From 4d9209089c91a91143ce8582bb074f2ac4972bd2 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 17:02:55 +1000 Subject: [PATCH 001/123] feat!: remove EQL v2-only published packages (protect, schema, protect-dynamodb) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR 1 of the EQL v2 final removal (#707). Delete the closed v2-only dependency chain — @cipherstash/protect-dynamodb → @cipherstash/protect → @cipherstash/schema — and every reference to it. Nothing outside the three imported them (@cipherstash/stack depends only on the separate @cipherstash/protect-ffi). They are superseded by @cipherstash/stack: - @cipherstash/protect -> @cipherstash/stack - @cipherstash/schema -> @cipherstash/stack/schema - @cipherstash/protect-dynamodb -> @cipherstash/stack/dynamodb (encryptedDynamoDB) Existing EQL v2 ciphertext stays decryptable through @cipherstash/stack; this removes the v2 authoring/emission surface, not the read path. Reference cleanup (dangling refs that would break build/CI): - e2e/package.json @cipherstash/protect dep edge - root package.json build:js turbo filter - tests.yml protect/protect-dynamodb .env steps (would fail `touch` on gone dirs) and the bun-job test loop - rebuild-docs.yml trigger tag (@cipherstash/protect@* -> @cipherstash/stack@*) - integration-{drizzle,prisma-next,supabase}.yml packages/schema/** path filters - lint-no-hardcoded-runners allowlist entry - e2e package-managers BIN fixture (dead) + two stale source comments Changeset / RC housekeeping: - delete schema-stevec-standard-pin.md (only target was the deleted schema) - prune the three from pre.json initialVersions - add deletion-notice changeset on @cipherstash/stack + @cipherstash/nextjs Meta honesty: SECURITY.md package list, AGENTS.md Repository Layout, nextjs package description. --- .changeset/pre.json | 4 - .changeset/remove-eql-v2-packages.md | 22 + .changeset/schema-stevec-standard-pin.md | 8 - .github/workflows/integration-drizzle.yml | 2 - .github/workflows/integration-prisma-next.yml | 2 - .github/workflows/integration-supabase.yml | 2 - .github/workflows/rebuild-docs.yml | 2 +- .github/workflows/tests.yml | 36 +- AGENTS.md | 11 +- SECURITY.md | 3 - e2e/package.json | 1 - e2e/tests/package-managers.e2e.test.ts | 7 +- package.json | 2 +- packages/nextjs/package.json | 2 +- packages/protect-dynamodb/.npmignore | 5 - packages/protect-dynamodb/CHANGELOG.md | 213 -- packages/protect-dynamodb/README.md | 358 --- .../protect-dynamodb/__tests__/audit.test.ts | 316 --- .../__tests__/dynamodb.test.ts | 331 --- .../__tests__/error-codes.test.ts | 125 - .../__tests__/helpers.test.ts | 89 - packages/protect-dynamodb/package.json | 54 - packages/protect-dynamodb/src/helpers.ts | 237 -- packages/protect-dynamodb/src/index.ts | 77 - .../src/operations/base-operation.ts | 64 - .../src/operations/bulk-decrypt-models.ts | 67 - .../src/operations/bulk-encrypt-models.ts | 67 - .../src/operations/decrypt-model.ts | 66 - .../src/operations/encrypt-model.ts | 62 - .../src/operations/search-terms.ts | 77 - packages/protect-dynamodb/src/types.ts | 66 - packages/protect-dynamodb/tsconfig.json | 28 - packages/protect-dynamodb/tsup.config.ts | 8 - packages/protect/.npmignore | 5 - packages/protect/CHANGELOG.md | 439 --- packages/protect/README.md | 1102 -------- packages/protect/__tests__/audit.test.ts | 472 ---- .../protect/__tests__/basic-protect.test.ts | 44 - .../protect/__tests__/bulk-protect.test.ts | 597 ---- .../__tests__/deprecated/search-terms.test.ts | 140 - .../encrypt-query-searchable-json.test.ts | 1061 -------- .../__tests__/encrypt-query-stevec.test.ts | 392 --- .../protect/__tests__/encrypt-query.test.ts | 946 ------- .../protect/__tests__/error-codes.test.ts | 369 --- packages/protect/__tests__/fixtures/index.ts | 145 - packages/protect/__tests__/helpers.test.ts | 274 -- .../__tests__/infer-index-type.test.ts | 94 - .../protect/__tests__/json-protect.test.ts | 1217 --------- .../protect/__tests__/jsonb-helpers.test.ts | 203 -- .../protect/__tests__/k-discriminator.test.ts | 55 - packages/protect/__tests__/keysets.test.ts | 97 - .../protect/__tests__/lock-context.test.ts | 208 -- .../protect/__tests__/nested-models.test.ts | 958 ------- .../protect/__tests__/number-protect.test.ts | 823 ------ .../protect/__tests__/protect-ops.test.ts | 843 ------ .../__tests__/searchable-json-pg.test.ts | 2390 ----------------- packages/protect/__tests__/supabase.test.ts | 343 --- packages/protect/package.json | 82 - packages/protect/src/bin/runner.ts | 35 - packages/protect/src/bin/stash.ts | 502 ---- packages/protect/src/client.ts | 18 - .../protect/src/ffi/helpers/error-code.ts | 12 - .../src/ffi/helpers/infer-index-type.ts | 120 - .../protect/src/ffi/helpers/type-guards.ts | 18 - .../protect/src/ffi/helpers/validation.ts | 94 - packages/protect/src/ffi/index.ts | 428 --- packages/protect/src/ffi/model-helpers.ts | 952 ------- .../src/ffi/operations/base-operation.ts | 50 - .../src/ffi/operations/batch-encrypt-query.ts | 235 -- .../src/ffi/operations/bulk-decrypt-models.ts | 112 - .../src/ffi/operations/bulk-decrypt.ts | 178 -- .../src/ffi/operations/bulk-encrypt-models.ts | 131 - .../src/ffi/operations/bulk-encrypt.ts | 213 -- .../src/ffi/operations/decrypt-model.ts | 109 - .../protect/src/ffi/operations/decrypt.ts | 130 - .../ffi/operations/deprecated/search-terms.ts | 131 - .../src/ffi/operations/encrypt-model.ts | 128 - .../src/ffi/operations/encrypt-query.ts | 178 -- .../protect/src/ffi/operations/encrypt.ts | 158 -- packages/protect/src/helpers/index.ts | 207 -- packages/protect/src/helpers/jsonb.ts | 99 - packages/protect/src/identify/index.ts | 128 - packages/protect/src/index.ts | 180 -- packages/protect/src/stash/index.ts | 459 ---- packages/protect/src/types.ts | 241 -- packages/protect/tsconfig.json | 28 - packages/protect/tsup.config.ts | 30 - packages/schema/.npmignore | 5 - packages/schema/CHANGELOG.md | 104 - packages/schema/README.md | 307 --- packages/schema/__tests__/schema.test.ts | 146 - .../schema/__tests__/searchable-json.test.ts | 46 - packages/schema/package.json | 50 - packages/schema/src/index.ts | 378 --- packages/schema/tsconfig.json | 28 - packages/schema/tsup.config.ts | 8 - .../stack/src/encryption/helpers/index.ts | 7 +- packages/stack/src/wasm-inline.ts | 2 +- pnpm-lock.yaml | 161 -- scripts/lint-no-hardcoded-runners.mjs | 1 - 100 files changed, 35 insertions(+), 22225 deletions(-) create mode 100644 .changeset/remove-eql-v2-packages.md delete mode 100644 .changeset/schema-stevec-standard-pin.md delete mode 100644 packages/protect-dynamodb/.npmignore delete mode 100644 packages/protect-dynamodb/CHANGELOG.md delete mode 100644 packages/protect-dynamodb/README.md delete mode 100644 packages/protect-dynamodb/__tests__/audit.test.ts delete mode 100644 packages/protect-dynamodb/__tests__/dynamodb.test.ts delete mode 100644 packages/protect-dynamodb/__tests__/error-codes.test.ts delete mode 100644 packages/protect-dynamodb/__tests__/helpers.test.ts delete mode 100644 packages/protect-dynamodb/package.json delete mode 100644 packages/protect-dynamodb/src/helpers.ts delete mode 100644 packages/protect-dynamodb/src/index.ts delete mode 100644 packages/protect-dynamodb/src/operations/base-operation.ts delete mode 100644 packages/protect-dynamodb/src/operations/bulk-decrypt-models.ts delete mode 100644 packages/protect-dynamodb/src/operations/bulk-encrypt-models.ts delete mode 100644 packages/protect-dynamodb/src/operations/decrypt-model.ts delete mode 100644 packages/protect-dynamodb/src/operations/encrypt-model.ts delete mode 100644 packages/protect-dynamodb/src/operations/search-terms.ts delete mode 100644 packages/protect-dynamodb/src/types.ts delete mode 100644 packages/protect-dynamodb/tsconfig.json delete mode 100644 packages/protect-dynamodb/tsup.config.ts delete mode 100644 packages/protect/.npmignore delete mode 100644 packages/protect/CHANGELOG.md delete mode 100644 packages/protect/README.md delete mode 100644 packages/protect/__tests__/audit.test.ts delete mode 100644 packages/protect/__tests__/basic-protect.test.ts delete mode 100644 packages/protect/__tests__/bulk-protect.test.ts delete mode 100644 packages/protect/__tests__/deprecated/search-terms.test.ts delete mode 100644 packages/protect/__tests__/encrypt-query-searchable-json.test.ts delete mode 100644 packages/protect/__tests__/encrypt-query-stevec.test.ts delete mode 100644 packages/protect/__tests__/encrypt-query.test.ts delete mode 100644 packages/protect/__tests__/error-codes.test.ts delete mode 100644 packages/protect/__tests__/fixtures/index.ts delete mode 100644 packages/protect/__tests__/helpers.test.ts delete mode 100644 packages/protect/__tests__/infer-index-type.test.ts delete mode 100644 packages/protect/__tests__/json-protect.test.ts delete mode 100644 packages/protect/__tests__/jsonb-helpers.test.ts delete mode 100644 packages/protect/__tests__/k-discriminator.test.ts delete mode 100644 packages/protect/__tests__/keysets.test.ts delete mode 100644 packages/protect/__tests__/lock-context.test.ts delete mode 100644 packages/protect/__tests__/nested-models.test.ts delete mode 100644 packages/protect/__tests__/number-protect.test.ts delete mode 100644 packages/protect/__tests__/protect-ops.test.ts delete mode 100644 packages/protect/__tests__/searchable-json-pg.test.ts delete mode 100644 packages/protect/__tests__/supabase.test.ts delete mode 100644 packages/protect/package.json delete mode 100644 packages/protect/src/bin/runner.ts delete mode 100644 packages/protect/src/bin/stash.ts delete mode 100644 packages/protect/src/client.ts delete mode 100644 packages/protect/src/ffi/helpers/error-code.ts delete mode 100644 packages/protect/src/ffi/helpers/infer-index-type.ts delete mode 100644 packages/protect/src/ffi/helpers/type-guards.ts delete mode 100644 packages/protect/src/ffi/helpers/validation.ts delete mode 100644 packages/protect/src/ffi/index.ts delete mode 100644 packages/protect/src/ffi/model-helpers.ts delete mode 100644 packages/protect/src/ffi/operations/base-operation.ts delete mode 100644 packages/protect/src/ffi/operations/batch-encrypt-query.ts delete mode 100644 packages/protect/src/ffi/operations/bulk-decrypt-models.ts delete mode 100644 packages/protect/src/ffi/operations/bulk-decrypt.ts delete mode 100644 packages/protect/src/ffi/operations/bulk-encrypt-models.ts delete mode 100644 packages/protect/src/ffi/operations/bulk-encrypt.ts delete mode 100644 packages/protect/src/ffi/operations/decrypt-model.ts delete mode 100644 packages/protect/src/ffi/operations/decrypt.ts delete mode 100644 packages/protect/src/ffi/operations/deprecated/search-terms.ts delete mode 100644 packages/protect/src/ffi/operations/encrypt-model.ts delete mode 100644 packages/protect/src/ffi/operations/encrypt-query.ts delete mode 100644 packages/protect/src/ffi/operations/encrypt.ts delete mode 100644 packages/protect/src/helpers/index.ts delete mode 100644 packages/protect/src/helpers/jsonb.ts delete mode 100644 packages/protect/src/identify/index.ts delete mode 100644 packages/protect/src/index.ts delete mode 100644 packages/protect/src/stash/index.ts delete mode 100644 packages/protect/src/types.ts delete mode 100644 packages/protect/tsconfig.json delete mode 100644 packages/protect/tsup.config.ts delete mode 100644 packages/schema/.npmignore delete mode 100644 packages/schema/CHANGELOG.md delete mode 100644 packages/schema/README.md delete mode 100644 packages/schema/__tests__/schema.test.ts delete mode 100644 packages/schema/__tests__/searchable-json.test.ts delete mode 100644 packages/schema/package.json delete mode 100644 packages/schema/src/index.ts delete mode 100644 packages/schema/tsconfig.json delete mode 100644 packages/schema/tsup.config.ts diff --git a/.changeset/pre.json b/.changeset/pre.json index cdaa6fd7c..b874a8b82 100644 --- a/.changeset/pre.json +++ b/.changeset/pre.json @@ -11,9 +11,6 @@ "@cipherstash/migrate": "0.2.0", "@cipherstash/nextjs": "4.1.1", "@cipherstash/prisma-next": "0.3.2", - "@cipherstash/protect": "12.0.1", - "@cipherstash/protect-dynamodb": "12.0.1", - "@cipherstash/schema": "3.0.1", "@cipherstash/stack": "0.19.0", "@cipherstash/stack-drizzle": "0.0.0", "@cipherstash/stack-supabase": "0.0.0", @@ -70,7 +67,6 @@ "remove-legacy-drizzle-package", "remove-secrets-leftovers", "rename-db-install-to-eql-install", - "schema-stevec-standard-pin", "skills-eql-v3-accuracy", "skills-identity-docs-refresh", "stack-1-0-0-rc", diff --git a/.changeset/remove-eql-v2-packages.md b/.changeset/remove-eql-v2-packages.md new file mode 100644 index 000000000..3cc56bc0a --- /dev/null +++ b/.changeset/remove-eql-v2-packages.md @@ -0,0 +1,22 @@ +--- +'@cipherstash/stack': patch +'@cipherstash/nextjs': patch +--- + +Remove the EQL v2-only published packages `@cipherstash/protect`, +`@cipherstash/schema`, and `@cipherstash/protect-dynamodb` from the repository +and the release train. They formed a closed dependency chain +(`@cipherstash/protect-dynamodb` → `@cipherstash/protect` → `@cipherstash/schema`) +and are superseded by `@cipherstash/stack`: + +- `@cipherstash/protect` (core encryption) → `@cipherstash/stack`, which now + carries the encryption client directly. +- `@cipherstash/schema` (schema builders) → `@cipherstash/stack/schema`. +- `@cipherstash/protect-dynamodb` (standalone DynamoDB adapter) → + `@cipherstash/stack/dynamodb` (`encryptedDynamoDB`), the maintained + implementation. + +Already-published versions remain installable from npm; the git history +preserves the source for any emergency maintenance. Existing EQL v2 ciphertext +stays decryptable through `@cipherstash/stack` — this removes the v2 authoring +and emission surface, not the read path. diff --git a/.changeset/schema-stevec-standard-pin.md b/.changeset/schema-stevec-standard-pin.md deleted file mode 100644 index d96bf3ca6..000000000 --- a/.changeset/schema-stevec-standard-pin.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -'@cipherstash/schema': patch ---- - -`searchableJson()` now pins the SteVec encoding mode to `standard` explicitly. -protect-ffi 0.29 flipped the library default to `compat` (the EQL v3 -encoding); pinning keeps the v2 wire format byte-stable so existing encrypted -JSON columns stay queryable and comparable. diff --git a/.github/workflows/integration-drizzle.yml b/.github/workflows/integration-drizzle.yml index ef8a42adc..7dd8310ba 100644 --- a/.github/workflows/integration-drizzle.yml +++ b/.github/workflows/integration-drizzle.yml @@ -23,7 +23,6 @@ on: # suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/stack/integration/**' # The WASM family suite (integration/wasm/**) exercises this entry: - 'packages/stack/src/wasm-inline.ts' @@ -46,7 +45,6 @@ on: # suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/stack/integration/**' # The WASM family suite (integration/wasm/**) exercises this entry: - 'packages/stack/src/wasm-inline.ts' diff --git a/.github/workflows/integration-prisma-next.yml b/.github/workflows/integration-prisma-next.yml index f3b128e01..acfb99b64 100644 --- a/.github/workflows/integration-prisma-next.yml +++ b/.github/workflows/integration-prisma-next.yml @@ -25,7 +25,6 @@ on: # suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' - 'local/docker-compose.postgres.yml' @@ -45,7 +44,6 @@ on: # suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' - 'local/docker-compose.postgres.yml' diff --git a/.github/workflows/integration-supabase.yml b/.github/workflows/integration-supabase.yml index 7ab83ca46..98b281655 100644 --- a/.github/workflows/integration-supabase.yml +++ b/.github/workflows/integration-supabase.yml @@ -19,7 +19,6 @@ on: # trigger the live suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/stack/integration/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' @@ -38,7 +37,6 @@ on: # trigger the live suite that would catch it. - 'packages/stack/src/encryption/**' - 'packages/stack/src/schema/**' - - 'packages/schema/**' - 'packages/stack/integration/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' diff --git a/.github/workflows/rebuild-docs.yml b/.github/workflows/rebuild-docs.yml index 80b320950..858732977 100644 --- a/.github/workflows/rebuild-docs.yml +++ b/.github/workflows/rebuild-docs.yml @@ -3,7 +3,7 @@ name: Rebuild Docs on: push: tags: - - '@cipherstash/protect@*' + - '@cipherstash/stack@*' jobs: trigger-docs-rebuild: diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index f13e60086..bf0c851f5 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -142,15 +142,6 @@ jobs: - name: Test — lint script self-tests run: pnpm run test:scripts - - name: Create .env file in ./packages/protect/ - run: | - touch ./packages/protect/.env - echo "CS_WORKSPACE_CRN=${{ vars.CS_WORKSPACE_CRN }}" >> ./packages/protect/.env - echo "CS_CLIENT_ID=${{ vars.CS_CLIENT_ID }}" >> ./packages/protect/.env - echo "CS_CLIENT_KEY=${{ secrets.CS_CLIENT_KEY }}" >> ./packages/protect/.env - echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/protect/.env - echo "DATABASE_URL=postgres://cipherstash:password@localhost:5432/cipherstash" >> ./packages/protect/.env - - name: Create .env file in ./packages/stack/ run: | touch ./packages/stack/.env @@ -160,14 +151,6 @@ jobs: echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/stack/.env echo "DATABASE_URL=postgres://cipherstash:password@localhost:5432/cipherstash" >> ./packages/stack/.env - - name: Create .env file in ./packages/protect-dynamodb/ - run: | - touch ./packages/protect-dynamodb/.env - echo "CS_WORKSPACE_CRN=${{ vars.CS_WORKSPACE_CRN }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_ID=${{ vars.CS_CLIENT_ID }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_KEY=${{ secrets.CS_CLIENT_KEY }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/protect-dynamodb/.env - # Run TurboRepo tests - name: Run tests run: pnpm run test @@ -353,15 +336,6 @@ jobs: - name: Install dependencies run: pnpm install --frozen-lockfile - - name: Create .env file in ./packages/protect/ - run: | - touch ./packages/protect/.env - echo "CS_WORKSPACE_CRN=${{ vars.CS_WORKSPACE_CRN }}" >> ./packages/protect/.env - echo "CS_CLIENT_ID=${{ vars.CS_CLIENT_ID }}" >> ./packages/protect/.env - echo "CS_CLIENT_KEY=${{ secrets.CS_CLIENT_KEY }}" >> ./packages/protect/.env - echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/protect/.env - echo "DATABASE_URL=postgres://cipherstash:password@localhost:5432/cipherstash" >> ./packages/protect/.env - - name: Create .env file in ./packages/stack/ run: | touch ./packages/stack/.env @@ -371,21 +345,13 @@ jobs: echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/stack/.env echo "DATABASE_URL=postgres://cipherstash:password@localhost:5432/cipherstash" >> ./packages/stack/.env - - name: Create .env file in ./packages/protect-dynamodb/ - run: | - touch ./packages/protect-dynamodb/.env - echo "CS_WORKSPACE_CRN=${{ vars.CS_WORKSPACE_CRN }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_ID=${{ vars.CS_CLIENT_ID }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_KEY=${{ secrets.CS_CLIENT_KEY }}" >> ./packages/protect-dynamodb/.env - echo "CS_CLIENT_ACCESS_KEY=${{ secrets.CS_CLIENT_ACCESS_KEY }}" >> ./packages/protect-dynamodb/.env - # Build with Node (turbo/tsup need Node), then run tests with Bun - name: Build packages run: pnpm turbo build --filter './packages/*' - name: Run tests with Bun run: | - for dir in packages/schema packages/protect packages/stack packages/protect-dynamodb packages/stack-forge; do + for dir in packages/stack packages/stack-forge; do if [ -f "$dir/vitest.config.ts" ] || [ -f "$dir/package.json" ]; then echo "--- Testing $dir ---" (cd "$dir" && bunx --bun vitest run) || true diff --git a/AGENTS.md b/AGENTS.md index 8c0566c11..6da0a2859 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -73,20 +73,13 @@ If these variables are missing, tests that require live encryption will fail or - `packages/stack`: Main package (`@cipherstash/stack`) containing the encryption client and all integrations - Subpath exports: `@cipherstash/stack`, `@cipherstash/stack/client`, `@cipherstash/stack/identity`, `@cipherstash/stack/schema`, `@cipherstash/stack/eql/v3`, `@cipherstash/stack/v3`, `@cipherstash/stack/types`, `@cipherstash/stack/dynamodb`, `@cipherstash/stack/encryption`, `@cipherstash/stack/errors`, `@cipherstash/stack/adapter-kit`, `@cipherstash/stack/wasm-inline` (the Drizzle and Supabase integrations moved to their own packages — see below) -- `packages/protect`: Core encryption library (internal, re-exported via `@cipherstash/stack`) - - `src/index.ts`: Public API (`Encryption`, exports) - - `src/ffi/index.ts`: `EncryptionClient` implementation, bridges to `@cipherstash/protect-ffi` - - `src/ffi/operations/*`: Encrypt/decrypt/model/bulk/query operations (thenable pattern with optional `.withLockContext()`) - - `__tests__/*`: End-to-end and API contract tests (Vitest) - `packages/cli`: The `stash` CLI — auth, init, encryption schema, and database setup (`stash eql install`). Has its own `AGENTS.md`. - `packages/wizard`: AI-powered encryption setup (`@cipherstash/wizard`) - `packages/migrate`: Plaintext-to-encrypted column migration (`@cipherstash/migrate`) — resumable backfill, per-column state - `packages/prisma-next`: Prisma Next integration (`@cipherstash/prisma-next`) — searchable field-level encryption for Postgres. **EQL v3 only**: per-domain constructors (`cipherstash.TextSearch()` / `text()` / `bigIntOrd()` / …) and `cipherstashFromStack` (the `./v3` and `./stack` entries). The EQL v2 surface was removed — the adapter's baseline migration installs the EQL v3 bundle only (works on Supabase as a non-superuser) - `packages/stack-drizzle`: Drizzle ORM integration (`@cipherstash/stack-drizzle`), depends on `@cipherstash/stack` — EQL v2 (`.`) and EQL v3 (`./v3`). Split out of `@cipherstash/stack`. - `packages/stack-supabase`: Supabase integration (`@cipherstash/stack-supabase`), depends on `@cipherstash/stack` — `encryptedSupabase` (v2) and `encryptedSupabaseV3` (v3). Split out of `@cipherstash/stack`. -- `packages/schema`: Schema builder utilities and types (`encryptedTable`, `encryptedColumn`, `encryptedField`) - `packages/nextjs`: Next.js helpers and Clerk integration (`./clerk` export) -- `packages/protect-dynamodb`: DynamoDB helpers (`protectDynamoDB`), built on `@cipherstash/protect`. A fork of the shipping adapter, kept for existing consumers of the standalone package; EQL v2 only. The maintained implementation is `packages/stack/src/dynamodb` (`encryptedDynamoDB`) - `packages/utils`: Shared config (`utils/config`) and logger (`utils/logger`) - `packages/bench`: Performance / index-engagement benchmarks (private, not published) - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) @@ -225,11 +218,11 @@ pnpm changeset:publish ## Adding Features Safely (LLM checklist) 1. Identify the target package(s) in `packages/*` and confirm whether changes affect public APIs or payload shapes. -2. If modifying `packages/protect` operations or `EncryptionClient`, ensure: +2. If modifying `packages/stack` encryption operations or `EncryptionClient`, ensure: - The Result contract and error type strings remain stable. - `.withLockContext()` remains available for affected operations. - ESM/CJS exports continue to work (don't break `require`). -3. If changing schema behavior (`packages/schema`), update type definitions and ensure validation still works in `EncryptionClient.init`. +3. If changing schema behavior (`packages/stack` schema builders, `@cipherstash/stack/schema`), update type definitions and ensure validation still works in `EncryptionClient.init`. 4. Add/extend tests in the same package. For features that require live credentials, guard with env checks or provide mock-friendly paths. 5. Run: - `pnpm run code:fix` diff --git a/SECURITY.md b/SECURITY.md index 85fd4e22b..47b725910 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -11,10 +11,7 @@ This repository is the CipherStash Stack monorepo for JavaScript/TypeScript. It | ------- | ----------- | | `@cipherstash/stack` | Main package: encryption client, schema, EQL v3 typed client | | `stash` | CipherStash CLI | -| `@cipherstash/protect` | Core encryption library (re-exported via `@cipherstash/stack`) | -| `@cipherstash/schema` | Schema builder utilities | | `@cipherstash/nextjs` | Next.js helpers | -| `@cipherstash/protect-dynamodb` | DynamoDB helpers | | `@cipherstash/migrate` | Plaintext-to-encrypted column migration tooling | | `@cipherstash/prisma-next` | Prisma Next integration (searchable field-level encryption for Postgres) | | `@cipherstash/stack-drizzle` | Drizzle ORM integration for `@cipherstash/stack` (EQL v2 + v3) | diff --git a/e2e/package.json b/e2e/package.json index 7fd7205eb..06cf30efb 100644 --- a/e2e/package.json +++ b/e2e/package.json @@ -9,7 +9,6 @@ }, "dependencies": { "stash": "workspace:*", - "@cipherstash/protect": "workspace:*", "@cipherstash/stack": "workspace:*", "@cipherstash/wizard": "workspace:*" }, diff --git a/e2e/tests/package-managers.e2e.test.ts b/e2e/tests/package-managers.e2e.test.ts index cf5e0951f..bfe42de02 100644 --- a/e2e/tests/package-managers.e2e.test.ts +++ b/e2e/tests/package-managers.e2e.test.ts @@ -25,10 +25,9 @@ const RUNNER: Record = { const BIN = { cli: resolve(REPO_ROOT, 'packages/cli/dist/bin/stash.js'), wizard: resolve(REPO_ROOT, 'packages/wizard/dist/bin/wizard.js'), - protect: resolve(REPO_ROOT, 'packages/protect/dist/bin/stash.js'), - // The legacy @cipherstash/drizzle `generate-eql-migration` bin is gone with - // the package (protect sunsets at 1.0; @cipherstash/stack-drizzle is the - // successor). + // The legacy @cipherstash/protect `stash` bin and the @cipherstash/drizzle + // `generate-eql-migration` bin are both gone with their packages + // (@cipherstash/stack and @cipherstash/stack-drizzle are the successors). } as const const UA: Record = { diff --git a/package.json b/package.json index 75cbff39a..c0e4cd9b2 100644 --- a/package.json +++ b/package.json @@ -21,7 +21,7 @@ "license": "MIT", "scripts": { "build": "turbo build --filter './packages/*'", - "build:js": "turbo build --filter './packages/protect' --filter './packages/nextjs'", + "build:js": "turbo build --filter './packages/nextjs'", "changeset": "changeset", "changeset:version": "changeset version", "changeset:publish": "changeset publish", diff --git a/packages/nextjs/package.json b/packages/nextjs/package.json index c318db4c1..2d7deebad 100644 --- a/packages/nextjs/package.json +++ b/packages/nextjs/package.json @@ -1,7 +1,7 @@ { "name": "@cipherstash/nextjs", "version": "4.1.1", - "description": "Nextjs package for use with @cipherstash/protect", + "description": "Nextjs package for use with @cipherstash/stack", "keywords": [ "encrypted", "typescript", diff --git a/packages/protect-dynamodb/.npmignore b/packages/protect-dynamodb/.npmignore deleted file mode 100644 index 3490e24db..000000000 --- a/packages/protect-dynamodb/.npmignore +++ /dev/null @@ -1,5 +0,0 @@ -.env -.turbo -node_modules -cipherstash.secret.toml -cipherstash.toml \ No newline at end of file diff --git a/packages/protect-dynamodb/CHANGELOG.md b/packages/protect-dynamodb/CHANGELOG.md deleted file mode 100644 index 6be07a477..000000000 --- a/packages/protect-dynamodb/CHANGELOG.md +++ /dev/null @@ -1,213 +0,0 @@ -# @cipherstash/protect-dynamodb - -## 12.0.2-rc.0 - -### Patch Changes - -- @cipherstash/protect@12.0.2-rc.0 - -## 12.0.1 - -### Patch Changes - -- aa9c4b1: Documentation: refresh package READMEs after the protectjs → stack repository rename. Fixed repository and license links, replaced dead in-repo docs links with cipherstash.com/docs URLs, rewrote the incorrect @cipherstash/nextjs README, and added guidance pointing new projects to @cipherstash/stack. -- Updated dependencies [aa9c4b1] - - @cipherstash/protect@12.0.1 - -## 12.0.0 - -### Patch Changes - -- f743fcc: Upgrade `@cipherstash/protect-ffi` to `0.23.0` and the bundled CipherStash EQL extension to `eql-2.3.1`. - - Breaking upstream changes adopted in this release: - - - **Encrypt-config schema version**: `buildEncryptConfig` now emits `{ v: 1, ... }` (was `{ v: 2, ... }`). protect-ffi `0.22.0` started validating this field and rejects any value other than `1` with the new `UNSUPPORTED_CONFIG_VERSION` error code. - - **Storage and query payloads are now distinct types** (protect-ffi `0.23.0`): the previously-conflated `Encrypted` type splits into `Encrypted` (storage-only, `c` required) and a new `EncryptedQuery` (search terms — scalar `unique`/`match`/`ore` lookups and `ste_vec_selector` JSON path queries; no `c`). JSON containment queries (`ste_vec_term`) still return a storage-shaped `Encrypted` payload. `encryptQuery` / `encryptQueryBulk` now return `Encrypted | EncryptedQuery`, and the stack's `EncryptedSearchTerm` / `EncryptedQueryResult` unions widen to match. `decrypt` rejects query payloads at the type level. The DynamoDB `SearchTermsOperation` narrows via `'hm' in term` rather than `term.hm`. - - **SteVec encoding default flipped**: protect-ffi's default `mode` for `ste_vec` indexes changed from `compat` to `standard`. The two encodings are not cross-compatible. Existing JSON-searchable data that was indexed under `compat` will need to be re-encrypted to be queryable. The stack adopts the new `standard` default — there is no longer a way to pin `compat` from the SDK. - - **EQL extension bumped to `eql-2.3.1`**: the new SteVec `standard` encoding requires matching support in the database EQL extension. The CLI's bundled SQL (`packages/cli/src/sql/*.sql`) and the `@cipherstash/prisma-next` install bundle (`migrations/20260601T0000_install_eql_bundle/ops.json` + `eql-install.generated.ts`) are updated to `eql-2.3.1`. Databases installed with an older EQL extension must be reinstalled (`stash db install`) before containment / contained-by queries against SteVec columns will work. `eql-2.3.1` ships the `_encrypted_check_c` fix for SteVec storage payloads ([cipherstash/encrypt-query-language#232](https://github.com/cipherstash/encrypt-query-language/issues/232)). - - **New error codes**: `ProtectErrorCode` (re-exported from `@cipherstash/protect-ffi`) gains `MATCH_REQUIRES_TEXT` and `UNSUPPORTED_CONFIG_VERSION`. Exhaustive switches over `ProtectErrorCode` will need additional cases. - - **`match` index validation**: protect-ffi now rejects `match` indexes on columns whose `cast_as` is not text-family (`'text'` / `'string'`) with `MATCH_REQUIRES_TEXT`. The stack's `freeTextSearch()` builder is unaffected because it only targets string-typed columns. - - **`Encrypted` ciphertext shape**: protect-ffi's `Encrypted` type is now a discriminated union keyed on `k` (`'ct'` for scalars, `'sv'` for SteVec). SteVec storage payloads now place the root document ciphertext at `sv[0].c`. The stack's `isEncryptedPayload` runtime check continues to work because storage payloads still carry `c` (scalar) or `sv` (SteVec). The DynamoDB helpers (`toEncryptedDynamoItem`, `SearchTermsOperation`) now narrow on `k` before reading variant-only fields. - - **Config-validation error message wording**: error messages for config-validation failures now come from upstream `ConfigError`. `ProtectError.code` values are preserved; consumers that string-match on `err.message` for config-validation errors must update. - -- Updated dependencies [f743fcc] - - @cipherstash/protect@12.0.0 - -## 11.0.2 - -### Patch Changes - -- Updated dependencies [a8dbb65] - - @cipherstash/protect@11.1.2 - -## 11.0.1 - -### Patch Changes - -- Updated dependencies [afe6810] - - @cipherstash/protect@11.1.1 - -## 11.0.0 - -### Patch Changes - -- Updated dependencies [068f820] - - @cipherstash/protect@11.1.0 - -## 10.0.0 - -### Patch Changes - -- Updated dependencies [b0e56b8] - - @cipherstash/protect@10.6.0 - -## 9.0.0 - -### Patch Changes - -- Updated dependencies [db72e2c] -- Updated dependencies [e769740] - - @cipherstash/protect@10.5.0 - -## 8.0.0 - -### Patch Changes - -- Updated dependencies [9ccaf68] - - @cipherstash/protect@10.4.0 - -## 7.0.0 - -### Patch Changes - -- Updated dependencies [a1fce2b] -- Updated dependencies [622b684] - - @cipherstash/protect@10.3.0 - -## 6.0.1 - -### Patch Changes - -- @cipherstash/protect@10.2.1 - -## 6.0.0 - -### Patch Changes - -- Updated dependencies [de029de] - - @cipherstash/protect@10.2.0 - -## 5.1.1 - -### Patch Changes - -- Updated dependencies [ff4421f] - - @cipherstash/protect@10.1.1 - -## 5.1.0 - -### Patch Changes - -- Updated dependencies [6b87c17] - - @cipherstash/protect@10.1.0 - -## 5.0.2 - -### Patch Changes - -- @cipherstash/protect@10.0.2 - -## 5.0.1 - -### Patch Changes - -- @cipherstash/protect@10.0.1 - -## 5.0.0 - -### Major Changes - -- 788dbfc: Added JSON and INT data type support and update FFI to v0.17.1 with x86_64 musl environment platform support. - - - Update @cipherstash/protect-ffi from 0.16.0 to 0.17.1 with support for x86_64 musl platforms. - - Add searchableJson() method to schema for JSON field indexing (the search operations still don't work but this interface exists) - - Refactor type system: EncryptedPayload → Encrypted, add JsPlaintext - - Add comprehensive test suites for JSON, integer, and basic encryption - - Update encryption format to use 'k' property for searchable JSON - - Remove deprecated search terms tests for JSON fields - - Simplify schema data types to text, int, json only - - Update model helpers to handle new encryption format - - Fix type safety issues in bulk operations and model encryption - -### Patch Changes - -- Updated dependencies [788dbfc] - - @cipherstash/protect@10.0.0 - -## 4.0.0 - -### Patch Changes - -- Updated dependencies [c7ed7ab] -- Updated dependencies [211e979] - - @cipherstash/protect@9.6.0 - -## 3.0.0 - -### Minor Changes - -- 6f45b02: Fully implemented audit metadata functionality. - -### Patch Changes - -- Updated dependencies [6f45b02] - - @cipherstash/protect@9.5.0 - -## 2.0.1 - -### Patch Changes - -- @cipherstash/protect@9.4.1 - -## 2.0.0 - -### Patch Changes - -- Updated dependencies [1cc4772] - - @cipherstash/protect@9.4.0 - -## 1.0.0 - -### Minor Changes - -- 01fed9e: Added audit support for all protect and protect-dynamodb interfaces. - -### Patch Changes - -- Updated dependencies [01fed9e] - - @cipherstash/protect@9.3.0 - -## 0.3.0 - -### Minor Changes - -- 2b63ee1: Support nested protect schema in dynamodb helper functions. -- e33fbaf: Fixed bug when handling schema definitions without an equality flag. - -## 0.2.0 - -### Minor Changes - -- 5fc0150: Fix build and publish. - -## 1.0.0 - -### Minor Changes - -- c8468ee: Released initial version of the DynamoDB helper interface. - -### Patch Changes - -- Updated dependencies [c8468ee] - - @cipherstash/protect@9.1.0 diff --git a/packages/protect-dynamodb/README.md b/packages/protect-dynamodb/README.md deleted file mode 100644 index 1489f336a..000000000 --- a/packages/protect-dynamodb/README.md +++ /dev/null @@ -1,358 +0,0 @@ -# CipherStash DynamoDB Helpers - -Helpers for using [CipherStash encryption](https://github.com/cipherstash/stack) with DynamoDB. - -> [!TIP] -> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which provides this functionality as `encryptedDynamoDB` from `@cipherstash/stack/dynamodb`. See the [DynamoDB docs](https://cipherstash.com/docs/stack/cipherstash/encryption/dynamodb). This package documents the legacy `@cipherstash/protect` API. - -[![Built by CipherStash](https://raw.githubusercontent.com/cipherstash/meta/refs/heads/main/csbadge.svg)](https://cipherstash.com) -[![NPM version](https://img.shields.io/npm/v/@cipherstash/protect-dynamodb.svg?style=for-the-badge&labelColor=000000)](https://www.npmjs.com/package/@cipherstash/protect-dynamodb) -[![License](https://img.shields.io/npm/l/@cipherstash/protect-dynamodb.svg?style=for-the-badge&labelColor=000000)](https://github.com/cipherstash/stack/blob/main/LICENSE.md) - -## Installation - -```bash -npm install @cipherstash/protect-dynamodb -# or -yarn add @cipherstash/protect-dynamodb -# or -pnpm add @cipherstash/protect-dynamodb -``` - -## Quick Start - -```typescript -import { protectDynamoDB } from '@cipherstash/protect-dynamodb' -import { protect, csColumn, csTable } from '@cipherstash/protect' -import { PutCommand, GetCommand } from '@aws-sdk/lib-dynamodb' - -// Define your protected table schema -const users = csTable('users', { - email: csColumn('email').equality(), -}) - -// Initialize the Protect client -const protectClient = await protect({ - schemas: [users], -}) - -// Create the DynamoDB helper instance -const protectDynamo = protectDynamoDB({ - protectClient, -}) - -// Encrypt and store a user -const user = { - email: 'user@example.com', -} - -const encryptResult = await protectDynamo.encryptModel(user, users) -if (encryptResult.failure) { - throw new Error(`Failed to encrypt user: ${encryptResult.failure.message}`) -} - -// Store in DynamoDB -await docClient.send(new PutCommand({ - TableName: 'Users', - Item: encryptResult.data, -})) - -// Create search terms for querying -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} - -// Query using the search term -const [emailHmac] = searchTermsResult.data -const result = await docClient.send(new GetCommand({ - TableName: 'Users', - Key: { email__hmac: emailHmac }, -})) - -if (!result.Item) { - throw new Error('Item not found') -} - -// Decrypt the result -const decryptResult = await protectDynamo.decryptModel( - result.Item, - users, -) - -if (decryptResult.failure) { - throw new Error(`Failed to decrypt user: ${decryptResult.failure.message}`) -} - -const decryptedUser = decryptResult.data -``` - -## Features - -### Encryption and Decryption - -The package provides methods to encrypt and decrypt data for DynamoDB: - -- `encryptModel`: Encrypts a single model -- `bulkEncryptModels`: Encrypts multiple models in bulk -- `decryptModel`: Decrypts a single model -- `bulkDecryptModels`: Decrypts multiple models in bulk - -All methods return a `Result` type that must be checked for failures: - -```typescript -const result = await protectDynamo.encryptModel(user, users) -if (result.failure) { - // Handle error - console.error(result.failure.message) -} else { - // Use encrypted data - const encryptedData = result.data -} -``` - -### Search Terms - -Create search terms for querying encrypted data: - -- `createSearchTerms`: Creates search terms for one or more columns - -```typescript -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} - -const [emailHmac] = searchTermsResult.data -``` - -### DynamoDB Integration - -The package automatically handles: -- Converting encrypted data to DynamoDB's format -- Adding HMAC attributes for searchable fields -- Preserving unencrypted fields -- Converting DynamoDB items back to encrypted format for decryption - -## Usage Patterns - -### Simple Table with Encrypted Fields - -```typescript -const users = csTable('users', { - email: csColumn('email').equality(), -}) - -// Encrypt and store -const encryptResult = await protectDynamo.encryptModel({ - pk: 'user#1', - email: 'user@example.com', -}, users) - -if (encryptResult.failure) { - throw new Error(`Failed to encrypt user: ${encryptResult.failure.message}`) -} - -// Query using search terms -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} -``` - -### Encrypted Partition Key - -```typescript -// Table with encrypted partition key -const table = { - TableName: 'Users', - AttributeDefinitions: [ - { - AttributeName: 'email__hmac', - AttributeType: 'S', - }, - ], - KeySchema: [ - { - AttributeName: 'email__hmac', - KeyType: 'HASH', - }, - ], -} - -// Create search terms for querying -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} - -const [emailHmac] = searchTermsResult.data -``` - -### Encrypted Sort Key - -```typescript -// Table with encrypted sort key -const table = { - TableName: 'Users', - AttributeDefinitions: [ - { - AttributeName: 'pk', - AttributeType: 'S', - }, - { - AttributeName: 'email__hmac', - AttributeType: 'S', - }, - ], - KeySchema: [ - { - AttributeName: 'pk', - KeyType: 'HASH', - }, - { - AttributeName: 'email__hmac', - KeyType: 'RANGE', - }, - ], -} - -// Create search terms for querying -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} - -const [emailHmac] = searchTermsResult.data -``` - -### Global Secondary Index with Encrypted Key - -```typescript -// Table with GSI using encrypted key -const table = { - TableName: 'Users', - AttributeDefinitions: [ - { - AttributeName: 'pk', - AttributeType: 'S', - }, - { - AttributeName: 'email__hmac', - AttributeType: 'S', - }, - ], - KeySchema: [ - { - AttributeName: 'pk', - KeyType: 'HASH', - }, - ], - GlobalSecondaryIndexes: [ - { - IndexName: 'EmailIndex', - KeySchema: [ - { - AttributeName: 'email__hmac', - KeyType: 'HASH', - }, - ], - Projection: { - ProjectionType: 'INCLUDE', - NonKeyAttributes: ['email__source'], - }, - }, - ], -} - -// Create search terms for querying -const searchTermsResult = await protectDynamo.createSearchTerms([ - { - value: 'user@example.com', - column: users.email, - table: users, - }, -]) - -if (searchTermsResult.failure) { - throw new Error(`Failed to create search terms: ${searchTermsResult.failure.message}`) -} - -const [emailHmac] = searchTermsResult.data -``` - -## Error Handling - -All methods return a `Result` type from `@byteslice/result` that must be checked for failures: - -```typescript -const result = await protectDynamo.encryptModel(user, users) - -if (result.failure) { - // Handle error - console.error(result.failure.message) -} else { - // Use encrypted data - const encryptedData = result.data -} -``` - -## Type Safety - -The package is fully typed and supports TypeScript: - -```typescript -type User = { - pk: string - email: string -} - -// Type-safe encryption -const encryptResult = await protectDynamo.encryptModel(user, users) -if (encryptResult.failure) { - throw new Error(`Failed to encrypt user: ${encryptResult.failure.message}`) -} -const encryptedUser = encryptResult.data - -// Type-safe decryption -const decryptResult = await protectDynamo.decryptModel(item, users) -if (decryptResult.failure) { - throw new Error(`Failed to decrypt user: ${decryptResult.failure.message}`) -} -const decryptedUser = decryptResult.data -``` diff --git a/packages/protect-dynamodb/__tests__/audit.test.ts b/packages/protect-dynamodb/__tests__/audit.test.ts deleted file mode 100644 index 619ff8754..000000000 --- a/packages/protect-dynamodb/__tests__/audit.test.ts +++ /dev/null @@ -1,316 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable, csValue, protect } from '@cipherstash/protect' -import { beforeAll, describe, expect, it } from 'vitest' -import { protectDynamoDB } from '../src' - -const schema = csTable('dynamo_cipherstash_test', { - email: csColumn('email').equality(), - firstName: csColumn('firstName').equality(), - lastName: csColumn('lastName').equality(), - phoneNumber: csColumn('phoneNumber'), - example: { - protected: csValue('example.protected'), - deep: { - protected: csValue('example.deep.protected'), - }, - }, -}) - -describe('protect dynamodb helpers', () => { - let protectClient: Awaited> - let protectDynamo: ReturnType - - beforeAll(async () => { - protectClient = await protect({ - schemas: [schema], - }) - - protectDynamo = protectDynamoDB({ - protectClient, - }) - }) - - it('should encrypt columns', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test.user@example.com', - address: '123 Main Street', - createdAt: '2024-08-15T22:14:49.948Z', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - companyName: 'Acme Corp', - batteryBrands: ['Brand1', 'Brand2'], - metadata: { role: 'admin' }, - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - }, - }, - } - - const result = await protectDynamo.encryptModel(testData, schema).audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_encrypt_model' }, - }) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify equality columns are encrypted - expect(encryptedData).toHaveProperty('email__source') - expect(encryptedData).toHaveProperty('email__hmac') - expect(encryptedData).toHaveProperty('firstName__source') - expect(encryptedData).toHaveProperty('firstName__hmac') - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - expect(encryptedData).toHaveProperty('phoneNumber__source') - expect(encryptedData).not.toHaveProperty('phoneNumber__hmac') - expect(encryptedData.example).toHaveProperty('protected__source') - expect(encryptedData.example.deep).toHaveProperty('protected__source') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.address).toBe('123 Main Street') - expect(encryptedData.createdAt).toBe('2024-08-15T22:14:49.948Z') - expect(encryptedData.companyName).toBe('Acme Corp') - expect(encryptedData.batteryBrands).toEqual(['Brand1', 'Brand2']) - expect(encryptedData.example.notProtected).toBe('I am not protected') - expect(encryptedData.example.deep.notProtected).toBe('deep not protected') - expect(encryptedData.metadata).toEqual({ role: 'admin' }) - }) - - it('should handle null and undefined values', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: null, - firstName: undefined, - lastName: 'Smith', - phoneNumber: null, - metadata: { role: null }, - example: { - protected: null, - notProtected: 'I am not protected', - deep: { - protected: undefined, - notProtected: 'deep not protected', - }, - }, - } - - const result = await protectDynamo.encryptModel(testData, schema).audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_encrypt_model' }, - }) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify null/undefined equality columns are handled - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.phoneNumber).toBeNull() - expect(encryptedData.email).toBeNull() - expect(encryptedData.firstName).toBeUndefined() - expect(encryptedData.metadata).toEqual({ role: null }) - expect(encryptedData.example.protected).toBeNull() - expect(encryptedData.example.deep.protected).toBeUndefined() - expect(encryptedData.example.deep.notProtected).toBe('deep not protected') - }) - - it('should handle empty strings and special characters', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: '', - firstName: 'John!@#$%^&*()', - lastName: 'Smith ', - phoneNumber: '', - metadata: { role: 'admin!@#$%^&*()' }, - } - - const result = await protectDynamo.encryptModel(testData, schema).audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_encrypt_model' }, - }) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify equality columns are encrypted - expect(encryptedData).toHaveProperty('email__source') - expect(encryptedData).toHaveProperty('email__hmac') - expect(encryptedData).toHaveProperty('firstName__source') - expect(encryptedData).toHaveProperty('firstName__hmac') - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - expect(encryptedData).toHaveProperty('phoneNumber__source') - expect(encryptedData).not.toHaveProperty('phoneNumber__hmac') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.metadata).toEqual({ role: 'admin!@#$%^&*()' }) - }) - - it('should handle bulk encryption', async () => { - const testData = [ - { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test1@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - }, - { - id: '02ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test2@example.com', - firstName: 'Jane', - lastName: 'Doe', - phoneNumber: '555-555-5556', - }, - ] - - const result = await protectDynamo - .bulkEncryptModels(testData, schema) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'dynamo_bulk_encrypt_models', - }, - }) - if (result.failure) { - throw new Error(`Bulk encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify both items are encrypted - expect(encryptedData).toHaveLength(2) - - // Verify first item - expect(encryptedData[0]).toHaveProperty('email__source') - expect(encryptedData[0]).toHaveProperty('email__hmac') - expect(encryptedData[0]).toHaveProperty('firstName__source') - expect(encryptedData[0]).toHaveProperty('firstName__hmac') - expect(encryptedData[0]).toHaveProperty('lastName__source') - expect(encryptedData[0]).toHaveProperty('lastName__hmac') - expect(encryptedData[0]).toHaveProperty('phoneNumber__source') - - // Verify second item - expect(encryptedData[1]).toHaveProperty('email__source') - expect(encryptedData[1]).toHaveProperty('email__hmac') - expect(encryptedData[1]).toHaveProperty('firstName__source') - expect(encryptedData[1]).toHaveProperty('firstName__hmac') - expect(encryptedData[1]).toHaveProperty('lastName__source') - expect(encryptedData[1]).toHaveProperty('lastName__hmac') - expect(encryptedData[1]).toHaveProperty('phoneNumber__source') - }) - - it('should handle decryption of encrypted data', async () => { - const originalData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test.user@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - }, - }, - } - - // First encrypt - const encryptResult = await protectDynamo - .encryptModel(originalData, schema) - .audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_encrypt_model' }, - }) - - if (encryptResult.failure) { - throw new Error(`Encryption failed: ${encryptResult.failure.message}`) - } - - // Then decrypt - const decryptResult = await protectDynamo - .decryptModel(encryptResult.data, schema) - .audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_decrypt_model' }, - }) - if (decryptResult.failure) { - throw new Error(`Decryption failed: ${decryptResult.failure.message}`) - } - - const decryptedData = decryptResult.data - - // Verify all fields match original data - expect(decryptedData).toEqual(originalData) - }) - - it('should handle decryption of bulk encrypted data', async () => { - const originalData = [ - { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test1@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - }, - { - id: '02ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test2@example.com', - firstName: 'Jane', - lastName: 'Doe', - phoneNumber: '555-555-5556', - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - }, - }, - }, - ] - - // First encrypt - const encryptResult = await protectDynamo - .bulkEncryptModels(originalData, schema) - .audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_bulk_encrypt_models' }, - }) - if (encryptResult.failure) { - throw new Error( - `Bulk encryption failed: ${encryptResult.failure.message}`, - ) - } - - // Then decrypt - const decryptResult = await protectDynamo - .bulkDecryptModels(encryptResult.data, schema) - .audit({ - metadata: { sub: 'cj@cjb.io', type: 'dynamo_bulk_decrypt_models' }, - }) - if (decryptResult.failure) { - throw new Error( - `Bulk decryption failed: ${decryptResult.failure.message}`, - ) - } - - const decryptedData = decryptResult.data - - // Verify all items match original data - expect(decryptedData).toEqual(originalData) - }) -}) diff --git a/packages/protect-dynamodb/__tests__/dynamodb.test.ts b/packages/protect-dynamodb/__tests__/dynamodb.test.ts deleted file mode 100644 index 0b8ee27af..000000000 --- a/packages/protect-dynamodb/__tests__/dynamodb.test.ts +++ /dev/null @@ -1,331 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable, csValue, protect } from '@cipherstash/protect' -import { beforeAll, describe, expect, it } from 'vitest' -import { protectDynamoDB } from '../src' - -const schema = csTable('dynamo_cipherstash_test', { - email: csColumn('email').equality(), - firstName: csColumn('firstName').equality(), - lastName: csColumn('lastName').equality(), - phoneNumber: csColumn('phoneNumber'), - json: csColumn('json').dataType('json'), - jsonSearchable: csColumn('jsonSearchable').dataType('json'), - //.searchableJson('users/jsonSearchable'), - example: { - protected: csValue('example.protected'), - deep: { - protected: csValue('example.deep.protected'), - protectNestedJson: csValue('example.deep.protectNestedJson').dataType( - 'json', - ), - }, - }, -}) - -describe('protect dynamodb helpers', () => { - let protectClient: Awaited> - let protectDynamo: ReturnType - - beforeAll(async () => { - protectClient = await protect({ - schemas: [schema], - }) - - protectDynamo = protectDynamoDB({ - protectClient, - }) - }) - - it('should encrypt and decrypt a model', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test.user@example.com', - address: '123 Main Street', - createdAt: '2024-08-15T22:14:49.948Z', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - json: { - name: 'John Doe', - age: 30, - preferences: { - theme: 'dark', - notifications: true, - }, - }, - jsonSearchable: { - name: 'John Doe', - age: 30, - preferences: { - theme: 'dark', - notifications: true, - }, - }, - companyName: 'Acme Corp', - batteryBrands: ['Brand1', 'Brand2'], - metadata: { role: 'admin' }, - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - protectNestedJson: { - hello: 'world', - }, - }, - }, - } - - const result = await protectDynamo.encryptModel(testData, schema) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify equality columns are encrypted - expect(encryptedData).toHaveProperty('email__source') - expect(encryptedData).toHaveProperty('email__hmac') - expect(encryptedData).toHaveProperty('firstName__source') - expect(encryptedData).toHaveProperty('firstName__hmac') - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - expect(encryptedData).toHaveProperty('phoneNumber__source') - expect(encryptedData).not.toHaveProperty('phoneNumber__hmac') - expect(encryptedData.example).toHaveProperty('protected__source') - expect(encryptedData.example.deep).toHaveProperty('protected__source') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.address).toBe('123 Main Street') - expect(encryptedData.createdAt).toBe('2024-08-15T22:14:49.948Z') - expect(encryptedData.companyName).toBe('Acme Corp') - expect(encryptedData.batteryBrands).toEqual(['Brand1', 'Brand2']) - expect(encryptedData.example.notProtected).toBe('I am not protected') - expect(encryptedData.example.deep.notProtected).toBe('deep not protected') - expect(encryptedData.metadata).toEqual({ role: 'admin' }) - - const decryptResult = await protectDynamo.decryptModel( - encryptedData, - schema, - ) - if (decryptResult.failure) { - throw new Error(`Decryption failed: ${decryptResult.failure.message}`) - } - - expect(decryptResult.data).toEqual(testData) - }) - - it('should handle null and undefined values', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: null, - firstName: undefined, - lastName: 'Smith', - phoneNumber: null, - metadata: { role: null }, - example: { - protected: null, - notProtected: 'I am not protected', - deep: { - protected: undefined, - notProtected: 'deep not protected', - }, - }, - } - - const result = await protectDynamo.encryptModel(testData, schema) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify null/undefined equality columns are handled - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.phoneNumber).toBeNull() - expect(encryptedData.email).toBeNull() - expect(encryptedData.firstName).toBeUndefined() - expect(encryptedData.metadata).toEqual({ role: null }) - expect(encryptedData.example.protected).toBeNull() - expect(encryptedData.example.deep.protected).toBeUndefined() - expect(encryptedData.example.deep.notProtected).toBe('deep not protected') - }) - - it('should handle empty strings and special characters', async () => { - const testData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: '', - firstName: 'John!@#$%^&*()', - lastName: 'Smith ', - phoneNumber: '', - metadata: { role: 'admin!@#$%^&*()' }, - } - - const result = await protectDynamo.encryptModel(testData, schema) - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify equality columns are encrypted - expect(encryptedData).toHaveProperty('email__source') - expect(encryptedData).toHaveProperty('email__hmac') - expect(encryptedData).toHaveProperty('firstName__source') - expect(encryptedData).toHaveProperty('firstName__hmac') - expect(encryptedData).toHaveProperty('lastName__source') - expect(encryptedData).toHaveProperty('lastName__hmac') - expect(encryptedData).toHaveProperty('phoneNumber__source') - expect(encryptedData).not.toHaveProperty('phoneNumber__hmac') - - // Verify other fields remain unchanged - expect(encryptedData.id).toBe('01ABCDEFGHIJKLMNOPQRSTUVWX') - expect(encryptedData.metadata).toEqual({ role: 'admin!@#$%^&*()' }) - }) - - it('should handle bulk encryption', async () => { - const testData = [ - { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test1@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - }, - { - id: '02ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test2@example.com', - firstName: 'Jane', - lastName: 'Doe', - phoneNumber: '555-555-5556', - }, - ] - - const result = await protectDynamo.bulkEncryptModels(testData, schema) - if (result.failure) { - throw new Error(`Bulk encryption failed: ${result.failure.message}`) - } - - const encryptedData = result.data - - // Verify both items are encrypted - expect(encryptedData).toHaveLength(2) - - // Verify first item - expect(encryptedData[0]).toHaveProperty('email__source') - expect(encryptedData[0]).toHaveProperty('email__hmac') - expect(encryptedData[0]).toHaveProperty('firstName__source') - expect(encryptedData[0]).toHaveProperty('firstName__hmac') - expect(encryptedData[0]).toHaveProperty('lastName__source') - expect(encryptedData[0]).toHaveProperty('lastName__hmac') - expect(encryptedData[0]).toHaveProperty('phoneNumber__source') - - // Verify second item - expect(encryptedData[1]).toHaveProperty('email__source') - expect(encryptedData[1]).toHaveProperty('email__hmac') - expect(encryptedData[1]).toHaveProperty('firstName__source') - expect(encryptedData[1]).toHaveProperty('firstName__hmac') - expect(encryptedData[1]).toHaveProperty('lastName__source') - expect(encryptedData[1]).toHaveProperty('lastName__hmac') - expect(encryptedData[1]).toHaveProperty('phoneNumber__source') - }) - - it('should handle decryption of encrypted data', async () => { - const originalData = { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test.user@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - }, - }, - } - - // First encrypt - const encryptResult = await protectDynamo.encryptModel(originalData, schema) - - if (encryptResult.failure) { - throw new Error(`Encryption failed: ${encryptResult.failure.message}`) - } - - // Then decrypt - const decryptResult = await protectDynamo.decryptModel( - encryptResult.data, - schema, - ) - if (decryptResult.failure) { - throw new Error(`Decryption failed: ${decryptResult.failure.message}`) - } - - const decryptedData = decryptResult.data - - // Verify all fields match original data - expect(decryptedData).toEqual(originalData) - }) - - it('should handle decryption of bulk encrypted data', async () => { - const originalData = [ - { - id: '01ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test1@example.com', - firstName: 'John', - lastName: 'Smith', - phoneNumber: '555-555-5555', - }, - { - id: '02ABCDEFGHIJKLMNOPQRSTUVWX', - email: 'test2@example.com', - firstName: 'Jane', - lastName: 'Doe', - phoneNumber: '555-555-5556', - example: { - protected: 'hello world', - notProtected: 'I am not protected', - deep: { - protected: 'deep protected', - notProtected: 'deep not protected', - }, - }, - }, - ] - - // First encrypt - const encryptResult = await protectDynamo.bulkEncryptModels( - originalData, - schema, - ) - if (encryptResult.failure) { - throw new Error( - `Bulk encryption failed: ${encryptResult.failure.message}`, - ) - } - - // Then decrypt - const decryptResult = await protectDynamo.bulkDecryptModels( - encryptResult.data, - schema, - ) - if (decryptResult.failure) { - throw new Error( - `Bulk decryption failed: ${decryptResult.failure.message}`, - ) - } - - const decryptedData = decryptResult.data - - // Verify all items match original data - expect(decryptedData).toEqual(originalData) - }) -}) diff --git a/packages/protect-dynamodb/__tests__/error-codes.test.ts b/packages/protect-dynamodb/__tests__/error-codes.test.ts deleted file mode 100644 index 918331c95..000000000 --- a/packages/protect-dynamodb/__tests__/error-codes.test.ts +++ /dev/null @@ -1,125 +0,0 @@ -import 'dotenv/config' -import type { ProtectClient } from '@cipherstash/protect' -import { - csColumn, - csTable, - FfiProtectError, - protect, -} from '@cipherstash/protect' -import { beforeAll, describe, expect, it } from 'vitest' -import { protectDynamoDB } from '../src' -import type { ProtectDynamoDBError } from '../src/types' - -const FFI_TEST_TIMEOUT = 30_000 - -describe('ProtectDynamoDB Error Code Preservation', () => { - let protectClient: ProtectClient - let protectDynamo: ReturnType - - const testSchema = csTable('test_table', { - email: csColumn('email').equality(), - }) - - const badSchema = csTable('test_table', { - nonexistent: csColumn('nonexistent_column'), - }) - - beforeAll(async () => { - protectClient = await protect({ schemas: [testSchema] }) - protectDynamo = protectDynamoDB({ protectClient }) - }) - - describe('handleError FFI error code extraction', () => { - it('FfiProtectError has code property accessible', () => { - const ffiError = new FfiProtectError({ - code: 'UNKNOWN_COLUMN', - message: 'Test error', - }) - expect(ffiError.code).toBe('UNKNOWN_COLUMN') - expect(ffiError instanceof FfiProtectError).toBe(true) - }) - }) - - describe('encryptModel error codes', () => { - it( - 'preserves FFI error codes', - async () => { - const model = { nonexistent: 'test value' } - - const result = await protectDynamo.encryptModel(model, badSchema) - - expect(result.failure).toBeDefined() - expect((result.failure as ProtectDynamoDBError).code).toBe( - 'UNKNOWN_COLUMN', - ) - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('decryptModel error codes', () => { - it( - 'uses PROTECT_DYNAMODB_ERROR for IO/parsing errors without FFI codes', - async () => { - // Malformed ciphertext causes IO/parsing errors that don't have FFI error codes - const malformedItem = { - email__source: 'invalid_ciphertext_data', - } - - const result = await protectDynamo.decryptModel( - malformedItem, - testSchema, - ) - - expect(result.failure).toBeDefined() - // FFI returns undefined code for IO/parsing errors, so we fall back to generic code - expect((result.failure as ProtectDynamoDBError).code).toBe( - 'PROTECT_DYNAMODB_ERROR', - ) - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkEncryptModels error codes', () => { - it( - 'preserves FFI error codes', - async () => { - const models = [{ nonexistent: 'value1' }, { nonexistent: 'value2' }] - - const result = await protectDynamo.bulkEncryptModels(models, badSchema) - - expect(result.failure).toBeDefined() - expect((result.failure as ProtectDynamoDBError).code).toBe( - 'UNKNOWN_COLUMN', - ) - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkDecryptModels error codes', () => { - it( - 'uses PROTECT_DYNAMODB_ERROR for IO/parsing errors without FFI codes', - async () => { - // Malformed ciphertext causes IO/parsing errors that don't have FFI error codes - const malformedItems = [ - { email__source: 'invalid1' }, - { email__source: 'invalid2' }, - ] - - const result = await protectDynamo.bulkDecryptModels( - malformedItems, - testSchema, - ) - - expect(result.failure).toBeDefined() - // FFI returns undefined code for IO/parsing errors, so we fall back to generic code - expect((result.failure as ProtectDynamoDBError).code).toBe( - 'PROTECT_DYNAMODB_ERROR', - ) - }, - FFI_TEST_TIMEOUT, - ) - }) -}) diff --git a/packages/protect-dynamodb/__tests__/helpers.test.ts b/packages/protect-dynamodb/__tests__/helpers.test.ts deleted file mode 100644 index aeb22bdc6..000000000 --- a/packages/protect-dynamodb/__tests__/helpers.test.ts +++ /dev/null @@ -1,89 +0,0 @@ -import { csColumn, csTable } from '@cipherstash/protect' -import { describe, expect, it } from 'vitest' -import { - ciphertextAttrSuffix, - searchTermAttrSuffix, - toItemWithEqlPayloads, -} from '../src/helpers' - -describe('toItemWithEqlPayloads', () => { - it('wraps searchable JSON columns as ste_vec (k: "sv") EQL payloads', () => { - // Regression test for `cast_as === 'json'` (previously 'jsonb'). The - // searchableJson() builder produces cast_as: 'json' + ste_vec index, and - // this branch must emit k: 'sv' so DynamoDB items round-trip through - // decrypt without losing the ste_vec discriminant. - const schema = csTable('users', { - preferences: csColumn('preferences').searchableJson(), - }) - - const stored = [{ s: 'selector', t: 'term' }] - const item = { - id: 'user-1', - [`preferences${ciphertextAttrSuffix}`]: stored, - } - - const result = toItemWithEqlPayloads(item, schema) - - expect(result).toEqual({ - id: 'user-1', - preferences: { - i: { c: 'preferences', t: 'users' }, - v: 2, - k: 'sv', - sv: stored, - }, - }) - }) - - it('wraps non-ste_vec columns as scalar ciphertext (k: "ct") EQL payloads', () => { - const schema = csTable('users', { - email: csColumn('email').equality(), - }) - - const ciphertext = 'mp_base85_ciphertext' - const item = { - id: 'user-1', - [`email${ciphertextAttrSuffix}`]: ciphertext, - [`email${searchTermAttrSuffix}`]: 'hmac-value', - } - - const result = toItemWithEqlPayloads(item, schema) - - // HMAC attribute is stripped; ciphertext is wrapped as k: 'ct'. - expect(result).toEqual({ - id: 'user-1', - email: { - i: { c: 'email', t: 'users' }, - v: 2, - k: 'ct', - c: ciphertext, - }, - }) - }) - - it('wraps non-searchable JSON columns as scalar ciphertext (k: "ct")', () => { - // A plain `dataType('json')` column has cast_as: 'json' but no ste_vec - // index — it must take the default `k: 'ct'` branch, not `k: 'sv'`. - const schema = csTable('users', { - metadata: csColumn('metadata').dataType('json'), - }) - - const ciphertext = 'mp_base85_ciphertext' - const item = { - id: 'user-1', - [`metadata${ciphertextAttrSuffix}`]: ciphertext, - } - - const result = toItemWithEqlPayloads(item, schema) - - expect(result).toEqual({ - id: 'user-1', - metadata: { - i: { c: 'metadata', t: 'users' }, - v: 2, - k: 'ct', - c: ciphertext, - }, - }) - }) -}) diff --git a/packages/protect-dynamodb/package.json b/packages/protect-dynamodb/package.json deleted file mode 100644 index 2e1d99d01..000000000 --- a/packages/protect-dynamodb/package.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "name": "@cipherstash/protect-dynamodb", - "version": "12.0.2-rc.0", - "description": "Protect.js DynamoDB Helpers", - "keywords": [ - "dynamodb", - "cipherstash", - "protect", - "encrypt", - "decrypt", - "security" - ], - "bugs": { - "url": "https://github.com/cipherstash/stack/issues" - }, - "repository": { - "type": "git", - "url": "git+https://github.com/cipherstash/stack.git", - "directory": "packages/protect-dynamodb" - }, - "license": "MIT", - "author": "CipherStash ", - "type": "module", - "exports": { - ".": { - "types": "./dist/index.d.ts", - "import": "./dist/index.js", - "require": "./dist/index.cjs" - } - }, - "scripts": { - "build": "tsup", - "dev": "tsup --watch", - "test": "vitest run", - "release": "tsup" - }, - "devDependencies": { - "@cipherstash/protect": "workspace:*", - "dotenv": "^17.4.2", - "tsup": "catalog:repo", - "tsx": "catalog:repo", - "typescript": "catalog:repo", - "vitest": "catalog:repo" - }, - "peerDependencies": { - "@cipherstash/protect": "workspace:*" - }, - "publishConfig": { - "access": "public" - }, - "dependencies": { - "@byteslice/result": "^0.2.0" - } -} diff --git a/packages/protect-dynamodb/src/helpers.ts b/packages/protect-dynamodb/src/helpers.ts deleted file mode 100644 index 505e5c54b..000000000 --- a/packages/protect-dynamodb/src/helpers.ts +++ /dev/null @@ -1,237 +0,0 @@ -import type { - Encrypted, - ProtectErrorCode, - ProtectTable, - ProtectTableColumn, -} from '@cipherstash/protect' -import { FfiProtectError } from '@cipherstash/protect' -import type { ProtectDynamoDBError } from './types' -export const ciphertextAttrSuffix = '__source' -export const searchTermAttrSuffix = '__hmac' - -export class ProtectDynamoDBErrorImpl - extends Error - implements ProtectDynamoDBError -{ - constructor( - message: string, - public code: ProtectErrorCode | 'PROTECT_DYNAMODB_ERROR', - public details?: Record, - ) { - super(message) - this.name = 'ProtectDynamoDBError' - } -} - -export function handleError( - error: unknown, - context: string, - options?: { - logger?: { - error: (message: string, error: Error) => void - } - errorHandler?: (error: ProtectDynamoDBError) => void - }, -): ProtectDynamoDBError { - // Preserve FFI error code if available, otherwise use generic DynamoDB error code - // Check for FfiProtectError instance or plain ProtectError objects with code property - const errorObj = error as Record - const errorCode = - error instanceof FfiProtectError - ? error.code - : errorObj && - typeof errorObj === 'object' && - 'code' in errorObj && - typeof errorObj.code === 'string' - ? (errorObj.code as ProtectErrorCode) - : 'PROTECT_DYNAMODB_ERROR' - - const errorMessage = - error instanceof Error - ? error.message - : errorObj && typeof errorObj.message === 'string' - ? errorObj.message - : String(error) - - const protectError = new ProtectDynamoDBErrorImpl(errorMessage, errorCode, { - context, - }) - - if (options?.errorHandler) { - options.errorHandler(protectError) - } - - if (options?.logger) { - options.logger.error(`Error in ${context}`, protectError) - } - - return protectError -} - -export function deepClone(obj: T): T { - if (obj === null || typeof obj !== 'object') { - return obj - } - - if (Array.isArray(obj)) { - return obj.map((item) => deepClone(item)) as unknown as T - } - - return Object.entries(obj as Record).reduce( - (acc, [key, value]) => ({ - // biome-ignore lint/performance/noAccumulatingSpread: TODO later - ...acc, - [key]: deepClone(value), - }), - {} as T, - ) -} - -export function toEncryptedDynamoItem( - encrypted: Record, - encryptedAttrs: string[], -): Record { - function processValue( - attrName: string, - attrValue: unknown, - isNested: boolean, - ): Record { - if (attrValue === null || attrValue === undefined) { - return { [attrName]: attrValue } - } - - // Handle encrypted payload - if ( - encryptedAttrs.includes(attrName) || - (isNested && - typeof attrValue === 'object' && - 'c' in (attrValue as object)) - ) { - const encryptPayload = attrValue as Encrypted - if (encryptPayload?.k === 'ct' && encryptPayload.c) { - const result: Record = {} - if (encryptPayload.hm) { - result[`${attrName}${searchTermAttrSuffix}`] = encryptPayload.hm - } - result[`${attrName}${ciphertextAttrSuffix}`] = encryptPayload.c - return result - } - - if (encryptPayload?.k === 'sv' && encryptPayload.sv) { - const result: Record = {} - result[`${attrName}${ciphertextAttrSuffix}`] = encryptPayload.sv - return result - } - } - - // Handle nested objects recursively - if (typeof attrValue === 'object' && !Array.isArray(attrValue)) { - const nestedResult = Object.entries( - attrValue as Record, - ).reduce( - (acc, [key, val]) => { - const processed = processValue(key, val, true) - return Object.assign({}, acc, processed) - }, - {} as Record, - ) - return { [attrName]: nestedResult } - } - - // Handle non-encrypted values - return { [attrName]: attrValue } - } - - return Object.entries(encrypted).reduce( - (putItem, [attrName, attrValue]) => { - const processed = processValue(attrName, attrValue, false) - return Object.assign({}, putItem, processed) - }, - {} as Record, - ) -} - -export function toItemWithEqlPayloads( - decrypted: Record, - encryptSchemas: ProtectTable, -): Record { - function processValue( - attrName: string, - attrValue: unknown, - isNested: boolean, - ): Record { - if (attrValue === null || attrValue === undefined) { - return { [attrName]: attrValue } - } - - // Skip HMAC fields - if (attrName.endsWith(searchTermAttrSuffix)) { - return {} - } - - const encryptConfig = encryptSchemas.build() - const encryptedAttrs = Object.keys(encryptConfig.columns) - const columnName = attrName.slice(0, -ciphertextAttrSuffix.length) - - // Handle encrypted payload - if ( - attrName.endsWith(ciphertextAttrSuffix) && - (encryptedAttrs.includes(columnName) || isNested) - ) { - // TODO: Implement the standardized typing for Encrypted Payloads through the FFI interface - const i = { c: columnName, t: encryptConfig.tableName } - const v = 2 - - // Nested values are not searchable, so we can just return the standard EQL payload. - // Worth noting, that encryptConfig.columns[columnName] will be undefined if isNested is true. - if ( - !isNested && - encryptConfig.columns[columnName].cast_as === 'json' && - encryptConfig.columns[columnName].indexes.ste_vec - ) { - return { - [columnName]: { - i, - v, - k: 'sv', - sv: attrValue, - }, - } - } - - return { - [columnName]: { - i, - v, - k: 'ct', - c: attrValue, - }, - } - } - - // Handle nested objects recursively - if (typeof attrValue === 'object' && !Array.isArray(attrValue)) { - const nestedResult = Object.entries( - attrValue as Record, - ).reduce( - (acc, [key, val]) => { - const processed = processValue(key, val, true) - return Object.assign({}, acc, processed) - }, - {} as Record, - ) - return { [attrName]: nestedResult } - } - - // Handle non-encrypted values - return { [attrName]: attrValue } - } - - return Object.entries(decrypted).reduce( - (formattedItem, [attrName, attrValue]) => { - const processed = processValue(attrName, attrValue, false) - return Object.assign({}, formattedItem, processed) - }, - {} as Record, - ) -} diff --git a/packages/protect-dynamodb/src/index.ts b/packages/protect-dynamodb/src/index.ts deleted file mode 100644 index fe18ddfa7..000000000 --- a/packages/protect-dynamodb/src/index.ts +++ /dev/null @@ -1,77 +0,0 @@ -import type { - Encrypted, - ProtectTable, - ProtectTableColumn, - SearchTerm, -} from '@cipherstash/protect' -import { BulkDecryptModelsOperation } from './operations/bulk-decrypt-models' -import { BulkEncryptModelsOperation } from './operations/bulk-encrypt-models' -import { DecryptModelOperation } from './operations/decrypt-model' -import { EncryptModelOperation } from './operations/encrypt-model' -import { SearchTermsOperation } from './operations/search-terms' -import type { ProtectDynamoDBConfig, ProtectDynamoDBInstance } from './types' - -export function protectDynamoDB( - config: ProtectDynamoDBConfig, -): ProtectDynamoDBInstance { - const { protectClient, options } = config - - return { - encryptModel>( - item: T, - protectTable: ProtectTable, - ) { - return new EncryptModelOperation( - protectClient, - item, - protectTable, - options, - ) - }, - - bulkEncryptModels>( - items: T[], - protectTable: ProtectTable, - ) { - return new BulkEncryptModelsOperation( - protectClient, - items, - protectTable, - options, - ) - }, - - decryptModel>( - item: Record, - protectTable: ProtectTable, - ) { - return new DecryptModelOperation( - protectClient, - item, - protectTable, - options, - ) - }, - - bulkDecryptModels>( - items: Record[], - protectTable: ProtectTable, - ) { - return new BulkDecryptModelsOperation( - protectClient, - items, - protectTable, - options, - ) - }, - - /** - * @deprecated Use `protectClient.encryptQuery(terms)` instead and extract the `hm` field for DynamoDB key lookups. - */ - createSearchTerms(terms: SearchTerm[]) { - return new SearchTermsOperation(protectClient, terms, options) - }, - } -} - -export * from './types' diff --git a/packages/protect-dynamodb/src/operations/base-operation.ts b/packages/protect-dynamodb/src/operations/base-operation.ts deleted file mode 100644 index e9ea0bb64..000000000 --- a/packages/protect-dynamodb/src/operations/base-operation.ts +++ /dev/null @@ -1,64 +0,0 @@ -import type { Result } from '@byteslice/result' -import type { ProtectDynamoDBError } from '../types' - -export type AuditConfig = { - metadata?: Record -} - -export type AuditData = { - metadata?: Record -} - -export type DynamoDBOperationOptions = { - logger?: { - error: (message: string, error: Error) => void - } - errorHandler?: (error: ProtectDynamoDBError) => void -} - -export abstract class DynamoDBOperation { - protected auditMetadata?: Record - protected logger?: DynamoDBOperationOptions['logger'] - protected errorHandler?: DynamoDBOperationOptions['errorHandler'] - - constructor(options?: DynamoDBOperationOptions) { - this.logger = options?.logger - this.errorHandler = options?.errorHandler - } - - /** - * Attach audit metadata to this operation. Can be chained. - */ - audit(config: AuditConfig): this { - this.auditMetadata = config.metadata - return this - } - - /** - * Get the audit metadata for this operation. - */ - protected getAuditData(): AuditData { - return { - metadata: this.auditMetadata, - } - } - - /** - * Execute the operation and return a Result - */ - abstract execute(): Promise> - - /** - * Make the operation thenable - */ - public then, TResult2 = never>( - onfulfilled?: - | (( - value: Result, - ) => TResult1 | PromiseLike) - | null, - onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null, - ): Promise { - return this.execute().then(onfulfilled, onrejected) - } -} diff --git a/packages/protect-dynamodb/src/operations/bulk-decrypt-models.ts b/packages/protect-dynamodb/src/operations/bulk-decrypt-models.ts deleted file mode 100644 index f27e6dd4a..000000000 --- a/packages/protect-dynamodb/src/operations/bulk-decrypt-models.ts +++ /dev/null @@ -1,67 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { - Decrypted, - Encrypted, - ProtectClient, - ProtectTable, - ProtectTableColumn, -} from '@cipherstash/protect' -import { handleError, toItemWithEqlPayloads } from '../helpers' -import type { ProtectDynamoDBError } from '../types' -import { - DynamoDBOperation, - type DynamoDBOperationOptions, -} from './base-operation' - -export class BulkDecryptModelsOperation< - T extends Record, -> extends DynamoDBOperation[]> { - private protectClient: ProtectClient - private items: Record[] - private protectTable: ProtectTable - - constructor( - protectClient: ProtectClient, - items: Record[], - protectTable: ProtectTable, - options?: DynamoDBOperationOptions, - ) { - super(options) - this.protectClient = protectClient - this.items = items - this.protectTable = protectTable - } - - public async execute(): Promise< - Result[], ProtectDynamoDBError> - > { - return await withResult( - async () => { - const itemsWithEqlPayloads = this.items.map((item) => - toItemWithEqlPayloads(item, this.protectTable), - ) - - const decryptResult = await this.protectClient - .bulkDecryptModels(itemsWithEqlPayloads as T[]) - .audit(this.getAuditData()) - - if (decryptResult.failure) { - // Create an Error object that preserves the FFI error code - // This is necessary because withResult's ensureError wraps non-Error objects - const error = new Error(decryptResult.failure.message) as Error & { - code?: string - } - error.code = decryptResult.failure.code - throw error - } - - return decryptResult.data - }, - (error) => - handleError(error, 'bulkDecryptModels', { - logger: this.logger, - errorHandler: this.errorHandler, - }), - ) - } -} diff --git a/packages/protect-dynamodb/src/operations/bulk-encrypt-models.ts b/packages/protect-dynamodb/src/operations/bulk-encrypt-models.ts deleted file mode 100644 index 855f9d142..000000000 --- a/packages/protect-dynamodb/src/operations/bulk-encrypt-models.ts +++ /dev/null @@ -1,67 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { - ProtectClient, - ProtectTable, - ProtectTableColumn, -} from '@cipherstash/protect' -import { deepClone, handleError, toEncryptedDynamoItem } from '../helpers' -import type { ProtectDynamoDBError } from '../types' -import { - DynamoDBOperation, - type DynamoDBOperationOptions, -} from './base-operation' - -export class BulkEncryptModelsOperation< - T extends Record, -> extends DynamoDBOperation { - private protectClient: ProtectClient - private items: T[] - private protectTable: ProtectTable - - constructor( - protectClient: ProtectClient, - items: T[], - protectTable: ProtectTable, - options?: DynamoDBOperationOptions, - ) { - super(options) - this.protectClient = protectClient - this.items = items - this.protectTable = protectTable - } - - public async execute(): Promise> { - return await withResult( - async () => { - const encryptResult = await this.protectClient - .bulkEncryptModels( - this.items.map((item) => deepClone(item)), - this.protectTable, - ) - .audit(this.getAuditData()) - - if (encryptResult.failure) { - // Create an Error object that preserves the FFI error code - // This is necessary because withResult's ensureError wraps non-Error objects - const error = new Error(encryptResult.failure.message) as Error & { - code?: string - } - error.code = encryptResult.failure.code - throw error - } - - const data = encryptResult.data.map((item) => deepClone(item)) - const encryptedAttrs = Object.keys(this.protectTable.build().columns) - - return data.map( - (encrypted) => toEncryptedDynamoItem(encrypted, encryptedAttrs) as T, - ) - }, - (error) => - handleError(error, 'bulkEncryptModels', { - logger: this.logger, - errorHandler: this.errorHandler, - }), - ) - } -} diff --git a/packages/protect-dynamodb/src/operations/decrypt-model.ts b/packages/protect-dynamodb/src/operations/decrypt-model.ts deleted file mode 100644 index c6980108b..000000000 --- a/packages/protect-dynamodb/src/operations/decrypt-model.ts +++ /dev/null @@ -1,66 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { - Decrypted, - Encrypted, - ProtectClient, - ProtectTable, - ProtectTableColumn, -} from '@cipherstash/protect' -import { handleError, toItemWithEqlPayloads } from '../helpers' -import type { ProtectDynamoDBError } from '../types' -import { - DynamoDBOperation, - type DynamoDBOperationOptions, -} from './base-operation' - -export class DecryptModelOperation< - T extends Record, -> extends DynamoDBOperation> { - private protectClient: ProtectClient - private item: Record - private protectTable: ProtectTable - - constructor( - protectClient: ProtectClient, - item: Record, - protectTable: ProtectTable, - options?: DynamoDBOperationOptions, - ) { - super(options) - this.protectClient = protectClient - this.item = item - this.protectTable = protectTable - } - - public async execute(): Promise, ProtectDynamoDBError>> { - return await withResult( - async () => { - const withEqlPayloads = toItemWithEqlPayloads( - this.item, - this.protectTable, - ) - - const decryptResult = await this.protectClient - .decryptModel(withEqlPayloads as T) - .audit(this.getAuditData()) - - if (decryptResult.failure) { - // Create an Error object that preserves the FFI error code - // This is necessary because withResult's ensureError wraps non-Error objects - const error = new Error(decryptResult.failure.message) as Error & { - code?: string - } - error.code = decryptResult.failure.code - throw error - } - - return decryptResult.data - }, - (error) => - handleError(error, 'decryptModel', { - logger: this.logger, - errorHandler: this.errorHandler, - }), - ) - } -} diff --git a/packages/protect-dynamodb/src/operations/encrypt-model.ts b/packages/protect-dynamodb/src/operations/encrypt-model.ts deleted file mode 100644 index 82ee1bd85..000000000 --- a/packages/protect-dynamodb/src/operations/encrypt-model.ts +++ /dev/null @@ -1,62 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { - ProtectClient, - ProtectTable, - ProtectTableColumn, -} from '@cipherstash/protect' -import { deepClone, handleError, toEncryptedDynamoItem } from '../helpers' -import type { ProtectDynamoDBError } from '../types' -import { - DynamoDBOperation, - type DynamoDBOperationOptions, -} from './base-operation' - -export class EncryptModelOperation< - T extends Record, -> extends DynamoDBOperation { - private protectClient: ProtectClient - private item: T - private protectTable: ProtectTable - - constructor( - protectClient: ProtectClient, - item: T, - protectTable: ProtectTable, - options?: DynamoDBOperationOptions, - ) { - super(options) - this.protectClient = protectClient - this.item = item - this.protectTable = protectTable - } - - public async execute(): Promise> { - return await withResult( - async () => { - const encryptResult = await this.protectClient - .encryptModel(deepClone(this.item), this.protectTable) - .audit(this.getAuditData()) - - if (encryptResult.failure) { - // Create an Error object that preserves the FFI error code - // This is necessary because withResult's ensureError wraps non-Error objects - const error = new Error(encryptResult.failure.message) as Error & { - code?: string - } - error.code = encryptResult.failure.code - throw error - } - - const data = deepClone(encryptResult.data) - const encryptedAttrs = Object.keys(this.protectTable.build().columns) - - return toEncryptedDynamoItem(data, encryptedAttrs) as T - }, - (error) => - handleError(error, 'encryptModel', { - logger: this.logger, - errorHandler: this.errorHandler, - }), - ) - } -} diff --git a/packages/protect-dynamodb/src/operations/search-terms.ts b/packages/protect-dynamodb/src/operations/search-terms.ts deleted file mode 100644 index 4da416bae..000000000 --- a/packages/protect-dynamodb/src/operations/search-terms.ts +++ /dev/null @@ -1,77 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { - isEncryptedScalarQuery, - type ProtectClient, - type SearchTerm, -} from '@cipherstash/protect' -import { handleError } from '../helpers' -import type { ProtectDynamoDBError } from '../types' -import { - DynamoDBOperation, - type DynamoDBOperationOptions, -} from './base-operation' - -/** - * @deprecated Use `protectClient.encryptQuery(terms)` instead and extract the `hm` field for DynamoDB key lookups. - * - * @example - * ```typescript - * // Before (deprecated) - * const result = await protectDynamo.createSearchTerms([{ value, column, table }]) - * const hmac = result.data[0] - * - * // After (new API) - * const [encrypted] = await protectClient.encryptQuery([{ value, column, table, queryType: 'equality' }]) - * const hmac = encrypted.hm - * ``` - */ -export class SearchTermsOperation extends DynamoDBOperation { - private protectClient: ProtectClient - private terms: SearchTerm[] - - constructor( - protectClient: ProtectClient, - terms: SearchTerm[], - options?: DynamoDBOperationOptions, - ) { - super(options) - this.protectClient = protectClient - this.terms = terms - } - - public async execute(): Promise> { - return await withResult( - async () => { - const searchTermsResult = await this.protectClient - .createSearchTerms(this.terms) - .audit(this.getAuditData()) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - return searchTermsResult.data.map((term) => { - if (typeof term === 'string') { - throw new Error( - 'expected encrypted search term to be an EncryptedPayload', - ) - } - - // DynamoDB lookups go through equality queries → the FFI returns an - // EncryptedScalarQuery carrying `hm`. Anything else (scalar storage, - // a `bf`/`ob` query, or a SteVec payload) is a misconfiguration. - if (!isEncryptedScalarQuery(term) || !('hm' in term)) { - throw new Error('expected encrypted search term to have an HMAC') - } - - return term.hm - }) - }, - (error) => - handleError(error, 'createSearchTerms', { - logger: this.logger, - errorHandler: this.errorHandler, - }), - ) - } -} diff --git a/packages/protect-dynamodb/src/types.ts b/packages/protect-dynamodb/src/types.ts deleted file mode 100644 index 6cd7338ba..000000000 --- a/packages/protect-dynamodb/src/types.ts +++ /dev/null @@ -1,66 +0,0 @@ -import type { - Encrypted, - ProtectClient, - ProtectErrorCode, - ProtectTable, - ProtectTableColumn, - SearchTerm, -} from '@cipherstash/protect' -import type { BulkDecryptModelsOperation } from './operations/bulk-decrypt-models' -import type { BulkEncryptModelsOperation } from './operations/bulk-encrypt-models' -import type { DecryptModelOperation } from './operations/decrypt-model' -import type { EncryptModelOperation } from './operations/encrypt-model' -import type { SearchTermsOperation } from './operations/search-terms' - -export interface ProtectDynamoDBConfig { - protectClient: ProtectClient - options?: { - logger?: { - error: (message: string, error: Error) => void - } - errorHandler?: (error: ProtectDynamoDBError) => void - } -} - -export interface ProtectDynamoDBError extends Error { - code: ProtectErrorCode | 'PROTECT_DYNAMODB_ERROR' - details?: Record -} - -export interface ProtectDynamoDBInstance { - encryptModel>( - item: T, - protectTable: ProtectTable, - ): EncryptModelOperation - - bulkEncryptModels>( - items: T[], - protectTable: ProtectTable, - ): BulkEncryptModelsOperation - - decryptModel>( - item: Record, - protectTable: ProtectTable, - ): DecryptModelOperation - - bulkDecryptModels>( - items: Record[], - protectTable: ProtectTable, - ): BulkDecryptModelsOperation - - /** - * @deprecated Use `protectClient.encryptQuery(terms)` instead and extract the `hm` field for DynamoDB key lookups. - * - * @example - * ```typescript - * // Before (deprecated) - * const result = await protectDynamo.createSearchTerms([{ value, column, table }]) - * const hmac = result.data[0] - * - * // After (new API) - * const [encrypted] = await protectClient.encryptQuery([{ value, column, table, queryType: 'equality' }]) - * const hmac = encrypted.hm - * ``` - */ - createSearchTerms(terms: SearchTerm[]): SearchTermsOperation -} diff --git a/packages/protect-dynamodb/tsconfig.json b/packages/protect-dynamodb/tsconfig.json deleted file mode 100644 index 6da81c927..000000000 --- a/packages/protect-dynamodb/tsconfig.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "compilerOptions": { - // Enable latest features - "lib": ["ESNext", "DOM"], - "target": "ESNext", - "module": "ESNext", - "moduleDetection": "force", - "jsx": "react-jsx", - "allowJs": true, - "esModuleInterop": true, - - // Bundler mode - "moduleResolution": "bundler", - "allowImportingTsExtensions": true, - "verbatimModuleSyntax": true, - "noEmit": true, - - // Best practices - "strict": true, - "skipLibCheck": true, - "noFallthroughCasesInSwitch": true, - - // Some stricter flags (disabled by default) - "noUnusedLocals": false, - "noUnusedParameters": false, - "noPropertyAccessFromIndexSignature": false - } -} diff --git a/packages/protect-dynamodb/tsup.config.ts b/packages/protect-dynamodb/tsup.config.ts deleted file mode 100644 index 0ced0a354..000000000 --- a/packages/protect-dynamodb/tsup.config.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { defineConfig } from 'tsup' - -export default defineConfig({ - entry: ['src/index.ts'], - format: ['cjs', 'esm'], - sourcemap: true, - dts: true, -}) diff --git a/packages/protect/.npmignore b/packages/protect/.npmignore deleted file mode 100644 index 3490e24db..000000000 --- a/packages/protect/.npmignore +++ /dev/null @@ -1,5 +0,0 @@ -.env -.turbo -node_modules -cipherstash.secret.toml -cipherstash.toml \ No newline at end of file diff --git a/packages/protect/CHANGELOG.md b/packages/protect/CHANGELOG.md deleted file mode 100644 index 9c79eaf46..000000000 --- a/packages/protect/CHANGELOG.md +++ /dev/null @@ -1,439 +0,0 @@ -# @cipherstash/protect - -## 12.0.2-rc.0 - -### Patch Changes - -- Updated dependencies [229ce59] - - @cipherstash/schema@3.0.2-rc.0 - -## 12.0.1 - -### Patch Changes - -- aa9c4b1: Documentation: refresh package READMEs after the protectjs → stack repository rename. Fixed repository and license links, replaced dead in-repo docs links with cipherstash.com/docs URLs, rewrote the incorrect @cipherstash/nextjs README, and added guidance pointing new projects to @cipherstash/stack. -- Updated dependencies [aa9c4b1] - - @cipherstash/schema@3.0.1 - -## 12.0.0 - -### Major Changes - -- f743fcc: Upgrade `@cipherstash/protect-ffi` to `0.23.0` and the bundled CipherStash EQL extension to `eql-2.3.1`. - - Breaking upstream changes adopted in this release: - - - **Encrypt-config schema version**: `buildEncryptConfig` now emits `{ v: 1, ... }` (was `{ v: 2, ... }`). protect-ffi `0.22.0` started validating this field and rejects any value other than `1` with the new `UNSUPPORTED_CONFIG_VERSION` error code. - - **Storage and query payloads are now distinct types** (protect-ffi `0.23.0`): the previously-conflated `Encrypted` type splits into `Encrypted` (storage-only, `c` required) and a new `EncryptedQuery` (search terms — scalar `unique`/`match`/`ore` lookups and `ste_vec_selector` JSON path queries; no `c`). JSON containment queries (`ste_vec_term`) still return a storage-shaped `Encrypted` payload. `encryptQuery` / `encryptQueryBulk` now return `Encrypted | EncryptedQuery`, and the stack's `EncryptedSearchTerm` / `EncryptedQueryResult` unions widen to match. `decrypt` rejects query payloads at the type level. The DynamoDB `SearchTermsOperation` narrows via `'hm' in term` rather than `term.hm`. - - **SteVec encoding default flipped**: protect-ffi's default `mode` for `ste_vec` indexes changed from `compat` to `standard`. The two encodings are not cross-compatible. Existing JSON-searchable data that was indexed under `compat` will need to be re-encrypted to be queryable. The stack adopts the new `standard` default — there is no longer a way to pin `compat` from the SDK. - - **EQL extension bumped to `eql-2.3.1`**: the new SteVec `standard` encoding requires matching support in the database EQL extension. The CLI's bundled SQL (`packages/cli/src/sql/*.sql`) and the `@cipherstash/prisma-next` install bundle (`migrations/20260601T0000_install_eql_bundle/ops.json` + `eql-install.generated.ts`) are updated to `eql-2.3.1`. Databases installed with an older EQL extension must be reinstalled (`stash db install`) before containment / contained-by queries against SteVec columns will work. `eql-2.3.1` ships the `_encrypted_check_c` fix for SteVec storage payloads ([cipherstash/encrypt-query-language#232](https://github.com/cipherstash/encrypt-query-language/issues/232)). - - **New error codes**: `ProtectErrorCode` (re-exported from `@cipherstash/protect-ffi`) gains `MATCH_REQUIRES_TEXT` and `UNSUPPORTED_CONFIG_VERSION`. Exhaustive switches over `ProtectErrorCode` will need additional cases. - - **`match` index validation**: protect-ffi now rejects `match` indexes on columns whose `cast_as` is not text-family (`'text'` / `'string'`) with `MATCH_REQUIRES_TEXT`. The stack's `freeTextSearch()` builder is unaffected because it only targets string-typed columns. - - **`Encrypted` ciphertext shape**: protect-ffi's `Encrypted` type is now a discriminated union keyed on `k` (`'ct'` for scalars, `'sv'` for SteVec). SteVec storage payloads now place the root document ciphertext at `sv[0].c`. The stack's `isEncryptedPayload` runtime check continues to work because storage payloads still carry `c` (scalar) or `sv` (SteVec). The DynamoDB helpers (`toEncryptedDynamoItem`, `SearchTermsOperation`) now narrow on `k` before reading variant-only fields. - - **Config-validation error message wording**: error messages for config-validation failures now come from upstream `ConfigError`. `ProtectError.code` values are preserved; consumers that string-match on `err.message` for config-validation errors must update. - -### Patch Changes - -- Updated dependencies [f743fcc] - - @cipherstash/schema@3.0.0 - -## 11.1.2 - -### Patch Changes - -- a8dbb65: Render every user-facing CLI string and execute every shell-out under the detected package manager (`npx` / `bunx` / `pnpm dlx` / `yarn dlx`), completing the work started in #379. Affected surfaces: `@cipherstash/cli` top-level + `auth` + `env` help, `db install` Drizzle migration steps, `db migrate` not-implemented warning, the Supabase migration SQL header, the Supabase status fallback exec, the `@cipherstash/protect` `stash` Stricli help (set/get/list/delete), the `@cipherstash/wizard` usage line and agent command allowlist, and the `@cipherstash/drizzle` `generate-eql-migration` help + drizzle-kit invocation. A new `pnpm run lint:runners` lint runs in CI and fails on any reintroduction of a hardcoded runner literal. - -## 11.1.1 - -### Patch Changes - -- afe6810: Bump protect-ffi version - -## 11.1.0 - -### Minor Changes - -- 068f820: Release the consolidated CipherStash CLI npm package. - -## 11.0.0 - -### Major Changes - -- b0e56b8: Upgrade protect-ffi to 0.21.0 and enable array_index_mode for searchable JSON - - - Upgrade `@cipherstash/protect-ffi` to 0.21.0 across all packages - - Enable `array_index_mode: 'all'` on STE vec indexes so JSON array operations - (jsonb_array_elements, jsonb_array_length, array containment) work correctly - - Delegate credential resolution entirely to protect-ffi's `withEnvCredentials` - - Download latest EQL at build/runtime instead of bundling hardcoded SQL files - -### Patch Changes - -- Updated dependencies [b0e56b8] - - @cipherstash/schema@2.2.0 - -## 10.5.0 - -### Minor Changes - -- db72e2c: Add `encryptQuery` API for encrypting query terms with explicit query type selection. - - - New `encryptQuery()` method replaces `createSearchTerms()` with improved query type handling - - Supports `equality`, `freeTextSearch`, and `orderAndRange` query types - - Deprecates `createSearchTerms()` - use `encryptQuery()` instead - - Updates drizzle operators to use correct index selection via `queryType` parameter - -- e769740: Add encrypted JSONB query support with `searchableJson()` (recommended). - - - New `searchableJson()` schema method enables encrypted JSONB path and containment queries - - Automatic query operation inference: string values become JSONPath selector queries, objects/arrays become containment queries - - Also supports explicit `queryType: 'steVecSelector'` and `queryType: 'steVecTerm'` for advanced use cases - - JSONB path utilities (`toJsonPath`, `buildNestedObject`, `parseJsonbPath`) for building encrypted JSON column queries - -### Patch Changes - -- Updated dependencies [e769740] - - @cipherstash/schema@2.1.0 - -## 10.4.0 - -### Minor Changes - -- 9ccaf68: Allow stash cli tool to read env files from .env.\*. - -## 10.3.0 - -### Minor Changes - -- a1fce2b: Add Stash interface and CLI tool. - -### Patch Changes - -- 622b684: Update @cipherstash/protect-ffi to 0.19.0 - -## 10.2.1 - -### Patch Changes - -- Updated dependencies [532ac3a] - - @cipherstash/schema@2.0.2 - -## 10.2.0 - -### Minor Changes - -- de029de: Add client safe exports. - -## 10.1.1 - -### Patch Changes - -- ff4421f: Expanded typedoc documentation -- Updated dependencies [ff4421f] - - @cipherstash/schema@2.0.1 - -## 10.1.0 - -### Minor Changes - -- 6b87c17: Added support for multi-tenant encryption with configurable keysets. - -## 10.0.2 - -### Patch Changes - -- Updated dependencies [9005484] - - @cipherstash/schema@2.0.0 - -## 10.0.1 - -### Patch Changes - -- Updated dependencies [d8ed4d4] - - @cipherstash/schema@1.1.0 - -## 10.0.0 - -### Major Changes - -- 788dbfc: Added JSON and INT data type support and update FFI to v0.17.1 with x86_64 musl environment platform support. - - - Update @cipherstash/protect-ffi from 0.16.0 to 0.17.1 with support for x86_64 musl platforms. - - Add searchableJson() method to schema for JSON field indexing (the search operations still don't work but this interface exists) - - Refactor type system: EncryptedPayload → Encrypted, add JsPlaintext - - Add comprehensive test suites for JSON, integer, and basic encryption - - Update encryption format to use 'k' property for searchable JSON - - Remove deprecated search terms tests for JSON fields - - Simplify schema data types to text, int, json only - - Update model helpers to handle new encryption format - - Fix type safety issues in bulk operations and model encryption - -### Patch Changes - -- Updated dependencies [788dbfc] - - @cipherstash/schema@1.0.0 - -## 9.6.0 - -### Minor Changes - -- c7ed7ab: Support TypeORM example with ES2022. -- 211e979: Added support for ES2022 and later. - -## 9.5.0 - -### Minor Changes - -- 6f45b02: Fully implemented audit metadata functionality. - -## 9.4.1 - -### Patch Changes - -- Updated dependencies [d0b02ea] - - @cipherstash/schema@0.1.0 - -## 9.4.0 - -### Minor Changes - -- 1cc4772: Released support for bulk encryption and decryption. - -## 9.3.0 - -### Minor Changes - -- 01fed9e: Added audit support for all protect and protect-dynamodb interfaces. - -## 9.2.0 - -### Minor Changes - -- 587f222: Added support for deeply nested protect schemas to support more complex model objects. - -## 9.1.0 - -### Minor Changes - -- c8468ee: Released initial version of the DynamoDB helper interface. - -## 9.0.0 - -### Major Changes - -- 1bc55a0: Implemented a more configurable pattern for the Protect client. - - This release introduces a new `ProtectClientConfig` type that can be used to configure the Protect client. - This is useful if you want to configure the Protect client specific to your application, and will future proof any additional configuration options that are added in the future. - - ```ts - import { protect, type ProtectClientConfig } from "@cipherstash/protect"; - - const config: ProtectClientConfig = { - schemas: [users, orders], - workspaceCrn: "your-workspace-crn", - accessKey: "your-access-key", - clientId: "your-client-id", - clientKey: "your-client-key", - }; - - const protectClient = await protect(config); - ``` - - The now deprecated method of passing your tables to the `protect` client is no longer supported. - - ```ts - import { protect, type ProtectClientConfig } from "@cipherstash/protect"; - - // old method (no longer supported) - const protectClient = await protect(users, orders); - - // required method - const config: ProtectClientConfig = { - schemas: [users, orders], - }; - - const protectClient = await protect(config); - ``` - -## 8.4.0 - -### Minor Changes - -- a471821: Fixed a bug in the model interface to correctly handle undefined and null values. - -## 8.3.0 - -### Minor Changes - -- 628acdc: Implemented createSearchTerms for a streamlined way of working with encrypted search terms. - -## 8.2.0 - -### Minor Changes - -- 0883e16: Fix cipherstash.toml and cipherstash.secret.toml file loading by bumping to @cipherstash/protect-ffi v0.14.2 - -## 8.1.0 - -### Minor Changes - -- 95c891d: Implemented CipherStash CRN in favor of workspace ID. - - - Replaces the environment variable `CS_WORKSPACE_ID` with `CS_WORKSPACE_CRN` - - Replaces `workspace_id` with `workspace_crn` in the `cipherstash.toml` file - -- 18d3653: Fixed handling composite types for EQL v2. - -## 8.0.0 - -### Major Changes - -- 8a4ea80: Implement EQL v2 data structure. - - - Support for Protect.js searchable encryption when using Supabase. - - Encrypted payloads are now composite types which support searchable encryption with EQL v2 functions. - - The `data` property is an object that matches the EQL v2 data structure. - -## 7.0.0 - -### Major Changes - -- 2cb2d84: Replaced bulk operations with model operations. - -## 6.3.0 - -### Minor Changes - -- a564f21: Bumped versions of dependencies to address CWE-346. - -## 6.2.0 - -### Minor Changes - -- fe4b443: Added symbolic link for protect readme. - -## 6.1.0 - -### Minor Changes - -- 43e1acb: \* Added support for searching encrypted data - - Added a schema strategy for defining your schema - - Required schema to initialize the protect client - -## 6.0.0 - -### Major Changes - -- f4d8334: Released protectjs-ffi with toml file configuration support. - Added a `withResult` pattern to all public facing functions for better error handling. - Updated all documentation to reflect the new configuration pattern. - -## 5.2.0 - -### Minor Changes - -- 499c246: Implemented protectjs-ffi. - -## 5.1.0 - -### Minor Changes - -- 5a34e76: Rebranded logging context and fixed tests. - -## 5.0.0 - -### Major Changes - -- 76599e5: Rebrand jseql to protect. - -## 4.0.0 - -### Major Changes - -- 5c08fe5: Enforced lock context to be called as a proto function rather than an optional argument for crypto functions. - There was a bug that caused the lock context to be interpreted as undefined when the users intention was to use it causing the encryption/decryption to fail. - This is a breaking change for users who were using the lock context as an optional argument. - To use the lock context, call the `withLockContext` method on the encrypt, decrypt, and bulk encrypt/decrypt functions, passing the lock context as a parameter rather than as an optional argument. - -## 3.9.0 - -### Minor Changes - -- e885975: Fixed improper use of throwing errors, and log with jseql logger. - -## 3.8.0 - -### Minor Changes - -- eeaec18: Implemented typing and import synatx for es6. - -## 3.7.0 - -### Minor Changes - -- 7b8ec52: Implement packageless logging framework. - -## 3.6.0 - -### Minor Changes - -- 7480cfd: Fixed node:util package bundling. - -## 3.5.0 - -### Minor Changes - -- c0123be: Replaced logtape with native node debuglog. - -## 3.4.0 - -### Minor Changes - -- 9a3132c: Implemented bulk encryption and decryptions. -- 9a3132c: Fixed the logtape peer dependency version. - -## 3.3.0 - -### Minor Changes - -- 80ee5af: Fixed bugs when implmenting the lock context with CTS v2 tokens. - -## 3.2.0 - -### Minor Changes - -- 0526f60: Use the latest jseql-ffi (0.4.0) -- fbb2bcb: Implemented CTS v2 for identity lock. - -## 3.1.0 - -### Minor Changes - -- 71ce612: Released support for LockContext initializer. -- e484718: Refactored init function to not require envrionment variables as arguments. -- e484718: Replaces jset with vitest for better typescript support. - -## 3.0.0 - -### Major Changes - -- 2eefb5f: Implemented jseql-ffi for inline crypto. - -## 2.1.0 - -### Minor Changes - -- 0536f03: Implemented new CsPlaintextV1Schema type and schema. - -## 2.0.0 - -### Major Changes - -- bea60c4: Added release management. - -## 1.0.0 - -### Major Changes - -- Released the initial version of jseql. diff --git a/packages/protect/README.md b/packages/protect/README.md deleted file mode 100644 index 06e45b53e..000000000 --- a/packages/protect/README.md +++ /dev/null @@ -1,1102 +0,0 @@ -

- CipherStash Logo -
- Protect.js -

-

- End-to-end field level encryption for JavaScript/TypeScript apps with zero‑knowledge key management. Search encrypted data without decrypting it. -
-

-

-
⭐ Please star this repo if you find it useful!
-
- - - -> [!TIP] -> `@cipherstash/protect` is the core encryption library that powers [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack). For new projects we recommend `@cipherstash/stack`, which re-exports this functionality alongside integrations for Drizzle, Supabase, DynamoDB, secrets, and identity-aware encryption. See the [CipherStash docs](https://cipherstash.com/docs) for the current API. - -Protect.js lets you encrypt every value with its own key—without sacrificing performance or usability. Encryption happens in your app; ciphertext is stored in your database. - -Per‑value unique keys are powered by CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms) bulk key operations, backed by a root key in [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html). - -Encrypted data is structured as an [EQL](https://github.com/cipherstash/encrypt-query-language) JSON payload and can be stored in any database that supports JSONB. - -> [!IMPORTANT] -> Searching, sorting, and filtering on encrypted data is currently only supported when storing encrypted data in PostgreSQL. -> Read more about [searching encrypted data](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption). - -Looking for DynamoDB support? Check out the [Protect.js for DynamoDB helper library](https://www.npmjs.com/package/@cipherstash/protect-dynamodb). - -## Quick start (60 seconds) - -Create an account and workspace in the [CipherStash dashboard](https://cipherstash.com/signup), then follow the onboarding guide to generate your client credentials and store them in your `.env` file. - -Install the package: - -```bash -npm install @cipherstash/protect -``` - -Start encrypting data: - -```ts -import { protect } from "@cipherstash/protect"; -import { csTable, csColumn } from "@cipherstash/protect"; - -// 1) Define a schema -const users = csTable("users", { email: csColumn("email") }); - -// 2) Create a client (requires CS_* env vars) -const client = await protect({ schemas: [users] }); - -// 3) Encrypt → store JSONB payload -const encrypted = await client.encrypt("alice@example.com", { - table: users, - column: users.email, -}); - -if (encrypted.failure) { - // You decide how to handle the failure and the user experience -} - -// 4) Decrypt later -const decrypted = await client.decrypt(encrypted.data); -``` - -## Architecture (high level) - -![Protect.js Architecture Diagram](https://raw.githubusercontent.com/cipherstash/stack/45e40dc888ddc412c361139192ed24668d3db60d/docs/images/protectjs-architecture.png) - -## Table of contents - -- [Quick start (60 seconds)](#quick-start-60-seconds) -- [Architecture (high level)](#architecture-high-level) -- [Features](#features) -- [Installing Protect.js](#installing-protectjs) -- [Getting started](#getting-started) -- [Identity-aware encryption](#identity-aware-encryption) -- [Supported data types](#supported-data-types) -- [Searchable encryption](#searchable-encryption) -- [Multi-tenant encryption](#multi-tenant-encryption) -- [Logging](#logging) -- [CipherStash Client](#cipherstash-client) -- [Example applications](#example-applications) -- [Builds and bundling](#builds-and-bundling) -- [Contributing](#contributing) -- [License](#license) - -For more specific documentation, refer to the [docs](https://cipherstash.com/docs). - -## Features - -Protect.js protects data in using industry-standard AES encryption. -Protect.js uses [ZeroKMS](https://cipherstash.com/products/zerokms) for bulk encryption and decryption operations. -This enables every encrypted value, in every column, in every row in your database to have a unique key — without sacrificing performance. - -**Features:** - -- **Bulk encryption and decryption**: Protect.js uses [ZeroKMS](https://cipherstash.com/products/zerokms) for encrypting and decrypting thousands of records at once, while using a unique key for every value. -- **Single item encryption and decryption**: Just looking for a way to encrypt and decrypt single values? Protect.js has you covered. -- **Really fast:** ZeroKMS's performance makes using millions of unique keys feasible and performant for real-world applications built with Protect.js. -- **Identity-aware encryption**: Lock down access to sensitive data by requiring a valid JWT to perform a decryption. -- **Audit trail**: Every decryption event will be logged in ZeroKMS to help you prove compliance. -- **Searchable encryption**: Protect.js supports searching encrypted data in PostgreSQL. -- **TypeScript support**: Strongly typed with TypeScript interfaces and types. - -**Use cases:** - -- **Trusted data access**: make sure only your end-users can access their sensitive data stored in your product. -- **Meet compliance requirements faster:** meet and exceed the data encryption requirements of SOC2 and ISO27001. -- **Reduce the blast radius of data breaches:** limit the impact of exploited vulnerabilities to only the data your end-users can decrypt. - -## Installing Protect.js - -Install the [`@cipherstash/protect` package](https://www.npmjs.com/package/@cipherstash/protect) with your package manager of choice: - -```bash -npm install @cipherstash/protect -# or -yarn add @cipherstash/protect -# or -pnpm add @cipherstash/protect -``` - -> [!TIP] -> [Bun](https://bun.sh/) is not currently supported due to a lack of [Node-API compatibility](https://github.com/oven-sh/bun/issues/158). Under the hood, Protect.js uses [CipherStash Client](#cipherstash-client) which is written in Rust and embedded using [Neon](https://github.com/neon-bindings/neon). - -### Opt-out of bundling - -> [!IMPORTANT] -> **You need to opt-out of bundling when using Protect.js.** - -Protect.js uses Node.js specific features and requires the use of the [native Node.js `require`](https://nodejs.org/api/modules.html#requireid). - -When using Protect.js, you need to opt-out of bundling for tools like [Webpack](https://webpack.js.org/configuration/externals/), [esbuild](https://webpack.js.org/configuration/externals/), or [Next.js](https://nextjs.org/docs/app/api-reference/config/next-config-js/serverExternalPackages). - -Read more about [building and bundling with Protect.js](#builds-and-bundling). - -## Getting started - -- 🆕 **Existing app?** Skip to [the next step](#configuration). -- 🌱 **Clean slate?** Check out the [getting started tutorial](https://cipherstash.com/docs/stack/quickstart). - -### Configuration - -If you haven't already, sign up for a [CipherStash account](https://cipherstash.com/signup). -Once you have an account, you will create a Workspace which is scoped to your application environment. - -Follow the onboarding steps to get your first set of credentials required to use Protect.js. -By the end of the onboarding, you will have the following environment variables: - -```bash -CS_WORKSPACE_CRN= # The workspace identifier -CS_CLIENT_ID= # The client identifier -CS_CLIENT_KEY= # The client key which is used as key material in combination with ZeroKMS -CS_CLIENT_ACCESS_KEY= # The API key used for authenticating with the CipherStash API -``` - -Save these environment variables to a `.env` file in your project. - -### Basic file structure - -The following is the basic file structure of the project. -In the `src/protect/` directory, we have the table definition in `schema.ts` and the protect client in `index.ts`. - -``` -📦 - ├ 📂 src - │ ├ 📂 protect - │ │ ├ 📜 index.ts - │ │ └ 📜 schema.ts - │ └ 📜 index.ts - ├ 📜 .env - ├ 📜 cipherstash.toml - ├ 📜 cipherstash.secret.toml - ├ 📜 package.json - └ 📜 tsconfig.json -``` - -### Define your schema - -Protect.js uses a schema to define the tables and columns that you want to encrypt and decrypt. - -Define your tables and columns by adding this to `src/protect/schema.ts`: - -```ts -import { csTable, csColumn } from "@cipherstash/protect"; - -export const users = csTable("users", { - email: csColumn("email"), -}); - -export const orders = csTable("orders", { - address: csColumn("address"), -}); -``` - -**Searchable encryption:** - -If you want to search encrypted data in your PostgreSQL database, you must declare the indexes in schema in `src/protect/schema.ts`: - -```ts -import { csTable, csColumn } from "@cipherstash/protect"; - -export const users = csTable("users", { - email: csColumn("email").freeTextSearch().equality().orderAndRange(), -}); - -export const orders = csTable("orders", { - address: csColumn("address"), -}); - -export const documents = csTable("documents", { - metadata: csColumn("metadata").searchableJson(), // Enables encrypted JSONB queries -}); -``` - -Read more about [defining your schema](https://cipherstash.com/docs/stack/cipherstash/encryption/schema). - -### Initialize the Protect client - -To import the `protect` function and initialize a client with your defined schema, add the following to `src/protect/index.ts`: - -```ts -import { protect, type ProtectClientConfig } from "@cipherstash/protect"; -import { users, orders } from "./schema"; - -const config: ProtectClientConfig = { - schemas: [users, orders], -} - -// Pass all your tables to the protect function to initialize the client -export const protectClient = await protect(config); -``` - -The `protect` function requires at least one `csTable` be provided in the `schemas` array. - -### Encrypt data - -Protect.js provides the `encrypt` function on `protectClient` to encrypt data. -`encrypt` takes a plaintext string, and an object with the table and column as parameters. - -To start encrypting data, add the following to `src/index.ts`: - -```typescript -import { users } from "./protect/schema"; -import { protectClient } from "./protect"; - -const encryptResult = await protectClient.encrypt("secret@squirrel.example", { - column: users.email, - table: users, -}); - -if (encryptResult.failure) { - // Handle the failure - console.log( - "error when encrypting:", - encryptResult.failure.type, - encryptResult.failure.message - ); -} - -console.log("EQL Payload containing ciphertexts:", encryptResult.data); -``` - -The `encrypt` function will return a `Result` object with either a `data` key, or a `failure` key. -The `encryptResult` will return one of the following: - -```typescript -// Success -{ - data: EncryptedPayload -} - -// Failure -{ - failure: { - type: 'EncryptionError', - message: 'A message about the error' - } -} -``` - -### Decrypt data - -Protect.js provides the `decrypt` function on `protectClient` to decrypt data. -`decrypt` takes an encrypted data object as a parameter. - -To start decrypting data, add the following to `src/index.ts`: - -```typescript -import { protectClient } from "./protect"; - -// encryptResult is the EQL payload from the previous step -const decryptResult = await protectClient.decrypt(encryptResult.data); - -if (decryptResult.failure) { - // Handle the failure - console.log( - "error when decrypting:", - decryptResult.failure.type, - decryptResult.failure.message - ); -} - -const plaintext = decryptResult.data; -console.log("plaintext:", plaintext); -``` - -The `decrypt` function returns a `Result` object with either a `data` key, or a `failure` key. -The `decryptResult` will return one of the following: - -```typescript -// Success -{ - data: 'secret@squirrel.example' -} - -// Failure -{ - failure: { - type: 'DecryptionError', - message: 'A message about the error' - } -} -``` - -### Working with models and objects - -Protect.js provides model-level encryption methods that make it easy to encrypt and decrypt entire objects. -These methods automatically handle the encryption of fields defined in your schema. - -If you are working with a large data set, the model operations are significantly faster than encrypting and decrypting individual objects as they are able to perform bulk operations. - -> [!TIP] -> CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms) is optimized for bulk operations. -> -> All the model operations are able to take advantage of this performance for real-world use cases by only making a single call to ZeroKMS regardless of the number of objects you are encrypting or decrypting while still using a unique key for each record. - -#### Encrypting a model - -Use the `encryptModel` method to encrypt a model's fields that are defined in your schema: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -// Your model with plaintext values -const user = { - id: "1", - email: "user@example.com", - address: "123 Main St", - createdAt: new Date("2024-01-01"), -}; - -const encryptedResult = await protectClient.encryptModel(user, users); - -if (encryptedResult.failure) { - // Handle the failure - console.log( - "error when encrypting:", - encryptedResult.failure.type, - encryptedResult.failure.message - ); -} - -const encryptedUser = encryptedResult.data; -console.log("encrypted user:", encryptedUser); -``` - -The `encryptModel` function will only encrypt fields that are defined in your schema. -Other fields (like `id` and `createdAt` in the example above) will remain unchanged. - -#### Type safety with models - -Protect.js provides strong TypeScript support for model operations. -You can specify your model's type to ensure end-to-end type safety: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -// Define your model type -type User = { - id: string; - email: string | null; - address: string | null; - createdAt: Date; - updatedAt: Date; - metadata?: { - preferences?: { - notifications: boolean; - theme: string; - }; - }; -}; - -// The encryptModel method will ensure type safety -const encryptedResult = await protectClient.encryptModel(user, users); - -if (encryptedResult.failure) { - // Handle the failure -} - -const encryptedUser = encryptedResult.data; -// TypeScript knows that encryptedUser matches the User type structure -// but with encrypted fields for those defined in the schema - -// Decryption maintains type safety -const decryptedResult = await protectClient.decryptModel(encryptedUser); - -if (decryptedResult.failure) { - // Handle the failure -} - -const decryptedUser = decryptedResult.data; -// decryptedUser is fully typed as User - -// Bulk operations also support type safety -const bulkEncryptedResult = await protectClient.bulkEncryptModels( - userModels, - users -); - -const bulkDecryptedResult = await protectClient.bulkDecryptModels( - bulkEncryptedResult.data -); -``` - -The type system ensures that: - -- Input models match your defined type structure -- Only fields defined in your schema are encrypted -- Encrypted and decrypted results maintain the correct type structure -- Optional and nullable fields are properly handled -- Nested object structures are preserved -- Additional properties not defined in the schema remain unchanged - -This type safety helps catch potential issues at compile time and provides better IDE support with autocompletion and type hints. - -> [!TIP] -> When using TypeScript with an ORM, you can reuse your ORM's model types directly with Protect.js's model operations. - -Example with Drizzle infered types: - -```typescript -import { protectClient } from "./protect"; -import { jsonb, pgTable, serial, InferSelectModel } from "drizzle-orm/pg-core"; -import { csTable, csColumn } from "@cipherstash/protect"; - -const protectUsers = csTable("users", { - email: csColumn("email"), -}); - -const users = pgTable("users", { - id: serial("id").primaryKey(), - email: jsonb("email").notNull(), -}); - -type User = InferSelectModel; - -const user = { - id: "1", - email: "user@example.com", -}; - -// Drizzle User type works directly with model operations -const encryptedResult = await protectClient.encryptModel( - user, - protectUsers -); -``` - -#### Decrypting a model - -Use the `decryptModel` method to decrypt a model's encrypted fields: - -```typescript -import { protectClient } from "./protect"; - -const decryptedResult = await protectClient.decryptModel(encryptedUser); - -if (decryptedResult.failure) { - // Handle the failure - console.log( - "error when decrypting:", - decryptedResult.failure.type, - decryptedResult.failure.message - ); -} - -const decryptedUser = decryptedResult.data; -console.log("decrypted user:", decryptedUser); -``` - -#### Bulk model operations - -For better performance when working with multiple models, use the `bulkEncryptModels` and `bulkDecryptModels` methods: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -// Array of models with plaintext values -const userModels = [ - { - id: "1", - email: "user1@example.com", - address: "123 Main St", - }, - { - id: "2", - email: "user2@example.com", - address: "456 Oak Ave", - }, -]; - -// Encrypt multiple models at once -const encryptedResult = await protectClient.bulkEncryptModels( - userModels, - users -); - -if (encryptedResult.failure) { - // Handle the failure -} - -const encryptedUsers = encryptedResult.data; - -// Decrypt multiple models at once -const decryptedResult = await protectClient.bulkDecryptModels(encryptedUsers); - -if (decryptedResult.failure) { - // Handle the failure -} - -const decryptedUsers = decryptedResult.data; -``` - -The model encryption methods provide a higher-level interface that's particularly useful when working with ORMs or when you need to encrypt multiple fields in an object. -They automatically handle the mapping between your model's structure and the encrypted fields defined in your schema. - -### Bulk operations - -Protect.js provides direct access to ZeroKMS bulk operations through the `bulkEncrypt` and `bulkDecrypt` methods. These methods are ideal when you need maximum performance and want to handle the correlation between encrypted/decrypted values and your application data manually. - -> [!TIP] -> The bulk operations provide the most direct interface to ZeroKMS's blazing fast bulk encryption and decryption capabilities. Each value gets a unique key while maintaining optimal performance through a single call to ZeroKMS. - -#### Bulk encryption - -Use the `bulkEncrypt` method to encrypt multiple plaintext values at once: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -// Array of plaintext values with optional IDs for correlation -const plaintexts = [ - { id: "user1", plaintext: "alice@example.com" }, - { id: "user2", plaintext: "bob@example.com" }, - { id: "user3", plaintext: "charlie@example.com" }, -]; - -const encryptedResult = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, -}); - -if (encryptedResult.failure) { - // Handle the failure - console.log( - "error when bulk encrypting:", - encryptedResult.failure.type, - encryptedResult.failure.message - ); -} - -const encryptedData = encryptedResult.data; -console.log("encrypted data:", encryptedData); -``` - -The `bulkEncrypt` method returns an array of objects with the following structure: - -```typescript -[ - { id: "user1", data: EncryptedPayload }, - { id: "user2", data: EncryptedPayload }, - { id: "user3", data: EncryptedPayload }, -] -``` - -You can also encrypt without IDs if you don't need correlation: - -```typescript -const plaintexts = [ - { plaintext: "alice@example.com" }, - { plaintext: "bob@example.com" }, - { plaintext: "charlie@example.com" }, -]; - -const encryptedResult = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, -}); -``` - -#### Bulk decryption - -Use the `bulkDecrypt` method to decrypt multiple encrypted values at once: - -```typescript -import { protectClient } from "./protect"; - -// encryptedData is the result from bulkEncrypt -const decryptedResult = await protectClient.bulkDecrypt(encryptedData); - -if (decryptedResult.failure) { - // Handle the failure - console.log( - "error when bulk decrypting:", - decryptedResult.failure.type, - decryptedResult.failure.message - ); -} - -const decryptedData = decryptedResult.data; -console.log("decrypted data:", decryptedData); -``` - -The `bulkDecrypt` method returns an array of objects with the following structure: - -```typescript -[ - { id: "user1", data: "alice@example.com" }, - { id: "user2", data: "bob@example.com" }, - { id: "user3", data: "charlie@example.com" }, -] -``` - -#### Response structure - -The `bulkDecrypt` method returns a `Result` object that represents the overall operation status. When successful from an HTTP and execution perspective, the `data` field contains an array where each item can have one of two outcomes: - -- **Success**: The item has a `data` field containing the decrypted plaintext -- **Failure**: The item has an `error` field containing a specific error message explaining why that particular decryption failed - -```typescript -// Example response structure -{ - data: [ - { id: "user1", data: "alice@example.com" }, // Success - { id: "user2", error: "Invalid ciphertext format" }, // Failure - { id: "user3", data: "charlie@example.com" }, // Success - ] -} -``` - -> [!NOTE] -> The underlying ZeroKMS response uses HTTP status code 207 (Multi-Status) to indicate that the bulk operation completed, but individual items within the batch may have succeeded or failed. This allows you to handle partial failures gracefully while still processing the successful decryptions. - -You can handle mixed results by checking each item: - -```typescript -const decryptedResult = await protectClient.bulkDecrypt(encryptedData); - -if (decryptedResult.failure) { - // Handle overall operation failure - console.log("Bulk decryption failed:", decryptedResult.failure.message); - return; -} - -// Process individual results -decryptedResult.data.forEach((item) => { - if ('data' in item) { - // Success - item.data contains the decrypted plaintext - console.log(`Decrypted ${item.id}:`, item.data); - } else if ('error' in item) { - // Failure - item.error contains the specific error message - console.log(`Failed to decrypt ${item.id}:`, item.error); - } -}); -``` - -#### Handling null values - -Bulk operations properly handle null values in both encryption and decryption: - -```typescript -const plaintexts = [ - { id: "user1", plaintext: "alice@example.com" }, - { id: "user2", plaintext: null }, - { id: "user3", plaintext: "charlie@example.com" }, -]; - -const encryptedResult = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, -}); - -// Null values are preserved in the encrypted result -// encryptedResult.data[1].data will be null - -const decryptedResult = await protectClient.bulkDecrypt(encryptedResult.data); - -// Null values are preserved in the decrypted result -// decryptedResult.data[1].data will be null -``` - -#### Using bulk operations with lock contexts - -Bulk operations support identity-aware encryption through lock contexts: - -```typescript -import { LockContext } from "@cipherstash/protect/identify"; - -const lc = new LockContext(); -const lockContext = await lc.identify(userJwt); - -if (lockContext.failure) { - // Handle the failure -} - -const plaintexts = [ - { id: "user1", plaintext: "alice@example.com" }, - { id: "user2", plaintext: "bob@example.com" }, -]; - -// Encrypt with lock context -const encryptedResult = await protectClient - .bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - .withLockContext(lockContext.data); - -// Decrypt with lock context -const decryptedResult = await protectClient - .bulkDecrypt(encryptedResult.data) - .withLockContext(lockContext.data); -``` - -#### Performance considerations - -Bulk operations are optimized for performance and can handle thousands of values efficiently: - -```typescript -// Create a large array of values -const plaintexts = Array.from({ length: 1000 }, (_, i) => ({ - id: `user${i}`, - plaintext: `user${i}@example.com`, -})); - -// Single call to ZeroKMS for all 1000 values -const encryptedResult = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, -}); - -// Single call to ZeroKMS for all 1000 values -const decryptedResult = await protectClient.bulkDecrypt(encryptedResult.data); -``` - -The bulk operations maintain the same security guarantees as individual operations - each value gets a unique key - while providing optimal performance through ZeroKMS's bulk processing capabilities. - -### Store encrypted data in a database - -Encrypted data can be stored in any database that supports JSONB, noting that searchable encryption is only supported in PostgreSQL at the moment. - -To store the encrypted data, specify the column type as `jsonb`. - -```sql -CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email jsonb NOT NULL, -); -``` - -#### Searchable encryption in PostgreSQL - -To enable searchable encryption in PostgreSQL, [install the EQL custom types and functions](https://github.com/cipherstash/encrypt-query-language?tab=readme-ov-file#installation). - -1. Download the EQL install script. The version is pinned to match this release of Protect.js — install exactly this version: - - ```sh - curl -sLo cipherstash-encrypt.sql https://github.com/cipherstash/encrypt-query-language/releases/download/eql-2.3.1/cipherstash-encrypt.sql - ``` - - Using [Supabase](https://supabase.com/)? We ship an EQL release specifically for Supabase. - Download the matching Supabase EQL install script: - - ```sh - curl -sLo cipherstash-encrypt-supabase.sql https://github.com/cipherstash/encrypt-query-language/releases/download/eql-2.3.1/cipherstash-encrypt-supabase.sql - ``` - -2. Run this command to install the custom types and functions: - - ```sh - psql -f cipherstash-encrypt.sql - ``` - - or with Supabase: - - ```sh - psql -f cipherstash-encrypt-supabase.sql - ``` - -EQL is now installed in your database and you can enable searchable encryption by adding the `eql_v2_encrypted` type to a column. - -```sql -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v2_encrypted -); -``` - -> [!WARNING] -> The `eql_v2_encrypted` type is a [composite type](https://www.postgresql.org/docs/current/rowtypes.html) and each ORM/client has a different way of handling inserts and selects. -> We've documented how to handle inserts and selects for the different ORMs/clients in the [docs](https://cipherstash.com/docs). - -Read more about [how to search encrypted data](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption) in the docs. - -## Identity-aware encryption - -> [!IMPORTANT] -> Right now identity-aware encryption is only supported if you are using [Clerk](https://clerk.com/) as your identity provider. -> Read more about [lock contexts with Clerk and Next.js](https://cipherstash.com/docs/stack/cipherstash/encryption/identity). - -Protect.js can add an additional layer of protection to your data by requiring a valid JWT to perform a decryption. - -This ensures that only the user who encrypted data is able to decrypt it. - -Protect.js does this through a mechanism called a _lock context_. - -### Lock context - -Lock contexts ensure that only specific users can access sensitive data. - -> [!CAUTION] -> You must use the same lock context to encrypt and decrypt data. -> If you use different lock contexts, you will be unable to decrypt the data. - -To use a lock context, initialize a `LockContext` object with the identity claims. - -```typescript -import { LockContext } from "@cipherstash/protect/identify"; - -// protectClient from the previous steps -const lc = new LockContext(); -``` - -> [!NOTE] -> When initializing a `LockContext`, the default context is set to use the `sub` Identity Claim. - -### Identifying a user for a lock context - -A lock context needs to be locked to a user. -To identify the user, call the `identify` method on the lock context object, and pass a valid JWT from a user's session: - -```typescript -const identifyResult = await lc.identify(jwt); - -// The identify method returns the same Result pattern as the encrypt and decrypt methods. -if (identifyResult.failure) { - // Hanlde the failure -} - -const lockContext = identifyResult.data; -``` - -### Encrypting data with a lock context - -To encrypt data with a lock context, call the optional `withLockContext` method on the `encrypt` function and pass the lock context object as a parameter: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -const encryptResult = await protectClient - .encrypt("plaintext", { - table: users, - column: users.email, - }) - .withLockContext(lockContext); - -if (encryptResult.failure) { - // Handle the failure -} - -console.log("EQL Payload containing ciphertexts:", encryptResult.data); -``` - -### Decrypting data with a lock context - -To decrypt data with a lock context, call the optional `withLockContext` method on the `decrypt` function and pass the lock context object as a parameter: - -```typescript -import { protectClient } from "./protect"; - -const decryptResult = await protectClient - .decrypt(encryptResult.data) - .withLockContext(lockContext); - -if (decryptResult.failure) { - // Handle the failure -} - -const plaintext = decryptResult.data; -``` - -### Model encryption with lock context - -All model operations support lock contexts for identity-aware encryption: - -```typescript -import { protectClient } from "./protect"; -import { users } from "./protect/schema"; - -const myUsers = [ - { - id: "1", - email: "user@example.com", - address: "123 Main St", - createdAt: new Date("2024-01-01"), - }, - { - id: "2", - email: "user2@example.com", - address: "456 Oak Ave", - }, -]; - -// Encrypt a model with lock context -const encryptedResult = await protectClient - .encryptModel(myUsers[0], users) - .withLockContext(lockContext); - -if (encryptedResult.failure) { - // Handle the failure -} - -// Decrypt a model with lock context -const decryptedResult = await protectClient - .decryptModel(encryptedResult.data) - .withLockContext(lockContext); - -// Bulk operations also support lock contexts -const bulkEncryptedResult = await protectClient - .bulkEncryptModels(myUsers, users) - .withLockContext(lockContext); - -const bulkDecryptedResult = await protectClient - .bulkDecryptModels(bulkEncryptedResult.data) - .withLockContext(lockContext); -``` - -## Supported data types - -Protect.js currently supports encrypting and decrypting text. -Other data types like booleans, dates, ints, floats, and JSON are well-supported in other CipherStash products, and will be coming to Protect.js soon. - -Until support for other data types are available, you can express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/stack/issues/48). - -## Searchable encryption - -Protect.js supports searching encrypted data in PostgreSQL: - -- **Text columns**: Use `.equality()`, `.freeTextSearch()`, and `.orderAndRange()` for exact match, text search, and sorting/range queries. -- **JSONB columns**: Use `.searchableJson()` (recommended) to enable encrypted JSONPath selector and containment queries on JSON data. The query operation is automatically inferred from the plaintext type. - -Read more about [searching encrypted data](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption) and the [PostgreSQL implementation details](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption) in the docs. - -## Multi-tenant encryption - -Protect.js supports multi-tenant encryption by using keysets. -Each keyset is cryptographically isolated from other keysets which esentially means that each tenant has their own unique keyspace. -If you are using a multi-tenant application, you can use keysets to encrypt data for each tenant creating a strong security boundary. - -In the [CipherStash Dashboard](https://dashboard.cipherstash.com/workspaces/_/encryption/keysets), you can create and manage keysets and then use the keyset identifier to encrypt data for each tenant when initializing the Protect.js client. - -```typescript -import { protect } from "@cipherstash/protect"; -import { users } from "./protect/schema"; - -const protectClient = await protect({ - schemas: [users], - keyset: { - // Must be a valid UUID which can be found in the CipherStash Dashboard - id: '123e4567-e89b-12d3-a456-426614174000' - }, -}) - -// or with a keyset name - -const protectClient = await protect({ - schemas: [users], - keyset: { - name: 'Company A' - }, -}) -``` - -> [!IMPORTANT] -> When creating a new keyset, make sure to grant your client access to the keyset or client initialization will fail. -> Read more about [managing keyset access](https://cipherstash.com/docs/platform/workspaces/key-sets). - -## Logging - -> [!TIP] -> `@cipherstash/protect` will NEVER log plaintext data. -> This is by design to prevent sensitive data from leaking into logs. - -`@cipherstash/protect` and `@cipherstash/nextjs` will log to the console with a log level of `info` by default. -To enable the logger, configure the following environment variable: - -```bash -PROTECT_LOG_LEVEL=debug # Enable debug logging -PROTECT_LOG_LEVEL=info # Enable info logging -PROTECT_LOG_LEVEL=error # Enable error logging -``` - -## CipherStash Client - -Protect.js is built on top of the CipherStash Client Rust SDK which is embedded with the `@cipherstash/protect-ffi` package. -The `@cipherstash/protect-ffi` source code is available on [GitHub](https://github.com/cipherstash/stack-ffi). - -Read more about configuring the CipherStash Client in the [configuration docs](https://cipherstash.com/docs/stack/reference). - -## Example applications - -Looking for examples of how to use Protect.js? -Check out the [example applications](./examples): - -- [Basic example](/examples/basic) demonstrates how to perform encryption operations -- [Drizzle example](/examples/drizzle) demonstrates how to use Protect.js with an ORM -- [Next.js and lock contexts example using Clerk](/examples/nextjs-clerk) demonstrates how to protect data with identity-aware encryption - -`@cipherstash/protect` can be used with most ORMs. -If you're interested in using `@cipherstash/protect` with a specific ORM, please [create an issue](https://github.com/cipherstash/stack/issues/new). - -## Builds and bundling - -`@cipherstash/protect` is a native Node.js module, and relies on native Node.js `require` to load the package. - -Here are a few resources to help based on your tool set: - -- [Required Next.js configuration](https://cipherstash.com/docs/stack/deploy/bundling). -- [SST and AWS serverless functions](https://cipherstash.com/docs/stack/deploy/bundling). - -> [!TIP] -> Deploying to Linux (e.g., AWS Lambda) with npm lockfile v3 and seeing runtime module load errors? See the troubleshooting guide: [bundling guide](https://cipherstash.com/docs/stack/deploy/bundling). - -## Contributing - -Please read the [contribution guide](CONTRIBUTE.md). - -## License - -Protect.js is [MIT licensed](./LICENSE.md). - ---- - -### Didn't find what you wanted? - -[Click here to let us know what was missing from our docs.](https://github.com/cipherstash/stack/issues/new?template=docs-feedback.yml&title=[Docs:]%20Feedback%20on%20README.md) diff --git a/packages/protect/__tests__/audit.test.ts b/packages/protect/__tests__/audit.test.ts deleted file mode 100644 index 6d508f585..000000000 --- a/packages/protect/__tests__/audit.test.ts +++ /dev/null @@ -1,472 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { LockContext, protect } from '../src' - -const users = csTable('users', { - auditable: csColumn('auditable'), - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), -}) - -type User = { - id: string - email?: string | null - address?: string | null - auditable?: string | null - createdAt?: Date - updatedAt?: Date - number?: number -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('encryption and decryption with audit', () => { - it('should encrypt and decrypt a payload with audit metadata', async () => { - const email = 'very_secret_data' - - const ciphertext = await protectClient - .encrypt(email, { - column: users.auditable, - table: users, - }) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'encrypt', - }, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const plaintext = await protectClient.decrypt(ciphertext.data).audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'decrypt', - }, - }) - - expect(plaintext).toEqual({ - data: email, - }) - }, 30000) - - it('should encrypt and decrypt a model with audit metadata', async () => { - // Create a model with decrypted values - const decryptedModel: User = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - auditable: 'sensitive_data', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - } - - // Encrypt the model with audit - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'encrypt_model', - }, - }) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.auditable).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.number).toBe(1) - - // Decrypt the model with audit - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'decrypt_model', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null values in a model with audit metadata', async () => { - // Create a model with null values - const decryptedModel: User = { - id: '1', - email: null, - address: null, - auditable: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - } - - // Encrypt the model with audit - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'encrypt_model_nulls', - }, - }) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify null fields are preserved - expect(encryptedModel.data.email).toBeNull() - expect(encryptedModel.data.address).toBeNull() - expect(encryptedModel.data.auditable).toBeNull() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.number).toBe(1) - - // Decrypt the model with audit - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'decrypt_model_nulls', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) -}) - -describe('bulk encryption with audit', () => { - it('should bulk encrypt and decrypt models with audit metadata', async () => { - // Create models with decrypted values - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - address: '123 Main St', - auditable: 'sensitive_data_1', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - }, - { - id: '2', - email: 'test2@example.com', - address: '456 Oak St', - auditable: 'sensitive_data_2', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - }, - ] - - // Encrypt the models with audit - const encryptedModels = await protectClient - .bulkEncryptModels(decryptedModels, users) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_encrypt_models', - }, - }) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toHaveProperty('c') - expect(encryptedModels.data[0].auditable).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[1].auditable).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].number).toBe(1) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].number).toBe(2) - - // Decrypt the models with audit - const decryptedResult = await protectClient - .bulkDecryptModels(encryptedModels.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_decrypt_models', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) - - it('should handle mixed null and non-null values in bulk operations with audit', async () => { - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - address: null, - auditable: 'sensitive_data_1', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - }, - { - id: '2', - email: null, - address: '123 Main St', - auditable: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - }, - { - id: '3', - email: 'test3@example.com', - address: '456 Oak St', - auditable: 'sensitive_data_3', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 3, - }, - ] - - // Encrypt the models with audit - const encryptedModels = await protectClient - .bulkEncryptModels(decryptedModels, users) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_encrypt_mixed_nulls', - }, - }) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toBeNull() - expect(encryptedModels.data[0].auditable).toHaveProperty('c') - expect(encryptedModels.data[1].email).toBeNull() - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[1].auditable).toBeNull() - expect(encryptedModels.data[2].email).toHaveProperty('c') - expect(encryptedModels.data[2].address).toHaveProperty('c') - expect(encryptedModels.data[2].auditable).toHaveProperty('c') - - // Decrypt the models with audit - const decryptedResult = await protectClient - .bulkDecryptModels(decryptedModels) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_decrypt_mixed_nulls', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) - - it('should return empty array if models is empty with audit', async () => { - // Encrypt empty array of models with audit - const encryptedModels = await protectClient - .bulkEncryptModels([], users) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_encrypt_empty', - }, - }) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - expect(encryptedModels.data).toEqual([]) - - // Decrypt empty array of models with audit - const decryptedResult = await protectClient - .bulkDecryptModels([]) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_decrypt_empty', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual([]) - }, 30000) -}) - -describe('audit with lock context', () => { - it('should encrypt and decrypt a model with both audit and lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create a model with decrypted values - const decryptedModel: User = { - id: '1', - email: 'test@example.com', - auditable: 'sensitive_with_context', - } - - // Encrypt the model with both audit and lock context - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'encrypt_with_context', - }, - }) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Decrypt the model with both audit and lock context - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .withLockContext(lockContext.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'decrypt_with_context', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should bulk encrypt and decrypt models with both audit and lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create models with decrypted values - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - auditable: 'bulk_sensitive_1', - }, - { - id: '2', - email: 'test2@example.com', - auditable: 'bulk_sensitive_2', - }, - ] - - // Encrypt the models with both audit and lock context - const encryptedModels = await protectClient - .bulkEncryptModels(decryptedModels, users) - .withLockContext(lockContext.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_encrypt_with_context', - }, - }) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Decrypt the models with both audit and lock context - const decryptedResult = await protectClient - .bulkDecryptModels(encryptedModels.data) - .withLockContext(lockContext.data) - .audit({ - metadata: { - sub: 'cj@cjb.io', - type: 'bulk_decrypt_with_context', - }, - }) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) -}) diff --git a/packages/protect/__tests__/basic-protect.test.ts b/packages/protect/__tests__/basic-protect.test.ts deleted file mode 100644 index 6277a842a..000000000 --- a/packages/protect/__tests__/basic-protect.test.ts +++ /dev/null @@ -1,44 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - json: csColumn('json').dataType('json'), -}) - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('encryption and decryption', () => { - it('should encrypt and decrypt a payload', async () => { - const email = 'hello@example.com' - - const ciphertext = await protectClient.encrypt(email, { - column: users.email, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const a = ciphertext.data - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: email, - }) - }, 30000) -}) diff --git a/packages/protect/__tests__/bulk-protect.test.ts b/packages/protect/__tests__/bulk-protect.test.ts deleted file mode 100644 index 893bea86a..000000000 --- a/packages/protect/__tests__/bulk-protect.test.ts +++ /dev/null @@ -1,597 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { type EncryptedPayload, LockContext, protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), -}) - -type User = { - id: string - email?: string | null - createdAt?: Date - updatedAt?: Date - address?: string | null - number?: number -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('bulk encryption and decryption', () => { - describe('bulk encrypt', () => { - it('should bulk encrypt an array of plaintexts with IDs', async () => { - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: 'bob@example.com' }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - - // Verify all encrypted values are different - expect(encryptedData.data[0].data?.c).not.toBe( - encryptedData.data[1].data?.c, - ) - expect(encryptedData.data[1].data?.c).not.toBe( - encryptedData.data[2].data?.c, - ) - expect(encryptedData.data[0].data?.c).not.toBe( - encryptedData.data[2].data?.c, - ) - }, 30000) - - it('should bulk encrypt an array of plaintexts without IDs', async () => { - const plaintexts = [ - { plaintext: 'alice@example.com' }, - { plaintext: 'bob@example.com' }, - { plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', undefined) - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', undefined) - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('c') - expect(encryptedData.data[2]).toHaveProperty('id', undefined) - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('c') - }, 30000) - - it('should handle null values in bulk encrypt', async () => { - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toBeNull() - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('c') - }, 30000) - - it('should handle all null values in bulk encrypt', async () => { - const plaintexts = [ - { id: 'user1', plaintext: null }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: null }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toBeNull() - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toBeNull() - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toBeNull() - }, 30000) - - it('should handle empty array in bulk encrypt', async () => { - const plaintexts: Array<{ id?: string; plaintext: string | null }> = [] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - expect(encryptedData.data).toHaveLength(0) - }, 30000) - }) - - describe('bulk decrypt', () => { - it('should bulk decrypt an array of encrypted payloads with IDs', async () => { - // First encrypt some data - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: 'bob@example.com' }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify structure - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 'alice@example.com') - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', 'bob@example.com') - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty( - 'data', - 'charlie@example.com', - ) - }, 30000) - - it('should bulk decrypt an array of encrypted payloads without IDs', async () => { - // First encrypt some data - const plaintexts = [ - { plaintext: 'alice@example.com' }, - { plaintext: 'bob@example.com' }, - { plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify structure - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', undefined) - expect(decryptedData.data[0]).toHaveProperty('data', 'alice@example.com') - expect(decryptedData.data[1]).toHaveProperty('id', undefined) - expect(decryptedData.data[1]).toHaveProperty('data', 'bob@example.com') - expect(decryptedData.data[2]).toHaveProperty('id', undefined) - expect(decryptedData.data[2]).toHaveProperty( - 'data', - 'charlie@example.com', - ) - }, 30000) - - it('should handle null values in bulk decrypt', async () => { - // First encrypt some data with nulls - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify structure - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 'alice@example.com') - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', null) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty( - 'data', - 'charlie@example.com', - ) - }, 30000) - - it('should handle all null values in bulk decrypt', async () => { - // First encrypt some data with all nulls - const plaintexts = [ - { id: 'user1', plaintext: null }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: null }, - ] - - const encryptedData = await protectClient.bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify structure - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', null) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', null) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty('data', null) - }, 30000) - - it('should handle empty array in bulk decrypt', async () => { - const encryptedPayloads: Array<{ id?: string; data: EncryptedPayload }> = - [] - - const decryptedData = await protectClient.bulkDecrypt(encryptedPayloads) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - expect(decryptedData.data).toHaveLength(0) - }, 30000) - }) - - describe('bulk operations with lock context', () => { - it('should bulk encrypt and decrypt with lock context', async () => { - // This test requires a valid JWT token, so we'll skip it in CI - // TODO: Add proper JWT token handling for CI - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: 'bob@example.com' }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - // Encrypt with lock context - const encryptedData = await protectClient - .bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - .withLockContext(lockContext.data) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('c') - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('c') - - // Decrypt with lock context - const decryptedData = await protectClient - .bulkDecrypt(encryptedData.data) - .withLockContext(lockContext.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 'alice@example.com') - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', 'bob@example.com') - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty( - 'data', - 'charlie@example.com', - ) - }, 30000) - - it('should handle null values with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const plaintexts = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: 'charlie@example.com' }, - ] - - // Encrypt with lock context - const encryptedData = await protectClient - .bulkEncrypt(plaintexts, { - column: users.email, - table: users, - }) - .withLockContext(lockContext.data) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify null is preserved - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toBeNull() - - // Decrypt with lock context - const decryptedData = await protectClient - .bulkDecrypt(encryptedData.data) - .withLockContext(lockContext.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify null is preserved - expect(decryptedData.data[1]).toHaveProperty('data') - expect(decryptedData.data[1].data).toBeNull() - }, 30000) - - it('should decrypt mixed lock context payloads with specific lock context', async () => { - const userJwt = process.env.USER_JWT - const user2Jwt = process.env.USER_2_JWT - - if (!userJwt || !user2Jwt) { - console.log( - 'Skipping mixed lock context test - missing USER_JWT or USER_2_JWT', - ) - return - } - - const lc = new LockContext() - const lc2 = new LockContext() - const lockContext1 = await lc.identify(userJwt) - const lockContext2 = await lc2.identify(user2Jwt) - - if (lockContext1.failure) { - throw new Error(`[protect]: ${lockContext1.failure.message}`) - } - - if (lockContext2.failure) { - throw new Error(`[protect]: ${lockContext2.failure.message}`) - } - - // Encrypt first value with USER_JWT lock context - const encryptedData1 = await protectClient - .bulkEncrypt([{ id: 'user1', plaintext: 'alice@example.com' }], { - column: users.email, - table: users, - }) - .withLockContext(lockContext1.data) - - if (encryptedData1.failure) { - throw new Error(`[protect]: ${encryptedData1.failure.message}`) - } - - // Encrypt second value with USER_2_JWT lock context - const encryptedData2 = await protectClient - .bulkEncrypt([{ id: 'user2', plaintext: 'bob@example.com' }], { - column: users.email, - table: users, - }) - .withLockContext(lockContext2.data) - - if (encryptedData2.failure) { - throw new Error(`[protect]: ${encryptedData2.failure.message}`) - } - - // Combine both encrypted payloads - const combinedEncryptedData = [ - ...encryptedData1.data, - ...encryptedData2.data, - ] - - // Decrypt with USER_2_JWT lock context - const decryptedData = await protectClient - .bulkDecrypt(combinedEncryptedData) - .withLockContext(lockContext2.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify both payloads are returned - expect(decryptedData.data).toHaveLength(2) - - // First payload should fail to decrypt since it was encrypted with different lock context - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('error') - expect(decryptedData.data[0]).not.toHaveProperty('data') - - // Second payload should be decrypted since it was encrypted with the same lock context - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', 'bob@example.com') - expect(decryptedData.data[1]).not.toHaveProperty('error') - }, 30000) - }) - - describe('bulk operations round-trip', () => { - it('should maintain data integrity through encrypt/decrypt cycle', async () => { - const originalData = [ - { id: 'user1', plaintext: 'alice@example.com' }, - { id: 'user2', plaintext: 'bob@example.com' }, - { id: 'user3', plaintext: null }, - { id: 'user4', plaintext: 'dave@example.com' }, - ] - - // Encrypt - const encryptedData = await protectClient.bulkEncrypt(originalData, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Decrypt - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify round-trip integrity - expect(decryptedData.data).toHaveLength(originalData.length) - - for (let i = 0; i < originalData.length; i++) { - expect(decryptedData.data[i].id).toBe(originalData[i].id) - expect(decryptedData.data[i].data).toBe(originalData[i].plaintext) - } - }, 30000) - - it('should handle large arrays efficiently', async () => { - const originalData = Array.from({ length: 100 }, (_, i) => ({ - id: `user${i}`, - plaintext: `user${i}@example.com`, - })) - - // Encrypt - const encryptedData = await protectClient.bulkEncrypt(originalData, { - column: users.email, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Decrypt - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify all data is preserved - expect(decryptedData.data).toHaveLength(100) - - for (let i = 0; i < 100; i++) { - expect(decryptedData.data[i].id).toBe(`user${i}`) - expect(decryptedData.data[i].data).toBe(`user${i}@example.com`) - } - }, 30000) - }) -}) diff --git a/packages/protect/__tests__/deprecated/search-terms.test.ts b/packages/protect/__tests__/deprecated/search-terms.test.ts deleted file mode 100644 index d3c12a773..000000000 --- a/packages/protect/__tests__/deprecated/search-terms.test.ts +++ /dev/null @@ -1,140 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { describe, expect, it } from 'vitest' -import { protect, type SearchTerm } from '../../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - age: csColumn('age').dataType('number').equality(), - score: csColumn('score').dataType('number').equality(), -}) - -describe('createSearchTerms (deprecated - backward compatibility)', () => { - it('should create search terms with default return type', async () => { - const protectClient = await protect({ schemas: [users] }) - - const searchTerms = [ - { - value: 'hello', - column: users.email, - table: users, - }, - { - value: 'world', - column: users.address, - table: users, - }, - ] as SearchTerm[] - - const searchTermsResult = await protectClient.createSearchTerms(searchTerms) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - expect(searchTermsResult.data).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - v: 2, - }), - ]), - ) - }, 30000) - - it('should create search terms with composite-literal return type', async () => { - const protectClient = await protect({ schemas: [users] }) - - const searchTerms = [ - { - value: 'hello', - column: users.email, - table: users, - returnType: 'composite-literal', - }, - ] as SearchTerm[] - - const searchTermsResult = await protectClient.createSearchTerms(searchTerms) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - const result = searchTermsResult.data[0] as string - expect(result).toMatch(/^\(.*\)$/) - expect(() => JSON.parse(result.slice(1, -1))).not.toThrow() - }, 30000) - - it('should create search terms with escaped-composite-literal return type', async () => { - const protectClient = await protect({ schemas: [users] }) - - const searchTerms = [ - { - value: 'hello', - column: users.email, - table: users, - returnType: 'escaped-composite-literal', - }, - ] as SearchTerm[] - - const searchTermsResult = await protectClient.createSearchTerms(searchTerms) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - const result = searchTermsResult.data[0] as string - expect(result).toMatch(/^".*"$/) - const unescaped = JSON.parse(result) - expect(unescaped).toMatch(/^\(.*\)$/) - expect(() => JSON.parse(unescaped.slice(1, -1))).not.toThrow() - }, 30000) - - it('should create search terms with composite-literal return type for numbers', async () => { - const protectClient = await protect({ schemas: [users] }) - - const searchTerms = [ - { - value: 42, - column: users.age, - table: users, - returnType: 'composite-literal' as const, - }, - ] - - const searchTermsResult = await protectClient.createSearchTerms(searchTerms) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - const result = searchTermsResult.data[0] as string - expect(result).toMatch(/^\(.*\)$/) - expect(() => JSON.parse(result.slice(1, -1))).not.toThrow() - }, 30000) - - it('should create search terms with escaped-composite-literal return type for numbers', async () => { - const protectClient = await protect({ schemas: [users] }) - - const searchTerms = [ - { - value: 99, - column: users.score, - table: users, - returnType: 'escaped-composite-literal' as const, - }, - ] - - const searchTermsResult = await protectClient.createSearchTerms(searchTerms) - - if (searchTermsResult.failure) { - throw new Error(`[protect]: ${searchTermsResult.failure.message}`) - } - - const result = searchTermsResult.data[0] as string - expect(result).toMatch(/^".*"$/) - const unescaped = JSON.parse(result) - expect(unescaped).toMatch(/^\(.*\)$/) - expect(() => JSON.parse(unescaped.slice(1, -1))).not.toThrow() - }, 30000) -}) diff --git a/packages/protect/__tests__/encrypt-query-searchable-json.test.ts b/packages/protect/__tests__/encrypt-query-searchable-json.test.ts deleted file mode 100644 index faa25cb84..000000000 --- a/packages/protect/__tests__/encrypt-query-searchable-json.test.ts +++ /dev/null @@ -1,1061 +0,0 @@ -import 'dotenv/config' -import { beforeAll, describe, expect, it } from 'vitest' -import { ProtectErrorTypes, protect } from '../src' - -type ProtectClient = Awaited> - -import { - createFailingMockLockContext, - createMockLockContext, - createMockLockContextWithNullContext, - expectFailure, - jsonbSchema, - metadata, - unwrapResult, -} from './fixtures' - -/* - * The `searchableJson` queryType provides a friendlier API by auto-inferring the - * underlying query operation from the plaintext type. It's equivalent to omitting - * queryType on ste_vec columns, but explicit for code clarity. - * - String values → ste_vec_selector (JSONPath queries) - * - Object/Array values → ste_vec_term (containment queries) - */ - -/** Assert encrypted selector output has valid shape */ -function expectSelector(data: any) { - expect(data).toHaveProperty('s') - expect(typeof data.s).toBe('string') - expect(data.s.length).toBeGreaterThan(0) -} - -/** Assert encrypted term output has valid shape */ -function expectTerm(data: any) { - expect(data).toHaveProperty('sv') - expect(Array.isArray(data.sv)).toBe(true) -} - -describe('encryptQuery with searchableJson queryType', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - // Core functionality: auto-inference from plaintext type - - it('auto-infers ste_vec_selector for string plaintext (JSONPath)', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectSelector(data) - }, 30000) - - it('auto-infers ste_vec_term for object plaintext (containment)', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('auto-infers ste_vec_term for nested object', async () => { - const result = await protectClient.encryptQuery( - { user: { profile: { role: 'admin' } } }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('auto-infers ste_vec_term for array plaintext', async () => { - const result = await protectClient.encryptQuery(['admin', 'user'], { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('returns null for null plaintext', async () => { - const result = await protectClient.encryptQuery(null, { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - // Edge cases: number/boolean require wrapping (same as steVecTerm) - - it('fails for bare number plaintext (requires wrapping)', async () => { - const result = await protectClient.encryptQuery(42, { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - expectFailure(result, /Wrap the number in a JSON object/) - }, 30000) - - it('fails for bare boolean plaintext (requires wrapping)', async () => { - const result = await protectClient.encryptQuery(true, { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - expectFailure(result, /Wrap the boolean in a JSON object/) - }, 30000) -}) - -describe('encryptQuery with searchableJson column and omitted queryType', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('auto-infers ste_vec_selector for string plaintext (JSONPath)', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectSelector(data) - }, 30000) - - it('auto-infers ste_vec_term for object plaintext (containment)', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('returns null for null plaintext', async () => { - const result = await protectClient.encryptQuery(null, { - column: jsonbSchema.metadata, - table: jsonbSchema, - }) - - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - it('fails for bare number plaintext (requires wrapping)', async () => { - const result = await protectClient.encryptQuery(42, { - column: jsonbSchema.metadata, - table: jsonbSchema, - }) - - expectFailure(result, /Wrap the number in a JSON object/) - }, 30000) - - it('fails for bare boolean plaintext (requires wrapping)', async () => { - const result = await protectClient.encryptQuery(true, { - column: jsonbSchema.metadata, - table: jsonbSchema, - }) - - expectFailure(result, /Wrap the boolean in a JSON object/) - }, 30000) -}) - -describe('searchableJson validation', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('throws when used on column without ste_vec index', async () => { - const result = await protectClient.encryptQuery('$.path', { - column: metadata.raw, // raw column has no ste_vec index - table: metadata, - queryType: 'searchableJson', - }) - - expectFailure(result) - }, 30000) -}) - -describe('searchableJson batch operations', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('handles mixed plaintext types in single batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.user.email', // string → ste_vec_selector - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: { role: 'admin' }, // object → ste_vec_term - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: ['tag1', 'tag2'], // array → ste_vec_term - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(3) - expect(data[0]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expectSelector(data[0]) - expect(data[1]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expectTerm(data[1]) - expect(data[2]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expectTerm(data[2]) - }, 30000) - - it('handles null values in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(3) - expect(data[0]).toBeNull() - expect(data[1]).not.toBeNull() - expectSelector(data[1]) - expect(data[2]).toBeNull() - }, 30000) - - it('can be mixed with explicit steVecSelector/steVecTerm in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.path1', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', // auto-infer - }, - { - value: '$.path2', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', // explicit - }, - { - value: { key: 'value' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', // explicit - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(3) - expectSelector(data[0]) - expectSelector(data[1]) - expectTerm(data[2]) - }, 30000) - - it('can omit queryType for searchableJson in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.path1', - column: jsonbSchema.metadata, - table: jsonbSchema, - }, - { - value: { key: 'value' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(3) - expectSelector(data[0]) - expectTerm(data[1]) - expect(data[2]).toBeNull() - }, 30000) -}) - -describe('searchableJson with returnType formatting', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('supports composite-literal returnType', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'composite-literal', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(1) - expect(typeof data[0]).toBe('string') - // Format: ("json") - expect(data[0]).toMatch(/^\(".*"\)$/) - }, 30000) - - it('supports escaped-composite-literal returnType', async () => { - const result = await protectClient.encryptQuery([ - { - value: { role: 'admin' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'escaped-composite-literal', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(1) - expect(typeof data[0]).toBe('string') - // Format: "(\"json\")" - outer quotes with escaped inner quotes - expect(data[0]).toMatch(/^"\(.*\)"$/) - }, 30000) - - describe('single-value returnType', () => { - it('returns composite-literal for selector', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'composite-literal', - }) - - const data = unwrapResult(result) - expect(typeof data).toBe('string') - expect(data).toMatch(/^\(".*"\)$/) - }, 30000) - - it('returns composite-literal for term', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'composite-literal', - }, - ) - - const data = unwrapResult(result) - expect(typeof data).toBe('string') - expect(data).toMatch(/^\(".*"\)$/) - }, 30000) - - it('returns escaped-composite-literal for selector', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'escaped-composite-literal', - }) - - const data = unwrapResult(result) - expect(typeof data).toBe('string') - expect(data as string).toMatch(/^"\(.*\)"$/) - // JSON.parse should yield the composite-literal format - const parsed = JSON.parse(data as string) - expect(parsed).toMatch(/^\(.*\)$/) - }, 30000) - - it('returns escaped-composite-literal for term', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - returnType: 'escaped-composite-literal', - }, - ) - - const data = unwrapResult(result) - expect(typeof data).toBe('string') - expect(data as string).toMatch(/^"\(.*\)"$/) - const parsed = JSON.parse(data as string) - expect(parsed).toMatch(/^\(.*\)$/) - }, 30000) - - it('returns Encrypted object when returnType is omitted', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) as any - expect(typeof data).toBe('object') - expect(data).toHaveProperty('i') - expect(data.i).toHaveProperty('t') - expect(data.i).toHaveProperty('c') - }, 30000) - }) -}) - -describe('searchableJson with LockContext', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('exposes withLockContext method', async () => { - const operation = protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - expect(operation.withLockContext).toBeDefined() - expect(typeof operation.withLockContext).toBe('function') - }) - - it('executes string plaintext with LockContext mock', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - expect(mockLockContext.getLockContext).toHaveBeenCalledTimes(1) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectSelector(data) - }, 30000) - - it('executes object plaintext with LockContext mock', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // LockContext should be called even if the actual encryption fails - // with a mock token (ste_vec_term operations may require real auth) - expect(mockLockContext.getLockContext).toHaveBeenCalledTimes(1) - - // Ensure the operation actually completed (has either data or failure) - expect(result.data !== undefined || result.failure !== undefined).toBe(true) - - // The result may fail due to mock token, but we verify LockContext integration worked - if (result.data) { - expect(result.data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(result.data) - } - }, 30000) - - it('executes batch with LockContext mock', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: { role: 'admin' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // LockContext should be called even if the actual encryption fails - // with a mock token (ste_vec_term operations may require real auth) - expect(mockLockContext.getLockContext).toHaveBeenCalledTimes(1) - - // The result may fail due to mock token, but we verify LockContext integration worked - if (result.data) { - expect(result.data).toHaveLength(2) - } - }, 30000) - - it('handles LockContext failure gracefully', async () => { - const mockLockContext = createFailingMockLockContext( - ProtectErrorTypes.CtsTokenError, - 'Mock LockContext failure', - ) - - const operation = protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - expectFailure( - result, - 'Mock LockContext failure', - ProtectErrorTypes.CtsTokenError, - ) - }, 30000) - - it('handles null value with LockContext', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery(null, { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // Null values should return null without calling LockContext - // since there's nothing to encrypt - expect(mockLockContext.getLockContext).not.toHaveBeenCalled() - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - it('handles explicit null context from getLockContext gracefully', async () => { - const mockLockContext = createMockLockContextWithNullContext() - - const operation = protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // Should succeed - null context should not be passed to FFI - const data = unwrapResult(result) - expect(data).toHaveLength(1) - expect(data[0]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expectSelector(data[0]) - }, 30000) -}) - -describe('searchableJson equivalence', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('produces identical metadata to omitting queryType for string', async () => { - const explicitResult = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const implicitResult = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - }) - - // Both should succeed and have identical metadata structure - const explicitData = unwrapResult(explicitResult) - const implicitData = unwrapResult(implicitResult) - - expect(explicitData.i).toEqual(implicitData.i) - expect(explicitData.v).toEqual(implicitData.v) - expectSelector(explicitData) - expectSelector(implicitData) - }, 30000) - - it('produces identical metadata to omitting queryType for object', async () => { - const explicitResult = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const implicitResult = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - }, - ) - - const explicitData = unwrapResult(explicitResult) - const implicitData = unwrapResult(implicitResult) - - expect(explicitData.i).toEqual(implicitData.i) - expect(explicitData.v).toEqual(implicitData.v) - expectTerm(explicitData) - expectTerm(implicitData) - }, 30000) - - it('produces identical metadata to explicit steVecSelector for string', async () => { - const searchableJsonResult = await protectClient.encryptQuery( - '$.user.email', - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const steVecSelectorResult = await protectClient.encryptQuery( - '$.user.email', - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - ) - - const searchableJsonData = unwrapResult(searchableJsonResult) - const steVecSelectorData = unwrapResult(steVecSelectorResult) - - expect(searchableJsonData.i).toEqual(steVecSelectorData.i) - expect(searchableJsonData.v).toEqual(steVecSelectorData.v) - expectSelector(searchableJsonData) - expectSelector(steVecSelectorData) - }, 30000) - - it('produces identical metadata to explicit steVecTerm for object', async () => { - const searchableJsonResult = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const steVecTermResult = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - ) - - const searchableJsonData = unwrapResult(searchableJsonResult) - const steVecTermData = unwrapResult(steVecTermResult) - - expect(searchableJsonData.i).toEqual(steVecTermData.i) - expect(searchableJsonData.v).toEqual(steVecTermData.v) - expectTerm(searchableJsonData) - expectTerm(steVecTermData) - }, 30000) -}) - -describe('searchableJson edge cases', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - // Valid edge cases that should succeed - - it('succeeds for empty object', async () => { - const result = await protectClient.encryptQuery( - {}, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('succeeds for empty array', async () => { - const result = await protectClient.encryptQuery([], { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('succeeds for object with wrapped number', async () => { - const result = await protectClient.encryptQuery( - { value: 42 }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('succeeds for object with wrapped boolean', async () => { - const result = await protectClient.encryptQuery( - { active: true }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('succeeds for object with null value', async () => { - const result = await protectClient.encryptQuery( - { field: null }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - it('succeeds for deeply nested object (3+ levels)', async () => { - const result = await protectClient.encryptQuery( - { - level1: { - level2: { - level3: { - level4: { - value: 'deep', - }, - }, - }, - }, - }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectTerm(data) - }, 30000) - - // String edge cases for JSONPath selectors - - it('succeeds for JSONPath with array index notation', async () => { - const result = await protectClient.encryptQuery('$.items[0].name', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectSelector(data) - }, 30000) - - it('succeeds for JSONPath with wildcard', async () => { - const result = await protectClient.encryptQuery('$.items[*].name', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - expectSelector(data) - }, 30000) -}) - -describe('searchableJson batch edge cases', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('handles single-item batch identically to scalar', async () => { - const scalarResult = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }) - - const batchResult = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const scalarData = unwrapResult(scalarResult) - const batchData = unwrapResult(batchResult) - - expect(batchData).toHaveLength(1) - expect(batchData[0].i).toEqual(scalarData.i) - expect(batchData[0].v).toEqual(scalarData.v) - expectSelector(scalarData) - expectSelector(batchData[0]) - }, 30000) - - it('handles all-null batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(3) - expect(data[0]).toBeNull() - expect(data[1]).toBeNull() - expect(data[2]).toBeNull() - }, 30000) - - it('handles empty batch', async () => { - const result = await protectClient.encryptQuery([]) - - const data = unwrapResult(result) - expect(data).toHaveLength(0) - }, 30000) - - it('handles large batch (10+ items)', async () => { - const items = Array.from({ length: 12 }, (_, i) => ({ - value: i % 2 === 0 ? `$.path${i}` : { index: i }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson' as const, - })) - - const result = await protectClient.encryptQuery(items) - - const data = unwrapResult(result) - expect(data).toHaveLength(12) - data.forEach((item: any, idx: number) => { - expect(item).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - v: 2, - }) - if (idx % 2 === 0) { - expectSelector(item) - } else { - expectTerm(item) - } - }) - }, 30000) - - it('handles multiple interspersed nulls at various positions', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: { role: 'admin' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'searchableJson', - }, - ]) - - const data = unwrapResult(result) - expect(data).toHaveLength(5) - expect(data[0]).toBeNull() - expect(data[1]).not.toBeNull() - expectSelector(data[1]) - expect(data[2]).toBeNull() - expect(data[3]).not.toBeNull() - expectTerm(data[3]) - expect(data[4]).toBeNull() - }, 30000) -}) diff --git a/packages/protect/__tests__/encrypt-query-stevec.test.ts b/packages/protect/__tests__/encrypt-query-stevec.test.ts deleted file mode 100644 index 7c536d725..000000000 --- a/packages/protect/__tests__/encrypt-query-stevec.test.ts +++ /dev/null @@ -1,392 +0,0 @@ -import 'dotenv/config' -import { beforeAll, describe, expect, it } from 'vitest' -import { protect } from '../src' - -type ProtectClient = Awaited> - -import { expectFailure, jsonbSchema, metadata, unwrapResult } from './fixtures' - -describe('encryptQuery with steVecSelector', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('encrypts a JSONPath selector', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('encrypts nested path selector', async () => { - const result = await protectClient.encryptQuery('$.user.profile.settings', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('fails for non-string plaintext with steVecSelector (object)', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - ) - - expectFailure(result) - }, 30000) -}) - -describe('encryptQuery with steVecTerm', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('encrypts an object for containment query', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('encrypts nested object for containment', async () => { - const result = await protectClient.encryptQuery( - { user: { profile: { role: 'admin' } } }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('encrypts array for containment query', async () => { - const result = await protectClient.encryptQuery([1, 2, 3], { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('rejects string plaintext with steVecTerm', async () => { - // steVecTerm requires object or array, not string - // For path queries like '$.field', use steVecSelector instead - const result = await protectClient.encryptQuery('search text', { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }) - - expectFailure(result, /expected JSON object or array/) - }, 30000) -}) - -describe('encryptQuery STE Vec validation', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('throws when steVecSelector used on non-ste_vec column', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: metadata.raw, // raw column has no ste_vec index - table: metadata, - queryType: 'steVecSelector', - }) - - expectFailure(result) - }, 30000) - - it('throws when steVecTerm used on non-ste_vec column', async () => { - const result = await protectClient.encryptQuery( - { field: 'value' }, - { - column: metadata.raw, // raw column has no ste_vec index - table: metadata, - queryType: 'steVecTerm', - }, - ) - - expectFailure(result) - }, 30000) -}) - -describe('encryptQuery batch with STE Vec', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema, metadata] }) - }) - - it('handles mixed query types in batch (steVecSelector + steVecTerm)', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - { - value: { role: 'admin' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expect(data[1]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - }, 30000) - - it('handles multiple steVecSelector queries in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - { - value: '$.settings.theme', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - expect(data[1]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - }, 30000) - - it('handles null values with steVecSelector in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - { - value: '$.user.email', - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecSelector', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).toBeNull() - expect(data[1]).not.toBeNull() - expect(data[1]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - }, 30000) - - it('handles null values with steVecTerm in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - { - value: { role: 'admin' }, - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).toBeNull() - expect(data[1]).not.toBeNull() - expect(data[1]).toMatchObject({ i: { t: 'documents', c: 'metadata' } }) - }, 30000) -}) - -describe('encryptQuery with queryType inference', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('infers steVecSelector for string plaintext without queryType', async () => { - const result = await protectClient.encryptQuery('$.user.email', { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - should infer steVecSelector from string - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('infers steVecTerm for object plaintext without queryType', async () => { - const result = await protectClient.encryptQuery( - { role: 'admin' }, - { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - should infer steVecTerm from object - }, - ) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('infers steVecTerm for array plaintext without queryType', async () => { - const result = await protectClient.encryptQuery(['admin', 'user'], { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - should infer steVecTerm from array - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) - - it('infers steVecTerm for number plaintext but FFI requires wrapping', async () => { - // Numbers infer steVecTerm but FFI requires wrapping in object/array - const result = await protectClient.encryptQuery(42, { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - infers steVecTerm, FFI rejects with helpful message - }) - - expectFailure(result, /Wrap the number in a JSON object/) - }, 30000) - - it('infers steVecTerm for boolean plaintext but FFI requires wrapping', async () => { - // Booleans infer steVecTerm but FFI requires wrapping in object/array - const result = await protectClient.encryptQuery(true, { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - infers steVecTerm, FFI rejects with helpful message - }) - - expectFailure(result, /Wrap the boolean in a JSON object/) - }, 30000) - - it('returns null for null plaintext (no inference needed)', async () => { - const result = await protectClient.encryptQuery(null, { - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType and null plaintext - should return null - }) - - // Null returns null, doesn't throw - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - it('uses explicit queryType over plaintext inference', async () => { - // String plaintext would normally infer steVecSelector, but explicit steVecTerm should be used - // Note: steVecTerm with string fails FFI validation, so we test the opposite direction - // Using a number (which would infer steVecTerm) with explicit steVecSelector would also fail - // So we verify with array + steVecTerm (already tested) and trust unit test coverage for precedence - const result = await protectClient.encryptQuery([42], { - column: jsonbSchema.metadata, - table: jsonbSchema, - queryType: 'steVecTerm', // Explicit - matches inference but proves explicit path works - }) - - const data = unwrapResult(result) - expect(data).toBeDefined() - expect(data).toMatchObject({ - i: { t: 'documents', c: 'metadata' }, - }) - }, 30000) -}) - -describe('encryptQuery batch with queryType inference', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ schemas: [jsonbSchema] }) - }) - - it('infers queryOp for each term independently in batch', async () => { - const results = await protectClient.encryptQuery([ - { - value: '$.user.email', // string → steVecSelector - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - }, - { - value: { role: 'admin' }, // object → steVecTerm - column: jsonbSchema.metadata, - table: jsonbSchema, - // No queryType - }, - ]) - - const data = unwrapResult(results) - expect(data).toHaveLength(2) - expect(data[0]).toBeDefined() - expect(data[1]).toBeDefined() - }, 30000) -}) diff --git a/packages/protect/__tests__/encrypt-query.test.ts b/packages/protect/__tests__/encrypt-query.test.ts deleted file mode 100644 index 1f64000b2..000000000 --- a/packages/protect/__tests__/encrypt-query.test.ts +++ /dev/null @@ -1,946 +0,0 @@ -import 'dotenv/config' -import { beforeAll, describe, expect, it } from 'vitest' -import { ProtectErrorTypes, protect } from '../src' -import type { ProtectClient } from '../src/ffi' -import { - articles, - createFailingMockLockContext, - createMockLockContext, - createMockLockContextWithNullContext, - expectFailure, - metadata, - products, - unwrapResult, - users, -} from './fixtures' - -describe('encryptQuery', () => { - let protectClient: ProtectClient - - beforeAll(async () => { - protectClient = await protect({ - schemas: [users, articles, products, metadata], - }) - }) - - describe('single value encryption with explicit queryType', () => { - it('encrypts for equality query type', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const data = unwrapResult(result) - - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(data).toHaveProperty('hm') - }, 30000) - - it('encrypts for freeTextSearch query type', async () => { - const result = await protectClient.encryptQuery('hello world', { - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }) - - const data = unwrapResult(result) - - expect(data).toMatchObject({ - i: { t: 'users', c: 'bio' }, - v: 2, - }) - expect(data).toHaveProperty('bf') - }, 30000) - - it('encrypts for orderAndRange query type', async () => { - const result = await protectClient.encryptQuery(25, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - const data = unwrapResult(result) - - expect(data).toMatchObject({ - i: { t: 'users', c: 'age' }, - v: 2, - }) - expect(data).toHaveProperty('ob') - }, 30000) - }) - - describe('auto-inference when queryType omitted', () => { - it('auto-infers equality for column with .equality()', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('hm') - }, 30000) - - it('auto-infers freeTextSearch for match-only column', async () => { - const result = await protectClient.encryptQuery('search content', { - column: articles.content, - table: articles, - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('bf') - }, 30000) - - it('auto-infers orderAndRange for ore-only column', async () => { - const result = await protectClient.encryptQuery(99.99, { - column: products.price, - table: products, - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('ob') - }, 30000) - }) - - describe('edge cases', () => { - it('handles null values', async () => { - const result = await protectClient.encryptQuery(null, { - column: users.email, - table: users, - queryType: 'equality', - }) - - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - it('rejects NaN values', async () => { - const result = await protectClient.encryptQuery(Number.NaN, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - expectFailure(result, 'NaN') - }, 30000) - - it('rejects Infinity values', async () => { - const result = await protectClient.encryptQuery( - Number.POSITIVE_INFINITY, - { - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ) - - expectFailure(result, 'Infinity') - }, 30000) - - it('rejects negative Infinity values', async () => { - const result = await protectClient.encryptQuery( - Number.NEGATIVE_INFINITY, - { - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ) - - expectFailure(result, 'Infinity') - }, 30000) - }) - - describe('validation errors', () => { - it('fails when queryType does not match column config', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'freeTextSearch', // email only has equality - }) - - expectFailure(result, 'not configured') - }, 30000) - - it('fails when column has no indexes configured', async () => { - const result = await protectClient.encryptQuery('raw data', { - column: metadata.raw, - table: metadata, - }) - - expectFailure(result, 'no indexes configured') - }, 30000) - - it('provides descriptive error for queryType mismatch', async () => { - const result = await protectClient.encryptQuery(42, { - column: users.age, - table: users, - queryType: 'equality', // age only has orderAndRange - }) - - expectFailure(result, 'unique') - expectFailure(result, 'not configured', ProtectErrorTypes.EncryptionError) - }, 30000) - }) - - describe('value/index type compatibility', () => { - it('fails when encrypting number with match index (explicit queryType)', async () => { - const result = await protectClient.encryptQuery(123, { - column: articles.content, // match-only column - table: articles, - queryType: 'freeTextSearch', - }) - - expectFailure(result, 'match') - expectFailure(result, 'numeric') - }, 30000) - - it('fails when encrypting number with auto-inferred match index', async () => { - const result = await protectClient.encryptQuery(123, { - column: articles.content, // match-only column, will infer 'match' - table: articles, - }) - - expectFailure(result, 'match') - }, 30000) - - it('fails in batch when number used with match index', async () => { - const result = await protectClient.encryptQuery([ - { value: 123, column: articles.content, table: articles }, - ]) - - expectFailure(result, 'match') - }, 30000) - - it('allows string with match index', async () => { - const result = await protectClient.encryptQuery('search text', { - column: articles.content, - table: articles, - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('bf') // bloom filter - }, 30000) - - it('allows number with ore index', async () => { - const result = await protectClient.encryptQuery(42, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('ob') // order bits - }, 30000) - }) - - describe('numeric edge cases', () => { - it('encrypts MAX_SAFE_INTEGER', async () => { - const result = await protectClient.encryptQuery(Number.MAX_SAFE_INTEGER, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'age' }, - v: 2, - }) - expect(data).toHaveProperty('ob') - }, 30000) - - it('encrypts MIN_SAFE_INTEGER', async () => { - const result = await protectClient.encryptQuery(Number.MIN_SAFE_INTEGER, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'age' }, - v: 2, - }) - expect(data).toHaveProperty('ob') - }, 30000) - - it('encrypts negative zero', async () => { - const result = await protectClient.encryptQuery(-0, { - column: users.age, - table: users, - queryType: 'orderAndRange', - }) - - const data = unwrapResult(result) - expect(data).toHaveProperty('ob') - }, 30000) - }) - - describe('string edge cases', () => { - it('encrypts empty string', async () => { - const result = await protectClient.encryptQuery('', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(data).toHaveProperty('hm') - }, 30000) - - it('encrypts unicode/emoji strings', async () => { - const result = await protectClient.encryptQuery('Hello 世界 🌍🚀', { - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'bio' }, - v: 2, - }) - expect(data).toHaveProperty('bf') - }, 30000) - - it('encrypts strings with SQL special characters', async () => { - const result = await protectClient.encryptQuery( - "'; DROP TABLE users; --", - { - column: users.email, - table: users, - queryType: 'equality', - }, - ) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(data).toHaveProperty('hm') - }, 30000) - }) - - describe('encryptQuery bulk (array overload)', () => { - it('encrypts multiple terms in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'user@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: 'search term', - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }, - { - value: 42, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(3) - expect(data[0]).toMatchObject({ i: { t: 'users', c: 'email' } }) - expect(data[1]).toMatchObject({ i: { t: 'users', c: 'bio' } }) - expect(data[2]).toMatchObject({ i: { t: 'users', c: 'age' } }) - }, 30000) - - it('handles empty array', async () => { - // Empty arrays without opts are treated as empty batch for backward compatibility - const result = await protectClient.encryptQuery([]) - - const data = unwrapResult(result) - expect(data).toEqual([]) - }, 30000) - - it('handles null values in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: null, - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).not.toBeNull() - expect(data[1]).toBeNull() - }, 30000) - - it('auto-infers queryType when omitted', async () => { - const result = await protectClient.encryptQuery([ - { value: 'user@example.com', column: users.email, table: users }, - { value: 42, column: users.age, table: users }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(2) - expect(data[0]).toHaveProperty('hm') - expect(data[1]).toHaveProperty('ob') - }, 30000) - - it('rejects NaN/Infinity values in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: Number.NaN, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - { - value: Number.POSITIVE_INFINITY, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - expect(result.failure).toBeDefined() - }, 30000) - - it('rejects negative Infinity in batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: Number.NEGATIVE_INFINITY, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - expectFailure(result, 'Infinity') - }, 30000) - }) - - describe('bulk index preservation', () => { - it('preserves exact positions with multiple nulls interspersed', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: 'user@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: null, - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }, - { - value: null, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - { - value: 42, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(5) - expect(data[0]).toBeNull() - expect(data[1]).not.toBeNull() - expect(data[1]).toHaveProperty('hm') - expect(data[2]).toBeNull() - expect(data[3]).toBeNull() - expect(data[4]).not.toBeNull() - expect(data[4]).toHaveProperty('ob') - }, 30000) - - it('handles single-item array', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'single@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(1) - expect(data[0]).toMatchObject({ i: { t: 'users', c: 'email' } }) - expect(data[0]).toHaveProperty('hm') - }, 30000) - - it('handles all-null array', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: null, - column: users.bio, - table: users, - queryType: 'freeTextSearch', - }, - { - value: null, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(3) - expect(data[0]).toBeNull() - expect(data[1]).toBeNull() - expect(data[2]).toBeNull() - }, 30000) - }) - - describe('audit support', () => { - it('passes audit metadata for single query', async () => { - const result = await protectClient - .encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - .audit({ metadata: { userId: 'test-user' } }) - - const data = unwrapResult(result) - expect(data).toMatchObject({ i: { t: 'users', c: 'email' } }) - }, 30000) - - it('passes audit metadata for bulk query', async () => { - const result = await protectClient - .encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - ]) - .audit({ metadata: { userId: 'test-user' } }) - - const data = unwrapResult(result) - expect(data).toHaveLength(1) - }, 30000) - }) - - describe('returnType formatting', () => { - it('returns Encrypted by default (no returnType)', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(1) - expect(data[0]).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(typeof data[0]).toBe('object') - }, 30000) - - it('returns composite-literal format when specified', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - returnType: 'composite-literal', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(1) - expect(typeof data[0]).toBe('string') - // Format: ("json") - expect(data[0]).toMatch(/^\(".*"\)$/) - }, 30000) - - it('returns escaped-composite-literal format when specified', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - returnType: 'escaped-composite-literal', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(1) - expect(typeof data[0]).toBe('string') - // Format: "(\"json\")" - outer quotes with escaped inner quotes - expect(data[0]).toMatch(/^"\(.*\)"$/) - }, 30000) - - it('returns eql format when explicitly specified', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - returnType: 'eql', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(1) - expect(data[0]).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(typeof data[0]).toBe('object') - }, 30000) - - it('handles mixed returnType values in same batch', async () => { - const result = await protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, // default - { - value: 'search term', - column: users.bio, - table: users, - queryType: 'freeTextSearch', - returnType: 'composite-literal', - }, - { - value: 42, - column: users.age, - table: users, - queryType: 'orderAndRange', - returnType: 'escaped-composite-literal', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(3) - - // First: default (Encrypted object) - expect(typeof data[0]).toBe('object') - expect(data[0]).toMatchObject({ i: { t: 'users', c: 'email' } }) - - // Second: composite-literal (string) - expect(typeof data[1]).toBe('string') - expect(data[1]).toMatch(/^\(".*"\)$/) - - // Third: escaped-composite-literal (string) - expect(typeof data[2]).toBe('string') - expect(data[2]).toMatch(/^"\(.*\)"$/) - }, 30000) - - it('handles returnType with null values', async () => { - const result = await protectClient.encryptQuery([ - { - value: null, - column: users.email, - table: users, - queryType: 'equality', - returnType: 'composite-literal', - }, - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - returnType: 'composite-literal', - }, - { - value: null, - column: users.bio, - table: users, - queryType: 'freeTextSearch', - returnType: 'escaped-composite-literal', - }, - ]) - - const data = unwrapResult(result) - - expect(data).toHaveLength(3) - expect(data[0]).toBeNull() - expect(typeof data[1]).toBe('string') - expect(data[1]).toMatch(/^\(".*"\)$/) - expect(data[2]).toBeNull() - }, 30000) - }) - - describe('single-value returnType formatting', () => { - it('returns Encrypted by default (no returnType)', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const data = unwrapResult(result) - - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(typeof data).toBe('object') - }, 30000) - - it('returns composite-literal format when specified', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - returnType: 'composite-literal', - }) - - const data = unwrapResult(result) - - expect(typeof data).toBe('string') - // Format: ("json") - expect(data).toMatch(/^\(".*"\)$/) - }, 30000) - - it('returns escaped-composite-literal format when specified', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - returnType: 'escaped-composite-literal', - }) - - const data = unwrapResult(result) - - expect(typeof data).toBe('string') - // Format: "(\"json\")" - outer quotes with escaped inner quotes - expect(data).toMatch(/^"\(.*\)"$/) - }, 30000) - - it('returns eql format when explicitly specified', async () => { - const result = await protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - returnType: 'eql', - }) - - const data = unwrapResult(result) - - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(typeof data).toBe('object') - }, 30000) - - it('handles null value with composite-literal returnType', async () => { - const result = await protectClient.encryptQuery(null, { - column: users.email, - table: users, - queryType: 'equality', - returnType: 'composite-literal', - }) - - const data = unwrapResult(result) - - expect(data).toBeNull() - }, 30000) - }) - - describe('LockContext support', () => { - it('single query with LockContext calls getLockContext', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - expect(withContext).toHaveProperty('execute') - expect(typeof withContext.execute).toBe('function') - }, 30000) - - it('bulk query with LockContext calls getLockContext', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - ]) - - const withContext = operation.withLockContext(mockLockContext as any) - expect(withContext).toHaveProperty('execute') - expect(typeof withContext.execute).toBe('function') - }, 30000) - - it('executes single query with LockContext mock', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - expect(mockLockContext.getLockContext).toHaveBeenCalledTimes(1) - - const data = unwrapResult(result) - expect(data).toMatchObject({ - i: { t: 'users', c: 'email' }, - v: 2, - }) - expect(data).toHaveProperty('hm') - }, 30000) - - it('executes bulk query with LockContext mock', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - { - value: 42, - column: users.age, - table: users, - queryType: 'orderAndRange', - }, - ]) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - expect(mockLockContext.getLockContext).toHaveBeenCalledTimes(1) - - const data = unwrapResult(result) - expect(data).toHaveLength(2) - expect(data[0]).toHaveProperty('hm') - expect(data[1]).toHaveProperty('ob') - }, 30000) - - it('handles LockContext failure gracefully', async () => { - const mockLockContext = createFailingMockLockContext( - ProtectErrorTypes.CtsTokenError, - 'Mock LockContext failure', - ) - - const operation = protectClient.encryptQuery('test@example.com', { - column: users.email, - table: users, - queryType: 'equality', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - expectFailure( - result, - 'Mock LockContext failure', - ProtectErrorTypes.CtsTokenError, - ) - }, 30000) - - it('handles null value with LockContext', async () => { - const mockLockContext = createMockLockContext() - - const operation = protectClient.encryptQuery(null, { - column: users.email, - table: users, - queryType: 'equality', - }) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // Null values should return null without calling LockContext - // since there's nothing to encrypt - const data = unwrapResult(result) - expect(data).toBeNull() - }, 30000) - - it('handles explicit null context from getLockContext gracefully', async () => { - // Simulate a runtime scenario where context is null (bypasses TypeScript) - const mockLockContext = createMockLockContextWithNullContext() - - const operation = protectClient.encryptQuery([ - { - value: 'test@example.com', - column: users.email, - table: users, - queryType: 'equality', - }, - ]) - - const withContext = operation.withLockContext(mockLockContext as any) - const result = await withContext.execute() - - // Should succeed - null context should not be passed to FFI - const data = unwrapResult(result) - expect(data).toHaveLength(1) - expect(data[0]).toHaveProperty('hm') - }, 30000) - }) -}) diff --git a/packages/protect/__tests__/error-codes.test.ts b/packages/protect/__tests__/error-codes.test.ts deleted file mode 100644 index d0a3d8596..000000000 --- a/packages/protect/__tests__/error-codes.test.ts +++ /dev/null @@ -1,369 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import type { ProtectClient } from '../src' -import { FfiProtectError, ProtectErrorTypes, protect } from '../src' - -/** FFI tests require longer timeout due to client initialization */ -const FFI_TEST_TIMEOUT = 30_000 - -/** - * Tests for FFI error code preservation in ProtectError. - * These tests verify that specific FFI error codes are preserved when errors occur, - * enabling programmatic error handling. - */ -describe('FFI Error Code Preservation', () => { - let protectClient: ProtectClient - - // Schema with a valid column for testing - const testSchema = csTable('test_table', { - email: csColumn('email').equality(), - bio: csColumn('bio').freeTextSearch(), - age: csColumn('age').dataType('number').orderAndRange(), - metadata: csColumn('metadata').searchableJson(), - }) - - // Schema without indexes for testing non-FFI validation - const noIndexSchema = csTable('no_index_table', { - raw: csColumn('raw'), - }) - - // Schema with non-existent column for triggering FFI UNKNOWN_COLUMN error - const badModelSchema = csTable('test_table', { - nonexistent: csColumn('nonexistent_column'), - }) - - beforeAll(async () => { - protectClient = await protect({ schemas: [testSchema, noIndexSchema] }) - }) - - describe('FfiProtectError class', () => { - it('constructs with code and message', () => { - const error = new FfiProtectError({ - code: 'UNKNOWN_COLUMN', - message: 'Test error', - }) - expect(error.code).toBe('UNKNOWN_COLUMN') - expect(error.message).toBe('Test error') - }) - }) - - describe('encryptQuery error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for non-existent column', - async () => { - // Create a fake column that doesn't exist in the schema - const fakeColumn = csColumn('nonexistent_column').equality() - - const result = await protectClient.encryptQuery('test', { - column: fakeColumn, - table: testSchema, - queryType: 'equality', - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - - it( - 'returns undefined code for columns without indexes (non-FFI validation)', - async () => { - // This error is caught during pre-FFI validation, not by FFI itself - const result = await protectClient.encryptQuery('test', { - column: noIndexSchema.raw, - table: noIndexSchema, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.message).toContain('no indexes configured') - // Pre-FFI validation errors don't have FFI error codes - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - - it( - 'returns undefined code for non-FFI validation errors', - async () => { - // NaN validation happens before FFI call - const result = await protectClient.encryptQuery(Number.NaN, { - column: testSchema.age, - table: testSchema, - queryType: 'orderAndRange', - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - // Non-FFI errors should have undefined code - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('batch encryptQuery error codes', () => { - it( - 'preserves error code in batch operations', - async () => { - const fakeColumn = csColumn('nonexistent_column').equality() - - const result = await protectClient.encryptQuery([ - { - value: 'test', - column: fakeColumn, - table: testSchema, - queryType: 'equality', - }, - ]) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - - it( - 'returns undefined code for non-FFI batch errors', - async () => { - const result = await protectClient.encryptQuery([ - { - value: Number.NaN, - column: testSchema.age, - table: testSchema, - queryType: 'orderAndRange', - }, - ]) - - expect(result.failure).toBeDefined() - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('encrypt error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for non-existent column in encrypt', - async () => { - const fakeColumn = csColumn('nonexistent_column') - - const result = await protectClient.encrypt('test', { - column: fakeColumn, - table: testSchema, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - - it( - 'returns undefined code for non-FFI encrypt errors', - async () => { - const result = await protectClient.encrypt(Number.NaN, { - column: testSchema.age, - table: testSchema, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - // NaN validation happens before FFI call - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('decrypt error codes', () => { - it( - 'returns undefined code for malformed ciphertext (non-FFI validation)', - async () => { - // This error occurs during ciphertext parsing, not FFI decryption - const invalidCiphertext = { - i: { t: 'test_table', c: 'nonexistent' }, - v: 2, - c: 'invalid_ciphertext_data', - } - - const result = await protectClient.decrypt(invalidCiphertext) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.DecryptionError) - // Malformed ciphertext errors are caught before FFI and don't have codes - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkEncrypt error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for non-existent column', - async () => { - const fakeColumn = csColumn('nonexistent_column') - - const result = await protectClient.bulkEncrypt( - [{ plaintext: 'test1' }, { plaintext: 'test2' }], - { - column: fakeColumn, - table: testSchema, - }, - ) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - - it( - 'returns undefined code for non-FFI validation errors', - async () => { - const result = await protectClient.bulkEncrypt( - [{ plaintext: Number.NaN }], - { - column: testSchema.age, - table: testSchema, - }, - ) - - expect(result.failure).toBeDefined() - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkDecrypt error codes', () => { - it( - 'returns undefined code for malformed ciphertexts (non-FFI validation)', - async () => { - // bulkDecrypt uses the "fallible" FFI API (decryptBulkFallible) which normally: - // - Succeeds at the operation level - // - Returns per-item results with either { data } or { error } - // - // However, malformed ciphertexts cause parsing errors BEFORE the fallible API, - // which throws and triggers a top-level failure (not per-item errors). - // These pre-FFI errors don't have structured FFI error codes. - const invalidCiphertexts = [ - { data: { i: { t: 'test_table', c: 'email' }, v: 2, c: 'invalid1' } }, - { data: { i: { t: 'test_table', c: 'email' }, v: 2, c: 'invalid2' } }, - ] - - const result = await protectClient.bulkDecrypt(invalidCiphertexts) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.DecryptionError) - // FFI parsing errors don't have structured error codes - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('encryptModel error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for model with non-existent column', - async () => { - const model = { nonexistent: 'test value' } - - const result = await protectClient.encryptModel(model, badModelSchema) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('decryptModel error codes', () => { - it( - 'returns undefined code for malformed model (non-FFI validation)', - async () => { - const malformedModel = { - email: { - i: { t: 'test_table', c: 'email' }, - v: 2, - c: 'invalid_ciphertext', - }, - } - - const result = await protectClient.decryptModel(malformedModel) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.DecryptionError) - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkEncryptModels error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for models with non-existent column', - async () => { - const models = [{ nonexistent: 'value1' }, { nonexistent: 'value2' }] - - const result = await protectClient.bulkEncryptModels( - models, - badModelSchema, - ) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('bulkDecryptModels error codes', () => { - it( - 'returns undefined code for malformed models (non-FFI validation)', - async () => { - const malformedModels = [ - { - email: { i: { t: 'test_table', c: 'email' }, v: 2, c: 'invalid1' }, - }, - { - email: { i: { t: 'test_table', c: 'email' }, v: 2, c: 'invalid2' }, - }, - ] - - const result = await protectClient.bulkDecryptModels(malformedModels) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.DecryptionError) - expect(result.failure?.code).toBeUndefined() - }, - FFI_TEST_TIMEOUT, - ) - }) - - describe('searchTerms (deprecated) error codes', () => { - it( - 'returns UNKNOWN_COLUMN code for non-existent column', - async () => { - const fakeColumn = csColumn('nonexistent_column').equality() - - const result = await protectClient.createSearchTerms([ - { value: 'test', column: fakeColumn, table: testSchema }, - ]) - - expect(result.failure).toBeDefined() - expect(result.failure?.type).toBe(ProtectErrorTypes.EncryptionError) - expect(result.failure?.code).toBe('UNKNOWN_COLUMN') - }, - FFI_TEST_TIMEOUT, - ) - }) -}) diff --git a/packages/protect/__tests__/fixtures/index.ts b/packages/protect/__tests__/fixtures/index.ts deleted file mode 100644 index fa892b375..000000000 --- a/packages/protect/__tests__/fixtures/index.ts +++ /dev/null @@ -1,145 +0,0 @@ -import { csColumn, csTable } from '@cipherstash/schema' -import { expect, vi } from 'vitest' - -// ============ Schema Fixtures ============ - -/** - * Users table with multiple index types for testing - */ -export const users = csTable('users', { - email: csColumn('email').equality(), - bio: csColumn('bio').freeTextSearch(), - age: csColumn('age').dataType('number').orderAndRange(), -}) - -/** - * Articles table with only freeTextSearch (for auto-inference test) - */ -export const articles = csTable('articles', { - content: csColumn('content').freeTextSearch(), -}) - -/** - * Products table with only orderAndRange (for auto-inference test) - */ -export const products = csTable('products', { - price: csColumn('price').dataType('number').orderAndRange(), -}) - -/** - * Metadata table with no indexes (for validation error test) - */ -export const metadata = csTable('metadata', { - raw: csColumn('raw'), -}) - -/** - * Documents table with searchable JSON column (for STE Vec queries) - */ -export const jsonbSchema = csTable('documents', { - id: csColumn('id'), - metadata: csColumn('metadata').searchableJson(), -}) - -/** - * Schema fixture with mixed column types including JSON. - */ -export const mixedSchema = csTable('records', { - id: csColumn('id'), - email: csColumn('email').equality(), - name: csColumn('name').freeTextSearch(), - metadata: csColumn('metadata').searchableJson(), -}) - -// ============ Mock Factories ============ - -/** - * Creates a mock LockContext with successful response - */ -export function createMockLockContext(overrides?: { - accessToken?: string - expiry?: number - identityClaim?: string[] -}) { - return { - getLockContext: vi.fn().mockResolvedValue({ - data: { - ctsToken: { - accessToken: overrides?.accessToken ?? 'mock-token', - expiry: overrides?.expiry ?? Date.now() + 3600000, - }, - context: { - identityClaim: overrides?.identityClaim ?? ['sub'], - }, - }, - }), - } -} - -/** - * Creates a mock LockContext with explicit null context (simulates runtime edge case) - */ -export function createMockLockContextWithNullContext() { - return { - getLockContext: vi.fn().mockResolvedValue({ - data: { - ctsToken: { - accessToken: 'mock-token', - expiry: Date.now() + 3600000, - }, - context: null, // Explicit null - simulating runtime edge case - }, - }), - } -} - -/** - * Creates a mock LockContext that returns a failure - */ -export function createFailingMockLockContext( - errorType: string, - message: string, -) { - return { - getLockContext: vi.fn().mockResolvedValue({ - failure: { type: errorType, message }, - }), - } -} - -// ============ Test Helpers ============ - -/** - * Unwraps a Result type, throwing an error if it's a failure. - * Use this to simplify test assertions when you expect success. - */ -export function unwrapResult(result: { - data?: T - failure?: { message: string } -}): T { - if (result.failure) { - throw new Error(result.failure.message) - } - return result.data as T -} - -/** - * Asserts that a result is a failure with optional message and type matching - */ -export function expectFailure( - result: { failure?: { message: string; type?: string } }, - messagePattern?: string | RegExp, - expectedType?: string, -) { - expect(result.failure).toBeDefined() - if (messagePattern) { - if (typeof messagePattern === 'string') { - expect(result.failure?.message).toContain(messagePattern) - } else { - expect(result.failure?.message).toMatch(messagePattern) - } - } - if (expectedType) { - expect(result.failure?.type).toBe(expectedType) - } -} diff --git a/packages/protect/__tests__/helpers.test.ts b/packages/protect/__tests__/helpers.test.ts deleted file mode 100644 index 566ff2554..000000000 --- a/packages/protect/__tests__/helpers.test.ts +++ /dev/null @@ -1,274 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { - bulkModelsToEncryptedPgComposites, - encryptedToCompositeLiteral, - encryptedToPgComposite, - isEncryptedPayload, - isEncryptedScalarQuery, - modelToEncryptedPgComposites, -} from '../src/helpers' - -describe('helpers', () => { - describe('encryptedToPgComposite', () => { - it('should convert encrypted payload to pg composite', () => { - const encrypted = { - v: 1, - c: 'ciphertext', - i: { - c: 'iv', - t: 't', - }, - k: 'k', - ob: ['a', 'b'], - bf: [1, 2, 3], - hm: 'hm', - } - - const pgComposite = encryptedToPgComposite(encrypted) - expect(pgComposite).toEqual({ - data: encrypted, - }) - }) - }) - - describe('encryptedToCompositeLiteral', () => { - it('should convert encrypted payload to pg composite literal string', () => { - const encrypted = { - v: 1, - c: 'ciphertext', - i: { - c: 'iv', - t: 't', - }, - } - - const literal = encryptedToCompositeLiteral(encrypted) - // Should produce PostgreSQL composite literal format: ("json_string") - expect(literal).toMatch(/^\(.*\)$/) - // The inner content should be a valid JSON string (double-stringified) - const innerContent = literal.slice(1, -1) // Remove outer parentheses - expect(() => JSON.parse(innerContent)).not.toThrow() - // Parsing the inner content should give us the original JSON - const parsedJson = JSON.parse(JSON.parse(innerContent)) - expect(parsedJson).toEqual(encrypted) - }) - }) - - describe('isEncryptedPayload', () => { - it('should return true for valid encrypted payload', () => { - const encrypted = { - v: 1, - c: 'ciphertext', - i: { c: 'iv', t: 't' }, - } - expect(isEncryptedPayload(encrypted)).toBe(true) - }) - - it('should return false for null', () => { - expect(isEncryptedPayload(null)).toBe(false) - }) - - it('should return false for non-encrypted object', () => { - expect(isEncryptedPayload({ foo: 'bar' })).toBe(false) - }) - }) - - describe('isEncryptedScalarQuery', () => { - const baseIndex = { c: 'email', t: 'users' } - - it('returns true for a unique (hm) scalar query term', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - hm: 'abc123', - }), - ).toBe(true) - }) - - it('returns true for a match (bf) scalar query term', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - bf: [1, 2, 3], - }), - ).toBe(true) - }) - - it('returns true for an ore (ob) scalar query term', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - ob: ['a', 'b'], - }), - ).toBe(true) - }) - - it('returns false when the storage ciphertext (c) is present', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - c: 'ciphertext', - hm: 'abc123', - }), - ).toBe(false) - }) - - it('returns false when more than one lookup term is present', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - hm: 'abc123', - bf: [1, 2, 3], - }), - ).toBe(false) - }) - - it('returns false when no lookup term is present', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: baseIndex, - }), - ).toBe(false) - }) - - it('returns false for a ste_vec query term (k: "sv")', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'sv', - i: baseIndex, - sv: [{ s: 'selector', t: 'term' }], - }), - ).toBe(false) - }) - - it('returns false when version (v) is missing or not a number', () => { - expect( - isEncryptedScalarQuery({ - k: 'ct', - i: baseIndex, - hm: 'abc123', - }), - ).toBe(false) - expect( - isEncryptedScalarQuery({ - v: '2', - k: 'ct', - i: baseIndex, - hm: 'abc123', - }), - ).toBe(false) - }) - - it('returns false when index (i) is missing or null', () => { - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - hm: 'abc123', - }), - ).toBe(false) - expect( - isEncryptedScalarQuery({ - v: 2, - k: 'ct', - i: null, - hm: 'abc123', - }), - ).toBe(false) - }) - - it('returns false for null, primitives, and non-objects', () => { - expect(isEncryptedScalarQuery(null)).toBe(false) - expect(isEncryptedScalarQuery(undefined)).toBe(false) - expect(isEncryptedScalarQuery('string')).toBe(false) - expect(isEncryptedScalarQuery(42)).toBe(false) - }) - }) - - describe('modelToEncryptedPgComposites', () => { - it('should transform model with encrypted fields', () => { - const model = { - name: 'John', - email: { - v: 1, - c: 'encrypted_email', - i: { c: 'iv', t: 't' }, - }, - age: 30, - } - - const result = modelToEncryptedPgComposites(model) - expect(result).toEqual({ - name: 'John', - email: { - data: { - v: 1, - c: 'encrypted_email', - i: { c: 'iv', t: 't' }, - }, - }, - age: 30, - }) - }) - }) - - describe('bulkModelsToEncryptedPgComposites', () => { - it('should transform multiple models with encrypted fields', () => { - const models = [ - { - name: 'John', - email: { - v: 1, - c: 'encrypted_email1', - i: { c: 'iv', t: 't' }, - }, - }, - { - name: 'Jane', - email: { - v: 1, - c: 'encrypted_email2', - i: { c: 'iv', t: 't' }, - }, - }, - ] - - const result = bulkModelsToEncryptedPgComposites(models) - expect(result).toEqual([ - { - name: 'John', - email: { - data: { - v: 1, - c: 'encrypted_email1', - i: { c: 'iv', t: 't' }, - }, - }, - }, - { - name: 'Jane', - email: { - data: { - v: 1, - c: 'encrypted_email2', - i: { c: 'iv', t: 't' }, - }, - }, - }, - ]) - }) - }) -}) diff --git a/packages/protect/__tests__/infer-index-type.test.ts b/packages/protect/__tests__/infer-index-type.test.ts deleted file mode 100644 index ac9c9340f..000000000 --- a/packages/protect/__tests__/infer-index-type.test.ts +++ /dev/null @@ -1,94 +0,0 @@ -import { csColumn, csTable } from '@cipherstash/schema' -import { describe, expect, it } from 'vitest' -import { inferQueryOpFromPlaintext } from '../src/ffi/helpers/infer-index-type' -import { inferIndexType, validateIndexType } from '../src/index' - -describe('infer-index-type helpers', () => { - const users = csTable('users', { - email: csColumn('email').equality(), - bio: csColumn('bio').freeTextSearch(), - age: csColumn('age').orderAndRange(), - name: csColumn('name').equality().freeTextSearch(), - }) - - describe('inferIndexType', () => { - it('returns unique for equality-only column', () => { - expect(inferIndexType(users.email)).toBe('unique') - }) - - it('returns match for freeTextSearch-only column', () => { - expect(inferIndexType(users.bio)).toBe('match') - }) - - it('returns ore for orderAndRange-only column', () => { - expect(inferIndexType(users.age)).toBe('ore') - }) - - it('returns unique when multiple indexes (priority: unique > match > ore)', () => { - expect(inferIndexType(users.name)).toBe('unique') - }) - - it('returns match when freeTextSearch and orderAndRange (priority: match > ore)', () => { - const schema = csTable('t', { - col: csColumn('col').freeTextSearch().orderAndRange(), - }) - expect(inferIndexType(schema.col)).toBe('match') - }) - - it('throws for column with no indexes', () => { - const noIndex = csTable('t', { col: csColumn('col') }) - expect(() => inferIndexType(noIndex.col)).toThrow('no indexes configured') - }) - - it('returns ste_vec for searchableJson-only column', () => { - const schema = csTable('t', { col: csColumn('col').searchableJson() }) - expect(inferIndexType(schema.col)).toBe('ste_vec') - }) - }) - - describe('validateIndexType', () => { - it('does not throw for valid index type', () => { - expect(() => validateIndexType(users.email, 'unique')).not.toThrow() - }) - - it('throws for unconfigured index type', () => { - expect(() => validateIndexType(users.email, 'match')).toThrow( - 'not configured', - ) - }) - - it('accepts ste_vec when configured', () => { - const schema = csTable('t', { col: csColumn('col').searchableJson() }) - expect(() => validateIndexType(schema.col, 'ste_vec')).not.toThrow() - }) - - it('rejects ste_vec when not configured', () => { - const schema = csTable('t', { col: csColumn('col').equality() }) - expect(() => validateIndexType(schema.col, 'ste_vec')).toThrow( - 'not configured', - ) - }) - }) - - describe('inferQueryOpFromPlaintext', () => { - it('returns ste_vec_selector for string plaintext', () => { - expect(inferQueryOpFromPlaintext('$.user.email')).toBe('ste_vec_selector') - }) - - it('returns ste_vec_term for object plaintext', () => { - expect(inferQueryOpFromPlaintext({ role: 'admin' })).toBe('ste_vec_term') - }) - - it('returns ste_vec_term for array plaintext', () => { - expect(inferQueryOpFromPlaintext(['admin', 'user'])).toBe('ste_vec_term') - }) - - it('returns ste_vec_term for number plaintext', () => { - expect(inferQueryOpFromPlaintext(42)).toBe('ste_vec_term') - }) - - it('returns ste_vec_term for boolean plaintext', () => { - expect(inferQueryOpFromPlaintext(true)).toBe('ste_vec_term') - }) - }) -}) diff --git a/packages/protect/__tests__/json-protect.test.ts b/packages/protect/__tests__/json-protect.test.ts deleted file mode 100644 index 22fc203ec..000000000 --- a/packages/protect/__tests__/json-protect.test.ts +++ /dev/null @@ -1,1217 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable, csValue } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { LockContext, protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - json: csColumn('json').dataType('json'), - metadata: { - profile: csValue('metadata.profile').dataType('json'), - settings: { - preferences: csValue('metadata.settings.preferences').dataType('json'), - }, - }, -}) - -type User = { - id: string - email?: string | null - createdAt?: Date - updatedAt?: Date - address?: string | null - json?: Record | null - metadata?: { - profile?: Record | null - settings?: { - preferences?: Record | null - } - } -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('JSON encryption and decryption', () => { - it('should encrypt and decrypt a simple JSON payload', async () => { - const json = { - name: 'John Doe', - age: 30, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should encrypt and decrypt a complex JSON payload', async () => { - const json = { - user: { - id: 123, - name: 'Jane Smith', - email: 'jane@example.com', - preferences: { - theme: 'dark', - notifications: true, - language: 'en-US', - }, - tags: ['premium', 'verified'], - metadata: { - created: '2023-01-01T00:00:00Z', - lastLogin: '2023-12-01T10:30:00Z', - }, - }, - settings: { - privacy: { - public: false, - shareData: true, - }, - features: { - beta: true, - experimental: false, - }, - }, - array: [1, 2, 3, { nested: 'value' }], - nullValue: null, - booleanValue: true, - numberValue: 42.5, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle null JSON payload', async () => { - const ciphertext = await protectClient.encrypt(null, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify null is preserved - expect(ciphertext.data).toBeNull() - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: null, - }) - }, 30000) - - it('should handle empty JSON object', async () => { - const json = {} - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with special characters', async () => { - const json = { - message: 'Hello "world" with \'quotes\' and \\backslashes\\', - unicode: '🚀 emoji and ñ special chars', - symbols: '!@#$%^&*()_+-=[]{}|;:,.<>?/~`', - multiline: 'Line 1\nLine 2\tTabbed', - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) -}) - -describe('JSON model encryption and decryption', () => { - it('should encrypt and decrypt a model with JSON field', async () => { - const decryptedModel = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - json: { - name: 'John Doe', - age: 30, - preferences: { - theme: 'dark', - notifications: true, - }, - }, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.address).toHaveProperty('k', 'ct') - expect(encryptedModel.data.json).toHaveProperty('k', 'ct') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null JSON in model', async () => { - const decryptedModel = { - id: '2', - email: 'test2@example.com', - address: '456 Oak St', - json: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.address).toHaveProperty('k', 'ct') - expect(encryptedModel.data.json).toBeNull() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle undefined JSON in model', async () => { - const decryptedModel = { - id: '3', - email: 'test3@example.com', - address: '789 Pine St', - json: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.address).toHaveProperty('k', 'ct') - expect(encryptedModel.data.json).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) -}) - -describe('JSON bulk encryption and decryption', () => { - it('should bulk encrypt and decrypt JSON payloads', async () => { - const jsonPayloads = [ - { id: 'user1', plaintext: { name: 'Alice', age: 25 } }, - { id: 'user2', plaintext: { name: 'Bob', age: 30 } }, - { id: 'user3', plaintext: { name: 'Charlie', age: 35 } }, - ] - - const encryptedData = await protectClient.bulkEncrypt(jsonPayloads, { - column: users.json, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('k', 'ct') - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', { - name: 'Alice', - age: 25, - }) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', { - name: 'Bob', - age: 30, - }) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty('data', { - name: 'Charlie', - age: 35, - }) - }, 30000) - - it('should handle mixed null and non-null JSON in bulk operations', async () => { - const jsonPayloads = [ - { id: 'user1', plaintext: { name: 'Alice', age: 25 } }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: { name: 'Charlie', age: 35 } }, - ] - - const encryptedData = await protectClient.bulkEncrypt(jsonPayloads, { - column: users.json, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toBeNull() - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('k', 'ct') - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', { - name: 'Alice', - age: 25, - }) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', null) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty('data', { - name: 'Charlie', - age: 35, - }) - }, 30000) - - it('should bulk encrypt and decrypt models with JSON fields', async () => { - const decryptedModels = [ - { - id: '1', - email: 'test1@example.com', - address: '123 Main St', - json: { - name: 'Alice', - preferences: { theme: 'dark' }, - }, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - }, - { - id: '2', - email: 'test2@example.com', - address: '456 Oak St', - json: { - name: 'Bob', - preferences: { theme: 'light' }, - }, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('k', 'ct') - expect(encryptedModels.data[0].address).toHaveProperty('k', 'ct') - expect(encryptedModels.data[0].json).toHaveProperty('k', 'ct') - expect(encryptedModels.data[1].email).toHaveProperty('k', 'ct') - expect(encryptedModels.data[1].address).toHaveProperty('k', 'ct') - expect(encryptedModels.data[1].json).toHaveProperty('k', 'ct') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) -}) - -describe('JSON encryption with lock context', () => { - it('should encrypt and decrypt JSON with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const json = { - name: 'John Doe', - age: 30, - preferences: { - theme: 'dark', - notifications: true, - }, - } - - const ciphertext = await protectClient - .encrypt(json, { - column: users.json, - table: users, - }) - .withLockContext(lockContext.data) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient - .decrypt(ciphertext.data) - .withLockContext(lockContext.data) - - if (plaintext.failure) { - throw new Error(`[protect]: ${plaintext.failure.message}`) - } - - expect(plaintext.data).toEqual(json) - }, 30000) - - it('should encrypt JSON model with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const decryptedModel = { - id: '1', - email: 'test@example.com', - json: { - name: 'John Doe', - preferences: { theme: 'dark' }, - }, - } - - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.json).toHaveProperty('k', 'ct') - - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should bulk encrypt JSON with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const jsonPayloads = [ - { id: 'user1', plaintext: { name: 'Alice', age: 25 } }, - { id: 'user2', plaintext: { name: 'Bob', age: 30 } }, - ] - - const encryptedData = await protectClient - .bulkEncrypt(jsonPayloads, { - column: users.json, - table: users, - }) - .withLockContext(lockContext.data) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(2) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('k', 'ct') - - // Decrypt with lock context - const decryptedData = await protectClient - .bulkDecrypt(encryptedData.data) - .withLockContext(lockContext.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(2) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', { - name: 'Alice', - age: 25, - }) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', { - name: 'Bob', - age: 30, - }) - }, 30000) -}) - -describe('JSON nested object encryption', () => { - it('should encrypt and decrypt nested JSON objects', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - metadata: { - profile: { - name: 'John Doe', - age: 30, - preferences: { - theme: 'dark', - notifications: true, - }, - }, - settings: { - preferences: { - language: 'en-US', - timezone: 'UTC', - }, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.metadata?.profile).toHaveProperty('k', 'ct') - expect(encryptedModel.data.metadata?.settings?.preferences).toHaveProperty( - 'c', - ) - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null values in nested JSON objects', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '2', - email: 'test2@example.com', - metadata: { - profile: null, - settings: { - preferences: null, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify null fields are preserved - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.metadata?.profile).toBeNull() - expect(encryptedModel.data.metadata?.settings?.preferences).toBeNull() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle undefined values in nested JSON objects', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '3', - email: 'test3@example.com', - metadata: { - profile: undefined, - settings: { - preferences: undefined, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify undefined fields are preserved - expect(encryptedModel.data.email).toHaveProperty('k', 'ct') - expect(encryptedModel.data.metadata?.profile).toBeUndefined() - expect(encryptedModel.data.metadata?.settings?.preferences).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) -}) - -describe('JSON edge cases and error handling', () => { - it('should handle very large JSON objects', async () => { - const largeJson = { - data: Array.from({ length: 1000 }, (_, i) => ({ - id: i, - name: `User ${i}`, - email: `user${i}@example.com`, - metadata: { - preferences: { - theme: i % 2 === 0 ? 'dark' : 'light', - notifications: i % 3 === 0, - }, - }, - })), - metadata: { - total: 1000, - created: new Date().toISOString(), - }, - } - - const ciphertext = await protectClient.encrypt(largeJson, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: largeJson, - }) - }, 30000) - - it('should handle JSON with circular references (should fail gracefully)', async () => { - const circularObj: Record = { name: 'test' } - circularObj.self = circularObj - - try { - await protectClient.encrypt(circularObj, { - column: users.json, - table: users, - }) - // This should not reach here as JSON.stringify should fail - expect(true).toBe(false) - } catch (error) { - // Expected to fail due to circular reference - expect(error).toBeDefined() - } - }, 30000) - - it('should handle JSON with special data types', async () => { - const json = { - string: 'hello', - number: 42, - boolean: true, - null: null, - array: [1, 2, 3], - object: { nested: 'value' }, - date: new Date('2023-01-01T00:00:00Z'), - // Note: Functions and undefined are not JSON serializable - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - // Date objects get serialized to strings in JSON - const expectedJson = { - ...json, - date: '2023-01-01T00:00:00.000Z', - } - - expect(plaintext).toEqual({ - data: expectedJson, - }) - }, 30000) -}) - -describe('JSON performance tests', () => { - it('should handle large numbers of JSON objects efficiently', async () => { - const largeJsonArray = Array.from({ length: 100 }, (_, i) => ({ - id: i, - data: { - name: `User ${i}`, - preferences: { - theme: i % 2 === 0 ? 'dark' : 'light', - notifications: i % 3 === 0, - }, - metadata: { - created: new Date().toISOString(), - version: i, - }, - }, - })) - - const jsonPayloads = largeJsonArray.map((item, index) => ({ - id: `user${index}`, - plaintext: item, - })) - - const encryptedData = await protectClient.bulkEncrypt(jsonPayloads, { - column: users.json, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(100) - - // Decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify all data is preserved - expect(decryptedData.data).toHaveLength(100) - - for (let i = 0; i < 100; i++) { - expect(decryptedData.data[i].id).toBe(`user${i}`) - expect(decryptedData.data[i].data).toEqual(largeJsonArray[i]) - } - }, 5000) -}) - -describe('JSON advanced scenarios', () => { - it('should handle JSON with deeply nested arrays', async () => { - const json = { - users: [ - { - id: 1, - name: 'Alice', - roles: [ - { name: 'admin', permissions: ['read', 'write', 'delete'] }, - { name: 'user', permissions: ['read'] }, - ], - }, - { - id: 2, - name: 'Bob', - roles: [{ name: 'user', permissions: ['read'] }], - }, - ], - metadata: { - total: 2, - lastUpdated: new Date().toISOString(), - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with mixed data types in arrays', async () => { - const json = { - mixedArray: ['string', 42, true, null, { nested: 'object' }, [1, 2, 3]], - metadata: { - types: ['string', 'number', 'boolean', 'null', 'object', 'array'], - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with Unicode and international characters', async () => { - const json = { - international: { - chinese: '你好世界', - japanese: 'こんにちは世界', - korean: '안녕하세요 세계', - arabic: 'مرحبا بالعالم', - russian: 'Привет мир', - emoji: '🚀 🌍 💻 🎉', - }, - metadata: { - languages: ['Chinese', 'Japanese', 'Korean', 'Arabic', 'Russian'], - encoding: 'UTF-8', - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with scientific notation and large numbers', async () => { - const json = { - numbers: { - integer: 1234567890, - float: Math.PI, - scientific: 1.23e10, - negative: -9876543210, - zero: 0, - verySmall: 1.23e-10, - }, - metadata: { - precision: 'high', - format: 'scientific', - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with boolean edge cases', async () => { - const json = { - booleans: { - true: true, - false: false, - stringTrue: 'true', - stringFalse: 'false', - numberOne: 1, - numberZero: 0, - emptyString: '', - nullValue: null, - }, - metadata: { - type: 'boolean_edge_cases', - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) -}) - -describe('JSON error handling and edge cases', () => { - it('should handle malformed JSON gracefully', async () => { - // This test ensures the library handles JSON serialization errors - const invalidJson = { - valid: 'data', - // This will cause JSON.stringify to fail - circular: null as unknown, - } - - // Create a circular reference - invalidJson.circular = invalidJson - - try { - await protectClient.encrypt(invalidJson, { - column: users.json, - table: users, - }) - expect(true).toBe(false) // Should not reach here - } catch (error) { - expect(error).toBeDefined() - expect(error).toBeInstanceOf(Error) - } - }, 30000) - - it('should handle empty arrays and objects', async () => { - const json = { - emptyArray: [], - emptyObject: {}, - nestedEmpty: { - array: [], - object: {}, - }, - mixedEmpty: { - data: 'present', - empty: [], - null: null, - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with very long strings', async () => { - const longString = 'A'.repeat(10000) // 10KB string - const json = { - longString, - metadata: { - length: longString.length, - type: 'long_string', - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) - - it('should handle JSON with all primitive types', async () => { - const json = { - string: 'hello world', - number: 42, - float: 3.14, - boolean: true, - null: null, - array: [1, 2, 3], - object: { key: 'value' }, - nested: { - level1: { - level2: { - level3: 'deep value', - }, - }, - }, - } - - const ciphertext = await protectClient.encrypt(json, { - column: users.json, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('k', 'ct') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: json, - }) - }, 30000) -}) diff --git a/packages/protect/__tests__/jsonb-helpers.test.ts b/packages/protect/__tests__/jsonb-helpers.test.ts deleted file mode 100644 index 9f12d131f..000000000 --- a/packages/protect/__tests__/jsonb-helpers.test.ts +++ /dev/null @@ -1,203 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { buildNestedObject, parseJsonbPath, toJsonPath } from '../src' - -describe('toJsonPath', () => { - it('converts simple path to JSONPath format', () => { - expect(toJsonPath('name')).toBe('$.name') - }) - - it('converts nested path to JSONPath format', () => { - expect(toJsonPath('user.email')).toBe('$.user.email') - }) - - it('converts deeply nested path', () => { - expect(toJsonPath('user.profile.settings.theme')).toBe( - '$.user.profile.settings.theme', - ) - }) - - it('returns unchanged if already in JSONPath format', () => { - expect(toJsonPath('$.user.email')).toBe('$.user.email') - }) - - it('normalizes bare $ prefix', () => { - expect(toJsonPath('$user.email')).toBe('$.user.email') - }) - - it('handles path starting with dot', () => { - expect(toJsonPath('.user.email')).toBe('$.user.email') - }) - - it('handles root path', () => { - expect(toJsonPath('$')).toBe('$') - }) - - it('handles empty string', () => { - expect(toJsonPath('')).toBe('$') - }) - - it('handles array index in path', () => { - expect(toJsonPath('user.roles[0]')).toBe('$.user.roles[0]') - }) - - it('handles array index with nested property', () => { - expect(toJsonPath('items[0].name')).toBe('$.items[0].name') - }) - - it('handles already-prefixed path with array index', () => { - expect(toJsonPath('$.data[2]')).toBe('$.data[2]') - }) - - it('handles nested array indices', () => { - expect(toJsonPath('matrix[0][1]')).toBe('$.matrix[0][1]') - }) - - it('handles array index at root level', () => { - expect(toJsonPath('[0].name')).toBe('$[0].name') - }) - - it('preserves already-prefixed root array index', () => { - expect(toJsonPath('$[0]')).toBe('$[0]') - }) - - it('preserves already-prefixed root array index with property', () => { - expect(toJsonPath('$[0].name')).toBe('$[0].name') - }) - - it('handles large array index', () => { - expect(toJsonPath('items[999].value')).toBe('$.items[999].value') - }) - - it('handles deeply nested path after array index', () => { - expect(toJsonPath('data[0].user.profile.settings')).toBe( - '$.data[0].user.profile.settings', - ) - }) - - it('handles root array with nested array', () => { - expect(toJsonPath('[0].items[1].name')).toBe('$[0].items[1].name') - }) -}) - -describe('buildNestedObject', () => { - it('builds single-level object', () => { - expect(buildNestedObject('name', 'alice')).toEqual({ name: 'alice' }) - }) - - it('builds two-level nested object', () => { - expect(buildNestedObject('user.role', 'admin')).toEqual({ - user: { role: 'admin' }, - }) - }) - - it('builds deeply nested object', () => { - expect(buildNestedObject('a.b.c.d', 'value')).toEqual({ - a: { b: { c: { d: 'value' } } }, - }) - }) - - it('handles numeric values', () => { - expect(buildNestedObject('user.age', 30)).toEqual({ - user: { age: 30 }, - }) - }) - - it('handles boolean values', () => { - expect(buildNestedObject('user.active', true)).toEqual({ - user: { active: true }, - }) - }) - - it('handles null values', () => { - expect(buildNestedObject('user.data', null)).toEqual({ - user: { data: null }, - }) - }) - - it('handles object values', () => { - const value = { nested: 'object' } - expect(buildNestedObject('user.config', value)).toEqual({ - user: { config: { nested: 'object' } }, - }) - }) - - it('handles array values', () => { - expect(buildNestedObject('user.tags', ['admin', 'user'])).toEqual({ - user: { tags: ['admin', 'user'] }, - }) - }) - - it('strips JSONPath prefix from path', () => { - expect(buildNestedObject('$.user.role', 'admin')).toEqual({ - user: { role: 'admin' }, - }) - }) - - it('throws on empty path', () => { - expect(() => buildNestedObject('', 'value')).toThrow('Path cannot be empty') - }) - - it('throws on root-only path', () => { - expect(() => buildNestedObject('$', 'value')).toThrow( - 'Path must contain at least one segment', - ) - }) - - it('throws on __proto__ segment', () => { - expect(() => buildNestedObject('__proto__.polluted', 'yes')).toThrow( - 'Path contains forbidden segment: __proto__', - ) - }) - - it('throws on prototype segment', () => { - expect(() => buildNestedObject('user.prototype.hack', 'yes')).toThrow( - 'Path contains forbidden segment: prototype', - ) - }) - - it('throws on constructor segment', () => { - expect(() => buildNestedObject('constructor', 'yes')).toThrow( - 'Path contains forbidden segment: constructor', - ) - }) - - it('throws on nested forbidden segment', () => { - expect(() => buildNestedObject('a.b.__proto__', 'yes')).toThrow( - 'Path contains forbidden segment: __proto__', - ) - }) -}) - -describe('parseJsonbPath', () => { - it('parses simple path', () => { - expect(parseJsonbPath('name')).toEqual(['name']) - }) - - it('parses nested path', () => { - expect(parseJsonbPath('user.email')).toEqual(['user', 'email']) - }) - - it('parses deeply nested path', () => { - expect(parseJsonbPath('a.b.c.d')).toEqual(['a', 'b', 'c', 'd']) - }) - - it('strips JSONPath prefix', () => { - expect(parseJsonbPath('$.user.email')).toEqual(['user', 'email']) - }) - - it('strips bare $ prefix', () => { - expect(parseJsonbPath('$user.email')).toEqual(['user', 'email']) - }) - - it('handles empty string', () => { - expect(parseJsonbPath('')).toEqual([]) - }) - - it('handles root only', () => { - expect(parseJsonbPath('$')).toEqual([]) - }) - - it('filters empty segments', () => { - expect(parseJsonbPath('user..email')).toEqual(['user', 'email']) - }) -}) diff --git a/packages/protect/__tests__/k-discriminator.test.ts b/packages/protect/__tests__/k-discriminator.test.ts deleted file mode 100644 index 002ff842f..000000000 --- a/packages/protect/__tests__/k-discriminator.test.ts +++ /dev/null @@ -1,55 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { protect } from '../src' - -const users = csTable('users', { - email: csColumn('email'), -}) - -describe('k-field discriminator (EQL v2.3)', () => { - let protectClient: Awaited> - - beforeAll(async () => { - protectClient = await protect({ schemas: [users] }) - }) - - it('encrypts scalar data with k: "ct" discriminator', async () => { - const testData = 'test@example.com' - - const result = await protectClient.encrypt(testData, { - column: users.email, - table: users, - }) - - if (result.failure) { - throw new Error(`Encryption failed: ${result.failure.message}`) - } - - expect(result.data).toHaveProperty('k', 'ct') - expect(result.data).toHaveProperty('c') - expect(result.data).toHaveProperty('v') - expect(result.data).toHaveProperty('i') - }, 30000) - - it('decrypts a payload round-trips back to the original plaintext', async () => { - const testData = 'roundtrip@example.com' - - const encrypted = await protectClient.encrypt(testData, { - column: users.email, - table: users, - }) - - if (encrypted.failure) { - throw new Error(`Encryption failed: ${encrypted.failure.message}`) - } - - const result = await protectClient.decrypt(encrypted.data!) - - if (result.failure) { - throw new Error(`Decryption failed: ${result.failure.message}`) - } - - expect(result.data).toBe(testData) - }, 30000) -}) diff --git a/packages/protect/__tests__/keysets.test.ts b/packages/protect/__tests__/keysets.test.ts deleted file mode 100644 index e8f4e0fed..000000000 --- a/packages/protect/__tests__/keysets.test.ts +++ /dev/null @@ -1,97 +0,0 @@ -import 'dotenv/config' -import { ensureKeyset } from '@cipherstash/protect-ffi' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { protect } from '../src' - -const users = csTable('users', { - email: csColumn('email'), -}) - -let testKeysetId: string - -beforeAll(async () => { - const keyset = await ensureKeyset({ name: 'Test' }) - testKeysetId = keyset.id -}) - -describe('encryption and decryption with keyset id', () => { - it('should encrypt and decrypt a payload', async () => { - const protectClient = await protect({ - schemas: [users], - keyset: { - id: testKeysetId, - }, - }) - - const email = 'hello@example.com' - - const ciphertext = await protectClient.encrypt(email, { - column: users.email, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const a = ciphertext.data - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: email, - }) - }, 30000) -}) - -describe('encryption and decryption with keyset name', () => { - it('should encrypt and decrypt a payload', async () => { - const protectClient = await protect({ - schemas: [users], - keyset: { - name: 'Test', - }, - }) - - const email = 'hello@example.com' - - const ciphertext = await protectClient.encrypt(email, { - column: users.email, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const a = ciphertext.data - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: email, - }) - }, 30000) -}) - -describe('encryption and decryption with invalid keyset id', () => { - it('should throw an error', async () => { - await expect( - protect({ - schemas: [users], - keyset: { - id: 'invalid-uuid', - }, - }), - ).rejects.toThrow( - '[protect]: Invalid UUID provided for keyset id. Must be a valid UUID.', - ) - }) -}) diff --git a/packages/protect/__tests__/lock-context.test.ts b/packages/protect/__tests__/lock-context.test.ts deleted file mode 100644 index 4a0894b29..000000000 --- a/packages/protect/__tests__/lock-context.test.ts +++ /dev/null @@ -1,208 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { protect } from '../src' -import { LockContext } from '../src/identify' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), -}) - -type User = { - id: string - email?: string | null - createdAt?: Date - updatedAt?: Date - address?: string | null - number?: number -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('encryption and decryption with lock context', () => { - it('should encrypt and decrypt a payload with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const email = 'hello@example.com' - - const ciphertext = await protectClient - .encrypt(email, { - column: users.email, - table: users, - }) - .withLockContext(lockContext.data) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - const plaintext = await protectClient - .decrypt(ciphertext.data) - .withLockContext(lockContext.data) - - if (plaintext.failure) { - throw new Error(`[protect]: ${plaintext.failure.message}`) - } - - expect(plaintext.data).toEqual(email) - }, 30000) - - it('should encrypt and decrypt a model with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create a model with decrypted values - const decryptedModel = { - id: '1', - email: 'plaintext', - } - - // Encrypt the model with lock context - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Decrypt the model with lock context - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual({ - id: '1', - email: 'plaintext', - }) - }, 30000) - - it('should encrypt with context and be unable to decrypt without context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create a model with decrypted values - const decryptedModel = { - id: '1', - email: 'plaintext', - } - - // Encrypt the model with lock context - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - try { - await protectClient.decryptModel(encryptedModel.data) - } catch (error) { - const e = error as Error - expect(e.message.startsWith('Failed to retrieve key')).toEqual(true) - } - }, 30000) - - it('should bulk encrypt and decrypt models with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create models with decrypted values - const decryptedModels = [ - { - id: '1', - email: 'test', - }, - { - id: '2', - email: 'test2', - }, - ] - - // Encrypt the models with lock context - const encryptedModels = await protectClient - .bulkEncryptModels(decryptedModels, users) - .withLockContext(lockContext.data) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Decrypt the models with lock context - const decryptedResult = await protectClient - .bulkDecryptModels(encryptedModels.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual([ - { - id: '1', - email: 'test', - }, - { - id: '2', - email: 'test2', - }, - ]) - }, 30000) -}) diff --git a/packages/protect/__tests__/nested-models.test.ts b/packages/protect/__tests__/nested-models.test.ts deleted file mode 100644 index 8f44f809b..000000000 --- a/packages/protect/__tests__/nested-models.test.ts +++ /dev/null @@ -1,958 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable, csValue } from '@cipherstash/schema' -import { describe, expect, it, vi } from 'vitest' -import { LockContext, protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - name: csColumn('name').freeTextSearch(), - example: { - field: csValue('example.field'), - nested: { - deeper: csValue('example.nested.deeper'), - }, - }, -}) - -type User = { - id: string - email?: string | null - createdAt?: Date - updatedAt?: Date - address?: string | null - notEncrypted?: string | null - example: { - field: string | undefined | null - nested?: { - deeper: string | undefined | null - plaintext?: string | undefined | null - notInSchema?: { - deeper: string | undefined | null - } - deeperNotInSchema?: string | undefined | null - extra?: { - plaintext: string | undefined | null - } - } - plaintext?: string | undefined | null - fieldNotInSchema?: string | undefined | null - notInSchema?: { - deeper: string | undefined | null - } - } -} - -describe('encrypt models with nested fields', () => { - it('should encrypt and decrypt a single value from a nested schema', async () => { - const protectClient = await protect({ schemas: [users] }) - - const encryptResponse = await protectClient.encrypt('hello world', { - column: users.example.field, - table: users, - }) - - if (encryptResponse.failure) { - throw new Error(`[protect]: ${encryptResponse.failure.message}`) - } - - // Verify encrypted field - expect(encryptResponse.data).toHaveProperty('c') - - const decryptResponse = await protectClient.decrypt(encryptResponse.data) - - if (decryptResponse.failure) { - throw new Error(`[protect]: ${decryptResponse.failure.message}`) - } - - expect(decryptResponse).toEqual({ - data: 'hello world', - }) - }) - - it('should encrypt and decrypt a model with nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - notEncrypted: 'not encrypted', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - example: { - field: 'test', - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.example.field).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.notEncrypted).toBe('not encrypted') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null values in nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '2', - email: null, - address: null, - example: { - field: null, - nested: { - deeper: null, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify null fields are preserved - expect(encryptedModel.data.email).toBeNull() - expect(encryptedModel.data.address).toBeNull() - expect(encryptedModel.data.example.field).toBeNull() - expect(encryptedModel.data.example.nested?.deeper).toBeNull() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle undefined values in nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '3', - example: { - field: undefined, - nested: { - deeper: undefined, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify undefined fields are preserved - expect(encryptedModel.data.email).toBeUndefined() - expect(encryptedModel.data.example.field).toBeUndefined() - expect(encryptedModel.data.example.nested?.deeper).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle mixed null and undefined values in nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '4', - email: 'test@example.com', - address: undefined, - notEncrypted: 'not encrypted', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - example: { - field: null, - nested: { - deeper: undefined, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - - // Verify null/undefined fields are preserved - expect(encryptedModel.data.address).toBeUndefined() - expect(encryptedModel.data.example.field).toBeNull() - expect(encryptedModel.data.example.nested?.deeper).toBeUndefined() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('4') - expect(encryptedModel.data.notEncrypted).toBe('not encrypted') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle deeply nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '3', - example: { - field: 'outer', - nested: { - deeper: 'inner value', - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.example.field).toHaveProperty('c') - expect(encryptedModel.data.example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('3') - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle missing optional nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '5', - example: { - field: 'present', - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.example.field).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('5') - expect(encryptedModel.data.example.nested).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - describe('bulk operations with nested fields', () => { - it('should handle bulk encryption and decryption of models with nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - example: { - field: 'test1', - nested: { - deeper: 'value1', - }, - }, - }, - { - id: '2', - email: 'test2@example.com', - example: { - field: 'test2', - nested: { - deeper: 'value2', - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].example.field).toHaveProperty('c') - expect(encryptedModels.data[0].example.nested?.deeper).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[1].id).toBe('2') - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }, 30000) - - it('should handle bulk operations with null and undefined values in nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: null, - example: { - field: null, - nested: { - deeper: undefined, - }, - }, - }, - { - id: '2', - email: undefined, - example: { - field: undefined, - nested: { - deeper: null, - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify null/undefined fields are preserved - expect(encryptedModels.data[0].email).toBeNull() - expect(encryptedModels.data[0].example.field).toBeNull() - expect(encryptedModels.data[0].example.nested?.deeper).toBeUndefined() - expect(encryptedModels.data[1].email).toBeUndefined() - expect(encryptedModels.data[1].example.field).toBeUndefined() - expect(encryptedModels.data[1].example.nested?.deeper).toBeNull() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[1].id).toBe('2') - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }, 30000) - - it('should handle bulk operations with missing optional nested fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - example: { - field: 'test1', - }, - }, - { - id: '2', - email: 'test2@example.com', - example: { - field: 'test2', - nested: { - deeper: 'value2', - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].example.nested).toBeUndefined() - expect(encryptedModels.data[1].id).toBe('2') - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }, 30000) - - it('should handle empty array in bulk operations', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - expect(encryptedModels.data).toEqual([]) - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual([]) - }, 30000) - }) -}) - -describe('nested fields with a plaintext field', () => { - it('should handle nested fields with a plaintext field', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - notEncrypted: 'not encrypted', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - example: { - field: 'test', - plaintext: 'plaintext', - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.example.field).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.notEncrypted).toBe('not encrypted') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.example.plaintext).toBe('plaintext') - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }) - - it('should handle multiple plaintext fields at different nesting levels', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - notEncrypted: 'not encrypted', - example: { - field: 'encrypted field', - plaintext: 'top level plaintext', - nested: { - deeper: 'encrypted deeper', - plaintext: 'nested plaintext', - extra: { - plaintext: 'deeply nested plaintext', - }, - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.example.field).toHaveProperty('c') - expect(encryptedModel.data.example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.notEncrypted).toBe('not encrypted') - expect(encryptedModel.data.example.plaintext).toBe('top level plaintext') - expect(encryptedModel.data.example.nested?.plaintext).toBe( - 'nested plaintext', - ) - expect(encryptedModel.data.example.nested?.extra?.plaintext).toBe( - 'deeply nested plaintext', - ) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }) - - it('should handle partial path matches in nested objects', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - example: { - field: 'encrypted field', - nested: { - deeper: 'encrypted deeper', - // This should not be encrypted as it's not in the schema - notInSchema: { - deeper: 'not encrypted', - }, - }, - // This should not be encrypted as it's not in the schema - notInSchema: { - deeper: 'not encrypted', - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.example.field).toHaveProperty('c') - expect(encryptedModel.data.example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.example.nested?.notInSchema?.deeper).toBe( - 'not encrypted', - ) - expect(encryptedModel.data.example.notInSchema?.deeper).toBe( - 'not encrypted', - ) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }) - - it('should handle mixed encrypted and plaintext fields with similar paths', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - example: { - field: 'encrypted field', - fieldNotInSchema: 'not encrypted', - nested: { - deeper: 'encrypted deeper', - deeperNotInSchema: 'not encrypted', - }, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.example.field).toHaveProperty('c') - expect(encryptedModel.data.example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.example.fieldNotInSchema).toBe('not encrypted') - expect(encryptedModel.data.example.nested?.deeperNotInSchema).toBe( - 'not encrypted', - ) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }) - - describe('bulk operations with plaintext fields', () => { - it('should handle bulk encryption and decryption with plaintext fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - address: '123 Main St', - example: { - field: 'encrypted field 1', - plaintext: 'plaintext 1', - nested: { - deeper: 'encrypted deeper 1', - plaintext: 'nested plaintext 1', - }, - }, - }, - { - id: '2', - email: 'test2@example.com', - address: '456 Main St', - example: { - field: 'encrypted field 2', - plaintext: 'plaintext 2', - nested: { - deeper: 'encrypted deeper 2', - plaintext: 'nested plaintext 2', - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toHaveProperty('c') - expect(encryptedModels.data[0].example.field).toHaveProperty('c') - expect(encryptedModels.data[0].example.nested?.deeper).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[1].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].example.plaintext).toBe('plaintext 1') - expect(encryptedModels.data[0].example.nested?.plaintext).toBe( - 'nested plaintext 1', - ) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].example.plaintext).toBe('plaintext 2') - expect(encryptedModels.data[1].example.nested?.plaintext).toBe( - 'nested plaintext 2', - ) - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }) - - it('should handle bulk operations with mixed encrypted and non-encrypted fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - example: { - field: 'encrypted field 1', - fieldNotInSchema: 'not encrypted 1', - nested: { - deeper: 'encrypted deeper 1', - deeperNotInSchema: 'not encrypted deeper 1', - }, - }, - }, - { - id: '2', - email: 'test2@example.com', - example: { - field: 'encrypted field 2', - fieldNotInSchema: 'not encrypted 2', - nested: { - deeper: 'encrypted deeper 2', - deeperNotInSchema: 'not encrypted deeper 2', - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].example.field).toHaveProperty('c') - expect(encryptedModels.data[0].example.nested?.deeper).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].example.fieldNotInSchema).toBe( - 'not encrypted 1', - ) - expect(encryptedModels.data[0].example.nested?.deeperNotInSchema).toBe( - 'not encrypted deeper 1', - ) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].example.fieldNotInSchema).toBe( - 'not encrypted 2', - ) - expect(encryptedModels.data[1].example.nested?.deeperNotInSchema).toBe( - 'not encrypted deeper 2', - ) - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }) - - it('should handle bulk operations with deeply nested plaintext fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModels: User[] = [ - { - id: '1', - email: 'test1@example.com', - example: { - field: 'encrypted field 1', - nested: { - deeper: 'encrypted deeper 1', - extra: { - plaintext: 'deeply nested plaintext 1', - }, - }, - }, - }, - { - id: '2', - email: 'test2@example.com', - example: { - field: 'encrypted field 2', - nested: { - deeper: 'encrypted deeper 2', - extra: { - plaintext: 'deeply nested plaintext 2', - }, - }, - }, - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].example.field).toHaveProperty('c') - expect(encryptedModels.data[0].example.nested?.deeper).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].example.field).toHaveProperty('c') - expect(encryptedModels.data[1].example.nested?.deeper).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].example.nested?.extra?.plaintext).toBe( - 'deeply nested plaintext 1', - ) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].example.nested?.extra?.plaintext).toBe( - 'deeply nested plaintext 2', - ) - - const decryptedResults = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResults.failure) { - throw new Error(`[protect]: ${decryptedResults.failure.message}`) - } - - expect(decryptedResults.data).toEqual(decryptedModels) - }) - }) -}) diff --git a/packages/protect/__tests__/number-protect.test.ts b/packages/protect/__tests__/number-protect.test.ts deleted file mode 100644 index 754e42e6d..000000000 --- a/packages/protect/__tests__/number-protect.test.ts +++ /dev/null @@ -1,823 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable, csValue } from '@cipherstash/schema' -import { beforeAll, describe, expect, it, test } from 'vitest' -import { LockContext, protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - age: csColumn('age').dataType('number').equality().orderAndRange(), - score: csColumn('score').dataType('number').equality().orderAndRange(), - metadata: { - count: csValue('metadata.count').dataType('number'), - level: csValue('metadata.level').dataType('number'), - }, -}) - -type User = { - id: string - email?: string - createdAt?: Date - updatedAt?: Date - address?: string - age?: number - score?: number - metadata?: { - count?: number - level?: number - } -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -const cases = [ - 25, - 0, - -42, - 2147483647, - 77.9, - 0.0, - -117.123456, - 1e15, - -1e15, // Very large floats - 9007199254740991, // Max safe integer in JavaScript -] - -describe('Number encryption and decryption', () => { - test.each(cases)('should encrypt and decrypt a number: %d', async (age) => { - const ciphertext = await protectClient.encrypt(age, { - column: users.age, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: age, - }) - }, 30000) - - it('should handle null integer', async () => { - const ciphertext = await protectClient.encrypt(null, { - column: users.age, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify null is preserved - expect(ciphertext.data).toBeNull() - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: null, - }) - }, 30000) - - // Special case - it('should treat a negative zero valued float as 0.0', async () => { - const score = -0.0 - - const ciphertext = await protectClient.encrypt(score, { - column: users.score, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: 0.0, - }) - }, 30000) - - // Special case - it('should error for a NaN float', async () => { - const score = Number.NaN - - const result = await protectClient.encrypt(score, { - column: users.score, - table: users, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.message).toContain('Cannot encrypt NaN value') - }, 30000) - - // Special case - it('should error for Infinity', async () => { - const score = Number.POSITIVE_INFINITY - - const result = await protectClient.encrypt(score, { - column: users.score, - table: users, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.message).toContain('Cannot encrypt Infinity value') - }, 30000) - - // Special case - it('should error for -Infinity', async () => { - const score = Number.NEGATIVE_INFINITY - - const result = await protectClient.encrypt(score, { - column: users.score, - table: users, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.message).toContain('Cannot encrypt Infinity value') - }, 30000) -}) - -describe('Model encryption and decryption', () => { - it('should encrypt and decrypt a model with number fields', async () => { - const decryptedModel = { - id: '1', - email: 'test@example.com', - address: '123 Main St', - age: 30, - score: 95, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.age).toHaveProperty('c') - expect(encryptedModel.data.score).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null numbers in model', async () => { - const decryptedModel: User = { - id: '2', - email: 'test2@example.com', - address: '456 Oak St', - age: undefined, - score: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.age).toBeUndefined() - expect(encryptedModel.data.score).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle undefined numbers in model', async () => { - const decryptedModel = { - id: '3', - email: 'test3@example.com', - address: '789 Pine St', - age: undefined, - score: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - expect(encryptedModel.data.age).toBeUndefined() - expect(encryptedModel.data.score).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) -}) - -describe('Bulk encryption and decryption', () => { - it('should bulk encrypt and decrypt number payloads', async () => { - const intPayloads = [ - { id: 'user1', plaintext: 25 }, - { id: 'user2', plaintext: 30.7 }, - { id: 'user3', plaintext: -35.123 }, - ] - - const encryptedData = await protectClient.bulkEncrypt(intPayloads, { - column: users.age, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('c') - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('c') - - // EQL v2.3: scalar encryptions carry the `k: 'ct'` discriminator - expect(encryptedData.data[0].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[1].data).toHaveProperty('k', 'ct') - expect(encryptedData.data[2].data).toHaveProperty('k', 'ct') - - // Verify all encrypted values are different - const getCiphertext = (data: { c?: unknown } | null | undefined) => data?.c - - expect(getCiphertext(encryptedData.data[0].data)).not.toBe( - getCiphertext(encryptedData.data[1].data), - ) - expect(getCiphertext(encryptedData.data[1].data)).not.toBe( - getCiphertext(encryptedData.data[2].data), - ) - expect(getCiphertext(encryptedData.data[0].data)).not.toBe( - getCiphertext(encryptedData.data[2].data), - ) - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 25) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', 30.7) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty('data', -35.123) - }, 30000) - - it('should handle mixed null and non-null numbers in bulk operations', async () => { - const intPayloads = [ - { id: 'user1', plaintext: 25 }, - { id: 'user2', plaintext: null }, - { id: 'user3', plaintext: 35 }, - ] - - const encryptedData = await protectClient.bulkEncrypt(intPayloads, { - column: users.age, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(3) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toBeNull() - expect(encryptedData.data[2]).toHaveProperty('id', 'user3') - expect(encryptedData.data[2]).toHaveProperty('data') - expect(encryptedData.data[2].data).toHaveProperty('c') - - // Now decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(3) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 25) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', null) - expect(decryptedData.data[2]).toHaveProperty('id', 'user3') - expect(decryptedData.data[2]).toHaveProperty('data', 35) - }, 30000) - - it('should bulk encrypt and decrypt models with number fields', async () => { - const decryptedModels = [ - { - id: '1', - email: 'test1@example.com', - address: '123 Main St', - age: 25, - score: 85, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - }, - { - id: '2', - email: 'test2@example.com', - address: '456 Oak St', - age: 30, - score: 90, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toHaveProperty('c') - expect(encryptedModels.data[0].age).toHaveProperty('c') - expect(encryptedModels.data[0].score).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[1].age).toHaveProperty('c') - expect(encryptedModels.data[1].score).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) -}) - -describe('Encryption with lock context', () => { - it('should encrypt and decrypt number with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const age = 42 - - const ciphertext = await protectClient - .encrypt(age, { - column: users.age, - table: users, - }) - .withLockContext(lockContext.data) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const plaintext = await protectClient - .decrypt(ciphertext.data) - .withLockContext(lockContext.data) - - if (plaintext.failure) { - throw new Error(`[protect]: ${plaintext.failure.message}`) - } - - expect(plaintext.data).toEqual(age) - }, 30000) - - it('should encrypt model with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const decryptedModel = { - id: '1', - email: 'test@example.com', - age: 30, - score: 95, - } - - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.age).toHaveProperty('c') - expect(encryptedModel.data.score).toHaveProperty('c') - - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should bulk encrypt numbers with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const intPayloads = [ - { id: 'user1', plaintext: 25 }, - { id: 'user2', plaintext: 30 }, - ] - - const encryptedData = await protectClient - .bulkEncrypt(intPayloads, { - column: users.age, - table: users, - }) - .withLockContext(lockContext.data) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(2) - expect(encryptedData.data[0]).toHaveProperty('id', 'user1') - expect(encryptedData.data[0]).toHaveProperty('data') - expect(encryptedData.data[0].data).toHaveProperty('c') - expect(encryptedData.data[1]).toHaveProperty('id', 'user2') - expect(encryptedData.data[1]).toHaveProperty('data') - expect(encryptedData.data[1].data).toHaveProperty('c') - - // Decrypt with lock context - const decryptedData = await protectClient - .bulkDecrypt(encryptedData.data) - .withLockContext(lockContext.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify decrypted data - expect(decryptedData.data).toHaveLength(2) - expect(decryptedData.data[0]).toHaveProperty('id', 'user1') - expect(decryptedData.data[0]).toHaveProperty('data', 25) - expect(decryptedData.data[1]).toHaveProperty('id', 'user2') - expect(decryptedData.data[1]).toHaveProperty('data', 30) - }, 30000) -}) - -describe('Nested object encryption', () => { - it('should encrypt and decrypt nested number objects', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '1', - email: 'test@example.com', - metadata: { - count: 100, - level: 5, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.metadata?.count).toHaveProperty('c') - expect(encryptedModel.data.metadata?.level).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle null values in nested objects with number fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel: User = { - id: '2', - email: 'test2@example.com', - metadata: { - count: undefined, - level: undefined, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify null fields are preserved - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.metadata?.count).toBeUndefined() - expect(encryptedModel.data.metadata?.level).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) - - it('should handle undefined values in nested objects with number fields', async () => { - const protectClient = await protect({ schemas: [users] }) - - const decryptedModel = { - id: '3', - email: 'test3@example.com', - metadata: { - count: undefined, - level: undefined, - }, - } - - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify undefined fields are preserved - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.metadata?.count).toBeUndefined() - expect(encryptedModel.data.metadata?.level).toBeUndefined() - - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModel) - }, 30000) -}) - -describe('encryptQuery for numbers', () => { - it('should create encrypted query for number fields', async () => { - const result = await protectClient.encryptQuery([ - { value: 25, column: users.age, table: users, queryType: 'equality' }, - { value: 100, column: users.score, table: users, queryType: 'equality' }, - ]) - - if (result.failure) { - throw new Error(`[protect]: ${result.failure.message}`) - } - - expect(result.data).toHaveLength(2) - expect(result.data[0]).toHaveProperty('v', 2) - expect(result.data[1]).toHaveProperty('v', 2) - }, 30000) -}) - -describe('Performance tests', () => { - it('should handle large numbers of numbers efficiently', async () => { - const largeNumArray = Array.from({ length: 100 }, (_, i) => ({ - id: i, - data: { - age: i + 18, // Ages 18-117 - score: (i % 100) + 1, // Scores 1-100 - }, - })) - - const numPayloads = largeNumArray.map((item, index) => ({ - id: `user${index}`, - plaintext: item.data.age, - })) - - const encryptedData = await protectClient.bulkEncrypt(numPayloads, { - column: users.age, - table: users, - }) - - if (encryptedData.failure) { - throw new Error(`[protect]: ${encryptedData.failure.message}`) - } - - // Verify structure - expect(encryptedData.data).toHaveLength(100) - - // Decrypt the data - const decryptedData = await protectClient.bulkDecrypt(encryptedData.data) - - if (decryptedData.failure) { - throw new Error(`[protect]: ${decryptedData.failure.message}`) - } - - // Verify all data is preserved - expect(decryptedData.data).toHaveLength(100) - - for (let i = 0; i < 100; i++) { - expect(decryptedData.data[i].id).toBe(`user${i}`) - expect(decryptedData.data[i].data).toEqual(largeNumArray[i].data.age) - } - }, 60000) -}) - -describe('Advanced scenarios', () => { - it('should handle boundary values', async () => { - const boundaryValues = [ - Number.MIN_SAFE_INTEGER, - -2147483648, // Min 32-bit signed integer - -1, - 0, - 1, - 2147483647, // Max 32-bit signed integer - Number.MAX_SAFE_INTEGER, - ] - - for (const value of boundaryValues) { - const ciphertext = await protectClient.encrypt(value, { - column: users.age, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify encrypted field - expect(ciphertext.data).toHaveProperty('c') - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: value, - }) - } - }, 30000) -}) - -const invalidPlaintexts = [ - '400', - 'aaa', - '100a', - '73.51', - {}, - [], - [123], - { num: 123 }, -] - -describe('Invalid or uncoercable values', () => { - test.each(invalidPlaintexts)('should fail to encrypt', async (input) => { - const result = await protectClient.encrypt(input, { - column: users.age, - table: users, - }) - - expect(result.failure).toBeDefined() - expect(result.failure?.message).toContain('Cannot convert') - }, 30000) -}) diff --git a/packages/protect/__tests__/protect-ops.test.ts b/packages/protect/__tests__/protect-ops.test.ts deleted file mode 100644 index c7a2e2765..000000000 --- a/packages/protect/__tests__/protect-ops.test.ts +++ /dev/null @@ -1,843 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { beforeAll, describe, expect, it } from 'vitest' -import { LockContext, protect } from '../src' - -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), -}) - -type User = { - id: string - email?: string | null - createdAt?: Date - updatedAt?: Date - address?: string | null - number?: number -} - -let protectClient: Awaited> - -beforeAll(async () => { - protectClient = await protect({ - schemas: [users], - }) -}) - -describe('encryption and decryption edge cases', () => { - it('should return null if plaintext is null', async () => { - const ciphertext = await protectClient.encrypt(null, { - column: users.email, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - // Verify null is preserved - expect(ciphertext.data).toBeNull() - - const plaintext = await protectClient.decrypt(ciphertext.data) - - expect(plaintext).toEqual({ - data: null, - }) - }, 30000) - - it('should encrypt and decrypt a model', async () => { - // Create a model with decrypted values - const decryptedModel = { - id: '1', - email: 'plaintext', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - address: '123 Main St', - number: 1, - } - - // Encrypt the model - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify encrypted fields - expect(encryptedModel.data.email).toHaveProperty('c') - expect(encryptedModel.data.address).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.number).toBe(1) - - // Decrypt the model - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual({ - id: '1', - email: 'plaintext', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - address: '123 Main St', - number: 1, - }) - }, 30000) - - it('should handle null values in a model', async () => { - // Create a model with null values - const decryptedModel = { - id: '1', - email: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: null, - } - - // Encrypt the model - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify null fields are preserved - expect(encryptedModel.data.email).toBeNull() - expect(encryptedModel.data.address).toBeNull() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.number).toBe(1) - - // Decrypt the model - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual({ - id: '1', - email: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: null, - }) - }, 30000) - - it('should handle undefined values in a model', async () => { - // Create a model with undefined values - const decryptedModel = { - id: '1', - email: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: null, - } - - // Encrypt the model - const encryptedModel = await protectClient.encryptModel( - decryptedModel, - users, - ) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Verify undefined fields are preserved - expect(encryptedModel.data.email).toBeUndefined() - expect(encryptedModel.data.address).toBeNull() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModel.data.id).toBe('1') - expect(encryptedModel.data.createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModel.data.number).toBe(1) - - // Decrypt the model - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual({ - id: '1', - email: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: null, - }) - }, 30000) -}) - -describe('bulk encryption', () => { - it('should bulk encrypt and decrypt models', async () => { - // Create models with decrypted values - const decryptedModels = [ - { - id: '1', - email: 'test', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: '123 Main St', - }, - { - id: '2', - email: 'test2', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - address: null, - }, - ] - - // Encrypt the models - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toHaveProperty('c') - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].address).toBeNull() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].number).toBe(1) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].number).toBe(2) - - // Decrypt the models - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual([ - { - id: '1', - email: 'test', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - address: '123 Main St', - }, - { - id: '2', - email: 'test2', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - address: null, - }, - ]) - }, 30000) - - it('should return empty array if models is empty', async () => { - // Encrypt empty array of models - const encryptedModels = await protectClient.bulkEncryptModels( - [], - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - expect(encryptedModels.data).toEqual([]) - }, 30000) - - it('should return empty array if decrypting empty array of models', async () => { - // Decrypt empty array of models - const decryptedResult = await protectClient.bulkDecryptModels([]) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual([]) - }, 30000) -}) - -describe('bulk encryption edge cases', () => { - it('should handle mixed null and non-null values in bulk operations', async () => { - const decryptedModels = [ - { - id: '1', - email: 'test1', - address: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - }, - { - id: '2', - email: null, - address: '123 Main St', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - }, - { - id: '3', - email: 'test3', - address: '456 Oak St', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 3, - }, - ] - - // Encrypt the models - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toBeNull() - expect(encryptedModels.data[1].email).toBeNull() - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[2].email).toHaveProperty('c') - expect(encryptedModels.data[2].address).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].number).toBe(1) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].number).toBe(2) - expect(encryptedModels.data[2].id).toBe('3') - expect(encryptedModels.data[2].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].number).toBe(3) - - // Decrypt the models - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) - - it('should handle mixed undefined and non-undefined values in bulk operations', async () => { - const decryptedModels = [ - { - id: '1', - email: 'test1', - address: undefined, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - }, - { - id: '2', - email: null, - address: '123 Main St', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - }, - { - id: '3', - email: 'test3', - address: '456 Oak St', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 3, - }, - ] - - // Encrypt the models - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toHaveProperty('c') - expect(encryptedModels.data[0].address).toBeUndefined() - expect(encryptedModels.data[1].email).toBeNull() - expect(encryptedModels.data[1].address).toHaveProperty('c') - expect(encryptedModels.data[2].email).toHaveProperty('c') - expect(encryptedModels.data[2].address).toHaveProperty('c') - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].number).toBe(1) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].number).toBe(2) - expect(encryptedModels.data[2].id).toBe('3') - expect(encryptedModels.data[2].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].number).toBe(3) - - // Decrypt the models - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) - - it('should handle empty models in bulk operations', async () => { - const decryptedModels = [ - { - id: '1', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 1, - }, // No encrypted fields - { - id: '2', - email: 'test2', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 2, - }, - { - id: '3', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: 3, - }, // No encrypted fields - ] - - // Encrypt the models - const encryptedModels = await protectClient.bulkEncryptModels( - decryptedModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Verify encrypted fields for each model - expect(encryptedModels.data[0].email).toBeUndefined() - expect(encryptedModels.data[0].address).toBeUndefined() - expect(encryptedModels.data[1].email).toHaveProperty('c') - expect(encryptedModels.data[1].address).toBeUndefined() - expect(encryptedModels.data[2].email).toBeUndefined() - expect(encryptedModels.data[2].address).toBeUndefined() - - // Verify non-encrypted fields remain unchanged - expect(encryptedModels.data[0].id).toBe('1') - expect(encryptedModels.data[0].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[0].number).toBe(1) - expect(encryptedModels.data[1].id).toBe('2') - expect(encryptedModels.data[1].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[1].number).toBe(2) - expect(encryptedModels.data[2].id).toBe('3') - expect(encryptedModels.data[2].createdAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].updatedAt).toEqual(new Date('2021-01-01')) - expect(encryptedModels.data[2].number).toBe(3) - - // Decrypt the models - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(decryptedModels) - }, 30000) -}) - -describe('error handling', () => { - it('should handle invalid encrypted payloads', async () => { - const validModel = { - id: '1', - email: 'test@example.com', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - address: '123 Main St', - number: 1, - } - - // First encrypt a valid model - const encryptedModel = await protectClient.encryptModel( - validModel, - users, - ) - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Create an invalid model by removing required fields - const invalidModel = { - id: '1', - // Missing required fields - } - - try { - await protectClient.decryptModel(invalidModel as User) - throw new Error('Expected decryption to fail') - } catch (error) { - expect(error).toBeDefined() - } - }, 30000) - - it('should handle missing required fields', async () => { - const model = { - id: '1', - email: null, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - address: null, - number: 1, - } - - try { - await protectClient.encryptModel(model, users) - throw new Error('Expected encryption to fail') - } catch (error) { - expect(error).toBeDefined() - } - }, 30000) -}) - -describe('type safety', () => { - it('should maintain type safety with complex nested objects', async () => { - const model = { - id: '1', - email: 'test@example.com', - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - address: '123 Main St', - number: 1, - metadata: { - preferences: { - notifications: true, - theme: 'dark', - }, - }, - } - - // Encrypt the model - const encryptedModel = await protectClient.encryptModel(model, users) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Decrypt the model - const decryptedResult = await protectClient.decryptModel( - encryptedModel.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(model) - }, 30000) -}) - -describe('performance', () => { - it('should handle large numbers of models efficiently', async () => { - const largeModels = Array(10) - .fill(null) - .map((_, i) => ({ - id: i.toString(), - email: `test${i}@example.com`, - address: `Address ${i}`, - createdAt: new Date('2021-01-01'), - updatedAt: new Date('2021-01-01'), - number: i, - })) - - // Encrypt the models - const encryptedModels = await protectClient.bulkEncryptModels( - largeModels, - users, - ) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Decrypt the models - const decryptedResult = await protectClient.bulkDecryptModels( - encryptedModels.data, - ) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual(largeModels) - }, 60000) -}) - -describe('encryption and decryption with lock context', () => { - it('should encrypt and decrypt a payload with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - const email = 'hello@example.com' - - const ciphertext = await protectClient - .encrypt(email, { - column: users.email, - table: users, - }) - .withLockContext(lockContext.data) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - const plaintext = await protectClient - .decrypt(ciphertext.data) - .withLockContext(lockContext.data) - - if (plaintext.failure) { - throw new Error(`[protect]: ${plaintext.failure.message}`) - } - - expect(plaintext.data).toEqual(email) - }, 30000) - - it('should encrypt and decrypt a model with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create a model with decrypted values - const decryptedModel = { - id: '1', - email: 'plaintext', - } - - // Encrypt the model with lock context - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - // Decrypt the model with lock context - const decryptedResult = await protectClient - .decryptModel(encryptedModel.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual({ - id: '1', - email: 'plaintext', - }) - }, 30000) - - it('should encrypt with context and be unable to decrypt without context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create a model with decrypted values - const decryptedModel = { - id: '1', - email: 'plaintext', - } - - // Encrypt the model with lock context - const encryptedModel = await protectClient - .encryptModel(decryptedModel, users) - .withLockContext(lockContext.data) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - try { - await protectClient.decryptModel(encryptedModel.data) - } catch (error) { - const e = error as Error - expect(e.message.startsWith('Failed to retrieve key')).toEqual(true) - } - }, 30000) - - it('should bulk encrypt and decrypt models with lock context', async () => { - const userJwt = process.env.USER_JWT - - if (!userJwt) { - console.log('Skipping lock context test - no USER_JWT provided') - return - } - - const lc = new LockContext() - const lockContext = await lc.identify(userJwt) - - if (lockContext.failure) { - throw new Error(`[protect]: ${lockContext.failure.message}`) - } - - // Create models with decrypted values - const decryptedModels = [ - { - id: '1', - email: 'test', - }, - { - id: '2', - email: 'test2', - }, - ] - - // Encrypt the models with lock context - const encryptedModels = await protectClient - .bulkEncryptModels(decryptedModels, users) - .withLockContext(lockContext.data) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - // Decrypt the models with lock context - const decryptedResult = await protectClient - .bulkDecryptModels(encryptedModels.data) - .withLockContext(lockContext.data) - - if (decryptedResult.failure) { - throw new Error(`[protect]: ${decryptedResult.failure.message}`) - } - - expect(decryptedResult.data).toEqual([ - { - id: '1', - email: 'test', - }, - { - id: '2', - email: 'test2', - }, - ]) - }, 30000) -}) - -describe('special characters', () => { - it('should encrypt and decrypt multiple special characters together', async () => { - const plaintext = - 'complex@string-with/slashes\\backslashes.and#symbols$%&+!@#$%^&*()_+-=[]{}|;:,.<>?/~`' - - const ciphertext = await protectClient.encrypt(plaintext, { - column: users.email, - table: users, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - const decrypted = await protectClient.decrypt(ciphertext.data) - - expect(decrypted).toEqual({ - data: plaintext, - }) - }, 30000) -}) diff --git a/packages/protect/__tests__/searchable-json-pg.test.ts b/packages/protect/__tests__/searchable-json-pg.test.ts deleted file mode 100644 index 49b13af02..000000000 --- a/packages/protect/__tests__/searchable-json-pg.test.ts +++ /dev/null @@ -1,2390 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import postgres from 'postgres' -import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { LockContext, protect } from '../src' - -if (!process.env.DATABASE_URL) { - throw new Error('Missing env.DATABASE_URL') -} - -// Disable prepared statements — required for pooled connections (PgBouncer in transaction mode) -const sql = postgres(process.env.DATABASE_URL, { prepare: false }) - -const table = csTable('protect-ci-jsonb', { - metadata: csColumn('metadata').searchableJson(), -}) - -const TEST_RUN_ID = `test-run-${Date.now()}-${Math.random().toString(36).slice(2, 8)}` -const userJwt = process.env.USER_JWT - -type ProtectClient = Awaited> -let protectClient: ProtectClient - -// ─── Helpers ───────────────────────────────────────────────────────── - -async function insertRow(plaintext: any) { - const encrypted = await protectClient.encryptModel( - { metadata: plaintext }, - table, - ) - if (encrypted.failure) throw new Error(encrypted.failure.message) - - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (${sql.json(encrypted.data.metadata)}::eql_v2_encrypted, ${TEST_RUN_ID}) - RETURNING id - ` - return { id: inserted.id, encrypted } -} - -async function verifyRow(row: any, expected: any) { - expect(row).toBeDefined() - const decrypted = await protectClient.decryptModel({ metadata: row.metadata }) - if (decrypted.failure) throw new Error(decrypted.failure.message) - expect(decrypted.data.metadata).toEqual(expected) -} - -async function encryptQueryTerm( - value: any, - queryType: 'steVecSelector' | 'steVecTerm' | 'searchableJson', - returnType: - | 'composite-literal' - | 'escaped-composite-literal' = 'composite-literal', -) { - const result = await protectClient.encryptQuery(value, { - column: table.metadata, - table: table, - queryType, - returnType, - }) - if (result.failure) throw new Error(result.failure.message) - return result.data -} - -beforeAll(async () => { - protectClient = await protect({ schemas: [table] }) - - await sql` - CREATE TABLE IF NOT EXISTS "protect-ci-jsonb" ( - id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, - metadata eql_v2_encrypted, - test_run_id TEXT - ) - ` -}, 30000) - -afterAll(async () => { - await sql`DELETE FROM "protect-ci-jsonb" WHERE test_run_id = ${TEST_RUN_ID}` - await sql.end() -}, 30000) - -describe('searchableJson postgres integration', () => { - // ─── Storage: encrypt → insert → select → decrypt ────────────────── - - describe('storage: encrypt → insert → select → decrypt', () => { - it('round-trips a flat JSON object', async () => { - const plaintext = { user: { email: 'flat-rt@test.com' }, role: 'admin' } - const { id } = await insertRow(plaintext) - - const rows = await sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE id = ${id} - ` - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - - it('round-trips nested JSON with arrays', async () => { - const plaintext = { - user: { - profile: { role: 'admin', permissions: ['read', 'write'] }, - tags: [{ name: 'vip' }, { name: 'beta' }], - }, - items: [{ id: 1, name: 'widget' }], - } - const { id } = await insertRow(plaintext) - - const rows = await sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE id = ${id} - ` - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - - it('round-trips null values', async () => { - const encrypted = await protectClient.encrypt(null, { - column: table.metadata, - table: table, - }) - - if (encrypted.failure) throw new Error(encrypted.failure.message) - expect(encrypted.data).toBeNull() - - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (NULL, ${TEST_RUN_ID}) - RETURNING id - ` - - const rows = await sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE id = ${inserted.id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].metadata).toBeNull() - }, 30000) - }) - - // ─── jsonb_path_query: path-based selector queries ───────────────── - - describe('jsonb_path_query: path-based selector queries', () => { - it('finds row by simple top-level path ($.role)', async () => { - const plaintext = { role: 'path-toplevel-test', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds row by nested path ($.user.email)', async () => { - const plaintext = { - user: { email: 'nested-path@test.com' }, - type: 'nested-path', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds row by deeply nested path ($.a.b.c)', async () => { - const plaintext = { a: { b: { c: 'deep-value' } }, marker: 'deep-path' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.a.b.c', 'steVecSelector') - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching path returns zero rows', async () => { - // Insert a doc that does NOT have $.nonexistent.path - const plaintext = { exists: true, marker: 'no-match-test' } - await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent.path', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - // No row should have this path - expect(rows.length).toBe(0) - }, 30000) - - it('multiple docs — only matching doc returned', async () => { - // Insert two docs: one with $.target.value, one without - const plaintextWithPath = { - target: { value: 'found-it' }, - marker: 'has-target', - } - const plaintextWithoutPath = { - other: { key: 'nope' }, - marker: 'no-target', - } - - const { id: idWith } = await insertRow(plaintextWithPath) - const { id: idWithout } = await insertRow(plaintextWithoutPath) - - const selectorTerm = await encryptQueryTerm( - '$.target.value', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - // The doc with $.target.value should be found - const matchingRow = rows.find((r) => r.id === idWith) - expect(matchingRow).toBeDefined() - - // The doc without $.target.value should NOT be found - const nonMatchingRow = rows.find((r) => r.id === idWithout) - expect(nonMatchingRow).toBeUndefined() - - // Decrypt and verify the matching row - await verifyRow(matchingRow!, plaintextWithPath) - }, 30000) - - it('finds row by simple top-level path (Simple)', async () => { - const plaintext = { role: 'path-tl-simple', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, '${selectorTerm}'::eql_v2_encrypted) as result - WHERE t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds row by nested path (Simple)', async () => { - const plaintext = { - user: { email: 'nested-simple@test.com' }, - type: 'nested-path-simple', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, '${selectorTerm}'::eql_v2_encrypted) as result - WHERE t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds with deep nested path (Simple)', async () => { - const plaintext = { - target: { nested: { value: 'deep-simple' } }, - marker: 'jpq-deep-simple', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.target.nested.value', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, '${selectorTerm}'::eql_v2_encrypted) as result - WHERE t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching path returns zero rows (Simple)', async () => { - const plaintext = { data: true, marker: 'jpq-nomatch-simple' } - await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.missing.path', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, '${selectorTerm}'::eql_v2_encrypted) as result - WHERE t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBe(0) - }, 30000) - }) - - // ─── Containment: @> term queries ────────────────────────────────── - - describe('containment: @> term queries', () => { - it('matches by key/value pair', async () => { - const plaintext = { role: 'admin-containment', department: 'engineering' } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'admin-containment' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('matches by nested object structure', async () => { - const plaintext = { - user: { profile: { role: 'superadmin' } }, - active: true, - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { user: { profile: { role: 'superadmin' } } }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching term returns zero rows', async () => { - const plaintext = { status: 'active', tier: 'free' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { status: 'nonexistent-value-xyz' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - }) - - // ─── Mixed and batch operations ──────────────────────────────────── - - describe('mixed and batch operations', () => { - it('batch encrypts selector + containment terms together', async () => { - const plaintext = { - user: { email: 'batch@test.com' }, - role: 'editor', - kind: 'batch-mixed', - } - const { id } = await insertRow(plaintext) - - const queryResult = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }, - { - value: { role: 'editor' }, - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ]) - - if (queryResult.failure) throw new Error(queryResult.failure.message) - const [selectorTerm, containmentTerm] = queryResult.data - - // Selector query: jsonb_path_query - const selectorRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(selectorRows.length).toBeGreaterThanOrEqual(1) - const selectorMatch = selectorRows.find((r) => r.id === id) - expect(selectorMatch).toBeDefined() - await verifyRow(selectorMatch!, plaintext) - - // Containment query: @> - const containmentRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${containmentTerm}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(containmentRows.length).toBeGreaterThanOrEqual(1) - const containmentMatch = containmentRows.find((r) => r.id === id) - expect(containmentMatch).toBeDefined() - await verifyRow(containmentMatch!, plaintext) - }, 30000) - - it('inferred vs explicit queryType produce same results', async () => { - const plaintext = { category: 'equivalence-test', priority: 'high' } - const { id } = await insertRow(plaintext) - - // Selector: inferred (searchableJson) vs explicit (steVecSelector) - const inferredSelectorTerm = await encryptQueryTerm( - '$.category', - 'searchableJson', - ) - const explicitSelectorTerm = await encryptQueryTerm( - '$.category', - 'steVecSelector', - ) - - const inferredRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${inferredSelectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - const explicitRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${explicitSelectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(inferredRows.length).toBe(explicitRows.length) - expect(inferredRows.length).toBeGreaterThanOrEqual(1) - - // Both should find our inserted row - const inferredMatch = inferredRows.find((r) => r.id === id) - const explicitMatch = explicitRows.find((r) => r.id === id) - expect(inferredMatch).toBeDefined() - expect(explicitMatch).toBeDefined() - - // Decrypt and compare — both should yield identical plaintext - const inferredDecrypted = await protectClient.decryptModel({ - metadata: inferredMatch!.metadata, - }) - const explicitDecrypted = await protectClient.decryptModel({ - metadata: explicitMatch!.metadata, - }) - if (inferredDecrypted.failure) - throw new Error(inferredDecrypted.failure.message) - if (explicitDecrypted.failure) - throw new Error(explicitDecrypted.failure.message) - - expect(inferredDecrypted.data.metadata).toEqual( - explicitDecrypted.data.metadata, - ) - expect(inferredDecrypted.data.metadata).toEqual(plaintext) - - // Containment: inferred (searchableJson) vs explicit (steVecTerm) - const inferredContainmentTerm = await encryptQueryTerm( - { category: 'equivalence-test' }, - 'searchableJson', - ) - const explicitContainmentTerm = await encryptQueryTerm( - { category: 'equivalence-test' }, - 'steVecTerm', - ) - - const inferredTermRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${inferredContainmentTerm}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - const explicitTermRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${explicitContainmentTerm}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(inferredTermRows.length).toBe(explicitTermRows.length) - expect(inferredTermRows.length).toBeGreaterThanOrEqual(1) - - const inferredTermMatch = inferredTermRows.find((r) => r.id === id) - const explicitTermMatch = explicitTermRows.find((r) => r.id === id) - expect(inferredTermMatch).toBeDefined() - expect(explicitTermMatch).toBeDefined() - - const inferredTermDecrypted = await protectClient.decryptModel({ - metadata: inferredTermMatch!.metadata, - }) - const explicitTermDecrypted = await protectClient.decryptModel({ - metadata: explicitTermMatch!.metadata, - }) - if (inferredTermDecrypted.failure) - throw new Error(inferredTermDecrypted.failure.message) - if (explicitTermDecrypted.failure) - throw new Error(explicitTermDecrypted.failure.message) - - expect(inferredTermDecrypted.data.metadata).toEqual( - explicitTermDecrypted.data.metadata, - ) - expect(inferredTermDecrypted.data.metadata).toEqual(plaintext) - }, 30000) - }) - - // ─── Escaped-composite-literal format ───────────────────────────── - - describe('escaped-composite-literal format', () => { - it('escaped selector → unwrap → query PG', async () => { - const plaintext = { - user: { email: 'escaped-sel@test.com' }, - marker: 'escaped-selector', - } - const { id } = await insertRow(plaintext) - - // Encrypt with both formats - const compositeData = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - 'composite-literal', - ) - const escapedData = (await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - 'escaped-composite-literal', - )) as string - - // Verify escaped format and unwrap - expect(typeof escapedData).toBe('string') - expect(escapedData).toMatch(/^"\(.*\)"$/) - const unwrapped = JSON.parse(escapedData) - - expect(unwrapped).toBe(compositeData) - - // Use composite-literal form to query PG - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${compositeData}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('escaped containment → unwrap → query PG', async () => { - const plaintext = { - role: 'escaped-containment-test', - department: 'security', - } - const { id } = await insertRow(plaintext) - - const escapedData = (await encryptQueryTerm( - { role: 'escaped-containment-test' }, - 'steVecTerm', - 'escaped-composite-literal', - )) as string - - // Verify escaped format and unwrap - expect(typeof escapedData).toBe('string') - expect(escapedData).toMatch(/^"\(.*\)"$/) - const unwrapped = JSON.parse(escapedData) - - // Unwrapped escaped format should be a valid composite-literal - expect(typeof unwrapped).toBe('string') - expect(unwrapped).toMatch(/^\(.*\)$/) - - // Use unwrapped composite-literal form to query PG - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${unwrapped}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('batch escaped format', async () => { - const plaintext = { - user: { email: 'batch-escaped@test.com' }, - role: 'batch-escaped-role', - marker: 'batch-escaped', - } - const { id } = await insertRow(plaintext) - - const queryResult = await protectClient.encryptQuery([ - { - value: '$.user.email', - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'escaped-composite-literal', - }, - { - value: { role: 'batch-escaped-role' }, - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'escaped-composite-literal', - }, - ]) - if (queryResult.failure) throw new Error(queryResult.failure.message) - - expect(queryResult.data).toHaveLength(2) - for (const item of queryResult.data) { - expect(typeof item).toBe('string') - expect(item).toMatch(/^"\(.*\)"$/) - } - - // Unwrap escaped format - const selectorUnwrapped = JSON.parse(queryResult.data[0] as string) - const containmentUnwrapped = JSON.parse(queryResult.data[1] as string) - - // Selector query - const selectorRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorUnwrapped}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(selectorRows.length).toBeGreaterThanOrEqual(1) - const selectorMatch = selectorRows.find((r) => r.id === id) - expect(selectorMatch).toBeDefined() - await verifyRow(selectorMatch!, plaintext) - - // Containment query - const containmentRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${containmentUnwrapped}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(containmentRows.length).toBeGreaterThanOrEqual(1) - const containmentMatch = containmentRows.find((r) => r.id === id) - expect(containmentMatch).toBeDefined() - await verifyRow(containmentMatch!, plaintext) - }, 30000) - }) - - // ─── LockContext integration ────────────────────────────────────── - - describe.skipIf(!userJwt)('LockContext integration', () => { - it('selector with LockContext', async () => { - const lc = new LockContext() - const lockContext = await lc.identify(userJwt!) - if (lockContext.failure) throw new Error(lockContext.failure.message) - - const plaintext = { - user: { email: 'lc-selector@test.com' }, - marker: 'lock-context-selector', - } - - const encrypted = await protectClient - .encryptModel({ metadata: plaintext }, table) - .withLockContext(lockContext.data) - if (encrypted.failure) throw new Error(encrypted.failure.message) - - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (${sql.json(encrypted.data.metadata)}::eql_v2_encrypted, ${TEST_RUN_ID}) - RETURNING id - ` - - const selectorResult = await protectClient - .encryptQuery('$.user.email', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }) - .withLockContext(lockContext.data) - .execute() - if (selectorResult.failure) - throw new Error(selectorResult.failure.message) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorResult.data}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === inserted.id) - expect(matchingRow).toBeDefined() - - const decrypted = await protectClient - .decryptModel({ metadata: matchingRow!.metadata }) - .withLockContext(lockContext.data) - if (decrypted.failure) throw new Error(decrypted.failure.message) - expect(decrypted.data.metadata).toEqual(plaintext) - }, 60000) - - it('containment with LockContext', async () => { - const lc = new LockContext() - const lockContext = await lc.identify(userJwt!) - if (lockContext.failure) throw new Error(lockContext.failure.message) - - const plaintext = { role: 'lc-containment-test', department: 'auth' } - - const encrypted = await protectClient - .encryptModel({ metadata: plaintext }, table) - .withLockContext(lockContext.data) - if (encrypted.failure) throw new Error(encrypted.failure.message) - - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (${sql.json(encrypted.data.metadata)}::eql_v2_encrypted, ${TEST_RUN_ID}) - RETURNING id - ` - - const containmentResult = await protectClient - .encryptQuery( - { role: 'lc-containment-test' }, - { - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ) - .withLockContext(lockContext.data) - .execute() - if (containmentResult.failure) - throw new Error(containmentResult.failure.message) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${containmentResult.data}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === inserted.id) - expect(matchingRow).toBeDefined() - - const decrypted = await protectClient - .decryptModel({ metadata: matchingRow!.metadata }) - .withLockContext(lockContext.data) - if (decrypted.failure) throw new Error(decrypted.failure.message) - expect(decrypted.data.metadata).toEqual(plaintext) - }, 60000) - - it('batch with LockContext', async () => { - const lc = new LockContext() - const lockContext = await lc.identify(userJwt!) - if (lockContext.failure) throw new Error(lockContext.failure.message) - - const plaintext = { - user: { email: 'lc-batch@test.com' }, - role: 'lc-batch-role', - kind: 'lock-context-batch', - } - - const encrypted = await protectClient - .encryptModel({ metadata: plaintext }, table) - .withLockContext(lockContext.data) - if (encrypted.failure) throw new Error(encrypted.failure.message) - - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (${sql.json(encrypted.data.metadata)}::eql_v2_encrypted, ${TEST_RUN_ID}) - RETURNING id - ` - - const batchResult = await protectClient - .encryptQuery([ - { - value: '$.user.email', - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }, - { - value: { role: 'lc-batch-role' }, - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ]) - .withLockContext(lockContext.data) - .execute() - if (batchResult.failure) throw new Error(batchResult.failure.message) - - const [selectorTerm, containmentTerm] = batchResult.data - - // Selector query - const selectorRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(selectorRows.length).toBeGreaterThanOrEqual(1) - const selectorMatch = selectorRows.find((r) => r.id === inserted.id) - expect(selectorMatch).toBeDefined() - - const selectorDecrypted = await protectClient - .decryptModel({ metadata: selectorMatch!.metadata }) - .withLockContext(lockContext.data) - if (selectorDecrypted.failure) - throw new Error(selectorDecrypted.failure.message) - expect(selectorDecrypted.data.metadata).toEqual(plaintext) - - // Containment query - const containmentRows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${containmentTerm}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(containmentRows.length).toBeGreaterThanOrEqual(1) - const containmentMatch = containmentRows.find((r) => r.id === inserted.id) - expect(containmentMatch).toBeDefined() - - const containmentDecrypted = await protectClient - .decryptModel({ metadata: containmentMatch!.metadata }) - .withLockContext(lockContext.data) - if (containmentDecrypted.failure) - throw new Error(containmentDecrypted.failure.message) - expect(containmentDecrypted.data.metadata).toEqual(plaintext) - }, 60000) - }) - - // ─── Concurrent query operations ───────────────────────────────── - - describe('concurrent query operations', () => { - it('parallel selector queries', async () => { - // Insert 3 docs with distinct structures - const docs = [ - { alpha: { key: 'concurrent-sel-1' }, marker: 'concurrent-1' }, - { beta: { key: 'concurrent-sel-2' }, marker: 'concurrent-2' }, - { gamma: { key: 'concurrent-sel-3' }, marker: 'concurrent-3' }, - ] - - const insertedIds: number[] = [] - for (const plaintext of docs) { - const { id } = await insertRow(plaintext) - insertedIds.push(id) - } - - // Parallel encrypt 3 selector queries - const [q1, q2, q3] = await Promise.all([ - protectClient.encryptQuery('$.alpha.key', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }), - protectClient.encryptQuery('$.beta.key', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }), - protectClient.encryptQuery('$.gamma.key', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }), - ]) - - if (q1.failure) throw new Error(q1.failure.message) - if (q2.failure) throw new Error(q2.failure.message) - if (q3.failure) throw new Error(q3.failure.message) - - // Execute each against PG - const [rows1, rows2, rows3] = await Promise.all([ - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${q1.data}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - `, - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${q2.data}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - `, - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${q3.data}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - `, - ]) - - // Each query should find its respective doc and not others - expect(rows1.find((r) => r.id === insertedIds[0])).toBeDefined() - expect(rows1.find((r) => r.id === insertedIds[1])).toBeUndefined() - expect(rows1.find((r) => r.id === insertedIds[2])).toBeUndefined() - expect(rows2.find((r) => r.id === insertedIds[1])).toBeDefined() - expect(rows2.find((r) => r.id === insertedIds[0])).toBeUndefined() - expect(rows2.find((r) => r.id === insertedIds[2])).toBeUndefined() - expect(rows3.find((r) => r.id === insertedIds[2])).toBeDefined() - expect(rows3.find((r) => r.id === insertedIds[0])).toBeUndefined() - expect(rows3.find((r) => r.id === insertedIds[1])).toBeUndefined() - - // Decrypt and validate each matched row - const match1 = rows1.find((r) => r.id === insertedIds[0])! - const decrypted1 = await protectClient.decryptModel({ - metadata: match1.metadata, - }) - if (decrypted1.failure) throw new Error(decrypted1.failure.message) - expect(decrypted1.data.metadata).toEqual(docs[0]) - - const match2 = rows2.find((r) => r.id === insertedIds[1])! - const decrypted2 = await protectClient.decryptModel({ - metadata: match2.metadata, - }) - if (decrypted2.failure) throw new Error(decrypted2.failure.message) - expect(decrypted2.data.metadata).toEqual(docs[1]) - - const match3 = rows3.find((r) => r.id === insertedIds[2])! - const decrypted3 = await protectClient.decryptModel({ - metadata: match3.metadata, - }) - if (decrypted3.failure) throw new Error(decrypted3.failure.message) - expect(decrypted3.data.metadata).toEqual(docs[2]) - }, 60000) - - it('parallel containment queries', async () => { - const docs = [ - { role: 'concurrent-contain-1', tier: 'gold' }, - { role: 'concurrent-contain-2', tier: 'silver' }, - ] - - const insertedIds: number[] = [] - for (const plaintext of docs) { - const { id } = await insertRow(plaintext) - insertedIds.push(id) - } - - // Parallel encrypt 2 containment queries - const [c1, c2] = await Promise.all([ - protectClient.encryptQuery( - { role: 'concurrent-contain-1' }, - { - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ), - protectClient.encryptQuery( - { role: 'concurrent-contain-2' }, - { - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ), - ]) - - if (c1.failure) throw new Error(c1.failure.message) - if (c2.failure) throw new Error(c2.failure.message) - - const [rows1, rows2] = await Promise.all([ - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE metadata @> ${c1.data}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - `, - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE metadata @> ${c2.data}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - `, - ]) - - // Each finds only its target doc - expect(rows1.find((r) => r.id === insertedIds[0])).toBeDefined() - expect(rows1.find((r) => r.id === insertedIds[1])).toBeUndefined() - expect(rows2.find((r) => r.id === insertedIds[1])).toBeDefined() - expect(rows2.find((r) => r.id === insertedIds[0])).toBeUndefined() - - // Decrypt and validate each matched row - const match1 = rows1.find((r) => r.id === insertedIds[0])! - const decrypted1 = await protectClient.decryptModel({ - metadata: match1.metadata, - }) - if (decrypted1.failure) throw new Error(decrypted1.failure.message) - expect(decrypted1.data.metadata).toEqual(docs[0]) - - const match2 = rows2.find((r) => r.id === insertedIds[1])! - const decrypted2 = await protectClient.decryptModel({ - metadata: match2.metadata, - }) - if (decrypted2.failure) throw new Error(decrypted2.failure.message) - expect(decrypted2.data.metadata).toEqual(docs[1]) - }, 60000) - - it('parallel mixed encrypt+query', async () => { - const plaintext = { - user: { email: 'concurrent-mixed@test.com' }, - role: 'concurrent-mixed-role', - kind: 'mixed-concurrent', - } - - // Parallel: encryptModel + selector encryptQuery + containment encryptQuery - const [encryptedModel, selectorResult, containmentResult] = - await Promise.all([ - protectClient.encryptModel({ metadata: plaintext }, table), - protectClient.encryptQuery('$.user.email', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - returnType: 'composite-literal', - }), - protectClient.encryptQuery( - { role: 'concurrent-mixed-role' }, - { - column: table.metadata, - table: table, - queryType: 'steVecTerm', - returnType: 'composite-literal', - }, - ), - ]) - - if (encryptedModel.failure) - throw new Error(encryptedModel.failure.message) - if (selectorResult.failure) - throw new Error(selectorResult.failure.message) - if (containmentResult.failure) - throw new Error(containmentResult.failure.message) - - // Insert the encrypted doc - const [inserted] = await sql` - INSERT INTO "protect-ci-jsonb" (metadata, test_run_id) - VALUES (${sql.json(encryptedModel.data.metadata)}::eql_v2_encrypted, ${TEST_RUN_ID}) - RETURNING id - ` - - // Query with both terms - const [selectorRows, containmentRows] = await Promise.all([ - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorResult.data}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - `, - sql` - SELECT id, (metadata).data as metadata FROM "protect-ci-jsonb" - WHERE metadata @> ${containmentResult.data}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - `, - ]) - - // Both should find the inserted row - expect(selectorRows.find((r) => r.id === inserted.id)).toBeDefined() - expect(containmentRows.find((r) => r.id === inserted.id)).toBeDefined() - // Verify result sets are bounded (not returning all rows) - expect(selectorRows.length).toBeGreaterThanOrEqual(1) - expect(containmentRows.length).toBeGreaterThanOrEqual(1) - - // Decrypt and validate both matched rows - const selectorMatch = selectorRows.find((r) => r.id === inserted.id)! - const selectorDecrypted = await protectClient.decryptModel({ - metadata: selectorMatch.metadata, - }) - if (selectorDecrypted.failure) - throw new Error(selectorDecrypted.failure.message) - expect(selectorDecrypted.data.metadata).toEqual(plaintext) - - const containmentMatch = containmentRows.find( - (r) => r.id === inserted.id, - )! - const containmentDecrypted = await protectClient.decryptModel({ - metadata: containmentMatch.metadata, - }) - if (containmentDecrypted.failure) - throw new Error(containmentDecrypted.failure.message) - expect(containmentDecrypted.data.metadata).toEqual(plaintext) - }, 60000) - }) - - // ─── Contained-by: <@ term queries ──────────────────────────────── - - describe('contained-by: <@ term queries', () => { - it('matches by key/value pair (Extended)', async () => { - const plaintext = { role: 'contained-by-kv', department: 'eng' } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'contained-by-kv' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('matches by nested object (Extended)', async () => { - const plaintext = { - user: { profile: { role: 'contained-by-nested' } }, - active: true, - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { user: { profile: { role: 'contained-by-nested' } } }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching value returns zero rows (Extended)', async () => { - const plaintext = { status: 'active-cb', tier: 'free' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { status: 'nonexistent-cb-xyz' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - - it('matches by key/value pair (Simple)', async () => { - const plaintext = { role: 'contained-by-kv-simple', department: 'ops' } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'contained-by-kv-simple' }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('matches by nested object (Simple)', async () => { - const plaintext = { - user: { profile: { role: 'contained-by-nested-simple' } }, - active: true, - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { user: { profile: { role: 'contained-by-nested-simple' } } }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching value returns zero rows (Simple)', async () => { - const plaintext = { status: 'active-cb-simple', tier: 'premium' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { status: 'nonexistent-cb-simple-xyz' }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBe(0) - }, 30000) - }) - - // ─── jsonb_path_query_first: scalar path queries ────────────────── - - describe('jsonb_path_query_first: scalar path queries', () => { - it('finds row by string field (Extended)', async () => { - const plaintext = { role: 'qf-string', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds row by nested path (Extended)', async () => { - const plaintext = { - user: { email: 'qf-nested@test.com' }, - type: 'qf-nested', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns no rows for unknown path (Extended)', async () => { - const plaintext = { exists: true, marker: 'qf-nomatch' } - await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent.path', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - - it('finds row by string field (Simple)', async () => { - const plaintext = { role: 'qf-string-simple', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, '${selectorTerm}'::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('finds row by nested path (Simple)', async () => { - const plaintext = { - user: { email: 'qf-nested-simple@test.com' }, - type: 'qf-nested-simple', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, '${selectorTerm}'::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns no rows for unknown path (Simple)', async () => { - const plaintext = { exists: true, marker: 'qf-nomatch-simple' } - await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent.path', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_query_first(t.metadata, '${selectorTerm}'::eql_v2_encrypted) IS NOT NULL - AND t.test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBe(0) - }, 30000) - }) - - // ─── jsonb_path_exists: boolean path queries ────────────────────── - - describe('jsonb_path_exists: boolean path queries', () => { - it('returns true for existing field (Extended)', async () => { - const plaintext = { role: 'pe-exists', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_exists(t.metadata, ${selectorTerm}::eql_v2_encrypted) - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns true for nested path (Extended)', async () => { - const plaintext = { - user: { email: 'pe-nested@test.com' }, - type: 'pe-nested', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_exists(t.metadata, ${selectorTerm}::eql_v2_encrypted) - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns false for unknown path (Extended)', async () => { - const plaintext = { exists: true, marker: 'pe-nomatch' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent.path', - 'steVecSelector', - ) - - const rows = await sql` - SELECT id, eql_v2.jsonb_path_exists(t.metadata, ${selectorTerm}::eql_v2_encrypted) as path_exists - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].path_exists).toBe(false) - }, 30000) - - it('returns true for existing field (Simple)', async () => { - const plaintext = { role: 'pe-exists-simple', extra: 'data' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_exists(t.metadata, '${selectorTerm}'::eql_v2_encrypted) - AND test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns true for nested path (Simple)', async () => { - const plaintext = { - user: { email: 'pe-nested-simple@test.com' }, - type: 'pe-nested-simple', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_exists(t.metadata, '${selectorTerm}'::eql_v2_encrypted) - AND test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('returns false for unknown path (Simple)', async () => { - const plaintext = { exists: true, marker: 'pe-nomatch-simple' } - await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent.path', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE eql_v2.jsonb_path_exists(t.metadata, '${selectorTerm}'::eql_v2_encrypted) - AND test_run_id = '${TEST_RUN_ID}'`, - ) - - expect(rows.length).toBe(0) - }, 30000) - }) - - describe('jsonb_array_elements + jsonb_array_length: array queries', () => { - it('returns null length for missing path (Extended)', async () => { - const plaintext = { exists: true, marker: 'al-nomatch' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent', - 'steVecSelector', - ) - - const rows = await sql` - SELECT t.id, - eql_v2.jsonb_array_length( - eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted) - ) as arr_len - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].arr_len).toBeNull() - - const dataRows = await sql` - SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = ${id} - ` - expect(dataRows).toHaveLength(1) - await verifyRow(dataRows[0], plaintext) - }, 30000) - - // [@] notation (proxy convention) produces the selector hash matching is_array=true STE vec entries. - // EQL v2.3: walk the raw sv array on the jsonb payload so each element is a plain jsonb object - // (not the wrapped eql_v2_encrypted composite — its `.data` notation only works inside the function). - it('[@] selector matches is_array=true entries in STE vec', async () => { - const plaintext = { colors: ['a', 'b'], marker: 'diag-sv' } - const { id } = await insertRow(plaintext) - - const entries = await sql` - SELECT - elem->>'s' as selector, - (elem->>'a')::boolean as is_array - FROM "protect-ci-jsonb" t, - LATERAL jsonb_array_elements((t.metadata).data -> 'sv') AS elem - WHERE t.id = ${id} - ` - - const arrayEntries = entries.filter((e: any) => e.is_array === true) - expect(arrayEntries.length).toBeGreaterThan(0) - - const selectorAt = await encryptQueryTerm('$.colors[@]', 'steVecSelector') - const hashAt = await sql` - SELECT (${selectorAt}::eql_v2_encrypted).data->>'s' as s - ` - - expect(hashAt[0].s).toBe(arrayEntries[0].selector) - }, 30000) - - // EQL v2.3: `jsonb_path_query_first` returns at most one sv entry (LIMIT 1) — counting / expanding - // uses `jsonb_path_query`, which aggregates matching array entries into a single row whose `data` - // carries `sv: [...]` + `a: 1`. `jsonb_array_length` / `jsonb_array_elements` walk that inner sv. - it('returns correct length for known array (Extended)', async () => { - const plaintext = { colors: ['a', 'b', 'c', 'd'], marker: 'al-known' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.colors[@]', - 'steVecSelector', - ) - - const rows = await sql` - SELECT eql_v2.jsonb_array_length(elem) AS arr_len - FROM "protect-ci-jsonb" t, - LATERAL eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) AS elem - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].arr_len).toBe(4) - - const dataRows = await sql` - SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = ${id} - ` - expect(dataRows).toHaveLength(1) - await verifyRow(dataRows[0], plaintext) - }, 30000) - - it('returns correct length for known array (Simple)', async () => { - const plaintext = { colors: ['x', 'y', 'z'], marker: 'al-known-s' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.colors[@]', - 'steVecSelector', - ) - - const rows = await sql.unsafe( - `SELECT eql_v2.jsonb_array_length(elem) AS arr_len - FROM "protect-ci-jsonb" t, - LATERAL eql_v2.jsonb_path_query(t.metadata, $1::eql_v2_encrypted) AS elem - WHERE t.id = $2`, - [selectorTerm, id], - ) - - expect(rows).toHaveLength(1) - expect(rows[0].arr_len).toBe(3) - - const dataRows = await sql.unsafe( - `SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = $1`, - [id], - ) - expect(dataRows).toHaveLength(1) - await verifyRow(dataRows[0], plaintext) - }, 30000) - - it('expands array via jsonb_array_elements (Extended)', async () => { - const plaintext = { tags: ['ae-a', 'ae-b', 'ae-c'], marker: 'ae-expand' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.tags[@]', 'steVecSelector') - - const rows = await sql` - SELECT eql_v2.jsonb_array_elements(elem) AS item - FROM "protect-ci-jsonb" t, - LATERAL eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) AS elem - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(3) - - const dataRows = await sql` - SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = ${id} - ` - expect(dataRows).toHaveLength(1) - await verifyRow(dataRows[0], plaintext) - }, 30000) - - it('expands array via jsonb_array_elements (Simple)', async () => { - const plaintext = { - tags: ['ae-s-a', 'ae-s-b', 'ae-s-c'], - marker: 'ae-expand-s', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.tags[@]', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT eql_v2.jsonb_array_elements(elem) AS item - FROM "protect-ci-jsonb" t, - LATERAL eql_v2.jsonb_path_query(t.metadata, $1::eql_v2_encrypted) AS elem - WHERE t.id = $2`, - [selectorTerm, id], - ) - - expect(rows).toHaveLength(3) - - const dataRows = await sql.unsafe( - `SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = $1`, - [id], - ) - expect(dataRows).toHaveLength(1) - await verifyRow(dataRows[0], plaintext) - }, 30000) - }) - - describe('containment: @> with array values', () => { - it('matches array subset (Extended)', async () => { - const plaintext = { - tags: ['ac-alpha', 'ac-beta', 'ac-gamma'], - marker: 'ac-subset', - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['ac-alpha'] }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching array value returns no rows (Extended)', async () => { - const plaintext = { tags: ['ac-exist'], marker: 'ac-nomatch' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['ac-nonexistent'] }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - - it('matches array subset (Simple)', async () => { - const plaintext = { - tags: ['ac-simple-x', 'ac-simple-y'], - marker: 'ac-simple', - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['ac-simple-x'] }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> $1::eql_v2_encrypted - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching array value returns no rows (Simple)', async () => { - const plaintext = { tags: ['ac-s-exist'], marker: 'ac-s-nomatch' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['ac-s-absent'] }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> $1::eql_v2_encrypted - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBe(0) - }, 30000) - - it('matches nested array subset (Extended)', async () => { - const plaintext = { - user: { roles: ['ac-nested-admin', 'ac-nested-editor'] }, - marker: 'ac-nested', - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { user: { roles: ['ac-nested-admin'] } }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> ${containmentTerm}::eql_v2_encrypted - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - }) - - describe('contained-by: <@ with array values', () => { - it('matches array superset (Extended)', async () => { - const plaintext = { - tags: ['cb-one', 'cb-two', 'cb-three'], - marker: 'cb-superset', - } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['cb-one'] }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching array returns no rows (Extended)', async () => { - const plaintext = { tags: ['cb-exist'], marker: 'cb-nomatch' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['cb-absent'] }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - - it('matches array superset (Simple)', async () => { - const plaintext = { tags: ['cb-s-one', 'cb-s-two'], marker: 'cb-s-super' } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['cb-s-one'] }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('non-matching array returns no rows (Simple)', async () => { - const plaintext = { tags: ['cb-s-exist'], marker: 'cb-s-nomatch' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { tags: ['cb-s-absent'] }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBe(0) - }, 30000) - }) - - describe('storage: array round-trips (gaps only)', () => { - it('round-trips object with empty string array', async () => { - const plaintext = { tags: [], marker: 'rt-empty-string-arr' } - const { id } = await insertRow(plaintext) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - - it('round-trips nested empty object array', async () => { - const plaintext = { data: { items: [] }, marker: 'rt-empty-obj-arr' } - const { id } = await insertRow(plaintext) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - }) - - // ─── Containment: operand and protocol matrix ────────────────────── - - describe('containment: operand and protocol matrix', () => { - it('@> matches key/value (Simple)', async () => { - const plaintext = { role: 'cm-admin-s', dept: 'cm-eng-s' } - const { id } = await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'cm-admin-s' }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> $1::eql_v2_encrypted - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('@> non-matching returns no rows (Simple)', async () => { - const plaintext = { role: 'cm-exist-s' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'cm-nope-s' }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.metadata @> $1::eql_v2_encrypted - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBe(0) - }, 30000) - - it('term <@ column matches subset (Extended)', async () => { - const plaintext = { role: 'cm-sub', marker: 'cm-sub-marker' } - const { id } = await insertRow(plaintext) - - // Query term is a SUBSET of the stored data - const containmentTerm = await encryptQueryTerm( - { role: 'cm-sub' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('term <@ column non-matching (Extended)', async () => { - const plaintext = { role: 'cm-sub-x' } - await insertRow(plaintext) - - const containmentTerm = await encryptQueryTerm( - { role: 'cm-sub-miss' }, - 'steVecTerm', - ) - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE ${containmentTerm}::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBe(0) - }, 30000) - - it('term <@ column matches subset (Simple)', async () => { - const plaintext = { role: 'cm-sub-s', marker: 'cm-sub-s-marker' } - const { id } = await insertRow(plaintext) - - // Query term is a SUBSET of the stored data - const containmentTerm = await encryptQueryTerm( - { role: 'cm-sub-s' }, - 'steVecTerm', - ) - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE $1::eql_v2_encrypted <@ t.metadata - AND t.test_run_id = $2`, - [containmentTerm, TEST_RUN_ID], - ) - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r: any) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - }) - - // ─── Field access: -> operator ───────────────────────────────────── - - describe('field access: -> operator', () => { - it('extracts field by encrypted selector (Extended)', async () => { - const plaintext = { - role: 'fa-enc', - dept: 'fa-dept', - marker: 'fa-enc-sel', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT t.metadata -> ${selectorTerm}::eql_v2_encrypted as extracted - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].extracted).not.toBeNull() - - const fullRows = await sql` - SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = ${id} - ` - await verifyRow(fullRows[0], plaintext) - }, 30000) - - it('extracts field by encrypted selector (Simple)', async () => { - const plaintext = { - role: 'fa-enc-s', - dept: 'fa-dept-s', - marker: 'fa-enc-sel-s', - } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT t.metadata -> $1::eql_v2_encrypted as extracted - FROM "protect-ci-jsonb" t - WHERE t.id = $2`, - [selectorTerm, id], - ) - - expect(rows).toHaveLength(1) - expect(rows[0].extracted).not.toBeNull() - - const fullRows = await sql` - SELECT (metadata).data as metadata FROM "protect-ci-jsonb" t WHERE t.id = ${id} - ` - await verifyRow(fullRows[0], plaintext) - }, 30000) - - it('returns null for non-existent field (Extended)', async () => { - const plaintext = { role: 'fa-null', marker: 'fa-null-marker' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm( - '$.nonexistent', - 'steVecSelector', - ) - - const rows = await sql` - SELECT t.metadata -> ${selectorTerm}::eql_v2_encrypted as extracted - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].extracted).toBeNull() - }, 30000) - - it('extracted field can be round-tripped (Extended)', async () => { - const plaintext = { - role: 'fa-roundtrip', - dept: 'fa-rt-dept', - marker: 'fa-rt-marker', - } - const { id } = await insertRow(plaintext) - - // Extract the role field via -> operator - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT t.metadata -> ${selectorTerm}::eql_v2_encrypted as extracted, - (t.metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE t.id = ${id} - ` - - expect(rows).toHaveLength(1) - expect(rows[0].extracted).not.toBeNull() - - // Decrypt the full document and verify the extracted field matches - await verifyRow(rows[0], plaintext) - }, 30000) - }) - - // ─── WHERE comparison: = equality ────────────────────────────────── - - // EQL v2.3: `=` on `eql_v2_encrypted` reduces to `hmac_256(a) = hmac_256(b)` and silently - // returns 0 rows when either side lacks `hm` (previously raised). For sv-element equality on - // oc-bearing selectors (strings / numbers), cast the extracted entry to `eql_v2.ste_vec_entry` - // — that operator is XOR-aware over `hm`/`oc` via `eq_term`. - describe('WHERE comparison: = equality (sv-element via ste_vec_entry)', () => { - it('jsonb_path_query_first = self-comparison (Extended)', async () => { - const plaintext = { role: 'eq-jpqf', marker: 'eq-jpqf-marker' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE (eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - = (eql_v2.jsonb_path_query_first(t.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - AND t.id = ${id} - ` - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - - it('jsonb_path_query_first = self-comparison (Simple)', async () => { - const plaintext = { role: 'eq-jpqf-s', marker: 'eq-jpqf-s-marker' } - const { id } = await insertRow(plaintext) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql.unsafe( - `SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t - WHERE (eql_v2.jsonb_path_query_first(t.metadata, $1::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - = (eql_v2.jsonb_path_query_first(t.metadata, $1::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - AND t.id = $2`, - [selectorTerm, id], - ) - - expect(rows).toHaveLength(1) - await verifyRow(rows[0], plaintext) - }, 30000) - - it('equality across two documents with same plaintext matches both', async () => { - const doc1 = { role: 'eq-cross-same', dept: 'eq-cross-d1' } - const doc2 = { role: 'eq-cross-same', dept: 'eq-cross-d2' } - - const { id: id1 } = await insertRow(doc1) - const { id: id2 } = await insertRow(doc2) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT a.id as id_a, b.id as id_b - FROM "protect-ci-jsonb" a, "protect-ci-jsonb" b - WHERE (eql_v2.jsonb_path_query_first(a.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - = (eql_v2.jsonb_path_query_first(b.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - AND a.id = ${id1} - AND b.id = ${id2} - ` - - expect(rows).toHaveLength(1) - expect(rows[0]).toMatchObject({ id_a: id1, id_b: id2 }) - }, 30000) - - it('equality across two documents with different plaintext returns empty', async () => { - const doc1 = { role: 'eq-cross-mismatch-1', marker: 'eq-mm-1' } - const doc2 = { role: 'eq-cross-mismatch-2', marker: 'eq-mm-2' } - - const { id: id1 } = await insertRow(doc1) - const { id: id2 } = await insertRow(doc2) - - const selectorTerm = await encryptQueryTerm('$.role', 'steVecSelector') - - const rows = await sql` - SELECT a.id as id_a, b.id as id_b - FROM "protect-ci-jsonb" a, "protect-ci-jsonb" b - WHERE (eql_v2.jsonb_path_query_first(a.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - = (eql_v2.jsonb_path_query_first(b.metadata, ${selectorTerm}::eql_v2_encrypted)).data::eql_v2.ste_vec_entry - AND a.id = ${id1} - AND b.id = ${id2} - ` - - expect(rows).toHaveLength(0) - }, 30000) - }) - - // ─── eql (default) return type ────────────────────────────────────── - - describe('eql (default) return type', () => { - it('selector query using raw eql return type', async () => { - const plaintext = { - user: { email: 'eql-raw-sel@test.com' }, - marker: 'eql-raw-sel', - } - const { id } = await insertRow(plaintext) - - // Omit returnType — single-value encryptQuery returns raw Encrypted object - const queryResult = await protectClient.encryptQuery('$.user.email', { - column: table.metadata, - table: table, - queryType: 'steVecSelector', - }) - if (queryResult.failure) throw new Error(queryResult.failure.message) - const rawResult = queryResult.data - - // Must use sql.json() to pass raw Encrypted object to PG - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${sql.json(rawResult)}::eql_v2_encrypted) as result - WHERE t.test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - - it('containment query using raw eql return type', async () => { - const plaintext = { role: 'eql-raw-contain', marker: 'eql-raw-ct' } - const { id } = await insertRow(plaintext) - - // Omit returnType — single-value encryptQuery returns raw Encrypted object - const queryResult = await protectClient.encryptQuery( - { role: 'eql-raw-contain' }, - { - column: table.metadata, - table: table, - queryType: 'steVecTerm', - }, - ) - if (queryResult.failure) throw new Error(queryResult.failure.message) - const rawResult = queryResult.data - - // Must use sql.json() to pass raw Encrypted object to PG - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" - WHERE metadata @> ${sql.json(rawResult)}::eql_v2_encrypted - AND test_run_id = ${TEST_RUN_ID} - ` - - expect(rows.length).toBeGreaterThanOrEqual(1) - const matchingRow = rows.find((r) => r.id === id) - expect(matchingRow).toBeDefined() - await verifyRow(matchingRow!, plaintext) - }, 30000) - }) - - // ─── Concurrent encrypt + decrypt stress ──────────────────────────── - - describe('concurrent encrypt + decrypt stress', () => { - it('concurrent encrypt + decrypt stress (10 parallel)', async () => { - const docs = Array.from({ length: 10 }, (_, i) => ({ - user: { email: `stress-${i}@test.com` }, - role: `stress-role-${i}`, - index: i, - marker: `stress-${i}`, - })) - - // Insert all 10 docs - const insertedIds: number[] = [] - for (const plaintext of docs) { - const { id } = await insertRow(plaintext) - insertedIds.push(id) - } - - // 10 parallel encrypt-query-decrypt pipelines - const results = await Promise.all( - docs.map(async (plaintext, i) => { - // Encrypt a selector query - const selectorTerm = await encryptQueryTerm( - '$.user.email', - 'steVecSelector', - ) - - // Query PG - const rows = await sql` - SELECT id, (metadata).data as metadata - FROM "protect-ci-jsonb" t, - eql_v2.jsonb_path_query(t.metadata, ${selectorTerm}::eql_v2_encrypted) as result - WHERE t.id = ${insertedIds[i]} - ` - - expect(rows).toHaveLength(1) - - // Decrypt - const decrypted = await protectClient.decryptModel({ - metadata: rows[0].metadata, - }) - if (decrypted.failure) throw new Error(decrypted.failure.message) - - return decrypted.data.metadata - }), - ) - - // Assert all 10 return correct plaintext - expect(results).toHaveLength(10) - results.forEach((result, i) => { - expect(result).toEqual(docs[i]) - }) - }, 120000) - }) -}) diff --git a/packages/protect/__tests__/supabase.test.ts b/packages/protect/__tests__/supabase.test.ts deleted file mode 100644 index 8face3bd5..000000000 --- a/packages/protect/__tests__/supabase.test.ts +++ /dev/null @@ -1,343 +0,0 @@ -import 'dotenv/config' -import { csColumn, csTable } from '@cipherstash/schema' -import { createClient } from '@supabase/supabase-js' -import postgres from 'postgres' -import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { - bulkModelsToEncryptedPgComposites, - type Encrypted, - encryptedToPgComposite, - isEncryptedPayload, - modelToEncryptedPgComposites, - protect, -} from '../src' - -// supabase.test.ts needs a live Supabase project, so the suite is skipped -// when the Supabase environment is not configured (e.g. in CI, pending a -// containerised Supabase setup). It runs locally when SUPABASE_URL, -// SUPABASE_ANON_KEY, and DATABASE_URL are all set. -const SUPABASE_ENABLED = Boolean( - process.env.SUPABASE_URL && - process.env.SUPABASE_ANON_KEY && - process.env.DATABASE_URL, -) - -const supabase = SUPABASE_ENABLED - ? createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!) - : (undefined as unknown as ReturnType) - -const table = csTable('protect-ci', { - encrypted: csColumn('encrypted').freeTextSearch().equality(), - age: csColumn('age').dataType('number').equality(), - score: csColumn('score').dataType('number').equality(), -}) - -// Hard code this as the CI database doesn't support order by on encrypted columns -const SKIP_ORDER_BY_TEST = true - -// Unique identifier for this test run to isolate data from concurrent test runs -// This is stored in a dedicated test_run_id column to avoid polluting test data -const TEST_RUN_ID = `test-run-${Date.now()}-${Math.random().toString(36).slice(2, 8)}` - -// Track all inserted IDs for cleanup -const insertedIds: number[] = [] - -beforeAll(async () => { - // Idempotent fixture setup. The `protect-ci` table is shared across the - // drizzle + protect/supabase + stack/supabase integration suites; each - // suite's beforeAll runs the same CREATE TABLE so a fresh database is - // ready without manual DBA work. The schema is the union of every - // column those suites read or write. - const sql = postgres(process.env.DATABASE_URL as string, { prepare: false }) - try { - await sql` - CREATE TABLE IF NOT EXISTS "protect-ci" ( - id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, - email eql_v2_encrypted, - age eql_v2_encrypted, - score eql_v2_encrypted, - profile eql_v2_encrypted, - encrypted eql_v2_encrypted, - "otherField" TEXT, - created_at TIMESTAMPTZ DEFAULT NOW(), - test_run_id TEXT - ) - ` - // Backfill any column added after the table was first created on a - // long-lived CI database. CREATE TABLE IF NOT EXISTS is a no-op on - // those, so new columns need an explicit ADD COLUMN IF NOT EXISTS. - await sql` - ALTER TABLE "protect-ci" ADD COLUMN IF NOT EXISTS "otherField" TEXT - ` - // Tell PostgREST to refresh its schema cache so the supabase-js client - // can see a freshly created table without waiting for the polling - // interval. No-op on plain Postgres (no listener bound). - await sql`NOTIFY pgrst, 'reload schema'` - } finally { - await sql.end() - } - - // Clean up any data from this specific test run (safe for concurrent runs) - const { error } = await supabase - .from('protect-ci') - .delete() - .eq('test_run_id', TEST_RUN_ID) - - if (error) { - console.warn(`[protect]: Failed to clean up test data: ${error.message}`) - } -}, 30000) - -afterAll(async () => { - // Clean up all data from this test run - if (insertedIds.length > 0) { - const { error } = await supabase - .from('protect-ci') - .delete() - .in('id', insertedIds) - if (error) { - console.error(`[protect]: Failed to clean up test data: ${error.message}`) - } - } -}) - -describe.skipIf(!SUPABASE_ENABLED)('supabase', () => { - it('should insert and select encrypted data', async () => { - const protectClient = await protect({ schemas: [table] }) - - const e = 'hello world' - - const ciphertext = await protectClient.encrypt(e, { - column: table.encrypted, - table: table, - }) - - if (ciphertext.failure) { - throw new Error(`[protect]: ${ciphertext.failure.message}`) - } - - const { data: insertedData, error: insertError } = await supabase - .from('protect-ci') - .insert({ - encrypted: encryptedToPgComposite(ciphertext.data), - test_run_id: TEST_RUN_ID, - }) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(insertedData[0].id) - - const { data, error } = await supabase - .from('protect-ci') - .select('id, encrypted::jsonb') - .eq('id', insertedData[0].id) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - const dataToDecrypt = data[0].encrypted as Encrypted - const plaintext = await protectClient.decrypt(dataToDecrypt) - - expect(plaintext).toEqual({ - data: e, - }) - }, 30000) - - it('should insert and select encrypted model data', async () => { - const protectClient = await protect({ schemas: [table] }) - - const model = { - encrypted: 'hello world', - otherField: 'not encrypted', - } - - const encryptedModel = await protectClient.encryptModel(model, table) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - const { data: insertedData, error: insertError } = await supabase - .from('protect-ci') - .insert([ - { - ...modelToEncryptedPgComposites(encryptedModel.data), - test_run_id: TEST_RUN_ID, - }, - ]) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(insertedData[0].id) - - const { data, error } = await supabase - .from('protect-ci') - .select('id, encrypted::jsonb, otherField') - .eq('id', insertedData[0].id) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - if (!isEncryptedPayload(data[0].encrypted)) { - throw new Error('Expected encrypted payload') - } - - const decryptedModel = await protectClient.decryptModel(data[0]) - - if (decryptedModel.failure) { - throw new Error(`[protect]: ${decryptedModel.failure.message}`) - } - - expect({ - encrypted: decryptedModel.data.encrypted, - otherField: data[0].otherField, - }).toEqual(model) - }, 30000) - - it('should insert and select bulk encrypted model data', async () => { - const protectClient = await protect({ schemas: [table] }) - - const models = [ - { - encrypted: 'hello world 1', - otherField: 'not encrypted 1', - }, - { - encrypted: 'hello world 2', - otherField: 'not encrypted 2', - }, - ] - - const encryptedModels = await protectClient.bulkEncryptModels(models, table) - - if (encryptedModels.failure) { - throw new Error(`[protect]: ${encryptedModels.failure.message}`) - } - - const dataToInsert = bulkModelsToEncryptedPgComposites( - encryptedModels.data, - ).map((row) => ({ - ...row, - test_run_id: TEST_RUN_ID, - })) - - const { data: insertedData, error: insertError } = await supabase - .from('protect-ci') - .insert(dataToInsert) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(...insertedData.map((d: { id: number }) => d.id)) - - const { data, error } = await supabase - .from('protect-ci') - .select('id, encrypted::jsonb, otherField') - .in( - 'id', - insertedData.map((d: { id: number }) => d.id), - ) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - const decryptedModels = await protectClient.bulkDecryptModels(data) - - if (decryptedModels.failure) { - throw new Error(`[protect]: ${decryptedModels.failure.message}`) - } - - expect( - decryptedModels.data.map((d) => { - return { - encrypted: d.encrypted, - otherField: d.otherField, - } - }), - ).toEqual(models) - }, 30000) - - it('should insert and query encrypted number data with equality', async () => { - const protectClient = await protect({ schemas: [table] }) - - const testAge = 25 - const model = { - age: testAge, - otherField: 'not encrypted', - } - - const encryptedModel = await protectClient.encryptModel(model, table) - - if (encryptedModel.failure) { - throw new Error(`[protect]: ${encryptedModel.failure.message}`) - } - - const insertResult = await supabase - .from('protect-ci') - .insert([ - { - ...modelToEncryptedPgComposites(encryptedModel.data), - test_run_id: TEST_RUN_ID, - }, - ]) - .select('id') - - if (insertResult.error) { - throw new Error(`[protect]: ${insertResult.error.message}`) - } - - const insertedRecordId = insertResult.data[0].id - insertedIds.push(insertedRecordId) - - // Create encrypted query for equality search with composite-literal returnType - const encryptedResult = await protectClient.encryptQuery([ - { - value: testAge, - column: table.age, - table: table, - queryType: 'equality', - returnType: 'composite-literal', - }, - ]) - - if (encryptedResult.failure) { - throw new Error(`[protect]: ${encryptedResult.failure.message}`) - } - - const [searchTerm] = encryptedResult.data - - // Query filtering by both encrypted age AND our specific test run's ID - // This ensures we don't pick up stale data from other test runs - const { data, error } = await supabase - .from('protect-ci') - .select('id, age::jsonb, otherField') - .eq('age', searchTerm) - .eq('test_run_id', TEST_RUN_ID) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - // Verify we found our specific row with encrypted age match - expect(data).toHaveLength(1) - - const decryptedModel = await protectClient.decryptModel(data[0]) - - if (decryptedModel.failure) { - throw new Error(`[protect]: ${decryptedModel.failure.message}`) - } - - expect(decryptedModel.data.age).toBe(testAge) - }, 30000) -}) diff --git a/packages/protect/package.json b/packages/protect/package.json deleted file mode 100644 index 2864adee6..000000000 --- a/packages/protect/package.json +++ /dev/null @@ -1,82 +0,0 @@ -{ - "name": "@cipherstash/protect", - "version": "12.0.2-rc.0", - "description": "CipherStash Protect for JavaScript", - "keywords": [ - "encrypted", - "query", - "language", - "typescript", - "ts", - "protect" - ], - "bugs": { - "url": "https://github.com/cipherstash/stack/issues" - }, - "repository": { - "type": "git", - "url": "git+https://github.com/cipherstash/stack.git", - "directory": "packages/protect" - }, - "license": "MIT", - "author": "CipherStash ", - "type": "module", - "bin": { - "stash": "./dist/bin/stash.js" - }, - "main": "./dist/index.cjs", - "types": "./dist/index.d.ts", - "exports": { - ".": { - "types": "./dist/index.d.ts", - "import": "./dist/index.js", - "require": "./dist/index.cjs" - }, - "./client": { - "types": "./dist/client.d.ts", - "import": "./dist/client.js", - "require": "./dist/client.cjs" - }, - "./identify": { - "types": "./dist/identify/index.d.ts", - "import": "./dist/identify/index.js", - "require": "./dist/identify/index.cjs" - }, - "./stash": { - "types": "./dist/stash/index.d.ts", - "import": "./dist/stash/index.js", - "require": "./dist/stash/index.cjs" - } - }, - "scripts": { - "build": "tsup", - "postbuild": "chmod +x ./dist/bin/stash.js", - "dev": "tsup --watch", - "test": "vitest run", - "release": "tsup" - }, - "devDependencies": { - "@supabase/supabase-js": "^2.110.2", - "execa": "^9.5.2", - "json-schema-to-typescript": "^15.0.2", - "postgres": "^3.4.7", - "tsup": "catalog:repo", - "tsx": "catalog:repo", - "typescript": "catalog:repo", - "vitest": "catalog:repo" - }, - "publishConfig": { - "access": "public" - }, - "dependencies": { - "@byteslice/result": "^0.2.0", - "@cipherstash/protect-ffi": "0.23.0", - "@cipherstash/schema": "workspace:*", - "@stricli/core": "^1.2.9", - "dotenv": "17.4.2", - "zod": "^3.25.76" - }, - "optionalDependencies": { - "@rollup/rollup-linux-x64-gnu": "4.62.2" - } -} diff --git a/packages/protect/src/bin/runner.ts b/packages/protect/src/bin/runner.ts deleted file mode 100644 index c313f5c5e..000000000 --- a/packages/protect/src/bin/runner.ts +++ /dev/null @@ -1,35 +0,0 @@ -import { existsSync } from 'node:fs' -import { resolve } from 'node:path' - -type Pm = 'npm' | 'pnpm' | 'yarn' | 'bun' - -function fromUserAgent(): Pm | undefined { - const ua = process.env.npm_config_user_agent ?? '' - if (ua.startsWith('bun/')) return 'bun' - if (ua.startsWith('pnpm/')) return 'pnpm' - if (ua.startsWith('yarn/')) return 'yarn' - return undefined -} - -function fromLockfile(cwd: string): Pm | undefined { - if ( - existsSync(resolve(cwd, 'bun.lockb')) || - existsSync(resolve(cwd, 'bun.lock')) - ) - return 'bun' - if (existsSync(resolve(cwd, 'pnpm-lock.yaml'))) return 'pnpm' - if (existsSync(resolve(cwd, 'yarn.lock'))) return 'yarn' - if (existsSync(resolve(cwd, 'package-lock.json'))) return 'npm' - return undefined -} - -export function detectRunner(): string { - const pm = fromUserAgent() ?? fromLockfile(process.cwd()) ?? 'npm' - return pm === 'bun' - ? 'bunx' - : pm === 'pnpm' - ? 'pnpm dlx' - : pm === 'yarn' - ? 'yarn dlx' - : 'npx' -} diff --git a/packages/protect/src/bin/stash.ts b/packages/protect/src/bin/stash.ts deleted file mode 100644 index 7506cbae6..000000000 --- a/packages/protect/src/bin/stash.ts +++ /dev/null @@ -1,502 +0,0 @@ -import { config } from 'dotenv' - -config() - -import readline from 'node:readline' -import { - buildApplication, - buildCommand, - buildRouteMap, - run, -} from '@stricli/core' -import { Stash } from '../stash/index.js' -import { detectRunner } from './runner.js' - -// ANSI color codes for beautiful terminal output -const colors = { - reset: '\x1b[0m', - bold: '\x1b[1m', - dim: '\x1b[2m', - green: '\x1b[32m', - red: '\x1b[31m', - yellow: '\x1b[33m', - blue: '\x1b[34m', - cyan: '\x1b[36m', - magenta: '\x1b[35m', -} - -const style = { - success: (text: string) => - `${colors.green}${colors.bold}✓${colors.reset} ${colors.green}${text}${colors.reset}`, - error: (text: string) => - `${colors.red}${colors.bold}✗${colors.reset} ${colors.red}${text}${colors.reset}`, - info: (text: string) => - `${colors.blue}${colors.bold}ℹ${colors.reset} ${colors.blue}${text}${colors.reset}`, - warning: (text: string) => - `${colors.yellow}${colors.bold}⚠${colors.reset} ${colors.yellow}${text}${colors.reset}`, - title: (text: string) => `${colors.bold}${colors.cyan}${text}${colors.reset}`, - label: (text: string) => `${colors.dim}${text}${colors.reset}`, - value: (text: string) => `${colors.bold}${text}${colors.reset}`, - bullet: () => `${colors.green}•${colors.reset}`, -} - -// Detect the package manager and build the CLI reference -const runner = detectRunner() -const cliRef = `${runner} stash` - -/** - * Get configuration from environment variables - */ -function getConfig(environment: string): Stash['config'] { - const workspaceCRN = process.env.CS_WORKSPACE_CRN - const clientId = process.env.CS_CLIENT_ID - const clientKey = process.env.CS_CLIENT_KEY - const apiKey = process.env.CS_CLIENT_ACCESS_KEY - const accessKey = process.env.CS_ACCESS_KEY - - const missing: string[] = [] - if (!workspaceCRN) missing.push('CS_WORKSPACE_CRN') - if (!clientId) missing.push('CS_CLIENT_ID') - if (!clientKey) missing.push('CS_CLIENT_KEY') - if (!apiKey) missing.push('CS_CLIENT_ACCESS_KEY') - - if (missing.length > 0) { - console.error( - style.error( - `Missing required environment variables: ${missing.join(', ')}`, - ), - ) - console.error( - `\n${style.info('Please set the following environment variables:')}`, - ) - for (const varName of missing) { - console.error(` ${style.bullet()} ${varName}`) - } - process.exit(1) - } - - if (!workspaceCRN || !clientId || !clientKey || !apiKey) { - // This should never happen due to the check above, but TypeScript needs it - throw new Error('Missing required configuration') - } - - return { - workspaceCRN, - clientId, - clientKey, - apiKey, - accessKey, - environment, - } -} - -/** - * Create a Stash instance with proper error handling - */ -function createStash(environment: string): Stash { - const config = getConfig(environment) - return new Stash(config) -} - -/** - * Prompt user for confirmation - */ -function askConfirmation(prompt: string): Promise { - const rl = readline.createInterface({ - input: process.stdin, - output: process.stdout, - }) - - return new Promise((resolve) => { - rl.question(prompt, (answer) => { - rl.close() - const normalized = answer.trim().toLowerCase() - resolve(normalized === 'y' || normalized === 'yes') - }) - }) -} - -/** - * Set command - Store an encrypted secret - */ -const setCommand = buildCommand({ - func: async (flags: { name: string; value: string; environment: string }) => { - const { name, value, environment } = flags - const stash = createStash(environment) - - console.log( - `${style.info(`Encrypting and storing secret "${name}" in environment "${environment}"...`)}`, - ) - - const result = await stash.set(name, value) - if (result.failure) { - console.error( - style.error(`Failed to set secret: ${result.failure.message}`), - ) - process.exit(1) - } - - console.log( - style.success( - `Secret "${name}" stored successfully in environment "${environment}"`, - ), - ) - }, - parameters: { - flags: { - name: { - kind: 'parsed', - parse: String, - brief: 'Name of the secret to store', - }, - value: { - kind: 'parsed', - parse: String, - brief: 'Plaintext value to encrypt and store', - }, - environment: { - kind: 'parsed', - parse: String, - brief: 'Environment name (e.g., production, staging, development)', - }, - }, - aliases: { - n: 'name', - V: 'value', - e: 'environment', - }, - }, - docs: { - brief: 'Store an encrypted secret in CipherStash', - fullDescription: ` -Store a secret value that will be encrypted locally before being sent to the CipherStash API. -The secret is encrypted end-to-end, ensuring your plaintext never leaves your machine unencrypted. - -Examples: - ${cliRef} secrets set --name DATABASE_URL --value "postgres://..." --environment production - ${cliRef} secrets set -n DATABASE_URL -V "postgres://..." -e production - ${cliRef} secrets set --name API_KEY --value "sk-123..." --environment staging - `.trim(), - }, -}) - -/** - * Get command - Retrieve and decrypt a secret - */ -const getCommand = buildCommand({ - func: async (flags: { name: string; environment: string }) => { - const { name, environment } = flags - const stash = createStash(environment) - - console.log( - `${style.info(`Retrieving secret "${name}" from environment "${environment}"...`)}`, - ) - - const result = await stash.get(name) - if (result.failure) { - console.error( - style.error(`Failed to get secret: ${result.failure.message}`), - ) - process.exit(1) - } - - console.log(`\n${style.title('Secret Value:')}`) - console.log(style.value(result.data)) - }, - parameters: { - flags: { - name: { - kind: 'parsed', - parse: String, - brief: 'Name of the secret to retrieve', - }, - environment: { - kind: 'parsed', - parse: String, - brief: 'Environment name (e.g., production, staging, development)', - }, - }, - aliases: { - n: 'name', - e: 'environment', - }, - }, - docs: { - brief: 'Retrieve and decrypt a secret from CipherStash', - fullDescription: ` -Retrieve a secret from CipherStash and decrypt it locally. The secret value is decrypted -on your machine, ensuring end-to-end security. - -Examples: - ${cliRef} secrets get --name DATABASE_URL --environment production - ${cliRef} secrets get -n DATABASE_URL -e production - ${cliRef} secrets get --name API_KEY --environment staging - `.trim(), - }, -}) - -/** - * List command - List all secrets in an environment - */ -const listCommand = buildCommand({ - func: async (flags: { environment: string }) => { - const { environment } = flags - const stash = createStash(environment) - - console.log( - `${style.info(`Listing secrets in environment "${environment}"...`)}`, - ) - - const result = await stash.list() - if (result.failure) { - console.error( - style.error(`Failed to list secrets: ${result.failure.message}`), - ) - process.exit(1) - } - - if (result.data.length === 0) { - console.log( - `\n${style.warning(`No secrets found in environment "${environment}"`)}`, - ) - return - } - - console.log(`\n${style.title(`Secrets in environment "${environment}":`)}`) - console.log('') - - for (const secret of result.data) { - const name = style.value(secret.name) - const metadata: string[] = [] - if (secret.createdAt) { - metadata.push( - `${style.label('created:')} ${new Date(secret.createdAt).toLocaleString()}`, - ) - } - if (secret.updatedAt) { - metadata.push( - `${style.label('updated:')} ${new Date(secret.updatedAt).toLocaleString()}`, - ) - } - - const metaStr = - metadata.length > 0 - ? ` ${colors.dim}(${metadata.join(', ')})${colors.reset}` - : '' - console.log(` ${style.bullet()} ${name}${metaStr}`) - } - - console.log('') - console.log( - style.label( - `Total: ${result.data.length} secret${result.data.length === 1 ? '' : 's'}`, - ), - ) - }, - parameters: { - flags: { - environment: { - kind: 'parsed', - parse: String, - brief: 'Environment name (e.g., production, staging, development)', - }, - }, - aliases: { - e: 'environment', - }, - }, - docs: { - brief: 'List all secrets in an environment', - fullDescription: ` -List all secrets stored in the specified environment. Only secret names and metadata -are returned; values remain encrypted and are not displayed. - -Examples: - ${cliRef} secrets list --environment production - ${cliRef} secrets list -e production - ${cliRef} secrets list --environment staging - `.trim(), - }, -}) - -/** - * Delete command - Delete a secret from the vault - */ -const deleteCommand = buildCommand({ - func: async (flags: { name: string; environment: string; yes?: boolean }) => { - const { name, environment, yes } = flags - const stash = createStash(environment) - - // Ask for confirmation unless --yes flag is set - if (!yes) { - const confirmation = await askConfirmation( - `${style.warning(`Are you sure you want to delete secret "${name}" from environment "${environment}"? This action cannot be undone. (yes/no): `)}`, - ) - - if (!confirmation) { - console.log(style.info('Deletion cancelled.')) - return - } - } - - console.log( - `${style.info(`Deleting secret "${name}" from environment "${environment}"...`)}`, - ) - - const result = await stash.delete(name) - if (result.failure) { - console.error( - style.error(`Failed to delete secret: ${result.failure.message}`), - ) - process.exit(1) - } - - console.log( - style.success( - `Secret "${name}" deleted successfully from environment "${environment}"`, - ), - ) - }, - parameters: { - flags: { - name: { - kind: 'parsed', - parse: String, - brief: 'Name of the secret to delete', - }, - environment: { - kind: 'parsed', - parse: String, - brief: 'Environment name (e.g., production, staging, development)', - }, - yes: { - kind: 'boolean', - optional: true, - brief: 'Skip confirmation prompt', - }, - }, - aliases: { - n: 'name', - e: 'environment', - y: 'yes', - }, - }, - docs: { - brief: 'Delete a secret from CipherStash', - fullDescription: ` -Permanently delete a secret from the specified environment. This action cannot be undone. -By default, you will be prompted for confirmation before deletion. Use --yes to skip the confirmation. - -Examples: - ${cliRef} secrets delete --name DATABASE_URL --environment production - ${cliRef} secrets delete -n DATABASE_URL -e production --yes - ${cliRef} secrets delete --name API_KEY --environment staging -y - `.trim(), - }, -}) - -/** - * Secrets route map - Groups all secret management commands - */ -const secretsRouteMap = buildRouteMap({ - routes: { - set: setCommand, - get: getCommand, - list: listCommand, - delete: deleteCommand, - }, - docs: { - brief: 'Manage encrypted secrets in CipherStash', - fullDescription: ` -The secrets command group provides operations for managing encrypted secrets stored in CipherStash. -All secrets are encrypted locally before being sent to the API, ensuring end-to-end encryption. - -Available Commands: - set Store an encrypted secret - get Retrieve and decrypt a secret - list List all secrets in an environment - delete Delete a secret from the vault - -Environment Variables: - CS_WORKSPACE_CRN CipherStash workspace CRN (required) - CS_CLIENT_ID CipherStash client ID (required) - CS_CLIENT_KEY CipherStash client key (required) - CS_CLIENT_ACCESS_KEY CipherStash client access key (required) - -Examples: - ${cliRef} secrets set --name DATABASE_URL --value "postgres://..." --environment production - ${cliRef} secrets set -n DATABASE_URL -V "postgres://..." -e production - ${cliRef} secrets get --name DATABASE_URL --environment production - ${cliRef} secrets get -n DATABASE_URL -e production - ${cliRef} secrets list --environment production - ${cliRef} secrets list -e production - ${cliRef} secrets delete --name DATABASE_URL --environment production - ${cliRef} secrets delete -n DATABASE_URL -e production --yes - ${cliRef} secrets delete -n DATABASE_URL -e production -y - `.trim(), - }, -}) - -/** - * Root command - Entry point for the CLI - */ -const rootRouteMap = buildRouteMap({ - routes: { - secrets: secretsRouteMap, - }, - docs: { - brief: 'CipherStash Protect - Encrypted secrets management', - fullDescription: ` -CipherStash Protect CLI - -Manage encrypted secrets with end-to-end encryption. Secrets are encrypted locally -before being sent to the CipherStash API, ensuring your plaintext never leaves -your machine unencrypted. - -Quick Start: - 1. Set required environment variables (CS_WORKSPACE_CRN, CS_CLIENT_ID, etc.) - 2. Use '${cliRef} secrets set' to store your first secret - 3. Use '${cliRef} secrets get' to retrieve secrets when needed - -Commands: - secrets Manage encrypted secrets - -Run '${cliRef} --help' for more information about a command. - `.trim(), - }, -}) - -/** - * Build the CLI application - */ -const app = buildApplication(rootRouteMap, { - name: 'stash', - versionInfo: { currentVersion: '10.2.1' }, - scanner: { caseStyle: 'allow-kebab-for-camel' }, -}) - -/** - * Main entry point - */ -async function main(): Promise { - try { - await run(app, process.argv.slice(2), { - process, - async forCommand() { - return { - process, - } - }, - }) - } catch (error) { - const message = error instanceof Error ? error.message : String(error) - console.error(style.error(`Unexpected error: ${message}`)) - process.exit(1) - } -} - -void main().catch((error: unknown) => { - const message = error instanceof Error ? error.message : String(error) - console.error(style.error(`Fatal error: ${message}`)) - process.exit(1) -}) diff --git a/packages/protect/src/client.ts b/packages/protect/src/client.ts deleted file mode 100644 index 9d165ecc1..000000000 --- a/packages/protect/src/client.ts +++ /dev/null @@ -1,18 +0,0 @@ -/** - * Client-safe exports for @cipherstash/protect - * - * This entry point exports types and utilities that can be used in client-side code - * without requiring the @cipherstash/protect-ffi native module. - * - * Use this import path: `@cipherstash/protect/client` - */ - -export type { - ProtectColumn, - ProtectTable, - ProtectTableColumn, - ProtectValue, -} from '@cipherstash/schema' -// Schema types and utilities - client-safe -export { csColumn, csTable, csValue } from '@cipherstash/schema' -export type { ProtectClient } from './ffi' diff --git a/packages/protect/src/ffi/helpers/error-code.ts b/packages/protect/src/ffi/helpers/error-code.ts deleted file mode 100644 index 0552e84a1..000000000 --- a/packages/protect/src/ffi/helpers/error-code.ts +++ /dev/null @@ -1,12 +0,0 @@ -import { - ProtectError as FfiProtectError, - type ProtectErrorCode, -} from '@cipherstash/protect-ffi' - -/** - * Extracts FFI error code from an error if it's an FFI error, otherwise returns undefined. - * Used to preserve specific error codes in ProtectError responses. - */ -export function getErrorCode(error: unknown): ProtectErrorCode | undefined { - return error instanceof FfiProtectError ? error.code : undefined -} diff --git a/packages/protect/src/ffi/helpers/infer-index-type.ts b/packages/protect/src/ffi/helpers/infer-index-type.ts deleted file mode 100644 index 3700ff166..000000000 --- a/packages/protect/src/ffi/helpers/infer-index-type.ts +++ /dev/null @@ -1,120 +0,0 @@ -import type { JsPlaintext, QueryOpName } from '@cipherstash/protect-ffi' -import type { ProtectColumn } from '@cipherstash/schema' -import type { FfiIndexTypeName, QueryTypeName } from '../../types' -import { queryTypeToFfi, queryTypeToQueryOp } from '../../types' - -/** - * Infer the primary index type from a column's configured indexes. - * Priority: unique > match > ore > ste_vec (for scalar queries) - */ -export function inferIndexType(column: ProtectColumn): FfiIndexTypeName { - const config = column.build() - const indexes = config.indexes - - if (!indexes || Object.keys(indexes).length === 0) { - throw new Error(`Column "${column.getName()}" has no indexes configured`) - } - - // Priority order for inference - if (indexes.unique) return 'unique' - if (indexes.match) return 'match' - if (indexes.ore) return 'ore' - if (indexes.ste_vec) return 'ste_vec' - - throw new Error( - `Column "${column.getName()}" has no suitable index for queries`, - ) -} - -/** - * Infer the FFI query operation from plaintext type for STE Vec queries. - * - String → ste_vec_selector (JSONPath queries like '$.user.email') - * - Object/Array/Number/Boolean → ste_vec_term (containment queries) - */ -export function inferQueryOpFromPlaintext(plaintext: JsPlaintext): QueryOpName { - if (typeof plaintext === 'string') { - return 'ste_vec_selector' - } - // Objects, arrays, numbers, booleans are all valid JSONB containment values - if ( - typeof plaintext === 'object' || - typeof plaintext === 'number' || - typeof plaintext === 'boolean' || - typeof plaintext === 'bigint' - ) { - return 'ste_vec_term' - } - // This should never happen with valid JsPlaintext, but keep for safety - return 'ste_vec_term' -} - -/** - * Validate that the specified index type is configured on the column - */ -export function validateIndexType( - column: ProtectColumn, - indexType: FfiIndexTypeName, -): void { - const config = column.build() - const indexes = config.indexes ?? {} - - const indexMap: Record = { - unique: !!indexes.unique, - match: !!indexes.match, - ore: !!indexes.ore, - ste_vec: !!indexes.ste_vec, - } - - if (!indexMap[indexType]) { - throw new Error( - `Index type "${indexType}" is not configured on column "${column.getName()}"`, - ) - } -} - -/** - * Resolve the index type and query operation for a query. - * Validates the index type is configured on the column when queryType is explicit. - * For ste_vec columns without explicit queryType, infers queryOp from plaintext shape. - * - * @param column - The column to resolve the index type for - * @param queryType - Optional explicit query type (if provided, validates against column config) - * @param plaintext - Optional plaintext value for queryOp inference on ste_vec columns - * @returns The FFI index type name and optional query operation name - * @throws Error if ste_vec is inferred but queryOp cannot be determined - */ -export function resolveIndexType( - column: ProtectColumn, - queryType?: QueryTypeName, - plaintext?: JsPlaintext | null, -): { indexType: FfiIndexTypeName; queryOp?: QueryOpName } { - const indexType = queryType - ? queryTypeToFfi[queryType] - : inferIndexType(column) - - if (queryType) { - validateIndexType(column, indexType) - - // For searchableJson, infer queryOp from plaintext type (not from mapping) - if (queryType === 'searchableJson') { - if (plaintext === undefined || plaintext === null) { - return { indexType } - } - return { indexType, queryOp: inferQueryOpFromPlaintext(plaintext) } - } - - return { indexType, queryOp: queryTypeToQueryOp[queryType] } - } - - // ste_vec inferred without explicit queryType → must infer from plaintext - if (indexType === 'ste_vec') { - if (plaintext === undefined || plaintext === null) { - // Null plaintext handled by caller (returns null early) - no inference needed - return { indexType } - } - return { indexType, queryOp: inferQueryOpFromPlaintext(plaintext) } - } - - // Non-ste_vec → no queryOp needed - return { indexType } -} diff --git a/packages/protect/src/ffi/helpers/type-guards.ts b/packages/protect/src/ffi/helpers/type-guards.ts deleted file mode 100644 index 86fb2fece..000000000 --- a/packages/protect/src/ffi/helpers/type-guards.ts +++ /dev/null @@ -1,18 +0,0 @@ -import type { ScalarQueryTerm } from '../../types' - -/** - * Type guard to check if a value is an array of ScalarQueryTerm objects. - * Used to discriminate between single value and bulk encryption in encryptQuery overloads. - */ -export function isScalarQueryTermArray( - value: unknown, -): value is readonly ScalarQueryTerm[] { - return ( - Array.isArray(value) && - value.length > 0 && - typeof value[0] === 'object' && - value[0] !== null && - 'column' in value[0] && - 'table' in value[0] - ) -} diff --git a/packages/protect/src/ffi/helpers/validation.ts b/packages/protect/src/ffi/helpers/validation.ts deleted file mode 100644 index d544bcf9c..000000000 --- a/packages/protect/src/ffi/helpers/validation.ts +++ /dev/null @@ -1,94 +0,0 @@ -import type { Result } from '@byteslice/result' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { FfiIndexTypeName } from '../../types' - -/** - * Validates that a value is not NaN or Infinity. - * Returns a failure Result if validation fails, undefined otherwise. - * Use this in async flows that return Result types. - * - * Uses `never` as the success type so the result can be assigned to any Result. - * - * @internal - */ -export function validateNumericValue( - value: unknown, -): Result | undefined { - if (typeof value === 'number' && Number.isNaN(value)) { - return { - failure: { - type: ProtectErrorTypes.EncryptionError, - message: '[protect]: Cannot encrypt NaN value', - }, - } - } - if (typeof value === 'number' && !Number.isFinite(value)) { - return { - failure: { - type: ProtectErrorTypes.EncryptionError, - message: '[protect]: Cannot encrypt Infinity value', - }, - } - } - return undefined -} - -/** - * Validates that a value is not NaN or Infinity. - * Throws an error if validation fails. - * Use this in sync flows where exceptions are caught. - * - * @internal - */ -export function assertValidNumericValue(value: unknown): void { - if (typeof value === 'number' && Number.isNaN(value)) { - throw new Error('[protect]: Cannot encrypt NaN value') - } - if (typeof value === 'number' && !Number.isFinite(value)) { - throw new Error('[protect]: Cannot encrypt Infinity value') - } -} - -/** - * Validates that the value type is compatible with the index type. - * Match index (freeTextSearch) only supports string values. - * Returns a failure Result if validation fails, undefined otherwise. - * Use this in async flows that return Result types. - * - * @internal - */ -export function validateValueIndexCompatibility( - value: unknown, - indexType: FfiIndexTypeName, - columnName: string, -): Result | undefined { - if (typeof value === 'number' && indexType === 'match') { - return { - failure: { - type: ProtectErrorTypes.EncryptionError, - message: `[protect]: Cannot use 'match' index with numeric value on column "${columnName}". The 'freeTextSearch' index only supports string values. Configure the column with 'orderAndRange()' or 'equality()' for numeric queries.`, - }, - } - } - return undefined -} - -/** - * Validates that the value type is compatible with the index type. - * Match index (freeTextSearch) only supports string values. - * Throws an error if validation fails. - * Use this in sync flows where exceptions are caught. - * - * @internal - */ -export function assertValueIndexCompatibility( - value: unknown, - indexType: FfiIndexTypeName, - columnName: string, -): void { - if (typeof value === 'number' && indexType === 'match') { - throw new Error( - `[protect]: Cannot use 'match' index with numeric value on column "${columnName}". The 'freeTextSearch' index only supports string values. Configure the column with 'orderAndRange()' or 'equality()' for numeric queries.`, - ) - } -} diff --git a/packages/protect/src/ffi/index.ts b/packages/protect/src/ffi/index.ts deleted file mode 100644 index 9c0505bc3..000000000 --- a/packages/protect/src/ffi/index.ts +++ /dev/null @@ -1,428 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { type JsPlaintext, newClient } from '@cipherstash/protect-ffi' -import { - type EncryptConfig, - encryptConfigSchema, - type ProtectTable, - type ProtectTableColumn, -} from '@cipherstash/schema' -import { logger } from '../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '..' -import { toFfiKeysetIdentifier } from '../helpers' -import type { - BulkDecryptPayload, - BulkEncryptPayload, - Client, - Decrypted, - Encrypted, - EncryptOptions, - EncryptQueryOptions, - KeysetIdentifier, - ScalarQueryTerm, - SearchTerm, -} from '../types' -import { isScalarQueryTermArray } from './helpers/type-guards' -import { BatchEncryptQueryOperation } from './operations/batch-encrypt-query' -import { BulkDecryptOperation } from './operations/bulk-decrypt' -import { BulkDecryptModelsOperation } from './operations/bulk-decrypt-models' -import { BulkEncryptOperation } from './operations/bulk-encrypt' -import { BulkEncryptModelsOperation } from './operations/bulk-encrypt-models' -import { DecryptOperation } from './operations/decrypt' -import { DecryptModelOperation } from './operations/decrypt-model' -import { SearchTermsOperation } from './operations/deprecated/search-terms' -import { EncryptOperation } from './operations/encrypt' -import { EncryptModelOperation } from './operations/encrypt-model' -import { EncryptQueryOperation } from './operations/encrypt-query' - -export const noClientError = () => - new Error( - 'The EQL client has not been initialized. Please call init() before using the client.', - ) - -/** The ProtectClient is the main entry point for interacting with the CipherStash Protect.js library. - * It provides methods for encrypting and decrypting individual values, as well as models (objects) and bulk operations. - * - * The client must be initialized using the {@link protect} function before it can be used. - */ -export class ProtectClient { - private client: Client - private encryptConfig: EncryptConfig | undefined - - /** - * Initializes the ProtectClient with the provided configuration. - * @internal - * @param config - The configuration object for initializing the client. - * @returns A promise that resolves to a {@link Result} containing the initialized ProtectClient or a {@link ProtectError}. - **/ - async init(config: { - encryptConfig: EncryptConfig - workspaceCrn?: string - accessKey?: string - clientId?: string - clientKey?: string - keyset?: KeysetIdentifier - }): Promise> { - return await withResult( - async () => { - const validated: EncryptConfig = encryptConfigSchema.parse( - config.encryptConfig, - ) - - logger.debug( - 'Initializing the Protect.js client with the following encrypt config:', - { - encryptConfig: validated, - }, - ) - - // newClient handles env var fallback internally via withEnvCredentials, - // so we pass config values through without manual fallback here. - this.client = await newClient({ - encryptConfig: validated, - clientOpts: { - workspaceCrn: config.workspaceCrn, - accessKey: config.accessKey, - clientId: config.clientId, - clientKey: config.clientKey, - keyset: toFfiKeysetIdentifier(config.keyset), - }, - }) - - this.encryptConfig = validated - - logger.info('Successfully initialized the Protect.js client.') - return this - }, - (error: unknown) => ({ - type: ProtectErrorTypes.ClientInitError, - message: (error as Error).message, - }), - ) - } - - /** - * Encrypt a value - returns a promise which resolves to an encrypted value. - * - * @param plaintext - The plaintext value to be encrypted. Can be null. - * @param opts - Options specifying the column and table for encryption. - * @returns An EncryptOperation that can be awaited or chained with additional methods. - * - * @example - * The following example demonstrates how to encrypt a value using the Protect client. - * It includes defining an encryption schema with {@link csTable} and {@link csColumn}, - * initializing the client with {@link protect}, and performing the encryption. - * - * `encrypt` returns an {@link EncryptOperation} which can be awaited to get a {@link Result} - * which can either be the encrypted value or a {@link ProtectError}. - * - * ```typescript - * // Define encryption schema - * import { csTable, csColumn } from "@cipherstash/protect" - * const userSchema = csTable("users", { - * email: csColumn("email"), - * }); - * - * // Initialize Protect client - * const protectClient = await protect({ schemas: [userSchema] }) - * - * // Encrypt a value - * const encryptedResult = await protectClient.encrypt( - * "person@example.com", - * { column: userSchema.email, table: userSchema } - * ) - * - * // Handle encryption result - * if (encryptedResult.failure) { - * throw new Error(`Encryption failed: ${encryptedResult.failure.message}`); - * } - * - * console.log("Encrypted data:", encryptedResult.data); - * ``` - * - * @example - * When encrypting data, a {@link LockContext} can be provided to tie the encryption to a specific user or session. - * This ensures that the same lock context is required for decryption. - * - * The following example demonstrates how to create a lock context using a user's JWT token - * and use it during encryption. - * - * ```typescript - * // Define encryption schema and initialize client as above - * - * // Create a lock for the user's `sub` claim from their JWT - * const lc = new LockContext(); - * const lockContext = await lc.identify(userJwt); - * - * if (lockContext.failure) { - * // Handle the failure - * } - * - * // Encrypt a value with the lock context - * // Decryption will then require the same lock context - * const encryptedResult = await protectClient.encrypt( - * "person@example.com", - * { column: userSchema.email, table: userSchema } - * ) - * .withLockContext(lockContext) - * ``` - * - * @see {@link Result} - * @see {@link csTable} - * @see {@link LockContext} - * @see {@link EncryptOperation} - */ - encrypt( - plaintext: JsPlaintext | null, - opts: EncryptOptions, - ): EncryptOperation { - return new EncryptOperation(this.client, plaintext, opts) - } - - /** - * Encrypt a query value - returns a promise which resolves to an encrypted query value. - * - * @param plaintext - The plaintext value to be encrypted for querying. Can be null. - * @param opts - Options specifying the column, table, and optional queryType for encryption. - * @returns An EncryptQueryOperation that can be awaited or chained with additional methods. - * - * @example - * The following example demonstrates how to encrypt a query value using the Protect client. - * - * ```typescript - * // Define encryption schema - * import { csTable, csColumn } from "@cipherstash/protect" - * const userSchema = csTable("users", { - * email: csColumn("email").equality(), - * }); - * - * // Initialize Protect client - * const protectClient = await protect({ schemas: [userSchema] }) - * - * // Encrypt a query value - * const encryptedResult = await protectClient.encryptQuery( - * "person@example.com", - * { column: userSchema.email, table: userSchema, queryType: 'equality' } - * ) - * - * // Handle encryption result - * if (encryptedResult.failure) { - * throw new Error(`Encryption failed: ${encryptedResult.failure.message}`); - * } - * - * console.log("Encrypted query:", encryptedResult.data); - * ``` - * - * @example - * The queryType can be auto-inferred from the column's configured indexes: - * - * ```typescript - * // When queryType is omitted, it will be inferred from the column's indexes - * const encryptedResult = await protectClient.encryptQuery( - * "person@example.com", - * { column: userSchema.email, table: userSchema } - * ) - * ``` - * - * @see {@link EncryptQueryOperation} - * - * **JSONB columns (searchableJson):** - * When `queryType` is omitted on a `searchableJson()` column, the query operation is inferred: - * - String plaintext → `steVecSelector` (JSONPath queries like `'$.user.email'`) - * - Object/Array plaintext → `steVecTerm` (containment queries like `{ role: 'admin' }`) - */ - encryptQuery( - plaintext: JsPlaintext | null, - opts: EncryptQueryOptions, - ): EncryptQueryOperation - - /** - * Encrypt multiple values for use in queries (batch operation). - * @param terms - Array of query terms to encrypt - */ - encryptQuery(terms: readonly ScalarQueryTerm[]): BatchEncryptQueryOperation - - encryptQuery( - plaintextOrTerms: JsPlaintext | null | readonly ScalarQueryTerm[], - opts?: EncryptQueryOptions, - ): EncryptQueryOperation | BatchEncryptQueryOperation { - // Discriminate between ScalarQueryTerm[] and JsPlaintext (which can also be an array) - // using a type guard function - if (isScalarQueryTermArray(plaintextOrTerms)) { - return new BatchEncryptQueryOperation(this.client, plaintextOrTerms) - } - - // Handle empty arrays: if opts provided, treat as single value; otherwise batch mode - // This maintains backward compatibility for encryptQuery([]) while allowing - // encryptQuery([], opts) to encrypt an empty array as a single value - if ( - Array.isArray(plaintextOrTerms) && - plaintextOrTerms.length === 0 && - !opts - ) { - return new BatchEncryptQueryOperation( - this.client, - [] as readonly ScalarQueryTerm[], - ) - } - - return new EncryptQueryOperation( - this.client, - plaintextOrTerms as JsPlaintext | null, - opts!, - ) - } - - /** - * Decryption - returns a promise which resolves to a decrypted value. - * - * @param encryptedData - The encrypted data to be decrypted. - * @returns A DecryptOperation that can be awaited or chained with additional methods. - * - * @example - * The following example demonstrates how to decrypt a value that was previously encrypted using {@link encrypt} client. - * It includes encrypting a value first, then decrypting it, and handling the result. - * - * ```typescript - * const encryptedData = await eqlClient.encrypt( - * "person@example.com", - * { column: "email", table: "users" } - * ) - * const decryptResult = await eqlClient.decrypt(encryptedData) - * if (decryptResult.failure) { - * throw new Error(`Decryption failed: ${decryptResult.failure.message}`); - * } - * console.log("Decrypted data:", decryptResult.data); - * ``` - * - * @example - * Provide a lock context when decrypting: - * ```typescript - * await eqlClient.decrypt(encryptedData) - * .withLockContext(lockContext) - * ``` - * - * @see {@link LockContext} - * @see {@link DecryptOperation} - */ - decrypt(encryptedData: Encrypted): DecryptOperation { - return new DecryptOperation(this.client, encryptedData) - } - - /** - * Encrypt a model based on its encryptConfig. - * - * @example - * ```typescript - * type User = { - * id: string; - * email: string; // encrypted - * } - * - * // Define the schema for the users table - * const usersSchema = csTable('users', { - * email: csColumn('email').freeTextSearch().equality().orderAndRange(), - * }) - * - * // Initialize the Protect client - * const protectClient = await protect({ schemas: [usersSchema] }) - * - * // Encrypt a user model - * const encryptedModel = await protectClient.encryptModel( - * { id: 'user_123', email: 'person@example.com' }, - * usersSchema, - * ) - * ``` - */ - encryptModel>( - input: Decrypted, - table: ProtectTable, - ): EncryptModelOperation { - return new EncryptModelOperation(this.client, input, table) - } - - /** - * Decrypt a model with encrypted values - * Usage: - * await eqlClient.decryptModel(encryptedModel) - * await eqlClient.decryptModel(encryptedModel).withLockContext(lockContext) - */ - decryptModel>( - input: T, - ): DecryptModelOperation { - return new DecryptModelOperation(this.client, input) - } - - /** - * Bulk encrypt models with decrypted values - * Usage: - * await eqlClient.bulkEncryptModels(decryptedModels, table) - * await eqlClient.bulkEncryptModels(decryptedModels, table).withLockContext(lockContext) - */ - bulkEncryptModels>( - input: Array>, - table: ProtectTable, - ): BulkEncryptModelsOperation { - return new BulkEncryptModelsOperation(this.client, input, table) - } - - /** - * Bulk decrypt models with encrypted values - * Usage: - * await eqlClient.bulkDecryptModels(encryptedModels) - * await eqlClient.bulkDecryptModels(encryptedModels).withLockContext(lockContext) - */ - bulkDecryptModels>( - input: Array, - ): BulkDecryptModelsOperation { - return new BulkDecryptModelsOperation(this.client, input) - } - - /** - * Bulk encryption - returns a thenable object. - * Usage: - * await eqlClient.bulkEncrypt(plaintexts, { column, table }) - * await eqlClient.bulkEncrypt(plaintexts, { column, table }).withLockContext(lockContext) - */ - bulkEncrypt( - plaintexts: BulkEncryptPayload, - opts: EncryptOptions, - ): BulkEncryptOperation { - return new BulkEncryptOperation(this.client, plaintexts, opts) - } - - /** - * Bulk decryption - returns a thenable object. - * Usage: - * await eqlClient.bulkDecrypt(encryptedPayloads) - * await eqlClient.bulkDecrypt(encryptedPayloads).withLockContext(lockContext) - */ - bulkDecrypt(encryptedPayloads: BulkDecryptPayload): BulkDecryptOperation { - return new BulkDecryptOperation(this.client, encryptedPayloads) - } - - /** - * Create search terms to use in a query searching encrypted data - * - * @deprecated Use `encryptQuery(terms)` instead. - * - * Migration example: - * ```typescript - * // Before (deprecated) - * const result = await client.createSearchTerms([ - * { value: 'test', column: users.email, table: users } - * ]) - * - * // After - * const result = await client.encryptQuery([ - * { value: 'test', column: users.email, table: users, queryType: 'equality' } - * ]) - * ``` - * - * Usage: - * await eqlClient.createSearchTerms(searchTerms) - * await eqlClient.createSearchTerms(searchTerms).withLockContext(lockContext) - */ - createSearchTerms(terms: SearchTerm[]): SearchTermsOperation { - return new SearchTermsOperation(this.client, terms) - } -} diff --git a/packages/protect/src/ffi/model-helpers.ts b/packages/protect/src/ffi/model-helpers.ts deleted file mode 100644 index ff6823426..000000000 --- a/packages/protect/src/ffi/model-helpers.ts +++ /dev/null @@ -1,952 +0,0 @@ -import { - type Encrypted as CipherStashEncrypted, - type DecryptBulkOptions, - decryptBulk, - encryptBulk, - type JsPlaintext, -} from '@cipherstash/protect-ffi' -import type { ProtectTable, ProtectTableColumn } from '@cipherstash/schema' -import { isEncryptedPayload } from '../helpers' -import type { GetLockContextResponse } from '../identify' -import type { Client, Decrypted, Encrypted } from '../types' -import type { AuditData } from './operations/base-operation' - -/** - * Helper function to extract encrypted fields from a model - */ -export function extractEncryptedFields>( - model: T, -): Record { - const result: Record = {} - - for (const [key, value] of Object.entries(model)) { - if (isEncryptedPayload(value)) { - result[key] = value - } - } - - return result -} - -/** - * Helper function to extract non-encrypted fields from a model - */ -export function extractOtherFields>( - model: T, -): Record { - const result: Record = {} - - for (const [key, value] of Object.entries(model)) { - if (!isEncryptedPayload(value)) { - result[key] = value - } - } - - return result -} - -/** - * Helper function to merge encrypted and non-encrypted fields into a model - */ -export function mergeFields( - otherFields: Record, - encryptedFields: Record, -): T { - return { ...otherFields, ...encryptedFields } as T -} - -/** - * Base interface for bulk operation payloads - */ -interface BulkOperationPayload { - id: string - [key: string]: unknown -} - -/** - * Interface for bulk operation key mapping - */ -interface BulkOperationKeyMap { - modelIndex: number - fieldKey: string -} - -/** - * Helper function to handle single model bulk operations with mapping - */ -async function handleSingleModelBulkOperation< - T extends BulkOperationPayload, - R, ->( - items: T[], - operation: (items: T[]) => Promise, - keyMap: Record, -): Promise> { - if (items.length === 0) { - return {} - } - - const results = await operation(items) - const mappedResults: Record = {} - - results.forEach((result, index) => { - const originalKey = keyMap[index.toString()] - mappedResults[originalKey] = result - }) - - return mappedResults -} - -/** - * Helper function to handle multiple model bulk operations with mapping - */ -async function handleMultiModelBulkOperation( - items: T[], - operation: (items: T[]) => Promise, - keyMap: Record, -): Promise> { - if (items.length === 0) { - return {} - } - - const results = await operation(items) - const mappedResults: Record = {} - - results.forEach((result, index) => { - const key = index.toString() - const { modelIndex, fieldKey } = keyMap[key] - mappedResults[`${modelIndex}-${fieldKey}`] = result - }) - - return mappedResults -} - -/** - * Helper function to prepare fields for decryption - */ -function prepareFieldsForDecryption>( - model: T, -): { - otherFields: Record - operationFields: Record - keyMap: Record - nullFields: Record -} { - const otherFields = { ...model } as Record - const operationFields: Record = {} - const nullFields: Record = {} - const keyMap: Record = {} - let index = 0 - - const processNestedFields = (obj: Record, prefix = '') => { - for (const [key, value] of Object.entries(obj)) { - const fullKey = prefix ? `${prefix}.${key}` : key - - if (value === null || value === undefined) { - nullFields[fullKey] = value - continue - } - - if (typeof value === 'object' && !isEncryptedPayload(value)) { - // Recursively process nested objects - processNestedFields(value as Record, fullKey) - } else if (isEncryptedPayload(value)) { - // This is an encrypted field - const id = index.toString() - keyMap[id] = fullKey - operationFields[fullKey] = value - index++ - - // Remove from otherFields - const parts = fullKey.split('.') - let current = otherFields - for (let i = 0; i < parts.length - 1; i++) { - current = current[parts[i]] as Record - } - delete current[parts[parts.length - 1]] - } - } - } - - processNestedFields(model) - return { otherFields, operationFields, keyMap, nullFields } -} - -/** - * Helper function to prepare fields for encryption - */ -function prepareFieldsForEncryption>( - model: T, - table: ProtectTable, -): { - otherFields: Record - operationFields: Record - keyMap: Record - nullFields: Record -} { - const otherFields = { ...model } as Record - const operationFields: Record = {} - const nullFields: Record = {} - const keyMap: Record = {} - let index = 0 - - const processNestedFields = ( - obj: Record, - prefix = '', - columnPaths: string[] = [], - ) => { - for (const [key, value] of Object.entries(obj)) { - const fullKey = prefix ? `${prefix}.${key}` : key - - if (value === null || value === undefined) { - nullFields[fullKey] = value - continue - } - - if ( - typeof value === 'object' && - !isEncryptedPayload(value) && - !columnPaths.includes(fullKey) - ) { - // Only process nested objects if they're in the schema - if (columnPaths.some((path) => path.startsWith(fullKey))) { - processNestedFields( - value as Record, - fullKey, - columnPaths, - ) - } - } else if (columnPaths.includes(fullKey)) { - // Only process fields that are explicitly defined in the schema - const id = index.toString() - keyMap[id] = fullKey - operationFields[fullKey] = value - index++ - - // Remove from otherFields - const parts = fullKey.split('.') - let current = otherFields - for (let i = 0; i < parts.length - 1; i++) { - current = current[parts[i]] as Record - } - delete current[parts[parts.length - 1]] - } - } - } - - // Get all column paths from the table schema - const columnPaths = Object.keys(table.build().columns) - processNestedFields(model, '', columnPaths) - - return { otherFields, operationFields, keyMap, nullFields } -} - -/** - * Helper function to convert a model with encrypted fields to a decrypted model - */ -export async function decryptModelFields>( - model: T, - client: Client, - auditData?: AuditData, -): Promise> { - if (!client) { - throw new Error('Client not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareFieldsForDecryption(model) - - const bulkDecryptPayload = Object.entries(operationFields).map( - ([key, value]) => ({ - id: key, - ciphertext: value as CipherStashEncrypted, - }), - ) - - const decryptedFields = await handleSingleModelBulkOperation( - bulkDecryptPayload, - (items) => - decryptBulk(client, { - ciphertexts: items, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - // Reconstruct the object with proper nesting - const result: Record = { ...otherFields } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the decrypted fields - for (const [key, value] of Object.entries(decryptedFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as Decrypted -} - -/** - * Helper function to convert a decrypted model to a model with encrypted fields - */ -export async function encryptModelFields>( - model: Decrypted, - table: ProtectTable, - client: Client, - auditData?: AuditData, -): Promise { - if (!client) { - throw new Error('Client not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareFieldsForEncryption(model, table) - - const bulkEncryptPayload = Object.entries(operationFields).map( - ([key, value]) => ({ - id: key, - plaintext: value as string, - table: table.tableName, - column: key, - }), - ) - - const encryptedData = await handleSingleModelBulkOperation( - bulkEncryptPayload, - (items) => - encryptBulk(client, { - plaintexts: items, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - // Reconstruct the object with proper nesting - const result: Record = { ...otherFields } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the encrypted fields - for (const [key, value] of Object.entries(encryptedData)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as T -} - -/** - * Helper function to convert a model with encrypted fields to a decrypted model with lock context - */ -export async function decryptModelFieldsWithLockContext< - T extends Record, ->( - model: T, - client: Client, - lockContext: GetLockContextResponse, - auditData?: AuditData, -): Promise> { - if (!client) { - throw new Error('Client not initialized') - } - - if (!lockContext) { - throw new Error('Lock context is not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareFieldsForDecryption(model) - - const bulkDecryptPayload = Object.entries(operationFields).map( - ([key, value]) => ({ - id: key, - ciphertext: value as CipherStashEncrypted, - lockContext: lockContext.context, - }), - ) - - const decryptedFields = await handleSingleModelBulkOperation( - bulkDecryptPayload, - (items) => - decryptBulk(client, { - ciphertexts: items, - serviceToken: lockContext.ctsToken, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - // Reconstruct the object with proper nesting - const result: Record = { ...otherFields } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the decrypted fields - for (const [key, value] of Object.entries(decryptedFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as Decrypted -} - -/** - * Helper function to convert a decrypted model to a model with encrypted fields with lock context - */ -export async function encryptModelFieldsWithLockContext< - T extends Record, ->( - model: Decrypted, - table: ProtectTable, - client: Client, - lockContext: GetLockContextResponse, - auditData?: AuditData, -): Promise { - if (!client) { - throw new Error('Client not initialized') - } - - if (!lockContext) { - throw new Error('Lock context is not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareFieldsForEncryption(model, table) - - const bulkEncryptPayload = Object.entries(operationFields).map( - ([key, value]) => ({ - id: key, - plaintext: value as string, - table: table.tableName, - column: key, - lockContext: lockContext.context, - }), - ) - - const encryptedData = await handleSingleModelBulkOperation( - bulkEncryptPayload, - (items) => - encryptBulk(client, { - plaintexts: items, - serviceToken: lockContext.ctsToken, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - // Reconstruct the object with proper nesting - const result: Record = { ...otherFields } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the encrypted fields - for (const [key, value] of Object.entries(encryptedData)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as T -} - -/** - * Helper function to prepare multiple models for bulk operation - */ -function prepareBulkModelsForOperation>( - models: T[], - table?: ProtectTable, -): { - otherFields: Record[] - operationFields: Record[] - keyMap: Record - nullFields: Record[] -} { - const otherFields: Record[] = [] - const operationFields: Record[] = [] - const nullFields: Record[] = [] - const keyMap: Record = {} - let index = 0 - - for (let modelIndex = 0; modelIndex < models.length; modelIndex++) { - const model = models[modelIndex] - const modelOtherFields = { ...model } as Record - const modelOperationFields: Record = {} - const modelNullFields: Record = {} - - const processNestedFields = ( - obj: Record, - prefix = '', - columnPaths: string[] = [], - ) => { - for (const [key, value] of Object.entries(obj)) { - const fullKey = prefix ? `${prefix}.${key}` : key - - if (value === null || value === undefined) { - modelNullFields[fullKey] = value - continue - } - - if ( - typeof value === 'object' && - !isEncryptedPayload(value) && - !columnPaths.includes(fullKey) - ) { - // Only process nested objects if they're in the schema - if (columnPaths.some((path) => path.startsWith(fullKey))) { - processNestedFields( - value as Record, - fullKey, - columnPaths, - ) - } - } else if (columnPaths.includes(fullKey)) { - // Only process fields that are explicitly defined in the schema - const id = index.toString() - keyMap[id] = { modelIndex, fieldKey: fullKey } - modelOperationFields[fullKey] = value - index++ - - // Remove from otherFields - const parts = fullKey.split('.') - let current = modelOtherFields - for (let i = 0; i < parts.length - 1; i++) { - current = current[parts[i]] as Record - } - delete current[parts[parts.length - 1]] - } - } - } - - if (table) { - // Get all column paths from the table schema - const columnPaths = Object.keys(table.build().columns) - processNestedFields(model, '', columnPaths) - } else { - // For decryption, process all encrypted fields - const processEncryptedFields = ( - obj: Record, - prefix = '', - columnPaths: string[] = [], - ) => { - for (const [key, value] of Object.entries(obj)) { - const fullKey = prefix ? `${prefix}.${key}` : key - - if (value === null || value === undefined) { - modelNullFields[fullKey] = value - continue - } - - if ( - typeof value === 'object' && - !isEncryptedPayload(value) && - !columnPaths.includes(fullKey) - ) { - // Recursively process nested objects - processEncryptedFields( - value as Record, - fullKey, - columnPaths, - ) - } else if (isEncryptedPayload(value)) { - // This is an encrypted field - const id = index.toString() - keyMap[id] = { modelIndex, fieldKey: fullKey } - modelOperationFields[fullKey] = value - index++ - - // Remove from otherFields - const parts = fullKey.split('.') - let current = modelOtherFields - for (let i = 0; i < parts.length - 1; i++) { - current = current[parts[i]] as Record - } - delete current[parts[parts.length - 1]] - } - } - } - processEncryptedFields(model) - } - - otherFields.push(modelOtherFields) - operationFields.push(modelOperationFields) - nullFields.push(modelNullFields) - } - - return { otherFields, operationFields, keyMap, nullFields } -} - -/** - * Helper function to convert multiple decrypted models to models with encrypted fields - */ -export async function bulkEncryptModels>( - models: Decrypted[], - table: ProtectTable, - client: Client, - auditData?: AuditData, -): Promise { - if (!client) { - throw new Error('Client not initialized') - } - - if (!models || models.length === 0) { - return [] - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareBulkModelsForOperation(models, table) - - const bulkEncryptPayload = operationFields.flatMap((fields, modelIndex) => - Object.entries(fields).map(([key, value]) => ({ - id: `${modelIndex}-${key}`, - plaintext: value as string, - table: table.tableName, - column: key, - })), - ) - - const encryptedData = await handleMultiModelBulkOperation( - bulkEncryptPayload, - (items) => - encryptBulk(client, { - plaintexts: items, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - return models.map((_, modelIndex) => { - const result: Record = { ...otherFields[modelIndex] } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields[modelIndex])) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the encrypted fields - const modelData = Object.fromEntries( - Object.entries(encryptedData) - .filter(([key]) => { - const [idx] = key.split('-') - return Number.parseInt(idx) === modelIndex - }) - .map(([key, value]) => { - const [_, fieldKey] = key.split('-') - return [fieldKey, value] - }), - ) - - for (const [key, value] of Object.entries(modelData)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as T - }) -} - -/** - * Helper function to convert multiple models with encrypted fields to decrypted models - */ -export async function bulkDecryptModels>( - models: T[], - client: Client, - auditData?: AuditData, -): Promise[]> { - if (!client) { - throw new Error('Client not initialized') - } - - if (!models || models.length === 0) { - return [] - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareBulkModelsForOperation(models) - - const bulkDecryptPayload = operationFields.flatMap((fields, modelIndex) => - Object.entries(fields).map(([key, value]) => ({ - id: `${modelIndex}-${key}`, - ciphertext: value as CipherStashEncrypted, - })), - ) - - const decryptedFields = await handleMultiModelBulkOperation( - bulkDecryptPayload, - (items) => - decryptBulk(client, { - ciphertexts: items, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Helper function to set a nested value - const setNestedValue = ( - obj: Record, - path: string[], - value: unknown, - ) => { - let current = obj - for (let i = 0; i < path.length - 1; i++) { - const part = path[i] - if (!(part in current)) { - current[part] = {} - } - current = current[part] as Record - } - current[path[path.length - 1]] = value - } - - return models.map((_, modelIndex) => { - const result: Record = { ...otherFields[modelIndex] } - - // First, reconstruct the null/undefined fields - for (const [key, value] of Object.entries(nullFields[modelIndex])) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - // Then, reconstruct the decrypted fields - const modelData = Object.fromEntries( - Object.entries(decryptedFields) - .filter(([key]) => { - const [idx] = key.split('-') - return Number.parseInt(idx) === modelIndex - }) - .map(([key, value]) => { - const [_, fieldKey] = key.split('-') - return [fieldKey, value] - }), - ) - - for (const [key, value] of Object.entries(modelData)) { - const parts = key.split('.') - setNestedValue(result, parts, value) - } - - return result as Decrypted - }) -} - -/** - * Helper function to convert multiple models with encrypted fields to decrypted models with lock context - */ -export async function bulkDecryptModelsWithLockContext< - T extends Record, ->( - models: T[], - client: Client, - lockContext: GetLockContextResponse, - auditData?: AuditData, -): Promise[]> { - if (!client) { - throw new Error('Client not initialized') - } - - if (!lockContext) { - throw new Error('Lock context is not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareBulkModelsForOperation(models) - - const bulkDecryptPayload = operationFields.flatMap((fields, modelIndex) => - Object.entries(fields).map(([key, value]) => ({ - id: `${modelIndex}-${key}`, - ciphertext: value as CipherStashEncrypted, - lockContext: lockContext.context, - })), - ) - - const decryptedFields = await handleMultiModelBulkOperation( - bulkDecryptPayload, - (items) => - decryptBulk(client, { - ciphertexts: items, - serviceToken: lockContext.ctsToken, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Reconstruct models - return models.map((_, modelIndex) => ({ - ...otherFields[modelIndex], - ...nullFields[modelIndex], - ...Object.fromEntries( - Object.entries(decryptedFields) - .filter(([key]) => { - const [idx] = key.split('-') - return Number.parseInt(idx) === modelIndex - }) - .map(([key, value]) => { - const [_, fieldKey] = key.split('-') - return [fieldKey, value] - }), - ), - })) as Decrypted[] -} - -/** - * Helper function to convert multiple decrypted models to models with encrypted fields with lock context - */ -export async function bulkEncryptModelsWithLockContext< - T extends Record, ->( - models: Decrypted[], - table: ProtectTable, - client: Client, - lockContext: GetLockContextResponse, - auditData?: AuditData, -): Promise { - if (!client) { - throw new Error('Client not initialized') - } - - if (!lockContext) { - throw new Error('Lock context is not initialized') - } - - const { otherFields, operationFields, keyMap, nullFields } = - prepareBulkModelsForOperation(models, table) - - const bulkEncryptPayload = operationFields.flatMap((fields, modelIndex) => - Object.entries(fields).map(([key, value]) => ({ - id: `${modelIndex}-${key}`, - plaintext: value as string, - table: table.tableName, - column: key, - lockContext: lockContext.context, - })), - ) - - const encryptedData = await handleMultiModelBulkOperation( - bulkEncryptPayload, - (items) => - encryptBulk(client, { - plaintexts: items, - serviceToken: lockContext.ctsToken, - unverifiedContext: auditData?.metadata, - }), - keyMap, - ) - - // Reconstruct models - return models.map((_, modelIndex) => ({ - ...otherFields[modelIndex], - ...nullFields[modelIndex], - ...Object.fromEntries( - Object.entries(encryptedData) - .filter(([key]) => { - const [idx] = key.split('-') - return Number.parseInt(idx) === modelIndex - }) - .map(([key, value]) => { - const [_, fieldKey] = key.split('-') - return [fieldKey, value] - }), - ), - })) as T[] -} diff --git a/packages/protect/src/ffi/operations/base-operation.ts b/packages/protect/src/ffi/operations/base-operation.ts deleted file mode 100644 index 8e4dd36b8..000000000 --- a/packages/protect/src/ffi/operations/base-operation.ts +++ /dev/null @@ -1,50 +0,0 @@ -import type { Result } from '@byteslice/result' -import type { ProtectError } from '../..' - -export type AuditConfig = { - metadata?: Record -} - -export type AuditData = { - metadata?: Record -} - -export abstract class ProtectOperation { - protected auditMetadata?: Record - - /** - * Attach audit metadata to this operation. Can be chained. - * @param config Configuration for ZeroKMS audit logging - * @param config.metadata Arbitrary JSON object for appending metadata to the audit log - */ - audit(config: AuditConfig): this { - this.auditMetadata = config.metadata - return this - } - - /** - * Get the audit data for this operation. - */ - public getAuditData(): AuditData { - return { - metadata: this.auditMetadata, - } - } - - /** - * Execute the operation and return a Result - */ - abstract execute(): Promise> - - /** - * Make the operation thenable - */ - public then, TResult2 = never>( - onfulfilled?: - | ((value: Result) => TResult1 | PromiseLike) - | null, - onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null, - ): Promise { - return this.execute().then(onfulfilled, onrejected) - } -} diff --git a/packages/protect/src/ffi/operations/batch-encrypt-query.ts b/packages/protect/src/ffi/operations/batch-encrypt-query.ts deleted file mode 100644 index a8ebca064..000000000 --- a/packages/protect/src/ffi/operations/batch-encrypt-query.ts +++ /dev/null @@ -1,235 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { - Encrypted as CipherStashEncrypted, - EncryptedQuery as CipherStashEncryptedQuery, -} from '@cipherstash/protect-ffi' -import { - encryptQueryBulk as ffiEncryptQueryBulk, - type JsPlaintext, - type QueryPayload, -} from '@cipherstash/protect-ffi' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import { formatEncryptedResult } from '../../helpers' -import type { Context, LockContext } from '../../identify' -import type { Client, EncryptedQueryResult, ScalarQueryTerm } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { resolveIndexType } from '../helpers/infer-index-type' -import { - assertValidNumericValue, - assertValueIndexCompatibility, -} from '../helpers/validation' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -/** - * Separates null/undefined values from non-null terms in the input array. - * Returns a set of indices where values are null/undefined and an array of non-null terms with their original indices. - */ -function filterNullTerms(terms: readonly ScalarQueryTerm[]): { - nullIndices: Set - nonNullTerms: { term: ScalarQueryTerm; originalIndex: number }[] -} { - const nullIndices = new Set() - const nonNullTerms: { term: ScalarQueryTerm; originalIndex: number }[] = [] - - terms.forEach((term, index) => { - if (term.value === null || term.value === undefined) { - nullIndices.add(index) - } else { - nonNullTerms.push({ term, originalIndex: index }) - } - }) - - return { nullIndices, nonNullTerms } -} - -/** - * Validates and transforms a single term into a QueryPayload. - * Throws an error if the value is NaN or Infinity. - * Optionally includes lockContext if provided. - */ -function buildQueryPayload( - term: ScalarQueryTerm, - lockContext?: Context, -): QueryPayload { - assertValidNumericValue(term.value) - - const { indexType, queryOp } = resolveIndexType( - term.column, - term.queryType, - term.value, - ) - - // Validate value/index compatibility - assertValueIndexCompatibility(term.value, indexType, term.column.getName()) - - const payload: QueryPayload = { - plaintext: term.value as JsPlaintext, - column: term.column.getName(), - table: term.table.tableName, - indexType, - queryOp, - } - - if (lockContext != null) { - payload.lockContext = lockContext - } - - return payload -} - -/** - * Reconstructs the results array with nulls in their original positions. - * Non-null encrypted values are placed at their original indices. - * Applies formatting based on term.returnType. - */ -function assembleResults( - totalLength: number, - encryptedValues: (CipherStashEncrypted | CipherStashEncryptedQuery)[], - nonNullTerms: { term: ScalarQueryTerm; originalIndex: number }[], -): EncryptedQueryResult[] { - const results: EncryptedQueryResult[] = new Array(totalLength).fill(null) - - // Fill in encrypted values at their original positions, applying formatting - nonNullTerms.forEach(({ term, originalIndex }, i) => { - const encrypted = encryptedValues[i] - - results[originalIndex] = formatEncryptedResult(encrypted, term.returnType) - }) - - return results -} - -/** - * @internal Use {@link ProtectClient.encryptQuery} with array input instead. - */ -export class BatchEncryptQueryOperation extends ProtectOperation< - EncryptedQueryResult[] -> { - constructor( - private client: Client, - private terms: readonly ScalarQueryTerm[], - ) { - super() - } - - public withLockContext( - lockContext: LockContext, - ): BatchEncryptQueryOperationWithLockContext { - return new BatchEncryptQueryOperationWithLockContext( - this.client, - this.terms, - lockContext, - this.auditMetadata, - ) - } - - public async execute(): Promise< - Result - > { - logger.debug('Encrypting query terms', { count: this.terms.length }) - - if (this.terms.length === 0) { - return { data: [] } - } - - const { nullIndices, nonNullTerms } = filterNullTerms(this.terms) - - if (nonNullTerms.length === 0) { - return { data: this.terms.map(() => null) } - } - - return await withResult( - async () => { - if (!this.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const queries: QueryPayload[] = nonNullTerms.map(({ term }) => - buildQueryPayload(term), - ) - - const encrypted = await ffiEncryptQueryBulk(this.client, { - queries, - unverifiedContext: metadata, - }) - - return assembleResults(this.terms.length, encrypted, nonNullTerms) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} - -/** - * @internal Use {@link ProtectClient.encryptQuery} with array input and `.withLockContext()` instead. - */ -export class BatchEncryptQueryOperationWithLockContext extends ProtectOperation< - EncryptedQueryResult[] -> { - constructor( - private client: Client, - private terms: readonly ScalarQueryTerm[], - private lockContext: LockContext, - auditMetadata?: Record, - ) { - super() - this.auditMetadata = auditMetadata - } - - public async execute(): Promise< - Result - > { - logger.debug('Encrypting query terms with lock context', { - count: this.terms.length, - }) - - if (this.terms.length === 0) { - return { data: [] } - } - - // Check for all-null terms BEFORE fetching lockContext to avoid unnecessary network call - const { nullIndices, nonNullTerms } = filterNullTerms(this.terms) - - if (nonNullTerms.length === 0) { - return { data: this.terms.map(() => null) } - } - - const lockContextResult = await this.lockContext.getLockContext() - if (lockContextResult.failure) { - return { failure: lockContextResult.failure } - } - - const { ctsToken, context } = lockContextResult.data - - return await withResult( - async () => { - if (!this.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const queries: QueryPayload[] = nonNullTerms.map(({ term }) => - buildQueryPayload(term, context), - ) - - const encrypted = await ffiEncryptQueryBulk(this.client, { - queries, - serviceToken: ctsToken, - unverifiedContext: metadata, - }) - - return assembleResults(this.terms.length, encrypted, nonNullTerms) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/bulk-decrypt-models.ts b/packages/protect/src/ffi/operations/bulk-decrypt-models.ts deleted file mode 100644 index dda93bff7..000000000 --- a/packages/protect/src/ffi/operations/bulk-decrypt-models.ts +++ /dev/null @@ -1,112 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Decrypted } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { - bulkDecryptModels, - bulkDecryptModelsWithLockContext, -} from '../model-helpers' -import { ProtectOperation } from './base-operation' - -export class BulkDecryptModelsOperation< - T extends Record, -> extends ProtectOperation[]> { - private client: Client - private models: T[] - - constructor(client: Client, models: T[]) { - super() - this.client = client - this.models = models - } - - public withLockContext( - lockContext: LockContext, - ): BulkDecryptModelsOperationWithLockContext { - return new BulkDecryptModelsOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise[], ProtectError>> { - logger.debug('Bulk decrypting models WITHOUT a lock context') - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - const auditData = this.getAuditData() - - return await bulkDecryptModels(this.models, this.client, auditData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - models: T[] - } { - return { - client: this.client, - models: this.models, - } - } -} - -export class BulkDecryptModelsOperationWithLockContext< - T extends Record, -> extends ProtectOperation[]> { - private operation: BulkDecryptModelsOperation - private lockContext: LockContext - - constructor( - operation: BulkDecryptModelsOperation, - lockContext: LockContext, - ) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise[], ProtectError>> { - return await withResult( - async () => { - const { client, models } = this.operation.getOperation() - - logger.debug('Bulk decrypting models WITH a lock context') - - if (!client) { - throw noClientError() - } - - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const auditData = this.getAuditData() - - return await bulkDecryptModelsWithLockContext( - models, - client, - context.data, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/bulk-decrypt.ts b/packages/protect/src/ffi/operations/bulk-decrypt.ts deleted file mode 100644 index e34c85dcf..000000000 --- a/packages/protect/src/ffi/operations/bulk-decrypt.ts +++ /dev/null @@ -1,178 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { - type Encrypted as CipherStashEncrypted, - type DecryptResult, - decryptBulkFallible, -} from '@cipherstash/protect-ffi' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { Context, LockContext } from '../../identify' -import type { BulkDecryptedData, BulkDecryptPayload, Client } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -// Helper functions for better composability -const createDecryptPayloads = ( - encryptedPayloads: BulkDecryptPayload, - lockContext?: Context, -) => { - return encryptedPayloads - .map((item, index) => ({ ...item, originalIndex: index })) - .filter(({ data }) => data !== null) - .map(({ id, data, originalIndex }) => ({ - id, - ciphertext: data as CipherStashEncrypted, - originalIndex, - ...(lockContext && { lockContext }), - })) -} - -const createNullResult = ( - encryptedPayloads: BulkDecryptPayload, -): BulkDecryptedData => { - return encryptedPayloads.map(({ id }) => ({ - id, - data: null, - })) -} - -const mapDecryptedDataToResult = ( - encryptedPayloads: BulkDecryptPayload, - decryptedData: DecryptResult[], -): BulkDecryptedData => { - const result: BulkDecryptedData = new Array(encryptedPayloads.length) - let decryptedIndex = 0 - - for (let i = 0; i < encryptedPayloads.length; i++) { - if (encryptedPayloads[i].data === null) { - result[i] = { id: encryptedPayloads[i].id, data: null } - } else { - const decryptResult = decryptedData[decryptedIndex] - if ('error' in decryptResult) { - result[i] = { - id: encryptedPayloads[i].id, - error: decryptResult.error, - } - } else { - result[i] = { - id: encryptedPayloads[i].id, - data: decryptResult.data, - } - } - decryptedIndex++ - } - } - - return result -} - -export class BulkDecryptOperation extends ProtectOperation { - private client: Client - private encryptedPayloads: BulkDecryptPayload - - constructor(client: Client, encryptedPayloads: BulkDecryptPayload) { - super() - this.client = client - this.encryptedPayloads = encryptedPayloads - } - - public withLockContext( - lockContext: LockContext, - ): BulkDecryptOperationWithLockContext { - return new BulkDecryptOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Bulk decrypting data WITHOUT a lock context') - return await withResult( - async () => { - if (!this.client) throw noClientError() - if (!this.encryptedPayloads || this.encryptedPayloads.length === 0) - return [] - - const nonNullPayloads = createDecryptPayloads(this.encryptedPayloads) - - if (nonNullPayloads.length === 0) { - return createNullResult(this.encryptedPayloads) - } - - const { metadata } = this.getAuditData() - - const decryptedData = await decryptBulkFallible(this.client, { - ciphertexts: nonNullPayloads, - unverifiedContext: metadata, - }) - - return mapDecryptedDataToResult(this.encryptedPayloads, decryptedData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - encryptedPayloads: BulkDecryptPayload - } { - return { - client: this.client, - encryptedPayloads: this.encryptedPayloads, - } - } -} - -export class BulkDecryptOperationWithLockContext extends ProtectOperation { - private operation: BulkDecryptOperation - private lockContext: LockContext - - constructor(operation: BulkDecryptOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, encryptedPayloads } = this.operation.getOperation() - logger.debug('Bulk decrypting data WITH a lock context') - - if (!client) throw noClientError() - if (!encryptedPayloads || encryptedPayloads.length === 0) return [] - - const context = await this.lockContext.getLockContext() - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const nonNullPayloads = createDecryptPayloads( - encryptedPayloads, - context.data.context, - ) - - if (nonNullPayloads.length === 0) { - return createNullResult(encryptedPayloads) - } - - const { metadata } = this.getAuditData() - - const decryptedData = await decryptBulkFallible(client, { - ciphertexts: nonNullPayloads, - serviceToken: context.data.ctsToken, - unverifiedContext: metadata, - }) - - return mapDecryptedDataToResult(encryptedPayloads, decryptedData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/bulk-encrypt-models.ts b/packages/protect/src/ffi/operations/bulk-encrypt-models.ts deleted file mode 100644 index 2bb69446b..000000000 --- a/packages/protect/src/ffi/operations/bulk-encrypt-models.ts +++ /dev/null @@ -1,131 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { ProtectTable, ProtectTableColumn } from '@cipherstash/schema' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Decrypted } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { - bulkEncryptModels, - bulkEncryptModelsWithLockContext, -} from '../model-helpers' -import { ProtectOperation } from './base-operation' - -export class BulkEncryptModelsOperation< - T extends Record, -> extends ProtectOperation { - private client: Client - private models: Decrypted[] - private table: ProtectTable - - constructor( - client: Client, - models: Decrypted[], - table: ProtectTable, - ) { - super() - this.client = client - this.models = models - this.table = table - } - - public withLockContext( - lockContext: LockContext, - ): BulkEncryptModelsOperationWithLockContext { - return new BulkEncryptModelsOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Bulk encrypting models WITHOUT a lock context', { - table: this.table.tableName, - }) - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - const auditData = this.getAuditData() - - return await bulkEncryptModels( - this.models, - this.table, - this.client, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - models: Decrypted[] - table: ProtectTable - } { - return { - client: this.client, - models: this.models, - table: this.table, - } - } -} - -export class BulkEncryptModelsOperationWithLockContext< - T extends Record, -> extends ProtectOperation { - private operation: BulkEncryptModelsOperation - private lockContext: LockContext - - constructor( - operation: BulkEncryptModelsOperation, - lockContext: LockContext, - ) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, models, table } = this.operation.getOperation() - - logger.debug('Bulk encrypting models WITH a lock context', { - table: table.tableName, - }) - - if (!client) { - throw noClientError() - } - - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const auditData = this.getAuditData() - - return await bulkEncryptModelsWithLockContext( - models, - table, - client, - context.data, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/bulk-encrypt.ts b/packages/protect/src/ffi/operations/bulk-encrypt.ts deleted file mode 100644 index 8b9dd78db..000000000 --- a/packages/protect/src/ffi/operations/bulk-encrypt.ts +++ /dev/null @@ -1,213 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { encryptBulk, type JsPlaintext } from '@cipherstash/protect-ffi' -import type { - ProtectColumn, - ProtectTable, - ProtectTableColumn, - ProtectValue, -} from '@cipherstash/schema' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { Context, LockContext } from '../../identify' -import type { - BulkEncryptedData, - BulkEncryptPayload, - Client, - Encrypted, - EncryptOptions, -} from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -// Helper functions for better composability -const createEncryptPayloads = ( - plaintexts: BulkEncryptPayload, - column: ProtectColumn | ProtectValue, - table: ProtectTable, - lockContext?: Context, -) => { - return plaintexts - .map((item, index) => ({ ...item, originalIndex: index })) - .filter(({ plaintext }) => plaintext !== null) - .map(({ id, plaintext, originalIndex }) => ({ - id, - plaintext: plaintext as JsPlaintext, - column: column.getName(), - table: table.tableName, - originalIndex, - ...(lockContext && { lockContext }), - })) -} - -const createNullResult = ( - plaintexts: BulkEncryptPayload, -): BulkEncryptedData => { - return plaintexts.map(({ id }) => ({ id, data: null })) -} - -const mapEncryptedDataToResult = ( - plaintexts: BulkEncryptPayload, - encryptedData: Encrypted[], -): BulkEncryptedData => { - const result: BulkEncryptedData = new Array(plaintexts.length) - let encryptedIndex = 0 - - for (let i = 0; i < plaintexts.length; i++) { - if (plaintexts[i].plaintext === null) { - result[i] = { id: plaintexts[i].id, data: null } - } else { - result[i] = { - id: plaintexts[i].id, - data: encryptedData[encryptedIndex], - } - encryptedIndex++ - } - } - - return result -} - -export class BulkEncryptOperation extends ProtectOperation { - private client: Client - private plaintexts: BulkEncryptPayload - private column: ProtectColumn | ProtectValue - private table: ProtectTable - - constructor( - client: Client, - plaintexts: BulkEncryptPayload, - opts: EncryptOptions, - ) { - super() - this.client = client - this.plaintexts = plaintexts - this.column = opts.column - this.table = opts.table - } - - public withLockContext( - lockContext: LockContext, - ): BulkEncryptOperationWithLockContext { - return new BulkEncryptOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Bulk encrypting data WITHOUT a lock context', { - column: this.column.getName(), - table: this.table.tableName, - }) - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - if (!this.plaintexts || this.plaintexts.length === 0) { - return [] - } - - const nonNullPayloads = createEncryptPayloads( - this.plaintexts, - this.column, - this.table, - ) - - if (nonNullPayloads.length === 0) { - return createNullResult(this.plaintexts) - } - - const { metadata } = this.getAuditData() - - const encryptedData = await encryptBulk(this.client, { - plaintexts: nonNullPayloads, - unverifiedContext: metadata, - }) - - return mapEncryptedDataToResult(this.plaintexts, encryptedData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - plaintexts: BulkEncryptPayload - column: ProtectColumn | ProtectValue - table: ProtectTable - } { - return { - client: this.client, - plaintexts: this.plaintexts, - column: this.column, - table: this.table, - } - } -} - -export class BulkEncryptOperationWithLockContext extends ProtectOperation { - private operation: BulkEncryptOperation - private lockContext: LockContext - - constructor(operation: BulkEncryptOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, plaintexts, column, table } = - this.operation.getOperation() - - logger.debug('Bulk encrypting data WITH a lock context', { - column: column.getName(), - table: table.tableName, - }) - - if (!client) { - throw noClientError() - } - if (!plaintexts || plaintexts.length === 0) { - return [] - } - - const context = await this.lockContext.getLockContext() - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const nonNullPayloads = createEncryptPayloads( - plaintexts, - column, - table, - context.data.context, - ) - - if (nonNullPayloads.length === 0) { - return createNullResult(plaintexts) - } - - const { metadata } = this.getAuditData() - - const encryptedData = await encryptBulk(client, { - plaintexts: nonNullPayloads, - serviceToken: context.data.ctsToken, - unverifiedContext: metadata, - }) - - return mapEncryptedDataToResult(plaintexts, encryptedData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/decrypt-model.ts b/packages/protect/src/ffi/operations/decrypt-model.ts deleted file mode 100644 index 3bdf3c34b..000000000 --- a/packages/protect/src/ffi/operations/decrypt-model.ts +++ /dev/null @@ -1,109 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Decrypted } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { - decryptModelFields, - decryptModelFieldsWithLockContext, -} from '../model-helpers' -import { ProtectOperation } from './base-operation' - -export class DecryptModelOperation< - T extends Record, -> extends ProtectOperation> { - private client: Client - private model: T - - constructor(client: Client, model: T) { - super() - this.client = client - this.model = model - } - - public withLockContext( - lockContext: LockContext, - ): DecryptModelOperationWithLockContext { - return new DecryptModelOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise, ProtectError>> { - logger.debug('Decrypting model WITHOUT a lock context') - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - const auditData = this.getAuditData() - - return await decryptModelFields(this.model, this.client, auditData) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - model: T - } { - return { - client: this.client, - model: this.model, - } - } -} - -export class DecryptModelOperationWithLockContext< - T extends Record, -> extends ProtectOperation> { - private operation: DecryptModelOperation - private lockContext: LockContext - - constructor(operation: DecryptModelOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise, ProtectError>> { - return await withResult( - async () => { - const { client, model } = this.operation.getOperation() - - logger.debug('Decrypting model WITH a lock context') - - if (!client) { - throw noClientError() - } - - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const auditData = this.getAuditData() - - return await decryptModelFieldsWithLockContext( - model, - client, - context.data, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/decrypt.ts b/packages/protect/src/ffi/operations/decrypt.ts deleted file mode 100644 index 98591f8ed..000000000 --- a/packages/protect/src/ffi/operations/decrypt.ts +++ /dev/null @@ -1,130 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { - decrypt as ffiDecrypt, - type JsPlaintext, -} from '@cipherstash/protect-ffi' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Encrypted } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -/** - * Decrypts an encrypted payload using the provided client. - * This is the type returned by the {@link ProtectClient.decrypt | decrypt} method of the {@link ProtectClient}. - */ -export class DecryptOperation extends ProtectOperation { - private client: Client - private encryptedData: Encrypted - - constructor(client: Client, encryptedData: Encrypted) { - super() - this.client = client - this.encryptedData = encryptedData - } - - public withLockContext( - lockContext: LockContext, - ): DecryptOperationWithLockContext { - return new DecryptOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - if (this.encryptedData === null) { - return null - } - - const { metadata } = this.getAuditData() - - logger.debug('Decrypting data WITHOUT a lock context', { - metadata, - }) - - return await ffiDecrypt(this.client, { - ciphertext: this.encryptedData, - unverifiedContext: metadata, - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - encryptedData: Encrypted - auditData?: Record - } { - return { - client: this.client, - encryptedData: this.encryptedData, - auditData: this.getAuditData(), - } - } -} - -export class DecryptOperationWithLockContext extends ProtectOperation { - private operation: DecryptOperation - private lockContext: LockContext - - constructor(operation: DecryptOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - const auditData = operation.getAuditData() - if (auditData) { - this.audit(auditData) - } - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, encryptedData } = this.operation.getOperation() - - if (!client) { - throw noClientError() - } - - if (encryptedData === null) { - return null - } - - const { metadata } = this.getAuditData() - - logger.debug('Decrypting data WITH a lock context', { - metadata, - }) - - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - return await ffiDecrypt(client, { - ciphertext: encryptedData, - unverifiedContext: metadata, - lockContext: context.data.context, - serviceToken: context.data.ctsToken, - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.DecryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/deprecated/search-terms.ts b/packages/protect/src/ffi/operations/deprecated/search-terms.ts deleted file mode 100644 index f0b2c2603..000000000 --- a/packages/protect/src/ffi/operations/deprecated/search-terms.ts +++ /dev/null @@ -1,131 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { encryptQueryBulk, type QueryPayload } from '@cipherstash/protect-ffi' -import { logger } from '../../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../../..' -import type { LockContext } from '../../../identify' -import type { Client, EncryptedSearchTerm, SearchTerm } from '../../../types' -import { noClientError } from '../..' -import { getErrorCode } from '../../helpers/error-code' -import { inferIndexType } from '../../helpers/infer-index-type' -import { ProtectOperation } from '../base-operation' - -/** - * @deprecated Use `BatchEncryptQueryOperation` instead. - * This class is maintained for backward compatibility only. - */ -export class SearchTermsOperation extends ProtectOperation< - EncryptedSearchTerm[] -> { - constructor( - private client: Client, - private terms: SearchTerm[], - ) { - super() - } - - public withLockContext( - lockContext: LockContext, - ): SearchTermsOperationWithLockContext { - return new SearchTermsOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Creating search terms (deprecated API)', { - count: this.terms.length, - }) - - return await withResult( - async () => { - if (!this.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const queries: QueryPayload[] = this.terms.map((term) => ({ - plaintext: term.value, - column: term.column.getName(), - table: term.table.tableName, - indexType: inferIndexType(term.column), - })) - - const encryptedTerms = await encryptQueryBulk(this.client, { - queries, - unverifiedContext: metadata, - }) - - return this.terms.map((term, index) => { - if (term.returnType === 'composite-literal') { - return `(${JSON.stringify(JSON.stringify(encryptedTerms[index]))})` - } - if (term.returnType === 'escaped-composite-literal') { - return `${JSON.stringify(`(${JSON.stringify(JSON.stringify(encryptedTerms[index]))})`)}` - } - return encryptedTerms[index] - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} - -export class SearchTermsOperationWithLockContext extends ProtectOperation< - EncryptedSearchTerm[] -> { - constructor( - private operation: SearchTermsOperation, - private lockContext: LockContext, - ) { - super() - this.auditMetadata = (operation as any).auditMetadata - } - - public async execute(): Promise> { - const lockContextResult = await this.lockContext.getLockContext() - if (lockContextResult.failure) { - return { failure: lockContextResult.failure } - } - - const { ctsToken, context } = lockContextResult.data - const op = this.operation as any - - return await withResult( - async () => { - if (!op.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const queries: QueryPayload[] = op.terms.map((term: SearchTerm) => ({ - plaintext: term.value, - column: term.column.getName(), - table: term.table.tableName, - indexType: inferIndexType(term.column), - lockContext: context, - })) - - const encryptedTerms = await encryptQueryBulk(op.client, { - queries, - serviceToken: ctsToken, - unverifiedContext: metadata, - }) - - return op.terms.map((term: SearchTerm, index: number) => { - if (term.returnType === 'composite-literal') { - return `(${JSON.stringify(JSON.stringify(encryptedTerms[index]))})` - } - if (term.returnType === 'escaped-composite-literal') { - return `${JSON.stringify(`(${JSON.stringify(JSON.stringify(encryptedTerms[index]))})`)}` - } - return encryptedTerms[index] - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/encrypt-model.ts b/packages/protect/src/ffi/operations/encrypt-model.ts deleted file mode 100644 index 54f1bb59e..000000000 --- a/packages/protect/src/ffi/operations/encrypt-model.ts +++ /dev/null @@ -1,128 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import type { ProtectTable, ProtectTableColumn } from '@cipherstash/schema' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Decrypted } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { - encryptModelFields, - encryptModelFieldsWithLockContext, -} from '../model-helpers' -import { ProtectOperation } from './base-operation' - -export class EncryptModelOperation< - T extends Record, -> extends ProtectOperation { - private client: Client - private model: Decrypted - private table: ProtectTable - - constructor( - client: Client, - model: Decrypted, - table: ProtectTable, - ) { - super() - this.client = client - this.model = model - this.table = table - } - - public withLockContext( - lockContext: LockContext, - ): EncryptModelOperationWithLockContext { - return new EncryptModelOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Encrypting model WITHOUT a lock context', { - table: this.table.tableName, - }) - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - const auditData = this.getAuditData() - - return await encryptModelFields( - this.model, - this.table, - this.client, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - model: Decrypted - table: ProtectTable - } { - return { - client: this.client, - model: this.model, - table: this.table, - } - } -} - -export class EncryptModelOperationWithLockContext< - T extends Record, -> extends ProtectOperation { - private operation: EncryptModelOperation - private lockContext: LockContext - - constructor(operation: EncryptModelOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, model, table } = this.operation.getOperation() - - logger.debug('Encrypting model WITH a lock context', { - table: table.tableName, - }) - - if (!client) { - throw noClientError() - } - - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - const auditData = this.getAuditData() - - return await encryptModelFieldsWithLockContext( - model, - table, - client, - context.data, - auditData, - ) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/encrypt-query.ts b/packages/protect/src/ffi/operations/encrypt-query.ts deleted file mode 100644 index bb633ccea..000000000 --- a/packages/protect/src/ffi/operations/encrypt-query.ts +++ /dev/null @@ -1,178 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { - encryptQuery as ffiEncryptQuery, - type JsPlaintext, -} from '@cipherstash/protect-ffi' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import { formatEncryptedResult } from '../../helpers' -import type { LockContext } from '../../identify' -import type { - Client, - EncryptedQueryResult, - EncryptQueryOptions, -} from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { resolveIndexType } from '../helpers/infer-index-type' -import { - assertValueIndexCompatibility, - validateNumericValue, -} from '../helpers/validation' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -/** - * @internal Use {@link ProtectClient.encryptQuery} instead. - */ -export class EncryptQueryOperation extends ProtectOperation { - constructor( - private client: Client, - private plaintext: JsPlaintext | null, - private opts: EncryptQueryOptions, - ) { - super() - } - - public withLockContext( - lockContext: LockContext, - ): EncryptQueryOperationWithLockContext { - return new EncryptQueryOperationWithLockContext( - this.client, - this.plaintext, - this.opts, - lockContext, - this.auditMetadata, - ) - } - - public async execute(): Promise> { - logger.debug('Encrypting query', { - column: this.opts.column.getName(), - table: this.opts.table.tableName, - queryType: this.opts.queryType, - }) - - if (this.plaintext === null || this.plaintext === undefined) { - return { data: null } - } - - const validationError = validateNumericValue(this.plaintext) - if (validationError?.failure) { - return { failure: validationError.failure } - } - - return await withResult( - async () => { - if (!this.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const { indexType, queryOp } = resolveIndexType( - this.opts.column, - this.opts.queryType, - this.plaintext, - ) - - // Validate value/index compatibility - assertValueIndexCompatibility( - this.plaintext, - indexType, - this.opts.column.getName(), - ) - - const encrypted = await ffiEncryptQuery(this.client, { - plaintext: this.plaintext as JsPlaintext, - column: this.opts.column.getName(), - table: this.opts.table.tableName, - indexType, - queryOp, - unverifiedContext: metadata, - }) - - return formatEncryptedResult(encrypted, this.opts.returnType) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation() { - return { client: this.client, plaintext: this.plaintext, ...this.opts } - } -} - -/** - * @internal Use {@link ProtectClient.encryptQuery} with `.withLockContext()` instead. - */ -export class EncryptQueryOperationWithLockContext extends ProtectOperation { - constructor( - private client: Client, - private plaintext: JsPlaintext | null, - private opts: EncryptQueryOptions, - private lockContext: LockContext, - auditMetadata?: Record, - ) { - super() - this.auditMetadata = auditMetadata - } - - public async execute(): Promise> { - if (this.plaintext === null || this.plaintext === undefined) { - return { data: null } - } - - const validationError = validateNumericValue(this.plaintext) - if (validationError?.failure) { - return { failure: validationError.failure } - } - - const lockContextResult = await this.lockContext.getLockContext() - if (lockContextResult.failure) { - return { failure: lockContextResult.failure } - } - - const { ctsToken, context } = lockContextResult.data - - return await withResult( - async () => { - if (!this.client) throw noClientError() - - const { metadata } = this.getAuditData() - - const { indexType, queryOp } = resolveIndexType( - this.opts.column, - this.opts.queryType, - this.plaintext, - ) - - // Validate value/index compatibility - assertValueIndexCompatibility( - this.plaintext, - indexType, - this.opts.column.getName(), - ) - - const encrypted = await ffiEncryptQuery(this.client, { - plaintext: this.plaintext as JsPlaintext, - column: this.opts.column.getName(), - table: this.opts.table.tableName, - indexType, - queryOp, - lockContext: context, - serviceToken: ctsToken, - unverifiedContext: metadata, - }) - - return formatEncryptedResult(encrypted, this.opts.returnType) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/ffi/operations/encrypt.ts b/packages/protect/src/ffi/operations/encrypt.ts deleted file mode 100644 index d91afa757..000000000 --- a/packages/protect/src/ffi/operations/encrypt.ts +++ /dev/null @@ -1,158 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { - encrypt as ffiEncrypt, - type JsPlaintext, -} from '@cipherstash/protect-ffi' -import type { - ProtectColumn, - ProtectTable, - ProtectTableColumn, - ProtectValue, -} from '@cipherstash/schema' -import { logger } from '../../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '../..' -import type { LockContext } from '../../identify' -import type { Client, Encrypted, EncryptOptions } from '../../types' -import { getErrorCode } from '../helpers/error-code' -import { noClientError } from '../index' -import { ProtectOperation } from './base-operation' - -export class EncryptOperation extends ProtectOperation { - private client: Client - private plaintext: JsPlaintext | null - private column: ProtectColumn | ProtectValue - private table: ProtectTable - - constructor( - client: Client, - plaintext: JsPlaintext | null, - opts: EncryptOptions, - ) { - super() - this.client = client - this.plaintext = plaintext - this.column = opts.column - this.table = opts.table - } - - public withLockContext( - lockContext: LockContext, - ): EncryptOperationWithLockContext { - return new EncryptOperationWithLockContext(this, lockContext) - } - - public async execute(): Promise> { - logger.debug('Encrypting data WITHOUT a lock context', { - column: this.column.getName(), - table: this.table.tableName, - }) - - return await withResult( - async () => { - if (!this.client) { - throw noClientError() - } - - if (this.plaintext === null) { - return null - } - - if ( - typeof this.plaintext === 'number' && - Number.isNaN(this.plaintext) - ) { - throw new Error('[protect]: Cannot encrypt NaN value') - } - - if ( - typeof this.plaintext === 'number' && - !Number.isFinite(this.plaintext) - ) { - throw new Error('[protect]: Cannot encrypt Infinity value') - } - - const { metadata } = this.getAuditData() - - return await ffiEncrypt(this.client, { - plaintext: this.plaintext, - column: this.column.getName(), - table: this.table.tableName, - unverifiedContext: metadata, - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } - - public getOperation(): { - client: Client - plaintext: JsPlaintext | null - column: ProtectColumn | ProtectValue - table: ProtectTable - } { - return { - client: this.client, - plaintext: this.plaintext, - column: this.column, - table: this.table, - } - } -} - -export class EncryptOperationWithLockContext extends ProtectOperation { - private operation: EncryptOperation - private lockContext: LockContext - - constructor(operation: EncryptOperation, lockContext: LockContext) { - super() - this.operation = operation - this.lockContext = lockContext - } - - public async execute(): Promise> { - return await withResult( - async () => { - const { client, plaintext, column, table } = - this.operation.getOperation() - - logger.debug('Encrypting data WITH a lock context', { - column: column, - table: table, - }) - - if (!client) { - throw noClientError() - } - - if (plaintext === null) { - return null - } - - const { metadata } = this.getAuditData() - const context = await this.lockContext.getLockContext() - - if (context.failure) { - throw new Error(`[protect]: ${context.failure.message}`) - } - - return await ffiEncrypt(client, { - plaintext, - column: column.getName(), - table: table.tableName, - lockContext: context.data.context, - serviceToken: context.data.ctsToken, - unverifiedContext: metadata, - }) - }, - (error: unknown) => ({ - type: ProtectErrorTypes.EncryptionError, - message: (error as Error).message, - code: getErrorCode(error), - }), - ) - } -} diff --git a/packages/protect/src/helpers/index.ts b/packages/protect/src/helpers/index.ts deleted file mode 100644 index aae2909b5..000000000 --- a/packages/protect/src/helpers/index.ts +++ /dev/null @@ -1,207 +0,0 @@ -import type { - Encrypted as CipherStashEncrypted, - EncryptedQuery as CipherStashEncryptedQuery, - EncryptedScalarQuery, - KeysetIdentifier as KeysetIdentifierFfi, -} from '@cipherstash/protect-ffi' -import type { - Encrypted, - EncryptedQueryResult, - KeysetIdentifier, -} from '../types' - -/** - * The shape `encryptQuery` / `encryptQueryBulk` can return: a full storage - * payload (`Encrypted`, returned for `ste_vec_term` containment queries) or a - * query-only payload with no ciphertext (`EncryptedQuery`, returned for - * scalar `unique`/`match`/`ore` lookups and `ste_vec_selector` path queries). - * - * TODO: duplicated in `@cipherstash/stack` — see - * `packages/stack/src/encryption/helpers/index.ts`. Both copies should be - * removed once `@cipherstash/protect-ffi` exports a named alias for the - * `encryptQuery` return type (https://github.com/cipherstash/stack/pull/473). - */ -type EncryptedQueryTerm = CipherStashEncrypted | CipherStashEncryptedQuery - -export type EncryptedPgComposite = { - data: Encrypted -} - -/** - * Helper function to transform an encrypted payload into a PostgreSQL composite type. - * Use this when inserting data via Supabase or similar clients. - */ -export function encryptedToPgComposite(obj: Encrypted): EncryptedPgComposite { - return { - data: obj, - } -} - -/** - * Helper function to transform an encrypted payload into a PostgreSQL composite literal string. - * Use this when querying with `.eq()` or similar equality operations in Supabase. - * - * @deprecated Use `encryptQuery()` with `returnType: 'composite-literal'` instead. - * @example - * ```typescript - * // Before (deprecated): - * const [encrypted] = await protectClient.encryptQuery([ - * { value: searchValue, column, table, queryType: 'equality' } - * ]) - * const literal = encryptedToCompositeLiteral(encrypted) - * await supabase.from('table').select().eq('column', literal) - * - * // After (recommended): - * const [searchTerm] = await protectClient.encryptQuery([ - * { value: searchValue, column, table, queryType: 'equality', returnType: 'composite-literal' } - * ]) - * await supabase.from('table').select().eq('column', searchTerm) - * ``` - */ -export function encryptedToCompositeLiteral(obj: EncryptedQueryTerm): string { - if (obj === null) { - throw new Error('encryptedToCompositeLiteral: obj cannot be null') - } - return `(${JSON.stringify(JSON.stringify(obj))})` -} - -/** - * Helper function to transform an encrypted payload into an escaped PostgreSQL composite literal string. - * Use this when you need the composite literal format to be escaped as a string value. - * - * @deprecated Use `encryptQuery()` with `returnType: 'escaped-composite-literal'` instead. - * See also: `encryptedToCompositeLiteral` for parallel deprecation guidance. - * @example - * ```typescript - * // Before (deprecated): - * const [encrypted] = await protectClient.encryptQuery([ - * { value: searchValue, column, table, queryType: 'equality' } - * ]) - * const escapedLiteral = encryptedToEscapedCompositeLiteral(encrypted) - * - * // After (recommended): - * const [searchTerm] = await protectClient.encryptQuery([ - * { value: searchValue, column, table, queryType: 'equality', returnType: 'escaped-composite-literal' } - * ]) - * ``` - */ -export function encryptedToEscapedCompositeLiteral( - obj: EncryptedQueryTerm, -): string { - if (obj === null) { - throw new Error('encryptedToEscapedCompositeLiteral: obj cannot be null') - } - return JSON.stringify(encryptedToCompositeLiteral(obj)) -} - -export function formatEncryptedResult( - encrypted: EncryptedQueryTerm, - returnType?: string, -): EncryptedQueryResult { - if (returnType === 'composite-literal') { - return encryptedToCompositeLiteral(encrypted) - } - if (returnType === 'escaped-composite-literal') { - return encryptedToEscapedCompositeLiteral(encrypted) - } - return encrypted -} - -/** - * Helper function to transform a model's encrypted fields into PostgreSQL composite types - */ -export function modelToEncryptedPgComposites>( - model: T, -): T { - const result: Record = {} - - for (const [key, value] of Object.entries(model)) { - if (isEncryptedPayload(value)) { - result[key] = encryptedToPgComposite(value) - } else { - result[key] = value - } - } - - return result as T -} - -/** - * Helper function to transform multiple models' encrypted fields into PostgreSQL composite types - */ -export function bulkModelsToEncryptedPgComposites< - T extends Record, ->(models: T[]): T[] { - return models.map((model) => modelToEncryptedPgComposites(model)) -} - -export function toFfiKeysetIdentifier( - keyset: KeysetIdentifier | undefined, -): KeysetIdentifierFfi | undefined { - if (!keyset) return undefined - - if ('name' in keyset) { - return { Name: keyset.name } - } - - return { Uuid: keyset.id } -} - -/** - * Helper function to check if a value is an encrypted payload - */ -export function isEncryptedPayload(value: unknown): value is Encrypted { - if (value === null) return false - - // TODO: this can definitely be improved - if (typeof value === 'object') { - const obj = value as Encrypted - return ( - obj !== null && 'v' in obj && ('c' in obj || 'sv' in obj) && 'i' in obj - ) - } - - return false -} - -/** - * Type guard narrowing a value to {@link EncryptedScalarQuery} — the scalar - * query term (`unique` / `match` / `ore` lookup) returned by `encryptQuery` / - * `encryptQueryBulk`. Unlike a storage payload it carries no ciphertext (`c`); - * it carries exactly one lookup term: `hm`, `bf`, or `ob`. - * - * Use this to discriminate a scalar query term from a storage payload - * (`EncryptedScalar`/`EncryptedSteVec`) or a `ste_vec_selector` query. - */ -export function isEncryptedScalarQuery( - value: unknown, -): value is EncryptedScalarQuery { - if (value === null || typeof value !== 'object') return false - - const obj = value as Record - - // `k: 'ct'` is the scalar discriminant; a query term never carries the - // ciphertext (`c`) that every storage payload has. - if (obj.k !== 'ct' || 'c' in obj) return false - if ( - typeof obj.v !== 'number' || - typeof obj.i !== 'object' || - obj.i === null - ) { - return false - } - - // Exactly one lookup term: `hm` (unique), `bf` (match), or `ob` (ore). - const lookupTerms = [ - typeof obj.hm === 'string', - Array.isArray(obj.bf), - Array.isArray(obj.ob), - ].filter(Boolean) - return lookupTerms.length === 1 -} - -export { - buildNestedObject, - parseJsonbPath, - toJsonPath, -} from './jsonb' diff --git a/packages/protect/src/helpers/jsonb.ts b/packages/protect/src/helpers/jsonb.ts deleted file mode 100644 index 018ea99c4..000000000 --- a/packages/protect/src/helpers/jsonb.ts +++ /dev/null @@ -1,99 +0,0 @@ -/** - * JSONB path utilities for converting between path formats. - * - * These utilities support dot-notation and basic JSONPath-style array indices (e.g., "[0]"). - * Only limited validation is performed (forbidden prototype keys); callers should still - * ensure segments are valid property names. - */ - -/** - * Convert a dot-notation path to JSONPath selector format. - * - * @example - * toJsonPath("user.email") // "$.user.email" - * toJsonPath("$.user.email") // "$.user.email" (unchanged) - * toJsonPath(".user.email") // "$.user.email" - * toJsonPath("name") // "$.name" - */ -export function toJsonPath(path: string): string { - if (!path || path === '$') return '$' - if (path.startsWith('$[')) return path - if (path.startsWith('$.')) return path - if (path.startsWith('$')) return `$.${path.slice(1)}` - if (path.startsWith('.')) return `$${path}` - if (path.startsWith('[')) return `$${path}` - return `$.${path}` -} - -/** - * Parse a JSONB path string into segments. - * Handles both dot notation and JSONPath format. - * - * Returns an empty array for empty, null, or undefined input (defensive for JS consumers). - * - * @example - * parseJsonbPath("user.email") // ["user", "email"] - * parseJsonbPath("$.user.email") // ["user", "email"] - * parseJsonbPath("name") // ["name"] - * parseJsonbPath("$.name") // ["name"] - */ -export function parseJsonbPath(path: string): string[] { - if (!path || typeof path !== 'string') return [] - - // Remove leading $. or $ prefix - const normalized = path.replace(/^\$\.?/, '') - - if (!normalized) return [] - - return normalized.split('.').filter(Boolean) -} - -/** - * Build a nested object from a dot-notation path and value. - * - * @example - * buildNestedObject("user.role", "admin") - * // Returns: { user: { role: "admin" } } - * - * buildNestedObject("name", "alice") - * // Returns: { name: "alice" } - * - * buildNestedObject("a.b.c", 123) - * // Returns: { a: { b: { c: 123 } } } - */ -const FORBIDDEN_KEYS = ['__proto__', 'prototype', 'constructor'] - -function validateSegment(segment: string): void { - if (FORBIDDEN_KEYS.includes(segment)) { - throw new Error(`Path contains forbidden segment: ${segment}`) - } -} - -export function buildNestedObject( - path: string, - value: unknown, -): Record { - if (!path) { - throw new Error('Path cannot be empty') - } - - const segments = parseJsonbPath(path) - if (segments.length === 0) { - throw new Error('Path must contain at least one segment') - } - - const result: Record = Object.create(null) - let current = result - - for (let i = 0; i < segments.length - 1; i++) { - const key = segments[i] - validateSegment(key) - current[key] = Object.create(null) - current = current[key] as Record - } - - const leafKey = segments[segments.length - 1] - validateSegment(leafKey) - current[leafKey] = value - return result -} diff --git a/packages/protect/src/identify/index.ts b/packages/protect/src/identify/index.ts deleted file mode 100644 index a1ff22be4..000000000 --- a/packages/protect/src/identify/index.ts +++ /dev/null @@ -1,128 +0,0 @@ -import { type Result, withResult } from '@byteslice/result' -import { loadWorkSpaceId } from '../../../utils/config' -import { logger } from '../../../utils/logger' -import { type ProtectError, ProtectErrorTypes } from '..' - -export type CtsRegions = 'ap-southeast-2' - -export type IdentifyOptions = { - fetchFromCts?: boolean -} - -export type CtsToken = { - accessToken: string - expiry: number -} - -export type Context = { - identityClaim: string[] -} - -export type LockContextOptions = { - context?: Context - ctsToken?: CtsToken -} - -export type GetLockContextResponse = { - ctsToken: CtsToken - context: Context -} - -export class LockContext { - private ctsToken: CtsToken | undefined - private workspaceId: string - private context: Context - - constructor({ - context = { identityClaim: ['sub'] }, - ctsToken, - }: LockContextOptions = {}) { - const workspaceId = loadWorkSpaceId() - - if (!workspaceId) { - throw new Error( - 'You have not defined a workspace ID in your config file, or the CS_WORKSPACE_ID environment variable.', - ) - } - - if (ctsToken) { - this.ctsToken = ctsToken - } - - this.workspaceId = workspaceId - this.context = context - logger.debug('Successfully initialized the EQL lock context.') - } - - async identify(jwtToken: string): Promise> { - const workspaceId = this.workspaceId - - const ctsEndpoint = - process.env.CS_CTS_ENDPOINT || - 'https://ap-southeast-2.aws.auth.viturhosted.net' - - const ctsFetchResult = await withResult( - () => - fetch(`${ctsEndpoint}/api/authorize`, { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - }, - body: JSON.stringify({ - workspaceId, - oidcToken: jwtToken, - }), - }), - (error) => ({ - type: ProtectErrorTypes.CtsTokenError, - message: error.message, - }), - ) - - if (ctsFetchResult.failure) { - return ctsFetchResult - } - - const identifiedLockContext = await withResult( - async () => { - const ctsToken = (await ctsFetchResult.data.json()) as CtsToken - - if (!ctsToken.accessToken) { - throw new Error( - 'The response from the CipherStash API did not contain an access token. Please contact support.', - ) - } - - this.ctsToken = ctsToken - return this - }, - (error) => ({ - type: ProtectErrorTypes.CtsTokenError, - message: error.message, - }), - ) - - return identifiedLockContext - } - - getLockContext(): Promise> { - return withResult( - () => { - if (!this.ctsToken?.accessToken && !this.ctsToken?.expiry) { - throw new Error( - 'The CTS token is not set. Please call identify() with a users JWT token, or pass an existing CTS token to the LockContext constructor before calling getLockContext().', - ) - } - - return { - context: this.context, - ctsToken: this.ctsToken, - } - }, - (error) => ({ - type: ProtectErrorTypes.CtsTokenError, - message: error.message, - }), - ) - } -} diff --git a/packages/protect/src/index.ts b/packages/protect/src/index.ts deleted file mode 100644 index c9505f07d..000000000 --- a/packages/protect/src/index.ts +++ /dev/null @@ -1,180 +0,0 @@ -import type { ProtectTable, ProtectTableColumn } from '@cipherstash/schema' -import { buildEncryptConfig } from '@cipherstash/schema' -import { ProtectClient } from './ffi' -import type { KeysetIdentifier } from './types' - -// Re-export FFI error types for programmatic error handling -export { - ProtectError as FfiProtectError, - type ProtectErrorCode, -} from '@cipherstash/protect-ffi' - -export const ProtectErrorTypes = { - ClientInitError: 'ClientInitError', - EncryptionError: 'EncryptionError', - DecryptionError: 'DecryptionError', - LockContextError: 'LockContextError', - CtsTokenError: 'CtsTokenError', -} - -export interface ProtectError { - type: (typeof ProtectErrorTypes)[keyof typeof ProtectErrorTypes] - message: string - code?: import('@cipherstash/protect-ffi').ProtectErrorCode -} - -type AtLeastOneCsTable = [T, ...T[]] - -export type ProtectClientConfig = { - schemas: AtLeastOneCsTable> - - /** - * The CipherStash workspace CRN (Cloud Resource Name). - * Format: `crn:.aws:`. - * Can also be set via the `CS_WORKSPACE_CRN` environment variable. - */ - workspaceCrn?: string - - /** - * The API access key used for authenticating with the CipherStash API. - * Can also be set via the `CS_CLIENT_ACCESS_KEY` environment variable. - * Obtain this from the CipherStash dashboard after creating a workspace. - */ - accessKey?: string - - /** - * The client identifier used to authenticate with CipherStash services. - * Can also be set via the `CS_CLIENT_ID` environment variable. - * Generated during workspace onboarding in the CipherStash dashboard. - */ - clientId?: string - - /** - * The client key material used in combination with ZeroKMS for encryption operations. - * Can also be set via the `CS_CLIENT_KEY` environment variable. - * Generated during workspace onboarding in the CipherStash dashboard. - */ - clientKey?: string - - /** - * An optional keyset identifier for multi-tenant encryption. - * Each keyset provides cryptographic isolation, giving each tenant its own keyspace. - * Specify by name (`{ name: "tenant-a" }`) or UUID (`{ id: "..." }`). - * Keysets are created and managed in the CipherStash dashboard. - */ - keyset?: KeysetIdentifier -} - -function isValidUuid(uuid: string): boolean { - const uuidRegex = - /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i - return uuidRegex.test(uuid) -} - -/* Initialize a Protect client with the provided configuration. - - @param config - The configuration object for initializing the Protect client. - - @see {@link ProtectClientConfig} for details on the configuration options. - - @returns A Promise that resolves to an instance of ProtectClient. - - @throws Will throw an error if no schemas are provided or if the keyset ID is not a valid UUID. -*/ -export const protect = async ( - config: ProtectClientConfig, -): Promise => { - const { schemas } = config - - if (!schemas.length) { - throw new Error( - '[protect]: At least one csTable must be provided to initialize the protect client', - ) - } - - if ( - config.keyset && - 'id' in config.keyset && - !isValidUuid(config.keyset.id) - ) { - throw new Error( - '[protect]: Invalid UUID provided for keyset id. Must be a valid UUID.', - ) - } - - const clientConfig = { - workspaceCrn: config.workspaceCrn, - accessKey: config.accessKey, - clientId: config.clientId, - clientKey: config.clientKey, - keyset: config.keyset, - } - - const client = new ProtectClient() - const encryptConfig = buildEncryptConfig(...schemas) - - const result = await client.init({ - encryptConfig, - ...clientConfig, - }) - - if (result.failure) { - throw new Error(`[protect]: ${result.failure.message}`) - } - - return result.data -} - -export type { Result } from '@byteslice/result' -export type { - ProtectColumn, - ProtectTable, - ProtectTableColumn, - ProtectValue, -} from '@cipherstash/schema' -export { csColumn, csTable, csValue } from '@cipherstash/schema' -export type { ProtectClient } from './ffi' -// Helpers -export { - inferIndexType, - validateIndexType, -} from './ffi/helpers/infer-index-type' -export type { ProtectOperation } from './ffi/operations/base-operation' -export { - BatchEncryptQueryOperation, - BatchEncryptQueryOperationWithLockContext, -} from './ffi/operations/batch-encrypt-query' -export type { BulkDecryptOperation } from './ffi/operations/bulk-decrypt' -export type { BulkDecryptModelsOperation } from './ffi/operations/bulk-decrypt-models' -export type { BulkEncryptOperation } from './ffi/operations/bulk-encrypt' -export type { BulkEncryptModelsOperation } from './ffi/operations/bulk-encrypt-models' -export type { DecryptOperation } from './ffi/operations/decrypt' -export type { DecryptModelOperation } from './ffi/operations/decrypt-model' -export type { EncryptOperation } from './ffi/operations/encrypt' -export type { EncryptModelOperation } from './ffi/operations/encrypt-model' -// Operations -export { - EncryptQueryOperation, - EncryptQueryOperationWithLockContext, -} from './ffi/operations/encrypt-query' -export * from './helpers' -// LockContext related type exports -export type { - Context, - CtsRegions, - CtsToken, - GetLockContextResponse, - IdentifyOptions, - LockContextOptions, -} from './identify' -// LockContext class export (value export for instantiation) -export { LockContext } from './identify' -// Types -export type { - EncryptQueryOptions, - FfiIndexTypeName, - QueryTypeName, - ScalarQueryTerm, -} from './types' -export * from './types' -export { queryTypes, queryTypeToFfi } from './types' diff --git a/packages/protect/src/stash/index.ts b/packages/protect/src/stash/index.ts deleted file mode 100644 index 73338dc83..000000000 --- a/packages/protect/src/stash/index.ts +++ /dev/null @@ -1,459 +0,0 @@ -import type { Result } from '@byteslice/result' -import { csColumn, csTable } from '@cipherstash/schema' -import { encryptedToPgComposite, type ProtectClient, protect } from '../index' -import type { Encrypted } from '../types' - -export type SecretName = string -export type SecretValue = string - -/** - * Configuration options for initializing the Stash client - */ -export interface StashConfig { - workspaceCRN: string - clientId: string - clientKey: string - environment: string - apiKey: string - accessKey?: string -} - -/** - * Secret metadata returned from the API - */ -export interface SecretMetadata { - id?: string - name: string - environment: string - createdAt?: string - updatedAt?: string -} - -/** - * API response for listing secrets - */ -export interface ListSecretsResponse { - environment: string - secrets: SecretMetadata[] -} - -/** - * API response for getting a secret - */ -export interface GetSecretResponse { - name: string - environment: string - encryptedValue: { - data: Encrypted - } - createdAt?: string - updatedAt?: string -} - -export interface DecryptedSecretResponse { - name: string - environment: string - value: string - createdAt?: string - updatedAt?: string -} - -/** - * The Stash client provides a high-level API for managing encrypted secrets - * stored in CipherStash. Secrets are encrypted locally before being sent to - * the API, ensuring end-to-end encryption. - */ -export class Stash { - private protectClient: ProtectClient | null = null - private config: StashConfig - private readonly apiBaseUrl = - process.env.STASH_API_URL || 'https://getstash.sh/api/secrets' - private readonly secretsSchema = csTable('secrets', { - value: csColumn('value'), - }) - - /** - * Extracts the workspace ID from a CRN string. - * CRN format: crn:region.aws:ID - * - * @param crn The CRN string to extract from - * @returns The workspace ID portion of the CRN - */ - private extractWorkspaceIdFromCrn(crn: string): string { - const match = crn.match(/crn:[^:]+:([^:]+)$/) - if (!match) { - throw new Error('Invalid CRN format') - } - return match[1] - } - - constructor(config: StashConfig) { - this.config = config - } - - /** - * Initialize the Stash client and underlying Protect client - */ - private async ensureInitialized(): Promise { - if (this.protectClient) { - return - } - - this.protectClient = await protect({ - schemas: [this.secretsSchema], - workspaceCrn: this.config.workspaceCRN, - clientId: this.config.clientId, - clientKey: this.config.clientKey, - accessKey: this.config.apiKey, - keyset: { - name: this.config.environment, - }, - }) - } - - /** - * Get the authorization header for API requests - */ - private getAuthHeader(): string { - return `Bearer ${this.config.apiKey}` - } - - /** - * Make an API request with error handling - */ - private async apiRequest( - method: string, - path: string, - body?: unknown, - ): Promise> { - try { - const url = `${this.apiBaseUrl}${path}` - const headers: Record = { - 'Content-Type': 'application/json', - Authorization: this.getAuthHeader(), - } - - const response = await fetch(url, { - method, - headers, - body: body ? JSON.stringify(body) : undefined, - }) - - if (!response.ok) { - const errorText = await response.text() - let errorMessage = `API request failed with status ${response.status}` - try { - const errorJson = JSON.parse(errorText) - errorMessage = errorJson.message || errorMessage - } catch { - errorMessage = errorText || errorMessage - } - - return { - failure: { - type: 'ApiError', - message: errorMessage, - }, - } - } - - const data = await response.json() - return { data } - } catch (error) { - return { - failure: { - type: 'NetworkError', - message: - error instanceof Error - ? error.message - : 'Unknown network error occurred', - }, - } - } - } - - /** - * Store an encrypted secret in the vault. - * The value is encrypted locally before being sent to the API. - * - * @param name - The name of the secret - * @param value - The plaintext value to encrypt and store - * @returns A Result indicating success or failure - * - * @example - * ```typescript - * const stash = new Stash({ ... }) - * const result = await stash.set('DATABASE_URL', 'postgres://user:pass@localhost:5432/mydb') - * if (result.failure) { - * console.error('Failed to set secret:', result.failure.message) - * } - * ``` - */ - async set( - name: SecretName, - value: SecretValue, - ): Promise> { - await this.ensureInitialized() - - if (!this.protectClient) { - return { - failure: { - type: 'ClientError', - message: 'Failed to initialize Protect client', - }, - } - } - - // Encrypt the value locally - const encryptResult = await this.protectClient.encrypt(value, { - column: this.secretsSchema.value, - table: this.secretsSchema, - }) - - if (encryptResult.failure) { - return { - failure: { - type: 'EncryptionError', - message: encryptResult.failure.message, - }, - } - } - - // Extract workspaceId from CRN - const workspaceId = this.extractWorkspaceIdFromCrn(this.config.workspaceCRN) - - // Send encrypted value to API - return await this.apiRequest('POST', '/set', { - workspaceId, - environment: this.config.environment, - name, - encryptedValue: encryptedToPgComposite(encryptResult.data), - }) - } - - /** - * Retrieve and decrypt a secret from the vault. - * The secret is decrypted locally after retrieval. - * - * @param name - The name of the secret to retrieve - * @returns A Result containing the decrypted value or an error - * - * @example - * ```typescript - * const stash = new Stash({ ... }) - * const result = await stash.get('DATABASE_URL') - * if (result.failure) { - * console.error('Failed to get secret:', result.failure.message) - * } else { - * console.log('Secret value:', result.data) - * } - * ``` - */ - async get( - name: SecretName, - ): Promise> { - await this.ensureInitialized() - - if (!this.protectClient) { - return { - failure: { - type: 'ClientError', - message: 'Failed to initialize Protect client', - }, - } - } - - // Extract workspaceId from CRN - const workspaceId = this.extractWorkspaceIdFromCrn(this.config.workspaceCRN) - - // Fetch encrypted value from API - const apiResult = await this.apiRequest('POST', '/get', { - workspaceId, - environment: this.config.environment, - name, - }) - - if (apiResult.failure) { - return apiResult - } - - // Decrypt the value locally - const decryptResult = await this.protectClient.decrypt( - apiResult.data.encryptedValue.data, - ) - - if (decryptResult.failure) { - return { - failure: { - type: 'DecryptionError', - message: decryptResult.failure.message, - }, - } - } - - if (typeof decryptResult.data !== 'string') { - return { - failure: { - type: 'DecryptionError', - message: 'Decrypted value is not a string', - }, - } - } - - return { data: decryptResult.data } - } - - /** - * Retrieve and decrypt many secrets from the vault. - * The secrets are decrypted locally after retrieval. - * This method only triggers a single network request to the ZeroKMS. - * - * @param names - The names of the secrets to retrieve - * @returns A Result containing an object mapping secret names to their decrypted values - * - * @example - * ```typescript - * const stash = new Stash({ ... }) - * const result = await stash.getMany(['DATABASE_URL', 'API_KEY']) - * if (result.failure) { - * console.error('Failed to get secrets:', result.failure.message) - * } else { - * const dbUrl = result.data.DATABASE_URL // Access by name - * const apiKey = result.data.API_KEY - * } - * ``` - */ - async getMany( - names: SecretName[], - ): Promise< - Result, { type: string; message: string }> - > { - await this.ensureInitialized() - - if (!this.protectClient) { - return { - failure: { - type: 'ClientError', - message: 'Failed to initialize Protect client', - }, - } - } - - // Extract workspaceId from CRN - const workspaceId = this.extractWorkspaceIdFromCrn(this.config.workspaceCRN) - - // Fetch encrypted value from API - const apiResult = await this.apiRequest( - 'POST', - '/get-many', - { - workspaceId, - environment: this.config.environment, - names, - }, - ) - - if (apiResult.failure) { - return apiResult - } - - const dataToDecrypt = apiResult.data.map((item) => ({ - name: item.name, - value: item.encryptedValue.data, - })) - - const decryptResult = - await this.protectClient.bulkDecryptModels(dataToDecrypt) - - if (decryptResult.failure) { - return { - failure: { - type: 'DecryptionError', - message: decryptResult.failure.message, - }, - } - } - - console.log('Decrypt result:', JSON.stringify(decryptResult.data, null, 2)) - - // Transform array of decrypted secrets into an object keyed by secret name - const decryptedSecrets = - decryptResult.data as unknown as DecryptedSecretResponse[] - const secretsMap: Record = {} - - for (const secret of decryptedSecrets) { - if (secret.name && secret.value) { - secretsMap[secret.name] = secret.value - } - } - - return { data: secretsMap } - } - - /** - * List all secrets in the environment. - * Only names and metadata are returned; values remain encrypted. - * - * @returns A Result containing the list of secrets or an error - * - * @example - * ```typescript - * const stash = new Stash({ ... }) - * const result = await stash.list() - * if (result.failure) { - * console.error('Failed to list secrets:', result.failure.message) - * } else { - * console.log('Secrets:', result.data) - * } - * ``` - */ - async list(): Promise< - Result - > { - // Extract workspaceId from CRN - const workspaceId = this.extractWorkspaceIdFromCrn(this.config.workspaceCRN) - - const apiResult = await this.apiRequest( - 'POST', - '/list', - { - workspaceId, - environment: this.config.environment, - }, - ) - - if (apiResult.failure) { - return apiResult - } - - return { data: apiResult.data.secrets } - } - - /** - * Delete a secret from the vault. - * - * @param name - The name of the secret to delete - * @returns A Result indicating success or failure - * - * @example - * ```typescript - * const stash = new Stash({ ... }) - * const result = await stash.delete('DATABASE_URL') - * if (result.failure) { - * console.error('Failed to delete secret:', result.failure.message) - * } - * ``` - */ - async delete( - name: SecretName, - ): Promise> { - // Extract workspaceId from CRN - const workspaceId = this.extractWorkspaceIdFromCrn(this.config.workspaceCRN) - - return await this.apiRequest('POST', '/delete', { - workspaceId, - environment: this.config.environment, - name, - }) - } -} diff --git a/packages/protect/src/types.ts b/packages/protect/src/types.ts deleted file mode 100644 index de8e061db..000000000 --- a/packages/protect/src/types.ts +++ /dev/null @@ -1,241 +0,0 @@ -import type { - Encrypted as CipherStashEncrypted, - EncryptedQuery as CipherStashEncryptedQuery, - JsPlaintext, - newClient, - QueryOpName, -} from '@cipherstash/protect-ffi' -import type { - ProtectColumn, - ProtectTable, - ProtectTableColumn, - ProtectValue, -} from '@cipherstash/schema' - -/** - * Type to represent the client object - */ -export type Client = Awaited> | undefined - -/** - * Type to represent an encrypted payload stored in the database. Always carries - * a ciphertext — scalar payloads at the root (`c`), SteVec payloads at - * `sv[0].c`. For search-term payloads returned by `encryptQuery`, see - * {@link EncryptedQuery}. - */ -export type Encrypted = CipherStashEncrypted | null - -/** - * Type to represent an encrypted query term (search needle) returned by - * `encryptQuery` / `encryptQueryBulk` for scalar (`unique` / `match` / `ore`) - * lookups and `ste_vec_selector` JSON path queries. Carries no ciphertext — it - * is matched against stored values, never decrypted. JSON containment queries - * (`ste_vec_term`) return a storage-shaped {@link Encrypted} payload instead. - */ -export type EncryptedQuery = CipherStashEncryptedQuery | null - -/** - * Represents an encrypted payload in the database - * @deprecated Use `Encrypted` instead - */ -export type EncryptedPayload = Encrypted | null - -/** - * Represents an encrypted data object in the database - * @deprecated Use `Encrypted` instead - */ -export type EncryptedData = Encrypted | null - -/** - * Represents a value that will be encrypted and used in a search - */ -export type SearchTerm = { - value: JsPlaintext - column: ProtectColumn - table: ProtectTable - returnType?: 'eql' | 'composite-literal' | 'escaped-composite-literal' -} - -export type KeysetIdentifier = - | { - name: string - } - | { - id: string - } - -/** - * The return type of the search term based on the return type specified in the `SearchTerm` type. - * - `eql` → an `Encrypted` storage payload (for `ste_vec_term` containment) or an - * {@link EncryptedQuery} term (for scalar lookups and `ste_vec_selector` queries). - * - `composite-literal` → `string` where the value is a composite literal. - * - `escaped-composite-literal` → `string` where the value is an escaped composite literal. - */ -export type EncryptedSearchTerm = Encrypted | EncryptedQuery | string - -/** - * Result type for encryptQuery batch operations. - * Can be an `Encrypted` storage payload (e.g. for `ste_vec_term`), an - * {@link EncryptedQuery} term (for scalar lookups and `ste_vec_selector`), - * a `string` (for composite-literal formats), or `null` (for null inputs). - */ -export type EncryptedQueryResult = Encrypted | EncryptedQuery | string | null - -/** - * Represents a payload to be encrypted using the `encrypt` function - */ -export type EncryptPayload = JsPlaintext | null - -/** - * Represents the options for encrypting a payload using the `encrypt` function - */ -export type EncryptOptions = { - column: ProtectColumn | ProtectValue - table: ProtectTable -} - -/** - * Type to identify encrypted fields in a model - */ -export type EncryptedFields = { - [K in keyof T as T[K] extends Encrypted ? K : never]: T[K] -} - -/** - * Type to identify non-encrypted fields in a model - */ -export type OtherFields = { - [K in keyof T as T[K] extends Encrypted ? never : K]: T[K] -} - -/** - * Type to represent decrypted fields in a model - */ -export type DecryptedFields = { - [K in keyof T as T[K] extends Encrypted ? K : never]: string -} - -/** - * Represents a model with plaintext (decrypted) values instead of the EQL/JSONB types - */ -export type Decrypted = OtherFields & DecryptedFields - -/** - * Types for bulk encryption and decryption operations. - */ -export type BulkEncryptPayload = Array<{ - id?: string - plaintext: JsPlaintext | null -}> - -export type BulkEncryptedData = Array<{ id?: string; data: Encrypted }> -export type BulkDecryptPayload = Array<{ id?: string; data: Encrypted }> -export type BulkDecryptedData = Array> - -type DecryptionSuccess = { - error?: never - data: T - id?: string -} - -type DecryptionError = { - error: T - id?: string - data?: never -} - -export type DecryptionResult = DecryptionSuccess | DecryptionError - -/** - * User-facing query type names for encrypting query values. - * - * - `'equality'`: For exact match queries. {@link https://cipherstash.com/docs/platform/searchable-encryption/supported-queries/exact | Exact Queries} - * - `'freeTextSearch'`: For text search queries. {@link https://cipherstash.com/docs/platform/searchable-encryption/supported-queries/match | Match Queries} - * - `'orderAndRange'`: For comparison and range queries. {@link https://cipherstash.com/docs/platform/searchable-encryption/supported-queries/range | Range Queries} - * - `'steVecSelector'`: For JSONPath selector queries (e.g., '$.user.email') - * - `'steVecTerm'`: For containment queries (e.g., { role: 'admin' }) - * - `'searchableJson'`: Auto-infers selector or term based on plaintext type (recommended) - * - String values → ste_vec_selector (JSONPath queries) - * - Object/Array/Number/Boolean → ste_vec_term (containment queries) - * - * Note: For columns with an ste_vec index, `'searchableJson'` behaves identically to omitting - * `queryType` entirely - both auto-infer the query operation from the plaintext type. Using - * `'searchableJson'` explicitly is useful for code clarity and self-documenting intent. - */ -export type QueryTypeName = - | 'orderAndRange' - | 'freeTextSearch' - | 'equality' - | 'steVecSelector' - | 'steVecTerm' - | 'searchableJson' - -/** - * Internal FFI index type names. - * @internal - */ -export type FfiIndexTypeName = 'ore' | 'match' | 'unique' | 'ste_vec' - -/** - * Query type constants for use with encryptQuery(). - */ -export const queryTypes = { - orderAndRange: 'orderAndRange', - freeTextSearch: 'freeTextSearch', - equality: 'equality', - steVecSelector: 'steVecSelector', - steVecTerm: 'steVecTerm', - searchableJson: 'searchableJson', -} as const satisfies Record - -/** - * Maps user-friendly query type names to FFI index type names. - * @internal - */ -export const queryTypeToFfi: Record = { - orderAndRange: 'ore', - freeTextSearch: 'match', - equality: 'unique', - steVecSelector: 'ste_vec', - steVecTerm: 'ste_vec', - searchableJson: 'ste_vec', -} - -/** - * Maps query type names to FFI query operation names. - * Returns undefined for query types that don't need a specific queryOp. - * @internal - */ -export const queryTypeToQueryOp: Partial> = { - steVecSelector: 'ste_vec_selector', - steVecTerm: 'ste_vec_term', -} - -/** - * Base type for query term options shared between single and bulk operations. - * @internal - */ -export type QueryTermBase = { - column: ProtectColumn - table: ProtectTable - queryType?: QueryTypeName // Optional - auto-infers if omitted - /** - * The format for the returned encrypted value: - * - `'eql'` (default) - Returns raw Encrypted object - * - `'composite-literal'` - Returns PostgreSQL composite literal format `("json")` - * - `'escaped-composite-literal'` - Returns escaped format `"(\"json\")"` - */ - returnType?: 'eql' | 'composite-literal' | 'escaped-composite-literal' -} - -/** - * Options for encrypting a single query term. - */ -export type EncryptQueryOptions = QueryTermBase - -/** - * Individual query term for bulk operations. - */ -export type ScalarQueryTerm = QueryTermBase & { - value: JsPlaintext | null -} diff --git a/packages/protect/tsconfig.json b/packages/protect/tsconfig.json deleted file mode 100644 index 639824185..000000000 --- a/packages/protect/tsconfig.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "compilerOptions": { - // Enable latest features - "lib": ["ES2022", "DOM"], - "target": "ES2022", - "module": "ESNext", - "moduleDetection": "force", - "jsx": "react-jsx", - "allowJs": true, - "esModuleInterop": true, - - // Bundler mode - "moduleResolution": "bundler", - "allowImportingTsExtensions": true, - "verbatimModuleSyntax": true, - "noEmit": true, - - // Best practices - "strict": true, - "skipLibCheck": true, - "noFallthroughCasesInSwitch": true, - - // Some stricter flags (disabled by default) - "noUnusedLocals": false, - "noUnusedParameters": false, - "noPropertyAccessFromIndexSignature": false - } -} diff --git a/packages/protect/tsup.config.ts b/packages/protect/tsup.config.ts deleted file mode 100644 index 8fdee46c6..000000000 --- a/packages/protect/tsup.config.ts +++ /dev/null @@ -1,30 +0,0 @@ -import { defineConfig } from 'tsup' - -export default defineConfig([ - { - entry: [ - 'src/index.ts', - 'src/client.ts', - 'src/identify/index.ts', - 'src/stash/index.ts', - ], - format: ['cjs', 'esm'], - sourcemap: true, - dts: true, - target: 'es2022', - tsconfig: './tsconfig.json', - }, - { - entry: ['src/bin/stash.ts'], - outDir: 'dist/bin', - format: ['esm'], - target: 'es2022', - banner: { - js: '#!/usr/bin/env node', - }, - dts: false, - sourcemap: true, - external: ['dotenv'], - noExternal: [], - }, -]) diff --git a/packages/schema/.npmignore b/packages/schema/.npmignore deleted file mode 100644 index 3490e24db..000000000 --- a/packages/schema/.npmignore +++ /dev/null @@ -1,5 +0,0 @@ -.env -.turbo -node_modules -cipherstash.secret.toml -cipherstash.toml \ No newline at end of file diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md deleted file mode 100644 index 678414865..000000000 --- a/packages/schema/CHANGELOG.md +++ /dev/null @@ -1,104 +0,0 @@ -# @cipherstash/schema - -## 3.0.2-rc.0 - -### Patch Changes - -- 229ce59: `searchableJson()` now pins the SteVec encoding mode to `standard` explicitly. - protect-ffi 0.29 flipped the library default to `compat` (the EQL v3 - encoding); pinning keeps the v2 wire format byte-stable so existing encrypted - JSON columns stay queryable and comparable. - -## 3.0.1 - -### Patch Changes - -- aa9c4b1: Documentation: refresh package READMEs after the protectjs → stack repository rename. Fixed repository and license links, replaced dead in-repo docs links with cipherstash.com/docs URLs, rewrote the incorrect @cipherstash/nextjs README, and added guidance pointing new projects to @cipherstash/stack. - -## 3.0.0 - -### Major Changes - -- f743fcc: Upgrade `@cipherstash/protect-ffi` to `0.23.0` and the bundled CipherStash EQL extension to `eql-2.3.1`. - - Breaking upstream changes adopted in this release: - - - **Encrypt-config schema version**: `buildEncryptConfig` now emits `{ v: 1, ... }` (was `{ v: 2, ... }`). protect-ffi `0.22.0` started validating this field and rejects any value other than `1` with the new `UNSUPPORTED_CONFIG_VERSION` error code. - - **Storage and query payloads are now distinct types** (protect-ffi `0.23.0`): the previously-conflated `Encrypted` type splits into `Encrypted` (storage-only, `c` required) and a new `EncryptedQuery` (search terms — scalar `unique`/`match`/`ore` lookups and `ste_vec_selector` JSON path queries; no `c`). JSON containment queries (`ste_vec_term`) still return a storage-shaped `Encrypted` payload. `encryptQuery` / `encryptQueryBulk` now return `Encrypted | EncryptedQuery`, and the stack's `EncryptedSearchTerm` / `EncryptedQueryResult` unions widen to match. `decrypt` rejects query payloads at the type level. The DynamoDB `SearchTermsOperation` narrows via `'hm' in term` rather than `term.hm`. - - **SteVec encoding default flipped**: protect-ffi's default `mode` for `ste_vec` indexes changed from `compat` to `standard`. The two encodings are not cross-compatible. Existing JSON-searchable data that was indexed under `compat` will need to be re-encrypted to be queryable. The stack adopts the new `standard` default — there is no longer a way to pin `compat` from the SDK. - - **EQL extension bumped to `eql-2.3.1`**: the new SteVec `standard` encoding requires matching support in the database EQL extension. The CLI's bundled SQL (`packages/cli/src/sql/*.sql`) and the `@cipherstash/prisma-next` install bundle (`migrations/20260601T0000_install_eql_bundle/ops.json` + `eql-install.generated.ts`) are updated to `eql-2.3.1`. Databases installed with an older EQL extension must be reinstalled (`stash db install`) before containment / contained-by queries against SteVec columns will work. `eql-2.3.1` ships the `_encrypted_check_c` fix for SteVec storage payloads ([cipherstash/encrypt-query-language#232](https://github.com/cipherstash/encrypt-query-language/issues/232)). - - **New error codes**: `ProtectErrorCode` (re-exported from `@cipherstash/protect-ffi`) gains `MATCH_REQUIRES_TEXT` and `UNSUPPORTED_CONFIG_VERSION`. Exhaustive switches over `ProtectErrorCode` will need additional cases. - - **`match` index validation**: protect-ffi now rejects `match` indexes on columns whose `cast_as` is not text-family (`'text'` / `'string'`) with `MATCH_REQUIRES_TEXT`. The stack's `freeTextSearch()` builder is unaffected because it only targets string-typed columns. - - **`Encrypted` ciphertext shape**: protect-ffi's `Encrypted` type is now a discriminated union keyed on `k` (`'ct'` for scalars, `'sv'` for SteVec). SteVec storage payloads now place the root document ciphertext at `sv[0].c`. The stack's `isEncryptedPayload` runtime check continues to work because storage payloads still carry `c` (scalar) or `sv` (SteVec). The DynamoDB helpers (`toEncryptedDynamoItem`, `SearchTermsOperation`) now narrow on `k` before reading variant-only fields. - - **Config-validation error message wording**: error messages for config-validation failures now come from upstream `ConfigError`. `ProtectError.code` values are preserved; consumers that string-match on `err.message` for config-validation errors must update. - -## 2.2.0 - -### Minor Changes - -- b0e56b8: Upgrade protect-ffi to 0.21.0 and enable array_index_mode for searchable JSON - - - Upgrade `@cipherstash/protect-ffi` to 0.21.0 across all packages - - Enable `array_index_mode: 'all'` on STE vec indexes so JSON array operations - (jsonb_array_elements, jsonb_array_length, array containment) work correctly - - Delegate credential resolution entirely to protect-ffi's `withEnvCredentials` - - Download latest EQL at build/runtime instead of bundling hardcoded SQL files - -## 2.1.0 - -### Minor Changes - -- e769740: Add encrypted JSONB query support with `searchableJson()` (recommended). - - - New `searchableJson()` schema method enables encrypted JSONB path and containment queries - - Automatic query operation inference: string values become JSONPath selector queries, objects/arrays become containment queries - - Also supports explicit `queryType: 'steVecSelector'` and `queryType: 'steVecTerm'` for advanced use cases - - JSONB path utilities (`toJsonPath`, `buildNestedObject`, `parseJsonbPath`) for building encrypted JSON column queries - -## 2.0.2 - -### Patch Changes - -- 532ac3a: Corrected types documentation in README to match Typedoc. - `int` -> `number` - `text` -> `string` - -## 2.0.1 - -### Patch Changes - -- ff4421f: Expanded typedoc documentation - -## 2.0.0 - -### Major Changes - -- 9005484: Include EQL 2.1.8 in package distribution - -## 1.1.0 - -### Minor Changes - -- d8ed4d4: Exported all types for packages looking for deeper integrations with Protect.js. - -## 1.0.0 - -### Major Changes - -- 788dbfc: Added JSON and INT data type support and update FFI to v0.17.1 with x86_64 musl environment platform support. - - - Update @cipherstash/protect-ffi from 0.16.0 to 0.17.1 with support for x86_64 musl platforms. - - Add searchableJson() method to schema for JSON field indexing (the search operations still don't work but this interface exists) - - Refactor type system: EncryptedPayload → Encrypted, add JsPlaintext - - Add comprehensive test suites for JSON, integer, and basic encryption - - Update encryption format to use 'k' property for searchable JSON - - Remove deprecated search terms tests for JSON fields - - Simplify schema data types to text, int, json only - - Update model helpers to handle new encryption format - - Fix type safety issues in bulk operations and model encryption - -## 0.1.0 - -### Minor Changes - -- d0b02ea: Released initial package for CipherStash Encrypt schemas. diff --git a/packages/schema/README.md b/packages/schema/README.md deleted file mode 100644 index f9cc98288..000000000 --- a/packages/schema/README.md +++ /dev/null @@ -1,307 +0,0 @@ -# @cipherstash/schema - -A TypeScript schema builder for CipherStash encryption that enables you to define encryption schemas with searchable encryption capabilities. - -## Overview - -`@cipherstash/schema` is a standalone package that provides the schema building functionality used by `@cipherstash/protect`. While not required for typical usage, this package is available if you need to build encryption configuration schemas directly or want to understand the underlying schema structure. - -> [!TIP] -> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which exposes this functionality as `encryptedTable` / `encryptedColumn` from `@cipherstash/stack/schema`. The `csTable` / `csColumn` API documented below is the legacy naming used by `@cipherstash/protect`. - -## Installation - -```bash -npm install @cipherstash/schema -# or -yarn add @cipherstash/schema -# or -pnpm add @cipherstash/schema -``` - -## Quick Start - -```typescript -import { csTable, csColumn, buildEncryptConfig } from '@cipherstash/schema' - -// Define your schema -const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - name: csColumn('name').freeTextSearch(), - age: csColumn('age').orderAndRange(), -}) - -// Build the encryption configuration -const config = buildEncryptConfig(users) -console.log(config) -``` - -## Core Functions - -### `csTable(tableName, columns)` - -Creates a table definition with encrypted columns. - -```typescript -import { csTable, csColumn } from '@cipherstash/schema' - -const users = csTable('users', { - email: csColumn('email'), - name: csColumn('name'), -}) -``` - -### `csColumn(columnName)` - -Creates a column definition with configurable indexes and data types. - -```typescript -import { csColumn } from '@cipherstash/schema' - -const emailColumn = csColumn('email') - .freeTextSearch() // Enable text search - .equality() // Enable exact matching - .orderAndRange() // Enable sorting and range queries - .dataType('string') // Set data type -``` - -### `csValue(valueName)` - -Creates a value definition for nested objects (up to 3 levels deep). - -```typescript -import { csTable, csColumn, csValue } from '@cipherstash/schema' - -const users = csTable('users', { - email: csColumn('email').equality(), - profile: { - name: csValue('profile.name'), - address: { - street: csValue('profile.address.street'), - location: { - coordinates: csValue('profile.address.location.coordinates'), - }, - }, - }, -}) -``` - -### `buildEncryptConfig(...tables)` - -Builds the final encryption configuration from table definitions. - -```typescript -import { buildEncryptConfig } from '@cipherstash/schema' - -const config = buildEncryptConfig(users, orders, products) -``` - -## Index Types - -### Equality Index (`.equality()`) - -Enables exact matching queries. - -```typescript -const emailColumn = csColumn('email').equality() -// SQL equivalent: WHERE email = 'example@example.com' -``` - -### Free Text Search (`.freeTextSearch()`) - -Enables text search with configurable options. - -```typescript -const descriptionColumn = csColumn('description').freeTextSearch({ - tokenizer: { kind: 'ngram', token_length: 3 }, - token_filters: [{ kind: 'downcase' }], - k: 6, - m: 2048, - include_original: true, -}) -// SQL equivalent: WHERE description LIKE '%example%' -``` - -### Order and Range (`.orderAndRange()`) - -Enables sorting and range queries. - -```typescript -const priceColumn = csColumn('price').orderAndRange() -// SQL equivalent: ORDER BY price ASC, WHERE price > 100 -``` - -## Data Types - -Set the data type for a column using `.dataType()`: - -```typescript -const column = csColumn('field') - .dataType('string') // text (default) - .dataType('number') // Javascript number (i.e. integer or float) - .dataType('jsonb') // JSON binary -``` - -## Nested Objects - -Support for nested object encryption (up to 3 levels deep): - -```typescript -const users = csTable('users', { - email: csColumn('email').equality(), - profile: { - name: csValue('profile.name'), - address: { - street: csValue('profile.address.street'), - city: csValue('profile.address.city'), - location: { - coordinates: csValue('profile.address.location.coordinates'), - }, - }, - }, -}) -``` - -> **Note**: Nested objects are not searchable and are not recommended for SQL databases. Use separate columns for searchable fields. - -## Advanced Configuration - -### Custom Token Filters - -```typescript -const column = csColumn('field').equality([ - { kind: 'downcase' } -]) -``` - -### Custom Match Options - -```typescript -const column = csColumn('field').freeTextSearch({ - tokenizer: { kind: 'standard' }, - token_filters: [{ kind: 'downcase' }], - k: 8, - m: 4096, - include_original: false, -}) -``` - -## Type Safety - -The schema builder provides full TypeScript support: - -```typescript -import { csTable, csColumn, type ProtectTableColumn } from '@cipherstash/schema' - -const users = csTable('users', { - email: csColumn('email').equality(), - name: csColumn('name').freeTextSearch(), -} as const) - -// TypeScript will infer the correct types -type UsersTable = typeof users -``` - -## Integration with Protect.js - -While this package can be used standalone, it's typically used through `@cipherstash/protect`: - -```typescript -import { csTable, csColumn } from '@cipherstash/protect' - -const users = csTable('users', { - email: csColumn('email').equality().freeTextSearch(), -}) - -const protectClient = await protect({ - schemas: [users], -}) -``` - -## Generated Configuration - -The `buildEncryptConfig` function generates a configuration object like this: - -```typescript -{ - v: 2, - tables: { - users: { - email: { - cast_as: 'text', - indexes: { - unique: { token_filters: [] }, - match: { - tokenizer: { kind: 'ngram', token_length: 3 }, - token_filters: [{ kind: 'downcase' }], - k: 6, - m: 2048, - include_original: true, - }, - ore: {}, - }, - }, - }, - }, -} -``` - -## Use Cases - -- **Standalone schema building**: When you need to generate encryption configurations outside of Protect.js -- **Custom tooling**: Building tools that work with CipherStash encryption schemas -- **Schema validation**: Validating schema structures before using them with Protect.js -- **Documentation generation**: Creating documentation from schema definitions - -## API Reference - -### `csTable(tableName: string, columns: ProtectTableColumn)` - -Creates a table definition. - -**Parameters:** -- `tableName`: The name of the table in the database -- `columns`: Object defining the columns and their configurations - -**Returns:** `ProtectTable & T` - -### `csColumn(columnName: string)` - -Creates a column definition. - -**Parameters:** -- `columnName`: The name of the column in the database - -**Returns:** `ProtectColumn` - -**Methods:** -- `.dataType(castAs: CastAs)`: Set the data type -- `.equality(tokenFilters?: TokenFilter[])`: Enable equality index -- `.freeTextSearch(opts?: MatchIndexOpts)`: Enable text search -- `.orderAndRange()`: Enable order and range index -- `.searchableJson()`: Enable searchable JSON index - -### `csValue(valueName: string)` - -Creates a value definition for nested objects. - -**Parameters:** -- `valueName`: Dot-separated path to the value (e.g., 'profile.name') - -**Returns:** `ProtectValue` - -**Methods:** -- `.dataType(castAs: CastAs)`: Set the data type - -### `buildEncryptConfig(...tables: ProtectTable[])` - -Builds the encryption configuration. - -**Parameters:** -- `...tables`: Variable number of table definitions - -**Returns:** `EncryptConfig` - -## License - -MIT License - see [LICENSE.md](../../LICENSE.md) for details. diff --git a/packages/schema/__tests__/schema.test.ts b/packages/schema/__tests__/schema.test.ts deleted file mode 100644 index 373f63585..000000000 --- a/packages/schema/__tests__/schema.test.ts +++ /dev/null @@ -1,146 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { buildEncryptConfig, csColumn, csTable, csValue } from '../src' - -describe('Schema with nested columns', () => { - it('should handle nested column structures in encrypt config', () => { - const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndRange(), - address: csColumn('address').freeTextSearch(), - example: { - field: csValue('example.field'), - nested: { - deep: csValue('example.nested.deep'), - }, - }, - } as const) - - const config = buildEncryptConfig(users) - - // Verify basic structure - expect(config).toEqual({ - v: 1, - tables: { - users: expect.any(Object), - }, - }) - - // Verify all columns are present with correct names - const columns = config.tables.users - expect(Object.keys(columns)).toEqual([ - 'email', - 'address', - 'example.field', - 'example.nested.deep', - ]) - - // Verify email column configuration - expect(columns.email).toEqual({ - cast_as: 'string', - indexes: { - match: expect.any(Object), - unique: expect.any(Object), - ore: {}, - }, - }) - - // Verify address column configuration - expect(columns.address).toEqual({ - cast_as: 'string', - indexes: { - match: expect.any(Object), - }, - }) - - // Verify nested field configuration - expect(columns['example.field']).toEqual({ - cast_as: 'string', - indexes: {}, - }) - - // Verify deeply nested field configuration - expect(columns['example.nested.deep']).toEqual({ - cast_as: 'string', - indexes: {}, - }) - }) - - it('should handle multiple tables with nested columns', () => { - const users = csTable('users', { - email: csColumn('email').equality(), - profile: { - name: csValue('profile.name'), - }, - } as const) - - const posts = csTable('posts', { - title: csColumn('title').freeTextSearch(), - metadata: { - tags: csValue('metadata.tags'), - }, - } as const) - - const config = buildEncryptConfig(users, posts) - - // Verify both tables are present - expect(Object.keys(config.tables)).toEqual(['users', 'posts']) - - // Verify users table columns - expect(Object.keys(config.tables.users)).toEqual(['email', 'profile.name']) - expect(config.tables.users.email.indexes).toHaveProperty('unique') - - // Verify posts table columns - expect(Object.keys(config.tables.posts)).toEqual(['title', 'metadata.tags']) - expect(config.tables.posts.title.indexes).toHaveProperty('match') - }) - - it('should handle complex nested structures with multiple index types', () => { - const complex = csTable('complex', { - id: csColumn('id').equality(), - content: { - text: csValue('content.text'), - metadata: { - tags: csValue('content.metadata.tags'), - stats: { - views: csValue('content.metadata.stats.views'), - }, - }, - }, - } as const) - - const config = buildEncryptConfig(complex) - - // Verify all columns are present - expect(Object.keys(config.tables.complex)).toEqual([ - 'id', - 'content.text', - 'content.metadata.tags', - 'content.metadata.stats.views', - ]) - - // Verify complex nested column with multiple indexes - expect(config.tables.complex['content.metadata.tags']).toEqual({ - cast_as: 'string', - indexes: {}, - }) - - // Verify deeply nested column with order and range - expect(config.tables.complex['content.metadata.stats.views']).toEqual({ - cast_as: 'string', - indexes: {}, - }) - }) - - // NOTE: Leaving this test commented out until stevec indexing for JSON is supported. - /*it('should handle ste_vec index for JSON columns', () => { - const users = csTable('users', { - json: csColumn('json').dataType('jsonb').searchableJson(), - } as const) - - const config = buildEncryptConfig(users) - - expect(config.tables.users.json.indexes).toHaveProperty('ste_vec') - expect(config.tables.users.json.indexes.ste_vec?.prefix).toEqual( - 'users/json', - ) - })*/ -}) diff --git a/packages/schema/__tests__/searchable-json.test.ts b/packages/schema/__tests__/searchable-json.test.ts deleted file mode 100644 index 8f14ec7cb..000000000 --- a/packages/schema/__tests__/searchable-json.test.ts +++ /dev/null @@ -1,46 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { buildEncryptConfig, csColumn, csTable } from '../src/index' - -describe('searchableJson()', () => { - it('sets cast_as to json and ste_vec marker on column build', () => { - const column = csColumn('metadata').searchableJson() - const config = column.build() - - expect(config.cast_as).toBe('json') - expect(config.indexes.ste_vec?.prefix).toBe('enabled') - }) - - it('is chainable', () => { - const column = csColumn('metadata') - expect(column.searchableJson()).toBe(column) - }) -}) - -describe('ProtectTable.build() with searchableJson', () => { - it('transforms prefix to table/column format', () => { - const users = csTable('users', { - metadata: csColumn('metadata').searchableJson(), - }) - const built = users.build() - - expect(built.columns.metadata.cast_as).toBe('json') - expect(built.columns.metadata.indexes.ste_vec?.prefix).toBe( - 'users/metadata', - ) - }) -}) - -describe('buildEncryptConfig with searchableJson', () => { - it('emits ste_vec index with table/column prefix', () => { - const users = csTable('users', { - metadata: csColumn('metadata').searchableJson(), - }) - - const config = buildEncryptConfig(users) - - expect(config.tables.users.metadata.cast_as).toBe('json') - expect(config.tables.users.metadata.indexes.ste_vec?.prefix).toBe( - 'users/metadata', - ) - }) -}) diff --git a/packages/schema/package.json b/packages/schema/package.json deleted file mode 100644 index f718c249f..000000000 --- a/packages/schema/package.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "name": "@cipherstash/schema", - "version": "3.0.2-rc.0", - "description": "CipherStash schema builder for TypeScript", - "keywords": [ - "encrypted", - "protect", - "schema", - "builder" - ], - "bugs": { - "url": "https://github.com/cipherstash/stack/issues" - }, - "repository": { - "type": "git", - "url": "git+https://github.com/cipherstash/stack.git", - "directory": "packages/schema" - }, - "license": "MIT", - "author": "CipherStash ", - "type": "module", - "exports": { - ".": { - "types": "./dist/index.d.ts", - "import": "./dist/index.js", - "require": "./dist/index.cjs" - } - }, - "files": [ - "dist", - "README.md" - ], - "scripts": { - "build": "tsup", - "dev": "tsup --watch", - "test": "vitest run", - "release": "tsup" - }, - "devDependencies": { - "tsup": "catalog:repo", - "typescript": "catalog:repo", - "vitest": "catalog:repo" - }, - "dependencies": { - "zod": "^3.25.76" - }, - "publishConfig": { - "access": "public" - } -} diff --git a/packages/schema/src/index.ts b/packages/schema/src/index.ts deleted file mode 100644 index 1b61526a3..000000000 --- a/packages/schema/src/index.ts +++ /dev/null @@ -1,378 +0,0 @@ -import { z } from 'zod' - -// ------------------------ -// Zod schemas -// ------------------------ - -/** - * Allowed cast types for CipherStash schema fields. - * - * **Possible values:** - * - `"bigint"` - * - `"boolean"` - * - `"date"` - * - `"number"` - * - `"string"` - * - `"json"` - * - * @remarks - * This is a Zod enum used at runtime to validate schema definitions. - * Use {@link CastAs} when typing your own code. - */ -export const castAsEnum = z - .enum(['bigint', 'boolean', 'date', 'number', 'string', 'json']) - .default('string') - -const tokenFilterSchema = z.object({ - kind: z.literal('downcase'), -}) - -const tokenizerSchema = z - .union([ - z.object({ - kind: z.literal('standard'), - }), - z.object({ - kind: z.literal('ngram'), - token_length: z.number(), - }), - ]) - .default({ kind: 'ngram', token_length: 3 }) - .optional() - -const oreIndexOptsSchema = z.object({}) - -const uniqueIndexOptsSchema = z.object({ - token_filters: z.array(tokenFilterSchema).default([]).optional(), -}) - -const matchIndexOptsSchema = z.object({ - tokenizer: tokenizerSchema, - token_filters: z.array(tokenFilterSchema).default([]).optional(), - k: z.number().default(6).optional(), - m: z.number().default(2048).optional(), - include_original: z.boolean().default(false).optional(), -}) - -const arrayIndexModeSchema = z.union([ - z.literal('all'), - z.literal('none'), - z.object({ - item: z.boolean().optional(), - wildcard: z.boolean().optional(), - position: z.boolean().optional(), - }), -]) - -const steVecIndexOptsSchema = z.object({ - prefix: z.string(), - array_index_mode: arrayIndexModeSchema.optional(), - mode: z.enum(['compat', 'standard']).optional(), -}) - -const indexesSchema = z - .object({ - ore: oreIndexOptsSchema.optional(), - unique: uniqueIndexOptsSchema.optional(), - match: matchIndexOptsSchema.optional(), - ste_vec: steVecIndexOptsSchema.optional(), - }) - .default({}) - -const columnSchema = z - .object({ - cast_as: castAsEnum, - indexes: indexesSchema, - }) - .default({}) - -const tableSchema = z.record(columnSchema).default({}) - -const tablesSchema = z.record(tableSchema).default({}) - -// `v` is locked to `1` because protect-ffi `0.22.0+` rejects any other value at -// `newClient` with `UNSUPPORTED_CONFIG_VERSION`. Enforcing it here means a bad -// hand-rolled config fails at zod-validation time instead of crossing into the -// native module first. -export const encryptConfigSchema = z.object({ - v: z.literal(1), - tables: tablesSchema, -}) - -// ------------------------ -// Type definitions -// ------------------------ - -/** - * Type-safe alias for {@link castAsEnum} used to specify the *unencrypted* data type of a column or value. - * This is important because once encrypted, all data is stored as binary blobs. - * - * @see {@link castAsEnum} for possible values. - */ -export type CastAs = z.infer -export type TokenFilter = z.infer -export type MatchIndexOpts = z.infer -export type SteVecIndexOpts = z.infer -export type UniqueIndexOpts = z.infer -export type OreIndexOpts = z.infer -export type ColumnSchema = z.infer - -export type ProtectTableColumn = { - [key: string]: - | ProtectColumn - | { - [key: string]: - | ProtectValue - | { - [key: string]: - | ProtectValue - | { - [key: string]: ProtectValue - } - } - } -} -export type EncryptConfig = z.infer - -// ------------------------ -// Interface definitions -// ------------------------ -export class ProtectValue { - private valueName: string - private castAsValue: CastAs - - constructor(valueName: string) { - this.valueName = valueName - this.castAsValue = 'string' - } - - /** - * Set or override the cast_as value. - */ - dataType(castAs: CastAs) { - this.castAsValue = castAs - return this - } - - build() { - return { - cast_as: this.castAsValue, - indexes: {}, - } - } - - getName() { - return this.valueName - } -} - -export class ProtectColumn { - private columnName: string - private castAsValue: CastAs - private indexesValue: { - ore?: OreIndexOpts - unique?: UniqueIndexOpts - match?: Required - ste_vec?: SteVecIndexOpts - } = {} - - constructor(columnName: string) { - this.columnName = columnName - this.castAsValue = 'string' - } - - /** - * Set or override the cast_as value. - */ - dataType(castAs: CastAs) { - this.castAsValue = castAs - return this - } - - /** - * Enable ORE indexing (Order-Revealing Encryption). - */ - orderAndRange() { - this.indexesValue.ore = {} - return this - } - - /** - * Enable an Exact index. Optionally pass tokenFilters. - */ - equality(tokenFilters?: TokenFilter[]) { - this.indexesValue.unique = { - token_filters: tokenFilters ?? [], - } - return this - } - - /** - * Enable a Match index. Allows passing of custom match options. - */ - freeTextSearch(opts?: MatchIndexOpts) { - // Provide defaults - this.indexesValue.match = { - tokenizer: opts?.tokenizer ?? { kind: 'ngram', token_length: 3 }, - token_filters: opts?.token_filters ?? [ - { - kind: 'downcase', - }, - ], - k: opts?.k ?? 6, - m: opts?.m ?? 2048, - include_original: opts?.include_original ?? true, - } - return this - } - - /** - * Configure this column for searchable encrypted JSON. - * Enables path queries ($.user.email) and containment queries ({ role: 'admin' }). - * Automatically sets cast_as to 'json'. - */ - searchableJson() { - this.castAsValue = 'json' - // `mode: 'standard'` pins the per-entry ordering term to CLLW-ORE (`oc`), - // the only encoding the eql_v2 SQL compares. protect-ffi 0.29 flipped the - // library default to `compat` (CLLW-OPE, `op`) for EQL v3; without the pin - // every v2 containment query silently matches nothing — and existing v2 - // rows encrypted under `standard` are not cross-comparable with `compat` - // anyway, so this also keeps the v2 wire format byte-stable. - this.indexesValue.ste_vec = { - prefix: 'enabled', - array_index_mode: 'all', - mode: 'standard', - } - return this - } - - build() { - return { - cast_as: this.castAsValue, - indexes: this.indexesValue, - } - } - - getName() { - return this.columnName - } -} - -interface TableDefinition { - tableName: string - columns: Record -} - -export class ProtectTable { - constructor( - public readonly tableName: string, - private readonly columnBuilders: T, - ) {} - - /** - * Build a TableDefinition object: tableName + built column configs. - */ - build(): TableDefinition { - const builtColumns: Record = {} - - const processColumn = ( - builder: - | ProtectColumn - | Record< - string, - | ProtectValue - | Record< - string, - | ProtectValue - | Record> - > - >, - colName: string, - ) => { - if (builder instanceof ProtectColumn) { - const builtColumn = builder.build() - - // Hanlde building the ste_vec index for JSON columns so users don't have to pass the prefix. - if ( - builtColumn.cast_as === 'json' && - builtColumn.indexes.ste_vec?.prefix === 'enabled' - ) { - builtColumns[colName] = { - ...builtColumn, - indexes: { - ...builtColumn.indexes, - ste_vec: { - ...builtColumn.indexes.ste_vec, - prefix: `${this.tableName}/${colName}`, - }, - }, - } - } else { - builtColumns[colName] = builtColumn - } - } else { - for (const [key, value] of Object.entries(builder)) { - if (value instanceof ProtectValue) { - builtColumns[value.getName()] = value.build() - } else { - processColumn(value, key) - } - } - } - } - - for (const [colName, builder] of Object.entries(this.columnBuilders)) { - processColumn(builder, colName) - } - - return { - tableName: this.tableName, - columns: builtColumns, - } - } -} - -// ------------------------ -// User facing functions -// ------------------------ -export function csTable( - tableName: string, - columns: T, -): ProtectTable & T { - const tableBuilder = new ProtectTable(tableName, columns) as ProtectTable & - T - - for (const [colName, colBuilder] of Object.entries(columns)) { - ;(tableBuilder as ProtectTableColumn)[colName] = colBuilder - } - - return tableBuilder -} - -export function csColumn(columnName: string) { - return new ProtectColumn(columnName) -} - -export function csValue(valueName: string) { - return new ProtectValue(valueName) -} - -// ------------------------ -// Internal functions -// ------------------------ -export function buildEncryptConfig( - ...protectTables: Array> -): EncryptConfig { - const config: EncryptConfig = { - v: 1, - tables: {}, - } - - for (const tb of protectTables) { - const tableDef = tb.build() - config.tables[tableDef.tableName] = tableDef.columns - } - - return config -} diff --git a/packages/schema/tsconfig.json b/packages/schema/tsconfig.json deleted file mode 100644 index 6da81c927..000000000 --- a/packages/schema/tsconfig.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "compilerOptions": { - // Enable latest features - "lib": ["ESNext", "DOM"], - "target": "ESNext", - "module": "ESNext", - "moduleDetection": "force", - "jsx": "react-jsx", - "allowJs": true, - "esModuleInterop": true, - - // Bundler mode - "moduleResolution": "bundler", - "allowImportingTsExtensions": true, - "verbatimModuleSyntax": true, - "noEmit": true, - - // Best practices - "strict": true, - "skipLibCheck": true, - "noFallthroughCasesInSwitch": true, - - // Some stricter flags (disabled by default) - "noUnusedLocals": false, - "noUnusedParameters": false, - "noPropertyAccessFromIndexSignature": false - } -} diff --git a/packages/schema/tsup.config.ts b/packages/schema/tsup.config.ts deleted file mode 100644 index 0ced0a354..000000000 --- a/packages/schema/tsup.config.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { defineConfig } from 'tsup' - -export default defineConfig({ - entry: ['src/index.ts'], - format: ['cjs', 'esm'], - sourcemap: true, - dts: true, -}) diff --git a/packages/stack/src/encryption/helpers/index.ts b/packages/stack/src/encryption/helpers/index.ts index 8b431d8d2..74c59f1a2 100644 --- a/packages/stack/src/encryption/helpers/index.ts +++ b/packages/stack/src/encryption/helpers/index.ts @@ -11,10 +11,9 @@ import type { Encrypted, EncryptedQueryResult, KeysetIdentifier } from '@/types' * payload, or a v3 ciphertext-free scalar/SteVec query term (including the * bare selector hash and `eql_v3.query_json` containment needle). * - * TODO: duplicated in `@cipherstash/protect` — see - * `packages/protect/src/helpers/index.ts`. Both copies should be removed once - * `@cipherstash/protect-ffi` exports a named alias for the `encryptQuery` - * return type (https://github.com/cipherstash/stack/pull/473). + * TODO: replace this local union once `@cipherstash/protect-ffi` exports a + * named alias for the `encryptQuery` return type + * (https://github.com/cipherstash/stack/pull/473). */ type EncryptedQueryTerm = | CipherStashEncrypted diff --git a/packages/stack/src/wasm-inline.ts b/packages/stack/src/wasm-inline.ts index 42d1d75c2..0b7e89663 100644 --- a/packages/stack/src/wasm-inline.ts +++ b/packages/stack/src/wasm-inline.ts @@ -882,7 +882,7 @@ export class WasmEncryptionClient { (term) => term.value !== null && term.value !== undefined, async (live) => // The FFI's batch field is `queries` (matching the native - // ffiEncryptQueryBulk call in packages/protect). + // ffiEncryptQueryBulk call in the encryption client). (await wasmEncryptQueryBulk( // biome-ignore lint/plugin: the FFI handle is an opaque wasm-bindgen pointer with no JS-side type this.client as never, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 105cb6953..eb397065d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -86,9 +86,6 @@ importers: e2e: dependencies: - '@cipherstash/protect': - specifier: workspace:* - version: link:../packages/protect '@cipherstash/stack': specifier: workspace:* version: link:../packages/stack @@ -465,97 +462,6 @@ importers: specifier: catalog:repo version: 3.2.7(@types/node@26.1.1)(jiti@2.7.0)(lightningcss@1.30.2)(terser@5.44.1)(tsx@4.23.0)(yaml@2.9.0) - packages/protect: - dependencies: - '@byteslice/result': - specifier: ^0.2.0 - version: 0.2.0 - '@cipherstash/protect-ffi': - specifier: 0.23.0 - version: 0.23.0 - '@cipherstash/schema': - specifier: workspace:* - version: link:../schema - '@stricli/core': - specifier: ^1.2.9 - version: 1.2.9 - dotenv: - specifier: 17.4.2 - version: 17.4.2 - zod: - specifier: ^3.25.76 - version: 3.25.76 - devDependencies: - '@supabase/supabase-js': - specifier: ^2.110.2 - version: 2.110.2 - execa: - specifier: ^9.5.2 - version: 9.6.1 - json-schema-to-typescript: - specifier: ^15.0.2 - version: 15.0.4 - postgres: - specifier: ^3.4.7 - version: 3.4.9 - tsup: - specifier: catalog:repo - version: 8.5.1(jiti@2.7.0)(postcss@8.5.14)(tsx@4.23.0)(typescript@5.9.3)(yaml@2.9.0) - tsx: - specifier: catalog:repo - version: 4.23.0 - typescript: - specifier: catalog:repo - version: 5.9.3 - vitest: - specifier: catalog:repo - version: 3.2.7(@types/node@26.1.1)(jiti@2.7.0)(lightningcss@1.30.2)(terser@5.44.1)(tsx@4.23.0)(yaml@2.9.0) - optionalDependencies: - '@rollup/rollup-linux-x64-gnu': - specifier: 4.62.2 - version: 4.62.2 - - packages/protect-dynamodb: - dependencies: - '@byteslice/result': - specifier: ^0.2.0 - version: 0.2.0 - devDependencies: - '@cipherstash/protect': - specifier: workspace:* - version: link:../protect - dotenv: - specifier: ^17.4.2 - version: 17.4.2 - tsup: - specifier: catalog:repo - version: 8.5.1(jiti@2.7.0)(postcss@8.5.14)(tsx@4.23.0)(typescript@5.9.3)(yaml@2.9.0) - tsx: - specifier: catalog:repo - version: 4.23.0 - typescript: - specifier: catalog:repo - version: 5.9.3 - vitest: - specifier: catalog:repo - version: 3.2.7(@types/node@26.1.1)(jiti@2.7.0)(lightningcss@1.30.2)(terser@5.44.1)(tsx@4.23.0)(yaml@2.9.0) - - packages/schema: - dependencies: - zod: - specifier: ^3.25.76 - version: 3.25.76 - devDependencies: - tsup: - specifier: catalog:repo - version: 8.5.1(jiti@2.7.0)(postcss@8.5.14)(tsx@4.23.0)(typescript@5.9.3)(yaml@2.9.0) - typescript: - specifier: catalog:repo - version: 5.9.3 - vitest: - specifier: catalog:repo - version: 3.2.7(@types/node@26.1.1)(jiti@2.7.0)(lightningcss@1.30.2)(terser@5.44.1)(tsx@4.23.0)(yaml@2.9.0) - packages/stack: dependencies: '@byteslice/result': @@ -1076,69 +982,36 @@ packages: '@cipherstash/eql@3.0.2': resolution: {integrity: sha512-E85o0aoOqgCW6RReLtJ0YLh/ExRlmDJo7LlJGpWPoMTVaw+CW8o11DJ4oJIF1vFtuxSVxNULuPzzBuVmpTvvcA==} - '@cipherstash/protect-ffi-darwin-arm64@0.23.0': - resolution: {integrity: sha512-DhkKC+trOfk3RLDvPXqGsrpWdVnLAMEVLUI59OuR9tdTcJeiABtbQx8VaXdbzvNxnbkoDnOqbFRE5D11Z7nerQ==} - cpu: [arm64] - os: [darwin] - '@cipherstash/protect-ffi-darwin-arm64@0.30.0': resolution: {integrity: sha512-MbUfH0em2ysQxEV8+ed6442Q/EPQbAh5p17p1p4+JLUgbQJwPvxP8gm0yjeVrYBuGnZodsMqZwgOgEGt7360Ig==} cpu: [arm64] os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.23.0': - resolution: {integrity: sha512-YuEn2RDHOaj9s8qDIX9cpQuBmsN2SZp/RjiNX72LxhV7JEDJuLSt0ySrl+k6MHoLiZotjkp7I1u6tq3vuLCC0Q==} - cpu: [x64] - os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.30.0': resolution: {integrity: sha512-GD2RXtjLvQaxWOPq2kbZEoNKTYWpDNTPQJ/tb9djpJF4RaR15g/jYhvk7Nqe2v3o6gre5RJBUpsaTQqC+G+L9A==} cpu: [x64] os: [darwin] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.23.0': - resolution: {integrity: sha512-I1kID2JqWnJUd0VHzNQo4gxeOAhEgzeXg3Fn0iDHnGKy+HDHd7+t/qEei9YLrV0wAXDnDFRhXXWRRVs8CxDzZA==} - cpu: [arm64] - os: [linux] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.30.0': resolution: {integrity: sha512-R/DWCDQDDx/hRksRueJLvqxyQCSpGOZIummKqcVvBNkIhF/6Aos1SZ+zXTHAZ4gvBlfu3ovnRfMMI6eAstZDog==} cpu: [arm64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.23.0': - resolution: {integrity: sha512-n4aCDK0os4iY1BQIHVVUBgt8WnfIb8R3gLXTRrTkMVug0dcoQ0ZZaL5ltIUgFGJG4bvfW8+7zWLRZ51CZkqKsQ==} - cpu: [x64] - os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.30.0': resolution: {integrity: sha512-uLi9kiKkJ9jUki75hgvLnDsDrJOj0RtZ8G0fVaKl4nnhZjCIVqpkn0EMlf0GOq2Bj8WiGSnca7kTGrhVkvgY9w==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-musl@0.23.0': - resolution: {integrity: sha512-62WG6ayFJ/1+M7W/AWGEDskLo6aHtr8PFoHoXkSTdhWf29RPb4+yU1pPNYVCitVWB1sdGs+lXSO6MFD4N6IIXw==} - cpu: [x64] - os: [linux] - '@cipherstash/protect-ffi-linux-x64-musl@0.30.0': resolution: {integrity: sha512-ZrPyxc+qi9u9LsyGYbdqoi6oMPeBBx+E+/uZuAzVtp115k8CsI8xlcrmNswQ6Z74EgvAUfLb1ykTn5/P9Ge0rw==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-win32-x64-msvc@0.23.0': - resolution: {integrity: sha512-z+jErHcPw1RwiwhSqqx/QzKqkk06gulh6YJl4TlSBPlJPjhR30TEcxQpQ2zf7kuv86JqsBRHv8UazLNePSiEww==} - cpu: [x64] - os: [win32] - '@cipherstash/protect-ffi-win32-x64-msvc@0.30.0': resolution: {integrity: sha512-xew3mH+jf9RlfuIjXQ7KFWTs44Wjur//Ed1U48yFK3zmsyzkZGYVwfN8zieKpKgKqvj844s5x/0kvFXtrJDB0A==} cpu: [x64] os: [win32] - '@cipherstash/protect-ffi@0.23.0': - resolution: {integrity: sha512-Ca8MKLrrumC561VoPDOhuUZcF8C8YenqO1Ig9hSJSRUB+jFeIJXeyn7glExsvKYWtxOx/pRub9FV8A0RyuPHMg==} - '@cipherstash/protect-ffi@0.30.0': resolution: {integrity: sha512-Xh8X/71ZOW6B6iEKPQlG4KrDgsKYZBw29ohsFnmMKOaTWVR7EodCE6MpSRDJ4uc+zYftp3QWsvUDEFz9gDpg6w==} @@ -2107,9 +1980,6 @@ packages: '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} - '@stricli/core@1.2.9': - resolution: {integrity: sha512-vKniyS+0GtarIaUpMRMQ5+Ck9mXeaBqdN/7otznkBTWU0EXfkMJAHt2QdOCmOudR9eciQZQf09DmYm14yfPWzw==} - '@supabase/auth-js@2.110.2': resolution: {integrity: sha512-Qj7a6EDP+AMMQFWqGv+qFa8r6re//dk+qQI5bA0KK+PZmnI3JPu97TDeNt6SMiQ2FkklP79hP2yDFYSnA989OA==} engines: {node: '>=22.0.0'} @@ -4203,53 +4073,24 @@ snapshots: '@cipherstash/eql@3.0.2': {} - '@cipherstash/protect-ffi-darwin-arm64@0.23.0': - optional: true - '@cipherstash/protect-ffi-darwin-arm64@0.30.0': optional: true - '@cipherstash/protect-ffi-darwin-x64@0.23.0': - optional: true - '@cipherstash/protect-ffi-darwin-x64@0.30.0': optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.23.0': - optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.30.0': optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.23.0': - optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.30.0': optional: true - '@cipherstash/protect-ffi-linux-x64-musl@0.23.0': - optional: true - '@cipherstash/protect-ffi-linux-x64-musl@0.30.0': optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.23.0': - optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.30.0': optional: true - '@cipherstash/protect-ffi@0.23.0': - dependencies: - '@neon-rs/load': 0.1.82 - optionalDependencies: - '@cipherstash/protect-ffi-darwin-arm64': 0.23.0 - '@cipherstash/protect-ffi-darwin-x64': 0.23.0 - '@cipherstash/protect-ffi-linux-arm64-gnu': 0.23.0 - '@cipherstash/protect-ffi-linux-x64-gnu': 0.23.0 - '@cipherstash/protect-ffi-linux-x64-musl': 0.23.0 - '@cipherstash/protect-ffi-win32-x64-msvc': 0.23.0 - '@cipherstash/protect-ffi@0.30.0': dependencies: '@neon-rs/load': 0.1.82 @@ -5134,8 +4975,6 @@ snapshots: '@standard-schema/spec@1.1.0': {} - '@stricli/core@1.2.9': {} - '@supabase/auth-js@2.110.2': dependencies: tslib: 2.8.1 diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index c513a28ed..e8bbc8b49 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -10,7 +10,6 @@ const ALLOWLISTED_PATHS = new Set([ 'packages/wizard/src/lib/detect.ts', // npm row of the PM table 'packages/cli/src/commands/init/utils.ts', // runnerCommand `case 'npm'` 'packages/cli/src/commands/init/lib/setup-prompt.ts', // execCommand `case 'npm':` switch - 'packages/protect/src/bin/runner.ts', // Pre-allowlisted: helper for Task 11 'packages/drizzle/src/bin/runner.ts', // Pre-allowlisted: helper for Task 13 'scripts/lint-no-hardcoded-runners.mjs', // this script's own docs ]) From c40854ba26055472fbd63b8bc10f0b4589d75688 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 17:21:29 +1000 Subject: [PATCH 002/123] docs: add EQL v2 removal PR1 plan --- .../2026-07-22-eql-v2-removal-pr1-plan.md | 70 +++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md diff --git a/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md b/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md new file mode 100644 index 000000000..f37c3219b --- /dev/null +++ b/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md @@ -0,0 +1,70 @@ +# EQL v2 removal — PR 1 step-plan (delete published v2 packages) + +Executes PR 1 of `docs/plans/2026-07-22-eql-v2-final-removal-design.md`. +Branch: `feat/remove-eql-v2-pr1-delete-packages` (worktree). Scoped against `origin/main` = `6ce53817`. + +## Goal +Delete the closed v2-only dependency chain — `@cipherstash/protect-dynamodb` → +`@cipherstash/protect` → `@cipherstash/schema` — and remove every reference so the +build, lockfile, and changeset state stay consistent. Mergeable in isolation. + +## Verified scope (live-code survey, not just design line-counts) +Closed-chain claim CONFIRMED for code imports: nothing outside the three imports them, +and `@cipherstash/stack` depends only on `@cipherstash/protect-ffi` (a different, external +package), not on any of the three. + +Corrections vs. the design's PR-1 paragraph: +- `.changeset/config.json` — the three are NOT in the `fixed` group and `ignore` is `[]`. + No config edit required (design assumed a fixed-group removal). +- `pnpm-workspace.yaml` uses globs (`packages/*`), no explicit entries to remove. +- No `tsconfig` `references` anywhere — nothing to unpick. +- Extra build blockers the design folded under "root config": `e2e/package.json` dep edge + and root `package.json` `build:js` turbo filter. +- Extra changeset-state fixes: pending `schema-stevec-standard-pin.md` targets the doomed + `@cipherstash/schema`; `pre.json` pins all three in `initialVersions`. + +## Steps + +### 1. Delete the three package directories +- `rm -rf packages/protect-dynamodb packages/protect packages/schema` + +### 2. Fix build blockers (dangling references that break compile/CI) +- `e2e/package.json` — remove the `"@cipherstash/protect": "workspace:*"` dependency line. +- root `package.json` — `build:js`: drop `--filter './packages/protect'`, keep `./packages/nextjs`. + +### 3. Clean stale (non-breaking) references +- `scripts/lint-no-hardcoded-runners.mjs` — remove the `packages/protect/src/bin/runner.ts` + allowlist entry (verify the script doesn't assert the path exists — if it does, this is + actually a blocker). +- `packages/nextjs/package.json` — description references `@cipherstash/protect`; repoint to + `@cipherstash/stack`. +- `skills/stash-drizzle/SKILL.md:38` — inspect the `@cipherstash/protect` mention; fix only if + the deletion made it wrong (a historical note about the legacy protect-based package may stay). + +### 4. Changeset / RC-mode housekeeping +- Delete `.changeset/schema-stevec-standard-pin.md` (only target is the deleted `@cipherstash/schema`; + already consumed in a prior rc per `pre.json`). +- `.changeset/pre.json` — remove the three from `initialVersions`; remove `schema-stevec-standard-pin` + from the `changesets` array (keeps it consistent with the deleted file). +- Add deletion-notice changeset `.changeset/remove-eql-v2-packages.md`: + `'@cipherstash/stack': patch` (successor surface for all three; group already major via + `stack-1-0-0-rc`), prose body naming each removed package and its migration path + (`@cipherstash/protect` → `@cipherstash/stack`; `@cipherstash/schema` → `@cipherstash/stack/schema`; + `@cipherstash/protect-dynamodb` → `@cipherstash/stack/dynamodb` `encryptedDynamoDB`). + Follows the `remove-legacy-drizzle-package.md` precedent. + +### 5. Meta-file honesty (trim what described the removed packages) +- `SECURITY.md` — drop the three rows from the package list. +- `AGENTS.md` — Repository Layout entries (protect, schema, protect-dynamodb) + prose mentions; + keep the "maintained implementation is `packages/stack/src/dynamodb`" guidance. + +### 6. Regenerate lockfile +- `pnpm install` (updates `pnpm-lock.yaml` for removed packages + e2e edge). CI is + `--frozen-lockfile`, so the committed lockfile must match. + +## Verification (green gate before commit) +- `pnpm changeset status` — no changeset references a missing package. +- `pnpm run build` — whole-repo turbo build; proves no dangling import/reference. +- `pnpm run code:check` — biome, error-free. +- `git grep -n "@cipherstash/protect\b\|@cipherstash/schema\|@cipherstash/protect-dynamodb"` — + only intentional survivors (e.g. migration-path prose, `protect-ffi` which is unrelated). From 24258a41a4ae7b7789c215a7829f58a00162d4a8 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 17:34:35 +1000 Subject: [PATCH 003/123] docs: correct plan changeset targets and stale-ref grep pattern - Note the intentional @cipherstash/nextjs patch (package.json description edit) - Use PCRE negative lookahead so the stale-reference check excludes protect-ffi --- docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md b/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md index f37c3219b..54e53c516 100644 --- a/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md +++ b/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md @@ -48,7 +48,9 @@ Corrections vs. the design's PR-1 paragraph: from the `changesets` array (keeps it consistent with the deleted file). - Add deletion-notice changeset `.changeset/remove-eql-v2-packages.md`: `'@cipherstash/stack': patch` (successor surface for all three; group already major via - `stack-1-0-0-rc`), prose body naming each removed package and its migration path + `stack-1-0-0-rc`) and `'@cipherstash/nextjs': patch` (its `package.json` description changes + from `@cipherstash/protect` to `@cipherstash/stack` — a published-metadata edit), prose body + naming each removed package and its migration path (`@cipherstash/protect` → `@cipherstash/stack`; `@cipherstash/schema` → `@cipherstash/stack/schema`; `@cipherstash/protect-dynamodb` → `@cipherstash/stack/dynamodb` `encryptedDynamoDB`). Follows the `remove-legacy-drizzle-package.md` precedent. @@ -66,5 +68,7 @@ Corrections vs. the design's PR-1 paragraph: - `pnpm changeset status` — no changeset references a missing package. - `pnpm run build` — whole-repo turbo build; proves no dangling import/reference. - `pnpm run code:check` — biome, error-free. -- `git grep -n "@cipherstash/protect\b\|@cipherstash/schema\|@cipherstash/protect-dynamodb"` — - only intentional survivors (e.g. migration-path prose, `protect-ffi` which is unrelated). +- `git grep -nP "@cipherstash/protect(?!-ffi)|@cipherstash/schema|@cipherstash/protect-dynamodb"` — + only intentional survivors (e.g. migration-path prose). The `(?!-ffi)` lookahead (PCRE, hence + `-P`) excludes the unrelated `@cipherstash/protect-ffi`; a plain `\b` would not, since a word + boundary sits between `protect` and `-ffi`. From 44714711bba1e269483228fc2c733d0854b849b9 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 17:33:04 +1000 Subject: [PATCH 004/123] refactor(migrate): drop EQL v2 from the domain-type classifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the `eql_v2_encrypted → 2` branch from `classifyEqlDomain`, so the migrate domain-type resolution (`detectColumnEqlVersion`, `listEncryptedColumns`, `resolveEncryptedColumn`) recognises only the self-describing `eql_v3_*` domains. v3 is the sole generation this workspace authors and backfills; a legacy v2 column's version is carried by the manifest's recorded `eqlVersion` (the CLI status renderers already fall back to it), so v2 status output is unchanged. This drops v2 *classification*, not the v2 read path — existing v2 ciphertext stays decryptable via `@cipherstash/stack`. `EqlVersion` keeps its `2` member for manifest-sourced legacy values and the exported function signatures are unchanged. Tests in `version.test.ts` are updated to assert v2 domains are no longer classified (excluded from `listEncryptedColumns`, `null` from the detectors). Decision 6 guard: `classifyEqlDomain` is a source-column classifier for backfill planning and read-only CLI status display — no decrypt/round-trip consumes its `2` result — so dropping v2 here is safe. NOTE: `packages/migrate/src/eql.ts` (the `eql_v2.*` config-lifecycle wrappers) is NOT deleted in this PR. Although it carries no decrypt path, it is hard- imported by the CLI's v2 cut-over/config commands (`encrypt cutover`, `db activate`, `db push`); deleting it would break the CLI build, which is out of this migrate-scoped PR. Its removal must be sequenced with the CLI v2-command removal. PR 2 of the EQL v2 final removal (#707). --- .../remove-eql-v2-migrate-classifier.md | 17 ++++++++++ .../migrate/src/__tests__/version.test.ts | 34 ++++++++++++++----- packages/migrate/src/version.ts | 15 +++++--- 3 files changed, 53 insertions(+), 13 deletions(-) create mode 100644 .changeset/remove-eql-v2-migrate-classifier.md diff --git a/.changeset/remove-eql-v2-migrate-classifier.md b/.changeset/remove-eql-v2-migrate-classifier.md new file mode 100644 index 000000000..45d163efa --- /dev/null +++ b/.changeset/remove-eql-v2-migrate-classifier.md @@ -0,0 +1,17 @@ +--- +'@cipherstash/migrate': patch +--- + +Drop EQL v2 from the domain-type classifier. `classifyEqlDomain` (and the +`detectColumnEqlVersion` / `listEncryptedColumns` / `resolveEncryptedColumn` +resolution built on it) no longer recognise the legacy `eql_v2_encrypted` +domain — v3 is the sole generation this workspace authors and backfills, so a +column's version is now determined solely from its self-describing `eql_v3_*` +domain type. A legacy v2 column's version is carried by the manifest's recorded +`eqlVersion` instead (the CLI's `encrypt status` / `status` renderers already +fall back to it), so status output is unchanged for v2 columns. + +This removes v2 *classification*, not the v2 read path: existing v2 ciphertext +remains decryptable through `@cipherstash/stack`. `EqlVersion` keeps its `2` +member for manifest-sourced legacy values; the exported function signatures are +unchanged. diff --git a/packages/migrate/src/__tests__/version.test.ts b/packages/migrate/src/__tests__/version.test.ts index b6cfdb9d6..96e5cfbb6 100644 --- a/packages/migrate/src/__tests__/version.test.ts +++ b/packages/migrate/src/__tests__/version.test.ts @@ -18,8 +18,13 @@ function mockClient(rows: Array>) { } describe('classifyEqlDomain', () => { - it('maps eql_v2_encrypted to 2', () => { - expect(classifyEqlDomain('eql_v2_encrypted')).toBe(2) + it('no longer classifies eql_v2_encrypted — v3 is the sole authored generation', () => { + // The v2 branch was removed: this workspace authors/backfills v3 only, so + // the domain classifier recognises `eql_v3_*` alone. A legacy v2 column's + // version now comes from the manifest's recorded `eqlVersion`, not here. + // (Existing v2 ciphertext stays decryptable — only classification of v2 as + // an authorable generation is dropped.) + expect(classifyEqlDomain('eql_v2_encrypted')).toBeNull() }) it('maps any eql_v3_* domain to 3', () => { @@ -46,10 +51,20 @@ describe('classifyEqlDomain', () => { describe('detectColumnEqlVersion', () => { it('classifies from the domain type', async () => { - const { client } = mockClient([{ domain_name: 'eql_v2_encrypted' }]) + const { client } = mockClient([{ domain_name: 'eql_v3_text_search' }]) expect( await detectColumnEqlVersion(client, 'users', 'email_encrypted'), - ).toBe(2) + ).toBe(3) + }) + + it('returns null for a legacy eql_v2_encrypted domain (v2 no longer classified)', async () => { + // A v2 column still exists physically, but the classifier no longer treats + // it as an authorable EQL generation — callers fall back to the manifest's + // recorded eqlVersion. + const { client } = mockClient([{ domain_name: 'eql_v2_encrypted' }]) + expect( + await detectColumnEqlVersion(client, 'users', 'ssn_encrypted'), + ).toBeNull() }) it('returns null for a plaintext column (base type, not a domain)', async () => { @@ -88,16 +103,17 @@ describe('detectColumnEqlVersion', () => { }) describe('listEncryptedColumns', () => { - it('returns only EQL-domain columns, classified', async () => { + it('returns only EQL v3-domain columns, classified (legacy v2 domains excluded)', async () => { const { client } = mockClient([ { column: 'id', domain_name: 'int8' }, { column: 'email', domain_name: 'text' }, { column: 'email_enc', domain_name: 'eql_v3_text_search' }, + // A legacy v2 column is no longer classified as EQL, so it drops out of + // the encrypted-column listing entirely. { column: 'ssn_encrypted', domain_name: 'eql_v2_encrypted' }, ]) expect(await listEncryptedColumns(client, 'users')).toEqual([ { column: 'email_enc', domain: 'eql_v3_text_search', version: 3 }, - { column: 'ssn_encrypted', domain: 'eql_v2_encrypted', version: 2 }, ]) }) }) @@ -153,10 +169,10 @@ describe('pickEncryptedColumn', () => { }) it('never resolves the plaintext column to itself', () => { - // Post-cutover v2: `email` itself carries the v2 domain. It is the - // ciphertext, not a counterpart of itself. + // A column that IS the plaintext argument cannot be its own encrypted + // counterpart, even when it's the table's sole EQL-domain column. expect( - pickEncryptedColumn([col('email', 'eql_v2_encrypted', 2)], 'email'), + pickEncryptedColumn([col('email', 'eql_v3_encrypted')], 'email'), ).toBeNull() }) diff --git a/packages/migrate/src/version.ts b/packages/migrate/src/version.ts index dccfe7e35..60e1d9d41 100644 --- a/packages/migrate/src/version.ts +++ b/packages/migrate/src/version.ts @@ -17,7 +17,7 @@ export type EqlVersion = 2 | 3 export interface EncryptedColumnInfo { /** The column's name, exactly as Postgres reports it. */ column: string - /** The EQL domain name, e.g. `eql_v2_encrypted` or `eql_v3_text_search`. */ + /** The EQL domain name, e.g. `eql_v3_text_search` or `eql_v3_integer_ord`. */ domain: string version: EqlVersion } @@ -30,12 +30,19 @@ export interface EncryptedColumnInfo { * rule lives, and why detection never relies on column NAMES: the * `_encrypted` naming is a convention, neither enforced nor required. * - * - `eql_v2_encrypted` → 2 * - `eql_v3_*` (e.g. `eql_v3_text_search`, `eql_v3_integer_ord`) → 3 - * - anything else → `null` (not an EQL column) + * - anything else → `null` (not an EQL v3 column) + * + * v3 is the sole generation this workspace authors and backfills, so the + * classifier only recognises `eql_v3_*` domains. A legacy `eql_v2_encrypted` + * column is therefore no longer classified here — its version is carried by + * the manifest's recorded `eqlVersion` instead (see the `?? eqlVersion` + * fallbacks in the CLI's `encrypt status` / `status` renderers). Existing v2 + * ciphertext stays decryptable; only its *classification as an authorable + * generation* is dropped. `EqlVersion` keeps the `2` member for those + * manifest-sourced legacy values. */ export function classifyEqlDomain(domain: string): EqlVersion | null { - if (domain === 'eql_v2_encrypted') return 2 // Underscore included: a bare `startsWith('eql_v3')` would also claim // hypothetical future generations like `eql_v30_*`. if (domain.startsWith('eql_v3_')) return 3 From 8832d3588e06f68e2a80313cfa2835988a277506 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:28:18 +1000 Subject: [PATCH 005/123] feat(stack)!: EQL v3 audit-on-decrypt + collapse EncryptionV3 into Encryption PR 3 of the EQL v2 removal (#707). Makes EQL v3 the sole generation the client authors and writes, while preserving the minimal v2 READ path on the core client and through the DynamoDB adapter. - Add MappedDecryptOperation: the typed client's decryptModel/bulkDecryptModels are now audit-chainable (.audit()/.withLockContext()) instead of a bare Promise, restoring audited decrypt including through encryptedDynamoDB. - Collapse EncryptionV3 into an overloaded Encryption; EncryptionV3 is now a deprecated, type-identical alias. An explicit config.eqlVersion is honored (the migration escape hatch is retained). - BREAKING: remove the DynamoDB v2 WRITE overloads (encryptModel/bulkEncryptModels); decrypt still reads existing v2 items. - Deprecate (retain) ClientConfig.eqlVersion and the ./schema v2 builders for legacy v2 read/migrate; siblings still consume them, so full removal is deferred to a later PR. - Tests: flip the audit-surface type assertions, add runtime audit-forwarding tests (both chaining orders), and v2-read acceptance integration tests (core + DynamoDB). - Update stash-dynamodb and stash-encryption skills and AGENTS.md. Changesets: @cipherstash/stack minor (audit-on-decrypt), @cipherstash/stack major (DynamoDB v2-write removal), stash patch (skills). --- .changeset/stack-audit-on-decrypt.md | 31 ++++ .changeset/stack-dynamodb-v2-write-removal.md | 17 +++ .changeset/stack-skills-eql-v3-audit.md | 13 ++ AGENTS.md | 4 +- .../decrypt-audit-forwarding.test.ts | 140 ++++++++++++++++++ .../dynamodb/client-compat.test-d.ts | 58 +++++--- .../dynamodb/resolve-decrypt.test.ts | 61 ++++++++ .../stack/__tests__/init-strategy.test.ts | 12 +- .../stack/__tests__/typed-client-v3.test-d.ts | 19 +++ .../stack/__tests__/typed-client-v3.test.ts | 30 +++- .../v3-matrix/matrix-audit.test-d.ts | 27 ++-- .../v3-matrix/matrix-lock-context.test.ts | 8 +- .../schema-v3-client.integration.test.ts | 13 +- .../v2-decrypt-compat.integration.test.ts | 94 ++++++++++++ packages/stack/src/dynamodb/index.ts | 8 +- packages/stack/src/dynamodb/types.ts | 31 +--- packages/stack/src/encryption/index.ts | 51 ++++++- .../encryption/operations/mapped-decrypt.ts | 92 ++++++++++++ packages/stack/src/encryption/v3.ts | 82 +++++----- packages/stack/src/schema/index.ts | 24 +++ packages/stack/src/types.ts | 6 + skills/stash-dynamodb/SKILL.md | 57 +++---- skills/stash-encryption/SKILL.md | 54 +++---- 23 files changed, 752 insertions(+), 180 deletions(-) create mode 100644 .changeset/stack-audit-on-decrypt.md create mode 100644 .changeset/stack-dynamodb-v2-write-removal.md create mode 100644 .changeset/stack-skills-eql-v3-audit.md create mode 100644 packages/stack/__tests__/decrypt-audit-forwarding.test.ts create mode 100644 packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts create mode 100644 packages/stack/src/encryption/operations/mapped-decrypt.ts diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md new file mode 100644 index 000000000..d832b793d --- /dev/null +++ b/.changeset/stack-audit-on-decrypt.md @@ -0,0 +1,31 @@ +--- +'@cipherstash/stack': minor +--- + +The typed EQL v3 client's `decryptModel` / `bulkDecryptModels` are now +audit-chainable. They return a chainable operation (a `MappedDecryptOperation`) +instead of a bare `Promise>`, so you can attach audit metadata and a +lock context before awaiting: + +```typescript +await client + .decryptModel(item, table) + .audit({ metadata: { requestId } }) + .withLockContext({ identityClaim: ["sub"] }) +``` + +Both chaining orders forward the metadata to ZeroKMS, and the `Date` +reconstruction still applies to the successful result. Await-only call sites are +unchanged — the operation is still thenable to the same `Result`. This restores +audited decrypt through the DynamoDB adapter (`encryptedDynamoDB(...).decryptModel`) +for a v3 client, which previously had nowhere to carry the metadata. + +`EncryptionV3` is now a deprecated, type-identical alias of `Encryption`: +`Encryption` is overloaded so an array of concrete EQL v3 tables yields the same +strongly-typed client. Use `Encryption` for new code. As part of this collapse +`EncryptionV3` no longer independently pins the wire format — like `Encryption`, +it now honours an explicit `config.eqlVersion` (the retained migration escape +hatch). The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 +builders remain available (now marked `@deprecated`) for reading and migrating +legacy v2 data; the client authors EQL v3 only. Their full removal is deferred to +a later PR. diff --git a/.changeset/stack-dynamodb-v2-write-removal.md b/.changeset/stack-dynamodb-v2-write-removal.md new file mode 100644 index 000000000..0f5058e34 --- /dev/null +++ b/.changeset/stack-dynamodb-v2-write-removal.md @@ -0,0 +1,17 @@ +--- +'@cipherstash/stack': major +--- + +**Breaking (DynamoDB adapter):** `encryptedDynamoDB(...).encryptModel` and +`bulkEncryptModels` no longer accept an EQL v2 table — write is EQL v3 only. The +v2 write type overloads have been removed, narrowing encrypt to `AnyV3Table`. + +**Decrypt still reads existing v2 items.** `decryptModel` / `bulkDecryptModels` +continue to accept an EQL v2 table (`encryptedColumn` / `encryptedField` from +`@cipherstash/stack/schema`), so previously stored v2 DynamoDB items remain +readable — the adapter keeps its v2 envelope reconstruction. Only the v2 write +surface is gone. + +Migrate v2 write call sites to an EQL v3 table (`encryptedTable` + `types.*` from +`@cipherstash/stack/eql/v3`). To keep reading old data, pass the v2 table to the +decrypt methods. diff --git a/.changeset/stack-skills-eql-v3-audit.md b/.changeset/stack-skills-eql-v3-audit.md new file mode 100644 index 000000000..f63f9b085 --- /dev/null +++ b/.changeset/stack-skills-eql-v3-audit.md @@ -0,0 +1,13 @@ +--- +'stash': patch +--- + +Skills refresh for the EQL v3 collapse (ships in the `stash` tarball): + +- `stash-dynamodb`: audited decrypt now works on the typed client — + `client.decryptModel(item, table).audit({ … })` — so the old "use + `Encryption({ config: { eqlVersion: 3 } })` for audited decrypts" caveat is + removed. Encrypt/write is EQL v3 only; decrypt still reads existing v2 items. +- `stash-encryption`: canonical examples use `Encryption` (with `EncryptionV3` + noted as a deprecated alias); the DynamoDB notes state encrypt is v3-only while + decrypt still reads v2. diff --git a/AGENTS.md b/AGENTS.md index 6da0a2859..d88560f4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -142,8 +142,8 @@ Three rules to remember when editing CI or pnpm config: ## Key Concepts and APIs -- **Initialization**: `EncryptionV3({ schemas })` returns the typed EQL v3 client. (`Encryption({ schemas })` is the legacy v2 client.) Provide at least one `encryptedTable`. -- **Schema (EQL v3, the documented approach)**: Define tables/columns with `encryptedTable` and the `types.*` concrete-domain factories from `@cipherstash/stack/eql/v3` (`types.TextSearch`, `types.IntegerOrd`, `types.Json`, …) — each domain's query capabilities are fixed by its type; there are no chainable capability tuners. Build the client with `EncryptionV3` from `@cipherstash/stack/v3`. (Legacy EQL v2 — `Encryption` + `encryptedColumn(...).equality().freeTextSearch().orderAndRange().searchableJson()` from `@cipherstash/stack/schema` — remains for existing deployments. DynamoDB accepts both v3 and v2 tables; nested fields work in both — v2 via `encryptedField` groups, v3 via a flat dotted column path (`'profile.ssn': types.TextEq(...)`), since `EncryptedV3TableColumn` admits no nested groups.) +- **Initialization**: `Encryption({ schemas })` is the single client factory. It is overloaded: an array of concrete EQL v3 tables yields the strongly-typed EQL v3 client; a v2/loose schema set yields the nominal client. `EncryptionV3` (from `@cipherstash/stack/v3`) is a deprecated, type-identical alias of `Encryption`. Provide at least one `encryptedTable`. +- **Schema (EQL v3, the documented approach)**: Define tables/columns with `encryptedTable` and the `types.*` concrete-domain factories from `@cipherstash/stack/eql/v3` (`types.TextSearch`, `types.IntegerOrd`, `types.Json`, …) — each domain's query capabilities are fixed by its type; there are no chainable capability tuners. Build the client with `Encryption` (import `encryptedTable`/`types` from `@cipherstash/stack/v3`). The client authors EQL v3 only; the `config.eqlVersion` field and the `@cipherstash/stack/schema` EQL v2 builders (`encryptedColumn(...).equality().freeTextSearch().orderAndRange().searchableJson()`) remain (now `@deprecated`) for reading/migrating legacy v2 data. `decrypt`/`decryptModel` read both v2 and v3 payloads. DynamoDB **writes EQL v3 only; decrypt still reads existing v2 items**; nested fields work in both — v2 via `encryptedField` groups (read only), v3 via a flat dotted column path (`'profile.ssn': types.TextEq(...)`), since `EncryptedV3TableColumn` admits no nested groups. - **Operations** (all return Result-like objects and support chaining `.withLockContext(lockContext)` and `.audit()` when applicable): - `encrypt(plaintext, { table, column })` - `decrypt(encryptedPayload)` diff --git a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts new file mode 100644 index 000000000..9f3c09288 --- /dev/null +++ b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts @@ -0,0 +1,140 @@ +/** + * Runtime proof that audit metadata attached to the typed EQL v3 client's + * `decryptModel` / `bulkDecryptModels` reaches ZeroKMS — the core half of + * acceptance #2b. Before PR 3 the typed client `await`ed the underlying decrypt + * and mapped the value, which collapsed the chain and dropped `.audit()`; the + * `MappedDecryptOperation` wrapper restores it. + * + * The metadata surfaces as `unverifiedContext` on the mocked protect-ffi + * `decryptBulk` call. Both chaining orders are covered (Risk R3): + * `.audit().withLockContext()` and `.withLockContext().audit()` must each + * forward the metadata (and the lock context's identity claim). + * + * Credential-free: protect-ffi is mocked, so there is no ZeroKMS round-trip. + */ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { Encryption } from '@/index' + +// A protect-ffi-shaped encrypted payload so the SDK's `isEncryptedPayload` +// check detects the model field as encrypted and routes it to `decryptBulk`. +const enc = () => ({ v: 2, i: { t: 'users', c: 'email' }, c: 'ciphertext' }) + +vi.mock('@cipherstash/protect-ffi', () => ({ + newClient: vi.fn(async () => ({ __mock: 'client' })), + encrypt: vi.fn(async () => enc()), + decrypt: vi.fn(async () => 'decrypted'), + encryptBulk: vi.fn(async (_c: unknown, opts: { plaintexts: unknown[] }) => + opts.plaintexts.map(enc), + ), + decryptBulk: vi.fn(async (_c: unknown, opts: { ciphertexts: unknown[] }) => + opts.ciphertexts.map(() => 'decrypted'), + ), + decryptBulkFallible: vi.fn( + async (_c: unknown, opts: { ciphertexts: unknown[] }) => + opts.ciphertexts.map(() => ({ data: 'decrypted' })), + ), + encryptQuery: vi.fn(async () => enc()), + encryptQueryBulk: vi.fn(async (_c: unknown, opts: { queries: unknown[] }) => + opts.queries.map(enc), + ), +})) + +import * as ffi from '@cipherstash/protect-ffi' +// Imported after the mock so the v3 table builder is available; `Encryption` +// returns the typed client for an all-v3 schema set. +import { encryptedTable, types } from '@/encryption/v3' + +const users = encryptedTable('users', { + email: types.TextEq('email'), +}) + +const IDENTITY_CLAIM = { identityClaim: ['sub'] } + +// biome-ignore lint/suspicious/noExplicitAny: test helper unwraps Result +function unwrap(result: any) { + if (result.failure) { + throw new Error(`operation failed: ${result.failure.message}`) + } + return result.data +} + +/** Options the FFI decrypt was last called with (second arg). */ +// biome-ignore lint/suspicious/noExplicitAny: reading recorded mock args +const lastDecryptOpts = () => (ffi.decryptBulk as any).mock.calls.at(-1)[1] + +/** The lock context is carried per-ciphertext, not on the top-level opts. */ +const lastCiphertextLockContext = () => + lastDecryptOpts().ciphertexts[0]?.lockContext + +let client: Awaited>> + +beforeEach(async () => { + vi.clearAllMocks() + process.env.CS_WORKSPACE_CRN = 'crn:ap-southeast-2.aws:test-workspace' + client = await Encryption({ schemas: [users] }) +}) + +describe('typed v3 client: audit metadata forwards through decryptModel', () => { + it('forwards .audit({ metadata }) as unverifiedContext (no lock context)', async () => { + unwrap( + await client + .decryptModel({ email: enc() }, users) + .audit({ metadata: { sub: 'u1' } }), + ) + + expect(ffi.decryptBulk).toHaveBeenCalledTimes(1) + expect(lastDecryptOpts().unverifiedContext).toEqual({ sub: 'u1' }) + }) + + it('forwards metadata AND identity claim with .audit().withLockContext()', async () => { + unwrap( + await client + .decryptModel({ email: enc() }, users) + .audit({ metadata: { m: 1 } }) + .withLockContext(IDENTITY_CLAIM), + ) + + expect(ffi.decryptBulk).toHaveBeenCalledTimes(1) + const opts = lastDecryptOpts() + expect(opts.unverifiedContext).toEqual({ m: 1 }) + expect(lastCiphertextLockContext()).toEqual(IDENTITY_CLAIM) + }) + + it('forwards metadata AND identity claim with .withLockContext().audit()', async () => { + unwrap( + await client + .decryptModel({ email: enc() }, users) + .withLockContext(IDENTITY_CLAIM) + .audit({ metadata: { m: 2 } }), + ) + + expect(ffi.decryptBulk).toHaveBeenCalledTimes(1) + const opts = lastDecryptOpts() + expect(opts.unverifiedContext).toEqual({ m: 2 }) + expect(lastCiphertextLockContext()).toEqual(IDENTITY_CLAIM) + }) + + it('forwards metadata via the lockContext argument (no chaining)', async () => { + unwrap( + await client + .decryptModel({ email: enc() }, users, IDENTITY_CLAIM) + .audit({ metadata: { m: 3 } }), + ) + + expect(ffi.decryptBulk).toHaveBeenCalledTimes(1) + const opts = lastDecryptOpts() + expect(opts.unverifiedContext).toEqual({ m: 3 }) + expect(lastCiphertextLockContext()).toEqual(IDENTITY_CLAIM) + }) + + it('forwards .audit({ metadata }) on bulkDecryptModels', async () => { + unwrap( + await client + .bulkDecryptModels([{ email: enc() }], users) + .audit({ metadata: { b: 4 } }), + ) + + expect(ffi.decryptBulk).toHaveBeenCalledTimes(1) + expect(lastDecryptOpts().unverifiedContext).toEqual({ b: 4 }) + }) +}) diff --git a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts index 2c322357a..971b641a2 100644 --- a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts +++ b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts @@ -79,27 +79,25 @@ describe('encryptedDynamoDB accepts both client shapes without a cast', () => { const dynamo = encryptedDynamoDB({ encryptionClient: typedClient }) -describe('all four methods accept both a v3 and a v2 table', () => { - it('encryptModel', () => { +describe('write is EQL v3 only; read accepts both a v3 and a v2 table', () => { + it('encryptModel accepts a v3 table and rejects a v2 table', () => { expectTypeOf(dynamo.encryptModel).toBeCallableWith( { pk: 'a', email: 'a@b.com' }, usersV3, ) - expectTypeOf(dynamo.encryptModel).toBeCallableWith( - { pk: 'a', email: 'a@b.com' }, - usersV2, - ) + // Write is EQL v3 only — the v2 write overload was removed, so a v2 table is + // rejected on encrypt (decrypt below still accepts it). + // @ts-expect-error - encryptModel no longer accepts an EQL v2 table + dynamo.encryptModel({ pk: 'a', email: 'a@b.com' }, usersV2) }) - it('bulkEncryptModels', () => { + it('bulkEncryptModels accepts a v3 table and rejects a v2 table', () => { expectTypeOf(dynamo.bulkEncryptModels).toBeCallableWith( [{ pk: 'a', email: 'a@b.com' }], usersV3, ) - expectTypeOf(dynamo.bulkEncryptModels).toBeCallableWith( - [{ pk: 'a', email: 'a@b.com' }], - usersV2, - ) + // @ts-expect-error - bulkEncryptModels no longer accepts an EQL v2 table + dynamo.bulkEncryptModels([{ pk: 'a', email: 'a@b.com' }], usersV2) }) it('decryptModel', () => { @@ -332,18 +330,6 @@ describe('a required-nullable v3 column keeps __source required', () => { }) }) -describe('the v2 overload still returns the input model', () => { - it('keeps an existing v2 caller compiling unchanged', async () => { - const result = await dynamo.encryptModel<{ pk: string; email?: string }>( - { pk: 'a', email: 'a@b.com' }, - usersV2, - ) - if (result.failure) throw new Error(result.failure.message) - - expectTypeOf(result.data).toEqualTypeOf<{ pk: string; email?: string }>() - }) -}) - describe('operations chain and resolve', () => { it('.audit() returns the operation so it stays chainable', () => { const op = dynamo.encryptModel({ pk: 'a', email: 'a@b.com' }, usersV3) @@ -352,6 +338,32 @@ describe('operations chain and resolve', () => { >() }) + it('.audit() is chainable on decryptModel and returns the operation', () => { + // The DynamoDB decrypt op is chainable; audit metadata now forwards to the + // underlying client decrypt regardless of client shape (see + // resolve-decrypt.test.ts / decrypt-audit-forwarding.test.ts for the runtime + // proof). Here we lock the type-level surface. + const op = dynamo.decryptModel({ pk: 'a', email__source: 'ct' }, usersV3) + expectTypeOf(op).toHaveProperty('audit') + expectTypeOf(op.audit({ metadata: { sub: 'u1' } })).toEqualTypeOf< + typeof op + >() + }) + + it('awaiting decryptModel yields a discriminated Result', async () => { + const result = await dynamo.decryptModel( + { pk: 'a', email__source: 'ct' }, + usersV3, + ) + + if (result.failure) { + expectTypeOf(result.failure.message).toEqualTypeOf() + return + } + + expectTypeOf(result.data.email).toEqualTypeOf() + }) + it('awaiting yields a discriminated Result', async () => { const result = await dynamo.encryptModel( { pk: 'a', email: 'a@b.com' }, diff --git a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts index f742afb9d..8d5eb2b1b 100644 --- a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts +++ b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts @@ -5,8 +5,12 @@ * reachable through a live ZeroKMS decrypt; these move that assurance onto the * pure CI lane. No credentials, no network. */ +import type { Result } from '@byteslice/result' import { afterEach, describe, expect, it, vi } from 'vitest' import { resolveDecryptResult, throwPreservingCode } from '@/dynamodb/helpers' +import { EncryptionOperation } from '@/encryption/operations/base-operation' +import { MappedDecryptOperation } from '@/encryption/operations/mapped-decrypt' +import { type EncryptionError, EncryptionErrorTypes } from '@/errors' import { logger } from '@/utils/logger' // The metadata-drop tests `vi.spyOn(logger, 'debug')` — the same shared singleton @@ -92,6 +96,63 @@ describe('resolveDecryptResult', () => { spy.mockRestore() } }) + + it('forwards audit metadata through a MappedDecryptOperation and applies its map', async () => { + // The typed EQL v3 client returns a `MappedDecryptOperation` on decrypt. + // resolveDecryptResult sees its `.audit()` and chains it; the wrapper + // forwards the metadata to the underlying op (whose `execute` reads it) and + // maps the successful result. This is the DynamoDB half of acceptance #2b. + let seenMetadata: Record | undefined + class Underlying extends EncryptionOperation<{ v: number }> { + override async execute(): Promise< + Result<{ v: number }, EncryptionError> + > { + seenMetadata = this.getAuditData().metadata + return { data: { v: 1 } } + } + } + + const mapped = new MappedDecryptOperation< + { v: number }, + { mapped: number } + >(new Underlying(), (value) => ({ mapped: value.v + 1 }), { + failure: { + type: EncryptionErrorTypes.DecryptionError, + message: 'unknown table', + }, + }) + + const result = await resolveDecryptResult(mapped, { metadata: { m: 7 } }) + + expect(result).toEqual({ data: { mapped: 2 } }) + expect(seenMetadata).toEqual({ m: 7 }) + }) + + it('returns the precomputed failure from a MappedDecryptOperation with no map (unknown table)', async () => { + class Underlying extends EncryptionOperation<{ v: number }> { + override async execute(): Promise< + Result<{ v: number }, EncryptionError> + > { + return { data: { v: 1 } } + } + } + + const unknownTableFailure = { + failure: { + type: EncryptionErrorTypes.DecryptionError, + message: 'unknown table', + }, + } + const mapped = new MappedDecryptOperation< + { v: number }, + { mapped: number } + >(new Underlying(), undefined, unknownTableFailure) + + const result = await resolveDecryptResult(mapped, { metadata: { m: 7 } }) + + expect(result.failure?.message).toBe('unknown table') + expect(result.data).toBeUndefined() + }) }) describe('throwPreservingCode', () => { diff --git a/packages/stack/__tests__/init-strategy.test.ts b/packages/stack/__tests__/init-strategy.test.ts index ddef9f0f5..1b29b4359 100644 --- a/packages/stack/__tests__/init-strategy.test.ts +++ b/packages/stack/__tests__/init-strategy.test.ts @@ -254,15 +254,17 @@ describe('Encryption config.eqlVersion', () => { expect(lastNewClientOpts().eqlVersion).toBe(3) }) - it('EncryptionV3 overrides an explicit eqlVersion — v3 is an invariant', async () => { - // A v2-mode client cannot resolve v3 concrete-type columns (every encrypt - // fails inside the FFI), so EncryptionV3 pins the wire format rather than - // honouring a caller's eqlVersion. + it('EncryptionV3 is a deprecated alias of Encryption — it honours an explicit eqlVersion identically', async () => { + // Post-collapse `EncryptionV3` IS `Encryption` (a deprecated alias), so it no + // longer independently pins the wire format: an explicit `eqlVersion: 2` over + // a v3 schema set is honoured exactly as `Encryption` honours it (the + // migration escape hatch above). A v2-mode client still cannot encrypt v3 + // concrete-type columns — this only pins the wire flag reaching the FFI. await EncryptionV3({ schemas: [v3Table() as never], config: { eqlVersion: 2 }, }) - expect(lastNewClientOpts().eqlVersion).toBe(3) + expect(lastNewClientOpts().eqlVersion).toBe(2) }) }) diff --git a/packages/stack/__tests__/typed-client-v3.test-d.ts b/packages/stack/__tests__/typed-client-v3.test-d.ts index 03b929c9b..58419bd1b 100644 --- a/packages/stack/__tests__/typed-client-v3.test-d.ts +++ b/packages/stack/__tests__/typed-client-v3.test-d.ts @@ -146,6 +146,25 @@ describe('typed v3 client — model decrypt yields precise plaintext', () => { users, ) }) + + it('decryptModel / bulkDecryptModels are chainable with .audit() and .withLockContext()', () => { + // The typed client's decrypt methods return a chainable operation (not a bare + // Promise), so audit metadata and lock context can be attached before await. + const op = client.decryptModel({ id: 'u1', email: {} as Encrypted }, users) + expectTypeOf(op).toHaveProperty('audit') + expectTypeOf(op).toHaveProperty('withLockContext') + // Both stay chainable — same operation type back. + expectTypeOf(op.audit({ metadata: { sub: 'u1' } })).toEqualTypeOf< + typeof op + >() + + const bulkOp = client.bulkDecryptModels( + [{ id: 'u1', email: {} as Encrypted }], + users, + ) + expectTypeOf(bulkOp).toHaveProperty('audit') + expectTypeOf(bulkOp).toHaveProperty('withLockContext') + }) }) describe('typed v3 client — soundness', () => { diff --git a/packages/stack/__tests__/typed-client-v3.test.ts b/packages/stack/__tests__/typed-client-v3.test.ts index 5969989b7..457fc8378 100644 --- a/packages/stack/__tests__/typed-client-v3.test.ts +++ b/packages/stack/__tests__/typed-client-v3.test.ts @@ -11,14 +11,32 @@ const table = encryptedTable('t', { }) /** - * A minimal client stub whose model-decrypt methods resolve to a fixed - * `Result` payload. `typedClient` only `await`s these, so a plain Promise is a - * sufficient thenable. + * A minimal operation stub resolving to a fixed `Result`. `typedClient` now + * wraps the underlying decrypt op in a `MappedDecryptOperation` and calls + * `.execute()` on it (rather than awaiting a bare promise), so the stub must be + * operation-like: `.execute()` plus the chainable `.audit()` / `.withLockContext()` + * the wrapper may delegate to. + */ +function fakeOp(result: R) { + return { + execute: () => Promise.resolve(result), + audit() { + return this + }, + withLockContext() { + return this + }, + } +} + +/** + * A minimal client stub whose model-decrypt methods return an operation + * resolving to a fixed `Result` payload. */ function fakeClient(data: Record): EncryptionClient { return { - decryptModel: () => Promise.resolve({ data }), - bulkDecryptModels: () => Promise.resolve({ data: [data] }), + decryptModel: () => fakeOp({ data }), + bulkDecryptModels: () => fakeOp({ data: [data] }), } as unknown as EncryptionClient } @@ -77,7 +95,7 @@ describe('typedClient — decrypt reconstruction', () => { it('propagates a failure result unchanged', async () => { const failing = { decryptModel: () => - Promise.resolve({ + fakeOp({ failure: { type: 'DecryptionError', message: 'boom' }, }), } as unknown as EncryptionClient diff --git a/packages/stack/__tests__/v3-matrix/matrix-audit.test-d.ts b/packages/stack/__tests__/v3-matrix/matrix-audit.test-d.ts index cdaa95db9..e3b56d954 100644 --- a/packages/stack/__tests__/v3-matrix/matrix-audit.test-d.ts +++ b/packages/stack/__tests__/v3-matrix/matrix-audit.test-d.ts @@ -1,9 +1,10 @@ /** - * Type-level pin of a real v3 asymmetry: audit metadata is available on the - * encrypt-side operations (which are chainable) but NOT on `decryptModel` / - * `bulkDecryptModels`, which return a bare `Promise>` rather than a - * chainable operation. Documented here as an executable invariant so the gap - * (v2's `decryptModel().audit(...)` has no v3 equivalent) can't silently change. + * Type-level pin that audit + lock-context are chainable on BOTH the encrypt-side + * operations AND `decryptModel` / `bulkDecryptModels`. The typed client's decrypt + * methods now return a chainable {@link MappedDecryptOperation} (was a bare + * `Promise>`), restoring the v2-era `decryptModel().audit(...)` surface + * on the v3 client. Documented here as an executable invariant so the capability + * can't silently regress. * * Runs via `pnpm test:types`. */ @@ -22,10 +23,16 @@ describe('v3 typed client audit/lock-context chainability (types)', () => { expectTypeOf(op).toHaveProperty('withLockContext') }) - it('does NOT expose .audit()/.withLockContext() on decryptModel (bare Promise)', () => { - const result = typed.decryptModel({ email: {} as never }, users) - // A Promise, not a chainable operation — no audit/lock-context hook. - expectTypeOf(result).not.toHaveProperty('audit') - expectTypeOf(result).not.toHaveProperty('withLockContext') + it('exposes .audit()/.withLockContext() on decryptModel (chainable operation)', () => { + const op = typed.decryptModel({ email: {} as never }, users) + // A chainable operation, not a bare Promise — audit + lock-context hooks. + expectTypeOf(op).toHaveProperty('audit') + expectTypeOf(op).toHaveProperty('withLockContext') + }) + + it('exposes .audit()/.withLockContext() on bulkDecryptModels (chainable operation)', () => { + const op = typed.bulkDecryptModels([{ email: {} as never }], users) + expectTypeOf(op).toHaveProperty('audit') + expectTypeOf(op).toHaveProperty('withLockContext') }) }) diff --git a/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts b/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts index d8041e338..e7456b947 100644 --- a/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts +++ b/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts @@ -37,7 +37,7 @@ vi.mock('@cipherstash/protect-ffi', () => ({ })) import * as ffi from '@cipherstash/protect-ffi' -import { encryptedTable, typedClient, types } from '@/encryption/v3' +import { encryptedTable, types } from '@/encryption/v3' const users = encryptedTable('users', { email: types.TextEq('email'), @@ -68,14 +68,16 @@ function unwrap(result: any) { // biome-ignore lint/suspicious/noExplicitAny: reading recorded mock args const lastOpts = (fn: any) => fn.mock.calls.at(-1)[1] -let typed: ReturnType +// `Encryption` returns the typed client directly for an all-v3 schema set (the +// collapse of `EncryptionV3`), so there is no separate `typedClient` wrap here. +let typed: Awaited>> let prevWorkspaceCrn: string | undefined beforeEach(async () => { vi.clearAllMocks() prevWorkspaceCrn = process.env.CS_WORKSPACE_CRN process.env.CS_WORKSPACE_CRN = 'crn:ap-southeast-2.aws:test-workspace' - typed = typedClient(await Encryption({ schemas: [users] as never }), users) + typed = await Encryption({ schemas: [users] }) }) afterEach(() => { diff --git a/packages/stack/integration/shared/schema-v3-client.integration.test.ts b/packages/stack/integration/shared/schema-v3-client.integration.test.ts index 839ded21a..a3b31b3a5 100644 --- a/packages/stack/integration/shared/schema-v3-client.integration.test.ts +++ b/packages/stack/integration/shared/schema-v3-client.integration.test.ts @@ -1,7 +1,6 @@ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' import type { EncryptionClient } from '@/encryption' -import { typedClient } from '@/encryption/v3' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' @@ -173,7 +172,9 @@ describe('eql_v3 client integration', () => { // `Date` from the encrypt-config `cast_as` (`reconstructRow`), keyed by the // JS property (`createdOn`) even though the DB column is `created_on`. it('round-trips a representative date storage domain via decryptModel', async () => { - const typed = typedClient(protectClient, users) + // `Encryption` returns the typed client for an all-v3 schema set, so + // `protectClient` already carries the typed model methods — no extra wrap. + const typed = protectClient // Zero milliseconds so the FFI dropping sub-second precision (`...00Z` vs // `...000Z`) does not perturb the reconstructed instant. const day = new Date('2026-07-01T00:00:00.000Z') @@ -203,7 +204,9 @@ describe('eql_v3 client integration', () => { // ENCRYPTED by the model path — not silently passed through as plaintext // because the field key (`createdOn`) fails to match the DB-keyed config. it('encrypts a property-vs-DB-name column through encryptModel (no plaintext leak)', async () => { - const typed = typedClient(protectClient, users) + // `Encryption` returns the typed client for an all-v3 schema set, so + // `protectClient` already carries the typed model methods — no extra wrap. + const typed = protectClient const day = new Date('2026-07-01T00:00:00.000Z') const encrypted = unwrapResult( @@ -232,7 +235,9 @@ describe('eql_v3 client integration', () => { // ms-zeroed `12:34:56` value must round-trip exactly. (Was skipped while every // timestamp domain wrongly set `cast_as: 'date'`; re-enabled with that fix.) it('round-trips a timestamp occurredAt column through the model path', async () => { - const typed = typedClient(protectClient, users) + // `Encryption` returns the typed client for an all-v3 schema set, so + // `protectClient` already carries the typed model methods — no extra wrap. + const typed = protectClient // Zero milliseconds: the FFI drops sub-second precision, so a ms-bearing // instant would perturb the reconstructed value. const moment = new Date('2026-07-01T12:34:56.000Z') diff --git a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts new file mode 100644 index 000000000..46567cfaa --- /dev/null +++ b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts @@ -0,0 +1,94 @@ +/** + * Acceptance #1a + #1b — EQL v2 READ compatibility after the v3 collapse (PR 3). + * + * PR 3 makes EQL v3 the only generation the client authors/writes, but MUST keep + * reading previously stored EQL v2 payloads — on the core client (guardrail 1) + * AND through the DynamoDB adapter (guardrail 2). This suite mints v2 data with a + * v2-mode `Encryption` client (the retained `config: { eqlVersion: 2 }` escape + * hatch) and proves it still round-trips: + * + * - #1a: a v2 ciphertext + a v2 model decrypt through the collapsed `Encryption` + * client's `decrypt` / `decryptModel`. + * - #1b: a stored v2 DynamoDB item — split with the exported + * `toEncryptedDynamoItem(payload, attrs, false)` — decrypts through + * `encryptedDynamoDB(...).decryptModel(item, v2Table)`, exercising the v2 + * envelope reconstruction (`toItemWithEqlPayloads`, `v === 2` / `k: 'ct'`). + * + * Both fail if the v2 read path is removed. Live: requires real ZeroKMS + * credentials (the integration harness provisions them and throws otherwise). + */ +import { unwrapResult } from '@cipherstash/test-kit' +import { beforeAll, describe, expect, it } from 'vitest' +import { encryptedDynamoDB } from '@/dynamodb' +import { toEncryptedDynamoItem } from '@/dynamodb/helpers' +import type { EncryptionClient } from '@/encryption' +import { Encryption } from '@/index' +// The deprecated v2 authoring builders remain for reading/migrating legacy data. +import { encryptedColumn, encryptedTable } from '@/schema' + +// A minimal EQL v2 table (v2 builders → no `buildColumnKeyMap`, so `Encryption` +// returns the nominal client, not the typed one). +const usersV2 = encryptedTable('v2_read_compat_users', { + email: encryptedColumn('email').equality(), +}) + +const SECRET = 'ada@example.com' + +// A v2-mode client: the explicit `eqlVersion: 2` escape hatch forces v2 wire so +// this suite mints genuinely-v2 payloads, independent of schema auto-detection. +let v2Client: EncryptionClient + +beforeAll(async () => { + v2Client = await Encryption({ + schemas: [usersV2], + config: { eqlVersion: 2 }, + }) +}) + +describe('#1a — core client reads a stored EQL v2 payload', () => { + it('round-trips a v2 ciphertext through decrypt', async () => { + const encrypted = unwrapResult( + await v2Client.encrypt(SECRET, { table: usersV2, column: usersV2.email }), + ) + // Guard against a false pass: it must be an actual ciphertext. + expect(encrypted).toHaveProperty('c') + + const decrypted = unwrapResult(await v2Client.decrypt(encrypted)) + expect(decrypted).toBe(SECRET) + }, 30000) + + it('round-trips a v2 model through decryptModel', async () => { + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + expect(encryptedModel.email).toHaveProperty('c') + + const decrypted = unwrapResult(await v2Client.decryptModel(encryptedModel)) + expect(decrypted).toEqual({ pk: 'a', email: SECRET }) + }, 30000) +}) + +describe('#1b — DynamoDB adapter reads a stored EQL v2 item', () => { + it('decrypts a v2-split DynamoDB item via encryptedDynamoDB.decryptModel', async () => { + // Stage a stored v2 item: encrypt a v2 model, then split it into DynamoDB + // attributes exactly as the (removed-at-the-type-level, retained-at-runtime) + // v2 write path would have — `toEncryptedDynamoItem(..., false)`. + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + const storedV2Item = toEncryptedDynamoItem(encryptedModel, ['email'], false) + + // The email column becomes `email__source` (+ `email__hmac` for equality); + // the plaintext key does not survive the split. + expect(storedV2Item).toHaveProperty('email__source') + expect(storedV2Item).not.toHaveProperty('email') + expect(storedV2Item.pk).toBe('a') + + const dynamo = encryptedDynamoDB({ encryptionClient: v2Client }) + const decrypted = unwrapResult( + await dynamo.decryptModel(storedV2Item, usersV2), + ) + + expect(decrypted).toMatchObject({ pk: 'a', email: SECRET }) + }, 30000) +}) diff --git a/packages/stack/src/dynamodb/index.ts b/packages/stack/src/dynamodb/index.ts index bce47df8b..9e2c60aad 100644 --- a/packages/stack/src/dynamodb/index.ts +++ b/packages/stack/src/dynamodb/index.ts @@ -68,9 +68,11 @@ function assertClientTableVersionMatch( * and `bulkDecryptModels` methods that transparently encrypt/decrypt DynamoDB * items according to the provided table schema. * - * Accepts EQL v3 tables (`types.*` domains) and EQL v2 tables - * (`encryptedColumn`/`encryptedField`) alike — the table decides which wire - * format is synthesized on read. + * **Encrypt/write is EQL v3 only** — `encryptModel` / `bulkEncryptModels` accept + * only EQL v3 tables (`types.*` domains). **Decrypt still reads existing EQL v2 + * items**: `decryptModel` / `bulkDecryptModels` continue to accept an EQL v2 + * table (`encryptedColumn`/`encryptedField`) so previously stored v2 data + * remains readable. The table decides which wire format is reconstructed on read. * * Only equality is meaningful on DynamoDB: an `hm` term is stored alongside the * ciphertext as `__hmac` and can back a key condition. Ordering and diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index 1bfbca989..d661928a3 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -37,12 +37,12 @@ export type AnyEncryptedTable = * nominal `TypedEncryptionClient` parameter would reject a client built for * a narrower schema tuple. * - * The two clients differ at runtime on the decrypt paths — the nominal client - * returns a chainable operation carrying `.audit()`, the typed wrapper returns - * a plain `Promise>` and takes the table as a second argument. The - * operation classes handle both; see `DecryptModelOperation`. The consequence - * for callers is that **audit metadata on decrypt requires the nominal - * client** — with a client from `EncryptionV3` there is nowhere to put it. + * Both clients now return a chainable operation on the decrypt paths — the + * nominal client's `DecryptModelOperation` and the typed wrapper's + * `MappedDecryptOperation` each carry `.audit()` (the typed wrapper also takes + * the table as a second argument). The operation classes handle both; see + * `DecryptModelOperation` and `resolveDecryptResult`. Audit metadata on decrypt + * is therefore forwarded regardless of which client shape is supplied. */ export type DynamoDBEncryptionClient = { encryptModel(input: never, table: never): unknown @@ -169,8 +169,8 @@ type Simplify = { [K in keyof T]: T[K] } * * A declared column `email` does NOT survive as `email`: the adapter deletes it * and writes `email__source` (plus `email__hmac` for equality domains). Typing - * the result as the input model — what the v2 overload still does — is a lie - * that type-checks `result.data.email` (always `undefined` at runtime) and + * the result as the input model — what the removed v2 write overload did — is a + * lie that type-checks `result.data.email` (always `undefined` at runtime) and * rejects `result.data.email__source` (the value you actually want). * * Keys that name no column pass through untouched — partition/sort keys, GSI @@ -240,16 +240,6 @@ export interface EncryptedDynamoDBInstance { item: V3ModelInput, table: Table, ): EncryptModelOperation> - /** - * EQL v2. Unchanged, so existing callers keep compiling — v2 columns do not - * carry their index configuration in the type, so the storage split cannot be - * derived. The returned `T` is the INPUT model, not what is on the wire; read - * `__source` / `__hmac` through a type of your own. - */ - encryptModel>( - item: T, - table: EncryptedTable, - ): EncryptModelOperation /** EQL v3. See {@link EncryptedDynamoDBInstance.encryptModel}. */ bulkEncryptModels< @@ -259,11 +249,6 @@ export interface EncryptedDynamoDBInstance { items: Array>, table: Table, ): BulkEncryptModelsOperation> - /** EQL v2. See {@link EncryptedDynamoDBInstance.encryptModel}. */ - bulkEncryptModels>( - items: T[], - table: EncryptedTable, - ): BulkEncryptModelsOperation /** * EQL v3: `item` is the stored attribute map (`__source` / diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index a66817e79..968bea2e1 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -1,6 +1,7 @@ import { type Result, withResult } from '@byteslice/result' import { newClient } from '@cipherstash/protect-ffi' import { validate as uuidValidate } from 'uuid' +import type { AnyV3Table } from '@/eql/v3' import { type EncryptionError, EncryptionErrorTypes } from '@/errors' // `LockContext` is imported type-only so the TSDoc {@link} references in the // comments below resolve; it is erased at compile time. @@ -20,6 +21,7 @@ import type { BulkDecryptPayload, BulkEncryptPayload, Client, + ClientConfig, Encrypted, EncryptedFromBuildableTable, EncryptionClientConfig, @@ -43,6 +45,13 @@ import { DecryptModelOperation } from './operations/decrypt-model' import { EncryptOperation } from './operations/encrypt' import { EncryptModelOperation } from './operations/encrypt-model' import { EncryptQueryOperation } from './operations/encrypt-query' +// `typedClient` wraps the nominal client into the strongly-typed EQL v3 client. +// This is a deliberate circular import with `./v3` (which imports `Encryption` +// from here): both `Encryption` and `typedClient` are hoisted `function` +// declarations, and the only module-eval cross-reference — `EncryptionV3 = +// Encryption` in `./v3` — reads the hoisted `Encryption`, so neither binding is +// observed uninitialised regardless of which module evaluates first. +import { type TypedEncryptionClient, typedClient } from './v3' // Re-export the operation classes returned by EncryptionClient methods so they // are part of the public API and appear in the generated reference, allowing @@ -852,9 +861,24 @@ export function __resetStrategyDeprecationWarningForTests(): void { * @see {@link ClientConfig.authStrategy} for the auth strategy field. * @see {@link EncryptionClient} for available methods after initialization. */ -export const Encryption = async ( +// Overload 1 — v3-typed: an array literal of concrete EQL v3 tables (from +// `@cipherstash/stack/v3`) yields the strongly-typed {@link TypedEncryptionClient}, +// the collapse of the former `EncryptionV3`. The wire format is forced to v3. +export function Encryption(config: { + schemas: S + config?: ClientConfig +}): Promise> +// Overload 2 — nominal: loose/dynamic schemas (introspection-derived, e.g. +// stack-supabase) or EQL v2 tables yield the generation-neutral +// {@link EncryptionClient}, which auto-detects its wire version. Declared LAST +// so `Parameters` / `ReturnType` resolve +// to this signature (stack-supabase casts its schemas against it). +export function Encryption( config: EncryptionClientConfig, -): Promise => { +): Promise +export async function Encryption( + config: EncryptionClientConfig, +): Promise { const { schemas, config: clientConfig } = config if (!schemas.length) { @@ -885,10 +909,16 @@ export const Encryption = async ( const client = new EncryptionClient() const encryptConfig = buildEncryptConfig(...schemas) - // Resolve the wire format for this client: an explicit - // `config.eqlVersion` wins; otherwise it is auto-detected from the - // schema set (v3 tables → 3, v2 tables → the FFI's v2 default). A mixed - // v2 + v3 schema set throws — one client emits exactly one wire format. + // A schema set of exclusively concrete EQL v3 tables is the typed-client path + // (the collapse of the former `EncryptionV3`). Every other set — EQL v2 tables, + // or the introspection-derived loose tables stack-supabase passes — stays + // nominal. + const isV3Only = schemas.every(hasBuildColumnKeyMap) + + // Resolve the wire format: an explicit `config.eqlVersion` wins (the retained + // migration escape hatch — e.g. deliberately writing v2 wire from a v3 schema + // set), otherwise it is auto-detected (v3 tables → 3, v2 tables → the FFI's v2 + // default). A mixed v2 + v3 schema set throws inside `resolveEqlVersion`. const eqlVersion = resolveEqlVersion(schemas, clientConfig?.eqlVersion) const result = await client.init({ @@ -902,5 +932,12 @@ export const Encryption = async ( throw new Error(`[encryption]: ${result.failure.message}`) } - return result.data + // Return the typed client only when the client is genuinely in EQL v3 mode: an + // all-v3 schema set that resolved to the v3 wire format. A caller that forced + // `eqlVersion: 2` over v3 schemas gets the nominal client for that deliberate, + // low-level v2-wire migration path (the typed client cannot encrypt v3 columns + // in v2 mode anyway). + return isV3Only && eqlVersion === 3 + ? typedClient(result.data, ...(schemas as unknown as readonly AnyV3Table[])) + : result.data } diff --git a/packages/stack/src/encryption/operations/mapped-decrypt.ts b/packages/stack/src/encryption/operations/mapped-decrypt.ts new file mode 100644 index 000000000..378b30ab4 --- /dev/null +++ b/packages/stack/src/encryption/operations/mapped-decrypt.ts @@ -0,0 +1,92 @@ +import type { Result } from '@byteslice/result' +import type { EncryptionError } from '@/errors' +import type { LockContextInput } from '@/identity' +import { type AuditConfig, EncryptionOperation } from './base-operation' + +/** + * The subset of an underlying decrypt operation the mapped wrapper drives: a + * chainable {@link EncryptionOperation} that MAY additionally expose + * `.withLockContext()`. A lock-bound underlying op (e.g. + * `DecryptModelOperationWithLockContext`) has already consumed its lock context + * and offers no further `withLockContext`, so the method is optional here. + */ +type UnderlyingDecryptOperation = EncryptionOperation & { + withLockContext?: (lockContext: LockContextInput) => EncryptionOperation +} + +/** + * The public contract of a decrypt-model operation returned by the typed client: + * awaitable to a `Result`, and chainable with `.audit()` / + * `.withLockContext()`. Hides the underlying pre-map type parameter. + */ +export interface AuditableDecryptModelOperation + extends EncryptionOperation { + audit(config: AuditConfig): this + withLockContext( + lockContext: LockContextInput, + ): AuditableDecryptModelOperation +} + +/** + * A chainable decrypt operation that maps a successful result through a pure, + * precomputed function while delegating audit metadata and lock context to the + * underlying operation it wraps. + * + * This is what lets the typed EQL v3 client carry `.audit()` / + * `.withLockContext()` on `decryptModel` / `bulkDecryptModels`: the earlier + * implementation `await`ed the underlying op and mapped the resolved value, + * which collapsed the chain to a plain `Promise>` and dropped both + * capabilities. Here the mapping runs inside `execute()` instead, so the + * operation stays an {@link EncryptionOperation} the whole way. + * + * - `.audit()` forwards to the underlying op, so the op that actually runs the + * decrypt sees the metadata (via `getAuditData()`), in EITHER chaining order — + * `.audit().withLockContext()` propagates because `withLockContext` copies the + * audit data forward, and `.withLockContext().audit()` propagates because the + * wrapper forwards `.audit()` onto the now-lock-bound underlying op. + * - `.withLockContext()` rebuilds the wrapper over `underlying.withLockContext(lc)`, + * preserving the same `map` and unknown-table failure. + * - `execute()` never throws: an unknown table (no `map`) returns the precomputed + * `failure` Result, and `map` is a precomputed reconstructor — pure, no + * `build()` — so it cannot reject the Result contract. + */ +export class MappedDecryptOperation extends EncryptionOperation { + constructor( + private readonly underlying: UnderlyingDecryptOperation, + // Precomputed reconstructor. `undefined` when the table is unknown to the + // client — `execute()` then short-circuits to `unknownTableFailure`. + private readonly map: ((value: In) => Out) | undefined, + private readonly unknownTableFailure: { failure: EncryptionError }, + ) { + super() + } + + override audit(config: AuditConfig): this { + // Delegate to the op that runs the decrypt so its `execute()` sees the + // metadata; the wrapper's own `execute()` reads nothing off `this`. + this.underlying.audit(config) + return this + } + + withLockContext( + lockContext: LockContextInput, + ): MappedDecryptOperation { + // A lock-bound underlying op exposes no `withLockContext`; there is nothing + // to re-bind, so keep the current underlying op. + const bound = this.underlying.withLockContext + ? this.underlying.withLockContext(lockContext) + : this.underlying + return new MappedDecryptOperation(bound, this.map, this.unknownTableFailure) + } + + override async execute(): Promise> { + if (!this.map) { + return this.unknownTableFailure + } + const result = await this.underlying.execute() + if (result.failure) { + return result + } + return { data: this.map(result.data) } + } +} diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 14d934556..34c4602c8 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -1,4 +1,3 @@ -import type { Result } from '@byteslice/result' import type { AnyV3Table, ColumnsOf, @@ -15,7 +14,6 @@ import type { LockContextInput } from '@/identity' import type { BulkDecryptPayload, BulkEncryptPayload, - ClientConfig, Encrypted, EncryptedReturnType, EncryptOptions, @@ -33,6 +31,10 @@ import { type EncryptOperation, type EncryptQueryOperation, } from './index' +import { + type AuditableDecryptModelOperation, + MappedDecryptOperation, +} from './operations/mapped-decrypt' /** * A strongly-typed view over an {@link EncryptionClient} for EQL v3 schemas. @@ -101,22 +103,24 @@ export interface TypedEncryptionClient { * columns are reconstructed from the encrypt-config `cast_as`. * * Pass `lockContext` to decrypt identity-bound data — the same context that - * was supplied at encrypt time must be provided here. + * was supplied at encrypt time must be provided here (or chain + * `.withLockContext()` on the returned operation). * - * Unlike the encrypt operations this returns a plain `Promise>` - * rather than a chainable operation, because it maps the resolved value. + * Returns a chainable {@link AuditableDecryptModelOperation}: await it for the + * `Result`, or chain `.audit({ metadata })` / `.withLockContext()` first. The + * per-row `Date` reconstruction is applied to the successful result. */ decryptModel>( input: T, table: Table, lockContext?: LockContextInput, - ): Promise, EncryptionError>> + ): AuditableDecryptModelOperation> bulkDecryptModels
>( input: Array, table: Table, lockContext?: LockContextInput, - ): Promise>, EncryptionError>> + ): AuditableDecryptModelOperation>> // Parity passthroughs — not v3-strengthened, delegated as-is. bulkEncrypt( @@ -253,25 +257,31 @@ export function typedClient( bulkEncryptModels: (input, table) => client.bulkEncryptModels(input as never, table as never) as never, decrypt: (encrypted) => client.decrypt(encrypted), - decryptModel: async (input, table, lockContext) => { + decryptModel: (input, table, lockContext) => { + // `reconstruct` is undefined for a table this client was not initialized + // with; the mapped op then resolves to `unknownTableFailure` on execute. const reconstruct = reconstructors.get(table.tableName) - if (!reconstruct) return unknownTableFailure as never const op = client.decryptModel(input as never) - const result = await (lockContext ? op.withLockContext(lockContext) : op) - if (result.failure) return result as never - return { data: reconstruct(result.data) } as never + const base = lockContext ? op.withLockContext(lockContext) : op + return new MappedDecryptOperation( + base, + reconstruct, + unknownTableFailure, + ) as never }, - bulkDecryptModels: async (input, table, lockContext) => { + bulkDecryptModels: (input, table, lockContext) => { const reconstruct = reconstructors.get(table.tableName) - if (!reconstruct) return unknownTableFailure as never const op = client.bulkDecryptModels(input as never) - const result = await (lockContext ? op.withLockContext(lockContext) : op) - if (result.failure) return result as never - return { - data: result.data.map((row) => - reconstruct(row as Record), - ), - } as never + const base = lockContext ? op.withLockContext(lockContext) : op + // The underlying op resolves to an array of rows; reconstruct each. + const mapRows = reconstruct + ? (rows: Array>) => rows.map(reconstruct) + : undefined + return new MappedDecryptOperation( + base, + mapRows, + unknownTableFailure, + ) as never }, bulkEncrypt: (plaintexts, opts) => client.bulkEncrypt(plaintexts, opts), bulkDecrypt: (payloads) => client.bulkDecrypt(payloads), @@ -280,38 +290,22 @@ export function typedClient( } /** - * Build a {@link TypedEncryptionClient} for EQL v3 schemas — the strongly-typed - * counterpart to {@link Encryption}. Mirrors its config, then retypes the client - * against the provided v3 `schemas`. + * @deprecated Use {@link Encryption} instead — it is now overloaded so an array + * of concrete EQL v3 tables yields the same strongly-typed client this used to. + * `EncryptionV3` is a type-identical alias of `Encryption`, retained for + * backwards compatibility, and will be removed in a future release. * * @example * ```typescript - * import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + * import { Encryption, encryptedTable, types } from "@cipherstash/stack/v3" * * const users = encryptedTable("users", { email: types.TextSearch("email") }) - * const client = await EncryptionV3({ schemas: [users] }) + * const client = await Encryption({ schemas: [users] }) * * await client.encrypt("a@b.com", { table: users, column: users.email }) * ``` */ -export async function EncryptionV3< - const S extends readonly AnyV3Table[], ->(config: { - schemas: S - config?: ClientConfig -}): Promise> { - const client = await Encryption({ - schemas: config.schemas as unknown as Parameters< - typeof Encryption - >[0]['schemas'], - // Force the v3 EQL wire format. protect-ffi's newClient defaults to - // eqlVersion 2; a v2-mode client cannot resolve v3 concrete-type columns - // and fails every encrypt with "Cannot convert undefined or null to - // object". This is a v3-only invariant, so it overrides any user value. - config: { ...config.config, eqlVersion: 3 }, - }) - return typedClient(client, ...config.schemas) -} +export const EncryptionV3 = Encryption // Single import surface: re-export the v3 `types` namespace + table API + type // helpers so `@cipherstash/stack/v3` provides everything needed to author and diff --git a/packages/stack/src/schema/index.ts b/packages/stack/src/schema/index.ts index 27fa88d47..a53ece0a2 100644 --- a/packages/stack/src/schema/index.ts +++ b/packages/stack/src/schema/index.ts @@ -219,6 +219,9 @@ export type EncryptConfig = z.infer * Builder for a nested encrypted field (encrypted but not searchable). * Create with {@link encryptedField}. Use inside nested objects in {@link encryptedTable}; * supports `.dataType()` for plaintext type. No index methods (equality, orderAndRange, etc.). + * + * @deprecated EQL v2 authoring. The client authors EQL v3; this class backs the + * v2 `encryptedField` builder, retained only for reading/migrating legacy v2 data. */ export class EncryptedField { private valueName: string @@ -263,6 +266,12 @@ export class EncryptedField { } } +/** + * Builder for an EQL v2 encrypted column. Create with {@link encryptedColumn}. + * + * @deprecated EQL v2 authoring. The client authors EQL v3; this class backs the + * v2 `encryptedColumn` builder, retained only for reading/migrating legacy v2 data. + */ export class EncryptedColumn { private columnName: string private castAsValue: CastAs @@ -602,6 +611,11 @@ export type InferEncrypted> = * so you can reference columns as `users.email` when calling `encrypt`, `decrypt`, * and `encryptQuery`. * + * @deprecated EQL v2 authoring. The client authors EQL v3 — build tables with + * `encryptedTable` + the `types.*` factories from `@cipherstash/stack/v3`. This + * v2 builder remains only for reading/migrating legacy v2 data and will be + * removed once the v2 adapters are gone. + * * @param tableName - The name of the database table this schema represents. * @param columns - An object whose keys are logical column names and values are * {@link EncryptedColumn} from {@link encryptedColumn}, or nested objects whose @@ -649,6 +663,11 @@ export function encryptedTable( * `.searchableJson()`) and/or `.dataType()` to configure searchable encryption * and the plaintext data type. * + * @deprecated EQL v2 authoring. The client authors EQL v3 — define columns with + * the `types.*` concrete-domain factories from `@cipherstash/stack/v3`. This v2 + * builder remains only for reading/migrating legacy v2 data and will be removed + * once the v2 adapters are gone. + * * @param columnName - The name of the database column to encrypt. * @returns A new `EncryptedColumn` builder. * @@ -672,6 +691,11 @@ export function encryptedColumn(columnName: string) { * for nested fields that are encrypted but not searchable (no indexes). Use `.dataType()` * to specify the plaintext type. * + * @deprecated EQL v2 authoring. The client authors EQL v3 — model nested fields + * with a dotted v3 column path (`'profile.ssn': types.TextEq(...)`) from + * `@cipherstash/stack/v3`. This v2 builder remains only for reading/migrating + * legacy v2 data and will be removed once the v2 adapters are gone. + * * @param valueName - The name of the value field. * @returns A new `EncryptedField` builder. * diff --git a/packages/stack/src/types.ts b/packages/stack/src/types.ts index c516d848e..6eee471b2 100644 --- a/packages/stack/src/types.ts +++ b/packages/stack/src/types.ts @@ -171,6 +171,12 @@ export type ClientConfig = { strategy?: AuthStrategy /** + * @deprecated The client authors EQL v3 — an all-v3 schema set forces `3` + * automatically and yields the typed client. This field remains only to read + * or write legacy EQL v2 during migration (e.g. `eqlVersion: 2` with a v2 + * schema set), and will be removed once the v2 adapters are gone. New code + * should not set it. + * * The EQL wire version the client emits — one FFI client always emits * exactly one wire format. * diff --git a/skills/stash-dynamodb/SKILL.md b/skills/stash-dynamodb/SKILL.md index cf3d44d47..2369d75b6 100644 --- a/skills/stash-dynamodb/SKILL.md +++ b/skills/stash-dynamodb/SKILL.md @@ -44,18 +44,20 @@ Non-encrypted attributes pass through unchanged. On decryption, the `__source` a ## Choosing a Schema Version -`encryptedDynamoDB` accepts both EQL v3 and EQL v2 tables. **Use EQL v3 for new projects.** +**Encrypt/write is EQL v3 only.** `encryptModel` / `bulkEncryptModels` accept only EQL v3 tables (`types.*` domains). **Decrypt still reads existing EQL v2 items** — `decryptModel` / `bulkDecryptModels` continue to accept an EQL v2 table so previously stored data stays readable. Author all new tables with EQL v3; keep a v2 table around only to read old data. -| | EQL v3 (recommended) | EQL v2 (existing deployments) | +| | EQL v3 (author + read) | EQL v2 (read existing data only) | |---|---|---| | Import | `@cipherstash/stack/v3` | `@cipherstash/stack` + `@cipherstash/stack/schema` | | Schema | `encryptedTable` + `types.*` | `encryptedTable` + `encryptedColumn` | -| Client | `EncryptionV3({ schemas })` | `Encryption({ schemas })` | +| Client | `Encryption({ schemas })` (or the deprecated `EncryptionV3`) | `Encryption({ schemas })` | +| encrypt | ✅ | ❌ removed — write is v3 only | +| decrypt | ✅ | ✅ reads stored v2 items | | Nested fields | Flat dotted path (`"profile.ssn"`) | Nested group + `encryptedField` | -There is no infrastructure migration between them — DynamoDB has no EQL extension to install and no schema to alter — but there is no automatic *data* migration path either. The two write **different wire formats**, so items written under one version cannot be decrypted by the other. Pick one per table and stay on it; to switch an existing table you must re-encrypt every item. +There is no infrastructure migration between the versions — DynamoDB has no EQL extension to install and no schema to alter — and there is no automatic *data* migration either. The two use **different wire formats**; a stored v2 item still decrypts through a v2 table, but new writes are v3. To fully move a table to v3, re-encrypt every item with a v3 schema. -The client must be built for the table. Build the client with the same v3 table you hand to `encryptedDynamoDB` — `EncryptionV3({ schemas: [users] })` (or `Encryption({ schemas: [users], config: { eqlVersion: 3 } })`). Passing a v3 table to a client that never registered it (a client built for a different schema set) throws a clear error naming the table on the first operation, instead of failing later with an opaque FFI deserialization error. +The client must be built for the table. Build the client with the same v3 table you hand to `encryptedDynamoDB` — `Encryption({ schemas: [users] })` returns the typed v3 client for a concrete v3 schema set. Passing a v3 table to a client that never registered it (a client built for a different schema set) throws a clear error naming the table on the first operation, instead of failing later with an opaque FFI deserialization error. **Nested attributes work in both versions**, with different authoring syntax. @@ -156,29 +158,34 @@ Which domains give you a `__hmac` attribute you can query on: ```typescript import { DynamoDBClient } from "@aws-sdk/client-dynamodb" import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb" -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack" import { encryptedDynamoDB } from "@cipherstash/stack/dynamodb" const dynamoClient = new DynamoDBClient({ region: "us-east-1" }) const docClient = DynamoDBDocumentClient.from(dynamoClient) -const encryptionClient = await EncryptionV3({ schemas: [users] }) +const encryptionClient = await Encryption({ schemas: [users] }) const dynamo = encryptedDynamoDB({ encryptionClient }) ``` -> **Audit metadata on decrypt:** the client from `EncryptionV3` has no audit -> surface on its decrypt methods, so `.audit()` on `decryptModel` / -> `bulkDecryptModels` resolves normally but the metadata is not recorded. If you -> need audited decrypts, build the client with `Encryption({ schemas: [users], -> config: { eqlVersion: 3 } })` instead — same v3 wire format, chainable decrypt. +> **Audit metadata on decrypt works.** `decryptModel` / `bulkDecryptModels` are +> audit-chainable — `dynamo.decryptModel(item, table).audit({ metadata })` forwards +> the metadata to ZeroKMS, whether the client came from `Encryption` or the +> deprecated `EncryptionV3` alias. (`EncryptionV3` is now a deprecated, type-identical +> alias of `Encryption`; prefer `Encryption`.) + +### EQL v2 Schema (reading existing deployments) -### EQL v2 Schema (existing deployments) +A v2 table is **read-only** through this adapter now: pass it to `decryptModel` / +`bulkDecryptModels` to read items written before the v3 cutover. `encryptModel` / +`bulkEncryptModels` no longer accept a v2 table — author new writes with an EQL v3 +table. ```typescript import { encryptedTable, encryptedColumn, encryptedField } from "@cipherstash/stack/schema" import { Encryption } from "@cipherstash/stack" -const users = encryptedTable("users", { +const usersV2 = encryptedTable("users", { email: encryptedColumn("email").equality(), // searchable via HMAC name: encryptedColumn("name"), // encrypt-only, no search metadata: encryptedColumn("metadata").dataType("json"), @@ -187,8 +194,11 @@ const users = encryptedTable("users", { }, }) -const encryptionClient = await Encryption({ schemas: [users] }) +const encryptionClient = await Encryption({ schemas: [usersV2] }) const dynamo = encryptedDynamoDB({ encryptionClient }) + +// Read existing v2 items — decrypt still accepts the v2 table. +const decrypted = await dynamo.decryptModel(storedV2Item, usersV2) ``` > **Note:** `encryptedColumn` also supports `.orderAndRange()`, `.freeTextSearch()`, and `.searchableJson()` index methods, but only `.equality()` produces HMAC values usable for DynamoDB key condition queries. @@ -456,7 +466,7 @@ if (result.failure) { import { encryptedDynamoDB } from "@cipherstash/stack/dynamodb" const dynamo = encryptedDynamoDB({ - // From EncryptionV3(...) or Encryption(...) + // From Encryption(...) (or the deprecated EncryptionV3(...) alias) encryptionClient, options: { // optional logger: { error: (message, error) => void }, @@ -467,9 +477,7 @@ const dynamo = encryptedDynamoDB({ ### Instance Methods -`table` is either an EQL v3 table (`types.*` domains) or an EQL v2 one. - -Each method is overloaded on the table's EQL version. +**Encrypt accepts an EQL v3 table only. Decrypt accepts an EQL v3 OR an EQL v2 table** (so stored v2 items stay readable). The decrypt methods are overloaded on the table's EQL version. **EQL v3** — the item is checked against the table's column domains, and the result is typed as the attribute map that is actually stored: a declared column `email` becomes `email__source` (plus `email__hmac` if its domain mints one), NOT `email`. @@ -482,16 +490,14 @@ Each method is overloaded on the table's EQL version. Let `T` be inferred from the argument; do not pass explicit type arguments on the v3 path. -**EQL v2** — unchanged. `T` is the model you pass in, so the result type does NOT reflect the `__source` / `__hmac` split; declare a type of your own to read those attributes. +**EQL v2 — decrypt (read) only.** The v2 write overloads were removed; `encryptModel` / `bulkEncryptModels` no longer accept a v2 table. `decryptModel` / `bulkDecryptModels` still do, so existing v2 items decrypt. `T` is the model shape you expect back; declare a type of your own to read the stored `__source` / `__hmac` attributes. | Method | Signature | Resolves to | |---|---|---| -| `encryptModel` | `(item: T, v2Table)` | `T` | -| `bulkEncryptModels` | `(items: T[], v2Table)` | `T[]` | | `decryptModel` | `(item: Record, v2Table)` | `T` | | `bulkDecryptModels` | `(items: Record[], v2Table)` | `T[]` | -All operations are thenable (awaitable) and support `.audit({ metadata })` chaining. See the note in Setup about decrypt-side audit metadata and the `EncryptionV3` client. +All operations are thenable (awaitable) and support `.audit({ metadata })` chaining — including the decrypt methods, whose metadata now forwards to ZeroKMS regardless of client shape (see the Setup note). Types exported from `@cipherstash/stack/dynamodb`: `EncryptedDynamoDBInstance`, `EncryptedDynamoDBConfig`, `EncryptedDynamoDBError`, `AnyEncryptedTable`, `DynamoDBEncryptionClient`, `EncryptedAttributes`, `DecryptedAttributes`, `AuditConfig`. @@ -529,7 +535,8 @@ const v2Hmac = v2Result.data.hm ```typescript import { DynamoDBClient } from "@aws-sdk/client-dynamodb" import { DynamoDBDocumentClient, PutCommand, GetCommand, QueryCommand } from "@aws-sdk/lib-dynamodb" -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, types } from "@cipherstash/stack/v3" import { encryptedDynamoDB } from "@cipherstash/stack/dynamodb" // Schema @@ -541,7 +548,7 @@ const users = encryptedTable("users", { // Clients const dynamoClient = new DynamoDBClient({ region: "us-east-1" }) const docClient = DynamoDBDocumentClient.from(dynamoClient) -const encryptionClient = await EncryptionV3({ schemas: [users] }) +const encryptionClient = await Encryption({ schemas: [users] }) const dynamo = encryptedDynamoDB({ encryptionClient }) // Write diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 227df0643..c140d2bfd 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -1,13 +1,13 @@ --- name: stash-encryption -description: Implement field-level encryption with @cipherstash/stack using the EQL v3 typed schema. Covers the types.* column catalog (concrete Postgres domains with fixed query capabilities), the strongly-typed EncryptionV3 client, encrypt/decrypt and model operations, searchable encryption (equality, free-text, range), encrypted JSON (containment and JSONPath selectors), bulk operations, identity-aware encryption with lock contexts, multi-tenant keysets, and the rollout/cutover lifecycle. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. +description: Implement field-level encryption with @cipherstash/stack using the EQL v3 typed schema. Covers the types.* column catalog (concrete Postgres domains with fixed query capabilities), the strongly-typed Encryption client (with EncryptionV3 as a deprecated alias), encrypt/decrypt and model operations, searchable encryption (equality, free-text, range), encrypted JSON (containment and JSONPath selectors), bulk operations, identity-aware encryption with lock contexts, multi-tenant keysets, and the rollout/cutover lifecycle. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. --- # CipherStash Stack - Encryption Comprehensive guide for implementing field-level encryption with `@cipherstash/stack`. Every value is encrypted with its own unique key via ZeroKMS (backed by AWS KMS). Encryption happens client-side before data leaves the application. -Encrypted columns are **EQL v3 concrete Postgres domains** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...): each column's query capabilities are fixed by the domain type you pick in the schema, and the `EncryptionV3` client derives precise TypeScript types from that schema — wrong-typed plaintext is a compile error, not a runtime surprise. +Encrypted columns are **EQL v3 concrete Postgres domains** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...): each column's query capabilities are fixed by the domain type you pick in the schema, and the `Encryption` client (typed for an all-v3 schema set) derives precise TypeScript types from that schema — wrong-typed plaintext is a compile error, not a runtime surprise. > An older schema surface (EQL v2, with chainable capability builders) still exists for existing deployments — see "Legacy: EQL v2" at the end. Everything else in this skill is the v3 surface. New projects should use v3. @@ -25,13 +25,14 @@ Encrypted columns are **EQL v3 concrete Postgres domains** (`public.eql_v3_text_ ## Quick Start ```typescript -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, types } from "@cipherstash/stack/v3" const users = encryptedTable("users", { email: types.TextSearch("email"), }) -const client = await EncryptionV3({ schemas: [users] }) +const client = await Encryption({ schemas: [users] }) const encrypted = await client.encrypt("secret@example.com", { table: users, @@ -145,7 +146,7 @@ profile. ### Programmatic Config ```typescript -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -181,16 +182,16 @@ The SDK never logs plaintext data. | Import Path | Provides | |---|---| -| `@cipherstash/stack/v3` | `EncryptionV3` factory, `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3. | +| `@cipherstash/stack/v3` | `EncryptionV3` (a deprecated alias of `Encryption`), `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3 schema authoring. | | `@cipherstash/stack/eql/v3` | `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, `V3ModelInput`, ...) | -| `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the untyped `Encryption` function (also accepts v3 schemas), legacy v2 re-exports | +| `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the `Encryption` function (typed for an all-v3 schema set; nominal for v2/loose schemas), legacy v2 re-exports | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/errors` | `EncryptionErrorTypes`, `StackError`, error subtypes, `getErrorMessage` | | `@cipherstash/stack/types` | All TypeScript types | | `@cipherstash/stack-drizzle/v3` | Drizzle ORM integration for EQL v3 schemas (see the `stash-drizzle` skill) | | `@cipherstash/stack-supabase` | `encryptedSupabaseV3` wrapper for Supabase (see the `stash-supabase` skill) | | `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. | -| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — **still requires the legacy v2 schema surface**; see "Legacy: EQL v2" below | +| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items. See the `stash-dynamodb` skill | | `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | ## Schema Definition @@ -289,12 +290,13 @@ CREATE TABLE users ( ); ``` -## Client Initialization: `EncryptionV3` +## Client Initialization: `Encryption` -`EncryptionV3` from `@cipherstash/stack/v3` returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities: +`Encryption` from `@cipherstash/stack` is overloaded: hand it an array of concrete EQL v3 tables and it returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities. `encryptedTable` / `types` are imported from `@cipherstash/stack/v3`: ```typescript -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, types } from "@cipherstash/stack/v3" const users = encryptedTable("users", { email: types.TextSearch("email"), @@ -302,18 +304,19 @@ const users = encryptedTable("users", { balance: types.BigintEq("balance"), }) -const client = await EncryptionV3({ schemas: [users] }) +const client = await Encryption({ schemas: [users] }) ``` -- The wire format is pinned to EQL v3 (`eqlVersion: 3`); you don't set it yourself. -- `EncryptionV3()` throws on init error (bad credentials, missing config, invalid keyset UUID). At least one schema is required. -- `typedClient(client, ...schemas)` (also exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface. -- The untyped `Encryption({ schemas })` from `@cipherstash/stack` also accepts v3 tables (it auto-detects them and sets the v3 wire format), but you lose the per-column typing — prefer `EncryptionV3`. **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. +- The wire format is auto-detected as EQL v3 for an all-v3 schema set; you don't set it yourself. +- `Encryption()` throws on init error (bad credentials, missing config, invalid keyset UUID). At least one schema is required. +- `EncryptionV3` (from `@cipherstash/stack/v3`) is a **deprecated, type-identical alias** of `Encryption`, kept for backwards compatibility. New code should use `Encryption`. +- `typedClient(client, ...schemas)` (exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface, if you built one via a lower-level path. +- **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. ```typescript // Error handling try { - const client = await EncryptionV3({ schemas: [users] }) + const client = await Encryption({ schemas: [users] }) } catch (error) { console.error("Init failed:", error.message) } @@ -426,7 +429,7 @@ for (const item of decrypted.data) { **On the WASM entry (`@cipherstash/stack/wasm-inline`), the batch shape differs** — do not copy the shape above onto the edge. The `{ data } | { failure }` Result is the same, but there are no `{ id, plaintext }` envelopes: each entry carries its own table and column, and the payload is a plain index-aligned array. > [!IMPORTANT] -> The `client` below is a **different client** from the one used everywhere else in this skill. The edge entry has its own `Encryption` factory — the native `EncryptionV3` client's `bulkEncrypt` takes `(plaintexts, { table, column })` and will fail at runtime if given the per-item shape below. Construct the WASM client explicitly: +> The `client` below is a **different client** from the one used everywhere else in this skill. The edge entry has its own `Encryption` factory — the native `Encryption` client's `bulkEncrypt` takes `(plaintexts, { table, column })` and will fail at runtime if given the per-item shape below. Construct the WASM client explicitly: ```typescript // Deno / Workers / Supabase Edge Functions — note the import path @@ -624,7 +627,8 @@ The client authenticates to ZeroKMS through `config.authStrategy`. Leave it unse ```typescript import { OidcFederationStrategy } from "@cipherstash/stack" -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, types } from "@cipherstash/stack/v3" const users = encryptedTable("users", { email: types.TextSearch("email") }) @@ -638,7 +642,7 @@ if (strategy.failure) { throw new Error(`[auth] ${strategy.failure.type}: ${strategy.failure.error.message}`) } -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { authStrategy: strategy.data }, }) @@ -703,13 +707,13 @@ Isolate encryption keys per tenant: ```typescript // By name -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { keyset: { name: "Company A" } }, }) // By UUID -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" } }, }) @@ -987,7 +991,7 @@ await runBackfill({ }) ``` -Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. `runBackfill` is version-agnostic — for an EQL v3 column pass an `EncryptionV3` client (from `@cipherstash/stack/v3`) and it writes v3 envelopes straight into the concrete `eql_v3_*` domain column. +Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. `runBackfill` is version-agnostic — for an EQL v3 column pass an `Encryption` client built from a v3 schema set and it writes v3 envelopes straight into the concrete `eql_v3_*` domain column. ### Invariants the rollout preserves @@ -1004,7 +1008,7 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a | Drizzle ORM | `@cipherstash/stack-drizzle/v3` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | | Supabase | `encryptedSupabaseV3` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | | Prisma | `@cipherstash/prisma-next` — searchable field-level encryption for Postgres | — | -| DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — **v2 schema surface only**, see the exception note under "Legacy: EQL v2" | `stash-dynamodb` | +| DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — encrypt is **EQL v3 only**; decrypt still reads existing v2 items | `stash-dynamodb` | ## Complete API Reference @@ -1052,4 +1056,4 @@ const client = await Encryption({ schemas: [users] }) The scalar v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Drizzle/Supabase integrations (`@cipherstash/stack-drizzle`, `encryptedSupabase`), and `stash eql install --eql-version 2` — still exists for existing deployments. Legacy v2 `searchableJson()` is the exception: protect-ffi 0.30 cannot emit the removed selector envelope, so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) -> **Exception — DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) **still requires the v2 schema surface** — `encryptedTable`/`encryptedColumn`/`encryptedField` from `@cipherstash/stack/schema`. Do not use v3 `types.*` schemas with it. See the `stash-dynamodb` skill; v3 support is tracked in [cipherstash/stack#657](https://github.com/cipherstash/stack/issues/657). +> **DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) now **encrypts EQL v3 only** — author tables with `types.*` from `@cipherstash/stack/eql/v3`. Its decrypt methods still accept a v2 table so previously stored v2 items remain readable. See the `stash-dynamodb` skill. From 5a0ed58ad43b026c87baf10a2d774e23b9834c63 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:44:48 +1000 Subject: [PATCH 006/123] fix(stack): typed client tolerates nominal-arity decrypt calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verification of the EncryptionV3->Encryption collapse surfaced a regression: because Encryption now returns the TYPED client for a v3 schema set, stack-supabase's encryptedSupabaseV3 (which casts to the nominal overload and calls decryptModel(row)/bulkDecryptModels(rows) one-arg) hit the typed methods with no table argument and threw on undefined.tableName. The sibling build passed because it is a type/runtime mismatch with no v3 runtime coverage. Fix: the typed client's decryptModel/bulkDecryptModels now tolerate a missing table, degrading to nominal behaviour (decrypt without date reconstruction) instead of dereferencing undefined — restoring exactly what stack-supabase received before the collapse. table stays required for genuinely-typed callers. - Add regression tests: typed client one-arg decryptModel/bulkDecryptModels decrypt without throwing (no reconstruction). - Correct now-stale customer-visible source docs flagged in review: the resolveDecryptResult docblock, the encryptedDynamoDB version-mismatch error string and @example, and the EncryptedDynamoDBConfig JSDoc (all pointed at EncryptionV3 / needless eqlVersion:3). Update construct-guard.test.ts to the new error wording. Follows 686004f8. --- .../dynamodb/construct-guard.test.ts | 4 +- .../stack/__tests__/typed-client-v3.test.ts | 48 +++++++++++++++++++ packages/stack/src/dynamodb/helpers.ts | 10 ++-- packages/stack/src/dynamodb/index.ts | 9 ++-- packages/stack/src/dynamodb/types.ts | 7 ++- packages/stack/src/encryption/v3.ts | 41 ++++++++++++---- 6 files changed, 97 insertions(+), 22 deletions(-) diff --git a/packages/stack/__tests__/dynamodb/construct-guard.test.ts b/packages/stack/__tests__/dynamodb/construct-guard.test.ts index 88c95e997..9ab09ea72 100644 --- a/packages/stack/__tests__/dynamodb/construct-guard.test.ts +++ b/packages/stack/__tests__/dynamodb/construct-guard.test.ts @@ -68,7 +68,9 @@ describe('encryptedDynamoDB client/table version guard', () => { message = (e as Error).message } expect(message).toContain('users_v3') - expect(message).toMatch(/EncryptionV3|eqlVersion/) + // Post-collapse the fix guidance points at `Encryption` (which auto-selects + // the v3 wire format for a v3 schema set), not `EncryptionV3`/`eqlVersion`. + expect(message).toMatch(/build it with Encryption\(/) }) it('guards every operation method, not just encryptModel', () => { diff --git a/packages/stack/__tests__/typed-client-v3.test.ts b/packages/stack/__tests__/typed-client-v3.test.ts index 457fc8378..c506d787f 100644 --- a/packages/stack/__tests__/typed-client-v3.test.ts +++ b/packages/stack/__tests__/typed-client-v3.test.ts @@ -140,4 +140,52 @@ describe('typedClient — decrypt reconstruction', () => { const data = result.data as Record expect(data.when).toBeInstanceOf(Date) }) + + // `Encryption` now returns THIS typed client for a v3 schema set, so a consumer + // typed against the nominal overload (e.g. stack-supabase's query builder, + // which casts to it and calls the one-arg `decryptModel(row)` / + // `bulkDecryptModels(rows)`) reaches the typed methods with NO table argument. + // They must decrypt without throwing — degrading to nominal behaviour (no date + // reconstruction) — not dereference `undefined.tableName`. + it('tolerates a one-arg (nominal-style) decryptModel call with no table', async () => { + const client = typedClient( + fakeClient({ when: '2020-01-02T03:04:05.000Z', note: 'hi' }), + table, + ) + // The typed signature forbids the one-arg form; a nominal-typed caller does + // it at runtime. Exercise that runtime path. + // biome-ignore lint/suspicious/noExplicitAny: exercising the nominal-arity runtime path + const decryptOneArg = client.decryptModel as any + + const result = await decryptOneArg({ + when: '2020-01-02T03:04:05.000Z', + note: 'hi', + }) + expect(result.failure).toBeFalsy() + if (result.failure) return + + const data = result.data as Record + // No table → no reconstruction: `when` stays the raw string, exactly as the + // nominal client would return it. + expect(data.when).toBe('2020-01-02T03:04:05.000Z') + expect(data.note).toBe('hi') + }) + + it('tolerates a one-arg (nominal-style) bulkDecryptModels call with no table', async () => { + const client = typedClient( + fakeClient({ when: '2021-06-01T00:00:00.000Z', note: 'x' }), + table, + ) + // biome-ignore lint/suspicious/noExplicitAny: exercising the nominal-arity runtime path + const bulkOneArg = client.bulkDecryptModels as any + + const result = await bulkOneArg([{ when: '2021-06-01T00:00:00.000Z' }]) + expect(result.failure).toBeFalsy() + if (result.failure) return + + const rows = result.data as Array> + expect(rows).toHaveLength(1) + // No table → no reconstruction: raw string, not a Date. + expect(rows[0].when).toBe('2021-06-01T00:00:00.000Z') + }) }) diff --git a/packages/stack/src/dynamodb/helpers.ts b/packages/stack/src/dynamodb/helpers.ts index 68966d409..67be28a68 100644 --- a/packages/stack/src/dynamodb/helpers.ts +++ b/packages/stack/src/dynamodb/helpers.ts @@ -86,11 +86,11 @@ export function handleError( /** * Resolve a decrypt call against either client shape. * - * The nominal `EncryptionClient` returns a chainable operation that carries - * `.audit()`; the `TypedEncryptionClient` from `EncryptionV3` returns a plain - * `Promise>`. Chain the audit metadata when the client can carry it, - * otherwise await the promise directly — a typed client has no audit surface - * on decrypt, so the metadata has nowhere to go. + * Both the nominal `EncryptionClient` and the typed client return a chainable + * operation carrying `.audit()` on decrypt (the typed client's is a + * `MappedDecryptOperation`). Chain the audit metadata onto it; the branch that + * awaits a bare promise remains only for a non-conforming custom client that + * exposes no `.audit()`. Audit metadata is forwarded regardless of client shape. */ export async function resolveDecryptResult( operation: unknown, diff --git a/packages/stack/src/dynamodb/index.ts b/packages/stack/src/dynamodb/index.ts index 9e2c60aad..a950c8f7c 100644 --- a/packages/stack/src/dynamodb/index.ts +++ b/packages/stack/src/dynamodb/index.ts @@ -57,7 +57,7 @@ function assertClientTableVersionMatch( if (table.tableName in config.tables) return throw new Error( - `encryptedDynamoDB: EQL version mismatch — the EQL v3 table "${table.tableName}" is not registered with this encryption client, so the client is not in EQL v3 mode for it. A v3 table requires a v3-mode client: build it with EncryptionV3({ schemas: [
] }) (or Encryption({ schemas: [
], config: { eqlVersion: 3 } })) and pass that client to encryptedDynamoDB. Otherwise encrypt/decrypt fails later inside the FFI with an opaque deserialization error.`, + `encryptedDynamoDB: EQL version mismatch — the EQL v3 table "${table.tableName}" is not registered with this encryption client, so the client is not in EQL v3 mode for it. A v3 table requires a v3-mode client: build it with Encryption({ schemas: [
] }) and pass that client to encryptedDynamoDB. Otherwise encrypt/decrypt fails later inside the FFI with an opaque deserialization error.`, ) } @@ -85,7 +85,8 @@ function assertClientTableVersionMatch( * * @example EQL v3 * ```typescript - * import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + * import { Encryption } from "@cipherstash/stack" + * import { encryptedTable, types } from "@cipherstash/stack/v3" * import { encryptedDynamoDB } from "@cipherstash/stack/dynamodb" * * const users = encryptedTable("users", { @@ -93,13 +94,13 @@ function assertClientTableVersionMatch( * name: types.Text("name"), // storage only * }) * - * const client = await EncryptionV3({ schemas: [users] }) + * const client = await Encryption({ schemas: [users] }) * const dynamo = encryptedDynamoDB({ encryptionClient: client }) * * const encrypted = await dynamo.encryptModel({ email: "a@b.com" }, users) * ``` * - * @example EQL v2 (existing deployments) + * @example EQL v2 (reading existing deployments — decrypt only) * ```typescript * import { Encryption } from "@cipherstash/stack" * import { encryptedDynamoDB } from "@cipherstash/stack/dynamodb" diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index d661928a3..9dfebb8cb 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -94,10 +94,9 @@ export type CallableEncryptionClient = { export interface EncryptedDynamoDBConfig { /** - * Either the nominal client from `Encryption(...)` / `Encryption({ schemas, - * config: { eqlVersion: 3 } })`, or the typed client from `EncryptionV3(...)`. - * For EQL v3 tables the client must be in v3 mode — `EncryptionV3` forces - * this; with `Encryption` you must pass `config: { eqlVersion: 3 }` yourself. + * The client from `Encryption(...)` (or the deprecated `EncryptionV3(...)` + * alias). For an EQL v3 schema set `Encryption` auto-selects the v3 wire format + * and returns the typed client — no `config: { eqlVersion: 3 }` needed. */ encryptionClient: EncryptionClient | DynamoDBEncryptionClient options?: { diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 34c4602c8..e36c90731 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -218,6 +218,16 @@ export function typedClient( }, } + // Pass-through maps for a one-arg (nominal-style) decrypt call, where `table` + // is absent: decrypt WITHOUT date reconstruction, exactly as the nominal + // `EncryptionClient` does. This client is now what `Encryption` returns for a + // v3 schema set, so a consumer typed against the nominal overload (e.g. + // stack-supabase's query builder, which casts to it) can call `decryptModel(x)` + // / `bulkDecryptModels(xs)` with no table. Degrade gracefully instead of + // dereferencing `undefined.tableName`. + const passthroughRow = (row: Record) => row + const passthroughRows = (rows: Array>) => rows + // Overloaded so the implementation is checked against BOTH forms directly — // no whole-value cast. The two public signatures mirror the interface member; // the hidden implementation signature is broad and forwards to the nominal @@ -258,9 +268,14 @@ export function typedClient( client.bulkEncryptModels(input as never, table as never) as never, decrypt: (encrypted) => client.decrypt(encrypted), decryptModel: (input, table, lockContext) => { - // `reconstruct` is undefined for a table this client was not initialized - // with; the mapped op then resolves to `unknownTableFailure` on execute. - const reconstruct = reconstructors.get(table.tableName) + // `table` is absent on a nominal-style one-arg call (see `passthroughRow`). + // Given a table: reconstruct dates from its cast_as, or — if it was never + // registered — leave `map` undefined so the mapped op resolves to + // `unknownTableFailure` on execute. + const maybeTable = table as AnyV3Table | undefined + const reconstruct = maybeTable + ? reconstructors.get(maybeTable.tableName) + : passthroughRow const op = client.decryptModel(input as never) const base = lockContext ? op.withLockContext(lockContext) : op return new MappedDecryptOperation( @@ -270,13 +285,23 @@ export function typedClient( ) as never }, bulkDecryptModels: (input, table, lockContext) => { - const reconstruct = reconstructors.get(table.tableName) + const maybeTable = table as AnyV3Table | undefined const op = client.bulkDecryptModels(input as never) const base = lockContext ? op.withLockContext(lockContext) : op - // The underlying op resolves to an array of rows; reconstruct each. - const mapRows = reconstruct - ? (rows: Array>) => rows.map(reconstruct) - : undefined + // No table → pass rows through (nominal behaviour). Registered table → + // reconstruct each row. Unregistered table → `undefined` map → + // `unknownTableFailure` on execute. + let mapRows: + | (( + rows: Array>, + ) => Array>) + | undefined + if (!maybeTable) { + mapRows = passthroughRows + } else { + const reconstruct = reconstructors.get(maybeTable.tableName) + mapRows = reconstruct ? (rows) => rows.map(reconstruct) : undefined + } return new MappedDecryptOperation( base, mapRows, From f587d7b81a08f698afb5323b536dad38002800f8 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:52:05 +1000 Subject: [PATCH 007/123] refactor(stack-supabase): fold v3 query builder into base Re-parameterize EncryptedQueryBuilderImpl over AnyV3Table and inline every EncryptedQueryBuilderV3Impl dialect override, then delete query-builder-v3.ts. Pure refactor: no runtime behaviour or wire encoding change. The decrypt path stays generation-agnostic (decryptModel/bulkDecryptModels), so stored EQL v2 payloads still decrypt (Decision 6). --- .../stack-supabase/src/query-builder-v3.ts | 989 ------------------ packages/stack-supabase/src/query-builder.ts | 938 ++++++++++++++--- 2 files changed, 818 insertions(+), 1109 deletions(-) delete mode 100644 packages/stack-supabase/src/query-builder-v3.ts diff --git a/packages/stack-supabase/src/query-builder-v3.ts b/packages/stack-supabase/src/query-builder-v3.ts deleted file mode 100644 index 8df37536c..000000000 --- a/packages/stack-supabase/src/query-builder-v3.ts +++ /dev/null @@ -1,989 +0,0 @@ -import { - DATE_LIKE_CASTS, - EncryptedV3Column, - logger, - matchNeedleError, - parseSelectorSegments, - reconstructSelectorDocument, - unsupportedLeafReason, -} from '@cipherstash/stack/adapter-kit' -import type { EncryptionClient } from '@cipherstash/stack/encryption' -import type { AnyV3Table } from '@cipherstash/stack/eql/v3' -import { - type EncryptionError, - EncryptionErrorTypes, -} from '@cipherstash/stack/errors' -import type { - ColumnSchema, - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' -import type { - BuildableQueryColumn, - Encrypted, - EncryptedQueryResult, - QueryTypeName, - ScalarQueryTerm, -} from '@cipherstash/stack/types' -import { addJsonbCastsV3, selectKeyToDbV3 } from './helpers' -import { - EncryptedQueryBuilderImpl, - EncryptionFailedError, -} from './query-builder' -import type { - DbName, - DbSelect, - FilterOp, - SupabaseClientLike, - SupabaseQueryBuilder, -} from './types' - -/** cast_as kinds that reconstruct to a JS `Date` — shared with the typed v3 - * client's decrypt-model path (see `encryption/v3.ts`). */ -const DATE_LIKE_CAST_SET = new Set(DATE_LIKE_CASTS) - -/** - * The subset of a v3 column builder the dialect relies on. Structural rather - * than the concrete class union so the runtime `instanceof EncryptedV3Column` - * gate and this type stay independent. - */ -type V3ColumnLike = { - getName(): string - getEqlType(): string - getQueryCapabilities(): { - equality: boolean - orderAndRange: boolean - freeTextSearch: boolean - /** Optional: only `public.eql_v3_json_search` (`types.Json`) carries it. */ - searchableJson?: boolean - } - build(): ColumnSchema -} - -/** - * Validate an encrypted-JSON containment operand: a NON-EMPTY plain object or a - * non-empty array. Everything else is rejected with an actionable steer: - * - * - Scalars/strings: the caller meant free-text (`matches` on a text column) or - * a selector — a raw JSON string is NOT parsed, by design (parsing would make - * `'{"a":1}'` and `{a:1}` silently different queries on other surfaces). - * - Non-plain objects (`Date`, `Map`, `RegExp`, class instances): these JSON- - * serialize to scalars or `{}` — not the sub-document the caller believes. - * - `{}` and `[]`: jsonb containment holds for EVERY document (`doc @> '{}'`), - * so an accidentally-empty needle would silently return (and decrypt) the - * whole table. The Drizzle adapter rejects the same needle for the same - * reason — the two first-party adapters must agree that this is an error. - */ -function assertJsonContainmentOperand(column: string, value: unknown): void { - const isPlainObject = - value !== null && - typeof value === 'object' && - !Array.isArray(value) && - (Object.getPrototypeOf(value) === Object.prototype || - Object.getPrototypeOf(value) === null) - if (!isPlainObject && !Array.isArray(value)) { - // Array.isArray is false on this branch by construction, so the label only - // distinguishes null / non-plain object / scalar. - const got = - value === null - ? 'null' - : typeof value === 'object' - ? (value as object).constructor?.name || 'a non-plain object' - : typeof value - throw new Error( - `[supabase v3]: encrypted JSON containment on column "${column}" takes a sub-document (plain object or array) to match, got ${got}.`, - ) - } - const empty = Array.isArray(value) - ? value.length === 0 - : Object.keys(value as object).length === 0 - if (empty) { - throw new Error( - `[supabase v3]: encrypted JSON containment on column "${column}" cannot take an empty ${Array.isArray(value) ? 'array' : 'object'} needle: it matches every row. Pass a non-empty sub-document, or omit the predicate to select all rows.`, - ) - } -} - -/** - * Reject a declared property name that is also a DIFFERENT physical column. - * - * `select('*')` expands the introspected DB names into property names, so a - * column renamed `created_at → createdAt` and a distinct plaintext column - * literally named `createdAt` both emit the token `createdAt`, which - * `addJsonbCastsV3` turns into `createdAt:created_at::jsonb` — twice. PostgREST - * returns the encrypted column under that key and the plaintext one is never - * selected, silently yielding the wrong value for a field the row type - * guarantees. - * - * Nothing downstream can disambiguate the two, and `EncryptedTable.build()`'s - * duplicate check only fires when two BUILDERS share a `getName()`. Refuse to - * construct instead. - */ -function assertNoPropertyDbNameCollision( - tableName: string, - propToDb: Record, - allColumns: string[] | null, -): void { - if (!allColumns) return - const dbNames = new Set(allColumns) - - for (const [property, dbName] of Object.entries(propToDb)) { - if (property === dbName) continue - if (!dbNames.has(property)) continue - throw new Error( - `[supabase v3]: property "${property}" on table "${tableName}" renames DB column "${dbName}", but "${property}" is also a distinct column in the database — the two collide in select('*'). Rename the property, or drop the declared rename.`, - ) - } -} - -/** - * EQL v3 dialect of {@link EncryptedQueryBuilderImpl} for native concrete-domain - * columns (`public.*` type domains, `eql_v3` operators). The query mechanism is - * v2's — direct EQL operators over PostgREST — with four narrow forks: - * - * - **Column recognition / naming** — v3 columns are `EncryptedV3Column` - * builders and may map a JS property name to a different DB column name - * (`buildColumnKeyMap`). Filters, select casts, and mutations resolve - * property → DB name; select casts alias the DB column back to the property - * (`prop:db_name::jsonb`) so result rows keep property keys. - * - **Mutation encoding** — the raw encrypted payload object is sent (the - * `public.*` domains are `DOMAIN … AS jsonb`), not v2's `{ data: … }` - * composite wrap. - * - **Query-term encoding** — scalar equality/range filters use the FULL - * storage envelope from `encrypt()`, serialized as jsonb text. - * - * NOT because narrowed terms fail the domain CHECK: the bundle defines a - * `public._query` companion for each storage domain, whose CHECK - * requires `NOT (VALUE ? 'c')` — i.e. it accepts exactly the no-ciphertext - * shape `encryptQuery` produces. Those domains are simply unreachable from - * here. PostgREST has no syntax to cast a filter VALUE, and an uncast literal - * is ambiguous between the `_query` and `jsonb` `@>`/`=` overloads (42725 — - * the bundle says so itself, see `cipherstash-encrypt-v3.sql`, the - * `_query_types.sql` note). The reachable overload is the `jsonb` one, whose - * body coerces its operand to the STORAGE domain, which does require `c`. - * (protect-ffi can mint narrowed `eql_v3.query_` operands via - * `encryptQuery`, but with no way to cast a PostgREST filter value they - * stay unreachable from this adapter.) - * - * The full envelope satisfies scalar storage-domain CHECKs by construction, - * and equality/range operators extract the term they need. - * - * EQL 3.0.2 removed the storage/jsonb escape hatch for free-text and JSON - * operators: those now require typed query-domain operands. The factory reads - * the installed EQL version and this builder fails those operators before - * encryption, so a decryptable storage envelope never enters a GET URL. - * - **Legacy `matches`, not `like`/`ilike`/`contains`** — on pre-3.0.2 EQL, - * encrypted free-text search is - * FUZZY BLOOM TOKEN MATCHING, not containment: the bundle declares `@>` on each - * match domain (`CREATE OPERATOR @> … FUNCTION = eql_v3.matches`, the SQL - * function's name), whose body is `match_term(a) @> match_term(b)` — `smallint[]` - * containment of the two bloom filters. It is order- and multiplicity- - * insensitive and one-sided (a `true` may be a false positive). PostgREST - * reaches it as `cs`. The operator is named `matches` to signal that; `contains` - * is reserved for exact (native) containment on plaintext columns. - * - * Match is tokenized + downcased, so `%` is NOT a wildcard. `like`/`ilike` on an - * encrypted column are delegated to `matches` as an APPROXIMATE compatibility - * shim (surrounding `%` stripped, internal `%`/`_` rejected, one warning) and - * pass through as real SQL LIKE on a plaintext column. - * - * Substrings DO match: the needle blooms to its own trigrams, and containment - * holds whenever every one of them is present in the stored value's bloom — - * i.e. for any substring of at least `token_length` (3) characters. Shorter - * needles bloom to nothing (`bf @> '{}'` is true for every row) and are - * rejected up front by `matchNeedleError`, not answered. - * - * Decrypted rows additionally get `Date` reconstruction from the - * encrypt-config `cast_as`, mirroring the typed v3 client. - */ -export class EncryptedQueryBuilderV3Impl< - T extends Record = Record, -> extends EncryptedQueryBuilderImpl { - private v3Table: AnyV3Table - /** JS property name → DB column name, for every encrypted column. */ - private propToDb: Record - /** DB column name → JS property name — the inverse of {@link propToDb}, used - * to expand `select('*')` back into property names. Null prototype: a DB - * column literally named `constructor` / `toString` would otherwise resolve - * to an inherited `Object.prototype` member and be emitted as a select token. */ - private dbToProp: Record - /** Built column schemas keyed by DB column name (for `cast_as`). */ - private columnSchemas: Record - /** Column builders keyed by BOTH property name and DB name. */ - private v3Columns: Record - /** EQL 3.0.2+ requires query-domain casts PostgREST cannot express. */ - private queryDomainsRequired: boolean - - constructor( - tableName: string, - table: AnyV3Table, - encryptionClient: EncryptionClient, - supabaseClient: SupabaseClientLike, - allColumns: string[] | null = null, - queryDomainsRequired = false, - ) { - super( - tableName, - // The base class only ever calls BuildableTable members on the schema - // (build / encryptModel plumbing); every v2-specific behaviour is - // overridden below. - table as unknown as EncryptedTable, - encryptionClient, - supabaseClient, - allColumns, - ) - - this.v3Table = table - this.queryDomainsRequired = queryDomainsRequired - this.propToDb = table.buildColumnKeyMap() - this.columnSchemas = table.build().columns - - this.dbToProp = Object.create(null) as Record - for (const [property, dbName] of Object.entries(this.propToDb)) { - this.dbToProp[dbName] = property - } - - assertNoPropertyDbNameCollision(tableName, this.propToDb, allColumns) - - // Null-prototype: keyed by DB column names, and `validateTransforms` reads - // it without an own-key guard — an inherited `constructor`/`toString` would - // otherwise resolve truthy for a plaintext column of that name. - this.v3Columns = Object.create(null) as Record - for (const [property, builder] of Object.entries(table.columnBuilders)) { - if (builder instanceof EncryptedV3Column) { - const col = builder as unknown as V3ColumnLike - this.v3Columns[property] = col - this.v3Columns[col.getName()] = col - } - } - - // The base class derives encrypted column names from build(), which v3 - // keys by DB name. Filters and select strings address columns by JS - // property name, so recognition must cover both. - this.encryptedColumnNames = Object.keys(this.v3Columns) - } - - // --------------------------------------------------------------------------- - // Dialect overrides - // --------------------------------------------------------------------------- - - protected override getColumnMap(): Record { - return this.v3Columns as unknown as Record - } - - /** Resolve a JS property name to its DB column name. `Object.hasOwn` guards - * the inherited-member hazard described on {@link EncryptedTable.buildColumnKeyMap}. */ - private dbNameFor(name: string): string { - return Object.hasOwn(this.propToDb, name) ? this.propToDb[name] : name - } - - protected override filterColumnName(column: string): DbName { - return this.dbNameFor(column) as DbName - } - - /** - * `ORDER BY` on an OPE-backed column is supported; on every other encrypted - * column it is rejected. - * - * A bare `ORDER BY col` IS wrong. The `*_ord` domains are - * `CREATE DOMAIN … AS jsonb`, and the bundle declares no btree operator class - * on any domain — it actively lints against one (`domain_opclass`), because an - * opclass on a domain bypasses operator resolution. So the sort resolves - * through jsonb's default `jsonb_cmp` and compares the envelope's keys in - * storage order, starting at the random ciphertext `c`. No error, and a - * stable, meaningless row order. (Measured: over 10 rows it returns - * `r00,r04,r08,r01,…` where the plaintext order is `r00..r09`.) - * - * But the correct sort key is reachable without a function call. `eql_v3.ord_term` - * returns the domain's `op` term, and OPE is order-preserving by construction: - * ordering by the term reproduces the plaintext order. PostgREST cannot emit - * `ORDER BY eql_v3.ord_term(col)`, but it CAN emit a jsonb path — - * `order=col->op.asc` — which selects exactly that term. Measured against a - * live PostgREST: `order=amount->op.asc` and `.desc` both reproduce the - * plaintext order for `integer_ord` and `text_search`, over 10 rows. - * - * So the guard is on the ordering FLAVOUR, not on encryption: - * - * - `ope` present → order by `col->op`. Every plain `_ord` domain, plus - * `text_ord` and `text_search`. - * - `ore` present → reject. The `ob` term is an array of ORE blocks whose - * comparison needs the superuser-only opclass; a jsonb-path sort over it is - * meaningless. (Such a column cannot hold data on managed Postgres at all: - * its domain CHECK raises `ore_domain_unavailable`.) - * - neither → reject. Storage-only, equality-only and match-only columns - * carry no ordering term to sort by. - * - * A column absent from {@link v3Columns} is a plaintext passthrough and orders - * normally. This runtime guard is the only protection the untyped - * (no-`schemas`) surface has. - */ - protected override validateTransforms(): void { - for (const t of this.transforms) { - if (t.kind !== 'order') continue - const column = this.v3Columns[t.column] - if (!column) continue - - const indexes = this.columnSchemas[column.getName()]?.indexes - if (indexes?.ope) continue - - const reason = indexes?.ore - ? 'its ORE ordering term (`ob`) needs the superuser-only ORE operator class, which PostgREST cannot reach through a jsonb path' - : 'it carries no ordering term to sort by' - - throw new Error( - `[supabase v3]: cannot order by encrypted column "${column.getName()}" (${column.getEqlType()}) — ${reason}. ` + - 'Order by a plaintext column, or use an OPE-backed ordering domain ' + - '(`*_ord`, `text_ord`, `text_search`), or use the EQL v3 Drizzle integration.', - ) - } - } - - /** - * Encrypted ordering columns sort by their `op` term, not by the envelope. - * - * `order=col->op` is the one ordering expression PostgREST can emit that - * reaches the OPE term. It must NOT leak into filters — those compare whole - * envelopes through the `eql_v3.*` operators — which is why this is its own - * seam rather than a change to `filterColumnName`. - * - * The canonical EQL form is `ORDER BY eql_v3.ord_term(col)`, which returns - * `eql_v3_internal.ope_cllw` — a domain over `bytea`, ordered by the native - * btree. PostgREST cannot call a function, so it orders the `op` term where it - * sits, inside the envelope. The two agree because the term is what - * `ord_term()` returns. - * - * `->` (jsonb) rather than `->>` (text) keeps the comparison on the typed - * value. Note this does NOT avoid the database collation: Postgres compares - * jsonb strings with `varstr_cmp` under the default collation, exactly as it - * does text. What makes the ordering collation-independent is the term itself - * — fixed-width lowercase hex (`[0-9a-f]`, 130 chars for `integer_ord`, 82 for - * `text_search`) — and every collation orders digits before letters and hex - * letters among themselves. `match-bloom`'s sibling assertion pins that shape. - * - * This runs at column-name-mapping time (`transformToDbSpace`), BEFORE - * `buildAndExecuteQuery` calls `validateTransforms`. For an encrypted column - * with no `ope` index it therefore returns a bare `dbName` here — a name that - * would sort by `jsonb_cmp` over the ciphertext if it reached PostgREST — but - * it never does: `validateTransforms` throws (with a domain-specific reason) - * before the query executes, so the bare name is only ever an intermediate - * value on a request that is about to be rejected. - */ - protected override orderColumnName(column: string): DbName { - const dbName = this.dbNameFor(column) - const encrypted = this.v3Columns[column] - if (!encrypted) return dbName as DbName - - return ( - this.columnSchemas[dbName]?.indexes?.ope ? `${dbName}->op` : dbName - ) as DbName - } - - /** - * Resolve a raw `.filter()` operator to the capability it exercises. Unlike - * v2, a supported v3 operand is a full storage envelope, so `queryType` - * never selects a narrowing — it only tells {@link encryptCollectedTerms} - * which capability to demand of the column. Getting it wrong therefore - * produces a wrong accept/reject, not a wrong ciphertext: the base class's - * `'equality'` default rejects `.filter('bio', 'cs', …)` on a - * `public.eql_v3_text_match` column, the one query that column can answer. - * - * Unknown operators throw rather than silently defaulting to equality, which - * would encrypt a term the column may not even be able to compare. - */ - protected override queryTypeForRawOp(operator: string): QueryTypeName { - switch (operator) { - case 'cs': - return 'freeTextSearch' - case 'gt': - case 'gte': - case 'lt': - case 'lte': - return 'orderAndRange' - case 'eq': - case 'neq': - case 'in': - case 'is': - return 'equality' - default: - throw new Error( - `[supabase v3]: unsupported raw filter operator "${operator}" on an encrypted column`, - ) - } - } - - protected override buildSelectString(): DbSelect | null { - if (this.selectColumns === null) return null - return addJsonbCastsV3(this.selectColumns, this.propToDb) - } - - /** - * Expand the introspected column list (DB names) into JS property names. - * - * Load-bearing for `select('*')` on a DECLARED table that renames a column. - * `addJsonbCastsV3` only emits the `prop:db_name::jsonb` alias — the thing - * that makes PostgREST return the column under its property name — when the - * token it sees is a property name. Feeding it the raw DB name instead takes - * the unaliased `dbNames.has(...)` branch, so the row comes back keyed - * `created_at` while the declared row type promises `createdAt`, silently - * yielding `undefined` for a field TypeScript guarantees. - * - * A DB column with no encrypted builder (plaintext passthrough, and every - * synthesized column, where property == DB name) maps to itself. - */ - protected override expandAllColumns(columns: string[]): string[] { - return columns.map((dbName) => - Object.hasOwn(this.dbToProp, dbName) ? this.dbToProp[dbName] : dbName, - ) - } - - /** v3 domains are plain jsonb — send the raw payload, keyed by DB name. */ - protected override transformEncryptedMutationModel( - model: Record, - ): Record { - const out: Record = Object.create(null) - for (const [key, value] of Object.entries(model)) { - out[this.dbNameFor(key)] = value - } - return out - } - - protected override transformEncryptedMutationModels( - models: Record[], - ): Record[] { - return models.map((model) => this.transformEncryptedMutationModel(model)) - } - - /** - * Validate a term's query type against its column's declared capabilities. - * Pure validation: `encrypt`/`bulkEncrypt` never receive the query type. On - * EQL 3.0.2+, free-text/JSON terms are rejected before this storage-encryption - * path can place ciphertext in a GET URL. - */ - private assertTermQueryable(term: ScalarQueryTerm): V3ColumnLike { - const column = term.column as unknown as V3ColumnLike - let queryType = term.queryType ?? 'equality' - const capabilities = column.getQueryCapabilities() - - // The `cs` wire operator is capability-overloaded: bloom free-text on a - // match/search TEXT column, encrypted ste_vec containment on a `types.Json` - // DOCUMENT column. Both arrive here as `freeTextSearch` (contains/matches/ - // raw `cs` all map there); resolve to the capability the column actually - // carries. The two are mutually exclusive by construction, so this can - // never reinterpret a real free-text column. - if ( - queryType === 'freeTextSearch' && - !capabilities.freeTextSearch && - capabilities.searchableJson - ) { - queryType = 'searchableJson' - } - - if ( - queryType !== 'equality' && - queryType !== 'orderAndRange' && - queryType !== 'freeTextSearch' && - queryType !== 'searchableJson' - ) { - throw new Error( - `[supabase v3]: query type "${queryType}" is not supported on EQL v3 columns`, - ) - } - - if (!capabilities[queryType]) { - throw new Error( - `[supabase v3]: column "${column.getName()}" (${column.getEqlType()}) does not support ${queryType} queries — declare the column with a domain that carries that capability`, - ) - } - - if (queryType === 'freeTextSearch' || queryType === 'searchableJson') { - // This is the common boundary for every spelling that collects an - // encrypted match/containment term: matches(), contains(), not(), raw - // filter(), and both forms of or(). Method-level checks provide earlier - // errors for the direct helpers, but cannot cover the inherited raw - // filter paths on their own. - this.assertPostgrestCanQueryEncryptedOperator('filter', column.getName()) - } - - if (queryType === 'searchableJson') { - // THE single enforced operand boundary for encrypted-JSON containment. - // Terms reach this resolver from every spelling — contains(), raw - // .filter(col,'cs',…), not(col,'contains'|'matches',…), and .or() - // string/structured conditions — and only contains() has a method-level - // guard. Without this check a raw string (e.g. a free-text term ported - // from a text column, or an .or() condition value, which is always a - // string) would be storage-encrypted as a JSON SCALAR and silently match - // nothing; pre-#650 every such spelling failed loudly on capability. - assertJsonContainmentOperand(column.getName(), term.value) - } - - // Free-text (bloom) needle floor. A needle shorter than the tokenizer's - // token_length produces NO tokens, so `bf @> '{}'` holds for every row and - // the query would silently return (and the caller decrypt) the whole table - // — a fail-open over-exposure. Reject it up front, mirroring the Drizzle v3 - // adapter (matchNeedleError) so both first-party surfaces guard identically. - // JSON containment terms (searchableJson) are validated separately above. - if (queryType === 'freeTextSearch') { - const match = column.build().indexes?.match - const reason = match ? matchNeedleError(term.value, match) : undefined - if (reason) { - throw new Error( - `[supabase v3]: cannot search column "${column.getName()}": ${reason}`, - ) - } - } - - return column - } - - private encryptionFailure(message: string, cause?: EncryptionError): never { - logger.error( - `Supabase: failed to encrypt query terms for table "${this.tableName}"`, - ) - // Most callers pass the operation's own `EncryptionError`; the contract- - // violation cases (bulk length mismatch, null envelope) have none, so - // synthesize one — a broken query encryption is still an encryption failure, - // and callers branch on `error.encryptionError` regardless. - throw new EncryptionFailedError( - `Failed to encrypt query terms: ${message}`, - cause ?? { type: EncryptionErrorTypes.EncryptionError, message }, - ) - } - - /** - * Encrypt every filter operand as a full storage envelope, serialized to jsonb - * text for the PostgREST filter value. - * - * Terms are grouped by column and each group takes ONE `bulkEncrypt` crossing. - * `in(col, [a, b, c])` collects one term per element (the list must never be - * encrypted whole), so encrypting per term spent N ZeroKMS/FFI round-trips - * where one would do. `bulkEncrypt` carries a single `{table, column}` for the - * whole payload, so the grouping is mandatory, not an optimisation: one bulk - * call over a mixed-column term array would stamp one column onto every - * plaintext. Results are scattered back onto the terms' original indices, - * which is the contract `termMap` downstream relies on. - * - * Mirrors `eql/v3/drizzle/operators.ts` `encryptOperands` — same batching - * contract, same length assertion, same fallback. Kept separate because that - * one encrypts a single-column operand list and returns `SQL[]`, while this - * must group a multi-column term array and preserve positions. - */ - protected override async encryptCollectedTerms( - terms: ScalarQueryTerm[], - ): Promise { - const groups = new Map< - V3ColumnLike, - { indices: number[]; values: ScalarQueryTerm['value'][] } - >() - terms.forEach((term, index) => { - const column = this.assertTermQueryable(term) - const group = groups.get(column) ?? { indices: [], values: [] } - group.indices.push(index) - group.values.push(term.value) - groups.set(column, group) - }) - - const bulkEncrypt = this.encryptionClient.bulkEncrypt?.bind( - this.encryptionClient, - ) - // Each term becomes the `JSON.stringify`'d storage envelope — a `string`, - // which is one arm of `EncryptedQueryResult`. PostgREST cannot cast a filter - // value to the `eql_v3.query_` twins, so v3 sends full envelopes where - // v2 sends `encryptQuery` composite literals; both are `EncryptedQueryResult`. - const results = new Array(terms.length) - - await Promise.all( - Array.from(groups, async ([column, { indices, values }]) => { - const encrypted = bulkEncrypt - ? await this.bulkEncryptGroup(bulkEncrypt, column, values) - : await this.encryptGroupPerTerm(column, values) - - encrypted.forEach((envelope, i) => { - results[indices[i]] = JSON.stringify(envelope) - }) - }), - ) - - return results - } - - /** One FFI crossing for a column's whole operand list. */ - private async bulkEncryptGroup( - bulkEncrypt: NonNullable, - column: V3ColumnLike, - values: ScalarQueryTerm['value'][], - ): Promise> { - const baseOp = bulkEncrypt( - values.map((plaintext) => ({ plaintext })) as never, - { column, table: this.v3Table } as never, - ) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) - this.encryptionFailure(result.failure.message, result.failure) - - // `bulkEncrypt` is position-stable, so a length mismatch means the contract - // was violated. Truncating instead would silently widen an `in` predicate - // (or narrow a `not.in`) to whatever came back. `result.data` is now - // `BulkEncryptedData` — `{ id?, data: Encrypted | null }[]` — not `unknown`. - const encrypted = result.data - if (encrypted.length !== values.length) { - this.encryptionFailure( - `bulk encryption returned ${encrypted.length} terms for ${values.length} values on column "${column.getName()}".`, - ) - } - return encrypted.map((term, i) => { - // `BulkEncryptedData` types the element as `Encrypted | null`. A `null` - // envelope here would be `JSON.stringify`'d to the literal string `"null"` - // and sent as the filter operand — silently matching whatever `"null"` - // encodes to rather than failing. A query term should never encrypt to a - // null envelope, so treat it as a contract violation, not a value. - if (term.data === null) { - this.encryptionFailure( - `bulk encryption returned a null envelope at position ${i} for column "${column.getName()}".`, - ) - } - return term.data - }) - } - - /** Fallback for a client that predates `bulkEncrypt`. */ - private async encryptGroupPerTerm( - column: V3ColumnLike, - values: ScalarQueryTerm['value'][], - ): Promise { - return Promise.all( - values.map(async (value) => { - const baseOp = this.encryptionClient.encrypt(value, { - column, - table: this.v3Table, - }) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) { - this.encryptionFailure(result.failure.message, result.failure) - } - return result.data - }), - ) - } - - /** Warn once per (op, column) that a `like`/`ilike` was delegated to `matches`. */ - private static readonly warnedLikeDelegation = new Set() - - /** True when `column` is one of this table's encrypted v3 columns. */ - private isEncryptedV3Column(column: string): boolean { - return Boolean(this.v3Columns[column]) - } - - /** True when `column` is an encrypted `types.Json` document column. */ - private isSearchableJsonColumn(column: string): boolean { - const builder: V3ColumnLike | undefined = this.v3Columns[column] - return Boolean(builder?.getQueryCapabilities().searchableJson) - } - - private assertPostgrestCanQueryEncryptedOperator( - method: string, - column: string, - ): void { - if (!this.queryDomainsRequired) return - throw new Error( - `[supabase v3]: ${method}() on encrypted column "${column}" is unavailable with EQL 3.0.2+: the SQL operator requires an eql_v3.query_* cast that PostgREST cannot express. Use the Drizzle or Prisma Next adapter, or a scoped SQL/RPC function.`, - ) - } - - /** - * `contains` on the v3 surface is EXACT containment: native jsonb/array `@>` - * on a plaintext column, ENCRYPTED ste_vec `@>` on a `types.Json` column (the - * sub-document operand is storage-encrypted whole; every leaf must match at - * its path — #650). On an encrypted match/search TEXT column containment is - * not the operation (that is the fuzzy `matches`), so refuse loudly rather - * than silently emit a bloom match under a name that promises exactness. - */ - override contains(column: string, value: unknown): this { - if (this.isSearchableJsonColumn(column)) { - this.assertPostgrestCanQueryEncryptedOperator('contains', column) - // Same validator the term resolver enforces — failing here just surfaces - // the error at the call site instead of at execution. - assertJsonContainmentOperand(column, value) - return super.contains(column, value) - } - if (this.isEncryptedV3Column(column)) { - throw new Error( - `[supabase v3]: contains() is native (exact) containment and does not apply to encrypted column "${column}". Use matches() for encrypted free-text search.`, - ) - } - return super.contains(column, value) - } - - /** - * `matches` is the encrypted free-text operator: fuzzy bloom-filter token - * matching, one-sided (may false-positive), NOT containment. It requires an - * encrypted match/search column; on a plaintext column, `contains` (native - * `@>`) is what the caller means — and on an encrypted JSON column, - * `contains`/`selectorEq` are (matching a document is containment, not - * free-text). Guarded here because both spellings collect the same - * `freeTextSearch` term, which the capability resolver would otherwise - * silently accept as containment of the raw string. - */ - override matches(column: string, value: unknown): this { - if (this.isSearchableJsonColumn(column)) { - throw new Error( - `[supabase v3]: matches() is encrypted free-text search and does not apply to encrypted JSON column "${column}". Use contains("${column}", subDocument) or selectorEq("${column}", path, value).`, - ) - } - if (!this.isEncryptedV3Column(column)) { - throw new Error( - `[supabase v3]: matches() is encrypted free-text search and requires an encrypted column; "${column}" is not one. Use contains() for native containment.`, - ) - } - this.assertPostgrestCanQueryEncryptedOperator('matches', column) - return super.matches(column, value) - } - - /** - * `not(col, 'contains', …)` on an encrypted TEXT column would negate a fuzzy - * bloom match under the `contains` name — the exact confusion #617 removes — - * because the base `not()` path rewrites the `contains` spelling to the `cs` - * wire operator. Reject it and steer to the `matches` spelling (or the raw - * `cs` operator, which is honest about the wire op). - * - * On an encrypted JSON column negated containment IS the honest exact - * operation (`not.cs` over ste_vec containment — {@link selectorNe} compiles - * to it), so it passes through. Plaintext columns keep native negated - * containment, and every other operator is delegated unchanged. - */ - override not(column: string, operator: string, value: unknown): this { - if ( - operator === 'contains' && - this.isEncryptedV3Column(column) && - !this.isSearchableJsonColumn(column) - ) { - throw new Error( - `[supabase v3]: not("${column}", 'contains', …) does not apply to encrypted column "${column}" — that is fuzzy free-text matching, not containment. Use not("${column}", 'matches', …) or the raw 'cs' operator.`, - ) - } - // Mirror of the matches() guard: a `matches` spelling on a JSON column - // would otherwise resolve to containment (the two share the `cs` wire op), - // silently negating an EXACT operation under a name that promises FUZZY. - if (operator === 'matches' && this.isSearchableJsonColumn(column)) { - throw new Error( - `[supabase v3]: not("${column}", 'matches', …) does not apply to encrypted JSON column "${column}" — matches() is free-text search. Use not("${column}", 'contains', subDocument) or selectorNe("${column}", path, value).`, - ) - } - return super.not(column, operator, value) - } - - /** - * Validate + reconstruct a selector needle: `('$.user.role', 'admin')` → - * `{user: {role: 'admin'}}`. Shared by {@link selectorEq}/{@link selectorNe}; - * throws with column context for a non-JSON column, an invalid path, or a - * non-scalar leaf. - */ - private selectorNeedle( - method: string, - column: string, - path: string, - value: unknown, - ): Record { - if (!this.isSearchableJsonColumn(column)) { - throw new Error( - `[supabase v3]: ${method}() requires an encrypted JSON (types.Json) column; "${column}" is not one.`, - ) - } - // Selector comparisons compare a scalar LEAF (null included in the shared - // helper's rejection; eq/ne arm — `ordering: false`; - // PostgREST cannot express selector ordering yet, see - // cipherstash/encrypt-query-language#407). - const leafReason = unsupportedLeafReason(value, false) - if (leafReason) { - throw new Error( - `[supabase v3]: ${method}("${column}", "${path}", …): ${leafReason}`, - ) - } - // Stricter than the shared helper (whose Date/bigint arms serve the Drizzle - // surface): a stored JsonDocument leaf is a JSON scalar, so a Date/bigint - // needle could never match one — reject with the serialization steer - // instead of running a query that structurally returns nothing. - if ( - typeof value !== 'string' && - typeof value !== 'number' && - typeof value !== 'boolean' - ) { - throw new Error( - `[supabase v3]: ${method}("${column}", "${path}", …): a JSON document leaf is a JSON scalar (string/number/boolean); got ${value instanceof Date ? 'a Date — pass date.toISOString() (or the stored form)' : typeof value}.`, - ) - } - let segments: string[] - try { - segments = parseSelectorSegments(path) - } catch (err) { - throw new Error( - `[supabase v3]: ${method}("${column}", …): ${err instanceof Error ? err.message : String(err)}`, - ) - } - return reconstructSelectorDocument(segments, value) - } - - /** - * Encrypted JSONPath-selector equality: matches rows whose document carries - * exactly `value` at `path`. Equality at a path IS containment of the - * path-shaped needle (`{user: {role: 'admin'}}`), so this compiles to - * {@link contains} — the ste_vec entry at the selector matches on its - * equality/ordering term. Selector ORDERING (`gt`/`lt`/…) is not expressible - * over PostgREST until the bundle grows a needle-comparison overload - * (cipherstash/encrypt-query-language#407); the Drizzle adapter's - * `ops.selector()` supports it today. - */ - selectorEq(column: string, path: string, value: unknown): this { - this.assertPostgrestCanQueryEncryptedOperator('selectorEq', column) - const needle = this.selectorNeedle('selectorEq', column, path, value) - return super.contains(column, needle) - } - - /** - * Encrypted JSONPath-selector inequality: rows whose document does NOT carry - * `value` at `path` — INCLUDING rows where the path is absent AND rows whose - * document column is SQL NULL, matching the Drizzle selector's `ne` (whose - * `OR entry IS NULL` arm covers both absence cases). A bare `not.cs` would - * drop NULL documents under three-valued logic (`NOT (NULL @> x)` is NULL), - * so this compiles to a structured OR: - * `column.is.null, column.not.cs.` — the containment condition's - * operand is encrypted through the normal or-condition term path. - */ - selectorNe(column: string, path: string, value: unknown): this { - this.assertPostgrestCanQueryEncryptedOperator('selectorNe', column) - const needle = this.selectorNeedle('selectorNe', column, path, value) - return super.or([ - { column, op: 'is', value: null }, - { column, op: 'contains', negate: true, value: needle }, - ]) - } - - /** - * `like`/`ilike` on an ENCRYPTED column are a best-effort compatibility shim, - * delegated to `matches`. EQL v3 free-text search is fuzzy bloom token - * matching, not SQL pattern matching, so the result is APPROXIMATE — matching - * is case-insensitive and one-sided (may false-positive), and anchoring is - * lost. Leading/trailing `%` are stripped; an internal `%` or any `_` cannot be - * approximated by trigram matching and throws. A plaintext column keeps real - * SQL LIKE. - */ - override like(column: string, pattern: string): this { - if (!this.isEncryptedV3Column(column)) return super.like(column, pattern) - return this.matches(column, this.likeNeedle(column, 'like', pattern)) - } - - override ilike(column: string, pattern: string): this { - if (!this.isEncryptedV3Column(column)) return super.ilike(column, pattern) - return this.matches(column, this.likeNeedle(column, 'ilike', pattern)) - } - - /** - * Reduce a SQL LIKE pattern to a fuzzy-match needle, or throw when it cannot be - * approximated. Strips surrounding `%` (prefix/suffix wildcards, which fuzzy - * matching subsumes); an internal `%` or any `_` is unapproximable. Warns once - * per (op, column) that the delegation is approximate. - */ - private likeNeedle(column: string, op: string, pattern: string): string { - const needle = pattern.replace(/^%+/, '').replace(/%+$/, '') - if (needle.includes('%') || pattern.includes('_')) { - throw new Error( - `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an internal "%" or any "_"). Use matches("${column}", term) with a literal search term.`, - ) - } - const key = `${op}:${column}` - if (!EncryptedQueryBuilderV3Impl.warnedLikeDelegation.has(key)) { - EncryptedQueryBuilderV3Impl.warnedLikeDelegation.add(key) - logger.warn( - `[supabase v3]: "${op}" on encrypted column "${column}" is delegated to matches() (fuzzy bloom token search). Results are APPROXIMATE — case-insensitive, one-sided (may false-positive), and wildcards/anchoring are not honored. Call matches() directly to make this explicit.`, - ) - } - return needle - } - - /** - * Encrypted `matches` goes through the bloom-filter `@>`, which the bundle - * declares on the domain as PostgREST's `cs`. The operand is the full storage - * envelope; `eql_v3.matches` (the SQL function) extracts the `bf` array from - * both sides. - * - * Emitted via `filter(col, 'cs', json)` rather than `q.contains(col, json)`: - * postgrest-js's `contains` re-serializes a non-string operand, and our - * operand is already `JSON.stringify`d. Plaintext `contains` (not encrypted) - * falls through to the base's native path. - */ - protected override applyContainsFilter( - q: SupabaseQueryBuilder, - column: DbName, - value: unknown, - wasEncrypted: boolean, - ): SupabaseQueryBuilder { - if (wasEncrypted) { - this.assertPostgrestCanQueryEncryptedOperator('filter', column) - return q.filter(column, 'cs', value) - } - return super.applyContainsFilter(q, column, value, wasEncrypted) - } - - /** - * `.or()` string conditions carry raw PostgREST operators, so a free-text - * condition arrives as `cs` — not a {@link FilterOp}. Resolve it through the - * same table the raw `.filter()` path uses, so `.or('amount.cs.5')` on an - * `integer_ord` column is rejected by the capability guard rather than - * silently encrypted as an equality term. A structured `{ op: 'matches' }` - * condition maps to free-text directly. - */ - protected override queryTypeForOrOp(op: FilterOp): QueryTypeName { - if (op === 'matches') return 'freeTextSearch' - // Structured conditions may carry the `contains` METHOD spelling (the wire - // token becomes `cs` in rebuildOrString). It maps to the same capability - // gate as `cs`; on a JSON column the term resolver then re-types it to - // searchableJson and validates the operand. selectorNe's IS-NULL-inclusive - // or-form relies on this arm. - if (op === 'contains') return 'freeTextSearch' - return this.queryTypeForRawOp(op) - } - - /** Rebuild `Date` values from the encrypt-config `cast_as` (date/timestamp), - * mirroring the typed v3 client's decrypt-model path. */ - protected override postprocessDecryptedRow( - row: Record, - ): Record { - // Every key an encrypted column can appear under: the keys this select - // actually produces (including caller-chosen aliases like `ts:createdAt`), - // plus the static property and DB names as a fallback for paths that record - // no select. Aliases win. Derived here from `this.selectColumns` (the row in - // hand) rather than cached from `buildSelectString`, so a reused builder can - // never postprocess a row with a previous operation's stale select map. - const keyToDb: Record = Object.assign( - Object.create(null), - this.selectColumns === null - ? undefined - : selectKeyToDbV3(this.selectColumns, this.propToDb), - ) - for (const [property, dbName] of Object.entries(this.propToDb)) { - keyToDb[property] ??= dbName - keyToDb[dbName] ??= dbName - } - - const out: Record = { ...row } - for (const [key, dbName] of Object.entries(keyToDb)) { - const castAs = this.columnSchemas[dbName]?.cast_as - if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue - const value = out[key] - if (value == null || value instanceof Date) continue - if (typeof value === 'string' || typeof value === 'number') { - out[key] = new Date(value) - } - } - return out - } -} diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index 555df6276..f09314fd3 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -1,34 +1,39 @@ import type { JsPlaintext } from '@cipherstash/protect-ffi' import type { AuditConfig } from '@cipherstash/stack/adapter-kit' import { - bulkModelsToEncryptedPgComposites, + DATE_LIKE_CASTS, + EncryptedV3Column, logger, - modelToEncryptedPgComposites, + matchNeedleError, + parseSelectorSegments, + reconstructSelectorDocument, + unsupportedLeafReason, } from '@cipherstash/stack/adapter-kit' import type { EncryptionClient } from '@cipherstash/stack/encryption' -import type { EncryptionError } from '@cipherstash/stack/errors' +import type { AnyV3Table } from '@cipherstash/stack/eql/v3' +import { + type EncryptionError, + EncryptionErrorTypes, +} from '@cipherstash/stack/errors' import type { LockContextInput } from '@cipherstash/stack/identity' -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' -import { EncryptedColumn } from '@cipherstash/stack/schema' +import type { ColumnSchema } from '@cipherstash/stack/schema' import type { BuildableQueryColumn, + Encrypted, EncryptedQueryResult, QueryTypeName, ScalarQueryTerm, } from '@cipherstash/stack/types' import { - addJsonbCasts, + addJsonbCastsV3, formatContainmentOperand, formatInListOperand, - getEncryptedColumnNames, isEncryptableTerm, isEncryptedColumn, mapFilterOpToQueryType, parseOrString, rebuildOrString, + selectKeyToDbV3, } from './helpers' import type { DbConflictList, @@ -57,27 +62,159 @@ import type { TransformOp, } from './types' +/** cast_as kinds that reconstruct to a JS `Date` — shared with the typed v3 + * client's decrypt-model path (see `encryption/v3.ts`). */ +const DATE_LIKE_CAST_SET = new Set(DATE_LIKE_CASTS) + +/** + * The subset of a v3 column builder the dialect relies on. Structural rather + * than the concrete class union so the runtime `instanceof EncryptedV3Column` + * gate and this type stay independent. + */ +type V3ColumnLike = { + getName(): string + getEqlType(): string + getQueryCapabilities(): { + equality: boolean + orderAndRange: boolean + freeTextSearch: boolean + /** Optional: only `public.eql_v3_json_search` (`types.Json`) carries it. */ + searchableJson?: boolean + } + build(): ColumnSchema +} + +/** + * Validate an encrypted-JSON containment operand: a NON-EMPTY plain object or a + * non-empty array. Everything else is rejected with an actionable steer: + * + * - Scalars/strings: the caller meant free-text (`matches` on a text column) or + * a selector — a raw JSON string is NOT parsed, by design (parsing would make + * `'{"a":1}'` and `{a:1}` silently different queries on other surfaces). + * - Non-plain objects (`Date`, `Map`, `RegExp`, class instances): these JSON- + * serialize to scalars or `{}` — not the sub-document the caller believes. + * - `{}` and `[]`: jsonb containment holds for EVERY document (`doc @> '{}'`), + * so an accidentally-empty needle would silently return (and decrypt) the + * whole table. The Drizzle adapter rejects the same needle for the same + * reason — the two first-party adapters must agree that this is an error. + */ +function assertJsonContainmentOperand(column: string, value: unknown): void { + const isPlainObject = + value !== null && + typeof value === 'object' && + !Array.isArray(value) && + (Object.getPrototypeOf(value) === Object.prototype || + Object.getPrototypeOf(value) === null) + if (!isPlainObject && !Array.isArray(value)) { + // Array.isArray is false on this branch by construction, so the label only + // distinguishes null / non-plain object / scalar. + const got = + value === null + ? 'null' + : typeof value === 'object' + ? (value as object).constructor?.name || 'a non-plain object' + : typeof value + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" takes a sub-document (plain object or array) to match, got ${got}.`, + ) + } + const empty = Array.isArray(value) + ? value.length === 0 + : Object.keys(value as object).length === 0 + if (empty) { + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" cannot take an empty ${Array.isArray(value) ? 'array' : 'object'} needle: it matches every row. Pass a non-empty sub-document, or omit the predicate to select all rows.`, + ) + } +} + +/** + * Reject a declared property name that is also a DIFFERENT physical column. + * + * `select('*')` expands the introspected DB names into property names, so a + * column renamed `created_at → createdAt` and a distinct plaintext column + * literally named `createdAt` both emit the token `createdAt`, which + * `addJsonbCastsV3` turns into `createdAt:created_at::jsonb` — twice. PostgREST + * returns the encrypted column under that key and the plaintext one is never + * selected, silently yielding the wrong value for a field the row type + * guarantees. + * + * Nothing downstream can disambiguate the two, and `EncryptedTable.build()`'s + * duplicate check only fires when two BUILDERS share a `getName()`. Refuse to + * construct instead. + */ +function assertNoPropertyDbNameCollision( + tableName: string, + propToDb: Record, + allColumns: string[] | null, +): void { + if (!allColumns) return + const dbNames = new Set(allColumns) + + for (const [property, dbName] of Object.entries(propToDb)) { + if (property === dbName) continue + if (!dbNames.has(property)) continue + throw new Error( + `[supabase v3]: property "${property}" on table "${tableName}" renames DB column "${dbName}", but "${property}" is also a distinct column in the database — the two collide in select('*'). Rename the property, or drop the declared rename.`, + ) + } +} + /** * A deferred query builder that wraps Supabase's query builder to automatically - * handle encryption and decryption of data. + * handle encryption and decryption of data for native EQL v3 concrete-domain + * columns (`public.*` type domains, `eql_v3` operators). * * All chained operations are recorded synchronously. When the builder is awaited, * it encrypts mutation data, adds `::jsonb` casts, batch-encrypts filter values, * executes the real Supabase query, and decrypts results. + * + * v3 columns are `EncryptedV3Column` builders and may map a JS property name to a + * different DB column name (`buildColumnKeyMap`). Filters, select casts, and + * mutations resolve property → DB name; select casts alias the DB column back to + * the property (`prop:db_name::jsonb`) so result rows keep property keys. The raw + * encrypted payload object is sent on mutations (the `public.*` domains are + * `DOMAIN … AS jsonb`), and scalar equality/range filters use the FULL storage + * envelope from `encrypt()`, serialized as jsonb text. + * + * EQL 3.0.2 removed the storage/jsonb escape hatch for free-text and JSON + * operators: those now require typed query-domain operands PostgREST cannot + * express. The factory reads the installed EQL version and this builder fails + * those operators before encryption, so a decryptable storage envelope never + * enters a GET URL. + * + * Decrypted rows additionally get `Date` reconstruction from the encrypt-config + * `cast_as`, mirroring the typed v3 client. `decryptModel`/`bulkDecryptModels` + * are generation-agnostic in `@cipherstash/stack`, so a stored EQL v2 payload + * still decrypts through this builder's read path. */ export class EncryptedQueryBuilderImpl< T extends Record = Record, > { protected tableName: string - protected schema: EncryptedTable + protected table: AnyV3Table protected encryptionClient: EncryptionClient protected supabaseClient: SupabaseClientLike protected encryptedColumnNames: string[] /** All column names for the table (encrypted + plaintext), in ordinal order, * used to expand `select('*')`. `null` when the caller supplied no column - * list (v2, or a v3 client that could not introspect). */ + * list (a v3 client that could not introspect). */ protected allColumns: string[] | null = null + /** JS property name → DB column name, for every encrypted column. */ + private propToDb: Record + /** DB column name → JS property name — the inverse of {@link propToDb}, used + * to expand `select('*')` back into property names. Null prototype: a DB + * column literally named `constructor` / `toString` would otherwise resolve + * to an inherited `Object.prototype` member and be emitted as a select token. */ + private dbToProp: Record + /** Built column schemas keyed by DB column name (for `cast_as`). */ + private columnSchemas: Record + /** Column builders keyed by BOTH property name and DB name. */ + private v3Columns: Record + /** EQL 3.0.2+ requires query-domain casts PostgREST cannot express. */ + private queryDomainsRequired: boolean + // Recorded operations protected mutation: MutationOp | null = null protected selectColumns: string | null = null @@ -99,17 +236,43 @@ export class EncryptedQueryBuilderImpl< constructor( tableName: string, - schema: EncryptedTable, + table: AnyV3Table, encryptionClient: EncryptionClient, supabaseClient: SupabaseClientLike, allColumns: string[] | null = null, + queryDomainsRequired = false, ) { this.tableName = tableName - this.schema = schema + this.table = table this.encryptionClient = encryptionClient this.supabaseClient = supabaseClient - this.encryptedColumnNames = getEncryptedColumnNames(schema) this.allColumns = allColumns + this.queryDomainsRequired = queryDomainsRequired + this.propToDb = table.buildColumnKeyMap() + this.columnSchemas = table.build().columns + + this.dbToProp = Object.create(null) as Record + for (const [property, dbName] of Object.entries(this.propToDb)) { + this.dbToProp[dbName] = property + } + + assertNoPropertyDbNameCollision(tableName, this.propToDb, allColumns) + + // Null-prototype: keyed by DB column names, and `validateTransforms` reads + // it without an own-key guard — an inherited `constructor`/`toString` would + // otherwise resolve truthy for a plaintext column of that name. + this.v3Columns = Object.create(null) as Record + for (const [property, builder] of Object.entries(table.columnBuilders)) { + if (builder instanceof EncryptedV3Column) { + const col = builder as unknown as V3ColumnLike + this.v3Columns[property] = col + this.v3Columns[col.getName()] = col + } + } + + // Filters and select strings address columns by JS property name AND by DB + // name, so recognition must cover both. + this.encryptedColumnNames = Object.keys(this.v3Columns) } // --------------------------------------------------------------------------- @@ -135,14 +298,23 @@ export class EncryptedQueryBuilderImpl< } /** - * Turn the introspected column list (DB names) into select tokens. The base - * returns them unchanged — v2 never supplies a column list, so this is dead - * for v2. The v3 dialect overrides it to emit JS property names, which is - * what makes `addJsonbCastsV3` alias a renamed column back to its property - * (`createdAt:created_at::jsonb`) rather than returning it under its DB name. + * Expand the introspected column list (DB names) into JS property names. + * + * Load-bearing for `select('*')` on a DECLARED table that renames a column. + * `addJsonbCastsV3` only emits the `prop:db_name::jsonb` alias — the thing + * that makes PostgREST return the column under its property name — when the + * token it sees is a property name. Feeding it the raw DB name instead takes + * the unaliased `dbNames.has(...)` branch, so the row comes back keyed + * `created_at` while the declared row type promises `createdAt`, silently + * yielding `undefined` for a field TypeScript guarantees. + * + * A DB column with no encrypted builder (plaintext passthrough, and every + * synthesized column, where property == DB name) maps to itself. */ protected expandAllColumns(columns: string[]): string[] { - return columns + return columns.map((dbName) => + Object.hasOwn(this.dbToProp, dbName) ? this.dbToProp[dbName] : dbName, + ) } insert( @@ -229,28 +401,79 @@ export class EncryptedQueryBuilderImpl< return this } + /** + * `like`/`ilike` on an ENCRYPTED column are a best-effort compatibility shim, + * delegated to `matches`. EQL v3 free-text search is fuzzy bloom token + * matching, not SQL pattern matching, so the result is APPROXIMATE — matching + * is case-insensitive and one-sided (may false-positive), and anchoring is + * lost. Leading/trailing `%` are stripped; an internal `%` or any `_` cannot be + * approximated by trigram matching and throws. A plaintext column keeps real + * SQL LIKE. + */ like(column: string, pattern: string): this { - this.filters.push({ op: 'like', column, value: pattern }) - return this + if (!this.isEncryptedV3Column(column)) { + this.filters.push({ op: 'like', column, value: pattern }) + return this + } + return this.matches(column, this.likeNeedle(column, 'like', pattern)) } ilike(column: string, pattern: string): this { - this.filters.push({ op: 'ilike', column, value: pattern }) - return this + if (!this.isEncryptedV3Column(column)) { + this.filters.push({ op: 'ilike', column, value: pattern }) + return this + } + return this.matches(column, this.likeNeedle(column, 'ilike', pattern)) } + /** + * `contains` on the v3 surface is EXACT containment: native jsonb/array `@>` + * on a plaintext column, ENCRYPTED ste_vec `@>` on a `types.Json` column (the + * sub-document operand is storage-encrypted whole; every leaf must match at + * its path — #650). On an encrypted match/search TEXT column containment is + * not the operation (that is the fuzzy `matches`), so refuse loudly rather + * than silently emit a bloom match under a name that promises exactness. + */ contains(column: string, value: unknown): this { + if (this.isSearchableJsonColumn(column)) { + this.assertPostgrestCanQueryEncryptedOperator('contains', column) + // Same validator the term resolver enforces — failing here just surfaces + // the error at the call site instead of at execution. + assertJsonContainmentOperand(column, value) + this.filters.push({ op: 'contains', column, value }) + return this + } + if (this.isEncryptedV3Column(column)) { + throw new Error( + `[supabase v3]: contains() is native (exact) containment and does not apply to encrypted column "${column}". Use matches() for encrypted free-text search.`, + ) + } this.filters.push({ op: 'contains', column, value }) return this } /** - * Encrypted free-text token match (v3 encrypted columns). Emits the same - * `cs`/`@>` wire operator as `contains`, but on a match-indexed encrypted - * column it is fuzzy bloom-filter token matching, not containment — see the v3 - * builder. The v3 dialect encrypts the operand as a free-text query term. + * `matches` is the encrypted free-text operator: fuzzy bloom-filter token + * matching, one-sided (may false-positive), NOT containment. It requires an + * encrypted match/search column; on a plaintext column, `contains` (native + * `@>`) is what the caller means — and on an encrypted JSON column, + * `contains`/`selectorEq` are (matching a document is containment, not + * free-text). Guarded here because both spellings collect the same + * `freeTextSearch` term, which the capability resolver would otherwise + * silently accept as containment of the raw string. */ matches(column: string, value: unknown): this { + if (this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: matches() is encrypted free-text search and does not apply to encrypted JSON column "${column}". Use contains("${column}", subDocument) or selectorEq("${column}", path, value).`, + ) + } + if (!this.isEncryptedV3Column(column)) { + throw new Error( + `[supabase v3]: matches() is encrypted free-text search and requires an encrypted column; "${column}" is not one. Use contains() for native containment.`, + ) + } + this.assertPostgrestCanQueryEncryptedOperator('matches', column) this.filters.push({ op: 'matches', column, value }) return this } @@ -270,7 +493,36 @@ export class EncryptedQueryBuilderImpl< return this } + /** + * `not(col, 'contains', …)` on an encrypted TEXT column would negate a fuzzy + * bloom match under the `contains` name — the exact confusion #617 removes — + * because the `not()` path rewrites the `contains` spelling to the `cs` wire + * operator. Reject it and steer to the `matches` spelling (or the raw `cs` + * operator, which is honest about the wire op). + * + * On an encrypted JSON column negated containment IS the honest exact + * operation (`not.cs` over ste_vec containment — {@link selectorNe} compiles + * to it), so it passes through. Plaintext columns keep native negated + * containment, and every other operator is recorded unchanged. + */ not(column: string, operator: string, value: unknown): this { + if ( + operator === 'contains' && + this.isEncryptedV3Column(column) && + !this.isSearchableJsonColumn(column) + ) { + throw new Error( + `[supabase v3]: not("${column}", 'contains', …) does not apply to encrypted column "${column}" — that is fuzzy free-text matching, not containment. Use not("${column}", 'matches', …) or the raw 'cs' operator.`, + ) + } + // Mirror of the matches() guard: a `matches` spelling on a JSON column + // would otherwise resolve to containment (the two share the `cs` wire op), + // silently negating an EXACT operation under a name that promises FUZZY. + if (operator === 'matches' && this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: not("${column}", 'matches', …) does not apply to encrypted JSON column "${column}" — matches() is free-text search. Use not("${column}", 'contains', subDocument) or selectorNe("${column}", path, value).`, + ) + } this.notFilters.push({ column, op: operator as FilterOp, value }) return this } @@ -299,6 +551,41 @@ export class EncryptedQueryBuilderImpl< return this } + /** + * Encrypted JSONPath-selector equality: matches rows whose document carries + * exactly `value` at `path`. Equality at a path IS containment of the + * path-shaped needle (`{user: {role: 'admin'}}`), so this compiles to + * {@link contains} — the ste_vec entry at the selector matches on its + * equality/ordering term. Selector ORDERING (`gt`/`lt`/…) is not expressible + * over PostgREST until the bundle grows a needle-comparison overload + * (cipherstash/encrypt-query-language#407); the Drizzle adapter's + * `ops.selector()` supports it today. + */ + selectorEq(column: string, path: string, value: unknown): this { + this.assertPostgrestCanQueryEncryptedOperator('selectorEq', column) + const needle = this.selectorNeedle('selectorEq', column, path, value) + return this.contains(column, needle) + } + + /** + * Encrypted JSONPath-selector inequality: rows whose document does NOT carry + * `value` at `path` — INCLUDING rows where the path is absent AND rows whose + * document column is SQL NULL, matching the Drizzle selector's `ne` (whose + * `OR entry IS NULL` arm covers both absence cases). A bare `not.cs` would + * drop NULL documents under three-valued logic (`NOT (NULL @> x)` is NULL), + * so this compiles to a structured OR: + * `column.is.null, column.not.cs.` — the containment condition's + * operand is encrypted through the normal or-condition term path. + */ + selectorNe(column: string, path: string, value: unknown): this { + this.assertPostgrestCanQueryEncryptedOperator('selectorNe', column) + const needle = this.selectorNeedle('selectorNe', column, path, value) + return this.or([ + { column, op: 'is', value: null }, + { column, op: 'contains', negate: true, value: needle }, + ]) + } + // --------------------------------------------------------------------------- // Transform methods (passthrough) // --------------------------------------------------------------------------- @@ -432,10 +719,10 @@ export class EncryptedQueryBuilderImpl< ) // A failure inside any of the encrypt/decrypt steps above is thrown as an - // `EncryptionFailedError` wrapping the operation's `EncryptionError` (or, in - // the v3 dialect, a synthesized one for its contract-violation cases). - // Thread it through so callers can branch on `error.encryptionError`; a plain - // PostgREST/API error is not an `EncryptionFailedError` and leaves it unset. + // `EncryptionFailedError` wrapping the operation's `EncryptionError` (or a + // synthesized one for its contract-violation cases). Thread it through so + // callers can branch on `error.encryptionError`; a plain PostgREST/API + // error is not an `EncryptionFailedError` and leaves it unset. const error: EncryptedSupabaseError = { message, encryptionError: @@ -473,7 +760,7 @@ export class EncryptedQueryBuilderImpl< if (Array.isArray(data)) { // Bulk encrypt - const baseOp = this.encryptionClient.bulkEncryptModels(data, this.schema) + const baseOp = this.encryptionClient.bulkEncryptModels(data, this.table) const op = this.lockContext ? baseOp.withLockContext(this.lockContext) : baseOp @@ -495,7 +782,7 @@ export class EncryptedQueryBuilderImpl< } // Single model - const baseOp = this.encryptionClient.encryptModel(data, this.schema) + const baseOp = this.encryptionClient.encryptModel(data, this.table) const op = this.lockContext ? baseOp.withLockContext(this.lockContext) : baseOp @@ -517,22 +804,24 @@ export class EncryptedQueryBuilderImpl< } /** - * Encode an encrypted model for the Supabase request body. v2 wraps each - * encrypted value in the `{ data: ... }` object expected by the - * `eql_v2_encrypted` composite type. The v3 dialect overrides this — native - * `eql_v3.*` domains are plain jsonb, so the raw payload is sent instead + * Encode an encrypted model for the Supabase request body. The native + * `eql_v3.*` domains are plain jsonb, so the raw encrypted payload is sent * (keyed by DB column name). */ protected transformEncryptedMutationModel( model: Record, ): Record { - return modelToEncryptedPgComposites(model) + const out: Record = Object.create(null) + for (const [key, value] of Object.entries(model)) { + out[this.dbNameFor(key)] = value + } + return out } protected transformEncryptedMutationModels( models: Record[], ): Record[] { - return bulkModelsToEncryptedPgComposites(models) + return models.map((model) => this.transformEncryptedMutationModel(model)) } // --------------------------------------------------------------------------- @@ -541,7 +830,7 @@ export class EncryptedQueryBuilderImpl< protected buildSelectString(): DbSelect | null { if (this.selectColumns === null) return null - return addJsonbCasts(this.selectColumns, this.encryptedColumnNames) + return addJsonbCastsV3(this.selectColumns, this.propToDb) } // --------------------------------------------------------------------------- @@ -566,7 +855,7 @@ export class EncryptedQueryBuilderImpl< terms.push({ value, column, - table: this.schema, + table: this.table, queryType, returnType: 'composite-literal', }) @@ -757,35 +1046,225 @@ export class EncryptedQueryBuilderImpl< } /** - * Encrypt the collected filter terms, returning one encoded value per term - * (in order). v2 batch-encrypts via `encryptQuery` with the - * `composite-literal` return type — the `("json")` string the - * `eql_v2_encrypted` composite operators compare. The v3 dialect overrides - * this to produce full-envelope jsonb operands instead. + * Encrypt every filter operand as a full storage envelope, serialized to jsonb + * text for the PostgREST filter value. + * + * Terms are grouped by column and each group takes ONE `bulkEncrypt` crossing. + * `in(col, [a, b, c])` collects one term per element (the list must never be + * encrypted whole), so encrypting per term spent N ZeroKMS/FFI round-trips + * where one would do. `bulkEncrypt` carries a single `{table, column}` for the + * whole payload, so the grouping is mandatory, not an optimisation: one bulk + * call over a mixed-column term array would stamp one column onto every + * plaintext. Results are scattered back onto the terms' original indices, + * which is the contract `termMap` downstream relies on. + * + * Mirrors `eql/v3/drizzle/operators.ts` `encryptOperands` — same batching + * contract, same length assertion, same fallback. Kept separate because that + * one encrypts a single-column operand list and returns `SQL[]`, while this + * must group a multi-column term array and preserve positions. */ protected async encryptCollectedTerms( terms: ScalarQueryTerm[], ): Promise { - // Batch encrypt all terms in one call - const baseOp = this.encryptionClient.encryptQuery(terms) + const groups = new Map< + V3ColumnLike, + { indices: number[]; values: ScalarQueryTerm['value'][] } + >() + terms.forEach((term, index) => { + const column = this.assertTermQueryable(term) + const group = groups.get(column) ?? { indices: [], values: [] } + group.indices.push(index) + group.values.push(term.value) + groups.set(column, group) + }) + + const bulkEncrypt = this.encryptionClient.bulkEncrypt?.bind( + this.encryptionClient, + ) + // Each term becomes the `JSON.stringify`'d storage envelope — a `string`, + // which is one arm of `EncryptedQueryResult`. PostgREST cannot cast a filter + // value to the `eql_v3.query_` twins, so v3 sends full envelopes, + // serialized to jsonb text. + const results = new Array(terms.length) + + await Promise.all( + Array.from(groups, async ([column, { indices, values }]) => { + const encrypted = bulkEncrypt + ? await this.bulkEncryptGroup(bulkEncrypt, column, values) + : await this.encryptGroupPerTerm(column, values) + + encrypted.forEach((envelope, i) => { + results[indices[i]] = JSON.stringify(envelope) + }) + }), + ) + + return results + } + + /** + * Validate a term's query type against its column's declared capabilities. + * Pure validation: `encrypt`/`bulkEncrypt` never receive the query type. On + * EQL 3.0.2+, free-text/JSON terms are rejected before this storage-encryption + * path can place ciphertext in a GET URL. + */ + private assertTermQueryable(term: ScalarQueryTerm): V3ColumnLike { + const column = term.column as unknown as V3ColumnLike + let queryType = term.queryType ?? 'equality' + const capabilities = column.getQueryCapabilities() + + // The `cs` wire operator is capability-overloaded: bloom free-text on a + // match/search TEXT column, encrypted ste_vec containment on a `types.Json` + // DOCUMENT column. Both arrive here as `freeTextSearch` (contains/matches/ + // raw `cs` all map there); resolve to the capability the column actually + // carries. The two are mutually exclusive by construction, so this can + // never reinterpret a real free-text column. + if ( + queryType === 'freeTextSearch' && + !capabilities.freeTextSearch && + capabilities.searchableJson + ) { + queryType = 'searchableJson' + } + + if ( + queryType !== 'equality' && + queryType !== 'orderAndRange' && + queryType !== 'freeTextSearch' && + queryType !== 'searchableJson' + ) { + throw new Error( + `[supabase v3]: query type "${queryType}" is not supported on EQL v3 columns`, + ) + } + + if (!capabilities[queryType]) { + throw new Error( + `[supabase v3]: column "${column.getName()}" (${column.getEqlType()}) does not support ${queryType} queries — declare the column with a domain that carries that capability`, + ) + } + + if (queryType === 'freeTextSearch' || queryType === 'searchableJson') { + // This is the common boundary for every spelling that collects an + // encrypted match/containment term: matches(), contains(), not(), raw + // filter(), and both forms of or(). Method-level checks provide earlier + // errors for the direct helpers, but cannot cover the raw filter paths on + // their own. + this.assertPostgrestCanQueryEncryptedOperator('filter', column.getName()) + } + + if (queryType === 'searchableJson') { + // THE single enforced operand boundary for encrypted-JSON containment. + // Terms reach this resolver from every spelling — contains(), raw + // .filter(col,'cs',…), not(col,'contains'|'matches',…), and .or() + // string/structured conditions — and only contains() has a method-level + // guard. Without this check a raw string (e.g. a free-text term ported + // from a text column, or an .or() condition value, which is always a + // string) would be storage-encrypted as a JSON SCALAR and silently match + // nothing; pre-#650 every such spelling failed loudly on capability. + assertJsonContainmentOperand(column.getName(), term.value) + } + + // Free-text (bloom) needle floor. A needle shorter than the tokenizer's + // token_length produces NO tokens, so `bf @> '{}'` holds for every row and + // the query would silently return (and the caller decrypt) the whole table + // — a fail-open over-exposure. Reject it up front, mirroring the Drizzle v3 + // adapter (matchNeedleError) so both first-party surfaces guard identically. + // JSON containment terms (searchableJson) are validated separately above. + if (queryType === 'freeTextSearch') { + const match = column.build().indexes?.match + const reason = match ? matchNeedleError(term.value, match) : undefined + if (reason) { + throw new Error( + `[supabase v3]: cannot search column "${column.getName()}": ${reason}`, + ) + } + } + + return column + } + + private encryptionFailure(message: string, cause?: EncryptionError): never { + logger.error( + `Supabase: failed to encrypt query terms for table "${this.tableName}"`, + ) + // Most callers pass the operation's own `EncryptionError`; the contract- + // violation cases (bulk length mismatch, null envelope) have none, so + // synthesize one — a broken query encryption is still an encryption failure, + // and callers branch on `error.encryptionError` regardless. + throw new EncryptionFailedError( + `Failed to encrypt query terms: ${message}`, + cause ?? { type: EncryptionErrorTypes.EncryptionError, message }, + ) + } + + /** One FFI crossing for a column's whole operand list. */ + private async bulkEncryptGroup( + bulkEncrypt: NonNullable, + column: V3ColumnLike, + values: ScalarQueryTerm['value'][], + ): Promise> { + const baseOp = bulkEncrypt( + values.map((plaintext) => ({ plaintext })) as never, + { column, table: this.table } as never, + ) const op = this.lockContext ? baseOp.withLockContext(this.lockContext) : baseOp if (this.auditConfig) op.audit(this.auditConfig) const result = await op - if (result.failure) { - logger.error( - `Supabase: failed to encrypt query terms for table "${this.tableName}"`, - ) - - throw new EncryptionFailedError( - `Failed to encrypt query terms: ${result.failure.message}`, - result.failure, + if (result.failure) + this.encryptionFailure(result.failure.message, result.failure) + + // `bulkEncrypt` is position-stable, so a length mismatch means the contract + // was violated. Truncating instead would silently widen an `in` predicate + // (or narrow a `not.in`) to whatever came back. `result.data` is now + // `BulkEncryptedData` — `{ id?, data: Encrypted | null }[]` — not `unknown`. + const encrypted = result.data + if (encrypted.length !== values.length) { + this.encryptionFailure( + `bulk encryption returned ${encrypted.length} terms for ${values.length} values on column "${column.getName()}".`, ) } + return encrypted.map((term, i) => { + // `BulkEncryptedData` types the element as `Encrypted | null`. A `null` + // envelope here would be `JSON.stringify`'d to the literal string `"null"` + // and sent as the filter operand — silently matching whatever `"null"` + // encodes to rather than failing. A query term should never encrypt to a + // null envelope, so treat it as a contract violation, not a value. + if (term.data === null) { + this.encryptionFailure( + `bulk encryption returned a null envelope at position ${i} for column "${column.getName()}".`, + ) + } + return term.data + }) + } - return result.data + /** Fallback for a client that predates `bulkEncrypt`. */ + private async encryptGroupPerTerm( + column: V3ColumnLike, + values: ScalarQueryTerm['value'][], + ): Promise { + return Promise.all( + values.map(async (value) => { + const baseOp = this.encryptionClient.encrypt(value, { + column, + table: this.table, + }) + const op = this.lockContext + ? baseOp.withLockContext(this.lockContext) + : baseOp + if (this.auditConfig) op.audit(this.auditConfig) + + const result = await op + if (result.failure) { + this.encryptionFailure(result.failure.message, result.failure) + } + return result.data + }), + ) } // --------------------------------------------------------------------------- @@ -803,9 +1282,9 @@ export class EncryptedQueryBuilderImpl< * the order in which capability errors surface. * * Safe to run BEFORE encryption: `getColumnMap()`/`encryptedColumnNames` are - * keyed by both property and DB name in v3 (and property == DB name in v2), - * so column lookup resolves identically either side of the translation, and - * `tableColumns[prop]` is the very same builder object as `tableColumns[db]`. + * keyed by both property and DB name, so column lookup resolves identically + * either side of the translation, and `tableColumns[prop]` is the very same + * builder object as `tableColumns[db]`. */ protected toDbSpace(): DbQuerySpace { return { @@ -856,12 +1335,43 @@ export class EncryptedQueryBuilderImpl< } /** - * The column expression `order()` sends to PostgREST. Its own seam, separate - * from {@link filterColumnName}: v3 orders an encrypted column by a jsonb path - * into its ordering term, which must not leak into filters. + * Encrypted ordering columns sort by their `op` term, not by the envelope. + * + * `order=col->op` is the one ordering expression PostgREST can emit that + * reaches the OPE term. It must NOT leak into filters — those compare whole + * envelopes through the `eql_v3.*` operators — which is why this is its own + * seam rather than a change to `filterColumnName`. + * + * The canonical EQL form is `ORDER BY eql_v3.ord_term(col)`, which returns + * `eql_v3_internal.ope_cllw` — a domain over `bytea`, ordered by the native + * btree. PostgREST cannot call a function, so it orders the `op` term where it + * sits, inside the envelope. The two agree because the term is what + * `ord_term()` returns. + * + * `->` (jsonb) rather than `->>` (text) keeps the comparison on the typed + * value. Note this does NOT avoid the database collation: Postgres compares + * jsonb strings with `varstr_cmp` under the default collation, exactly as it + * does text. What makes the ordering collation-independent is the term itself + * — fixed-width lowercase hex (`[0-9a-f]`, 130 chars for `integer_ord`, 82 for + * `text_search`) — and every collation orders digits before letters and hex + * letters among themselves. `match-bloom`'s sibling assertion pins that shape. + * + * This runs at column-name-mapping time (`transformToDbSpace`), BEFORE + * `buildAndExecuteQuery` calls `validateTransforms`. For an encrypted column + * with no `ope` index it therefore returns a bare `dbName` here — a name that + * would sort by `jsonb_cmp` over the ciphertext if it reached PostgREST — but + * it never does: `validateTransforms` throws (with a domain-specific reason) + * before the query executes, so the bare name is only ever an intermediate + * value on a request that is about to be rejected. */ protected orderColumnName(column: string): DbName { - return this.filterColumnName(column) + const dbName = this.dbNameFor(column) + const encrypted = this.v3Columns[column] + if (!encrypted) return dbName as DbName + + return ( + this.columnSchemas[dbName]?.indexes?.ope ? `${dbName}->op` : dbName + ) as DbName } private transformToDbSpace(t: TransformOp): DbTransformOp { @@ -889,8 +1399,6 @@ export class EncryptedQueryBuilderImpl< switch (m.kind) { case 'insert': case 'upsert': - // `resolveMutationOptions` returns the SAME reference when no column - // needed renaming, which v2 relies on. return { ...m, options: this.resolveMutationOptions(m.options) } case 'update': case 'delete': @@ -1189,8 +1697,8 @@ export class EncryptedQueryBuilderImpl< } else { // Every condition names a plaintext column, whose property name IS // its DB name — nothing to map. Forward the caller's ORIGINAL string - // byte-for-byte: v2 relies on this for nested `and()` and quoted - // values that `parseOrString`/`rebuildOrString` cannot round-trip. + // byte-for-byte: relied on for nested `and()` and quoted values that + // `parseOrString`/`rebuildOrString` cannot round-trip. q = q.or(of_.original as DbFilterString, { referencedTable: of_.referencedTable, }) @@ -1233,30 +1741,31 @@ export class EncryptedQueryBuilderImpl< } // --------------------------------------------------------------------------- - // Dialect seams — every default preserves the v2 behaviour byte-for-byte. - // The v3 builder (see ./query-builder-v3) overrides these for native - // `eql_v3.*` domain columns. + // Dialect seams for native `eql_v3.*` domain columns. // --------------------------------------------------------------------------- + /** Resolve a JS property name to its DB column name. `Object.hasOwn` guards + * the inherited-member hazard described on {@link EncryptedTable.buildColumnKeyMap}. */ + private dbNameFor(name: string): string { + return Object.hasOwn(this.propToDb, name) ? this.propToDb[name] : name + } + /** - * Map a filter's column name to the DB column name PostgREST must see. - * v2 schemas key columns by their DB name already, so this is the identity; - * the v3 dialect resolves a JS property name to its DB name. + * Map a filter's column name to the DB column name PostgREST must see — + * resolving a JS property name to its DB name. * * This is the ONLY place a {@link DbName} is minted. The * {@link SupabaseQueryBuilder} seam accepts nothing else, so every column * name reaching PostgREST must pass through here. */ protected filterColumnName(column: string): DbName { - return column as DbName + return this.dbNameFor(column) as DbName } /** * Resolve the column names carried by a mutation's options. `onConflict` is a * comma-separated column list, so it needs the same property→DB mapping as a - * filter. Returns the original object when nothing changed, so v2 — where - * {@link filterColumnName} is the identity — passes the caller's reference on - * untouched. + * filter. Returns the original object when nothing changed. */ protected resolveMutationOptions< O extends { onConflict?: string } | undefined, @@ -1274,26 +1783,86 @@ export class EncryptedQueryBuilderImpl< } /** - * Validate the accumulated transforms before the query is built. Called from - * inside {@link execute}'s try, so a throw surfaces as a `status: 500` error - * result (or rethrows under `throwOnError`), matching the filter-path - * capability guard. v2 imposes no constraints. + * `ORDER BY` on an OPE-backed column is supported; on every other encrypted + * column it is rejected. + * + * A bare `ORDER BY col` IS wrong. The `*_ord` domains are + * `CREATE DOMAIN … AS jsonb`, and the bundle declares no btree operator class + * on any domain — it actively lints against one (`domain_opclass`), because an + * opclass on a domain bypasses operator resolution. So the sort resolves + * through jsonb's default `jsonb_cmp` and compares the envelope's keys in + * storage order, starting at the random ciphertext `c`. No error, and a + * stable, meaningless row order. + * + * But the correct sort key is reachable without a function call. `eql_v3.ord_term` + * returns the domain's `op` term, and OPE is order-preserving by construction: + * ordering by the term reproduces the plaintext order. PostgREST cannot emit + * `ORDER BY eql_v3.ord_term(col)`, but it CAN emit a jsonb path — + * `order=col->op.asc` — which selects exactly that term. + * + * So the guard is on the ordering FLAVOUR, not on encryption: + * + * - `ope` present → order by `col->op`. Every plain `_ord` domain, plus + * `text_ord` and `text_search`. + * - `ore` present → reject. The `ob` term is an array of ORE blocks whose + * comparison needs the superuser-only opclass; a jsonb-path sort over it is + * meaningless. + * - neither → reject. Storage-only, equality-only and match-only columns + * carry no ordering term to sort by. + * + * A column absent from {@link v3Columns} is a plaintext passthrough and orders + * normally. This runtime guard is the only protection the untyped + * (no-`schemas`) surface has. */ - protected validateTransforms(): void {} + protected validateTransforms(): void { + for (const t of this.transforms) { + if (t.kind !== 'order') continue + const column = this.v3Columns[t.column] + if (!column) continue + + const indexes = this.columnSchemas[column.getName()]?.indexes + if (indexes?.ope) continue + + const reason = indexes?.ore + ? 'its ORE ordering term (`ob`) needs the superuser-only ORE operator class, which PostgREST cannot reach through a jsonb path' + : 'it carries no ordering term to sort by' + + throw new Error( + `[supabase v3]: cannot order by encrypted column "${column.getName()}" (${column.getEqlType()}) — ${reason}. ` + + 'Order by a plaintext column, or use an OPE-backed ordering domain ' + + '(`*_ord`, `text_ord`, `text_search`), or use the EQL v3 Drizzle integration.', + ) + } + } /** - * The CipherStash query type to encrypt a raw `.filter(column, operator, …)` - * term under. `operator` is an arbitrary PostgREST operator string, not a - * {@link FilterOp}, so it cannot go through `mapFilterOpToQueryType`. + * Resolve a raw `.filter()` operator to the capability it exercises. A + * supported v3 operand is a full storage envelope, so `queryType` never + * selects a narrowing — it only tells {@link assertTermQueryable} which + * capability to demand of the column. * - * v2 encrypts every raw filter as an equality term. That is wrong — a raw - * `.filter('amount', 'gte', …)` wants an ORE term — but in v2 `queryType` - * selects the `encryptQuery` narrowing, so correcting it changes the - * ciphertext on the wire. Preserved verbatim here and tracked separately; - * the v3 dialect, where `queryType` is only a capability gate, overrides it. + * Unknown operators throw rather than silently defaulting to equality, which + * would encrypt a term the column may not even be able to compare. */ - protected queryTypeForRawOp(_operator: string): QueryTypeName { - return 'equality' + protected queryTypeForRawOp(operator: string): QueryTypeName { + switch (operator) { + case 'cs': + return 'freeTextSearch' + case 'gt': + case 'gte': + case 'lt': + case 'lte': + return 'orderAndRange' + case 'eq': + case 'neq': + case 'in': + case 'is': + return 'equality' + default: + throw new Error( + `[supabase v3]: unsupported raw filter operator "${operator}" on an encrypted column`, + ) + } } /** @@ -1317,10 +1886,9 @@ export class EncryptedQueryBuilderImpl< } /** - * Apply a `like`/`ilike` filter. v2 relies on the `~~` operator defined on - * `eql_v2_encrypted`; the v3 dialect overrides this for encrypted columns - * because the `eql_v3.*` domains expose free-text match via `@>` - * (PostgREST `cs`) rather than a LIKE operator. + * Apply a `like`/`ilike` filter. On an encrypted column `like`/`ilike` were + * rewritten to `matches` at record time, so a `like`/`ilike` pending filter + * only ever names a plaintext column, which keeps real SQL LIKE. */ protected applyPatternFilter( q: SupabaseQueryBuilder, @@ -1336,20 +1904,27 @@ export class EncryptedQueryBuilderImpl< /** * Apply a `contains` filter. On a plaintext column this is PostgREST's native - * jsonb/array containment. The v3 dialect overrides it for encrypted columns, - * where `cs` resolves to the `@>` operator the EQL bundle declares on the - * domain, backed by `eql_v3.matches` (bloom-filter containment). + * jsonb/array containment. On an encrypted column `cs` resolves to the `@>` + * operator the EQL bundle declares on the domain, backed by `eql_v3.matches` + * (bloom-filter containment) — and the operand is the full storage envelope, + * already `JSON.stringify`d, emitted via `filter(col, 'cs', json)` rather than + * `q.contains` (postgrest-js's `contains` re-serializes a non-string operand). * - * A structured operand is serialized here rather than by postgrest-js, which - * joins array elements on `,` without quoting them — so `['with,comma']` would - * reach Postgres as two elements. Scalars keep the native path. + * A structured plaintext operand is serialized here rather than by + * postgrest-js, which joins array elements on `,` without quoting them — so + * `['with,comma']` would reach Postgres as two elements. Scalars keep the + * native path. */ protected applyContainsFilter( q: SupabaseQueryBuilder, column: DbName, value: unknown, - _wasEncrypted: boolean, + wasEncrypted: boolean, ): SupabaseQueryBuilder { + if (wasEncrypted) { + this.assertPostgrestCanQueryEncryptedOperator('filter', column) + return q.filter(column, 'cs', value) + } const literal = formatContainmentOperand(value) return literal !== null ? q.filter(column, 'cs', literal) @@ -1359,10 +1934,17 @@ export class EncryptedQueryBuilderImpl< /** * The CipherStash query type for an `.or()` condition's operator on an * encrypted column. String-form conditions carry raw PostgREST operators - * (`cs`), which are not {@link FilterOp}s; the v3 dialect maps those. + * (`cs`), which are not {@link FilterOp}s. */ protected queryTypeForOrOp(op: FilterOp): QueryTypeName { - return mapFilterOpToQueryType(op) + if (op === 'matches') return 'freeTextSearch' + // Structured conditions may carry the `contains` METHOD spelling (the wire + // token becomes `cs` in rebuildOrString). It maps to the same capability + // gate as `cs`; on a JSON column the term resolver then re-types it to + // searchableJson and validates the operand. selectorNe's IS-NULL-inclusive + // or-form relies on this arm. + if (op === 'contains') return 'freeTextSearch' + return this.queryTypeForRawOp(op) } /** @@ -1375,13 +1957,41 @@ export class EncryptedQueryBuilderImpl< } /** - * Post-process a decrypted result row. The v3 dialect reconstructs `Date` - * values from the encrypt-config `cast_as`; v2 returns rows unchanged. + * Post-process a decrypted result row: rebuild `Date` values from the + * encrypt-config `cast_as` (date/timestamp), mirroring the typed v3 client's + * decrypt-model path. */ protected postprocessDecryptedRow( row: Record, ): Record { - return row + // Every key an encrypted column can appear under: the keys this select + // actually produces (including caller-chosen aliases like `ts:createdAt`), + // plus the static property and DB names as a fallback for paths that record + // no select. Aliases win. Derived here from `this.selectColumns` (the row in + // hand) rather than cached from `buildSelectString`, so a reused builder can + // never postprocess a row with a previous operation's stale select map. + const keyToDb: Record = Object.assign( + Object.create(null), + this.selectColumns === null + ? undefined + : selectKeyToDbV3(this.selectColumns, this.propToDb), + ) + for (const [property, dbName] of Object.entries(this.propToDb)) { + keyToDb[property] ??= dbName + keyToDb[dbName] ??= dbName + } + + const out: Record = { ...row } + for (const [key, dbName] of Object.entries(keyToDb)) { + const castAs = this.columnSchemas[dbName]?.cast_as + if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue + const value = out[key] + if (value == null || value instanceof Date) continue + if (typeof value === 'string' || typeof value === 'number') { + out[key] = new Date(value) + } + } + return out } // --------------------------------------------------------------------------- @@ -1523,17 +2133,105 @@ export class EncryptedQueryBuilderImpl< // --------------------------------------------------------------------------- protected getColumnMap(): Record { - const map: Record = {} - const schema = this.schema as unknown as Record + return this.v3Columns as unknown as Record + } - for (const colName of this.encryptedColumnNames) { - const col = schema[colName] - if (col instanceof EncryptedColumn) { - map[colName] = col - } + /** Warn once per (op, column) that a `like`/`ilike` was delegated to `matches`. */ + private static readonly warnedLikeDelegation = new Set() + + /** True when `column` is one of this table's encrypted v3 columns. */ + private isEncryptedV3Column(column: string): boolean { + return Boolean(this.v3Columns[column]) + } + + /** True when `column` is an encrypted `types.Json` document column. */ + private isSearchableJsonColumn(column: string): boolean { + const builder: V3ColumnLike | undefined = this.v3Columns[column] + return Boolean(builder?.getQueryCapabilities().searchableJson) + } + + private assertPostgrestCanQueryEncryptedOperator( + method: string, + column: string, + ): void { + if (!this.queryDomainsRequired) return + throw new Error( + `[supabase v3]: ${method}() on encrypted column "${column}" is unavailable with EQL 3.0.2+: the SQL operator requires an eql_v3.query_* cast that PostgREST cannot express. Use the Drizzle or Prisma Next adapter, or a scoped SQL/RPC function.`, + ) + } + + /** + * Validate + reconstruct a selector needle: `('$.user.role', 'admin')` → + * `{user: {role: 'admin'}}`. Shared by {@link selectorEq}/{@link selectorNe}; + * throws with column context for a non-JSON column, an invalid path, or a + * non-scalar leaf. + */ + private selectorNeedle( + method: string, + column: string, + path: string, + value: unknown, + ): Record { + if (!this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: ${method}() requires an encrypted JSON (types.Json) column; "${column}" is not one.`, + ) } + // Selector comparisons compare a scalar LEAF (null included in the shared + // helper's rejection; eq/ne arm — `ordering: false`; + // PostgREST cannot express selector ordering yet, see + // cipherstash/encrypt-query-language#407). + const leafReason = unsupportedLeafReason(value, false) + if (leafReason) { + throw new Error( + `[supabase v3]: ${method}("${column}", "${path}", …): ${leafReason}`, + ) + } + // Stricter than the shared helper (whose Date/bigint arms serve the Drizzle + // surface): a stored JsonDocument leaf is a JSON scalar, so a Date/bigint + // needle could never match one — reject with the serialization steer + // instead of running a query that structurally returns nothing. + if ( + typeof value !== 'string' && + typeof value !== 'number' && + typeof value !== 'boolean' + ) { + throw new Error( + `[supabase v3]: ${method}("${column}", "${path}", …): a JSON document leaf is a JSON scalar (string/number/boolean); got ${value instanceof Date ? 'a Date — pass date.toISOString() (or the stored form)' : typeof value}.`, + ) + } + let segments: string[] + try { + segments = parseSelectorSegments(path) + } catch (err) { + throw new Error( + `[supabase v3]: ${method}("${column}", …): ${err instanceof Error ? err.message : String(err)}`, + ) + } + return reconstructSelectorDocument(segments, value) + } - return map + /** + * Reduce a SQL LIKE pattern to a fuzzy-match needle, or throw when it cannot be + * approximated. Strips surrounding `%` (prefix/suffix wildcards, which fuzzy + * matching subsumes); an internal `%` or any `_` is unapproximable. Warns once + * per (op, column) that the delegation is approximate. + */ + private likeNeedle(column: string, op: string, pattern: string): string { + const needle = pattern.replace(/^%+/, '').replace(/%+$/, '') + if (needle.includes('%') || pattern.includes('_')) { + throw new Error( + `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an internal "%" or any "_"). Use matches("${column}", term) with a literal search term.`, + ) + } + const key = `${op}:${column}` + if (!EncryptedQueryBuilderImpl.warnedLikeDelegation.has(key)) { + EncryptedQueryBuilderImpl.warnedLikeDelegation.add(key) + logger.warn( + `[supabase v3]: "${op}" on encrypted column "${column}" is delegated to matches() (fuzzy bloom token search). Results are APPROXIMATE — case-insensitive, one-sided (may false-positive), and wildcards/anchoring are not honored. Call matches() directly to make this explicit.`, + ) + } + return needle } } From fdffe69642201a0fc7820c691796b117a177b9d3 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:52:05 +1000 Subject: [PATCH 008/123] feat(stack-supabase)!: remove EQL v2 authoring surface, de-suffix v3 API encryptedSupabase is now the introspecting EQL v3 factory (was encryptedSupabaseV3); encryptedSupabaseV3 kept as a @deprecated type-identical alias. The legacy v2 encryptedSupabase({ encryptionClient, supabaseClient }).from(table, schema) wrapper and EncryptedSupabaseConfig are removed. All *V3 type exports de-suffixed to canonical names with @deprecated *V3 aliases retained. v2 decrypt is unaffected. --- packages/stack-supabase/README.md | 19 +- packages/stack-supabase/src/index.ts | 150 ++++++---------- packages/stack-supabase/src/introspect.ts | 6 +- packages/stack-supabase/src/types.ts | 207 +++++++++++++--------- 4 files changed, 197 insertions(+), 185 deletions(-) diff --git a/packages/stack-supabase/README.md b/packages/stack-supabase/README.md index 746cfc902..70aaf1981 100644 --- a/packages/stack-supabase/README.md +++ b/packages/stack-supabase/README.md @@ -9,9 +9,9 @@ Depends on `@cipherstash/stack`; install both: npm install @cipherstash/stack @cipherstash/stack-supabase @supabase/supabase-js ``` -## EQL v3 (recommended) +## EQL v3 -`encryptedSupabaseV3` introspects the database at connect time (native +`encryptedSupabase` introspects the database at connect time (native `public.eql_v3_*` column domains) — no schema argument, `select('*')` support, equality/range filters, and encrypted `order()` on OPE columns. @@ -22,18 +22,23 @@ Use the Drizzle or Prisma Next adapter, or a carefully scoped direct SQL/RPC path. ```ts -import { encryptedSupabaseV3 } from '@cipherstash/stack-supabase' +import { encryptedSupabase } from '@cipherstash/stack-supabase' -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) +const es = await encryptedSupabase(supabaseUrl, supabaseKey) await es.from('users').select('id, email').eq('email', 'a@b.com') ``` +`encryptedSupabaseV3` remains as a `@deprecated`, type-identical alias of +`encryptedSupabase`, so existing imports keep working. + Introspection needs a direct Postgres connection (`DATABASE_URL`), so `pg` is an optional peer and the factory cannot run in a Worker or the browser. -## EQL v2 (legacy) +## EQL v2 (removed) -`encryptedSupabase` wraps a supabase-js client with a v2 schema; still shipped for -existing v2 deployments. +The legacy EQL v2 authoring wrapper — `encryptedSupabase({ encryptionClient, +supabaseClient }).from(tableName, schema)` — has been removed; this package now +authors and queries EQL v3 only. Migrate existing v2 columns to an `eql_v3_*` +domain, or pin the last release that shipped the v2 wrapper. See the `stash-supabase` agent skill and https://cipherstash.com/docs for the full guide. diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index ec9847261..b295a587a 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -1,78 +1,18 @@ import { Encryption } from '@cipherstash/stack' -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' import type { UnmodelledColumn } from './introspect' import { eqlRequiresQueryDomains, introspect } from './introspect' import { EncryptedQueryBuilderImpl } from './query-builder' -import { EncryptedQueryBuilderV3Impl } from './query-builder-v3' import { mergeDeclaredTables, synthesizeTables } from './schema-builder' import type { - EncryptedQueryBuilder, - EncryptedSupabaseConfig, + EncryptedQueryBuilderUntyped, EncryptedSupabaseInstance, - EncryptedSupabaseV3Instance, - EncryptedSupabaseV3Options, + EncryptedSupabaseOptions, SupabaseClientLike, - TypedEncryptedSupabaseV3Instance, + TypedEncryptedSupabaseInstance, V3Schemas, } from './types' import { verifyDeclaredSchemas } from './verify' -/** - * Create an encrypted Supabase wrapper that transparently handles encryption - * and decryption for queries on encrypted columns. - * - * @param config - Configuration containing the encryption client and Supabase client. - * @returns An object with a `from()` method that mirrors `supabase.from()` but - * auto-encrypts mutations, adds `::jsonb` casts, encrypts filter values, and - * decrypts results. - * - * @example - * ```typescript - * import { Encryption } from '@cipherstash/stack' - * import { encryptedSupabase } from '@cipherstash/stack-supabase' - * import { encryptedTable, encryptedColumn } from '@cipherstash/stack/schema' - * - * const users = encryptedTable('users', { - * name: encryptedColumn('name').freeTextSearch().equality(), - * email: encryptedColumn('email').freeTextSearch().equality(), - * }) - * - * const client = await Encryption({ schemas: [users] }) - * const eSupabase = encryptedSupabase({ encryptionClient: client, supabaseClient: supabase }) - * - * // INSERT - auto-encrypts, auto-converts to PG composite - * await eSupabase.from('users', users) - * .insert({ name: 'John', email: 'john@example.com', age: 30 }) - * - * // SELECT with filter - auto-casts ::jsonb, auto-encrypts search term, auto-decrypts - * const { data } = await eSupabase.from('users', users) - * .select('id, email, name') - * .eq('email', 'john@example.com') - * ``` - */ -export function encryptedSupabase( - config: EncryptedSupabaseConfig, -): EncryptedSupabaseInstance { - const { encryptionClient, supabaseClient } = config - - return { - from = Record>( - tableName: string, - schema: EncryptedTable, - ) { - return new EncryptedQueryBuilderImpl( - tableName, - schema, - encryptionClient, - supabaseClient, - ) - }, - } -} - /** * Throw if `tableName` carries an EQL v3 column this SDK version cannot model. * @@ -101,11 +41,17 @@ function assertTableIsModelled( } /** - * Create an encrypted Supabase wrapper for **EQL v3** schemas by introspecting - * the database at connect time. Detects EQL v3 columns by their Postgres domain - * and derives each column's encryption config from it — callers no longer pass a - * schema to `from()`. Supplying `schemas` is optional: it adds compile-time - * types and verifies the declared tables against the database at construction. + * Create an encrypted Supabase wrapper over **native EQL v3 column domains** by + * introspecting the database at connect time. Detects EQL v3 columns by their + * Postgres domain and derives each column's encryption config from it — callers + * do not pass a schema to `from()`. Supplying `schemas` is optional: it adds + * compile-time types and verifies the declared tables against the database at + * construction. + * + * Encrypted data is stored as EQL v3 payloads. The generation-agnostic decrypt + * path in `@cipherstash/stack` still reads existing EQL v2 payloads, but this + * wrapper only AUTHORS EQL v3 — the legacy v2 authoring surface (a hand-written + * client-side schema and `from(tableName, schema)`) has been removed. * * Requires a Postgres connection (`options.databaseUrl` or `DATABASE_URL`) for * introspection, so it cannot run in a Worker or the browser. @@ -125,45 +71,45 @@ function assertTableIsModelled( * * @example * ```typescript - * const supabase = await encryptedSupabaseV3(supabaseUrl, supabaseKey) + * const supabase = await encryptedSupabase(supabaseUrl, supabaseKey) * await supabase.from('users').insert({ email: 'alice@example.com' }) * const { data } = await supabase.from('users').select().eq('email', 'alice@example.com') * ``` */ -export async function encryptedSupabaseV3( +export async function encryptedSupabase( supabaseUrl: string, supabaseKey: string, - options: EncryptedSupabaseV3Options & { schemas: S }, -): Promise> -export async function encryptedSupabaseV3( + options: EncryptedSupabaseOptions & { schemas: S }, +): Promise> +export async function encryptedSupabase( supabaseUrl: string, supabaseKey: string, - options?: EncryptedSupabaseV3Options, -): Promise -export async function encryptedSupabaseV3( + options?: EncryptedSupabaseOptions, +): Promise +export async function encryptedSupabase( supabaseClient: SupabaseClientLike, - options: EncryptedSupabaseV3Options & { schemas: S }, -): Promise> -export async function encryptedSupabaseV3( + options: EncryptedSupabaseOptions & { schemas: S }, +): Promise> +export async function encryptedSupabase( supabaseClient: SupabaseClientLike, - options?: EncryptedSupabaseV3Options, -): Promise -// The implementation's option params are `EncryptedSupabaseV3Options +// The implementation's option params are `EncryptedSupabaseOptions`, NOT ``. The no-schemas overloads take -// `EncryptedSupabaseV3Options` — i.e. ``, whose `schemas` is typed +// `EncryptedSupabaseOptions` — i.e. ``, whose `schemas` is typed // `undefined` — and TS2394s against an implementation param whose `schemas` is // typed `V3Schemas`. Widening the type argument to the full constraint makes // every overload relatable to the implementation signature. -export async function encryptedSupabaseV3( +export async function encryptedSupabase( clientOrUrl: SupabaseClientLike | string, - keyOrOptions?: string | EncryptedSupabaseV3Options, - maybeOptions?: EncryptedSupabaseV3Options, + keyOrOptions?: string | EncryptedSupabaseOptions, + maybeOptions?: EncryptedSupabaseOptions, ): Promise< - EncryptedSupabaseV3Instance | TypedEncryptedSupabaseV3Instance + EncryptedSupabaseInstance | TypedEncryptedSupabaseInstance > { // 1. Resolve the Supabase client + options from the overload shape. let supabaseClient: SupabaseClientLike - let options: EncryptedSupabaseV3Options + let options: EncryptedSupabaseOptions if (typeof clientOrUrl === 'string') { const url = clientOrUrl const key = keyOrOptions as string @@ -180,10 +126,10 @@ export async function encryptedSupabaseV3( if (code !== 'MODULE_NOT_FOUND' && code !== 'ERR_MODULE_NOT_FOUND') throw err throw new Error( - "[supabase v3]: encryptedSupabaseV3(url, key) needs '@supabase/supabase-js' " + + "[supabase v3]: encryptedSupabase(url, key) needs '@supabase/supabase-js' " + 'to build the client, but that optional peer dependency is not installed. ' + 'Install it (`npm install @supabase/supabase-js`), or pass an existing ' + - 'client: encryptedSupabaseV3(supabaseClient, options).', + 'client: encryptedSupabase(supabaseClient, options).', { cause: err }, ) } @@ -191,7 +137,7 @@ export async function encryptedSupabaseV3( } else { supabaseClient = clientOrUrl options = - (keyOrOptions as EncryptedSupabaseV3Options) ?? {} + (keyOrOptions as EncryptedSupabaseOptions) ?? {} } // 2. Resolve the database URL for introspection. @@ -273,34 +219,46 @@ export async function encryptedSupabaseV3( // ciphertext for one. Never make it optional. assertTableIsModelled(tableName, unmodelled) const allColumns = synth.allColumns.get(tableName) ?? null - return new EncryptedQueryBuilderV3Impl( + return new EncryptedQueryBuilderImpl( tableName, table, encryptionClient, supabaseClient, allColumns, queryDomainsRequired, - ) as unknown as EncryptedQueryBuilder> + ) as unknown as EncryptedQueryBuilderUntyped> }, } return instance as unknown as - | EncryptedSupabaseV3Instance - | TypedEncryptedSupabaseV3Instance + | EncryptedSupabaseInstance + | TypedEncryptedSupabaseInstance } +/** + * @deprecated Use {@link encryptedSupabase}. `encryptedSupabaseV3` is a + * type-identical alias kept for existing imports; the `V3` suffix is redundant + * now that EQL v3 is the only generation this wrapper authors. + */ +export const encryptedSupabaseV3 = encryptedSupabase + export type { EncryptedQueryBuilder, EncryptedQueryBuilderCore, + EncryptedQueryBuilderUntyped, + // Deprecated `*V3` aliases (Decision 5 — supabase keeps type-identical aliases). EncryptedQueryBuilderV3, EncryptedQueryBuilderV3Untyped, - EncryptedSupabaseConfig, EncryptedSupabaseError, EncryptedSupabaseInstance, + EncryptedSupabaseOptions, EncryptedSupabaseResponse, EncryptedSupabaseV3Instance, EncryptedSupabaseV3Options, + FilterableKeys, + FreeTextSearchableKeys, PendingOrCondition, SupabaseClientLike, + TypedEncryptedSupabaseInstance, TypedEncryptedSupabaseV3Instance, V3FilterableKeys, V3FreeTextSearchableKeys, diff --git a/packages/stack-supabase/src/introspect.ts b/packages/stack-supabase/src/introspect.ts index 18501468c..d1619f358 100644 --- a/packages/stack-supabase/src/introspect.ts +++ b/packages/stack-supabase/src/introspect.ts @@ -189,10 +189,10 @@ export async function loadPg( if (code !== 'MODULE_NOT_FOUND' && code !== 'ERR_MODULE_NOT_FOUND') throw err throw new Error( - '[supabase v3]: encryptedSupabaseV3 introspects the database over a direct ' + + '[supabase v3]: encryptedSupabase introspects the database over a direct ' + "Postgres connection, but the optional peer dependency 'pg' is not installed. " + - 'Install it (`npm install pg`). This also means encryptedSupabaseV3 cannot run ' + - 'in a Worker or the browser — use encryptedSupabase (EQL v2) there.', + 'Install it (`npm install pg`). This also means encryptedSupabase cannot run ' + + 'in a Worker or the browser, where a direct Postgres connection is unavailable.', { cause: err }, ) } diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index 0ea50ca9d..cc5378d56 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -19,32 +19,16 @@ import type { V3Schemas } from './schema-builder' // Config & instance // --------------------------------------------------------------------------- -export type EncryptedSupabaseConfig = { - encryptionClient: EncryptionClient - supabaseClient: SupabaseClientLike -} - -export interface EncryptedSupabaseInstance { - from = Record>( - tableName: string, - schema: EncryptedTable, - ): EncryptedQueryBuilder -} - -// --------------------------------------------------------------------------- -// EQL v3 config & instance -// --------------------------------------------------------------------------- - export type { V3Schemas } /** - * Options for {@link import('./index').encryptedSupabaseV3}. + * Options for {@link import('./index').encryptedSupabase}. * * @typeParam S - declared v3 tables. When present, `from()` is constrained to * the declared table names and returns typed builders, and the tables are * verified against the database at construction. */ -export type EncryptedSupabaseV3Options< +export type EncryptedSupabaseOptions< S extends V3Schemas | undefined = undefined, > = { /** Postgres connection string for introspection. Defaults to @@ -61,7 +45,7 @@ export type EncryptedSupabaseV3Options< * Declaring a `text_search` column does NOT change its match behaviour: a * declared and a synthesized `text_search` column build byte-identically, and * neither `types.TextSearch` nor `EncryptedTextSearchColumn` accepts match - * options. See the `contains` note on `EncryptedQueryBuilderV3Impl`. + * options. See the `contains` note on `EncryptedQueryBuilderImpl`. */ schemas?: S } @@ -82,7 +66,7 @@ type V3ColumnsOfTable
= Table extends { * the filterable keys so a filter on one is a type error, matching the runtime * guard in the v3 term encryption path. */ -export type NonQueryableV3Keys
= { +export type NonQueryableKeys
= { [K in Extract, string>]: [ QueryTypesForColumn[K]>, ] extends [never] @@ -124,7 +108,7 @@ type NonScalarQueryableV3Keys
= { * before #650's `searchableJson` arm the two sets coincided). Plaintext * (non-schema) columns pass through untouched, exactly as in v2. */ -export type V3FilterableKeys< +export type FilterableKeys< Table extends AnyV3Table, Row extends Record, > = Exclude, NonScalarQueryableV3Keys
> @@ -150,7 +134,7 @@ type NonFreeTextSearchV3Keys
= { * table's encrypted columns that lack a match index. Plaintext columns pass * through, where `contains` is PostgREST's native jsonb/array containment. */ -export type V3FreeTextSearchableKeys< +export type FreeTextSearchableKeys< Table extends AnyV3Table, Row extends Record, > = Exclude, NonFreeTextSearchV3Keys
> @@ -159,13 +143,13 @@ export type V3FreeTextSearchableKeys< * Row keys `matches()` accepts: ONLY the table's ENCRYPTED columns that carry a * `freeTextSearch` capability (`public.eql_v3_text_match` / `text_search`). * - * Unlike {@link V3FreeTextSearchableKeys} (which additionally lets plaintext keys + * Unlike {@link FreeTextSearchableKeys} (which additionally lets plaintext keys * through, because the old `contains` also served native containment), this * excludes plaintext columns entirely — `matches()` is encrypted free-text only, * so calling it on a plaintext column is a compile error, not a runtime throw. * Derived from the encrypted-column keys minus the non-free-text ones. */ -export type V3EncryptedFreeTextKeys< +export type EncryptedFreeTextKeys< Table extends AnyV3Table, Row extends Record, > = Exclude< @@ -194,9 +178,9 @@ type NonSearchableJsonV3Keys
= { * columns whose domain is `public.eql_v3_json_search` (`types.Json`). Plaintext * columns are excluded — on those, `contains()` is PostgREST-native containment * and the selector methods do not apply. Mirror of - * {@link V3EncryptedFreeTextKeys} for the `searchableJson` capability. + * {@link EncryptedFreeTextKeys} for the `searchableJson` capability. */ -export type V3SearchableJsonKeys< +export type SearchableJsonKeys< Table extends AnyV3Table, Row extends Record, > = Exclude< @@ -265,7 +249,7 @@ type PlaintextContainsValue = V extends readonly unknown[] * excluded here to match the runtime rejection in `validateTransforms`, * rather than type-checking clean and throwing at execute time. */ -export type NonOrderableV3Keys
= { +export type NonOrderableKeys
= { [K in Extract< keyof V3ColumnsOfTable
, string @@ -285,25 +269,25 @@ export type NonOrderableV3Keys
= { * `jsonb_cmp` and compares the random ciphertext first. But the builder does not * emit a bare `ORDER BY`: for an encrypted ordering column it emits the jsonb * path `col->op`, which selects the OPE term, and OPE is order-preserving. See - * `EncryptedQueryBuilderV3Impl.orderColumnName`. + * `EncryptedQueryBuilderImpl.orderColumnName`. * * ORE-backed (`*_ord_ore`) columns are excluded at compile time by - * {@link NonOrderableV3Keys} — the builder sorts through a jsonb path that + * {@link NonOrderableKeys} — the builder sorts through a jsonb path that * cannot reach their superuser-only ORE opclass, so `.order()` on one is a type * error, matching the runtime rejection in `validateTransforms` (defense in * depth for the untyped `.order(someString)` path). */ -export type V3OrderableKeys< +export type OrderableKeys< Table extends AnyV3Table, Row extends Record, -> = Exclude, NonOrderableV3Keys
> +> = Exclude, NonOrderableKeys
> /** * Row keys that are NOT encrypted v3 columns. Used where a method's operand is a * SQL value rather than a ciphertext envelope — `is(col, true)` in particular, * since an encrypted column holds jsonb and can never be `IS TRUE`. */ -export type V3PlaintextKeys< +export type PlaintextKeys< Table extends AnyV3Table, Row extends Record, > = Exclude< @@ -313,86 +297,86 @@ export type V3PlaintextKeys< /** * The v3 builder type: the shared {@link EncryptedQueryBuilderCore} surface with - * filter methods narrowed to {@link V3FilterableKeys} and `order()` to - * {@link V3OrderableKeys}. + * filter methods narrowed to {@link FilterableKeys} and `order()` to + * {@link OrderableKeys}. * * `like`/`ilike` are absent by construction. EQL v3 free-text search is fuzzy * bloom-filter token matching (`@>`), not SQL wildcard matching — `%` is * tokenized like any other character, so a `like` pattern is a category error. * The v3 dialect of Drizzle omits them for the same reason. Use `matches`. */ -export interface EncryptedQueryBuilderV3< +export interface EncryptedQueryBuilder< Table extends AnyV3Table, Row extends Record, > extends EncryptedQueryBuilderCore< Row, - V3FilterableKeys & StringKeyOf, - EncryptedQueryBuilderV3, - V3OrderableKeys & StringKeyOf, + FilterableKeys & StringKeyOf, + EncryptedQueryBuilder, + OrderableKeys & StringKeyOf, // `is(col, true)` is legal only on a PLAINTEXT column: an encrypted column // holds a jsonb envelope, never a SQL boolean. The two axes were threaded // separately "so they can diverge", and now they have — `order()` admits // encrypted ordering columns (sorted by their `op` term), `is(col, true)` // still admits none. - V3PlaintextKeys & StringKeyOf + PlaintextKeys & StringKeyOf > { /** Encrypted free-text token match on legacy EQL versions. EQL 3.0.2+ * requires a query-domain cast PostgREST cannot express, so this fails fast. */ - matches & StringKeyOf>( + matches & StringKeyOf>( column: K, value: string, - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Native (exact) jsonb/array containment (`@>`). Plaintext columns only — an * encrypted column is a compile error (use {@link matches}). A scalar plaintext * column resolves its operand to `never` (`@>` is array/jsonb only). */ - contains & StringKeyOf>( + contains & StringKeyOf>( column: K, value: PlaintextContainsValue, - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Encrypted JSON containment on legacy EQL versions. EQL 3.0.2+ requires an * `eql_v3.query_json` cast PostgREST cannot express, so this fails fast before * encrypting an operand into the request URL. */ - contains & StringKeyOf>( + contains & StringKeyOf>( column: K, value: EncryptedJsonContainsValue, - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Encrypted JSONPath equality on legacy EQL versions. EQL 3.0.2+ fails fast * because PostgREST cannot express the required query-domain cast. */ - selectorEq & StringKeyOf>( + selectorEq & StringKeyOf>( column: K, path: string, value: SelectorLeafValue, - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Encrypted JSONPath inequality on legacy EQL versions. EQL 3.0.2+ fails * fast because PostgREST cannot express the required query-domain cast. */ - selectorNe & StringKeyOf>( + selectorNe & StringKeyOf>( column: K, path: string, value: SelectorLeafValue, - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Raw legacy containment spelling. EQL 3.0.2+ rejects this before sending. */ - filter & StringKeyOf>( + filter & StringKeyOf>( column: K, operator: 'cs', value: EncryptedJsonContainsValue, - ): EncryptedQueryBuilderV3 - filter & StringKeyOf>( + ): EncryptedQueryBuilder + filter & StringKeyOf>( column: K, operator: string, value: Row[K], - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder /** Negated legacy containment spelling. EQL 3.0.2+ rejects this before * sending. */ - not & StringKeyOf>( + not & StringKeyOf>( column: K, operator: 'contains', value: EncryptedJsonContainsValue, - ): EncryptedQueryBuilderV3 - not & StringKeyOf>( + ): EncryptedQueryBuilder + not & StringKeyOf>( column: K, operator: string, value: Row[K], - ): EncryptedQueryBuilderV3 + ): EncryptedQueryBuilder } /** @@ -407,12 +391,12 @@ export interface EncryptedQueryBuilderV3< * union (which subsumes the encrypted column's `string`); the runtime resolves * the column and picks the encoding (and rejects the wrong-column-kind pairing). */ -export interface EncryptedQueryBuilderV3Untyped< +export interface EncryptedQueryBuilderUntyped< Row extends Record, > extends EncryptedQueryBuilderCore< Row, StringKeyOf, - EncryptedQueryBuilderV3Untyped + EncryptedQueryBuilderUntyped > { /** Fuzzy free-text token match on an encrypted match/search column. The * operand is always the string term to tokenize (never an array/object), even @@ -420,33 +404,33 @@ export interface EncryptedQueryBuilderV3Untyped< matches>( column: K, value: string, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped /** Native jsonb/array containment on plaintext columns. Encrypted JSON * containment fails fast on EQL 3.0.2+. */ contains>( column: K, value: NativeContainsValue, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped /** Legacy encrypted JSONPath equality; fails fast on EQL 3.0.2+. */ selectorEq>( column: K, path: string, value: SelectorLeafValue, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped /** Legacy encrypted JSONPath inequality; fails fast on EQL 3.0.2+. */ selectorNe>( column: K, path: string, value: SelectorLeafValue, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped } /** Untyped instance (no `schemas`): rows default to `Record` * and `from` accepts any table name. */ -export interface EncryptedSupabaseV3Instance { +export interface EncryptedSupabaseInstance { from = Record>( tableName: string, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped } /** Typed instance (with `schemas: S`): a declared table name resolves to the @@ -463,13 +447,13 @@ export interface EncryptedSupabaseV3Instance { * Overload order matters: the literal-constrained signature is declared first, * so TypeScript prefers it whenever the argument is a declared key and only * falls through to `string` otherwise. */ -export interface TypedEncryptedSupabaseV3Instance { +export interface TypedEncryptedSupabaseInstance { from( table: K, - ): EncryptedQueryBuilderV3> + ): EncryptedQueryBuilder> from = Record>( table: string, - ): EncryptedQueryBuilderV3Untyped + ): EncryptedQueryBuilderUntyped } // --------------------------------------------------------------------------- @@ -801,7 +785,7 @@ export interface EncryptedQueryBuilderCore< FK extends StringKeyOf, Self, /** Keys `order()` accepts. Defaults to `FK`, so the v2 surface is unchanged; - * v3 narrows it to plaintext columns (see {@link V3OrderableKeys}). */ + * v3 narrows it to plaintext columns (see {@link OrderableKeys}). */ OK extends StringKeyOf = FK, /** Keys the BOOLEAN form of `is()` accepts. Defaults to `FK`, so the v2 * surface is unchanged; v3 narrows it to plaintext columns. Distinct from @@ -905,19 +889,84 @@ export interface EncryptedQueryBuilderCore< csv(): Self abortSignal(signal: AbortSignal): Self throwOnError(): Self - /** Escape hatch: re-types the rows and drops back to the v2 builder surface. */ - returns>(): EncryptedQueryBuilder + /** Escape hatch: re-types the rows and drops back to the untyped v3 builder + * surface. */ + returns>(): EncryptedQueryBuilderUntyped /** Bind identity-aware encryption. Accepts either a plain * `{ identityClaim }` (the common form) or a `LockContext` instance. */ withLockContext(lockContext: LockContextInput): Self audit(config: AuditConfig): Self } -/** The v2 builder: free-text search via SQL wildcard matching. */ -export interface EncryptedQueryBuilder< - T extends Record = Record, - FK extends StringKeyOf = StringKeyOf, -> extends EncryptedQueryBuilderCore> { - like(column: K, pattern: string): EncryptedQueryBuilder - ilike(column: K, pattern: string): EncryptedQueryBuilder -} +// --------------------------------------------------------------------------- +// Deprecated `*V3` aliases (Decision 5 — supabase keeps type-identical aliases). +// The v3 names are now the unsuffixed canonical exports; these aliases keep +// existing `*V3` imports compiling. +// --------------------------------------------------------------------------- + +/** @deprecated Use {@link EncryptedSupabaseOptions}. */ +export type EncryptedSupabaseV3Options< + S extends V3Schemas | undefined = undefined, +> = EncryptedSupabaseOptions + +/** @deprecated Use {@link NonQueryableKeys}. */ +export type NonQueryableV3Keys
= + NonQueryableKeys
+ +/** @deprecated Use {@link FilterableKeys}. */ +export type V3FilterableKeys< + Table extends AnyV3Table, + Row extends Record, +> = FilterableKeys + +/** @deprecated Use {@link FreeTextSearchableKeys}. */ +export type V3FreeTextSearchableKeys< + Table extends AnyV3Table, + Row extends Record, +> = FreeTextSearchableKeys + +/** @deprecated Use {@link EncryptedFreeTextKeys}. */ +export type V3EncryptedFreeTextKeys< + Table extends AnyV3Table, + Row extends Record, +> = EncryptedFreeTextKeys + +/** @deprecated Use {@link SearchableJsonKeys}. */ +export type V3SearchableJsonKeys< + Table extends AnyV3Table, + Row extends Record, +> = SearchableJsonKeys + +/** @deprecated Use {@link NonOrderableKeys}. */ +export type NonOrderableV3Keys
= + NonOrderableKeys
+ +/** @deprecated Use {@link OrderableKeys}. */ +export type V3OrderableKeys< + Table extends AnyV3Table, + Row extends Record, +> = OrderableKeys + +/** @deprecated Use {@link PlaintextKeys}. */ +export type V3PlaintextKeys< + Table extends AnyV3Table, + Row extends Record, +> = PlaintextKeys + +/** @deprecated Use {@link EncryptedQueryBuilder}. */ +export type EncryptedQueryBuilderV3< + Table extends AnyV3Table, + Row extends Record, +> = EncryptedQueryBuilder + +/** @deprecated Use {@link EncryptedQueryBuilderUntyped}. */ +export type EncryptedQueryBuilderV3Untyped< + Row extends Record, +> = EncryptedQueryBuilderUntyped + +/** @deprecated Use {@link EncryptedSupabaseInstance}. */ +export type EncryptedSupabaseV3Instance = EncryptedSupabaseInstance + +/** @deprecated Use {@link TypedEncryptedSupabaseInstance}. */ +export type TypedEncryptedSupabaseV3Instance = + TypedEncryptedSupabaseInstance From a9a6b78caef32b272a4cc12f49ff5e832bfa89b8 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:52:05 +1000 Subject: [PATCH 009/123] test(stack-supabase): update suites for folded v3 builder + de-suffix Point tests at the folded EncryptedQueryBuilderImpl, drop the removed v2 authoring tests (v2 live suite, v2 wire-encoding block, v2 builder type test), convert the shared execute() error-threading tests to a v3 table, and add canonical-name type assertions. --- .../supabase-encryption-error.test.ts | 37 +- .../__tests__/supabase-v3-builder.test.ts | 292 +-------------- .../__tests__/supabase-v3-factory.test.ts | 2 +- .../__tests__/supabase-v3-json.test.ts | 2 +- .../__tests__/supabase-v3-matrix.test.ts | 2 +- .../__tests__/supabase-v3-select-star.test.ts | 2 +- .../__tests__/supabase-v3-wire.test.ts | 2 +- .../__tests__/supabase-v3.test-d.ts | 45 ++- .../stack-supabase/__tests__/supabase.test.ts | 335 ------------------ .../integration/wire.integration.test.ts | 2 +- 10 files changed, 51 insertions(+), 670 deletions(-) delete mode 100644 packages/stack-supabase/__tests__/supabase.test.ts diff --git a/packages/stack-supabase/__tests__/supabase-encryption-error.test.ts b/packages/stack-supabase/__tests__/supabase-encryption-error.test.ts index 5154b8c1e..b9acdd008 100644 --- a/packages/stack-supabase/__tests__/supabase-encryption-error.test.ts +++ b/packages/stack-supabase/__tests__/supabase-encryption-error.test.ts @@ -1,13 +1,8 @@ import type { EncryptionClient } from '@cipherstash/stack/encryption' import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { EncryptionErrorTypes } from '@cipherstash/stack/errors' -import { - encryptedColumn, - encryptedTable as encryptedTableV2, -} from '@cipherstash/stack/schema' import { describe, expect, it } from 'vitest' import { EncryptedQueryBuilderImpl } from '../src/query-builder' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' import { createMockEncryptionClient, createMockSupabase, @@ -18,18 +13,14 @@ import { /** * Regression coverage for #626: the query builder's catch block used to hardcode * `encryptionError: undefined`, so the typed `EncryptedSupabaseError.encryptionError` - * field was dead. The v2 tests pin that a genuine encryption failure now threads its - * `EncryptionError` through the shared base `execute()` catch, while a plain - * (non-encryption) throw leaves it unset. The v3 tests cover the dialect's own - * `encryptionFailure` path, which synthesizes an `EncryptionError` for its two - * query-term contract-violation cases (length mismatch, null envelope) that have - * no operation failure to wrap. + * field was dead. The `execute()` tests pin that a genuine encryption failure now + * threads its `EncryptionError` through the shared `execute()` catch, while a plain + * (non-encryption) throw leaves it unset. The `encryptionFailure` tests cover the + * dialect's own `encryptionFailure` path, which synthesizes an `EncryptionError` + * for its two query-term contract-violation cases (length mismatch, null envelope) + * that have no operation failure to wrap. */ -const usersV2 = encryptedTableV2('users', { - email: encryptedColumn('email').freeTextSearch().equality(), -}) - const usersV3 = encryptedTable('users', { email: types.TextEq('email'), }) @@ -48,7 +39,7 @@ function failingOperation(failure: { type: string; message: string }) { } describe('EncryptedSupabaseError.encryptionError (#626)', () => { - it('v2: threads the EncryptionError through on an encryption failure', async () => { + it('threads the EncryptionError through on an encryption failure', async () => { const failure = { type: EncryptionErrorTypes.EncryptionError, message: 'zerokms unreachable', @@ -62,7 +53,7 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { const { client: supabase } = createMockSupabase() const builder = new EncryptedQueryBuilderImpl( 'users', - usersV2, + usersV3, encryptionClient as unknown as EncryptionClient, supabase, ) @@ -77,7 +68,7 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { ) }) - it('v2: leaves encryptionError unset on a plain (non-encryption) error', async () => { + it('leaves encryptionError unset on a plain (non-encryption) error', async () => { const encryptionClient = createMockEncryptionClient() const { client: supabase } = createMockSupabase() // Make the underlying supabase call throw a non-encryption error: an insert @@ -88,7 +79,7 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { const builder = new EncryptedQueryBuilderImpl( 'users', - usersV2, + usersV3, encryptionClient, supabase, ) @@ -105,8 +96,8 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { // wrap for its contract-violation cases, so it synthesizes an EncryptionError. // Drive it directly: a two-element `in` list whose bulkEncrypt returns one // term trips the length-mismatch check. (The base `execute()` threading is - // already covered by the v2 test above; overriding `encryptModel` here would - // only re-run that shared path, not this v3-specific branch.) + // already covered by the `execute()` tests above; overriding `encryptModel` + // here would only re-run that shared path, not this v3-specific branch.) const encryptionClient = createMockEncryptionClient() as unknown as { bulkEncrypt: (...args: unknown[]) => unknown } @@ -114,7 +105,7 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { operation([{ data: fakeEnvelope('ada', 'email') }]) const { client: supabase } = createMockSupabase() - const { error, status } = await new EncryptedQueryBuilderV3Impl( + const { error, status } = await new EncryptedQueryBuilderImpl( 'users', usersV3, encryptionClient as unknown as EncryptionClient, @@ -144,7 +135,7 @@ describe('EncryptedSupabaseError.encryptionError (#626)', () => { operation([{ data: null }, { data: fakeEnvelope('grace', 'email') }]) const { client: supabase } = createMockSupabase() - const { error, status } = await new EncryptedQueryBuilderV3Impl( + const { error, status } = await new EncryptedQueryBuilderImpl( 'users', usersV3, encryptionClient as unknown as EncryptionClient, diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index b510a9baa..92e4a6649 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -1,11 +1,6 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' -import { - encryptedColumn, - encryptedTable as encryptedTableV2, -} from '@cipherstash/stack/schema' import { describe, expect, it, vi } from 'vitest' -import { encryptedSupabase } from '../src/index.js' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { createMockEncryptionClient, createMockSupabase, @@ -29,11 +24,6 @@ const users = encryptedTable('users', { bio: types.TextMatch('bio'), }) -const usersV2 = encryptedTableV2('users', { - email: encryptedColumn('email').freeTextSearch().equality(), - age: encryptedColumn('age').dataType('number').equality().orderAndRange(), -}) - // DB column names as introspection would report them (id/note are plaintext). const USERS_ALL_COLUMNS = [ 'id', @@ -1176,286 +1166,6 @@ describe('encryptedSupabaseV3 wire encoding', () => { }) }) -// --------------------------------------------------------------------------- -// v2 regression — the dialect seams must leave the v2 wire encoding untouched -// --------------------------------------------------------------------------- - -describe('encryptedSupabase (v2) wire encoding is unchanged by the dialect seams', () => { - function v2Instance(resultData: unknown = []) { - const supabase = createMockSupabase(resultData) - const es = encryptedSupabase({ - encryptionClient: createMockEncryptionClient(), - supabaseClient: supabase.client, - }) - return { es, supabase } - } - - it('wraps encrypted mutation values in the { data } composite shape', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).insert({ email: 'a@b.com', note: 'x' }) - - const [insert] = supabase.callsFor('insert') - const body = insert.args[0] as Record - expect(body.email).toHaveProperty('data') - expect(isFakeEnvelope((body.email as Record).data)).toBe( - true, - ) - expect(body.note).toBe('x') - }) - - it('encodes filter terms as composite literals via encryptQuery', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id, email').eq('email', 'a@b.com') - - const [eq] = supabase.callsFor('eq') - expect(eq.args).toEqual(['email', '("a@b.com")']) - }) - - it('keeps like on encrypted columns as like', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id, email').like('email', 'a@b') - - expect(supabase.callsFor('like')).toHaveLength(1) - expect(supabase.callsFor('filter')).toHaveLength(0) - }) - - it('adds plain ::jsonb casts without aliasing', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id, email, age') - - const [select] = supabase.callsFor('select') - expect(select.args[0]).toBe('id, email::jsonb, age::jsonb') - }) - - it('passes order() column names through unchanged', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id, age').order('age') - - const [order] = supabase.callsFor('order') - expect(order.args[0]).toBe('age') - }) - - it('passes the onConflict option through by reference', async () => { - const { es, supabase } = v2Instance() - - const options = { onConflict: 'email' } - await es.from('users', usersV2).upsert({ email: 'a@b.com' }, options) - - const [upsert] = supabase.callsFor('upsert') - expect(upsert.args[1]).toBe(options) - }) - - it('passes an all-plaintext or() string through verbatim', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id').or('id.eq.1,note.eq.x') - - const [or] = supabase.callsFor('or') - expect(or.args[0]).toBe('id.eq.1,note.eq.x') - }) - - // `contains` is not a PostgREST operator in EITHER dialect — the structured - // or() path emitted `note.contains.x` on v2 too, since the base builder never - // translated the token. Fixed in `rebuildOrString`, so both dialects inherit it. - it('translates a structured or() contains to cs', async () => { - const { es, supabase } = v2Instance() - - await es - .from('users', usersV2) - .select('id') - .or([{ column: 'note', op: 'contains', value: 'x' }]) - - expect(supabase.callsFor('or')[0].args[0]).toBe('note.cs.x') - }) - - // ------------------------------------------------------------------------- - // Characterization tests for the paths `toDbSpace()` will rewrite. Each pins - // the correlation between the term collector (`encryptFilterValues`) and the - // applier (`applyFilters`), which agree only by array index / column name. - // ------------------------------------------------------------------------- - - it('match() encrypts encrypted keys and passes plaintext through', async () => { - const { es, supabase } = v2Instance() - - await es - .from('users', usersV2) - .select('id') - .match({ email: 'a@b.com', note: 'plain' }) - - const [match] = supabase.callsFor('match') - const query = match.args[0] as Record - expect(query.email).toBe('("a@b.com")') - expect(query.note).toBe('plain') - // Key order survives the Record -> entries -> Record round-trip - expect(Object.keys(query)).toEqual(['email', 'note']) - }) - - it('not() encrypts on encrypted columns and passes plaintext through', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id').not('email', 'eq', 'a@b.com') - await es.from('users', usersV2).select('id').not('note', 'eq', 'plain') - - const [encrypted, plain] = supabase.callsFor('not') - expect(encrypted.args).toEqual(['email', 'eq', '("a@b.com")']) - expect(plain.args).toEqual(['note', 'eq', 'plain']) - }) - - // The v2 composite literal `("a@b.com")` is itself quote-bearing, so it needs - // the same escaped operand as v3 — postgrest-js's `in()` would emit - // `in.("("a@b.com")")` and PostgREST would reject it. - it('in() encrypts each element and leaves plaintext arrays alone', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id').in('email', ['a@b.com', 'c@d']) - await es.from('users', usersV2).select('id').in('note', ['x', 'y']) - - const [encrypted] = supabase.callsFor('filter') - expect(encrypted.args).toEqual([ - 'email', - 'in', - '("(\\"a@b.com\\")","(\\"c@d\\")")', - ]) - - const [plain] = supabase.callsFor('in') - expect(plain.args[1]).toEqual(['x', 'y']) - }) - - it('is() leaves the value untouched on an encrypted column', async () => { - const { es, supabase } = v2Instance() - - await es.from('users', usersV2).select('id').is('email', null) - - const [is] = supabase.callsFor('is') - expect(is.args).toEqual(['email', null]) - }) - - it('filter() encrypts the operand on an encrypted column', async () => { - const { es, supabase } = v2Instance() - - await es - .from('users', usersV2) - .select('id') - .filter('email', 'eq', 'a@b.com') - await es.from('users', usersV2).select('id').filter('note', 'eq', 'plain') - - const [encrypted, plain] = supabase.callsFor('filter') - expect(encrypted.args).toEqual(['email', 'eq', '("a@b.com")']) - expect(plain.args).toEqual(['note', 'eq', 'plain']) - }) - - // The single most important characterization test: a strict nonempty SUBSET - // of the or-string's conditions is encrypted, so the condition index `j` must - // agree between the two `parseOrString` calls that `toDbSpace()` collapses - // into one. - it('or() rebuilds a mixed encrypted/plaintext string, keeping each condition on its own column', async () => { - const { es, supabase } = v2Instance() - - await es - .from('users', usersV2) - .select('id') - .or('email.eq.a@b.com,note.eq.x') - - const [or] = supabase.callsFor('or') - const emitted = or.args[0] as string - const [emailCond, noteCond] = emitted.split(',') - expect(emailCond).toContain('email.eq.') - expect(emailCond).toContain('a@b.com') - expect(noteCond).toBe('note.eq.x') - }) - - it('keeps every filter array correlated in a combined query', async () => { - const { es, supabase } = v2Instance() - - await es - .from('users', usersV2) - .select('id, email, age') - .eq('email', 'a@b.com') - .not('age', 'eq', 30) - .or('email.eq.c@d,note.eq.x') - .match({ email: 'e@f.com' }) - .filter('age', 'gte', 18) - .order('age') - .limit(10) - - expect(supabase.callsFor('eq')[0].args).toEqual(['email', '("a@b.com")']) - expect(supabase.callsFor('not')[0].args).toEqual(['age', 'eq', '("30")']) - expect(supabase.callsFor('or')[0].args[0]).toContain('note.eq.x') - expect( - (supabase.callsFor('match')[0].args[0] as Record).email, - ).toBe('("e@f.com")') - expect(supabase.callsFor('filter')[0].args).toEqual([ - 'age', - 'gte', - '("18")', - ]) - expect(supabase.callsFor('order')[0].args[0]).toBe('age') - expect(supabase.callsFor('limit')[0].args[0]).toBe(10) - }) - - // Released-side regressions. `is` is a SQL predicate — PostgREST accepts only - // null/true/false — and a null operand is SQL NULL, never a value to search - // for. Only the regular `.is()` filter skipped encryption; every other - // collector encrypted whatever it was handed, emitting operands PostgREST - // rejects. - describe('is / null operands are never encrypted', () => { - it('does not encrypt not(col, is, null)', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').not('age', 'is', null) - expect(supabase.callsFor('not')[0].args).toEqual(['age', 'is', null]) - }) - - it('does not encrypt a raw filter() is operand', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').filter('age', 'is', null) - expect(supabase.callsFor('filter')[0].args).toEqual(['age', 'is', null]) - }) - - it('forwards an or() is condition unencrypted', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').or('age.is.null') - expect(supabase.callsFor('or')[0].args[0]).toBe('age.is.null') - }) - - it('does not encrypt a null eq() operand', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').eq('email', null) - expect(supabase.callsFor('eq')[0].args).toEqual(['email', null]) - }) - - it('does not encrypt a null match() value', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').match({ email: null }) - expect(supabase.callsFor('match')[0].args[0]).toEqual({ email: null }) - }) - - it('does not encrypt null elements of an in() list', async () => { - const { es, supabase } = v2Instance() - await es - .from('users', usersV2) - .select('id') - .in('email', ['a@b.com', null]) - // `null` stays a bare PostgREST `null`, never a ciphertext. - expect(supabase.callsFor('filter')[0].args).toEqual([ - 'email', - 'in', - '("(\\"a@b.com\\")",null)', - ]) - }) - - it('treats is() as a predicate even with a non-null operand', async () => { - const { es, supabase } = v2Instance() - await es.from('users', usersV2).select('id').is('email', false) - expect(supabase.callsFor('is')[0].args).toEqual(['email', false]) - }) - }) -}) - // --------------------------------------------------------------------------- // Property → DB name collision // diff --git a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts index 622af6227..b9cebc982 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts @@ -3,7 +3,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import type { SupabaseClientLike } from '../src/index.js' import { encryptedSupabaseV3 } from '../src/index.js' import type { IntrospectionData } from '../src/introspect' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' // --- Mocks ----------------------------------------------------------------- // diff --git a/packages/stack-supabase/__tests__/supabase-v3-json.test.ts b/packages/stack-supabase/__tests__/supabase-v3-json.test.ts index f829f0024..bbf747f57 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-json.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-json.test.ts @@ -12,7 +12,7 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { createMockEncryptionClient, createMockSupabase, diff --git a/packages/stack-supabase/__tests__/supabase-v3-matrix.test.ts b/packages/stack-supabase/__tests__/supabase-v3-matrix.test.ts index b4306a094..6f566dbb0 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-matrix.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-matrix.test.ts @@ -29,7 +29,7 @@ import { V3_MATRIX, } from '@cipherstash/test-kit/catalog' import { describe, expect, it } from 'vitest' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { createMockEncryptionClient, createMockSupabase, diff --git a/packages/stack-supabase/__tests__/supabase-v3-select-star.test.ts b/packages/stack-supabase/__tests__/supabase-v3-select-star.test.ts index 1fd76f220..2d7a09a3b 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-select-star.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-select-star.test.ts @@ -1,7 +1,7 @@ import type { EncryptionClient } from '@cipherstash/stack/encryption' import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' /** * Supabase double that records the select string AND simulates the part of diff --git a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts index 61953f6a7..6b33b7c66 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts @@ -10,7 +10,7 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { createWirePostgrest } from './helpers/postgrest-wire' import { createMockEncryptionClient } from './helpers/supabase-mock' diff --git a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts index 2bd1a5724..9d0717ea5 100644 --- a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts +++ b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts @@ -9,6 +9,7 @@ import { } from '@cipherstash/stack/schema' import { describe, expectTypeOf, it } from 'vitest' import { + type EncryptedQueryBuilder, type EncryptedQueryBuilderV3, type EncryptedSupabaseResponse, encryptedSupabase, @@ -335,21 +336,6 @@ describe('encryptedSupabaseV3 untyped surface (no schemas)', () => { builder.contains('tags', 'vip') }) - it('keeps like/ilike on the v2 builder', () => { - const v2Users = v2EncryptedTable('users', { - email: encryptedColumn('email').freeTextSearch(), - }) - const v2 = encryptedSupabase({ - encryptionClient: {} as never, - supabaseClient, - }) - const builder = v2.from<{ email: string }>('users', v2Users) - builder.like('email', '%ada%') - builder.ilike('email', '%ada%') - // @ts-expect-error — contains is the v3 dialect's method - builder.contains('email', 'ada') - }) - it('supports a no-arg select(), like supabase-js', async () => { const supabase = await encryptedSupabaseV3(supabaseClient) supabase.from('users').select() @@ -445,3 +431,32 @@ describe('withLockContext accepts the plain { identityClaim } form (not only Loc mixedBuilder.withLockContext({ claim: ['sub'] }) }) }) + +// --------------------------------------------------------------------------- +// De-suffixed canonical names (encryptedSupabase / EncryptedQueryBuilder) +// +// `encryptedSupabaseV3` and `EncryptedQueryBuilderV3` remain as type-identical +// `@deprecated` aliases (exercised throughout the tests above); these assert the +// unsuffixed names are the same surface. +// --------------------------------------------------------------------------- + +describe('canonical (unsuffixed) exports', () => { + it('encryptedSupabase is the same factory as encryptedSupabaseV3', () => { + expectTypeOf(encryptedSupabase).toEqualTypeOf(encryptedSupabaseV3) + }) + + it('EncryptedQueryBuilder is the same type as EncryptedQueryBuilderV3', () => { + expectTypeOf>().toEqualTypeOf< + EncryptedQueryBuilderV3 + >() + }) + + it('a typed builder narrows to the documented shape', async () => { + const supabase = await encryptedSupabase(supabaseClient, { + schemas: { users }, + }) + expectTypeOf(supabase.from('users')).toEqualTypeOf< + EncryptedQueryBuilder + >() + }) +}) diff --git a/packages/stack-supabase/__tests__/supabase.test.ts b/packages/stack-supabase/__tests__/supabase.test.ts deleted file mode 100644 index 1023bc387..000000000 --- a/packages/stack-supabase/__tests__/supabase.test.ts +++ /dev/null @@ -1,335 +0,0 @@ -import 'dotenv/config' - -import { Encryption } from '@cipherstash/stack' -import { encryptedColumn, encryptedTable } from '@cipherstash/stack/schema' -import { createClient } from '@supabase/supabase-js' -import postgres from 'postgres' -import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { encryptedSupabase } from '../src/index.js' - -// supabase.test.ts needs a live Supabase project, so the suite is skipped -// when the Supabase environment is not configured (e.g. in CI, pending a -// containerised Supabase setup). It runs locally when SUPABASE_URL, -// SUPABASE_ANON_KEY, and DATABASE_URL are all set. -const SUPABASE_ENABLED = Boolean( - process.env.SUPABASE_URL && - process.env.SUPABASE_ANON_KEY && - process.env.DATABASE_URL, -) - -const supabase = SUPABASE_ENABLED - ? createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!) - : (undefined as unknown as ReturnType) - -const table = encryptedTable('protect-ci', { - encrypted: encryptedColumn('encrypted').freeTextSearch().equality(), - age: encryptedColumn('age').dataType('number').equality(), - // orderAndRange backs the encrypted range test below — the "range filtering - // works on Supabase" claim needs a CI-covered v2 baseline, not just the - // one-off live spike (see the v3 suite's matching range test). - score: encryptedColumn('score').dataType('number').equality().orderAndRange(), -}) - -// Row type for the protect-ci table -type ProtectCiRow = { - id: number - encrypted: string - age: number - score: number - otherField: string - test_run_id: string -} - -// Unique identifier for this test run to isolate data from concurrent test runs -// This is stored in a dedicated test_run_id column to avoid polluting test data -const TEST_RUN_ID = `test-run-${Date.now()}-${Math.random().toString(36).slice(2, 8)}` - -// Track all inserted IDs for cleanup -const insertedIds: number[] = [] - -beforeAll(async () => { - // Idempotent fixture setup. The `protect-ci` table is shared across the - // drizzle + protect/supabase + stack/supabase integration suites; each - // suite's beforeAll runs the same CREATE TABLE so a fresh database is - // ready without manual DBA work. The schema is the union of every - // column those suites read or write. - const sql = postgres(process.env.DATABASE_URL as string, { prepare: false }) - try { - await sql` - CREATE TABLE IF NOT EXISTS "protect-ci" ( - id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, - email eql_v2_encrypted, - age eql_v2_encrypted, - score eql_v2_encrypted, - profile eql_v2_encrypted, - encrypted eql_v2_encrypted, - "otherField" TEXT, - created_at TIMESTAMPTZ DEFAULT NOW(), - test_run_id TEXT - ) - ` - // Backfill any column added after the table was first created on a - // long-lived CI database. CREATE TABLE IF NOT EXISTS is a no-op on - // those, so new columns need an explicit ADD COLUMN IF NOT EXISTS. - await sql` - ALTER TABLE "protect-ci" ADD COLUMN IF NOT EXISTS "otherField" TEXT - ` - // Tell PostgREST to refresh its schema cache so the supabase-js client - // can see a freshly created table without waiting for the polling - // interval. No-op on plain Postgres (no listener bound). - await sql`NOTIFY pgrst, 'reload schema'` - } finally { - await sql.end() - } - - // Clean up any data from this specific test run (safe for concurrent runs) - const { error } = await supabase - .from('protect-ci') - .delete() - .eq('test_run_id', TEST_RUN_ID) - - if (error) { - console.warn(`[protect]: Failed to clean up test data: ${error.message}`) - } -}, 30000) - -afterAll(async () => { - // Clean up all data from this test run - if (insertedIds.length > 0) { - const { error } = await supabase - .from('protect-ci') - .delete() - .in('id', insertedIds) - if (error) { - console.error(`[protect]: Failed to clean up test data: ${error.message}`) - } - } -}, 30000) - -describe.skipIf(!SUPABASE_ENABLED)( - 'supabase (encryptedSupabase wrapper)', - () => { - it('should insert and select encrypted data', async () => { - const protectClient = await Encryption({ schemas: [table] }) - const eSupabase = encryptedSupabase({ - encryptionClient: protectClient, - supabaseClient: supabase, - }) - - const plaintext = 'hello world' - - // Insert — auto-encrypts the `encrypted` column, auto-converts to PG composite - const { data: insertedData, error: insertError } = await eSupabase - .from('protect-ci', table) - .insert({ - encrypted: plaintext, - test_run_id: TEST_RUN_ID, - }) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(insertedData![0].id) - - // Select — auto-adds ::jsonb cast to `encrypted`, auto-decrypts result - const { data, error } = await eSupabase - .from('protect-ci', table) - .select('id, encrypted') - .eq('id', insertedData![0].id) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - expect(data).toHaveLength(1) - expect(data![0].encrypted).toBe(plaintext) - }, 30000) - - it('should insert and select encrypted model data', async () => { - const protectClient = await Encryption({ schemas: [table] }) - const eSupabase = encryptedSupabase({ - encryptionClient: protectClient, - supabaseClient: supabase, - }) - - const model = { - encrypted: 'hello world', - otherField: 'not encrypted', - } - - // Insert — auto-encrypts `encrypted`, passes `otherField` through - const { data: insertedData, error: insertError } = await eSupabase - .from('protect-ci', table) - .insert({ - ...model, - test_run_id: TEST_RUN_ID, - }) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(insertedData![0].id) - - // Select — auto-adds ::jsonb to `encrypted`, auto-decrypts - const { data, error } = await eSupabase - .from('protect-ci', table) - .select('id, encrypted, otherField') - .eq('id', insertedData![0].id) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - expect(data).toHaveLength(1) - expect({ - encrypted: data![0].encrypted, - otherField: data![0].otherField, - }).toEqual(model) - }, 30000) - - it('should insert and select bulk encrypted model data', async () => { - const protectClient = await Encryption({ schemas: [table] }) - const eSupabase = encryptedSupabase({ - encryptionClient: protectClient, - supabaseClient: supabase, - }) - - const models = [ - { - encrypted: 'hello world 1', - otherField: 'not encrypted 1', - }, - { - encrypted: 'hello world 2', - otherField: 'not encrypted 2', - }, - ] - - // Bulk insert — auto-encrypts all models - const { data: insertedData, error: insertError } = await eSupabase - .from('protect-ci', table) - .insert(models.map((m) => ({ ...m, test_run_id: TEST_RUN_ID }))) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(...insertedData!.map((d) => d.id)) - - // Select — auto-decrypts all results - const { data, error } = await eSupabase - .from('protect-ci', table) - .select('id, encrypted, otherField') - .in( - 'id', - insertedData!.map((d) => d.id), - ) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - expect( - data!.map((d) => ({ - encrypted: d.encrypted, - otherField: d.otherField, - })), - ).toEqual(models) - }, 30000) - - it('should insert and query encrypted number data with equality', async () => { - const protectClient = await Encryption({ schemas: [table] }) - const eSupabase = encryptedSupabase({ - encryptionClient: protectClient, - supabaseClient: supabase, - }) - - const testAge = 25 - const model = { - age: testAge, - otherField: 'not encrypted', - } - - // Insert — auto-encrypts `age` - const { data: insertedData, error: insertError } = await eSupabase - .from('protect-ci', table) - .insert({ - ...model, - test_run_id: TEST_RUN_ID, - }) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(insertedData![0].id) - - // Query by encrypted `age` — auto-encrypts the search term - const { data, error } = await eSupabase - .from('protect-ci', table) - .select('id, age, otherField') - .eq('age', testAge) - .eq('test_run_id', TEST_RUN_ID) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - // Verify we found our specific row with encrypted age match - expect(data).toHaveLength(1) - expect(data![0].age).toBe(testAge) - }, 30000) - - // Encrypted RANGE filtering on Supabase previously had no CI coverage — - // the custom ORE operator's resolution over PostgREST rested on a one-off - // live spike. This is the v2 baseline the v3 suite's range test mirrors. - // Range needs real ORE terms (live ZeroKMS), same gating as the suite. - // Note: range FILTERING only — ORDER BY on an encrypted column is - // unsupported on Supabase (operator families need superuser). - it('should filter encrypted number data with a range (gte/lte)', async () => { - const protectClient = await Encryption({ schemas: [table] }) - const eSupabase = encryptedSupabase({ - encryptionClient: protectClient, - supabaseClient: supabase, - }) - - const models = [ - { score: 10, otherField: 'low' }, - { score: 25, otherField: 'mid' }, - { score: 90, otherField: 'high' }, - ] - - const { data: insertedData, error: insertError } = await eSupabase - .from('protect-ci', table) - .insert(models.map((m) => ({ ...m, test_run_id: TEST_RUN_ID }))) - .select('id') - - if (insertError) { - throw new Error(`[protect]: ${insertError.message}`) - } - - insertedIds.push(...insertedData!.map((d) => d.id)) - - const { data, error } = await eSupabase - .from('protect-ci', table) - .select('id, score, otherField') - .gte('score', 20) - .lte('score', 30) - .eq('test_run_id', TEST_RUN_ID) - - if (error) { - throw new Error(`[protect]: ${error.message}`) - } - - expect(data).toHaveLength(1) - expect(data![0].score).toBe(25) - expect(data![0].otherField).toBe('mid') - }, 30000) - }, -) diff --git a/packages/stack-supabase/integration/wire.integration.test.ts b/packages/stack-supabase/integration/wire.integration.test.ts index 808f69412..cd10218a0 100644 --- a/packages/stack-supabase/integration/wire.integration.test.ts +++ b/packages/stack-supabase/integration/wire.integration.test.ts @@ -33,7 +33,7 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { databaseUrl } from '@cipherstash/test-kit' import postgres from 'postgres' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { makePostgrestClient, reloadSchemaCache } from './helpers/pgrest' import { narrowedQueryTerm, storageEnvelope } from './helpers/v3-envelope' From e0dea47633be8f1711946f586ad403aeed406e02 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:52:05 +1000 Subject: [PATCH 010/123] docs(stack-supabase): update stash-supabase skill + changesets for v2 removal --- .../remove-eql-v2-supabase-authoring.md | 39 +++++ .changeset/remove-eql-v2-supabase-skill.md | 11 ++ skills/stash-supabase/SKILL.md | 142 ++++++++---------- 3 files changed, 110 insertions(+), 82 deletions(-) create mode 100644 .changeset/remove-eql-v2-supabase-authoring.md create mode 100644 .changeset/remove-eql-v2-supabase-skill.md diff --git a/.changeset/remove-eql-v2-supabase-authoring.md b/.changeset/remove-eql-v2-supabase-authoring.md new file mode 100644 index 000000000..50b5d6335 --- /dev/null +++ b/.changeset/remove-eql-v2-supabase-authoring.md @@ -0,0 +1,39 @@ +--- +'@cipherstash/stack-supabase': major +--- + +Remove the EQL v2 authoring surface and de-suffix the v3 API to the canonical +unsuffixed names (part of the EQL v2 removal, #707). + +- **`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory** + (formerly `encryptedSupabaseV3`). `encryptedSupabaseV3` remains as a + type-identical `@deprecated` alias, so existing imports keep working. +- **The legacy v2 `encryptedSupabase({ encryptionClient, supabaseClient })` + wrapper is removed** — with it the two-argument `from(tableName, schema)` form + and the hand-written client-side v2 schema. Its `EncryptedSupabaseConfig` and + the v2 `EncryptedSupabaseInstance`/`EncryptedQueryBuilder` type shapes are gone; + the unsuffixed type names now denote the v3 surface. +- **The `*V3` type exports are de-suffixed** to their canonical names — + `EncryptedSupabaseV3Options` → `EncryptedSupabaseOptions`, + `EncryptedSupabaseV3Instance` → `EncryptedSupabaseInstance`, + `TypedEncryptedSupabaseV3Instance` → `TypedEncryptedSupabaseInstance`, + `EncryptedQueryBuilderV3` → `EncryptedQueryBuilder`, + `EncryptedQueryBuilderV3Untyped` → `EncryptedQueryBuilderUntyped`, + `V3FilterableKeys` → `FilterableKeys`, `V3OrderableKeys` → `OrderableKeys`, and + the rest of the `*V3` key-helper types. Each keeps a type-identical + `@deprecated` `*V3` alias. + +**Not affected: reading existing data.** Only the v2 *authoring/emission* surface +is removed. Decryption in `@cipherstash/stack` is generation-agnostic, so rows +written as EQL v2 payloads still decrypt through the wrapper's read path. + +Internally the v3 query builder (`query-builder-v3.ts`) was folded into the base +`EncryptedQueryBuilderImpl`, which is now natively EQL v3; no runtime behaviour or +wire encoding changed. + +**Migration:** rename `encryptedSupabaseV3` → `encryptedSupabase` (or keep using +the alias). If you still use the v2 `encryptedSupabase({ encryptionClient, +supabaseClient }).from(table, schema)` wrapper, migrate the table to an +`eql_v3_*` column domain and switch to the introspecting factory — +`await encryptedSupabase(supabaseUrl, supabaseKey)` — see the `stash-supabase` +skill and https://cipherstash.com/docs. diff --git a/.changeset/remove-eql-v2-supabase-skill.md b/.changeset/remove-eql-v2-supabase-skill.md new file mode 100644 index 000000000..c4c33dedd --- /dev/null +++ b/.changeset/remove-eql-v2-supabase-skill.md @@ -0,0 +1,11 @@ +--- +'stash': patch +--- + +Update the bundled `stash-supabase` agent skill for the EQL v2 removal (#707): +`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory (with +`encryptedSupabaseV3` kept as a `@deprecated` alias), and the legacy v2 +`encryptedSupabase({ encryptionClient, supabaseClient })` authoring wrapper has +been removed. The skill's examples, exported-type list, and migration/cutover +guidance are corrected accordingly. Skills ship inside the `stash` tarball, so +the stale v2 guidance would otherwise land in a user's project. diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 57f7099da..fdbb2d62b 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -1,18 +1,21 @@ --- name: stash-supabase -description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabaseV3 wrapper over native EQL v3 column domains, transparent encryption/decryption on insert/update/select, encrypted scalar filters (eq, gt/gte/lt/lte, in, or), ordering on encrypted columns, EQL 3.0.2 PostgREST query-domain limitations, identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. +description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabase wrapper over native EQL v3 column domains, transparent encryption/decryption on insert/update/select, encrypted scalar filters (eq, gt/gte/lt/lte, in, or), ordering on encrypted columns, EQL 3.0.2 PostgREST query-domain limitations, identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. --- # CipherStash Stack - Supabase Integration Guide for integrating CipherStash field-level encryption with Supabase using -the `encryptedSupabaseV3` wrapper over native EQL v3 column domains. The +the `encryptedSupabase` wrapper over native EQL v3 column domains. The wrapper provides transparent encryption on mutations and decryption on selects, with support for equality, range, and ordering. -A legacy EQL v2 wrapper (`encryptedSupabase`) still ships for existing -deployments — see "Legacy: EQL v2" at the end. New projects should use -`encryptedSupabaseV3`. +> **Naming note.** `encryptedSupabase` is the current EQL v3 factory. +> `encryptedSupabaseV3` remains as a `@deprecated`, type-identical alias, so +> existing imports keep working — prefer `encryptedSupabase` in new code. The +> old EQL v2 authoring wrapper (`encryptedSupabase({ encryptionClient, +> supabaseClient }).from(table, schema)`) has been **removed** — see +> "Legacy: EQL v2" at the end. ## When to Use This Skill @@ -134,17 +137,17 @@ SQL-standard type names (`integer`, `smallint`, `real`, `double`, `boolean`, ### 3. Initialize the wrapper ```typescript -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" // Introspects the database via options.databaseUrl or DATABASE_URL -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) -// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) +const es = await encryptedSupabase(supabaseUrl, supabaseKey) +// or wrap an existing client: await encryptedSupabase(supabaseClient, options) await es.from("users").insert({ email: "a@b.com", amount: 30 }) await es.from("users").select("id, email, amount").eq("email", "a@b.com") ``` -`encryptedSupabaseV3` **introspects the database at connect time**: it +`encryptedSupabase` **introspects the database at connect time**: it detects EQL v3 columns by their Postgres domain, derives each column's encryption config from the domain, and builds the encryption client internally — there is no client-side schema to hand-maintain. Introspection @@ -165,7 +168,7 @@ tables against the database at construction: ```typescript import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" const users = encryptedTable("users", { email: types.TextSearch("email"), // public.eql_v3_text_search — eq + range + free-text @@ -173,7 +176,7 @@ const users = encryptedTable("users", { joined: types.TimestampOrd("joined_at") // public.eql_v3_timestamp_ord — eq + range, decrypts to Date }) -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { +const es = await encryptedSupabase(supabaseUrl, supabaseKey, { schemas: { users }, }) @@ -443,7 +446,7 @@ third-party OIDC JWT (Clerk, Supabase, Auth0, ...) with ```typescript import { OidcFederationStrategy } from "@cipherstash/stack" -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" const strategy = OidcFederationStrategy.create( process.env.CS_WORKSPACE_CRN!, @@ -451,7 +454,7 @@ const strategy = OidcFederationStrategy.create( ) if (strategy.failure) throw new Error(strategy.failure.error.message) -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { +const es = await encryptedSupabase(supabaseUrl, supabaseKey, { config: { authStrategy: strategy.data }, }) ``` @@ -502,7 +505,7 @@ const { data, error } = await es ```typescript import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" // Optional declared schema — compile-time types. Introspection alone // (no `schemas`) also works. @@ -511,7 +514,7 @@ const users = encryptedTable("users", { amount: types.IntegerOrd("amount"), }) -const es = await encryptedSupabaseV3( +const es = await encryptedSupabase( process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!, { schemas: { users } }, // databaseUrl defaults to DATABASE_URL @@ -581,9 +584,15 @@ at all — only `.is(column, null)`. `@cipherstash/stack-supabase` also exports the following types: -- `EncryptedSupabaseV3Options`, `EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, `EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3Schemas` -- `SupabaseClientLike` -- `EncryptedSupabaseConfig`, `EncryptedSupabaseInstance`, `EncryptedQueryBuilder`, `PendingOrCondition` (legacy EQL v2) +- `EncryptedSupabaseOptions`, `EncryptedSupabaseInstance`, `TypedEncryptedSupabaseInstance`, `EncryptedQueryBuilder`, `EncryptedQueryBuilderUntyped`, `EncryptedQueryBuilderCore`, `V3Schemas` +- `FilterableKeys`, `FreeTextSearchableKeys` +- `EncryptedSupabaseResponse`, `EncryptedSupabaseError`, `PendingOrCondition`, `SupabaseClientLike` + +Each `*V3`-suffixed name from earlier releases (`EncryptedSupabaseV3Options`, +`EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, +`EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3FilterableKeys`, +`V3FreeTextSearchableKeys`) is still exported as a `@deprecated`, type-identical +alias of its unsuffixed counterpart above. ## Migrating an Existing Column to Encrypted @@ -636,7 +645,7 @@ ALTER TABLE users Apply with `supabase db reset` locally or `supabase migration up` against the remote project. -No client-side schema change is required — `encryptedSupabaseV3` introspects +No client-side schema change is required — `encryptedSupabase` introspects the new column's domain at the next client startup. If you use declared `schemas`, add the column so it is typed: @@ -665,7 +674,7 @@ Find **every** code path that writes to `users.email` and update it to also writ ```typescript // src/db/users.ts -import { es } from './clients' // encryptedSupabaseV3 instance +import { es } from './clients' // encryptedSupabase instance export async function insertUser(email: string) { return es.from('users').insert({ @@ -714,60 +723,20 @@ If something goes wrong (e.g. you discover the dual-write code wasn't actually l **EQL v3 (the schema above): there is no cut-over.** The encrypted column keeps its own name — point your application at `email_encrypted` through the -`encryptedSupabaseV3` wrapper, deploy, verify reads decrypt correctly, then skip +`encryptedSupabase` wrapper, deploy, verify reads decrypt correctly, then skip ahead to the drop step. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). -The rest of this subsection is the **EQL v2** path (an `eql_v2_encrypted` twin -queried through the legacy `encryptedSupabase` wrapper), kept for existing v2 -deployments. - -First, if you use declared `schemas`, update them to the post-cutover shape — the encrypted column will live under the original column name: - -```typescript -// src/encryption/schema.ts (post-cutover) -export const users = encryptedTable('users', { - email: types.TextSearch('email'), -}) -``` - -(Without declared schemas, introspection picks up the renamed column at the next client startup.) - -> **Known gap (EQL v2, SDK-only users):** `stash encrypt cutover` requires a pending EQL configuration, which is set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. EQL v3 columns never hit this — cut-over doesn't apply to them. -> -> **Using CipherStash Proxy?** Re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): -> -> ```bash -> stash db push -> # → writes the new config as `pending`. Active config (still pointing at -> # `email_encrypted`) keeps serving while we complete the cutover. -> ``` - -Now run the cutover: - -```bash -stash encrypt cutover --table users --column email -``` - -Inside one transaction it: (1) renames `email` → `email_plaintext` and `email_encrypted` → `email`, (2) promotes the pending EQL config to `active` (and the prior active to `inactive`), (3) records a `cut_over` event in `cs_migrations`. - -App code that does `select('email')` now returns ciphertext that must be decrypted via the `encryptedSupabaseV3` wrapper. **This is the moment that breaks read paths if they aren't going through the wrapper.** - -Update read paths to use the wrapper: - -```typescript -// Before -const { data } = await supabase.from('users').select('email').eq('id', id).single() - -// After — the wrapper decrypts transparently -const { data } = await es.from('users').select('email').eq('id', id).single() -``` - -For supported scalar queries that filter on `email`, the wrapper handles the -encrypted operators internally — calls such as `.eq()` and `.gte()` keep the -same shape, but values are encrypted before reaching the database. See -`## Query Filters` above for the EQL 3.0.2 PostgREST limitations. +> **EQL v2 cutover (`stash encrypt cutover`) via this SDK is no longer +> supported.** `@cipherstash/stack-supabase` authors and queries EQL v3 only — +> the v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper that +> read the post-cutover `eql_v2_encrypted` column has been removed. The CLI +> lifecycle commands still auto-detect a v2 column, but the SDK read path for one +> is gone. If you have an in-flight v2 cutover, either pin the last +> `@cipherstash/stack-supabase` release that shipped the v2 wrapper to finish it, +> or (recommended) create an `eql_v3_*` twin and run the v3 rollout above. New +> encryption should always target an `eql_v3_*` domain. #### Drop: remove the plaintext column @@ -796,15 +765,24 @@ All three are read-only. ## Legacy: EQL v2 -Earlier versions of this integration stored ciphertext in `jsonb` / -composite `eql_v2_encrypted` columns (enabled via `CREATE EXTENSION eql_v2` -or the v2 EQL bundle) and queried them through the `encryptedSupabase({ -supabaseClient, encryptionClient })` factory, which takes a hand-written -client-side schema and a two-argument `from(tableName, schema)`. That surface -still ships in `@cipherstash/stack-supabase` and is unchanged — keep using it -for existing v2 deployments — but it is not the recommended path for new -projects: use `encryptedSupabaseV3`. The CLI rollout tooling (`stash encrypt -backfill` / `cutover` / `drop`) supports both generations and auto-detects which -one a column uses, so a v2 twin is no longer needed to get CLI-managed -backfill — see the EQL version note in the migration section above. For the v2 -wrapper's full API and semantics, see the docs at https://cipherstash.com/docs. +Earlier versions of this integration stored ciphertext in composite +`eql_v2_encrypted` columns (enabled via `CREATE EXTENSION eql_v2` or the v2 EQL +bundle) and both wrote and read them through the `encryptedSupabase({ +supabaseClient, encryptionClient })` factory — a hand-written client-side schema +and a two-argument `from(tableName, schema)`. + +**That v2 wrapper has been removed.** `@cipherstash/stack-supabase` now authors +and queries EQL v3 only, via the introspecting `encryptedSupabase(url, key)` / +`encryptedSupabase(client, options)` factory described above. There is no longer +a code path in this package that emits or reads `eql_v2_encrypted` columns. + +Existing v2 deployments have two options: + +- **Migrate to EQL v3** (recommended): add an `eql_v3_*` twin column and run the + rollout in "Migrating an Existing Column to Encrypted" above. +- **Stay on v2 for now:** pin the last `@cipherstash/stack-supabase` release that + shipped the v2 wrapper. The CLI rollout tooling (`stash encrypt backfill` / + `cutover` / `drop`) still auto-detects a column's EQL generation. + +For the removed v2 wrapper's historical API and semantics, see the docs at +https://cipherstash.com/docs. From 8b69a34ad9d2d9ba106145dc8916fd62a52fa98d Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 11:26:16 +1000 Subject: [PATCH 011/123] docs(stack-supabase): clarify v2 decrypt path in changeset v2 read via this adapter is intentionally removed; v2 ciphertext still decrypts through the core @cipherstash/stack client. Mixed-generation handling is customer-side (install both), per #707 out-of-scope stance. --- .changeset/remove-eql-v2-supabase-authoring.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.changeset/remove-eql-v2-supabase-authoring.md b/.changeset/remove-eql-v2-supabase-authoring.md index 50b5d6335..ad783dc52 100644 --- a/.changeset/remove-eql-v2-supabase-authoring.md +++ b/.changeset/remove-eql-v2-supabase-authoring.md @@ -23,9 +23,15 @@ unsuffixed names (part of the EQL v2 removal, #707). the rest of the `*V3` key-helper types. Each keeps a type-identical `@deprecated` `*V3` alias. -**Not affected: reading existing data.** Only the v2 *authoring/emission* surface -is removed. Decryption in `@cipherstash/stack` is generation-agnostic, so rows -written as EQL v2 payloads still decrypt through the wrapper's read path. +**Reading existing v2 data.** Only the v2 *authoring/emission* surface is removed +— no v2 ciphertext is stranded. Decryption in `@cipherstash/stack` is +generation-agnostic, so EQL v2 payloads still decrypt through the core client +(`decrypt` / `decryptModel`). This adapter, however, is now EQL v3 only and will +not auto-read an `eql_v2_encrypted` column: to read legacy v2 data during +migration, decrypt fetched rows with `@cipherstash/stack` directly, or run a +v2-configured setup alongside the v3 one and route per column. Mixed-generation +handling is a customer-side concern (install both and handle it explicitly), not +adapter auto-detection. Internally the v3 query builder (`query-builder-v3.ts`) was folded into the base `EncryptedQueryBuilderImpl`, which is now natively EQL v3; no runtime behaviour or From a3d6abb8c19cfc1a5bf156b4573f23ff540f606f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 11:58:31 +1000 Subject: [PATCH 012/123] refactor(stack-supabase): split query-builder monolith, ratchet FTA cap 91 -> 71 The EQL v2 removal folded `query-builder-v3.ts` into `query-builder.ts` (2b4e2e92). FTA penalises size superlinearly, so two files that each passed the complexity gate became one that did not: 90.88 + 73.95 -> 105.77, against a cap of 91. That failed the "Analyze v3 complexity" check on #769. `fta-v3.yml` is explicit that the cap is "a ratchet, not an aspiration", so decompose rather than raise it. The 2331-line class becomes a pipeline: column-map.ts ColumnMap - name + capability resolution, the concern every stage needed (was six correlated fields) query-encrypt.ts filter-operand terms: collect, validate, batch-encrypt query-mutation.ts row-data encryption for insert/update/upsert query-dbspace.ts property-space -> DB-space, once query-filters.ts operand substitution onto the PostgREST query query-results.ts decryption + Date reconstruction query-builder.ts the class: recorded state, fluent surface, execute() Worst file 105.77 -> 70.12 (query-encrypt.ts); cap lowered 91 -> 71. Also removed, all no-ops from the pre-fold v2/v3 inheritance: - `notFilterOperator` - an identity function with an unused parameter - `applyPatternFilter`'s dead `_wasEncrypted` parameter - `protected` throughout, now `private`; nothing extends the class and collapsed the six-times-repeated withLockContext/audit/await dance into one `withOpContext` helper - a skipped lock context would encrypt under the wrong data key, so the three steps must stay identical. `EncryptedQueryBuilderImpl` keeps its name, export site and constructor; nothing here is part of the package's public surface (index.ts does not re-export it). 466 tests pass unchanged. One test reached the unsupported- queryType backstop by subclassing to override `queryTypeForRawOp`, which is now a module function; `assertTermQueryable` is exported instead and the test calls it directly, dropping a subclass its own comment called "breaking the internal contract". Docs: correct comments asserting adapter-side EQL v2 reads. The adapter is v3 only and does not auto-read `eql_v2_encrypted` columns - introspection matches `public.eql_v3_*` domains exclusively, so such a column never enters the encrypt config. No ciphertext is stranded: core decryption stays generation-agnostic. Also retires the two-dialect scaffolding in types.ts, whose header claimed v3 does free-text via `contains` - contradicting the typed surface forty lines above it and re-introducing the exact confusion #617 removed. `EncryptedQueryBuilderCore`'s OK/BK defaults are kept: they are live for the untyped surface, only their v2-era rationale was wrong. --- .github/workflows/fta-v3.yml | 17 +- .../__tests__/supabase-v3-builder.test.ts | 54 +- packages/stack-supabase/package.json | 2 +- packages/stack-supabase/src/column-map.ts | 218 +++ packages/stack-supabase/src/helpers.ts | 2 +- packages/stack-supabase/src/index.ts | 13 +- packages/stack-supabase/src/query-builder.ts | 1716 ++--------------- packages/stack-supabase/src/query-dbspace.ts | 141 ++ packages/stack-supabase/src/query-encrypt.ts | 659 +++++++ packages/stack-supabase/src/query-filters.ts | 388 ++++ packages/stack-supabase/src/query-mutation.ts | 79 + packages/stack-supabase/src/query-results.ts | 205 ++ packages/stack-supabase/src/types.ts | 77 +- 13 files changed, 1915 insertions(+), 1656 deletions(-) create mode 100644 packages/stack-supabase/src/column-map.ts create mode 100644 packages/stack-supabase/src/query-dbspace.ts create mode 100644 packages/stack-supabase/src/query-encrypt.ts create mode 100644 packages/stack-supabase/src/query-filters.ts create mode 100644 packages/stack-supabase/src/query-mutation.ts create mode 100644 packages/stack-supabase/src/query-results.ts diff --git a/.github/workflows/fta-v3.yml b/.github/workflows/fta-v3.yml index 80e494fec..dd3108b43 100644 --- a/.github/workflows/fta-v3.yml +++ b/.github/workflows/fta-v3.yml @@ -68,10 +68,19 @@ jobs: # blocking gate. No `continue-on-error`. # One step per package so a failure names the offending package. Each caps # at its current worst file (a ratchet, not an aspiration): stack src/eql/v3 - # at 69, and the split adapter packages at their monolith maxima — drizzle - # 89 (operators.ts), supabase 91 (query-builder.ts). Lower a cap whenever a - # file is refactored below the next threshold; the v2 query-builder/operators - # monoliths are the debt to whittle down toward stack's 69. + # at 69, drizzle 89 (operators.ts, still a monolith), and supabase 71 + # (query-encrypt.ts at 70.12, after the query-builder monolith was split + # across column-map / query-encrypt / query-mutation / query-dbspace / + # query-filters / query-results). Lower a cap whenever a file is refactored + # below the next threshold; drizzle's operators.ts is the remaining debt to + # whittle down toward stack's 69. + # + # NB supabase's top three now cluster tightly (query-encrypt 70.12, + # query-builder 70.05, helpers 69.13), so 71 is a ~0.9 margin. A cap set + # flush against its max is fragile — the 91 this replaced was 0.12 above + # its max and a single refactor blew straight through it. If a formatting + # reflow trips this step without a real complexity increase, re-measure + # (`npx fta src --format table`) before assuming the code got worse. - name: Analyze stack (eql/v3) complexity run: pnpm --filter @cipherstash/stack run analyze:complexity diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index 92e4a6649..60d26c8f8 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -1,6 +1,8 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it, vi } from 'vitest' +import { ColumnMap } from '../src/column-map' import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' +import { assertTermQueryable } from '../src/query-encrypt' import { createMockEncryptionClient, createMockSupabase, @@ -1475,37 +1477,33 @@ describe('v3 raw filter() resolves the query type from the operator', () => { ]) }) - // `encryptCollectedTerms` rejects any queryType outside the four supported EQL + // `assertTermQueryable` rejects any queryType outside the four supported EQL // v3 kinds. No public call path can produce a fifth — `mapFilterOpToQueryType`, // `queryTypeForRawOp` and `queryTypeForOrOp` are exhaustive — so this backstop - // is unreachable without breaking the internal contract, which is exactly what - // the subclass below does. Keep the guard: it is what a future producer - // gaining a new QueryTypeName would trip over. (`searchableJson` used to be - // the exemplar here until #650 made it a real, supported kind.) - it('rejects a query type outside the supported EQL v3 kinds', async () => { - const supabase = createMockSupabase() + // is only reachable by handing the resolver a term the internal contract says + // cannot exist. Keep the guard: it is what a future producer gaining a new + // QueryTypeName would trip over. (`searchableJson` used to be the exemplar + // here until #650 made it a real, supported kind.) + it('rejects a query type outside the supported EQL v3 kinds', () => { + const term = { + value: 'a@b.com', + column: users.columnBuilders.email, + table: users, + queryType: 'steVecSelector', + returnType: 'composite-literal', + } as unknown as Parameters[0] - class BogusQueryType extends EncryptedQueryBuilderV3Impl { - protected override queryTypeForRawOp(_operator: string) { - return 'steVecSelector' as never - } - } - - const builder = new BogusQueryType( - 'users', - users, - createMockEncryptionClient(), - supabase.client, - USERS_ALL_COLUMNS, - ) - - const { error, status } = await builder - .select('id') - .filter('email', 'eq', 'a@b.com') - - expect(status).toBe(500) - expect(error?.message).toContain('query type "steVecSelector"') - expect(error?.message).toContain('not supported on EQL v3 columns') + expect(() => + assertTermQueryable(term, { + tableName: 'users', + table: users, + encryptionClient: createMockEncryptionClient(), + lockContext: null, + auditConfig: null, + columns: new ColumnMap('users', users, USERS_ALL_COLUMNS), + queryDomainsRequired: false, + }), + ).toThrow(/query type "steVecSelector".*not supported on EQL v3 columns/s) }) }) diff --git a/packages/stack-supabase/package.json b/packages/stack-supabase/package.json index 4b0a5bc46..dbd36fe63 100644 --- a/packages/stack-supabase/package.json +++ b/packages/stack-supabase/package.json @@ -51,7 +51,7 @@ "test": "vitest run", "test:types": "vitest --run --typecheck.only", "test:integration": "vitest run --config integration/vitest.config.ts", - "analyze:complexity": "fta src --score-cap 91" + "analyze:complexity": "fta src --score-cap 71" }, "dependencies": { "@cipherstash/stack": "workspace:*" diff --git a/packages/stack-supabase/src/column-map.ts b/packages/stack-supabase/src/column-map.ts new file mode 100644 index 000000000..8920d87d1 --- /dev/null +++ b/packages/stack-supabase/src/column-map.ts @@ -0,0 +1,218 @@ +import { EncryptedV3Column } from '@cipherstash/stack/adapter-kit' +import type { AnyV3Table } from '@cipherstash/stack/eql/v3' +import type { ColumnSchema } from '@cipherstash/stack/schema' +import type { BuildableQueryColumn } from '@cipherstash/stack/types' +import type { DbName } from './types' + +/** + * The subset of a v3 column builder the dialect relies on. Structural rather + * than the concrete class union so the runtime `instanceof EncryptedV3Column` + * gate and this type stay independent. + */ +export type V3ColumnLike = { + getName(): string + getEqlType(): string + getQueryCapabilities(): { + equality: boolean + orderAndRange: boolean + freeTextSearch: boolean + /** Optional: only `public.eql_v3_json_search` (`types.Json`) carries it. */ + searchableJson?: boolean + } + build(): ColumnSchema +} + +/** + * Reject a declared property name that is also a DIFFERENT physical column. + * + * `select('*')` expands the introspected DB names into property names, so a + * column renamed `created_at → createdAt` and a distinct plaintext column + * literally named `createdAt` both emit the token `createdAt`, which + * `addJsonbCastsV3` turns into `createdAt:created_at::jsonb` — twice. PostgREST + * returns the encrypted column under that key and the plaintext one is never + * selected, silently yielding the wrong value for a field the row type + * guarantees. + * + * Nothing downstream can disambiguate the two, and `EncryptedTable.build()`'s + * duplicate check only fires when two BUILDERS share a `getName()`. Refuse to + * construct instead. + */ +function assertNoPropertyDbNameCollision( + tableName: string, + propToDb: Record, + allColumns: string[] | null, +): void { + if (!allColumns) return + const dbNames = new Set(allColumns) + + for (const [property, dbName] of Object.entries(propToDb)) { + if (property === dbName) continue + if (!dbNames.has(property)) continue + throw new Error( + `[supabase v3]: property "${property}" on table "${tableName}" renames DB column "${dbName}", but "${property}" is also a distinct column in the database — the two collide in select('*'). Rename the property, or drop the declared rename.`, + ) + } +} + +/** + * Name and capability resolution for one table's columns. + * + * Every stage of the query pipeline addresses columns by BOTH the JS property + * name a caller wrote and the DB name PostgREST must see, and each needs to ask + * whether a given name is an encrypted v3 column and what it can be queried by. + * Concentrating that here means the encrypt, DB-space, filter-apply and decrypt + * modules take one collaborator instead of six correlated fields that must be + * kept in lockstep. + */ +export class ColumnMap { + /** JS property name → DB column name, for every encrypted column. */ + readonly propToDb: Record + /** Built column schemas keyed by DB column name (for `cast_as`, `indexes`). */ + readonly columnSchemas: Record + /** Every name an encrypted column answers to — property AND DB spelling. + * Filters and select strings address columns by both, so recognition must + * cover both. */ + readonly encryptedColumnNames: string[] + + /** DB column name → JS property name — the inverse of {@link propToDb}, used + * to expand `select('*')` back into property names. Null prototype: a DB + * column literally named `constructor` / `toString` would otherwise resolve + * to an inherited `Object.prototype` member and be emitted as a select token. */ + private readonly dbToProp: Record + /** Column builders keyed by BOTH property name and DB name. */ + private readonly v3Columns: Record + + constructor( + tableName: string, + table: AnyV3Table, + allColumns: string[] | null, + ) { + this.propToDb = table.buildColumnKeyMap() + this.columnSchemas = table.build().columns + + this.dbToProp = Object.create(null) as Record + for (const [property, dbName] of Object.entries(this.propToDb)) { + this.dbToProp[dbName] = property + } + + assertNoPropertyDbNameCollision(tableName, this.propToDb, allColumns) + + // Null-prototype: keyed by DB column names, and `validateTransforms` reads + // it without an own-key guard — an inherited `constructor`/`toString` would + // otherwise resolve truthy for a plaintext column of that name. + this.v3Columns = Object.create(null) as Record + for (const [property, builder] of Object.entries(table.columnBuilders)) { + if (builder instanceof EncryptedV3Column) { + const col = builder as unknown as V3ColumnLike + this.v3Columns[property] = col + this.v3Columns[col.getName()] = col + } + } + + this.encryptedColumnNames = Object.keys(this.v3Columns) + } + + /** Resolve a JS property name to its DB column name. `Object.hasOwn` guards + * the inherited-member hazard described on {@link EncryptedTable.buildColumnKeyMap}. */ + dbNameFor(name: string): string { + return Object.hasOwn(this.propToDb, name) ? this.propToDb[name] : name + } + + /** + * Map a filter's column name to the DB column name PostgREST must see — + * resolving a JS property name to its DB name. + * + * This is the ONLY place a {@link DbName} is minted. The + * {@link SupabaseQueryBuilder} seam accepts nothing else, so every column + * name reaching PostgREST must pass through here. + */ + filterColumnName(column: string): DbName { + return this.dbNameFor(column) as DbName + } + + /** + * Encrypted ordering columns sort by their `op` term, not by the envelope. + * + * `order=col->op` is the one ordering expression PostgREST can emit that + * reaches the OPE term. It must NOT leak into filters — those compare whole + * envelopes through the `eql_v3.*` operators — which is why this is its own + * seam rather than a change to {@link filterColumnName}. + * + * The canonical EQL form is `ORDER BY eql_v3.ord_term(col)`, which returns + * `eql_v3_internal.ope_cllw` — a domain over `bytea`, ordered by the native + * btree. PostgREST cannot call a function, so it orders the `op` term where it + * sits, inside the envelope. The two agree because the term is what + * `ord_term()` returns. + * + * `->` (jsonb) rather than `->>` (text) keeps the comparison on the typed + * value. Note this does NOT avoid the database collation: Postgres compares + * jsonb strings with `varstr_cmp` under the default collation, exactly as it + * does text. What makes the ordering collation-independent is the term itself + * — fixed-width lowercase hex (`[0-9a-f]`, 130 chars for `integer_ord`, 82 for + * `text_search`) — and every collation orders digits before letters and hex + * letters among themselves. `match-bloom`'s sibling assertion pins that shape. + * + * This runs at column-name-mapping time (`transformToDbSpace`), BEFORE + * `buildAndExecuteQuery` calls `validateTransforms`. For an encrypted column + * with no `ope` index it therefore returns a bare `dbName` here — a name that + * would sort by `jsonb_cmp` over the ciphertext if it reached PostgREST — but + * it never does: `validateTransforms` throws (with a domain-specific reason) + * before the query executes, so the bare name is only ever an intermediate + * value on a request that is about to be rejected. + */ + orderColumnName(column: string): DbName { + const dbName = this.dbNameFor(column) + const encrypted = this.v3Columns[column] + if (!encrypted) return dbName as DbName + + return ( + this.columnSchemas[dbName]?.indexes?.ope ? `${dbName}->op` : dbName + ) as DbName + } + + /** + * Expand the introspected column list (DB names) into JS property names. + * + * Load-bearing for `select('*')` on a DECLARED table that renames a column. + * `addJsonbCastsV3` only emits the `prop:db_name::jsonb` alias — the thing + * that makes PostgREST return the column under its property name — when the + * token it sees is a property name. Feeding it the raw DB name instead takes + * the unaliased `dbNames.has(...)` branch, so the row comes back keyed + * `created_at` while the declared row type promises `createdAt`, silently + * yielding `undefined` for a field TypeScript guarantees. + * + * A DB column with no encrypted builder (plaintext passthrough, and every + * synthesized column, where property == DB name) maps to itself. + */ + expandAllColumns(columns: string[]): string[] { + return columns.map((dbName) => + Object.hasOwn(this.dbToProp, dbName) ? this.dbToProp[dbName] : dbName, + ) + } + + /** True when `column` is one of this table's encrypted v3 columns. */ + isEncryptedV3Column(column: string): boolean { + return Boolean(this.v3Columns[column]) + } + + /** True when `column` is an encrypted `types.Json` document column. */ + isSearchableJsonColumn(column: string): boolean { + const builder: V3ColumnLike | undefined = this.v3Columns[column] + return Boolean(builder?.getQueryCapabilities().searchableJson) + } + + /** The encrypted builder for `column`, by either spelling. */ + encryptedColumn(column: string): V3ColumnLike | undefined { + return this.v3Columns[column] + } + + /** The built schema for a DB column name — `cast_as` and `indexes`. */ + schemaFor(dbName: string): ColumnSchema | undefined { + return this.columnSchemas[dbName] + } + + /** The encrypted builders as the term collector's column lookup. */ + queryColumnMap(): Record { + return this.v3Columns as unknown as Record + } +} diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index 7d2ae5fc6..f8e929cee 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -109,7 +109,7 @@ function lookupDbName( * property name. * - A DB column name used directly is cast in place (`db_name::jsonb`). * - Tokens that already carry a cast, or contain parens/dots (functions, - * foreign tables), are left untouched — same rules as the v2 helper. + * foreign tables), are left untouched. */ export function addJsonbCastsV3( columns: string, diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index b295a587a..4befe9726 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -48,10 +48,15 @@ function assertTableIsModelled( * compile-time types and verifies the declared tables against the database at * construction. * - * Encrypted data is stored as EQL v3 payloads. The generation-agnostic decrypt - * path in `@cipherstash/stack` still reads existing EQL v2 payloads, but this - * wrapper only AUTHORS EQL v3 — the legacy v2 authoring surface (a hand-written - * client-side schema and `from(tableName, schema)`) has been removed. + * Encrypted data is stored as EQL v3 payloads. This wrapper is EQL v3 only — it + * both authors and reads v3, and the legacy v2 authoring surface (a hand-written + * client-side schema and `from(tableName, schema)`) has been removed. It does + * not auto-read an `eql_v2_encrypted` column: introspection recognises the + * `public.eql_v3_*` domains exclusively, so a v2 column never enters the + * encrypt config and is returned as an untouched passthrough. No v2 ciphertext + * is stranded — decryption in `@cipherstash/stack` is generation-agnostic, so + * legacy payloads still decrypt through the core client (`decrypt` / + * `decryptModel`). Handle mixed-generation data explicitly on the caller side. * * Requires a Postgres connection (`options.databaseUrl` or `DATABASE_URL`) for * introspection, so it cannot run in a Worker or the browser. diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index f09314fd3..2b299965e 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -1,51 +1,34 @@ -import type { JsPlaintext } from '@cipherstash/protect-ffi' import type { AuditConfig } from '@cipherstash/stack/adapter-kit' import { - DATE_LIKE_CASTS, - EncryptedV3Column, logger, - matchNeedleError, parseSelectorSegments, reconstructSelectorDocument, unsupportedLeafReason, } from '@cipherstash/stack/adapter-kit' import type { EncryptionClient } from '@cipherstash/stack/encryption' import type { AnyV3Table } from '@cipherstash/stack/eql/v3' -import { - type EncryptionError, - EncryptionErrorTypes, -} from '@cipherstash/stack/errors' import type { LockContextInput } from '@cipherstash/stack/identity' -import type { ColumnSchema } from '@cipherstash/stack/schema' -import type { - BuildableQueryColumn, - Encrypted, - EncryptedQueryResult, - QueryTypeName, - ScalarQueryTerm, -} from '@cipherstash/stack/types' +import { ColumnMap } from './column-map' +import { addJsonbCastsV3 } from './helpers' +import { toDbSpace } from './query-dbspace' +import { + assertJsonContainmentOperand, + assertPostgrestCanQueryEncryptedOperator, + type EncryptedFilterState, + type EncryptionContext, + EncryptionFailedError, + encryptFilterValues, +} from './query-encrypt' +import { applyFilters } from './query-filters' +import { encryptMutationData } from './query-mutation' import { - addJsonbCastsV3, - formatContainmentOperand, - formatInListOperand, - isEncryptableTerm, - isEncryptedColumn, - mapFilterOpToQueryType, - parseOrString, - rebuildOrString, - selectKeyToDbV3, -} from './helpers' + type DecryptContext, + decryptResults, + type RawSupabaseResult, +} from './query-results' import type { - DbConflictList, - DbFilterString, - DbMutationOp, - DbMutationOptions, - DbName, - DbPendingOrCondition, - DbPendingOrFilter, DbQuerySpace, DbSelect, - DbTransformOp, EncryptedSupabaseError, EncryptedSupabaseResponse, FilterOp, @@ -56,109 +39,17 @@ import type { PendingOrCondition, PendingOrFilter, PendingRawFilter, + RecordedOps, ResultMode, SupabaseClientLike, SupabaseQueryBuilder, TransformOp, } from './types' -/** cast_as kinds that reconstruct to a JS `Date` — shared with the typed v3 - * client's decrypt-model path (see `encryption/v3.ts`). */ -const DATE_LIKE_CAST_SET = new Set(DATE_LIKE_CASTS) - -/** - * The subset of a v3 column builder the dialect relies on. Structural rather - * than the concrete class union so the runtime `instanceof EncryptedV3Column` - * gate and this type stay independent. - */ -type V3ColumnLike = { - getName(): string - getEqlType(): string - getQueryCapabilities(): { - equality: boolean - orderAndRange: boolean - freeTextSearch: boolean - /** Optional: only `public.eql_v3_json_search` (`types.Json`) carries it. */ - searchableJson?: boolean - } - build(): ColumnSchema -} - -/** - * Validate an encrypted-JSON containment operand: a NON-EMPTY plain object or a - * non-empty array. Everything else is rejected with an actionable steer: - * - * - Scalars/strings: the caller meant free-text (`matches` on a text column) or - * a selector — a raw JSON string is NOT parsed, by design (parsing would make - * `'{"a":1}'` and `{a:1}` silently different queries on other surfaces). - * - Non-plain objects (`Date`, `Map`, `RegExp`, class instances): these JSON- - * serialize to scalars or `{}` — not the sub-document the caller believes. - * - `{}` and `[]`: jsonb containment holds for EVERY document (`doc @> '{}'`), - * so an accidentally-empty needle would silently return (and decrypt) the - * whole table. The Drizzle adapter rejects the same needle for the same - * reason — the two first-party adapters must agree that this is an error. - */ -function assertJsonContainmentOperand(column: string, value: unknown): void { - const isPlainObject = - value !== null && - typeof value === 'object' && - !Array.isArray(value) && - (Object.getPrototypeOf(value) === Object.prototype || - Object.getPrototypeOf(value) === null) - if (!isPlainObject && !Array.isArray(value)) { - // Array.isArray is false on this branch by construction, so the label only - // distinguishes null / non-plain object / scalar. - const got = - value === null - ? 'null' - : typeof value === 'object' - ? (value as object).constructor?.name || 'a non-plain object' - : typeof value - throw new Error( - `[supabase v3]: encrypted JSON containment on column "${column}" takes a sub-document (plain object or array) to match, got ${got}.`, - ) - } - const empty = Array.isArray(value) - ? value.length === 0 - : Object.keys(value as object).length === 0 - if (empty) { - throw new Error( - `[supabase v3]: encrypted JSON containment on column "${column}" cannot take an empty ${Array.isArray(value) ? 'array' : 'object'} needle: it matches every row. Pass a non-empty sub-document, or omit the predicate to select all rows.`, - ) - } -} +export { EncryptionFailedError } from './query-encrypt' -/** - * Reject a declared property name that is also a DIFFERENT physical column. - * - * `select('*')` expands the introspected DB names into property names, so a - * column renamed `created_at → createdAt` and a distinct plaintext column - * literally named `createdAt` both emit the token `createdAt`, which - * `addJsonbCastsV3` turns into `createdAt:created_at::jsonb` — twice. PostgREST - * returns the encrypted column under that key and the plaintext one is never - * selected, silently yielding the wrong value for a field the row type - * guarantees. - * - * Nothing downstream can disambiguate the two, and `EncryptedTable.build()`'s - * duplicate check only fires when two BUILDERS share a `getName()`. Refuse to - * construct instead. - */ -function assertNoPropertyDbNameCollision( - tableName: string, - propToDb: Record, - allColumns: string[] | null, -): void { - if (!allColumns) return - const dbNames = new Set(allColumns) - - for (const [property, dbName] of Object.entries(propToDb)) { - if (property === dbName) continue - if (!dbNames.has(property)) continue - throw new Error( - `[supabase v3]: property "${property}" on table "${tableName}" renames DB column "${dbName}", but "${property}" is also a distinct column in the database — the two collide in select('*'). Rename the property, or drop the declared rename.`, - ) - } -} +/** Warn once per (op, column) that a `like`/`ilike` was delegated to `matches`. */ +const warnedLikeDelegation = new Set() /** * A deferred query builder that wraps Supabase's query builder to automatically @@ -184,55 +75,51 @@ function assertNoPropertyDbNameCollision( * enters a GET URL. * * Decrypted rows additionally get `Date` reconstruction from the encrypt-config - * `cast_as`, mirroring the typed v3 client. `decryptModel`/`bulkDecryptModels` - * are generation-agnostic in `@cipherstash/stack`, so a stored EQL v2 payload - * still decrypts through this builder's read path. + * `cast_as`, mirroring the typed v3 client. This builder authors and reads EQL + * v3 only: legacy `eql_v2_encrypted` columns are not recognised by introspection, + * so they never enter the encrypt config and are returned as untouched + * passthroughs. Decrypt v2 data with the core `@cipherstash/stack` client. + * + * The pipeline is split across sibling modules — `./column-map` (name and + * capability resolution), `./query-encrypt` (mutation data and filter terms), + * `./query-dbspace` (property → DB space), `./query-filters` (operand + * substitution), `./query-results` (decryption) — and orchestrated by + * {@link execute} below. */ export class EncryptedQueryBuilderImpl< T extends Record = Record, > { - protected tableName: string - protected table: AnyV3Table - protected encryptionClient: EncryptionClient - protected supabaseClient: SupabaseClientLike - protected encryptedColumnNames: string[] + private tableName: string + private table: AnyV3Table + private encryptionClient: EncryptionClient + private supabaseClient: SupabaseClientLike + /** Name and capability resolution for this table's columns. */ + private columns: ColumnMap /** All column names for the table (encrypted + plaintext), in ordinal order, * used to expand `select('*')`. `null` when the caller supplied no column * list (a v3 client that could not introspect). */ - protected allColumns: string[] | null = null - - /** JS property name → DB column name, for every encrypted column. */ - private propToDb: Record - /** DB column name → JS property name — the inverse of {@link propToDb}, used - * to expand `select('*')` back into property names. Null prototype: a DB - * column literally named `constructor` / `toString` would otherwise resolve - * to an inherited `Object.prototype` member and be emitted as a select token. */ - private dbToProp: Record - /** Built column schemas keyed by DB column name (for `cast_as`). */ - private columnSchemas: Record - /** Column builders keyed by BOTH property name and DB name. */ - private v3Columns: Record + private allColumns: string[] | null = null /** EQL 3.0.2+ requires query-domain casts PostgREST cannot express. */ private queryDomainsRequired: boolean // Recorded operations - protected mutation: MutationOp | null = null - protected selectColumns: string | null = null - protected selectOptions: + private mutation: MutationOp | null = null + private selectColumns: string | null = null + private selectOptions: | { head?: boolean; count?: 'exact' | 'planned' | 'estimated' } | undefined = undefined - protected filters: PendingFilter[] = [] - protected orFilters: PendingOrFilter[] = [] - protected matchFilters: PendingMatchFilter[] = [] - protected notFilters: PendingNotFilter[] = [] - protected rawFilters: PendingRawFilter[] = [] - protected transforms: TransformOp[] = [] - protected resultMode: ResultMode = 'array' - protected shouldThrowOnError = false + private filters: PendingFilter[] = [] + private orFilters: PendingOrFilter[] = [] + private matchFilters: PendingMatchFilter[] = [] + private notFilters: PendingNotFilter[] = [] + private rawFilters: PendingRawFilter[] = [] + private transforms: TransformOp[] = [] + private resultMode: ResultMode = 'array' + private shouldThrowOnError = false // Encryption-specific state - protected lockContext: LockContextInput | null = null - protected auditConfig: AuditConfig | null = null + private lockContext: LockContextInput | null = null + private auditConfig: AuditConfig | null = null constructor( tableName: string, @@ -248,31 +135,7 @@ export class EncryptedQueryBuilderImpl< this.supabaseClient = supabaseClient this.allColumns = allColumns this.queryDomainsRequired = queryDomainsRequired - this.propToDb = table.buildColumnKeyMap() - this.columnSchemas = table.build().columns - - this.dbToProp = Object.create(null) as Record - for (const [property, dbName] of Object.entries(this.propToDb)) { - this.dbToProp[dbName] = property - } - - assertNoPropertyDbNameCollision(tableName, this.propToDb, allColumns) - - // Null-prototype: keyed by DB column names, and `validateTransforms` reads - // it without an own-key guard — an inherited `constructor`/`toString` would - // otherwise resolve truthy for a plaintext column of that name. - this.v3Columns = Object.create(null) as Record - for (const [property, builder] of Object.entries(table.columnBuilders)) { - if (builder instanceof EncryptedV3Column) { - const col = builder as unknown as V3ColumnLike - this.v3Columns[property] = col - this.v3Columns[col.getName()] = col - } - } - - // Filters and select strings address columns by JS property name AND by DB - // name, so recognition must cover both. - this.encryptedColumnNames = Object.keys(this.v3Columns) + this.columns = new ColumnMap(tableName, table, allColumns) } // --------------------------------------------------------------------------- @@ -289,7 +152,9 @@ export class EncryptedQueryBuilderImpl< "encryptedSupabase does not support select('*'). Please list columns explicitly so that encrypted columns can be cast with ::jsonb.", ) } - this.selectColumns = this.expandAllColumns(this.allColumns).join(', ') + this.selectColumns = this.columns + .expandAllColumns(this.allColumns) + .join(', ') } else { this.selectColumns = columns } @@ -297,26 +162,6 @@ export class EncryptedQueryBuilderImpl< return this } - /** - * Expand the introspected column list (DB names) into JS property names. - * - * Load-bearing for `select('*')` on a DECLARED table that renames a column. - * `addJsonbCastsV3` only emits the `prop:db_name::jsonb` alias — the thing - * that makes PostgREST return the column under its property name — when the - * token it sees is a property name. Feeding it the raw DB name instead takes - * the unaliased `dbNames.has(...)` branch, so the row comes back keyed - * `created_at` while the declared row type promises `createdAt`, silently - * yielding `undefined` for a field TypeScript guarantees. - * - * A DB column with no encrypted builder (plaintext passthrough, and every - * synthesized column, where property == DB name) maps to itself. - */ - protected expandAllColumns(columns: string[]): string[] { - return columns.map((dbName) => - Object.hasOwn(this.dbToProp, dbName) ? this.dbToProp[dbName] : dbName, - ) - } - insert( data: Partial | Partial[], options?: { @@ -411,7 +256,7 @@ export class EncryptedQueryBuilderImpl< * SQL LIKE. */ like(column: string, pattern: string): this { - if (!this.isEncryptedV3Column(column)) { + if (!this.columns.isEncryptedV3Column(column)) { this.filters.push({ op: 'like', column, value: pattern }) return this } @@ -419,7 +264,7 @@ export class EncryptedQueryBuilderImpl< } ilike(column: string, pattern: string): this { - if (!this.isEncryptedV3Column(column)) { + if (!this.columns.isEncryptedV3Column(column)) { this.filters.push({ op: 'ilike', column, value: pattern }) return this } @@ -435,15 +280,15 @@ export class EncryptedQueryBuilderImpl< * than silently emit a bloom match under a name that promises exactness. */ contains(column: string, value: unknown): this { - if (this.isSearchableJsonColumn(column)) { - this.assertPostgrestCanQueryEncryptedOperator('contains', column) + if (this.columns.isSearchableJsonColumn(column)) { + this.assertPostgrestCanQueryEncrypted('contains', column) // Same validator the term resolver enforces — failing here just surfaces // the error at the call site instead of at execution. assertJsonContainmentOperand(column, value) this.filters.push({ op: 'contains', column, value }) return this } - if (this.isEncryptedV3Column(column)) { + if (this.columns.isEncryptedV3Column(column)) { throw new Error( `[supabase v3]: contains() is native (exact) containment and does not apply to encrypted column "${column}". Use matches() for encrypted free-text search.`, ) @@ -463,17 +308,17 @@ export class EncryptedQueryBuilderImpl< * silently accept as containment of the raw string. */ matches(column: string, value: unknown): this { - if (this.isSearchableJsonColumn(column)) { + if (this.columns.isSearchableJsonColumn(column)) { throw new Error( `[supabase v3]: matches() is encrypted free-text search and does not apply to encrypted JSON column "${column}". Use contains("${column}", subDocument) or selectorEq("${column}", path, value).`, ) } - if (!this.isEncryptedV3Column(column)) { + if (!this.columns.isEncryptedV3Column(column)) { throw new Error( `[supabase v3]: matches() is encrypted free-text search and requires an encrypted column; "${column}" is not one. Use contains() for native containment.`, ) } - this.assertPostgrestCanQueryEncryptedOperator('matches', column) + this.assertPostgrestCanQueryEncrypted('matches', column) this.filters.push({ op: 'matches', column, value }) return this } @@ -508,8 +353,8 @@ export class EncryptedQueryBuilderImpl< not(column: string, operator: string, value: unknown): this { if ( operator === 'contains' && - this.isEncryptedV3Column(column) && - !this.isSearchableJsonColumn(column) + this.columns.isEncryptedV3Column(column) && + !this.columns.isSearchableJsonColumn(column) ) { throw new Error( `[supabase v3]: not("${column}", 'contains', …) does not apply to encrypted column "${column}" — that is fuzzy free-text matching, not containment. Use not("${column}", 'matches', …) or the raw 'cs' operator.`, @@ -518,7 +363,7 @@ export class EncryptedQueryBuilderImpl< // Mirror of the matches() guard: a `matches` spelling on a JSON column // would otherwise resolve to containment (the two share the `cs` wire op), // silently negating an EXACT operation under a name that promises FUZZY. - if (operator === 'matches' && this.isSearchableJsonColumn(column)) { + if (operator === 'matches' && this.columns.isSearchableJsonColumn(column)) { throw new Error( `[supabase v3]: not("${column}", 'matches', …) does not apply to encrypted JSON column "${column}" — matches() is free-text search. Use not("${column}", 'contains', subDocument) or selectorNe("${column}", path, value).`, ) @@ -562,7 +407,7 @@ export class EncryptedQueryBuilderImpl< * `ops.selector()` supports it today. */ selectorEq(column: string, path: string, value: unknown): this { - this.assertPostgrestCanQueryEncryptedOperator('selectorEq', column) + this.assertPostgrestCanQueryEncrypted('selectorEq', column) const needle = this.selectorNeedle('selectorEq', column, path, value) return this.contains(column, needle) } @@ -578,7 +423,7 @@ export class EncryptedQueryBuilderImpl< * operand is encrypted through the normal or-condition term path. */ selectorNe(column: string, path: string, value: unknown): this { - this.assertPostgrestCanQueryEncryptedOperator('selectorNe', column) + this.assertPostgrestCanQueryEncrypted('selectorNe', column) const needle = this.selectorNeedle('selectorNe', column, path, value) return this.or([ { column, op: 'is', value: null }, @@ -686,21 +531,23 @@ export class EncryptedQueryBuilderImpl< // Core execution // --------------------------------------------------------------------------- - protected async execute(): Promise> { + private async execute(): Promise> { try { logger.debug(`Supabase encrypted query on table "${this.tableName}".`) + const ctx = this.encryptionContext() + // 1. Encrypt mutation data - const encryptedMutation = await this.encryptMutationData() + const encryptedMutation = await encryptMutationData(this.mutation, ctx) // 2. Build select string with ::jsonb casts const selectString = this.buildSelectString() // 3. Translate every recorded column name into DB-space, once. - const dbSpace = this.toDbSpace() + const dbSpace = toDbSpace(this.recordedOps(), this.columns) // 4. Batch-encrypt filter values - const encryptedFilters = await this.encryptFilterValues(dbSpace) + const encryptedFilters = await encryptFilterValues(dbSpace, ctx) // 5. Build and execute real Supabase query const result = await this.buildAndExecuteQuery( @@ -711,7 +558,12 @@ export class EncryptedQueryBuilderImpl< ) // 6. Decrypt results - return await this.decryptResults(result) + return await decryptResults(result, { + ...ctx, + selectColumns: this.selectColumns, + resultMode: this.resultMode, + hasMutation: this.mutation !== null, + } satisfies DecryptContext) } catch (err) { const message = err instanceof Error ? err.message : String(err) logger.error( @@ -745,676 +597,47 @@ export class EncryptedQueryBuilderImpl< } } - // --------------------------------------------------------------------------- - // Step 1: Encrypt mutation data - // --------------------------------------------------------------------------- - - protected async encryptMutationData(): Promise< - Record | Record[] | null - > { - if (!this.mutation) return null - - if (this.mutation.kind === 'delete') return null - - const data = this.mutation.data - - if (Array.isArray(data)) { - // Bulk encrypt - const baseOp = this.encryptionClient.bulkEncryptModels(data, this.table) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) { - logger.error( - `Supabase: failed to encrypt models for table "${this.tableName}"`, - ) - - throw new EncryptionFailedError( - `Failed to encrypt models: ${result.failure.message}`, - result.failure, - ) - } - - return this.transformEncryptedMutationModels(result.data) - } - - // Single model - const baseOp = this.encryptionClient.encryptModel(data, this.table) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) { - logger.error( - `Supabase: failed to encrypt model for table "${this.tableName}"`, - ) - - throw new EncryptionFailedError( - `Failed to encrypt model: ${result.failure.message}`, - result.failure, - ) + /** The shared slice of builder state every encrypt/decrypt step needs. Built + * per `execute()`, so `lockContext`/`auditConfig` are read at execution time. */ + private encryptionContext(): EncryptionContext { + return { + tableName: this.tableName, + table: this.table, + encryptionClient: this.encryptionClient, + lockContext: this.lockContext, + auditConfig: this.auditConfig, + columns: this.columns, + queryDomainsRequired: this.queryDomainsRequired, } - - return this.transformEncryptedMutationModel(result.data) } - /** - * Encode an encrypted model for the Supabase request body. The native - * `eql_v3.*` domains are plain jsonb, so the raw encrypted payload is sent - * (keyed by DB column name). - */ - protected transformEncryptedMutationModel( - model: Record, - ): Record { - const out: Record = Object.create(null) - for (const [key, value] of Object.entries(model)) { - out[this.dbNameFor(key)] = value + /** The recorded query in property space, for `toDbSpace`. */ + private recordedOps(): RecordedOps { + return { + filters: this.filters, + matchFilters: this.matchFilters, + notFilters: this.notFilters, + rawFilters: this.rawFilters, + orFilters: this.orFilters, + transforms: this.transforms, + mutation: this.mutation, } - return out - } - - protected transformEncryptedMutationModels( - models: Record[], - ): Record[] { - return models.map((model) => this.transformEncryptedMutationModel(model)) } // --------------------------------------------------------------------------- // Step 2: Build select string with casts // --------------------------------------------------------------------------- - protected buildSelectString(): DbSelect | null { + private buildSelectString(): DbSelect | null { if (this.selectColumns === null) return null - return addJsonbCastsV3(this.selectColumns, this.propToDb) - } - - // --------------------------------------------------------------------------- - // Step 3: Encrypt filter values - // --------------------------------------------------------------------------- - - protected async encryptFilterValues( - dbSpace: DbQuerySpace, - ): Promise { - // Collect all terms that need encryption - const terms: ScalarQueryTerm[] = [] - const termMap: TermMapping[] = [] - - const tableColumns = this.getColumnMap() - - const pushTerm = ( - value: JsPlaintext, - column: ScalarQueryTerm['column'], - queryType: QueryTypeName, - mapping: TermMapping, - ) => { - terms.push({ - value, - column, - table: this.table, - queryType, - returnType: 'composite-literal', - }) - termMap.push(mapping) - } - - /** - * Collect one term per element of an `in`-list operand. - * - * Element-wise is the only correct encoding: encrypting the array as ONE - * value collapses `(a,b)` into a single ciphertext that matches nothing. A - * null element is SQL NULL and passes through unencrypted; the applier - * restores it by index, which is why the mapping carries `inIndex`. - * - * Shared by the regular-`in`, `not(…,'in',…)` and or-condition paths. They - * drifted apart once already — the `not` path went unfixed while the other - * two encrypted element-wise — so they are kept in lockstep here rather than - * spelled out three times. - */ - const collectInListTerms = ( - op: FilterOp, - values: readonly unknown[], - column: ScalarQueryTerm['column'], - queryType: QueryTypeName, - mappingFor: (inIndex: number) => TermMapping, - ) => { - for (let j = 0; j < values.length; j++) { - if (!isEncryptableTerm(op, values[j])) continue - pushTerm(values[j] as JsPlaintext, column, queryType, mappingFor(j)) - } - } - - // Regular filters - for (let i = 0; i < dbSpace.filters.length; i++) { - const f = dbSpace.filters[i] - if (!isEncryptedColumn(f.column, this.encryptedColumnNames)) continue - - const column = tableColumns[f.column] - if (!column) continue - - if (f.op === 'in' && Array.isArray(f.value)) { - collectInListTerms( - f.op, - f.value, - column, - mapFilterOpToQueryType(f.op), - (inIndex) => ({ source: 'filter', filterIndex: i, inIndex }), - ) - } else if (!isEncryptableTerm(f.op, f.value)) { - // `is` predicate or null operand — forwarded unencrypted. - } else { - pushTerm(f.value as JsPlaintext, column, mapFilterOpToQueryType(f.op), { - source: 'filter', - filterIndex: i, - }) - } - } - - // Match filters - for (let i = 0; i < dbSpace.matchFilters.length; i++) { - const mf = dbSpace.matchFilters[i] - for (const { column: colName, value } of mf.entries) { - if (!isEncryptedColumn(colName, this.encryptedColumnNames)) continue - // `match` carries no operator; equality is implied. - if (!isEncryptableTerm('eq', value)) continue - const column = tableColumns[colName] - if (!column) continue - - pushTerm(value as JsPlaintext, column, 'equality', { - source: 'match', - matchIndex: i, - column: colName, - }) - } - } - - // Not filters - for (let i = 0; i < dbSpace.notFilters.length; i++) { - const nf = dbSpace.notFilters[i] - if (!isEncryptedColumn(nf.column, this.encryptedColumnNames)) continue - if (!isEncryptableTerm(nf.op, nf.value)) continue - const column = tableColumns[nf.column] - if (!column) continue - - if (nf.op === 'in') { - // A PostgREST list literal (`'(a,b)'`) cannot be encrypted element-wise, - // and encrypting it whole matches nothing. Refuse it rather than emit a - // filter that silently returns no rows. - if (!Array.isArray(nf.value)) { - throw new Error( - `not("${nf.column}", "in", …) on an encrypted column requires an array of values, ` + - `not a PostgREST list literal — each element must be encrypted separately`, - ) - } - collectInListTerms( - nf.op, - nf.value, - column, - mapFilterOpToQueryType(nf.op), - (inIndex) => ({ source: 'not', notIndex: i, inIndex }), - ) - continue - } - - pushTerm(nf.value as JsPlaintext, column, mapFilterOpToQueryType(nf.op), { - source: 'not', - notIndex: i, - }) - } - - // Or filters — conditions were parsed once, in `toDbSpace`. The string and - // structured forms differ only in their `source` tag; the encryption rules, - // including the `in`-list split below, are identical. - for (let i = 0; i < dbSpace.orFilters.length; i++) { - const of_ = dbSpace.orFilters[i] - const source = of_.kind === 'string' ? 'or-string' : 'or-structured' - - for (let j = 0; j < of_.conditions.length; j++) { - const cond = of_.conditions[j] - if (!isEncryptedColumn(cond.column, this.encryptedColumnNames)) continue - const column = tableColumns[cond.column] - if (!column) continue - - // `queryTypeForOrOp`, not `mapFilterOpToQueryType`: an or-condition may - // carry a raw PostgREST operator (`cs`), which is not a `FilterOp`. - const queryType = this.queryTypeForOrOp(cond.op) - const mappingFor = (inIndex?: number): TermMapping => ({ - source, - orIndex: i, - conditionIndex: j, - inIndex, - }) - - if (cond.op === 'in' && Array.isArray(cond.value)) { - collectInListTerms(cond.op, cond.value, column, queryType, mappingFor) - continue - } - - if (!isEncryptableTerm(cond.op, cond.value)) continue - pushTerm(cond.value as JsPlaintext, column, queryType, mappingFor()) - } - } - - // Raw filters - for (let i = 0; i < dbSpace.rawFilters.length; i++) { - const rf = dbSpace.rawFilters[i] - if (!isEncryptedColumn(rf.column, this.encryptedColumnNames)) continue - const column = tableColumns[rf.column] - if (!column) continue - - if (rf.operator === 'in') { - // Same contract as the `not(…, 'in', …)` path: a PostgREST list literal - // (`'("a","b")'`) cannot be encrypted element-wise, and encrypting it - // whole matches nothing. Refuse it rather than emit a filter that - // silently returns no rows. - if (!Array.isArray(rf.value)) { - throw new Error( - `filter("${rf.column}", "in", …) on an encrypted column requires an array of values, ` + - `not a PostgREST list literal — each element must be encrypted separately`, - ) - } - collectInListTerms( - 'in', - rf.value, - column, - this.queryTypeForRawOp(rf.operator), - (inIndex) => ({ source: 'raw', rawIndex: i, inIndex }), - ) - continue - } - - if (!isEncryptableTerm(rf.operator, rf.value)) continue - - pushTerm( - rf.value as JsPlaintext, - column, - this.queryTypeForRawOp(rf.operator), - { source: 'raw', rawIndex: i }, - ) - } - - if (terms.length === 0) { - return { encryptedValues: [], termMap: [] } - } - - const encryptedValues = await this.encryptCollectedTerms(terms) - return { encryptedValues, termMap } - } - - /** - * Encrypt every filter operand as a full storage envelope, serialized to jsonb - * text for the PostgREST filter value. - * - * Terms are grouped by column and each group takes ONE `bulkEncrypt` crossing. - * `in(col, [a, b, c])` collects one term per element (the list must never be - * encrypted whole), so encrypting per term spent N ZeroKMS/FFI round-trips - * where one would do. `bulkEncrypt` carries a single `{table, column}` for the - * whole payload, so the grouping is mandatory, not an optimisation: one bulk - * call over a mixed-column term array would stamp one column onto every - * plaintext. Results are scattered back onto the terms' original indices, - * which is the contract `termMap` downstream relies on. - * - * Mirrors `eql/v3/drizzle/operators.ts` `encryptOperands` — same batching - * contract, same length assertion, same fallback. Kept separate because that - * one encrypts a single-column operand list and returns `SQL[]`, while this - * must group a multi-column term array and preserve positions. - */ - protected async encryptCollectedTerms( - terms: ScalarQueryTerm[], - ): Promise { - const groups = new Map< - V3ColumnLike, - { indices: number[]; values: ScalarQueryTerm['value'][] } - >() - terms.forEach((term, index) => { - const column = this.assertTermQueryable(term) - const group = groups.get(column) ?? { indices: [], values: [] } - group.indices.push(index) - group.values.push(term.value) - groups.set(column, group) - }) - - const bulkEncrypt = this.encryptionClient.bulkEncrypt?.bind( - this.encryptionClient, - ) - // Each term becomes the `JSON.stringify`'d storage envelope — a `string`, - // which is one arm of `EncryptedQueryResult`. PostgREST cannot cast a filter - // value to the `eql_v3.query_` twins, so v3 sends full envelopes, - // serialized to jsonb text. - const results = new Array(terms.length) - - await Promise.all( - Array.from(groups, async ([column, { indices, values }]) => { - const encrypted = bulkEncrypt - ? await this.bulkEncryptGroup(bulkEncrypt, column, values) - : await this.encryptGroupPerTerm(column, values) - - encrypted.forEach((envelope, i) => { - results[indices[i]] = JSON.stringify(envelope) - }) - }), - ) - - return results - } - - /** - * Validate a term's query type against its column's declared capabilities. - * Pure validation: `encrypt`/`bulkEncrypt` never receive the query type. On - * EQL 3.0.2+, free-text/JSON terms are rejected before this storage-encryption - * path can place ciphertext in a GET URL. - */ - private assertTermQueryable(term: ScalarQueryTerm): V3ColumnLike { - const column = term.column as unknown as V3ColumnLike - let queryType = term.queryType ?? 'equality' - const capabilities = column.getQueryCapabilities() - - // The `cs` wire operator is capability-overloaded: bloom free-text on a - // match/search TEXT column, encrypted ste_vec containment on a `types.Json` - // DOCUMENT column. Both arrive here as `freeTextSearch` (contains/matches/ - // raw `cs` all map there); resolve to the capability the column actually - // carries. The two are mutually exclusive by construction, so this can - // never reinterpret a real free-text column. - if ( - queryType === 'freeTextSearch' && - !capabilities.freeTextSearch && - capabilities.searchableJson - ) { - queryType = 'searchableJson' - } - - if ( - queryType !== 'equality' && - queryType !== 'orderAndRange' && - queryType !== 'freeTextSearch' && - queryType !== 'searchableJson' - ) { - throw new Error( - `[supabase v3]: query type "${queryType}" is not supported on EQL v3 columns`, - ) - } - - if (!capabilities[queryType]) { - throw new Error( - `[supabase v3]: column "${column.getName()}" (${column.getEqlType()}) does not support ${queryType} queries — declare the column with a domain that carries that capability`, - ) - } - - if (queryType === 'freeTextSearch' || queryType === 'searchableJson') { - // This is the common boundary for every spelling that collects an - // encrypted match/containment term: matches(), contains(), not(), raw - // filter(), and both forms of or(). Method-level checks provide earlier - // errors for the direct helpers, but cannot cover the raw filter paths on - // their own. - this.assertPostgrestCanQueryEncryptedOperator('filter', column.getName()) - } - - if (queryType === 'searchableJson') { - // THE single enforced operand boundary for encrypted-JSON containment. - // Terms reach this resolver from every spelling — contains(), raw - // .filter(col,'cs',…), not(col,'contains'|'matches',…), and .or() - // string/structured conditions — and only contains() has a method-level - // guard. Without this check a raw string (e.g. a free-text term ported - // from a text column, or an .or() condition value, which is always a - // string) would be storage-encrypted as a JSON SCALAR and silently match - // nothing; pre-#650 every such spelling failed loudly on capability. - assertJsonContainmentOperand(column.getName(), term.value) - } - - // Free-text (bloom) needle floor. A needle shorter than the tokenizer's - // token_length produces NO tokens, so `bf @> '{}'` holds for every row and - // the query would silently return (and the caller decrypt) the whole table - // — a fail-open over-exposure. Reject it up front, mirroring the Drizzle v3 - // adapter (matchNeedleError) so both first-party surfaces guard identically. - // JSON containment terms (searchableJson) are validated separately above. - if (queryType === 'freeTextSearch') { - const match = column.build().indexes?.match - const reason = match ? matchNeedleError(term.value, match) : undefined - if (reason) { - throw new Error( - `[supabase v3]: cannot search column "${column.getName()}": ${reason}`, - ) - } - } - - return column - } - - private encryptionFailure(message: string, cause?: EncryptionError): never { - logger.error( - `Supabase: failed to encrypt query terms for table "${this.tableName}"`, - ) - // Most callers pass the operation's own `EncryptionError`; the contract- - // violation cases (bulk length mismatch, null envelope) have none, so - // synthesize one — a broken query encryption is still an encryption failure, - // and callers branch on `error.encryptionError` regardless. - throw new EncryptionFailedError( - `Failed to encrypt query terms: ${message}`, - cause ?? { type: EncryptionErrorTypes.EncryptionError, message }, - ) - } - - /** One FFI crossing for a column's whole operand list. */ - private async bulkEncryptGroup( - bulkEncrypt: NonNullable, - column: V3ColumnLike, - values: ScalarQueryTerm['value'][], - ): Promise> { - const baseOp = bulkEncrypt( - values.map((plaintext) => ({ plaintext })) as never, - { column, table: this.table } as never, - ) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) - this.encryptionFailure(result.failure.message, result.failure) - - // `bulkEncrypt` is position-stable, so a length mismatch means the contract - // was violated. Truncating instead would silently widen an `in` predicate - // (or narrow a `not.in`) to whatever came back. `result.data` is now - // `BulkEncryptedData` — `{ id?, data: Encrypted | null }[]` — not `unknown`. - const encrypted = result.data - if (encrypted.length !== values.length) { - this.encryptionFailure( - `bulk encryption returned ${encrypted.length} terms for ${values.length} values on column "${column.getName()}".`, - ) - } - return encrypted.map((term, i) => { - // `BulkEncryptedData` types the element as `Encrypted | null`. A `null` - // envelope here would be `JSON.stringify`'d to the literal string `"null"` - // and sent as the filter operand — silently matching whatever `"null"` - // encodes to rather than failing. A query term should never encrypt to a - // null envelope, so treat it as a contract violation, not a value. - if (term.data === null) { - this.encryptionFailure( - `bulk encryption returned a null envelope at position ${i} for column "${column.getName()}".`, - ) - } - return term.data - }) - } - - /** Fallback for a client that predates `bulkEncrypt`. */ - private async encryptGroupPerTerm( - column: V3ColumnLike, - values: ScalarQueryTerm['value'][], - ): Promise { - return Promise.all( - values.map(async (value) => { - const baseOp = this.encryptionClient.encrypt(value, { - column, - table: this.table, - }) - const op = this.lockContext - ? baseOp.withLockContext(this.lockContext) - : baseOp - if (this.auditConfig) op.audit(this.auditConfig) - - const result = await op - if (result.failure) { - this.encryptionFailure(result.failure.message, result.failure) - } - return result.data - }), - ) - } - - // --------------------------------------------------------------------------- - // Phase boundary: property-space -> DB-space - // --------------------------------------------------------------------------- - - /** - * Translate every recorded column name from JS property space into DB space, - * once. Downstream (`encryptFilterValues`, `applyFilters`, - * `buildAndExecuteQuery`) consumes only the branded result, so a column can - * no longer reach PostgREST untranslated — that is a compile error. - * - * Total: `filterColumnName`, `parseOrString`, and `resolveMutationOptions` - * never throw, so this introduces no new early-throw point and cannot perturb - * the order in which capability errors surface. - * - * Safe to run BEFORE encryption: `getColumnMap()`/`encryptedColumnNames` are - * keyed by both property and DB name, so column lookup resolves identically - * either side of the translation, and `tableColumns[prop]` is the very same - * builder object as `tableColumns[db]`. - */ - protected toDbSpace(): DbQuerySpace { - return { - filters: this.filters.map((f) => ({ - ...f, - column: this.filterColumnName(f.column), - })), - matchFilters: this.matchFilters.map((mf) => ({ - entries: Object.entries(mf.query).map(([column, value]) => ({ - column: this.filterColumnName(column), - value, - })), - })), - notFilters: this.notFilters.map((nf) => ({ - ...nf, - column: this.filterColumnName(nf.column), - })), - rawFilters: this.rawFilters.map((rf) => ({ - ...rf, - column: this.filterColumnName(rf.column), - })), - orFilters: this.orFilters.map((of_) => this.orFilterToDbSpace(of_)), - transforms: this.transforms.map((t) => this.transformToDbSpace(t)), - mutation: this.mutation ? this.mutationToDbSpace(this.mutation) : null, - } - } - - /** Column names only. Which conditions were encrypted is never decided here: - * it stays derived at apply time from the substitution maps, so this pass - * never has to agree with the encryption predicate. The operator token is - * settled later still, in `rebuildOrString`, where `contains` becomes `cs` - * for encrypted and plaintext conditions alike. */ - private orFilterToDbSpace(of_: PendingOrFilter): DbPendingOrFilter { - const toDbCondition = (c: PendingOrCondition): DbPendingOrCondition => ({ - ...c, - column: this.filterColumnName(c.column), - }) - - if (of_.kind === 'string') { - return { - kind: 'string', - original: of_.value, - conditions: parseOrString(of_.value).map(toDbCondition), - referencedTable: of_.referencedTable, - } - } - return { kind: 'structured', conditions: of_.conditions.map(toDbCondition) } - } - - /** - * Encrypted ordering columns sort by their `op` term, not by the envelope. - * - * `order=col->op` is the one ordering expression PostgREST can emit that - * reaches the OPE term. It must NOT leak into filters — those compare whole - * envelopes through the `eql_v3.*` operators — which is why this is its own - * seam rather than a change to `filterColumnName`. - * - * The canonical EQL form is `ORDER BY eql_v3.ord_term(col)`, which returns - * `eql_v3_internal.ope_cllw` — a domain over `bytea`, ordered by the native - * btree. PostgREST cannot call a function, so it orders the `op` term where it - * sits, inside the envelope. The two agree because the term is what - * `ord_term()` returns. - * - * `->` (jsonb) rather than `->>` (text) keeps the comparison on the typed - * value. Note this does NOT avoid the database collation: Postgres compares - * jsonb strings with `varstr_cmp` under the default collation, exactly as it - * does text. What makes the ordering collation-independent is the term itself - * — fixed-width lowercase hex (`[0-9a-f]`, 130 chars for `integer_ord`, 82 for - * `text_search`) — and every collation orders digits before letters and hex - * letters among themselves. `match-bloom`'s sibling assertion pins that shape. - * - * This runs at column-name-mapping time (`transformToDbSpace`), BEFORE - * `buildAndExecuteQuery` calls `validateTransforms`. For an encrypted column - * with no `ope` index it therefore returns a bare `dbName` here — a name that - * would sort by `jsonb_cmp` over the ciphertext if it reached PostgREST — but - * it never does: `validateTransforms` throws (with a domain-specific reason) - * before the query executes, so the bare name is only ever an intermediate - * value on a request that is about to be rejected. - */ - protected orderColumnName(column: string): DbName { - const dbName = this.dbNameFor(column) - const encrypted = this.v3Columns[column] - if (!encrypted) return dbName as DbName - - return ( - this.columnSchemas[dbName]?.indexes?.ope ? `${dbName}->op` : dbName - ) as DbName - } - - private transformToDbSpace(t: TransformOp): DbTransformOp { - switch (t.kind) { - case 'order': - return { ...t, column: this.orderColumnName(t.column) } - // `returns` is in the union but never pushed (`returns()` is a cast). - case 'limit': - case 'range': - case 'single': - case 'maybeSingle': - case 'csv': - case 'abortSignal': - case 'throwOnError': - case 'returns': - return t - default: { - const exhaustive: never = t - return exhaustive - } - } - } - - private mutationToDbSpace(m: MutationOp): DbMutationOp { - switch (m.kind) { - case 'insert': - case 'upsert': - return { ...m, options: this.resolveMutationOptions(m.options) } - case 'update': - case 'delete': - return m // options carry no column names - default: { - const exhaustive: never = m - return exhaustive - } - } + return addJsonbCastsV3(this.selectColumns, this.columns.propToDb) } // --------------------------------------------------------------------------- - // Step 4: Build and execute real Supabase query + // Step 5: Build and execute real Supabase query // --------------------------------------------------------------------------- - protected async buildAndExecuteQuery( + private async buildAndExecuteQuery( encryptedMutation: | Record | Record[] @@ -1454,7 +677,13 @@ export class EncryptedQueryBuilderImpl< } // Apply resolved filters - query = this.applyFilters(query, encryptedFilters, dbSpace) + query = applyFilters( + query, + encryptedFilters, + dbSpace, + this.columns, + this.queryDomainsRequired, + ) // Apply transforms — column names already in DB-space. for (const t of dbSpace.transforms) { @@ -1490,298 +719,6 @@ export class EncryptedQueryBuilderImpl< return result } - // --------------------------------------------------------------------------- - // Apply filters with encrypted values substituted - // --------------------------------------------------------------------------- - - protected applyFilters( - query: SupabaseQueryBuilder, - encryptedFilters: EncryptedFilterState, - dbSpace: DbQuerySpace, - ): SupabaseQueryBuilder { - let q = query - - // Build lookup maps for quick access to encrypted values - const filterValueMap = new Map() - const filterInMap = new Map() // "filterIndex:inIndex" -> value - const matchValueMap = new Map() // "matchIndex:column" -> value - const notValueMap = new Map() - const notInMap = new Map() // "notIndex:inIndex" -> value - const rawValueMap = new Map() - const rawInMap = new Map() // "rawIndex:inIndex" -> value - const orStringConditionMap = new Map() // "orIndex:condIndex" -> value - const orStructuredConditionMap = new Map() - - for (let i = 0; i < encryptedFilters.termMap.length; i++) { - const mapping = encryptedFilters.termMap[i] - const encValue = encryptedFilters.encryptedValues[i] - - switch (mapping.source) { - case 'filter': - if (mapping.inIndex !== undefined) { - filterInMap.set( - `${mapping.filterIndex}:${mapping.inIndex}`, - encValue, - ) - } else { - filterValueMap.set(mapping.filterIndex, encValue) - } - break - case 'match': - matchValueMap.set(`${mapping.matchIndex}:${mapping.column}`, encValue) - break - case 'not': - if (mapping.inIndex !== undefined) { - notInMap.set(`${mapping.notIndex}:${mapping.inIndex}`, encValue) - } else { - notValueMap.set(mapping.notIndex, encValue) - } - break - case 'raw': - if (mapping.inIndex !== undefined) { - rawInMap.set(`${mapping.rawIndex}:${mapping.inIndex}`, encValue) - } else { - rawValueMap.set(mapping.rawIndex, encValue) - } - break - // `inIndex` widens the key to address one element of an `in` list, so a - // whole-condition value and a per-element value never collide. - case 'or-string': - orStringConditionMap.set(orKey(mapping), encValue) - break - case 'or-structured': - orStructuredConditionMap.set(orKey(mapping), encValue) - break - } - } - - // Apply regular filters - for (let i = 0; i < dbSpace.filters.length; i++) { - const f = dbSpace.filters[i] - let value = f.value - - if (filterValueMap.has(i)) { - value = filterValueMap.get(i) - } else if (f.op === 'in' && Array.isArray(f.value)) { - // Reconstruct array with encrypted values substituted - value = f.value.map((v, j) => { - const key = `${i}:${j}` - return filterInMap.has(key) ? filterInMap.get(key) : v - }) - } - - const column = f.column - const wasEncrypted = filterValueMap.has(i) - - switch (f.op) { - case 'eq': - q = q.eq(column, value) - break - case 'neq': - q = q.neq(column, value) - break - case 'gt': - q = q.gt(column, value) - break - case 'gte': - q = q.gte(column, value) - break - case 'lt': - q = q.lt(column, value) - break - case 'lte': - q = q.lte(column, value) - break - case 'like': - case 'ilike': - q = this.applyPatternFilter(q, column, f.op, value, wasEncrypted) - break - // `matches` (encrypted free-text) and `contains` (plaintext / encrypted - // JSON) share the `cs`/`@>` wire operator; the operand encoding is the - // same, so both emit through the one containment applier. - case 'contains': - case 'matches': - q = this.applyContainsFilter(q, column, value, wasEncrypted) - break - case 'is': - q = q.is(column, value) - break - case 'in': - // `wasEncrypted` above is false for in-lists: their ciphertexts land - // in `filterInMap`, keyed per element. - q = this.applyInFilter( - q, - column, - value as unknown[], - Array.isArray(f.value) && - f.value.some((_, j) => filterInMap.has(`${i}:${j}`)), - ) - break - } - } - - // Apply match filters - for (let i = 0; i < dbSpace.matchFilters.length; i++) { - const mf = dbSpace.matchFilters[i] - const resolvedQuery: Record = {} - - for (const { column: colName, value: originalValue } of mf.entries) { - const key = `${i}:${colName}` - resolvedQuery[colName] = matchValueMap.has(key) - ? matchValueMap.get(key) - : originalValue - } - - q = q.match(resolvedQuery) - } - - // Apply not filters - for (let i = 0; i < dbSpace.notFilters.length; i++) { - const nf = dbSpace.notFilters[i] - - if (nf.op === 'in' && Array.isArray(nf.value)) { - const values = nf.value.map((v, j) => - notInMap.has(`${i}:${j}`) ? notInMap.get(`${i}:${j}`) : v, - ) - q = q.not(nf.column, 'in', formatInListOperand(values)) - continue - } - - const wasEncrypted = notValueMap.has(i) - const value = wasEncrypted ? notValueMap.get(i) : nf.value - - // `contains` is a supabase-js METHOD name, not a PostgREST operator, and - // `q.not()` interpolates its operand with `String(value)` — so an array - // arrives brace-less and an object as `[object Object]`. Build the - // containment literal ourselves and emit the `cs` token, exactly as the - // `.or()` path does. A scalar (including the encrypted envelope, already - // serialized) yields `null` and is forwarded untouched. - if (nf.op === 'contains' || nf.op === 'matches') { - const literal = formatContainmentOperand(value) - q = q.not(nf.column, 'cs', literal ?? value) - continue - } - - q = q.not(nf.column, this.notFilterOperator(nf.op, wasEncrypted), value) - } - - // Apply or filters - for (let i = 0; i < dbSpace.orFilters.length; i++) { - const of_ = dbSpace.orFilters[i] - - if (of_.kind === 'string') { - // Already parsed (once) and translated by `toDbSpace`. - const parsed = [...of_.conditions] - - for (let j = 0; j < parsed.length; j++) { - const sub = substituteOrValue(orStringConditionMap, i, j, parsed[j]) - if (sub) { - parsed[j] = { ...parsed[j], value: sub.value } - } - } - - // Rebuild whenever a condition REFERENCES an encrypted column — not - // merely when a value was encrypted. An `is`/null operand on an - // encrypted column encrypts nothing, so keying on "was a value - // substituted" would send that condition down the verbatim path below - // and forward the caller's JS property name to a DB that only knows the - // column's real name. `toDbSpace` has already translated `parsed`. - const referencesEncrypted = parsed.some((c) => - isEncryptedColumn(c.column, this.encryptedColumnNames), - ) - - if (referencesEncrypted) { - q = q.or(rebuildOrString(parsed), { - referencedTable: of_.referencedTable, - }) - } else { - // Every condition names a plaintext column, whose property name IS - // its DB name — nothing to map. Forward the caller's ORIGINAL string - // byte-for-byte: relied on for nested `and()` and quoted values that - // `parseOrString`/`rebuildOrString` cannot round-trip. - q = q.or(of_.original as DbFilterString, { - referencedTable: of_.referencedTable, - }) - } - } else { - // Structured: convert to string - const conditions = of_.conditions.map((cond, j) => { - const sub = substituteOrValue(orStructuredConditionMap, i, j, cond) - return sub ? { ...cond, value: sub.value } : cond - }) - - q = q.or(rebuildOrString(conditions)) - } - } - - // Apply raw filters - for (let i = 0; i < dbSpace.rawFilters.length; i++) { - const rf = dbSpace.rawFilters[i] - - // An encrypted `in` list was encrypted element-wise; reassemble it into - // the quoted PostgREST list literal, exactly as the `not` path does. A - // plaintext column keeps its operand untouched. - if ( - rf.operator === 'in' && - Array.isArray(rf.value) && - isEncryptedColumn(rf.column, this.encryptedColumnNames) - ) { - const values = rf.value.map((v, j) => - rawInMap.has(`${i}:${j}`) ? rawInMap.get(`${i}:${j}`) : v, - ) - q = q.filter(rf.column, rf.operator, formatInListOperand(values)) - continue - } - - const value = rawValueMap.has(i) ? rawValueMap.get(i) : rf.value - q = q.filter(rf.column, rf.operator, value) - } - - return q - } - - // --------------------------------------------------------------------------- - // Dialect seams for native `eql_v3.*` domain columns. - // --------------------------------------------------------------------------- - - /** Resolve a JS property name to its DB column name. `Object.hasOwn` guards - * the inherited-member hazard described on {@link EncryptedTable.buildColumnKeyMap}. */ - private dbNameFor(name: string): string { - return Object.hasOwn(this.propToDb, name) ? this.propToDb[name] : name - } - - /** - * Map a filter's column name to the DB column name PostgREST must see — - * resolving a JS property name to its DB name. - * - * This is the ONLY place a {@link DbName} is minted. The - * {@link SupabaseQueryBuilder} seam accepts nothing else, so every column - * name reaching PostgREST must pass through here. - */ - protected filterColumnName(column: string): DbName { - return this.dbNameFor(column) as DbName - } - - /** - * Resolve the column names carried by a mutation's options. `onConflict` is a - * comma-separated column list, so it needs the same property→DB mapping as a - * filter. Returns the original object when nothing changed. - */ - protected resolveMutationOptions< - O extends { onConflict?: string } | undefined, - >(options: O): DbMutationOptions | undefined { - if (!options?.onConflict) return options as DbMutationOptions | undefined - const mapped = options.onConflict - .split(',') - .map((column) => this.filterColumnName(column.trim())) - .join(',') as DbConflictList - return ( - mapped === options.onConflict - ? options - : { ...options, onConflict: mapped } - ) as DbMutationOptions - } - /** * `ORDER BY` on an OPE-backed column is supported; on every other encrypted * column it is rejected. @@ -1810,17 +747,17 @@ export class EncryptedQueryBuilderImpl< * - neither → reject. Storage-only, equality-only and match-only columns * carry no ordering term to sort by. * - * A column absent from {@link v3Columns} is a plaintext passthrough and orders + * A column with no encrypted builder is a plaintext passthrough and orders * normally. This runtime guard is the only protection the untyped * (no-`schemas`) surface has. */ - protected validateTransforms(): void { + private validateTransforms(): void { for (const t of this.transforms) { if (t.kind !== 'order') continue - const column = this.v3Columns[t.column] + const column = this.columns.encryptedColumn(t.column) if (!column) continue - const indexes = this.columnSchemas[column.getName()]?.indexes + const indexes = this.columns.schemaFor(column.getName())?.indexes if (indexes?.ope) continue const reason = indexes?.ore @@ -1835,328 +772,18 @@ export class EncryptedQueryBuilderImpl< } } - /** - * Resolve a raw `.filter()` operator to the capability it exercises. A - * supported v3 operand is a full storage envelope, so `queryType` never - * selects a narrowing — it only tells {@link assertTermQueryable} which - * capability to demand of the column. - * - * Unknown operators throw rather than silently defaulting to equality, which - * would encrypt a term the column may not even be able to compare. - */ - protected queryTypeForRawOp(operator: string): QueryTypeName { - switch (operator) { - case 'cs': - return 'freeTextSearch' - case 'gt': - case 'gte': - case 'lt': - case 'lte': - return 'orderAndRange' - case 'eq': - case 'neq': - case 'in': - case 'is': - return 'equality' - default: - throw new Error( - `[supabase v3]: unsupported raw filter operator "${operator}" on an encrypted column`, - ) - } - } - - /** - * Apply an `in` filter. - * - * A plaintext list goes to postgrest-js's `in()`, which quotes elements that - * contain `,()`. An ENCRYPTED list cannot: every element is a - * `JSON.stringify`d envelope, and `in()` wraps it in `"…"` without escaping - * the quotes inside it, so PostgREST terminates the value at the envelope's - * first `"`. Emit the operand ourselves and hand it to `filter()`, which - * forwards it verbatim. - */ - protected applyInFilter( - q: SupabaseQueryBuilder, - column: DbName, - values: unknown[], - wasEncrypted: boolean, - ): SupabaseQueryBuilder { - if (!wasEncrypted) return q.in(column, values) - return q.filter(column, 'in', formatInListOperand(values)) - } - - /** - * Apply a `like`/`ilike` filter. On an encrypted column `like`/`ilike` were - * rewritten to `matches` at record time, so a `like`/`ilike` pending filter - * only ever names a plaintext column, which keeps real SQL LIKE. - */ - protected applyPatternFilter( - q: SupabaseQueryBuilder, - column: DbName, - op: 'like' | 'ilike', - value: unknown, - _wasEncrypted: boolean, - ): SupabaseQueryBuilder { - return op === 'like' - ? q.like(column, value as string) - : q.ilike(column, value as string) - } - - /** - * Apply a `contains` filter. On a plaintext column this is PostgREST's native - * jsonb/array containment. On an encrypted column `cs` resolves to the `@>` - * operator the EQL bundle declares on the domain, backed by `eql_v3.matches` - * (bloom-filter containment) — and the operand is the full storage envelope, - * already `JSON.stringify`d, emitted via `filter(col, 'cs', json)` rather than - * `q.contains` (postgrest-js's `contains` re-serializes a non-string operand). - * - * A structured plaintext operand is serialized here rather than by - * postgrest-js, which joins array elements on `,` without quoting them — so - * `['with,comma']` would reach Postgres as two elements. Scalars keep the - * native path. - */ - protected applyContainsFilter( - q: SupabaseQueryBuilder, - column: DbName, - value: unknown, - wasEncrypted: boolean, - ): SupabaseQueryBuilder { - if (wasEncrypted) { - this.assertPostgrestCanQueryEncryptedOperator('filter', column) - return q.filter(column, 'cs', value) - } - const literal = formatContainmentOperand(value) - return literal !== null - ? q.filter(column, 'cs', literal) - : q.contains(column, value) - } - - /** - * The CipherStash query type for an `.or()` condition's operator on an - * encrypted column. String-form conditions carry raw PostgREST operators - * (`cs`), which are not {@link FilterOp}s. - */ - protected queryTypeForOrOp(op: FilterOp): QueryTypeName { - if (op === 'matches') return 'freeTextSearch' - // Structured conditions may carry the `contains` METHOD spelling (the wire - // token becomes `cs` in rebuildOrString). It maps to the same capability - // gate as `cs`; on a JSON column the term resolver then re-types it to - // searchableJson and validates the operand. selectorNe's IS-NULL-inclusive - // or-form relies on this arm. - if (op === 'contains') return 'freeTextSearch' - return this.queryTypeForRawOp(op) - } - - /** - * The PostgREST operator to use for a `.not()` filter. Every {@link FilterOp} - * except `contains` spells the same as its PostgREST operator; `contains` is - * handled before this is reached, because it also needs its operand rewritten. - */ - protected notFilterOperator(op: FilterOp, _wasEncrypted: boolean): string { - return op - } - - /** - * Post-process a decrypted result row: rebuild `Date` values from the - * encrypt-config `cast_as` (date/timestamp), mirroring the typed v3 client's - * decrypt-model path. - */ - protected postprocessDecryptedRow( - row: Record, - ): Record { - // Every key an encrypted column can appear under: the keys this select - // actually produces (including caller-chosen aliases like `ts:createdAt`), - // plus the static property and DB names as a fallback for paths that record - // no select. Aliases win. Derived here from `this.selectColumns` (the row in - // hand) rather than cached from `buildSelectString`, so a reused builder can - // never postprocess a row with a previous operation's stale select map. - const keyToDb: Record = Object.assign( - Object.create(null), - this.selectColumns === null - ? undefined - : selectKeyToDbV3(this.selectColumns, this.propToDb), - ) - for (const [property, dbName] of Object.entries(this.propToDb)) { - keyToDb[property] ??= dbName - keyToDb[dbName] ??= dbName - } - - const out: Record = { ...row } - for (const [key, dbName] of Object.entries(keyToDb)) { - const castAs = this.columnSchemas[dbName]?.cast_as - if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue - const value = out[key] - if (value == null || value instanceof Date) continue - if (typeof value === 'string' || typeof value === 'number') { - out[key] = new Date(value) - } - } - return out - } - - // --------------------------------------------------------------------------- - // Step 5: Decrypt results - // --------------------------------------------------------------------------- - - protected async decryptResults( - result: RawSupabaseResult, - ): Promise> { - // If there's an error from Supabase, pass it through - if (result.error) { - return { - data: null, - error: { - message: result.error.message, - details: result.error.details, - hint: result.error.hint, - code: result.error.code, - }, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - // No data to decrypt - if (result.data === null || result.data === undefined) { - return { - data: null, - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - // Determine if we need to decrypt - const hasSelect = this.selectColumns !== null - const hasMutationWithReturning = this.mutation !== null && hasSelect - - if (!hasSelect && !hasMutationWithReturning) { - // No select means no data to decrypt (e.g., insert without .select()) - return { - data: result.data as T[], - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - // Decrypt based on result mode - if (this.resultMode === 'single' || this.resultMode === 'maybeSingle') { - if (result.data === null) { - return { - data: null, - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - // Single result — decrypt one model - const baseDecryptOp = this.encryptionClient.decryptModel( - result.data as Record, - ) - const decryptOp = this.lockContext - ? baseDecryptOp.withLockContext(this.lockContext) - : baseDecryptOp - if (this.auditConfig) decryptOp.audit(this.auditConfig) - - const decrypted = await decryptOp - if (decrypted.failure) { - logger.error( - `Supabase: failed to decrypt model for table "${this.tableName}"`, - ) - - throw new EncryptionFailedError( - `Failed to decrypt model: ${decrypted.failure.message}`, - decrypted.failure, - ) - } - - return { - data: this.postprocessDecryptedRow( - decrypted.data as Record, - ) as unknown as T[], - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - // Array result — bulk decrypt - const dataArray = result.data as Record[] - if (dataArray.length === 0) { - return { - data: [] as unknown as T[], - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - - const baseBulkDecryptOp = this.encryptionClient.bulkDecryptModels(dataArray) - const bulkDecryptOp = this.lockContext - ? baseBulkDecryptOp.withLockContext(this.lockContext) - : baseBulkDecryptOp - if (this.auditConfig) bulkDecryptOp.audit(this.auditConfig) - - const decrypted = await bulkDecryptOp - if (decrypted.failure) { - logger.error( - `Supabase: failed to decrypt models for table "${this.tableName}"`, - ) - - throw new EncryptionFailedError( - `Failed to decrypt models: ${decrypted.failure.message}`, - decrypted.failure, - ) - } - - return { - data: decrypted.data.map((row) => - this.postprocessDecryptedRow(row as Record), - ) as unknown as T[], - error: null, - count: result.count ?? null, - status: result.status, - statusText: result.statusText, - } - } - // --------------------------------------------------------------------------- // Helpers // --------------------------------------------------------------------------- - protected getColumnMap(): Record { - return this.v3Columns as unknown as Record - } - - /** Warn once per (op, column) that a `like`/`ilike` was delegated to `matches`. */ - private static readonly warnedLikeDelegation = new Set() - - /** True when `column` is one of this table's encrypted v3 columns. */ - private isEncryptedV3Column(column: string): boolean { - return Boolean(this.v3Columns[column]) - } - - /** True when `column` is an encrypted `types.Json` document column. */ - private isSearchableJsonColumn(column: string): boolean { - const builder: V3ColumnLike | undefined = this.v3Columns[column] - return Boolean(builder?.getQueryCapabilities().searchableJson) - } - - private assertPostgrestCanQueryEncryptedOperator( + private assertPostgrestCanQueryEncrypted( method: string, column: string, ): void { - if (!this.queryDomainsRequired) return - throw new Error( - `[supabase v3]: ${method}() on encrypted column "${column}" is unavailable with EQL 3.0.2+: the SQL operator requires an eql_v3.query_* cast that PostgREST cannot express. Use the Drizzle or Prisma Next adapter, or a scoped SQL/RPC function.`, + assertPostgrestCanQueryEncryptedOperator( + this.queryDomainsRequired, + method, + column, ) } @@ -2172,7 +799,7 @@ export class EncryptedQueryBuilderImpl< path: string, value: unknown, ): Record { - if (!this.isSearchableJsonColumn(column)) { + if (!this.columns.isSearchableJsonColumn(column)) { throw new Error( `[supabase v3]: ${method}() requires an encrypted JSON (types.Json) column; "${column}" is not one.`, ) @@ -2225,8 +852,8 @@ export class EncryptedQueryBuilderImpl< ) } const key = `${op}:${column}` - if (!EncryptedQueryBuilderImpl.warnedLikeDelegation.has(key)) { - EncryptedQueryBuilderImpl.warnedLikeDelegation.add(key) + if (!warnedLikeDelegation.has(key)) { + warnedLikeDelegation.add(key) logger.warn( `[supabase v3]: "${op}" on encrypted column "${column}" is delegated to matches() (fuzzy bloom token search). Results are APPROXIMATE — case-insensitive, one-sided (may false-positive), and wildcards/anchoring are not honored. Call matches() directly to make this explicit.`, ) @@ -2234,98 +861,3 @@ export class EncryptedQueryBuilderImpl< return needle } } - -// --------------------------------------------------------------------------- -// Internal types -// --------------------------------------------------------------------------- - -type TermMapping = - | { source: 'filter'; filterIndex: number; inIndex?: number } - | { source: 'match'; matchIndex: number; column: string } - | { source: 'not'; notIndex: number; inIndex?: number } - | { source: 'raw'; rawIndex: number; inIndex?: number } - | { - source: 'or-string' - orIndex: number - conditionIndex: number - inIndex?: number - } - | { - source: 'or-structured' - orIndex: number - conditionIndex: number - inIndex?: number - } - -type EncryptedFilterState = { - // `EncryptedQueryResult[]`, not `unknown[]` — `encryptCollectedTerms` returns - // that type, and typing the field to match is what lets the restored envelope - // type reach the use site (`encryptedValues[i]`) instead of widening back to - // `unknown` at this boundary. - encryptedValues: EncryptedQueryResult[] - termMap: TermMapping[] -} - -/** Key an `.or()` condition, or one element of its `in` list. */ -function orKey(mapping: { - orIndex: number - conditionIndex: number - inIndex?: number -}): string { - const base = `${mapping.orIndex}:${mapping.conditionIndex}` - return mapping.inIndex === undefined ? base : `${base}:${mapping.inIndex}` -} - -/** - * Substitute encrypted operands back into one `.or()` condition, returning - * `undefined` when nothing was encrypted for it. - * - * An `in` list is reconstructed element-by-element so `formatOrValue` re-emits - * the `(a,b)` list form. Substituting the array as a single value would collapse - * it to one ciphertext that matches nothing. - */ -function substituteOrValue( - map: Map, - orIndex: number, - conditionIndex: number, - cond: { op: FilterOp; value: unknown }, -): { value: unknown } | undefined { - const whole = orKey({ orIndex, conditionIndex }) - if (map.has(whole)) return { value: map.get(whole) } - - if (cond.op === 'in' && Array.isArray(cond.value)) { - let substituted = false - const value = cond.value.map((element, inIndex) => { - const key = orKey({ orIndex, conditionIndex, inIndex }) - if (!map.has(key)) return element - substituted = true - return map.get(key) - }) - if (substituted) return { value } - } - - return undefined -} - -type RawSupabaseResult = { - data: unknown - error: { - message: string - details?: string - hint?: string - code?: string - } | null - count?: number | null - status: number - statusText: string -} - -export class EncryptionFailedError extends Error { - public encryptionError: EncryptionError - - constructor(message: string, encryptionError: EncryptionError) { - super(message) - this.name = 'EncryptionFailedError' - this.encryptionError = encryptionError - } -} diff --git a/packages/stack-supabase/src/query-dbspace.ts b/packages/stack-supabase/src/query-dbspace.ts new file mode 100644 index 000000000..ae68df562 --- /dev/null +++ b/packages/stack-supabase/src/query-dbspace.ts @@ -0,0 +1,141 @@ +import type { ColumnMap } from './column-map' +import { parseOrString } from './helpers' +import type { + DbConflictList, + DbMutationOp, + DbMutationOptions, + DbPendingOrCondition, + DbPendingOrFilter, + DbQuerySpace, + DbTransformOp, + MutationOp, + PendingOrCondition, + PendingOrFilter, + RecordedOps, + TransformOp, +} from './types' + +/** + * Resolve the column names carried by a mutation's options. `onConflict` is a + * comma-separated column list, so it needs the same property→DB mapping as a + * filter. Returns the original object when nothing changed. + */ +export function resolveMutationOptions< + O extends { onConflict?: string } | undefined, +>(options: O, columns: ColumnMap): DbMutationOptions | undefined { + if (!options?.onConflict) return options as DbMutationOptions | undefined + const mapped = options.onConflict + .split(',') + .map((column) => columns.filterColumnName(column.trim())) + .join(',') as DbConflictList + return ( + mapped === options.onConflict ? options : { ...options, onConflict: mapped } + ) as DbMutationOptions +} + +/** Column names only. Which conditions were encrypted is never decided here: + * it stays derived at apply time from the substitution maps, so this pass + * never has to agree with the encryption predicate. The operator token is + * settled later still, in `rebuildOrString`, where `contains` becomes `cs` + * for encrypted and plaintext conditions alike. */ +function orFilterToDbSpace( + of_: PendingOrFilter, + columns: ColumnMap, +): DbPendingOrFilter { + const toDbCondition = (c: PendingOrCondition): DbPendingOrCondition => ({ + ...c, + column: columns.filterColumnName(c.column), + }) + + if (of_.kind === 'string') { + return { + kind: 'string', + original: of_.value, + conditions: parseOrString(of_.value).map(toDbCondition), + referencedTable: of_.referencedTable, + } + } + return { kind: 'structured', conditions: of_.conditions.map(toDbCondition) } +} + +function transformToDbSpace(t: TransformOp, columns: ColumnMap): DbTransformOp { + switch (t.kind) { + case 'order': + return { ...t, column: columns.orderColumnName(t.column) } + // `returns` is in the union but never pushed (`returns()` is a cast). + case 'limit': + case 'range': + case 'single': + case 'maybeSingle': + case 'csv': + case 'abortSignal': + case 'throwOnError': + case 'returns': + return t + default: { + const exhaustive: never = t + return exhaustive + } + } +} + +function mutationToDbSpace(m: MutationOp, columns: ColumnMap): DbMutationOp { + switch (m.kind) { + case 'insert': + case 'upsert': + return { ...m, options: resolveMutationOptions(m.options, columns) } + case 'update': + case 'delete': + return m // options carry no column names + default: { + const exhaustive: never = m + return exhaustive + } + } +} + +/** + * Translate every recorded column name from JS property space into DB space, + * once. Downstream (`encryptFilterValues`, `applyFilters`, + * `buildAndExecuteQuery`) consumes only the branded result, so a column can + * no longer reach PostgREST untranslated — that is a compile error. + * + * Total: `filterColumnName`, `parseOrString`, and `resolveMutationOptions` + * never throw, so this introduces no new early-throw point and cannot perturb + * the order in which capability errors surface. + * + * Safe to run BEFORE encryption: the column map is keyed by both property and + * DB name, so column lookup resolves identically either side of the + * translation, and `tableColumns[prop]` is the very same builder object as + * `tableColumns[db]`. + */ +export function toDbSpace( + recorded: RecordedOps, + columns: ColumnMap, +): DbQuerySpace { + return { + filters: recorded.filters.map((f) => ({ + ...f, + column: columns.filterColumnName(f.column), + })), + matchFilters: recorded.matchFilters.map((mf) => ({ + entries: Object.entries(mf.query).map(([column, value]) => ({ + column: columns.filterColumnName(column), + value, + })), + })), + notFilters: recorded.notFilters.map((nf) => ({ + ...nf, + column: columns.filterColumnName(nf.column), + })), + rawFilters: recorded.rawFilters.map((rf) => ({ + ...rf, + column: columns.filterColumnName(rf.column), + })), + orFilters: recorded.orFilters.map((of_) => orFilterToDbSpace(of_, columns)), + transforms: recorded.transforms.map((t) => transformToDbSpace(t, columns)), + mutation: recorded.mutation + ? mutationToDbSpace(recorded.mutation, columns) + : null, + } +} diff --git a/packages/stack-supabase/src/query-encrypt.ts b/packages/stack-supabase/src/query-encrypt.ts new file mode 100644 index 000000000..56ac8d653 --- /dev/null +++ b/packages/stack-supabase/src/query-encrypt.ts @@ -0,0 +1,659 @@ +import type { JsPlaintext } from '@cipherstash/protect-ffi' +import type { AuditConfig } from '@cipherstash/stack/adapter-kit' +import { logger, matchNeedleError } from '@cipherstash/stack/adapter-kit' +import type { EncryptionClient } from '@cipherstash/stack/encryption' +import type { AnyV3Table } from '@cipherstash/stack/eql/v3' +import { + type EncryptionError, + EncryptionErrorTypes, +} from '@cipherstash/stack/errors' +import type { LockContextInput } from '@cipherstash/stack/identity' +import type { + Encrypted, + EncryptedQueryResult, + QueryTypeName, + ScalarQueryTerm, +} from '@cipherstash/stack/types' +import type { ColumnMap, V3ColumnLike } from './column-map' +import { + isEncryptableTerm, + isEncryptedColumn, + mapFilterOpToQueryType, +} from './helpers' +import type { DbQuerySpace, FilterOp } from './types' + +export class EncryptionFailedError extends Error { + public encryptionError: EncryptionError + + constructor(message: string, encryptionError: EncryptionError) { + super(message) + this.name = 'EncryptionFailedError' + this.encryptionError = encryptionError + } +} + +export type TermMapping = + | { source: 'filter'; filterIndex: number; inIndex?: number } + | { source: 'match'; matchIndex: number; column: string } + | { source: 'not'; notIndex: number; inIndex?: number } + | { source: 'raw'; rawIndex: number; inIndex?: number } + | { + source: 'or-string' + orIndex: number + conditionIndex: number + inIndex?: number + } + | { + source: 'or-structured' + orIndex: number + conditionIndex: number + inIndex?: number + } + +export type EncryptedFilterState = { + // `EncryptedQueryResult[]`, not `unknown[]` — `encryptCollectedTerms` returns + // that type, and typing the field to match is what lets the restored envelope + // type reach the use site (`encryptedValues[i]`) instead of widening back to + // `unknown` at this boundary. + encryptedValues: EncryptedQueryResult[] + termMap: TermMapping[] +} + +/** + * Everything an encryption step needs from the builder. Assembled per + * `execute()`, so `lockContext`/`auditConfig` are read at execution time — not + * captured when the builder was constructed. + */ +export type EncryptionContext = { + tableName: string + table: AnyV3Table + encryptionClient: EncryptionClient + lockContext: LockContextInput | null + auditConfig: AuditConfig | null + columns: ColumnMap + /** EQL 3.0.2+ requires query-domain casts PostgREST cannot express. */ + queryDomainsRequired: boolean +} + +/** + * Apply the builder's lock context and audit config to a pending operation. + * + * Every encrypt/decrypt crossing in this adapter goes through the same three + * steps, and they must stay identical: an operation that silently skipped the + * lock context would encrypt under the wrong data key. + */ +export function withOpContext( + baseOp: PromiseLike & { + withLockContext( + lockContext: LockContextInput, + ): PromiseLike & { audit(config: AuditConfig): unknown } + audit(config: AuditConfig): unknown + }, + ctx: Pick, +): PromiseLike { + const op = ctx.lockContext ? baseOp.withLockContext(ctx.lockContext) : baseOp + if (ctx.auditConfig) op.audit(ctx.auditConfig) + return op +} + +/** + * EQL 3.0.2 removed the storage/jsonb escape hatch for free-text and JSON + * operators: those now require typed query-domain operands PostgREST cannot + * express. Fail before encryption, so a decryptable storage envelope never + * enters a GET URL. + */ +export function assertPostgrestCanQueryEncryptedOperator( + queryDomainsRequired: boolean, + method: string, + column: string, +): void { + if (!queryDomainsRequired) return + throw new Error( + `[supabase v3]: ${method}() on encrypted column "${column}" is unavailable with EQL 3.0.2+: the SQL operator requires an eql_v3.query_* cast that PostgREST cannot express. Use the Drizzle or Prisma Next adapter, or a scoped SQL/RPC function.`, + ) +} + +/** + * Validate an encrypted-JSON containment operand: a NON-EMPTY plain object or a + * non-empty array. Everything else is rejected with an actionable steer: + * + * - Scalars/strings: the caller meant free-text (`matches` on a text column) or + * a selector — a raw JSON string is NOT parsed, by design (parsing would make + * `'{"a":1}'` and `{a:1}` silently different queries on other surfaces). + * - Non-plain objects (`Date`, `Map`, `RegExp`, class instances): these JSON- + * serialize to scalars or `{}` — not the sub-document the caller believes. + * - `{}` and `[]`: jsonb containment holds for EVERY document (`doc @> '{}'`), + * so an accidentally-empty needle would silently return (and decrypt) the + * whole table. The Drizzle adapter rejects the same needle for the same + * reason — the two first-party adapters must agree that this is an error. + */ +export function assertJsonContainmentOperand( + column: string, + value: unknown, +): void { + const isPlainObject = + value !== null && + typeof value === 'object' && + !Array.isArray(value) && + (Object.getPrototypeOf(value) === Object.prototype || + Object.getPrototypeOf(value) === null) + if (!isPlainObject && !Array.isArray(value)) { + // Array.isArray is false on this branch by construction, so the label only + // distinguishes null / non-plain object / scalar. + const got = + value === null + ? 'null' + : typeof value === 'object' + ? (value as object).constructor?.name || 'a non-plain object' + : typeof value + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" takes a sub-document (plain object or array) to match, got ${got}.`, + ) + } + const empty = Array.isArray(value) + ? value.length === 0 + : Object.keys(value as object).length === 0 + if (empty) { + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" cannot take an empty ${Array.isArray(value) ? 'array' : 'object'} needle: it matches every row. Pass a non-empty sub-document, or omit the predicate to select all rows.`, + ) + } +} + +/** + * Resolve a raw `.filter()` operator to the capability it exercises. A + * supported v3 operand is a full storage envelope, so `queryType` never + * selects a narrowing — it only tells {@link assertTermQueryable} which + * capability to demand of the column. + * + * Unknown operators throw rather than silently defaulting to equality, which + * would encrypt a term the column may not even be able to compare. + */ +export function queryTypeForRawOp(operator: string): QueryTypeName { + switch (operator) { + case 'cs': + return 'freeTextSearch' + case 'gt': + case 'gte': + case 'lt': + case 'lte': + return 'orderAndRange' + case 'eq': + case 'neq': + case 'in': + case 'is': + return 'equality' + default: + throw new Error( + `[supabase v3]: unsupported raw filter operator "${operator}" on an encrypted column`, + ) + } +} + +/** + * The CipherStash query type for an `.or()` condition's operator on an + * encrypted column. String-form conditions carry raw PostgREST operators + * (`cs`), which are not {@link FilterOp}s. + */ +export function queryTypeForOrOp(op: FilterOp): QueryTypeName { + if (op === 'matches') return 'freeTextSearch' + // Structured conditions may carry the `contains` METHOD spelling (the wire + // token becomes `cs` in rebuildOrString). It maps to the same capability + // gate as `cs`; on a JSON column the term resolver then re-types it to + // searchableJson and validates the operand. selectorNe's IS-NULL-inclusive + // or-form relies on this arm. + if (op === 'contains') return 'freeTextSearch' + return queryTypeForRawOp(op) +} + +function encryptionFailure( + tableName: string, + message: string, + cause?: EncryptionError, +): never { + logger.error( + `Supabase: failed to encrypt query terms for table "${tableName}"`, + ) + // Most callers pass the operation's own `EncryptionError`; the contract- + // violation cases (bulk length mismatch, null envelope) have none, so + // synthesize one — a broken query encryption is still an encryption failure, + // and callers branch on `error.encryptionError` regardless. + throw new EncryptionFailedError( + `Failed to encrypt query terms: ${message}`, + cause ?? { type: EncryptionErrorTypes.EncryptionError, message }, + ) +} + +// --------------------------------------------------------------------------- +// Step 3: Encrypt filter values +// --------------------------------------------------------------------------- + +export async function encryptFilterValues( + dbSpace: DbQuerySpace, + ctx: EncryptionContext, +): Promise { + // Collect all terms that need encryption + const terms: ScalarQueryTerm[] = [] + const termMap: TermMapping[] = [] + + const tableColumns = ctx.columns.queryColumnMap() + const encryptedColumnNames = ctx.columns.encryptedColumnNames + + const pushTerm = ( + value: JsPlaintext, + column: ScalarQueryTerm['column'], + queryType: QueryTypeName, + mapping: TermMapping, + ) => { + terms.push({ + value, + column, + table: ctx.table, + queryType, + returnType: 'composite-literal', + }) + termMap.push(mapping) + } + + /** + * Collect one term per element of an `in`-list operand. + * + * Element-wise is the only correct encoding: encrypting the array as ONE + * value collapses `(a,b)` into a single ciphertext that matches nothing. A + * null element is SQL NULL and passes through unencrypted; the applier + * restores it by index, which is why the mapping carries `inIndex`. + * + * Shared by the regular-`in`, `not(…,'in',…)` and or-condition paths. They + * drifted apart once already — the `not` path went unfixed while the other + * two encrypted element-wise — so they are kept in lockstep here rather than + * spelled out three times. + */ + const collectInListTerms = ( + op: FilterOp, + values: readonly unknown[], + column: ScalarQueryTerm['column'], + queryType: QueryTypeName, + mappingFor: (inIndex: number) => TermMapping, + ) => { + for (let j = 0; j < values.length; j++) { + if (!isEncryptableTerm(op, values[j])) continue + pushTerm(values[j] as JsPlaintext, column, queryType, mappingFor(j)) + } + } + + // Regular filters + for (let i = 0; i < dbSpace.filters.length; i++) { + const f = dbSpace.filters[i] + if (!isEncryptedColumn(f.column, encryptedColumnNames)) continue + + const column = tableColumns[f.column] + if (!column) continue + + if (f.op === 'in' && Array.isArray(f.value)) { + collectInListTerms( + f.op, + f.value, + column, + mapFilterOpToQueryType(f.op), + (inIndex) => ({ source: 'filter', filterIndex: i, inIndex }), + ) + } else if (!isEncryptableTerm(f.op, f.value)) { + // `is` predicate or null operand — forwarded unencrypted. + } else { + pushTerm(f.value as JsPlaintext, column, mapFilterOpToQueryType(f.op), { + source: 'filter', + filterIndex: i, + }) + } + } + + // Match filters + for (let i = 0; i < dbSpace.matchFilters.length; i++) { + const mf = dbSpace.matchFilters[i] + for (const { column: colName, value } of mf.entries) { + if (!isEncryptedColumn(colName, encryptedColumnNames)) continue + // `match` carries no operator; equality is implied. + if (!isEncryptableTerm('eq', value)) continue + const column = tableColumns[colName] + if (!column) continue + + pushTerm(value as JsPlaintext, column, 'equality', { + source: 'match', + matchIndex: i, + column: colName, + }) + } + } + + // Not filters + for (let i = 0; i < dbSpace.notFilters.length; i++) { + const nf = dbSpace.notFilters[i] + if (!isEncryptedColumn(nf.column, encryptedColumnNames)) continue + if (!isEncryptableTerm(nf.op, nf.value)) continue + const column = tableColumns[nf.column] + if (!column) continue + + if (nf.op === 'in') { + // A PostgREST list literal (`'(a,b)'`) cannot be encrypted element-wise, + // and encrypting it whole matches nothing. Refuse it rather than emit a + // filter that silently returns no rows. + if (!Array.isArray(nf.value)) { + throw new Error( + `not("${nf.column}", "in", …) on an encrypted column requires an array of values, ` + + `not a PostgREST list literal — each element must be encrypted separately`, + ) + } + collectInListTerms( + nf.op, + nf.value, + column, + mapFilterOpToQueryType(nf.op), + (inIndex) => ({ source: 'not', notIndex: i, inIndex }), + ) + continue + } + + pushTerm(nf.value as JsPlaintext, column, mapFilterOpToQueryType(nf.op), { + source: 'not', + notIndex: i, + }) + } + + // Or filters — conditions were parsed once, in `toDbSpace`. The string and + // structured forms differ only in their `source` tag; the encryption rules, + // including the `in`-list split below, are identical. + for (let i = 0; i < dbSpace.orFilters.length; i++) { + const of_ = dbSpace.orFilters[i] + const source = of_.kind === 'string' ? 'or-string' : 'or-structured' + + for (let j = 0; j < of_.conditions.length; j++) { + const cond = of_.conditions[j] + if (!isEncryptedColumn(cond.column, encryptedColumnNames)) continue + const column = tableColumns[cond.column] + if (!column) continue + + // `queryTypeForOrOp`, not `mapFilterOpToQueryType`: an or-condition may + // carry a raw PostgREST operator (`cs`), which is not a `FilterOp`. + const queryType = queryTypeForOrOp(cond.op) + const mappingFor = (inIndex?: number): TermMapping => ({ + source, + orIndex: i, + conditionIndex: j, + inIndex, + }) + + if (cond.op === 'in' && Array.isArray(cond.value)) { + collectInListTerms(cond.op, cond.value, column, queryType, mappingFor) + continue + } + + if (!isEncryptableTerm(cond.op, cond.value)) continue + pushTerm(cond.value as JsPlaintext, column, queryType, mappingFor()) + } + } + + // Raw filters + for (let i = 0; i < dbSpace.rawFilters.length; i++) { + const rf = dbSpace.rawFilters[i] + if (!isEncryptedColumn(rf.column, encryptedColumnNames)) continue + const column = tableColumns[rf.column] + if (!column) continue + + if (rf.operator === 'in') { + // Same contract as the `not(…, 'in', …)` path: a PostgREST list literal + // (`'("a","b")'`) cannot be encrypted element-wise, and encrypting it + // whole matches nothing. Refuse it rather than emit a filter that + // silently returns no rows. + if (!Array.isArray(rf.value)) { + throw new Error( + `filter("${rf.column}", "in", …) on an encrypted column requires an array of values, ` + + `not a PostgREST list literal — each element must be encrypted separately`, + ) + } + collectInListTerms( + 'in', + rf.value, + column, + queryTypeForRawOp(rf.operator), + (inIndex) => ({ source: 'raw', rawIndex: i, inIndex }), + ) + continue + } + + if (!isEncryptableTerm(rf.operator, rf.value)) continue + + pushTerm(rf.value as JsPlaintext, column, queryTypeForRawOp(rf.operator), { + source: 'raw', + rawIndex: i, + }) + } + + if (terms.length === 0) { + return { encryptedValues: [], termMap: [] } + } + + const encryptedValues = await encryptCollectedTerms(terms, ctx) + return { encryptedValues, termMap } +} + +/** + * Encrypt every filter operand as a full storage envelope, serialized to jsonb + * text for the PostgREST filter value. + * + * Terms are grouped by column and each group takes ONE `bulkEncrypt` crossing. + * `in(col, [a, b, c])` collects one term per element (the list must never be + * encrypted whole), so encrypting per term spent N ZeroKMS/FFI round-trips + * where one would do. `bulkEncrypt` carries a single `{table, column}` for the + * whole payload, so the grouping is mandatory, not an optimisation: one bulk + * call over a mixed-column term array would stamp one column onto every + * plaintext. Results are scattered back onto the terms' original indices, + * which is the contract `termMap` downstream relies on. + * + * Mirrors `eql/v3/drizzle/operators.ts` `encryptOperands` — same batching + * contract, same length assertion, same fallback. Kept separate because that + * one encrypts a single-column operand list and returns `SQL[]`, while this + * must group a multi-column term array and preserve positions. + */ +async function encryptCollectedTerms( + terms: ScalarQueryTerm[], + ctx: EncryptionContext, +): Promise { + const groups = new Map< + V3ColumnLike, + { indices: number[]; values: ScalarQueryTerm['value'][] } + >() + terms.forEach((term, index) => { + const column = assertTermQueryable(term, ctx) + const group = groups.get(column) ?? { indices: [], values: [] } + group.indices.push(index) + group.values.push(term.value) + groups.set(column, group) + }) + + const bulkEncrypt = ctx.encryptionClient.bulkEncrypt?.bind( + ctx.encryptionClient, + ) + // Each term becomes the `JSON.stringify`'d storage envelope — a `string`, + // which is one arm of `EncryptedQueryResult`. PostgREST cannot cast a filter + // value to the `eql_v3.query_` twins, so v3 sends full envelopes, + // serialized to jsonb text. + const results = new Array(terms.length) + + await Promise.all( + Array.from(groups, async ([column, { indices, values }]) => { + const encrypted = bulkEncrypt + ? await bulkEncryptGroup(bulkEncrypt, column, values, ctx) + : await encryptGroupPerTerm(column, values, ctx) + + encrypted.forEach((envelope, i) => { + results[indices[i]] = JSON.stringify(envelope) + }) + }), + ) + + return results +} + +/** + * Validate a term's query type against its column's declared capabilities. + * Pure validation: `encrypt`/`bulkEncrypt` never receive the query type. On + * EQL 3.0.2+, free-text/JSON terms are rejected before this storage-encryption + * path can place ciphertext in a GET URL. + * + * Exported for direct testing: no public call path can produce an unsupported + * `queryType` (`mapFilterOpToQueryType`, {@link queryTypeForRawOp} and + * {@link queryTypeForOrOp} are exhaustive), so that backstop is only reachable + * by calling this with a hand-built term. + */ +export function assertTermQueryable( + term: ScalarQueryTerm, + ctx: EncryptionContext, +): V3ColumnLike { + const column = term.column as unknown as V3ColumnLike + let queryType = term.queryType ?? 'equality' + const capabilities = column.getQueryCapabilities() + + // The `cs` wire operator is capability-overloaded: bloom free-text on a + // match/search TEXT column, encrypted ste_vec containment on a `types.Json` + // DOCUMENT column. Both arrive here as `freeTextSearch` (contains/matches/ + // raw `cs` all map there); resolve to the capability the column actually + // carries. The two are mutually exclusive by construction, so this can + // never reinterpret a real free-text column. + if ( + queryType === 'freeTextSearch' && + !capabilities.freeTextSearch && + capabilities.searchableJson + ) { + queryType = 'searchableJson' + } + + if ( + queryType !== 'equality' && + queryType !== 'orderAndRange' && + queryType !== 'freeTextSearch' && + queryType !== 'searchableJson' + ) { + throw new Error( + `[supabase v3]: query type "${queryType}" is not supported on EQL v3 columns`, + ) + } + + if (!capabilities[queryType]) { + throw new Error( + `[supabase v3]: column "${column.getName()}" (${column.getEqlType()}) does not support ${queryType} queries — declare the column with a domain that carries that capability`, + ) + } + + if (queryType === 'freeTextSearch' || queryType === 'searchableJson') { + // This is the common boundary for every spelling that collects an + // encrypted match/containment term: matches(), contains(), not(), raw + // filter(), and both forms of or(). Method-level checks provide earlier + // errors for the direct helpers, but cannot cover the raw filter paths on + // their own. + assertPostgrestCanQueryEncryptedOperator( + ctx.queryDomainsRequired, + 'filter', + column.getName(), + ) + } + + if (queryType === 'searchableJson') { + // THE single enforced operand boundary for encrypted-JSON containment. + // Terms reach this resolver from every spelling — contains(), raw + // .filter(col,'cs',…), not(col,'contains'|'matches',…), and .or() + // string/structured conditions — and only contains() has a method-level + // guard. Without this check a raw string (e.g. a free-text term ported + // from a text column, or an .or() condition value, which is always a + // string) would be storage-encrypted as a JSON SCALAR and silently match + // nothing; pre-#650 every such spelling failed loudly on capability. + assertJsonContainmentOperand(column.getName(), term.value) + } + + // Free-text (bloom) needle floor. A needle shorter than the tokenizer's + // token_length produces NO tokens, so `bf @> '{}'` holds for every row and + // the query would silently return (and the caller decrypt) the whole table + // — a fail-open over-exposure. Reject it up front, mirroring the Drizzle v3 + // adapter (matchNeedleError) so both first-party surfaces guard identically. + // JSON containment terms (searchableJson) are validated separately above. + if (queryType === 'freeTextSearch') { + const match = column.build().indexes?.match + const reason = match ? matchNeedleError(term.value, match) : undefined + if (reason) { + throw new Error( + `[supabase v3]: cannot search column "${column.getName()}": ${reason}`, + ) + } + } + + return column +} + +/** One FFI crossing for a column's whole operand list. */ +async function bulkEncryptGroup( + bulkEncrypt: NonNullable, + column: V3ColumnLike, + values: ScalarQueryTerm['value'][], + ctx: EncryptionContext, +): Promise> { + const result = await withOpContext( + bulkEncrypt( + values.map((plaintext) => ({ plaintext })) as never, + { + column, + table: ctx.table, + } as never, + ), + ctx, + ) + if (result.failure) + encryptionFailure(ctx.tableName, result.failure.message, result.failure) + + // `bulkEncrypt` is position-stable, so a length mismatch means the contract + // was violated. Truncating instead would silently widen an `in` predicate + // (or narrow a `not.in`) to whatever came back. `result.data` is now + // `BulkEncryptedData` — `{ id?, data: Encrypted | null }[]` — not `unknown`. + const encrypted = result.data + if (encrypted.length !== values.length) { + encryptionFailure( + ctx.tableName, + `bulk encryption returned ${encrypted.length} terms for ${values.length} values on column "${column.getName()}".`, + ) + } + return encrypted.map((term, i) => { + // `BulkEncryptedData` types the element as `Encrypted | null`. A `null` + // envelope here would be `JSON.stringify`'d to the literal string `"null"` + // and sent as the filter operand — silently matching whatever `"null"` + // encodes to rather than failing. A query term should never encrypt to a + // null envelope, so treat it as a contract violation, not a value. + if (term.data === null) { + encryptionFailure( + ctx.tableName, + `bulk encryption returned a null envelope at position ${i} for column "${column.getName()}".`, + ) + } + return term.data + }) +} + +/** Fallback for a client that predates `bulkEncrypt`. */ +async function encryptGroupPerTerm( + column: V3ColumnLike, + values: ScalarQueryTerm['value'][], + ctx: EncryptionContext, +): Promise { + return Promise.all( + values.map(async (value) => { + const result = await withOpContext( + ctx.encryptionClient.encrypt(value, { + column, + table: ctx.table, + }), + ctx, + ) + if (result.failure) { + encryptionFailure(ctx.tableName, result.failure.message, result.failure) + } + return result.data + }), + ) +} diff --git a/packages/stack-supabase/src/query-filters.ts b/packages/stack-supabase/src/query-filters.ts new file mode 100644 index 000000000..fdad1d10a --- /dev/null +++ b/packages/stack-supabase/src/query-filters.ts @@ -0,0 +1,388 @@ +import type { ColumnMap } from './column-map' +import { + formatContainmentOperand, + formatInListOperand, + isEncryptedColumn, + rebuildOrString, +} from './helpers' +import { + assertPostgrestCanQueryEncryptedOperator, + type EncryptedFilterState, +} from './query-encrypt' +import type { + DbFilterString, + DbName, + DbQuerySpace, + FilterOp, + SupabaseQueryBuilder, +} from './types' + +/** Key an `.or()` condition, or one element of its `in` list. */ +function orKey(mapping: { + orIndex: number + conditionIndex: number + inIndex?: number +}): string { + const base = `${mapping.orIndex}:${mapping.conditionIndex}` + return mapping.inIndex === undefined ? base : `${base}:${mapping.inIndex}` +} + +/** + * Substitute encrypted operands back into one `.or()` condition, returning + * `undefined` when nothing was encrypted for it. + * + * An `in` list is reconstructed element-by-element so `formatOrValue` re-emits + * the `(a,b)` list form. Substituting the array as a single value would collapse + * it to one ciphertext that matches nothing. + */ +function substituteOrValue( + map: Map, + orIndex: number, + conditionIndex: number, + cond: { op: FilterOp; value: unknown }, +): { value: unknown } | undefined { + const whole = orKey({ orIndex, conditionIndex }) + if (map.has(whole)) return { value: map.get(whole) } + + if (cond.op === 'in' && Array.isArray(cond.value)) { + let substituted = false + const value = cond.value.map((element, inIndex) => { + const key = orKey({ orIndex, conditionIndex, inIndex }) + if (!map.has(key)) return element + substituted = true + return map.get(key) + }) + if (substituted) return { value } + } + + return undefined +} + +/** + * Apply an `in` filter. + * + * A plaintext list goes to postgrest-js's `in()`, which quotes elements that + * contain `,()`. An ENCRYPTED list cannot: every element is a + * `JSON.stringify`d envelope, and `in()` wraps it in `"…"` without escaping + * the quotes inside it, so PostgREST terminates the value at the envelope's + * first `"`. Emit the operand ourselves and hand it to `filter()`, which + * forwards it verbatim. + */ +function applyInFilter( + q: SupabaseQueryBuilder, + column: DbName, + values: unknown[], + wasEncrypted: boolean, +): SupabaseQueryBuilder { + if (!wasEncrypted) return q.in(column, values) + return q.filter(column, 'in', formatInListOperand(values)) +} + +/** + * Apply a `like`/`ilike` filter. On an encrypted column `like`/`ilike` were + * rewritten to `matches` at record time, so a `like`/`ilike` pending filter + * only ever names a plaintext column, which keeps real SQL LIKE. + */ +function applyPatternFilter( + q: SupabaseQueryBuilder, + column: DbName, + op: 'like' | 'ilike', + value: unknown, +): SupabaseQueryBuilder { + return op === 'like' + ? q.like(column, value as string) + : q.ilike(column, value as string) +} + +/** + * Apply a `contains` filter. On a plaintext column this is PostgREST's native + * jsonb/array containment. On an encrypted column `cs` resolves to the `@>` + * operator the EQL bundle declares on the domain, backed by `eql_v3.matches` + * (bloom-filter containment) — and the operand is the full storage envelope, + * already `JSON.stringify`d, emitted via `filter(col, 'cs', json)` rather than + * `q.contains` (postgrest-js's `contains` re-serializes a non-string operand). + * + * A structured plaintext operand is serialized here rather than by + * postgrest-js, which joins array elements on `,` without quoting them — so + * `['with,comma']` would reach Postgres as two elements. Scalars keep the + * native path. + */ +function applyContainsFilter( + q: SupabaseQueryBuilder, + column: DbName, + value: unknown, + wasEncrypted: boolean, + queryDomainsRequired: boolean, +): SupabaseQueryBuilder { + if (wasEncrypted) { + assertPostgrestCanQueryEncryptedOperator( + queryDomainsRequired, + 'filter', + column, + ) + return q.filter(column, 'cs', value) + } + const literal = formatContainmentOperand(value) + return literal !== null + ? q.filter(column, 'cs', literal) + : q.contains(column, value) +} + +/** + * Apply every recorded filter to the real Supabase query, substituting the + * encrypted operand wherever one was produced for that position. + */ +export function applyFilters( + query: SupabaseQueryBuilder, + encryptedFilters: EncryptedFilterState, + dbSpace: DbQuerySpace, + columns: ColumnMap, + queryDomainsRequired: boolean, +): SupabaseQueryBuilder { + let q = query + const encryptedColumnNames = columns.encryptedColumnNames + + // Build lookup maps for quick access to encrypted values + const filterValueMap = new Map() + const filterInMap = new Map() // "filterIndex:inIndex" -> value + const matchValueMap = new Map() // "matchIndex:column" -> value + const notValueMap = new Map() + const notInMap = new Map() // "notIndex:inIndex" -> value + const rawValueMap = new Map() + const rawInMap = new Map() // "rawIndex:inIndex" -> value + const orStringConditionMap = new Map() // "orIndex:condIndex" -> value + const orStructuredConditionMap = new Map() + + for (let i = 0; i < encryptedFilters.termMap.length; i++) { + const mapping = encryptedFilters.termMap[i] + const encValue = encryptedFilters.encryptedValues[i] + + switch (mapping.source) { + case 'filter': + if (mapping.inIndex !== undefined) { + filterInMap.set(`${mapping.filterIndex}:${mapping.inIndex}`, encValue) + } else { + filterValueMap.set(mapping.filterIndex, encValue) + } + break + case 'match': + matchValueMap.set(`${mapping.matchIndex}:${mapping.column}`, encValue) + break + case 'not': + if (mapping.inIndex !== undefined) { + notInMap.set(`${mapping.notIndex}:${mapping.inIndex}`, encValue) + } else { + notValueMap.set(mapping.notIndex, encValue) + } + break + case 'raw': + if (mapping.inIndex !== undefined) { + rawInMap.set(`${mapping.rawIndex}:${mapping.inIndex}`, encValue) + } else { + rawValueMap.set(mapping.rawIndex, encValue) + } + break + // `inIndex` widens the key to address one element of an `in` list, so a + // whole-condition value and a per-element value never collide. + case 'or-string': + orStringConditionMap.set(orKey(mapping), encValue) + break + case 'or-structured': + orStructuredConditionMap.set(orKey(mapping), encValue) + break + } + } + + // Apply regular filters + for (let i = 0; i < dbSpace.filters.length; i++) { + const f = dbSpace.filters[i] + let value = f.value + + if (filterValueMap.has(i)) { + value = filterValueMap.get(i) + } else if (f.op === 'in' && Array.isArray(f.value)) { + // Reconstruct array with encrypted values substituted + value = f.value.map((v, j) => { + const key = `${i}:${j}` + return filterInMap.has(key) ? filterInMap.get(key) : v + }) + } + + const column = f.column + const wasEncrypted = filterValueMap.has(i) + + switch (f.op) { + case 'eq': + q = q.eq(column, value) + break + case 'neq': + q = q.neq(column, value) + break + case 'gt': + q = q.gt(column, value) + break + case 'gte': + q = q.gte(column, value) + break + case 'lt': + q = q.lt(column, value) + break + case 'lte': + q = q.lte(column, value) + break + case 'like': + case 'ilike': + q = applyPatternFilter(q, column, f.op, value) + break + // `matches` (encrypted free-text) and `contains` (plaintext / encrypted + // JSON) share the `cs`/`@>` wire operator; the operand encoding is the + // same, so both emit through the one containment applier. + case 'contains': + case 'matches': + q = applyContainsFilter( + q, + column, + value, + wasEncrypted, + queryDomainsRequired, + ) + break + case 'is': + q = q.is(column, value) + break + case 'in': + // `wasEncrypted` above is false for in-lists: their ciphertexts land + // in `filterInMap`, keyed per element. + q = applyInFilter( + q, + column, + value as unknown[], + Array.isArray(f.value) && + f.value.some((_, j) => filterInMap.has(`${i}:${j}`)), + ) + break + } + } + + // Apply match filters + for (let i = 0; i < dbSpace.matchFilters.length; i++) { + const mf = dbSpace.matchFilters[i] + const resolvedQuery: Record = {} + + for (const { column: colName, value: originalValue } of mf.entries) { + const key = `${i}:${colName}` + resolvedQuery[colName] = matchValueMap.has(key) + ? matchValueMap.get(key) + : originalValue + } + + q = q.match(resolvedQuery) + } + + // Apply not filters + for (let i = 0; i < dbSpace.notFilters.length; i++) { + const nf = dbSpace.notFilters[i] + + if (nf.op === 'in' && Array.isArray(nf.value)) { + const values = nf.value.map((v, j) => + notInMap.has(`${i}:${j}`) ? notInMap.get(`${i}:${j}`) : v, + ) + q = q.not(nf.column, 'in', formatInListOperand(values)) + continue + } + + const wasEncrypted = notValueMap.has(i) + const value = wasEncrypted ? notValueMap.get(i) : nf.value + + // `contains` is a supabase-js METHOD name, not a PostgREST operator, and + // `q.not()` interpolates its operand with `String(value)` — so an array + // arrives brace-less and an object as `[object Object]`. Build the + // containment literal ourselves and emit the `cs` token, exactly as the + // `.or()` path does. A scalar (including the encrypted envelope, already + // serialized) yields `null` and is forwarded untouched. + if (nf.op === 'contains' || nf.op === 'matches') { + const literal = formatContainmentOperand(value) + q = q.not(nf.column, 'cs', literal ?? value) + continue + } + + // Every `FilterOp` except `contains` spells the same as its PostgREST + // operator, and `contains` was handled above (it also needs its operand + // rewritten), so the recorded op is the wire op. + q = q.not(nf.column, nf.op, value) + } + + // Apply or filters + for (let i = 0; i < dbSpace.orFilters.length; i++) { + const of_ = dbSpace.orFilters[i] + + if (of_.kind === 'string') { + // Already parsed (once) and translated by `toDbSpace`. + const parsed = [...of_.conditions] + + for (let j = 0; j < parsed.length; j++) { + const sub = substituteOrValue(orStringConditionMap, i, j, parsed[j]) + if (sub) { + parsed[j] = { ...parsed[j], value: sub.value } + } + } + + // Rebuild whenever a condition REFERENCES an encrypted column — not + // merely when a value was encrypted. An `is`/null operand on an + // encrypted column encrypts nothing, so keying on "was a value + // substituted" would send that condition down the verbatim path below + // and forward the caller's JS property name to a DB that only knows the + // column's real name. `toDbSpace` has already translated `parsed`. + const referencesEncrypted = parsed.some((c) => + isEncryptedColumn(c.column, encryptedColumnNames), + ) + + if (referencesEncrypted) { + q = q.or(rebuildOrString(parsed), { + referencedTable: of_.referencedTable, + }) + } else { + // Every condition names a plaintext column, whose property name IS + // its DB name — nothing to map. Forward the caller's ORIGINAL string + // byte-for-byte: relied on for nested `and()` and quoted values that + // `parseOrString`/`rebuildOrString` cannot round-trip. + q = q.or(of_.original as DbFilterString, { + referencedTable: of_.referencedTable, + }) + } + } else { + // Structured: convert to string + const conditions = of_.conditions.map((cond, j) => { + const sub = substituteOrValue(orStructuredConditionMap, i, j, cond) + return sub ? { ...cond, value: sub.value } : cond + }) + + q = q.or(rebuildOrString(conditions)) + } + } + + // Apply raw filters + for (let i = 0; i < dbSpace.rawFilters.length; i++) { + const rf = dbSpace.rawFilters[i] + + // An encrypted `in` list was encrypted element-wise; reassemble it into + // the quoted PostgREST list literal, exactly as the `not` path does. A + // plaintext column keeps its operand untouched. + if ( + rf.operator === 'in' && + Array.isArray(rf.value) && + isEncryptedColumn(rf.column, encryptedColumnNames) + ) { + const values = rf.value.map((v, j) => + rawInMap.has(`${i}:${j}`) ? rawInMap.get(`${i}:${j}`) : v, + ) + q = q.filter(rf.column, rf.operator, formatInListOperand(values)) + continue + } + + const value = rawValueMap.has(i) ? rawValueMap.get(i) : rf.value + q = q.filter(rf.column, rf.operator, value) + } + + return q +} diff --git a/packages/stack-supabase/src/query-mutation.ts b/packages/stack-supabase/src/query-mutation.ts new file mode 100644 index 000000000..154f3244a --- /dev/null +++ b/packages/stack-supabase/src/query-mutation.ts @@ -0,0 +1,79 @@ +import { logger } from '@cipherstash/stack/adapter-kit' +import type { ColumnMap } from './column-map' +import { + type EncryptionContext, + EncryptionFailedError, + withOpContext, +} from './query-encrypt' +import type { MutationOp } from './types' + +/** + * Encode an encrypted model for the Supabase request body. The native + * `eql_v3.*` domains are plain jsonb, so the raw encrypted payload is sent + * (keyed by DB column name). + */ +function transformEncryptedMutationModel( + model: Record, + columns: ColumnMap, +): Record { + const out: Record = Object.create(null) + for (const [key, value] of Object.entries(model)) { + out[columns.dbNameFor(key)] = value + } + return out +} + +/** + * Encrypt a mutation's row data — the values being STORED, as opposed to the + * filter operands being searched by (`./query-encrypt`). `delete` carries no + * data, and a builder with no recorded mutation encrypts nothing. + */ +export async function encryptMutationData( + mutation: MutationOp | null, + ctx: EncryptionContext, +): Promise | Record[] | null> { + if (!mutation) return null + if (mutation.kind === 'delete') return null + + const data = mutation.data + + if (Array.isArray(data)) { + // Bulk encrypt + const result = await withOpContext( + ctx.encryptionClient.bulkEncryptModels(data, ctx.table), + ctx, + ) + if (result.failure) { + logger.error( + `Supabase: failed to encrypt models for table "${ctx.tableName}"`, + ) + + throw new EncryptionFailedError( + `Failed to encrypt models: ${result.failure.message}`, + result.failure, + ) + } + + return result.data.map((model) => + transformEncryptedMutationModel(model, ctx.columns), + ) + } + + // Single model + const result = await withOpContext( + ctx.encryptionClient.encryptModel(data, ctx.table), + ctx, + ) + if (result.failure) { + logger.error( + `Supabase: failed to encrypt model for table "${ctx.tableName}"`, + ) + + throw new EncryptionFailedError( + `Failed to encrypt model: ${result.failure.message}`, + result.failure, + ) + } + + return transformEncryptedMutationModel(result.data, ctx.columns) +} diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts new file mode 100644 index 000000000..8c1812bd8 --- /dev/null +++ b/packages/stack-supabase/src/query-results.ts @@ -0,0 +1,205 @@ +import { DATE_LIKE_CASTS, logger } from '@cipherstash/stack/adapter-kit' +import { selectKeyToDbV3 } from './helpers' +import { + type EncryptionContext, + EncryptionFailedError, + withOpContext, +} from './query-encrypt' +import type { EncryptedSupabaseResponse, ResultMode } from './types' + +/** cast_as kinds that reconstruct to a JS `Date` — shared with the typed v3 + * client's decrypt-model path (see `encryption/v3.ts`). */ +const DATE_LIKE_CAST_SET = new Set(DATE_LIKE_CASTS) + +export type RawSupabaseResult = { + data: unknown + error: { + message: string + details?: string + hint?: string + code?: string + } | null + count?: number | null + status: number + statusText: string +} + +/** What the decrypt step needs beyond the shared encryption context. */ +export type DecryptContext = EncryptionContext & { + selectColumns: string | null + resultMode: ResultMode + hasMutation: boolean +} + +/** + * Post-process a decrypted result row: rebuild `Date` values from the + * encrypt-config `cast_as` (date/timestamp), mirroring the typed v3 client's + * decrypt-model path. + */ +function postprocessDecryptedRow( + row: Record, + ctx: DecryptContext, +): Record { + // Every key an encrypted column can appear under: the keys this select + // actually produces (including caller-chosen aliases like `ts:createdAt`), + // plus the static property and DB names as a fallback for paths that record + // no select. Aliases win. Derived here from `ctx.selectColumns` (the row in + // hand) rather than cached from `buildSelectString`, so a reused builder can + // never postprocess a row with a previous operation's stale select map. + const propToDb = ctx.columns.propToDb + const keyToDb: Record = Object.assign( + Object.create(null), + ctx.selectColumns === null + ? undefined + : selectKeyToDbV3(ctx.selectColumns, propToDb), + ) + for (const [property, dbName] of Object.entries(propToDb)) { + keyToDb[property] ??= dbName + keyToDb[dbName] ??= dbName + } + + const out: Record = { ...row } + for (const [key, dbName] of Object.entries(keyToDb)) { + const castAs = ctx.columns.schemaFor(dbName)?.cast_as + if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue + const value = out[key] + if (value == null || value instanceof Date) continue + if (typeof value === 'string' || typeof value === 'number') { + out[key] = new Date(value) + } + } + return out +} + +/** + * Decrypt a PostgREST response into the caller's row type. + * + * Reads EQL v3 only. A column carrying legacy EQL v2 ciphertext is never in + * this adapter's encrypt config — introspection recognises `public.eql_v3_*` + * domains exclusively — so it is returned as an untouched passthrough rather + * than decrypted. To read v2 data, decrypt fetched rows with the core + * `@cipherstash/stack` client, whose decrypt path is generation-agnostic. + */ +export async function decryptResults>( + result: RawSupabaseResult, + ctx: DecryptContext, +): Promise> { + // If there's an error from Supabase, pass it through + if (result.error) { + return { + data: null, + error: { + message: result.error.message, + details: result.error.details, + hint: result.error.hint, + code: result.error.code, + }, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + // No data to decrypt + if (result.data === null || result.data === undefined) { + return { + data: null, + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + // Determine if we need to decrypt + const hasSelect = ctx.selectColumns !== null + const hasMutationWithReturning = ctx.hasMutation && hasSelect + + if (!hasSelect && !hasMutationWithReturning) { + // No select means no data to decrypt (e.g., insert without .select()) + return { + data: result.data as T[], + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + // Decrypt based on result mode + if (ctx.resultMode === 'single' || ctx.resultMode === 'maybeSingle') { + if (result.data === null) { + return { + data: null, + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + // Single result — decrypt one model + const decrypted = await withOpContext( + ctx.encryptionClient.decryptModel(result.data as Record), + ctx, + ) + if (decrypted.failure) { + logger.error( + `Supabase: failed to decrypt model for table "${ctx.tableName}"`, + ) + + throw new EncryptionFailedError( + `Failed to decrypt model: ${decrypted.failure.message}`, + decrypted.failure, + ) + } + + return { + data: postprocessDecryptedRow( + decrypted.data as Record, + ctx, + ) as unknown as T[], + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + // Array result — bulk decrypt + const dataArray = result.data as Record[] + if (dataArray.length === 0) { + return { + data: [] as unknown as T[], + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } + } + + const decrypted = await withOpContext( + ctx.encryptionClient.bulkDecryptModels(dataArray), + ctx, + ) + if (decrypted.failure) { + logger.error( + `Supabase: failed to decrypt models for table "${ctx.tableName}"`, + ) + + throw new EncryptionFailedError( + `Failed to decrypt models: ${decrypted.failure.message}`, + decrypted.failure, + ) + } + + return { + data: decrypted.data.map((row) => + postprocessDecryptedRow(row as Record, ctx), + ) as unknown as T[], + error: null, + count: result.count ?? null, + status: result.status, + statusText: result.statusText, + } +} diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index cc5378d56..c74fc9e6b 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -106,7 +106,7 @@ type NonScalarQueryableV3Keys
= { * the table's encrypted columns with no scalar capability (storage-only * columns, and `types.Json` documents — see {@link NonScalarQueryableV3Keys}; * before #650's `searchableJson` arm the two sets coincided). Plaintext - * (non-schema) columns pass through untouched, exactly as in v2. + * (non-schema) columns pass through untouched. */ export type FilterableKeys< Table extends AnyV3Table, @@ -380,11 +380,12 @@ export interface EncryptedQueryBuilder< } /** - * The v3 builder for a table with no declared schema. Without capability + * The builder for a table with no declared schema. Without capability * information `contains` cannot be narrowed to match-indexed columns — the - * runtime guard in the term-encryption path is the only protection — but the - * DIALECT is still v3, so `like`/`ilike` are absent here too. Typing this as - * {@link EncryptedQueryBuilder} would hand back the v2 surface. + * runtime guard in the term-encryption path is the only protection — and + * `order()`/`is(col, true)` cannot be narrowed either, so this surface takes + * {@link EncryptedQueryBuilderCore}'s `OK`/`BK` defaults. `like`/`ilike` are + * absent here as on the typed surface. * * For the same reason nothing here can tell an encrypted match column from a * plaintext jsonb one, so `matches`/`contains` accept the full native operand @@ -631,7 +632,7 @@ export type DbMutationOptions = Record & { // --------------------------------------------------------------------------- // DB-space IR — the recorded query, with every column name translated. // -// `toDbSpace()` (see ./query-builder) maps the property-space IR above into +// `toDbSpace()` (see ./query-dbspace) maps the property-space IR above into // this one, exactly once, before any column name can reach PostgREST. The // branded `column` fields make that translation a compile-time obligation: // `applyFilters`/`buildAndExecuteQuery` consume only these types, so feeding @@ -681,6 +682,19 @@ export type DbMutationOp = | Extract | Extract +/** The whole recorded query, in PROPERTY space — the builder's chained state as + * handed to `toDbSpace()`. The mirror of {@link DbQuerySpace} on the untranslated + * side of that boundary. */ +export type RecordedOps = { + filters: PendingFilter[] + matchFilters: PendingMatchFilter[] + notFilters: PendingNotFilter[] + rawFilters: PendingRawFilter[] + orFilters: PendingOrFilter[] + transforms: TransformOp[] + mutation: MutationOp | null +} + /** The whole recorded query, in DB-space. */ export type DbQuerySpace = { filters: DbPendingFilter[] @@ -771,33 +785,43 @@ export type { type StringKeyOf = Extract /** - * Every builder method shared by the v2 and v3 dialects. `Self` is the concrete - * builder each method returns, so a dialect that omits a method (v3 omits - * `like`/`ilike`) does not have it laundered back in by a chained call whose - * return type widened to the base interface. + * Every builder method shared by the TYPED ({@link EncryptedQueryBuilder}) and + * UNTYPED ({@link EncryptedQueryBuilderUntyped}) surfaces. Both are EQL v3 — + * they differ only in how much they can narrow, not in dialect. + * + * `Self` is the concrete builder each method returns, so a surface that omits a + * method does not have it laundered back in by a chained call whose return type + * widened to the base interface. * - * Free-text search is the ONLY axis on which the two dialects differ: v2 - * matches with SQL wildcards (`like`/`ilike` → `~~`), v3 with token containment - * (`contains` → `@>`). Each adds its own method below. + * Free-text search lives on the sub-interfaces rather than here, because its + * key set differs between the two: `matches()` narrows to the encrypted + * match/search columns on the typed surface, and to every row key on the + * untyped one. Neither surface carries `like`/`ilike` — EQL v3 free-text is + * fuzzy bloom-token matching, not SQL pattern matching, so the builder rewrites + * a `like` on an encrypted column to `matches` at record time (see + * `query-builder.ts`). They survive in this file only as the internal + * {@link FilterOp} union and on the raw {@link SupabaseQueryBuilder} seam, both + * of which still serve plaintext columns. */ export interface EncryptedQueryBuilderCore< T extends Record, FK extends StringKeyOf, Self, - /** Keys `order()` accepts. Defaults to `FK`, so the v2 surface is unchanged; - * v3 narrows it to plaintext columns (see {@link OrderableKeys}). */ + /** Keys `order()` accepts. The typed surface narrows it to the orderable + * columns (see {@link OrderableKeys}); it defaults to `FK` for the untyped + * surface, which has no capability information to narrow with. */ OK extends StringKeyOf = FK, - /** Keys the BOOLEAN form of `is()` accepts. Defaults to `FK`, so the v2 - * surface is unchanged; v3 narrows it to plaintext columns. Distinct from - * `OK` on purpose: "sortable" and "IS TRUE-able" are different capability - * axes that happen to select the same keys today, and narrowing `order()` - * later must not silently narrow `is()` with it. */ + /** Keys the BOOLEAN form of `is()` accepts. The typed surface narrows it to + * plaintext columns; it defaults to `FK` for the untyped surface, as `OK` + * does. Distinct from `OK` on purpose: "sortable" and "IS TRUE-able" are + * different capability axes that happen to select the same keys today, and + * narrowing `order()` later must not silently narrow `is()` with it. */ BK extends StringKeyOf = FK, > extends PromiseLike> { /** `columns` defaults to `'*'`, matching supabase-js. A `'*'` select expands - * to the introspected column list when one is available (v3), and otherwise - * throws — v2 has no column list to cast, so `select()` and `select('*')` - * both throw there. */ + * to the introspected column list; when none is available (a client that + * could not introspect) both `select()` and `select('*')` throw, because an + * unexpanded `*` cannot cast the encrypted columns with `::jsonb`. */ select( columns?: string, options?: { head?: boolean; count?: 'exact' | 'planned' | 'estimated' }, @@ -863,9 +887,10 @@ export interface EncryptedQueryBuilderCore< options?: { referencedTable?: string; foreignTable?: string }, ): Self match(query: Partial): Self - // `OK`, not `FK`: v3 cannot order by ANY encrypted column, because PostgREST - // cannot emit `ORDER BY eql_v3.ord_term(col)` and a bare `ORDER BY` sorts the - // ciphertext envelope. `OK` defaults to `FK`, so the v2 surface is unchanged. + // `OK`, not `FK`: an encrypted column is orderable only when its domain + // carries an OPE term (PostgREST reaches it as `col->op`); a bare `ORDER BY` + // would sort the ciphertext envelope. `OK` defaults to `FK` on the untyped + // surface, where the runtime `validateTransforms` guard is the only check. order( column: K, options?: { From 3d3486290bb00f63da7df9d7e8a8caf82b7815e6 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 23:32:03 +1000 Subject: [PATCH 013/123] feat(stack-drizzle)!: remove EQL v2 root and collapse ./v3 to root Delete the EQL v2 authoring surface (`src/index.ts` `encryptedType` + config extraction, `src/operators.ts`, `src/schema-extraction.ts`) and promote the EQL v3 implementation from `./v3` to the package root as a hard break with no alias. - Drop the `./v3` subpath from the `exports` map and remove `typesVersions`; the v3 impl now lives at `src/*` and is the sole `.` export. - De-suffix the public API: `createEncryptionOperatorsV3` -> `createEncryptionOperators`, `extractEncryptionSchemaV3` -> `extractEncryptionSchema`. No `*V3` aliases (they would type-check v2 call sites against v3 semantics). - Delete the two v2 operator test files; move the v3 test suite to `__tests__/*` and repoint imports. ESM+CJS both preserved. - Fix the `@cipherstash/bench` importer: rewrite the Drizzle table to v3 `types.*` domains + `EncryptionV3`, switch operator usages off the removed v2-only ops (`like`/`ilike`, jsonb-path) to `matches`/`contains`/`selector`, and update the bench fixture schema.sql to the eql_v3 domains/index terms. - Update the README and the bundled `stash-drizzle` skill to the collapsed root imports; remove v2 authoring guidance. Existing v2 ciphertext still decrypts via `@cipherstash/stack`; only the Drizzle-side v2 authoring/query-building is removed (Decision 6). BREAKING CHANGE: `@cipherstash/stack-drizzle` no longer exports an EQL v2 surface and the `./v3` subpath is removed. Import the v3 API from the package root with the de-suffixed names. --- .changeset/remove-eql-v2-drizzle-root.md | 22 + .../__benches__/drizzle/operators.bench.ts | 4 +- .../drizzle/operators.explain.test.ts | 42 +- packages/bench/package.json | 1 + packages/bench/sql/schema.sql | 28 +- packages/bench/src/drizzle/setup.ts | 48 +- packages/bench/src/harness/seed.ts | 19 +- packages/stack-drizzle/README.md | 27 +- .../__tests__/{v3 => }/bigint.test.ts | 10 +- .../__tests__/{v3 => }/codec.test.ts | 2 +- .../__tests__/{v3 => }/column.test.ts | 2 +- .../drizzle-operators-bigint.test.ts | 59 - .../__tests__/drizzle-operators-jsonb.test.ts | 211 -- .../__tests__/{v3 => }/exports.test.ts | 6 +- .../__tests__/{v3 => }/indexes.test.ts | 4 +- .../__tests__/{v3 => }/operators.test-d.ts | 16 +- .../__tests__/{v3 => }/operators.test.ts | 30 +- .../{v3 => }/schema-extraction.test.ts | 22 +- .../__tests__/{v3 => }/selector.test.ts | 8 +- .../__tests__/{v3 => }/sql-dialect.test.ts | 2 +- .../__tests__/{v3 => }/types.test-d.ts | 2 +- .../__tests__/{v3 => }/types.test.ts | 4 +- packages/stack-drizzle/integration/adapter.ts | 16 +- .../stack-drizzle/integration/json-adapter.ts | 12 +- .../lock-context.integration.test.ts | 14 +- .../null-persistence.integration.test.ts | 14 +- .../relational.integration.test.ts | 16 +- packages/stack-drizzle/package.json | 17 - packages/stack-drizzle/src/{v3 => }/codec.ts | 0 packages/stack-drizzle/src/{v3 => }/column.ts | 0 packages/stack-drizzle/src/index.ts | 233 +- .../stack-drizzle/src/{v3 => }/indexes.ts | 2 +- packages/stack-drizzle/src/operators.ts | 2586 +++++------------ .../stack-drizzle/src/schema-extraction.ts | 140 +- .../stack-drizzle/src/{v3 => }/sql-dialect.ts | 0 packages/stack-drizzle/src/{v3 => }/types.ts | 0 packages/stack-drizzle/src/v3/index.ts | 9 - packages/stack-drizzle/src/v3/operators.ts | 959 ------ .../stack-drizzle/src/v3/schema-extraction.ts | 50 - packages/stack-drizzle/tsup.config.ts | 2 +- skills/stash-drizzle/SKILL.md | 99 +- 41 files changed, 1048 insertions(+), 3690 deletions(-) create mode 100644 .changeset/remove-eql-v2-drizzle-root.md rename packages/stack-drizzle/__tests__/{v3 => }/bigint.test.ts (94%) rename packages/stack-drizzle/__tests__/{v3 => }/codec.test.ts (98%) rename packages/stack-drizzle/__tests__/{v3 => }/column.test.ts (99%) delete mode 100644 packages/stack-drizzle/__tests__/drizzle-operators-bigint.test.ts delete mode 100644 packages/stack-drizzle/__tests__/drizzle-operators-jsonb.test.ts rename packages/stack-drizzle/__tests__/{v3 => }/exports.test.ts (91%) rename packages/stack-drizzle/__tests__/{v3 => }/indexes.test.ts (98%) rename packages/stack-drizzle/__tests__/{v3 => }/operators.test-d.ts (87%) rename packages/stack-drizzle/__tests__/{v3 => }/operators.test.ts (97%) rename packages/stack-drizzle/__tests__/{v3 => }/schema-extraction.test.ts (82%) rename packages/stack-drizzle/__tests__/{v3 => }/selector.test.ts (96%) rename packages/stack-drizzle/__tests__/{v3 => }/sql-dialect.test.ts (98%) rename packages/stack-drizzle/__tests__/{v3 => }/types.test-d.ts (97%) rename packages/stack-drizzle/__tests__/{v3 => }/types.test.ts (87%) rename packages/stack-drizzle/src/{v3 => }/codec.ts (100%) rename packages/stack-drizzle/src/{v3 => }/column.ts (100%) rename packages/stack-drizzle/src/{v3 => }/indexes.ts (99%) rename packages/stack-drizzle/src/{v3 => }/sql-dialect.ts (100%) rename packages/stack-drizzle/src/{v3 => }/types.ts (100%) delete mode 100644 packages/stack-drizzle/src/v3/index.ts delete mode 100644 packages/stack-drizzle/src/v3/operators.ts delete mode 100644 packages/stack-drizzle/src/v3/schema-extraction.ts diff --git a/.changeset/remove-eql-v2-drizzle-root.md b/.changeset/remove-eql-v2-drizzle-root.md new file mode 100644 index 000000000..66e61576f --- /dev/null +++ b/.changeset/remove-eql-v2-drizzle-root.md @@ -0,0 +1,22 @@ +--- +'@cipherstash/stack-drizzle': major +'stash': patch +--- + +Remove the EQL v2 authoring surface from `@cipherstash/stack-drizzle` and collapse the EQL v3 `./v3` subpath into the package root. + +**Breaking (`@cipherstash/stack-drizzle`):** + +- The EQL v2 root exports are gone: `encryptedType`, the v2 `extractEncryptionSchema`, the v2 `createEncryptionOperators` (including the `like` / `ilike` operators), and `EncryptionConfigError`. Authoring or querying `eql_v2_encrypted` columns through Drizzle is no longer supported. +- The `./v3` subpath is **removed** from the package `exports` map and `typesVersions`. The EQL v3 implementation is now the package root, and the `*V3` names are de-suffixed (`createEncryptionOperatorsV3` → `createEncryptionOperators`, `extractEncryptionSchemaV3` → `extractEncryptionSchema`). This is a **hard break with no alias**: post-collapse the root names would collide with the removed v2 names, and keeping an alias would silently type-check v2 call sites against v3 semantics. + +**Migration** — import the v3 surface from the package root instead of `./v3`, and drop the `V3` suffix: + +```diff +- import { types, extractEncryptionSchemaV3, createEncryptionOperatorsV3 } from '@cipherstash/stack-drizzle/v3' ++ import { types, extractEncryptionSchema, createEncryptionOperators } from '@cipherstash/stack-drizzle' +``` + +The `types.*` column factories, `makeEqlV3Column` / `getEqlV3Column` / `isEqlV3Column`, the codec helpers (`v3ToDriver` / `v3FromDriver` / `EqlV3CodecError`), and `EncryptionOperatorError` are unchanged apart from moving to the root. + +Existing EQL v2 ciphertext remains decryptable via `@cipherstash/stack` — only the Drizzle-side v2 authoring and query-building is removed. The bundled `stash-drizzle` skill is updated to the collapsed root imports. diff --git a/packages/bench/__benches__/drizzle/operators.bench.ts b/packages/bench/__benches__/drizzle/operators.bench.ts index 4dd7c99af..53422497f 100644 --- a/packages/bench/__benches__/drizzle/operators.bench.ts +++ b/packages/bench/__benches__/drizzle/operators.bench.ts @@ -44,8 +44,8 @@ describe('drizzle', () => { await handle.db.select().from(benchTable).where(where) }) - bench('like (prefix)', async () => { - const where = (await ops.like(benchTable.encText, '%value-00000%')) as SQL + bench('matches (free-text)', async () => { + const where = (await ops.matches(benchTable.encText, 'value')) as SQL await handle.db.select().from(benchTable).where(where) }) diff --git a/packages/bench/__tests__/drizzle/operators.explain.test.ts b/packages/bench/__tests__/drizzle/operators.explain.test.ts index 9aec76d4f..d3f58857e 100644 --- a/packages/bench/__tests__/drizzle/operators.explain.test.ts +++ b/packages/bench/__tests__/drizzle/operators.explain.test.ts @@ -104,13 +104,12 @@ async function tryExplainWhere(name: string, where: SQL): Promise { // --- #421: equality + array operators ------------------------------------- // -// `bench_text_hmac_idx` (functional hash on eql_v2.hmac_256) is the expected -// fast path. Pre-fix Drizzle emits bare `=` / `<>` / `IN (...)` which falls -// back to seq scan. Post-fix it emits `eql_v2.hmac_256(col) = -// eql_v2.hmac_256(value)` and the index scan kicks in. +// `bench_text_hmac_idx` (functional hash on eql_v3.eq_term) is the expected +// fast path. The v3 operators emit `eql_v3.eq_term(col) = eql_v3.hmac_256(value)` +// so the functional hash index scan kicks in instead of a seq scan. // // `eq` and `inArray` are naturally high-selectivity (only a few rows match), -// so the planner should pick the hmac index — assertion enforces it. +// so the planner should pick the eq_term index — assertion enforces it. // // `ne` and `notInArray` are naturally low-selectivity (almost all rows match); // even with the hmac index available the planner correctly chooses a seq @@ -161,14 +160,10 @@ describe('#421: equality and array operators', () => { // We don't yet know which call-shaped forms the planner inlines. Record plan // shape; assertions land in a follow-up once #422 closes. describe('#422: call-shaped operators (recorded, not asserted)', () => { - it('records like / ilike plan shapes', async () => { + it('records matches plan shape', async () => { await tryExplainWhere( - 'like', - (await ops.like(benchTable.encText, '%value-00000%')) as SQL, - ) - await tryExplainWhere( - 'ilike', - (await ops.ilike(benchTable.encText, '%VALUE-00000%')) as SQL, + 'matches', + (await ops.matches(benchTable.encText, 'value')) as SQL, ) }) @@ -190,20 +185,15 @@ describe('#422: call-shaped operators (recorded, not asserted)', () => { ) }) - it('records jsonb operator plan shapes', async () => { - for (const [name, build] of [ - [ - 'jsonbPathQueryFirst', - () => ops.jsonbPathQueryFirst(benchTable.encJsonb, '$.idx'), - ], - ['jsonbGet', () => ops.jsonbGet(benchTable.encJsonb, '$.idx')], - [ - 'jsonbPathExists', - () => ops.jsonbPathExists(benchTable.encJsonb, '$.idx'), - ], - ] as const) { - await tryExplainWhere(name, await build()) - } + it('records encrypted-JSONB plan shapes (contains / selector)', async () => { + await tryExplainWhere( + 'contains', + (await ops.contains(benchTable.encJsonb, { idx: 42 })) as SQL, + ) + await tryExplainWhere( + 'selector.gt', + (await ops.selector(benchTable.encJsonb, '$.idx').gt(5000)) as SQL, + ) }) it('records ORDER BY plan shape (asc / desc)', async () => { diff --git a/packages/bench/package.json b/packages/bench/package.json index 1aa5b847a..8c95c944a 100644 --- a/packages/bench/package.json +++ b/packages/bench/package.json @@ -5,6 +5,7 @@ "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", "type": "module", "scripts": { + "build": "tsc --noEmit", "db:setup": "tsx src/cli/setup.ts", "db:reset": "tsx src/cli/reset.ts", "test:local": "vitest run", diff --git a/packages/bench/sql/schema.sql b/packages/bench/sql/schema.sql index cf861d2be..9390fa5e8 100644 --- a/packages/bench/sql/schema.sql +++ b/packages/bench/sql/schema.sql @@ -1,30 +1,32 @@ -- Bench fixture schema. -- Single bench table covering text / int / jsonb encrypted columns plus the --- three canonical EQL functional indexes: hmac_256 (hash), bloom_filter (GIN), --- ste_vec (GIN). +-- three canonical EQL v3 functional indexes over the column-side index terms: +-- eq_term (hash), match_term (GIN bloom), ste_vec (GIN). -- --- We deliberately do NOT create the `eql_v2.encrypted_operator_class` btree --- indexes that ore-benches uses. Encrypted composites for full-feature columns --- (equality + match + ORE) blow past the 2704-byte btree page-size limit, and --- those indexes don't exist on Supabase anyway — the bench's whole job is to --- validate that the functional-index path works. +-- Mirrors `src/drizzle/setup.ts`: the columns are concrete `eql_v3_*` Postgres +-- domains (see `types.TextSearch` / `types.IntegerOrd` / `types.Json`). Install +-- the domains and index-term functions first with `stash eql install --eql-version 3`. +-- +-- We deliberately do NOT create btree operator-class indexes: encrypted terms for +-- full-feature columns blow past the 2704-byte btree page-size limit, and the +-- bench's whole job is to validate that the functional-index path works. DROP TABLE IF EXISTS bench; CREATE TABLE bench ( id SERIAL PRIMARY KEY, - enc_text eql_v2_encrypted NOT NULL, - enc_int eql_v2_encrypted NOT NULL, - enc_jsonb eql_v2_encrypted NOT NULL + enc_text public.eql_v3_text_search NOT NULL, + enc_int public.eql_v3_integer_ord NOT NULL, + enc_jsonb public.eql_v3_json_search NOT NULL ); CREATE INDEX bench_text_hmac_idx - ON bench USING hash (eql_v2.hmac_256(enc_text)); + ON bench USING hash (eql_v3.eq_term(enc_text)); CREATE INDEX bench_text_bloom_idx - ON bench USING gin (eql_v2.bloom_filter(enc_text)); + ON bench USING gin (eql_v3.match_term(enc_text)); CREATE INDEX bench_jsonb_stevec_idx - ON bench USING gin (eql_v2.ste_vec(enc_jsonb)); + ON bench USING gin (eql_v3.ste_vec(enc_jsonb)); ANALYZE bench; diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index 944ece227..267ffc951 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -1,9 +1,5 @@ -import { Encryption } from '@cipherstash/stack' -import type { EncryptionClient } from '@cipherstash/stack/encryption' -import { - encryptedType, - extractEncryptionSchema, -} from '@cipherstash/stack-drizzle' +import { EncryptionV3 } from '@cipherstash/stack/v3' +import { extractEncryptionSchema, types } from '@cipherstash/stack-drizzle' import { drizzle } from 'drizzle-orm/node-postgres' import { pgTable, serial } from 'drizzle-orm/pg-core' import pg from 'pg' @@ -12,34 +8,23 @@ import { getDatabaseUrl } from '../harness/db.js' /** * Drizzle schema for the bench table. Mirrors `sql/schema.sql`. * - * `id` is `serial`; the encrypted columns are `eql_v2_encrypted` composites - * driven by `@cipherstash/stack-drizzle`'s `encryptedType`. + * `id` is `serial`; the encrypted columns are concrete `eql_v3_*` Postgres + * domains emitted by `@cipherstash/stack-drizzle`'s `types.*` factories. * - * Index config flags (`equality`, `freeTextSearch`, `orderAndRange`, - * `searchableJson`) are deliberately all on — the bench needs to exercise - * every query family that lands on the table. + * The domains are chosen to exercise every query family the bench lands on the + * table: `TextSearch` (equality + free-text + order/range), `IntegerOrd` + * (equality + order/range), and `Json` (encrypted-JSONB containment + selector). */ export const benchTable = pgTable('bench', { id: serial('id').primaryKey(), - encText: encryptedType('enc_text', { - equality: true, - freeTextSearch: true, - orderAndRange: true, - }), - encInt: encryptedType('enc_int', { - dataType: 'number', - equality: true, - orderAndRange: true, - }), - encJsonb: encryptedType<{ idx: number; group: number }>('enc_jsonb', { - dataType: 'json', - searchableJson: true, - }), + encText: types.TextSearch('enc_text'), + encInt: types.IntegerOrd('enc_int'), + encJsonb: types.Json('enc_jsonb'), }) /** - * Encryption schema for the stack `Encryption()` client. Derived from the - * Drizzle table above so the two can't drift apart. + * Encryption schema for the `EncryptionV3()` client. Derived from the Drizzle + * table above so the two can't drift apart. */ export const encryptionBenchTable = extractEncryptionSchema(benchTable) @@ -49,11 +34,14 @@ export type BenchPlaintextRow = { enc_jsonb: { idx: number; group: number } } +/** The typed EQL v3 client this bench drives. */ +export type BenchEncryptionClient = Awaited> + export type BenchHandle = { pgClient: pg.Client pool: pg.Pool db: ReturnType - encryptionClient: EncryptionClient + encryptionClient: BenchEncryptionClient } /** @@ -69,7 +57,9 @@ export async function buildBench(): Promise { const db = drizzle(pool) - const encryptionClient = await Encryption({ schemas: [encryptionBenchTable] }) + const encryptionClient = await EncryptionV3({ + schemas: [encryptionBenchTable], + }) return { pgClient, pool, db, encryptionClient } } diff --git a/packages/bench/src/harness/seed.ts b/packages/bench/src/harness/seed.ts index c25d21c7a..e9fc828b6 100644 --- a/packages/bench/src/harness/seed.ts +++ b/packages/bench/src/harness/seed.ts @@ -53,11 +53,10 @@ export async function seed( plaintexts.push(makePlaintextRow(rowsBefore + i)) } - const encResult = - await h.encryptionClient.bulkEncryptModels( - plaintexts, - encryptionBenchTable, - ) + const encResult = await h.encryptionClient.bulkEncryptModels( + plaintexts, + encryptionBenchTable, + ) if (encResult.failure) { throw new Error( `[bench:seed] bulkEncryptModels failed: ${encResult.failure.message}`, @@ -65,12 +64,12 @@ export async function seed( } // bulkEncryptModels returns rows keyed by the encryptedTable column names - // (snake_case here). Drizzle's `benchTable` uses camelCase TS field names — - // remap before insert. + // (snake_case here) with encrypted EQL v3 envelopes as values. Drizzle's + // `benchTable` uses camelCase TS field names — remap before insert. const encRows = encResult.data.map((r) => ({ - encText: r.enc_text as unknown as string, - encInt: r.enc_int as unknown as number, - encJsonb: r.enc_jsonb as unknown as { idx: number; group: number }, + encText: r.enc_text, + encInt: r.enc_int, + encJsonb: r.enc_jsonb, })) for (let i = 0; i < encRows.length; i += INSERT_BATCH) { diff --git a/packages/stack-drizzle/README.md b/packages/stack-drizzle/README.md index 841726a30..ae5907733 100644 --- a/packages/stack-drizzle/README.md +++ b/packages/stack-drizzle/README.md @@ -71,7 +71,7 @@ npm install @cipherstash/stack @cipherstash/stack-drizzle drizzle-orm Full guide: [Drizzle quickstart →][drizzle-docs] -## Full example (EQL v3, the `/v3` subpath) +## Full example (EQL v3) Each encrypted column is a concrete `public.eql_v3_*` Postgres domain whose query capabilities are fixed by the `types.*` factory you choose — no per-column config object: @@ -81,9 +81,9 @@ import { pgTable, integer } from 'drizzle-orm/pg-core' import { EncryptionV3 } from '@cipherstash/stack/v3' import { types, - extractEncryptionSchemaV3, - createEncryptionOperatorsV3, -} from '@cipherstash/stack-drizzle/v3' + extractEncryptionSchema, + createEncryptionOperators, +} from '@cipherstash/stack-drizzle' const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), @@ -91,9 +91,9 @@ const users = pgTable('users', { age: types.IntegerOrd('age'), // equality + order/range }) -const schema = extractEncryptionSchemaV3(users) +const schema = extractEncryptionSchema(users) const client = await EncryptionV3({ schemas: [schema] }) -const ops = createEncryptionOperatorsV3(client) +const ops = createEncryptionOperators(client) // Insert — encrypt models first (bulk helpers batch key operations // through ZeroKMS instead of one round trip per row) @@ -128,7 +128,7 @@ indexes up like any others: ```ts import { integer, pgTable } from 'drizzle-orm/pg-core' -import { encryptedIndexes, types } from '@cipherstash/stack-drizzle/v3' +import { encryptedIndexes, types } from '@cipherstash/stack-drizzle' export const users = pgTable( 'users', @@ -160,19 +160,6 @@ key revocation, and [identity-bound decryption][identity] (lock contexts) work w database ever holding a secret. Runs on plain PostgreSQL, Supabase, and RDS/Aurora; the SQL install needs no superuser. -## EQL v2 (package root) — legacy - -The v2 integration predates the typed v3 domains and is kept for existing projects. New -projects should use v3 above. - -```ts -import { encryptedType, extractEncryptionSchema, createEncryptionOperators } from '@cipherstash/stack-drizzle' -import { Encryption } from '@cipherstash/stack' -``` - -`encryptedType` defines an `eql_v2_encrypted` column; `createEncryptionOperators` returns -query operators (`eq`, `like`, `gt`, `inArray`, …) that transparently encrypt search values. - ## Docs - [Drizzle integration guide →][drizzle-docs] diff --git a/packages/stack-drizzle/__tests__/v3/bigint.test.ts b/packages/stack-drizzle/__tests__/bigint.test.ts similarity index 94% rename from packages/stack-drizzle/__tests__/v3/bigint.test.ts rename to packages/stack-drizzle/__tests__/bigint.test.ts index e31a6a4e2..7cfbf9f3d 100644 --- a/packages/stack-drizzle/__tests__/v3/bigint.test.ts +++ b/packages/stack-drizzle/__tests__/bigint.test.ts @@ -1,12 +1,12 @@ import { type SQL, sql } from 'drizzle-orm' import { integer, PgDialect, pgTable } from 'drizzle-orm/pg-core' import { describe, expect, it, vi } from 'vitest' -import { getEqlV3Column, isEqlV3Column } from '../../src/v3/column' +import { getEqlV3Column, isEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, + createEncryptionOperators, EncryptionOperatorError, -} from '../../src/v3/operators' -import { types } from '../../src/v3/types' +} from '../src/operators' +import { types } from '../src/types' // A representative query TERM — what `client.encryptQuery` returns for a bigint // operand: a ciphertext-free term, deliberately NOT a plaintext bigint. @@ -25,7 +25,7 @@ function chainable(result: unknown) { function setup() { const encryptQuery = vi.fn(() => chainable(Promise.resolve({ data: TERM }))) - const ops = createEncryptionOperatorsV3({ encryptQuery }) + const ops = createEncryptionOperators({ encryptQuery }) const dialect = new PgDialect() const render = (s: SQL) => dialect.sqlToQuery(s) return { ops, encryptQuery, render } diff --git a/packages/stack-drizzle/__tests__/v3/codec.test.ts b/packages/stack-drizzle/__tests__/codec.test.ts similarity index 98% rename from packages/stack-drizzle/__tests__/v3/codec.test.ts rename to packages/stack-drizzle/__tests__/codec.test.ts index 3049c2c2a..6d44985a4 100644 --- a/packages/stack-drizzle/__tests__/v3/codec.test.ts +++ b/packages/stack-drizzle/__tests__/codec.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import { EqlV3CodecError, v3FromDriver, v3ToDriver } from '../../src/v3/codec' +import { EqlV3CodecError, v3FromDriver, v3ToDriver } from '../src/codec' // A realistic `public.eql_v3_text_eq` envelope: schema version, table/column // identifier, mp_base85 ciphertext, HMAC term. The trivial `{v:1,c:'ct'}` diff --git a/packages/stack-drizzle/__tests__/v3/column.test.ts b/packages/stack-drizzle/__tests__/column.test.ts similarity index 99% rename from packages/stack-drizzle/__tests__/v3/column.test.ts rename to packages/stack-drizzle/__tests__/column.test.ts index 9ca47384d..dfca7f9b8 100644 --- a/packages/stack-drizzle/__tests__/v3/column.test.ts +++ b/packages/stack-drizzle/__tests__/column.test.ts @@ -12,7 +12,7 @@ import { getEqlV3Column, isEqlV3Column, makeEqlV3Column, -} from '../../src/v3/column' +} from '../src/column' describe('makeEqlV3Column', () => { it('sets dataType() to the BARE eql_v3 domain (no schema qualifier)', () => { diff --git a/packages/stack-drizzle/__tests__/drizzle-operators-bigint.test.ts b/packages/stack-drizzle/__tests__/drizzle-operators-bigint.test.ts deleted file mode 100644 index e71e47622..000000000 --- a/packages/stack-drizzle/__tests__/drizzle-operators-bigint.test.ts +++ /dev/null @@ -1,59 +0,0 @@ -import type { EncryptionClient } from '@cipherstash/stack/encryption' -import { PgDialect, pgTable } from 'drizzle-orm/pg-core' -import { describe, expect, it, vi } from 'vitest' -import { createEncryptionOperators, encryptedType } from '../src/index.js' - -// Regression coverage for the `bigint` (int8) plaintext path through the v3 -// Drizzle operators. Before the fix, `toPlaintext` fell through to -// `String(value)`, so `ops.eq(bigintCol, 1n)` was encrypted as the TEXT "1" and -// silently mismatched the column's bigint domain. The operators must forward a -// native `bigint` to `encryptQuery` (whose term type is `Plaintext`), so -// protect-ffi 0.28 encrypts it against the int8 domain and bounds-checks it. - -const ENCRYPTED_VALUE = '{"v":"encrypted-value"}' - -function setup() { - const encryptQuery = vi.fn(async (termsOrValue: unknown) => - Array.isArray(termsOrValue) - ? { data: termsOrValue.map(() => ENCRYPTED_VALUE) } - : { data: ENCRYPTED_VALUE }, - ) - const client = { encryptQuery } as unknown as EncryptionClient - const encryptionOps = createEncryptionOperators(client) - const dialect = new PgDialect() - return { encryptQuery, encryptionOps, dialect } -} - -const accounts = pgTable('accounts', { - balance: encryptedType('balance', { - dataType: 'bigint', - equality: true, - orderAndRange: true, - }), -}) - -describe('Drizzle v3 operators preserve bigint plaintext (int8 columns)', () => { - it('eq forwards a native bigint to encryptQuery, not a String()-coerced value', async () => { - const { encryptQuery, encryptionOps } = setup() - - await encryptionOps.eq(accounts.balance, 1n) - - expect(encryptQuery).toHaveBeenCalledTimes(1) - const terms = encryptQuery.mock.calls[0]?.[0] as Array<{ value: unknown }> - expect(terms).toHaveLength(1) - expect(typeof terms[0]?.value).toBe('bigint') - expect(terms[0]?.value).toBe(1n) - // The pre-fix bug: a stringified bigint. - expect(terms[0]?.value).not.toBe('1') - }) - - it('preserves i64 boundary magnitudes losslessly (beyond Number.MAX_SAFE_INTEGER)', async () => { - const { encryptQuery, encryptionOps } = setup() - const big = 9223372036854775807n // i64::MAX - - await encryptionOps.gt(accounts.balance, big) - - const terms = encryptQuery.mock.calls[0]?.[0] as Array<{ value: unknown }> - expect(terms[0]?.value).toBe(big) - }) -}) diff --git a/packages/stack-drizzle/__tests__/drizzle-operators-jsonb.test.ts b/packages/stack-drizzle/__tests__/drizzle-operators-jsonb.test.ts deleted file mode 100644 index 47ab6bce0..000000000 --- a/packages/stack-drizzle/__tests__/drizzle-operators-jsonb.test.ts +++ /dev/null @@ -1,211 +0,0 @@ -import type { EncryptionClient } from '@cipherstash/stack/encryption' -import { PgDialect, pgTable } from 'drizzle-orm/pg-core' -import { describe, expect, it, vi } from 'vitest' -import { - createEncryptionOperators, - EncryptionOperatorError, - encryptedType, -} from '../src/index.js' - -const ENCRYPTED_VALUE = '{"v":"encrypted-value"}' - -function createMockEncryptionClient() { - const encryptQuery = vi.fn(async (termsOrValue: unknown) => { - if (Array.isArray(termsOrValue)) { - return { data: termsOrValue.map(() => ENCRYPTED_VALUE) } - } - return { data: ENCRYPTED_VALUE } - }) - - return { - client: { encryptQuery } as unknown as EncryptionClient, - encryptQuery, - } -} - -function setup() { - const { client, encryptQuery } = createMockEncryptionClient() - const encryptionOps = createEncryptionOperators(client) - const dialect = new PgDialect() - return { client, encryptQuery, encryptionOps, dialect } -} - -const docsTable = pgTable('json_docs', { - metadata: encryptedType>('metadata', { - dataType: 'json', - searchableJson: true, - }), - noJsonConfig: encryptedType('no_json_config', { - equality: true, - }), -}) - -describe('createEncryptionOperators JSONB selector typing', () => { - it('casts jsonbPathQueryFirst selector params to eql_v2_encrypted', async () => { - const { encryptQuery, encryptionOps, dialect } = setup() - - const condition = await encryptionOps.jsonbPathQueryFirst( - docsTable.metadata, - '$.profile.email', - ) - const query = dialect.sqlToQuery(condition) - - expect(query.sql).toMatch( - /eql_v2\.jsonb_path_query_first\([^,]+,\s*\$\d+::eql_v2_encrypted\)/, - ) - expect(query.params).toHaveLength(1) - expect(query.params[0]).toContain('encrypted-value') - expect(encryptQuery).toHaveBeenCalledTimes(1) - expect(encryptQuery.mock.calls[0]?.[0]).toMatchObject([ - { queryType: 'steVecSelector' }, - ]) - }) - - it('casts jsonbPathExists selector params to eql_v2_encrypted', async () => { - const { encryptQuery, encryptionOps, dialect } = setup() - - const condition = await encryptionOps.jsonbPathExists( - docsTable.metadata, - '$.profile.email', - ) - const query = dialect.sqlToQuery(condition) - - expect(query.sql).toMatch( - /eql_v2\.jsonb_path_exists\([^,]+,\s*\$\d+::eql_v2_encrypted\)/, - ) - expect(query.params).toHaveLength(1) - expect(query.params[0]).toContain('encrypted-value') - expect(encryptQuery).toHaveBeenCalledTimes(1) - expect(encryptQuery.mock.calls[0]?.[0]).toMatchObject([ - { queryType: 'steVecSelector' }, - ]) - }) - - it('casts jsonbGet selector params to eql_v2_encrypted', async () => { - const { encryptQuery, encryptionOps, dialect } = setup() - - const condition = await encryptionOps.jsonbGet( - docsTable.metadata, - '$.profile.email', - ) - const query = dialect.sqlToQuery(condition) - - expect(query.sql).toMatch(/->\s*\$\d+::eql_v2_encrypted/) - expect(query.params).toHaveLength(1) - expect(query.params[0]).toContain('encrypted-value') - expect(encryptQuery).toHaveBeenCalledTimes(1) - expect(encryptQuery.mock.calls[0]?.[0]).toMatchObject([ - { queryType: 'steVecSelector' }, - ]) - }) -}) - -describe('JSONB operator error paths', () => { - it('throws EncryptionOperatorError when column lacks searchableJson config', () => { - const { encryptionOps } = setup() - - expect(() => - encryptionOps.jsonbPathQueryFirst(docsTable.noJsonConfig, '$.path'), - ).toThrow(EncryptionOperatorError) - - expect(() => - encryptionOps.jsonbPathQueryFirst(docsTable.noJsonConfig, '$.path'), - ).toThrow(/searchableJson/) - }) - - it('throws EncryptionOperatorError for jsonbPathExists without searchableJson', () => { - const { encryptionOps } = setup() - - expect(() => - encryptionOps.jsonbPathExists(docsTable.noJsonConfig, '$.path'), - ).toThrow(EncryptionOperatorError) - }) - - it('throws EncryptionOperatorError for jsonbGet without searchableJson', () => { - const { encryptionOps } = setup() - - expect(() => - encryptionOps.jsonbGet(docsTable.noJsonConfig, '$.path'), - ).toThrow(EncryptionOperatorError) - }) - - it('error includes column name and operator context', () => { - const { encryptionOps } = setup() - - try { - encryptionOps.jsonbPathQueryFirst(docsTable.noJsonConfig, '$.path') - expect.fail('Should have thrown') - } catch (error) { - expect(error).toBeInstanceOf(EncryptionOperatorError) - const opError = error as EncryptionOperatorError - expect(opError.context?.columnName).toBe('no_json_config') - expect(opError.context?.operator).toBe('jsonbPathQueryFirst') - } - }) -}) - -describe('JSONB batched operations', () => { - it('batches jsonbPathQueryFirst and jsonbGet in encryptionOps.and()', async () => { - const { encryptQuery, encryptionOps, dialect } = setup() - - const condition = await encryptionOps.and( - encryptionOps.jsonbPathQueryFirst(docsTable.metadata, '$.profile.email'), - encryptionOps.jsonbGet(docsTable.metadata, '$.profile.name'), - ) - const query = dialect.sqlToQuery(condition) - - expect(query.sql).toContain('eql_v2.jsonb_path_query_first') - expect(query.sql).toContain('->') - // Verify batch encryption happened (at least one call with 2 terms) - expect( - encryptQuery.mock.calls.some( - (call: unknown[]) => Array.isArray(call[0]) && call[0].length === 2, - ), - ).toBe(true) - }) - - it('batches jsonbPathExists and jsonbPathQueryFirst in encryptionOps.or()', async () => { - const { encryptQuery, encryptionOps, dialect } = setup() - - const condition = await encryptionOps.or( - encryptionOps.jsonbPathExists(docsTable.metadata, '$.profile.email'), - encryptionOps.jsonbPathQueryFirst(docsTable.metadata, '$.profile.name'), - ) - const query = dialect.sqlToQuery(condition) - - expect(query.sql).toContain('eql_v2.jsonb_path_exists') - expect(query.sql).toContain('eql_v2.jsonb_path_query_first') - // Verify batch encryption happened (at least one call with 2 terms) - expect( - encryptQuery.mock.calls.some( - (call: unknown[]) => Array.isArray(call[0]) && call[0].length === 2, - ), - ).toBe(true) - }) - - it('generates SQL combining conditions with AND', async () => { - const { encryptionOps, dialect } = setup() - - const condition = await encryptionOps.and( - encryptionOps.jsonbPathQueryFirst(docsTable.metadata, '$.a'), - encryptionOps.jsonbPathExists(docsTable.metadata, '$.b'), - ) - const query = dialect.sqlToQuery(condition) - - // AND combines conditions - expect(query.sql).toContain(' and ') - }) - - it('generates SQL combining conditions with OR', async () => { - const { encryptionOps, dialect } = setup() - - const condition = await encryptionOps.or( - encryptionOps.jsonbPathQueryFirst(docsTable.metadata, '$.a'), - encryptionOps.jsonbPathExists(docsTable.metadata, '$.b'), - ) - const query = dialect.sqlToQuery(condition) - - // OR combines conditions - expect(query.sql).toContain(' or ') - }) -}) diff --git a/packages/stack-drizzle/__tests__/v3/exports.test.ts b/packages/stack-drizzle/__tests__/exports.test.ts similarity index 91% rename from packages/stack-drizzle/__tests__/v3/exports.test.ts rename to packages/stack-drizzle/__tests__/exports.test.ts index 2b3b41f74..4a32d0c3f 100644 --- a/packages/stack-drizzle/__tests__/v3/exports.test.ts +++ b/packages/stack-drizzle/__tests__/exports.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import * as barrel from '../../src/v3/index.js' +import * as barrel from '../src/index.js' // Exhaustive, so ADDING an export fails here too. The previous version asserted // only four names, leaving the codec/column re-exports unpinned and letting a @@ -7,9 +7,9 @@ import * as barrel from '../../src/v3/index.js' const PUBLIC_SURFACE = [ 'EncryptionOperatorError', 'EqlV3CodecError', - 'createEncryptionOperatorsV3', + 'createEncryptionOperators', 'encryptedIndexes', - 'extractEncryptionSchemaV3', + 'extractEncryptionSchema', 'getEqlV3Column', 'isEqlV3Column', 'makeEqlV3Column', diff --git a/packages/stack-drizzle/__tests__/v3/indexes.test.ts b/packages/stack-drizzle/__tests__/indexes.test.ts similarity index 98% rename from packages/stack-drizzle/__tests__/v3/indexes.test.ts rename to packages/stack-drizzle/__tests__/indexes.test.ts index d81a9bc8a..a3d57be21 100644 --- a/packages/stack-drizzle/__tests__/v3/indexes.test.ts +++ b/packages/stack-drizzle/__tests__/indexes.test.ts @@ -6,8 +6,8 @@ import { pgTable, } from 'drizzle-orm/pg-core' import { describe, expect, it } from 'vitest' -import { encryptedIndexes } from '../../src/v3/indexes' -import { types } from '../../src/v3/types' +import { encryptedIndexes } from '../src/indexes' +import { types } from '../src/types' // #753: the integration emitted the encrypted operators but no index DDL, so // encrypted predicates sequential-scanned by default. `encryptedIndexes` diff --git a/packages/stack-drizzle/__tests__/v3/operators.test-d.ts b/packages/stack-drizzle/__tests__/operators.test-d.ts similarity index 87% rename from packages/stack-drizzle/__tests__/v3/operators.test-d.ts rename to packages/stack-drizzle/__tests__/operators.test-d.ts index 7c04db648..b5e1be918 100644 --- a/packages/stack-drizzle/__tests__/v3/operators.test-d.ts +++ b/packages/stack-drizzle/__tests__/operators.test-d.ts @@ -6,12 +6,12 @@ import type { LockContext } from '@cipherstash/stack/identity' import type { EncryptedQueryResult } from '@cipherstash/stack/types' import type { EncryptionV3 } from '@cipherstash/stack/v3' import { describe, expectTypeOf, it } from 'vitest' -import { createEncryptionOperatorsV3 } from '../../src/v3/index.js' +import { createEncryptionOperators } from '../src/index.js' /** - * Static regression guard for M1: `createEncryptionOperatorsV3` must accept the + * Static regression guard for M1: `createEncryptionOperators` must accept the * `TypedEncryptionClient` that `EncryptionV3` resolves to — the documented - * `createEncryptionOperatorsV3(await EncryptionV3({ schemas }))` usage — as well + * `createEncryptionOperators(await EncryptionV3({ schemas }))` usage — as well * as the nominal `EncryptionClient` and a hand-rolled `{ encryptQuery }` double, * none requiring a cast. Typing the parameter to `EncryptionClient` (the * original bug) makes the first call below a compile error, which this suite @@ -20,7 +20,7 @@ import { createEncryptionOperatorsV3 } from '../../src/v3/index.js' * forms; the doubles model that. Lives in a `*.test-d.ts` so it is inside the * existing typecheck scope without dragging the loose-typed runtime suites in. */ -describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { +describe('createEncryptionOperators - client parameter (M1)', () => { type V3Client = Awaited> // A query operation resolving `Result` — the surface the factory drives. @@ -31,11 +31,11 @@ describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { } it('accepts the client EncryptionV3 returns with no cast', () => { - expectTypeOf(createEncryptionOperatorsV3).toBeCallableWith({} as V3Client) + expectTypeOf(createEncryptionOperators).toBeCallableWith({} as V3Client) }) it('still accepts the nominal EncryptionClient', () => { - expectTypeOf(createEncryptionOperatorsV3).toBeCallableWith( + expectTypeOf(createEncryptionOperators).toBeCallableWith( {} as EncryptionClient, ) }) @@ -57,7 +57,7 @@ describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { // the operand-client contract, so a structural double must supply it too. encrypt: (_value: never, _opts: never): QueryOp => ({}) as never, } - expectTypeOf(createEncryptionOperatorsV3).toBeCallableWith(double) + expectTypeOf(createEncryptionOperators).toBeCallableWith(double) }) it('rejects an { encryptQuery } double that resolves `unknown` (the erasure regression)', () => { @@ -76,6 +76,6 @@ describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { } // @ts-expect-error — `encryptQuery` resolving `unknown` does not satisfy the // factory's `ChainableOperation` client contract. - createEncryptionOperatorsV3(erased) + createEncryptionOperators(erased) }) }) diff --git a/packages/stack-drizzle/__tests__/v3/operators.test.ts b/packages/stack-drizzle/__tests__/operators.test.ts similarity index 97% rename from packages/stack-drizzle/__tests__/v3/operators.test.ts rename to packages/stack-drizzle/__tests__/operators.test.ts index 3a5ccc120..bc2456173 100644 --- a/packages/stack-drizzle/__tests__/v3/operators.test.ts +++ b/packages/stack-drizzle/__tests__/operators.test.ts @@ -17,13 +17,13 @@ import { } from 'drizzle-orm' import { integer, PgDialect, pgTable } from 'drizzle-orm/pg-core' import { describe, expect, it, vi } from 'vitest' -import { makeEqlV3Column } from '../../src/v3/column' +import { makeEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, + createEncryptionOperators, EncryptionOperatorError, -} from '../../src/v3/operators' -import { extractEncryptionSchemaV3 } from '../../src/v3/schema-extraction' -import { types } from '../../src/v3/types' +} from '../src/operators' +import { extractEncryptionSchema } from '../src/schema-extraction' +import { types } from '../src/types' // A query TERM (from `encryptQuery`), not a storage envelope: it carries index // terms but NO ciphertext `c` — that is the whole point of the encrypt → @@ -75,7 +75,7 @@ function setup(termImpl: (value?: unknown) => unknown = () => TERM) { // The factory's `client` parameter is the structural `{ encryptQuery }` // surface, so this hand-rolled double satisfies it with no cast. const client = { encryptQuery } - const ops = createEncryptionOperatorsV3(client, { lockContext, audit }) + const ops = createEncryptionOperators(client, { lockContext, audit }) const dialect = new PgDialect() const render = (s: unknown) => dialect.sqlToQuery(s as SQL) return { ops, encryptQuery, render } @@ -138,7 +138,7 @@ const users = pgTable('users', { flag: types.Boolean('flag'), }) -describe('createEncryptionOperatorsV3 - equality', () => { +describe('createEncryptionOperators - equality', () => { it.each( equalityDomains, )('%s eq emits the latest two-arg eql_v3.eq with a query-term operand', async (eqlType, spec) => { @@ -198,7 +198,7 @@ describe('createEncryptionOperatorsV3 - equality', () => { await ops.eq(second.age, 37) expect(encryptQuery.mock.calls[1]?.[1]?.table.build()).toEqual( - extractEncryptionSchemaV3(second).build(), + extractEncryptionSchema(second).build(), ) }) @@ -228,7 +228,7 @@ describe('createEncryptionOperatorsV3 - equality', () => { }) }) -describe('createEncryptionOperatorsV3 - comparison & range', () => { +describe('createEncryptionOperators - comparison & range', () => { it.each([ ['gt', 'eql_v3.gt'], ['gte', 'eql_v3.gte'], @@ -355,7 +355,7 @@ describe('createEncryptionOperatorsV3 - comparison & range', () => { }) }) -describe('createEncryptionOperatorsV3 - free-text match', () => { +describe('createEncryptionOperators - free-text match', () => { it.each( matchDomains, )('%s matches emits latest eql_v3.matches with a query-term operand', async (eqlType, spec) => { @@ -413,7 +413,7 @@ describe('createEncryptionOperatorsV3 - free-text match', () => { }) }) -describe('createEncryptionOperatorsV3 - JSON containment', () => { +describe('createEncryptionOperators - JSON containment', () => { const JSON_TYPE = 'public.eql_v3_json_search' // json has no `eql_v3.matches` overload: containment is the `@>` operator, @@ -476,7 +476,7 @@ describe('createEncryptionOperatorsV3 - JSON containment', () => { }) }) -describe('createEncryptionOperatorsV3 - JSON selectors', () => { +describe('createEncryptionOperators - JSON selectors', () => { const JSON_TYPE = 'public.eql_v3_json_search' const column = matrixColumn(JSON_TYPE) const VALUE_SELECTOR = { sv: [{ s: 'value-selector' }] } @@ -565,7 +565,7 @@ describe('createEncryptionOperatorsV3 - JSON selectors', () => { }) }) -describe('createEncryptionOperatorsV3 - domains with no scalar query', () => { +describe('createEncryptionOperators - domains with no scalar query', () => { it.each(nonScalarQueryDomains)('%s eq throws', async (eqlType, spec) => { const { ops } = setup() await expect( @@ -588,7 +588,7 @@ describe('createEncryptionOperatorsV3 - domains with no scalar query', () => { }) }) -describe('createEncryptionOperatorsV3 - array, ordering, combinators', () => { +describe('createEncryptionOperators - array, ordering, combinators', () => { it('inArray ORs one eq term per value; empty array throws', async () => { const { ops, render } = setup() const q = render(await ops.inArray(users.nickname, ['ada', 'grace'])) @@ -784,7 +784,7 @@ describe('createEncryptionOperatorsV3 - array, ordering, combinators', () => { }) }) -describe('createEncryptionOperatorsV3 - gating errors', () => { +describe('createEncryptionOperators - gating errors', () => { it('wraps encryption failures with operator context', async () => { const { ops, encryptQuery } = setup() encryptQuery.mockReturnValueOnce( diff --git a/packages/stack-drizzle/__tests__/v3/schema-extraction.test.ts b/packages/stack-drizzle/__tests__/schema-extraction.test.ts similarity index 82% rename from packages/stack-drizzle/__tests__/v3/schema-extraction.test.ts rename to packages/stack-drizzle/__tests__/schema-extraction.test.ts index e3767fcc0..4c9c32a6d 100644 --- a/packages/stack-drizzle/__tests__/v3/schema-extraction.test.ts +++ b/packages/stack-drizzle/__tests__/schema-extraction.test.ts @@ -1,10 +1,10 @@ import { encryptedTable, types as v3Types } from '@cipherstash/stack/eql/v3' import { integer, pgTable } from 'drizzle-orm/pg-core' import { describe, expect, it } from 'vitest' -import { extractEncryptionSchemaV3 } from '../../src/v3/schema-extraction' -import { types } from '../../src/v3/types' +import { extractEncryptionSchema } from '../src/schema-extraction' +import { types } from '../src/types' -describe('extractEncryptionSchemaV3', () => { +describe('extractEncryptionSchema', () => { it('rebuilds an equivalent eql/v3 encryptedTable from every drizzle v3 factory', () => { const drizzleColumns = Object.fromEntries( Object.entries(types).map(([factoryName, factory]) => [ @@ -19,7 +19,7 @@ describe('extractEncryptionSchemaV3', () => { ]), ) - const extracted = extractEncryptionSchemaV3( + const extracted = extractEncryptionSchema( pgTable('users', drizzleColumns as never), ) const authored = encryptedTable('users', authoredColumns as never) @@ -34,7 +34,7 @@ describe('extractEncryptionSchemaV3', () => { age: types.IntegerOrd('age'), }) - const extracted = extractEncryptionSchemaV3(users) + const extracted = extractEncryptionSchema(users) const authored = encryptedTable('users', { email: v3Types.TextSearch('email'), age: v3Types.IntegerOrd('age'), @@ -51,8 +51,8 @@ describe('extractEncryptionSchemaV3', () => { email: types.IntegerOrd('email'), }) - const accountsSchema = extractEncryptionSchemaV3(accounts) - const metricsSchema = extractEncryptionSchemaV3(metrics) + const accountsSchema = extractEncryptionSchema(accounts) + const metricsSchema = extractEncryptionSchema(metrics) expect(accountsSchema.build()).toStrictEqual( encryptedTable('accounts', { @@ -72,7 +72,7 @@ describe('extractEncryptionSchemaV3', () => { emailAddress: types.TextEq('email_address'), }) - expect(extractEncryptionSchemaV3(users).build()).toStrictEqual( + expect(extractEncryptionSchema(users).build()).toStrictEqual( encryptedTable('users', { createdOn: v3Types.Date('created_on'), emailAddress: v3Types.TextEq('email_address'), @@ -89,7 +89,7 @@ describe('extractEncryptionSchemaV3', () => { }, } - expect(extractEncryptionSchemaV3(table as never).build()).toStrictEqual( + expect(extractEncryptionSchema(table as never).build()).toStrictEqual( encryptedTable('users', { createdOn: v3Types.Date('created_on'), }).build(), @@ -98,12 +98,12 @@ describe('extractEncryptionSchemaV3', () => { it('throws when the table has no encrypted v3 columns', () => { const plain = pgTable('plain', { id: integer() }) - expect(() => extractEncryptionSchemaV3(plain)).toThrow(/no encrypted v3/i) + expect(() => extractEncryptionSchema(plain)).toThrow(/no encrypted v3/i) }) it('throws the table-name error before checking for encrypted columns', () => { expect(() => - extractEncryptionSchemaV3({ + extractEncryptionSchema({ secret: { getSQLType: () => 'public.eql_v3_text_eq', name: 'secret' }, } as never), ).toThrow('Unable to read table name from Drizzle table.') diff --git a/packages/stack-drizzle/__tests__/v3/selector.test.ts b/packages/stack-drizzle/__tests__/selector.test.ts similarity index 96% rename from packages/stack-drizzle/__tests__/v3/selector.test.ts rename to packages/stack-drizzle/__tests__/selector.test.ts index e3c2296cf..cf2fec265 100644 --- a/packages/stack-drizzle/__tests__/v3/selector.test.ts +++ b/packages/stack-drizzle/__tests__/selector.test.ts @@ -1,12 +1,12 @@ import { integer, pgTable } from 'drizzle-orm/pg-core' import { describe, expect, it } from 'vitest' import { - createEncryptionOperatorsV3, + createEncryptionOperators, EncryptionOperatorError, parseSelectorSegments, reconstructSelectorDocument, -} from '../../src/v3/operators.js' -import { types } from '../../src/v3/types.js' +} from '../src/operators.js' +import { types } from '../src/types.js' describe('parseSelectorSegments', () => { it('parses $-rooted, bare, and whitespace-padded dot paths', () => { @@ -70,7 +70,7 @@ describe('ops.selector — up-front guards (no encryption reached)', () => { const failIfCalled = () => { throw new Error('guard should reject before encrypting') } - const ops = createEncryptionOperatorsV3({ + const ops = createEncryptionOperators({ encryptQuery: failIfCalled, encrypt: failIfCalled, } as never) diff --git a/packages/stack-drizzle/__tests__/v3/sql-dialect.test.ts b/packages/stack-drizzle/__tests__/sql-dialect.test.ts similarity index 98% rename from packages/stack-drizzle/__tests__/v3/sql-dialect.test.ts rename to packages/stack-drizzle/__tests__/sql-dialect.test.ts index 42feef81a..bd67750b3 100644 --- a/packages/stack-drizzle/__tests__/v3/sql-dialect.test.ts +++ b/packages/stack-drizzle/__tests__/sql-dialect.test.ts @@ -1,7 +1,7 @@ import { not, sql } from 'drizzle-orm' import { PgDialect } from 'drizzle-orm/pg-core' import { describe, expect, it } from 'vitest' -import { EQL_V3_FN_SCHEMA, v3Dialect } from '../../src/v3/sql-dialect' +import { EQL_V3_FN_SCHEMA, v3Dialect } from '../src/sql-dialect' const dialect = new PgDialect() const render = (s: ReturnType) => dialect.sqlToQuery(s).sql diff --git a/packages/stack-drizzle/__tests__/v3/types.test-d.ts b/packages/stack-drizzle/__tests__/types.test-d.ts similarity index 97% rename from packages/stack-drizzle/__tests__/v3/types.test-d.ts rename to packages/stack-drizzle/__tests__/types.test-d.ts index 55b8022c2..18f81ae88 100644 --- a/packages/stack-drizzle/__tests__/v3/types.test-d.ts +++ b/packages/stack-drizzle/__tests__/types.test-d.ts @@ -1,7 +1,7 @@ import type { types as v3Types } from '@cipherstash/stack/eql/v3' import type { Encrypted } from '@cipherstash/stack/types' import { describe, expectTypeOf, it } from 'vitest' -import { types } from '../../src/v3/types' +import { types } from '../src/types' describe('v3 drizzle types - type-level', () => { it('exposes exactly the same factory keys as @/eql/v3 types', () => { diff --git a/packages/stack-drizzle/__tests__/v3/types.test.ts b/packages/stack-drizzle/__tests__/types.test.ts similarity index 87% rename from packages/stack-drizzle/__tests__/v3/types.test.ts rename to packages/stack-drizzle/__tests__/types.test.ts index face747e1..b82251a9a 100644 --- a/packages/stack-drizzle/__tests__/v3/types.test.ts +++ b/packages/stack-drizzle/__tests__/types.test.ts @@ -1,7 +1,7 @@ import { types as v3Types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' -import { getEqlV3Column } from '../../src/v3/column' -import { types } from '../../src/v3/types' +import { getEqlV3Column } from '../src/column' +import { types } from '../src/types' describe('v3 drizzle types namespace', () => { it('exposes the same factory names as @/eql/v3 types', () => { diff --git a/packages/stack-drizzle/integration/adapter.ts b/packages/stack-drizzle/integration/adapter.ts index 524ad33d1..a2e1ea848 100644 --- a/packages/stack-drizzle/integration/adapter.ts +++ b/packages/stack-drizzle/integration/adapter.ts @@ -11,11 +11,11 @@ import { asc, type SQL } from 'drizzle-orm' import { type PgTable, pgTable, text } from 'drizzle-orm/pg-core' import { drizzle } from 'drizzle-orm/postgres-js' import postgres from 'postgres' -import { makeEqlV3Column } from '../src/v3/column' +import { makeEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, - extractEncryptionSchemaV3, -} from '../src/v3/index.js' + createEncryptionOperators, + extractEncryptionSchema, +} from '../src/index.js' /** * The Drizzle v3 adapter under real ZeroKMS ciphertext. @@ -58,9 +58,9 @@ export function makeDrizzleAdapter(): IntegrationAdapter { let sqlClient: postgres.Sql let db: ReturnType let client: Awaited> - let ops: ReturnType + let ops: ReturnType let table: AnyTable - let schema: ReturnType + let schema: ReturnType let tableName: string const column = (slug: string) => table[slug] @@ -140,7 +140,7 @@ export function makeDrizzleAdapter(): IntegrationAdapter { rowKey: text('row_key').primaryKey(), ...columns, }) - schema = extractEncryptionSchemaV3(table) + schema = extractEncryptionSchema(table) // The DDL comes from the column builders, not from a hand-written list, so // a domain rename cannot silently desync the table from the schema. @@ -158,7 +158,7 @@ export function makeDrizzleAdapter(): IntegrationAdapter { // The client must know this table's schema to encrypt for it. Rebuilt per // family, since each family has its own columns. client = await EncryptionV3({ schemas: [schema as never] }) - ops = createEncryptionOperatorsV3(client) + ops = createEncryptionOperators(client) }, async insertSingle(_spec: TableSpec, row: PlainRow) { diff --git a/packages/stack-drizzle/integration/json-adapter.ts b/packages/stack-drizzle/integration/json-adapter.ts index 181b98fb5..2ed37577b 100644 --- a/packages/stack-drizzle/integration/json-adapter.ts +++ b/packages/stack-drizzle/integration/json-adapter.ts @@ -11,10 +11,10 @@ import { type PgTable, pgTable, text } from 'drizzle-orm/pg-core' import { drizzle } from 'drizzle-orm/postgres-js' import postgres from 'postgres' import { - createEncryptionOperatorsV3, - extractEncryptionSchemaV3, + createEncryptionOperators, + extractEncryptionSchema, types, -} from '../src/v3/index.js' +} from '../src/index.js' // biome-ignore lint/suspicious/noExplicitAny: the table name is supplied by the shared suite at runtime type AnyTable = any @@ -26,7 +26,7 @@ export function makeDrizzleJsonAdapter(): JsonIntegrationAdapter { let tableName: string let table: AnyTable let client: Awaited> - let ops: ReturnType + let ops: ReturnType const rowsFor = async ( where: SQL | undefined, @@ -71,9 +71,9 @@ export function makeDrizzleJsonAdapter(): JsonIntegrationAdapter { rowKey: text('row_key').primaryKey(), document: types.Json('document'), }) - const schema = extractEncryptionSchemaV3(table) + const schema = extractEncryptionSchema(table) client = await EncryptionV3({ schemas: [schema] }) - ops = createEncryptionOperatorsV3(client) + ops = createEncryptionOperators(client) await sqlClient.unsafe(`DROP TABLE IF EXISTS ${tableName}`) await sqlClient.unsafe(` diff --git a/packages/stack-drizzle/integration/lock-context.integration.test.ts b/packages/stack-drizzle/integration/lock-context.integration.test.ts index 293c34853..8167c0edf 100644 --- a/packages/stack-drizzle/integration/lock-context.integration.test.ts +++ b/packages/stack-drizzle/integration/lock-context.integration.test.ts @@ -41,11 +41,11 @@ import { integer, pgTable, text } from 'drizzle-orm/pg-core' import { drizzle } from 'drizzle-orm/postgres-js' import postgres from 'postgres' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { makeEqlV3Column } from '../src/v3/column' +import { makeEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, - extractEncryptionSchemaV3, -} from '../src/v3/index.js' + createEncryptionOperators, + extractEncryptionSchema, +} from '../src/index.js' const sqlClient = postgres(databaseUrl(), { prepare: false }) @@ -67,12 +67,12 @@ const secretTable = pgTable(TABLE_NAME, { secret: makeEqlV3Column(V3_MATRIX['public.eql_v3_text_eq'].builder('secret')), } as never) -const schema = extractEncryptionSchemaV3(secretTable) +const schema = extractEncryptionSchema(secretTable) type SelectRow = { rowKey: string } let client: Awaited> -let ops: ReturnType +let ops: ReturnType let db: ReturnType /** @@ -159,7 +159,7 @@ beforeAll(async () => { schemas: [schema], config: { authStrategy: federation.data }, }) - ops = createEncryptionOperatorsV3(client) + ops = createEncryptionOperators(client) db = drizzle({ client: sqlClient }) await sqlClient.unsafe(` diff --git a/packages/stack-drizzle/integration/null-persistence.integration.test.ts b/packages/stack-drizzle/integration/null-persistence.integration.test.ts index 6534c60ac..f97c1d29f 100644 --- a/packages/stack-drizzle/integration/null-persistence.integration.test.ts +++ b/packages/stack-drizzle/integration/null-persistence.integration.test.ts @@ -20,11 +20,11 @@ import { integer, pgTable, text } from 'drizzle-orm/pg-core' import { drizzle } from 'drizzle-orm/postgres-js' import postgres from 'postgres' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { makeEqlV3Column } from '../src/v3/column' +import { makeEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, - extractEncryptionSchemaV3, -} from '../src/v3/index.js' + createEncryptionOperators, + extractEncryptionSchema, +} from '../src/index.js' const sqlClient = postgres(databaseUrl(), { prepare: false }) @@ -80,12 +80,12 @@ const TIERS = [ }, ] as const -const schema = extractEncryptionSchemaV3(nullableTable) +const schema = extractEncryptionSchema(nullableTable) type SelectRow = { rowKey: string } let client: Awaited> -let ops: ReturnType +let ops: ReturnType let db: ReturnType function unwrap(result: { data?: T; failure?: { message: string } }): T { @@ -108,7 +108,7 @@ async function selectRowKeys(condition: SQL): Promise { beforeAll(async () => { // EQL v3 is installed once per run by `global-setup.ts`. client = await EncryptionV3({ schemas: [schema] }) - ops = createEncryptionOperatorsV3(client) + ops = createEncryptionOperators(client) db = drizzle({ client: sqlClient }) const columnDefs = TIERS.map((t) => `"${t.db}" ${t.domain}`).join(',\n ') diff --git a/packages/stack-drizzle/integration/relational.integration.test.ts b/packages/stack-drizzle/integration/relational.integration.test.ts index ee36fcf7f..8e6eecb38 100644 --- a/packages/stack-drizzle/integration/relational.integration.test.ts +++ b/packages/stack-drizzle/integration/relational.integration.test.ts @@ -43,13 +43,13 @@ import { integer, pgTable, text } from 'drizzle-orm/pg-core' import { drizzle } from 'drizzle-orm/postgres-js' import postgres from 'postgres' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { makeEqlV3Column } from '../src/v3/column' +import { makeEqlV3Column } from '../src/column' import { - createEncryptionOperatorsV3, + createEncryptionOperators, EncryptionOperatorError, - extractEncryptionSchemaV3, + extractEncryptionSchema, types as v3drizzle, -} from '../src/v3/index.js' +} from '../src/index.js' const sqlClient = postgres(databaseUrl(), { prepare: false }) @@ -128,8 +128,8 @@ const BIGINT_LEDGER = -9223372036854775808n const BIGINT_B_BALANCE = -5n const BIGINT_B_LEDGER = 100n -const schema = extractEncryptionSchemaV3(matrixTable) -const bigintSchema = extractEncryptionSchemaV3(bigintTable) +const schema = extractEncryptionSchema(matrixTable) +const bigintSchema = extractEncryptionSchema(bigintTable) type PlainValue = string | number | bigint | boolean | Date type RowKey = (typeof ROWS)[number] @@ -137,7 +137,7 @@ type MatrixPlainRow = Record type SelectRow = { rowKey: string } type Db = ReturnType type Client = Awaited> -type Ops = ReturnType +type Ops = ReturnType type ComparisonOperator = 'gt' | 'gte' | 'lt' | 'lte' let client: Client @@ -213,7 +213,7 @@ function encryptedInsertRows(): MatrixPlainRow[] { beforeAll(async () => { client = await EncryptionV3({ schemas: [schema, bigintSchema] }) - ops = createEncryptionOperatorsV3(client) + ops = createEncryptionOperators(client) db = drizzle({ client: sqlClient }) const columnDefs = matrixEntries diff --git a/packages/stack-drizzle/package.json b/packages/stack-drizzle/package.json index dd004ad62..48320122c 100644 --- a/packages/stack-drizzle/package.json +++ b/packages/stack-drizzle/package.json @@ -43,25 +43,8 @@ "default": "./dist/index.cjs" } }, - "./v3": { - "import": { - "types": "./dist/v3/index.d.ts", - "default": "./dist/v3/index.js" - }, - "require": { - "types": "./dist/v3/index.d.cts", - "default": "./dist/v3/index.cjs" - } - }, "./package.json": "./package.json" }, - "typesVersions": { - "*": { - "v3": [ - "./dist/v3/index.d.ts" - ] - } - }, "scripts": { "prebuild": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\"", "build": "tsup", diff --git a/packages/stack-drizzle/src/v3/codec.ts b/packages/stack-drizzle/src/codec.ts similarity index 100% rename from packages/stack-drizzle/src/v3/codec.ts rename to packages/stack-drizzle/src/codec.ts diff --git a/packages/stack-drizzle/src/v3/column.ts b/packages/stack-drizzle/src/column.ts similarity index 100% rename from packages/stack-drizzle/src/v3/column.ts rename to packages/stack-drizzle/src/column.ts diff --git a/packages/stack-drizzle/src/index.ts b/packages/stack-drizzle/src/index.ts index ff3f12bb0..07afb0e90 100644 --- a/packages/stack-drizzle/src/index.ts +++ b/packages/stack-drizzle/src/index.ts @@ -1,234 +1,9 @@ -import type { - CastAs, - MatchIndexOpts, - TokenFilter, -} from '@cipherstash/stack/schema' -import { customType } from 'drizzle-orm/pg-core' - -export type { CastAs, MatchIndexOpts, TokenFilter } - -// The encrypted column type is created by the EQL install script in the -// `public` schema (see packages/cli/src/installer/index.ts). We return the -// bare identifier here — drizzle-kit's CREATE TABLE path emits it correctly -// (it only prepends a schema when `typeSchema` is set, which is only true -// for pgEnum columns). The ALTER COLUMN path is a different story: it -// unconditionally wraps the dataType() return in double-quotes and prepends -// `"{typeSchema}".`, and since custom types have no typeSchema, the output -// becomes `"undefined"."eql_v2_encrypted"`. Returning a pre-quoted -// `"public"."eql_v2_encrypted"` here does NOT fix that — drizzle-kit just -// double-escapes the quotes, producing `"undefined".""public"."eql_v2_encrypted""`. -// Instead, the CLI's `rewriteEncryptedAlterColumns` rewrites every broken -// ALTER COLUMN form into an ADD + DROP + RENAME sequence that does work. -const EQL_ENCRYPTED_DATA_TYPE = 'eql_v2_encrypted' - -/** - * Configuration for encrypted column indexes and data types - */ -export type EncryptedColumnConfig = { - /** - * Data type for the column (default: 'string') - */ - dataType?: CastAs - /** - * Enable free text search. Can be a boolean for default options, or an object for custom configuration. - */ - freeTextSearch?: boolean | MatchIndexOpts - /** - * Enable equality index. Can be a boolean for default options, or an array of token filters. - */ - equality?: boolean | TokenFilter[] - /** - * Enable order and range index for sorting and range queries. - */ - orderAndRange?: boolean - /** - * Enable searchable JSON index for JSONB path queries. - * Requires dataType: 'json'. - */ - searchableJson?: boolean -} - -/** - * Map to store configuration for encrypted columns - * Keyed by column name (the name passed to encryptedType) - */ -const columnConfigMap = new Map< - string, - EncryptedColumnConfig & { name: string } ->() - -/** - * Creates an encrypted column type for Drizzle ORM with configurable searchable encryption options. - * - * When data is encrypted, the actual stored value is an [EQL v2](/docs/reference/eql) encrypted composite type which includes any searchable encryption indexes defined for the column. - * Importantly, the original data type is not known until it is decrypted. Therefore, this function allows specifying - * the original data type via the `dataType` option in the configuration. - * This ensures that when data is decrypted, it can be correctly interpreted as the intended TypeScript type. - * - * @typeParam TData - The TypeScript type of the data stored in the column - * @param name - The column name in the database - * @param config - Optional configuration for data type and searchable encryption indexes - * @returns A Drizzle column type that can be used in pgTable definitions - * - * ## Searchable Encryption Options - * - * - `dataType`: Specifies the original data type of the column (e.g., 'string', 'number', 'json'). Default is 'string'. - * - `freeTextSearch`: Enables free text search index. Can be a boolean for default options, or an object for custom configuration. - * - `equality`: Enables equality index. Can be a boolean for default options, or an array of token filters. - * - `orderAndRange`: Enables order and range index for sorting and range queries. - * - `searchableJson`: Enables searchable JSON index for JSONB path queries on encrypted JSON columns. - * - * See {@link EncryptedColumnConfig}. - * - * @example - * Defining a drizzle table schema for postgres table with encrypted columns. - * - * ```typescript - * import { pgTable, integer, timestamp } from 'drizzle-orm/pg-core' - * import { encryptedType } from '@cipherstash/stack-drizzle' - * - * const users = pgTable('users', { - * email: encryptedType('email', { - * freeTextSearch: true, - * equality: true, - * orderAndRange: true, - * }), - * age: encryptedType('age', { - * dataType: 'number', - * equality: true, - * orderAndRange: true, - * }), - * profile: encryptedType('profile', { - * dataType: 'json', - * }), - * }) - * ``` - */ -export const encryptedType = ( - name: string, - config?: EncryptedColumnConfig, -) => { - // Create the Drizzle custom type - const customColumnType = customType<{ data: TData; driverData: string }>({ - dataType() { - return EQL_ENCRYPTED_DATA_TYPE - }, - toDriver(value: TData): string { - const jsonStr = JSON.stringify(value) - const escaped = jsonStr.replace(/"/g, '""') - return `("${escaped}")` - }, - fromDriver(value: string): TData { - const parseComposite = (str: string) => { - if (!str || str === '') return null - - const trimmed = str.trim() - - if (trimmed.startsWith('(') && trimmed.endsWith(')')) { - let inner = trimmed.slice(1, -1) - inner = inner.replace(/""/g, '"') - - if (inner.startsWith('"') && inner.endsWith('"')) { - const stripped = inner.slice(1, -1) - return JSON.parse(stripped) - } - - if (inner.startsWith('{') || inner.startsWith('[')) { - return JSON.parse(inner) - } - - return inner - } - - return JSON.parse(str) - } - - return parseComposite(value) as TData - }, - }) - - // Create the column instance - const column = customColumnType(name) - - // Store configuration keyed by column name - // This allows us to look it up during schema extraction - const fullConfig: EncryptedColumnConfig & { name: string } = { - name, - ...config, - } - - // Store in Map keyed by column name (will be looked up during extraction) - columnConfigMap.set(name, fullConfig) - - // Also store on property for immediate access (before pgTable processes it) - // We need to use any here because Drizzle columns don't have a type for custom properties - // biome-ignore lint/suspicious/noExplicitAny: Drizzle columns don't expose custom property types - ;(column as any)._encryptionConfig = fullConfig - - return column -} - -/** - * Get configuration for an encrypted column by checking if it's an encrypted type - * and looking up the config by column name - * @internal - */ -export function getEncryptedColumnConfig( - columnName: string, - column: unknown, -): (EncryptedColumnConfig & { name: string }) | undefined { - // Check if this is an encrypted column - if (column && typeof column === 'object') { - // We need to use any here to access Drizzle column properties - // biome-ignore lint/suspicious/noExplicitAny: Drizzle column types don't expose all properties - const columnAny = column as any - - // Check if it's an encrypted column by checking sqlName or dataType. - // We accept both the bare `eql_v2_encrypted` form (current) and the - // fully-qualified `"public"."eql_v2_encrypted"` form that @cipherstash/stack - // 0.15.0 briefly emitted, for back-compat with tables built against that - // release. - const isEncryptedTypeString = (value: unknown): boolean => - value === EQL_ENCRYPTED_DATA_TYPE || - value === '"public"."eql_v2_encrypted"' - - const isEncrypted = - isEncryptedTypeString(columnAny.sqlName) || - isEncryptedTypeString(columnAny.dataType) || - (columnAny.dataType && - typeof columnAny.dataType === 'function' && - isEncryptedTypeString(columnAny.dataType())) - - if (isEncrypted) { - // Try to get config from property (if still there) - if (columnAny._encryptionConfig) { - return columnAny._encryptionConfig - } - - // Look up config by column name (the name passed to encryptedType) - // The column.name should match what was passed to encryptedType - const lookupName = columnAny.name || columnName - return columnConfigMap.get(lookupName) - } - } - return undefined -} - -/** - * Create Drizzle query operators (`eq`, `lt`, `gt`, etc.) that work with - * encrypted columns. The returned operators encrypt query values before - * passing them to Drizzle, enabling searchable encryption in standard - * Drizzle queries. - */ +export { EqlV3CodecError, v3FromDriver, v3ToDriver } from './codec.js' +export { getEqlV3Column, isEqlV3Column, makeEqlV3Column } from './column.js' +export { encryptedIndexes } from './indexes.js' export { createEncryptionOperators, - EncryptionConfigError, EncryptionOperatorError, } from './operators.js' -/** - * Extract a CipherStash encryption schema from a Drizzle table definition. - * - * Inspects columns created with {@link encryptedType} and builds the equivalent - * `encryptedTable` / `encryptedColumn` schema automatically. - */ export { extractEncryptionSchema } from './schema-extraction.js' +export { types } from './types.js' diff --git a/packages/stack-drizzle/src/v3/indexes.ts b/packages/stack-drizzle/src/indexes.ts similarity index 99% rename from packages/stack-drizzle/src/v3/indexes.ts rename to packages/stack-drizzle/src/indexes.ts index 0649a090a..d43afa58f 100644 --- a/packages/stack-drizzle/src/v3/indexes.ts +++ b/packages/stack-drizzle/src/indexes.ts @@ -14,7 +14,7 @@ import { EQL_V3_FN_SCHEMA } from './sql-dialect.js' * * ```ts * import { integer, pgTable } from 'drizzle-orm/pg-core' - * import { encryptedIndexes, types } from '@cipherstash/stack-drizzle/v3' + * import { encryptedIndexes, types } from '@cipherstash/stack-drizzle' * * export const users = pgTable( * 'users', diff --git a/packages/stack-drizzle/src/operators.ts b/packages/stack-drizzle/src/operators.ts index 674bb3ba3..4f6df813d 100644 --- a/packages/stack-drizzle/src/operators.ts +++ b/packages/stack-drizzle/src/operators.ts @@ -1,90 +1,96 @@ -import type { EncryptionClient } from '@cipherstash/stack/encryption' +import type { Result } from '@byteslice/result' +import type { AuditConfig } from '@cipherstash/stack/adapter-kit' +import { + jsonPathOf, + matchNeedleError, + parseSelectorSegments, + reconstructSelectorDocument, + stripDomainSchema, + unsupportedLeafReason, +} from '@cipherstash/stack/adapter-kit' +import type { + AnyEncryptedV3Column, + AnyV3Table, +} from '@cipherstash/stack/eql/v3' +import type { EncryptionError } from '@cipherstash/stack/errors' +import type { LockContext } from '@cipherstash/stack/identity' +import type { ColumnSchema } from '@cipherstash/stack/schema' import type { - EncryptedColumn, - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' -import { type QueryTypeName, queryTypes } from '@cipherstash/stack/types' + EncryptedQueryResult, + QueryTypeName, +} from '@cipherstash/stack/types' import { and, - arrayContained, - arrayContains, - arrayOverlaps, asc, - between, - bindIfParam, + Column, desc, - eq, exists, - gt, - gte, - ilike, - inArray, + is, isNotNull, isNull, - like, - lt, - lte, - ne, not, - notBetween, notExists, - notIlike, - notInArray, or, type SQL, type SQLWrapper, sql, } from 'drizzle-orm' import type { PgTable } from 'drizzle-orm/pg-core' -import type { EncryptedColumnConfig } from './index.js' -import { getEncryptedColumnConfig } from './index.js' -import { extractEncryptionSchema } from './schema-extraction.js' - -// ============================================================================ -// Type Definitions and Type Guards -// ============================================================================ - -/** - * Branded type for Drizzle table with encrypted columns - */ -// biome-ignore lint/suspicious/noExplicitAny: Drizzle table types don't expose Symbol properties -type EncryptedDrizzleTable = PgTable & { - readonly __isEncryptedTable?: true -} +import { getEqlV3Column } from './column.js' +import { + extractEncryptionSchema, + getDrizzleTableName, +} from './schema-extraction.js' +import { type ComparisonOp, type EqualityOp, v3Dialect } from './sql-dialect.js' /** - * Type guard to check if a value is a Drizzle SQLWrapper + * The client capability this factory consumes: `encryptQuery`, in both its + * single (`value, opts`) and batch (`terms[]`) forms. Declared structurally — + * with maximally-permissive operands — so it is satisfied by the nominal + * `EncryptionClient`, by the `TypedEncryptionClient` that `EncryptionV3` returns + * (whatever its schema tuple), AND by a hand-rolled test double, none needing a + * cast. Typing the parameter to the nominal `TypedEncryptionClient` would + * reject a client built for a narrower schema tuple (it accepts fewer tables than + * `readonly AnyV3Table[]`); the structural surface sidesteps that variance. + * + * Every operand is a QUERY TERM, not a storage envelope: `encryptQuery` mints a + * ciphertext-free term (no `c`) carrying all of the column's configured index + * terms, which the operator layer casts to the column's `eql_v3.query_` + * type. This reaches the bundle's `(domain, query_)` overloads and keeps + * WHERE-clause payloads free of the ciphertext a query never needs (#622). + * + * `never` operands: the real client's `encryptQuery` is generic (`queryType` is + * constrained to the column's own query types), which a concrete signature here + * cannot match. `never` params keep the structural surface satisfiable by that + * generic method AND by a test double; the call sites cast their real operands. */ -function isSQLWrapper(value: unknown): value is SQLWrapper { - return ( - typeof value === 'object' && - value !== null && - 'sql' in value && - typeof (value as { sql: unknown }).sql !== 'undefined' - ) +type OperandEncryptionClient = { + encryptQuery( + value: never, + opts: never, + ): ChainableOperation + encryptQuery(terms: never): ChainableOperation } -/** - * Type guard to check if a value is a Drizzle table - */ -function isPgTable(value: unknown): value is EncryptedDrizzleTable { - return ( - typeof value === 'object' && - value !== null && - Symbol.for('drizzle:Name') in value - ) -} +// Path helpers now live in @cipherstash/stack/adapter-kit (shared with the +// Supabase adapter, #650); re-exported so existing imports keep working. +export { parseSelectorSegments, reconstructSelectorDocument } /** - * Custom error types for better debugging + * A dedicated error for v3 operator gating and operand-encryption failures, + * carrying the offending column/table/operator for diagnostics. + * + * INTENTIONAL FORK: this mirrors the v2 adapter's `EncryptionOperatorError` + * rather than sharing it. Unifying the two would couple `./drizzle` and + * `./eql/v3/drizzle` — two independently-versioned public entry points — so the + * duplication is deliberate, not an oversight. */ export class EncryptionOperatorError extends Error { constructor( message: string, public readonly context?: { - tableName?: string columnName?: string + tableName?: string operator?: string }, ) { @@ -93,1853 +99,861 @@ export class EncryptionOperatorError extends Error { } } -export class EncryptionConfigError extends EncryptionOperatorError { - constructor(message: string, context?: EncryptionOperatorError['context']) { - super(message, context) - this.name = 'EncryptionConfigError' - } -} - -// ============================================================================ -// Utility Functions -// ============================================================================ - -/** - * Helper to extract table name from a Drizzle table - */ -function getDrizzleTableName(drizzleTable: unknown): string | undefined { - if (!isPgTable(drizzleTable)) { - return undefined - } - // Access Symbol property using Record type to avoid indexing errors - const tableWithSymbol = drizzleTable as unknown as Record< - symbol, - string | undefined - > - return tableWithSymbol[Symbol.for('drizzle:Name')] -} - -/** - * Helper to get the drizzle table from a drizzle column - */ -function getDrizzleTableFromColumn(drizzleColumn: SQLWrapper): unknown { - const column = drizzleColumn as unknown as Record - return column.table as unknown -} - -/** - * Helper to extract encrypted table from a drizzle column by deriving it from the column's parent table - */ -function getEncryptedTableFromColumn( - drizzleColumn: SQLWrapper, - tableCache: Map>, -): EncryptedTable | undefined { - const drizzleTable = getDrizzleTableFromColumn(drizzleColumn) - if (!drizzleTable) { - return undefined - } - - const tableName = getDrizzleTableName(drizzleTable) - if (!tableName) { - return undefined - } - - // Check cache first - let encryptedTable = tableCache.get(tableName) - if (encryptedTable) { - return encryptedTable - } - - // Extract encryption schema from drizzle table and cache it - try { - // biome-ignore lint/suspicious/noExplicitAny: PgTable type doesn't expose all needed properties - encryptedTable = extractEncryptionSchema(drizzleTable as PgTable) - tableCache.set(tableName, encryptedTable) - return encryptedTable - } catch { - // Table doesn't have encrypted columns or extraction failed - return undefined - } -} - -/** - * Helper to get the encrypted column definition for a Drizzle column from the encrypted table - */ -function getEncryptedColumn( - drizzleColumn: SQLWrapper, - encryptedTable: EncryptedTable, -): EncryptedColumn | undefined { - const column = drizzleColumn as unknown as Record - const columnName = column.name as string | undefined - if (!columnName) { - return undefined - } - - const tableRecord = encryptedTable as unknown as Record - return tableRecord[columnName] as EncryptedColumn | undefined +interface ColumnContext { + builder: AnyEncryptedV3Column + table: AnyV3Table + indexes: ColumnSchema['indexes'] + columnName: string + tableName: string + /** The `eql_v3.query_` type an operand for this column casts to, so + * `encryptQuery`'s ciphertext-free term reaches the narrowed-query overloads. + * `null` for storage-only columns (no query domain); those never encrypt an + * operand — every operator gates on a query capability first. JSON columns + * override this at the call site (`query_json`, an irregular name). */ + queryCast: string | null } /** - * Column metadata extracted from a Drizzle column + * The `eql_v3.query_` cast for a column's storage domain — e.g. + * `public.eql_v3_text_search` → `eql_v3.query_text_search`. Uniform across the + * queryable column domains (`_eq`, `_ord`, `_ord_ore`, `_match`, `_search`); the + * two irregular cases are handled elsewhere: storage-only domains + * (`eql_v3_boolean`, the bare base types) have no query domain and return `null` + * (they are never queried), and `eql_v3_json_search` maps to `query_json`, cast + * explicitly on the JSON path. */ -interface ColumnInfo { - readonly encryptedColumn: EncryptedColumn | undefined - readonly config: (EncryptedColumnConfig & { name: string }) | undefined - readonly encryptedTable: EncryptedTable | undefined - readonly columnName: string - readonly tableName: string | undefined +function queryCastForDomain(eqlType: string): string | null { + const bare = stripDomainSchema(eqlType) // public.eql_v3_text_search → eql_v3_text_search + const prefix = 'eql_v3_' + if (!bare.startsWith(prefix)) return null + const suffix = bare.slice(prefix.length) + // No index suffix (bare storage-only domain like `boolean`, `text`) → no query + // domain exists. These are gated out before any operand is encrypted. The + // suffixes match the column factories in `@/eql/v3/columns` exactly: ope + // ordering is the `_ord` domain (not `_ord_ope`) and text search is `_search` + // (there is no `_search_ore` column), so those two never occur here. + if (!/_(eq|ord|ord_ore|match|search)$/.test(suffix)) { + return null + } + return `eql_v3.query_${suffix}` } -/** - * Helper to get the encrypted column and column config for a Drizzle column - * If encryptedTable is not provided, it will be derived from the column - */ -function getColumnInfo( - drizzleColumn: SQLWrapper, - encryptedTable: EncryptedTable | undefined, - tableCache: Map>, -): ColumnInfo { - const column = drizzleColumn as unknown as Record - const columnName = (column.name as string | undefined) || 'unknown' - - // If encryptedTable not provided, try to derive it from the column - let resolvedTable = encryptedTable - if (!resolvedTable) { - resolvedTable = getEncryptedTableFromColumn(drizzleColumn, tableCache) - } - - const drizzleTable = getDrizzleTableFromColumn(drizzleColumn) - const tableName = getDrizzleTableName(drizzleTable) - - if (!resolvedTable) { - // Column is not from an encrypted table - const config = getEncryptedColumnConfig(columnName, drizzleColumn) - return { - encryptedColumn: undefined, - config, - encryptedTable: undefined, - columnName, - tableName, - } - } - - const encryptedColumn = getEncryptedColumn(drizzleColumn, resolvedTable) - const config = getEncryptedColumnConfig(columnName, drizzleColumn) - - return { - encryptedColumn, - config, - encryptedTable: resolvedTable, - columnName, - tableName, - } +export type EncryptionOperatorCallOpts = { + lockContext?: LockContext + audit?: AuditConfig } /** - * Helper to convert a value to plaintext format + * An SDK encryption operation after its lock context has been applied: still + * auditable and awaitable, but not re-lockable. `withLockContext` returns this, + * not the full {@link ChainableOperation}, mirroring the real + * `EncryptOperationWithLockContext`, which drops `withLockContext` (you cannot + * lock-context twice). Modelling that is what lets the real client type satisfy + * the structural surface with no cast. */ -function toPlaintext(value: unknown): string | number | bigint { - if (typeof value === 'boolean') { - return value ? 1 : 0 - } - if ( - typeof value === 'string' || - typeof value === 'number' || - // `bigint` (int8 columns) must pass through as a native bigint, NOT via the - // `String(value)` fallthrough below — a stringified bigint would be - // encrypted as text and silently mismatch the column's bigint domain. The - // downstream `encryptQuery` term type is `Plaintext`, which carries bigint, - // and protect-ffi 0.28 i64-bounds-checks it at the boundary. - typeof value === 'bigint' - ) { - return value - } - if (value instanceof Date) { - return value.toISOString() - } - return String(value) +type AuditableOperation = { + audit(config: AuditConfig): AuditableOperation + then: PromiseLike>['then'] } /** - * Value to encrypt with its associated column + * The subset of an SDK encryption operation this factory drives: the fluent + * `withLockContext`/`audit` chain, and a `then` that resolves the operation's + * `Result`. Generic over the resolved payload `T` so the single `encryptQuery` + * carries an `EncryptedQueryResult` term and the batch form an + * `EncryptedQueryResult[]`, rather than the `unknown` this erased to before. + * + * Structural, not the concrete `EncryptQueryOperation` class, because the client + * is passed in and the factory must accept any implementation with this surface. */ -interface ValueToEncrypt { - readonly value: string | number | bigint - readonly column: SQLWrapper - readonly columnInfo: ColumnInfo - readonly queryType?: QueryTypeName - readonly originalIndex: number +type ChainableOperation = { + withLockContext(lockContext: LockContext): AuditableOperation + audit(config: AuditConfig): AuditableOperation + then: PromiseLike>['then'] } /** - * Helper to encrypt multiple values for use in a query - * Returns an array of encrypted search terms or original values if not encrypted + * Build v3-aware query operators (`eq`, `gte`, `matches`, `contains`, `asc`, …) bound to an + * encryption `client`. Each comparison/containment operator AUTO-ENCRYPTS its + * plaintext operand into an EQL v3 query term before handing it to Drizzle, so + * callers pass plaintext and the emitted SQL compares encrypted values. Every + * operator also gates on the target column's capabilities and throws + * {@link EncryptionOperatorError} when the column can't answer the operator + * (e.g. ordering a non-`ore` column). + * + * @param client - anything that can `encryptQuery` — the nominal + * `EncryptionClient` or the `TypedEncryptionClient` from `EncryptionV3` (no + * cast needed). + * @param defaults - lock context / audit applied to every operand encryption + * unless a per-call override is supplied. + * + * @example + * ```typescript + * const ops = createEncryptionOperators(await EncryptionV3({ schemas: [users] })) + * await db.select().from(users).where(await ops.eq(users.email, 'a@b.com')) + * ``` */ -async function encryptValues( - encryptionClient: EncryptionClient, - values: Array<{ - value: unknown - column: SQLWrapper - queryType?: QueryTypeName - }>, - encryptedTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise { - if (values.length === 0) { - return [] - } - - // Single pass: collect values to encrypt with their metadata - const valuesToEncrypt: ValueToEncrypt[] = [] - const results: unknown[] = new Array(values.length) - - for (let i = 0; i < values.length; i++) { - const { value, column, queryType } = values[i] - const columnInfo = getColumnInfo(column, encryptedTable, tableCache) - - if ( - !columnInfo.encryptedColumn || - !columnInfo.config || - !columnInfo.encryptedTable - ) { - // Column is not encrypted, return value as-is - results[i] = value - continue - } - - const plaintextValue = toPlaintext(value) - valuesToEncrypt.push({ - value: plaintextValue, - column, - columnInfo, - queryType, - originalIndex: i, - }) - } - - if (valuesToEncrypt.length === 0) { - return results - } - - // Group values by column to batch encrypt with same column/table - const columnGroups = new Map< - string, - { - column: EncryptedColumn - table: EncryptedTable - columnName: string - values: Array<{ - value: string | number | bigint - index: number - queryType?: QueryTypeName - }> - resultIndices: number[] - } - >() - - let valueIndex = 0 - for (const { - value, - columnInfo, - queryType, - originalIndex, - } of valuesToEncrypt) { - // Safe access with validation - we know these exist from earlier checks - if ( - !columnInfo.config || - !columnInfo.encryptedColumn || - !columnInfo.encryptedTable - ) { - continue - } - - const columnName = columnInfo.config.name - const groupKey = `${columnInfo.tableName ?? 'unknown'}/${columnName}` - let group = columnGroups.get(groupKey) - if (!group) { - group = { - column: columnInfo.encryptedColumn, - table: columnInfo.encryptedTable, - columnName, - values: [], - resultIndices: [], - } - columnGroups.set(groupKey, group) - } - group.values.push({ value, index: valueIndex++, queryType }) - group.resultIndices.push(originalIndex) - } - - // Encrypt all values for each column in batches - for (const [, group] of columnGroups) { - const { columnName } = group - try { - const terms = group.values.map((v) => ({ - value: v.value, - column: group.column, - table: group.table, - queryType: v.queryType, - })) - - const encryptedTerms = await encryptionClient.encryptQuery(terms) - - if (encryptedTerms.failure) { - throw new EncryptionOperatorError( - `Failed to encrypt query terms for column "${columnName}": ${encryptedTerms.failure.message}`, - { columnName }, - ) - } - - // Map results back to original indices - for (let i = 0; i < group.values.length; i++) { - const resultIndex = group.resultIndices[i] ?? -1 - if (resultIndex >= 0 && resultIndex < results.length) { - results[resultIndex] = encryptedTerms.data[i] - } - } - } catch (error) { - if (error instanceof EncryptionOperatorError) { - throw error - } - const errorMessage = - error instanceof Error ? error.message : String(error) +export function createEncryptionOperators( + client: OperandEncryptionClient, + defaults: EncryptionOperatorCallOpts = {}, +) { + const tableCache = new WeakMap() + // Per-column context memo. `resolveContext` is value-independent, so caching + // by column identity makes `inArray`/`notInArray` build the context (and its + // deep-cloned match block) once for the whole list instead of once per value. + const contextCache = new WeakMap() + + function drizzleTableOf(column: SQLWrapper): PgTable | undefined { + return is(column, Column) + ? (column.table as PgTable | undefined) + : undefined + } + + function resolveContext(column: SQLWrapper, operator: string): ColumnContext { + const cached = contextCache.get(column) + if (cached) return cached + + const columnName = is(column, Column) ? column.name : 'unknown' + const builder = getEqlV3Column(columnName, column) + if (!builder) { throw new EncryptionOperatorError( - `Unexpected error encrypting values for column "${columnName}": ${errorMessage}`, - { columnName }, + `Operator "${operator}" requires an encrypted v3 column, but "${columnName}" is not one.`, + { columnName, operator }, ) } - } - - return results -} -/** - * Helper to encrypt a single value for use in a query - * Returns the encrypted search term or the original value if not encrypted - */ -async function encryptValue( - encryptionClient: EncryptionClient, - value: unknown, - drizzleColumn: SQLWrapper, - encryptedTable: EncryptedTable | undefined, - tableCache: Map>, - queryType?: QueryTypeName, -): Promise { - const results = await encryptValues( - encryptionClient, - [{ value, column: drizzleColumn, queryType }], - encryptedTable, - tableCache, - ) - return results[0] -} - -// ============================================================================ -// Lazy Operator Pattern -// ============================================================================ - -/** - * Simplified lazy operator that defers encryption until awaited or batched - */ -interface LazyOperator { - readonly __isLazyOperator: true - readonly operator: string - readonly queryType?: QueryTypeName - readonly left: SQLWrapper - readonly right: unknown - readonly min?: unknown - readonly max?: unknown - readonly needsEncryption: boolean - readonly columnInfo: ColumnInfo - execute( - encrypted: unknown, - encryptedMin?: unknown, - encryptedMax?: unknown, - ): SQL -} - -/** - * Type guard for lazy operators - */ -function isLazyOperator(value: unknown): value is LazyOperator { - return ( - typeof value === 'object' && - value !== null && - '__isLazyOperator' in value && - (value as LazyOperator).__isLazyOperator === true - ) -} - -/** - * Creates a lazy operator that defers execution - */ -function createLazyOperator( - operator: string, - left: SQLWrapper, - right: unknown, - execute: ( - encrypted: unknown, - encryptedMin?: unknown, - encryptedMax?: unknown, - ) => SQL, - needsEncryption: boolean, - columnInfo: ColumnInfo, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, - min?: unknown, - max?: unknown, - queryType?: QueryTypeName, -): LazyOperator & Promise { - let resolvedSQL: SQL | undefined - let encryptionPromise: Promise | undefined - - const lazyOp: LazyOperator = { - __isLazyOperator: true, - operator, - queryType, - left, - right, - min, - max, - needsEncryption, - columnInfo, - execute, - } - - // Create a promise that will be resolved when encryption completes - const promise = new Promise((resolve, reject) => { - // Auto-execute when awaited directly - queueMicrotask(async () => { - if (resolvedSQL !== undefined) { - resolve(resolvedSQL) - return - } - - try { - if (!encryptionPromise) { - encryptionPromise = executeLazyOperatorDirect( - lazyOp, - encryptionClient, - defaultTable, - tableCache, - ) - } - const sql = await encryptionPromise - resolvedSQL = sql - resolve(sql) - } catch (error) { - reject(error) - } - }) - }) - - // Attach lazy operator properties to the promise - return Object.assign(promise, lazyOp) -} + const drizzleTable = drizzleTableOf(column) + const tableName = getDrizzleTableName(drizzleTable) ?? 'unknown' -/** - * Executes a lazy operator with pre-encrypted values (used in batched mode) - */ -async function executeLazyOperator( - lazyOp: LazyOperator, - encryptedValues?: { value: unknown; encrypted: unknown }[], -): Promise { - if (!lazyOp.needsEncryption) { - return lazyOp.execute(lazyOp.right) - } - - if (lazyOp.min !== undefined && lazyOp.max !== undefined) { - // Between operator - use provided encrypted values - let encryptedMin: unknown - let encryptedMax: unknown - - if (encryptedValues && encryptedValues.length >= 2) { - encryptedMin = encryptedValues[0]?.encrypted - encryptedMax = encryptedValues[1]?.encrypted - } else { - throw new EncryptionOperatorError( - 'Between operator requires both min and max encrypted values', - { - columnName: lazyOp.columnInfo.columnName, - tableName: lazyOp.columnInfo.tableName, - operator: lazyOp.operator, - }, - ) + let table = drizzleTable ? tableCache.get(drizzleTable) : undefined + if (!table && drizzleTable) { + table = extractEncryptionSchema(drizzleTable) + tableCache.set(drizzleTable, table) } - - if (encryptedMin === undefined || encryptedMax === undefined) { + if (!table) { throw new EncryptionOperatorError( - 'Between operator requires both min and max values to be encrypted', - { - columnName: lazyOp.columnInfo.columnName, - tableName: lazyOp.columnInfo.tableName, - operator: lazyOp.operator, - }, + `Unable to resolve the encrypted table for column "${columnName}".`, + { columnName, operator }, ) } - return lazyOp.execute(undefined, encryptedMin, encryptedMax) - } - - // Single value operator - let encrypted: unknown - - if (encryptedValues && encryptedValues.length > 0) { - encrypted = encryptedValues[0]?.encrypted - } else { - throw new EncryptionOperatorError( - 'Operator requires encrypted value but none provided', - { - columnName: lazyOp.columnInfo.columnName, - tableName: lazyOp.columnInfo.tableName, - operator: lazyOp.operator, - }, - ) - } - - if (encrypted === undefined) { - throw new EncryptionOperatorError( - 'Encryption failed or value was not encrypted', - { - columnName: lazyOp.columnInfo.columnName, - tableName: lazyOp.columnInfo.tableName, - operator: lazyOp.operator, - }, - ) - } - - return lazyOp.execute(encrypted) -} - -/** - * Executes a lazy operator directly by encrypting values on demand - * Used when operator is awaited directly (not batched) - */ -async function executeLazyOperatorDirect( - lazyOp: LazyOperator, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise { - if (!lazyOp.needsEncryption) { - return lazyOp.execute(lazyOp.right) - } - - if (lazyOp.min !== undefined && lazyOp.max !== undefined) { - // Between operator - encrypt min and max - const [encryptedMin, encryptedMax] = await encryptValues( - encryptionClient, - [ - { value: lazyOp.min, column: lazyOp.left, queryType: lazyOp.queryType }, - { value: lazyOp.max, column: lazyOp.left, queryType: lazyOp.queryType }, - ], - defaultTable, - tableCache, - ) - return lazyOp.execute(undefined, encryptedMin, encryptedMax) - } - - // Single value operator - const encrypted = await encryptValue( - encryptionClient, - lazyOp.right, - lazyOp.left, - defaultTable, - tableCache, - lazyOp.queryType, - ) - - return lazyOp.execute(encrypted) -} - -// ============================================================================ -// Operator Factory Functions -// ============================================================================ - -/** - * Creates a comparison operator (eq, ne, gt, gte, lt, lte) - */ -function createComparisonOperator( - operator: 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', - left: SQLWrapper, - right: unknown, - columnInfo: ColumnInfo, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise | SQL { - const { config } = columnInfo - - // Operators requiring orderAndRange index - const requiresOrderAndRange = ['gt', 'gte', 'lt', 'lte'].includes(operator) - - if (requiresOrderAndRange) { - if (!config?.orderAndRange) { - // Return regular Drizzle operator for non-encrypted columns - switch (operator) { - case 'gt': - return gt(left, right) - case 'gte': - return gte(left, right) - case 'lt': - return lt(left, right) - case 'lte': - return lte(left, right) - } - } - - // This will be replaced with encrypted value in executeLazyOperator - const executeFn = (encrypted: unknown) => { - if (encrypted === undefined) { - throw new EncryptionOperatorError( - `Encryption failed for ${operator} operator`, - { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, - operator, - }, - ) - } - return sql`eql_v2.${sql.raw(operator)}(${left}, ${bindIfParam(encrypted, left)})` - } - - return createLazyOperator( - operator, - left, - right, - executeFn, - true, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - undefined, // min - undefined, // max - queryTypes.orderAndRange, - ) as Promise - } - - // Equality operators (eq, ne) - const requiresEquality = ['eq', 'ne'].includes(operator) - - if (requiresEquality && config?.equality) { - const executeFn = (encrypted: unknown) => { - if (encrypted === undefined) { - throw new EncryptionOperatorError( - `Encryption failed for ${operator} operator`, - { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, - operator, - }, - ) - } - return operator === 'eq' ? eq(left, encrypted) : ne(left, encrypted) + const context: ColumnContext = { + builder, + table, + indexes: builder.build().indexes, + columnName, + tableName, + queryCast: queryCastForDomain(builder.getEqlType()), } - - return createLazyOperator( - operator, - left, - right, - executeFn, - true, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - undefined, // min - undefined, // max - queryTypes.equality, - ) as Promise + contextCache.set(column, context) + return context } - // Fallback to regular Drizzle operators - return operator === 'eq' ? eq(left, right) : ne(left, right) -} - -/** - * Creates a range operator (between, notBetween) - */ -function createRangeOperator( - operator: 'between' | 'notBetween', - left: SQLWrapper, - min: unknown, - max: unknown, - columnInfo: ColumnInfo, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise | SQL { - const { config } = columnInfo - - if (!config?.orderAndRange) { - return operator === 'between' - ? between(left, min, max) - : notBetween(left, min, max) - } - - const executeFn = ( - _encrypted: unknown, - encryptedMin?: unknown, - encryptedMax?: unknown, - ) => { - if (encryptedMin === undefined || encryptedMax === undefined) { + /** + * Gate an operator on the column's indexes. `indexes` is a disjunction — any + * one of them grants the capability — so equality (`unique` OR `ore`) and the + * single-index gates share one rule and one diagnostic shape. + */ + function requireIndex( + ctx: ColumnContext, + indexes: readonly ('unique' | 'ore' | 'ope' | 'match' | 'ste_vec')[], + operator: string, + capability: string, + ): void { + if (!indexes.some((index) => ctx.indexes[index])) { throw new EncryptionOperatorError( - `${operator} operator requires both min and max values`, - { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, - operator, - }, + `Operator "${operator}" requires ${capability} on column "${ctx.columnName}" (domain ${ctx.builder.getEqlType()} does not support it).`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, ) } - - const rangeCondition = sql`eql_v2.gte(${left}, ${bindIfParam(encryptedMin, left)}) AND eql_v2.lte(${left}, ${bindIfParam(encryptedMax, left)})` - - return operator === 'between' - ? rangeCondition - : sql`NOT (${rangeCondition})` } - return createLazyOperator( - operator, - left, - undefined, - executeFn, - true, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - min, - max, - queryTypes.orderAndRange, - ) as Promise -} - -/** - * Creates a text search operator (like, ilike, notIlike) - */ -function createTextSearchOperator( - operator: 'like' | 'ilike' | 'notIlike', - left: SQLWrapper, - right: unknown, - columnInfo: ColumnInfo, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise | SQL { - const { config } = columnInfo - - if (!config?.freeTextSearch) { - // Cast to satisfy TypeScript - const rightValue = right as string | SQLWrapper - switch (operator) { - case 'like': - return like(left as Parameters[0], rightValue) - case 'ilike': - return ilike(left as Parameters[0], rightValue) - case 'notIlike': - return notIlike(left as Parameters[0], rightValue) - } - } - - const executeFn = (encrypted: unknown) => { - if (encrypted === undefined) { + // Ordering flavour is pinned by the column's domain (eql-3.0.0): `_ord` + // domains carry `ope` (`op` CLLW-OPE term), `_ord_ore` domains carry `ore` + // (`ob` block-ORE term). Either satisfies the order/range operators, and an + // order-capable column answers equality via its ordering term too. + const EQUALITY_INDEXES = ['unique', 'ore', 'ope'] as const + const ORDERING_INDEXES = ['ore', 'ope'] as const + // Two DISTINCT operators, split by semantics (#617): + // - `matches` is bloom free-text (`match`, a `text_search`/`text_match` + // column): a one-sided, order- and multiplicity-insensitive token match that + // may false-positive. It emits `eql_v3.matches(col, operand)` (the SQL + // function keeps its bundle name) but is NOT containment. + // - `contains` is encrypted-JSONB containment (an `eql_v3_json_search` column, + // `ste_vec` index): exact jsonb `@>`, no false positives — genuine + // containment, so it keeps the `contains` name. + const MATCH_INDEXES = ['match'] as const + const JSON_CONTAINMENT_INDEXES = ['ste_vec'] as const + + function applyOperationOptions( + op: ChainableOperation, + opts?: EncryptionOperatorCallOpts, + ): AuditableOperation { + const lockContext = opts?.lockContext ?? defaults.lockContext + const audit = opts?.audit ?? defaults.audit + const withLock = lockContext ? op.withLockContext(lockContext) : op + if (audit) withLock.audit(audit) + return withLock + } + + function requireNonNullOperand( + ctx: ColumnContext, + value: unknown, + operator: string, + ): void { + if (value == null) { throw new EncryptionOperatorError( - `Encryption failed for ${operator} operator`, + `Operator "${operator}" cannot encrypt a null operand for column "${ctx.columnName}". Use isNull() or isNotNull() for NULL checks.`, { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, + columnName: ctx.columnName, + tableName: ctx.tableName, operator, }, ) } - - const sqlFn = sql`eql_v2.${sql.raw(operator === 'notIlike' ? 'ilike' : operator)}(${left}, ${bindIfParam(encrypted, left)})` - return operator === 'notIlike' ? sql`NOT (${sqlFn})` : sqlFn } - return createLazyOperator( - operator, - left, - right, - executeFn, - true, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - undefined, // min - undefined, // max - queryTypes.freeTextSearch, - ) as Promise -} - -/** - * Creates a JSONB operator that encrypts a JSON path selector and wraps it - * in the appropriate `eql_v2` function call. - * - * Supports `jsonbPathQueryFirst`, `jsonbGet`, and `jsonbPathExists`. - * The column must have `searchableJson` enabled in its {@link EncryptedColumnConfig}. - */ -function createJsonbOperator( - operator: 'jsonbPathQueryFirst' | 'jsonbGet' | 'jsonbPathExists', - left: SQLWrapper, - right: unknown, - columnInfo: ColumnInfo, - encryptionClient: EncryptionClient, - defaultTable: EncryptedTable | undefined, - tableCache: Map>, -): Promise { - const { config } = columnInfo - const encryptedSelector = (value: unknown) => - sql`${bindIfParam(value, left)}::eql_v2_encrypted` - - if (!config?.searchableJson) { - throw new EncryptionOperatorError( - `The ${operator} operator requires searchableJson to be enabled on the column configuration.`, - { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, - operator, - }, - ) - } - - const executeFn = (encrypted: unknown) => { - if (encrypted === undefined) { + /** + * Reject a free-text needle the column's match index cannot answer. A needle + * shorter than the tokenizer's `token_length` yields an empty bloom filter, + * and `stored_bf @> '{}'` holds for every row — so without this the query + * silently returns the whole table. + */ + function requireAnswerableNeedle( + ctx: ColumnContext, + value: unknown, + operator: string, + ): void { + const match = ctx.indexes.match + if (!match) return + const reason = matchNeedleError(value, match) + if (reason) { throw new EncryptionOperatorError( - `Encryption failed for ${operator} operator`, - { - columnName: columnInfo.columnName, - tableName: columnInfo.tableName, - operator, - }, + `Operator "${operator}" cannot search column "${ctx.columnName}": ${reason}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, ) } - switch (operator) { - case 'jsonbPathQueryFirst': - return sql`eql_v2.jsonb_path_query_first(${left}, ${encryptedSelector(encrypted)})` - case 'jsonbGet': - return sql`${left} -> ${encryptedSelector(encrypted)}` - case 'jsonbPathExists': - return sql`eql_v2.jsonb_path_exists(${left}, ${encryptedSelector(encrypted)})` - } } - return createLazyOperator( - operator, - left, - right, - executeFn, - true, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - undefined, - undefined, - queryTypes.steVecSelector, - ) as Promise -} - -// ============================================================================ -// Public API: createEncryptionOperators -// ============================================================================ - -/** - * Creates a set of encryption-aware operators that automatically encrypt values - * for encrypted columns before using them with Drizzle operators. - * - * For equality and text search operators (eq, ne, like, ilike, inArray, etc.): - * Values are encrypted and then passed to regular Drizzle operators, which use - * PostgreSQL's built-in operators for eql_v2_encrypted types. - * - * For order and range operators (gt, gte, lt, lte, between, notBetween): - * Values are encrypted and then use eql_v2.* functions (eql_v2.gt(), eql_v2.gte(), etc.) - * which are required for ORE (Order-Revealing Encryption) comparisons. - * - * @param encryptionClient - The EncryptionClient instance - * @returns An object with all Drizzle operators wrapped for encrypted columns - * - * @example - * ```ts - * // Initialize operators - * const ops = createEncryptionOperators(encryptionClient) - * - * // Equality search - automatically encrypts and uses PostgreSQL operators - * const results = await db - * .select() - * .from(usersTable) - * .where(await ops.eq(usersTable.email, 'user@example.com')) - * - * // Range query - automatically encrypts and uses eql_v2.gte() - * const olderUsers = await db - * .select() - * .from(usersTable) - * .where(await ops.gte(usersTable.age, 25)) - * ``` - */ -export function createEncryptionOperators(encryptionClient: EncryptionClient): { - // Comparison operators - /** - * Equality operator - encrypts value for encrypted columns. - * Requires either `equality` or `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users with a specific email address. - * ```ts - * const condition = await ops.eq(usersTable.email, 'user@example.com') - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - eq: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Not equal operator - encrypts value for encrypted columns. - * Requires either `equality` or `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users whose email address is not a specific value. - * ```ts - * const condition = await ops.ne(usersTable.email, 'user@example.com') - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - ne: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Greater than operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users older than a specific age. - * ```ts - * const condition = await ops.gt(usersTable.age, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - gt: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Greater than or equal operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users older than or equal to a specific age. - * ```ts - * const condition = await ops.gte(usersTable.age, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - gte: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Less than operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users younger than a specific age. - * ```ts - * const condition = await ops.lt(usersTable.age, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - lt: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Less than or equal operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users younger than or equal to a specific age. - * ```ts - * const condition = await ops.lte(usersTable.age, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - lte: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * Between operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users within a specific age range. - * ```ts - * const condition = await ops.between(usersTable.age, 20, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - between: (left: SQLWrapper, min: unknown, max: unknown) => Promise | SQL - - /** - * Not between operator for encrypted columns with ORE index. - * Requires `orderAndRange` to be set on {@link EncryptedColumnConfig}. - * - * @example - * Select users outside a specific age range. - * ```ts - * const condition = await ops.notBetween(usersTable.age, 20, 30) - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - notBetween: ( - left: SQLWrapper, - min: unknown, - max: unknown, - ) => Promise | SQL - - /** - * Like operator for encrypted columns with free text search. - * Requires `freeTextSearch` to be set on {@link EncryptedColumnConfig}. - * - * > [!IMPORTANT] - * > Case sensitivity on encrypted columns depends on the {@link EncryptedColumnConfig}. - * > Ensure that the column is configured for case-insensitive search if needed. - * - * @example - * Select users with email addresses matching a pattern. - * ```ts - * const condition = await ops.like(usersTable.email, '%@example.com') - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - like: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * ILike operator for encrypted columns with free text search. - * Requires `freeTextSearch` to be set on {@link EncryptedColumnConfig}. - * - * > [!IMPORTANT] - * > Case sensitivity on encrypted columns depends on the {@link EncryptedColumnConfig}. - * > Ensure that the column is configured for case-insensitive search if needed. - * - * @example - * Select users with email addresses matching a pattern (case-insensitive). - * ```ts - * const condition = await ops.ilike(usersTable.email, '%@example.com') - * const results = await db.select().from(usersTable).where(condition) - * ``` - */ - ilike: (left: SQLWrapper, right: unknown) => Promise | SQL - notIlike: (left: SQLWrapper, right: unknown) => Promise | SQL - - /** - * JSONB path query first operator for encrypted columns with searchable JSON. - * Requires `searchableJson` to be set on {@link EncryptedColumnConfig}. - * - * Encrypts the JSON path selector and calls `eql_v2.jsonb_path_query_first()`, - * casting the parameter to `eql_v2_encrypted`. - * - * @throws {EncryptionOperatorError} If the column does not have `searchableJson` enabled. - */ - jsonbPathQueryFirst: (left: SQLWrapper, right: unknown) => Promise - - /** - * JSONB get operator for encrypted columns with searchable JSON. - * Requires `searchableJson` to be set on {@link EncryptedColumnConfig}. - * - * Encrypts the JSON path selector and uses the `->` operator, - * casting the parameter to `eql_v2_encrypted`. - * - * @throws {EncryptionOperatorError} If the column does not have `searchableJson` enabled. - */ - jsonbGet: (left: SQLWrapper, right: unknown) => Promise - - /** - * JSONB path exists operator for encrypted columns with searchable JSON. - * Requires `searchableJson` to be set on {@link EncryptedColumnConfig}. - * - * Encrypts the JSON path selector and calls `eql_v2.jsonb_path_exists()`, - * casting the parameter to `eql_v2_encrypted`. - * - * @throws {EncryptionOperatorError} If the column does not have `searchableJson` enabled. - */ - jsonbPathExists: (left: SQLWrapper, right: unknown) => Promise - // Array operators - inArray: (left: SQLWrapper, right: unknown[] | SQLWrapper) => Promise - notInArray: (left: SQLWrapper, right: unknown[] | SQLWrapper) => Promise - // Sorting operators - asc: (column: SQLWrapper) => SQL - desc: (column: SQLWrapper) => SQL - and: ( - ...conditions: (SQL | SQLWrapper | Promise | undefined)[] - ) => Promise - or: ( - ...conditions: (SQL | SQLWrapper | Promise | undefined)[] - ) => Promise - // Operators that don't need encryption (pass through to Drizzle) - exists: typeof exists - notExists: typeof notExists - isNull: typeof isNull - isNotNull: typeof isNotNull - not: typeof not - // Array operators that work with arrays directly (not encrypted values) - arrayContains: typeof arrayContains - arrayContained: typeof arrayContained - arrayOverlaps: typeof arrayOverlaps -} { - // Create a cache for encrypted tables keyed by table name - const tableCache = new Map>() - const defaultTable: EncryptedTable | undefined = - undefined - - /** - * Equality operator - encrypts value and uses regular Drizzle operator - */ - const encryptedEq = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'eq', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + function operandFailure( + ctx: ColumnContext, + operator: string, + reason: string, + ): EncryptionOperatorError { + return new EncryptionOperatorError( + `Failed to encrypt query operand for "${ctx.columnName}": ${reason}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, ) } /** - * Not equal operator - encrypts value and uses regular Drizzle operator + * Render a query term as a cast operand: `''::eql_v3.query_`. + * The cast is what reaches the bundle's `(domain, query_)` overloads — + * a bare `::jsonb` would hit the storage-domain overload, whose CHECK demands + * the ciphertext `c` a query term deliberately omits. `queryCast` is derived + * from the column's own domain (see `queryCastForDomain`), so `sql.raw` is safe. */ - const encryptedNe = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'ne', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) + function castOperand( + ctx: ColumnContext, + operator: string, + term: EncryptedQueryResult, + ): SQL { + if (ctx.queryCast === null) { + throw operandFailure( + ctx, + operator, + `column domain "${ctx.builder.getEqlType()}" has no query operand type.`, + ) + } + return sql`${JSON.stringify(term)}::${sql.raw(ctx.queryCast)}` } - /** - * Greater than operator - uses eql_v2.gt() for encrypted columns with ORE index - */ - const encryptedGt = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'gt', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) - } + async function encryptOperand( + ctx: ColumnContext, + value: unknown, + operator: string, + queryType: QueryTypeName, + opts?: EncryptionOperatorCallOpts, + ): Promise { + requireNonNullOperand(ctx, value, operator) - /** - * Greater than or equal operator - uses eql_v2.gte() for encrypted columns with ORE index - */ - const encryptedGte = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'gte', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + const result = await applyOperationOptions( + client.encryptQuery( + value as never, + { + table: ctx.table, + column: ctx.builder, + queryType, + } as never, + ), + opts, ) + if (result.failure) { + throw operandFailure(ctx, operator, result.failure.message) + } + return castOperand(ctx, operator, result.data) } /** - * Less than operator - uses eql_v2.lt() for encrypted columns with ORE index + * Encrypt a whole operand list in ONE `encryptQuery` batch crossing (rather + * than one per value). The batch is position-stable, so the returned terms + * align index-for-index with `values`; a response of a different length means + * the contract was violated and is rejected rather than silently truncating + * the predicate (which would widen an `inArray` or narrow a `notInArray`). + * Null operands are rejected up front, so no term is filtered out of the batch. */ - const encryptedLt = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'lt', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) - } + async function encryptOperands( + ctx: ColumnContext, + values: unknown[], + operator: string, + queryType: QueryTypeName, + opts?: EncryptionOperatorCallOpts, + ): Promise { + for (const value of values) requireNonNullOperand(ctx, value, operator) - /** - * Less than or equal operator - uses eql_v2.lte() for encrypted columns with ORE index - */ - const encryptedLte = ( - left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createComparisonOperator( - 'lte', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + const terms = values.map((value) => ({ + value, + column: ctx.builder, + table: ctx.table, + queryType, + })) + const result = await applyOperationOptions( + client.encryptQuery(terms as never), + opts, ) - } + if (result.failure) { + throw operandFailure(ctx, operator, result.failure.message) + } - /** - * Between operator - uses eql_v2.gte() and eql_v2.lte() for encrypted columns with ORE index - */ - const encryptedBetween = ( - left: SQLWrapper, - min: unknown, - max: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createRangeOperator( - 'between', - left, - min, - max, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) + const encrypted = result.data as EncryptedQueryResult[] + if (encrypted.length !== values.length) { + throw operandFailure( + ctx, + operator, + `batch query encryption returned ${encrypted.length} terms for ${values.length} values.`, + ) + } + return encrypted.map((term) => castOperand(ctx, operator, term)) } - /** - * Not between operator - uses eql_v2.gte() and eql_v2.lte() for encrypted columns with ORE index - */ - const encryptedNotBetween = ( - left: SQLWrapper, - min: unknown, - max: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createRangeOperator( - 'notBetween', - left, - min, - max, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) - } + const colSql = (column: SQLWrapper): SQL => sql`${column}` - /** - * Like operator - encrypts value and uses eql_v2.like() for encrypted columns with match index - */ - const encryptedLike = ( + async function equality( + op: EqualityOp, left: SQLWrapper, right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createTextSearchOperator( - 'like', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, op) + requireIndex(ctx, EQUALITY_INDEXES, op, 'equality') + const enc = await encryptOperand(ctx, right, op, 'equality', opts) + return v3Dialect.equality(op, colSql(left), enc) } - /** - * Case-insensitive like operator - encrypts value and uses eql_v2.ilike() for encrypted columns with match index - */ - const encryptedIlike = ( + async function comparison( + op: ComparisonOp, left: SQLWrapper, right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createTextSearchOperator( - 'ilike', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, op) + requireIndex(ctx, ORDERING_INDEXES, op, 'order/range') + const enc = await encryptOperand(ctx, right, op, 'orderAndRange', opts) + return v3Dialect.comparison(op, colSql(left), enc) } - /** - * Not like operator (case insensitive) - encrypts value and uses eql_v2.ilike() for encrypted columns with match index - */ - const encryptedNotIlike = ( + async function range( left: SQLWrapper, - right: unknown, - ): Promise | SQL => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createTextSearchOperator( - 'notIlike', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, - ) - } - - /** - * JSONB path query first operator - encrypts the selector and calls - * `eql_v2.jsonb_path_query_first()` for encrypted columns with searchable JSON. - */ - const encryptedJsonbPathQueryFirst = ( + min: unknown, + max: unknown, + negate: boolean, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, operator) + requireIndex(ctx, ORDERING_INDEXES, operator, 'order/range') + // Independent operands — encrypt concurrently rather than paying two + // sequential round-trips to the crypto backend. + const [encMin, encMax] = await Promise.all([ + encryptOperand(ctx, min, operator, 'orderAndRange', opts), + encryptOperand(ctx, max, operator, 'orderAndRange', opts), + ]) + // `v3Dialect.range` is already parenthesised, so `NOT` binds to the whole + // conjunction without a wrapper here. + const condition = v3Dialect.range(colSql(left), encMin, encMax) + return negate ? sql`NOT ${condition}` : condition + } + + /** + * Fuzzy free-text token match on a `text_search`/`text_match` column. NOT + * containment: it tests whether the needle's downcased 3-gram set is a subset + * of the haystack's, via a bloom filter — order- and multiplicity-insensitive + * and one-sided (a `true` may be a false positive, a `false` never is). Emits + * `eql_v3.matches(col, operand)` (the SQL function's bundle name). + */ + async function matches( left: SQLWrapper, right: unknown, - ): Promise => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createJsonbOperator( - 'jsonbPathQueryFirst', - left, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, operator) + requireIndex(ctx, MATCH_INDEXES, operator, 'free-text search') + // The answerable-needle rule applies (a sub-`token_length` needle blooms to + // nothing and would match every row); the `query_` cast reaches the + // match overload. + requireAnswerableNeedle(ctx, right, operator) + const enc = await encryptOperand( + ctx, right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + operator, + 'freeTextSearch', + opts, ) + return v3Dialect.contains(colSql(left), enc) } /** - * JSONB get operator - encrypts the selector and uses the `->` operator - * for encrypted columns with searchable JSON. + * Exact encrypted-JSONB containment on an `eql_v3_json_search` (`ste_vec`) column: + * genuine jsonb `@>`, no false positives — hence it keeps the `contains` name. + * `eql_v3_json_search` has no `eql_v3.matches` overload; containment is the `@>` + * operator, whose `(eql_v3_json_search, eql_v3.query_json)` form takes a NARROWED + * query term (searchableJson → no ciphertext), cast to `eql_v3.query_json`. */ - const encryptedJsonbGet = ( + async function containsJsonOp( left: SQLWrapper, right: unknown, - ): Promise => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createJsonbOperator( - 'jsonbGet', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, operator) + requireIndex(ctx, JSON_CONTAINMENT_INDEXES, operator, 'JSON containment') + const needle = await encryptJsonContainmentTerm(ctx, right, operator, opts) + return v3Dialect.containsJson(colSql(left), needle) + } + + /** + * Build a `query_json` containment needle for a `json` column — the JSON query + * term carries no ciphertext and satisfies the `eql_v3.query_json` CHECK the + * `@>` overload needs. Cast here (not by `queryCastForDomain`): a json column's + * domain is `eql_v3_json_search` but its query operand type is the irregular + * `eql_v3.query_json`. + */ + async function encryptJsonContainmentTerm( + ctx: ColumnContext, + value: unknown, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + requireNonNullOperand(ctx, value, operator) + // Reject the empty-object needle. `doc @> '{}'` holds for EVERY document + // (jsonb `{} ⊆ anything`), so `contains(col, {})` would silently return the + // whole table — the same whole-table footgun the bloom path guards against + // with `requireAnswerableNeedle`. An accidental empty filter is a bug, not a + // match-all request; callers wanting every row should omit the predicate. + if ( + value !== null && + typeof value === 'object' && + !Array.isArray(value) && + Object.keys(value).length === 0 + ) { + throw new EncryptionOperatorError( + `Operator "${operator}" cannot take an empty object needle on column "${ctx.columnName}": it matches every row. Pass a non-empty sub-object, or omit the predicate to select all rows.`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) + } + const result = await applyOperationOptions( + client.encryptQuery( + value as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'searchableJson', + } as never, + ), + opts, ) - } - - /** - * JSONB path exists operator - encrypts the selector and calls - * `eql_v2.jsonb_path_exists()` for encrypted columns with searchable JSON. - */ - const encryptedJsonbPathExists = ( - left: SQLWrapper, - right: unknown, - ): Promise => { - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - return createJsonbOperator( - 'jsonbPathExists', - left, - right, - columnInfo, - encryptionClient, - defaultTable, - tableCache, + if (result.failure) { + throw operandFailure(ctx, operator, result.failure.message) + } + return sql`${JSON.stringify(result.data)}::eql_v3.query_json` + } + + /** + * JSONPath selector-with-constraint on an `eql_v3_json_search` (`ste_vec`) + * column. Equality uses a value-selector containment needle, allowing the + * functional GIN index to answer the query. Ordering extracts the path entry + * with a selector hash and compares it to a ciphertext-free scalar ordering + * term. No storage ciphertext is placed in the WHERE clause. + */ + async function selectorCompare( + col: SQLWrapper, + path: string, + op: EqualityOp | ComparisonOp, + value: unknown, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(col, operator) + requireIndex( + ctx, + JSON_CONTAINMENT_INDEXES, + operator, + 'JSON selector (searchableJson)', ) - } - - /** - * In array operator - encrypts all values in the array - */ - const encryptedInArray = async ( - left: SQLWrapper, - right: unknown[] | SQLWrapper, - ): Promise => { - // If right is a SQLWrapper (subquery), pass through to Drizzle - if (isSQLWrapper(right)) { - return inArray(left, right as unknown as Parameters[1]) - } + requireNonNullOperand(ctx, value, operator) - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - - if (!columnInfo.config?.equality || !Array.isArray(right)) { - return inArray(left, right as unknown[]) + // A selector compares a scalar leaf — reject non-scalars / non-orderable + // types up front with a clear error, not a deferred DB failure. + const ordering = op !== 'eq' && op !== 'ne' + const leafReason = unsupportedLeafReason(value, ordering) + if (leafReason) { + throw new EncryptionOperatorError( + `Operator "${operator}" cannot compare column "${ctx.columnName}": ${leafReason}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) } - // Encrypt all values in the array in a single batch - const encryptedValues = await encryptValues( - encryptionClient, - right.map((value) => ({ - value, - column: left, - queryType: queryTypes.equality, - })), - defaultTable, - tableCache, - ) - - // Use regular eq for each encrypted value - PostgreSQL operators handle it - const conditions = encryptedValues - .filter((encrypted) => encrypted !== undefined) - .map((encrypted) => eq(left, encrypted)) - - if (conditions.length === 0) { - return sql`false` + // Surface path-validation failures as EncryptionOperatorError with context. + let segments: string[] + try { + segments = parseSelectorSegments(path) + } catch (err) { + throw new EncryptionOperatorError( + `Operator "${operator}" on column "${ctx.columnName}": ${err instanceof Error ? err.message : String(err)}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) } - const combined = or(...conditions) - return combined ?? sql`false` - } + const canonicalPath = jsonPathOf(segments) - /** - * Not in array operator - */ - const encryptedNotInArray = async ( - left: SQLWrapper, - right: unknown[] | SQLWrapper, - ): Promise => { - // If right is a SQLWrapper (subquery), pass through to Drizzle - if (isSQLWrapper(right)) { - return notInArray( - left, - right as unknown as Parameters[1], + if (op === 'eq' || op === 'ne') { + const result = await applyOperationOptions( + client.encryptQuery( + { path: canonicalPath, value } as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'steVecValueSelector', + } as never, + ), + opts, + ) + if (result.failure) { + throw operandFailure(ctx, operator, result.failure.message) + } + const contains = v3Dialect.containsJson( + colSql(col), + sql`${JSON.stringify(result.data)}::eql_v3.query_json`, + ) + return op === 'eq' + ? contains + : sql`(NOT ${contains} OR ${colSql(col)} IS NULL)` + } + + // Selector hashing and scalar-term encryption are independent, so order + // them concurrently. `steVecTerm` accepts only the JSON-orderable scalar + // families (string and number), enforced above. + const [selResult, termResult] = await Promise.all([ + applyOperationOptions( + client.encryptQuery( + canonicalPath as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'steVecSelector', + } as never, + ), + opts, + ), + applyOperationOptions( + client.encryptQuery( + value as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'steVecTerm', + } as never, + ), + opts, + ), + ]) + if (selResult.failure) { + throw operandFailure(ctx, operator, selResult.failure.message) + } + if (termResult.failure) { + throw operandFailure(ctx, operator, termResult.failure.message) + } + + // A v3 selector term is the bare HMAC hash string; guard the shape so a + // wrapped envelope can't silently bind as a JSON blob and match no rows. + const selValue = selResult.data + if (typeof selValue !== 'string') { + throw operandFailure( + ctx, + operator, + `expected a bare selector hash, got ${typeof selValue}.`, ) } - const columnInfo = getColumnInfo(left, defaultTable, tableCache) - - if (!columnInfo.config?.equality || !Array.isArray(right)) { - return notInArray(left, right as unknown[]) - } + const selSql = sql`${selValue}::text` + const leftEntry = v3Dialect.selectorEntry(colSql(col), selSql) + const termCast = + typeof value === 'string' + ? 'eql_v3.query_text_ord' + : 'eql_v3.query_double_ord' + const rightTerm = sql`${JSON.stringify(termResult.data)}::${sql.raw(termCast)}` + return v3Dialect.comparison(op, leftEntry, rightTerm) + } - // Encrypt all values in the array in a single batch - const encryptedValues = await encryptValues( - encryptionClient, - right.map((value) => ({ - value, - column: left, - queryType: queryTypes.equality, - })), - defaultTable, - tableCache, + /** Comparison methods bound to a `col->'path'` selector, mirroring the scalar + * operators. Async: each encrypts its operand. */ + function selectorOps(col: SQLWrapper, path: string) { + const at = + (op: EqualityOp | ComparisonOp) => + (value: unknown, opts?: EncryptionOperatorCallOpts) => + selectorCompare(col, path, op, value, `selector(${path}).${op}`, opts) + return { + /** `col->'path' = value` (encrypted equality at the selector). A row whose + * document lacks `path` is excluded (it is not equal to `value`). */ + eq: at('eq'), + /** `col->'path' <> value`, INCLUDING rows whose document lacks `path` + * ("not equal to value" covers "has no value"). */ + ne: at('ne'), + /** `col->'path' > value` (encrypted ordering at the selector). */ + gt: at('gt'), + /** `col->'path' >= value`. */ + gte: at('gte'), + /** `col->'path' < value`. */ + lt: at('lt'), + /** `col->'path' <= value`. */ + lte: at('lte'), + /** Order rows ascending by the encrypted scalar at `path`. Missing paths + * produce SQL NULL and follow PostgreSQL's normal NULL ordering. */ + asc: (opts?: EncryptionOperatorCallOpts) => + selectorOrder(col, path, 'asc', opts), + /** Order rows descending by the encrypted scalar at `path`. Missing paths + * produce SQL NULL and follow PostgreSQL's normal NULL ordering. */ + desc: (opts?: EncryptionOperatorCallOpts) => + selectorOrder(col, path, 'desc', opts), + } + } + + /** Build `ORDER BY eql_v3.ord_term(col -> selector::text)`. + * The selector is encrypted, but the extracted SteVec entry already carries + * its OPE ordering term, so no plaintext comparison operand is needed. */ + async function selectorOrder( + col: SQLWrapper, + path: string, + direction: 'asc' | 'desc', + opts?: EncryptionOperatorCallOpts, + ): Promise { + const operator = `selector(${path}).${direction}` + const ctx = resolveContext(col, operator) + requireIndex( + ctx, + JSON_CONTAINMENT_INDEXES, + operator, + 'JSON selector (searchableJson)', ) - // Use regular ne for each encrypted value - PostgreSQL operators handle it - const conditions = encryptedValues - .filter((encrypted) => encrypted !== undefined) - .map((encrypted) => ne(left, encrypted)) - - if (conditions.length === 0) { - return sql`true` + let canonicalPath: string + try { + canonicalPath = jsonPathOf(parseSelectorSegments(path)) + } catch (err) { + throw new EncryptionOperatorError( + `Operator "${operator}" on column "${ctx.columnName}": ${err instanceof Error ? err.message : String(err)}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) } - const combined = and(...conditions) - return combined ?? sql`true` - } - - /** - * Ascending order helper - uses eql_v2.order_by() for encrypted columns with ORE index - */ - const encryptedAsc = (column: SQLWrapper): SQL => { - const columnInfo = getColumnInfo(column, defaultTable, tableCache) - - if (columnInfo.config?.orderAndRange) { - return asc(sql`eql_v2.order_by(${column})`) + const result = await applyOperationOptions( + client.encryptQuery( + canonicalPath as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'steVecSelector', + } as never, + ), + opts, + ) + if (result.failure) { + throw operandFailure(ctx, operator, result.failure.message) } - - return asc(column) - } - - /** - * Descending order helper - uses eql_v2.order_by() for encrypted columns with ORE index - */ - const encryptedDesc = (column: SQLWrapper): SQL => { - const columnInfo = getColumnInfo(column, defaultTable, tableCache) - - if (columnInfo.config?.orderAndRange) { - return desc(sql`eql_v2.order_by(${column})`) + if (typeof result.data !== 'string') { + throw operandFailure( + ctx, + operator, + `expected a bare selector hash, got ${typeof result.data}.`, + ) } - return desc(column) + const entry = v3Dialect.selectorEntry( + colSql(col), + sql`${result.data}::text`, + ) + const term = v3Dialect.orderBy(entry, 'ope') + return direction === 'asc' ? asc(term) : desc(term) } - /** - * Batched AND operator - collects lazy operators, batches encryption, and combines conditions - */ - const encryptedAnd = async ( - ...conditions: (SQL | SQLWrapper | Promise | undefined)[] - ): Promise => { - // Single pass: separate lazy operators from regular conditions - const lazyOperators: LazyOperator[] = [] - const regularConditions: (SQL | SQLWrapper | undefined)[] = [] - const regularPromises: Promise[] = [] - - for (const condition of conditions) { - if (condition === undefined) { - continue - } - - if (isLazyOperator(condition)) { - lazyOperators.push(condition) - } else if (condition instanceof Promise) { - // Check if promise is also a lazy operator - if (isLazyOperator(condition)) { - lazyOperators.push(condition) - } else { - regularPromises.push(condition) - } - } else { - regularConditions.push(condition) - } - } - - // If there are no lazy operators, just use Drizzle's and() - if (lazyOperators.length === 0) { - const allConditions: (SQL | SQLWrapper | undefined)[] = [ - ...regularConditions, - ...(await Promise.all(regularPromises)), - ] - return and(...allConditions) ?? sql`true` - } - - // Single pass: collect all values to encrypt with metadata - const valuesToEncrypt: Array<{ - value: unknown - column: SQLWrapper - columnInfo: ColumnInfo - queryType?: QueryTypeName - lazyOpIndex: number - isMin?: boolean - isMax?: boolean - }> = [] - - for (let i = 0; i < lazyOperators.length; i++) { - const lazyOp = lazyOperators[i] - if (!lazyOp.needsEncryption) { - continue - } - - if (lazyOp.min !== undefined && lazyOp.max !== undefined) { - valuesToEncrypt.push({ - value: lazyOp.min, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - isMin: true, - }) - valuesToEncrypt.push({ - value: lazyOp.max, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - isMax: true, - }) - } else if (lazyOp.right !== undefined) { - valuesToEncrypt.push({ - value: lazyOp.right, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - }) - } + async function inArrayOp( + left: SQLWrapper, + values: unknown[], + negate: boolean, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(left, operator) + if (values.length === 0) { + throw new EncryptionOperatorError( + `Operator "${operator}" requires a non-empty list of values for column "${ctx.columnName}".`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) } - - // Batch encrypt all values - const encryptedResults = await encryptValues( - encryptionClient, - valuesToEncrypt.map((v) => ({ - value: v.value, - column: v.column, - queryType: v.queryType, - })), - defaultTable, - tableCache, + // Gate and resolve the context once for the whole list, then encrypt it in + // a single `encryptQuery` batch crossing. + requireIndex(ctx, EQUALITY_INDEXES, operator, 'equality') + const op: EqualityOp = negate ? 'ne' : 'eq' + const encrypted = await encryptOperands( + ctx, + values, + operator, + 'equality', + opts, ) - - // Group encrypted values by lazy operator index - const encryptedByLazyOp = new Map< - number, - { value?: unknown; min?: unknown; max?: unknown } - >() - - for (let i = 0; i < valuesToEncrypt.length; i++) { - const { lazyOpIndex, isMin, isMax } = valuesToEncrypt[i] - const encrypted = encryptedResults[i] - - let group = encryptedByLazyOp.get(lazyOpIndex) - if (!group) { - group = {} - encryptedByLazyOp.set(lazyOpIndex, group) - } - - if (isMin) { - group.min = encrypted - } else if (isMax) { - group.max = encrypted - } else { - group.value = encrypted - } - } - - // Execute all lazy operators with their encrypted values - const sqlConditions: SQL[] = [] - for (let i = 0; i < lazyOperators.length; i++) { - const lazyOp = lazyOperators[i] - const encrypted = encryptedByLazyOp.get(i) - - let sqlCondition: SQL - if (lazyOp.needsEncryption && encrypted) { - const encryptedValues: Array<{ value: unknown; encrypted: unknown }> = - [] - if (encrypted.value !== undefined) { - encryptedValues.push({ - value: lazyOp.right, - encrypted: encrypted.value, - }) - } - if (encrypted.min !== undefined) { - encryptedValues.push({ value: lazyOp.min, encrypted: encrypted.min }) - } - if (encrypted.max !== undefined) { - encryptedValues.push({ value: lazyOp.max, encrypted: encrypted.max }) - } - sqlCondition = await executeLazyOperator(lazyOp, encryptedValues) - } else { - sqlCondition = lazyOp.execute(lazyOp.right) - } - - sqlConditions.push(sqlCondition) - } - - // Await any regular promises - const regularPromisesResults = await Promise.all(regularPromises) - - // Combine all conditions - const allConditions: (SQL | SQLWrapper | undefined)[] = [ - ...regularConditions, - ...sqlConditions, - ...regularPromisesResults, - ] - - return and(...allConditions) ?? sql`true` + const conditions = encrypted.map((enc) => + v3Dialect.equality(op, colSql(left), enc), + ) + // The empty-list guard above leaves `conditions` non-empty, so `and`/`or` + // never return undefined here. + return (negate ? and(...conditions) : or(...conditions)) as SQL } - /** - * Batched OR operator - collects lazy operators, batches encryption, and combines conditions - */ - const encryptedOr = async ( - ...conditions: (SQL | SQLWrapper | Promise | undefined)[] - ): Promise => { - const lazyOperators: LazyOperator[] = [] - const regularConditions: (SQL | SQLWrapper | undefined)[] = [] - const regularPromises: Promise[] = [] - - for (const condition of conditions) { - if (condition === undefined) { - continue - } - - if (isLazyOperator(condition)) { - lazyOperators.push(condition) - } else if (condition instanceof Promise) { - if (isLazyOperator(condition)) { - lazyOperators.push(condition) - } else { - regularPromises.push(condition) - } - } else { - regularConditions.push(condition) - } - } - - if (lazyOperators.length === 0) { - const allConditions: (SQL | SQLWrapper | undefined)[] = [ - ...regularConditions, - ...(await Promise.all(regularPromises)), - ] - return or(...allConditions) ?? sql`false` - } - - const valuesToEncrypt: Array<{ - value: unknown - column: SQLWrapper - columnInfo: ColumnInfo - queryType?: QueryTypeName - lazyOpIndex: number - isMin?: boolean - isMax?: boolean - }> = [] - - for (let i = 0; i < lazyOperators.length; i++) { - const lazyOp = lazyOperators[i] - if (!lazyOp.needsEncryption) { - continue - } - - if (lazyOp.min !== undefined && lazyOp.max !== undefined) { - valuesToEncrypt.push({ - value: lazyOp.min, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - isMin: true, - }) - valuesToEncrypt.push({ - value: lazyOp.max, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - isMax: true, - }) - } else if (lazyOp.right !== undefined) { - valuesToEncrypt.push({ - value: lazyOp.right, - column: lazyOp.left, - columnInfo: lazyOp.columnInfo, - queryType: lazyOp.queryType, - lazyOpIndex: i, - }) - } - } + function orderTerm(column: SQLWrapper, operator: string): SQL { + const ctx = resolveContext(column, operator) + requireIndex(ctx, ORDERING_INDEXES, operator, 'order/range') + return v3Dialect.orderBy(colSql(column), ctx.indexes.ore ? 'ore' : 'ope') + } - const encryptedResults = await encryptValues( - encryptionClient, - valuesToEncrypt.map((v) => ({ - value: v.value, - column: v.column, - queryType: v.queryType, - })), - defaultTable, - tableCache, + async function combine( + joiner: typeof and, + empty: SQL, + conditions: (SQL | SQLWrapper | Promise | undefined)[], + ): Promise { + const present = conditions.filter( + (c): c is SQL | SQLWrapper | Promise => c !== undefined, ) - - const encryptedByLazyOp = new Map< - number, - { value?: unknown; min?: unknown; max?: unknown } - >() - - for (let i = 0; i < valuesToEncrypt.length; i++) { - const { lazyOpIndex, isMin, isMax } = valuesToEncrypt[i] - const encrypted = encryptedResults[i] - - let group = encryptedByLazyOp.get(lazyOpIndex) - if (!group) { - group = {} - encryptedByLazyOp.set(lazyOpIndex, group) - } - - if (isMin) { - group.min = encrypted - } else if (isMax) { - group.max = encrypted - } else { - group.value = encrypted - } - } - - const sqlConditions: SQL[] = [] - for (let i = 0; i < lazyOperators.length; i++) { - const lazyOp = lazyOperators[i] - const encrypted = encryptedByLazyOp.get(i) - - let sqlCondition: SQL - if (lazyOp.needsEncryption && encrypted) { - const encryptedValues: Array<{ value: unknown; encrypted: unknown }> = - [] - if (encrypted.value !== undefined) { - encryptedValues.push({ - value: lazyOp.right, - encrypted: encrypted.value, - }) - } - if (encrypted.min !== undefined) { - encryptedValues.push({ value: lazyOp.min, encrypted: encrypted.min }) - } - if (encrypted.max !== undefined) { - encryptedValues.push({ value: lazyOp.max, encrypted: encrypted.max }) - } - sqlCondition = await executeLazyOperator(lazyOp, encryptedValues) - } else { - sqlCondition = lazyOp.execute(lazyOp.right) - } - - sqlConditions.push(sqlCondition) - } - - const regularPromisesResults = await Promise.all(regularPromises) - - const allConditions: (SQL | SQLWrapper | undefined)[] = [ - ...regularConditions, - ...sqlConditions, - ...regularPromisesResults, - ] - - return or(...allConditions) ?? sql`false` + const resolved = await Promise.all(present) + return joiner(...resolved) ?? empty } return { - // Comparison operators - eq: encryptedEq, - ne: encryptedNe, - gt: encryptedGt, - gte: encryptedGte, - lt: encryptedLt, - lte: encryptedLte, - - // Range operators - between: encryptedBetween, - notBetween: encryptedNotBetween, - - // Text search operators - like: encryptedLike, - ilike: encryptedIlike, - notIlike: encryptedNotIlike, - - // Searchable JSON operators - jsonbPathQueryFirst: encryptedJsonbPathQueryFirst, - jsonbGet: encryptedJsonbGet, - jsonbPathExists: encryptedJsonbPathExists, - - // Array operators - inArray: encryptedInArray, - notInArray: encryptedNotInArray, - - // Sorting operators - asc: encryptedAsc, - desc: encryptedDesc, - - // AND operator - batches encryption operations - and: encryptedAnd, - - // OR operator - batches encryption operations - or: encryptedOr, - - // Operators that don't need encryption (pass through to Drizzle) - exists, - notExists, + /** Equality: `column = value`. Encrypts `r` and emits `eql_v3.eq`. + * Requires a `unique` or `ore` index on the column. */ + eq: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + equality('eq', l, r, opts), + /** Inequality: `column <> value`. Encrypts `r` and emits `eql_v3.neq`. + * Requires a `unique` or `ore` index on the column. */ + ne: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + equality('ne', l, r, opts), + /** Greater-than: `column > value`. Encrypts `r` and emits `eql_v3.gt`. + * Requires an `ore` (order/range) index. */ + gt: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + comparison('gt', l, r, opts), + /** Greater-than-or-equal: `column >= value`. Encrypts `r` and emits + * `eql_v3.gte`. Requires an `ore` (order/range) index. */ + gte: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + comparison('gte', l, r, opts), + /** Less-than: `column < value`. Encrypts `r` and emits `eql_v3.lt`. + * Requires an `ore` (order/range) index. */ + lt: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + comparison('lt', l, r, opts), + /** Less-than-or-equal: `column <= value`. Encrypts `r` and emits + * `eql_v3.lte`. Requires an `ore` (order/range) index. */ + lte: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + comparison('lte', l, r, opts), + /** Inclusive range `min <= column <= max`. Encrypts both bounds + * concurrently. Requires an `ore` (order/range) index. */ + between: ( + l: SQLWrapper, + min: unknown, + max: unknown, + opts?: EncryptionOperatorCallOpts, + ) => range(l, min, max, false, 'between', opts), + /** Negated inclusive range `NOT (min <= column <= max)`. Encrypts both + * bounds concurrently. Requires an `ore` (order/range) index. */ + notBetween: ( + l: SQLWrapper, + min: unknown, + max: unknown, + opts?: EncryptionOperatorCallOpts, + ) => range(l, min, max, true, 'notBetween', opts), + /** Fuzzy free-text token match — the needle's 3-gram set is (bloom-)tested + * as a subset of the column's. NOT containment: order/multiplicity- + * insensitive and one-sided (a `true` may be a false positive). Encrypts + * `r`. Requires a `match` (free-text search) index. */ + matches: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + matches(l, r, 'matches', opts), + /** Exact encrypted-JSONB containment (`@>`): matches rows whose document + * contains the given sub-object. No false positives. Encrypts `r`. Requires + * a `ste_vec` index (a `types.Json` column). */ + contains: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => + containsJsonOp(l, r, 'contains', opts), + /** JSONPath selector-with-constraint on a `types.Json` (`ste_vec`) column. + * Returns comparison and ordering methods bound to `col->'path'` — e.g. + * `await ops.selector(users.doc, '$.age').gt(21)` emits + * `col->'' > `, while `.asc()` emits + * `ORDER BY eql_v3.ord_term(col->'')`. Its unique power over `contains` + * is ordering at a path (`gt`/`gte`/`lt`/`lte` and `asc`/`desc`); `eq`/`ne` + * are also provided. Dot-notation object paths only in v1. */ + selector: (l: SQLWrapper, path: string) => selectorOps(l, path), + /** Membership: ORs one encrypted `eq` term per value. The whole list is + * encrypted in one `encryptQuery` batch crossing. Rejects an empty list; + * requires a `unique` or `ore` index. */ + inArray: ( + l: SQLWrapper, + values: unknown[], + opts?: EncryptionOperatorCallOpts, + ) => inArrayOp(l, values, false, 'inArray', opts), + /** Non-membership: ANDs one encrypted `ne` term per value. The whole list + * is encrypted in one `encryptQuery` batch crossing. Rejects an empty list; + * requires a `unique` or `ore` index. */ + notInArray: ( + l: SQLWrapper, + values: unknown[], + opts?: EncryptionOperatorCallOpts, + ) => inArrayOp(l, values, true, 'notInArray', opts), + /** Ascending order by the encrypted order term (`eql_v3.ord_term` / + * `eql_v3.ord_term_ore`, by the column's ordering flavour). + * Synchronous (no operand to encrypt). Requires an ordering index. */ + asc: (c: SQLWrapper) => asc(orderTerm(c, 'asc')), + /** Descending order by the encrypted order term (`eql_v3.ord_term` / + * `eql_v3.ord_term_ore`, by the column's ordering flavour). + * Synchronous (no operand to encrypt). Requires an ordering index. */ + desc: (c: SQLWrapper) => desc(orderTerm(c, 'desc')), + /** Conjunction of the given conditions, awaiting any async operands and + * dropping `undefined`. Empty input resolves to `true`. */ + and: (...conds: (SQL | SQLWrapper | Promise | undefined)[]) => + combine(and, sql`true`, conds), + /** Disjunction of the given conditions, awaiting any async operands and + * dropping `undefined`. Empty input resolves to `false`. */ + or: (...conds: (SQL | SQLWrapper | Promise | undefined)[]) => + combine(or, sql`false`, conds), + /** Drizzle's `isNull`, re-exported unchanged — `column IS NULL` needs no + * encryption and works on any (nullable) encrypted column. */ isNull, + /** Drizzle's `isNotNull`, re-exported unchanged — `column IS NOT NULL` + * needs no encryption. */ isNotNull, + /** Drizzle's `not`, re-exported unchanged — negates an already-built + * (encrypted) predicate. Safe over any operator here, including `between`, + * whose fragment is self-parenthesising. */ not, - // Array operators that work with arrays directly (not encrypted values) - arrayContains, - arrayContained, - arrayOverlaps, + /** Drizzle's `exists`, re-exported unchanged — for correlated subqueries. */ + exists, + /** Drizzle's `notExists`, re-exported unchanged — for correlated + * subqueries. */ + notExists, } } diff --git a/packages/stack-drizzle/src/schema-extraction.ts b/packages/stack-drizzle/src/schema-extraction.ts index 90bb3ec1e..7e0b4226f 100644 --- a/packages/stack-drizzle/src/schema-extraction.ts +++ b/packages/stack-drizzle/src/schema-extraction.ts @@ -1,130 +1,50 @@ import { - type EncryptedColumn, - type EncryptedTable, - encryptedColumn, + type AnyEncryptedV3Column, + type AnyV3Table, encryptedTable, -} from '@cipherstash/stack/schema' -import type { PgCustomColumn, PgTable } from 'drizzle-orm/pg-core' -import { getEncryptedColumnConfig } from './index.js' +} from '@cipherstash/stack/eql/v3' +import type { PgTable } from 'drizzle-orm/pg-core' +import { getEqlV3Column } from './column.js' + +/** Drizzle stashes the SQL table name on this well-known symbol key. */ +const DRIZZLE_NAME = Symbol.for('drizzle:Name') /** - * Extracts the encrypted column keys from a Drizzle table type. - * Columns created with `encryptedType` are `PgCustomColumn` instances; - * this picks only those keys and maps them to `EncryptedColumn`. + * Read the SQL table name Drizzle stashes on a `pgTable`. Returns `undefined` + * for a non-object or a table not built with `pgTable()`. Shared by + * {@link extractEncryptionSchema} and the operator factory so the + * symbol-key introspection lives in exactly one place. */ -// biome-ignore lint/suspicious/noExplicitAny: PgCustomColumn requires a wide generic -type DrizzleEncryptedSchema = { - [K in keyof T as T[K] extends PgCustomColumn - ? K - : never]: EncryptedColumn +export function getDrizzleTableName(table: unknown): string | undefined { + if (!table || typeof table !== 'object') return undefined + const name = (table as Record)[DRIZZLE_NAME] + return typeof name === 'string' ? name : undefined } -/** - * Extracts an encryption schema from a Drizzle table definition. - * This function identifies columns created with `encryptedType` and - * builds a corresponding `EncryptedTable` with `encryptedColumn` definitions. - * - * @param table - The Drizzle table definition - * @returns A EncryptedTable that can be used with encryption client initialization - * - * @example - * ```ts - * const drizzleUsersTable = pgTable('users', { - * email: encryptedType('email', { freeTextSearch: true, equality: true }), - * age: encryptedType('age', { dataType: 'number', orderAndRange: true }), - * }) - * - * const encryptionSchema = extractEncryptionSchema(drizzleUsersTable) - * const client = await createEncryptionClient({ schemas: [encryptionSchema.build()] }) - * ``` - */ -// We use any for the PgTable generic because we need to access Drizzle's internal properties -// biome-ignore lint/suspicious/noExplicitAny: Drizzle table types don't expose Symbol properties -export function extractEncryptionSchema>( - table: T, -): EncryptedTable> & DrizzleEncryptedSchema { - // Drizzle tables store the name in a Symbol property - // biome-ignore lint/suspicious/noExplicitAny: Drizzle tables don't expose Symbol properties in types - const tableName = (table as any)[Symbol.for('drizzle:Name')] as - | string - | undefined +export function extractEncryptionSchema(table: PgTable): AnyV3Table { + const tableName = getDrizzleTableName(table) if (!tableName) { throw new Error( - 'Unable to extract table name from Drizzle table. Ensure you are using a table created with pgTable().', + 'Unable to read table name from Drizzle table. Use a table created with pgTable().', ) } - const columns: Record = {} - - // Iterate through table columns - for (const [columnName, column] of Object.entries(table)) { - // Skip if it's not a column (could be methods or other properties) - if (typeof column !== 'object' || column === null) { - continue - } - - // Check if this column has encrypted configuration - const config = getEncryptedColumnConfig(columnName, column) - - if (config) { - // Extract the actual column name from the column object (not the schema key) - // Drizzle columns have a 'name' property that contains the actual database column name - const actualColumnName = column.name || config.name - - // This is an encrypted column - build encryptedColumn using the actual column name - const csCol = encryptedColumn(actualColumnName) - - // Apply data type - if (config.dataType && config.dataType !== 'string') { - csCol.dataType(config.dataType) - } - - // Apply indexes based on configuration - if (config.orderAndRange) { - csCol.orderAndRange() - } - - if (config.equality) { - if (Array.isArray(config.equality)) { - // Custom token filters - csCol.equality(config.equality) - } else { - // Default equality (boolean true) - csCol.equality() - } - } - - if (config.freeTextSearch) { - if (typeof config.freeTextSearch === 'object') { - // Custom match options - csCol.freeTextSearch(config.freeTextSearch) - } else { - // Default freeTextSearch (boolean true) - csCol.freeTextSearch() - } - } - - if (config.searchableJson) { - if (config.dataType !== 'json') { - throw new Error( - `Column "${columnName}" has searchableJson enabled but dataType is "${config.dataType ?? 'string'}". searchableJson requires dataType: 'json'.`, - ) - } - csCol.searchableJson() - } - - columns[actualColumnName] = csCol - } + const columns: Record = {} + for (const [property, column] of Object.entries(table)) { + if (typeof column !== 'object' || column === null) continue + const columnName = + 'name' in column && typeof column.name === 'string' + ? column.name + : property + const builder = getEqlV3Column(columnName, column) + if (builder) columns[property] = builder } if (Object.keys(columns).length === 0) { throw new Error( - `No encrypted columns found in table "${tableName}". Use encryptedType() to define encrypted columns.`, + `No encrypted v3 columns found in table "${tableName}". Declare columns with the v3 drizzle \`types\` namespace.`, ) } - return encryptedTable(tableName, columns) as EncryptedTable< - DrizzleEncryptedSchema - > & - DrizzleEncryptedSchema + return encryptedTable(tableName, columns) } diff --git a/packages/stack-drizzle/src/v3/sql-dialect.ts b/packages/stack-drizzle/src/sql-dialect.ts similarity index 100% rename from packages/stack-drizzle/src/v3/sql-dialect.ts rename to packages/stack-drizzle/src/sql-dialect.ts diff --git a/packages/stack-drizzle/src/v3/types.ts b/packages/stack-drizzle/src/types.ts similarity index 100% rename from packages/stack-drizzle/src/v3/types.ts rename to packages/stack-drizzle/src/types.ts diff --git a/packages/stack-drizzle/src/v3/index.ts b/packages/stack-drizzle/src/v3/index.ts deleted file mode 100644 index 5e6c042c7..000000000 --- a/packages/stack-drizzle/src/v3/index.ts +++ /dev/null @@ -1,9 +0,0 @@ -export { EqlV3CodecError, v3FromDriver, v3ToDriver } from './codec.js' -export { getEqlV3Column, isEqlV3Column, makeEqlV3Column } from './column.js' -export { encryptedIndexes } from './indexes.js' -export { - createEncryptionOperatorsV3, - EncryptionOperatorError, -} from './operators.js' -export { extractEncryptionSchemaV3 } from './schema-extraction.js' -export { types } from './types.js' diff --git a/packages/stack-drizzle/src/v3/operators.ts b/packages/stack-drizzle/src/v3/operators.ts deleted file mode 100644 index f2e113855..000000000 --- a/packages/stack-drizzle/src/v3/operators.ts +++ /dev/null @@ -1,959 +0,0 @@ -import type { Result } from '@byteslice/result' -import type { AuditConfig } from '@cipherstash/stack/adapter-kit' -import { - jsonPathOf, - matchNeedleError, - parseSelectorSegments, - reconstructSelectorDocument, - stripDomainSchema, - unsupportedLeafReason, -} from '@cipherstash/stack/adapter-kit' -import type { - AnyEncryptedV3Column, - AnyV3Table, -} from '@cipherstash/stack/eql/v3' -import type { EncryptionError } from '@cipherstash/stack/errors' -import type { LockContext } from '@cipherstash/stack/identity' -import type { ColumnSchema } from '@cipherstash/stack/schema' -import type { - EncryptedQueryResult, - QueryTypeName, -} from '@cipherstash/stack/types' -import { - and, - asc, - Column, - desc, - exists, - is, - isNotNull, - isNull, - not, - notExists, - or, - type SQL, - type SQLWrapper, - sql, -} from 'drizzle-orm' -import type { PgTable } from 'drizzle-orm/pg-core' -import { getEqlV3Column } from './column.js' -import { - extractEncryptionSchemaV3, - getDrizzleTableName, -} from './schema-extraction.js' -import { type ComparisonOp, type EqualityOp, v3Dialect } from './sql-dialect.js' - -/** - * The client capability this factory consumes: `encryptQuery`, in both its - * single (`value, opts`) and batch (`terms[]`) forms. Declared structurally — - * with maximally-permissive operands — so it is satisfied by the nominal - * `EncryptionClient`, by the `TypedEncryptionClient` that `EncryptionV3` returns - * (whatever its schema tuple), AND by a hand-rolled test double, none needing a - * cast. Typing the parameter to the nominal `TypedEncryptionClient` would - * reject a client built for a narrower schema tuple (it accepts fewer tables than - * `readonly AnyV3Table[]`); the structural surface sidesteps that variance. - * - * Every operand is a QUERY TERM, not a storage envelope: `encryptQuery` mints a - * ciphertext-free term (no `c`) carrying all of the column's configured index - * terms, which the operator layer casts to the column's `eql_v3.query_` - * type. This reaches the bundle's `(domain, query_)` overloads and keeps - * WHERE-clause payloads free of the ciphertext a query never needs (#622). - * - * `never` operands: the real client's `encryptQuery` is generic (`queryType` is - * constrained to the column's own query types), which a concrete signature here - * cannot match. `never` params keep the structural surface satisfiable by that - * generic method AND by a test double; the call sites cast their real operands. - */ -type OperandEncryptionClient = { - encryptQuery( - value: never, - opts: never, - ): ChainableOperation - encryptQuery(terms: never): ChainableOperation -} - -// Path helpers now live in @cipherstash/stack/adapter-kit (shared with the -// Supabase adapter, #650); re-exported so existing imports keep working. -export { parseSelectorSegments, reconstructSelectorDocument } - -/** - * A dedicated error for v3 operator gating and operand-encryption failures, - * carrying the offending column/table/operator for diagnostics. - * - * INTENTIONAL FORK: this mirrors the v2 adapter's `EncryptionOperatorError` - * rather than sharing it. Unifying the two would couple `./drizzle` and - * `./eql/v3/drizzle` — two independently-versioned public entry points — so the - * duplication is deliberate, not an oversight. - */ -export class EncryptionOperatorError extends Error { - constructor( - message: string, - public readonly context?: { - columnName?: string - tableName?: string - operator?: string - }, - ) { - super(message) - this.name = 'EncryptionOperatorError' - } -} - -interface ColumnContext { - builder: AnyEncryptedV3Column - table: AnyV3Table - indexes: ColumnSchema['indexes'] - columnName: string - tableName: string - /** The `eql_v3.query_` type an operand for this column casts to, so - * `encryptQuery`'s ciphertext-free term reaches the narrowed-query overloads. - * `null` for storage-only columns (no query domain); those never encrypt an - * operand — every operator gates on a query capability first. JSON columns - * override this at the call site (`query_json`, an irregular name). */ - queryCast: string | null -} - -/** - * The `eql_v3.query_` cast for a column's storage domain — e.g. - * `public.eql_v3_text_search` → `eql_v3.query_text_search`. Uniform across the - * queryable column domains (`_eq`, `_ord`, `_ord_ore`, `_match`, `_search`); the - * two irregular cases are handled elsewhere: storage-only domains - * (`eql_v3_boolean`, the bare base types) have no query domain and return `null` - * (they are never queried), and `eql_v3_json_search` maps to `query_json`, cast - * explicitly on the JSON path. - */ -function queryCastForDomain(eqlType: string): string | null { - const bare = stripDomainSchema(eqlType) // public.eql_v3_text_search → eql_v3_text_search - const prefix = 'eql_v3_' - if (!bare.startsWith(prefix)) return null - const suffix = bare.slice(prefix.length) - // No index suffix (bare storage-only domain like `boolean`, `text`) → no query - // domain exists. These are gated out before any operand is encrypted. The - // suffixes match the column factories in `@/eql/v3/columns` exactly: ope - // ordering is the `_ord` domain (not `_ord_ope`) and text search is `_search` - // (there is no `_search_ore` column), so those two never occur here. - if (!/_(eq|ord|ord_ore|match|search)$/.test(suffix)) { - return null - } - return `eql_v3.query_${suffix}` -} - -export type EncryptionOperatorCallOpts = { - lockContext?: LockContext - audit?: AuditConfig -} - -/** - * An SDK encryption operation after its lock context has been applied: still - * auditable and awaitable, but not re-lockable. `withLockContext` returns this, - * not the full {@link ChainableOperation}, mirroring the real - * `EncryptOperationWithLockContext`, which drops `withLockContext` (you cannot - * lock-context twice). Modelling that is what lets the real client type satisfy - * the structural surface with no cast. - */ -type AuditableOperation = { - audit(config: AuditConfig): AuditableOperation - then: PromiseLike>['then'] -} - -/** - * The subset of an SDK encryption operation this factory drives: the fluent - * `withLockContext`/`audit` chain, and a `then` that resolves the operation's - * `Result`. Generic over the resolved payload `T` so the single `encryptQuery` - * carries an `EncryptedQueryResult` term and the batch form an - * `EncryptedQueryResult[]`, rather than the `unknown` this erased to before. - * - * Structural, not the concrete `EncryptQueryOperation` class, because the client - * is passed in and the factory must accept any implementation with this surface. - */ -type ChainableOperation = { - withLockContext(lockContext: LockContext): AuditableOperation - audit(config: AuditConfig): AuditableOperation - then: PromiseLike>['then'] -} - -/** - * Build v3-aware query operators (`eq`, `gte`, `matches`, `contains`, `asc`, …) bound to an - * encryption `client`. Each comparison/containment operator AUTO-ENCRYPTS its - * plaintext operand into an EQL v3 query term before handing it to Drizzle, so - * callers pass plaintext and the emitted SQL compares encrypted values. Every - * operator also gates on the target column's capabilities and throws - * {@link EncryptionOperatorError} when the column can't answer the operator - * (e.g. ordering a non-`ore` column). - * - * @param client - anything that can `encryptQuery` — the nominal - * `EncryptionClient` or the `TypedEncryptionClient` from `EncryptionV3` (no - * cast needed). - * @param defaults - lock context / audit applied to every operand encryption - * unless a per-call override is supplied. - * - * @example - * ```typescript - * const ops = createEncryptionOperatorsV3(await EncryptionV3({ schemas: [users] })) - * await db.select().from(users).where(await ops.eq(users.email, 'a@b.com')) - * ``` - */ -export function createEncryptionOperatorsV3( - client: OperandEncryptionClient, - defaults: EncryptionOperatorCallOpts = {}, -) { - const tableCache = new WeakMap() - // Per-column context memo. `resolveContext` is value-independent, so caching - // by column identity makes `inArray`/`notInArray` build the context (and its - // deep-cloned match block) once for the whole list instead of once per value. - const contextCache = new WeakMap() - - function drizzleTableOf(column: SQLWrapper): PgTable | undefined { - return is(column, Column) - ? (column.table as PgTable | undefined) - : undefined - } - - function resolveContext(column: SQLWrapper, operator: string): ColumnContext { - const cached = contextCache.get(column) - if (cached) return cached - - const columnName = is(column, Column) ? column.name : 'unknown' - const builder = getEqlV3Column(columnName, column) - if (!builder) { - throw new EncryptionOperatorError( - `Operator "${operator}" requires an encrypted v3 column, but "${columnName}" is not one.`, - { columnName, operator }, - ) - } - - const drizzleTable = drizzleTableOf(column) - const tableName = getDrizzleTableName(drizzleTable) ?? 'unknown' - - let table = drizzleTable ? tableCache.get(drizzleTable) : undefined - if (!table && drizzleTable) { - table = extractEncryptionSchemaV3(drizzleTable) - tableCache.set(drizzleTable, table) - } - if (!table) { - throw new EncryptionOperatorError( - `Unable to resolve the encrypted table for column "${columnName}".`, - { columnName, operator }, - ) - } - - const context: ColumnContext = { - builder, - table, - indexes: builder.build().indexes, - columnName, - tableName, - queryCast: queryCastForDomain(builder.getEqlType()), - } - contextCache.set(column, context) - return context - } - - /** - * Gate an operator on the column's indexes. `indexes` is a disjunction — any - * one of them grants the capability — so equality (`unique` OR `ore`) and the - * single-index gates share one rule and one diagnostic shape. - */ - function requireIndex( - ctx: ColumnContext, - indexes: readonly ('unique' | 'ore' | 'ope' | 'match' | 'ste_vec')[], - operator: string, - capability: string, - ): void { - if (!indexes.some((index) => ctx.indexes[index])) { - throw new EncryptionOperatorError( - `Operator "${operator}" requires ${capability} on column "${ctx.columnName}" (domain ${ctx.builder.getEqlType()} does not support it).`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - } - - // Ordering flavour is pinned by the column's domain (eql-3.0.0): `_ord` - // domains carry `ope` (`op` CLLW-OPE term), `_ord_ore` domains carry `ore` - // (`ob` block-ORE term). Either satisfies the order/range operators, and an - // order-capable column answers equality via its ordering term too. - const EQUALITY_INDEXES = ['unique', 'ore', 'ope'] as const - const ORDERING_INDEXES = ['ore', 'ope'] as const - // Two DISTINCT operators, split by semantics (#617): - // - `matches` is bloom free-text (`match`, a `text_search`/`text_match` - // column): a one-sided, order- and multiplicity-insensitive token match that - // may false-positive. It emits `eql_v3.matches(col, operand)` (the SQL - // function keeps its bundle name) but is NOT containment. - // - `contains` is encrypted-JSONB containment (an `eql_v3_json_search` column, - // `ste_vec` index): exact jsonb `@>`, no false positives — genuine - // containment, so it keeps the `contains` name. - const MATCH_INDEXES = ['match'] as const - const JSON_CONTAINMENT_INDEXES = ['ste_vec'] as const - - function applyOperationOptions( - op: ChainableOperation, - opts?: EncryptionOperatorCallOpts, - ): AuditableOperation { - const lockContext = opts?.lockContext ?? defaults.lockContext - const audit = opts?.audit ?? defaults.audit - const withLock = lockContext ? op.withLockContext(lockContext) : op - if (audit) withLock.audit(audit) - return withLock - } - - function requireNonNullOperand( - ctx: ColumnContext, - value: unknown, - operator: string, - ): void { - if (value == null) { - throw new EncryptionOperatorError( - `Operator "${operator}" cannot encrypt a null operand for column "${ctx.columnName}". Use isNull() or isNotNull() for NULL checks.`, - { - columnName: ctx.columnName, - tableName: ctx.tableName, - operator, - }, - ) - } - } - - /** - * Reject a free-text needle the column's match index cannot answer. A needle - * shorter than the tokenizer's `token_length` yields an empty bloom filter, - * and `stored_bf @> '{}'` holds for every row — so without this the query - * silently returns the whole table. - */ - function requireAnswerableNeedle( - ctx: ColumnContext, - value: unknown, - operator: string, - ): void { - const match = ctx.indexes.match - if (!match) return - const reason = matchNeedleError(value, match) - if (reason) { - throw new EncryptionOperatorError( - `Operator "${operator}" cannot search column "${ctx.columnName}": ${reason}`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - } - - function operandFailure( - ctx: ColumnContext, - operator: string, - reason: string, - ): EncryptionOperatorError { - return new EncryptionOperatorError( - `Failed to encrypt query operand for "${ctx.columnName}": ${reason}`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - - /** - * Render a query term as a cast operand: `''::eql_v3.query_`. - * The cast is what reaches the bundle's `(domain, query_)` overloads — - * a bare `::jsonb` would hit the storage-domain overload, whose CHECK demands - * the ciphertext `c` a query term deliberately omits. `queryCast` is derived - * from the column's own domain (see `queryCastForDomain`), so `sql.raw` is safe. - */ - function castOperand( - ctx: ColumnContext, - operator: string, - term: EncryptedQueryResult, - ): SQL { - if (ctx.queryCast === null) { - throw operandFailure( - ctx, - operator, - `column domain "${ctx.builder.getEqlType()}" has no query operand type.`, - ) - } - return sql`${JSON.stringify(term)}::${sql.raw(ctx.queryCast)}` - } - - async function encryptOperand( - ctx: ColumnContext, - value: unknown, - operator: string, - queryType: QueryTypeName, - opts?: EncryptionOperatorCallOpts, - ): Promise { - requireNonNullOperand(ctx, value, operator) - - const result = await applyOperationOptions( - client.encryptQuery( - value as never, - { - table: ctx.table, - column: ctx.builder, - queryType, - } as never, - ), - opts, - ) - if (result.failure) { - throw operandFailure(ctx, operator, result.failure.message) - } - return castOperand(ctx, operator, result.data) - } - - /** - * Encrypt a whole operand list in ONE `encryptQuery` batch crossing (rather - * than one per value). The batch is position-stable, so the returned terms - * align index-for-index with `values`; a response of a different length means - * the contract was violated and is rejected rather than silently truncating - * the predicate (which would widen an `inArray` or narrow a `notInArray`). - * Null operands are rejected up front, so no term is filtered out of the batch. - */ - async function encryptOperands( - ctx: ColumnContext, - values: unknown[], - operator: string, - queryType: QueryTypeName, - opts?: EncryptionOperatorCallOpts, - ): Promise { - for (const value of values) requireNonNullOperand(ctx, value, operator) - - const terms = values.map((value) => ({ - value, - column: ctx.builder, - table: ctx.table, - queryType, - })) - const result = await applyOperationOptions( - client.encryptQuery(terms as never), - opts, - ) - if (result.failure) { - throw operandFailure(ctx, operator, result.failure.message) - } - - const encrypted = result.data as EncryptedQueryResult[] - if (encrypted.length !== values.length) { - throw operandFailure( - ctx, - operator, - `batch query encryption returned ${encrypted.length} terms for ${values.length} values.`, - ) - } - return encrypted.map((term) => castOperand(ctx, operator, term)) - } - - const colSql = (column: SQLWrapper): SQL => sql`${column}` - - async function equality( - op: EqualityOp, - left: SQLWrapper, - right: unknown, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, op) - requireIndex(ctx, EQUALITY_INDEXES, op, 'equality') - const enc = await encryptOperand(ctx, right, op, 'equality', opts) - return v3Dialect.equality(op, colSql(left), enc) - } - - async function comparison( - op: ComparisonOp, - left: SQLWrapper, - right: unknown, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, op) - requireIndex(ctx, ORDERING_INDEXES, op, 'order/range') - const enc = await encryptOperand(ctx, right, op, 'orderAndRange', opts) - return v3Dialect.comparison(op, colSql(left), enc) - } - - async function range( - left: SQLWrapper, - min: unknown, - max: unknown, - negate: boolean, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, operator) - requireIndex(ctx, ORDERING_INDEXES, operator, 'order/range') - // Independent operands — encrypt concurrently rather than paying two - // sequential round-trips to the crypto backend. - const [encMin, encMax] = await Promise.all([ - encryptOperand(ctx, min, operator, 'orderAndRange', opts), - encryptOperand(ctx, max, operator, 'orderAndRange', opts), - ]) - // `v3Dialect.range` is already parenthesised, so `NOT` binds to the whole - // conjunction without a wrapper here. - const condition = v3Dialect.range(colSql(left), encMin, encMax) - return negate ? sql`NOT ${condition}` : condition - } - - /** - * Fuzzy free-text token match on a `text_search`/`text_match` column. NOT - * containment: it tests whether the needle's downcased 3-gram set is a subset - * of the haystack's, via a bloom filter — order- and multiplicity-insensitive - * and one-sided (a `true` may be a false positive, a `false` never is). Emits - * `eql_v3.matches(col, operand)` (the SQL function's bundle name). - */ - async function matches( - left: SQLWrapper, - right: unknown, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, operator) - requireIndex(ctx, MATCH_INDEXES, operator, 'free-text search') - // The answerable-needle rule applies (a sub-`token_length` needle blooms to - // nothing and would match every row); the `query_` cast reaches the - // match overload. - requireAnswerableNeedle(ctx, right, operator) - const enc = await encryptOperand( - ctx, - right, - operator, - 'freeTextSearch', - opts, - ) - return v3Dialect.contains(colSql(left), enc) - } - - /** - * Exact encrypted-JSONB containment on an `eql_v3_json_search` (`ste_vec`) column: - * genuine jsonb `@>`, no false positives — hence it keeps the `contains` name. - * `eql_v3_json_search` has no `eql_v3.matches` overload; containment is the `@>` - * operator, whose `(eql_v3_json_search, eql_v3.query_json)` form takes a NARROWED - * query term (searchableJson → no ciphertext), cast to `eql_v3.query_json`. - */ - async function containsJsonOp( - left: SQLWrapper, - right: unknown, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, operator) - requireIndex(ctx, JSON_CONTAINMENT_INDEXES, operator, 'JSON containment') - const needle = await encryptJsonContainmentTerm(ctx, right, operator, opts) - return v3Dialect.containsJson(colSql(left), needle) - } - - /** - * Build a `query_json` containment needle for a `json` column — the JSON query - * term carries no ciphertext and satisfies the `eql_v3.query_json` CHECK the - * `@>` overload needs. Cast here (not by `queryCastForDomain`): a json column's - * domain is `eql_v3_json_search` but its query operand type is the irregular - * `eql_v3.query_json`. - */ - async function encryptJsonContainmentTerm( - ctx: ColumnContext, - value: unknown, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - requireNonNullOperand(ctx, value, operator) - // Reject the empty-object needle. `doc @> '{}'` holds for EVERY document - // (jsonb `{} ⊆ anything`), so `contains(col, {})` would silently return the - // whole table — the same whole-table footgun the bloom path guards against - // with `requireAnswerableNeedle`. An accidental empty filter is a bug, not a - // match-all request; callers wanting every row should omit the predicate. - if ( - value !== null && - typeof value === 'object' && - !Array.isArray(value) && - Object.keys(value).length === 0 - ) { - throw new EncryptionOperatorError( - `Operator "${operator}" cannot take an empty object needle on column "${ctx.columnName}": it matches every row. Pass a non-empty sub-object, or omit the predicate to select all rows.`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - const result = await applyOperationOptions( - client.encryptQuery( - value as never, - { - table: ctx.table, - column: ctx.builder, - queryType: 'searchableJson', - } as never, - ), - opts, - ) - if (result.failure) { - throw operandFailure(ctx, operator, result.failure.message) - } - return sql`${JSON.stringify(result.data)}::eql_v3.query_json` - } - - /** - * JSONPath selector-with-constraint on an `eql_v3_json_search` (`ste_vec`) - * column. Equality uses a value-selector containment needle, allowing the - * functional GIN index to answer the query. Ordering extracts the path entry - * with a selector hash and compares it to a ciphertext-free scalar ordering - * term. No storage ciphertext is placed in the WHERE clause. - */ - async function selectorCompare( - col: SQLWrapper, - path: string, - op: EqualityOp | ComparisonOp, - value: unknown, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(col, operator) - requireIndex( - ctx, - JSON_CONTAINMENT_INDEXES, - operator, - 'JSON selector (searchableJson)', - ) - requireNonNullOperand(ctx, value, operator) - - // A selector compares a scalar leaf — reject non-scalars / non-orderable - // types up front with a clear error, not a deferred DB failure. - const ordering = op !== 'eq' && op !== 'ne' - const leafReason = unsupportedLeafReason(value, ordering) - if (leafReason) { - throw new EncryptionOperatorError( - `Operator "${operator}" cannot compare column "${ctx.columnName}": ${leafReason}`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - - // Surface path-validation failures as EncryptionOperatorError with context. - let segments: string[] - try { - segments = parseSelectorSegments(path) - } catch (err) { - throw new EncryptionOperatorError( - `Operator "${operator}" on column "${ctx.columnName}": ${err instanceof Error ? err.message : String(err)}`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - - const canonicalPath = jsonPathOf(segments) - - if (op === 'eq' || op === 'ne') { - const result = await applyOperationOptions( - client.encryptQuery( - { path: canonicalPath, value } as never, - { - table: ctx.table, - column: ctx.builder, - queryType: 'steVecValueSelector', - } as never, - ), - opts, - ) - if (result.failure) { - throw operandFailure(ctx, operator, result.failure.message) - } - const contains = v3Dialect.containsJson( - colSql(col), - sql`${JSON.stringify(result.data)}::eql_v3.query_json`, - ) - return op === 'eq' - ? contains - : sql`(NOT ${contains} OR ${colSql(col)} IS NULL)` - } - - // Selector hashing and scalar-term encryption are independent, so order - // them concurrently. `steVecTerm` accepts only the JSON-orderable scalar - // families (string and number), enforced above. - const [selResult, termResult] = await Promise.all([ - applyOperationOptions( - client.encryptQuery( - canonicalPath as never, - { - table: ctx.table, - column: ctx.builder, - queryType: 'steVecSelector', - } as never, - ), - opts, - ), - applyOperationOptions( - client.encryptQuery( - value as never, - { - table: ctx.table, - column: ctx.builder, - queryType: 'steVecTerm', - } as never, - ), - opts, - ), - ]) - if (selResult.failure) { - throw operandFailure(ctx, operator, selResult.failure.message) - } - if (termResult.failure) { - throw operandFailure(ctx, operator, termResult.failure.message) - } - - // A v3 selector term is the bare HMAC hash string; guard the shape so a - // wrapped envelope can't silently bind as a JSON blob and match no rows. - const selValue = selResult.data - if (typeof selValue !== 'string') { - throw operandFailure( - ctx, - operator, - `expected a bare selector hash, got ${typeof selValue}.`, - ) - } - - const selSql = sql`${selValue}::text` - const leftEntry = v3Dialect.selectorEntry(colSql(col), selSql) - const termCast = - typeof value === 'string' - ? 'eql_v3.query_text_ord' - : 'eql_v3.query_double_ord' - const rightTerm = sql`${JSON.stringify(termResult.data)}::${sql.raw(termCast)}` - return v3Dialect.comparison(op, leftEntry, rightTerm) - } - - /** Comparison methods bound to a `col->'path'` selector, mirroring the scalar - * operators. Async: each encrypts its operand. */ - function selectorOps(col: SQLWrapper, path: string) { - const at = - (op: EqualityOp | ComparisonOp) => - (value: unknown, opts?: EncryptionOperatorCallOpts) => - selectorCompare(col, path, op, value, `selector(${path}).${op}`, opts) - return { - /** `col->'path' = value` (encrypted equality at the selector). A row whose - * document lacks `path` is excluded (it is not equal to `value`). */ - eq: at('eq'), - /** `col->'path' <> value`, INCLUDING rows whose document lacks `path` - * ("not equal to value" covers "has no value"). */ - ne: at('ne'), - /** `col->'path' > value` (encrypted ordering at the selector). */ - gt: at('gt'), - /** `col->'path' >= value`. */ - gte: at('gte'), - /** `col->'path' < value`. */ - lt: at('lt'), - /** `col->'path' <= value`. */ - lte: at('lte'), - /** Order rows ascending by the encrypted scalar at `path`. Missing paths - * produce SQL NULL and follow PostgreSQL's normal NULL ordering. */ - asc: (opts?: EncryptionOperatorCallOpts) => - selectorOrder(col, path, 'asc', opts), - /** Order rows descending by the encrypted scalar at `path`. Missing paths - * produce SQL NULL and follow PostgreSQL's normal NULL ordering. */ - desc: (opts?: EncryptionOperatorCallOpts) => - selectorOrder(col, path, 'desc', opts), - } - } - - /** Build `ORDER BY eql_v3.ord_term(col -> selector::text)`. - * The selector is encrypted, but the extracted SteVec entry already carries - * its OPE ordering term, so no plaintext comparison operand is needed. */ - async function selectorOrder( - col: SQLWrapper, - path: string, - direction: 'asc' | 'desc', - opts?: EncryptionOperatorCallOpts, - ): Promise { - const operator = `selector(${path}).${direction}` - const ctx = resolveContext(col, operator) - requireIndex( - ctx, - JSON_CONTAINMENT_INDEXES, - operator, - 'JSON selector (searchableJson)', - ) - - let canonicalPath: string - try { - canonicalPath = jsonPathOf(parseSelectorSegments(path)) - } catch (err) { - throw new EncryptionOperatorError( - `Operator "${operator}" on column "${ctx.columnName}": ${err instanceof Error ? err.message : String(err)}`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - - const result = await applyOperationOptions( - client.encryptQuery( - canonicalPath as never, - { - table: ctx.table, - column: ctx.builder, - queryType: 'steVecSelector', - } as never, - ), - opts, - ) - if (result.failure) { - throw operandFailure(ctx, operator, result.failure.message) - } - if (typeof result.data !== 'string') { - throw operandFailure( - ctx, - operator, - `expected a bare selector hash, got ${typeof result.data}.`, - ) - } - - const entry = v3Dialect.selectorEntry( - colSql(col), - sql`${result.data}::text`, - ) - const term = v3Dialect.orderBy(entry, 'ope') - return direction === 'asc' ? asc(term) : desc(term) - } - - async function inArrayOp( - left: SQLWrapper, - values: unknown[], - negate: boolean, - operator: string, - opts?: EncryptionOperatorCallOpts, - ): Promise { - const ctx = resolveContext(left, operator) - if (values.length === 0) { - throw new EncryptionOperatorError( - `Operator "${operator}" requires a non-empty list of values for column "${ctx.columnName}".`, - { columnName: ctx.columnName, tableName: ctx.tableName, operator }, - ) - } - // Gate and resolve the context once for the whole list, then encrypt it in - // a single `encryptQuery` batch crossing. - requireIndex(ctx, EQUALITY_INDEXES, operator, 'equality') - const op: EqualityOp = negate ? 'ne' : 'eq' - const encrypted = await encryptOperands( - ctx, - values, - operator, - 'equality', - opts, - ) - const conditions = encrypted.map((enc) => - v3Dialect.equality(op, colSql(left), enc), - ) - // The empty-list guard above leaves `conditions` non-empty, so `and`/`or` - // never return undefined here. - return (negate ? and(...conditions) : or(...conditions)) as SQL - } - - function orderTerm(column: SQLWrapper, operator: string): SQL { - const ctx = resolveContext(column, operator) - requireIndex(ctx, ORDERING_INDEXES, operator, 'order/range') - return v3Dialect.orderBy(colSql(column), ctx.indexes.ore ? 'ore' : 'ope') - } - - async function combine( - joiner: typeof and, - empty: SQL, - conditions: (SQL | SQLWrapper | Promise | undefined)[], - ): Promise { - const present = conditions.filter( - (c): c is SQL | SQLWrapper | Promise => c !== undefined, - ) - const resolved = await Promise.all(present) - return joiner(...resolved) ?? empty - } - - return { - /** Equality: `column = value`. Encrypts `r` and emits `eql_v3.eq`. - * Requires a `unique` or `ore` index on the column. */ - eq: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - equality('eq', l, r, opts), - /** Inequality: `column <> value`. Encrypts `r` and emits `eql_v3.neq`. - * Requires a `unique` or `ore` index on the column. */ - ne: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - equality('ne', l, r, opts), - /** Greater-than: `column > value`. Encrypts `r` and emits `eql_v3.gt`. - * Requires an `ore` (order/range) index. */ - gt: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - comparison('gt', l, r, opts), - /** Greater-than-or-equal: `column >= value`. Encrypts `r` and emits - * `eql_v3.gte`. Requires an `ore` (order/range) index. */ - gte: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - comparison('gte', l, r, opts), - /** Less-than: `column < value`. Encrypts `r` and emits `eql_v3.lt`. - * Requires an `ore` (order/range) index. */ - lt: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - comparison('lt', l, r, opts), - /** Less-than-or-equal: `column <= value`. Encrypts `r` and emits - * `eql_v3.lte`. Requires an `ore` (order/range) index. */ - lte: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - comparison('lte', l, r, opts), - /** Inclusive range `min <= column <= max`. Encrypts both bounds - * concurrently. Requires an `ore` (order/range) index. */ - between: ( - l: SQLWrapper, - min: unknown, - max: unknown, - opts?: EncryptionOperatorCallOpts, - ) => range(l, min, max, false, 'between', opts), - /** Negated inclusive range `NOT (min <= column <= max)`. Encrypts both - * bounds concurrently. Requires an `ore` (order/range) index. */ - notBetween: ( - l: SQLWrapper, - min: unknown, - max: unknown, - opts?: EncryptionOperatorCallOpts, - ) => range(l, min, max, true, 'notBetween', opts), - /** Fuzzy free-text token match — the needle's 3-gram set is (bloom-)tested - * as a subset of the column's. NOT containment: order/multiplicity- - * insensitive and one-sided (a `true` may be a false positive). Encrypts - * `r`. Requires a `match` (free-text search) index. */ - matches: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - matches(l, r, 'matches', opts), - /** Exact encrypted-JSONB containment (`@>`): matches rows whose document - * contains the given sub-object. No false positives. Encrypts `r`. Requires - * a `ste_vec` index (a `types.Json` column). */ - contains: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => - containsJsonOp(l, r, 'contains', opts), - /** JSONPath selector-with-constraint on a `types.Json` (`ste_vec`) column. - * Returns comparison and ordering methods bound to `col->'path'` — e.g. - * `await ops.selector(users.doc, '$.age').gt(21)` emits - * `col->'' > `, while `.asc()` emits - * `ORDER BY eql_v3.ord_term(col->'')`. Its unique power over `contains` - * is ordering at a path (`gt`/`gte`/`lt`/`lte` and `asc`/`desc`); `eq`/`ne` - * are also provided. Dot-notation object paths only in v1. */ - selector: (l: SQLWrapper, path: string) => selectorOps(l, path), - /** Membership: ORs one encrypted `eq` term per value. The whole list is - * encrypted in one `encryptQuery` batch crossing. Rejects an empty list; - * requires a `unique` or `ore` index. */ - inArray: ( - l: SQLWrapper, - values: unknown[], - opts?: EncryptionOperatorCallOpts, - ) => inArrayOp(l, values, false, 'inArray', opts), - /** Non-membership: ANDs one encrypted `ne` term per value. The whole list - * is encrypted in one `encryptQuery` batch crossing. Rejects an empty list; - * requires a `unique` or `ore` index. */ - notInArray: ( - l: SQLWrapper, - values: unknown[], - opts?: EncryptionOperatorCallOpts, - ) => inArrayOp(l, values, true, 'notInArray', opts), - /** Ascending order by the encrypted order term (`eql_v3.ord_term` / - * `eql_v3.ord_term_ore`, by the column's ordering flavour). - * Synchronous (no operand to encrypt). Requires an ordering index. */ - asc: (c: SQLWrapper) => asc(orderTerm(c, 'asc')), - /** Descending order by the encrypted order term (`eql_v3.ord_term` / - * `eql_v3.ord_term_ore`, by the column's ordering flavour). - * Synchronous (no operand to encrypt). Requires an ordering index. */ - desc: (c: SQLWrapper) => desc(orderTerm(c, 'desc')), - /** Conjunction of the given conditions, awaiting any async operands and - * dropping `undefined`. Empty input resolves to `true`. */ - and: (...conds: (SQL | SQLWrapper | Promise | undefined)[]) => - combine(and, sql`true`, conds), - /** Disjunction of the given conditions, awaiting any async operands and - * dropping `undefined`. Empty input resolves to `false`. */ - or: (...conds: (SQL | SQLWrapper | Promise | undefined)[]) => - combine(or, sql`false`, conds), - /** Drizzle's `isNull`, re-exported unchanged — `column IS NULL` needs no - * encryption and works on any (nullable) encrypted column. */ - isNull, - /** Drizzle's `isNotNull`, re-exported unchanged — `column IS NOT NULL` - * needs no encryption. */ - isNotNull, - /** Drizzle's `not`, re-exported unchanged — negates an already-built - * (encrypted) predicate. Safe over any operator here, including `between`, - * whose fragment is self-parenthesising. */ - not, - /** Drizzle's `exists`, re-exported unchanged — for correlated subqueries. */ - exists, - /** Drizzle's `notExists`, re-exported unchanged — for correlated - * subqueries. */ - notExists, - } -} diff --git a/packages/stack-drizzle/src/v3/schema-extraction.ts b/packages/stack-drizzle/src/v3/schema-extraction.ts deleted file mode 100644 index 68f8796bb..000000000 --- a/packages/stack-drizzle/src/v3/schema-extraction.ts +++ /dev/null @@ -1,50 +0,0 @@ -import { - type AnyEncryptedV3Column, - type AnyV3Table, - encryptedTable, -} from '@cipherstash/stack/eql/v3' -import type { PgTable } from 'drizzle-orm/pg-core' -import { getEqlV3Column } from './column.js' - -/** Drizzle stashes the SQL table name on this well-known symbol key. */ -const DRIZZLE_NAME = Symbol.for('drizzle:Name') - -/** - * Read the SQL table name Drizzle stashes on a `pgTable`. Returns `undefined` - * for a non-object or a table not built with `pgTable()`. Shared by - * {@link extractEncryptionSchemaV3} and the operator factory so the - * symbol-key introspection lives in exactly one place. - */ -export function getDrizzleTableName(table: unknown): string | undefined { - if (!table || typeof table !== 'object') return undefined - const name = (table as Record)[DRIZZLE_NAME] - return typeof name === 'string' ? name : undefined -} - -export function extractEncryptionSchemaV3(table: PgTable): AnyV3Table { - const tableName = getDrizzleTableName(table) - if (!tableName) { - throw new Error( - 'Unable to read table name from Drizzle table. Use a table created with pgTable().', - ) - } - - const columns: Record = {} - for (const [property, column] of Object.entries(table)) { - if (typeof column !== 'object' || column === null) continue - const columnName = - 'name' in column && typeof column.name === 'string' - ? column.name - : property - const builder = getEqlV3Column(columnName, column) - if (builder) columns[property] = builder - } - - if (Object.keys(columns).length === 0) { - throw new Error( - `No encrypted v3 columns found in table "${tableName}". Declare columns with the v3 drizzle \`types\` namespace.`, - ) - } - - return encryptedTable(tableName, columns) -} diff --git a/packages/stack-drizzle/tsup.config.ts b/packages/stack-drizzle/tsup.config.ts index 19ce5967e..dcffd3486 100644 --- a/packages/stack-drizzle/tsup.config.ts +++ b/packages/stack-drizzle/tsup.config.ts @@ -1,7 +1,7 @@ import { defineConfig } from 'tsup' export default defineConfig({ - entry: ['src/index.ts', 'src/v3/index.ts'], + entry: ['src/index.ts'], outDir: 'dist', format: ['cjs', 'esm'], sourcemap: true, diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 7a9b3366a..3777aac76 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -1,11 +1,11 @@ --- name: stash-drizzle -description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle/v3 (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the EncryptionV3 typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. +description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the EncryptionV3 typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. --- # CipherStash Stack - Drizzle ORM Integration -Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle/v3` (EQL v3). Provides Drizzle-native encrypted column factories and query operators that transparently encrypt search values — Drizzle never sees plaintext in a query. +Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle` (EQL v3). Provides Drizzle-native encrypted column factories and query operators that transparently encrypt search values — Drizzle never sees plaintext in a query. In EQL v3 every encrypted column is a **concrete Postgres domain** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...) whose query capabilities are fixed by the type you pick — there is no capability config object. See the `stash-encryption` skill's "Schema Definition" section (the `types` catalog) for the full catalog and capability suffixes (`Eq`, `Ord`/`OrdOre`, `Match`, `Search`, `Json`). @@ -33,8 +33,8 @@ npm install @cipherstash/stack @cipherstash/stack-drizzle drizzle-orm The Drizzle integration ships as its own first-party package, `@cipherstash/stack-drizzle`, which depends on `@cipherstash/stack`. Install both. -The v3 surface documented here lives on the `@cipherstash/stack-drizzle/v3` subpath. -It is distinct from the older, separate `@cipherstash/drizzle` package (which is +The v3 surface documented here is exported from the `@cipherstash/stack-drizzle` +package root. It is distinct from the older, separate `@cipherstash/drizzle` package (which is `@cipherstash/protect`-based, with different symbol names) — that package is **deprecated and no longer published**; do not install it. This package replaces it. @@ -83,11 +83,11 @@ Encrypted predicates need functional indexes over the `eql_v3.*` extractors, and ## Schema Definition -Use the `types` namespace from `@cipherstash/stack-drizzle/v3` to define encrypted columns. Each factory maps 1:1 to a Postgres domain, and the column's query capabilities are fixed by the type: +Use the `types` namespace from `@cipherstash/stack-drizzle` to define encrypted columns. Each factory maps 1:1 to a Postgres domain, and the column's query capabilities are fixed by the type: ```typescript import { pgTable, integer, timestamp, varchar } from "drizzle-orm/pg-core" -import { types } from "@cipherstash/stack-drizzle/v3" +import { types } from "@cipherstash/stack-drizzle" const usersTable = pgTable("users", { id: integer("id").primaryKey().generatedAlwaysAsIdentity(), @@ -117,18 +117,18 @@ Capability suffixes at a glance (full catalog: `stash-encryption` skill, "Schema Value families: `Integer`/`Smallint`/`Numeric`/`Real`/`Double` (`number`), `Bigint` (`bigint`), `Date`/`Timestamp` (`Date`), `Text` (`string`), `Boolean` (`boolean`, storage only), `Json` (a JSON document — object or array, not a top-level scalar). -`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the Drizzle subpath is shorthand for the same thing. +`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the Drizzle package is shorthand for the same thing. ## Initialization ### 1. Extract Schema from Drizzle Table ```typescript -import { extractEncryptionSchemaV3, createEncryptionOperatorsV3 } from "@cipherstash/stack-drizzle/v3" +import { extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" import { EncryptionV3 } from "@cipherstash/stack/v3" // Convert the Drizzle table definition to a CipherStash v3 schema -const usersSchema = extractEncryptionSchemaV3(usersTable) +const usersSchema = extractEncryptionSchema(usersTable) ``` ### 2. Initialize the Encryption Client @@ -144,10 +144,10 @@ const encryptionClient = await EncryptionV3({ ### 3. Create Query Operators ```typescript -const ops = createEncryptionOperatorsV3(encryptionClient) +const ops = createEncryptionOperators(encryptionClient) ``` -`createEncryptionOperatorsV3(client, { lockContext, audit })` optionally sets defaults applied to every operand encryption; the async encrypting operators (`eq`, `ne`, `inArray`, `notInArray`, `gt`/`gte`/`lt`/`lte`, `between`/`notBetween`, `matches`, `contains`, and all methods returned by `selector(...)`) also take an optional trailing `{ lockContext, audit }` argument per call. Top-level `asc`/`desc` and the passthrough operators (`isNull`, `isNotNull`, `not`, `and`, `or`, `exists`, `notExists`) encrypt nothing and take no such argument. +`createEncryptionOperators(client, { lockContext, audit })` optionally sets defaults applied to every operand encryption; the async encrypting operators (`eq`, `ne`, `inArray`, `notInArray`, `gt`/`gte`/`lt`/`lte`, `between`/`notBetween`, `matches`, `contains`, and all methods returned by `selector(...)`) also take an optional trailing `{ lockContext, audit }` argument per call. Top-level `asc`/`desc` and the passthrough operators (`isNull`, `isNotNull`, `not`, `and`, `or`, `exists`, `notExists`) encrypt nothing and take no such argument. ### 4. Create Drizzle Instance @@ -451,7 +451,7 @@ Add an `email_encrypted` column **alongside** `email`. Crucially, the encrypted ```typescript // src/db/schema.ts -import { types } from '@cipherstash/stack-drizzle/v3' +import { types } from '@cipherstash/stack-drizzle' export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), @@ -465,10 +465,10 @@ Update the encryption client to harvest the encrypted columns from the table: ```typescript // src/encryption/index.ts import { EncryptionV3 } from '@cipherstash/stack/v3' -import { extractEncryptionSchemaV3 } from '@cipherstash/stack-drizzle/v3' +import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' import { users } from '../db/schema' -const usersEncryptionSchema = extractEncryptionSchemaV3(users) +const usersEncryptionSchema = extractEncryptionSchema(users) export const encryptionClient = await EncryptionV3({ schemas: [usersEncryptionSchema] }) ``` @@ -544,49 +544,14 @@ If something goes wrong (e.g. you discover the dual-write code wasn't actually l #### Switch reads to the encrypted column -**EQL v3 (the schema above): there is no cut-over.** The encrypted column keeps -its own name — you switch the application to it by name, verify reads, then drop -the plaintext column. Point your queries at `email_encrypted`, deploy, and -confirm reads decrypt correctly; then skip ahead to the drop step. Running -`stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +**EQL v3: there is no cut-over.** The encrypted column keeps its own name — you +switch the application to it by name, verify reads, then drop the plaintext +column. Running `stash encrypt cutover` on a **backfilled** v3 column reports +"not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). -The rest of this subsection is the **EQL v2** path (a `eql_v2_encrypted` twin), -kept for existing v2 deployments. - -First, update the Drizzle schema to the post-cutover shape — switch `email` to the encrypted type and remove the `email_encrypted` column. - -> **Using CipherStash Proxy?** -> -> If using Proxy, re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): -> -> ```bash -> stash db push -> # → writes the new config as `pending`. Active config (still pointing at -> # `email_encrypted`) keeps serving while we complete the cutover. -> ``` - -Now run the cutover: - -```bash -stash encrypt cutover --table users --column email -``` - -Inside one transaction it: (1) renames `email` → `email_plaintext` and `email_encrypted` → `email`, (2) promotes the pending EQL config to `active` (and the prior active to `inactive`), (3) records a `cut_over` event in `cs_migrations`. - -The Drizzle schema you just edited now matches the physical DB shape — `email` is the encrypted column. Keep the temporary `email_plaintext: text('email_plaintext')` declaration in the schema file until the drop step: - -```typescript -// src/db/schema.ts (post-cutover) -export const users = pgTable('users', { - id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: types.TextSearch('email'), - email_plaintext: text('email_plaintext'), // temporary; dropped next -}) -``` - -App code that does `SELECT email FROM users` now returns ciphertext that must be decrypted via the encryption client. **This is the moment that breaks read paths if they aren't decrypting.** - -Update read paths to decrypt: +Point your read paths at `email_encrypted` and decrypt the selected envelopes +with the encryption client. **This is the moment that breaks read paths if they +aren't decrypting.** ```typescript // Before @@ -597,10 +562,10 @@ const email = rows[0].email const rows = await db.select().from(users).where(eq(users.id, id)) const decrypted = await encryptionClient.decryptModel(rows[0], usersEncryptionSchema) if (decrypted.failure) throw new Error(decrypted.failure.message) -const email = decrypted.data.email +const email = decrypted.data.email_encrypted ``` -For queries that filter on `email`, switch to the encrypted operators from `createEncryptionOperatorsV3` — `eq`, `matches`, `gte`, etc. (See `## Query Encrypted Data` above.) +For queries that filter on the column, switch to the encrypted operators from `createEncryptionOperators` — `eq`, `matches`, `gte`, etc. (See `## Query Encrypted Data` above.) Deploy, confirm reads decrypt correctly, then drop the plaintext column. #### Drop: remove the plaintext column @@ -610,10 +575,10 @@ Once read paths are updated and you're confident reads are decrypting correctly, stash encrypt drop --table users --column email ``` -The CLI emits a Drizzle migration file with the drop. **Which column it drops depends on the EQL version**, which the CLI auto-detects: - -- **v3** — drops the original plaintext column, `ALTER TABLE users DROP COLUMN email;`. There was no rename, so no `email_plaintext` exists. Requires the column to be in the `backfilled` phase, plus a live coverage check. -- **v2** — drops the post-rename leftover, `ALTER TABLE users DROP COLUMN email_plaintext;`. Requires the `cut-over` phase. +The CLI emits a Drizzle migration file with the drop. For a v3 column it drops +the original plaintext column, `ALTER TABLE users DROP COLUMN email;` — there was +no rename, so no `email_plaintext` exists. Requires the column to be in the +`backfilled` phase, plus a live coverage check. Review and apply with `drizzle-kit migrate`, then update the schema to its final shape — the encrypted column is the only one left: @@ -678,11 +643,11 @@ All comparison/containment operators auto-encrypt their operands and are async; ### Other v3 Exports -`types`, `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchemaV3`, `createEncryptionOperatorsV3`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError` — all from `@cipherstash/stack-drizzle/v3`. +`types`, `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchema`, `createEncryptionOperators`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError` — all from `@cipherstash/stack-drizzle`. ## Error Handling -Operators throw `EncryptionOperatorError` (exported from `@cipherstash/stack-drizzle/v3`) whenever the query cannot be answered safely: +Operators throw `EncryptionOperatorError` (exported from `@cipherstash/stack-drizzle`) whenever the query cannot be answered safely: - the column is not an encrypted v3 column (there is no plaintext fallback); - the column's domain lacks the operator's capability (e.g. ordering a `TextEq` column, `eq` on a `Json` column); @@ -691,7 +656,7 @@ Operators throw `EncryptionOperatorError` (exported from `@cipherstash/stack-dri - operand encryption itself fails. ```typescript -import { EncryptionOperatorError } from "@cipherstash/stack-drizzle/v3" +import { EncryptionOperatorError } from "@cipherstash/stack-drizzle" class EncryptionOperatorError extends Error { context?: { @@ -706,6 +671,4 @@ There is no `EncryptionConfigError` on the v3 path — capability problems surfa Encryption client operations (`encryptModel`, `bulkDecryptModels`, ...) don't throw — they return `Result` objects with `data` or `failure`. Check `.failure` before using `.data`. -## Legacy: EQL v2 - -The original v2 integration — `encryptedType` config-flag columns, `extractEncryptionSchema`, and `createEncryptionOperators` (with `like`/`ilike`) from the `@cipherstash/stack-drizzle` package root, over the `eql_v2_encrypted` column type installed via `stash eql install --drizzle` (the deprecated standalone `@cipherstash/drizzle` package shipped its own `generate-eql-migration` bin for the same purpose) — still exists for existing deployments and is documented at https://cipherstash.com/docs. New projects must use the `/v3` surface documented above — `stash init --drizzle` generates an EQL **v3** migration via `stash eql migration --drizzle`. Reaching the v2 install path now requires opting in explicitly with `stash eql install --drizzle --eql-version 2`. +> **EQL v2 removed.** `@cipherstash/stack-drizzle` no longer ships an EQL v2 authoring surface: the pre-v3 `encryptedType` config-flag columns and the `like`/`ilike` operators are gone, and the `./v3` subpath has collapsed into the package root (`@cipherstash/stack-drizzle`). All exports documented here are EQL v3. Existing v2 ciphertext is still decryptable via `@cipherstash/stack`; only the Drizzle-side authoring/query-building of new v2 columns is removed. `stash init --drizzle` generates an EQL v3 migration via `stash eql migration --drizzle`. From 92a99e4f2ed9cf9774c3e497136f7d82b8e81670 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 12:10:46 +1000 Subject: [PATCH 014/123] fix(bench): infer typed v3 client through a helper after Encryption collapse PR 3 made EncryptionV3 an overloaded alias; ReturnType now resolves to the nominal overload (EncryptionClient) not the typed client. Infer through a single-signature helper so the bench handle keeps the typed client type. --- packages/bench/src/drizzle/setup.ts | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index 267ffc951..16ba56d2e 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -34,8 +34,20 @@ export type BenchPlaintextRow = { enc_jsonb: { idx: number; group: number } } +/** + * Build the typed EQL v3 client this bench drives. Wrapped in a + * single-signature helper because `EncryptionV3` is now overloaded (typed v3 + * vs. nominal) — `ReturnType` resolves to the *last* + * (nominal) overload, so we infer the return type through this helper instead. + */ +function makeEncryptionClient() { + return EncryptionV3({ schemas: [encryptionBenchTable] }) +} + /** The typed EQL v3 client this bench drives. */ -export type BenchEncryptionClient = Awaited> +export type BenchEncryptionClient = Awaited< + ReturnType +> export type BenchHandle = { pgClient: pg.Client @@ -57,9 +69,7 @@ export async function buildBench(): Promise { const db = drizzle(pool) - const encryptionClient = await EncryptionV3({ - schemas: [encryptionBenchTable], - }) + const encryptionClient = await makeEncryptionClient() return { pgClient, pool, db, encryptionClient } } From 3f0c616c653e59eea99f6c22ac522ea51e94e715 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 14:10:58 +1000 Subject: [PATCH 015/123] ci(bench): install EQL v3 through the real CLI instead of relying on the image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bench fixture schema moved to concrete EQL v3 domains (`public.eql_v3_text_search`, `eql_v3.eq_term(...)`), but tests-bench.yml started `postgres-eql:17-2.3.1` — which ships EQL v2 — and ran the suite immediately. `applySchema` failed with `type "public.eql_v3_text_search" does not exist`. Nothing tied the image tag to the EQL version the fixtures need, so the coupling broke silently. Install v3 the way every integration suite already does: a vitest `globalSetup` calling test-kit's `installEqlV3`, which shells out to the real `stash eql install --eql-version 3`. That needs only a database URL — no CipherStash credentials — so `db-only.test.ts` stays the credential-free smoke test it is meant to be, and CI and `pnpm test:local` become the same path rather than CI depending on a step the local flow never ran. The globalSetup imports `@cipherstash/test-kit/install` (a new narrow subpath export) rather than the barrel: the barrel reaches `needle-for.ts`, which consumes stack source through the `@/` alias, and bench has no `stackSourceAlias` in its vitest config. `install.ts` depends on node builtins only. Workflow changes: - Use the `integration-setup` composite, which already builds the `stash` CLI the install needs (pinned to this job's existing Node 22). - Start only the `postgres` service; docker-compose.yml also defines a PostgREST that bench never talks to. - Widen the trigger paths to follow the package graph. bench depends on `@cipherstash/stack` and `@cipherstash/stack-drizzle`, and now on the CLI's EQL installer, but the filter covered only `packages/bench/**` and `local/**` — a break upstream could never trigger this job. No changeset: `@cipherstash/bench` and `@cipherstash/test-kit` are both private, and workflow files are repo tooling. --- .github/workflows/tests-bench.yml | 53 +++++++++++++--------- packages/bench/package.json | 1 + packages/bench/src/harness/global-setup.ts | 31 +++++++++++++ packages/bench/vitest.config.ts | 8 ++++ packages/test-kit/package.json | 1 + pnpm-lock.yaml | 3 ++ 6 files changed, 76 insertions(+), 21 deletions(-) create mode 100644 packages/bench/src/harness/global-setup.ts diff --git a/.github/workflows/tests-bench.yml b/.github/workflows/tests-bench.yml index d36eee339..5f1d8c5ad 100644 --- a/.github/workflows/tests-bench.yml +++ b/.github/workflows/tests-bench.yml @@ -1,20 +1,37 @@ name: "Test Benchmark JS" +# Trigger paths follow the package GRAPH, not the bench directory. bench depends +# on `@cipherstash/stack` and `@cipherstash/stack-drizzle`, and (since the fixture +# schema moved to EQL v3 domains) on the CLI's EQL installer — a break in any of +# them can red this job, so an edit to any of them has to be able to trigger it. +# Scoped the same way the `integration-*` workflows scope theirs rather than +# taking whole packages, to keep the trigger surface honest. on: push: branches: - 'main' paths: - 'packages/bench/**' + - 'packages/stack/src/eql/v3/**' + - 'packages/stack-drizzle/**' + - 'packages/test-kit/**' + - 'packages/cli/src/installer/**' - 'local/**' - '.github/workflows/tests-bench.yml' + - '.github/actions/integration-setup/**' pull_request: branches: - "**" + # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases. paths: - 'packages/bench/**' + - 'packages/stack/src/eql/v3/**' + - 'packages/stack-drizzle/**' + - 'packages/test-kit/**' + - 'packages/cli/src/installer/**' - 'local/**' - '.github/workflows/tests-bench.yml' + - '.github/actions/integration-setup/**' jobs: tests-bench: @@ -25,25 +42,14 @@ jobs: - name: Checkout Repo uses: actions/checkout@v6 - - uses: pnpm/action-setup@v6.0.9 - name: Install pnpm - with: - run_install: false - - - name: Install Node.js - uses: actions/setup-node@v6.5.0 + # The shared integration preamble, which also builds the `stash` CLI — not + # incidental here: the bench `globalSetup` installs EQL v3 by shelling out + # to the real `stash eql install --eql-version 3`. + - uses: ./.github/actions/integration-setup with: + # This job's existing Node; the composite defaults to 24 for the + # integration suites. node-version: 22 - cache: 'pnpm' - - # node-pty's install hook falls back to `node-gyp rebuild` when no - # linux-x64 prebuild matches. pnpm/action-setup v6 no longer ships - # node-gyp on PATH, so install it explicitly. - - name: Install node-gyp - run: npm install -g node-gyp - - - name: Install dependencies - run: pnpm recursive install --frozen-lockfile # `@cipherstash/stack` ships dist/-based `exports`; bench imports # from `@cipherstash/stack` and `@cipherstash/stack-drizzle` (the Drizzle @@ -54,11 +60,16 @@ jobs: - name: Build stack + adapter packages run: pnpm exec turbo run build --filter @cipherstash/stack --filter @cipherstash/stack-drizzle - # Starts the pinned postgres-eql container (PostgreSQL 17 + EQL - # pre-installed) via local/docker-compose.yml; waits for healthcheck. - - name: Start local Postgres (EQL) + # Service-scoped `postgres`: local/docker-compose.yml also defines a + # PostgREST that bench never talks to. + # + # This container has EQL v2 baked in and NOTHING to do with the v3 domains + # the fixture schema needs — those come from the `globalSetup` install + # below. Do not add an EQL step here; keeping the install inside vitest is + # what makes `pnpm test:local` and CI the same path. + - name: Start local Postgres working-directory: local - run: docker compose up --wait --wait-timeout 60 + run: docker compose up --wait --wait-timeout 60 postgres # `pnpm test:local` resolves to `vitest run`; the trailing `db-only` # is a vitest path filter that narrows to __tests__/db-only.test.ts diff --git a/packages/bench/package.json b/packages/bench/package.json index 8c95c944a..4134c57ce 100644 --- a/packages/bench/package.json +++ b/packages/bench/package.json @@ -18,6 +18,7 @@ "pg": "^8.22.0" }, "devDependencies": { + "@cipherstash/test-kit": "workspace:*", "@types/node": "^22.20.1", "@types/pg": "^8.20.0", "tsx": "catalog:repo", diff --git a/packages/bench/src/harness/global-setup.ts b/packages/bench/src/harness/global-setup.ts new file mode 100644 index 000000000..ffb318a6e --- /dev/null +++ b/packages/bench/src/harness/global-setup.ts @@ -0,0 +1,31 @@ +// The `/install` subpath, NOT the barrel: the barrel reaches `needle-for.ts`, +// which imports stack source through the `@/` alias, and bench has no +// `stackSourceAlias` in its vitest config (it consumes stack through its +// published dist/ exports). `install.ts` depends on node builtins only. +import { installEqlV3 } from '@cipherstash/test-kit/install' +import { getDatabaseUrl } from './db.js' + +/** + * Install EQL v3 once per run, before any suite applies `sql/schema.sql`. + * + * The fixture schema declares its columns as the concrete `eql_v3_*` domains, so + * the bundle has to be in the database before the first `CREATE TABLE`. Doing it + * here rather than in a CI-only step means the local `pnpm test:local` path and + * the CI path are the same path — the previous arrangement had the workflow rely + * on an EQL-v2-preinstalled image while the schema had already moved to v3, and + * nothing connected the two. + * + * Runs through the real `stash eql install`, matching the integration suites' + * harness (`@cipherstash/test-kit`'s `integration/global-setup.ts`): an installer + * regression fails here instead of hiding behind a test-only SQL apply. + * + * Unlike those suites this needs NO CipherStash credentials — `installEqlV3` + * wants only a database URL — which keeps `db-only.test.ts` the credential-free + * smoke test it is meant to be. + */ +export async function setup(): Promise { + // EXPLICIT, never inferred from the environment. `dbVariant()` would guess + // from `PGRST_URL`, and bench's compose stack happens to define a PostgREST + // it never talks to — a guess here would silently apply the Supabase grants. + await installEqlV3(getDatabaseUrl(), 'postgres') +} diff --git a/packages/bench/vitest.config.ts b/packages/bench/vitest.config.ts index c0f631c04..ac80b923e 100644 --- a/packages/bench/vitest.config.ts +++ b/packages/bench/vitest.config.ts @@ -3,6 +3,14 @@ import { defineConfig } from 'vitest/config' export default defineConfig({ test: { include: ['__tests__/**/*.test.ts'], + // Installs EQL v3 before any suite applies `sql/schema.sql`, which declares + // its columns as `eql_v3_*` domains. See the file for why this is not a + // CI-only step. + globalSetup: ['./src/harness/global-setup.ts'], + // `@cipherstash/test-kit` is consumed as unbuilt TypeScript source, so it + // must not be externalized — same reason the integration suites carry this + // (`packages/test-kit/src/integration/config.ts`). + server: { deps: { inline: [/packages\/test-kit/] } }, testTimeout: 300_000, hookTimeout: 300_000, pool: 'forks', diff --git a/packages/test-kit/package.json b/packages/test-kit/package.json index 656ffc96c..8c4738389 100644 --- a/packages/test-kit/package.json +++ b/packages/test-kit/package.json @@ -8,6 +8,7 @@ "./integration-clerk": "./src/integration/clerk.ts", ".": "./src/index.ts", "./catalog": "./src/catalog.ts", + "./install": "./src/install.ts", "./json-suite": "./src/run-json-suite.ts", "./suite": "./src/run-family-suite.ts" }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index eb397065d..dac40d1ff 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -218,6 +218,9 @@ importers: specifier: ^8.22.0 version: 8.22.0 devDependencies: + '@cipherstash/test-kit': + specifier: workspace:* + version: link:../test-kit '@types/node': specifier: ^22.20.1 version: 22.20.1 From fe0afa2a4675604933f6e8a8c1e196ef4f382a4b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 15:52:53 +1000 Subject: [PATCH 016/123] =?UTF-8?q?fix(bench,cli,docs):=20address=20PR=20r?= =?UTF-8?q?eview=20=E2=80=94=20dead=20JSON=20index,=20broken=20init=20scaf?= =?UTF-8?q?fold,=20stale=20shipped=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six review findings, each with the regression test that would have caught it. - `packages/bench/sql/schema.sql` indexed `eql_v3.ste_vec(enc_jsonb)`, a mechanical rename of the v2 expression. It builds, and `db-only.test.ts`'s `pg_indexes` check passes, but nothing the adapter emits ever mentions it: `@>` on `eql_v3_json_search` inlines to `eql_v3.to_ste_vec_query(a)::jsonb` (the bundle's own comment says so). A permanently-dead index in the one fixture whose job is to prove index engagement. Now indexed on the inlined expression, with `scripts/__tests__/bench-index-expressions.test.mjs` pinning each bench index against the operator body in the vendored EQL bundle — no database, no credentials, which matters because the bench's own EXPLAIN assertions need credentials and never run in CI. - `stash init --drizzle` scaffolded `extractEncryptionSchemaV3` from `@cipherstash/stack-drizzle/v3` into the user's repo — both removed here, so a freshly-initialised project would not resolve. Flipped the drizzle half of the generated strings (the `@cipherstash/stack/v3` half stays for PR 7) and pinned it in `utils-codegen-drizzle.test.ts`. - `skills/stash-encryption`, `packages/stack/README.md` and `AGENTS.md` still documented the `./v3` subpath and the `*V3` names. Both skills and the stack README are shipped artifacts. `no-removed-drizzle-surface.test.mjs` scans the shipped file set (deliberately not CHANGELOGs or specs, which should still name the old surface). - `vitest.shared.ts` aliased `@cipherstash/stack-drizzle/v3` to the deleted `src/v3/index.ts`; `vitest-shared-alias.test.mjs` asserts every alias target exists on disk. - The bench `matches` operand was the shared `value` prefix, which every seeded row contains — the bloom index had nothing to narrow, so the number measured a full scan. Uses a full seeded value now. - The bench importer's only CI gate was the Bun job's `turbo build`, which swallows its own test failures. Added an explicit typecheck step to the main test job. The eq/matches index assertions are sound and now say why: the operator wrappers are `LANGUAGE sql IMMUTABLE STRICT` single-SELECT bodies, so the planner inlines them and applies the same inlining to the stored index expression. The old comment claimed the adapter emitted `eql_v3.eq_term(col) = eql_v3.hmac_256(value)` — it emits `eql_v3.eq(col, term)`, and `eql_v3.hmac_256` does not exist in the bundle at all. --- .changeset/remove-eql-v2-drizzle-root.md | 7 +- .github/workflows/tests.yml | 10 ++ AGENTS.md | 4 +- .../__benches__/drizzle/operators.bench.ts | 9 +- .../drizzle/operators.explain.test.ts | 15 +- packages/bench/sql/schema.sql | 25 ++- .../__tests__/utils-codegen-drizzle.test.ts | 64 ++++++++ .../cli/src/commands/init/lib/setup-prompt.ts | 2 +- packages/cli/src/commands/init/utils.ts | 12 +- packages/stack/README.md | 23 +-- .../bench-index-expressions.test.mjs | 142 ++++++++++++++++++ .../no-removed-drizzle-surface.test.mjs | 69 +++++++++ .../__tests__/vitest-shared-alias.test.mjs | 34 +++++ skills/stash-encryption/SKILL.md | 6 +- vitest.shared.ts | 7 +- 15 files changed, 393 insertions(+), 36 deletions(-) create mode 100644 packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts create mode 100644 scripts/__tests__/bench-index-expressions.test.mjs create mode 100644 scripts/__tests__/no-removed-drizzle-surface.test.mjs create mode 100644 scripts/__tests__/vitest-shared-alias.test.mjs diff --git a/.changeset/remove-eql-v2-drizzle-root.md b/.changeset/remove-eql-v2-drizzle-root.md index 66e61576f..64d918746 100644 --- a/.changeset/remove-eql-v2-drizzle-root.md +++ b/.changeset/remove-eql-v2-drizzle-root.md @@ -1,5 +1,6 @@ --- '@cipherstash/stack-drizzle': major +'@cipherstash/stack': patch 'stash': patch --- @@ -19,4 +20,8 @@ Remove the EQL v2 authoring surface from `@cipherstash/stack-drizzle` and collap The `types.*` column factories, `makeEqlV3Column` / `getEqlV3Column` / `isEqlV3Column`, the codec helpers (`v3ToDriver` / `v3FromDriver` / `EqlV3CodecError`), and `EncryptionOperatorError` are unchanged apart from moving to the root. -Existing EQL v2 ciphertext remains decryptable via `@cipherstash/stack` — only the Drizzle-side v2 authoring and query-building is removed. The bundled `stash-drizzle` skill is updated to the collapsed root imports. +Existing EQL v2 ciphertext remains decryptable via `@cipherstash/stack` — only the Drizzle-side v2 authoring and query-building is removed. + +**`stash` (patch):** `stash init --drizzle` scaffolded the removed surface — the generated encryption-client file and the Drizzle placeholder both imported `extractEncryptionSchemaV3` from `@cipherstash/stack-drizzle/v3`, so a freshly-initialised project would not resolve against this release. Both now emit the collapsed root import. The bundled `stash-drizzle` and `stash-encryption` skills are updated to match. + +**`@cipherstash/stack` (patch):** README only — its Drizzle section documented the removed `./v3` subpath and the `*V3` export names. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index bf0c851f5..2386d320b 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -136,6 +136,16 @@ jobs: - name: Typecheck (prisma-next — enforces v3 operator-capability gating) run: pnpm --filter @cipherstash/prisma-next run typecheck + # `packages/bench` is a live importer of `@cipherstash/stack` and + # `@cipherstash/stack-drizzle`, but it has no `test` script (its suites + # need a database), so `pnpm run test` never reaches it and an adapter + # rename could break it unnoticed. Its `build` is `tsc --noEmit`, and + # turbo's `^build` builds the two adapters first. This also runs in the + # Bun job's full `turbo build`, but that job swallows its own test + # failures — the importer guard should not depend on it. + - name: Typecheck (bench — guards the stack/stack-drizzle importers) + run: pnpm exec turbo run build --filter @cipherstash/bench + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/AGENTS.md b/AGENTS.md index d88560f4b..5fcc10440 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,7 +77,7 @@ If these variables are missing, tests that require live encryption will fail or - `packages/wizard`: AI-powered encryption setup (`@cipherstash/wizard`) - `packages/migrate`: Plaintext-to-encrypted column migration (`@cipherstash/migrate`) — resumable backfill, per-column state - `packages/prisma-next`: Prisma Next integration (`@cipherstash/prisma-next`) — searchable field-level encryption for Postgres. **EQL v3 only**: per-domain constructors (`cipherstash.TextSearch()` / `text()` / `bigIntOrd()` / …) and `cipherstashFromStack` (the `./v3` and `./stack` entries). The EQL v2 surface was removed — the adapter's baseline migration installs the EQL v3 bundle only (works on Supabase as a non-superuser) -- `packages/stack-drizzle`: Drizzle ORM integration (`@cipherstash/stack-drizzle`), depends on `@cipherstash/stack` — EQL v2 (`.`) and EQL v3 (`./v3`). Split out of `@cipherstash/stack`. +- `packages/stack-drizzle`: Drizzle ORM integration (`@cipherstash/stack-drizzle`), depends on `@cipherstash/stack` — **EQL v3 only**, on the package root (the v2 surface was removed and the old `./v3` subpath collapsed into `.`). Split out of `@cipherstash/stack`. - `packages/stack-supabase`: Supabase integration (`@cipherstash/stack-supabase`), depends on `@cipherstash/stack` — `encryptedSupabase` (v2) and `encryptedSupabaseV3` (v3). Split out of `@cipherstash/stack`. - `packages/nextjs`: Next.js helpers and Clerk integration (`./clerk` export) - `packages/utils`: Shared config (`utils/config`) and logger (`utils/logger`) @@ -154,7 +154,7 @@ Three rules to remember when editing CI or pnpm config: - `encryptQuery(terms[])` for batch query encryption - **Identity-aware encryption**: Authenticate the client as the end user with `OidcFederationStrategy` (`config.authStrategy`, re-exported from `@cipherstash/stack`), then chain `.withLockContext({ identityClaim })` on operations to bind the data key to a claim. The same claim must be used for encrypt and decrypt. (`LockContext.identify()` from `@cipherstash/stack/identity` is deprecated — the strategy now handles token acquisition; `.withLockContext()` also accepts a `LockContext`.) - **Integrations**: - - **Drizzle ORM**: `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` from `@cipherstash/stack-drizzle` (EQL v3 factories from `@cipherstash/stack-drizzle/v3`) + - **Drizzle ORM**: `types.*` column factories, `extractEncryptionSchema`, `createEncryptionOperators` from `@cipherstash/stack-drizzle` - **Supabase**: `encryptedSupabase` (v2) / `encryptedSupabaseV3` (v3) from `@cipherstash/stack-supabase` - **DynamoDB**: `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` diff --git a/packages/bench/__benches__/drizzle/operators.bench.ts b/packages/bench/__benches__/drizzle/operators.bench.ts index 53422497f..c7079a5a9 100644 --- a/packages/bench/__benches__/drizzle/operators.bench.ts +++ b/packages/bench/__benches__/drizzle/operators.bench.ts @@ -45,7 +45,14 @@ describe('drizzle', () => { }) bench('matches (free-text)', async () => { - const where = (await ops.matches(benchTable.encText, 'value')) as SQL + // A FULL seeded value, not the shared `value` prefix: every row is + // `value-<7 digits>`, so a bare `value` needle matches all 10k rows and the + // bloom index has nothing to narrow — the number would measure a full scan + // rather than the index path this bench exists to measure. + const where = (await ops.matches( + benchTable.encText, + 'value-0000042', + )) as SQL await handle.db.select().from(benchTable).where(where) }) diff --git a/packages/bench/__tests__/drizzle/operators.explain.test.ts b/packages/bench/__tests__/drizzle/operators.explain.test.ts index d3f58857e..546f4fb2c 100644 --- a/packages/bench/__tests__/drizzle/operators.explain.test.ts +++ b/packages/bench/__tests__/drizzle/operators.explain.test.ts @@ -105,8 +105,15 @@ async function tryExplainWhere(name: string, where: SQL): Promise { // --- #421: equality + array operators ------------------------------------- // // `bench_text_hmac_idx` (functional hash on eql_v3.eq_term) is the expected -// fast path. The v3 operators emit `eql_v3.eq_term(col) = eql_v3.hmac_256(value)` -// so the functional hash index scan kicks in instead of a seq scan. +// fast path. The v3 operators do NOT emit that expression directly — they emit +// the wrapper `eql_v3.eq(col, $1::eql_v3.query_text_search)`. It engages the +// index because the wrapper is `LANGUAGE sql IMMUTABLE STRICT` over a single +// SELECT, so the planner inlines it to `eql_v3.eq_term(col) = +// eql_v3.eq_term($1)` — and applies the same inlining to the stored index +// expression, which is how the two meet. Break the inlinability (plpgsql body, +// VOLATILE) and every assertion below silently degrades to a seq scan. +// `scripts/__tests__/bench-index-expressions.test.mjs` pins that contract +// against the shipped bundle without needing a database. // // `eq` and `inArray` are naturally high-selectivity (only a few rows match), // so the planner should pick the eq_term index — assertion enforces it. @@ -163,7 +170,9 @@ describe('#422: call-shaped operators (recorded, not asserted)', () => { it('records matches plan shape', async () => { await tryExplainWhere( 'matches', - (await ops.matches(benchTable.encText, 'value')) as SQL, + // A full seeded value — `value` alone is in every row, so the recorded + // plan would say nothing about the bloom index. See operators.bench.ts. + (await ops.matches(benchTable.encText, 'value-0000042')) as SQL, ) }) diff --git a/packages/bench/sql/schema.sql b/packages/bench/sql/schema.sql index 9390fa5e8..92e11c4e9 100644 --- a/packages/bench/sql/schema.sql +++ b/packages/bench/sql/schema.sql @@ -1,12 +1,31 @@ -- Bench fixture schema. -- Single bench table covering text / int / jsonb encrypted columns plus the --- three canonical EQL v3 functional indexes over the column-side index terms: --- eq_term (hash), match_term (GIN bloom), ste_vec (GIN). +-- three canonical EQL v3 functional indexes. -- -- Mirrors `src/drizzle/setup.ts`: the columns are concrete `eql_v3_*` Postgres -- domains (see `types.TextSearch` / `types.IntegerOrd` / `types.Json`). Install -- the domains and index-term functions first with `stash eql install --eql-version 3`. -- +-- Each index expression must be the expression the matching EQL v3 operator +-- INLINES TO, not merely a term extractor for the same column. The adapter +-- emits a function call (`eql_v3.eq(col, term)`, `eql_v3.matches(...)`, +-- `col @> needle`); every one of those is `LANGUAGE sql IMMUTABLE STRICT` +-- with a single-SELECT body, so the planner inlines it, and Postgres applies +-- the same inlining to the stored index expression. The two only meet if the +-- index is built on the inlined form: +-- +-- eql_v3.eq(a, b) -> eql_v3.eq_term(a) = eql_v3.eq_term(b) +-- eql_v3.matches(a, b) -> eql_v3.match_term(a) @> eql_v3.match_term(b) +-- a @> b (json) -> eql_v3.to_ste_vec_query(a)::jsonb +-- @> eql_v3.to_ste_vec_query(b)::jsonb +-- +-- That last one is why the JSON index is NOT on `eql_v3.ste_vec(...)`: that +-- function exists and the index builds happily, but nothing the adapter emits +-- ever mentions it, so the index is dead weight. The bundle says so itself, on +-- `eql_v3."@>"(eql_v3_json_search, eql_v3.query_json)`: "Inlines to native +-- `jsonb @>` over `eql_v3.to_ste_vec_query(a)::jsonb`, so a functional GIN +-- index on the same expression engages." +-- -- We deliberately do NOT create btree operator-class indexes: encrypted terms for -- full-feature columns blow past the 2704-byte btree page-size limit, and the -- bench's whole job is to validate that the functional-index path works. @@ -27,6 +46,6 @@ CREATE INDEX bench_text_bloom_idx ON bench USING gin (eql_v3.match_term(enc_text)); CREATE INDEX bench_jsonb_stevec_idx - ON bench USING gin (eql_v3.ste_vec(enc_jsonb)); + ON bench USING gin ((eql_v3.to_ste_vec_query(enc_jsonb)::jsonb)); ANALYZE bench; diff --git a/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts b/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts new file mode 100644 index 000000000..50acc3682 --- /dev/null +++ b/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, it } from 'vitest' +import type { SchemaDef } from '../types.js' +import { + generateClientFromSchemas, + generatePlaceholderClient, +} from '../utils.js' + +// `stash init --drizzle` writes these strings into the USER's repo as real +// source. Nothing type-checks a template literal, so a rename in +// `@cipherstash/stack-drizzle` cannot break the build here — it breaks the +// scaffolded project instead, in someone else's checkout. These assertions are +// the only thing standing between a de-suffixed export and a broken `stash +// init`. +// +// Pinned by this suite (all three removed when `./v3` collapsed into the root): +// - the `@cipherstash/stack-drizzle/v3` specifier +// - `extractEncryptionSchemaV3` +// - `createEncryptionOperatorsV3` + +const SCHEMAS: SchemaDef[] = [ + { + tableName: 'users', + columns: [ + { name: 'email', dataType: 'string', searchOps: ['equality'] }, + { name: 'age', dataType: 'number', searchOps: ['orderAndRange'] }, + ], + }, +] + +/** Every drizzle-flavoured string `stash init` can write into a user's repo. */ +const GENERATED_SOURCES: Array<[string, string]> = [ + ['generateClientFromSchemas', generateClientFromSchemas('drizzle', SCHEMAS)], + ['generatePlaceholderClient', generatePlaceholderClient('drizzle')], +] + +describe.each( + GENERATED_SOURCES, +)('drizzle scaffold — %s', (_name, generated) => { + it('imports the drizzle surface from the package ROOT, never the removed ./v3 subpath', () => { + expect(generated).not.toContain('@cipherstash/stack-drizzle/v3') + expect(generated).toContain('@cipherstash/stack-drizzle') + }) + + it('uses the de-suffixed export names, never the removed *V3 aliases', () => { + expect(generated).not.toContain('extractEncryptionSchemaV3') + expect(generated).not.toContain('createEncryptionOperatorsV3') + }) + + it('never scaffolds the removed EQL v2 authoring surface', () => { + expect(generated).not.toContain('encryptedType') + expect(generated).not.toContain('eql_v2_encrypted') + }) +}) + +describe('generateClientFromSchemas (drizzle)', () => { + const generated = generateClientFromSchemas('drizzle', SCHEMAS) + + it('names extractEncryptionSchema in both the import and the call site', () => { + expect(generated).toContain( + "import { types, extractEncryptionSchema } from '@cipherstash/stack-drizzle'", + ) + expect(generated).toContain('extractEncryptionSchema(usersTable)') + }) +}) diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index ae3fa5aa3..6380cacf2 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -277,7 +277,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { 'Use when the column **does not yet exist** in the database (no plaintext predecessor to preserve). This is normal Drizzle / Supabase work plus the encryption client patterns from the integration skill.', '', "1. **If this is the first encrypted column in the project, configure the bundler exclusion first.** `@cipherstash/stack` cannot be bundled (it wraps a native FFI module). Next.js: add `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` to `next.config.*`. Webpack: `externals`. esbuild: `external`. Vite SSR: `ssr.external`. Without this, the encryption client crashes at runtime with `Cannot find module '@cipherstash/protect-ffi-*'`. See the `stash-encryption` skill's Installation section for the full snippets.", - "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — `encryptedType` for Drizzle, `encryptedColumn` for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", + "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories for Drizzle, `encryptedColumn` for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`protectOps.eq`, etc. — see the integration skill).', diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 3045433d7..68526460e 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -279,13 +279,13 @@ ${columnDefs.join('\n')} createdAt: timestamp('created_at').defaultNow(), }) -const ${schemaVarName} = extractEncryptionSchemaV3(${varName})` +const ${schemaVarName} = extractEncryptionSchema(${varName})` }) const schemaVarNames = schemas.map((s) => `${toCamelCase(s.tableName)}Schema`) return `import { pgTable, integer, timestamp } from 'drizzle-orm/pg-core' -import { types, extractEncryptionSchemaV3 } from '@cipherstash/stack-drizzle/v3' +import { types, extractEncryptionSchema } from '@cipherstash/stack-drizzle' import { EncryptionV3 } from '@cipherstash/stack/v3' ${tableDefs.join('\n\n')} @@ -389,7 +389,7 @@ const DRIZZLE_PLACEHOLDER = `/** * pointing at this file. * * This project uses EQL v3. Encrypted columns are concrete Postgres domains - * built with the \`types.*\` factories from \`@cipherstash/stack-drizzle/v3\`. + * built with the \`types.*\` factories from \`@cipherstash/stack-drizzle\`. * Each domain's query capabilities are FIXED by the type you pick — there is * no capability config object. Choose the factory whose capabilities you need: * types.Text / types.Integer / … storage only (encrypt/decrypt, no queries) @@ -404,7 +404,7 @@ const DRIZZLE_PLACEHOLDER = `/** * Encrypted twin column for an existing populated column (path 3 — lifecycle): * * import { pgTable, integer, text } from 'drizzle-orm/pg-core' - * import { types } from '@cipherstash/stack-drizzle/v3' + * import { types } from '@cipherstash/stack-drizzle' * * export const users = pgTable('users', { * id: integer('id').primaryKey().generatedAlwaysAsIdentity(), @@ -421,12 +421,12 @@ const DRIZZLE_PLACEHOLDER = `/** * * Once you have encrypted tables declared, harvest them and pass to EncryptionV3(): * - * import { extractEncryptionSchemaV3 } from '@cipherstash/stack-drizzle/v3' + * import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' * import { EncryptionV3 } from '@cipherstash/stack/v3' * import { users, orders } from './db/schema' * * export const encryptionClient = await EncryptionV3({ - * schemas: [extractEncryptionSchemaV3(users), extractEncryptionSchemaV3(orders)], + * schemas: [extractEncryptionSchema(users), extractEncryptionSchema(orders)], * }) */ import { EncryptionV3 } from '@cipherstash/stack/v3' diff --git a/packages/stack/README.md b/packages/stack/README.md index f5450a725..8ca24e116 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -362,7 +362,7 @@ The one limitation is the ORE-backed `*OrdOre` domains: their ordering term need ### Drizzle Integration -The `@cipherstash/stack-drizzle/v3` subpath (of the separate `@cipherstash/stack-drizzle` package) provides Drizzle-native column factories, schema extraction, and auto-encrypting, capability-checked query operators. +The separate `@cipherstash/stack-drizzle` package provides Drizzle-native column factories, schema extraction, and auto-encrypting, capability-checked query operators. It is EQL v3 only, all on the package root — the EQL v2 surface was removed and the old `./v3` subpath collapsed into `.`. Declare a Drizzle table using the `types` factories — each factory emits its domain as the column's SQL type, so `drizzle-kit generate` produces `ADD COLUMN email public.eql_v3_text_search` etc.: @@ -371,9 +371,9 @@ import { pgTable, integer } from "drizzle-orm/pg-core" import { drizzle } from "drizzle-orm/postgres-js" import { types, - createEncryptionOperatorsV3, - extractEncryptionSchemaV3, -} from "@cipherstash/stack-drizzle/v3" + createEncryptionOperators, + extractEncryptionSchema, +} from "@cipherstash/stack-drizzle" import { EncryptionV3 } from "@cipherstash/stack/v3" // Capabilities come from the concrete type — no flags to configure. @@ -389,9 +389,9 @@ const users = pgTable("users", { Derive the v3 schema from the table, build the typed client, and create the operators: ```ts -const usersSchema = extractEncryptionSchemaV3(users) +const usersSchema = extractEncryptionSchema(users) const client = await EncryptionV3({ schemas: [usersSchema] }) -const ops = createEncryptionOperatorsV3(client) +const ops = createEncryptionOperators(client) const db = drizzle({ client: sqlClient }) ``` @@ -762,9 +762,8 @@ depend on `@cipherstash/stack` (they are no longer subpaths of it): | Package | Provides | |-------|-----| -| `@cipherstash/stack-drizzle/v3` | EQL v3 Drizzle integration: `types` column factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, `makeEqlV3Column`, `EncryptionOperatorError` | +| `@cipherstash/stack-drizzle` | EQL v3 Drizzle integration (package root, v3 only): `types` column factories, `createEncryptionOperators`, `extractEncryptionSchema`, `makeEqlV3Column`, `EncryptionOperatorError` | | `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabaseV3` (and the legacy v2 `encryptedSupabase`) | -| `@cipherstash/stack-drizzle` | Legacy EQL v2 Drizzle integration (root subpath): `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` | ## Legacy: EQL v2 @@ -781,9 +780,11 @@ emitted by protect-ffi 0.30 and must migrate to v3 `types.Json`: - **Query formatting**: v2 query terms can be rendered as strings with `returnType: 'composite-literal'` / `'escaped-composite-literal'` for string-based APIs. -- **Integrations**: the v2 Drizzle surface is the root of - `@cipherstash/stack-drizzle` (`encryptedType`, `extractEncryptionSchema`, - `createEncryptionOperators`); the v2 Supabase surface is `encryptedSupabase`. +- **Integrations**: the v2 Supabase surface is `encryptedSupabase`. There is no + v2 Drizzle surface any more — `@cipherstash/stack-drizzle` dropped + `encryptedType` and the v2 operators, so existing v2 Drizzle columns are + read-only: decrypt them through `@cipherstash/stack` and migrate to a v3 + domain. - **DynamoDB still requires v2**: `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` works with the v2 API only — v3 support is tracked in [#657](https://github.com/cipherstash/stack/issues/657). diff --git a/scripts/__tests__/bench-index-expressions.test.mjs b/scripts/__tests__/bench-index-expressions.test.mjs new file mode 100644 index 000000000..0ea8c9d10 --- /dev/null +++ b/scripts/__tests__/bench-index-expressions.test.mjs @@ -0,0 +1,142 @@ +import { readFileSync } from 'node:fs' +import { createRequire } from 'node:module' +import { dirname, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + +// The bench exists to prove the functional-index path engages. An index whose +// expression is merely "a term extractor for the right column" satisfies +// `CREATE INDEX` and satisfies the db-only smoke test's `pg_indexes` check, and +// is still never used — the planner only matches an index against the +// expression the operator INLINES TO. `eql_v3.ste_vec(enc_jsonb)` was exactly +// that: a valid, buildable, permanently-dead index, because `@>` on +// `eql_v3_json_search` inlines to `eql_v3.to_ste_vec_query(a)::jsonb`. +// +// Every EQL v3 operator wrapper is `LANGUAGE sql IMMUTABLE STRICT` with a +// single-SELECT body, which is what makes it inlinable — and Postgres runs the +// same simplification over stored index expressions, so the two meet only if +// the index names the function in the operator's BODY. +// +// This test reads the vendored EQL bundle (the same SQL `stash eql install` +// applies) and pins each bench index to the function its operator actually +// lowers to. It needs no database and no credentials, which is the point: the +// bench's own EXPLAIN assertions require CipherStash credentials and never run +// in CI. + +const require = createRequire(resolve(REPO_ROOT, 'packages/cli/package.json')) + +/** The EQL v3 bundle `stash eql install --eql-version 3` applies. */ +function bundleSql() { + const pkgJson = require.resolve('@cipherstash/eql/package.json') + return readFileSync( + resolve(dirname(pkgJson), 'dist/sql/cipherstash-encrypt.sql'), + 'utf8', + ) +} + +/** + * The body of one `CREATE FUNCTION` in the bundle, from its signature line to + * the `$$` that closes it. + */ +function functionBody(sql, signature) { + const start = sql.indexOf(signature) + expect(start, `bundle has no ${signature}`).toBeGreaterThan(-1) + const bodyStart = sql.indexOf('$$', start) + const bodyEnd = sql.indexOf('$$', bodyStart + 2) + return sql.slice(bodyStart + 2, bodyEnd) +} + +/** `eql_v3.(` calls appearing in a SQL fragment. */ +function eqlCalls(fragment) { + return [...fragment.matchAll(/eql_v3\.([a-z_0-9]+)\s*\(/g)].map((m) => m[1]) +} + +const SCHEMA_SQL = readFileSync( + resolve(REPO_ROOT, 'packages/bench/sql/schema.sql'), + 'utf8', +) + +/** `CREATE INDEX ... ()` pairs from the bench fixture. */ +function benchIndexes() { + const found = new Map() + for (const m of SCHEMA_SQL.matchAll( + /CREATE INDEX\s+(\w+)\s+ON bench USING \w+\s*\(([\s\S]*?)\);/g, + )) { + found.set(m[1], m[2].trim()) + } + return found +} + +/** + * Each bench index, and the operator wrapper whose inlined body it has to + * match. Signatures are the `query_` overloads — the narrowed query + * term is what the Drizzle adapter casts its operand to. + */ +const INDEX_CONTRACTS = [ + { + index: 'bench_text_hmac_idx', + operator: + 'CREATE FUNCTION eql_v3.eq(a public.eql_v3_text_search, b eql_v3.query_text_search)', + inlinesTo: 'eq_term', + }, + { + index: 'bench_text_bloom_idx', + operator: + 'CREATE FUNCTION eql_v3.matches(a public.eql_v3_text_search, b eql_v3.query_text_search)', + inlinesTo: 'match_term', + }, + { + index: 'bench_jsonb_stevec_idx', + operator: + 'CREATE FUNCTION eql_v3."@>"(a public.eql_v3_json_search, b eql_v3.query_json)', + inlinesTo: 'to_ste_vec_query', + }, +] + +describe('bench index expressions match the EQL v3 operator lowering', () => { + const sql = bundleSql() + const indexes = benchIndexes() + + it('parses all three bench indexes (guards a silently-empty regex)', () => { + expect([...indexes.keys()].sort()).toEqual( + INDEX_CONTRACTS.map((c) => c.index).sort(), + ) + }) + + it.each( + INDEX_CONTRACTS, + )('$index is built on the function $operator inlines to', ({ + index, + operator, + inlinesTo, + }) => { + const body = functionBody(sql, operator) + + // The operator really does lower to this function — if the bundle + // changes shape, this fails here rather than silently passing the + // schema check below against a stale assumption. + expect( + eqlCalls(body), + `${operator} no longer calls eql_v3.${inlinesTo}`, + ).toContain(inlinesTo) + + expect( + eqlCalls(indexes.get(index)), + `${index} must index eql_v3.${inlinesTo}(...) — the expression the ` + + 'operator inlines to — or the planner can never match it', + ).toEqual([inlinesTo]) + }) + + it('the operator wrappers are inlinable (LANGUAGE sql IMMUTABLE STRICT)', () => { + for (const { operator } of INDEX_CONTRACTS) { + const start = sql.indexOf(operator) + const header = sql.slice(start, sql.indexOf('$$', start)) + // A plpgsql or VOLATILE wrapper would never be inlined, so no functional + // index on its body could ever engage. + expect(header.toLowerCase(), operator).toMatch(/language sql/) + expect(header.toLowerCase(), operator).toMatch(/immutable/) + } + }) +}) diff --git a/scripts/__tests__/no-removed-drizzle-surface.test.mjs b/scripts/__tests__/no-removed-drizzle-surface.test.mjs new file mode 100644 index 000000000..64334bce4 --- /dev/null +++ b/scripts/__tests__/no-removed-drizzle-surface.test.mjs @@ -0,0 +1,69 @@ +import { execFileSync } from 'node:child_process' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + +/** + * The `@cipherstash/stack-drizzle` names removed when EQL v2 went away and + * `./v3` collapsed into the package root. Nothing type-checks a README, a + * SKILL.md, or a template literal, so these three drifted silently before — + * and `skills/` ships inside the `stash` tarball, which means the drift lands + * in a customer's repo rather than in CI. + */ +const REMOVED = [ + '@cipherstash/stack-drizzle/v3', + 'extractEncryptionSchemaV3', + 'createEncryptionOperatorsV3', +] + +/** + * Files whose contents are SHIPPED — published to npm, copied into a user's + * repo, or written there by `stash init`. Deliberately not the whole tree: + * CHANGELOGs and `docs/superpowers/specs/**` are historical records that + * SHOULD still name the old surface, and rewriting history to appease a lint + * is worse than the drift it prevents. + */ +// `:(glob)` magic so `*` stops at a path separator — without it git's default +// wildmatch crosses `/` and `lib/*.ts` sweeps in `lib/__tests__/*.test.ts`. +const SHIPPED_GLOBS = [ + ':(glob)skills/*/SKILL.md', + ':(glob)packages/*/README.md', + 'README.md', + 'AGENTS.md', + // `stash init` writes these strings into the user's project as real source. + 'packages/cli/src/commands/init/utils.ts', + ':(glob)packages/cli/src/commands/init/lib/*.ts', + ':(glob)packages/cli/src/commands/init/doctrine/*.md', +] + +/** Tracked files matching the shipped globs, via git so it honours .gitignore. */ +function shippedFiles() { + const out = execFileSync('git', ['ls-files', '-z', ...SHIPPED_GLOBS], { + cwd: REPO_ROOT, + encoding: 'utf8', + }) + return out.split('\0').filter(Boolean) +} + +describe('removed stack-drizzle surface is absent from shipped files', () => { + const files = shippedFiles() + + it('finds the shipped file set (guards against a silently-empty glob)', () => { + expect(files.length).toBeGreaterThan(5) + expect(files).toContain('skills/stash-drizzle/SKILL.md') + expect(files).toContain('packages/stack/README.md') + }) + + it.each(files)('%s', (file) => { + const body = readFileSync(resolve(REPO_ROOT, file), 'utf8') + const found = REMOVED.filter((name) => body.includes(name)) + expect( + found, + `${file} names removed @cipherstash/stack-drizzle exports: ${found.join(', ')}. ` + + 'The `./v3` subpath collapsed into the package root and the *V3 suffixes were dropped.', + ).toEqual([]) + }) +}) diff --git a/scripts/__tests__/vitest-shared-alias.test.mjs b/scripts/__tests__/vitest-shared-alias.test.mjs new file mode 100644 index 000000000..771d68f8a --- /dev/null +++ b/scripts/__tests__/vitest-shared-alias.test.mjs @@ -0,0 +1,34 @@ +import { existsSync } from 'node:fs' +import { describe, expect, it } from 'vitest' +import { sharedAlias, stackSourceAlias } from '../../vitest.shared.ts' + +// A vite alias pointing at a file that no longer exists fails LATE and badly: +// nothing validates the map, so the entry survives a package refactor and the +// next test to use that specifier gets a "failed to resolve import" on a path +// the author never wrote, instead of the honest "subpath is not exported". +// `@cipherstash/stack-drizzle/v3` sat here for exactly that reason after the +// `./v3` collapse. + +/** Directory aliases are written with a trailing slash (`'@/'`). */ +const targets = (map) => + Object.entries(map).map(([specifier, target]) => [ + specifier, + target.endsWith('/') ? target.slice(0, -1) : target, + ]) + +describe('vitest.shared.ts alias targets exist on disk', () => { + it('exports both maps, non-empty (guards a silently-renamed export)', () => { + expect(Object.keys(sharedAlias).length).toBeGreaterThan(5) + expect(Object.keys(stackSourceAlias).length).toBeGreaterThan(0) + }) + + it.each(targets(sharedAlias))('sharedAlias %s', (_specifier, target) => { + expect(existsSync(target), `missing alias target: ${target}`).toBe(true) + }) + + it.each( + targets(stackSourceAlias), + )('stackSourceAlias %s', (_specifier, target) => { + expect(existsSync(target), `missing alias target: ${target}`).toBe(true) + }) +}) diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index c140d2bfd..f212de55f 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -188,7 +188,7 @@ The SDK never logs plaintext data. | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/errors` | `EncryptionErrorTypes`, `StackError`, error subtypes, `getErrorMessage` | | `@cipherstash/stack/types` | All TypeScript types | -| `@cipherstash/stack-drizzle/v3` | Drizzle ORM integration for EQL v3 schemas (see the `stash-drizzle` skill) | +| `@cipherstash/stack-drizzle` | Drizzle ORM integration for EQL v3 schemas — the package root, EQL v3 only (see the `stash-drizzle` skill) | | `@cipherstash/stack-supabase` | `encryptedSupabaseV3` wrapper for Supabase (see the `stash-supabase` skill) | | `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. | | `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items. See the `stash-dynamodb` skill | @@ -1005,7 +1005,7 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a | Target | Package / entry point | Skill | |---|---|---| -| Drizzle ORM | `@cipherstash/stack-drizzle/v3` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | +| Drizzle ORM | `@cipherstash/stack-drizzle` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | | Supabase | `encryptedSupabaseV3` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | | Prisma | `@cipherstash/prisma-next` — searchable field-level encryption for Postgres | — | | DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — encrypt is **EQL v3 only**; decrypt still reads existing v2 items | `stash-dynamodb` | @@ -1054,6 +1054,6 @@ const users = encryptedTable("users", { const client = await Encryption({ schemas: [users] }) ``` -The scalar v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Drizzle/Supabase integrations (`@cipherstash/stack-drizzle`, `encryptedSupabase`), and `stash eql install --eql-version 2` — still exists for existing deployments. Legacy v2 `searchableJson()` is the exception: protect-ffi 0.30 cannot emit the removed selector envelope, so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) +The scalar v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Supabase integration (`encryptedSupabase`), and `stash eql install --eql-version 2` — still exists for existing deployments, with two carve-outs. **Drizzle has no v2 surface at all**: `@cipherstash/stack-drizzle` dropped `encryptedType` and the `like`/`ilike` operators, so a v2 Drizzle column is read-only — decrypt it through `@cipherstash/stack` rather than through the adapter. **Legacy v2 `searchableJson()`** cannot be emitted by protect-ffi 0.30 (the selector envelope was removed), so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) > **DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) now **encrypts EQL v3 only** — author tables with `types.*` from `@cipherstash/stack/eql/v3`. Its decrypt methods still accept a v2 table so previously stored v2 items remain readable. See the `stash-dynamodb` skill. diff --git a/vitest.shared.ts b/vitest.shared.ts index cbf78fd5b..56fee9202 100644 --- a/vitest.shared.ts +++ b/vitest.shared.ts @@ -78,11 +78,8 @@ export const sharedAlias: Record = { 'packages/stack-supabase/src/index.ts', ), // The Drizzle adapter package (was `@cipherstash/stack/drizzle` + - // `@cipherstash/stack/eql/v3/drizzle`). `/v3` first — longest prefix wins. - '@cipherstash/stack-drizzle/v3': resolve( - repoRoot, - 'packages/stack-drizzle/src/v3/index.ts', - ), + // `@cipherstash/stack/eql/v3/drizzle`). Root only: the `./v3` subpath + // collapsed into it, so there is no longer a longer prefix to order first. '@cipherstash/stack-drizzle': resolve( repoRoot, 'packages/stack-drizzle/src/index.ts', From 16c260f61cf4dd94698f6aab8946718c68b4af9d Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 14:59:31 +1000 Subject: [PATCH 017/123] docs(repo): repoint dead package paths + guard against recurrence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses PR #760 review feedback. - `docs/query-api-walkthrough.md` pointed at `packages/protect/src/ffi/*`, deleted by this PR. Replaced with a note rooting the doc's relative paths, and corrected the two other stale facts in the same block: the protect-ffi pin (0.24.0/0.23.0 → the actual 0.30.0) and row 1a's query-builder paths, which moved to stack-drizzle/stack-supabase in the #627 split. - `.github/dependabot.yml`'s ignore-rule comment narrated the #673 incident in the present tense against a package that no longer exists. Kept the lesson, marked the package as removed, and stated why the rule still holds for the surviving consumers. - `.github/workflows/tests.yml` looped over `packages/stack-forge`, which has never existed. The `[ -f ]` guard made it a no-op; dropped it. `scripts/lint-no-dead-package-paths.mjs` fails CI on any `packages/` reference that doesn't resolve to a directory, across docs, .github, skills and the root meta files. Design archives (docs/plans, docs/superpowers) and CHANGELOGs are exempt — they record history, not the current tree. It catches all three of the above; self-tests follow the existing scripts/__tests__ pattern. --- .github/dependabot.yml | 8 +- .github/workflows/tests.yml | 7 +- docs/query-api-walkthrough.md | 6 +- package.json | 1 + .../lint-no-dead-package-paths/dead-ref.md | 4 + .../lint-no-dead-package-paths/dead-ref.yml | 3 + .../dir-with-changelog/CHANGELOG.md | 3 + .../lint-no-dead-package-paths/dir/nested.md | 1 + .../lint-no-dead-package-paths/globs.md | 2 + .../lint-no-dead-package-paths/live-refs.md | 5 + .../many-dead-refs.md | 4 + .../lint-no-dead-package-paths/prefix.md | 3 + .../lint-no-dead-package-paths.test.mjs | 88 ++++++++++++++ scripts/lint-no-dead-package-paths.mjs | 107 ++++++++++++++++++ 14 files changed, 235 insertions(+), 7 deletions(-) create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.yml create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/dir-with-changelog/CHANGELOG.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/dir/nested.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/globs.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/live-refs.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/many-dead-refs.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/prefix.md create mode 100644 scripts/__tests__/lint-no-dead-package-paths.test.mjs create mode 100644 scripts/lint-no-dead-package-paths.mjs diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 5e475458e..3bcb587ca 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -48,9 +48,11 @@ updates: - dependency-name: "@cipherstash/auth-*" # Release-managed manually alongside stack releases. Grouped bumps are # actively harmful here: Dependabot's "group consistency" once upgraded - # the sunsetting packages/protect off its 0.23.0 pin (0.24+ renames the - # exports it imports) while DOWNGRADING the stack packages from 0.29.0 - # (see #673). + # the (since-removed) `protect` package off its 0.23.0 pin — 0.24+ renamed + # the exports it imported — while DOWNGRADING the stack packages from + # 0.29.0 (see #673). The surviving consumers (stack, stack-drizzle, + # stack-supabase) pin one version between them and must stay in lockstep, + # so the rule stands. - dependency-name: "@cipherstash/protect-ffi" # 0.x bumps ship breaking type changes (e.g. 0.2 → 0.3 tightened the # FailureOption constraint). Review and apply manually. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 2386d320b..7f2e277f3 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -149,6 +149,11 @@ jobs: - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners + # Deleting or renaming a package leaves `packages/` references + # dangling in docs and CI config. Nothing else catches it (#760 review). + - name: Lint — no references to deleted package directories + run: pnpm run lint:package-paths + - name: Test — lint script self-tests run: pnpm run test:scripts @@ -361,7 +366,7 @@ jobs: - name: Run tests with Bun run: | - for dir in packages/stack packages/stack-forge; do + for dir in packages/stack; do if [ -f "$dir/vitest.config.ts" ] || [ -f "$dir/package.json" ]; then echo "--- Testing $dir ---" (cd "$dir" && bunx --bun vitest run) || true diff --git a/docs/query-api-walkthrough.md b/docs/query-api-walkthrough.md index 8559a2ed7..ac44297f9 100644 --- a/docs/query-api-walkthrough.md +++ b/docs/query-api-walkthrough.md @@ -37,7 +37,7 @@ flowchart TD | # | Layer | Entry point | Role | |---|-------|-------------|------| | 1 | Public API | `encryption/index.ts:259/270` `encryptQuery()` | Overloaded: single value → `EncryptQueryOperation`; `ScalarQueryTerm[]` → `BatchEncryptQueryOperation`. | -| 1a | Query builders | `drizzle/operators.ts:976`, `supabase/query-builder.ts:44` | `eq/gt/...` operators & deferred builders that batch-encrypt RHS values, then emit a WHERE clause. | +| 1a | Query builders | `stack-drizzle/src/operators.ts`, `stack-supabase/src/query-builder.ts` | `eq/gt/...` operators & deferred builders that batch-encrypt RHS values, then emit a WHERE clause. | | 2 | Operations | `operations/encrypt-query.ts:41`, `operations/batch-encrypt-query.ts:115` | `execute()`: validate → resolve index → call FFI. `*WithLockContext` resolves `LockContextInput` via `resolveLockContext` before the FFI call. | | 3 | EQL resolution | `helpers/infer-index-type.ts:89`, `types.ts:292` | `resolveIndexType` + `queryTypeToFfi`/`queryTypeToQueryOp` map public `QueryTypeName` → FFI `indexType`/`queryOp`. | | 4 | FFI JS wrapper | `protect-ffi/lib/index.cjs:155` | `encryptQuery`/`encryptQueryBulk` → `wrapAsync(native.*)`. | @@ -73,5 +73,5 @@ flowchart LR - **Client init:** `EncryptionClient.init()` (`encryption/index.ts:81`) calls FFI `newClient()` once; the returned `Client` handle is passed into every `encryptQuery` call. - **`cipherstashclient`** = the CipherStash Client **Rust SDK**, compiled via Neon into the platform `.node` binary inside `@cipherstash/protect-ffi`. It performs the actual crypto and talks to ZeroKMS. - **Result shape:** `EncryptedQueryResult` (`types.ts:175`); shaped by `formatEncryptedResult(..., returnType)` (`eql` vs raw). -- **Version:** `package.json` pins `@cipherstash/protect-ffi@0.24.0` (installed tree observed at `0.23.0` — confirm before relying on it). -- `packages/protect/src/ffi/*` mirrors this flow under the older `protect` package name. +- **Version:** `packages/stack/package.json` pins `@cipherstash/protect-ffi@0.30.0` (`stack-drizzle` and `stack-supabase` pin the same version — they must move in lockstep). +- **Paths:** rows 1–3 and the notes above are relative to `packages/stack/src/`; row 1a is rooted at `packages/`; rows 4–5 are inside the installed `@cipherstash/protect-ffi`. Line numbers drift — search the symbol, don't trust the offset. diff --git a/package.json b/package.json index c0e4cd9b2..13cfd258b 100644 --- a/package.json +++ b/package.json @@ -29,6 +29,7 @@ "clean": "rimraf --glob **/.next **/.turbo **/dist **/node_modules", "code:fix": "biome check --write", "code:check": "biome check", + "lint:package-paths": "node scripts/lint-no-dead-package-paths.mjs", "lint:runners": "node scripts/lint-no-hardcoded-runners.mjs", "lint:workflow-cache": "node scripts/lint-no-workflow-caching.mjs", "release": "pnpm run build && changeset publish", diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.md new file mode 100644 index 000000000..4863381c2 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.md @@ -0,0 +1,4 @@ +# Dead reference + +- `packages/protect/src/ffi/*` mirrors this flow under the older name. +- `packages/stack` is fine. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.yml b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.yml new file mode 100644 index 000000000..4975c8373 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-ref.yml @@ -0,0 +1,3 @@ +ignore: + # Dependabot once bumped packages/protect off its pin. + - dependency-name: "@cipherstash/protect-ffi" diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir-with-changelog/CHANGELOG.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir-with-changelog/CHANGELOG.md new file mode 100644 index 000000000..66eedb203 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir-with-changelog/CHANGELOG.md @@ -0,0 +1,3 @@ +## 1.0.0 + +- Removed `packages/protect` and `packages/schema`. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir/nested.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir/nested.md new file mode 100644 index 000000000..e84569e11 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dir/nested.md @@ -0,0 +1 @@ +See `packages/protect/src/index.ts`. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/globs.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/globs.md new file mode 100644 index 000000000..809fcbf07 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/globs.md @@ -0,0 +1,2 @@ +Build everything with `turbo build --filter './packages/*'`. +The workspace glob is `packages/*`. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/live-refs.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/live-refs.md new file mode 100644 index 000000000..f6ea4400f --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/live-refs.md @@ -0,0 +1,5 @@ +# Live references + +- `packages/stack/src/encryption/index.ts` is the client. +- `packages/cli` owns the `stash` binary; see `./packages/cli/AGENTS.md`. +- `packages/stack-drizzle` and `packages/stack-supabase` are the split adapters. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/many-dead-refs.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/many-dead-refs.md new file mode 100644 index 000000000..f3c075aad --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/many-dead-refs.md @@ -0,0 +1,4 @@ +- packages/protect +- packages/schema +- packages/protect-dynamodb +- packages/stack diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/prefix.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/prefix.md new file mode 100644 index 000000000..6c9aa347d --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/prefix.md @@ -0,0 +1,3 @@ +- `packages/stack-drizzle` exists. +- `packages/stack-forge` does not. +- `packages/stack` exists. diff --git a/scripts/__tests__/lint-no-dead-package-paths.test.mjs b/scripts/__tests__/lint-no-dead-package-paths.test.mjs new file mode 100644 index 000000000..bc9ea927f --- /dev/null +++ b/scripts/__tests__/lint-no-dead-package-paths.test.mjs @@ -0,0 +1,88 @@ +import { execFileSync } from 'node:child_process' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const SCRIPT = resolve( + fileURLToPath(import.meta.url), + '../../lint-no-dead-package-paths.mjs', +) + +function run(...targets) { + try { + execFileSync('node', [SCRIPT, ...targets], { encoding: 'utf8' }) + return { exitCode: 0, output: '' } + } catch (err) { + return { + exitCode: err.status, + output: String(err.stdout) + String(err.stderr), + } + } +} + +const fx = (name) => + resolve( + fileURLToPath(import.meta.url), + `../fixtures/lint-no-dead-package-paths/${name}`, + ) + +describe('lint-no-dead-package-paths', () => { + it('passes on the repo as it stands', () => { + const r = run() + expect(r.output).toBe('') + expect(r.exitCode).toBe(0) + }) + + it('passes when every `packages/` reference resolves', () => { + expect(run(fx('live-refs.md')).exitCode).toBe(0) + }) + + it('fails on a reference to a deleted package', () => { + const r = run(fx('dead-ref.md')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/protect/) + }) + + it('names the file and line of each offender', () => { + const r = run(fx('dead-ref.md')) + expect(r.output).toMatch(/dead-ref\.md:3/) + }) + + it('reports every offender, not just the first', () => { + const r = run(fx('many-dead-refs.md')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/protect\b/) + expect(r.output).toMatch(/packages\/schema\b/) + expect(r.output).toMatch(/packages\/protect-dynamodb\b/) + }) + + it('flags dead paths in YAML comments too', () => { + const r = run(fx('dead-ref.yml')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/protect/) + }) + + it('matches the longest package name, not a shorter prefix', () => { + // `packages/stack-drizzle` must not be read as `packages/stack` + suffix, + // and a dead `packages/stack-forge` must not be excused by live + // `packages/stack`. + const r = run(fx('prefix.md')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/stack-forge/) + expect(r.output).not.toMatch(/packages\/stack-drizzle/) + }) + + it('ignores `packages/*` globs and `./packages/*` filters', () => { + expect(run(fx('globs.md')).exitCode).toBe(0) + }) + + it('walks a directory target', () => { + const r = run(fx('dir')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/nested\.md/) + }) + + it('skips CHANGELOG.md — released history names deleted packages', () => { + expect(run(fx('dir-with-changelog')).exitCode).toBe(0) + }) +}) diff --git a/scripts/lint-no-dead-package-paths.mjs b/scripts/lint-no-dead-package-paths.mjs new file mode 100644 index 000000000..8ca468d27 --- /dev/null +++ b/scripts/lint-no-dead-package-paths.mjs @@ -0,0 +1,107 @@ +import { readdirSync, readFileSync, statSync } from 'node:fs' +import { join, relative, resolve } from 'node:path' + +const REPO_ROOT = resolve(import.meta.dirname, '..') + +// Default targets — the surfaces a reader or agent navigates by. Override with +// argv[2..] for tests / ad-hoc checks. +// +// Deliberately NOT scanned: +// - `docs/plans/`, `docs/superpowers/` — design archives. They narrate the +// state of the tree at the time they were written, including packages the +// plan itself proposed removing. Rewriting history there is wrong. +// - `CHANGELOG.md` (anywhere) — released entries name what they removed. +const TARGETS = process.argv.slice(2).length + ? process.argv.slice(2) + : [ + 'AGENTS.md', + 'CLAUDE.md', + 'README.md', + 'SECURITY.md', + 'CONTRIBUTE.md', + 'docs', + '.github', + 'skills', + 'e2e/README.md', + 'packages/cli/AGENTS.md', + ] + +const SKIP_DIRS = new Set([ + 'node_modules', + 'dist', + 'plans', + 'superpowers', + '.git', +]) +const SKIP_FILES = new Set(['CHANGELOG.md']) +const TEXT_EXT = /\.(md|ya?ml|json|mjs|ts|txt)$/ + +// `packages/` where `` is a real directory name. The character +// class excludes `*`, so workspace globs (`packages/*`, `./packages/*`) are +// left alone, and `+` is greedy so `packages/stack-forge` is never excused by +// the live `packages/stack`. +const PACKAGE_REF = /packages\/([a-z0-9][a-z0-9._-]*)/g + +const livePackages = new Set( + readdirSync(resolve(REPO_ROOT, 'packages'), { withFileTypes: true }) + .filter((e) => e.isDirectory()) + .map((e) => e.name), +) + +function* walk(abs) { + const stat = statSync(abs) + if (stat.isFile()) { + yield abs + return + } + for (const entry of readdirSync(abs, { withFileTypes: true })) { + if (entry.isDirectory()) { + if (SKIP_DIRS.has(entry.name)) continue + yield* walk(join(abs, entry.name)) + } else if (!SKIP_FILES.has(entry.name) && TEXT_EXT.test(entry.name)) { + yield join(abs, entry.name) + } + } +} + +const offenders = [] +for (const target of TARGETS) { + const abs = resolve(REPO_ROOT, target) + let exists = true + try { + statSync(abs) + } catch { + exists = false + } + // A default target that doesn't exist is not an error — the list covers + // optional files (CONTRIBUTE.md, per-package AGENTS.md). + if (!exists) continue + + for (const file of walk(abs)) { + const rel = relative(REPO_ROOT, file) + if (SKIP_FILES.has(rel.split('/').pop())) continue + const lines = readFileSync(file, 'utf8').split('\n') + lines.forEach((line, idx) => { + PACKAGE_REF.lastIndex = 0 + for (const m of line.matchAll(PACKAGE_REF)) { + if (livePackages.has(m[1])) continue + offenders.push(`${rel}:${idx + 1}: \`packages/${m[1]}\` does not exist`) + } + }) + } +} + +if (offenders.length > 0) { + console.error( + `Found ${offenders.length} reference(s) to a non-existent package directory:\n`, + ) + for (const o of offenders) console.error(` ${o}`) + console.error( + '\nA package was renamed or removed without updating the docs and config\n' + + 'that point at it. Repoint each reference at the surviving path, or drop\n' + + 'the line if it no longer describes anything. Design archives\n' + + '(docs/plans, docs/superpowers) and CHANGELOGs are exempt — they record\n' + + 'history, not the current tree.', + ) + process.exit(1) +} From b214b5fa80872a020db2493ec15fd4677daaf4a1 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 15:07:51 +1000 Subject: [PATCH 018/123] refactor(cli): drop the unreachable v2 branch left by the classifier change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `classifyEqlDomain` now returns `3 | null`, so `listEncryptedColumns` can never emit `version: 2` and the `c.version === 2` exemption in `explainUnresolved` is dead. Removing it is provably behaviour-preserving: a post-cutover v2 table (`` carrying the v2 domain) now reaches `explainUnresolved` with an EMPTY candidate list, which the `candidates.length === 0` guard above already falls through on. - Remove the branch; rewrite the doc comment and the stale v2 parentheticals in `drop.ts` / `cutover.ts` to say why the post-cutover state now arrives as "no EQL columns". - Rewrite the drop test that hand-built a `version: 2` candidate — a state resolution can no longer produce, so it only exercised the dead branch — to the state that actually occurs. - Add unit tests pinning `explainUnresolved`'s contract, including that a candidate sharing the plaintext column's name still fails closed (the removed branch's only behaviour, and wrong at v3, which has no cut-over rename). - Correct the backfill manifest comments: `null` now also means a legacy `eql_v2_encrypted` domain, so a v2 column backfilled from here on records no `eqlVersion` and reports no version in `encrypt status`. The live-domain fallback yields null for that case too. Existing manifests are unaffected. Noted in the changeset. - Drop the PR-1 plan doc that landed in this PR's diff. --- .../remove-eql-v2-migrate-classifier.md | 5 +- .../2026-07-22-eql-v2-removal-pr1-plan.md | 74 ------------------- .../encrypt/__tests__/encrypt-v3.test.ts | 12 ++- packages/cli/src/commands/encrypt/backfill.ts | 20 +++-- packages/cli/src/commands/encrypt/cutover.ts | 6 +- packages/cli/src/commands/encrypt/drop.ts | 5 +- .../encrypt/lib/__tests__/resolve-eql.test.ts | 59 +++++++++++++++ .../src/commands/encrypt/lib/resolve-eql.ts | 24 +++--- 8 files changed, 106 insertions(+), 99 deletions(-) delete mode 100644 docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md create mode 100644 packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts diff --git a/.changeset/remove-eql-v2-migrate-classifier.md b/.changeset/remove-eql-v2-migrate-classifier.md index 45d163efa..e6ae23fb6 100644 --- a/.changeset/remove-eql-v2-migrate-classifier.md +++ b/.changeset/remove-eql-v2-migrate-classifier.md @@ -9,7 +9,10 @@ domain — v3 is the sole generation this workspace authors and backfills, so a column's version is now determined solely from its self-describing `eql_v3_*` domain type. A legacy v2 column's version is carried by the manifest's recorded `eqlVersion` instead (the CLI's `encrypt status` / `status` renderers already -fall back to it), so status output is unchanged for v2 columns. +fall back to it), so status output is unchanged for v2 columns already recorded +in `.cipherstash/migrations.json`. A v2 column backfilled from here on records +no `eqlVersion` and so reports no version in `stash encrypt status` — the v2 +lifecycle itself (cut-over, then dropping `_plaintext`) is unaffected. This removes v2 *classification*, not the v2 read path: existing v2 ciphertext remains decryptable through `@cipherstash/stack`. `EqlVersion` keeps its `2` diff --git a/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md b/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md deleted file mode 100644 index 54e53c516..000000000 --- a/docs/plans/2026-07-22-eql-v2-removal-pr1-plan.md +++ /dev/null @@ -1,74 +0,0 @@ -# EQL v2 removal — PR 1 step-plan (delete published v2 packages) - -Executes PR 1 of `docs/plans/2026-07-22-eql-v2-final-removal-design.md`. -Branch: `feat/remove-eql-v2-pr1-delete-packages` (worktree). Scoped against `origin/main` = `6ce53817`. - -## Goal -Delete the closed v2-only dependency chain — `@cipherstash/protect-dynamodb` → -`@cipherstash/protect` → `@cipherstash/schema` — and remove every reference so the -build, lockfile, and changeset state stay consistent. Mergeable in isolation. - -## Verified scope (live-code survey, not just design line-counts) -Closed-chain claim CONFIRMED for code imports: nothing outside the three imports them, -and `@cipherstash/stack` depends only on `@cipherstash/protect-ffi` (a different, external -package), not on any of the three. - -Corrections vs. the design's PR-1 paragraph: -- `.changeset/config.json` — the three are NOT in the `fixed` group and `ignore` is `[]`. - No config edit required (design assumed a fixed-group removal). -- `pnpm-workspace.yaml` uses globs (`packages/*`), no explicit entries to remove. -- No `tsconfig` `references` anywhere — nothing to unpick. -- Extra build blockers the design folded under "root config": `e2e/package.json` dep edge - and root `package.json` `build:js` turbo filter. -- Extra changeset-state fixes: pending `schema-stevec-standard-pin.md` targets the doomed - `@cipherstash/schema`; `pre.json` pins all three in `initialVersions`. - -## Steps - -### 1. Delete the three package directories -- `rm -rf packages/protect-dynamodb packages/protect packages/schema` - -### 2. Fix build blockers (dangling references that break compile/CI) -- `e2e/package.json` — remove the `"@cipherstash/protect": "workspace:*"` dependency line. -- root `package.json` — `build:js`: drop `--filter './packages/protect'`, keep `./packages/nextjs`. - -### 3. Clean stale (non-breaking) references -- `scripts/lint-no-hardcoded-runners.mjs` — remove the `packages/protect/src/bin/runner.ts` - allowlist entry (verify the script doesn't assert the path exists — if it does, this is - actually a blocker). -- `packages/nextjs/package.json` — description references `@cipherstash/protect`; repoint to - `@cipherstash/stack`. -- `skills/stash-drizzle/SKILL.md:38` — inspect the `@cipherstash/protect` mention; fix only if - the deletion made it wrong (a historical note about the legacy protect-based package may stay). - -### 4. Changeset / RC-mode housekeeping -- Delete `.changeset/schema-stevec-standard-pin.md` (only target is the deleted `@cipherstash/schema`; - already consumed in a prior rc per `pre.json`). -- `.changeset/pre.json` — remove the three from `initialVersions`; remove `schema-stevec-standard-pin` - from the `changesets` array (keeps it consistent with the deleted file). -- Add deletion-notice changeset `.changeset/remove-eql-v2-packages.md`: - `'@cipherstash/stack': patch` (successor surface for all three; group already major via - `stack-1-0-0-rc`) and `'@cipherstash/nextjs': patch` (its `package.json` description changes - from `@cipherstash/protect` to `@cipherstash/stack` — a published-metadata edit), prose body - naming each removed package and its migration path - (`@cipherstash/protect` → `@cipherstash/stack`; `@cipherstash/schema` → `@cipherstash/stack/schema`; - `@cipherstash/protect-dynamodb` → `@cipherstash/stack/dynamodb` `encryptedDynamoDB`). - Follows the `remove-legacy-drizzle-package.md` precedent. - -### 5. Meta-file honesty (trim what described the removed packages) -- `SECURITY.md` — drop the three rows from the package list. -- `AGENTS.md` — Repository Layout entries (protect, schema, protect-dynamodb) + prose mentions; - keep the "maintained implementation is `packages/stack/src/dynamodb`" guidance. - -### 6. Regenerate lockfile -- `pnpm install` (updates `pnpm-lock.yaml` for removed packages + e2e edge). CI is - `--frozen-lockfile`, so the committed lockfile must match. - -## Verification (green gate before commit) -- `pnpm changeset status` — no changeset references a missing package. -- `pnpm run build` — whole-repo turbo build; proves no dangling import/reference. -- `pnpm run code:check` — biome, error-free. -- `git grep -nP "@cipherstash/protect(?!-ffi)|@cipherstash/schema|@cipherstash/protect-dynamodb"` — - only intentional survivors (e.g. migration-path prose). The `(?!-ffi)` lookahead (PCRE, hence - `-P`) excludes the unrelated `@cipherstash/protect-ffi`; a plain `\b` would not, since a word - boundary sits between `protect` and `-ffi`. diff --git a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts index da2389aff..45def1ef2 100644 --- a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts @@ -430,10 +430,14 @@ describe('encrypt drop — EQL version awareness', () => { // After cutover renamed the ciphertext onto `email`, no counterpart is // resolvable BY DESIGN. The fail-closed guard must recognize this state // rather than blocking the one drop the lifecycle actually wants. - lifecycleMock.mockResolvedValue({ - info: null, - candidates: [{ column: 'email', domain: 'eql_v2_encrypted', version: 2 }], - }) + // + // `candidates` is EMPTY, not `[{ column: 'email', version: 2 }]`: the + // classifier no longer recognises `eql_v2_encrypted`, so a post-cutover v2 + // column drops out of `listEncryptedColumns` entirely. The state reaches + // `explainUnresolved` as "no EQL columns at all", which is exactly the + // case it already falls through on. This pins that the v2 lifecycle still + // works through the narrower classifier. + lifecycleMock.mockResolvedValue({ info: null, candidates: [] }) migrateMocks.progress.mockResolvedValueOnce({ phase: 'cut-over' }) await dropCommand({ table: 'users', column: 'email' }) diff --git a/packages/cli/src/commands/encrypt/backfill.ts b/packages/cli/src/commands/encrypt/backfill.ts index 7f7edb006..1698d8825 100644 --- a/packages/cli/src/commands/encrypt/backfill.ts +++ b/packages/cli/src/commands/encrypt/backfill.ts @@ -141,8 +141,11 @@ export async function backfillCommand(options: BackfillCommandOptions) { // v2 or v3 changes the rest of the LIFECYCLE (v3 has no cut-over — the // ladder is backfill → switch-by-name → drop), so detect it up front, // record it in the manifest, and tell the user which path they're on. - // `null` means the target column doesn't exist or isn't an EQL domain — - // let the existing checks below produce their specific errors. + // `null` means the target column doesn't exist, isn't an EQL domain, or is + // a legacy `eql_v2_encrypted` column (no longer classified — v3 is the sole + // authored generation). Every one of those falls through to the v2 ladder, + // which is the correct default for the v2 case and lets the existing checks + // below produce their specific errors for the other two. const eqlVersion = await detectColumnEqlVersion( db, options.table, @@ -553,12 +556,19 @@ function buildManifestEntry( // convention only, never relied upon. encryptedColumn, // v2's ladder ends with the rename cut-over; v3 has none — its end - // state is the plaintext column dropped. + // state is the plaintext column dropped. An unclassified column (null, + // which now includes a legacy v2 domain) takes the v2 ladder. targetPhase: eqlVersion === 3 ? 'dropped' : 'cut-over', } if (pkColumn) entry.pkColumn = pkColumn - // Absent means UNKNOWN (detection couldn't see the column), not v2 — - // readers fall back to the domain type in the database. + // Absent means the classifier returned null: the column isn't there, isn't + // an EQL domain, or carries the legacy `eql_v2_encrypted` domain (which + // `classifyEqlDomain` no longer recognises — v3 is the sole authored + // generation). Readers fall back to the live domain type, which yields null + // for those same three cases, so `encrypt status` reports no version rather + // than guessing one. Manifests written before v2 classification was dropped + // still carry `eqlVersion: 2`, so existing v2 columns keep their version; + // only a v2 column backfilled from here on records none. if (eqlVersion) entry.eqlVersion = eqlVersion return entry } diff --git a/packages/cli/src/commands/encrypt/cutover.ts b/packages/cli/src/commands/encrypt/cutover.ts index 6584ef075..676c4e244 100644 --- a/packages/cli/src/commands/encrypt/cutover.ts +++ b/packages/cli/src/commands/encrypt/cutover.ts @@ -78,8 +78,10 @@ export async function cutoverCommand(options: CutoverCommandOptions) { // Fail closed on ambiguity: `info === null` with EQL columns present // means we can't tell WHICH lifecycle applies — running the v2 config // machine against (possibly) v3 columns would only produce a misleading - // downstream error. (No EQL columns at all, or the post-cutover v2 - // same-name state, still falls through to the v2 preconditions below.) + // downstream error. (A table with no EQL v3 columns still falls through to + // the v2 preconditions below — which now also covers the post-cutover v2 + // same-name state, since an `eql_v2_encrypted` column is no longer + // classified and so never appears as a candidate.) const unresolved = explainUnresolved( options.table, options.column, diff --git a/packages/cli/src/commands/encrypt/drop.ts b/packages/cli/src/commands/encrypt/drop.ts index 2fe440bb0..9beb6953a 100644 --- a/packages/cli/src/commands/encrypt/drop.ts +++ b/packages/cli/src/commands/encrypt/drop.ts @@ -84,8 +84,9 @@ export async function dropCommand(options: DropCommandOptions) { // the wrong ciphertext and generate an irreversible drop of the wrong // data. (The post-cutover v2 state — `` itself carries the v2 // domain, counterpart legitimately unresolvable — falls through to the - // v2 path; live truth from the DB wins over the manifest's cached - // version throughout.) + // v2 path: the classifier recognises `eql_v3_*` only, so that column is + // not a candidate and the table reads as having no EQL columns. Live + // truth from the DB wins over the manifest's cached version throughout.) const unresolved = explainUnresolved( options.table, options.column, diff --git a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts new file mode 100644 index 000000000..1e5bc6cae --- /dev/null +++ b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts @@ -0,0 +1,59 @@ +/** + * Pins {@link explainUnresolved}'s fail-closed contract now that the domain + * classifier recognises `eql_v3_*` only. + * + * `listEncryptedColumns` can no longer emit `version: 2` — a legacy + * `eql_v2_encrypted` column is not classified as an EQL column at all, so it + * never reaches this function as a candidate. The post-cutover v2 state (the + * ciphertext renamed onto the plaintext column's own name) therefore arrives + * here as an EMPTY candidate list, which the first guard already falls through + * on. These tests exist so removing the now-unreachable `version === 2` branch + * is provably behaviour-preserving, and so a future v2 sweep cannot delete the + * empty-list guard the v2 lifecycle actually depends on. + */ + +import type { EncryptedColumnInfo } from '@cipherstash/migrate' +import { describe, expect, it } from 'vitest' +import { explainUnresolved } from '../resolve-eql.js' + +const v3 = ( + column: string, + domain = 'eql_v3_text_eq', +): EncryptedColumnInfo => ({ + column, + domain, + version: 3, +}) + +describe('explainUnresolved', () => { + it('falls through (null) when the table has no EQL columns at all', () => { + // Both the not-yet-backfilled case and the post-cutover v2 same-name case + // land here: the caller's own preconditions produce the accurate error. + expect(explainUnresolved('users', 'email', [])).toBeNull() + }) + + it('fails closed, naming every candidate, when none is identifiable', () => { + const message = explainUnresolved('users', 'email', [ + v3('a_enc'), + v3('b_enc', 'eql_v3_text_search'), + ]) + + expect(message).toContain('Cannot identify which encrypted column') + expect(message).toContain('a_enc (eql_v3_text_eq)') + expect(message).toContain('b_enc (eql_v3_text_search)') + expect(message).toContain('--encrypted-column') + }) + + it('gives no free pass to a candidate sharing the plaintext column name', () => { + // The removed branch exempted a SAME-NAME candidate, but only at + // `version === 2`. A v3 domain on the plaintext column's own name is not + // the post-cutover state (v3 has no cut-over rename), so it must still + // fail closed rather than let a destructive command guess a lifecycle. + const message = explainUnresolved('users', 'email', [ + v3('email'), + v3('email_enc'), + ]) + + expect(message).toContain('Cannot identify which encrypted column') + }) +}) diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index 9ce9afb6a..f3cb0afc2 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -68,16 +68,21 @@ export async function resolveColumnLifecycle( /** * Explain a failed resolution (`info === null`) to the user, or return - * `null` when the failure is fine to fall through to the v2 lifecycle: + * `null` when the failure is fine to fall through to the v2 lifecycle. * - * - No EQL columns at all → the v2 phase/config preconditions produce the - * accurate error ("not backfilled", "no pending config", …). - * - The plaintext column ITSELF carries the v2 domain → the normal - * post-cutover v2 state (`` was renamed onto the ciphertext), where - * "no counterpart" is expected, not a problem. + * The one fall-through case is "no EQL columns at all", which the v2 + * phase/config preconditions turn into an accurate error ("not backfilled", + * "no pending config", …). Since `classifyEqlDomain` recognises `eql_v3_*` + * only, that case now also covers the post-cutover v2 state — `` was + * renamed onto the ciphertext, and its `eql_v2_encrypted` domain is no longer + * classified, so the column never appears as a candidate. (It used to arrive + * here as a `version: 2` candidate and needed its own exemption.) * - * Anything else means EQL columns exist but none is identifiable — the - * caller must fail closed with this message rather than guess a lifecycle. + * A non-empty candidate list therefore means EQL v3 columns exist but none is + * identifiable — the caller must fail closed with this message rather than + * guess a lifecycle, including when one candidate happens to share the + * plaintext column's name (v3 has no cut-over rename, so that is not the + * post-cutover state). */ export function explainUnresolved( table: string, @@ -85,9 +90,6 @@ export function explainUnresolved( candidates: readonly EncryptedColumnInfo[], ): string | null { if (candidates.length === 0) return null - if (candidates.some((c) => c.column === column && c.version === 2)) { - return null - } const listed = candidates .map((c) => ` - ${c.column} (${c.domain})`) .join('\n') From 87a9b005906f5370838ab519249d14717f9ab31b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 11:23:20 +1000 Subject: [PATCH 019/123] chore(stack): address review nits - MappedDecryptOperation.execute returns a fresh Result on the unknown-table path, so ops can't alias/mutate a shared failure object. - Document the AnyV3Table cast in Encryption with a biome-ignore explaining the runtime isV3Only guard the compiler can't see. - Restore CS_WORKSPACE_CRN in decrypt-audit-forwarding.test.ts afterEach so it doesn't leak env state across suites. Verified the v2-read acceptance tests (integration/shared/v2-decrypt-compat) match the CI CS_IT_SUITE glob, so #1a/#1b run under test:integration. --- .../__tests__/decrypt-audit-forwarding.test.ts | 14 +++++++++++++- packages/stack/src/encryption/index.ts | 10 +++++++--- .../src/encryption/operations/mapped-decrypt.ts | 3 ++- 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts index 9f3c09288..bc7b7d236 100644 --- a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts +++ b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts @@ -12,7 +12,7 @@ * * Credential-free: protect-ffi is mocked, so there is no ZeroKMS round-trip. */ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { Encryption } from '@/index' // A protect-ffi-shaped encrypted payload so the SDK's `isEncryptedPayload` @@ -67,13 +67,25 @@ const lastCiphertextLockContext = () => lastDecryptOpts().ciphertexts[0]?.lockContext let client: Awaited>> +let prevWorkspaceCrn: string | undefined beforeEach(async () => { vi.clearAllMocks() + prevWorkspaceCrn = process.env.CS_WORKSPACE_CRN process.env.CS_WORKSPACE_CRN = 'crn:ap-southeast-2.aws:test-workspace' client = await Encryption({ schemas: [users] }) }) +afterEach(() => { + // Restore the prior value so this suite doesn't leak env state into other + // Vitest suites sharing the worker. + if (prevWorkspaceCrn === undefined) { + delete process.env.CS_WORKSPACE_CRN + } else { + process.env.CS_WORKSPACE_CRN = prevWorkspaceCrn + } +}) + describe('typed v3 client: audit metadata forwards through decryptModel', () => { it('forwards .audit({ metadata }) as unverifiedContext (no lock context)', async () => { unwrap( diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index 968bea2e1..45ffe5a9a 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -937,7 +937,11 @@ export async function Encryption( // `eqlVersion: 2` over v3 schemas gets the nominal client for that deliberate, // low-level v2-wire migration path (the typed client cannot encrypt v3 columns // in v2 mode anyway). - return isV3Only && eqlVersion === 3 - ? typedClient(result.data, ...(schemas as unknown as readonly AnyV3Table[])) - : result.data + if (isV3Only && eqlVersion === 3) { + // biome-ignore lint/plugin: the runtime `isV3Only` guard (every schema has + // buildColumnKeyMap) proves these are AnyV3Table — the compiler can't see it. + const v3Schemas = schemas as unknown as readonly AnyV3Table[] + return typedClient(result.data, ...v3Schemas) + } + return result.data } diff --git a/packages/stack/src/encryption/operations/mapped-decrypt.ts b/packages/stack/src/encryption/operations/mapped-decrypt.ts index 378b30ab4..8aa0fddd3 100644 --- a/packages/stack/src/encryption/operations/mapped-decrypt.ts +++ b/packages/stack/src/encryption/operations/mapped-decrypt.ts @@ -81,7 +81,8 @@ export class MappedDecryptOperation extends EncryptionOperation { override async execute(): Promise> { if (!this.map) { - return this.unknownTableFailure + // Fresh Result so no two ops can alias (or mutate) a shared failure object. + return { failure: { ...this.unknownTableFailure.failure } } } const result = await this.underlying.execute() if (result.failure) { From 90a1555ec0ff638fb1934a7814443837deab5e8f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 15:08:58 +1000 Subject: [PATCH 020/123] =?UTF-8?q?fix(stack):=20address=20PR=20768=20revi?= =?UTF-8?q?ew=20=E2=80=94=20stale=20decrypt-audit=20guidance,=20silent=20l?= =?UTF-8?q?ock-context=20re-bind?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - resolveDecryptResult: the inner comment and `logger.debug` message still said the typed client has no decrypt audit surface and told the reader to use `Encryption({ config: { eqlVersion: 3 } })`. Both shipped clients now carry `.audit()` on decrypt, so the branch only fires for a non-conforming custom client. Message rewritten to describe that, with a regression test asserting it names neither `eqlVersion` nor `EncryptionV3`. The same false claim had propagated to the resolve-decrypt test header, `dynamodb/types.ts`, a test literally named "though decrypt cannot carry it", and a comment in `stack-supabase` — all corrected. - MappedDecryptOperation.withLockContext: chaining a second lock context onto an already-bound op silently dropped it. The wrapper always exposes the method, unlike the nominal path where it is absent after binding, so the re-bind type-checks. It now throws. Verified no internal or sibling call site chains after a positional bind (stack-supabase calls `decryptModel(row)` one-arg then chains, so its underlying op is unbound). Tests cover both `decryptModel` and `bulkDecryptModels`. - Changeset bumped minor -> major. Making `Encryption({ schemas: [] })` return the typed client changes its return type AND adds `Date` reconstruction on the two-arg `decryptModel` for existing plain-`Encryption` v3 callers. The package already carried a major, so the released bump is unchanged — the changelog is just accurate now. - skills/stash-encryption documented `decryptModel`/`bulkDecryptModels` as returning `Promise>` and said only encrypt-side ops are chainable. It ships in the `stash` tarball, so that was wrong guidance in customer repos. Updated, including the one-or-the-other lock-context rule. --- .changeset/stack-audit-on-decrypt.md | 18 +++++++--- packages/stack-supabase/src/index.ts | 10 +++--- .../decrypt-audit-forwarding.test.ts | 22 ++++++++++++ .../dynamodb/encrypted-dynamodb-v3.test.ts | 10 +++--- .../dynamodb/resolve-decrypt.test.ts | 35 +++++++++++++++---- packages/stack/src/dynamodb/helpers.ts | 10 +++--- packages/stack/src/dynamodb/types.ts | 8 +++-- .../encryption/operations/mapped-decrypt.ts | 30 ++++++++++++---- skills/stash-encryption/SKILL.md | 6 ++-- 9 files changed, 112 insertions(+), 37 deletions(-) diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md index d832b793d..dd21fd17c 100644 --- a/.changeset/stack-audit-on-decrypt.md +++ b/.changeset/stack-audit-on-decrypt.md @@ -1,5 +1,5 @@ --- -'@cipherstash/stack': minor +'@cipherstash/stack': major --- The typed EQL v3 client's `decryptModel` / `bulkDecryptModels` are now @@ -20,9 +20,19 @@ unchanged — the operation is still thenable to the same `Result`. This restore audited decrypt through the DynamoDB adapter (`encryptedDynamoDB(...).decryptModel`) for a v3 client, which previously had nowhere to carry the metadata. -`EncryptionV3` is now a deprecated, type-identical alias of `Encryption`: -`Encryption` is overloaded so an array of concrete EQL v3 tables yields the same -strongly-typed client. Use `Encryption` for new code. As part of this collapse +Chaining `.withLockContext()` onto a decrypt operation that already took a lock +context positionally (`decryptModel(item, table, lc).withLockContext(other)`) now +throws instead of silently keeping the first. Pass the lock context one way or +the other, not both. + +**Breaking:** `EncryptionV3` is now a deprecated, type-identical alias of +`Encryption`: `Encryption` is overloaded so an array of concrete EQL v3 tables +yields the same strongly-typed client. Use `Encryption` for new code. If you were +already passing EQL v3 tables to plain `Encryption`, you now receive the typed +client rather than the nominal one — its `decryptModel` / `bulkDecryptModels` +return type changes, and the two-argument form reconstructs `Date` columns from +`cast_as` instead of leaving them as ISO strings. Code that read those columns as +strings needs updating. As part of this collapse `EncryptionV3` no longer independently pins the wire format — like `Encryption`, it now honours an explicit `config.eqlVersion` (the retained migration escape hatch). The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index 4befe9726..4dfec3030 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -176,11 +176,11 @@ export async function encryptedSupabase( } // 5. Build the raw (eqlVersion 3) encryption client from the merged tables. - // NB: `Encryption`, not `EncryptionV3` — the query builder consumes the raw - // chainable `EncryptionClient`, whereas `EncryptionV3` returns the typed - // wrapper whose `decryptModel` returns a plain Promise. Pass only - // tables that carry at least one encrypted column (`Encryption` requires a - // non-empty schema list). + // NB: the query builder consumes the raw chainable `EncryptionClient`, and + // calls `decryptModel(row)` with no table — the typed client degrades to + // nominal (passthrough) behaviour for that arity, so either shape works. + // Pass only tables that carry at least one encrypted column (`Encryption` + // requires a non-empty schema list). const encryptionSchemas = [...synth.tables.values()].filter( (t) => Object.keys(t.columnBuilders).length > 0, ) diff --git a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts index bc7b7d236..3953d0a8b 100644 --- a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts +++ b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts @@ -139,6 +139,28 @@ describe('typed v3 client: audit metadata forwards through decryptModel', () => expect(lastCiphertextLockContext()).toEqual(IDENTITY_CLAIM) }) + it('throws when a second lock context is chained onto an already-bound op', () => { + // The wrapper always exposes `withLockContext`, so a positional bind + // followed by a chained one type-checks. Silently keeping the first would + // drop the caller's intent (and fail later at ZeroKMS with an opaque + // rejection); reject the re-bind at the call site instead. + const op = client.decryptModel({ email: enc() }, users, IDENTITY_CLAIM) + + expect(() => op.withLockContext({ identityClaim: ['other'] })).toThrow( + /already bound to a lock context/i, + ) + }) + + it('throws on a re-bind for bulkDecryptModels too', () => { + const op = client.bulkDecryptModels([{ email: enc() }], users, { + identityClaim: ['sub'], + }) + + expect(() => op.withLockContext(IDENTITY_CLAIM)).toThrow( + /already bound to a lock context/i, + ) + }) + it('forwards .audit({ metadata }) on bulkDecryptModels', async () => { unwrap( await client diff --git a/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts b/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts index c32b41f10..4f1f42096 100644 --- a/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts +++ b/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts @@ -624,7 +624,7 @@ describe('audit metadata with a v3 table', liveSuiteOptions, () => { expect(decrypted.data).toEqual(item) }) - it('is accepted on the typed client, though decrypt cannot carry it', async () => { + it('is carried on every operation of the typed client too', async () => { const item: User = { pk: 'user#15', email: 'ken@example.com' } const encrypted = await typedDynamo @@ -632,9 +632,11 @@ describe('audit metadata with a v3 table', liveSuiteOptions, () => { .audit({ metadata }) if (encrypted.failure) throw new Error(encrypted.failure.message) - // The typed client's `decryptModel` returns a plain promise with no audit - // surface. The chain must still resolve correctly — the metadata is simply - // not forwarded. Use the nominal client if decrypt audit matters. + // The typed client's `decryptModel` now returns a `MappedDecryptOperation`, + // so the metadata forwards to ZeroKMS here as it does on the nominal client. + // That forwarding is proven credential-free in + // `__tests__/decrypt-audit-forwarding.test.ts`; this live test asserts the + // round-trip still decrypts with the chain attached. const decrypted = await typedDynamo .decryptModel(encrypted.data, users) .audit({ metadata }) diff --git a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts index 8d5eb2b1b..92044eb78 100644 --- a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts +++ b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts @@ -1,9 +1,11 @@ /** - * Pure unit tests for the two client-shape helpers that bridge the nominal - * `EncryptionClient` (chainable, carries `.audit()`) and the typed client from - * `EncryptionV3` (plain `Promise`). Both branches were previously only - * reachable through a live ZeroKMS decrypt; these move that assurance onto the - * pure CI lane. No credentials, no network. + * Pure unit tests for the two client-shape helpers behind the DynamoDB adapter's + * decrypt path. Both shipped clients — nominal `EncryptionClient` and the typed + * EQL v3 client (whose decrypt returns a `MappedDecryptOperation`) — are + * chainable and carry `.audit()`; the bare-promise branch remains only for a + * non-conforming custom client. Every branch was previously reachable only + * through a live ZeroKMS decrypt; these move that assurance onto the pure CI + * lane. No credentials, no network. */ import type { Result } from '@byteslice/result' import { afterEach, describe, expect, it, vi } from 'vitest' @@ -22,7 +24,7 @@ afterEach(() => { }) describe('resolveDecryptResult', () => { - it('awaits a plain promise when the operation has no .audit (typed client)', async () => { + it('awaits a plain promise when the operation has no .audit (custom client)', async () => { const result = await resolveDecryptResult( Promise.resolve({ data: { x: 1 } }), { metadata: { ignored: true } }, @@ -85,6 +87,27 @@ describe('resolveDecryptResult', () => { } }) + it('does not blame the typed client in the dropped-metadata message', async () => { + // Both shipped clients now carry `.audit()` on decrypt, so this branch is + // reachable only from a non-conforming custom client. The message used to + // tell the reader to switch to `Encryption({ config: { eqlVersion: 3 } })` + // for audited decrypts, which is no longer true of any shipped client. + const spy = vi.spyOn(logger, 'debug').mockImplementation(() => {}) + + try { + await resolveDecryptResult(Promise.resolve({ data: { x: 1 } }), { + metadata: { m: 42 }, + }) + + const message = spy.mock.calls.at(-1)?.[0] as string + expect(message).not.toMatch(/eqlVersion/) + expect(message).not.toMatch(/EncryptionV3/) + expect(message).not.toMatch(/typed client/) + } finally { + spy.mockRestore() + } + }) + it('does not log about audit metadata when none is passed', async () => { const spy = vi.spyOn(logger, 'debug').mockImplementation(() => {}) diff --git a/packages/stack/src/dynamodb/helpers.ts b/packages/stack/src/dynamodb/helpers.ts index 67be28a68..38c62d099 100644 --- a/packages/stack/src/dynamodb/helpers.ts +++ b/packages/stack/src/dynamodb/helpers.ts @@ -103,12 +103,12 @@ export async function resolveDecryptResult( } if (typeof chainable?.audit !== 'function' && auditData.metadata) { - // The typed EncryptionV3 client returns a plain promise with no decrypt - // audit surface, so the metadata has nowhere to go. Make the drop - // observable rather than silent — use the nominal client for audited - // decrypts. + // Every client this package ships carries `.audit()` on decrypt, so this + // only fires for a custom client whose decrypt returns something else — + // there is then nowhere to put the metadata. Make the drop observable + // rather than silent. logger.debug( - 'DynamoDB: decrypt audit metadata ignored — the typed client has no decrypt audit surface; use Encryption({ config: { eqlVersion: 3 } }) for audited decrypts.', + "DynamoDB: decrypt audit metadata ignored — this client's decrypt does not return a chainable operation with .audit(). Audited decrypts need a client built with Encryption({ schemas }).", ) } diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index 9dfebb8cb..bee2e952d 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -69,9 +69,11 @@ type ChainableEncryptOperation = { * satisfy. The operation classes therefore cast to this shape at the call site * — the same split the Drizzle v3 operators use. * - * `decryptModel` is intentionally untyped in its return: the nominal client - * returns a chainable operation, the typed client a plain promise. See - * `resolveDecryptResult`. + * `decryptModel` is intentionally untyped in its return: both shipped clients + * return a chainable operation, but different classes of one (the nominal + * client's `DecryptModelOperation`, the typed client's `MappedDecryptOperation`), + * and a custom client may return something else entirely. See + * `resolveDecryptResult`, which normalises all three. */ export type CallableEncryptionClient = { encryptModel( diff --git a/packages/stack/src/encryption/operations/mapped-decrypt.ts b/packages/stack/src/encryption/operations/mapped-decrypt.ts index 8aa0fddd3..e3ee40b50 100644 --- a/packages/stack/src/encryption/operations/mapped-decrypt.ts +++ b/packages/stack/src/encryption/operations/mapped-decrypt.ts @@ -22,6 +22,10 @@ type UnderlyingDecryptOperation = EncryptionOperation & { export interface AuditableDecryptModelOperation extends EncryptionOperation { audit(config: AuditConfig): this + /** + * @throws if the operation already took a lock context positionally — bind it + * once, either positionally or by chaining. + */ withLockContext( lockContext: LockContextInput, ): AuditableDecryptModelOperation @@ -45,7 +49,10 @@ export interface AuditableDecryptModelOperation * audit data forward, and `.withLockContext().audit()` propagates because the * wrapper forwards `.audit()` onto the now-lock-bound underlying op. * - `.withLockContext()` rebuilds the wrapper over `underlying.withLockContext(lc)`, - * preserving the same `map` and unknown-table failure. + * preserving the same `map` and unknown-table failure. It throws if the + * underlying op is already lock-bound (a positional lock context followed by a + * chained one), because the wrapper — unlike the nominal path — still exposes + * the method after binding, so the second call would otherwise be dropped. * - `execute()` never throws: an unknown table (no `map`) returns the precomputed * `failure` Result, and `map` is a precomputed reconstructor — pure, no * `build()` — so it cannot reject the Result contract. @@ -71,12 +78,21 @@ export class MappedDecryptOperation extends EncryptionOperation { withLockContext( lockContext: LockContextInput, ): MappedDecryptOperation { - // A lock-bound underlying op exposes no `withLockContext`; there is nothing - // to re-bind, so keep the current underlying op. - const bound = this.underlying.withLockContext - ? this.underlying.withLockContext(lockContext) - : this.underlying - return new MappedDecryptOperation(bound, this.map, this.unknownTableFailure) + // A lock-bound underlying op exposes no `withLockContext` — it has already + // consumed its context. Unlike the nominal path, where the method is simply + // absent after binding, the wrapper always exposes it, so a second bind + // type-checks. Silently keeping the first would drop the caller's intent and + // surface later as an opaque ZeroKMS rejection; reject it here instead. + if (!this.underlying.withLockContext) { + throw new Error( + '[encryption]: this decrypt operation is already bound to a lock context. Pass the lock context positionally OR chain .withLockContext() — not both.', + ) + } + return new MappedDecryptOperation( + this.underlying.withLockContext(lockContext), + this.map, + this.unknownTableFailure, + ) } override async execute(): Promise> { diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index f212de55f..749c38f22 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -1021,14 +1021,14 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a | `encryptQuery` | `(plaintext, { table, column, queryType?, returnType? })` — queryable columns only; `queryType` constrained to the column's capabilities | `EncryptQueryOperation` | | `encryptQuery` | `(terms: readonly ScalarQueryTerm[])` — batch form | `BatchEncryptQueryOperation` | | `encryptModel` | `(model, table)` — schema fields validated against inferred plaintext types | `EncryptModelOperation>` | -| `decryptModel` | `(model, table, lockContext?)` | `Promise, EncryptionError>>` | +| `decryptModel` | `(model, table, lockContext?)` | `AuditableDecryptModelOperation>` | | `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` | -| `bulkDecryptModels` | `(models, table, lockContext?)` | `Promise[], EncryptionError>>` | +| `bulkDecryptModels` | `(models, table, lockContext?)` | `AuditableDecryptModelOperation[]>` | | `bulkEncrypt` | `(plaintexts, { column, table })` — parity passthrough | `BulkEncryptOperation` | | `bulkDecrypt` | `(encryptedPayloads)` — parity passthrough | `BulkDecryptOperation` | | `getEncryptConfig` | `()` | The client's encrypt config | -The encrypt-side operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining; `decryptModel`/`bulkDecryptModels` return plain promises and take the lock context as an argument. +All of these operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining — including `decryptModel`/`bulkDecryptModels`, which also accept the lock context as a third argument. Use one or the other: chaining `.withLockContext()` onto a decrypt that already took a positional lock context throws. ### Schema Builders From 8ac4f6411ae6853a4a4d3c1a3ea459b825125c89 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 12:39:38 +1000 Subject: [PATCH 021/123] fix(stack-supabase)!: type single()/maybeSingle() as one row, not an array MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit review of the query-builder split. Six findings, all pre-dating the refactor; these are the two worth acting on. `single()`/`maybeSingle()` have always returned ONE object at runtime, but returned `Self`, so the builder kept advertising the array shape it was created with — `data` was typed `T[] | null` while holding a single row. Callers had to launder it, and the test suite documented the lie in a comment while casting through `unknown` to reach the row's fields. Both now return `EncryptedSingleQueryBuilder`, awaiting `EncryptedSupabaseResponse` (`data: T | null`) — which already covers the zero-row case for `maybeSingle()` and the error case for both, so no separate null modelling was needed. The impl class carries the awaited shape as a `TData` parameter so the promise cannot keep advertising `T[]` after the runtime has been switched to single-row mode; `returns()` preserves that shape. Filters and transforms are deliberately absent from the single-row builder, matching supabase-js: applying one after `single()` would change the query the single-row promise was made about. Also drops two unnecessary `as unknown as T[]` bridges in query-results.ts (`[] as T[]` and the bulk-decrypt map both compile directly). Not acted on, with reasons: - The missing `assertPostgrestCanQueryEncryptedOperator` in the not-filter branch is a false positive. The guard fires upstream in `assertTermQueryable` (`contains`/`matches` map to `freeTextSearch`), before encryption, and `supabase-v3-json.test.ts` already covers `.not(col,'contains')` and `.not(col,'matches')` under EQL 3.0.2. The suggested guard keys on `wasEncrypted`, which that path never reaches. - Routing plaintext `in` arrays through `formatInListOperand` would change behaviour: `.filter()` is the raw escape hatch and forwards verbatim, as supabase-js does. The encrypted path only intervenes because it must encrypt element-wise. - The `as never` on `bulkEncrypt` args and the `term.column` double assertion need `ScalarQueryTerm['column']` widened in `@cipherstash/stack` — a different package's public type. --- .changeset/supabase-single-row-typing.md | 32 +++++++++++++++++ .../__tests__/supabase-v3-builder.test.ts | 9 +++-- packages/stack-supabase/src/query-builder.ts | 36 +++++++++++++------ packages/stack-supabase/src/query-results.ts | 17 +++++---- packages/stack-supabase/src/types.ts | 32 +++++++++++++++-- 5 files changed, 103 insertions(+), 23 deletions(-) create mode 100644 .changeset/supabase-single-row-typing.md diff --git a/.changeset/supabase-single-row-typing.md b/.changeset/supabase-single-row-typing.md new file mode 100644 index 000000000..eff4d8e06 --- /dev/null +++ b/.changeset/supabase-single-row-typing.md @@ -0,0 +1,32 @@ +--- +'@cipherstash/stack-supabase': major +--- + +`single()` and `maybeSingle()` now type `data` as the ROW, not an array. + +Both have always returned one object at runtime, but the builder kept +advertising the array shape it was created with, so `data` was typed `T[] | null` +while holding a single row. Every caller had to launder it: + +```typescript +const { data } = await supabase.from('users').select('id, email').single() +// before: data is `User[] | null` — wrong; a cast was the only way through +const user = data as unknown as User +// after: data is `User | null` +data?.email +``` + +`single()`/`maybeSingle()` now return `EncryptedSingleQueryBuilder`, which +awaits to `EncryptedSupabaseResponse` (`data: T | null`). That covers the +zero-row case for `maybeSingle()` and the error case for both, so no separate +null modelling was needed. + +Filters and transforms are not chainable after `single()`/`maybeSingle()`, +matching supabase-js — applying one afterwards would change the query the +single-row promise was made about. `returns()` preserves the awaited shape, +so `.single().returns()` still awaits one row. + +**Migration:** delete the cast. Code that worked around the old typing with +`data as unknown as Row` (or read `data![0]`) should now use `data` directly; +the cast still compiles but is no longer needed, and `data![0]` becomes a type +error. diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index 60d26c8f8..ce9835a2f 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -949,11 +949,10 @@ describe('encryptedSupabaseV3 wire encoding', () => { expect(error).toBeNull() expect(supabase.callsFor('single')).toHaveLength(1) expect(Array.isArray(data)).toBe(false) - // `data` is declared `T[] | null` on the shared builder surface; single() - // narrows it to one row at runtime only. - const single = data as unknown as { email: string; createdAt: Date } - expect(single.email).toBe('a@b.com') - expect(single.createdAt).toBeInstanceOf(Date) + // `single()` narrows the awaited shape to ONE row at the type level too, + // so `data` is the row itself — no cast needed to reach its fields. + expect(data?.email).toBe('a@b.com') + expect(data?.createdAt).toBeInstanceOf(Date) }) it('maybeSingle() returns null for null result data without throwing', async () => { diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index 2b299965e..b2c9652ef 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -88,6 +88,11 @@ const warnedLikeDelegation = new Set() */ export class EncryptedQueryBuilderImpl< T extends Record = Record, + /** The shape this builder awaits to. `T[]` normally; narrowed to `T` by + * {@link single}/{@link maybeSingle}, which return ONE row. Carried as a + * parameter so the promise cannot keep advertising `T[]` after the runtime + * has been switched to single-row mode. */ + TData = T[], > { private tableName: string private table: AnyV3Table @@ -465,16 +470,19 @@ export class EncryptedQueryBuilderImpl< return this } - single(): this { + single(): EncryptedQueryBuilderImpl { this.resultMode = 'single' this.transforms.push({ kind: 'single' }) - return this + // Type-level narrowing only; builder state is preserved. `TData` appears in + // `then`/`execute` return positions, so the two instantiations are not + // mutually assignable and `this` cannot be re-typed without an assertion. + return this as unknown as EncryptedQueryBuilderImpl } - maybeSingle(): this { + maybeSingle(): EncryptedQueryBuilderImpl { this.resultMode = 'maybeSingle' this.transforms.push({ kind: 'maybeSingle' }) - return this + return this as unknown as EncryptedQueryBuilderImpl } csv(): this { @@ -493,9 +501,17 @@ export class EncryptedQueryBuilderImpl< return this } - returns>(): EncryptedQueryBuilderImpl { + /** Re-type the ROW. The awaited SHAPE is preserved: called after + * `single()`/`maybeSingle()` this still awaits one row, not `U[]`. */ + returns>(): EncryptedQueryBuilderImpl< + U, + TData extends readonly unknown[] ? U[] : U + > { // Type-level cast only; builder state is preserved - return this as unknown as EncryptedQueryBuilderImpl + return this as unknown as EncryptedQueryBuilderImpl< + U, + TData extends readonly unknown[] ? U[] : U + > } // --------------------------------------------------------------------------- @@ -516,10 +532,10 @@ export class EncryptedQueryBuilderImpl< // PromiseLike implementation (deferred execution) // --------------------------------------------------------------------------- - then, TResult2 = never>( + then, TResult2 = never>( onfulfilled?: | (( - value: EncryptedSupabaseResponse, + value: EncryptedSupabaseResponse, ) => TResult1 | PromiseLike) | null, onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null, @@ -531,7 +547,7 @@ export class EncryptedQueryBuilderImpl< // Core execution // --------------------------------------------------------------------------- - private async execute(): Promise> { + private async execute(): Promise> { try { logger.debug(`Supabase encrypted query on table "${this.tableName}".`) @@ -558,7 +574,7 @@ export class EncryptedQueryBuilderImpl< ) // 6. Decrypt results - return await decryptResults(result, { + return await decryptResults(result, { ...ctx, selectColumns: this.selectColumns, resultMode: this.resultMode, diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts index 8c1812bd8..5838e012d 100644 --- a/packages/stack-supabase/src/query-results.ts +++ b/packages/stack-supabase/src/query-results.ts @@ -80,10 +80,13 @@ function postprocessDecryptedRow( * than decrypted. To read v2 data, decrypt fetched rows with the core * `@cipherstash/stack` client, whose decrypt path is generation-agnostic. */ -export async function decryptResults>( +export async function decryptResults< + T extends Record, + TData = T[], +>( result: RawSupabaseResult, ctx: DecryptContext, -): Promise> { +): Promise> { // If there's an error from Supabase, pass it through if (result.error) { return { @@ -118,7 +121,7 @@ export async function decryptResults>( if (!hasSelect && !hasMutationWithReturning) { // No select means no data to decrypt (e.g., insert without .select()) return { - data: result.data as T[], + data: result.data as TData, error: null, count: result.count ?? null, status: result.status, @@ -155,10 +158,12 @@ export async function decryptResults>( } return { + // ONE row — `TData` is `T` on this path (`single`/`maybeSingle` narrow it), + // so no array wrapping and no cast pretending otherwise. data: postprocessDecryptedRow( decrypted.data as Record, ctx, - ) as unknown as T[], + ) as TData, error: null, count: result.count ?? null, status: result.status, @@ -170,7 +175,7 @@ export async function decryptResults>( const dataArray = result.data as Record[] if (dataArray.length === 0) { return { - data: [] as unknown as T[], + data: [] as TData, error: null, count: result.count ?? null, status: result.status, @@ -196,7 +201,7 @@ export async function decryptResults>( return { data: decrypted.data.map((row) => postprocessDecryptedRow(row as Record, ctx), - ) as unknown as T[], + ) as TData, error: null, count: result.count ?? null, status: result.status, diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index c74fc9e6b..dd4611273 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -461,6 +461,21 @@ export interface TypedEncryptedSupabaseInstance { // Response // --------------------------------------------------------------------------- +/** + * The builder returned by `single()`/`maybeSingle()`: awaits to a SINGLE row + * (`data: T | null`) instead of an array. + * + * Only the two post-hoc modifiers supabase-js also allows after `.single()` are + * carried over. Filters and transforms are deliberately absent — applying one + * after `single()` would change the query the single-row promise was made + * about. + */ +export interface EncryptedSingleQueryBuilder + extends PromiseLike> { + abortSignal(signal: AbortSignal): EncryptedSingleQueryBuilder + throwOnError(): EncryptedSingleQueryBuilder +} + export type EncryptedSupabaseResponse = { data: T | null error: EncryptedSupabaseError | null @@ -909,8 +924,21 @@ export interface EncryptedQueryBuilderCore< to: number, options?: { referencedTable?: string; foreignTable?: string }, ): Self - single(): Self - maybeSingle(): Self + /** + * Return ONE row rather than an array — so the awaited `data` is `T | null`, + * not `T[]`. Returns {@link EncryptedSingleQueryBuilder} rather than `Self` + * because that change of shape is the whole point of the call: typing it + * `Self` would keep promising `T[]` while the runtime hands back one object, + * forcing every caller through a cast (`data as unknown as Row`). + * + * Filters and transforms are not chainable afterwards, matching supabase-js — + * `single()` is applied last. + */ + single(): EncryptedSingleQueryBuilder + /** As {@link single}, but a zero-row result is `data: null` rather than an + * error. Same `T | null` awaited shape — `single()` reports the missing row + * through `error` instead. */ + maybeSingle(): EncryptedSingleQueryBuilder csv(): Self abortSignal(signal: AbortSignal): Self throwOnError(): Self From 42231e79d844609000ad97aecb2674ea23e97d81 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 17:44:14 +1000 Subject: [PATCH 022/123] test(cli): retarget the codegen drizzle assertions at the collapsed root MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI runs against the PR MERGE commit, so it saw a file this branch never had: `remove-v2` gained `utils-codegen.test.ts` (from the v3 domain-picker work) after this branch was cut, and it pins the generated `stash init` client to `extractEncryptionSchemaV3` / `@cipherstash/stack-drizzle/v3` — exactly the two names this PR removes. Textually the merge is clean; the conflict is semantic, which is why it only surfaced in CI. Points those assertions at the collapsed root and adds the matching negatives, so a regression to the `./v3` specifier fails here rather than in a scaffolded project. Narrowed `utils-codegen-drizzle.test.ts` to what the merged-in suite does not cover — `generatePlaceholderClient`, still untested — and updated it to the new `ColumnDef` shape (`domain`, not `dataType`/`searchOps`). --- .../__tests__/utils-codegen-drizzle.test.ts | 35 ++++++++++--------- .../init/__tests__/utils-codegen.test.ts | 11 ++++-- 2 files changed, 27 insertions(+), 19 deletions(-) diff --git a/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts b/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts index 50acc3682..d30246f14 100644 --- a/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts +++ b/packages/cli/src/commands/init/__tests__/utils-codegen-drizzle.test.ts @@ -8,21 +8,22 @@ import { // `stash init --drizzle` writes these strings into the USER's repo as real // source. Nothing type-checks a template literal, so a rename in // `@cipherstash/stack-drizzle` cannot break the build here — it breaks the -// scaffolded project instead, in someone else's checkout. These assertions are -// the only thing standing between a de-suffixed export and a broken `stash -// init`. +// scaffolded project instead, in someone else's checkout. // -// Pinned by this suite (all three removed when `./v3` collapsed into the root): -// - the `@cipherstash/stack-drizzle/v3` specifier -// - `extractEncryptionSchemaV3` -// - `createEncryptionOperatorsV3` +// `utils-codegen.test.ts` covers the generated client's happy path. This file +// covers the surface that had no test at all — `generatePlaceholderClient`, +// whose doc block is a copy-paste reference the handoff agent works from — and +// asserts, for BOTH strings, that nothing removed with EQL v2 survives: +// - the `@cipherstash/stack-drizzle/v3` specifier (`./v3` collapsed into `.`) +// - `extractEncryptionSchemaV3` / `createEncryptionOperatorsV3` +// - `encryptedType`, the v2 config-flag column factory const SCHEMAS: SchemaDef[] = [ { tableName: 'users', columns: [ - { name: 'email', dataType: 'string', searchOps: ['equality'] }, - { name: 'age', dataType: 'number', searchOps: ['orderAndRange'] }, + { name: 'email', domain: 'TextSearch' }, + { name: 'age', domain: 'IntegerOrd' }, ], }, ] @@ -36,9 +37,9 @@ const GENERATED_SOURCES: Array<[string, string]> = [ describe.each( GENERATED_SOURCES, )('drizzle scaffold — %s', (_name, generated) => { - it('imports the drizzle surface from the package ROOT, never the removed ./v3 subpath', () => { - expect(generated).not.toContain('@cipherstash/stack-drizzle/v3') + it('names the drizzle package root, never the removed ./v3 subpath', () => { expect(generated).toContain('@cipherstash/stack-drizzle') + expect(generated).not.toContain('@cipherstash/stack-drizzle/v3') }) it('uses the de-suffixed export names, never the removed *V3 aliases', () => { @@ -52,13 +53,13 @@ describe.each( }) }) -describe('generateClientFromSchemas (drizzle)', () => { - const generated = generateClientFromSchemas('drizzle', SCHEMAS) +describe('generatePlaceholderClient (drizzle)', () => { + const placeholder = generatePlaceholderClient('drizzle') - it('names extractEncryptionSchema in both the import and the call site', () => { - expect(generated).toContain( - "import { types, extractEncryptionSchema } from '@cipherstash/stack-drizzle'", + it('shows the harvest pattern with the collapsed root import', () => { + expect(placeholder).toContain( + "import { extractEncryptionSchema } from '@cipherstash/stack-drizzle'", ) - expect(generated).toContain('extractEncryptionSchema(usersTable)') + expect(placeholder).toContain('extractEncryptionSchema(users)') }) }) diff --git a/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts b/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts index 1221065d5..5aabc73d4 100644 --- a/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts +++ b/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts @@ -27,8 +27,15 @@ describe('generateClientFromSchemas', () => { const out = generateClientFromSchemas('drizzle', schemas) expect(out).toContain("email: types.TextSearch('email'),") expect(out).toContain("age: types.IntegerOrd('age'),") - expect(out).toContain('extractEncryptionSchemaV3') - expect(out).toContain("from '@cipherstash/stack-drizzle/v3'") + // The collapsed root: `@cipherstash/stack-drizzle` dropped its EQL v2 + // surface and folded `./v3` into `.`, de-suffixing the exports. The + // negatives matter as much as the positives — this string is written into + // the user's repo as real source, and nothing type-checks a template + // literal, so the removed names can only be caught here. + expect(out).toContain('extractEncryptionSchema(') + expect(out).toContain("from '@cipherstash/stack-drizzle'") + expect(out).not.toContain('extractEncryptionSchemaV3') + expect(out).not.toContain('@cipherstash/stack-drizzle/v3') }) it('carries no residual v2 capability vocabulary', () => { From 8cbe0073c5882b5f8c4440b8ce1b27c3f713c0ec Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 22 Jul 2026 21:57:02 +1000 Subject: [PATCH 023/123] feat(wizard): port migration rewriter to the EQL v3 domain family MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wizard now scaffolds EQL v3 columns, so `drizzle-kit generate` emits `ALTER COLUMN ... SET DATA TYPE eql_v3_` — which Postgres rejects (no cast from text/numeric to an EQL domain). The post-agent rewriter matched only the single `eql_v2_encrypted` type, so those v3 statements slipped through unrepaired and failed at migrate time. Port the rewriter to the whole `eql_v3_*` concrete-domain family alongside legacy `eql_v2_encrypted`, mirroring the sibling CLI fix (#693): every mangled form drizzle-kit emits (incl. the 0.31.0+ `"undefined".` prefix and schema-qualified pgSchema tables), near-miss flagging for `SET DATA TYPE ... USING ...` it cannot safely repair, statement-breakpoints, and a clearer data-destroying / empty-table-only warning that points populated tables at the staged `stash encrypt` flow. Database introspection (`isEqlEncrypted`) now recognises BOTH `eql_v2_encrypted` and the `eql_v3_*` family as already-encrypted, matching migrate's `classifyEqlDomain` v3 convention — so the agent won't scaffold over existing encrypted data of either generation (v2 ciphertext stays valid and detected). Add a rewrite-migrations test suite (adapted from the CLI's). --- .changeset/wizard-eql-v3-migration-rewrite.md | 27 ++ .../src/__tests__/rewrite-migrations.test.ts | 399 ++++++++++++++++++ packages/wizard/src/lib/post-agent.ts | 16 +- packages/wizard/src/lib/rewrite-migrations.ts | 251 ++++++++--- packages/wizard/src/tools/wizard-tools.ts | 21 +- 5 files changed, 661 insertions(+), 53 deletions(-) create mode 100644 .changeset/wizard-eql-v3-migration-rewrite.md create mode 100644 packages/wizard/src/__tests__/rewrite-migrations.test.ts diff --git a/.changeset/wizard-eql-v3-migration-rewrite.md b/.changeset/wizard-eql-v3-migration-rewrite.md new file mode 100644 index 000000000..4e9ae5861 --- /dev/null +++ b/.changeset/wizard-eql-v3-migration-rewrite.md @@ -0,0 +1,27 @@ +--- +'@cipherstash/wizard': minor +--- + +Teach the wizard's post-agent Drizzle step to repair EQL **v3** migrations, not +just legacy EQL v2. + +The wizard now scaffolds EQL v3 columns, so `drizzle-kit generate` emits +`ALTER TABLE … ALTER COLUMN … SET DATA TYPE eql_v3_` — which Postgres +rejects (there is no cast from `text`/`numeric` to an EQL domain). The migration +rewriter previously matched only the single `eql_v2_encrypted` type, so those v3 +statements slipped through unrepaired and failed at migrate time. + +The rewriter is ported to match the whole EQL v3 concrete-domain family +(`eql_v3_text_search`, `eql_v3_integer_ord`, …) alongside legacy +`eql_v2_encrypted`, across every mangled form drizzle-kit emits (including the +`"undefined".` prefix from 0.31.0+ and schema-qualified `pgSchema()` tables). It +now also flags near-miss `SET DATA TYPE … USING …` statements it cannot safely +repair instead of leaving broken SQL, and each rewritten file carries a clearer +warning that the ADD+DROP+RENAME is data-destroying and safe only on an empty +table — a populated table must use the staged `stash encrypt` flow. This +re-converges the rewriter with the sibling copy in the `stash` CLI. + +Database introspection also recognises v3 encrypted columns: `isEqlEncrypted` +now reports both `eql_v2_encrypted` and the `eql_v3_*` family as already +encrypted, so the agent won't scaffold over existing encrypted data of either +generation. diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts new file mode 100644 index 000000000..21ae01cd6 --- /dev/null +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -0,0 +1,399 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { rewriteEncryptedAlterColumns } from '../lib/rewrite-migrations.js' + +describe('rewriteEncryptedAlterColumns', () => { + let tmpDir: string + + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'wizard-rewrite-')) + }) + + afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }) + }) + + it('rewrites an in-place ALTER COLUMN with the bare v2 type name', async () => { + const original = `ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE eql_v2_encrypted;\n` + const filePath = path.join(tmpDir, '0002_alter.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "transactions" ADD COLUMN "amount__cipherstash_tmp" "public"."eql_v2_encrypted";', + ) + expect(updated).toContain( + 'ALTER TABLE "transactions" DROP COLUMN "amount";', + ) + expect(updated).toContain( + 'ALTER TABLE "transactions" RENAME COLUMN "amount__cipherstash_tmp" TO "amount";', + ) + expect(updated).not.toContain('SET DATA TYPE') + // Wizard-branded header. + expect(updated).toContain('-- Rewritten by @cipherstash/wizard') + }) + + it('rewrites a bare v3 domain (the generation the wizard now scaffolds)', async () => { + const original = `ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` + const filePath = path.join(tmpDir, '0002_v3.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('rewrites the schema-qualified form produced by drizzle-kit', async () => { + const original = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n' + const filePath = path.join(tmpDir, '0003_alter.sql') + fs.writeFileSync(filePath, original) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('rewrites a schema-qualified table produced by pgSchema()', async () => { + // drizzle-kit emits `"app"."users"` for a table declared in a pgSchema(); + // the old `\s+` between the table and ALTER COLUMN could never cross the `.`. + const original = + 'ALTER TABLE "app"."users" ALTER COLUMN "email" SET DATA TYPE "undefined"."eql_v3_text_search";\n' + const filePath = path.join(tmpDir, '0014_qualified.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + // Every emitted statement keeps the schema qualifier. + expect(updated).toContain( + 'ALTER TABLE "app"."users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).toContain('ALTER TABLE "app"."users" DROP COLUMN "email";') + expect(updated).toContain( + 'ALTER TABLE "app"."users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('rewrites the "undefined" schema form drizzle-kit emits for bare custom types', async () => { + const original = + 'ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE "undefined"."eql_v2_encrypted";\n' + const filePath = path.join(tmpDir, '0005_undef.sql') + fs.writeFileSync(filePath, original) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "transactions" ADD COLUMN "amount__cipherstash_tmp" "public"."eql_v2_encrypted";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('rewrites the double-quoted form produced by stack 0.15.0', async () => { + const original = + 'ALTER TABLE "transactions" ALTER COLUMN "description" SET DATA TYPE "undefined".""public"."eql_v2_encrypted"";\n' + const filePath = path.join(tmpDir, '0006_double.sql') + fs.writeFileSync(filePath, original) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "transactions" ADD COLUMN "description__cipherstash_tmp" "public"."eql_v2_encrypted";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('leaves unrelated migrations untouched', async () => { + const original = + 'CREATE TABLE "widgets" ("id" integer PRIMARY KEY, "name" text);\n' + const filePath = path.join(tmpDir, '0001_init.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('skips the file passed in options.skip', async () => { + const install = path.join(tmpDir, '0000_install-eql.sql') + const alter = path.join(tmpDir, '0002_alter.sql') + fs.writeFileSync(install, 'CREATE SCHEMA eql_v2;\n') + fs.writeFileSync( + alter, + 'ALTER TABLE "t" ALTER COLUMN "c" SET DATA TYPE eql_v2_encrypted;', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir, { + skip: install, + }) + expect(rewritten).toEqual([alter]) + expect(fs.readFileSync(install, 'utf-8')).toBe('CREATE SCHEMA eql_v2;\n') + }) + + it('returns an empty result when the directory does not exist', async () => { + const missing = path.join(tmpDir, 'does-not-exist') + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(missing) + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + }) + + // Every concrete `eql_v3_*` domain shipped by `@cipherstash/stack/eql/v3` + // (see `packages/stack/src/eql/v3/columns.ts`). Eight scalar bases carry the + // four storage/eq/ord flavours; text adds `_match`/`_search`; boolean and json + // stand alone. + const V3_SCALAR_BASES = [ + 'integer', + 'smallint', + 'bigint', + 'numeric', + 'real', + 'double', + 'date', + 'timestamp', + ] + const V3_DOMAINS = [ + ...V3_SCALAR_BASES.flatMap((base) => + ['', '_eq', '_ord', '_ord_ore'].map( + (flavour) => `eql_v3_${base}${flavour}`, + ), + ), + 'eql_v3_text', + 'eql_v3_text_eq', + 'eql_v3_text_match', + 'eql_v3_text_ord', + 'eql_v3_text_ord_ore', + 'eql_v3_text_search', + 'eql_v3_boolean', + 'eql_v3_json', + ] + + it.each(V3_DOMAINS)('rewrites an ALTER COLUMN to %s', async (domain) => { + const filePath = path.join(tmpDir, '0007_v3.sql') + fs.writeFileSync( + filePath, + `ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "undefined"."${domain}";\n`, + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + `ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."${domain}";`, + ) + expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') + expect(updated).toContain( + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + // DOMAIN_RE is derived from ENCRYPTED_DOMAIN, so drift between the two can't + // silently leave a domain unrewritten. Prove every domain the alternation + // recognises is actually extracted into the emitted ADD COLUMN. + it.each([ + ...V3_DOMAINS, + 'eql_v2_encrypted', + ])('extracts the bare domain %s from a mangled ALTER', async (domain) => { + const filePath = path.join(tmpDir, '0015_drift.sql') + fs.writeFileSync( + filePath, + `ALTER TABLE "t" ALTER COLUMN "c" SET DATA TYPE "undefined"."${domain}";\n`, + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + `ALTER TABLE "t" ADD COLUMN "c__cipherstash_tmp" "public"."${domain}";`, + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + // The mangled forms are the cross product of what `dataType()` returns and + // which drizzle-kit era renders it (see the file's comment table). + const MANGLED_FORMS: Array<[label: string, emitted: string]> = [ + ['plain, drizzle-kit <=0.30.6', 'eql_v3_text_search'], + [ + '"undefined"-prefixed, drizzle-kit >=0.31.0', + '"undefined"."eql_v3_text_search"', + ], + ['dotted, drizzle-kit <=0.30.6', 'public.eql_v3_text_search'], + [ + 'dotted inside "undefined", drizzle-kit >=0.31.0', + '"undefined"."public.eql_v3_text_search"', + ], + ['pre-quoted, drizzle-kit <=0.30.6', '"public"."eql_v3_text_search"'], + [ + 'pre-quoted inside "undefined", drizzle-kit >=0.31.0', + '"undefined".""public"."eql_v3_text_search""', + ], + ['bare-quoted (speculative)', '"eql_v3_text_search"'], + ] + + it.each(MANGLED_FORMS)('rewrites the v3 %s form', async (_label, emitted) => { + const filePath = path.join(tmpDir, '0008_form.sql') + fs.writeFileSync( + filePath, + `ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE ${emitted};\n`, + ) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).not.toContain('SET DATA TYPE') + }) + + it('names the target domain in the guidance comment', async () => { + const filePath = path.join(tmpDir, '0010_comment.sql') + fs.writeFileSync( + filePath, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "undefined"."eql_v3_integer_ord";\n', + ) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain('-- eql_v3_integer_ord.') + }) + + it('warns that the rewrite is data-destroying / empty-table-only', async () => { + const filePath = path.join(tmpDir, '0016_warn.sql') + fs.writeFileSync( + filePath, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain('safe ONLY if') + expect(updated).toContain('constraints, defaults, and indexes') + expect(updated).toContain('stash encrypt') + }) + + it('separates ADD/DROP/RENAME with --> statement-breakpoint', async () => { + const filePath = path.join(tmpDir, '0018_breakpoint.sql') + fs.writeFileSync( + filePath, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8').trimEnd() + const chunks = updated.split('--> statement-breakpoint') + expect(chunks).toHaveLength(3) + expect(chunks[0]).toContain('ADD COLUMN') + expect(chunks[1]).toContain('DROP COLUMN') + expect(chunks[2]).toContain('RENAME COLUMN') + }) + + it('rewrites each statement to its own domain when v2 and v3 are mixed', async () => { + const filePath = path.join(tmpDir, '0011_mixed.sql') + fs.writeFileSync( + filePath, + [ + 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v2_encrypted;', + 'ALTER TABLE "a" ALTER COLUMN "y" SET DATA TYPE "undefined"."eql_v3_json";', + ].join('\n'), + ) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "a" ADD COLUMN "x__cipherstash_tmp" "public"."eql_v2_encrypted";', + ) + expect(updated).toContain( + 'ALTER TABLE "a" ADD COLUMN "y__cipherstash_tmp" "public"."eql_v3_json";', + ) + }) + + it.each([ + ['a plaintext type', 'text'], + ['jsonb', 'jsonb'], + ['a lookalike from another EQL major', 'eql_v4_text_search'], + ['a lookalike prefix', 'not_eql_v3_text_search'], + ])('leaves an ALTER COLUMN to %s untouched', async (_label, type) => { + const original = `ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE ${type};\n` + const filePath = path.join(tmpDir, '0012_other.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('leaves a hand-authored SET DATA TYPE ... USING conversion untouched but flags it', async () => { + const original = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n' + const filePath = path.join(tmpDir, '0013_using.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].file).toBe(filePath) + expect(skipped[0].statement).toContain('SET DATA TYPE') + expect(skipped[0].statement).toContain('eql_v3_text_search') + // Left untouched on disk — we flag, we don't guess. + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('reports no skipped statements when the strict rewrite fully handled the file', async () => { + const filePath = path.join(tmpDir, '0021_handled.sql') + fs.writeFileSync( + filePath, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(skipped).toEqual([]) + }) + + it('handles multiple ALTER statements in one file', async () => { + const original = [ + 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v3_text_search;', + 'ALTER TABLE "a" ALTER COLUMN "y" SET DATA TYPE eql_v3_text_search;', + 'CREATE INDEX "a_z" ON "a" ("z");', + ].join('\n') + const filePath = path.join(tmpDir, '0004_multi.sql') + fs.writeFileSync(filePath, original) + + await rewriteEncryptedAlterColumns(tmpDir) + + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated.match(/ADD COLUMN/g)?.length).toBe(2) + expect(updated.match(/DROP COLUMN/g)?.length).toBe(2) + // Non-matching statement preserved + expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') + }) +}) diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index 34c18f2a7..24f8bb918 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -76,8 +76,10 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { cwd, ) - // Rewrite any `ALTER COLUMN ... SET DATA TYPE eql_v2_encrypted` that - // drizzle-kit just produced — those fail in Postgres. CIP-2991 + CIP-2994. + // Rewrite any `ALTER COLUMN ... SET DATA TYPE ` that + // drizzle-kit just produced — those fail in Postgres (no cast from + // text/numeric to an EQL domain). Covers the EQL v3 family the wizard now + // scaffolds, and legacy eql_v2_encrypted. CIP-2991 + CIP-2994 + #693. await rewriteEncryptedMigrations(cwd) const shouldMigrate = await p.confirm({ @@ -118,16 +120,22 @@ async function rewriteEncryptedMigrations(cwd: string): Promise { if (!existsSync(abs)) continue try { - const rewritten = await rewriteEncryptedAlterColumns(abs) + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(abs) if (rewritten.length > 0) { p.log.info( `Rewrote ${rewritten.length} migration file(s) in ${dir}/ to use ADD+DROP+RENAME for encrypted columns.`, ) for (const file of rewritten) p.log.step(` - ${file}`) p.log.warn( - 'If any of these tables already have rows, backfill the new column via @cipherstash/stack before running the migration in production. See the comments in the rewritten SQL.', + 'This rewrite is data-destroying — safe only on an EMPTY table. If any of these tables already have rows, do NOT run the migration; use the staged `stash encrypt` flow (add -> backfill via @cipherstash/stack -> cutover -> drop) instead. See the comments in the rewritten SQL.', ) } + if (skipped.length > 0) { + p.log.warn( + `${skipped.length} statement(s) look like an ALTER-to-encrypted the rewrite could not safely repair (e.g. a hand-authored SET DATA TYPE ... USING ...). Review them before migrating:`, + ) + for (const s of skipped) p.log.step(` - ${s.file}: ${s.statement}`) + } // Only rewrite the first dir that matches — running again on a // different candidate would double-transform already-rewritten SQL. return diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 24635f10e..bbe2a754f 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -2,59 +2,158 @@ import { readdir, readFile, writeFile } from 'node:fs/promises' import { join } from 'node:path' /** - * Matches drizzle-kit's generated in-place type change to the encrypted - * column type. drizzle-kit's ALTER COLUMN path wraps the customType - * `dataType()` return value in double-quotes and prepends `"{typeSchema}".`. - * Custom types have no `typeSchema`, so we see several mangled forms - * depending on what `dataType()` returned. We match all of them: + * The encrypted column types this rewrite applies to: the single EQL v2 type, + * and the whole EQL v3 concrete-domain family (`eql_v3_text_search`, + * `eql_v3_integer_ord`, …). The v3 side is matched by shape rather than by an + * enumerated list so a newly added domain is covered without touching this file + * — the `eql_v3_` prefix is unambiguous in a `SET DATA TYPE` position. * - * - bare `eql_v2_encrypted` → `"undefined"."eql_v2_encrypted"` - * - pre-quoted `"public"."eql_v2_encrypted"` (stack 0.15.0 regression) → - * `"undefined".""public"."eql_v2_encrypted""` - * - the plain `eql_v2_encrypted` and `"public"."eql_v2_encrypted"` forms, - * in case a future drizzle-kit release stops prepending undefined. + * v2 stays matched even though the wizard now scaffolds v3 exclusively: a user's + * migration directory can hold files generated against an older stack version, + * and existing v2 ciphertext stays valid — we still want their ALTER-to-encrypted + * statements repaired. + */ +const ENCRYPTED_DOMAIN = String.raw`eql_v2_encrypted|eql_v3_[a-z0-9_]+` + +/** + * Extracts the bare domain name out of whichever mangled form matched. Derived + * from {@link ENCRYPTED_DOMAIN} so the two can never drift out of sync — a new + * domain added to the alternation is extracted here for free. + */ +const DOMAIN_RE = new RegExp(ENCRYPTED_DOMAIN, 'i') + +/** + * The mangled forms drizzle-kit emits for a customType in an ALTER COLUMN. + * + * drizzle-kit's ALTER COLUMN path wraps the customType `dataType()` return + * value in double-quotes and prepends `"{typeSchema}".`. Custom types have no + * `typeSchema`, so the emitted SQL depends on BOTH what `dataType()` returned + * and which drizzle-kit renders it. Verified against 0.24.2, 0.28.1, 0.30.6, + * 0.31.0, 0.31.1 and 0.31.10 through `drizzle-kit/api`'s `generateMigration`: + * + * | `dataType()` returns | <= 0.30.6 | >= 0.31.0 | + * | ------------------------------- | --------------------------- | ---------------------------------- | + * | `eql_v3_text_search` (current) | `eql_v3_text_search` | `"undefined"."eql_v3_text_search"` | + * | `public.eql_v3_text_search` | `public.eql_v3_text_search` | `"undefined"."public.eql_v3_…"` | + * | `"public"."eql_v2_encrypted"` | `"public"."eql_v2_…"` | `"undefined".""public"."eql_v2_…"` | + * + * The `"undefined".` prefix is a **0.31.0-and-later** behaviour — 0.30.6 and + * earlier emit the plain form. All six forms stay matched because a user's + * migration directory can hold files generated by any drizzle-kit version + * against any stack version — including the qualified `public.eql_v3_*` that + * stack emitted before the unqualified fix. + * + * Ordered longest-first so the alternation cannot match a prefix of a longer + * form and leave a trailing fragment behind. + * + * NOTE: this file is the sibling of `packages/cli/src/commands/db/rewrite-migrations.ts`, + * which cli's `stash eql migration --drizzle` uses. Both are tightly coupled to + * drizzle-kit's output format — if drizzle-kit changes, both need updating + * together. Keep them in sync. + */ +const MANGLED_TYPE_FORMS = [ + String.raw`"undefined"\.""public"\."(?:${ENCRYPTED_DOMAIN})""`, + String.raw`"undefined"\."public\.(?:${ENCRYPTED_DOMAIN})"`, + String.raw`"undefined"\."(?:${ENCRYPTED_DOMAIN})"`, + String.raw`"public"\."(?:${ENCRYPTED_DOMAIN})"`, + String.raw`public\.(?:${ENCRYPTED_DOMAIN})`, + // Not emitted by any released drizzle-kit ALTER path, but it is the shape the + // CREATE TABLE path renders — guard it in case ALTER converges on it. + String.raw`"(?:${ENCRYPTED_DOMAIN})"`, + String.raw`(?:${ENCRYPTED_DOMAIN})`, +].join('|') + +/** + * Matches drizzle-kit's generated in-place type change to an encrypted column + * type, in any of the forms above. * * Captures: - * - $1: table name (without quotes) - * - $2: column name (without quotes) + * - $1: table name, OR the schema when a qualifier follows (both without quotes) + * - $2: table name when schema-qualified (`"app"."users"`), else undefined + * - $3: column name (without quotes) + * - $4: the mangled type blob — feed it to {@link DOMAIN_RE} for the bare name * - * Note: a copy of this lives in `stash` (`db/rewrite-migrations.ts`) - * because cli's `eql install --drizzle` uses the same fix. Both copies are - * tightly coupled to drizzle-kit's output format — if drizzle-kit changes, - * both need to be updated together. + * The optional `(?:\."([^"]+)")?` after the first quoted name matches the + * schema qualifier drizzle-kit emits for a `pgSchema()` table (`"app"."users"`). + */ +const ALTER_COLUMN_TO_ENCRYPTED_RE = new RegExp( + // `\s*;` — not `[^;]*;` — so the type blob must run straight into the + // statement terminator. drizzle-kit emits exactly that (no trailing clause), + // and the tighter tail means a HAND-AUTHORED `SET DATA TYPE … USING ;` + // is left untouched instead of being silently rewritten (and its USING + // discarded). + String.raw`ALTER TABLE "([^"]+)"(?:\."([^"]+)")?\s+ALTER COLUMN "([^"]+)"\s+SET DATA TYPE (${MANGLED_TYPE_FORMS})\s*;`, + 'gi', +) + +/** + * A deliberately BROAD scan for statements that look like an in-place change to + * an encrypted column but that the strict {@link ALTER_COLUMN_TO_ENCRYPTED_RE} + * did not rewrite — a hand-authored `SET DATA TYPE … USING …;`, or some future + * drizzle-kit form the strict matcher doesn't yet cover. * - * The copies have DIVERGED as of #693: the `stash` one also matches the EQL v3 - * domain family (`eql_v3_*`) and two further mangled forms, and documents which - * drizzle-kit versions emit which. This copy stays v2-only because the wizard - * only ever scaffolds v2 columns. Port the v3 handling here if that changes. + * It matches any single (`;`-terminated) statement that contains both + * `SET DATA TYPE` and an `eql_v2`/`eql_v3` domain token at a word boundary (so + * lookalikes like `eql_v4_…` or `not_eql_v3_…` don't trip it). Run AFTER the + * strict rewrite so genuinely-rewritten statements — which no longer contain + * `SET DATA TYPE` — are already gone; whatever this still finds is a near-miss + * we flag for human review rather than silently ship as broken SQL. */ -const ALTER_COLUMN_TO_ENCRYPTED_RE = - /ALTER TABLE "([^"]+)"\s+ALTER COLUMN "([^"]+)"\s+SET DATA TYPE (?:"undefined"\.""public"\."eql_v2_encrypted""|"undefined"\."eql_v2_encrypted"|"public"\."eql_v2_encrypted"|eql_v2_encrypted)[^;]*;/gi +const NEAR_MISS_RE = + /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi + +/** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ +export interface SkippedAlter { + /** Absolute path of the migration file the statement lives in. */ + file: string + /** The offending statement, verbatim (trimmed), for the user to review. */ + statement: string +} + +/** Outcome of a sweep: the files rewritten, and near-misses left for review. */ +export interface RewriteResult { + /** Absolute paths of files whose unsafe ALTER COLUMN(s) were rewritten. */ + rewritten: string[] + /** Near-miss statements the strict matcher passed over — flag, don't guess. */ + skipped: SkippedAlter[] +} /** - * Replace in-place `ALTER COLUMN ... SET DATA TYPE eql_v2_encrypted` statements - * with an ADD + DROP + RENAME sequence. + * Replace in-place `ALTER COLUMN ... SET DATA TYPE ` + * statements with an ADD + DROP + RENAME sequence. * - * **Why this exists (CIP-2991, CIP-2994):** Postgres has no implicit cast from - * `text`/`numeric` to `eql_v2_encrypted`, so `ALTER COLUMN ... SET DATA TYPE - * eql_v2_encrypted` fails with `cannot cast type ... to eql_v2_encrypted`. - * The fix that works on both empty and non-empty tables is to add a new - * encrypted column, backfill it, drop the original, and rename the new - * column into place. For empty tables the UPDATE is a no-op and the - * sequence is effectively equivalent to DROP+ADD. + * **Why this exists (CIP-2991, CIP-2994, #693):** Postgres has no implicit cast + * from `text`/`numeric` to an encrypted domain, so `ALTER COLUMN ... SET DATA + * TYPE eql_v2_encrypted` (or any `eql_v3_*` domain) fails at migrate time with + * `cannot cast type ... to `. This applies equally to the single EQL v2 + * type and the whole EQL v3 concrete-domain family. * - * We only rewrite the statement — the actual encryption of existing rows has - * to happen in application code (via `encryptModel` from - * `@cipherstash/stack`), which is why the UPDATE is emitted as a guidance - * comment rather than real SQL. Running this migration against a populated - * table leaves the new column NULL until the app backfills it. + * The rewrite is an ADD+DROP+RENAME, which is **equivalent to DROP+ADD**: it + * makes the column type valid but does NOT preserve the column's data. It is + * therefore safe ONLY on an EMPTY table. On a populated table the new column + * starts NULL and the original is dropped in the same migration, so the + * plaintext is destroyed. The commented UPDATE is a placeholder that can never + * become real SQL (the encrypted value is the EQL envelope produced by ZeroKMS + * via the client — there is no expression Postgres can evaluate to fill it), so + * a populated table must instead use the staged `stash encrypt` lifecycle + * (add → backfill via `@cipherstash/stack`'s `encryptModel` → cutover → drop), + * which keeps both columns alive across deploys. Each rewritten file carries a + * header comment saying exactly this. + * + * Returns {@link RewriteResult}: the files rewritten, plus `skipped` near-misses + * — statements that look like an ALTER-to-encrypted but fall outside the strict + * matcher (a hand-authored `SET DATA TYPE … USING …;`, or a future drizzle-kit + * form). Near-misses are left untouched on disk and surfaced non-fatally so the + * caller can tell the user to review them, rather than silently shipping broken + * SQL. */ export async function rewriteEncryptedAlterColumns( outDir: string, options: { skip?: string } = {}, -): Promise { +): Promise { const entries = await readdir(outDir).catch(() => []) const rewritten: string[] = [] + const skipped: SkippedAlter[] = [] for (const entry of entries) { if (!entry.endsWith('.sql')) continue @@ -62,35 +161,91 @@ export async function rewriteEncryptedAlterColumns( if (options.skip && filePath === options.skip) continue const original = await readFile(filePath, 'utf-8') - if (!ALTER_COLUMN_TO_ENCRYPTED_RE.test(original)) continue // Reset the regex's lastIndex — it's stateful on /g ALTER_COLUMN_TO_ENCRYPTED_RE.lastIndex = 0 const updated = original.replace( ALTER_COLUMN_TO_ENCRYPTED_RE, - (_match, table: string, column: string) => renderSafeAlter(table, column), + ( + match: string, + first: string, + second: string | undefined, + column: string, + mangledType: string, + ) => { + // When schema-qualified (`"app"."users"`) the first capture is the + // schema and the second is the table; otherwise the first is the table. + const schema = second === undefined ? undefined : first + const table = second === undefined ? first : second + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() + // Unreachable — the outer regex only matches when a domain is present — + // but leave the statement alone rather than emit a broken rewrite. + if (!domain) return match + return renderSafeAlter(table, column, domain, schema) + }, ) if (updated !== original) { await writeFile(filePath, updated, 'utf-8') rewritten.push(filePath) } + + // Broad secondary scan on the POST-rewrite content: anything still carrying + // `SET DATA TYPE` near an eql_v2/eql_v3 token slipped past the strict + // matcher. Flag it — non-fatally — rather than leave the user shipping SQL + // that fails at migrate time. + for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { + skipped.push({ file: filePath, statement: nearMiss[0].trim() }) + } } - return rewritten + return { rewritten, skipped } } -function renderSafeAlter(table: string, column: string): string { +/** + * The rewrite is identical for v2 and v3, and the ADD+DROP+RENAME sequence is + * equivalent to DROP+ADD: it makes the column type valid but does NOT preserve + * the column's data. It is therefore safe only on an EMPTY table. On a populated + * table the new column starts NULL and the old one is dropped in the same + * migration, so the plaintext is destroyed. + * + * The commented UPDATE is only a placeholder — it can never become real SQL. The + * encrypted value is the EQL envelope produced by ZeroKMS via the client + * (`encryptModel` / `bulkEncryptModels`); there is no expression Postgres can + * evaluate to fill it. (v3 stores that envelope as jsonb rather than v2's + * composite, but this is equally true on both surfaces.) + * + * So the guidance does NOT tell the user to backfill and run this migration — + * that would still lose data on cutover. It points a populated table at the + * staged `stash encrypt` lifecycle (add → backfill → cutover → drop), which + * keeps both columns alive across deploys. + */ +function renderSafeAlter( + table: string, + column: string, + domain: string, + schema?: string, +): string { const tmp = `${column}__cipherstash_tmp` + // Preserve the schema qualifier drizzle-kit emitted for pgSchema() tables so + // the rewritten statements target the same object. + const qualifiedTable = schema ? `"${schema}"."${table}"` : `"${table}"` return [ '-- Rewritten by @cipherstash/wizard: in-place ALTER COLUMN cannot cast to', - `-- eql_v2_encrypted. If "${table}" already has rows, backfill the new`, - "-- column via @cipherstash/stack's encryptModel in application code BEFORE", - '-- running this migration in production. Empty tables are safe as-is.', - `ALTER TABLE "${table}" ADD COLUMN "${tmp}" "public"."eql_v2_encrypted";`, - `-- UPDATE "${table}" SET "${tmp}" = /* encrypted value for ${column} */ NULL;`, - `ALTER TABLE "${table}" DROP COLUMN "${column}";`, - `ALTER TABLE "${table}" RENAME COLUMN "${tmp}" TO "${column}";`, + `-- ${domain}. This ADD+DROP+RENAME equals DROP+ADD and is safe ONLY if`, + `-- ${qualifiedTable} is empty. On a populated table it DESTROYS existing "${column}"`, + '-- data (the new column starts NULL) — do NOT run it there. Use the staged', + "-- `stash encrypt` path instead: add -> backfill via @cipherstash/stack's", + '-- encryptModel in application code -> cutover -> drop.', + '-- NOTE: constraints, defaults, and indexes on the original column are NOT', + '-- carried over by this ADD/DROP/RENAME — re-add any NOT NULL, DEFAULT,', + '-- UNIQUE, or index definitions manually.', + `ALTER TABLE ${qualifiedTable} ADD COLUMN "${tmp}" "public"."${domain}";`, + `-- UPDATE ${qualifiedTable} SET "${tmp}" = /* encrypted value for ${column} */ NULL`, + '--> statement-breakpoint', + `ALTER TABLE ${qualifiedTable} DROP COLUMN "${column}";`, + '--> statement-breakpoint', + `ALTER TABLE ${qualifiedTable} RENAME COLUMN "${tmp}" TO "${column}";`, ].join('\n') } diff --git a/packages/wizard/src/tools/wizard-tools.ts b/packages/wizard/src/tools/wizard-tools.ts index 3a589b7bf..f60673aa5 100644 --- a/packages/wizard/src/tools/wizard-tools.ts +++ b/packages/wizard/src/tools/wizard-tools.ts @@ -149,6 +149,25 @@ interface DbColumn { isEqlEncrypted: boolean } +/** + * Is a Postgres domain-type (`udt_name`) an EQL-encrypted column? + * + * Used purely for DETECTION — telling the agent "this column is already + * encrypted, don't scaffold over it". It therefore recognises BOTH generations: + * the legacy single `eql_v2_encrypted` type AND the EQL v3 concrete-domain + * family (`eql_v3_text_search`, `eql_v3_integer_ord`, …). The wizard now only + * *emits* v3, but a project may already carry v2-encrypted columns whose + * ciphertext stays valid — misreporting them as plaintext would let the agent + * clobber real encrypted data. + * + * The `eql_v3_` prefix (trailing underscore included) matches the convention in + * `@cipherstash/migrate`'s `classifyEqlDomain` — a bare `eql_v3` would also + * claim a hypothetical future major like `eql_v30_*`. + */ +function isEqlEncryptedDomain(udtName: string): boolean { + return udtName === 'eql_v2_encrypted' || udtName.startsWith('eql_v3_') +} + interface DbTable { tableName: string columns: DbColumn[] @@ -183,7 +202,7 @@ export async function introspectDatabase( columnName: row.column_name, dataType: row.data_type, udtName: row.udt_name, - isEqlEncrypted: row.udt_name === 'eql_v2_encrypted', + isEqlEncrypted: isEqlEncryptedDomain(row.udt_name), }) tableMap.set(row.table_name, cols) } From 761bdd9671d63ac5c46a4573750443e8618fc564 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 15:02:25 +1000 Subject: [PATCH 024/123] fix(wizard,cli): sweep every migration dir; trim near-miss preamble MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address code-review feedback on the v3 migration-rewriter port. **Sweep every candidate migration directory.** `rewriteEncryptedMigrations` returned after the FIRST candidate that merely existed, even when that directory contained zero matches — so an empty or already-rewritten `drizzle/` sitting next to a project's real `migrations/` left those migrations unrepaired, and they then failed at migrate time with the very `cannot cast type ...` error the rewriter exists to prevent. The comment justifying the early return ("running again on a different candidate would double-transform already-rewritten SQL") was wrong on both counts: distinct directories hold distinct files, and the rewrite is idempotent anyway — a rewritten statement no longer contains `SET DATA TYPE`, so neither the strict matcher nor the near-miss scan can match it a second time. Extract the sweep into an exported, log-free `sweepMigrationDirs(cwd, dirs)` so it is directly testable without mocking @clack/prompts. It sweeps every existing candidate and reports a directory that throws via `error` rather than stranding the ones after it; post-agent keeps the reporting. **Trim the near-miss statement preamble.** `NEAR_MISS_RE` opens with a lazy `[^;]*?` whose only left boundary is the previous `;` — or the start of file when there is none. The reported statement therefore dragged in every comment and blank line since then, so a near-miss in a file opening with a comment block was quoted back to the user with the whole header glued to its front. Strip leading blank/comment lines (incl. `--> statement-breakpoint`) so the statement reads as the offending statement alone. Detection is unchanged; only the text shown to the user differs. Applied to the `stash` CLI sibling too — it carried the identical defect and its header mandates keeping the two in sync. The directory sweep does not apply there: the CLI takes a single explicit `outDir`. **Document the hardcoded `"public".""`.** Behaviour unchanged and not a regression, but worth stating: the domain qualifier is an assumption (EQL installs into `public`), not something read back from the matched SQL — the `schema` capture is the TABLE's schema and says nothing about where the domain lives. Non-public domain installs would need it threaded in here and in the CLI sibling. Already pinned by the existing pgSchema() test. 9 new tests across the two packages, each watched failing first. --- .changeset/cli-near-miss-statement-trim.md | 15 ++ .changeset/wizard-eql-v3-migration-rewrite.md | 9 + .../src/__tests__/rewrite-migrations.test.ts | 60 +++++++ .../cli/src/commands/db/rewrite-migrations.ts | 26 ++- .../src/__tests__/rewrite-migrations.test.ts | 167 +++++++++++++++++- packages/wizard/src/lib/post-agent.ts | 61 +++---- packages/wizard/src/lib/rewrite-migrations.ts | 89 +++++++++- 7 files changed, 390 insertions(+), 37 deletions(-) create mode 100644 .changeset/cli-near-miss-statement-trim.md diff --git a/.changeset/cli-near-miss-statement-trim.md b/.changeset/cli-near-miss-statement-trim.md new file mode 100644 index 000000000..6cd7180aa --- /dev/null +++ b/.changeset/cli-near-miss-statement-trim.md @@ -0,0 +1,15 @@ +--- +'stash': patch +--- + +Trim the leading comment block from near-miss statements reported by the Drizzle +migration rewriter (`stash eql migration --drizzle`, `stash eql install`). + +The broad near-miss scan is anchored on the previous `;`, so a +`SET DATA TYPE … USING …` it could not safely repair was quoted back to the user +with every preceding comment and blank line glued to its front — in a file +opening with a comment block, that meant the whole header. The reported +statement is now the offending statement alone. Detection is unchanged; only the +text shown to the user is affected. + +Keeps this rewriter in sync with its sibling in `@cipherstash/wizard`. diff --git a/.changeset/wizard-eql-v3-migration-rewrite.md b/.changeset/wizard-eql-v3-migration-rewrite.md index 4e9ae5861..a627e5eb9 100644 --- a/.changeset/wizard-eql-v3-migration-rewrite.md +++ b/.changeset/wizard-eql-v3-migration-rewrite.md @@ -21,6 +21,15 @@ warning that the ADD+DROP+RENAME is data-destroying and safe only on an empty table — a populated table must use the staged `stash encrypt` flow. This re-converges the rewriter with the sibling copy in the `stash` CLI. +The post-agent step now sweeps **every** candidate migration directory +(`drizzle/`, `migrations/`, `src/db/migrations/`) rather than stopping at the +first one that exists. Previously an empty or already-rewritten `drizzle/` +sitting next to a project's real `migrations/` caused those migrations to be +skipped entirely, so they still failed at migrate time. A directory that can't +be read is reported and the remaining candidates are still swept. Reported +near-miss statements are also trimmed of any preceding comment block, so the +statement quoted back to the user is the offending statement alone. + Database introspection also recognises v3 encrypted columns: `isEqlEncrypted` now reports both `eql_v2_encrypted` and the `eql_v3_*` family as already encrypted, so the agent won't scaffold over existing encrypted data of either diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index e213cc886..173cba0fb 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -424,6 +424,66 @@ describe('rewriteEncryptedAlterColumns', () => { expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) }) + // A near-miss is quoted back to the user verbatim, so it must read as the + // offending statement alone. NEAR_MISS_RE opens with a lazy `[^;]*?`, which + // can only be bounded by the previous `;` — so without an explicit trim the + // reported "statement" drags in every comment and blank line since then. + it('reports a near-miss without the file-leading comment block', async () => { + const statement = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;' + const filePath = path.join(tmpDir, '0022_preamble.sql') + fs.writeFileSync( + filePath, + [ + '-- Custom SQL migration file, put your code below! --', + '-- Hand-converts the email column in place.', + '', + statement, + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { + const statement = + 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' + const filePath = path.join(tmpDir, '0023_breakpoint-preamble.sql') + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + statement, + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + + it('keeps a multi-line near-miss statement intact after the preamble trim', async () => { + const statement = [ + 'ALTER TABLE "users"', + ' ALTER COLUMN "email"', + ' SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;', + ].join('\n') + const filePath = path.join(tmpDir, '0024_multiline.sql') + fs.writeFileSync(filePath, `-- leading note\n\n${statement}\n`) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + it('reports no skipped statements for a clean file', async () => { const filePath = path.join(tmpDir, '0020_clean.sql') fs.writeFileSync(filePath, 'CREATE TABLE "t" ("id" integer);\n') diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index a96dc5229..966cd54d8 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -95,6 +95,27 @@ const ALTER_COLUMN_TO_ENCRYPTED_RE = new RegExp( const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi +/** + * Blank lines and `--` line comments (including drizzle-kit's + * `--> statement-breakpoint`) at the head of a {@link NEAR_MISS_RE} match. + * + * That regex opens with a lazy `[^;]*?`, whose only left boundary is the + * previous `;` — or the start of the file when there is no preceding statement. + * So the raw match drags in every comment and blank line since then, and a + * near-miss in a file that opens with a comment block gets reported to the user + * with that whole block glued to its front. Strip the preamble so the statement + * we quote back reads as the offending statement alone. + * + * Only line comments are handled — `/* … *\/` block comments are not something + * drizzle-kit emits, and a stray one costs cosmetics, not correctness. + */ +const STATEMENT_PREAMBLE_RE = /^(?:[^\S\n]*(?:--[^\n]*)?\n)+/ + +/** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ +function trimStatementPreamble(statement: string): string { + return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() +} + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ @@ -189,7 +210,10 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ file: filePath, statement: nearMiss[0].trim() }) + skipped.push({ + file: filePath, + statement: trimStatementPreamble(nearMiss[0]), + }) } } diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 21ae01cd6..41507fe86 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -2,7 +2,10 @@ import fs from 'node:fs' import os from 'node:os' import path from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { rewriteEncryptedAlterColumns } from '../lib/rewrite-migrations.js' +import { + rewriteEncryptedAlterColumns, + sweepMigrationDirs, +} from '../lib/rewrite-migrations.js' describe('rewriteEncryptedAlterColumns', () => { let tmpDir: string @@ -379,6 +382,62 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) }) + // A near-miss is quoted back to the user verbatim, so it must read as the + // offending statement alone. NEAR_MISS_RE opens with a lazy `[^;]*?`, which + // can only be bounded by the previous `;` — so without an explicit trim the + // reported "statement" drags in every comment and blank line since then. + it('reports a near-miss without the file-leading comment block', async () => { + const statement = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;' + const original = [ + '-- Custom SQL migration file, put your code below! --', + '-- Hand-converts the email column in place.', + '', + statement, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0022_preamble.sql') + fs.writeFileSync(filePath, original) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { + const statement = + 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' + const original = [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + statement, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0023_breakpoint-preamble.sql') + fs.writeFileSync(filePath, original) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + + it('keeps a multi-line near-miss statement intact after the preamble trim', async () => { + const statement = [ + 'ALTER TABLE "users"', + ' ALTER COLUMN "email"', + ' SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;', + ].join('\n') + const filePath = path.join(tmpDir, '0024_multiline.sql') + fs.writeFileSync(filePath, `-- leading note\n\n${statement}\n`) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].statement).toBe(statement) + }) + it('handles multiple ALTER statements in one file', async () => { const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v3_text_search;', @@ -397,3 +456,109 @@ describe('rewriteEncryptedAlterColumns', () => { expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') }) }) + +describe('sweepMigrationDirs', () => { + let tmpDir: string + + const ALTER = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n' + + /** Create `dir` under the sandbox, optionally seeding `name` with `sql`. */ + const seedDir = (dir: string, name?: string, sql?: string): string => { + const abs = path.join(tmpDir, dir) + fs.mkdirSync(abs, { recursive: true }) + if (name) fs.writeFileSync(path.join(abs, name), sql ?? ALTER) + return abs + } + + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'wizard-sweep-')) + }) + + afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }) + }) + + it('skips candidate directories that do not exist', async () => { + const abs = seedDir('migrations', '0001_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['drizzle', 'migrations']) + + expect(results.map((r) => r.dir)).toEqual(['migrations']) + expect(results[0].rewritten).toEqual([path.join(abs, '0001_alter.sql')]) + }) + + // The regression this test locks in: the old loop `return`ed after the FIRST + // existing candidate, so a project with an empty `drizzle/` alongside a real + // `migrations/` had its actual migrations silently left unrepaired. + it('keeps sweeping when an earlier candidate directory yields no matches', async () => { + seedDir('drizzle') + const abs = seedDir('migrations', '0002_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['drizzle', 'migrations']) + + expect(results.map((r) => r.dir)).toEqual(['drizzle', 'migrations']) + expect(results[0].rewritten).toEqual([]) + expect(results[1].rewritten).toEqual([path.join(abs, '0002_alter.sql')]) + expect( + fs.readFileSync(path.join(abs, '0002_alter.sql'), 'utf-8'), + ).not.toContain('SET DATA TYPE') + }) + + it('rewrites every candidate directory that holds encrypted alters', async () => { + const drizzle = seedDir('drizzle', '0001_alter.sql') + const nested = seedDir('src/db/migrations', '0002_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, [ + 'drizzle', + 'src/db/migrations', + ]) + + expect(results.flatMap((r) => r.rewritten)).toEqual([ + path.join(drizzle, '0001_alter.sql'), + path.join(nested, '0002_alter.sql'), + ]) + }) + + it('is idempotent — a second sweep rewrites nothing', async () => { + const abs = seedDir('drizzle', '0001_alter.sql') + + await sweepMigrationDirs(tmpDir, ['drizzle']) + const afterFirst = fs.readFileSync( + path.join(abs, '0001_alter.sql'), + 'utf-8', + ) + const results = await sweepMigrationDirs(tmpDir, ['drizzle']) + + expect(results[0].rewritten).toEqual([]) + expect(fs.readFileSync(path.join(abs, '0001_alter.sql'), 'utf-8')).toBe( + afterFirst, + ) + }) + + it('surfaces a failing directory as an error and still sweeps the rest', async () => { + // A directory named `*.sql` makes readFile throw EISDIR mid-sweep. + const broken = seedDir('drizzle') + fs.mkdirSync(path.join(broken, '0001_alter.sql')) + const abs = seedDir('migrations', '0002_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['drizzle', 'migrations']) + + expect(results[0].dir).toBe('drizzle') + expect(results[0].error).toBeDefined() + expect(results[1].rewritten).toEqual([path.join(abs, '0002_alter.sql')]) + }) + + it('reports near-misses per directory', async () => { + const abs = seedDir( + 'drizzle', + '0001_using.sql', + 'ALTER TABLE "t" ALTER COLUMN "c" SET DATA TYPE eql_v3_json USING (c)::jsonb;\n', + ) + + const results = await sweepMigrationDirs(tmpDir, ['drizzle']) + + expect(results[0].skipped).toHaveLength(1) + expect(results[0].skipped[0].file).toBe(path.join(abs, '0001_using.sql')) + }) +}) diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index 24f8bb918..e66af3b1a 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -6,11 +6,9 @@ */ import { execSync } from 'node:child_process' -import { existsSync } from 'node:fs' -import { resolve } from 'node:path' import * as p from '@clack/prompts' import type { GatheredContext } from './gather.js' -import { rewriteEncryptedAlterColumns } from './rewrite-migrations.js' +import { sweepMigrationDirs } from './rewrite-migrations.js' import type { DetectedPackageManager, Integration } from './types.js' interface PostAgentOptions { @@ -21,8 +19,10 @@ interface PostAgentOptions { } /** - * Candidate directories drizzle-kit may write migrations to. We check in - * order and rewrite the first one that exists; `drizzle` is the default. + * Candidate directories drizzle-kit may write migrations to. `drizzle` is the + * default, but a project's configured `out` is not discoverable from here, so + * every candidate that exists is swept — see {@link sweepMigrationDirs} for why + * stopping at the first one loses migrations. */ const DRIZZLE_OUT_DIRS = ['drizzle', 'migrations', 'src/db/migrations'] @@ -115,34 +115,29 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { } async function rewriteEncryptedMigrations(cwd: string): Promise { - for (const dir of DRIZZLE_OUT_DIRS) { - const abs = resolve(cwd, dir) - if (!existsSync(abs)) continue - - try { - const { rewritten, skipped } = await rewriteEncryptedAlterColumns(abs) - if (rewritten.length > 0) { - p.log.info( - `Rewrote ${rewritten.length} migration file(s) in ${dir}/ to use ADD+DROP+RENAME for encrypted columns.`, - ) - for (const file of rewritten) p.log.step(` - ${file}`) - p.log.warn( - 'This rewrite is data-destroying — safe only on an EMPTY table. If any of these tables already have rows, do NOT run the migration; use the staged `stash encrypt` flow (add -> backfill via @cipherstash/stack -> cutover -> drop) instead. See the comments in the rewritten SQL.', - ) - } - if (skipped.length > 0) { - p.log.warn( - `${skipped.length} statement(s) look like an ALTER-to-encrypted the rewrite could not safely repair (e.g. a hand-authored SET DATA TYPE ... USING ...). Review them before migrating:`, - ) - for (const s of skipped) p.log.step(` - ${s.file}: ${s.statement}`) - } - // Only rewrite the first dir that matches — running again on a - // different candidate would double-transform already-rewritten SQL. - return - } catch (err) { - const message = err instanceof Error ? err.message : String(err) - p.log.warn(`Could not rewrite migrations in ${dir}: ${message}`) - return + const results = await sweepMigrationDirs(cwd, DRIZZLE_OUT_DIRS) + + for (const { dir, rewritten, skipped, error } of results) { + if (error) { + p.log.warn(`Could not rewrite migrations in ${dir}: ${error}`) + continue + } + + if (rewritten.length > 0) { + p.log.info( + `Rewrote ${rewritten.length} migration file(s) in ${dir}/ to use ADD+DROP+RENAME for encrypted columns.`, + ) + for (const file of rewritten) p.log.step(` - ${file}`) + p.log.warn( + 'This rewrite is data-destroying — safe only on an EMPTY table. If any of these tables already have rows, do NOT run the migration; use the staged `stash encrypt` flow (add -> backfill via @cipherstash/stack -> cutover -> drop) instead. See the comments in the rewritten SQL.', + ) + } + + if (skipped.length > 0) { + p.log.warn( + `${skipped.length} statement(s) look like an ALTER-to-encrypted the rewrite could not safely repair (e.g. a hand-authored SET DATA TYPE ... USING ...). Review them before migrating:`, + ) + for (const s of skipped) p.log.step(` - ${s.file}: ${s.statement}`) } } } diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index bbe2a754f..ce3ce1ab6 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -1,5 +1,6 @@ +import { existsSync } from 'node:fs' import { readdir, readFile, writeFile } from 'node:fs/promises' -import { join } from 'node:path' +import { join, resolve } from 'node:path' /** * The encrypted column types this rewrite applies to: the single EQL v2 type, @@ -102,6 +103,27 @@ const ALTER_COLUMN_TO_ENCRYPTED_RE = new RegExp( const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi +/** + * Blank lines and `--` line comments (including drizzle-kit's + * `--> statement-breakpoint`) at the head of a {@link NEAR_MISS_RE} match. + * + * That regex opens with a lazy `[^;]*?`, whose only left boundary is the + * previous `;` — or the start of the file when there is no preceding statement. + * So the raw match drags in every comment and blank line since then, and a + * near-miss in a file that opens with a comment block gets reported to the user + * with that whole block glued to its front. Strip the preamble so the statement + * we quote back reads as the offending statement alone. + * + * Only line comments are handled — `/* … *\/` block comments are not something + * drizzle-kit emits, and a stray one costs cosmetics, not correctness. + */ +const STATEMENT_PREAMBLE_RE = /^(?:[^\S\n]*(?:--[^\n]*)?\n)+/ + +/** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ +function trimStatementPreamble(statement: string): string { + return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() +} + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ @@ -196,13 +218,67 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ file: filePath, statement: nearMiss[0].trim() }) + skipped.push({ + file: filePath, + statement: trimStatementPreamble(nearMiss[0]), + }) } } return { rewritten, skipped } } +/** One candidate migration directory's outcome from {@link sweepMigrationDirs}. */ +export interface DirRewriteResult extends RewriteResult { + /** The candidate directory as supplied (relative to `cwd`), for reporting. */ + dir: string + /** Set when this directory's sweep threw; the sweep continues regardless. */ + error?: string +} + +/** + * Sweep every candidate migration directory under `cwd`, rewriting each one's + * unsafe ALTER-to-encrypted statements. Directories that don't exist are + * skipped; the returned array carries one entry per directory that did. + * + * **Why every directory, not just the first:** a project's drizzle-kit `out` + * directory is not discoverable from here, so the caller passes a candidate + * list. Stopping at the first candidate that merely *exists* means an empty or + * already-rewritten `drizzle/` sitting next to the project's real `migrations/` + * silently leaves those migrations unrepaired — they then fail at migrate time + * with the very `cannot cast type ...` error this function exists to prevent. + * + * Sweeping all of them is safe because the rewrite is idempotent: a rewritten + * statement no longer contains `SET DATA TYPE`, so neither + * {@link ALTER_COLUMN_TO_ENCRYPTED_RE} nor {@link NEAR_MISS_RE} can match it a + * second time. Two candidates resolving to the same directory (via a symlink) + * therefore cost a redundant read, not a double transform. + * + * A directory that throws mid-sweep is reported via `error` rather than + * aborting — one unreadable candidate must not strand the others. + */ +export async function sweepMigrationDirs( + cwd: string, + dirs: readonly string[], +): Promise { + const results: DirRewriteResult[] = [] + + for (const dir of dirs) { + const abs = resolve(cwd, dir) + if (!existsSync(abs)) continue + + try { + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(abs) + results.push({ dir, rewritten, skipped }) + } catch (err) { + const message = err instanceof Error ? err.message : String(err) + results.push({ dir, rewritten: [], skipped: [], error: message }) + } + } + + return results +} + /** * The rewrite is identical for v2 and v3, and the ADD+DROP+RENAME sequence is * equivalent to DROP+ADD: it makes the column type valid but does NOT preserve @@ -241,6 +317,15 @@ function renderSafeAlter( '-- NOTE: constraints, defaults, and indexes on the original column are NOT', '-- carried over by this ADD/DROP/RENAME — re-add any NOT NULL, DEFAULT,', '-- UNIQUE, or index definitions manually.', + // The domain is emitted as `"public".""` unconditionally: EQL + // installs its domains into `public` (both `stash eql install` and the + // adapters' baseline migrations do), and the qualifier makes the rewrite + // independent of the session `search_path`. It is an ASSUMPTION, not + // something read back from the matched SQL — the `schema` capture above is + // the TABLE's schema (from a pgSchema() table) and says nothing about where + // the domain lives. If EQL ever supports installing into a non-`public` + // schema, this needs the install schema threaded in, here and in the + // sibling `packages/cli/src/commands/db/rewrite-migrations.ts`. `ALTER TABLE ${qualifiedTable} ADD COLUMN "${tmp}" "public"."${domain}";`, `-- UPDATE ${qualifiedTable} SET "${tmp}" = /* encrypted value for ${column} */ NULL`, '--> statement-breakpoint', From 0cef401c09376797e2ec24083b35b4348d139f01 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 23 Jul 2026 17:46:31 +1000 Subject: [PATCH 025/123] fix(wizard): resolve @cipherstash/auth via the node condition; gate typecheck MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the last open item from the #771 review — the 3 pre-existing `AutoStrategy` tsc errors flagged as "would bite a future strict type gate". **Root cause.** `@cipherstash/auth` uses conditional exports: `node` resolves to `index.d.ts` (the full surface, including `AutoStrategy`), `default` resolves to `wasm-types.d.ts` (which has no `AutoStrategy`). The wizard's tsconfig sets `moduleResolution: "bundler"`, which does NOT include the `node` condition, so tsc took the `default` branch and reported `AutoStrategy` as missing on all three call sites — `agent/fetch-prompt.ts`, `agent/interface.ts` and `lib/prerequisites.ts`. Nothing was actually broken at runtime: the wizard is a Node CLI and always loads the `node` branch. The types were simply resolved against the wrong entry point. **Fix.** Add `"customConditions": ["node"]`, matching what `packages/stack`, `packages/prisma-next`, `packages/test-kit`, `packages/stack-drizzle` and `packages/stack-supabase` already carry for the identical `protect-ffi` conditional-export problem. `tsc --noEmit` now exits 0. **Regression guard.** A type-level defect needs a type-level gate, or it comes back silently — the wizard is built by tsup with `dts: false`, so the build transpiles without ever typechecking, which is exactly why these three errors sat in `main` unnoticed. Add a `typecheck` script and wire it into `tests.yml` alongside the prisma-next gate (#684). Verified the gate fires: with `customConditions` removed the step exits 2 on all three errors; restored, it exits 0. No changeset — `dts: false` means the published tarball is byte-identical. Tooling only, no observable behaviour change. Green: wizard 265 pass / 5 env-skipped, typecheck 0 errors; stash 770 pass / 53 files; `code:check` 0 errors. --- .github/workflows/tests.yml | 6 ++++++ packages/wizard/package.json | 1 + packages/wizard/tsconfig.json | 11 +++++++++++ 3 files changed, 18 insertions(+) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 7f2e277f3..eb27ad8b2 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -146,6 +146,12 @@ jobs: - name: Typecheck (bench — guards the stack/stack-drizzle importers) run: pnpm exec turbo run build --filter @cipherstash/bench + # The wizard is built by tsup, which transpiles without typechecking, so + # nothing here caught the three `auth.AutoStrategy` resolution errors that + # sat in `main` until #771. Gate it so they cannot come back silently. + - name: Typecheck (wizard) + run: pnpm --filter @cipherstash/wizard run typecheck + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/packages/wizard/package.json b/packages/wizard/package.json index 467135fd3..f9d982171 100644 --- a/packages/wizard/package.json +++ b/packages/wizard/package.json @@ -26,6 +26,7 @@ "postbuild": "chmod +x ./dist/bin/wizard.js", "dev": "tsup --watch", "test": "vitest run", + "typecheck": "tsc --project tsconfig.json --noEmit", "lint": "biome check ." }, "dependencies": { diff --git a/packages/wizard/tsconfig.json b/packages/wizard/tsconfig.json index 1f60894ec..d88a9b847 100644 --- a/packages/wizard/tsconfig.json +++ b/packages/wizard/tsconfig.json @@ -7,6 +7,17 @@ "allowJs": true, "esModuleInterop": true, "moduleResolution": "bundler", + + // `@cipherstash/auth` uses conditional exports — `node` picks `index.d.ts` + // (the full surface, including `AutoStrategy`), `default` picks + // `wasm-types.d.ts` (which has no `AutoStrategy`). Bundler resolution does + // not include `node` by default, so without this the wizard's three + // `auth.AutoStrategy` call sites fail to typecheck even though they run + // fine — the wizard is a Node CLI and always loads the `node` branch. + // Matches `packages/stack`, `packages/prisma-next`, `packages/test-kit`, + // `packages/stack-drizzle` and `packages/stack-supabase`. + "customConditions": ["node"], + "allowImportingTsExtensions": true, "verbatimModuleSyntax": true, "noEmit": true, From 9ed73a72aeacf07883ecd78292c896a9ce2d75dc Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 11:13:12 +1000 Subject: [PATCH 026/123] docs(skills): repoint drizzle index guidance at the collapsed root Rebase fallout: main's `stash-indexing` skill (#773) and the indexing section it added to `stash-drizzle` both import `encryptedIndexes` from `@cipherstash/stack-drizzle/v3`, a subpath this branch removes when it collapses `./v3` into the package root. Both files ship to customer repos, so the stale specifier would not resolve in a freshly-initialised project. Caught by `scripts/__tests__/no-removed-drizzle-surface.test.mjs`, the guard this branch added for exactly this drift. --- skills/stash-drizzle/SKILL.md | 2 +- skills/stash-indexing/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 3777aac76..027deb6ab 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -377,7 +377,7 @@ if (!decrypted.failure) { Drizzle emits the encrypted query operators, but **no index DDL** — without functional indexes over the `eql_v3.*` term extractors, every encrypted predicate sequential-scans. `encryptedIndexes` derives the recommended indexes for every encrypted column in the table from its domain, so a schema column can't be forgotten: ```typescript -import { encryptedIndexes, types } from "@cipherstash/stack-drizzle/v3" +import { encryptedIndexes, types } from "@cipherstash/stack-drizzle" import { integer, pgTable } from "drizzle-orm/pg-core" export const users = pgTable( diff --git a/skills/stash-indexing/SKILL.md b/skills/stash-indexing/SKILL.md index 10b362d16..5e7b46cc9 100644 --- a/skills/stash-indexing/SKILL.md +++ b/skills/stash-indexing/SKILL.md @@ -254,7 +254,7 @@ Index not being used: **The integrations emit the query operators for you — none applies index DDL on its own. Making sure these indexes exist is always your job.** This skill is the general model — recipes, engagement rules, verification. How to apply it in a specific integration lives in that integration's skill: -- **Drizzle** — `encryptedIndexes(t)` from `@cipherstash/stack-drizzle/v3` derives the recommended indexes for every encrypted column in the table, or declare individual expression indexes in the schema DSL. See `stash-drizzle` § Indexing Encrypted Columns. +- **Drizzle** — `encryptedIndexes(t)` from `@cipherstash/stack-drizzle` derives the recommended indexes for every encrypted column in the table, or declare individual expression indexes in the schema DSL. See `stash-drizzle` § Indexing Encrypted Columns. - **Prisma Next** — Prisma's schema language cannot express functional indexes; the DDL goes in a migration in the adapter's flow. See `stash-prisma-next`. - **Supabase** — a `supabase/migrations/` file; no superuser needed (see above). See `stash-supabase`. - **Raw SQL / plain PostgreSQL** — the recipes in this skill, in whatever migration tool owns the schema. Never ad-hoc in production. From ccc7fbc1b776c5b2522181256805fc00a2d26c76 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 11:22:05 +1000 Subject: [PATCH 027/123] feat(cli,examples)!: de-suffix the init scaffold, rewrite examples/basic to v3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRs 3-5 of the EQL v2 removal froze the target names, but nothing type-checks the artefacts that describe them, so two were left wrong. `stash init` scaffolded `EncryptionV3` into customer source. That name is a deprecated alias now, and the scaffold is a template literal, so only an assertion can catch it — the codegen tests pinned the old form, so they are flipped with negatives added, mirroring the drizzle precedent from 7b783eef. `@cipherstash/stack/v3` exported `EncryptionV3` but not `Encryption`, so the scaffold could not emit a single clean import. Re-export it — the deprecation example in v3.ts already told users to import it from there. `examples/basic` had not compiled since the removal deleted `encryptedType` and the v2 `encryptedSupabase`. Ported to the v3 types.* factories. Its Supabase branch is deleted rather than ported: it imported a `contactsTable` that was never exported, so it was already dead before this work, and examples/supabase-worker carries the Supabase story. Root `build`/`test` filter to ./packages/*, so CI never compiled any example and the breakage sat on a green board. Gate examples/basic through a new turbo `typecheck` task so `^build` builds its deps first. Verified the gate fails on the exact regression that shipped. --- .github/workflows/tests.yml | 8 +++ examples/basic/encrypt.ts | 8 ++- examples/basic/index.ts | 27 -------- examples/basic/package.json | 5 +- examples/basic/src/encryption/index.ts | 21 +++--- examples/basic/src/lib/supabase/encrypted.ts | 9 --- examples/basic/src/lib/supabase/server.ts | 8 --- examples/basic/src/queries/contacts.ts | 68 ------------------- .../init/__tests__/utils-codegen.test.ts | 8 ++- .../src/commands/init/steps/install-eql.ts | 4 +- packages/cli/src/commands/init/utils.ts | 34 +++++----- packages/stack/src/encryption/v3.ts | 5 +- pnpm-lock.yaml | 6 +- turbo.json | 4 ++ 14 files changed, 62 insertions(+), 153 deletions(-) delete mode 100644 examples/basic/src/lib/supabase/encrypted.ts delete mode 100644 examples/basic/src/lib/supabase/server.ts delete mode 100644 examples/basic/src/queries/contacts.ts diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index eb27ad8b2..3a72ef269 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -152,6 +152,14 @@ jobs: - name: Typecheck (wizard) run: pnpm --filter @cipherstash/wizard run typecheck + # `examples/*` are standalone apps outside the `./packages/*` filter that + # root `build`/`test` use, so nothing in CI compiled them. `examples/basic` + # had been broken since the v2 removal deleted `encryptedType` and the v2 + # `encryptedSupabase` — on a fully green board. Gate it through turbo so + # `^build` builds stack/stack-drizzle/stash first. + - name: Typecheck (examples/basic — guards the v3 stack/stack-drizzle importers) + run: pnpm exec turbo run typecheck --filter @cipherstash/basic-example + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/examples/basic/encrypt.ts b/examples/basic/encrypt.ts index 140e22605..17f869fd5 100644 --- a/examples/basic/encrypt.ts +++ b/examples/basic/encrypt.ts @@ -1,9 +1,13 @@ import 'dotenv/config' -import { Encryption, encryptedColumn, encryptedTable } from '@cipherstash/stack' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +// EQL v3: a column's query capabilities are fixed by the domain you pick — +// there are no chainable capability tuners. `types.Text` is storage-only +// (encrypt/decrypt, no queries), which is all this demo needs. Reach for +// `types.TextEq` / `types.TextSearch` when you need to query the column. export const users = encryptedTable('users', { - name: encryptedColumn('name'), + name: types.Text('name'), }) export const client = await Encryption({ diff --git a/examples/basic/index.ts b/examples/basic/index.ts index 63bc282a6..b00ad13fa 100644 --- a/examples/basic/index.ts +++ b/examples/basic/index.ts @@ -1,7 +1,6 @@ import 'dotenv/config' import readline from 'node:readline' import { client, users } from './encrypt' -import { createContact, getAllContacts } from './src/queries/contacts' const rl = readline.createInterface({ input: process.stdin, @@ -69,32 +68,6 @@ async function main() { console.log('Bulk encrypted data:', bulkEncryptResult.data) - // Demonstrate Supabase integration with CipherStash encryption - console.log('\n--- Supabase Integration Demo ---') - - try { - // Example: Create a new contact (would insert into encrypted Supabase table) - console.log('Creating encrypted contact...') - const newContact = { - name: 'John Doe', - email: 'john@example.com', - role: 'Developer', // This field will be encrypted using CipherStash - } - - // Note: This would fail in this basic example since we don't have actual Supabase setup - // but shows the pattern for encrypted Supabase usage - console.log('Contact data to encrypt:', newContact) - - // Example: Fetch contacts (would decrypt results from Supabase) - console.log('Fetching encrypted contacts...') - // const contacts = await getAllContacts() - // console.log('Decrypted contacts:', contacts.data) - } catch (error) { - console.log( - 'Supabase demo skipped (no actual Supabase connection in this basic example)', - ) - } - rl.close() } diff --git a/examples/basic/package.json b/examples/basic/package.json index 1c1de60b3..4a62550ad 100644 --- a/examples/basic/package.json +++ b/examples/basic/package.json @@ -4,7 +4,8 @@ "version": "1.2.14-rc.4", "type": "module", "scripts": { - "start": "tsx index.ts" + "start": "tsx index.ts", + "typecheck": "tsc --project tsconfig.json --noEmit" }, "keywords": [], "author": "", @@ -13,7 +14,7 @@ "dependencies": { "@cipherstash/stack": "workspace:*", "@cipherstash/stack-drizzle": "workspace:*", - "@cipherstash/stack-supabase": "workspace:*", + "drizzle-orm": "^0.45.2", "dotenv": "^17.4.2", "pg": "8.22.0" }, diff --git a/examples/basic/src/encryption/index.ts b/examples/basic/src/encryption/index.ts index 4f78be08e..fca33a2bf 100644 --- a/examples/basic/src/encryption/index.ts +++ b/examples/basic/src/encryption/index.ts @@ -1,20 +1,15 @@ -import { Encryption } from '@cipherstash/stack' -import { - encryptedType, - extractEncryptionSchema, -} from '@cipherstash/stack-drizzle' +import { Encryption } from '@cipherstash/stack/v3' +import { extractEncryptionSchema, types } from '@cipherstash/stack-drizzle' import { integer, pgTable, timestamp } from 'drizzle-orm/pg-core' +// EQL v3 encrypted columns are concrete Postgres domains built with the +// `types.*` factories. The domain fixes the query capabilities: `TextSearch` +// is equality + order/range + free-text, the v3 equivalent of what the old v2 +// builder spelled `.equality().freeTextSearch()`. export const usersTable = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType('email', { - equality: true, - freeTextSearch: true, - }), - name: encryptedType('name', { - equality: true, - freeTextSearch: true, - }), + email: types.TextSearch('email'), + name: types.TextSearch('name'), createdAt: timestamp('created_at').defaultNow(), }) diff --git a/examples/basic/src/lib/supabase/encrypted.ts b/examples/basic/src/lib/supabase/encrypted.ts deleted file mode 100644 index 147930987..000000000 --- a/examples/basic/src/lib/supabase/encrypted.ts +++ /dev/null @@ -1,9 +0,0 @@ -import { encryptedSupabase } from '@cipherstash/stack-supabase' -import { contactsTable, encryptionClient } from '../../encryption/index' -import { createServerClient } from './server' - -const supabase = await createServerClient() -export const eSupabase = encryptedSupabase({ - encryptionClient, - supabaseClient: supabase, -}) diff --git a/examples/basic/src/lib/supabase/server.ts b/examples/basic/src/lib/supabase/server.ts deleted file mode 100644 index 967949176..000000000 --- a/examples/basic/src/lib/supabase/server.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { createClient } from '@supabase/supabase-js' - -export async function createServerClient() { - const supabaseUrl = process.env.NEXT_PUBLIC_SUPABASE_URL! - const supabaseKey = process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY! - - return createClient(supabaseUrl, supabaseKey) -} diff --git a/examples/basic/src/queries/contacts.ts b/examples/basic/src/queries/contacts.ts deleted file mode 100644 index ad8979b39..000000000 --- a/examples/basic/src/queries/contacts.ts +++ /dev/null @@ -1,68 +0,0 @@ -import { contactsTable } from '../encryption/index' -import { eSupabase } from '../lib/supabase/encrypted' - -// Example queries using encrypted Supabase wrapper - -export async function getAllContacts() { - const { data, error } = await eSupabase - .from('contacts', contactsTable) - .select('id, name, email, role') // explicit columns, no * - .order('created_at', { ascending: false }) - - return { data, error } -} - -export async function getContactsByRole(role: string) { - const { data, error } = await eSupabase - .from('contacts', contactsTable) - .select('id, name, email, role') - .eq('role', role) // auto-encrypted - - return { data, error } -} - -export async function searchContactsByName(searchTerm: string) { - const { data, error } = await eSupabase - .from('contacts', contactsTable) - .select('id, name, email, role') - .ilike('name', `%${searchTerm}%`) // auto-encrypted - - return { data, error } -} - -export async function createContact(contact: { - name: string - email: string - role: string -}) { - const { data, error } = await eSupabase - .from('contacts', contactsTable) - .insert(contact) // auto-encrypted - .select('id, name, email, role') - .single() - - return { data, error } -} - -export async function updateContact( - id: string, - updates: Partial<{ name: string; email: string; role: string }>, -) { - const { data, error } = await eSupabase - .from('contacts', contactsTable) - .update(updates) // auto-encrypted - .eq('id', id) - .select('id, name, email, role') - .single() - - return { data, error } -} - -export async function deleteContact(id: string) { - const { error } = await eSupabase - .from('contacts', contactsTable) - .delete() - .eq('id', id) - - return { error } -} diff --git a/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts b/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts index 5aabc73d4..59067ee14 100644 --- a/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts +++ b/packages/cli/src/commands/init/__tests__/utils-codegen.test.ts @@ -20,7 +20,12 @@ describe('generateClientFromSchemas', () => { expect(out).toContain("age: types.IntegerOrd('age'),") expect(out).toContain("verified: types.Boolean('verified'),") expect(out).toContain("from '@cipherstash/stack/v3'") - expect(out).toContain('EncryptionV3(') + // `Encryption` is the current name; `EncryptionV3` is a deprecated alias. + // Same reasoning as the drizzle negatives below — this is a template + // literal written into the user's repo as real source, so nothing but an + // assertion here catches a scaffold that teaches the deprecated name. + expect(out).toContain('Encryption(') + expect(out).not.toContain('EncryptionV3') }) it('emits the chosen v3 domain factory per column (drizzle)', () => { @@ -54,6 +59,7 @@ describe('generateClientFromSchemas', () => { const out = generateClientFromSchemas('supabase', schemas) expect(out).toContain("email: types.TextSearch('email'),") expect(out).toContain("from '@cipherstash/stack/v3'") + expect(out).not.toContain('EncryptionV3') expect(out).not.toContain('@cipherstash/stack-drizzle') }) diff --git a/packages/cli/src/commands/init/steps/install-eql.ts b/packages/cli/src/commands/init/steps/install-eql.ts index 9ee6bd880..f6c912466 100644 --- a/packages/cli/src/commands/init/steps/install-eql.ts +++ b/packages/cli/src/commands/init/steps/install-eql.ts @@ -108,8 +108,8 @@ export const installEqlStep: InitStep = { // at all. That pin made `stash init --drizzle` the one flow that provisions // a v2 database while every other integration (and a bare `stash eql // install`) gets v3, and it contradicted the stash-drizzle skill we install - // into the very same project — that skill documents the `/v3` surface - // (`types.*` domains, `EncryptionV3`) and would have the user's agent + // into the very same project — that skill documents the v3 surface + // (`types.*` domains, `Encryption`) and would have the user's agent // author v3 code against a v2 database. // // `stash eql migration --drizzle` (added in #691) closes that gap: v3 SQL, diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 68526460e..fc0e6aec1 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -286,11 +286,11 @@ const ${schemaVarName} = extractEncryptionSchema(${varName})` return `import { pgTable, integer, timestamp } from 'drizzle-orm/pg-core' import { types, extractEncryptionSchema } from '@cipherstash/stack-drizzle' -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { Encryption } from '@cipherstash/stack/v3' ${tableDefs.join('\n\n')} -export const encryptionClient = await EncryptionV3({ +export const encryptionClient = await Encryption({ schemas: [${schemaVarNames.join(', ')}], }) ` @@ -311,11 +311,11 @@ ${columnDefs.join('\n')} const tableVarNames = schemas.map((s) => `${toCamelCase(s.tableName)}Table`) - return `import { EncryptionV3, encryptedTable, types } from '@cipherstash/stack/v3' + return `import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' ${tableDefs.join('\n\n')} -export const encryptionClient = await EncryptionV3({ +export const encryptionClient = await Encryption({ schemas: [${tableVarNames.join(', ')}], }) ` @@ -366,7 +366,7 @@ export function generateClientFromSchemas( * schemas yet, and explicitly tells the agent that the user's existing * schema files remain authoritative. The agent's job during the handoff * is to declare encrypted columns directly in those files and update the - * `EncryptionV3({ schemas: [...] })` call below to reference them. + * `Encryption({ schemas: [...] })` call below to reference them. */ export function generatePlaceholderClient(integration: Integration): string { if (integration === 'drizzle') { @@ -381,7 +381,7 @@ const DRIZZLE_PLACEHOLDER = `/** * \`stash init\` wrote this file. It is intentionally NOT a real Drizzle * schema. Your existing schema files (typically under \`src/db/\`) remain * authoritative — your agent will edit those directly when you encrypt a - * column, then update the \`EncryptionV3({ schemas: [...] })\` call below + * column, then update the \`Encryption({ schemas: [...] })\` call below * to reference the encrypted tables you declared there. * * Until that happens, the encryption client is initialised with no @@ -419,19 +419,19 @@ const DRIZZLE_PLACEHOLDER = `/** * billing_address: types.TextEq('billing_address'), * }) * - * Once you have encrypted tables declared, harvest them and pass to EncryptionV3(): + * Once you have encrypted tables declared, harvest them and pass to Encryption(): * * import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' - * import { EncryptionV3 } from '@cipherstash/stack/v3' + * import { Encryption } from '@cipherstash/stack/v3' * import { users, orders } from './db/schema' * - * export const encryptionClient = await EncryptionV3({ + * export const encryptionClient = await Encryption({ * schemas: [extractEncryptionSchema(users), extractEncryptionSchema(orders)], * }) */ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { Encryption } from '@cipherstash/stack/v3' -export const encryptionClient = await EncryptionV3({ schemas: [] }) +export const encryptionClient = await Encryption({ schemas: [] }) ` const GENERIC_PLACEHOLDER = `/** @@ -440,7 +440,7 @@ const GENERIC_PLACEHOLDER = `/** * \`stash init\` wrote this file. It is intentionally NOT a real schema * definition. Your existing schema files remain authoritative — your * agent will declare encrypted columns there and update the - * \`EncryptionV3({ schemas: [...] })\` call below to reference them. + * \`Encryption({ schemas: [...] })\` call below to reference them. * * Until that happens, the encryption client is initialised with no * schemas, and \`stash encrypt\` commands will surface a clear error @@ -474,16 +474,16 @@ const GENERIC_PLACEHOLDER = `/** * billing_address: types.TextEq('billing_address'), * }) * - * Once you have encrypted tables declared, pass them to EncryptionV3(): + * Once you have encrypted tables declared, pass them to Encryption(): * - * import { EncryptionV3 } from '@cipherstash/stack/v3' + * import { Encryption } from '@cipherstash/stack/v3' * import { users, orders } from './db/schema' * - * export const encryptionClient = await EncryptionV3({ + * export const encryptionClient = await Encryption({ * schemas: [users, orders], * }) */ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { Encryption } from '@cipherstash/stack/v3' -export const encryptionClient = await EncryptionV3({ schemas: [] }) +export const encryptionClient = await Encryption({ schemas: [] }) ` diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index e36c90731..dc4602e13 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -334,5 +334,8 @@ export const EncryptionV3 = Encryption // Single import surface: re-export the v3 `types` namespace + table API + type // helpers so `@cipherstash/stack/v3` provides everything needed to author and -// use a schema. +// use a schema. `Encryption` comes along for the same reason — it is the +// current name for what `EncryptionV3` aliases, so authoring a v3 schema and +// building its client should not need a second import specifier. +export { Encryption } export * from '@/eql/v3' diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index dac40d1ff..8a666f68e 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -117,12 +117,12 @@ importers: '@cipherstash/stack-drizzle': specifier: workspace:* version: link:../../packages/stack-drizzle - '@cipherstash/stack-supabase': - specifier: workspace:* - version: link:../../packages/stack-supabase dotenv: specifier: ^17.4.2 version: 17.4.2 + drizzle-orm: + specifier: ^0.45.2 + version: 0.45.2(@types/pg@8.20.0)(gel@2.2.0)(mysql2@3.16.0)(pg@8.22.0)(postgres@3.4.9) pg: specifier: 8.22.0 version: 8.22.0 diff --git a/turbo.json b/turbo.json index 4ef8a864b..6ed7b4a76 100644 --- a/turbo.json +++ b/turbo.json @@ -19,6 +19,10 @@ "inputs": ["$TURBO_DEFAULT$", ".env*"], "cache": false }, + "typecheck": { + "dependsOn": ["^build"], + "inputs": ["$TURBO_DEFAULT$"] + }, "test:e2e": { "dependsOn": ["^build", "build"], "inputs": ["$TURBO_DEFAULT$", ".env*"], From b2f9d7a53ddcf4ff46a2ad3e6f50b6563a008182 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 11:31:28 +1000 Subject: [PATCH 028/123] docs(skills,meta): correct the naming the v2 removal inverted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRs 3-5 froze the API names but the prose describing them was not swept, so the shipped guidance told users the opposite of what the code does. The worst of it is in skills/, which ship inside the stash tarball and get copied into customer repos: stash-encryption and stash-cli both described `encryptedSupabase` as the legacy EQL v2 wrapper. It is the EQL v3 factory — the v2 wrapper was deleted in #769. stash-drizzle taught `EncryptionV3` as the canonical client, contradicting stash-encryption, which correctly calls it deprecated. Also corrects the @cipherstash/stack README, the npm landing page, which claimed DynamoDB "still requires v2" and pointed at #657 — #768 removed the v2 write overloads and #657 is closed. The root README quickstart still taught the v2 chainable builders as the primary example. Reframes the "Legacy: EQL v2" sections to say what is actually true now: v2 is a read path, not an authoring surface. Left alone deliberately: the v2 CLI/database sections in supabase-sdk.md and the v2 read path in stash-dynamodb — both accurate today, and the CLI text moves with the SQL teardown. `export { Encryption }` in v3.ts is ordered after the `export *` so Biome's organizeImports is satisfied without detaching the comment from its subject. --- .../remove-eql-v2-scaffold-examples-meta.md | 22 +++++++ AGENTS.md | 4 +- README.md | 8 +-- SECURITY.md | 4 +- docs/reference/supabase-sdk.md | 38 ++++++------ packages/bench/README.md | 5 +- packages/migrate/README.md | 2 +- packages/stack-drizzle/README.md | 4 +- packages/stack/README.md | 58 ++++++++++--------- packages/stack/src/encryption/v3.ts | 9 +-- skills/stash-cli/SKILL.md | 2 +- skills/stash-drizzle/SKILL.md | 12 ++-- skills/stash-encryption/SKILL.md | 6 +- skills/stash-prisma-next/SKILL.md | 2 +- 14 files changed, 102 insertions(+), 74 deletions(-) create mode 100644 .changeset/remove-eql-v2-scaffold-examples-meta.md diff --git a/.changeset/remove-eql-v2-scaffold-examples-meta.md b/.changeset/remove-eql-v2-scaffold-examples-meta.md new file mode 100644 index 000000000..38714d679 --- /dev/null +++ b/.changeset/remove-eql-v2-scaffold-examples-meta.md @@ -0,0 +1,22 @@ +--- +'stash': patch +'@cipherstash/stack': patch +--- + +De-suffix the v3 client name in generated code and shipped guidance. + +`stash init` scaffolded `import { EncryptionV3 } from '@cipherstash/stack/v3'` +into the client file it writes. `EncryptionV3` is a deprecated alias of +`Encryption`, so new projects were started on the deprecated name. The +scaffold now emits `Encryption`. + +`@cipherstash/stack/v3` now re-exports `Encryption` alongside the deprecated +`EncryptionV3` alias, so a v3 schema and its client come from one import +specifier — the deprecation notice already documented this import, but it did +not resolve. + +Corrects the bundled agent skills and package docs, which described +`encryptedSupabase` as the legacy EQL v2 wrapper. It is the EQL v3 factory; +the v2 wrapper was removed. Also drops the stale "DynamoDB still requires v2" +note from the `@cipherstash/stack` README — DynamoDB writes EQL v3 and reads +existing v2 items. diff --git a/AGENTS.md b/AGENTS.md index 5fcc10440..943dbb981 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,7 +78,7 @@ If these variables are missing, tests that require live encryption will fail or - `packages/migrate`: Plaintext-to-encrypted column migration (`@cipherstash/migrate`) — resumable backfill, per-column state - `packages/prisma-next`: Prisma Next integration (`@cipherstash/prisma-next`) — searchable field-level encryption for Postgres. **EQL v3 only**: per-domain constructors (`cipherstash.TextSearch()` / `text()` / `bigIntOrd()` / …) and `cipherstashFromStack` (the `./v3` and `./stack` entries). The EQL v2 surface was removed — the adapter's baseline migration installs the EQL v3 bundle only (works on Supabase as a non-superuser) - `packages/stack-drizzle`: Drizzle ORM integration (`@cipherstash/stack-drizzle`), depends on `@cipherstash/stack` — **EQL v3 only**, on the package root (the v2 surface was removed and the old `./v3` subpath collapsed into `.`). Split out of `@cipherstash/stack`. -- `packages/stack-supabase`: Supabase integration (`@cipherstash/stack-supabase`), depends on `@cipherstash/stack` — `encryptedSupabase` (v2) and `encryptedSupabaseV3` (v3). Split out of `@cipherstash/stack`. +- `packages/stack-supabase`: Supabase integration (`@cipherstash/stack-supabase`), depends on `@cipherstash/stack` — **EQL v3 only**: `encryptedSupabase` is the v3 factory (`encryptedSupabaseV3` remains as a `@deprecated` alias). Split out of `@cipherstash/stack`. - `packages/nextjs`: Next.js helpers and Clerk integration (`./clerk` export) - `packages/utils`: Shared config (`utils/config`) and logger (`utils/logger`) - `packages/bench`: Performance / index-engagement benchmarks (private, not published) @@ -155,7 +155,7 @@ Three rules to remember when editing CI or pnpm config: - **Identity-aware encryption**: Authenticate the client as the end user with `OidcFederationStrategy` (`config.authStrategy`, re-exported from `@cipherstash/stack`), then chain `.withLockContext({ identityClaim })` on operations to bind the data key to a claim. The same claim must be used for encrypt and decrypt. (`LockContext.identify()` from `@cipherstash/stack/identity` is deprecated — the strategy now handles token acquisition; `.withLockContext()` also accepts a `LockContext`.) - **Integrations**: - **Drizzle ORM**: `types.*` column factories, `extractEncryptionSchema`, `createEncryptionOperators` from `@cipherstash/stack-drizzle` - - **Supabase**: `encryptedSupabase` (v2) / `encryptedSupabaseV3` (v3) from `@cipherstash/stack-supabase` + - **Supabase**: `encryptedSupabase` from `@cipherstash/stack-supabase` (EQL v3; `encryptedSupabaseV3` is a `@deprecated` alias) - **DynamoDB**: `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` ## Critical Gotchas (read before coding) diff --git a/README.md b/README.md index 0753d24e1..489c7f83a 100644 --- a/README.md +++ b/README.md @@ -20,11 +20,11 @@ **Encryption** ```typescript -import { Encryption, encryptedTable, encryptedColumn } from "@cipherstash/stack"; +import { Encryption, encryptedTable, types } from "@cipherstash/stack/v3"; -// 1. Define your schema +// 1. Define your schema — the column type fixes its query capabilities const users = encryptedTable("users", { - email: encryptedColumn("email").equality().freeTextSearch(), + email: types.TextSearch("email"), // equality + order/range + free-text search }); // 2. Initialize the client @@ -67,7 +67,7 @@ bun add @cipherstash/stack ## Features - **[Searchable encryption](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption)**: query encrypted data with equality, free text search, range, and [JSONB queries](https://cipherstash.com/docs/stack/cipherstash/encryption/searchable-encryption#jsonb-queries-with-searchablejson). -- **[Type-safe schema](https://cipherstash.com/docs/stack/cipherstash/encryption/schema)**: define encrypted tables and columns with `encryptedTable` / `encryptedColumn` +- **[Type-safe schema](https://cipherstash.com/docs/stack/cipherstash/encryption/schema)**: define encrypted tables and columns with `encryptedTable` and the `types.*` concrete-domain factories - **[Model & bulk operations](https://cipherstash.com/docs/stack/cipherstash/encryption/encrypt-decrypt#model-operations)**: encrypt and decrypt entire objects or batches with `encryptModel` / `bulkEncryptModels`. - **[Identity-aware encryption](https://cipherstash.com/docs/stack/cipherstash/encryption/identity)**: authenticate as the end user with `OidcFederationStrategy` and bind the data key to their identity with `.withLockContext({ identityClaim })` for policy-based access control. diff --git a/SECURITY.md b/SECURITY.md index 47b725910..ba3996d97 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -14,8 +14,8 @@ This repository is the CipherStash Stack monorepo for JavaScript/TypeScript. It | `@cipherstash/nextjs` | Next.js helpers | | `@cipherstash/migrate` | Plaintext-to-encrypted column migration tooling | | `@cipherstash/prisma-next` | Prisma Next integration (searchable field-level encryption for Postgres) | -| `@cipherstash/stack-drizzle` | Drizzle ORM integration for `@cipherstash/stack` (EQL v2 + v3) | -| `@cipherstash/stack-supabase` | Supabase integration for `@cipherstash/stack` (EQL v2 + v3) | +| `@cipherstash/stack-drizzle` | Drizzle ORM integration for `@cipherstash/stack` (EQL v3) | +| `@cipherstash/stack-supabase` | Supabase integration for `@cipherstash/stack` (EQL v3) | | `@cipherstash/wizard` | AI-powered encryption setup | **Security fixes are released for the latest release line of each package.** Security reports are welcome for any version, but fixes land in the latest release — if you are running an older major version, plan to upgrade to receive them. diff --git a/docs/reference/supabase-sdk.md b/docs/reference/supabase-sdk.md index 6561f255e..7294a8eb9 100644 --- a/docs/reference/supabase-sdk.md +++ b/docs/reference/supabase-sdk.md @@ -4,14 +4,18 @@ are transparently encrypted on mutations, `::jsonb`-cast on selects, encrypted in filter terms, and decrypted in results. -Two entry points, one query mechanism: +One entry point, EQL v3 only: | Entry point | Schema DSL | Column storage | |---|---|---| -| `encryptedSupabase` | `@cipherstash/stack/schema` (EQL v2) | `eql_v2_encrypted` composite | -| `encryptedSupabaseV3` | `@cipherstash/stack/eql/v3` (EQL v3) | native `public.eql_v3_*` domains | +| `encryptedSupabase` | `@cipherstash/stack/eql/v3` (EQL v3) | native `public.eql_v3_*` domains | -Both filter via **direct EQL operators over PostgREST**: the wrapper encrypts +`encryptedSupabaseV3` remains as a `@deprecated`, type-identical alias. The old +EQL v2 authoring wrapper — `encryptedSupabase({ encryptionClient, +supabaseClient })` — has been removed; the name now binds to the v3 factory +below. + +It filters via **direct EQL operators over PostgREST**: the wrapper encrypts the filter term and emits an ordinary `col term` filter, which resolves to the custom operator defined on the encrypted type (equality by HMAC, range by the ordering term — CLLW-OPE on `_ord` domains, block-ORE on `_ord_ore` — @@ -19,7 +23,7 @@ free-text by bloom-filter containment). ## Quick start (EQL v3) -`encryptedSupabaseV3` is an async factory that **introspects the database at +`encryptedSupabase` is an async factory that **introspects the database at connect time**: it detects EQL v3 columns by their Postgres domain, derives each column's encryption config from the domain, and builds the encryption client internally. Introspection needs a direct Postgres connection @@ -27,14 +31,14 @@ client internally. Introspection needs a direct Postgres connection run in a Worker or the browser. ```typescript -import { encryptedSupabaseV3 } from '@cipherstash/stack-supabase' +import { encryptedSupabase } from '@cipherstash/stack-supabase' // Introspects the database via options.databaseUrl or DATABASE_URL -const es = await encryptedSupabaseV3( +const es = await encryptedSupabase( process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!, ) -// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) +// or wrap an existing client: await encryptedSupabase(supabaseClient, options) await es.from('users').insert({ email: 'a@b.com', amount: 30 }) @@ -49,16 +53,14 @@ await es.from('users').select('id, amount').gte('amount', 10).lte('amount', 100) `from(tableName)` takes only the table name — no schema argument; column capabilities come from the introspected domains. -The builder surface is shared across v2 and v3: -`.select/.insert/.update/.upsert/.delete`, +The builder surface is `.select/.insert/.update/.upsert/.delete`, `.eq/.neq/.in/.is/.gt/.gte/.lt/.lte/.match/.or/.not/.filter`, transforms (`.order/.limit/.range/.single/.maybeSingle/.csv/.abortSignal/.throwOnError`), -plus `.withLockContext(lockContext)` and `.audit(config)` — with one fork: -free-text search. v2 exposes `.like/.ilike` (SQL wildcard matching); v3 -exposes `.matches()` (fuzzy bloom token search) on encrypted columns, keeps -`.contains()` for native (exact) containment on plaintext columns, and treats -`like`/`ilike` on an encrypted column as an approximate shim that delegates to -`.matches()` (see "v3 encoding details" below). +plus `.withLockContext(lockContext)` and `.audit(config)`. For free-text +search it exposes `.matches()` (fuzzy bloom token search) on encrypted +columns, keeps `.contains()` for native (exact) containment on plaintext +columns, and treats `like`/`ilike` on an encrypted column as an approximate +shim that delegates to `.matches()` (see "v3 encoding details" below). ### Typing (v3) @@ -68,14 +70,14 @@ tables against the database at construction: ```typescript import { encryptedTable, types } from '@cipherstash/stack/eql/v3' -import { encryptedSupabaseV3 } from '@cipherstash/stack-supabase' +import { encryptedSupabase } from '@cipherstash/stack-supabase' const users = encryptedTable('users', { email: types.TextSearch('email'), // public.eql_v3_text_search amount: types.IntegerOrd('amount'), // public.eql_v3_integer_ord }) -const es = await encryptedSupabaseV3( +const es = await encryptedSupabase( process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!, { schemas: { users } }, diff --git a/packages/bench/README.md b/packages/bench/README.md index af5c3e38b..cd5409815 100644 --- a/packages/bench/README.md +++ b/packages/bench/README.md @@ -3,8 +3,9 @@ Performance / index-engagement benchmarks for stack integrations. This package validates that each integration emits SQL that engages the canonical -EQL functional indexes (`eql_v2.hmac_256`, `eql_v2.bloom_filter`, `eql_v2.ste_vec`) -on a Supabase-shaped install (no operator classes). It runs in two layers: +EQL functional indexes (`eql_v3.eq_term`, `eql_v3.match_term`, +`eql_v3.to_ste_vec_query`) on a Supabase-shaped install (no operator classes). +It runs in two layers: 1. **EXPLAIN-shape tests** (`__tests__/`) — vitest tests that assert on `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` output. Pass/fail. Cheap. diff --git a/packages/migrate/README.md b/packages/migrate/README.md index ea1fb6549..30d03a961 100644 --- a/packages/migrate/README.md +++ b/packages/migrate/README.md @@ -43,7 +43,7 @@ Creates `cipherstash.cs_migrations` idempotently. Normally called by `stash eql Chunked, resumable, idempotent backfill of plaintext → encrypted. Per chunk, in a single transaction: select next page → encrypt via `client.bulkEncryptModels` → `UPDATE … FROM (VALUES …)` → `INSERT` a `backfill_checkpoint` event. Guards with `encrypted IS NULL` so re-runs never double-write. - `db`: a `pg.PoolClient` (the runner drives transactions on it). -- `encryptionClient`: your initialised `@cipherstash/stack` client (or anything that exposes `bulkEncryptModels(models, table)` returning `{ data } | { failure }`). For an EQL v3 column pass an `EncryptionV3` client (from `@cipherstash/stack/v3`) — it pins the v3 wire format; the engine itself is version-agnostic and writes whatever envelope the client produces. +- `encryptionClient`: your initialised `@cipherstash/stack` client (or anything that exposes `bulkEncryptModels(models, table)` returning `{ data } | { failure }`). For an EQL v3 column pass an `Encryption` client (from `@cipherstash/stack/v3`) — it pins the v3 wire format; the engine itself is version-agnostic and writes whatever envelope the client produces. - `tableSchema`: the `EncryptedTable` for the target table from your encryption client file. - `signal`: optional `AbortSignal`. If aborted between chunks, the backfill exits cleanly and leaves a resumable checkpoint. diff --git a/packages/stack-drizzle/README.md b/packages/stack-drizzle/README.md index ae5907733..7db86277d 100644 --- a/packages/stack-drizzle/README.md +++ b/packages/stack-drizzle/README.md @@ -78,7 +78,7 @@ are fixed by the `types.*` factory you choose — no per-column config object: ```ts import { pgTable, integer } from 'drizzle-orm/pg-core' -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { Encryption } from '@cipherstash/stack/v3' import { types, extractEncryptionSchema, @@ -92,7 +92,7 @@ const users = pgTable('users', { }) const schema = extractEncryptionSchema(users) -const client = await EncryptionV3({ schemas: [schema] }) +const client = await Encryption({ schemas: [schema] }) const ops = createEncryptionOperators(client) // Insert — encrypt models first (bulk helpers batch key operations diff --git a/packages/stack/README.md b/packages/stack/README.md index 8ca24e116..942f65fc9 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -57,7 +57,7 @@ The wizard will authenticate you, walk you through choosing a database connectio Define a table with concrete EQL v3 column types, build the typed client, and encrypt: ```typescript -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack/v3" import { encryptedTable, types } from "@cipherstash/stack/eql/v3" // Define a schema — the column type fixes its query capabilities @@ -66,7 +66,7 @@ const users = encryptedTable("users", { }) // Create a typed client -const client = await EncryptionV3({ schemas: [users] }) +const client = await Encryption({ schemas: [users] }) // Encrypt a value const encrypted = await client.encrypt("hello@example.com", { @@ -374,7 +374,7 @@ import { createEncryptionOperators, extractEncryptionSchema, } from "@cipherstash/stack-drizzle" -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack/v3" // Capabilities come from the concrete type — no flags to configure. const users = pgTable("users", { @@ -390,7 +390,7 @@ Derive the v3 schema from the table, build the typed client, and create the oper ```ts const usersSchema = extractEncryptionSchema(users) -const client = await EncryptionV3({ schemas: [usersSchema] }) +const client = await Encryption({ schemas: [usersSchema] }) const ops = createEncryptionOperators(client) const db = drizzle({ client: sqlClient }) @@ -478,7 +478,7 @@ explicit strategies cover the other cases: ```typescript import { OidcFederationStrategy } from "@cipherstash/stack" -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack/v3" // The callback is re-invoked on every (re-)federation and must return the // CURRENT third-party OIDC JWT. @@ -488,7 +488,7 @@ const strategy = OidcFederationStrategy.create( ) if (strategy.failure) throw new Error(strategy.failure.error.message) -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { authStrategy: strategy.data }, }) @@ -598,10 +598,10 @@ See the [Going to Production](https://cipherstash.com/docs/stack/deploy/going-to Pass config directly when initializing the client: ```typescript -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack/v3" import { users } from "./schema" -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -618,7 +618,7 @@ const client = await EncryptionV3({ Isolate encryption keys per tenant using keysets: ```typescript -const client = await EncryptionV3({ +const client = await Encryption({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" }, @@ -626,7 +626,7 @@ const client = await EncryptionV3({ }) // or by name -const client2 = await EncryptionV3({ +const client2 = await Encryption({ schemas: [users], config: { keyset: { name: "Company A" }, @@ -683,10 +683,10 @@ if (result.failure) { ## API Reference -### `EncryptionV3(config)` - Initialize the typed client +### `Encryption(config)` - Initialize the typed client ```typescript -function EncryptionV3(config: { +function Encryption(config: { schemas: AnyV3Table[] config?: ClientConfig }): Promise @@ -749,9 +749,9 @@ type UserEncrypted = InferEncrypted | Import Path | Provides | |-------|-----| -| `@cipherstash/stack/v3` | `EncryptionV3` typed client factory, `typedClient`, plus re-exports of the EQL v3 authoring DSL | +| `@cipherstash/stack/v3` | `Encryption` typed client factory (`EncryptionV3` is a `@deprecated` alias), `typedClient`, plus re-exports of the EQL v3 authoring DSL | | `@cipherstash/stack/eql/v3` | EQL v3 authoring DSL: `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, ...) | -| `@cipherstash/stack` | `Encryption` function (legacy v2 entry point), auth strategies | +| `@cipherstash/stack` | `Encryption` client factory, auth strategies | | `@cipherstash/stack/schema` | Legacy v2 schema builders (see [Legacy: EQL v2](#legacy-eql-v2)) | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) | @@ -763,31 +763,33 @@ depend on `@cipherstash/stack` (they are no longer subpaths of it): | Package | Provides | |-------|-----| | `@cipherstash/stack-drizzle` | EQL v3 Drizzle integration (package root, v3 only): `types` column factories, `createEncryptionOperators`, `extractEncryptionSchema`, `makeEqlV3Column`, `EncryptionOperatorError` | -| `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabaseV3` (and the legacy v2 `encryptedSupabase`) | +| `@cipherstash/stack-supabase` | Supabase integration (v3 only): `encryptedSupabase` (`encryptedSupabaseV3` is a `@deprecated` alias) | ## Legacy: EQL v2 Before the concrete-domain types above, encrypted columns were declared with chainable capability builders and stored in a single `eql_v2_encrypted` -composite column type. The scalar surface remains supported for existing -deployments, but new work should use EQL v3. Legacy searchable JSON cannot be -emitted by protect-ffi 0.30 and must migrate to v3 `types.Json`: +composite column type. **v2 is now a read path, not an authoring surface**: +`decrypt` / `decryptModel` still read stored v2 payloads, so existing +deployments keep working, but new work must use EQL v3. Legacy searchable JSON +cannot be emitted by protect-ffi 0.30 and must migrate to v3 `types.Json`: - **Client and schema**: `Encryption` from `@cipherstash/stack` with `encryptedColumn("email").equality().freeTextSearch().orderAndRange()` and - the builders from `@cipherstash/stack/schema`. v2 and v3 tables cannot be - mixed in one client. + the builders from `@cipherstash/stack/schema`. These are still exported but + `@deprecated` — do not author new schemas with them. v2 and v3 tables cannot + be mixed in one client. - **Query formatting**: v2 query terms can be rendered as strings with `returnType: 'composite-literal'` / `'escaped-composite-literal'` for string-based APIs. -- **Integrations**: the v2 Supabase surface is `encryptedSupabase`. There is no - v2 Drizzle surface any more — `@cipherstash/stack-drizzle` dropped - `encryptedType` and the v2 operators, so existing v2 Drizzle columns are - read-only: decrypt them through `@cipherstash/stack` and migrate to a v3 - domain. -- **DynamoDB still requires v2**: `encryptedDynamoDB` from - `@cipherstash/stack/dynamodb` works with the v2 API only — v3 support is - tracked in [#657](https://github.com/cipherstash/stack/issues/657). +- **Integrations are v3 only.** `encryptedSupabase` is the EQL v3 factory — + there is no v2 Supabase wrapper any more. Likewise + `@cipherstash/stack-drizzle` dropped `encryptedType` and the v2 operators. + Existing v2 columns reached through either adapter are read-only: decrypt + them through `@cipherstash/stack` and migrate to a v3 domain. +- **DynamoDB writes EQL v3 only.** `encryptedDynamoDB` from + `@cipherstash/stack/dynamodb` encrypts with `types.*` v3 tables; its decrypt + methods still accept a v2 table so previously stored items remain readable. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index dc4602e13..070bce612 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -334,8 +334,9 @@ export const EncryptionV3 = Encryption // Single import surface: re-export the v3 `types` namespace + table API + type // helpers so `@cipherstash/stack/v3` provides everything needed to author and -// use a schema. `Encryption` comes along for the same reason — it is the -// current name for what `EncryptionV3` aliases, so authoring a v3 schema and -// building its client should not need a second import specifier. -export { Encryption } +// use a schema. export * from '@/eql/v3' +// `Encryption` comes along for the same reason — it is the current name for +// what `EncryptionV3` aliases, so authoring a v3 schema and building its +// client should not need a second import specifier. +export { Encryption } diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 430b41952..7cf99621a 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -475,7 +475,7 @@ stash encrypt cutover --table users --column email In one transaction it renames `` → `_plaintext` and `_encrypted` → ``, advances the pending config to `encrypting`, activates it, and appends a `cut_over` event. With a Proxy URL configured (`--proxy-url` or `CIPHERSTASH_PROXY_URL`) it then calls `eql_v2.reload_config()` so Proxy picks up the new shape. -> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabaseV3` wrapper for Supabase — `encryptedSupabase` on the legacy v2 surface, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. +> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. > > **Known gap (v2).** The pending-configuration precondition is satisfied by `stash db push`. SDK-only users (who otherwise never need `db push`) must therefore run it once before `encrypt cutover`. This gap is specific to the legacy v2 path and is not being decoupled — EQL v3 columns sidestep it entirely (no configuration table, no cut-over; see above), which is how [issue #585](https://github.com/cipherstash/stack/issues/585) was resolved when v3 became the default. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 027deb6ab..057280571 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -1,6 +1,6 @@ --- name: stash-drizzle -description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the EncryptionV3 typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. +description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the Encryption typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. --- # CipherStash Stack - Drizzle ORM Integration @@ -125,7 +125,7 @@ Value families: `Integer`/`Smallint`/`Numeric`/`Real`/`Double` (`number`), `Bigi ```typescript import { extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" -import { EncryptionV3 } from "@cipherstash/stack/v3" +import { Encryption } from "@cipherstash/stack/v3" // Convert the Drizzle table definition to a CipherStash v3 schema const usersSchema = extractEncryptionSchema(usersTable) @@ -134,12 +134,12 @@ const usersSchema = extractEncryptionSchema(usersTable) ### 2. Initialize the Encryption Client ```typescript -const encryptionClient = await EncryptionV3({ +const encryptionClient = await Encryption({ schemas: [usersSchema], }) ``` -`EncryptionV3` returns a strongly-typed client: plaintext types are pinned to each column's domain, and query methods only accept queryable columns. +`Encryption` returns a strongly-typed client: plaintext types are pinned to each column's domain, and query methods only accept queryable columns. ### 3. Create Query Operators @@ -464,13 +464,13 @@ Update the encryption client to harvest the encrypted columns from the table: ```typescript // src/encryption/index.ts -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { Encryption } from '@cipherstash/stack/v3' import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' import { users } from '../db/schema' const usersEncryptionSchema = extractEncryptionSchema(users) -export const encryptionClient = await EncryptionV3({ schemas: [usersEncryptionSchema] }) +export const encryptionClient = await Encryption({ schemas: [usersEncryptionSchema] }) ``` Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted public.eql_v3_text_search;`. Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 749c38f22..31f939907 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -189,7 +189,7 @@ The SDK never logs plaintext data. | `@cipherstash/stack/errors` | `EncryptionErrorTypes`, `StackError`, error subtypes, `getErrorMessage` | | `@cipherstash/stack/types` | All TypeScript types | | `@cipherstash/stack-drizzle` | Drizzle ORM integration for EQL v3 schemas — the package root, EQL v3 only (see the `stash-drizzle` skill) | -| `@cipherstash/stack-supabase` | `encryptedSupabaseV3` wrapper for Supabase (see the `stash-supabase` skill) | +| `@cipherstash/stack-supabase` | `encryptedSupabase` wrapper for Supabase — EQL v3 only (see the `stash-supabase` skill) | | `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. | | `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items. See the `stash-dynamodb` skill | | `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | @@ -1006,7 +1006,7 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a | Target | Package / entry point | Skill | |---|---|---| | Drizzle ORM | `@cipherstash/stack-drizzle` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | -| Supabase | `encryptedSupabaseV3` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | +| Supabase | `encryptedSupabase` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | | Prisma | `@cipherstash/prisma-next` — searchable field-level encryption for Postgres | — | | DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — encrypt is **EQL v3 only**; decrypt still reads existing v2 items | `stash-dynamodb` | @@ -1054,6 +1054,6 @@ const users = encryptedTable("users", { const client = await Encryption({ schemas: [users] }) ``` -The scalar v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Supabase integration (`encryptedSupabase`), and `stash eql install --eql-version 2` — still exists for existing deployments, with two carve-outs. **Drizzle has no v2 surface at all**: `@cipherstash/stack-drizzle` dropped `encryptedType` and the `like`/`ilike` operators, so a v2 Drizzle column is read-only — decrypt it through `@cipherstash/stack` rather than through the adapter. **Legacy v2 `searchableJson()`** cannot be emitted by protect-ffi 0.30 (the selector envelope was removed), so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) +**v2 is a read path now, not an authoring surface.** `decrypt` / `decryptModel` still read stored v2 payloads, and `stash eql install --eql-version 2` still installs the v2 SQL, so existing deployments keep working. What is gone: the **Supabase** and **Drizzle** adapters are EQL v3 only — `encryptedSupabase` is the v3 factory (there is no v2 form), and `@cipherstash/stack-drizzle` dropped `encryptedType` and the `like`/`ilike` operators. A v2 column reached through either adapter is read-only; decrypt it through `@cipherstash/stack` instead. The `@cipherstash/stack/schema` builders (`encryptedColumn(...).equality()`, …) remain exported but are `@deprecated` — do not author new schemas with them. **Legacy v2 `searchableJson()`** cannot be emitted by protect-ffi 0.30 (the selector envelope was removed), so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) > **DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) now **encrypts EQL v3 only** — author tables with `types.*` from `@cipherstash/stack/eql/v3`. Its decrypt methods still accept a v2 table so previously stored v2 items remain readable. See the `stash-dynamodb` skill. diff --git a/skills/stash-prisma-next/SKILL.md b/skills/stash-prisma-next/SKILL.md index a19bae603..632af947c 100644 --- a/skills/stash-prisma-next/SKILL.md +++ b/skills/stash-prisma-next/SKILL.md @@ -117,7 +117,7 @@ export const db = postgres({ `cipherstashFromStack({ contractJson })` derives the v3 encryption schemas from the contract (one `public.eql_v3_*` domain per column), constructs the -`@cipherstash/stack` `EncryptionV3` client from your `CS_*` env vars or local +`@cipherstash/stack` `Encryption` client from your `CS_*` env vars or local profile, builds the SDK adapter, and returns ready-to-spread `extensions` and `middleware`. From 6c70e29d3442248f87509fb7e89634c96b5ab904 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 18:59:46 +1000 Subject: [PATCH 029/123] fix(wizard,cli): never rewrite a commented-out or already-encrypted ALTER The ALTER-to-encrypted matcher was comment-blind and its replacement is multi-line, so a commented-out statement kept the `-- ` on line 1 only and emitted a live DROP COLUMN. It also captured only the target type, so changing an already-encrypted column's domain produced the same ADD+DROP+RENAME over ciphertext. Both copies of the rewriter are fixed; skipped statements now carry a reason, EACCES on a migration dir is reported, and the wizard's run-migration prompt defaults to No after a rewrite. --- .changeset/rewriter-never-drops-ciphertext.md | 20 ++ .../src/__tests__/rewrite-migrations.test.ts | 265 ++++++++++++++++++ packages/cli/src/commands/db/install.ts | 10 +- .../cli/src/commands/db/rewrite-migrations.ts | 265 ++++++++++++++++-- packages/cli/src/commands/eql/migration.ts | 10 +- .../wizard/src/__tests__/post-agent.test.ts | 75 +++++ .../src/__tests__/rewrite-migrations.test.ts | 265 ++++++++++++++++++ packages/wizard/src/lib/post-agent.ts | 39 ++- packages/wizard/src/lib/rewrite-migrations.ts | 265 ++++++++++++++++-- 9 files changed, 1162 insertions(+), 52 deletions(-) create mode 100644 .changeset/rewriter-never-drops-ciphertext.md diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md new file mode 100644 index 000000000..df4565d47 --- /dev/null +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -0,0 +1,20 @@ +--- +'@cipherstash/wizard': patch +'stash': patch +--- + +Fix a data-loss bug in the Drizzle migration rewriter: a **commented-out** +`ALTER … SET DATA TYPE` was rewritten into executable SQL. The matcher was +comment-blind and the replacement is multi-line, so the author's `-- ` survived +on the first line only — the `DROP COLUMN` on the next line emitted live and +dropped a populated column. Statements inside a `--` line comment or a `/* … */` +block are now left as written, and a `--` inside a string literal no longer +reads as a comment. + +The sweep also refuses to rewrite a column the migration corpus already gives an +encrypted type, so changing a column's encrypted domain no longer drops a column +full of ciphertext. Skipped statements report why they were left alone. + +An unreadable migration directory (`EACCES`) is reported rather than silently +treated as empty, and the wizard's `Run the migration now?` prompt defaults to +No whenever the sweep rewrote or flagged anything. diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 173cba0fb..7a00161ca 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -507,6 +507,271 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) }) + // A multi-line replacement inherits the author's `-- ` on line 1 ONLY, so + // rewriting a commented-out ALTER turns lines 2+ — including DROP COLUMN — + // into live SQL. Commented SQL never runs; leave it exactly as written. + describe('commented-out statements', () => { + it.each([ + ['a line comment', '-- '], + ['an indented line comment', ' -- '], + ['a drizzle statement-breakpoint style prefix', '--> '], + ])('leaves an ALTER behind %s untouched', async (_label, prefix) => { + const original = `${prefix}ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` + const filePath = path.join(tmpDir, '0030_commented.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves an ALTER inside a block comment untouched', async () => { + const original = [ + '/* superseded by 0031', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_block.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('leaves an ALTER inside a NESTED block comment untouched', async () => { + const original = [ + '/* outer /* inner */', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_nested.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('does not report a commented-out near-miss', async () => { + const filePath = path.join(tmpDir, '0030_commented-using.sql') + fs.writeFileSync( + filePath, + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toEqual([]) + }) + + // The comment scan must not be fooled by `--` inside a string literal, or + // it would skip a live statement and leave broken SQL to fail at migrate. + it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + const filePath = path.join(tmpDir, '0030_literal.sql') + fs.writeFileSync( + filePath, + [ + `INSERT INTO "notes" ("body") VALUES ('a -- b');`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain( + 'ALTER TABLE "users" DROP COLUMN "email";', + ) + }) + }) + + // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and + // unlike the plaintext case there is nothing left anywhere to backfill from. + describe('columns that are already encrypted', () => { + it('refuses to rewrite a domain change on a column created encrypted', async () => { + const create = path.join(tmpDir, '0000_create.sql') + fs.writeFileSync( + create, + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('refuses to rewrite a domain change on a column ADDed encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The + // column it renamed onto is encrypted, so a later domain change on it is + // just as destructive. + it('follows a RENAME from a previous sweep', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + it('reports the destructive statement once, not twice', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + [ + '-- Custom SQL migration file, put your code below! --', + '', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // The scoping matters: encrypting `contacts.email` must not be blocked by + // an unrelated `users.email` that happens to share a column name. + it('scopes the check to the table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_contacts.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "contacts" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The ordinary case this rewrite exists for: plaintext today, encrypted + // after the ALTER. Nothing to preserve, so rewrite it. + it('still rewrites a plaintext column created in an earlier migration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" (\n\t"id" integer PRIMARY KEY,\n\t"email" text\n);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A commented-out ADD never ran, so it says nothing about the live schema. + it('ignores an encrypted ADD COLUMN that is commented out', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + }) + + // `options.skip` excludes a file from being EDITED, not from describing the + // schema the other files are altering. + it('honours an encrypted column defined in the skipped file', async () => { + const skipPath = path.join(tmpDir, '0000_install.sql') + fs.writeFileSync( + skipPath, + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: skipPath, + }, + ) + + expect(rewritten).toEqual([]) + expect(skipped[0]?.reason).toBe('already-encrypted') + }) + }) + it('handles multiple ALTER statements in one file', async () => { const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v2_encrypted;', diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index 8b230552f..60cfb0aae 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -34,7 +34,10 @@ import { detectSupabaseProject, type SupabaseProjectInfo, } from './detect.js' -import { rewriteEncryptedAlterColumns } from './rewrite-migrations.js' +import { + describeSkipReason, + rewriteEncryptedAlterColumns, +} from './rewrite-migrations.js' import { SUPABASE_EQL_MIGRATION_FILENAME, writeSupabaseEqlMigration, @@ -657,10 +660,11 @@ export async function generateDrizzleMigration( if (skipped.length > 0) { sweepIncomplete = true p.log.warn( - `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep could not rewrite automatically. Review and fix them before running your migrations:`, + `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep left alone. Review and fix them before running your migrations:`, ) - for (const { file, statement } of skipped) { + for (const { file, statement, reason } of skipped) { p.log.step(` - ${file}: ${statement}`) + p.log.step(` ${describeSkipReason(reason)}`) } } } catch (error) { diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 966cd54d8..7dcb9cc4e 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -116,12 +116,195 @@ function trimStatementPreamble(statement: string): string { return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() } +/** + * True when the character at `index` sits inside a SQL comment — a `--` line + * comment, or a (nestable) `/* … *\/` block comment. + * + * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is + * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a + * commented-out `-- ALTER TABLE … SET DATA TYPE …;` therefore leaves the + * author's `-- ` prefix on line 1 only — lines 2+, including `DROP COLUMN`, + * become live executable SQL and destroy the column. Commented SQL is inert by + * definition, so the only correct move is to leave it exactly as written. + * + * Single-quoted literals are skipped so a `'--'` inside a string cannot open a + * phantom comment. Dollar-quoted bodies are NOT tracked: a `--` inside one + * reads as a comment here, which can only make us skip a rewrite (the statement + * then fails loudly at migrate time), never perform a destructive one. + */ +function isInsideComment(sql: string, index: number): boolean { + let i = 0 + while (i < index) { + if (sql.startsWith('--', i)) { + const eol = sql.indexOf('\n', i) + if (eol === -1 || eol >= index) return true + i = eol + 1 + } else if (sql.startsWith('/*', i)) { + // Postgres block comments nest, so track depth rather than stopping at + // the first `*/` — a nested close would otherwise end the comment early + // and let the text after it read as live SQL. + let depth = 1 + let j = i + 2 + while (j < sql.length && depth > 0) { + if (sql.startsWith('/*', j)) { + depth += 1 + j += 2 + } else if (sql.startsWith('*/', j)) { + depth -= 1 + j += 2 + } else { + j += 1 + } + } + if (j > index) return true + i = j + } else if (sql[i] === "'") { + const close = sql.indexOf("'", i + 1) + // Unterminated, or the literal runs past `index`: `index` is inside a + // string, which is not a comment. + if (close === -1 || close >= index) return false + i = close + 1 + } else { + i += 1 + } + } + return false +} + +/** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ +const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` + +/** + * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a + * delimiter so a bare domain cannot match a prefix of a longer identifier. + */ +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` + +/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +const ADD_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** `ALTER TABLE … RENAME COLUMN "a" TO "b"` — $1/$2 table, $3 from, $4 to. */ +const RENAME_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+RENAME COLUMN\s+"([^"]+)"\s+TO\s+"([^"]+)"`, + 'gi', +) + +/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ +const CREATE_TABLE_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, + 'gi', +) + +/** `"col" ` inside a CREATE TABLE body — $1 column. */ +const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** Splits a `TABLE_REF` capture pair into its schema and table halves. */ +function tableOf( + first: string, + second: string | undefined, +): { schema?: string; table: string } { + // When schema-qualified (`"app"."users"`) the first capture is the schema and + // the second is the table; otherwise the first is the table. + return second === undefined + ? { table: first } + : { schema: first, table: second } +} + +/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +function columnKey(table: string, column: string, schema?: string): string { + return JSON.stringify([schema ?? '', table, column]) +} + +/** + * Index every column the migration corpus gives an ENCRYPTED type, so the + * rewrite can tell the change it exists for (plaintext → encrypted) from one it + * must never touch (encrypted → encrypted). + * + * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. + * A column whose encrypted domain merely changes (`types.TextEq` → + * `types.TextSearch`) matches just as well as a plaintext column, and the + * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext + * left anywhere to backfill from, so unlike the plaintext case the data is not + * recoverable from the application at all. Changing an encrypted column's domain + * changes its index terms, so the data has to be re-encrypted through the client + * regardless; the sweep flags the statement and leaves it for the user. + * + * The index is corpus-wide rather than ordered by migration: over-detecting + * "encrypted" costs a flagged statement the user must handle by hand, while + * under-detecting costs irrecoverable ciphertext. Only comment-free statements + * count, for the same reason {@link isInsideComment} exists. + */ +function indexEncryptedColumns(contents: readonly string[]): Set { + const encrypted = new Set() + + for (const sql of contents) { + for (const created of sql.matchAll(CREATE_TABLE_RE)) { + if (isInsideComment(sql, created.index)) continue + const { schema, table } = tableOf(created[1], created[2]) + for (const column of created[3].matchAll( + CREATE_TABLE_ENCRYPTED_COLUMN_RE, + )) { + encrypted.add(columnKey(table, column[1], schema)) + } + } + + for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { + if (isInsideComment(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + encrypted.add(columnKey(table, added[3], schema)) + } + + // A rename carries the column's type with it — and `__cipherstash_tmp` + // renamed onto the real name is exactly what a previous sweep of this very + // directory emitted. Run after ADD so that tmp column is already indexed. + for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { + if (isInsideComment(sql, renamed.index)) continue + const { schema, table } = tableOf(renamed[1], renamed[2]) + if (encrypted.has(columnKey(table, renamed[3], schema))) { + encrypted.add(columnKey(table, renamed[4], schema)) + } + } + } + + return encrypted +} + +/** Why a recognised ALTER-to-encrypted statement was left alone. */ +export type SkipReason = + /** Outside the strict matcher — hand-authored `USING`, or an unknown form. */ + | 'unrecognised-form' + /** The column already holds an encrypted domain; rewriting drops ciphertext. */ + | 'already-encrypted' + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ file: string /** The offending statement, verbatim (trimmed), for the user to review. */ statement: string + /** Why it was left alone — the caller turns this into user-facing guidance. */ + reason: SkipReason +} + +/** + * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print + * next to the statement. Lives here so every caller says the same thing — the + * two reasons need very different action from the user, and a single generic + * "could not rewrite automatically" hides that. + */ +export function describeSkipReason(reason: SkipReason): string { + switch (reason) { + case 'already-encrypted': + return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" + case 'unrecognised-form': + return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + } } /** Outcome of a sweep: the files rewritten, and near-misses left for review. */ @@ -154,27 +337,57 @@ export interface RewriteResult { * which keeps both columns alive across deploys. Each rewritten file carries a * header comment saying exactly this. * - * Returns {@link RewriteResult}: the files rewritten, plus `skipped` near-misses - * — statements that look like an ALTER-to-encrypted but fall outside the strict - * matcher (a hand-authored `SET DATA TYPE … USING …;`, or a future drizzle-kit - * form). Near-misses are left untouched on disk and surfaced non-fatally so the - * caller can tell the user to review them, rather than silently shipping broken - * SQL. + * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements + * left for a human — ones outside the strict matcher (a hand-authored + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting + * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. + * Both are left untouched on disk and surfaced non-fatally so the caller can + * tell the user to review them, rather than silently shipping broken SQL or + * destroying data. Statements sitting inside a SQL comment are inert and are + * neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, options: { skip?: string } = {}, ): Promise { - const entries = await readdir(outDir).catch(() => []) + const entries = await readdir(outDir).catch( + (error: NodeJS.ErrnoException) => { + // A missing directory is simply nothing to sweep. Anything else — EACCES + // above all — is a sweep that did NOT happen, and the caller reports it + // rather than letting the user believe their migrations were checked. + if (error.code === 'ENOENT') return [] as string[] + throw error + }, + ) const rewritten: string[] = [] const skipped: SkippedAlter[] = [] + const seen = new Set() + + /** Record a skip once — the strict pass and the broad scan can both find it. */ + const skip = (file: string, statement: string, reason: SkipReason): void => { + // Keyed on collapsed whitespace: the two passes trim the same statement + // by slightly different rules, and the strict pass runs first so its + // more specific reason is the one kept. + const key = `${file} :: ${statement.replace(/\s+/g, ' ')}` + if (seen.has(key)) return + seen.add(key) + skipped.push({ file, statement, reason }) + } - for (const entry of entries) { - if (!entry.endsWith('.sql')) continue + const sqlFiles = entries.filter((entry) => entry.endsWith('.sql')).sort() + const contents = new Map() + for (const entry of sqlFiles) { const filePath = join(outDir, entry) - if (options.skip && filePath === options.skip) continue + contents.set(filePath, await readFile(filePath, 'utf-8')) + } - const original = await readFile(filePath, 'utf-8') + // Built from the WHOLE corpus, including `options.skip`: a column's current + // type comes from the migrations that ran before this one, not just the files + // we are allowed to edit. + const encryptedColumns = indexEncryptedColumns([...contents.values()]) + + for (const [filePath, original] of contents) { + if (options.skip && filePath === options.skip) continue // Reset the regex's lastIndex — it's stateful on /g ALTER_COLUMN_TO_ENCRYPTED_RE.lastIndex = 0 @@ -187,11 +400,21 @@ export async function rewriteEncryptedAlterColumns( second: string | undefined, column: string, mangledType: string, + offset: number, ) => { - // When schema-qualified (`"app"."users"`) the first capture is the - // schema and the second is the table; otherwise the first is the table. - const schema = second === undefined ? undefined : first - const table = second === undefined ? first : second + // Commented-out SQL never runs, and a multi-line replacement would only + // inherit the `-- ` on its first line — leaving the rest live. + if (isInsideComment(original, offset)) return match + + const { schema, table } = tableOf(first, second) + + // Already encrypted: the ADD+DROP+RENAME would drop the ciphertext and + // there is no plaintext left to backfill from. Flag, never guess. + if (encryptedColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'already-encrypted') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. @@ -210,10 +433,14 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ - file: filePath, - statement: trimStatementPreamble(nearMiss[0]), - }) + const statement = trimStatementPreamble(nearMiss[0]) + // Anchor the comment test on the `SET DATA TYPE` itself: the match starts + // at the previous `;`, so its own offset sits before any preamble. + const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) + if (isInsideComment(updated, nearMiss.index + Math.max(keyword, 0))) { + continue + } + skip(filePath, statement, 'unrecognised-form') } } diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index 8923f7478..c2adf39c5 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -10,7 +10,10 @@ import { printNextSteps, SAFE_MIGRATION_NAME, } from '@/commands/db/install.js' -import { rewriteEncryptedAlterColumns } from '@/commands/db/rewrite-migrations.js' +import { + describeSkipReason, + rewriteEncryptedAlterColumns, +} from '@/commands/db/rewrite-migrations.js' import { detectPackageManager, execArgv, @@ -245,10 +248,11 @@ async function generateDrizzleEqlMigration( if (skipped.length > 0) { sweepIncomplete = true p.log.warn( - `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep could not rewrite automatically. Review and fix them before running your migrations:`, + `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep left alone. Review and fix them before running your migrations:`, ) - for (const { file, statement } of skipped) { + for (const { file, statement, reason } of skipped) { p.log.step(` - ${file}: ${statement}`) + p.log.step(` ${describeSkipReason(reason)}`) } } } catch (error) { diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index 0ee7fc51d..d3de8d2ca 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -1,3 +1,6 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' import { beforeEach, describe, expect, it, vi } from 'vitest' import { runPostAgentSteps } from '../lib/post-agent.js' import type { DetectedPackageManager } from '../lib/types.js' @@ -5,7 +8,15 @@ import type { DetectedPackageManager } from '../lib/types.js' // Mock the child_process module vi.mock('node:child_process') +// Only `confirm` is replaced — the log/spinner calls stay real so the module's +// output paths still execute. +vi.mock('@clack/prompts', async (importOriginal) => { + const actual = await importOriginal() + return { ...actual, confirm: vi.fn(async () => false) } +}) + import * as childProcess from 'node:child_process' +import * as p from '@clack/prompts' const bun: DetectedPackageManager = { name: 'bun', @@ -99,3 +110,67 @@ describe('runPostAgentSteps execution commands', () => { expect(commands).toContain('bunx stash eql install') }) }) + +// The sweep's own warning says "do NOT run the migration" on a populated table. +// Defaulting the very next prompt to Yes invites the mistake the warning exists +// to prevent — so a sweep that touched anything flips the default to No. +describe('drizzle migrate prompt after a destructive rewrite', () => { + let cwd: string + + const runDrizzle = () => + runPostAgentSteps({ + cwd, + integration: 'drizzle', + packageManager: bun, + gathered: { + installCommand: 'bun add @cipherstash/stack', + hasStashConfig: true, + usesProxy: false, + } as never, + }) + + beforeEach(() => { + cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'wizard-post-agent-')) + vi.mocked(p.confirm).mockClear() + }) + + it('defaults to Yes when the sweep changed nothing', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_init.sql'), + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(true) + }) + + it('defaults to No, and says why, when a file was rewritten', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0001_encrypt.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('DESTROYS data') + }) + + it('defaults to No when a statement was flagged rather than rewritten', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0001_using.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + }) +}) diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 41507fe86..862643b68 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -438,6 +438,271 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].statement).toBe(statement) }) + // A multi-line replacement inherits the author's `-- ` on line 1 ONLY, so + // rewriting a commented-out ALTER turns lines 2+ — including DROP COLUMN — + // into live SQL. Commented SQL never runs; leave it exactly as written. + describe('commented-out statements', () => { + it.each([ + ['a line comment', '-- '], + ['an indented line comment', ' -- '], + ['a drizzle statement-breakpoint style prefix', '--> '], + ])('leaves an ALTER behind %s untouched', async (_label, prefix) => { + const original = `${prefix}ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` + const filePath = path.join(tmpDir, '0030_commented.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves an ALTER inside a block comment untouched', async () => { + const original = [ + '/* superseded by 0031', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_block.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('leaves an ALTER inside a NESTED block comment untouched', async () => { + const original = [ + '/* outer /* inner */', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_nested.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('does not report a commented-out near-miss', async () => { + const filePath = path.join(tmpDir, '0030_commented-using.sql') + fs.writeFileSync( + filePath, + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toEqual([]) + }) + + // The comment scan must not be fooled by `--` inside a string literal, or + // it would skip a live statement and leave broken SQL to fail at migrate. + it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + const filePath = path.join(tmpDir, '0030_literal.sql') + fs.writeFileSync( + filePath, + [ + `INSERT INTO "notes" ("body") VALUES ('a -- b');`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain( + 'ALTER TABLE "users" DROP COLUMN "email";', + ) + }) + }) + + // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and + // unlike the plaintext case there is nothing left anywhere to backfill from. + describe('columns that are already encrypted', () => { + it('refuses to rewrite a domain change on a column created encrypted', async () => { + const create = path.join(tmpDir, '0000_create.sql') + fs.writeFileSync( + create, + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('refuses to rewrite a domain change on a column ADDed encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The + // column it renamed onto is encrypted, so a later domain change on it is + // just as destructive. + it('follows a RENAME from a previous sweep', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + it('reports the destructive statement once, not twice', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + [ + '-- Custom SQL migration file, put your code below! --', + '', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // The scoping matters: encrypting `contacts.email` must not be blocked by + // an unrelated `users.email` that happens to share a column name. + it('scopes the check to the table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_contacts.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "contacts" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The ordinary case this rewrite exists for: plaintext today, encrypted + // after the ALTER. Nothing to preserve, so rewrite it. + it('still rewrites a plaintext column created in an earlier migration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" (\n\t"id" integer PRIMARY KEY,\n\t"email" text\n);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A commented-out ADD never ran, so it says nothing about the live schema. + it('ignores an encrypted ADD COLUMN that is commented out', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + }) + + // `options.skip` excludes a file from being EDITED, not from describing the + // schema the other files are altering. + it('honours an encrypted column defined in the skipped file', async () => { + const skipPath = path.join(tmpDir, '0000_install.sql') + fs.writeFileSync( + skipPath, + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: skipPath, + }, + ) + + expect(rewritten).toEqual([]) + expect(skipped[0]?.reason).toBe('already-encrypted') + }) + }) + it('handles multiple ALTER statements in one file', async () => { const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v3_text_search;', diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index e66af3b1a..ecaf77a62 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -8,7 +8,7 @@ import { execSync } from 'node:child_process' import * as p from '@clack/prompts' import type { GatheredContext } from './gather.js' -import { sweepMigrationDirs } from './rewrite-migrations.js' +import { describeSkipReason, sweepMigrationDirs } from './rewrite-migrations.js' import type { DetectedPackageManager, Integration } from './types.js' interface PostAgentOptions { @@ -80,11 +80,19 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { // drizzle-kit just produced — those fail in Postgres (no cast from // text/numeric to an EQL domain). Covers the EQL v3 family the wizard now // scaffolds, and legacy eql_v2_encrypted. CIP-2991 + CIP-2994 + #693. - await rewriteEncryptedMigrations(cwd) - + const sweep = await rewriteEncryptedMigrations(cwd) + + // A rewritten file is a DROP+ADD in disguise, and a flagged statement is one + // the sweep could not make safe at all. Either way the next keystroke can + // destroy data, so the prompt says so and defaults to NO — an + // `initialValue: true` immediately under a "do NOT run the migration" + // warning invites exactly the mistake the warning is about. + const destructive = sweep.rewritten > 0 || sweep.skipped > 0 const shouldMigrate = await p.confirm({ - message: `Run the migration now? (${runner} drizzle-kit migrate)`, - initialValue: true, + message: destructive + ? `Run the migration now? (${runner} drizzle-kit migrate) — see the warnings above: this migration DESTROYS data on any table that already holds rows` + : `Run the migration now? (${runner} drizzle-kit migrate)`, + initialValue: !destructive, }) if (!p.isCancel(shouldMigrate) && shouldMigrate) { @@ -114,10 +122,20 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { } } -async function rewriteEncryptedMigrations(cwd: string): Promise { +/** + * Sweep the candidate migration directories, reporting what happened, and + * return the totals so the caller can decide how dangerous "run it now" is. + */ +async function rewriteEncryptedMigrations( + cwd: string, +): Promise<{ rewritten: number; skipped: number }> { const results = await sweepMigrationDirs(cwd, DRIZZLE_OUT_DIRS) + const totals = { rewritten: 0, skipped: 0 } for (const { dir, rewritten, skipped, error } of results) { + totals.rewritten += rewritten.length + totals.skipped += skipped.length + if (error) { p.log.warn(`Could not rewrite migrations in ${dir}: ${error}`) continue @@ -135,11 +153,16 @@ async function rewriteEncryptedMigrations(cwd: string): Promise { if (skipped.length > 0) { p.log.warn( - `${skipped.length} statement(s) look like an ALTER-to-encrypted the rewrite could not safely repair (e.g. a hand-authored SET DATA TYPE ... USING ...). Review them before migrating:`, + `${skipped.length} statement(s) look like an ALTER-to-encrypted that the rewrite left alone. Review them before migrating:`, ) - for (const s of skipped) p.log.step(` - ${s.file}: ${s.statement}`) + for (const s of skipped) { + p.log.step(` - ${s.file}: ${s.statement}`) + p.log.step(` ${describeSkipReason(s.reason)}`) + } } } + + return totals } async function runStep( diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index ce3ce1ab6..3c395016b 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -124,12 +124,195 @@ function trimStatementPreamble(statement: string): string { return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() } +/** + * True when the character at `index` sits inside a SQL comment — a `--` line + * comment, or a (nestable) `/* … *\/` block comment. + * + * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is + * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a + * commented-out `-- ALTER TABLE … SET DATA TYPE …;` therefore leaves the + * author's `-- ` prefix on line 1 only — lines 2+, including `DROP COLUMN`, + * become live executable SQL and destroy the column. Commented SQL is inert by + * definition, so the only correct move is to leave it exactly as written. + * + * Single-quoted literals are skipped so a `'--'` inside a string cannot open a + * phantom comment. Dollar-quoted bodies are NOT tracked: a `--` inside one + * reads as a comment here, which can only make us skip a rewrite (the statement + * then fails loudly at migrate time), never perform a destructive one. + */ +function isInsideComment(sql: string, index: number): boolean { + let i = 0 + while (i < index) { + if (sql.startsWith('--', i)) { + const eol = sql.indexOf('\n', i) + if (eol === -1 || eol >= index) return true + i = eol + 1 + } else if (sql.startsWith('/*', i)) { + // Postgres block comments nest, so track depth rather than stopping at + // the first `*/` — a nested close would otherwise end the comment early + // and let the text after it read as live SQL. + let depth = 1 + let j = i + 2 + while (j < sql.length && depth > 0) { + if (sql.startsWith('/*', j)) { + depth += 1 + j += 2 + } else if (sql.startsWith('*/', j)) { + depth -= 1 + j += 2 + } else { + j += 1 + } + } + if (j > index) return true + i = j + } else if (sql[i] === "'") { + const close = sql.indexOf("'", i + 1) + // Unterminated, or the literal runs past `index`: `index` is inside a + // string, which is not a comment. + if (close === -1 || close >= index) return false + i = close + 1 + } else { + i += 1 + } + } + return false +} + +/** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ +const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` + +/** + * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a + * delimiter so a bare domain cannot match a prefix of a longer identifier. + */ +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` + +/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +const ADD_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** `ALTER TABLE … RENAME COLUMN "a" TO "b"` — $1/$2 table, $3 from, $4 to. */ +const RENAME_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+RENAME COLUMN\s+"([^"]+)"\s+TO\s+"([^"]+)"`, + 'gi', +) + +/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ +const CREATE_TABLE_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, + 'gi', +) + +/** `"col" ` inside a CREATE TABLE body — $1 column. */ +const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** Splits a `TABLE_REF` capture pair into its schema and table halves. */ +function tableOf( + first: string, + second: string | undefined, +): { schema?: string; table: string } { + // When schema-qualified (`"app"."users"`) the first capture is the schema and + // the second is the table; otherwise the first is the table. + return second === undefined + ? { table: first } + : { schema: first, table: second } +} + +/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +function columnKey(table: string, column: string, schema?: string): string { + return JSON.stringify([schema ?? '', table, column]) +} + +/** + * Index every column the migration corpus gives an ENCRYPTED type, so the + * rewrite can tell the change it exists for (plaintext → encrypted) from one it + * must never touch (encrypted → encrypted). + * + * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. + * A column whose encrypted domain merely changes (`types.TextEq` → + * `types.TextSearch`) matches just as well as a plaintext column, and the + * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext + * left anywhere to backfill from, so unlike the plaintext case the data is not + * recoverable from the application at all. Changing an encrypted column's domain + * changes its index terms, so the data has to be re-encrypted through the client + * regardless; the sweep flags the statement and leaves it for the user. + * + * The index is corpus-wide rather than ordered by migration: over-detecting + * "encrypted" costs a flagged statement the user must handle by hand, while + * under-detecting costs irrecoverable ciphertext. Only comment-free statements + * count, for the same reason {@link isInsideComment} exists. + */ +function indexEncryptedColumns(contents: readonly string[]): Set { + const encrypted = new Set() + + for (const sql of contents) { + for (const created of sql.matchAll(CREATE_TABLE_RE)) { + if (isInsideComment(sql, created.index)) continue + const { schema, table } = tableOf(created[1], created[2]) + for (const column of created[3].matchAll( + CREATE_TABLE_ENCRYPTED_COLUMN_RE, + )) { + encrypted.add(columnKey(table, column[1], schema)) + } + } + + for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { + if (isInsideComment(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + encrypted.add(columnKey(table, added[3], schema)) + } + + // A rename carries the column's type with it — and `__cipherstash_tmp` + // renamed onto the real name is exactly what a previous sweep of this very + // directory emitted. Run after ADD so that tmp column is already indexed. + for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { + if (isInsideComment(sql, renamed.index)) continue + const { schema, table } = tableOf(renamed[1], renamed[2]) + if (encrypted.has(columnKey(table, renamed[3], schema))) { + encrypted.add(columnKey(table, renamed[4], schema)) + } + } + } + + return encrypted +} + +/** Why a recognised ALTER-to-encrypted statement was left alone. */ +export type SkipReason = + /** Outside the strict matcher — hand-authored `USING`, or an unknown form. */ + | 'unrecognised-form' + /** The column already holds an encrypted domain; rewriting drops ciphertext. */ + | 'already-encrypted' + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ file: string /** The offending statement, verbatim (trimmed), for the user to review. */ statement: string + /** Why it was left alone — the caller turns this into user-facing guidance. */ + reason: SkipReason +} + +/** + * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print + * next to the statement. Lives here so every caller says the same thing — the + * two reasons need very different action from the user, and a single generic + * "could not rewrite automatically" hides that. + */ +export function describeSkipReason(reason: SkipReason): string { + switch (reason) { + case 'already-encrypted': + return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" + case 'unrecognised-form': + return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + } } /** Outcome of a sweep: the files rewritten, and near-misses left for review. */ @@ -162,27 +345,57 @@ export interface RewriteResult { * which keeps both columns alive across deploys. Each rewritten file carries a * header comment saying exactly this. * - * Returns {@link RewriteResult}: the files rewritten, plus `skipped` near-misses - * — statements that look like an ALTER-to-encrypted but fall outside the strict - * matcher (a hand-authored `SET DATA TYPE … USING …;`, or a future drizzle-kit - * form). Near-misses are left untouched on disk and surfaced non-fatally so the - * caller can tell the user to review them, rather than silently shipping broken - * SQL. + * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements + * left for a human — ones outside the strict matcher (a hand-authored + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting + * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. + * Both are left untouched on disk and surfaced non-fatally so the caller can + * tell the user to review them, rather than silently shipping broken SQL or + * destroying data. Statements sitting inside a SQL comment are inert and are + * neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, options: { skip?: string } = {}, ): Promise { - const entries = await readdir(outDir).catch(() => []) + const entries = await readdir(outDir).catch( + (error: NodeJS.ErrnoException) => { + // A missing directory is simply nothing to sweep. Anything else — EACCES + // above all — is a sweep that did NOT happen, and the caller reports it + // rather than letting the user believe their migrations were checked. + if (error.code === 'ENOENT') return [] as string[] + throw error + }, + ) const rewritten: string[] = [] const skipped: SkippedAlter[] = [] + const seen = new Set() + + /** Record a skip once — the strict pass and the broad scan can both find it. */ + const skip = (file: string, statement: string, reason: SkipReason): void => { + // Keyed on collapsed whitespace: the two passes trim the same statement + // by slightly different rules, and the strict pass runs first so its + // more specific reason is the one kept. + const key = `${file} :: ${statement.replace(/\s+/g, ' ')}` + if (seen.has(key)) return + seen.add(key) + skipped.push({ file, statement, reason }) + } - for (const entry of entries) { - if (!entry.endsWith('.sql')) continue + const sqlFiles = entries.filter((entry) => entry.endsWith('.sql')).sort() + const contents = new Map() + for (const entry of sqlFiles) { const filePath = join(outDir, entry) - if (options.skip && filePath === options.skip) continue + contents.set(filePath, await readFile(filePath, 'utf-8')) + } - const original = await readFile(filePath, 'utf-8') + // Built from the WHOLE corpus, including `options.skip`: a column's current + // type comes from the migrations that ran before this one, not just the files + // we are allowed to edit. + const encryptedColumns = indexEncryptedColumns([...contents.values()]) + + for (const [filePath, original] of contents) { + if (options.skip && filePath === options.skip) continue // Reset the regex's lastIndex — it's stateful on /g ALTER_COLUMN_TO_ENCRYPTED_RE.lastIndex = 0 @@ -195,11 +408,21 @@ export async function rewriteEncryptedAlterColumns( second: string | undefined, column: string, mangledType: string, + offset: number, ) => { - // When schema-qualified (`"app"."users"`) the first capture is the - // schema and the second is the table; otherwise the first is the table. - const schema = second === undefined ? undefined : first - const table = second === undefined ? first : second + // Commented-out SQL never runs, and a multi-line replacement would only + // inherit the `-- ` on its first line — leaving the rest live. + if (isInsideComment(original, offset)) return match + + const { schema, table } = tableOf(first, second) + + // Already encrypted: the ADD+DROP+RENAME would drop the ciphertext and + // there is no plaintext left to backfill from. Flag, never guess. + if (encryptedColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'already-encrypted') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. @@ -218,10 +441,14 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ - file: filePath, - statement: trimStatementPreamble(nearMiss[0]), - }) + const statement = trimStatementPreamble(nearMiss[0]) + // Anchor the comment test on the `SET DATA TYPE` itself: the match starts + // at the previous `;`, so its own offset sits before any preamble. + const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) + if (isInsideComment(updated, nearMiss.index + Math.max(keyword, 0))) { + continue + } + skip(filePath, statement, 'unrecognised-form') } } From 38665ce9325a8bfe4f181bb32ec49b978a12ca33 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 18:59:58 +1000 Subject: [PATCH 030/123] fix(stack)!: make Encryption's overloads agree with what it returns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v3 overload accepted `config: { eqlVersion: 2 }` and an empty schema tuple, neither of which yields the typed client it promised — the first returns the nominal client at runtime (silently discarding decryptModel's table and lock context), the second throws. It now takes a non-empty tuple and V3ClientConfig. `EncryptionClientFor` is exported because `ReturnType` reads the last overload and so always names the nominal client. decryptModel/bulkDecryptModels gain the table-less overload the runtime already supports — the path for reading legacy v2 models, whose table cannot belong to a v3 schema tuple. v2-decrypt-compat now covers a v3 client reading v2 ciphertext rather than only round-tripping v2 through the emission path being removed. --- .changeset/stack-audit-on-decrypt.md | 23 ++- .../prisma-next/src/stack/from-stack-v3.ts | 34 +++- .../__tests__/encryption-overloads.test-d.ts | 126 +++++++++++++ .../v2-decrypt-compat.integration.test.ts | 107 ++++++++++- packages/stack/src/encryption/index.ts | 11 +- packages/stack/src/encryption/v3.ts | 169 +++++++++++++----- packages/stack/src/types-public.ts | 1 + packages/stack/src/types.ts | 16 ++ 8 files changed, 423 insertions(+), 64 deletions(-) create mode 100644 packages/stack/__tests__/encryption-overloads.test-d.ts diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md index dd21fd17c..de4b34cdf 100644 --- a/.changeset/stack-audit-on-decrypt.md +++ b/.changeset/stack-audit-on-decrypt.md @@ -32,10 +32,25 @@ already passing EQL v3 tables to plain `Encryption`, you now receive the typed client rather than the nominal one — its `decryptModel` / `bulkDecryptModels` return type changes, and the two-argument form reconstructs `Date` columns from `cast_as` instead of leaving them as ISO strings. Code that read those columns as -strings needs updating. As part of this collapse -`EncryptionV3` no longer independently pins the wire format — like `Encryption`, -it now honours an explicit `config.eqlVersion` (the retained migration escape -hatch). The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 +strings needs updating. + +The v3 overload takes a non-empty tuple of tables and a `V3ClientConfig` — +`ClientConfig` without the deprecated `eqlVersion` escape hatch. So +`Encryption({ schemas: [] })` no longer type-checks (it used to compile and then +throw), and `config: { eqlVersion: 2 }` selects the nominal overload, which is +the client you actually get back. Callers passing a plain `AnyV3Table[]` rather +than an array literal must narrow it to `readonly [AnyV3Table, ...AnyV3Table[]]`. +`Awaited>` names the nominal client whatever you +pass, because `ReturnType` reads the last overload; use the exported +`EncryptionClientFor` to name the client for a schema tuple. + +`decryptModel` / `bulkDecryptModels` on the typed client also accept a call with +no table, matching the runtime, which has always allowed it — that is the path +for reading models written before the upgrade, above all legacy EQL v2 ones, +whose table cannot be a member of a v3 schema tuple. Prefer the two-argument +form whenever the table is registered. + +The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 builders remain available (now marked `@deprecated`) for reading and migrating legacy v2 data; the client authors EQL v3 only. Their full removal is deferred to a later PR. diff --git a/packages/prisma-next/src/stack/from-stack-v3.ts b/packages/prisma-next/src/stack/from-stack-v3.ts index 28ab70452..2f569b550 100644 --- a/packages/prisma-next/src/stack/from-stack-v3.ts +++ b/packages/prisma-next/src/stack/from-stack-v3.ts @@ -25,7 +25,7 @@ */ import type { AnyV3Table } from '@cipherstash/stack/eql/v3' -import type { ClientConfig } from '@cipherstash/stack/types' +import type { V3ClientConfig } from '@cipherstash/stack/types' import { EncryptionV3, type TypedEncryptionClient } from '@cipherstash/stack/v3' import type { SqlMiddleware, @@ -55,8 +55,14 @@ export interface CipherstashFromStackV3Options { */ readonly schemasV3?: ReadonlyArray - /** Pass-through to `EncryptionV3({ config })` (keyset overrides, logging, …). */ - readonly encryptionConfig?: ClientConfig + /** + * Pass-through to `EncryptionV3({ config })` (keyset overrides, logging, …). + * + * `V3ClientConfig`, not `ClientConfig`: this package is EQL v3 only, and the + * legacy `eqlVersion: 2` escape hatch returns the nominal (untyped) client at + * runtime, which is not what this entry point hands back. + */ + readonly encryptionConfig?: V3ClientConfig } export interface CipherstashFromStackV3Result { @@ -83,8 +89,11 @@ export async function cipherstashFromStack( ) } - const derived = deriveStackSchemasV3(opts.contractJson) - if (derived.length === 0) { + // Destructured rather than length-checked so the non-emptiness survives into + // the type layer: `Encryption` requires a non-empty schema tuple (an empty one + // used to type-check and then throw at runtime). + const [firstDerived, ...restDerived] = deriveStackSchemasV3(opts.contractJson) + if (firstDerived === undefined) { throw new Error( 'cipherstashFromStack: no v3 cipherstash columns found in contract.json. ' + 'Declare at least one v3 `cipherstash.*()` column (e.g. `cipherstash.TextSearch()`) in prisma/schema.prisma ' + @@ -92,7 +101,10 @@ export async function cipherstashFromStack( ) } - const schemas = resolveV3Schemas(derived, opts.schemasV3) + const schemas = resolveV3Schemas( + [firstDerived, ...restDerived], + opts.schemasV3, + ) const encryptionClient = await EncryptionV3({ schemas, @@ -132,15 +144,18 @@ function collectNonV3CipherstashCodecIds( return [...ids].sort() } +/** A schema list carrying its non-emptiness in the type, as `Encryption` wants. */ +type NonEmptyV3Schemas = readonly [AnyV3Table, ...AnyV3Table[]] + /** * Validate contract-declared tables against their overrides (exact * domain identity) and append override-only tables — same merge * semantics as the v2 `resolveSchemas`. */ function resolveV3Schemas( - derived: ReadonlyArray, + derived: NonEmptyV3Schemas, override: ReadonlyArray | undefined, -): ReadonlyArray { +): NonEmptyV3Schemas { if (override === undefined || override.length === 0) return derived const derivedByName = new Map(derived.map((t) => [t.tableName, t])) @@ -153,7 +168,8 @@ function resolveV3Schemas( } return [ - ...derived, + derived[0], + ...derived.slice(1), ...override.filter((t) => !derivedByName.has(t.tableName)), ] } diff --git a/packages/stack/__tests__/encryption-overloads.test-d.ts b/packages/stack/__tests__/encryption-overloads.test-d.ts new file mode 100644 index 000000000..e3afd5929 --- /dev/null +++ b/packages/stack/__tests__/encryption-overloads.test-d.ts @@ -0,0 +1,126 @@ +/** + * Type-level contract for the `Encryption` overload pair. + * + * `Encryption` is overloaded — an all-v3 schema tuple yields the typed client, + * everything else yields the nominal one — and the two are NOT mutually + * assignable. Overload selection is therefore load-bearing public API, and none + * of it is exercised by a runtime test. Each case below was a real defect found + * in review (#772): a call that type-checked as the typed client but returned + * the nominal one at runtime, a schema set the types accepted and the runtime + * threw on, and a `ReturnType` idiom that silently resolves to the wrong client. + */ +import { describe, expectTypeOf, it } from 'vitest' +import { Encryption, type EncryptionClient } from '@/encryption' +import { + type EncryptionClientFor, + encryptedTable, + type TypedEncryptionClient, +} from '@/encryption/v3' +import { types } from '@/eql/v3' +import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' + +const users = encryptedTable('users', { + email: types.TextEq('email'), + createdAt: types.TimestampOrd('created_at'), +}) + +const usersV2 = encryptedTableV2('users_v2', { + email: encryptedColumn('email').equality(), +}) + +describe('overload selection', () => { + it('an all-v3 schema tuple yields the typed client', async () => { + const client = await Encryption({ schemas: [users] }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) + + it('a v2 schema set yields the nominal client', async () => { + const client = await Encryption({ schemas: [usersV2] }) + expectTypeOf(client).toEqualTypeOf() + }) + + // S-6: `readonly []` satisfies `readonly AnyV3Table[]`, so an empty schema set + // used to compile and then throw at runtime. Both overloads now require at + // least one table. + it('rejects an empty schema set', () => { + // @ts-expect-error - at least one encryptedTable is required + Encryption({ schemas: [] }) + }) + + // S-4: forcing v2 wire over v3 schemas returns the NOMINAL client at runtime + // (the typed client cannot author v3 columns in v2 mode). The types used to + // claim the typed client, so `decryptModel(row, table, lockContext)` compiled + // and then silently dropped `table` and `lockContext`. + it('forcing eqlVersion 2 over v3 schemas yields the nominal client', async () => { + const client = await Encryption({ + schemas: [users], + config: { eqlVersion: 2 }, + }) + expectTypeOf(client).toEqualTypeOf() + // The typed-only three-arg decrypt is therefore not available — which is + // exactly what the runtime does. + // @ts-expect-error - the nominal client takes the model alone + client.decryptModel({ email: 'x' }, users) + }) + + it('an explicit eqlVersion 3 keeps the typed client', async () => { + const client = await Encryption({ + schemas: [users], + config: { eqlVersion: 3 }, + }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) +}) + +describe('naming the client type', () => { + // S-2: `ReturnType` reads the LAST overload, so this idiom resolves to the + // nominal client no matter what schemas you pass. Pinned rather than fixed — + // overload order cannot satisfy both forms — so the surprise is documented and + // cannot change silently. + it('ReturnType resolves to the NOMINAL client', () => { + expectTypeOf< + Awaited> + >().toEqualTypeOf() + }) + + it('EncryptionClientFor names the typed client for a v3 tuple', async () => { + const client: EncryptionClientFor = + await Encryption({ schemas: [users] }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) + + it('EncryptionClientFor falls back to the nominal client', () => { + expectTypeOf< + EncryptionClientFor + >().toEqualTypeOf() + }) +}) + +describe('reading legacy EQL v2 models through a typed client', () => { + // The compatibility promise: a v3-configured client must read rows written + // before the upgrade. Their table is not — and cannot be — a member of the + // client's v3 schema tuple, so the table-less call has to type-check. + it('accepts a table-less decryptModel', async () => { + const client = await Encryption({ schemas: [users] }) + expectTypeOf(client.decryptModel).toBeCallableWith({ + pk: 'a', + email: { k: 'ct', v: 2, c: 'ciphertext', i: { t: 'legacy', c: 'email' } }, + }) + expectTypeOf(client.bulkDecryptModels).toBeCallableWith([ + { pk: 'a', email: { k: 'ct', v: 2, c: 'x', i: { t: 'l', c: 'email' } } }, + ]) + }) + + it('still rejects a table this client was not built with', async () => { + const client = await Encryption({ schemas: [users] }) + const unregistered = encryptedTable('other', { note: types.TextEq('note') }) + // @ts-expect-error - `other` is not a member of the client's schema tuple + client.decryptModel({ note: 'x' }, unregistered) + }) +}) diff --git a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts index 46567cfaa..c1a96813b 100644 --- a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts +++ b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts @@ -1,11 +1,11 @@ /** - * Acceptance #1a + #1b — EQL v2 READ compatibility after the v3 collapse (PR 3). + * Acceptance #1a + #1b + #1c — EQL v2 READ compatibility after the v3 collapse. * - * PR 3 makes EQL v3 the only generation the client authors/writes, but MUST keep - * reading previously stored EQL v2 payloads — on the core client (guardrail 1) - * AND through the DynamoDB adapter (guardrail 2). This suite mints v2 data with a - * v2-mode `Encryption` client (the retained `config: { eqlVersion: 2 }` escape - * hatch) and proves it still round-trips: + * The client authors EQL v3 only, but MUST keep reading previously stored EQL v2 + * payloads — on the core client (guardrail 1) AND through the DynamoDB adapter + * (guardrail 2). This suite mints v2 data with a v2-mode `Encryption` client (the + * retained `config: { eqlVersion: 2 }` escape hatch) and proves it still + * round-trips: * * - #1a: a v2 ciphertext + a v2 model decrypt through the collapsed `Encryption` * client's `decrypt` / `decryptModel`. @@ -13,15 +13,28 @@ * `toEncryptedDynamoItem(payload, attrs, false)` — decrypts through * `encryptedDynamoDB(...).decryptModel(item, v2Table)`, exercising the v2 * envelope reconstruction (`toItemWithEqlPayloads`, `v === 2` / `k: 'ct'`). + * - #1c: the SAME v2 payloads decrypt through a **v3-configured** client that + * has never heard of the v2 table. * - * Both fail if the v2 read path is removed. Live: requires real ZeroKMS - * credentials (the integration harness provisions them and throws otherwise). + * #1c is the invariant customers actually depend on and the reason this file + * cannot be a v2-only suite: after the removal their client is v3, their stored + * data is v2, and #1a/#1b would keep passing even if a v3 client had lost the + * ability to read v2 — because both mint AND read through the v2-mode client. + * Whatever eventually deletes the `eqlVersion: 2` minting path must keep #1c + * alive (against a checked-in ciphertext fixture, or a raw `EncryptConfig`), + * not delete it along with the escape hatch. + * + * Live: requires real ZeroKMS credentials (the integration harness provisions + * them and throws otherwise). The credential-free half of the v2 read path — the + * DynamoDB envelope split/reconstruction over static payloads — is covered by + * `__tests__/dynamodb/helpers.test.ts`, which needs no network at all. */ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' import { encryptedDynamoDB } from '@/dynamodb' import { toEncryptedDynamoItem } from '@/dynamodb/helpers' import type { EncryptionClient } from '@/encryption' +import { encryptedTable as encryptedTableV3, types as typesV3 } from '@/eql/v3' import { Encryption } from '@/index' // The deprecated v2 authoring builders remain for reading/migrating legacy data. import { encryptedColumn, encryptedTable } from '@/schema' @@ -32,17 +45,34 @@ const usersV2 = encryptedTable('v2_read_compat_users', { email: encryptedColumn('email').equality(), }) +// A DIFFERENT table, in v3 builders, so the v3 client below carries no knowledge +// of the v2 table whatsoever — reading v2 data must not depend on the v2 schema +// still being registered. +const unrelatedV3 = encryptedTableV3('v2_read_compat_unrelated_v3', { + note: typesV3.TextEq('note'), +}) + const SECRET = 'ada@example.com' // A v2-mode client: the explicit `eqlVersion: 2` escape hatch forces v2 wire so // this suite mints genuinely-v2 payloads, independent of schema auto-detection. let v2Client: EncryptionClient +// The client a customer is left with after the removal: v3 schemas, default +// (v3) wire version. Every read in #1c goes through this one. +// +// Typed through a thunk rather than annotated `EncryptionClient`: a concrete v3 +// schema set selects the TYPED overload, and `TypedEncryptionClient` is not +// assignable to the nominal `EncryptionClient`. +const makeV3Client = () => Encryption({ schemas: [unrelatedV3] }) +let v3Client: Awaited> + beforeAll(async () => { v2Client = await Encryption({ schemas: [usersV2], config: { eqlVersion: 2 }, }) + v3Client = await Encryption({ schemas: [unrelatedV3] }) }) describe('#1a — core client reads a stored EQL v2 payload', () => { @@ -92,3 +122,64 @@ describe('#1b — DynamoDB adapter reads a stored EQL v2 item', () => { expect(decrypted).toMatchObject({ pk: 'a', email: SECRET }) }, 30000) }) + +// The real-world shape of the compatibility promise: the customer upgraded, so +// their client is v3-configured and knows nothing about the v2 table — but the +// rows already in their database are v2. +describe('#1c — a v3-configured client reads stored EQL v2 payloads', () => { + it('decrypts a v2 ciphertext minted before the upgrade', async () => { + const encrypted = unwrapResult( + await v2Client.encrypt(SECRET, { table: usersV2, column: usersV2.email }), + ) + // Guard against a false pass: this must be a genuine v2 payload, or the + // test proves nothing about v2 compatibility. + expect(encrypted).toMatchObject({ v: 2 }) + + const decrypted = unwrapResult(await v3Client.decrypt(encrypted)) + expect(decrypted).toBe(SECRET) + }, 30000) + + it('decrypts a v2 model minted before the upgrade', async () => { + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + expect(encryptedModel.email).toMatchObject({ v: 2 }) + + const decrypted = unwrapResult(await v3Client.decryptModel(encryptedModel)) + expect(decrypted).toEqual({ pk: 'a', email: SECRET }) + }, 30000) + + it('bulk-decrypts v2 ciphertexts minted before the upgrade', async () => { + const encrypted = unwrapResult( + await v2Client.bulkEncrypt( + [ + { id: '1', plaintext: SECRET }, + { id: '2', plaintext: 'grace@example.com' }, + ], + { table: usersV2, column: usersV2.email }, + ), + ) + + const decrypted = unwrapResult(await v3Client.bulkDecrypt(encrypted)) + expect(decrypted).toEqual([ + { id: '1', data: SECRET }, + { id: '2', data: 'grace@example.com' }, + ]) + }, 30000) + + it('decrypts a stored v2 DynamoDB item through a v3-configured adapter', async () => { + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + const storedV2Item = toEncryptedDynamoItem(encryptedModel, ['email'], false) + + // The adapter is built on the v3 client; only the TABLE argument still + // describes the legacy v2 shape, because that is what the item on disk is. + const dynamo = encryptedDynamoDB({ encryptionClient: v3Client }) + const decrypted = unwrapResult( + await dynamo.decryptModel(storedV2Item, usersV2), + ) + + expect(decrypted).toMatchObject({ pk: 'a', email: SECRET }) + }, 30000) +}) diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index 45ffe5a9a..5a47a930a 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -30,6 +30,7 @@ import type { KeysetIdentifier, Plaintext, ScalarQueryTerm, + V3ClientConfig, } from '@/types' import { hasBuildColumnKeyMap } from '@/types' import { logger } from '@/utils/logger' @@ -864,9 +865,15 @@ export function __resetStrategyDeprecationWarningForTests(): void { // Overload 1 — v3-typed: an array literal of concrete EQL v3 tables (from // `@cipherstash/stack/v3`) yields the strongly-typed {@link TypedEncryptionClient}, // the collapse of the former `EncryptionV3`. The wire format is forced to v3. -export function Encryption(config: { +// +// The schema tuple is constrained NON-EMPTY: `readonly AnyV3Table[]` admits +// `readonly []`, so `Encryption({ schemas: [] })` type-checked and then threw at +// runtime. The nominal overload has always required at least one table. +export function Encryption< + const S extends readonly [AnyV3Table, ...AnyV3Table[]], +>(config: { schemas: S - config?: ClientConfig + config?: V3ClientConfig }): Promise> // Overload 2 — nominal: loose/dynamic schemas (introspection-derived, e.g. // stack-supabase) or EQL v2 tables yield the generation-neutral diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 070bce612..3ba2f19ce 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -14,6 +14,7 @@ import type { LockContextInput } from '@/identity' import type { BulkDecryptPayload, BulkEncryptPayload, + Decrypted, Encrypted, EncryptedReturnType, EncryptOptions, @@ -116,12 +117,32 @@ export interface TypedEncryptionClient { lockContext?: LockContextInput, ): AuditableDecryptModelOperation> + /** + * Table-less form, mirroring the nominal {@link EncryptionClient}: decrypt + * whatever encrypted fields the model carries, with no `Date` reconstruction + * (there is no `cast_as` to reconstruct from) and no precise plaintext shape. + * + * This is the read path for rows that predate this client's schemas — legacy + * **EQL v2** models above all, whose table is not, and cannot be, a member of + * `S`. The runtime has always accepted the one-arg call; without this + * signature the type layer forbids the very compatibility the client + * promises. Prefer the two-arg form whenever the table IS registered. + */ + decryptModel>( + input: T, + ): AuditableDecryptModelOperation> + bulkDecryptModels
>( input: Array, table: Table, lockContext?: LockContextInput, ): AuditableDecryptModelOperation>> + /** Table-less bulk form — see the one-arg {@link decryptModel} overload. */ + bulkDecryptModels>( + input: Array, + ): AuditableDecryptModelOperation>> + // Parity passthroughs — not v3-strengthened, delegated as-is. bulkEncrypt( plaintexts: BulkEncryptPayload, @@ -258,6 +279,80 @@ export function typedClient( return client.encryptQuery(plaintextOrTerms as never, opts as never) } + // Overloaded declarations for the same reason as `encryptQuery` above: the + // table-ful and table-less forms have different return types, and a single + // arrow in the object literal cannot present both. + function decryptModel< + Table extends S[number], + T extends Record, + >( + input: T, + table: Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation> + function decryptModel>( + input: T, + ): AuditableDecryptModelOperation> + function decryptModel( + input: Record, + table?: AnyV3Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation { + // `table` is absent on a nominal-style one-arg call (see `passthroughRow`). + // Given a table: reconstruct dates from its cast_as, or — if it was never + // registered — leave `map` undefined so the mapped op resolves to + // `unknownTableFailure` on execute. + const reconstruct = table + ? reconstructors.get(table.tableName) + : passthroughRow + const op = client.decryptModel(input) + const base = lockContext ? op.withLockContext(lockContext) : op + return new MappedDecryptOperation( + base, + reconstruct, + unknownTableFailure, + ) as never + } + + function bulkDecryptModels< + Table extends S[number], + T extends Record, + >( + input: Array, + table: Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation>> + function bulkDecryptModels>( + input: Array, + ): AuditableDecryptModelOperation>> + function bulkDecryptModels( + input: Array>, + table?: AnyV3Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation { + const op = client.bulkDecryptModels(input) + const base = lockContext ? op.withLockContext(lockContext) : op + // No table → pass rows through (nominal behaviour). Registered table → + // reconstruct each row. Unregistered table → `undefined` map → + // `unknownTableFailure` on execute. + let mapRows: + | (( + rows: Array>, + ) => Array>) + | undefined + if (!table) { + mapRows = passthroughRows + } else { + const reconstruct = reconstructors.get(table.tableName) + mapRows = reconstruct ? (rows) => rows.map(reconstruct) : undefined + } + return new MappedDecryptOperation( + base, + mapRows, + unknownTableFailure, + ) as never + } + return { encrypt: (plaintext, opts) => client.encrypt(plaintext as never, opts as never), @@ -267,53 +362,45 @@ export function typedClient( bulkEncryptModels: (input, table) => client.bulkEncryptModels(input as never, table as never) as never, decrypt: (encrypted) => client.decrypt(encrypted), - decryptModel: (input, table, lockContext) => { - // `table` is absent on a nominal-style one-arg call (see `passthroughRow`). - // Given a table: reconstruct dates from its cast_as, or — if it was never - // registered — leave `map` undefined so the mapped op resolves to - // `unknownTableFailure` on execute. - const maybeTable = table as AnyV3Table | undefined - const reconstruct = maybeTable - ? reconstructors.get(maybeTable.tableName) - : passthroughRow - const op = client.decryptModel(input as never) - const base = lockContext ? op.withLockContext(lockContext) : op - return new MappedDecryptOperation( - base, - reconstruct, - unknownTableFailure, - ) as never - }, - bulkDecryptModels: (input, table, lockContext) => { - const maybeTable = table as AnyV3Table | undefined - const op = client.bulkDecryptModels(input as never) - const base = lockContext ? op.withLockContext(lockContext) : op - // No table → pass rows through (nominal behaviour). Registered table → - // reconstruct each row. Unregistered table → `undefined` map → - // `unknownTableFailure` on execute. - let mapRows: - | (( - rows: Array>, - ) => Array>) - | undefined - if (!maybeTable) { - mapRows = passthroughRows - } else { - const reconstruct = reconstructors.get(maybeTable.tableName) - mapRows = reconstruct ? (rows) => rows.map(reconstruct) : undefined - } - return new MappedDecryptOperation( - base, - mapRows, - unknownTableFailure, - ) as never - }, + decryptModel, + bulkDecryptModels, bulkEncrypt: (plaintexts, opts) => client.bulkEncrypt(plaintexts, opts), bulkDecrypt: (payloads) => client.bulkDecrypt(payloads), getEncryptConfig: () => client.getEncryptConfig(), } satisfies TypedEncryptionClient } +/** + * The client type {@link Encryption} resolves to for the schema tuple `S`. + * + * **Use this instead of `Awaited>`.** `Encryption` + * is overloaded, and TypeScript's `ReturnType` reads the LAST overload — the + * nominal one — so that expression yields `EncryptionClient` even for an all-v3 + * schema set, and assigning the real (typed) client to it is an error: + * + * ``` + * Type 'TypedEncryptionClient<…>' is missing the following properties + * from type 'EncryptionClient': client, encryptConfig, init + * ``` + * + * Overload order cannot fix that — whichever signature is last wins, so one of + * the two forms is always mis-resolved. Name the schema tuple instead: + * + * ```typescript + * const users = encryptedTable("users", { email: types.TextSearch("email") }) + * let client: EncryptionClientFor + * client = await Encryption({ schemas: [users] }) + * ``` + * + * The equivalent inline workaround — inferring through a single-signature + * helper, `Awaited>` — also works, and is what + * `packages/bench` does. + */ +export type EncryptionClientFor = + S extends readonly [AnyV3Table, ...AnyV3Table[]] + ? TypedEncryptionClient + : EncryptionClient + /** * @deprecated Use {@link Encryption} instead — it is now overloaded so an array * of concrete EQL v3 tables yields the same strongly-typed client this used to. diff --git a/packages/stack/src/types-public.ts b/packages/stack/src/types-public.ts index 2553d3d30..bd1ec8b8f 100644 --- a/packages/stack/src/types-public.ts +++ b/packages/stack/src/types-public.ts @@ -45,6 +45,7 @@ export type { QueryTypeName, ScalarQueryTerm, SearchTerm, + V3ClientConfig, } from '@/types' // Runtime values diff --git a/packages/stack/src/types.ts b/packages/stack/src/types.ts index 6eee471b2..7cca68207 100644 --- a/packages/stack/src/types.ts +++ b/packages/stack/src/types.ts @@ -203,6 +203,22 @@ export type ClientConfig = { eqlVersion?: 2 | 3 } +/** + * {@link ClientConfig} for a client that authors EQL v3 — the same options + * minus the legacy `eqlVersion: 2` escape hatch. + * + * `Encryption` accepts this (not the full `ClientConfig`) alongside an all-v3 + * schema set. Forcing v2 wire over v3 schemas returns the NOMINAL client at + * runtime, so admitting `2` there typed the call as `TypedEncryptionClient` + * while handing back a client that silently ignores the typed client's extra + * `decryptModel` arguments. Adapters that are v3-only (`@cipherstash/prisma-next`, + * `@cipherstash/stack-drizzle`) should take this type for their pass-through + * config for the same reason. + */ +export type V3ClientConfig = Omit & { + eqlVersion?: 3 +} + type AtLeastOneCsTable = [T, ...T[]] /** Structural contract for a column builder the client can consume for STORAGE From 92a91417bca7eb88c61cb17cf15e952dc7db2aa5 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 18:59:58 +1000 Subject: [PATCH 031/123] fix(stack-supabase): export the single-row builder, drop dead v2 remnants `EncryptedSingleQueryBuilder` was declared but missing from the type-export allowlist, and its changeset documented a `.single().returns()` that did not compile. Removes the unreachable v2 leftovers: `getEncryptedColumnNames`, the v2 `addJsonbCasts`, `returnType: 'composite-literal'` on v3 terms, and four unused type imports. Corrects the inverted postgrest-js parity comment. --- .changeset/supabase-single-row-typing.md | 14 +++-- .../__tests__/supabase-v3.test-d.ts | 58 ++++++++++++++++++ packages/stack-supabase/src/helpers.ts | 61 ------------------- packages/stack-supabase/src/index.ts | 1 + packages/stack-supabase/src/query-encrypt.ts | 6 +- packages/stack-supabase/src/types.ts | 28 +++++---- 6 files changed, 91 insertions(+), 77 deletions(-) diff --git a/.changeset/supabase-single-row-typing.md b/.changeset/supabase-single-row-typing.md index eff4d8e06..7df5932fe 100644 --- a/.changeset/supabase-single-row-typing.md +++ b/.changeset/supabase-single-row-typing.md @@ -21,12 +21,18 @@ awaits to `EncryptedSupabaseResponse` (`data: T | null`). That covers the zero-row case for `maybeSingle()` and the error case for both, so no separate null modelling was needed. -Filters and transforms are not chainable after `single()`/`maybeSingle()`, +Filters and transforms are no longer chainable after `single()`/`maybeSingle()`, matching supabase-js — applying one afterwards would change the query the -single-row promise was made about. `returns()` preserves the awaited shape, -so `.single().returns()` still awaits one row. +single-row promise was made about. `.single().eq(...)`, `.single().limit(...)` +and friends were previously accepted and are now compile errors. What only +re-types or re-configures the pending request is carried over: `returns()` +(preserving the awaited shape, so `.single().returns()` awaits one row), +`abortSignal()`, `throwOnError()`, `withLockContext()` and `audit()`. +`EncryptedSingleQueryBuilder` is exported so a stored builder can be +annotated. **Migration:** delete the cast. Code that worked around the old typing with `data as unknown as Row` (or read `data![0]`) should now use `data` directly; the cast still compiles but is no longer needed, and `data![0]` becomes a type -error. +error. Move any filter or transform chained after `single()`/`maybeSingle()` to +before it. diff --git a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts index 9d0717ea5..b4aad1436 100644 --- a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts +++ b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts @@ -11,6 +11,7 @@ import { describe, expectTypeOf, it } from 'vitest' import { type EncryptedQueryBuilder, type EncryptedQueryBuilderV3, + type EncryptedSingleQueryBuilder, type EncryptedSupabaseResponse, encryptedSupabase, encryptedSupabaseV3, @@ -460,3 +461,60 @@ describe('canonical (unsuffixed) exports', () => { >() }) }) + +// --------------------------------------------------------------------------- +// single() / maybeSingle() +// +// The single-row builder is a DIFFERENT type from the array builder, so every +// method it does and does not carry is public API. Filters and transforms are +// deliberately absent — one applied after `single()` would change the query the +// single-row promise was made about — but everything that only re-types or +// re-configures the pending request stays available. `returns()` in +// particular is documented in the changeset for this change and was unreachable +// from the public surface until now (#772 review, SB-1). +// --------------------------------------------------------------------------- + +/** A typed builder for the single-row assertions below. */ +type MixedRow = UserRow & { + tags: string[] + meta: Record + note: string +} + +describe('single-row builder surface', () => { + it('is exported so a stored builder can be annotated', () => { + expectTypeOf>().toMatchTypeOf< + PromiseLike> + >() + }) + + it('awaits to ONE row, not an array', async () => { + const { data } = await mixedBuilder.single() + expectTypeOf(data).toEqualTypeOf() + }) + + it('carries returns() with the single-row shape preserved', async () => { + const { data } = await mixedBuilder.single().returns() + expectTypeOf(data).toEqualTypeOf() + }) + + it('carries the encryption configurators, which are read at execute time', () => { + const builder = mixedBuilder.single() + expectTypeOf( + builder.withLockContext({ identityClaim: ['sub'] }), + ).toEqualTypeOf>() + expectTypeOf(builder.audit({ metadata: { a: 1 } })).toEqualTypeOf< + EncryptedSingleQueryBuilder + >() + }) + + it('does NOT carry filters or transforms', () => { + const builder = mixedBuilder.single() + // @ts-expect-error - a filter after single() would change the query + builder.eq('email', 'a@b.com') + // @ts-expect-error - a transform after single() would change the query + builder.limit(1) + // @ts-expect-error - single() is applied last + builder.single() + }) +}) diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index f8e929cee..23168ffe0 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -1,7 +1,3 @@ -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' import type { QueryTypeName } from '@cipherstash/stack/types' import type { DbFilterString, @@ -11,16 +7,6 @@ import type { PendingOrCondition, } from './types' -/** - * Get the names of all encrypted columns defined in a table schema. - */ -export function getEncryptedColumnNames( - schema: EncryptedTable, -): string[] { - const built = schema.build() - return Object.keys(built.columns) -} - /** * Check whether a column name refers to an encrypted column in the schema. */ @@ -31,53 +17,6 @@ export function isEncryptedColumn( return encryptedColumnNames.includes(columnName) } -/** - * Parse a Supabase select string and add `::jsonb` casts to encrypted columns. - * - * Input: `'id, email, name'` - * Output: `'id, email::jsonb, name::jsonb'` (if email and name are encrypted) - * - * Handles whitespace, already-cast columns, and embedded functions. - */ -export function addJsonbCasts( - columns: string, - encryptedColumnNames: string[], -): DbSelect { - // The mapping below emits DB-space tokens; the brand is asserted once, here. - return columns - .split(',') - .map((col) => { - const trimmed = col.trim() - - // Skip empty segments - if (!trimmed) return col - - // If it already has a cast (e.g. `email::jsonb`), skip - if (trimmed.includes('::')) return col - - // If it contains parens (function call) or dots (foreign table), skip - if (trimmed.includes('(') || trimmed.includes('.')) return col - - // Check if the column name (possibly with alias) is encrypted - // Handle `column_name` or `column_name as alias` - const parts = trimmed.split(/\s+/) - const colName = parts[0] - - if (isEncryptedColumn(colName, encryptedColumnNames)) { - // Preserve original whitespace before the column - const leadingWhitespace = col.match(/^(\s*)/)?.[1] ?? '' - if (parts.length > 1) { - // Has alias: `email as e` -> `email::jsonb as e` - return `${leadingWhitespace}${colName}::jsonb ${parts.slice(1).join(' ')}` - } - return `${leadingWhitespace}${colName}::jsonb` - } - - return col - }) - .join(',') as DbSelect -} - /** * Resolve a select token to its DB column name, or `undefined`. * diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index 4dfec3030..4710e6f52 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -253,6 +253,7 @@ export type { // Deprecated `*V3` aliases (Decision 5 — supabase keeps type-identical aliases). EncryptedQueryBuilderV3, EncryptedQueryBuilderV3Untyped, + EncryptedSingleQueryBuilder, EncryptedSupabaseError, EncryptedSupabaseInstance, EncryptedSupabaseOptions, diff --git a/packages/stack-supabase/src/query-encrypt.ts b/packages/stack-supabase/src/query-encrypt.ts index 56ac8d653..30e1cfeb4 100644 --- a/packages/stack-supabase/src/query-encrypt.ts +++ b/packages/stack-supabase/src/query-encrypt.ts @@ -245,12 +245,16 @@ export async function encryptFilterValues( queryType: QueryTypeName, mapping: TermMapping, ) => { + // No `returnType`: the v3 path encrypts these through `bulkEncrypt` as full + // storage envelopes (see `encryptCollectedTerms`), never through + // `encryptQuery`, so nothing ever reads it. It used to request the v2 + // composite-literal formatting that only the removed `encryptQuery` path + // applied. terms.push({ value, column, table: ctx.table, queryType, - returnType: 'composite-literal', }) termMap.push(mapping) } diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index dd4611273..d16dc6739 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -1,5 +1,4 @@ import type { AuditConfig } from '@cipherstash/stack/adapter-kit' -import type { EncryptionClient } from '@cipherstash/stack/encryption' import type { AnyV3Table, EqlTypeForColumn, @@ -7,11 +6,7 @@ import type { QueryTypesForColumn, } from '@cipherstash/stack/eql/v3' import type { EncryptionError } from '@cipherstash/stack/errors' -import type { LockContext, LockContextInput } from '@cipherstash/stack/identity' -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' +import type { LockContextInput } from '@cipherstash/stack/identity' import type { ClientConfig } from '@cipherstash/stack/types' import type { V3Schemas } from './schema-builder' @@ -465,15 +460,26 @@ export interface TypedEncryptedSupabaseInstance { * The builder returned by `single()`/`maybeSingle()`: awaits to a SINGLE row * (`data: T | null`) instead of an array. * - * Only the two post-hoc modifiers supabase-js also allows after `.single()` are - * carried over. Filters and transforms are deliberately absent — applying one - * after `single()` would change the query the single-row promise was made - * about. + * FILTERS and TRANSFORMS are deliberately absent — applying one after `single()` + * would change the query the single-row promise was made about. Everything that + * only re-types or re-configures the pending request is carried over, which is + * also what postgrest-js does: its `single()` returns a `PostgrestBuilder`, + * carrying `returns`/`overrideTypes`/`throwOnError`/`setHeader` (and NOT + * `abortSignal`, which lives on `PostgrestTransformBuilder`). This adapter keeps + * `abortSignal` as a deliberate superset — an abort is not a query change — and + * adds the two encryption-specific configurators, which the runtime reads at + * execute time and so remain valid after `single()`. */ export interface EncryptedSingleQueryBuilder extends PromiseLike> { abortSignal(signal: AbortSignal): EncryptedSingleQueryBuilder throwOnError(): EncryptedSingleQueryBuilder + /** Re-type the ROW. The single-row awaited shape is preserved — `U`, not `U[]`. */ + returns>(): EncryptedSingleQueryBuilder + /** Bind identity-aware encryption. Read at execute time, so order-independent. */ + withLockContext(lockContext: LockContextInput): EncryptedSingleQueryBuilder + /** Attach audit metadata. Read at execute time, so order-independent. */ + audit(config: AuditConfig): EncryptedSingleQueryBuilder } export type EncryptedSupabaseResponse = { @@ -630,7 +636,7 @@ declare const DbBrand: unique symbol */ export type DbName = string & { readonly [DbBrand]: 'column' } -/** A PostgREST select list, DB-space and `::jsonb`-cast. Minted by `addJsonbCasts`/`addJsonbCastsV3`. */ +/** A PostgREST select list, DB-space and `::jsonb`-cast. Minted by `addJsonbCastsV3`. */ export type DbSelect = string & { readonly [DbBrand]: 'select' } /** A PostgREST `or()` filter string in DB-space. Minted by `rebuildOrString`. */ From d30882f2b3303584ba1b6b45678fe888a46fb024 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 19:00:11 +1000 Subject: [PATCH 032/123] docs(skills,meta): stop describing EQL v2 as the current generation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doctrine written into customer repos still called the extension `eql_v2.*`, setup-prompt still named the removed `encryptedType` and the v2 `encryptedColumn` builder, and four skills claimed the rollout tooling auto-detects a column's version — detection is one-sided, so a v2 column classifies as unknown and records no version. Also fixes the `--drizzle`/`--migration` flags that now require `--eql-version 2`, the stale `# v2 (default)` in the Supabase reference, and the README section routing users onto the deprecated v2 builders. --- docs/reference/supabase-sdk.md | 9 ++++++--- .../commands/init/doctrine/AGENTS-doctrine.md | 4 ++-- .../cli/src/commands/init/lib/setup-prompt.ts | 4 ++-- packages/stack-drizzle/README.md | 6 +++++- packages/stack/README.md | 20 +++++++++++-------- skills/stash-cli/SKILL.md | 10 +++++----- skills/stash-drizzle/SKILL.md | 2 +- skills/stash-dynamodb/SKILL.md | 5 +++-- skills/stash-encryption/SKILL.md | 6 +++--- skills/stash-supabase/SKILL.md | 4 ++-- 10 files changed, 41 insertions(+), 29 deletions(-) diff --git a/docs/reference/supabase-sdk.md b/docs/reference/supabase-sdk.md index 7294a8eb9..220ea7734 100644 --- a/docs/reference/supabase-sdk.md +++ b/docs/reference/supabase-sdk.md @@ -10,6 +10,9 @@ One entry point, EQL v3 only: |---|---|---| | `encryptedSupabase` | `@cipherstash/stack/eql/v3` (EQL v3) | native `public.eql_v3_*` domains | +Rows already written as EQL v2 still decrypt through `@cipherstash/stack`; what +is gone is the ability to author new v2 columns here. + `encryptedSupabaseV3` remains as a `@deprecated`, type-identical alias. The old EQL v2 authoring wrapper — `encryptedSupabase({ encryptionClient, supabaseClient })` — has been removed; the name now binds to the v3 factory @@ -134,11 +137,11 @@ The domains use SQL-standard type names (`integer`, `smallint`, `real`, ### Install EQL ```bash -# v2 (default) +# v3 (the default) stash eql install --supabase -# v3 -stash eql install --eql-version 3 --supabase +# v2 (legacy installs only) +stash eql install --eql-version 2 --supabase ``` For **v2**, `--supabase` selects the opclass-stripped bundle (operator diff --git a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md index 411114a0d..7e6400669 100644 --- a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md +++ b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md @@ -7,7 +7,7 @@ This document is the **durable rule book** for any agent working on this codebas ## What you are working with - **Encryption client** — a per-project module that defines which tables and columns are encrypted, what data type each column holds, and which search operations are enabled. The path is in `.cipherstash/context.json` under `encryptionClientPath`. -- **EQL extension** — a Postgres extension installed via `stash eql install` that provides server-side functions for searchable encryption (`eql_v2.*`). Required before any migration that creates encrypted columns. +- **EQL extension** — installed via `stash eql install`, providing the column domains and server-side functions for searchable encryption. `stash eql install` installs **EQL v3** by default: the `public.eql_v3_*` column domains and the `eql_v3.*` functions behind them. (A project set up before v3 may instead carry the legacy `eql_v2` schema and `eql_v2_encrypted` column type; both generations decrypt, but new columns are v3.) Required before any migration that creates encrypted columns. - **Project context** — `.cipherstash/context.json` records what `stash init` discovered: integration, package manager, env key names (never values), schemas, install command, CLI version. Treat it as authoritative. - **Action plan** — `.cipherstash/setup-prompt.md` is the project-specific to-do list for the current setup run. Read it first. @@ -18,7 +18,7 @@ This document is the **durable rule book** for any agent working on this codebas 3. **Never read or echo secrets.** Env key *names* (`CS_WORKSPACE_CRN`, `CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, `DATABASE_URL`) are fine to reference in code and docs. Their *values* are not. New env keys go in `.env.example` with placeholders; instruct the user to add the real value locally. Never read, `cat`, `grep`, or echo `~/.cipherstash/secretkey.json` (the development key), `~/.cipherstash/auth.json` (OAuth token and JWTs), anything under `~/.cipherstash/workspaces/`, or a value-bearing env file (`.env`, `.env.local`, `.env.production`, …). `.env.example` is the exception — it holds placeholders, not values, and you are expected to edit it. The CLI reads the credentials itself; no command needs you to open them. If a command fails on authentication, re-run `stash auth login` rather than inspecting the profile. 4. **Never invent CipherStash APIs.** If you don't know how a function is called, read the relevant skill (see below) — don't guess. The TypeScript types in `@cipherstash/stack` are the source of truth for what's callable. 5. **Never run database introspection yourself.** Don't run `psql`, `\d`, `pg_dump`, `supabase db dump`, or `drizzle-kit introspect`. The CLI already did this; the result is in `context.json`. If you need fresh introspection, ask the user to re-run `stash init`. -6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The `eql_v2` schema and `eql_v2_*` functions (CLI-managed; missing function ⇒ `stash eql upgrade`, not a hand-edit). +6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The EQL schemas and functions installed by the CLI — `eql_v3`/`public.eql_v3_*` today, `eql_v2`/`eql_v2_*` on a legacy install (missing function ⇒ `stash eql upgrade`, not a hand-edit). 7. **`@cipherstash/stack` must be excluded from any bundler.** The package wraps a native FFI module (`@cipherstash/protect-ffi`) that cannot be bundled. The moment you `npm install @cipherstash/stack` in a project with a bundler, configure the exclusion *before* writing any code that imports it. Concretely: Next.js needs `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` in `next.config.{js,ts,mjs}`; webpack needs `externals` entries; esbuild needs `external`; Vite SSR needs `ssr.external`. Skipping this surfaces as `Cannot find module '@cipherstash/protect-ffi-*'` at runtime, often after the user has shipped to production. If you're declaring an encrypted column for the first time in a project, treat configuring this exclusion as part of the same change. ## Migrations — three phases, always reversible diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 6380cacf2..dd5ba0de7 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -277,7 +277,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { 'Use when the column **does not yet exist** in the database (no plaintext predecessor to preserve). This is normal Drizzle / Supabase work plus the encryption client patterns from the integration skill.', '', "1. **If this is the first encrypted column in the project, configure the bundler exclusion first.** `@cipherstash/stack` cannot be bundled (it wraps a native FFI module). Next.js: add `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` to `next.config.*`. Webpack: `externals`. esbuild: `external`. Vite SSR: `ssr.external`. Without this, the encryption client crashes at runtime with `Cannot find module '@cipherstash/protect-ffi-*'`. See the `stash-encryption` skill's Installation section for the full snippets.", - "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories for Drizzle, `encryptedColumn` for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", + "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories from `@cipherstash/stack-drizzle` for Drizzle, and the `types.*` factories from `@cipherstash/stack/eql/v3` (via `encryptedTable`, passed as `schemas`) for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`protectOps.eq`, etc. — see the integration skill).', @@ -303,7 +303,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '#### Backfill and switch — after dual-writes are live', '', `3. **Backfill.** Run \`${cli} encrypt backfill --table --column \`. The CLI prompts the user (or accepts \`--confirm-dual-writes-deployed\` non-interactively) to confirm dual-writes are live, then chunks through the existing rows. Resumable; checkpoints to \`cs_migrations\` after every chunk. SIGINT-safe.`, - `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2:** update the schema file to declare the encrypted column under its final name (drop the twin suffix, switch \`\` to \`encryptedType\`), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`).`, + `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`). The adapters no longer author v2 — \`@cipherstash/stack-drizzle\` removed \`encryptedType\` — so declare the column with a \`types.*\` v3 domain and reach v2 rows through \`@cipherstash/stack\`'s decrypt path.`, '5. **Wire the read path through the encryption client.** The read column now holds ciphertext. Read code paths must decrypt before returning the value to callers — `decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, or the equivalent `decrypt`/`bulkDecryptModels` calls. Without this step, your read paths return raw encrypted payloads to end users. The integration skill has the exact API.', '6. **Remove the dual-write code.** The plaintext column (still `` on v3; renamed `_plaintext` on v2) is no longer authoritative. Delete the dual-write logic from the persistence layer.', `7. **Drop.** Run \`${cli} encrypt drop --table --column \`. Generates a migration that removes the now-unused plaintext column (on v3 it first verifies no rows are still plaintext-only). Apply with the project's normal migration tooling.`, diff --git a/packages/stack-drizzle/README.md b/packages/stack-drizzle/README.md index 7db86277d..d261a97d1 100644 --- a/packages/stack-drizzle/README.md +++ b/packages/stack-drizzle/README.md @@ -107,7 +107,11 @@ if (!enc.failure) await db.insert(users).values(enc.data) const rows = await db .select() .from(users) - .where(await ops.eq(users.email, 'alice@example.com')) + .where(await ops.and( + ops.matches(users.email, 'alice'), // free-text token match over ciphertext + ops.between(users.age, 18, 65), + )) + .orderBy(ops.asc(users.age)) // Decrypt after select const dec = await client.bulkDecryptModels(rows, schema) diff --git a/packages/stack/README.md b/packages/stack/README.md index 942f65fc9..4930cb0dd 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -751,7 +751,7 @@ type UserEncrypted = InferEncrypted |-------|-----| | `@cipherstash/stack/v3` | `Encryption` typed client factory (`EncryptionV3` is a `@deprecated` alias), `typedClient`, plus re-exports of the EQL v3 authoring DSL | | `@cipherstash/stack/eql/v3` | EQL v3 authoring DSL: `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, ...) | -| `@cipherstash/stack` | `Encryption` client factory, auth strategies | +| `@cipherstash/stack` | `Encryption` — the single client factory (overloaded: an array of concrete EQL v3 tables yields the typed v3 client) — plus auth strategies | | `@cipherstash/stack/schema` | Legacy v2 schema builders (see [Legacy: EQL v2](#legacy-eql-v2)) | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) | @@ -795,16 +795,20 @@ Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/do ### Migrating from @cipherstash/protect -`@cipherstash/protect` users land on the legacy v2 surface first — the mapping -below is 1:1, and method signatures on the encryption client (`encrypt`, -`decrypt`, `encryptModel`, etc.) and the `Result` pattern (`data` / `failure`) -are unchanged. From there, adopt EQL v3 for new tables: +Method signatures on the encryption client (`encrypt`, `decrypt`, +`encryptModel`, ...) and the `Result` pattern (`data` / `failure`) are unchanged. +**Declare tables with the EQL v3 DSL** — the v2 builders below are `@deprecated` +and exist to read and migrate data already written as v2, not to author new +columns. A column's capabilities come from its `types.*` domain rather than +chained tuners: `csColumn("email").equality().freeTextSearch()` becomes +`types.TextSearch("email")`. -| `@cipherstash/protect` | `@cipherstash/stack` (legacy v2) | Import Path | +| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | |------------|-----------|-------| | `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | -| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | -| `csColumn(name)` | `encryptedColumn(name)` | `@cipherstash/stack/schema` | +| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/eql/v3` | +| `csColumn(name)` | `types.(name)` (e.g. `types.TextSearch`) | `@cipherstash/stack/eql/v3` | +| `csTable`/`csColumn` for READING legacy v2 data | `encryptedTable` / `encryptedColumn` (`@deprecated`) | `@cipherstash/stack/schema` | | `import { LockContext } from "@cipherstash/protect/identify"` | `import { LockContext } from "@cipherstash/stack/identity"` | `@cipherstash/stack/identity` | | N/A | CLI | `npx stash` | diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 7cf99621a..b35c9be4f 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -352,9 +352,9 @@ Gets a project from zero to installed EQL. It loads an existing `stash.config.ts | `--force` | Reinstall even if EQL is present | | `--dry-run` | Show what would happen | | `--supabase` | Supabase-compatible install (no operator families; grants `anon`, `authenticated`, `service_role`) | -| `--drizzle` | Generate a Drizzle migration (`--name`, `--out` tune it — `--name` accepts letters, numbers, `-`, `_` only; `--out` is passed to `drizzle-kit --out`, so set it to match your `drizzle.config.ts`) | -| `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly | -| `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`) | +| `--drizzle` | Generate a Drizzle migration (**v2 only — requires `--eql-version 2`**; for v3 use `eql migration --drizzle`). `--name`, `--out` tune it — `--name` accepts letters, numbers, `-`, `_` only; `--out` is passed to `drizzle-kit --out`, so set it to match your `drizzle.config.ts` | +| `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly. `--migration` is **v2 only — requires `--eql-version 2`**; for a v3 install as a migration use `eql migration` | +| `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`). **v2 only — requires `--eql-version 2`** | | `--exclude-operator-family` | Skip operator families (non-superuser roles) | | `--eql-version <2\|3>` | EQL generation. **Default `3`** (the native `public.eql_v3_*` domain schema — the documented approach). `2` is the legacy composite schema. | | `--latest` | Fetch latest EQL from GitHub instead of the bundled copy (**v2 only**) | @@ -433,7 +433,7 @@ For AI-guided integration that edits your existing schema files in place, prefer The database-side toolset that takes an existing plaintext column the rest of the way, **after** the rollout PR is deployed and dual-writes are live. It drives `@cipherstash/migrate`, recording every transition in `cipherstash.cs_migrations` (installed by `eql install`) and reading intent from `.cipherstash/migrations.json`. -The phase ladder depends on the column's EQL version, which the commands detect from the column's **domain type** (EQL v3 types are self-describing; the `_encrypted` naming is a convention only, never relied upon): +The phase ladder depends on the column's EQL version, which the commands read off the column's **domain type** — never off the `_encrypted` naming, which is a convention only. Detection is one-sided: a `public.eql_v3_*` domain is recognised as v3, and everything else (including a legacy `eql_v2_encrypted` column) falls through to the v2 ladder: - **EQL v3 (the default):** `schema-added → dual-writing → backfilling → backfilled → dropped`. There is no cut-over — the application switches to the encrypted column by name, then the plaintext column is dropped. - **EQL v2:** `schema-added → dual-writing → backfilling → backfilled → cut-over → dropped`, where cut-over renames the encrypted twin into the original column name. @@ -451,7 +451,7 @@ stash encrypt backfill --table users --column email --chunk-size 5000 Chunked, resumable, idempotent. Walks the table in keyset-pagination order, encrypts each chunk via `bulkEncryptModels`, and writes one `UPDATE ... FROM (VALUES ...)` per chunk in a transaction that also checkpoints to `cs_migrations`. SIGINT/SIGTERM finishes the current chunk and exits cleanly; re-running resumes. The `IS NOT NULL AND _encrypted IS NULL` guard makes concurrent runners and re-runs converge. -Backfill **auto-detects the target column's EQL version** from its Postgres domain type and records it (plus the version-appropriate target phase) in `.cipherstash/migrations.json`. On an EQL v3 column it finishes by printing the v3 next steps: switch the application to `_encrypted` by name, then `stash encrypt drop` — there is no cut-over. +Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgres domain type and records it (plus the target phase) in `.cipherstash/migrations.json`. A column that is not a v3 domain — including a legacy `eql_v2_encrypted` one — does not classify and takes the v2 lifecycle, recording no version. On an EQL v3 column it finishes by printing the v3 next steps: switch the application to `_encrypted` by name, then `stash encrypt drop` — there is no cut-over. **Dual-write precondition.** The application must already write both `` and `_encrypted` on every insert and update. Otherwise rows written *during* the backfill land in plaintext only, silently. The first run prompts (interactive) or requires `--confirm-dual-writes-deployed` (non-interactive), then records `dual_writing`. Resumes don't re-prompt. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 057280571..c73d1a70c 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -423,7 +423,7 @@ The hard case: a Drizzle table that already exists in production with live data CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (If using CipherStash Proxy, the rollout also includes `stash db push` to register the encryption config with EQL.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. -> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and auto-detects a column's version from its Postgres domain type — there is no flag. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which Drizzle tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. diff --git a/skills/stash-dynamodb/SKILL.md b/skills/stash-dynamodb/SKILL.md index 2369d75b6..af53d7fca 100644 --- a/skills/stash-dynamodb/SKILL.md +++ b/skills/stash-dynamodb/SKILL.md @@ -517,10 +517,11 @@ const result = await encryptionClient.encryptQuery( if (result.failure) throw new Error(result.failure.message) const hmac = result.data.hm // Use this in DynamoDB key conditions -// EQL v2 — pass queryType explicitly: +// EQL v2 — pass queryType explicitly (`usersV2` is the legacy table declared +// in the v2 read section above; `users` is the v3 one and infers its queryType): const v2Result = await encryptionClient.encryptQuery( "search-value", - { table: users, column: users.email, queryType: "equality" } + { table: usersV2, column: usersV2.email, queryType: "equality" } ) if (v2Result.failure) throw new Error(v2Result.failure.message) const v2Hmac = v2Result.data.hm diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 31f939907..98b4ac791 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -182,7 +182,7 @@ The SDK never logs plaintext data. | Import Path | Provides | |---|---| -| `@cipherstash/stack/v3` | `EncryptionV3` (a deprecated alias of `Encryption`), `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3 schema authoring. | +| `@cipherstash/stack/v3` | `Encryption` (the client factory), `EncryptionV3` (a deprecated alias of it), `typedClient`, `TypedEncryptionClient`, `EncryptionClientFor` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3 schema authoring. | | `@cipherstash/stack/eql/v3` | `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, `V3ModelInput`, ...) | | `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the `Encryption` function (typed for an all-v3 schema set; nominal for v2/loose schemas), legacy v2 re-exports | | `@cipherstash/stack/identity` | `LockContext` class and identity types | @@ -807,7 +807,7 @@ try { ## Rolling Encryption Out to Production -> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and auto-detects the column's version from its Postgres domain type — no flag. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). +> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. @@ -859,7 +859,7 @@ Once dual-writes are recorded as live in `cs_migrations`: | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | | Schema rename | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. | | `stash encrypt cutover` | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. | -| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | +| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | | Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | | `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index fdbb2d62b..8a4efd3d4 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -600,7 +600,7 @@ The hard case: a Supabase table that already exists with live data in a plaintex CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. -> **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and auto-detects a column's version from its Postgres domain type — there is no flag. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +> **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). > **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. @@ -715,7 +715,7 @@ stash encrypt backfill --table users --column email # (CI: pass --confirm-dual-writes-deployed instead.) ``` -Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It auto-detects whether the column is EQL v2 or v3 and records that in `cs_migrations`. +Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It detects a `public.eql_v3_*` column as EQL v3 and records that in `cs_migrations`; a legacy `eql_v2_encrypted` column does not classify and takes the v2 lifecycle with no version recorded. If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. From 6c5a0e7d4ef59f05b62a7db25a315cd20162c28c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 19:00:11 +1000 Subject: [PATCH 033/123] chore(ci,changesets): fix bench triggers, test-kit alias, overstated claims tests-bench.yml watched packages/stack/src/eql/v3/** but the bench client comes from src/encryption/**, so bench-relevant changes never triggered it. vitest.shared.ts gains the missing @cipherstash/test-kit/install alias. The drizzle changeset advertised encryptedIndexes on the deleted /v3 entry, and the DynamoDB one claimed a runtime guarantee the type narrowing does not make. --- .changeset/drizzle-encrypted-indexes.md | 2 +- .changeset/stack-dynamodb-v2-write-removal.md | 5 +++-- .github/workflows/tests-bench.yml | 8 ++++++++ vitest.shared.ts | 4 ++++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/.changeset/drizzle-encrypted-indexes.md b/.changeset/drizzle-encrypted-indexes.md index d9f390253..9af771af2 100644 --- a/.changeset/drizzle-encrypted-indexes.md +++ b/.changeset/drizzle-encrypted-indexes.md @@ -2,7 +2,7 @@ '@cipherstash/stack-drizzle': minor --- -New `encryptedIndexes` helper on the `/v3` entry: spread +New `encryptedIndexes` helper on the package root: spread `...encryptedIndexes(t)` in `pgTable`'s third-argument callback and it derives the recommended functional indexes for every encrypted column in the table — named `
__`, tracked by `drizzle-kit generate` like diff --git a/.changeset/stack-dynamodb-v2-write-removal.md b/.changeset/stack-dynamodb-v2-write-removal.md index 0f5058e34..445bd3729 100644 --- a/.changeset/stack-dynamodb-v2-write-removal.md +++ b/.changeset/stack-dynamodb-v2-write-removal.md @@ -3,8 +3,9 @@ --- **Breaking (DynamoDB adapter):** `encryptedDynamoDB(...).encryptModel` and -`bulkEncryptModels` no longer accept an EQL v2 table — write is EQL v3 only. The -v2 write type overloads have been removed, narrowing encrypt to `AnyV3Table`. +`bulkEncryptModels` no longer accept an EQL v2 table. The v2 write type overloads +have been removed, narrowing encrypt to `AnyV3Table`. The narrowing is +type-level — treat the type as the contract, not a runtime guard. **Decrypt still reads existing v2 items.** `decryptModel` / `bulkDecryptModels` continue to accept an EQL v2 table (`encryptedColumn` / `encryptedField` from diff --git a/.github/workflows/tests-bench.yml b/.github/workflows/tests-bench.yml index 5f1d8c5ad..5b8da9735 100644 --- a/.github/workflows/tests-bench.yml +++ b/.github/workflows/tests-bench.yml @@ -13,6 +13,10 @@ on: paths: - 'packages/bench/**' - 'packages/stack/src/eql/v3/**' + # The bench drives the typed v3 CLIENT, which lives in `encryption/`, not + # `eql/v3/` (that is the authoring DSL). Without this the job stayed green + # by never running for the changes most likely to break it. + - 'packages/stack/src/encryption/**' - 'packages/stack-drizzle/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' @@ -26,6 +30,10 @@ on: paths: - 'packages/bench/**' - 'packages/stack/src/eql/v3/**' + # The bench drives the typed v3 CLIENT, which lives in `encryption/`, not + # `eql/v3/` (that is the authoring DSL). Without this the job stayed green + # by never running for the changes most likely to break it. + - 'packages/stack/src/encryption/**' - 'packages/stack-drizzle/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' diff --git a/vitest.shared.ts b/vitest.shared.ts index 56fee9202..3b2c3554b 100644 --- a/vitest.shared.ts +++ b/vitest.shared.ts @@ -33,6 +33,10 @@ export const sharedAlias: Record = { repoRoot, 'packages/test-kit/src/catalog.ts', ), + '@cipherstash/test-kit/install': resolve( + repoRoot, + 'packages/test-kit/src/install.ts', + ), '@cipherstash/test-kit/integration-clerk': resolve( repoRoot, 'packages/test-kit/src/integration/clerk.ts', From 6c3fb757a8032027d1335547afb83654461d11d0 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 19:56:53 +1000 Subject: [PATCH 034/123] fix(wizard): default the migrate prompt to No when a sweep could not run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `rewriteEncryptedMigrations` folded a failed directory sweep away: the `error` branch logged a warning and `continue`d, contributing 0 to both totals, so `destructive = rewritten > 0 || skipped > 0` stayed false and `p.confirm` was handed `initialValue: true`. A directory the sweep could not read looked exactly like a clean one, and the case where hitting Enter is least safe got the friendliest prompt. `packages/cli` already settled this the other way — `eql/migration.ts` and `db/install.ts` track a `sweepIncomplete` flag that deliberately treats "failed outright" and "left near-misses" as one state. Match it. The sweep now reports `failedDirs`, and the caller separates `unsafe` (rewritten or flagged) from `unverified` (unswept). Either defaults the prompt to No, but only `unsafe` claims the migration destroys data — nothing is known about an unswept directory, so the prompt names it and says it went unchecked instead. The failure test is `error !== undefined`, not truthiness: `error` is `err.message`, and `new Error()` has an empty one, which would otherwise fall straight back onto the fail-open path. --- .changeset/rewriter-never-drops-ciphertext.md | 6 +- .../wizard/src/__tests__/post-agent.test.ts | 46 +++++++++++++++ packages/wizard/src/lib/post-agent.ts | 58 +++++++++++++++---- 3 files changed, 98 insertions(+), 12 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index df4565d47..9b54efbcc 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -16,5 +16,7 @@ encrypted type, so changing a column's encrypted domain no longer drops a column full of ciphertext. Skipped statements report why they were left alone. An unreadable migration directory (`EACCES`) is reported rather than silently -treated as empty, and the wizard's `Run the migration now?` prompt defaults to -No whenever the sweep rewrote or flagged anything. +treated as empty, and the wizard's `Run the migration now?` prompt defaults to No +whenever the sweep rewrote anything, flagged anything, or could not check a +directory at all — naming the directories that went unchecked, and making no +claim about data destruction for a directory nothing is known about. diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index d3de8d2ca..c13f83cef 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -15,8 +15,18 @@ vi.mock('@clack/prompts', async (importOriginal) => { return { ...actual, confirm: vi.fn(async () => false) } }) +// Wraps the REAL sweep, so every test below still exercises it for free. Only +// the empty-message case overrides it, because no real filesystem error is +// reachable with a blank `message`. +vi.mock('../lib/rewrite-migrations.js', async (importOriginal) => { + const actual = + await importOriginal() + return { ...actual, sweepMigrationDirs: vi.fn(actual.sweepMigrationDirs) } +}) + import * as childProcess from 'node:child_process' import * as p from '@clack/prompts' +import { sweepMigrationDirs } from '../lib/rewrite-migrations.js' const bun: DetectedPackageManager = { name: 'bun', @@ -173,4 +183,40 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] expect(options?.initialValue).toBe(false) }) + + // A directory whose sweep threw contributes 0 to both totals, so a failed + // sweep used to be indistinguishable from a clean one: prompt defaulting to + // Yes over migrations nobody checked. Unknown is not the same as safe. + it('defaults to No when a directory could not be swept at all', async () => { + // A directory named `*.sql` makes readFile throw EISDIR mid-sweep. + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.mkdirSync(path.join(cwd, 'drizzle', '0001_alter.sql')) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + // Nothing is known about that directory, so the prompt must not claim the + // migration destroys data — only that it went unchecked. + expect(String(options?.message)).not.toContain('DESTROYS data') + expect(String(options?.message)).toContain('drizzle/') + expect(String(options?.message)).toContain('could not check 1 directory') + }) + + // `error` is built as `err instanceof Error ? err.message : String(err)`, and + // `new Error()` has an empty message — so a thrown error can arrive as `''`. + // Testing it for truthiness rather than presence would drop that directory + // back into the fail-open default, which is the exact bug above wearing a + // different hat. + it('treats an empty error message as a failed sweep, not a clean one', async () => { + vi.mocked(sweepMigrationDirs).mockResolvedValueOnce([ + { dir: 'drizzle', rewritten: [], skipped: [], error: '' }, + ]) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('could not check 1 directory') + }) }) diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index ecaf77a62..e4d4e2154 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -87,12 +87,35 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { // destroy data, so the prompt says so and defaults to NO — an // `initialValue: true` immediately under a "do NOT run the migration" // warning invites exactly the mistake the warning is about. - const destructive = sweep.rewritten > 0 || sweep.skipped > 0 + const unsafe = sweep.rewritten > 0 || sweep.skipped > 0 + + // A directory whose sweep threw contributes 0 to both totals, so on its own + // it is indistinguishable from a clean sweep — except that it means the + // opposite: those migrations may still hold unrepaired `SET DATA TYPE` + // statements and nobody has looked. `stash eql migration` / `db install` + // treat "sweep failed outright" and "sweep left near-misses" as the same + // state for the same reason; unknown is not safe, so the default is NO here + // too. The wording differs from the destructive case on purpose: nothing is + // known about that directory, so claiming it destroys data would be a guess. + const unverifiedDirs = sweep.failedDirs + const unverified = unverifiedDirs.length > 0 + const unverifiedList = unverifiedDirs.map((dir) => `${dir}/`).join(', ') + const unverifiedCount = `${unverifiedDirs.length} director${ + unverifiedDirs.length === 1 ? 'y' : 'ies' + }` + if (unverified) { + p.log.warn( + `The ALTER COLUMN sweep did not fully complete — review the sibling migrations in ${unverifiedList} before running drizzle-kit migrate, or you may apply broken/unsafe SQL.`, + ) + } + const shouldMigrate = await p.confirm({ - message: destructive + message: unsafe ? `Run the migration now? (${runner} drizzle-kit migrate) — see the warnings above: this migration DESTROYS data on any table that already holds rows` - : `Run the migration now? (${runner} drizzle-kit migrate)`, - initialValue: !destructive, + : unverified + ? `Run the migration now? (${runner} drizzle-kit migrate) — the sweep could not check ${unverifiedCount} (${unverifiedList}); review those migrations before migrating, or you may apply broken/unsafe SQL` + : `Run the migration now? (${runner} drizzle-kit migrate)`, + initialValue: !unsafe && !unverified, }) if (!p.isCancel(shouldMigrate) && shouldMigrate) { @@ -125,19 +148,34 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { /** * Sweep the candidate migration directories, reporting what happened, and * return the totals so the caller can decide how dangerous "run it now" is. + * + * `failedDirs` names the directories that exist but whose sweep threw. It is a + * third state, not a variant of "nothing to do": those migrations may still + * contain unrepaired `SET DATA TYPE` statements and went unchecked, which the + * `rewritten`/`skipped` counts cannot express — both stay 0 for such a + * directory, exactly as they do for a clean one. */ -async function rewriteEncryptedMigrations( - cwd: string, -): Promise<{ rewritten: number; skipped: number }> { +async function rewriteEncryptedMigrations(cwd: string): Promise<{ + rewritten: number + skipped: number + failedDirs: string[] +}> { const results = await sweepMigrationDirs(cwd, DRIZZLE_OUT_DIRS) - const totals = { rewritten: 0, skipped: 0 } + const totals = { rewritten: 0, skipped: 0, failedDirs: [] as string[] } for (const { dir, rewritten, skipped, error } of results) { totals.rewritten += rewritten.length totals.skipped += skipped.length - if (error) { - p.log.warn(`Could not rewrite migrations in ${dir}: ${error}`) + // Presence, not truthiness: `error` is `err.message` for a thrown `Error`, + // and `new Error()` has an empty message. Testing `if (error)` would put a + // blank-message failure back on the fail-open path this whole branch exists + // to close. + if (error !== undefined) { + totals.failedDirs.push(dir) + p.log.warn( + `Could not rewrite migrations in ${dir}: ${error || 'unknown error'}`, + ) continue } From ffe49746c80b9c24dc25a1a27a087d8d8e08699b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 19:57:08 +1000 Subject: [PATCH 035/123] fix(stack-supabase): accept interface row types, not just type aliases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `from()`, `returns()` and `single().returns()` constrained the row parameter to `Record`. An `interface` carries no implicit index signature, so the most ordinary way to declare a row type failed with TS2344 while an equivalent `type` alias compiled. Upstream postgrest-js constrains `returns` to nothing at all, so the adapter was stricter than the API it mirrors. Relaxed to `object` — which still rejects scalar row types — across the whole chain, since `EncryptedQueryBuilderUntyped` and `EncryptedQueryBuilderCore` re-impose the constraint on the array builder's return type. `decryptResults` moves with them; its row parameter only supplies the default for `TData`, which every call site passes explicitly. The existing type tests missed this because they declare the row as `type UserRow = InferPlaintext`. The new case uses a real `interface` and carries a comment saying why it must stay one. Also corrects the `EncryptedSingleQueryBuilder` docs, which claimed everything re-typing or re-configuring the request is carried over past `single()`. `overrideTypes` and `setHeader` are not — and since `single()`/`maybeSingle()` return the same instance rather than a passthrough, calling them fails at runtime, not just in the checker. --- .changeset/supabase-interface-row-types.md | 30 ++++++++++ .../__tests__/supabase-v3.test-d.ts | 59 +++++++++++++++++-- packages/stack-supabase/src/query-builder.ts | 4 +- packages/stack-supabase/src/query-results.ts | 5 +- packages/stack-supabase/src/types.ts | 48 +++++++++------ 5 files changed, 118 insertions(+), 28 deletions(-) create mode 100644 .changeset/supabase-interface-row-types.md diff --git a/.changeset/supabase-interface-row-types.md b/.changeset/supabase-interface-row-types.md new file mode 100644 index 000000000..688927d2c --- /dev/null +++ b/.changeset/supabase-interface-row-types.md @@ -0,0 +1,30 @@ +--- +'@cipherstash/stack-supabase': minor +--- + +Row-type generics now accept an `interface`, not just a `type` alias. + +`from()`, `returns()` and `single().returns()` constrained their row +parameter to `Record`. An `interface` has no implicit index +signature, so the most ordinary way to declare a row type failed to compile: + +```typescript +interface User { id: string; email: string } + +// before: TS2344 — Index signature for type 'string' is missing in type 'User' +// after: fine +const { data } = await supabase.from('users').select('id, email') +``` + +A `type User = { … }` alias worked, which is why the existing type tests never +caught it. The constraint is now `object`, which still rejects `string`/`number` +row types. upstream `postgrest-js` constrains `returns` to nothing at all, so +this brings the adapter in line with the API it mirrors rather than being +stricter than it. + +Also corrects the `EncryptedSingleQueryBuilder` documentation, which claimed +that "everything that only re-types or re-configures the pending request is +carried over" after `single()`/`maybeSingle()`. `overrideTypes` and `setHeader` +are not carried over — they have no adapter equivalent, and since +`single()`/`maybeSingle()` return the same builder instance rather than a +passthrough, calling them would fail at runtime, not just fail to typecheck. diff --git a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts index b4aad1436..f1b819230 100644 --- a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts +++ b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts @@ -468,10 +468,12 @@ describe('canonical (unsuffixed) exports', () => { // The single-row builder is a DIFFERENT type from the array builder, so every // method it does and does not carry is public API. Filters and transforms are // deliberately absent — one applied after `single()` would change the query the -// single-row promise was made about — but everything that only re-types or -// re-configures the pending request stays available. `returns()` in -// particular is documented in the changeset for this change and was unreachable -// from the public surface until now (#772 review, SB-1). +// single-row promise was made about. What stays available is exactly `then`, +// `abortSignal`, `throwOnError`, `returns`, `withLockContext` and `audit`; this +// is NOT parity with postgrest-js, whose `overrideTypes` and `setHeader` have no +// adapter equivalent. `returns()` in particular is documented in the +// changeset for this change and was unreachable from the public surface until +// now (#772 review, SB-1). // --------------------------------------------------------------------------- /** A typed builder for the single-row assertions below. */ @@ -518,3 +520,52 @@ describe('single-row builder surface', () => { builder.single() }) }) + +// --------------------------------------------------------------------------- +// Row-type generics accept an `interface` +// +// DO NOT "simplify" `InterfaceRow` below into a `type` alias — the whole point +// of this block is the distinction. A `type` alias for an object literal gets an +// IMPLICIT INDEX SIGNATURE, so it satisfies `Record`; an +// `interface` does NOT, so an interface row type fails a +// `Row extends Record` constraint with TS2344 ("Index signature +// for type 'string' is missing"). Every other row-typed test in this file goes +// through `type UserRow = InferPlaintext` (line ~33) — an alias — +// which is exactly why none of them ever caught this. +// +// Interfaces are the ordinary way a Supabase user declares a row type (and what +// `supabase gen types` emits alongside its aliases), and upstream postgrest-js +// leaves the equivalent `returns()` type parameter entirely unconstrained, so +// the adapter must not be stricter than the API it mirrors. +// --------------------------------------------------------------------------- + +interface InterfaceRow { + id: string + email: string +} + +describe('row-type generics accept an interface (not just a type alias)', () => { + it('accepts an interface on the untyped instance from()', async () => { + const supabase = await encryptedSupabase(supabaseClient) + const { data } = await supabase.from('users').select('*') + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on the typed instance fallback from()', async () => { + const supabase = await encryptedSupabase(supabaseClient, { + schemas: { users }, + }) + const { data } = await supabase.from('orders').select('*') + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on returns()', async () => { + const { data } = await mixedBuilder.returns() + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on single().returns()', async () => { + const { data } = await mixedBuilder.single().returns() + expectTypeOf(data).toEqualTypeOf() + }) +}) diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index b2c9652ef..d7356a1a5 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -87,7 +87,7 @@ const warnedLikeDelegation = new Set() * {@link execute} below. */ export class EncryptedQueryBuilderImpl< - T extends Record = Record, + T extends object = Record, /** The shape this builder awaits to. `T[]` normally; narrowed to `T` by * {@link single}/{@link maybeSingle}, which return ONE row. Carried as a * parameter so the promise cannot keep advertising `T[]` after the runtime @@ -503,7 +503,7 @@ export class EncryptedQueryBuilderImpl< /** Re-type the ROW. The awaited SHAPE is preserved: called after * `single()`/`maybeSingle()` this still awaits one row, not `U[]`. */ - returns>(): EncryptedQueryBuilderImpl< + returns(): EncryptedQueryBuilderImpl< U, TData extends readonly unknown[] ? U[] : U > { diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts index 5838e012d..128c06fca 100644 --- a/packages/stack-supabase/src/query-results.ts +++ b/packages/stack-supabase/src/query-results.ts @@ -80,10 +80,7 @@ function postprocessDecryptedRow( * than decrypted. To read v2 data, decrypt fetched rows with the core * `@cipherstash/stack` client, whose decrypt path is generation-agnostic. */ -export async function decryptResults< - T extends Record, - TData = T[], ->( +export async function decryptResults( result: RawSupabaseResult, ctx: DecryptContext, ): Promise> { diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index d16dc6739..6755b8b56 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -387,9 +387,8 @@ export interface EncryptedQueryBuilder< * union (which subsumes the encrypted column's `string`); the runtime resolves * the column and picks the encoding (and rejects the wrong-column-kind pairing). */ -export interface EncryptedQueryBuilderUntyped< - Row extends Record, -> extends EncryptedQueryBuilderCore< +export interface EncryptedQueryBuilderUntyped + extends EncryptedQueryBuilderCore< Row, StringKeyOf, EncryptedQueryBuilderUntyped @@ -424,7 +423,7 @@ export interface EncryptedQueryBuilderUntyped< /** Untyped instance (no `schemas`): rows default to `Record` * and `from` accepts any table name. */ export interface EncryptedSupabaseInstance { - from = Record>( + from>( tableName: string, ): EncryptedQueryBuilderUntyped } @@ -447,7 +446,7 @@ export interface TypedEncryptedSupabaseInstance { from( table: K, ): EncryptedQueryBuilder> - from = Record>( + from>( table: string, ): EncryptedQueryBuilderUntyped } @@ -461,21 +460,32 @@ export interface TypedEncryptedSupabaseInstance { * (`data: T | null`) instead of an array. * * FILTERS and TRANSFORMS are deliberately absent — applying one after `single()` - * would change the query the single-row promise was made about. Everything that - * only re-types or re-configures the pending request is carried over, which is - * also what postgrest-js does: its `single()` returns a `PostgrestBuilder`, - * carrying `returns`/`overrideTypes`/`throwOnError`/`setHeader` (and NOT - * `abortSignal`, which lives on `PostgrestTransformBuilder`). This adapter keeps - * `abortSignal` as a deliberate superset — an abort is not a query change — and - * adds the two encryption-specific configurators, which the runtime reads at - * execute time and so remain valid after `single()`. + * would change the query the single-row promise was made about. + * + * What IS carried is exactly: `then` (via `PromiseLike`), `abortSignal`, + * `throwOnError`, `returns`, `withLockContext` and `audit`. That is a + * hand-written list, not a passthrough — `single()`/`maybeSingle()` return the + * same builder instance, so a method absent here is absent at runtime too. + * + * It is therefore NOT parity with postgrest-js, which carries a different set: + * its `single()` returns a `PostgrestBuilder`, carrying + * `returns`/`overrideTypes`/`throwOnError`/`setHeader` (and NOT `abortSignal`, + * which lives on `PostgrestTransformBuilder`). Relative to that, this adapter + * keeps `abortSignal` as a deliberate superset — an abort is not a query change + * — and adds the two encryption-specific configurators, which the runtime reads + * at execute time and so remain valid after `single()`; but postgrest-js's + * `overrideTypes` and `setHeader` have NO adapter equivalent, on this surface or + * any other. */ export interface EncryptedSingleQueryBuilder extends PromiseLike> { abortSignal(signal: AbortSignal): EncryptedSingleQueryBuilder throwOnError(): EncryptedSingleQueryBuilder - /** Re-type the ROW. The single-row awaited shape is preserved — `U`, not `U[]`. */ - returns>(): EncryptedSingleQueryBuilder + /** Re-type the ROW. The single-row awaited shape is preserved — `U`, not `U[]`. + * `object`, not `Record`: an `interface` row type has no + * implicit index signature and would be rejected by the latter (upstream + * postgrest-js constrains its `returns` type parameter not at all). */ + returns(): EncryptedSingleQueryBuilder /** Bind identity-aware encryption. Read at execute time, so order-independent. */ withLockContext(lockContext: LockContextInput): EncryptedSingleQueryBuilder /** Attach audit metadata. Read at execute time, so order-independent. */ @@ -825,7 +835,7 @@ type StringKeyOf = Extract * of which still serve plaintext columns. */ export interface EncryptedQueryBuilderCore< - T extends Record, + T extends object, FK extends StringKeyOf, Self, /** Keys `order()` accepts. The typed surface narrows it to the orderable @@ -949,8 +959,10 @@ export interface EncryptedQueryBuilderCore< abortSignal(signal: AbortSignal): Self throwOnError(): Self /** Escape hatch: re-types the rows and drops back to the untyped v3 builder - * surface. */ - returns>(): EncryptedQueryBuilderUntyped + * surface. `object`, not `Record`: an `interface` row type + * has no implicit index signature and would be rejected by the latter, while + * `object` still excludes `string`/`number` row types. */ + returns(): EncryptedQueryBuilderUntyped /** Bind identity-aware encryption. Accepts either a plain * `{ identityClaim }` (the common form) or a `LockContext` instance. */ withLockContext(lockContext: LockContextInput): Self From 73531570eaba736e3aa8788594d5219edfcd0173 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 19:57:24 +1000 Subject: [PATCH 036/123] docs(skills): scope the v2 rollout lifecycle, and fix the drop column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three rollout skills described the EQL v2 lifecycle as the unqualified default even though v3 is the default generation. Worst case, `stash encrypt drop` was documented as removing `_plaintext` — that is the v2 target. On a v3 column there is no `_plaintext`: `drop.ts:113-118` drops the ORIGINAL ``, inside a `DO` block that takes ACCESS EXCLUSIVE, re-counts unencrypted rows at apply time and raises rather than dropping if any remain. Also corrected: - "the pending row will be promoted by `stash encrypt cutover`" is false for v3 — cutover short-circuits before touching any config, so `stash db activate` is the only promotion path. - The Proxy call-outs told every reader to run `stash db push`. It manages `eql_v2_configuration`, which v3 does not ship: on a v3-only database it reports "Nothing to do." and exits 0. - The CLI sequence blocks were unlabelled v2 recipes contradicting the table above them. v3 now comes first, since it is the default, and the v2 block is scoped rather than deleted. - drizzle-kit emits the BARE `eql_v3_text_search`; the schema is stripped deliberately, so the documented DDL was invalid. Adds the unit coverage `resolveEqlVersion` has always claimed to have: its `@internal` marker says "exported for unit-test coverage of the detection matrix" and there was none, so the only exercise of the v3 wire choice needed live credentials. Pairs with a guard in the v2-decrypt compat suite pinning that client to v3 wire — without it the whole `#1c` block passes vacuously if detection regresses, because a v2-mode client reads v2 payloads natively. --- .changeset/skills-v3-lifecycle-honesty.md | 27 +++++ .../__tests__/resolve-eql-version.test.ts | 104 ++++++++++++++++++ .../v2-decrypt-compat.integration.test.ts | 28 +++++ skills/stash-drizzle/SKILL.md | 23 ++-- skills/stash-encryption/SKILL.md | 66 +++++++++-- skills/stash-supabase/SKILL.md | 12 +- 6 files changed, 235 insertions(+), 25 deletions(-) create mode 100644 .changeset/skills-v3-lifecycle-honesty.md create mode 100644 packages/stack/__tests__/resolve-eql-version.test.ts diff --git a/.changeset/skills-v3-lifecycle-honesty.md b/.changeset/skills-v3-lifecycle-honesty.md new file mode 100644 index 000000000..5808d215e --- /dev/null +++ b/.changeset/skills-v3-lifecycle-honesty.md @@ -0,0 +1,27 @@ +--- +'stash': patch +--- + +Correct the EQL v2/v3 rollout lifecycle in the bundled `stash-encryption`, +`stash-supabase` and `stash-drizzle` agent skills. Each described the **v2** +lifecycle as the unqualified default even though v3 is the default generation, +so an agent following the prose would run steps that do not apply — and, in one +case, expect the wrong column to be dropped. + +- `stash encrypt drop` was documented as removing `_plaintext`. That is the + **v2** target. On a v3 column there is no `_plaintext`: the command drops + the **original ``**, guarded by a `DO` block that takes `ACCESS EXCLUSIVE` + and re-counts unencrypted rows at apply time, raising instead of dropping if + any remain. Each step in the cutover table is now marked v2-only or v3, and the + drop preconditions (`cut-over` for v2, `backfilled` for v3) are stated. +- "The pending row will be promoted to active by `stash encrypt cutover`" was + false for v3, where cutover short-circuits before touching any configuration. + `stash db activate` is the only promotion path there. +- The CipherStash Proxy call-outs told every reader to run `stash db push`. + `db push`/`db activate` manage `eql_v2_configuration`, which EQL v3 does not + ship — on a v3-only database `db push` reports "Nothing to do." and exits 0, + and `db activate` errors. The call-outs are now scoped to the EQL v2 + Proxy + path. + +Skills ship inside the `stash` tarball and are copied into user projects at +`stash init`, so this guidance was being installed into customer repos. diff --git a/packages/stack/__tests__/resolve-eql-version.test.ts b/packages/stack/__tests__/resolve-eql-version.test.ts new file mode 100644 index 000000000..e2e93f0ce --- /dev/null +++ b/packages/stack/__tests__/resolve-eql-version.test.ts @@ -0,0 +1,104 @@ +/** + * The wire-format detection matrix behind `Encryption({ schemas })`. + * + * `resolveEqlVersion` decides, from the schema set alone, which EQL wire format + * the FFI client will emit. It carries an `@internal exported for unit-test + * coverage of the detection matrix` marker — this file is that coverage. It had + * none until now, which mattered: the only place the v3 wire choice was + * exercised was `integration/shared/v2-decrypt-compat.integration.test.ts`, and + * that suite needs live ZeroKMS credentials, so a regression here was invisible + * to `pnpm test`. + * + * Why a silent regression here is dangerous rather than merely wrong: if v3 + * detection broke, `resolveEqlVersion` would return `undefined` (the FFI's v2 + * default) instead of throwing, so a v3-schema client would quietly start + * writing v2 wire into `eql_v3_*` columns. Every v2-read compatibility test + * would keep passing, because reading v2 is exactly what a v2-mode client does + * natively. Pin the mapping directly. + * + * Credential-free by construction: `resolveEqlVersion` is pure, and inspects + * only `build()` output and the `buildColumnKeyMap` marker. + */ +import { describe, expect, it } from 'vitest' +import { resolveEqlVersion } from '@/encryption' +import { encryptedTable as encryptedTableV3, types as typesV3 } from '@/eql/v3' +// The deprecated v2 authoring builders remain for reading/migrating legacy data. +import { encryptedColumn, encryptedTable } from '@/schema' + +const usersV3 = encryptedTableV3('users_v3', { + email: typesV3.TextSearch('email'), +}) + +const ordersV3 = encryptedTableV3('orders_v3', { + total: typesV3.IntegerOrd('total'), +}) + +const usersV2 = encryptedTable('users_v2', { + email: encryptedColumn('email').equality(), +}) + +const documentsV2SteVec = encryptedTable('documents_v2', { + metadata: encryptedColumn('metadata').searchableJson(), +}) + +describe('resolveEqlVersion — wire format detection', () => { + it('resolves an all-v3 schema set to 3', () => { + expect(resolveEqlVersion([usersV3])).toBe(3) + }) + + it('resolves several v3 tables to 3', () => { + expect(resolveEqlVersion([usersV3, ordersV3])).toBe(3) + }) + + it('leaves a v2 scalar schema set on the FFI default by returning undefined', () => { + // NOT `2`: the FFI's own default is v2, and `undefined` is what the client + // passes through to mean "don't override it". + expect(resolveEqlVersion([usersV2])).toBeUndefined() + }) + + it('throws on a mixed v2 + v3 schema set — one client emits one wire format', () => { + expect(() => resolveEqlVersion([usersV3, usersV2])).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) + + it('throws on a mixed set regardless of schema order', () => { + expect(() => resolveEqlVersion([usersV2, usersV3])).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) +}) + +describe('resolveEqlVersion — legacy v2 searchable JSON', () => { + it('throws for a v2 ste_vec column, which protect-ffi 0.30 cannot emit', () => { + expect(() => resolveEqlVersion([documentsV2SteVec])).toThrow( + /searchableJson\(\) on the legacy EQL v2 schema is not supported/, + ) + }) + + it('still throws when an explicit eqlVersion is supplied', () => { + // The explicit escape hatch bypasses DETECTION, not validation — otherwise + // it would emit v3 data into an eql_v2 column. + expect(() => resolveEqlVersion([documentsV2SteVec], 2)).toThrow( + /searchableJson\(\) on the legacy EQL v2 schema is not supported/, + ) + }) +}) + +describe('resolveEqlVersion — explicit config.eqlVersion', () => { + it('honours an explicit 2 over v3 schemas, for minting v2 wire during a migration', () => { + expect(resolveEqlVersion([usersV3], 2)).toBe(2) + }) + + it('honours an explicit 3 over a v2 schema set', () => { + expect(resolveEqlVersion([usersV2], 3)).toBe(3) + }) + + it('does not let an explicit version rescue a mixed schema set', () => { + // Mixing is unfixable by declaration: the two generations target different + // column types, so no single wire format serves both. + expect(() => resolveEqlVersion([usersV3, usersV2], 3)).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) +}) diff --git a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts index c1a96813b..7926837a7 100644 --- a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts +++ b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts @@ -127,6 +127,34 @@ describe('#1b — DynamoDB adapter reads a stored EQL v2 item', () => { // their client is v3-configured and knows nothing about the v2 table — but the // rows already in their database are v2. describe('#1c — a v3-configured client reads stored EQL v2 payloads', () => { + // The precondition every other case in this block rests on. Without it the + // whole describe can pass vacuously: if v3 detection regressed, + // `resolveEqlVersion` would return `undefined` rather than throw, leaving + // `v3Client` on the FFI's v2 default — and a v2-mode client reads v2 payloads + // natively, so every assertion below would still be green while the thing + // under test (a *v3* client reading v2) was never exercised. `v3Client` is + // typed through a thunk, so the overload collapse wouldn't surface as a type + // error either. The credential-free half of this guard — the detection matrix + // itself — is `__tests__/resolve-eql-version.test.ts`. + it('is genuinely in EQL v3 mode, or the cases below prove nothing', async () => { + const v3Payload = unwrapResult( + await v3Client.encrypt('a v3-authored note', { + table: unrelatedV3, + column: unrelatedV3.note, + }), + ) + + expect(v3Payload).toMatchObject({ + v: 3, + i: { t: 'v2_read_compat_unrelated_v3', c: 'note' }, + }) + // The structural discriminator, and the stronger half of this guard: a v2 + // scalar carries `k: 'ct'`, a v3 scalar carries no `k` at all (see the + // narrowing note in `src/types.ts`). This still catches a regression that + // somehow preserved `v`. + expect(v3Payload).not.toHaveProperty('k') + }, 30000) + it('decrypts a v2 ciphertext minted before the upgrade', async () => { const encrypted = unwrapResult( await v2Client.encrypt(SECRET, { table: usersV2, column: usersV2.email }), diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index c73d1a70c..f52025bb4 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -421,7 +421,7 @@ Run `ANALYZE
` after the migration applies — an expression index gather The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (If using CipherStash Proxy, the rollout also includes `stash db push` to register the encryption config with EQL.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (On a legacy EQL v2 + CipherStash Proxy database, the rollout also includes `stash db push` to register the encryption config in `eql_v2_configuration`; EQL v3 ships no configuration table, so on the v3-only database the schema below assumes, `db push` reports "Nothing to do." and a v3 rollout never needs it.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. > **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). @@ -473,19 +473,19 @@ const usersEncryptionSchema = extractEncryptionSchema(users) export const encryptionClient = await Encryption({ schemas: [usersEncryptionSchema] }) ``` -Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted public.eql_v3_text_search;`. Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) +Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN "email_encrypted" "eql_v3_text_search";` — drizzle-kit emits the **bare** domain name, which resolves to the `public.eql_v3_text_search` domain via `search_path` (a schema-qualified custom type would be quoted as one identifier and fail, so the bare name is deliberate). Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) > **Using CipherStash Proxy?** > -> If your app queries encrypted data through CipherStash Proxy, register the new encryption config with EQL: +> `stash db push` registers the encryption config in `eql_v2_configuration`, which only exists on a database that has EQL v2 installed. The Database Setup above installs EQL v3 only, and `types.TextSearch('email_encrypted')` is a `public.eql_v3_text_search` column — v3 keeps a column's config in its domain type and ships no configuration table, so on that database `stash db push` prints "Nothing to do." and exits 0. There is nothing to push for this schema. > > ```bash -> stash db push +> stash db push # EQL v2 + Proxy databases only > ``` > -> If this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected. The pending row will be promoted to active by `stash encrypt cutover` in the cutover step. +> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected, and `stash db activate` promotes the pending row to active. > -> SDK-only users can skip this step. +> SDK-only users can skip this step on EQL v3 (this section's case) — there is nothing to push. On a legacy EQL v2 column they cannot: `stash encrypt cutover` requires a pending EQL config, so an SDK-only v2 rollout must still run `stash db push` once before cutover (see the SDK-only note under Backfill below). #### Dual-writing: write to both columns from app code @@ -576,9 +576,14 @@ stash encrypt drop --table users --column email ``` The CLI emits a Drizzle migration file with the drop. For a v3 column it drops -the original plaintext column, `ALTER TABLE users DROP COLUMN email;` — there was -no rename, so no `email_plaintext` exists. Requires the column to be in the -`backfilled` phase, plus a live coverage check. +the original plaintext column, `email` — there was no rename, so no +`email_plaintext` exists. The generated SQL is not a bare `ALTER TABLE`: it is a +`DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, +re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply +time*, `RAISE EXCEPTION`s if any remain, and only then executes the +`ALTER TABLE ... DROP COLUMN`. So a row written after generation can't be +silently destroyed. Requires the column to be in the `backfilled` phase, plus a +live coverage check at generation time. Review and apply with `drizzle-kit migrate`, then update the schema to its final shape — the encrypted column is the only one left: diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 98b4ac791..3c780c1cb 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -838,7 +838,7 @@ Everything that lands in the repo and ships in **one** PR: | Schema-add | Migration adds `_encrypted` (nullable `jsonb`) alongside the existing plaintext column. Plaintext column unchanged; application still writes only plaintext. | | Dual-write code | Application now writes both `` and `_encrypted` on every persistence path that mutates the row, in the same transaction, on every code branch. Reads still come from the plaintext column. | -> **If you use CipherStash Proxy:** After the schema-add, run `stash db push` to register the new column in `eql_v2_configuration`. With no active config yet it writes directly to `active`; with an existing active config it writes `pending` (cutover will promote it). Required for Proxy-based queries. +> **If you use CipherStash Proxy (the EQL v2 path):** `stash db push` and `stash db activate` manage `eql_v2_configuration`, so they only apply to a database that has EQL v2 installed — on a v3-only database (the default) `db push` reports "Nothing to do." and exits 0, and `db activate` errors out. EQL v3 ships no configuration table, so a v3 rollout has nothing to push. On a v2 + Proxy database, run `stash db push` after the schema-add to register the new column. With no active config yet it writes directly to `active`; with an existing active config it writes `pending`, which `stash db activate` promotes to active (the v2 `stash encrypt cutover` also promotes it as part of its rename). Required for Proxy-based queries. **The dual-write definition matters.** "Writes both columns" is not enough. The rule is: every persistence path that mutates this row writes both columns, in the same transaction, on every code branch. A single missed branch — a CSV import, an admin action, a background job, a third-party webhook handler — means rows inserted in production after deploy land in plaintext only, and backfill won't catch them. Grep for every site that writes the plaintext column before declaring rollout complete. @@ -857,11 +857,11 @@ Once dual-writes are recorded as live in `cs_migrations`: | Action | What changes | |---|---| | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | -| Schema rename | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. | -| `stash encrypt cutover` | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. | +| Schema rename (**v2 only**) | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. **v3:** there is no rename — leave `_encrypted` under its own name and point the schema/queries at that name instead. | +| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. | | Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | -| Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | -| `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | +| Remove dual-write code | The plaintext column is no longer authoritative — **v2:** it is now `_plaintext` (the cutover renamed it); **v3:** it is still the original ``, since nothing was renamed. Either way, delete the dual-write logic once reads are served from the encrypted column. | +| `stash encrypt drop` | Emits a migration that drops the plaintext column — and *which* column that is depends on the generation. **v2** (precondition: phase `cut-over`): a plain `ALTER TABLE … DROP COLUMN "_plaintext"`. **v3** (precondition: phase `backfilled`): it drops the **original ``** — there is no `_plaintext` — and the generated SQL is a `DO` block that takes `ACCESS EXCLUSIVE` on the table, re-counts rows with `` set and `_encrypted` NULL *at apply time*, and raises instead of dropping if any remain. Apply with the project's normal migration tooling. | **Create the functional indexes between backfill and the read switch** (EQL v3 columns). After `stash encrypt backfill` completes and before reads move to the encrypted column, create the `eql_v3.*` extractor indexes for every queried capability (and `ANALYZE`) — one bulk build instead of per-row maintenance during backfill, and the switched reads engage an index from the first query. Recipes in the `stash-indexing` skill. (Legacy v2 rollouts have no extractor indexes to create — skip this step.) @@ -870,7 +870,7 @@ Once dual-writes are recorded as live in `cs_migrations`: Three sources of truth, kept separate on purpose: - **`.cipherstash/migrations.json`** (repo) — *intent*. Which columns the developer wants to encrypt and at which phase, code-reviewable. -- **`eql_v2_configuration`** (DB, EQL-managed) — *EQL intent*. Which columns are encrypted and with which indexes; drives the CipherStash Proxy. +- **`eql_v2_configuration`** (DB, EQL-managed) — *EQL intent*. **EQL v2 + Proxy only.** Which columns are encrypted and with which indexes; drives the CipherStash Proxy. EQL v3 encodes a column's config in its Postgres domain type and ships no configuration table, so a v3-only database has just the other two. - **`cipherstash.cs_migrations`** (DB, CipherStash-managed) — *runtime state*. Append-only event log: phase transitions, backfill cursors, error rows. Latest row per `(table, column)` is the current state. `stash encrypt status` shows all three side-by-side and flags drift (e.g. EQL says registered, the physical `_encrypted` column is missing). `stash status` (the quest log) rolls them up into the per-column "what's the next move" view used during a rollout. @@ -879,6 +879,51 @@ Three sources of truth, kept separate on purpose: ### CLI sequence for a single column +#### EQL v3 (the default) + +```bash +# Run this often — it's the canonical "where am I?" command. +stash status + +# ---- ENCRYPTION ROLLOUT (one PR, one deploy) ---- +# 1. Add the encrypted twin column via your normal migration tooling +# (drizzle-kit / supabase migrations / etc.). +# 2. Edit application code so every persistence path writes both +# `` and `_encrypted` in the same transaction, on every +# code branch. +# 3. Ship the PR to production. + +# ---- ⛔ DEPLOY GATE ---- +# Verify dual-writes are live, then redraft the plan for cutover work: +stash status +stash plan + +# ---- ENCRYPTION CUTOVER ---- +stash encrypt backfill --table users --column email +# Prompts to confirm dual-writes are live (or pass +# --confirm-dual-writes-deployed in CI). Resumable; SIGINT-safe. + +# Recovery — if dual-writes weren't actually live when backfill ran, +# re-run with --force to encrypt every plaintext row regardless. +stash encrypt backfill --table users --column email --force + +# Create the `eql_v3.*` extractor indexes for every queried capability +# and ANALYZE the table, via your normal migration tooling. Do this +# after backfill and before reads move over. Recipes: `stash-indexing`. + +# Point the application at the encrypted column BY NAME — +# `email_encrypted`. There is no rename and no `stash encrypt cutover` +# on v3. Wire the read paths through the encryption client so they +# decrypt, deploy, and verify reads return plaintext. + +# Then remove the dual-write code and drop the plaintext column. +# The generated migration re-checks coverage under a lock at apply +# time and refuses to drop if any plaintext-only row remains: +stash encrypt drop --table users --column email +``` + +#### EQL v2 (legacy) + > **Known limitation (v2):** `stash encrypt cutover` requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked separately. (EQL v3 columns never hit this — cutover doesn't apply to them.) ```bash @@ -919,9 +964,9 @@ stash encrypt cutover --table users --column email stash encrypt drop --table users --column email ``` -#### If you use CipherStash Proxy +#### EQL v2 + CipherStash Proxy -Register and promote encryption config at each phase: +Register and promote encryption config at each phase. This applies only to a database with EQL v2 installed — `eql_v2_configuration` is where `stash db push` writes, and a v3-only database has no such table (`db push` reports "Nothing to do."). A v3 rollout uses the v3 sequence above whether or not Proxy is in front of it. ```bash # Run this often — it's the canonical "where am I?" command. @@ -933,8 +978,9 @@ stash status # 2. Register the new encryption config with EQL: stash db push # First push (no active config yet) → writes directly to active. -# Subsequent push (active already exists) → writes pending; cutover -# will promote it. +# Subsequent push (active already exists) → writes pending, which +# `stash db activate` promotes — as does the v2 `stash encrypt +# cutover` below, as part of its rename. # 3. Edit application code so every persistence path writes both # `` and `_encrypted` in the same transaction, on every # code branch. diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 8a4efd3d4..98e464535 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -598,11 +598,11 @@ alias of its unsuffixed counterpart above. The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + switch reads to the encrypted column + drop — under EQL v3, which this section's schema uses, the switch is an application-side change with no rename; under legacy EQL v2 it is a rename). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. > **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). -> **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. +> **Using CipherStash Proxy?** `stash db push` and `stash db activate` manage the `eql_v2_configuration` table, so they only apply to a database that has EQL v2 installed. EQL v3 ships no configuration table — on the v3-only database this section's schema assumes, `stash db push` reports "Nothing to do." and exits 0, and there is no cutover to run it before. If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) against a legacy EQL v2 database instead of the SDK, run `stash db push` after schema-add to register the encrypted column shape with EQL. > **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. @@ -658,13 +658,13 @@ export const users = encryptedTable('users', { }) ``` -> **Using CipherStash Proxy?** Register the new encryption config with EQL: +> **Using CipherStash Proxy?** `stash db push` registers the encryption config in `eql_v2_configuration` — an EQL v2 + Proxy artifact. The `public.eql_v3_text_search` column added above needs nothing registered: EQL v3 keeps a column's config in its domain type and ships no configuration table, so on a v3-only database `db push` prints "Nothing to do." and exits 0. > > ```bash -> stash db push +> stash db push # EQL v2 + Proxy databases only > ``` > -> If this is the project's first encrypted column, `db push` writes directly to the active EQL config. If an active config already exists, it writes the new config as `pending` — that's expected. Cutover (later) will promote it. +> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config. If an active config already exists, it writes the new config as `pending` — that's expected, and `stash db activate` promotes it to active. > > **SDK users:** Skip this step. Your encryption config lives in app code. @@ -748,7 +748,7 @@ stash encrypt drop --table users --column email The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version**, which the CLI auto-detects: -- **v3** — drops the original plaintext column, `ALTER TABLE users DROP COLUMN email;`. There was no rename, so no `email_plaintext` exists. Requires the `backfilled` phase plus a live coverage check. +- **v3** — drops the original plaintext column, `email`. There was no rename, so no `email_plaintext` exists. The SQL is not a bare `ALTER TABLE`: it's a `DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply time*, `RAISE EXCEPTION`s if any remain, and only then executes the `ALTER TABLE ... DROP COLUMN` — so a row written after generation can't be silently destroyed. Requires the `backfilled` phase plus a live coverage check at generation time. - **v2** — drops the post-rename leftover, `ALTER TABLE users DROP COLUMN email_plaintext;`. Requires the `cut-over` phase. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — the plaintext column is gone; only the encrypted column is written now, through the wrapper. From a5fab3c77fce6965fce57def85e4023dcdc057a6 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 20:46:36 +1000 Subject: [PATCH 037/123] docs: stop claiming the tooling detects a column's EQL v2 generation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `classifyEqlDomain` returns `3` for an `eql_v3_*` domain and `null` for everything else (`packages/migrate/src/version.ts:45-50`). Detection is one-sided: a v2 column is never detected as v2 — it classifies as unknown and reaches the v2 lifecycle by fallback. It also records no `eqlVersion` in the manifest, since `backfill.ts:572` only writes the field when it is truthy. Three shipped surfaces said otherwise: - `skills/stash-supabase/SKILL.md` claimed auto-detection of a v2 column in three places, one of them inside the "Stay on v2 for now" bullet — precisely the case it got wrong. The same file already carried the correct one-sided wording in its EQL version note, so the three now match it. This skill is copied into customer repos by `stash init`. - `packages/migrate/README.md` documented `detectColumnEqlVersion` as returning `2`, `3`, or `null`. It cannot return `2`. That is the hardest of the three to excuse — it states an exported function's return type. - `packages/stack/README.md`'s headline Supabase example imported and called `encryptedSupabaseV3`, the `@deprecated` alias, contradicting the package table and the v3-only note further down the same file. Documentation only — no behaviour change. --- .changeset/one-sided-eql-detection-docs.md | 31 ++++++++++++++++++++++ packages/migrate/README.md | 4 +-- packages/stack/README.md | 6 ++--- skills/stash-supabase/SKILL.md | 18 ++++++++----- 4 files changed, 48 insertions(+), 11 deletions(-) create mode 100644 .changeset/one-sided-eql-detection-docs.md diff --git a/.changeset/one-sided-eql-detection-docs.md b/.changeset/one-sided-eql-detection-docs.md new file mode 100644 index 000000000..0da0531a7 --- /dev/null +++ b/.changeset/one-sided-eql-detection-docs.md @@ -0,0 +1,31 @@ +--- +'stash': patch +'@cipherstash/migrate': patch +'@cipherstash/stack': patch +--- + +Correct shipped documentation that claimed the tooling detects a column's EQL +**v2** generation. It does not, and has not since `classifyEqlDomain` dropped v2: +detection is one-sided — a `public.eql_v3_*` Postgres domain classifies as **v3**, +and anything else (a plaintext column, or a legacy `eql_v2_encrypted` one) +classifies as *unknown* and falls through to the **v2** lifecycle. The v2 path is +reached by fallback, not by detection, and a v2 column records no `eqlVersion` in +`.cipherstash/migrations.json`, so `stash encrypt status` reports no version for +it. + +- `skills/stash-supabase/SKILL.md` said the CLI "still auto-detects a v2 column" + (twice, once inside the "Stay on v2 for now" bullet — exactly the case it got + wrong) and that `stash encrypt drop` picks its target from a version the CLI + "auto-detects". All three now describe the one-sided rule, matching the correct + wording already in the same file's EQL version note. This skill is copied into + customer repos by `stash init`, so the wrong version of it was being installed + as guidance. +- `packages/migrate/README.md` documented `detectColumnEqlVersion(client, table, + column)` as returning `2`, `3`, or `null`. It cannot return `2` — the return + type is now stated as `3` or `null`, with what a `null` means for the caller. + The lifecycle intro no longer presents the v2 ladder as a detection result. +- `packages/stack/README.md`'s Supabase example imported and called + `encryptedSupabaseV3`, the `@deprecated` alias, contradicting the same file's + package table and v3-only note. It now uses `encryptedSupabase`. + +Documentation only — no behaviour change. diff --git a/packages/migrate/README.md b/packages/migrate/README.md index 30d03a961..bd41337ff 100644 --- a/packages/migrate/README.md +++ b/packages/migrate/README.md @@ -6,7 +6,7 @@ Backs the `stash encrypt` CLI command group, but also exported for direct use ## Lifecycle -Each column walks through these phases — the ladder depends on the column's EQL version (auto-detected from its Postgres domain type via `detectColumnEqlVersion`): +Each column walks through these phases — the ladder depends on the column's EQL version, and detection is one-sided: `detectColumnEqlVersion` recognises an `eql_v3_*` Postgres domain as **v3**, and everything else as *unknown* (`null`). The v2 ladder is the fallback for an unknown column, not a detection result: ```text EQL v2: schema-added → dual-writing → backfilling → backfilled → cut-over → dropped @@ -59,7 +59,7 @@ Thin wrappers around `eql_v2.rename_encrypted_columns()` (the **v2** cut-over pr ### `detectColumnEqlVersion` / `resolveEncryptedColumn` / `listEncryptedColumns` / `classifyEqlDomain` -The EQL types are self-describing, and these are the domain-type primitives everything version-specific above branches on. `detectColumnEqlVersion(client, table, column)` inspects one column's Postgres domain type and returns `2`, `3`, or `null` (not an EQL column); resolution is case-exact (quoted-identifier semantics, matching the rest of the pipeline) and honours `search_path`. `resolveEncryptedColumn(client, table, plaintextColumn, hint?)` finds a plaintext column's encrypted counterpart from the domain types — an explicit hint (e.g. the manifest's recorded `encryptedColumn`) wins, then the `_encrypted` convention, then the table's sole EQL column; the name is never assumed. `listEncryptedColumns` returns every EQL-domain column on a table, classified. +The EQL types are self-describing, and these are the domain-type primitives everything version-specific above branches on. `detectColumnEqlVersion(client, table, column)` inspects one column's Postgres domain type and returns `3` (an `eql_v3_*` domain) or `null` — it never returns `2`. `null` means *unknown*: a plaintext column, or a legacy `eql_v2_encrypted` one, is indistinguishable here, and callers fall through to the v2 lifecycle (a v2 column's version is carried by the manifest's recorded `eqlVersion`, if it has one). Resolution is case-exact (quoted-identifier semantics, matching the rest of the pipeline) and honours `search_path`. `resolveEncryptedColumn(client, table, plaintextColumn, hint?)` finds a plaintext column's encrypted counterpart from the domain types — an explicit hint (e.g. the manifest's recorded `encryptedColumn`) wins, then the `_encrypted` convention, then the table's sole EQL column; the name is never assumed. `listEncryptedColumns` returns every EQL-domain column on a table, classified. ### `countEncrypted` / `countUnencrypted` diff --git a/packages/stack/README.md b/packages/stack/README.md index 4930cb0dd..ae45f1ea6 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -443,12 +443,12 @@ Notes: ### Supabase Integration -`encryptedSupabaseV3` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: +`encryptedSupabase` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: ```typescript -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) +const es = await encryptedSupabase(supabaseUrl, supabaseKey) await es.from("users").insert({ email: "a@b.com", age: 30 }) await es.from("users").select("id, email").eq("email", "a@b.com") diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 98e464535..2cb45f9a8 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -732,10 +732,12 @@ finished). > supported.** `@cipherstash/stack-supabase` authors and queries EQL v3 only — > the v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper that > read the post-cutover `eql_v2_encrypted` column has been removed. The CLI -> lifecycle commands still auto-detect a v2 column, but the SDK read path for one -> is gone. If you have an in-flight v2 cutover, either pin the last -> `@cipherstash/stack-supabase` release that shipped the v2 wrapper to finish it, -> or (recommended) create an `eql_v3_*` twin and run the v3 rollout above. New +> lifecycle commands still handle a v2 column — detection is one-sided, so a v2 +> column classifies as *unknown* and falls through to the v2 lifecycle — but the +> SDK read path for one is gone. If you have an in-flight v2 cutover, either pin +> the last `@cipherstash/stack-supabase` release that shipped the v2 wrapper to +> finish it, or (recommended) create an `eql_v3_*` twin and run the v3 rollout +> above. New > encryption should always target an `eql_v3_*` domain. #### Drop: remove the plaintext column @@ -746,7 +748,7 @@ Once read paths are routing through the wrapper and you're confident reads are d stash encrypt drop --table users --column email ``` -The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version**, which the CLI auto-detects: +The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version.** Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else classifies as *unknown* and falls through to the **v2** path: - **v3** — drops the original plaintext column, `email`. There was no rename, so no `email_plaintext` exists. The SQL is not a bare `ALTER TABLE`: it's a `DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply time*, `RAISE EXCEPTION`s if any remain, and only then executes the `ALTER TABLE ... DROP COLUMN` — so a row written after generation can't be silently destroyed. Requires the `backfilled` phase plus a live coverage check at generation time. - **v2** — drops the post-rename leftover, `ALTER TABLE users DROP COLUMN email_plaintext;`. Requires the `cut-over` phase. @@ -782,7 +784,11 @@ Existing v2 deployments have two options: rollout in "Migrating an Existing Column to Encrypted" above. - **Stay on v2 for now:** pin the last `@cipherstash/stack-supabase` release that shipped the v2 wrapper. The CLI rollout tooling (`stash encrypt backfill` / - `cutover` / `drop`) still auto-detects a column's EQL generation. + `cutover` / `drop`) still drives a v2 column, but only because detection is + one-sided: a v2 column is never detected as v2, it classifies as *unknown* and + falls through to the v2 lifecycle. No EQL version is recorded for it in + `.cipherstash/migrations.json`, so `stash encrypt status` reports no version + for that column. For the removed v2 wrapper's historical API and semantics, see the docs at https://cipherstash.com/docs. From ed41b935fa0fbb295e1a43f6c3d773813227c058 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 20:50:08 +1000 Subject: [PATCH 038/123] docs(changesets): fix claims that are false in the release they ship in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changeset defects, all of which render into the 1.0.0 CHANGELOG. `eql-v3-sole-docs.md` named the taught surface as `EncryptionV3` / `@cipherstash/stack-drizzle/v3` / `encryptedSupabaseV3` — the first two of which the docs no longer teach and the third of which is a `@deprecated` alias — and advertised a `/v3` subpath this release deletes. Its two "exceptions" were also stale in present tense: DynamoDB does not "still require the v2 schema surface" (writes are v3-only per `dynamodb/types.ts:240`; only the decrypt overloads keep a v2 table), which contradicted `stack-dynamodb-v2-write-removal.md` in the same release, and the rollout tooling does not "currently target v2 columns" — v3 is the recognised and default generation. `init-placeholder-eql-v3.md` said init scaffolds `EncryptionV3` and `extractEncryptionSchemaV3` from `/v3`; it emits `Encryption` and the collapsed root (`init/utils.ts:288-289`), contradicting `remove-eql-v2-drizzle-root.md:25`. That text is already published in `packages/cli/CHANGELOG.md`, and it was accurate for rc.0 — so it gets a "superseded later in this release" pointer rather than a rewrite. The same claims live in `init-drizzle-eql-v3.md`; editing one and not the other would have created a fresh contradiction. Adds the missing `@cipherstash/prisma-next` changeset for the `encryptionConfig` narrowing from `ClientConfig` to `V3ClientConfig` (`from-stack-v3.ts:65`), which makes `{ eqlVersion: 2 }` a TS2322. Ten changesets name that package and none mentioned it. Deliberately untouched: thirteen `/v3`-subpath and `*V3` references across seven other changesets. Those describe surfaces that genuinely shipped in rc.0-rc.4 and are already published; rewriting them would make the collated 1.0.0 CHANGELOG diverge from what users received. --- .changeset/eql-v3-sole-docs.md | 28 +++++++++++++------ .changeset/init-placeholder-eql-v3.md | 4 +++ .changeset/prisma-next-v3-client-config.md | 32 ++++++++++++++++++++++ 3 files changed, 55 insertions(+), 9 deletions(-) create mode 100644 .changeset/prisma-next-v3-client-config.md diff --git a/.changeset/eql-v3-sole-docs.md b/.changeset/eql-v3-sole-docs.md index 1e1a72020..540d34a1f 100644 --- a/.changeset/eql-v3-sole-docs.md +++ b/.changeset/eql-v3-sole-docs.md @@ -5,12 +5,22 @@ Docs: EQL v3 is now the sole documented approach. The `stash-encryption`, `stash-drizzle`, and `stash-supabase` skills and the `@cipherstash/stack` -README teach only the v3 typed surface (`EncryptionV3`, `types.*` concrete -domains, `@cipherstash/stack-drizzle/v3`, `encryptedSupabaseV3`); EQL v2 -shrinks to one short Legacy section per document. Two explicit exceptions are -called out: DynamoDB still requires the v2 schema surface (#657), and the -encrypt rollout tooling (`stash encrypt backfill`/`cutover`, -`@cipherstash/migrate`) currently targets v2 columns (#648) — its guidance is -kept under a version callout. Also corrects the legacy `@cipherstash/drizzle` -README's pointer to the removed `@cipherstash/stack/drizzle` subpath (now the -separate `@cipherstash/stack-drizzle` package). +README teach only the v3 typed surface (`Encryption`, the `types.*` concrete +domains, the `@cipherstash/stack-drizzle` package root, `encryptedSupabase`); +EQL v2 shrinks to one short Legacy section per document. Two places keep more +than a Legacy section, because EQL v2 is still reachable there: + +- **DynamoDB reads.** `encryptedDynamoDB` writes EQL v3 only, but `decryptModel` + / `bulkDecryptModels` still accept an EQL v2 table so previously stored v2 + items stay readable, so the `stash-dynamodb` skill keeps the v2 schema shape + documented for that read path (#657). +- **The encrypt rollout lifecycle.** `stash encrypt *` and `@cipherstash/migrate` + detect a column's generation from its Postgres domain type, with EQL v3 as the + recognised and default generation; a legacy `eql_v2_encrypted` column does not + classify and falls through to the v2 lifecycle, which ends in + `stash encrypt cutover` rather than the v3 `stash encrypt drop`. That + difference is kept under a version callout (#648). + +Also corrects the legacy `@cipherstash/drizzle` README's pointer to the removed +`@cipherstash/stack/drizzle` subpath (now the separate `@cipherstash/stack-drizzle` +package). diff --git a/.changeset/init-placeholder-eql-v3.md b/.changeset/init-placeholder-eql-v3.md index b82cf31cc..6be0fe42e 100644 --- a/.changeset/init-placeholder-eql-v3.md +++ b/.changeset/init-placeholder-eql-v3.md @@ -17,3 +17,7 @@ the concrete-domain `types.*` factories (`types.TextSearch`, `types.IntegerOrd`, `types.Text`, `types.Json`, …), and the `@cipherstash/stack-drizzle/v3` entry (`extractEncryptionSchemaV3`) for Drizzle. The `encryptionClient` export shape and the empty-schema "no schemas yet" error path are unchanged. + +**Superseded later in this release** by the `@cipherstash/stack-drizzle` EQL v2 +removal: the scaffolds now emit `Encryption` from `@cipherstash/stack/v3` and +`extractEncryptionSchema` from the collapsed `@cipherstash/stack-drizzle` root. diff --git a/.changeset/prisma-next-v3-client-config.md b/.changeset/prisma-next-v3-client-config.md new file mode 100644 index 000000000..8d17b9a54 --- /dev/null +++ b/.changeset/prisma-next-v3-client-config.md @@ -0,0 +1,32 @@ +--- +'@cipherstash/prisma-next': major +--- + +**Breaking:** `CipherstashFromStackV3Options.encryptionConfig` — the config +passed through to the encryption client by `cipherstashFromStack` — is narrowed +from `ClientConfig` to `V3ClientConfig` (`ClientConfig` without the legacy +`eqlVersion: 2` escape hatch). Forcing EQL v2 no longer type-checks: + +```ts +const cipherstash = await cipherstashFromStack({ + contractJson, + encryptionConfig: { eqlVersion: 2 }, + // ^^^^^^^^^^ error TS2322: Type '2' is not assignable to type '3'. +}) +``` + +The option never did what it looked like it did. This package is EQL v3 only, +and `eqlVersion: 2` selects `Encryption`'s nominal (untyped) overload at +runtime — not the `TypedEncryptionClient` that `cipherstashFromStack` returns as +`encryptionClient`. The field disagreed with the client you got back. + +**Migration:** drop the `eqlVersion` field. Every other `ClientConfig` option +(`workspaceCrn`, `clientId`, `clientKey`, `accessKey`, `authStrategy`, logging, +…) is unchanged and still accepted. + +To read legacy EQL v2 rows, decrypt through `@cipherstash/stack` rather than +asking this adapter for a v2 client: the decrypt path is generation-agnostic and +reads both v2 and v3 payloads. Use the returned `encryptionClient` — `decrypt(…)` +for a single value, or the no-table `decryptModel(row)` / `bulkDecryptModels(rows)` +form for whole models, which is the supported path for models written before the +v3 upgrade. From 8865866c7d972c6030445ee246a352c25d74cadf Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 20:53:30 +1000 Subject: [PATCH 039/123] fix(wizard,cli): treat quoted identifiers and string literals as inert MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two ways the comment guard could still be walked past into a live `DROP COLUMN`. Both reproduce in the wizard and the CLI copy. An apostrophe inside a double-quoted identifier opened a phantom string literal. `sql.indexOf("'", i + 1)` then ran to the NEXT apostrophe — typically the same identifier in a commented-out ALTER further down — concluded that ALTER was live, and rewrote it. A column named `"o'brien_data"` was enough, and the CREATE that declared it sits above the ALTER in any real corpus, so the ordering the bug needs is the ordering migrations naturally have. An `ALTER` quoted inside an `INSERT … VALUES ('…')` is data, but was rewritten as SQL. That splices `--> statement-breakpoint` markers inside the literal, so splitting the file the way drizzle's migrator does yields a bare, live `ALTER TABLE … DROP COLUMN …;` as a chunk of its own. It is non-destructive today only because the preceding chunk hits a syntax error first and drizzle aborts the transaction — a property of the runner, not of what we emitted. Escaping the injected text's own apostrophe would have fixed the syntax error and left the DROP COLUMN. `isInsideComment` becomes `isInsideCommentOrString`: double-quoted identifiers are consumed before `'` is considered, an unterminated or index-spanning literal counts as inert, and a doubled `''`/`""` is an escape. Dollar-quoted bodies stay untracked, which is now strictly fail-safe — an apostrophe in one can only cause a skip, never a destructive rewrite. The two implementations remain character-identical: diffing them comment-stripped yields only the three sanctioned deltas (the `existsSync`/`resolve` imports, the wizard-only `sweepMigrationDirs` block, and the attribution string). Neither new function appears in that diff. Also pins two cases that already behaved correctly but were untested: lowercase `alter table … set data type`, and a commented-out ALTER with CRLF line endings. --- .changeset/rewriter-never-drops-ciphertext.md | 14 ++- .../src/__tests__/rewrite-migrations.test.ts | 93 +++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 79 ++++++++++++---- .../src/__tests__/rewrite-migrations.test.ts | 93 +++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 79 ++++++++++++---- 5 files changed, 319 insertions(+), 39 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index 9b54efbcc..02807dd61 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -7,9 +7,17 @@ Fix a data-loss bug in the Drizzle migration rewriter: a **commented-out** `ALTER … SET DATA TYPE` was rewritten into executable SQL. The matcher was comment-blind and the replacement is multi-line, so the author's `-- ` survived on the first line only — the `DROP COLUMN` on the next line emitted live and -dropped a populated column. Statements inside a `--` line comment or a `/* … */` -block are now left as written, and a `--` inside a string literal no longer -reads as a comment. +dropped a populated column. + +A statement is now left exactly as written whenever it is inert — inside a `--` +line comment, inside a `/* … */` block, or inside a single-quoted string +literal, where an `ALTER` is data rather than SQL. (Rewriting one splices +`--> statement-breakpoint` markers *inside* the literal, so splitting the file +the way drizzle's migrator does yields a bare, live `DROP COLUMN` as a chunk of +its own.) Quoting is tokenised properly in the process: a `--` inside a string +no longer opens a comment, an apostrophe inside a quoted identifier such as +`"o'brien_data"` no longer opens a phantom string literal, and a doubled `''` or +`""` reads as an escape rather than a delimiter. The sweep also refuses to rewrite a column the migration corpus already gives an encrypted type, so changing a column's encrypted domain no longer drops a column diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 7a00161ca..c7277dba7 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -594,6 +594,78 @@ describe('rewriteEncryptedAlterColumns', () => { 'ALTER TABLE "users" DROP COLUMN "email";', ) }) + + // An apostrophe inside a DOUBLE-QUOTED identifier is not a string + // delimiter. Reading it as one opens a phantom literal whose "closing" + // quote is the apostrophe in the SAME identifier further down the file — + // PAST the commented-out ALTER — so the scan concludes the ALTER is live + // and rewrites it into a real DROP COLUMN. The CREATE that declared the + // column always sits above the ALTER, so a real corpus produces exactly + // this shape. + it('leaves a commented-out ALTER untouched when an earlier identifier holds an apostrophe', async () => { + const original = [ + 'CREATE TABLE "users" (', + '\t"id" serial PRIMARY KEY NOT NULL,', + '\t"o\'brien_data" text', + ');', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "o\'brien_data" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0031_apostrophe.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // A statement inside a single-quoted literal is DATA, not SQL. Rewriting it + // splices `--> statement-breakpoint` markers INSIDE the literal, so + // splitting the file the way drizzle's migrator does yields a bare, live + // `ALTER TABLE ... DROP COLUMN ...;` as a chunk of its own. + it('leaves an ALTER inside a string literal untouched', async () => { + const original = [ + `INSERT INTO "audit_log" ("note") VALUES ('the reverted migration read:`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + `(do not run it again)');`, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0032_string-literal.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + expect(updated).not.toContain('--> statement-breakpoint') + }) + + // Regression pin, not a bug fix — this already behaves. A commented-out + // ALTER in a CRLF file must come back byte-identical. + it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { + const original = [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\r\n') + const filePath = path.join(tmpDir, '0033_crlf.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) }) // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and @@ -789,4 +861,25 @@ describe('rewriteEncryptedAlterColumns', () => { // Non-matching statement preserved expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') }) + + // Regression pin, not a bug fix — the matchers carry `/gi`, so a + // hand-lowercased migration is rewritten just like drizzle-kit's output. + it('rewrites a lowercase alter table ... set data type', async () => { + const filePath = path.join(tmpDir, '0034_lowercase.sql') + fs.writeFileSync( + filePath, + 'alter table "users" alter column "email" set data type eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') + expect(updated).not.toMatch(/set data type/i) + }) }) diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 7dcb9cc4e..b936d80e4 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -117,8 +117,10 @@ function trimStatementPreamble(statement: string): string { } /** - * True when the character at `index` sits inside a SQL comment — a `--` line - * comment, or a (nestable) `/* … *\/` block comment. + * True when the character at `index` is INERT — it sits inside a SQL comment (a + * `--` line comment, or a nestable block comment) or inside a single-quoted + * string literal. Either way it is not a statement the database will execute, + * so the rewrite must leave it exactly as written. * * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a @@ -127,12 +129,26 @@ function trimStatementPreamble(statement: string): string { * become live executable SQL and destroy the column. Commented SQL is inert by * definition, so the only correct move is to leave it exactly as written. * - * Single-quoted literals are skipped so a `'--'` inside a string cannot open a - * phantom comment. Dollar-quoted bodies are NOT tracked: a `--` inside one - * reads as a comment here, which can only make us skip a rewrite (the statement + * **Why string literals count as inert too:** an ALTER quoted inside an + * `INSERT … VALUES ('…')` is DATA. Rewriting it splices `--> statement-breakpoint` + * markers INSIDE the literal, so splitting the file the way drizzle's migrator + * does yields a bare, live `ALTER TABLE … DROP COLUMN …;` as a chunk of its own. + * Escaping the injected text's own apostrophe (`@cipherstash/stack's`) would fix + * only the syntax error, not that live DROP COLUMN — so the statement is left + * as written, exactly like a commented one. + * + * Double-quoted identifiers are tokenised BEFORE `'` is considered: an + * apostrophe inside `"o'brien_data"` must not open a phantom literal, or the + * scan runs to the NEXT apostrophe — typically the same identifier in a + * commented-out ALTER further down — decides that ALTER is live, and rewrites + * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in + * an identifier) is an escape and does not close the token. + * + * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a + * comment/literal here, which can only make us skip a rewrite (the statement * then fails loudly at migrate time), never perform a destructive one. */ -function isInsideComment(sql: string, index: number): boolean { +function isInsideCommentOrString(sql: string, index: number): boolean { let i = 0 while (i < index) { if (sql.startsWith('--', i)) { @@ -158,12 +174,17 @@ function isInsideComment(sql: string, index: number): boolean { } if (j > index) return true i = j + } else if (sql[i] === '"') { + // A quoted identifier is live SQL, but its body is not: consuming it here + // — before the `'` branch below — is what stops an apostrophe inside one + // from opening a string literal that never really existed. + i = endOfQuoted(sql, i, '"') } else if (sql[i] === "'") { - const close = sql.indexOf("'", i + 1) + const end = endOfQuoted(sql, i, "'") // Unterminated, or the literal runs past `index`: `index` is inside a - // string, which is not a comment. - if (close === -1 || close >= index) return false - i = close + 1 + // string literal, which is every bit as inert as a comment. + if (end > index) return true + i = end } else { i += 1 } @@ -171,6 +192,25 @@ function isInsideComment(sql: string, index: number): boolean { return false } +/** + * The index just past the `quote`-delimited token that opens at `open`, or + * `sql.length` when it is never closed. A doubled delimiter inside the token is + * an escaped one (`''`, `""`) and does not end it. + */ +function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { + let i = open + 1 + while (i < sql.length) { + if (sql[i] !== quote) { + i += 1 + } else if (sql[i + 1] === quote) { + i += 2 + } else { + return i + 1 + } + } + return sql.length +} + /** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` @@ -238,14 +278,14 @@ function columnKey(table: string, column: string, schema?: string): string { * The index is corpus-wide rather than ordered by migration: over-detecting * "encrypted" costs a flagged statement the user must handle by hand, while * under-detecting costs irrecoverable ciphertext. Only comment-free statements - * count, for the same reason {@link isInsideComment} exists. + * count, for the same reason {@link isInsideCommentOrString} exists. */ function indexEncryptedColumns(contents: readonly string[]): Set { const encrypted = new Set() for (const sql of contents) { for (const created of sql.matchAll(CREATE_TABLE_RE)) { - if (isInsideComment(sql, created.index)) continue + if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) for (const column of created[3].matchAll( CREATE_TABLE_ENCRYPTED_COLUMN_RE, @@ -255,7 +295,7 @@ function indexEncryptedColumns(contents: readonly string[]): Set { } for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { - if (isInsideComment(sql, added.index)) continue + if (isInsideCommentOrString(sql, added.index)) continue const { schema, table } = tableOf(added[1], added[2]) encrypted.add(columnKey(table, added[3], schema)) } @@ -264,7 +304,7 @@ function indexEncryptedColumns(contents: readonly string[]): Set { // renamed onto the real name is exactly what a previous sweep of this very // directory emitted. Run after ADD so that tmp column is already indexed. for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { - if (isInsideComment(sql, renamed.index)) continue + if (isInsideCommentOrString(sql, renamed.index)) continue const { schema, table } = tableOf(renamed[1], renamed[2]) if (encrypted.has(columnKey(table, renamed[3], schema))) { encrypted.add(columnKey(table, renamed[4], schema)) @@ -343,8 +383,9 @@ export interface RewriteResult { * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. * Both are left untouched on disk and surfaced non-fatally so the caller can * tell the user to review them, rather than silently shipping broken SQL or - * destroying data. Statements sitting inside a SQL comment are inert and are - * neither rewritten nor reported. + * destroying data. Statements sitting inside a SQL comment — or inside a + * single-quoted string literal, where they are data rather than SQL — are inert + * and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, @@ -404,7 +445,7 @@ export async function rewriteEncryptedAlterColumns( ) => { // Commented-out SQL never runs, and a multi-line replacement would only // inherit the `-- ` on its first line — leaving the rest live. - if (isInsideComment(original, offset)) return match + if (isInsideCommentOrString(original, offset)) return match const { schema, table } = tableOf(first, second) @@ -437,7 +478,9 @@ export async function rewriteEncryptedAlterColumns( // Anchor the comment test on the `SET DATA TYPE` itself: the match starts // at the previous `;`, so its own offset sits before any preamble. const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) - if (isInsideComment(updated, nearMiss.index + Math.max(keyword, 0))) { + if ( + isInsideCommentOrString(updated, nearMiss.index + Math.max(keyword, 0)) + ) { continue } skip(filePath, statement, 'unrecognised-form') diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 862643b68..16d64f472 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -525,6 +525,78 @@ describe('rewriteEncryptedAlterColumns', () => { 'ALTER TABLE "users" DROP COLUMN "email";', ) }) + + // An apostrophe inside a DOUBLE-QUOTED identifier is not a string + // delimiter. Reading it as one opens a phantom literal whose "closing" + // quote is the apostrophe in the SAME identifier further down the file — + // PAST the commented-out ALTER — so the scan concludes the ALTER is live + // and rewrites it into a real DROP COLUMN. The CREATE that declared the + // column always sits above the ALTER, so a real corpus produces exactly + // this shape. + it('leaves a commented-out ALTER untouched when an earlier identifier holds an apostrophe', async () => { + const original = [ + 'CREATE TABLE "users" (', + '\t"id" serial PRIMARY KEY NOT NULL,', + '\t"o\'brien_data" text', + ');', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "o\'brien_data" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0031_apostrophe.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // A statement inside a single-quoted literal is DATA, not SQL. Rewriting it + // splices `--> statement-breakpoint` markers INSIDE the literal, so + // splitting the file the way drizzle's migrator does yields a bare, live + // `ALTER TABLE ... DROP COLUMN ...;` as a chunk of its own. + it('leaves an ALTER inside a string literal untouched', async () => { + const original = [ + `INSERT INTO "audit_log" ("note") VALUES ('the reverted migration read:`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + `(do not run it again)');`, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0032_string-literal.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + expect(updated).not.toContain('--> statement-breakpoint') + }) + + // Regression pin, not a bug fix — this already behaves. A commented-out + // ALTER in a CRLF file must come back byte-identical. + it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { + const original = [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\r\n') + const filePath = path.join(tmpDir, '0033_crlf.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) }) // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and @@ -720,6 +792,27 @@ describe('rewriteEncryptedAlterColumns', () => { // Non-matching statement preserved expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') }) + + // Regression pin, not a bug fix — the matchers carry `/gi`, so a + // hand-lowercased migration is rewritten just like drizzle-kit's output. + it('rewrites a lowercase alter table ... set data type', async () => { + const filePath = path.join(tmpDir, '0034_lowercase.sql') + fs.writeFileSync( + filePath, + 'alter table "users" alter column "email" set data type eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') + expect(updated).not.toMatch(/set data type/i) + }) }) describe('sweepMigrationDirs', () => { diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 3c395016b..e5a76112f 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -125,8 +125,10 @@ function trimStatementPreamble(statement: string): string { } /** - * True when the character at `index` sits inside a SQL comment — a `--` line - * comment, or a (nestable) `/* … *\/` block comment. + * True when the character at `index` is INERT — it sits inside a SQL comment (a + * `--` line comment, or a nestable block comment) or inside a single-quoted + * string literal. Either way it is not a statement the database will execute, + * so the rewrite must leave it exactly as written. * * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a @@ -135,12 +137,26 @@ function trimStatementPreamble(statement: string): string { * become live executable SQL and destroy the column. Commented SQL is inert by * definition, so the only correct move is to leave it exactly as written. * - * Single-quoted literals are skipped so a `'--'` inside a string cannot open a - * phantom comment. Dollar-quoted bodies are NOT tracked: a `--` inside one - * reads as a comment here, which can only make us skip a rewrite (the statement + * **Why string literals count as inert too:** an ALTER quoted inside an + * `INSERT … VALUES ('…')` is DATA. Rewriting it splices `--> statement-breakpoint` + * markers INSIDE the literal, so splitting the file the way drizzle's migrator + * does yields a bare, live `ALTER TABLE … DROP COLUMN …;` as a chunk of its own. + * Escaping the injected text's own apostrophe (`@cipherstash/stack's`) would fix + * only the syntax error, not that live DROP COLUMN — so the statement is left + * as written, exactly like a commented one. + * + * Double-quoted identifiers are tokenised BEFORE `'` is considered: an + * apostrophe inside `"o'brien_data"` must not open a phantom literal, or the + * scan runs to the NEXT apostrophe — typically the same identifier in a + * commented-out ALTER further down — decides that ALTER is live, and rewrites + * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in + * an identifier) is an escape and does not close the token. + * + * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a + * comment/literal here, which can only make us skip a rewrite (the statement * then fails loudly at migrate time), never perform a destructive one. */ -function isInsideComment(sql: string, index: number): boolean { +function isInsideCommentOrString(sql: string, index: number): boolean { let i = 0 while (i < index) { if (sql.startsWith('--', i)) { @@ -166,12 +182,17 @@ function isInsideComment(sql: string, index: number): boolean { } if (j > index) return true i = j + } else if (sql[i] === '"') { + // A quoted identifier is live SQL, but its body is not: consuming it here + // — before the `'` branch below — is what stops an apostrophe inside one + // from opening a string literal that never really existed. + i = endOfQuoted(sql, i, '"') } else if (sql[i] === "'") { - const close = sql.indexOf("'", i + 1) + const end = endOfQuoted(sql, i, "'") // Unterminated, or the literal runs past `index`: `index` is inside a - // string, which is not a comment. - if (close === -1 || close >= index) return false - i = close + 1 + // string literal, which is every bit as inert as a comment. + if (end > index) return true + i = end } else { i += 1 } @@ -179,6 +200,25 @@ function isInsideComment(sql: string, index: number): boolean { return false } +/** + * The index just past the `quote`-delimited token that opens at `open`, or + * `sql.length` when it is never closed. A doubled delimiter inside the token is + * an escaped one (`''`, `""`) and does not end it. + */ +function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { + let i = open + 1 + while (i < sql.length) { + if (sql[i] !== quote) { + i += 1 + } else if (sql[i + 1] === quote) { + i += 2 + } else { + return i + 1 + } + } + return sql.length +} + /** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` @@ -246,14 +286,14 @@ function columnKey(table: string, column: string, schema?: string): string { * The index is corpus-wide rather than ordered by migration: over-detecting * "encrypted" costs a flagged statement the user must handle by hand, while * under-detecting costs irrecoverable ciphertext. Only comment-free statements - * count, for the same reason {@link isInsideComment} exists. + * count, for the same reason {@link isInsideCommentOrString} exists. */ function indexEncryptedColumns(contents: readonly string[]): Set { const encrypted = new Set() for (const sql of contents) { for (const created of sql.matchAll(CREATE_TABLE_RE)) { - if (isInsideComment(sql, created.index)) continue + if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) for (const column of created[3].matchAll( CREATE_TABLE_ENCRYPTED_COLUMN_RE, @@ -263,7 +303,7 @@ function indexEncryptedColumns(contents: readonly string[]): Set { } for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { - if (isInsideComment(sql, added.index)) continue + if (isInsideCommentOrString(sql, added.index)) continue const { schema, table } = tableOf(added[1], added[2]) encrypted.add(columnKey(table, added[3], schema)) } @@ -272,7 +312,7 @@ function indexEncryptedColumns(contents: readonly string[]): Set { // renamed onto the real name is exactly what a previous sweep of this very // directory emitted. Run after ADD so that tmp column is already indexed. for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { - if (isInsideComment(sql, renamed.index)) continue + if (isInsideCommentOrString(sql, renamed.index)) continue const { schema, table } = tableOf(renamed[1], renamed[2]) if (encrypted.has(columnKey(table, renamed[3], schema))) { encrypted.add(columnKey(table, renamed[4], schema)) @@ -351,8 +391,9 @@ export interface RewriteResult { * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. * Both are left untouched on disk and surfaced non-fatally so the caller can * tell the user to review them, rather than silently shipping broken SQL or - * destroying data. Statements sitting inside a SQL comment are inert and are - * neither rewritten nor reported. + * destroying data. Statements sitting inside a SQL comment — or inside a + * single-quoted string literal, where they are data rather than SQL — are inert + * and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, @@ -412,7 +453,7 @@ export async function rewriteEncryptedAlterColumns( ) => { // Commented-out SQL never runs, and a multi-line replacement would only // inherit the `-- ` on its first line — leaving the rest live. - if (isInsideComment(original, offset)) return match + if (isInsideCommentOrString(original, offset)) return match const { schema, table } = tableOf(first, second) @@ -445,7 +486,9 @@ export async function rewriteEncryptedAlterColumns( // Anchor the comment test on the `SET DATA TYPE` itself: the match starts // at the previous `;`, so its own offset sits before any preamble. const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) - if (isInsideComment(updated, nearMiss.index + Math.max(keyword, 0))) { + if ( + isInsideCommentOrString(updated, nearMiss.index + Math.max(keyword, 0)) + ) { continue } skip(filePath, statement, 'unrecognised-form') From 2db9b42df11af70417cbb697eaf5621cba992414 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:13:24 +1000 Subject: [PATCH 040/123] fix(wizard,cli): an unterminated quoted identifier must fail safe too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of #780 found the previous commit reintroduced its own bug one branch over. The `'` branch treats an unterminated literal as inert and returns early; the `"` branch just advanced the cursor. On an unterminated identifier that set `i = sql.length`, so the `while (i < index)` loop exited and the function returned `false` — "live" — and every commented-out ALTER below it was rewritten into a real DROP COLUMN. Exactly the failure the apostrophe fix was for. Reachability is low: an unpaired `"` outside a comment or literal is malformed SQL, and drizzle does not generate one. But the sweep runs over whatever migration directories a project has, the failure is silent, and the cost is a dropped populated column — the same reasons the corpus heuristic elsewhere in this file is written to fail closed. Both branches now return early on an unterminated quote. An ALTER match offset never lands inside an identifier, so nothing changes for well-formed input. --- .changeset/rewriter-never-drops-ciphertext.md | 5 ++-- .../src/__tests__/rewrite-migrations.test.ts | 26 +++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 9 ++++++- .../src/__tests__/rewrite-migrations.test.ts | 26 +++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 9 ++++++- 5 files changed, 71 insertions(+), 4 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index 02807dd61..b3577f8b4 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -16,8 +16,9 @@ literal, where an `ALTER` is data rather than SQL. (Rewriting one splices the way drizzle's migrator does yields a bare, live `DROP COLUMN` as a chunk of its own.) Quoting is tokenised properly in the process: a `--` inside a string no longer opens a comment, an apostrophe inside a quoted identifier such as -`"o'brien_data"` no longer opens a phantom string literal, and a doubled `''` or -`""` reads as an escape rather than a delimiter. +`"o'brien_data"` no longer opens a phantom string literal, a doubled `''` or +`""` reads as an escape rather than a delimiter, and an unterminated quote of +either kind makes the rest of the file inert rather than live. The sweep also refuses to rewrite a column the migration corpus already gives an encrypted type, so changing a column's encrypted domain no longer drops a column diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index c7277dba7..4f449965e 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -648,6 +648,32 @@ describe('rewriteEncryptedAlterColumns', () => { expect(updated).not.toContain('--> statement-breakpoint') }) + // An UNTERMINATED quoted identifier must fail the same way an unterminated + // string literal does — by swallowing the rest of the file as inert. If it + // instead runs the scan cursor to the end, the loop exits and every + // commented-out ALTER below it is reported live and rewritten: the same + // destructive outcome as the apostrophe case above, one branch over. + it('leaves a commented-out ALTER untouched after an unterminated quoted identifier', async () => { + const original = [ + 'CREATE TABLE "users" ("id" serial PRIMARY KEY NOT NULL, "email" text);', + '--> statement-breakpoint', + 'SELECT "unclosed FROM users;', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_unterminated-identifier.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + // Regression pin, not a bug fix — this already behaves. A commented-out // ALTER in a CRLF file must come back byte-identical. it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index b936d80e4..550cb282a 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -178,7 +178,14 @@ function isInsideCommentOrString(sql: string, index: number): boolean { // A quoted identifier is live SQL, but its body is not: consuming it here // — before the `'` branch below — is what stops an apostrophe inside one // from opening a string literal that never really existed. - i = endOfQuoted(sql, i, '"') + const end = endOfQuoted(sql, i, '"') + // An unterminated identifier swallows the rest of the file, so treat it + // as inert exactly like the unterminated literal below. Running the + // cursor to the end instead would exit the loop and report `false` — + // "live" — which is how the apostrophe bug destroyed a column in the + // first place, one branch over. + if (end > index) return true + i = end } else if (sql[i] === "'") { const end = endOfQuoted(sql, i, "'") // Unterminated, or the literal runs past `index`: `index` is inside a diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 16d64f472..f5ef3ee45 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -579,6 +579,32 @@ describe('rewriteEncryptedAlterColumns', () => { expect(updated).not.toContain('--> statement-breakpoint') }) + // An UNTERMINATED quoted identifier must fail the same way an unterminated + // string literal does — by swallowing the rest of the file as inert. If it + // instead runs the scan cursor to the end, the loop exits and every + // commented-out ALTER below it is reported live and rewritten: the same + // destructive outcome as the apostrophe case above, one branch over. + it('leaves a commented-out ALTER untouched after an unterminated quoted identifier', async () => { + const original = [ + 'CREATE TABLE "users" ("id" serial PRIMARY KEY NOT NULL, "email" text);', + '--> statement-breakpoint', + 'SELECT "unclosed FROM users;', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_unterminated-identifier.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + // Regression pin, not a bug fix — this already behaves. A commented-out // ALTER in a CRLF file must come back byte-identical. it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index e5a76112f..73e1908b7 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -186,7 +186,14 @@ function isInsideCommentOrString(sql: string, index: number): boolean { // A quoted identifier is live SQL, but its body is not: consuming it here // — before the `'` branch below — is what stops an apostrophe inside one // from opening a string literal that never really existed. - i = endOfQuoted(sql, i, '"') + const end = endOfQuoted(sql, i, '"') + // An unterminated identifier swallows the rest of the file, so treat it + // as inert exactly like the unterminated literal below. Running the + // cursor to the end instead would exit the loop and report `false` — + // "live" — which is how the apostrophe bug destroyed a column in the + // first place, one branch over. + if (end > index) return true + i = end } else if (sql[i] === "'") { const end = endOfQuoted(sql, i, "'") // Unterminated, or the literal runs past `index`: `index` is inside a From de804c2cdd796a6a4a7205a4a336b698bf580a77 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:26:25 +1000 Subject: [PATCH 041/123] fix(stack): only forward an EQL v3 table to the client on DynamoDB decrypt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `#1c` integration case added by 7e0092f3 went red in CI, and it was right to. `encryptedDynamoDB` documents that writes are EQL v3 only but decrypt keeps accepting a v2 table so previously stored items stay readable. Both decrypt operations then forwarded that table to the encryption client unconditionally. The second argument is only meaningful to the typed v3 client, which resolves it against its own reconstructor map. A legacy v2 table is not in that map, so a v3-configured client reading a stored v2 item failed with "decryptModel received a table this client was not initialized with" — precisely the customer the compatibility promise is written for. The comment above the call claimed the argument was "ignored by the nominal one", which was true when only a nominal client existed and stopped being true when the typed client arrived. Forward the table only when `isV3Table`; otherwise use the table-less form and let the client derive the table from the payloads. `bulkDecryptModels` carried the identical unconditional forward with no test on it at all, so it is fixed in the same commit. Both paths now have credential-free coverage asserting what the adapter forwards. The only previous coverage was a live-ZeroKMS integration suite, which is why this reached released code: it is present on `main` as well as this branch. --- .../dynamodb-v2-read-table-forwarding.md | 27 +++++ .../dynamodb/v2-table-forwarding.test.ts | 108 ++++++++++++++++++ .../operations/bulk-decrypt-models.ts | 11 +- .../src/dynamodb/operations/decrypt-model.ts | 13 ++- 4 files changed, 153 insertions(+), 6 deletions(-) create mode 100644 .changeset/dynamodb-v2-read-table-forwarding.md create mode 100644 packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts diff --git a/.changeset/dynamodb-v2-read-table-forwarding.md b/.changeset/dynamodb-v2-read-table-forwarding.md new file mode 100644 index 000000000..82918224a --- /dev/null +++ b/.changeset/dynamodb-v2-read-table-forwarding.md @@ -0,0 +1,27 @@ +--- +'@cipherstash/stack': patch +--- + +Fix `encryptedDynamoDB`'s EQL v2 read path, which failed against a v3-configured +client — the case the v2 read support exists for. + +`decryptModel` and `bulkDecryptModels` forwarded the table to the encryption +client unconditionally. The second argument is only meaningful to the typed EQL +v3 client, which resolves it against its own reconstructor map; a legacy v2 +table is not in that map, so the call failed with: + +``` +[eql/v3]: decryptModel received a table this client was not initialized with +``` + +That contradicted the adapter's documented contract — writes are EQL v3 only, +but decrypt keeps accepting a v2 table so previously stored items stay readable. +In practice the failure hit exactly the customers the compatibility promise was +written for: upgraded to a v3 schema, with v2 items still in the table. + +The table is now forwarded only when it is an EQL v3 table. A v2 table takes the +table-less form, and the client derives the table from the payloads themselves. + +Both paths are covered by a credential-free test asserting what the adapter +forwards, so a regression no longer depends on a live-credential integration run +to surface. diff --git a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts new file mode 100644 index 000000000..d8f1bc872 --- /dev/null +++ b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts @@ -0,0 +1,108 @@ +/** + * Which table (if any) the DynamoDB adapter forwards to the encryption client. + * + * `encryptedDynamoDB` promises that `decryptModel` / `bulkDecryptModels` keep + * accepting an EQL **v2** table so previously stored v2 items stay readable + * (see the contract note in `src/dynamodb/index.ts`). That promise breaks if the + * adapter forwards the v2 table to a v3-configured client: the typed client + * looks the table up in its own reconstructor map, does not find it, and fails + * with "decryptModel received a table this client was not initialized with". + * + * The nominal client derives the table from the payloads and needs no second + * argument, and the typed client now exposes a table-less overload for exactly + * this case — so the correct forward is conditional on the table's generation, + * not unconditional. + * + * Credential-free by construction: the adapter never touches the AWS SDK, and + * these drive it with a recording stub client, so the assertion is on the call + * the adapter makes rather than on a live decrypt. That matters because the only + * other coverage of this path is an integration suite requiring live ZeroKMS. + */ +import { describe, expect, it } from 'vitest' +import { encryptedDynamoDB } from '@/dynamodb' +import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' +import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' + +const usersV2 = encryptedTableV2('users_v2', { + email: encryptedColumn('email').equality(), +}) + +const usersV3 = encryptedTableV3('users_v3', { + email: types.TextEq('email'), +}) + +/** + * A client that records how each decrypt method was called. `getEncryptConfig` + * reports `knownTables` so the construction-time version guard sees a client + * that knows the v3 table under test. + */ +function recordingClient(knownTables: string[]) { + const calls: { method: string; argCount: number; table: unknown }[] = [] + + const record = + (method: string) => + (...args: unknown[]) => { + calls.push({ method, argCount: args.length, table: args[1] }) + return Promise.resolve({ data: {} }) + } + + const client = { + getEncryptConfig: () => ({ + v: 1, + tables: Object.fromEntries(knownTables.map((t) => [t, {}])), + }), + encryptModel: record('encryptModel'), + bulkEncryptModels: record('bulkEncryptModels'), + decryptModel: record('decryptModel'), + bulkDecryptModels: record('bulkDecryptModels'), + } + + return { calls, client } +} + +describe('decryptModel table forwarding', () => { + it('does not forward an EQL v2 table to the client', async () => { + const { calls, client } = recordingClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.decryptModel({ pk: 'a' }, usersV2) + + expect(calls).toHaveLength(1) + expect(calls[0]?.method).toBe('decryptModel') + // The v2 table must not reach the client — a typed client would reject it. + expect(calls[0]?.table).toBeUndefined() + }) + + it('still forwards an EQL v3 table, which the typed client requires', async () => { + const { calls, client } = recordingClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.decryptModel({ pk: 'a' }, usersV3) + + expect(calls).toHaveLength(1) + expect(calls[0]?.table).toBe(usersV3) + }) +}) + +describe('bulkDecryptModels table forwarding', () => { + it('does not forward an EQL v2 table to the client', async () => { + const { calls, client } = recordingClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.bulkDecryptModels([{ pk: 'a' }], usersV2) + + expect(calls).toHaveLength(1) + expect(calls[0]?.method).toBe('bulkDecryptModels') + expect(calls[0]?.table).toBeUndefined() + }) + + it('still forwards an EQL v3 table, which the typed client requires', async () => { + const { calls, client } = recordingClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.bulkDecryptModels([{ pk: 'a' }], usersV3) + + expect(calls).toHaveLength(1) + expect(calls[0]?.table).toBe(usersV3) + }) +}) diff --git a/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts b/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts index d6d8b8b3f..4f1f4f21a 100644 --- a/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts +++ b/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts @@ -4,6 +4,7 @@ import { logger } from '@/utils/logger' import { buildReadContext, handleError, + isV3Table, resolveDecryptResult, throwPreservingCode, toItemWithEqlPayloads, @@ -53,9 +54,13 @@ export class BulkDecryptModelsOperation< const client = this.encryptionClient as CallableEncryptionClient const decryptResult = await resolveDecryptResult[]>( - // The second argument is required by the typed client and ignored by - // the nominal one, which derives the table from the payloads. - client.bulkDecryptModels(itemsWithEqlPayloads, this.table), + // Conditional for the same reason as `decryptModel` — see the note + // there. A v2 table forwarded to a v3-configured typed client is + // rejected by its reconstructor lookup, breaking the v2 read path + // this adapter documents as supported. + isV3Table(this.table) + ? client.bulkDecryptModels(itemsWithEqlPayloads, this.table) + : client.bulkDecryptModels(itemsWithEqlPayloads), this.getAuditData(), ) diff --git a/packages/stack/src/dynamodb/operations/decrypt-model.ts b/packages/stack/src/dynamodb/operations/decrypt-model.ts index aea746b36..1a725e56e 100644 --- a/packages/stack/src/dynamodb/operations/decrypt-model.ts +++ b/packages/stack/src/dynamodb/operations/decrypt-model.ts @@ -3,6 +3,7 @@ import type { Decrypted, EncryptedValue } from '@/types' import { logger } from '@/utils/logger' import { handleError, + isV3Table, resolveDecryptResult, throwPreservingCode, toItemWithEqlPayloads, @@ -47,9 +48,15 @@ export class DecryptModelOperation< const client = this.encryptionClient as CallableEncryptionClient const decryptResult = await resolveDecryptResult>( - // The second argument is required by the typed client and ignored by - // the nominal one, which derives the table from the payloads. - client.decryptModel(withEqlPayloads, this.table), + // The typed client REQUIRES the table; the nominal one derives it + // from the payloads and needs no second argument. Forwarding a v2 + // table unconditionally breaks this adapter's documented v2 read + // path: a v3-configured typed client looks the table up in its own + // reconstructor map, does not find it, and fails. Only a v3 table is + // ever meaningful to that lookup, so only a v3 table is passed. + isV3Table(this.table) + ? client.decryptModel(withEqlPayloads, this.table) + : client.decryptModel(withEqlPayloads), this.getAuditData(), ) From 1e4056bbad4616c1e0906300bdd08334a416e3aa Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:43:33 +1000 Subject: [PATCH 042/123] docs(stack-supabase): drop a comment about a removed code path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The note explained why `returnType` is absent, then justified it with what the field used to do on the `encryptQuery` path that the same PR deleted. `returnType` appears nowhere else in this package — it is not a field on anything this function builds — so the comment described the absence of something a reader cannot find, and pointed at code they cannot read. From ae5d1a44, "drop dead v2 remnants". --- packages/stack-supabase/src/query-encrypt.ts | 5 ----- 1 file changed, 5 deletions(-) diff --git a/packages/stack-supabase/src/query-encrypt.ts b/packages/stack-supabase/src/query-encrypt.ts index 30e1cfeb4..ae67e4bfe 100644 --- a/packages/stack-supabase/src/query-encrypt.ts +++ b/packages/stack-supabase/src/query-encrypt.ts @@ -245,11 +245,6 @@ export async function encryptFilterValues( queryType: QueryTypeName, mapping: TermMapping, ) => { - // No `returnType`: the v3 path encrypts these through `bulkEncrypt` as full - // storage envelopes (see `encryptCollectedTerms`), never through - // `encryptQuery`, so nothing ever reads it. It used to request the v2 - // composite-literal formatting that only the removed `encryptQuery` path - // applied. terms.push({ value, column, From c70da7c47bede6315432896559514d04ec1b1640 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:56:00 +1000 Subject: [PATCH 043/123] docs: drop three more comments that explain the present via deleted code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same shape as the `query-encrypt.ts` note removed in 4c176782: each justified current behaviour by narrating a code path this stack deleted, which a reader cannot look up. - `resolve-eql.ts` — a parenthetical about a `version: 2` candidate that "used to arrive here and needed its own exemption". The sentence before it already says the column never appears as a candidate, which is the whole point. - `dynamodb/types.ts` — an em-dash clause attributing the wrong typing to "the removed v2 write overload". The sentence is unchanged without it, and this JSDoc ships in the published `.d.ts`. - `install-eql.ts` — eight lines narrating an `eqlVersion: '2'` pin that no longer exists. Rewritten to state the constraint that still holds: `eql install --drizzle` is v2-only, so routing Drizzle through it would provision a v2 database and contradict the v3 skill installed into the same project. Comment-only; no non-comment line changed. Kept the past-tense notes that describe fixed BUGS rather than removed APIs — a reader can still check those against the code that would reintroduce them. --- .../cli/src/commands/encrypt/lib/resolve-eql.ts | 3 +-- packages/cli/src/commands/init/steps/install-eql.ts | 13 ++++++------- packages/stack/src/dynamodb/types.ts | 6 +++--- 3 files changed, 10 insertions(+), 12 deletions(-) diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index f3cb0afc2..b4e595a10 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -75,8 +75,7 @@ export async function resolveColumnLifecycle( * "no pending config", …). Since `classifyEqlDomain` recognises `eql_v3_*` * only, that case now also covers the post-cutover v2 state — `` was * renamed onto the ciphertext, and its `eql_v2_encrypted` domain is no longer - * classified, so the column never appears as a candidate. (It used to arrive - * here as a `version: 2` candidate and needed its own exemption.) + * classified, so the column never appears as a candidate. * * A non-empty candidate list therefore means EQL v3 columns exist but none is * identifiable — the caller must fail closed with this message rather than diff --git a/packages/cli/src/commands/init/steps/install-eql.ts b/packages/cli/src/commands/init/steps/install-eql.ts index f6c912466..31c9e5b9f 100644 --- a/packages/cli/src/commands/init/steps/install-eql.ts +++ b/packages/cli/src/commands/init/steps/install-eql.ts @@ -104,13 +104,12 @@ export const installEqlStep: InitStep = { // --drizzle`) rather than routing through `eql install`. // // `eql install --drizzle` is v2-only — under the v3 default it rejects the - // flag outright, so init used to pin `eqlVersion: '2'` to get a migration - // at all. That pin made `stash init --drizzle` the one flow that provisions - // a v2 database while every other integration (and a bare `stash eql - // install`) gets v3, and it contradicted the stash-drizzle skill we install - // into the very same project — that skill documents the v3 surface - // (`types.*` domains, `Encryption`) and would have the user's agent - // author v3 code against a v2 database. + // flag outright, so routing Drizzle through it would provision a v2 + // database while every other integration (and a bare `stash eql install`) + // gets v3. That also contradicts the stash-drizzle skill installed into the + // very same project, which documents the v3 surface (`types.*` domains, + // `Encryption`) and would have the user's agent author v3 code against a v2 + // database. // // `stash eql migration --drizzle` (added in #691) closes that gap: v3 SQL, // still migration-first, and it bundles the `cs_migrations` tracking schema diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index bee2e952d..cfe37bb68 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -170,9 +170,9 @@ type Simplify = { [K in keyof T]: T[K] } * * A declared column `email` does NOT survive as `email`: the adapter deletes it * and writes `email__source` (plus `email__hmac` for equality domains). Typing - * the result as the input model — what the removed v2 write overload did — is a - * lie that type-checks `result.data.email` (always `undefined` at runtime) and - * rejects `result.data.email__source` (the value you actually want). + * the result as the input model is a lie that type-checks `result.data.email` + * (always `undefined` at runtime) and rejects `result.data.email__source` (the + * value you actually want). * * Keys that name no column pass through untouched — partition/sort keys, GSI * attributes, anything else on the item. From d46c454059c47cf08caa41a661ddcd52c8206fb4 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 20:57:26 +1000 Subject: [PATCH 044/123] docs(specs): fail-closed inversion for the encrypted ALTER COLUMN rewrite --- ...026-07-24-a2-fail-closed-rewrite-design.md | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-24-a2-fail-closed-rewrite-design.md diff --git a/docs/superpowers/specs/2026-07-24-a2-fail-closed-rewrite-design.md b/docs/superpowers/specs/2026-07-24-a2-fail-closed-rewrite-design.md new file mode 100644 index 000000000..f44fa80f4 --- /dev/null +++ b/docs/superpowers/specs/2026-07-24-a2-fail-closed-rewrite-design.md @@ -0,0 +1,203 @@ +# A-2 — fail-closed inversion for the encrypted ALTER COLUMN rewrite + +**Date:** 2026-07-24 +**Addresses:** A-2 in `.work/2026-07-24-eql-v2-removal-verification.md` +**Base:** `05b7bf07` (A-1/A-3 tokenizer fix) +**Affects:** `packages/cli/src/commands/db/rewrite-migrations.ts`, `packages/wizard/src/lib/rewrite-migrations.ts` (the same file twice) + +## Problem + +`rewriteEncryptedAlterColumns` rewrites an in-place `ALTER COLUMN … SET DATA TYPE +` into an ADD + DROP + RENAME sequence. That sequence is +equivalent to DROP + ADD: it fixes the column's type but destroys its contents. + +The sweep decides whether to rewrite by consulting a corpus-wide index of columns +the migrations give an *encrypted* type (`indexEncryptedColumns`). A column found +there is skipped as `already-encrypted`, because dropping it would destroy +ciphertext with no plaintext left to backfill from. **Everything else is +rewritten** — including a column the corpus never mentions at all. + +That default is fail-open. A column absent from the index is not known to be +plaintext; it is simply unknown. Two reproduced triggers, both from the +verification doc: + +- **(a) Lone ALTER, no source declaration.** The corpus contains the ALTER and + nothing that declares the column. Rewritten today: 1 live `DROP COLUMN`. +- **(b) Cross-directory split.** The declaration lives in `drizzle/` and the + ALTER in `migrations/`. The index is built per directory, so the sweep of + `migrations/` never sees the declaration. Rewritten today: 1 live `DROP + COLUMN` — and the dropped column is already typed `eql_v3_text_eq`, i.e. + ciphertext. + +Trigger (b) is not contrived. `post-agent.ts:163` ships with +`DRIZZLE_OUT_DIRS = ['drizzle', 'migrations', 'src/db/migrations']`, so scanning +multiple directories with a per-directory index *is* the default configuration. + +## Decision + +Invert the default. Rewrite only a column the corpus positively declares and does +not give an encrypted type. Anything unknown is skipped and reported. + +Two alternatives were considered and rejected: + +- **Widen the index across directories** (build one corpus index in + `sweepMigrationDirs` and pass it into each per-directory rewrite). Fixes (b) + with a sharper `already-encrypted` message, but unioning unrelated migration + histories can over-detect *plaintext*, which is itself fail-open. Rejected: + fail-closed already makes (b) safe, and this trades a better message for a new + fail-open surface. +- **Require the declaration in the same file.** A column created in + `0001_init.sql` and altered in `0007_encrypt.sql` is the normal case, so this + would stop rewriting anything. + +## Mechanism + +The encrypted side is already precise: it matches against the known domain list +in `ENCRYPTED_DOMAIN`. So "plaintext" needs no type classification — it is the +residue. A column the corpus **declares** but that is not in the encrypted set is +plaintext. + +That turns "what type is this column?" (a SQL parse) into "does a declaration for +this column exist?" (a name match in a position the code already isolates). No +comma splitting, no paren-depth tracking, no quote state machine. + +`indexEncryptedColumns` becomes `indexColumnDeclarations`, one traversal +returning `{ encrypted, declared }`. `encrypted` is populated exactly as today — +the existing broad scan is deliberately over-detecting and must not regress. +`declared` is populated from two positions: + +1. **Inside a `CREATE TABLE` body**, already captured as group 3 of + `CREATE_TABLE_RE`: scan `"([^"]+)"\s+["a-z]` for declared names. +2. **`ALTER TABLE … ADD COLUMN "col" `**: generalise + `ADD_ENCRYPTED_COLUMN_RE` to accept any type with the same tail. + +`RENAME COLUMN "a" TO "b"` propagates `declared` the way it already propagates +`encrypted`. + +The `\s+["a-z]` tail is what makes the name match a declaration rather than a +mention. A type token always begins with a letter or a double quote, so +`"email" text` and `"email" "public"."eql_v3_text_search"` match, while these do +not — the name is followed by `)`, `,` or an operator: + +```sql +PRIMARY KEY ("id", "name") +FOREIGN KEY ("org_id") REFERENCES "orgs"("id") +CHECK ("age" > 0) +UNIQUE("email") +``` + +**The scan must stay anchored to those two positions.** A whole-file scan would +let `ALTER COLUMN "email" SET DATA TYPE …` match its own `"email" SET` and +declare the very column it is asking about — a self-fulfilling fail-open. + +**Known false positive, accepted.** In +`CONSTRAINT "users_email_unique" UNIQUE("email")` the constraint *name* is +followed by whitespace and a letter, so it registers as a declared column. It is +inert unless a constraint name exactly equals the name of a column that is also +encrypted and undeclared; drizzle's `
__unique` convention makes that +contrived. Documented in the docstring rather than engineered around. + +**Corpus-wide, not migration-ordered.** As with the existing encrypted index, a +declaration in a *later* migration than the ALTER still counts. This is a +pre-existing property, not introduced here. + +### Decision order in the rewrite callback + +Unchanged first step, two steps after it: + +1. `isInsideCommentOrString(original, offset)` → return the match untouched. +2. In `encrypted` → `skip(…, 'already-encrypted')`. +3. Not in `declared` → `skip(…, 'source-unknown')`. +4. Otherwise rewrite. + +Step 2 must precede step 3 so a column that is both declared and encrypted — the +output of a previous sweep, which emits `ADD COLUMN ` plus +`RENAME TO ` — still reports `already-encrypted`, the more specific +and more actionable reason. + +`isInsideCommentOrString` is not touched: it is the subject of A-1/A-3, already +landed in the base commit. + +## Skip reason + +`SkipReason` gains `'source-unknown'`. `describeSkipReason` has no `default` arm, +so the compiler forces the new arm to be written. All three consumers +(`install.ts:660`, `eql/migration.ts:248`, `post-agent.ts:192`) render +`describeSkipReason(reason)` verbatim and need no edit. + +The text names both causes, because the user's remedy differs: + +> the sweep could not find where this column was declared in this migration +> directory, so it cannot tell a plaintext column (safe to rewrite) from one that +> already holds ciphertext (where the rewrite would DROP it). Usually the +> column's `CREATE TABLE` / `ADD COLUMN` lives in a different directory, or the +> migration history was squashed. Check the column's current type in the +> database: if it is plaintext and the table is empty, apply the +> ADD/DROP/RENAME by hand; if it already holds ciphertext, use the staged +> `stash encrypt` lifecycle instead + +## Blast radius + +This is a behaviour change for correct corpora, not only for the bug. Any project +whose swept directory does not contain the column's declaration — squashed +migrations, a baseline pulled with `drizzle-kit pull`, a schema bootstrapped from +a hand-written SQL file — stops being rewritten and starts being flagged for +review. That is the trade being bought, and it is the correct direction: the cost +of a false skip is a statement the user fixes by hand, and the cost of a false +rewrite is irrecoverable data. + +## Testing + +Most existing tests in both files use a bare ALTER with no `CREATE` anywhere and +assert a rewrite; under the inversion they would all become `source-unknown`. +Rather than rewrite each fixture, add a helper that writes a `0000_init.sql` +declaring the plaintext source column. It sorts first and is never rewritten, so +`rewritten`/`skipped` assertions stay as they are and the diff is one line per +test. + +New cases, in **both** test files: + +- Lone ALTER, no declaration anywhere → `skipped` with `source-unknown`, file + unchanged on disk, zero `DROP COLUMN`. +- Declaration in a sibling file in the same directory → rewritten (the + non-regression case). +- Declaration living in the `options.skip` file → still counts as declared. +- `RENAME COLUMN` propagates declared-ness from the original name. +- A column named only inside `PRIMARY KEY (…)` / `REFERENCES "t"("col")` → + **not** declared, so `source-unknown`. +- A column that is both declared and encrypted → still `already-encrypted`, not + `source-unknown` (pins the ordering of steps 2 and 3). + +Wizard file only: + +- The cross-directory split from the verification doc, driven through + `sweepMigrationDirs`: encrypted declaration in `drizzle/`, ALTER in + `migrations/` → `source-unknown`, zero `DROP COLUMN` in the output. + +## Dual-copy sync + +The two files are the same file twice, differing only in the wizard's +`sweepMigrationDirs` / `DirRewriteResult`, its attribution string +(`@cipherstash/wizard` vs `stash`), and a few comment blocks. Apply one fix +twice, then diff the two whitespace-normalised files to confirm nothing new +diverged. + +## Docs and changeset + +- `skills/stash-cli/SKILL.md:393` and `skills/stash-drizzle/SKILL.md:64` both + state that each such statement is rewritten. Both need the qualifier that the + rewrite now requires the column's declaration to be present in the swept + directory, and otherwise flags the statement for review. +- Extend `.changeset/rewriter-never-drops-ciphertext.md` rather than adding a + second changeset: it already ships the sibling `already-encrypted` behaviour + for the same component in the same release, and two changesets would describe + overlapping behaviour. Both `stash` and `@cipherstash/wizard` are already + listed. + +## Out of scope + +- Any change to `isInsideCommentOrString` (A-1/A-3, landed in the base commit). +- Sharing a corpus index across directories in `sweepMigrationDirs` (rejected + above). +- The near-miss scan (`NEAR_MISS_RE`) and its `unrecognised-form` reason, which + are unaffected. From 46f30b4975d1e09b3bff8961cdc423a3bce9ad51 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:14:00 +1000 Subject: [PATCH 045/123] fix(cli): rewrite an encrypted ALTER only for a column the corpus declares A column missing from the encrypted index was treated as plaintext and rewritten into ADD+DROP+RENAME. Absence is not evidence: the declaration can live in a migration directory the sweep never reads, so the DROP COLUMN could land on a column already holding ciphertext. The corpus index now records declared columns as well as encrypted ones, and an ALTER whose column is declared nowhere is reported as source-unknown instead of rewritten. --- .../src/__tests__/rewrite-migrations.test.ts | 215 ++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 129 +++++++++-- 2 files changed, 323 insertions(+), 21 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 4f449965e..ac38926de 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -15,7 +15,28 @@ describe('rewriteEncryptedAlterColumns', () => { fs.rmSync(tmpDir, { recursive: true, force: true }) }) + /** + * Declare `columns` on `tableRef` as PLAINTEXT, in a migration that sorts + * before every fixture below. + * + * The sweep is fail-closed: it rewrites a column only when the corpus shows + * the column exists and is not already encrypted. A fixture that is just an + * ALTER declares nothing, so it is `source-unknown` by design — a test that + * exercises the REWRITE has to supply the `CREATE TABLE` a real drizzle + * corpus would carry. + * + * `tableRef` is written exactly as it appears in the ALTER, so a pgSchema() + * table passes `'"app"."users"'` and the declaration lands on the same key. + */ + const declarePlaintext = (tableRef: string, ...columns: string[]): void => { + const file = path.join(tmpDir, '0000_declare.sql') + const existing = fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : '' + const defs = columns.map((column) => `"${column}" text`).join(', ') + fs.writeFileSync(file, `${existing}CREATE TABLE ${tableRef} (${defs});\n`) + } + it('rewrites an in-place ALTER COLUMN with the bare type name', async () => { + declarePlaintext('"transactions"', 'amount') const original = `ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE eql_v2_encrypted;\n` const filePath = path.join(tmpDir, '0002_alter.sql') fs.writeFileSync(filePath, original) @@ -37,6 +58,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the schema-qualified form produced by drizzle-kit', async () => { + declarePlaintext('"users"', 'email') const original = 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v2_encrypted";\n' const filePath = path.join(tmpDir, '0003_alter.sql') @@ -54,6 +76,7 @@ describe('rewriteEncryptedAlterColumns', () => { it('rewrites a schema-qualified table produced by pgSchema()', async () => { // drizzle-kit emits `"app"."users"` for a table declared in a pgSchema(); // the old `\s+` between the table and ALTER COLUMN could never cross the `.`. + declarePlaintext('"app"."users"', 'email') const original = 'ALTER TABLE "app"."users" ALTER COLUMN "email" SET DATA TYPE "undefined"."eql_v3_text_search";\n' const filePath = path.join(tmpDir, '0014_qualified.sql') @@ -75,6 +98,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the "undefined" schema form drizzle-kit emits for bare custom types', async () => { + declarePlaintext('"transactions"', 'amount') const original = 'ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE "undefined"."eql_v2_encrypted";\n' const filePath = path.join(tmpDir, '0005_undef.sql') @@ -90,6 +114,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the double-quoted form produced by stack 0.15.0', async () => { + declarePlaintext('"transactions"', 'description') const original = 'ALTER TABLE "transactions" ALTER COLUMN "description" SET DATA TYPE "undefined".""public"."eql_v2_encrypted"";\n' const filePath = path.join(tmpDir, '0006_double.sql') @@ -117,6 +142,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('skips the file passed in options.skip', async () => { + declarePlaintext('"t"', 'c') const install = path.join(tmpDir, '0000_install-eql.sql') const alter = path.join(tmpDir, '0002_alter.sql') fs.writeFileSync(install, 'CREATE SCHEMA eql_v2;\n') @@ -169,6 +195,7 @@ describe('rewriteEncryptedAlterColumns', () => { ] it.each(V3_DOMAINS)('rewrites an ALTER COLUMN to %s', async (domain) => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0007_v3.sql') fs.writeFileSync( filePath, @@ -196,6 +223,7 @@ describe('rewriteEncryptedAlterColumns', () => { ...V3_DOMAINS, 'eql_v2_encrypted', ])('extracts the bare domain %s from a mangled ALTER', async (domain) => { + declarePlaintext('"t"', 'c') const filePath = path.join(tmpDir, '0015_drift.sql') fs.writeFileSync( filePath, @@ -243,6 +271,7 @@ describe('rewriteEncryptedAlterColumns', () => { ] it.each(MANGLED_FORMS)('rewrites the v3 %s form', async (_label, emitted) => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0008_form.sql') fs.writeFileSync( filePath, @@ -265,6 +294,7 @@ describe('rewriteEncryptedAlterColumns', () => { '"undefined"."public.eql_v2_encrypted"', ], ])('rewrites the previously unmatched v2 %s form', async (_label, emitted) => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0009_v2form.sql') fs.writeFileSync( filePath, @@ -281,6 +311,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('names the target domain in the guidance comment', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0010_comment.sql') fs.writeFileSync( filePath, @@ -295,6 +326,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('notes that constraints/defaults/indexes are not carried over', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0016_constraints.sql') fs.writeFileSync( filePath, @@ -309,6 +341,7 @@ describe('rewriteEncryptedAlterColumns', () => { it('does not terminate the commented UPDATE placeholder with a semicolon', async () => { // A runner that naively splits on `;` must not cut mid-comment. + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0017_semicolon.sql') fs.writeFileSync( filePath, @@ -328,6 +361,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('separates ADD/DROP/RENAME with --> statement-breakpoint, one exec stmt per chunk', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0018_breakpoint.sql') fs.writeFileSync( filePath, @@ -353,6 +387,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites each statement to its own domain when v2 and v3 are mixed', async () => { + declarePlaintext('"a"', 'x', 'y') const filePath = path.join(tmpDir, '0011_mixed.sql') fs.writeFileSync( filePath, @@ -495,6 +530,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('reports no skipped statements when the strict rewrite fully handled the file', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0021_handled.sql') fs.writeFileSync( filePath, @@ -577,6 +613,7 @@ describe('rewriteEncryptedAlterColumns', () => { // The comment scan must not be fooled by `--` inside a string literal, or // it would skip a live statement and leave broken SQL to fail at migrate. it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0030_literal.sql') fs.writeFileSync( filePath, @@ -793,6 +830,7 @@ describe('rewriteEncryptedAlterColumns', () => { // The scoping matters: encrypting `contacts.email` must not be blocked by // an unrelated `users.email` that happens to share a column name. it('scopes the check to the table', async () => { + declarePlaintext('"contacts"', 'email') fs.writeFileSync( path.join(tmpDir, '0000_add.sql'), 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', @@ -830,6 +868,7 @@ describe('rewriteEncryptedAlterColumns', () => { // A commented-out ADD never ran, so it says nothing about the live schema. it('ignores an encrypted ADD COLUMN that is commented out', async () => { + declarePlaintext('"users"', 'email') fs.writeFileSync( path.join(tmpDir, '0000_add.sql'), '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', @@ -868,9 +907,42 @@ describe('rewriteEncryptedAlterColumns', () => { expect(rewritten).toEqual([]) expect(skipped[0]?.reason).toBe('already-encrypted') }) + + // Ordering pin: this column is BOTH declared plaintext (0000) and made + // encrypted by a previous sweep (0001). `already-encrypted` is the more + // specific reason and must win over `source-unknown`. + it('reports already-encrypted even when the column was also declared plaintext', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" text);\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) }) it('handles multiple ALTER statements in one file', async () => { + declarePlaintext('"a"', 'x', 'y') const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v2_encrypted;', 'ALTER TABLE "a" ALTER COLUMN "y" SET DATA TYPE eql_v2_encrypted;', @@ -891,6 +963,7 @@ describe('rewriteEncryptedAlterColumns', () => { // Regression pin, not a bug fix — the matchers carry `/gi`, so a // hand-lowercased migration is rewritten just like drizzle-kit's output. it('rewrites a lowercase alter table ... set data type', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0034_lowercase.sql') fs.writeFileSync( filePath, @@ -908,4 +981,146 @@ describe('rewriteEncryptedAlterColumns', () => { expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') expect(updated).not.toMatch(/set data type/i) }) + + // A-2: the sweep is FAIL-CLOSED. A column the corpus never declares is + // UNKNOWN, not plaintext — it may already hold ciphertext, with its + // declaration sitting in a migration directory this sweep never sees. + describe('columns the corpus does not declare', () => { + it('refuses to rewrite a column the corpus never declares', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(`${alterSql}\n`) + expect(updated).not.toContain('DROP COLUMN') + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + it('rewrites a column declared plaintext by an ADD COLUMN', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" text;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The skipped file is still part of the corpus: a column's current type + // comes from the migrations that ran before this one, edit-eligible or not. + it('counts a declaration living in the file passed to options.skip', async () => { + const install = path.join(tmpDir, '0000_install-eql.sql') + fs.writeFileSync( + install, + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" text);\n', + ) + const alter = path.join(tmpDir, '0002_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: install, + }, + ) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + it('follows a RENAME when deciding a column is declared', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email_address" text);', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email_address" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A name inside a table/key constraint is a MENTION, not a declaration — + // it is followed by `)` or `,`, never by a type token. Counting one would + // put the rewrite back on the fail-open path. + it('does not treat a name inside PRIMARY KEY / REFERENCES as a declaration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "sessions" (', + '\t"id" integer,', + '\t"user_id" integer,', + '\tPRIMARY KEY ("id", "email"),', + '\tFOREIGN KEY ("user_id") REFERENCES "users"("email")', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "sessions" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // A column line commented out INSIDE a live CREATE TABLE never ran, so it + // declares nothing. + it('does not count a column commented out inside a live CREATE TABLE', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t-- "email" text,', + '\t"name" text', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('source-unknown') + }) + }) }) diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 550cb282a..789ca43a3 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -251,6 +251,40 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( 'gi', ) +/** + * A column name in a DECLARATION position — `"email" text`, + * `"email" "public"."eql_v3_text_search"`, `"amount" numeric(10, 2)`. + * + * The `\s+["a-z]` tail is the whole trick, and it is what separates a + * declaration from a MENTION. Every SQL type token begins with a letter or a + * double quote, so a name followed by `)`, `,` or an operator does not match: + * + * ```sql + * PRIMARY KEY ("id", "name") + * FOREIGN KEY ("org_id") REFERENCES "orgs"("id") + * CHECK ("age" > 0) + * ``` + * + * Deliberately NOT pinned to the encrypted domains. A column the corpus + * declares but does not give an encrypted type is, by residue, plaintext — so + * the fail-closed rule needs no type classification and no SQL parsing, only + * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. + * + * **Known false positive, accepted.** In + * `CONSTRAINT "users_email_unique" UNIQUE("email")` the constraint NAME is + * followed by whitespace and a letter, so it registers as a declared column. It + * is inert unless a constraint name exactly equals the name of a column that is + * also encrypted and undeclared, which drizzle's `
__unique` + * convention makes contrived. + */ +const DECLARED_COLUMN_RE = /"([^"]+)"\s+["a-z]/gi + +/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +const ADD_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+["a-z]`, + 'gi', +) + /** Splits a `TABLE_REF` capture pair into its schema and table halves. */ function tableOf( first: string, @@ -263,42 +297,72 @@ function tableOf( : { schema: first, table: second } } -/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +/** Identity of a column across the corpus, for {@link indexColumnDeclarations}. */ function columnKey(table: string, column: string, schema?: string): string { return JSON.stringify([schema ?? '', table, column]) } +/** What the migration corpus says about the columns it mentions. */ +interface ColumnIndex { + /** Columns the corpus gives an ENCRYPTED type. Rewriting one drops ciphertext. */ + encrypted: Set + /** Columns the corpus DECLARES at all, whatever type it gives them. */ + declared: Set +} + /** - * Index every column the migration corpus gives an ENCRYPTED type, so the - * rewrite can tell the change it exists for (plaintext → encrypted) from one it - * must never touch (encrypted → encrypted). + * Index what the migration corpus knows about each column, so the rewrite can + * tell the change it exists for (plaintext → encrypted) from the two it must + * never make: encrypted → encrypted, and a change to a column it knows nothing + * about. * - * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. - * A column whose encrypted domain merely changes (`types.TextEq` → + * **Why `encrypted` (#772 review, W-3):** the strict matcher captures only the + * TARGET type. A column whose encrypted domain merely changes (`types.TextEq` → * `types.TextSearch`) matches just as well as a plaintext column, and the * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext * left anywhere to backfill from, so unlike the plaintext case the data is not - * recoverable from the application at all. Changing an encrypted column's domain - * changes its index terms, so the data has to be re-encrypted through the client - * regardless; the sweep flags the statement and leaves it for the user. + * recoverable from the application at all. + * + * **Why `declared` (A-2):** absence from `encrypted` is not evidence of + * plaintext. It is evidence of nothing. The sweep runs per directory, and the + * shipped wizard default scans three of them, so a column's `CREATE TABLE` can + * simply live somewhere this sweep never reads — the column is then rewritten + * on an assumption, and the ADD+DROP+RENAME drops live ciphertext. Requiring a + * POSITIVE declaration makes the unknown case a flagged statement instead. + * + * Because the encrypted side is matched against the known domain list, the + * plaintext side needs no type classification: a column the corpus declares but + * does not give an encrypted type is plaintext by residue. * * The index is corpus-wide rather than ordered by migration: over-detecting - * "encrypted" costs a flagged statement the user must handle by hand, while - * under-detecting costs irrecoverable ciphertext. Only comment-free statements - * count, for the same reason {@link isInsideCommentOrString} exists. + * "encrypted" costs a flagged statement the user handles by hand, while + * under-detecting costs irrecoverable ciphertext. Only live statements count, + * for the same reason {@link isInsideCommentOrString} exists — inside a + * `CREATE TABLE` the check is applied per column line, so a commented-out + * column in an otherwise live table declares nothing. */ -function indexEncryptedColumns(contents: readonly string[]): Set { +function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { const encrypted = new Set() + const declared = new Set() for (const sql of contents) { for (const created of sql.matchAll(CREATE_TABLE_RE)) { if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) - for (const column of created[3].matchAll( - CREATE_TABLE_ENCRYPTED_COLUMN_RE, - )) { + const body = created[3] + + // The body is scanned as its own document: a `--` earlier in it comments + // out the rest of that line, and a CREATE inside a block comment or a + // string literal was already skipped above. + for (const column of body.matchAll(CREATE_TABLE_ENCRYPTED_COLUMN_RE)) { + if (isInsideCommentOrString(body, column.index)) continue encrypted.add(columnKey(table, column[1], schema)) } + + for (const column of body.matchAll(DECLARED_COLUMN_RE)) { + if (isInsideCommentOrString(body, column.index)) continue + declared.add(columnKey(table, column[1], schema)) + } } for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { @@ -307,19 +371,26 @@ function indexEncryptedColumns(contents: readonly string[]): Set { encrypted.add(columnKey(table, added[3], schema)) } + for (const added of sql.matchAll(ADD_COLUMN_RE)) { + if (isInsideCommentOrString(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + declared.add(columnKey(table, added[3], schema)) + } + // A rename carries the column's type with it — and `__cipherstash_tmp` // renamed onto the real name is exactly what a previous sweep of this very // directory emitted. Run after ADD so that tmp column is already indexed. for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { if (isInsideCommentOrString(sql, renamed.index)) continue const { schema, table } = tableOf(renamed[1], renamed[2]) - if (encrypted.has(columnKey(table, renamed[3], schema))) { - encrypted.add(columnKey(table, renamed[4], schema)) - } + const from = columnKey(table, renamed[3], schema) + const to = columnKey(table, renamed[4], schema) + if (encrypted.has(from)) encrypted.add(to) + if (declared.has(from)) declared.add(to) } } - return encrypted + return { encrypted, declared } } /** Why a recognised ALTER-to-encrypted statement was left alone. */ @@ -328,6 +399,8 @@ export type SkipReason = | 'unrecognised-form' /** The column already holds an encrypted domain; rewriting drops ciphertext. */ | 'already-encrypted' + /** The corpus never declares the column, so its current type is unknown. */ + | 'source-unknown' /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { @@ -351,6 +424,8 @@ export function describeSkipReason(reason: SkipReason): string { return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" case 'unrecognised-form': return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + case 'source-unknown': + return "the sweep could not find where this column was declared in this migration directory, so it cannot tell a plaintext column (safe to rewrite) from one that already holds ciphertext (where the rewrite would DROP it). Usually the column's `CREATE TABLE` / `ADD COLUMN` lives in a different directory, or the migration history was squashed. Check the column's current type in the database: if it is plaintext and the table is empty, apply the ADD/DROP/RENAME by hand; if it already holds ciphertext, use the staged `stash encrypt` lifecycle instead" } } @@ -432,7 +507,8 @@ export async function rewriteEncryptedAlterColumns( // Built from the WHOLE corpus, including `options.skip`: a column's current // type comes from the migrations that ran before this one, not just the files // we are allowed to edit. - const encryptedColumns = indexEncryptedColumns([...contents.values()]) + const { encrypted: encryptedColumns, declared: declaredColumns } = + indexColumnDeclarations([...contents.values()]) for (const [filePath, original] of contents) { if (options.skip && filePath === options.skip) continue @@ -463,6 +539,17 @@ export async function rewriteEncryptedAlterColumns( return match } + // Fail closed. Absence from `encryptedColumns` does not mean the column + // is plaintext — it means the corpus never said. The declaration can + // sit in a directory this sweep does not read (the wizard ships with + // three candidates and indexes each separately), so rewriting on the + // assumption is how a live `DROP COLUMN` reaches a populated — possibly + // already-encrypted — column. Flag it and let the user look. + if (!declaredColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'source-unknown') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. From 034e804f18f24118a8d48d7e1b1c80bf2ab4f42e Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:35:53 +1000 Subject: [PATCH 046/123] fix(cli): close the keyword-mention gap and comment-check regression in the rewrite sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code review of the fail-closed rewrite (3fedb33e) found two Important defects, both traced to the plan's own supplied code rather than its transcription: - DECLARED_COLUMN_RE matched any quoted name followed by whitespace and a letter, so a mention inside a predicate or constraint — e.g. CHECK ("email" IS NOT NULL), or a constraint named the same as a column — registered as a declaration with no real declaration ever existing, walking the fail-closed sweep back onto the fail-open path it exists to close. Fixed with a word-boundary-pinned negative lookahead over the SQL keywords reachable in that position, verified against every real type token that shares a keyword's leading letters (interval/inet/int vs IN, char/citext vs CHECK, eql_v2_encrypted/eql_v3_* vs EXCLUDE, etc). - The per-column comment check landed on BOTH body scans inside indexColumnDeclarations, but over-detecting "encrypted" (safe) and over-detecting "declared" (not safe) are not symmetric. Applying it to the encrypted scan turned a commented-out encrypted column inside an otherwise-live CREATE TABLE from over-detected (safe, pre-existing behaviour) to under-detected — exactly the "irrecoverable ciphertext" case the function's own docstring warns against. Removed the check from the encrypted scan only; kept it on the declared scan. Also corrected two doc comments that asserted the old (wrong) behaviour, and three minor JSDoc/comment inaccuracies (skip-reason count, missing source-unknown in the exported contract, a comment naming the wrong set). Added tests pinning both fixes: a CHECK predicate mention and a same-named constraint no longer count as declarations; a type name that merely starts with a blocked keyword's letters (interval/inet) still does; and a commented-out encrypted column beside a live redeclaration still reports already-encrypted rather than being rewritten. --- .../src/__tests__/rewrite-migrations.test.ts | 126 +++++++++++++++++- .../cli/src/commands/db/rewrite-migrations.ts | 89 +++++++++---- 2 files changed, 187 insertions(+), 28 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index ac38926de..c830b6001 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -939,6 +939,37 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toHaveLength(1) expect(skipped[0].reason).toBe('already-encrypted') }) + + // A commented-out encrypted column line inside an otherwise LIVE + // CREATE TABLE must still count as encrypted. The comment check applies + // only to the DECLARED scan (over-detecting plaintext costs data); the + // ENCRYPTED scan never re-checks comments inside a live CREATE TABLE, so + // this stays over-detecting — the safe direction — exactly as it was + // before the corpus learned to track declarations at all. + it('still counts a commented-out encrypted column inside a live CREATE TABLE as encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer,', + '\t-- "email" "public"."eql_v3_text_eq",', + '\t"email" text', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) }) it('handles multiple ALTER statements in one file', async () => { @@ -1067,9 +1098,12 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) }) - // A name inside a table/key constraint is a MENTION, not a declaration — - // it is followed by `)` or `,`, never by a type token. Counting one would - // put the rewrite back on the fail-open path. + // A name inside a table/key constraint is a MENTION, not a declaration. + // Here every mention is followed by `)` or `,`, which the declaration + // regex's tail alone already rejects. A mention followed by a SQL keyword + // instead (e.g. `CHECK ("email" IS NOT NULL)`) is a separate case, closed + // by the regex's keyword lookahead and pinned by its own tests below. + // Counting either would put the rewrite back on the fail-open path. it('does not treat a name inside PRIMARY KEY / REFERENCES as a declaration', async () => { fs.writeFileSync( path.join(tmpDir, '0000_create.sql'), @@ -1122,5 +1156,91 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toHaveLength(1) expect(skipped[0].reason).toBe('source-unknown') }) + + // A CHECK predicate mentioning a column has the same `"name" ` + // shape as a declaration — `"email" IS` — but IS is a predicate keyword, + // not a type token. Without the keyword lookahead this would read as a + // declaration and rewrite the column, dropping any ciphertext it already + // holds via a declaration this sweep never sees. + it('does not treat a CHECK predicate mention as a declaration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\tCONSTRAINT "c1" CHECK ("email" IS NOT NULL)', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(`${alterSql}\n`) + expect(updated).not.toContain('DROP COLUMN') + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // A constraint's NAME can coincide with a column's name — drizzle's + // `
__unique` convention makes this contrived, but the keyword + // lookahead closes it regardless: a constraint name is always followed + // immediately by its constraint-type keyword (UNIQUE here), which the + // lookahead excludes. + it('does not let a same-named constraint declare the column', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\tCONSTRAINT "email" UNIQUE("id")', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // The keyword lookahead is pinned to a word boundary specifically so it + // does not eat real type names that merely START WITH a blocked + // keyword's letters: `interval`/`inet` both start "in" (colliding with + // IN) but must still count as genuine declarations. + it('still declares a column whose type name starts with a blocked keyword', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "events" ("email" interval, "note" inet);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + [ + 'ALTER TABLE "events" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + 'ALTER TABLE "events" ALTER COLUMN "note" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) }) }) diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 789ca43a3..7e1de20ec 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -255,29 +255,57 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * A column name in a DECLARATION position — `"email" text`, * `"email" "public"."eql_v3_text_search"`, `"amount" numeric(10, 2)`. * - * The `\s+["a-z]` tail is the whole trick, and it is what separates a - * declaration from a MENTION. Every SQL type token begins with a letter or a - * double quote, so a name followed by `)`, `,` or an operator does not match: + * The `\s+["a-z]` tail is what separates a declaration from a MENTION. Most + * mentions are followed by `)`, `,`, or an operator, which the tail alone + * already rejects: * * ```sql * PRIMARY KEY ("id", "name") * FOREIGN KEY ("org_id") REFERENCES "orgs"("id") - * CHECK ("age" > 0) * ``` * + * But a mention inside a predicate or a table-level constraint is often + * followed by whitespace and a SQL KEYWORD — which has exactly the same + * `\s+[a-z]` shape as a real declaration, and drizzle-kit emits both forms + * inside a `CREATE TABLE` (a `CONSTRAINT … CHECK ()` body is + * user-authored, so the predicate form is reachable too): + * + * ```sql + * CHECK ("email" IS NOT NULL) + * CONSTRAINT "users_email_unique" UNIQUE ("email") + * ``` + * + * The negative lookahead rejects the keywords reachable this way: predicate + * keywords (`IS`, `IN`, `NOT`, `LIKE`, `ILIKE`, `BETWEEN`, `AND`, `OR`), + * ordering/collation (`COLLATE`, `ASC`, `DESC`, `NULLS`), and every + * table/column-constraint keyword (`UNIQUE`, `PRIMARY`, `FOREIGN`, `CHECK`, + * `EXCLUDE`, `REFERENCES`, `CONSTRAINT`, `USING`, `WITH`) — the last six also + * close the constraint-NAME case above, since Postgres always puts one of + * them immediately after a constraint's name. + * + * A real type token still matches because the lookahead is pinned to a word + * boundary (`\b`) right after each keyword, so it only rejects a keyword when + * that keyword is the WHOLE next word: `interval` and `inet` both start with + * `IN`'s letters, but their third character keeps them inside one word, so + * `\bIN\b` never matches there; the same reasoning clears `int`, + * `char`/`citext` against `CHECK`, and `eql_v2_encrypted`/`eql_v3_*` against + * `EXCLUDE`. + * * Deliberately NOT pinned to the encrypted domains. A column the corpus * declares but does not give an encrypted type is, by residue, plaintext — so * the fail-closed rule needs no type classification and no SQL parsing, only * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. * - * **Known false positive, accepted.** In - * `CONSTRAINT "users_email_unique" UNIQUE("email")` the constraint NAME is - * followed by whitespace and a letter, so it registers as a declared column. It - * is inert unless a constraint name exactly equals the name of a column that is - * also encrypted and undeclared, which drizzle's `
__unique` - * convention makes contrived. + * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: + * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, + * `OVERLAPS`, …) still lets a bare mention through, e.g. + * `CHECK ("email" SIMILAR TO '...')`. This is inert unless that mention's name + * exactly matches a DIFFERENT column that is both encrypted and genuinely + * undeclared anywhere else in the corpus — contrived, and strictly narrower + * than the gap this replaces. */ -const DECLARED_COLUMN_RE = /"([^"]+)"\s+["a-z]/gi +const DECLARED_COLUMN_RE = + /"([^"]+)"\s+(?!(?:IS|IN|NOT|LIKE|ILIKE|BETWEEN|AND|OR|COLLATE|ASC|DESC|NULLS|UNIQUE|PRIMARY|FOREIGN|CHECK|EXCLUDE|REFERENCES|CONSTRAINT|USING|WITH)\b)["a-z]/gi /** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ const ADD_COLUMN_RE = new RegExp( @@ -336,10 +364,10 @@ interface ColumnIndex { * * The index is corpus-wide rather than ordered by migration: over-detecting * "encrypted" costs a flagged statement the user handles by hand, while - * under-detecting costs irrecoverable ciphertext. Only live statements count, - * for the same reason {@link isInsideCommentOrString} exists — inside a - * `CREATE TABLE` the check is applied per column line, so a commented-out - * column in an otherwise live table declares nothing. + * under-detecting costs irrecoverable ciphertext. That asymmetry is why the + * two per-column scans inside a `CREATE TABLE` body are deliberately + * inconsistent about comments (see the comment at the loops themselves) — + * do not "fix" that inconsistency, it is the point. */ function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { const encrypted = new Set() @@ -354,8 +382,16 @@ function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { // The body is scanned as its own document: a `--` earlier in it comments // out the rest of that line, and a CREATE inside a block comment or a // string literal was already skipped above. + // + // Asymmetric on purpose. ENCRYPTED does NOT re-check comments here: a + // commented-out encrypted column inside an otherwise live CREATE TABLE + // still counts, exactly as it did before this sweep learned to declare + // anything — over-detecting "encrypted" only costs a flagged statement. + // DECLARED DOES re-check: a commented-out plaintext line never ran, so + // it must not count as a declaration — over-detecting "declared" is + // what would put a truly-undeclared, possibly already-encrypted column + // back on the fail-open path. for (const column of body.matchAll(CREATE_TABLE_ENCRYPTED_COLUMN_RE)) { - if (isInsideCommentOrString(body, column.index)) continue encrypted.add(columnKey(table, column[1], schema)) } @@ -415,7 +451,7 @@ export interface SkippedAlter { /** * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print * next to the statement. Lives here so every caller says the same thing — the - * two reasons need very different action from the user, and a single generic + * three reasons need very different action from the user, and a single generic * "could not rewrite automatically" hides that. */ export function describeSkipReason(reason: SkipReason): string { @@ -461,13 +497,16 @@ export interface RewriteResult { * * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements * left for a human — ones outside the strict matcher (a hand-authored - * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting - * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. - * Both are left untouched on disk and surfaced non-fatally so the caller can - * tell the user to review them, rather than silently shipping broken SQL or - * destroying data. Statements sitting inside a SQL comment — or inside a - * single-quoted string literal, where they are data rather than SQL — are inert - * and are neither rewritten nor reported. + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form); ones targeting a + * column that is ALREADY encrypted, where the rewrite would drop ciphertext; + * and ones targeting a column the corpus never DECLARES anywhere, where the + * rewrite would be guessing at a type it cannot see (likely the most common + * skip on a real corpus, since the sweep only ever reads part of the + * migration history). All three are left untouched on disk and surfaced + * non-fatally so the caller can tell the user to review them, rather than + * silently shipping broken SQL or destroying data. Statements sitting inside a + * SQL comment — or inside a single-quoted string literal, where they are data + * rather than SQL — are inert and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, @@ -539,7 +578,7 @@ export async function rewriteEncryptedAlterColumns( return match } - // Fail closed. Absence from `encryptedColumns` does not mean the column + // Fail closed. Absence from `declaredColumns` does not mean the column // is plaintext — it means the corpus never said. The declaration can // sit in a directory this sweep does not read (the wizard ships with // three candidates and indexes each separately), so rewriting on the From 923d6fbec3f355f3ab21e9ae50d1a52ca15b580d Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:56:24 +1000 Subject: [PATCH 047/123] fix(wizard): rewrite an encrypted ALTER only for a column the corpus declares Mirrors the CLI fail-closed fix (3fedb33e, bcc573c6) into the wizard's copy of the sweep. This is where the fail-open default bit hardest: the wizard ships scanning drizzle/, migrations/ and src/db/migrations/ with a per-directory index, so a column declared in one directory and altered in another was rewritten on an assumption, and the ADD+DROP+ RENAME dropped live ciphertext with nothing anywhere to backfill from. Also documents two residual gaps found in Task 1's review, in both copies: a REFERENCES ... ON DELETE CASCADE mention can coincide with a column name (DECLARED_COLUMN_RE's keyword lookahead doesn't cover ON), and ADD_COLUMN_RE needs no such lookahead since the token after an ADD COLUMN's column name is always its type. Ports the CLI's full current test coverage (not just the original 7 tests, since a second review-fix commit added 4 more pinning the keyword lookahead and the comment-check asymmetry) plus a wizard-specific regression test for the per-directory index gap. --- .../cli/src/commands/db/rewrite-migrations.ts | 23 +- .../src/__tests__/rewrite-migrations.test.ts | 375 +++++++++++++++++- packages/wizard/src/lib/rewrite-migrations.ts | 195 +++++++-- 3 files changed, 556 insertions(+), 37 deletions(-) diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 7e1de20ec..e82e5cedf 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -298,16 +298,27 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, - * `OVERLAPS`, …) still lets a bare mention through, e.g. - * `CHECK ("email" SIMILAR TO '...')`. This is inert unless that mention's name - * exactly matches a DIFFERENT column that is both encrypted and genuinely - * undeclared anywhere else in the corpus — contrived, and strictly narrower - * than the gap this replaces. + * `OVERLAPS`, `ON`, …) still lets a bare mention through, e.g. + * `CHECK ("email" SIMILAR TO '...')`, or a `REFERENCES "email" ON DELETE + * CASCADE` where `"email"` names the referenced TABLE rather than the column + * being declared — `"id" integer REFERENCES "email" ON DELETE CASCADE` inside + * a `CREATE TABLE` body still registers `email` as declared. That one only + * bites when a column shares a referenced table's name, so like the rest of + * this residue it is inert unless that mention's name exactly matches a + * DIFFERENT column that is both encrypted and genuinely undeclared anywhere + * else in the corpus — contrived, and strictly narrower than the gap this + * replaces. */ const DECLARED_COLUMN_RE = /"([^"]+)"\s+(?!(?:IS|IN|NOT|LIKE|ILIKE|BETWEEN|AND|OR|COLLATE|ASC|DESC|NULLS|UNIQUE|PRIMARY|FOREIGN|CHECK|EXCLUDE|REFERENCES|CONSTRAINT|USING|WITH)\b)["a-z]/gi -/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +/** + * `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. + * + * Needs no {@link DECLARED_COLUMN_RE}-style keyword lookahead: the token + * right after an ADD COLUMN's column name is always its type, so no SQL + * keyword can occupy that position. + */ const ADD_COLUMN_RE = new RegExp( String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+["a-z]`, 'gi', diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index f5ef3ee45..9edc41bb3 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -18,7 +18,28 @@ describe('rewriteEncryptedAlterColumns', () => { fs.rmSync(tmpDir, { recursive: true, force: true }) }) + /** + * Declare `columns` on `tableRef` as PLAINTEXT, in a migration that sorts + * before every fixture below. + * + * The sweep is fail-closed: it rewrites a column only when the corpus shows + * the column exists and is not already encrypted. A fixture that is just an + * ALTER declares nothing, so it is `source-unknown` by design — a test that + * exercises the REWRITE has to supply the `CREATE TABLE` a real drizzle + * corpus would carry. + * + * `tableRef` is written exactly as it appears in the ALTER, so a pgSchema() + * table passes `'"app"."users"'` and the declaration lands on the same key. + */ + const declarePlaintext = (tableRef: string, ...columns: string[]): void => { + const file = path.join(tmpDir, '0000_declare.sql') + const existing = fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : '' + const defs = columns.map((column) => `"${column}" text`).join(', ') + fs.writeFileSync(file, `${existing}CREATE TABLE ${tableRef} (${defs});\n`) + } + it('rewrites an in-place ALTER COLUMN with the bare v2 type name', async () => { + declarePlaintext('"transactions"', 'amount') const original = `ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE eql_v2_encrypted;\n` const filePath = path.join(tmpDir, '0002_alter.sql') fs.writeFileSync(filePath, original) @@ -42,6 +63,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites a bare v3 domain (the generation the wizard now scaffolds)', async () => { + declarePlaintext('"users"', 'email') const original = `ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` const filePath = path.join(tmpDir, '0002_v3.sql') fs.writeFileSync(filePath, original) @@ -57,6 +79,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the schema-qualified form produced by drizzle-kit', async () => { + declarePlaintext('"users"', 'email') const original = 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n' const filePath = path.join(tmpDir, '0003_alter.sql') @@ -74,6 +97,7 @@ describe('rewriteEncryptedAlterColumns', () => { it('rewrites a schema-qualified table produced by pgSchema()', async () => { // drizzle-kit emits `"app"."users"` for a table declared in a pgSchema(); // the old `\s+` between the table and ALTER COLUMN could never cross the `.`. + declarePlaintext('"app"."users"', 'email') const original = 'ALTER TABLE "app"."users" ALTER COLUMN "email" SET DATA TYPE "undefined"."eql_v3_text_search";\n' const filePath = path.join(tmpDir, '0014_qualified.sql') @@ -95,6 +119,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the "undefined" schema form drizzle-kit emits for bare custom types', async () => { + declarePlaintext('"transactions"', 'amount') const original = 'ALTER TABLE "transactions" ALTER COLUMN "amount" SET DATA TYPE "undefined"."eql_v2_encrypted";\n' const filePath = path.join(tmpDir, '0005_undef.sql') @@ -110,6 +135,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites the double-quoted form produced by stack 0.15.0', async () => { + declarePlaintext('"transactions"', 'description') const original = 'ALTER TABLE "transactions" ALTER COLUMN "description" SET DATA TYPE "undefined".""public"."eql_v2_encrypted"";\n' const filePath = path.join(tmpDir, '0006_double.sql') @@ -137,6 +163,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('skips the file passed in options.skip', async () => { + declarePlaintext('"t"', 'c') const install = path.join(tmpDir, '0000_install-eql.sql') const alter = path.join(tmpDir, '0002_alter.sql') fs.writeFileSync(install, 'CREATE SCHEMA eql_v2;\n') @@ -190,6 +217,7 @@ describe('rewriteEncryptedAlterColumns', () => { ] it.each(V3_DOMAINS)('rewrites an ALTER COLUMN to %s', async (domain) => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0007_v3.sql') fs.writeFileSync( filePath, @@ -217,6 +245,7 @@ describe('rewriteEncryptedAlterColumns', () => { ...V3_DOMAINS, 'eql_v2_encrypted', ])('extracts the bare domain %s from a mangled ALTER', async (domain) => { + declarePlaintext('"t"', 'c') const filePath = path.join(tmpDir, '0015_drift.sql') fs.writeFileSync( filePath, @@ -255,6 +284,7 @@ describe('rewriteEncryptedAlterColumns', () => { ] it.each(MANGLED_FORMS)('rewrites the v3 %s form', async (_label, emitted) => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0008_form.sql') fs.writeFileSync( filePath, @@ -271,6 +301,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('names the target domain in the guidance comment', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0010_comment.sql') fs.writeFileSync( filePath, @@ -284,6 +315,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('warns that the rewrite is data-destroying / empty-table-only', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0016_warn.sql') fs.writeFileSync( filePath, @@ -299,6 +331,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('separates ADD/DROP/RENAME with --> statement-breakpoint', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0018_breakpoint.sql') fs.writeFileSync( filePath, @@ -316,6 +349,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('rewrites each statement to its own domain when v2 and v3 are mixed', async () => { + declarePlaintext('"a"', 'x', 'y') const filePath = path.join(tmpDir, '0011_mixed.sql') fs.writeFileSync( filePath, @@ -370,6 +404,7 @@ describe('rewriteEncryptedAlterColumns', () => { }) it('reports no skipped statements when the strict rewrite fully handled the file', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0021_handled.sql') fs.writeFileSync( filePath, @@ -508,6 +543,7 @@ describe('rewriteEncryptedAlterColumns', () => { // The comment scan must not be fooled by `--` inside a string literal, or // it would skip a live statement and leave broken SQL to fail at migrate. it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0030_literal.sql') fs.writeFileSync( filePath, @@ -724,6 +760,7 @@ describe('rewriteEncryptedAlterColumns', () => { // The scoping matters: encrypting `contacts.email` must not be blocked by // an unrelated `users.email` that happens to share a column name. it('scopes the check to the table', async () => { + declarePlaintext('"contacts"', 'email') fs.writeFileSync( path.join(tmpDir, '0000_add.sql'), 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', @@ -761,6 +798,7 @@ describe('rewriteEncryptedAlterColumns', () => { // A commented-out ADD never ran, so it says nothing about the live schema. it('ignores an encrypted ADD COLUMN that is commented out', async () => { + declarePlaintext('"users"', 'email') fs.writeFileSync( path.join(tmpDir, '0000_add.sql'), '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', @@ -799,9 +837,73 @@ describe('rewriteEncryptedAlterColumns', () => { expect(rewritten).toEqual([]) expect(skipped[0]?.reason).toBe('already-encrypted') }) + + // Ordering pin: this column is BOTH declared plaintext (0000) and made + // encrypted by a previous sweep (0001). `already-encrypted` is the more + // specific reason and must win over `source-unknown`. + it('reports already-encrypted even when the column was also declared plaintext', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" text);\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // A commented-out encrypted column line inside an otherwise LIVE + // CREATE TABLE must still count as encrypted. The comment check applies + // only to the DECLARED scan (over-detecting plaintext costs data); the + // ENCRYPTED scan never re-checks comments inside a live CREATE TABLE, so + // this stays over-detecting — the safe direction — exactly as it was + // before the corpus learned to track declarations at all. + it('still counts a commented-out encrypted column inside a live CREATE TABLE as encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer,', + '\t-- "email" "public"."eql_v3_text_eq",', + '\t"email" text', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) }) it('handles multiple ALTER statements in one file', async () => { + declarePlaintext('"a"', 'x', 'y') const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v3_text_search;', 'ALTER TABLE "a" ALTER COLUMN "y" SET DATA TYPE eql_v3_text_search;', @@ -822,6 +924,7 @@ describe('rewriteEncryptedAlterColumns', () => { // Regression pin, not a bug fix — the matchers carry `/gi`, so a // hand-lowercased migration is rewritten just like drizzle-kit's output. it('rewrites a lowercase alter table ... set data type', async () => { + declarePlaintext('"users"', 'email') const filePath = path.join(tmpDir, '0034_lowercase.sql') fs.writeFileSync( filePath, @@ -839,13 +942,250 @@ describe('rewriteEncryptedAlterColumns', () => { expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') expect(updated).not.toMatch(/set data type/i) }) + + // A-2: the sweep is FAIL-CLOSED. A column the corpus never declares is + // UNKNOWN, not plaintext — it may already hold ciphertext, with its + // declaration sitting in a migration directory this sweep never sees. + describe('columns the corpus does not declare', () => { + it('refuses to rewrite a column the corpus never declares', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(`${alterSql}\n`) + expect(updated).not.toContain('DROP COLUMN') + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + it('rewrites a column declared plaintext by an ADD COLUMN', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" text;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The skipped file is still part of the corpus: a column's current type + // comes from the migrations that ran before this one, edit-eligible or not. + it('counts a declaration living in the file passed to options.skip', async () => { + const install = path.join(tmpDir, '0000_install-eql.sql') + fs.writeFileSync( + install, + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" text);\n', + ) + const alter = path.join(tmpDir, '0002_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: install, + }, + ) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + it('follows a RENAME when deciding a column is declared', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email_address" text);', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email_address" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A name inside a table/key constraint is a MENTION, not a declaration. + // Here every mention is followed by `)` or `,`, which the declaration + // regex's tail alone already rejects. A mention followed by a SQL keyword + // instead (e.g. `CHECK ("email" IS NOT NULL)`) is a separate case, closed + // by the regex's keyword lookahead and pinned by its own tests below. + // Counting either would put the rewrite back on the fail-open path. + it('does not treat a name inside PRIMARY KEY / REFERENCES as a declaration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "sessions" (', + '\t"id" integer,', + '\t"user_id" integer,', + '\tPRIMARY KEY ("id", "email"),', + '\tFOREIGN KEY ("user_id") REFERENCES "users"("email")', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "sessions" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // A column line commented out INSIDE a live CREATE TABLE never ran, so it + // declares nothing. + it('does not count a column commented out inside a live CREATE TABLE', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t-- "email" text,', + '\t"name" text', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('source-unknown') + }) + + // A CHECK predicate mentioning a column has the same `"name" ` + // shape as a declaration — `"email" IS` — but IS is a predicate keyword, + // not a type token. Without the keyword lookahead this would read as a + // declaration and rewrite the column, dropping any ciphertext it already + // holds via a declaration this sweep never sees. + it('does not treat a CHECK predicate mention as a declaration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\tCONSTRAINT "c1" CHECK ("email" IS NOT NULL)', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(`${alterSql}\n`) + expect(updated).not.toContain('DROP COLUMN') + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // A constraint's NAME can coincide with a column's name — drizzle's + // `
__unique` convention makes this contrived, but the keyword + // lookahead closes it regardless: a constraint name is always followed + // immediately by its constraint-type keyword (UNIQUE here), which the + // lookahead excludes. + it('does not let a same-named constraint declare the column', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\tCONSTRAINT "email" UNIQUE("id")', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // The keyword lookahead is pinned to a word boundary specifically so it + // does not eat real type names that merely START WITH a blocked + // keyword's letters: `interval`/`inet` both start "in" (colliding with + // IN) but must still count as genuine declarations. + it('still declares a column whose type name starts with a blocked keyword', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "events" ("email" interval, "note" inet);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + [ + 'ALTER TABLE "events" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + 'ALTER TABLE "events" ALTER COLUMN "note" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + }) }) describe('sweepMigrationDirs', () => { let tmpDir: string - const ALTER = - 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n' + const ALTER = [ + // Fail-closed: the sweep rewrites only a column the corpus DECLARES, and a + // real drizzle corpus carries the CREATE that declared it. + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" text);', + '--> statement-breakpoint', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') /** Create `dir` under the sandbox, optionally seeding `name` with `sql`. */ const seedDir = (dir: string, name?: string, sql?: string): string => { @@ -945,4 +1285,35 @@ describe('sweepMigrationDirs', () => { expect(results[0].skipped).toHaveLength(1) expect(results[0].skipped[0].file).toBe(path.join(abs, '0001_using.sql')) }) + + // A-2 trigger (b). The wizard ships scanning three candidate directories and + // indexes each separately, so a declaration in `drizzle/` is invisible to the + // sweep of `migrations/`. That column is already ciphertext; rewriting it + // emits a live DROP COLUMN with nothing anywhere to backfill from. + it('does not rewrite an ALTER whose column is declared in a sibling directory', async () => { + seedDir( + 'drizzle', + '0000_create.sql', + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" "public"."eql_v3_text_eq");\n', + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const migrations = seedDir( + 'migrations', + '0001_domain-change.sql', + `${alterSql}\n`, + ) + const alter = path.join(migrations, '0001_domain-change.sql') + + const results = await sweepMigrationDirs(tmpDir, ['drizzle', 'migrations']) + + const swept = results.find((result) => result.dir === 'migrations') + expect(swept?.rewritten).toEqual([]) + expect(swept?.skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'source-unknown' }, + ]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(`${alterSql}\n`) + expect(updated).not.toContain('DROP COLUMN') + }) }) diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 73e1908b7..4fb879f89 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -259,6 +259,79 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( 'gi', ) +/** + * A column name in a DECLARATION position — `"email" text`, + * `"email" "public"."eql_v3_text_search"`, `"amount" numeric(10, 2)`. + * + * The `\s+["a-z]` tail is what separates a declaration from a MENTION. Most + * mentions are followed by `)`, `,`, or an operator, which the tail alone + * already rejects: + * + * ```sql + * PRIMARY KEY ("id", "name") + * FOREIGN KEY ("org_id") REFERENCES "orgs"("id") + * ``` + * + * But a mention inside a predicate or a table-level constraint is often + * followed by whitespace and a SQL KEYWORD — which has exactly the same + * `\s+[a-z]` shape as a real declaration, and drizzle-kit emits both forms + * inside a `CREATE TABLE` (a `CONSTRAINT … CHECK ()` body is + * user-authored, so the predicate form is reachable too): + * + * ```sql + * CHECK ("email" IS NOT NULL) + * CONSTRAINT "users_email_unique" UNIQUE ("email") + * ``` + * + * The negative lookahead rejects the keywords reachable this way: predicate + * keywords (`IS`, `IN`, `NOT`, `LIKE`, `ILIKE`, `BETWEEN`, `AND`, `OR`), + * ordering/collation (`COLLATE`, `ASC`, `DESC`, `NULLS`), and every + * table/column-constraint keyword (`UNIQUE`, `PRIMARY`, `FOREIGN`, `CHECK`, + * `EXCLUDE`, `REFERENCES`, `CONSTRAINT`, `USING`, `WITH`) — the last six also + * close the constraint-NAME case above, since Postgres always puts one of + * them immediately after a constraint's name. + * + * A real type token still matches because the lookahead is pinned to a word + * boundary (`\b`) right after each keyword, so it only rejects a keyword when + * that keyword is the WHOLE next word: `interval` and `inet` both start with + * `IN`'s letters, but their third character keeps them inside one word, so + * `\bIN\b` never matches there; the same reasoning clears `int`, + * `char`/`citext` against `CHECK`, and `eql_v2_encrypted`/`eql_v3_*` against + * `EXCLUDE`. + * + * Deliberately NOT pinned to the encrypted domains. A column the corpus + * declares but does not give an encrypted type is, by residue, plaintext — so + * the fail-closed rule needs no type classification and no SQL parsing, only + * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. + * + * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: + * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, + * `OVERLAPS`, `ON`, …) still lets a bare mention through, e.g. + * `CHECK ("email" SIMILAR TO '...')`, or a `REFERENCES "email" ON DELETE + * CASCADE` where `"email"` names the referenced TABLE rather than the column + * being declared — `"id" integer REFERENCES "email" ON DELETE CASCADE` inside + * a `CREATE TABLE` body still registers `email` as declared. That one only + * bites when a column shares a referenced table's name, so like the rest of + * this residue it is inert unless that mention's name exactly matches a + * DIFFERENT column that is both encrypted and genuinely undeclared anywhere + * else in the corpus — contrived, and strictly narrower than the gap this + * replaces. + */ +const DECLARED_COLUMN_RE = + /"([^"]+)"\s+(?!(?:IS|IN|NOT|LIKE|ILIKE|BETWEEN|AND|OR|COLLATE|ASC|DESC|NULLS|UNIQUE|PRIMARY|FOREIGN|CHECK|EXCLUDE|REFERENCES|CONSTRAINT|USING|WITH)\b)["a-z]/gi + +/** + * `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. + * + * Needs no {@link DECLARED_COLUMN_RE}-style keyword lookahead: the token + * right after an ADD COLUMN's column name is always its type, so no SQL + * keyword can occupy that position. + */ +const ADD_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+["a-z]`, + 'gi', +) + /** Splits a `TABLE_REF` capture pair into its schema and table halves. */ function tableOf( first: string, @@ -271,42 +344,80 @@ function tableOf( : { schema: first, table: second } } -/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +/** Identity of a column across the corpus, for {@link indexColumnDeclarations}. */ function columnKey(table: string, column: string, schema?: string): string { return JSON.stringify([schema ?? '', table, column]) } +/** What the migration corpus says about the columns it mentions. */ +interface ColumnIndex { + /** Columns the corpus gives an ENCRYPTED type. Rewriting one drops ciphertext. */ + encrypted: Set + /** Columns the corpus DECLARES at all, whatever type it gives them. */ + declared: Set +} + /** - * Index every column the migration corpus gives an ENCRYPTED type, so the - * rewrite can tell the change it exists for (plaintext → encrypted) from one it - * must never touch (encrypted → encrypted). + * Index what the migration corpus knows about each column, so the rewrite can + * tell the change it exists for (plaintext → encrypted) from the two it must + * never make: encrypted → encrypted, and a change to a column it knows nothing + * about. * - * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. - * A column whose encrypted domain merely changes (`types.TextEq` → + * **Why `encrypted` (#772 review, W-3):** the strict matcher captures only the + * TARGET type. A column whose encrypted domain merely changes (`types.TextEq` → * `types.TextSearch`) matches just as well as a plaintext column, and the * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext * left anywhere to backfill from, so unlike the plaintext case the data is not - * recoverable from the application at all. Changing an encrypted column's domain - * changes its index terms, so the data has to be re-encrypted through the client - * regardless; the sweep flags the statement and leaves it for the user. + * recoverable from the application at all. + * + * **Why `declared` (A-2):** absence from `encrypted` is not evidence of + * plaintext. It is evidence of nothing. The sweep runs per directory, and the + * shipped wizard default scans three of them, so a column's `CREATE TABLE` can + * simply live somewhere this sweep never reads — the column is then rewritten + * on an assumption, and the ADD+DROP+RENAME drops live ciphertext. Requiring a + * POSITIVE declaration makes the unknown case a flagged statement instead. + * + * Because the encrypted side is matched against the known domain list, the + * plaintext side needs no type classification: a column the corpus declares but + * does not give an encrypted type is plaintext by residue. * * The index is corpus-wide rather than ordered by migration: over-detecting - * "encrypted" costs a flagged statement the user must handle by hand, while - * under-detecting costs irrecoverable ciphertext. Only comment-free statements - * count, for the same reason {@link isInsideCommentOrString} exists. + * "encrypted" costs a flagged statement the user handles by hand, while + * under-detecting costs irrecoverable ciphertext. That asymmetry is why the + * two per-column scans inside a `CREATE TABLE` body are deliberately + * inconsistent about comments (see the comment at the loops themselves) — + * do not "fix" that inconsistency, it is the point. */ -function indexEncryptedColumns(contents: readonly string[]): Set { +function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { const encrypted = new Set() + const declared = new Set() for (const sql of contents) { for (const created of sql.matchAll(CREATE_TABLE_RE)) { if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) - for (const column of created[3].matchAll( - CREATE_TABLE_ENCRYPTED_COLUMN_RE, - )) { + const body = created[3] + + // The body is scanned as its own document: a `--` earlier in it comments + // out the rest of that line, and a CREATE inside a block comment or a + // string literal was already skipped above. + // + // Asymmetric on purpose. ENCRYPTED does NOT re-check comments here: a + // commented-out encrypted column inside an otherwise live CREATE TABLE + // still counts, exactly as it did before this sweep learned to declare + // anything — over-detecting "encrypted" only costs a flagged statement. + // DECLARED DOES re-check: a commented-out plaintext line never ran, so + // it must not count as a declaration — over-detecting "declared" is + // what would put a truly-undeclared, possibly already-encrypted column + // back on the fail-open path. + for (const column of body.matchAll(CREATE_TABLE_ENCRYPTED_COLUMN_RE)) { encrypted.add(columnKey(table, column[1], schema)) } + + for (const column of body.matchAll(DECLARED_COLUMN_RE)) { + if (isInsideCommentOrString(body, column.index)) continue + declared.add(columnKey(table, column[1], schema)) + } } for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { @@ -315,19 +426,26 @@ function indexEncryptedColumns(contents: readonly string[]): Set { encrypted.add(columnKey(table, added[3], schema)) } + for (const added of sql.matchAll(ADD_COLUMN_RE)) { + if (isInsideCommentOrString(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + declared.add(columnKey(table, added[3], schema)) + } + // A rename carries the column's type with it — and `__cipherstash_tmp` // renamed onto the real name is exactly what a previous sweep of this very // directory emitted. Run after ADD so that tmp column is already indexed. for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { if (isInsideCommentOrString(sql, renamed.index)) continue const { schema, table } = tableOf(renamed[1], renamed[2]) - if (encrypted.has(columnKey(table, renamed[3], schema))) { - encrypted.add(columnKey(table, renamed[4], schema)) - } + const from = columnKey(table, renamed[3], schema) + const to = columnKey(table, renamed[4], schema) + if (encrypted.has(from)) encrypted.add(to) + if (declared.has(from)) declared.add(to) } } - return encrypted + return { encrypted, declared } } /** Why a recognised ALTER-to-encrypted statement was left alone. */ @@ -336,6 +454,8 @@ export type SkipReason = | 'unrecognised-form' /** The column already holds an encrypted domain; rewriting drops ciphertext. */ | 'already-encrypted' + /** The corpus never declares the column, so its current type is unknown. */ + | 'source-unknown' /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { @@ -350,7 +470,7 @@ export interface SkippedAlter { /** * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print * next to the statement. Lives here so every caller says the same thing — the - * two reasons need very different action from the user, and a single generic + * three reasons need very different action from the user, and a single generic * "could not rewrite automatically" hides that. */ export function describeSkipReason(reason: SkipReason): string { @@ -359,6 +479,8 @@ export function describeSkipReason(reason: SkipReason): string { return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" case 'unrecognised-form': return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + case 'source-unknown': + return "the sweep could not find where this column was declared in this migration directory, so it cannot tell a plaintext column (safe to rewrite) from one that already holds ciphertext (where the rewrite would DROP it). Usually the column's `CREATE TABLE` / `ADD COLUMN` lives in a different directory, or the migration history was squashed. Check the column's current type in the database: if it is plaintext and the table is empty, apply the ADD/DROP/RENAME by hand; if it already holds ciphertext, use the staged `stash encrypt` lifecycle instead" } } @@ -394,13 +516,16 @@ export interface RewriteResult { * * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements * left for a human — ones outside the strict matcher (a hand-authored - * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting - * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. - * Both are left untouched on disk and surfaced non-fatally so the caller can - * tell the user to review them, rather than silently shipping broken SQL or - * destroying data. Statements sitting inside a SQL comment — or inside a - * single-quoted string literal, where they are data rather than SQL — are inert - * and are neither rewritten nor reported. + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form); ones targeting a + * column that is ALREADY encrypted, where the rewrite would drop ciphertext; + * and ones targeting a column the corpus never DECLARES anywhere, where the + * rewrite would be guessing at a type it cannot see (likely the most common + * skip on a real corpus, since the sweep only ever reads part of the + * migration history). All three are left untouched on disk and surfaced + * non-fatally so the caller can tell the user to review them, rather than + * silently shipping broken SQL or destroying data. Statements sitting inside a + * SQL comment — or inside a single-quoted string literal, where they are data + * rather than SQL — are inert and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, @@ -440,7 +565,8 @@ export async function rewriteEncryptedAlterColumns( // Built from the WHOLE corpus, including `options.skip`: a column's current // type comes from the migrations that ran before this one, not just the files // we are allowed to edit. - const encryptedColumns = indexEncryptedColumns([...contents.values()]) + const { encrypted: encryptedColumns, declared: declaredColumns } = + indexColumnDeclarations([...contents.values()]) for (const [filePath, original] of contents) { if (options.skip && filePath === options.skip) continue @@ -471,6 +597,17 @@ export async function rewriteEncryptedAlterColumns( return match } + // Fail closed. Absence from `declaredColumns` does not mean the column + // is plaintext — it means the corpus never said. The declaration can + // sit in a directory this sweep does not read (the wizard ships with + // three candidates and indexes each separately), so rewriting on the + // assumption is how a live `DROP COLUMN` reaches a populated — possibly + // already-encrypted — column. Flag it and let the user look. + if (!declaredColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'source-unknown') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. From 01ad43bfe01675aa3ec43a0a900775faedec9c61 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:00:09 +1000 Subject: [PATCH 048/123] fix(cli): declare the column the fail-closed sweep now requires in migration tests The Drizzle migration-sweep test fixtures wrote a bare ALTER COLUMN ... SET DATA TYPE with nothing declaring the target column anywhere in the corpus. Since rewriteEncryptedAlterColumns is now fail-closed, such a statement is correctly left as a source-unknown skip, which broke two assertions that expected the rewrite to happen. These tests exist to check the sweep's behaviour (that it walks sibling migrations and skips the just-generated install migration), not the fail-closed rule itself, so fix the fixtures rather than the assertions: each now writes a sibling CREATE TABLE declaring the column plaintext, matching the declarePlaintext pattern already used in rewrite-migrations.test.ts. --- .../commands/eql/__tests__/migration.test.ts | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/packages/cli/src/commands/eql/__tests__/migration.test.ts b/packages/cli/src/commands/eql/__tests__/migration.test.ts index ba3f8d7d1..9f0fa1f90 100644 --- a/packages/cli/src/commands/eql/__tests__/migration.test.ts +++ b/packages/cli/src/commands/eql/__tests__/migration.test.ts @@ -217,6 +217,14 @@ describe('eqlMigrationCommand — Drizzle', () => { it('rewrites a sibling migration with a broken v3 ALTER COLUMN', async () => { const out = join(tmp, 'drizzle') mkdirSync(out, { recursive: true }) + // The sweep is fail-closed: it rewrites a column only when the corpus + // positively declares it (and it isn't already encrypted). A real drizzle + // corpus carries this declaration in an earlier migration — supply it so + // the fixture matches what the sweep actually requires. + writeFileSync( + join(out, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) const sibling = join(out, '0001_encrypt-email.sql') writeFileSync( sibling, @@ -239,10 +247,17 @@ describe('eqlMigrationCommand — Drizzle', () => { it('does not rewrite the EQL install migration it just generated', async () => { const out = join(tmp, 'drizzle') mkdirSync(out, { recursive: true }) - const generated = join(out, '0000_install-eql.sql') + // Fail-closed requires the corpus to positively declare the column before + // the sweep will touch it — supply the declaration a real drizzle corpus + // would carry, same as the sibling-rewrite test above. + writeFileSync( + join(out, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + const generated = join(out, '0001_install-eql.sql') // A sibling carrying the SAME statement — the differential that proves the // sweep ran at all, rather than no-opping over the whole directory. - const sibling = join(out, '0001_encrypt-email.sql') + const sibling = join(out, '0002_encrypt-email.sql') const unsafeAlter = 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "undefined"."eql_v3_text_search";\n' writeFileSync(sibling, unsafeAlter) From baa3163a1aa50c1e12ac6a9e9b28612a66615c1f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:11:00 +1000 Subject: [PATCH 049/123] fix(wizard): declare the column the fail-closed sweep now requires in post-agent test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'defaults to No, and says why, when a file was rewritten' fixture wrote a bare ALTER COLUMN ... SET DATA TYPE with nothing declaring the target column anywhere in the swept corpus. Since rewriteEncryptedAlterColumns is fail-closed, that statement was correctly left as a source-unknown skip — the test passed anyway only because post-agent.ts's `unsafe` check is `rewritten > 0 || skipped > 0`, so it silently became a duplicate of the skipped-path test below it, and the `rewritten > 0` half of that condition had zero coverage. Add a sibling migration declaring the column plaintext (matching the declarePlaintext pattern already used in rewrite-migrations.test.ts and the analogous cli/eql/migration.test.ts fix), so the sweep genuinely rewrites the column and the test exercises the branch its name claims. --- packages/wizard/src/__tests__/post-agent.test.ts | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index c13f83cef..afbba81c5 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -159,6 +159,15 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { it('defaults to No, and says why, when a file was rewritten', async () => { fs.mkdirSync(path.join(cwd, 'drizzle')) + // The sweep is fail-closed: it rewrites a column only when the corpus + // positively declares it (and it isn't already encrypted). A real drizzle + // corpus carries this declaration in an earlier migration — supply it so + // the fixture matches what the sweep actually requires, and the ALTER + // below is genuinely rewritten rather than skipped as source-unknown. + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) fs.writeFileSync( path.join(cwd, 'drizzle', '0001_encrypt.sql'), 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', From 9c864039df32632ec5aea94972a1300a20783f0d Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:17:06 +1000 Subject: [PATCH 050/123] docs(skills,changesets): the ALTER sweep repairs only what it can place Both skills said every recognised statement is rewritten. It now requires the column's declaration to be present in the swept directory and flags what it cannot place. --- .changeset/rewriter-never-drops-ciphertext.md | 12 ++++++++++++ skills/stash-cli/SKILL.md | 2 ++ skills/stash-drizzle/SKILL.md | 2 +- 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index b3577f8b4..116c48549 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -24,6 +24,18 @@ The sweep also refuses to rewrite a column the migration corpus already gives an encrypted type, so changing a column's encrypted domain no longer drops a column full of ciphertext. Skipped statements report why they were left alone. +The sweep is now fail-closed about the columns it does not recognise at all. +Previously a column missing from the corpus index was assumed to be plaintext +and rewritten; absence is not evidence, and the declaration can simply live in a +migration directory the sweep never reads — the wizard ships scanning three of +them and indexes each separately. Such a statement is now reported for review +rather than rewritten, so the ADD+DROP+RENAME can no longer drop a column that +already holds ciphertext. If your migration history is squashed, or the column's +`CREATE TABLE` lives outside the directory being swept, you will see the +statement flagged instead of repaired: check the column's current type and +either apply the rewrite by hand on an empty table, or use the staged +`stash encrypt` lifecycle. + An unreadable migration directory (`EACCES`) is reported rather than silently treated as empty, and the wizard's `Run the migration now?` prompt defaults to No whenever the sweep rewrote anything, flagged anything, or could not check a diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index b35c9be4f..39ea85498 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -392,6 +392,8 @@ Pass exactly one of `--drizzle` / `--prisma`. The generated migration also insta After writing the migration, `--drizzle` sweeps the output directory for sibling migrations containing an in-place `ALTER COLUMN … SET DATA TYPE ` — drizzle-kit emits these when you change a plaintext column to an encrypted one, and Postgres rejects them (there is no cast from `text`/`numeric` to an EQL type). Each is rewritten into an `ADD COLUMN` + `DROP` + `RENAME` sequence and the rewritten files are listed. This is equivalent to DROP+ADD — it fixes the type but does **not** preserve data — so it is safe **only on an empty table**. On a populated table the new column starts NULL and the old one is dropped in the same migration, destroying the plaintext; do not run it there. Use the staged `stash encrypt` path (add → backfill → cutover → drop) instead. +The sweep is fail-closed: it rewrites a statement only when the same directory also contains the `CREATE TABLE` or `ADD COLUMN` that declared the column, and the column is not already encrypted. A statement it cannot place — the declaration lives in another migration directory, or the history was squashed — is listed as needing review rather than rewritten, because a column of unknown type may already hold ciphertext that the rewrite's `DROP COLUMN` would destroy. + #### `eql upgrade` The install SQL is safe to re-run — columns and data survive — but it is not fully idempotent: it begins with `DROP SCHEMA IF EXISTS eql_v3 CASCADE`, which cascade-drops any **functional indexes** built on the `eql_v3` extractors (see `stash-indexing`). After an upgrade, recreate them: migration runners skip migrations already recorded as applied, so add a *new* migration that re-issues the `CREATE INDEX` statements (or run the DDL directly), then `ANALYZE` the affected tables. `upgrade` checks the current version, re-runs the install SQL, and reports the new one. If EQL isn't installed it points you at `eql install`. Same `--supabase`, `--exclude-operator-family`, `--eql-version`, `--latest`, `--dry-run`, `--database-url` flags. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index f52025bb4..50ae36fd2 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -61,7 +61,7 @@ stash eql migration --drizzle --supabase # also grants eql_v3 to anon/authenti The generated migration also installs the `cs_migrations` tracking schema, so a single `drizzle-kit migrate` covers everything `stash encrypt …` needs — no out-of-band `stash eql install`. EQL v3 ships one SQL bundle for every target including Supabase; `--supabase` only adds the PostgREST/RLS role grants (harmless when you connect directly as `postgres`). Requires `drizzle-kit` installed and configured. -**Changing an existing plaintext column to an encrypted one.** `drizzle-kit generate` emits an in-place `ALTER TABLE … ALTER COLUMN … SET DATA TYPE eql_v3_`, which Postgres rejects — there is no cast from `text`/`numeric` to an EQL domain. (On drizzle-kit 0.31.0 and later the emitted type is also mangled to `"undefined"."eql_v3_"`, since a `customType` has no `typeSchema`.) The `stash eql migration --drizzle` sweep repairs the invalid statement — the `stash-cli` skill covers what it rewrites and the rule that matters: the repair is data-destroying, so it is safe **only on an empty table**. For a table with live data, do **not** apply the swept migration; follow the staged flow in **Migrating an Existing Column to Encrypted** below instead. +**Changing an existing plaintext column to an encrypted one.** `drizzle-kit generate` emits an in-place `ALTER TABLE … ALTER COLUMN … SET DATA TYPE eql_v3_`, which Postgres rejects — there is no cast from `text`/`numeric` to an EQL domain. (On drizzle-kit 0.31.0 and later the emitted type is also mangled to `"undefined"."eql_v3_"`, since a `customType` has no `typeSchema`.) The `stash eql migration --drizzle` sweep repairs the invalid statement — the `stash-cli` skill covers what it rewrites and the rule that matters: the repair is data-destroying, so it is safe **only on an empty table**. It repairs only what it can place: the swept directory must also contain the migration that declared the column. If it does not, the statement is flagged for review instead of rewritten. For a table with live data, do **not** apply the swept migration; follow the staged flow in **Migrating an Existing Column to Encrypted** below instead. ### Column Storage From e6624074145c42c46fc883570e8ac66b077bf0e2 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:40:17 +1000 Subject: [PATCH 051/123] docs(changesets,skills): scope the drop-ciphertext guarantee to what the migration corpus can see The changeset and skills/stash-cli/SKILL.md both claimed the fail-closed sweep means the rewrite "can no longer drop a column that already holds ciphertext." That overclaims: the sweep requires a corpus *declaration*, not the database's *current* state, and the migration corpus is only a model of the database that can go stale. `stash encrypt cutover` renames columns directly in the database without writing drizzle SQL, so the corpus can still describe a cut-over column as plaintext; the same is true of any hand-authored psql or Supabase-dashboard change. Reworded both to say the guarantee holds for what the corpus shows, not for a database that has drifted from its migration history. --- .changeset/rewriter-never-drops-ciphertext.md | 18 +++++++++++++----- skills/stash-cli/SKILL.md | 2 +- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index 116c48549..66ad0f209 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -29,11 +29,19 @@ Previously a column missing from the corpus index was assumed to be plaintext and rewritten; absence is not evidence, and the declaration can simply live in a migration directory the sweep never reads — the wizard ships scanning three of them and indexes each separately. Such a statement is now reported for review -rather than rewritten, so the ADD+DROP+RENAME can no longer drop a column that -already holds ciphertext. If your migration history is squashed, or the column's -`CREATE TABLE` lives outside the directory being swept, you will see the -statement flagged instead of repaired: check the column's current type and -either apply the rewrite by hand on an empty table, or use the staged +rather than rewritten, so the ADD+DROP+RENAME no longer drops a column that the +migration corpus itself shows already holds ciphertext. That is a guarantee +about what the corpus says, not about the database: the sweep reasons entirely +from migration files, and a database that has drifted from its migration +history is outside what it can see. `stash encrypt cutover` is the sharpest +example — it renames columns directly in the database and never writes drizzle +SQL, so the corpus can still describe a column as plaintext after cutover has +made it ciphertext; the same is true of any change made by hand via psql or the +Supabase dashboard. If your migration history is squashed, the column's +`CREATE TABLE` lives outside the directory being swept, or the database has +simply drifted from what the migrations describe, you will see the statement +flagged instead of repaired: check the column's current type in the database +and either apply the rewrite by hand on an empty table, or use the staged `stash encrypt` lifecycle. An unreadable migration directory (`EACCES`) is reported rather than silently diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 39ea85498..aa1c03c27 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -392,7 +392,7 @@ Pass exactly one of `--drizzle` / `--prisma`. The generated migration also insta After writing the migration, `--drizzle` sweeps the output directory for sibling migrations containing an in-place `ALTER COLUMN … SET DATA TYPE ` — drizzle-kit emits these when you change a plaintext column to an encrypted one, and Postgres rejects them (there is no cast from `text`/`numeric` to an EQL type). Each is rewritten into an `ADD COLUMN` + `DROP` + `RENAME` sequence and the rewritten files are listed. This is equivalent to DROP+ADD — it fixes the type but does **not** preserve data — so it is safe **only on an empty table**. On a populated table the new column starts NULL and the old one is dropped in the same migration, destroying the plaintext; do not run it there. Use the staged `stash encrypt` path (add → backfill → cutover → drop) instead. -The sweep is fail-closed: it rewrites a statement only when the same directory also contains the `CREATE TABLE` or `ADD COLUMN` that declared the column, and the column is not already encrypted. A statement it cannot place — the declaration lives in another migration directory, or the history was squashed — is listed as needing review rather than rewritten, because a column of unknown type may already hold ciphertext that the rewrite's `DROP COLUMN` would destroy. +The sweep is fail-closed: it rewrites a statement only when the same directory also contains the `CREATE TABLE` or `ADD COLUMN` that declared the column, and the column is not already encrypted **in the corpus**. That is a guarantee about what the migration files say, not about the live database — the sweep never queries the database, so a column that has drifted from its migration history (for example, renamed directly by `stash encrypt cutover`, or altered by hand via psql or the Supabase dashboard) can already hold ciphertext while the corpus still describes it as plaintext. A statement it cannot place — the declaration lives in another migration directory, or the history was squashed — is listed as needing review rather than rewritten, because a column of unknown type may already hold ciphertext that the rewrite's `DROP COLUMN` would destroy. Either way, check the column's actual type in the database before applying a flagged or corpus-cleared rewrite by hand. #### `eql upgrade` From bdb94141def6d0c34e0e25f0631a065c74e8d520 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:40:31 +1000 Subject: [PATCH 052/123] fix(cli,wizard): collapse duplicate skip reports for block-comment-prefixed near-misses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A statement the strict matcher matched but skipped (already-encrypted or source-unknown) is left on disk unchanged, so it still contains `SET DATA TYPE` and the broad near-miss scan finds it again. When a `/* ... */` block comment sat glued to the front of that statement, STATEMENT_PREAMBLE_RE (which only stripped blank lines and `--` line comments) left the near-miss pass reporting a different statement string than the strict pass's `match.trim()`, so the dedup key never matched: the same statement was reported twice, once with the correct reason and once as `unrecognised-form` — contradictory advice (look for a hand-authored `USING` clause) for a statement that has none, with the block comment quoted back to the user besides. Extend the preamble regex to also strip a leading block comment so both passes agree on the statement text and the second report collapses into the first. Also tightens the `leaves an ALTER inside a NESTED block comment untouched` regression test to assert `skipped` is empty, not just `rewritten` — this branch's fail-closed change made the fixture's column undeclared, so the statement is skipped as source-unknown regardless of whether nested block comments are handled, and the old assertion passed either way. Also names, in the DECLARED_COLUMN_RE docstring, the dependency the "declared but not encrypted is plaintext by residue" claim has on MANGLED_TYPE_FORMS covering every encrypted shape: a domain installed into a non-`public` schema and an array of the domain both escape ENCRYPTED_TYPE_REF and get rewritten as if plaintext. Neither is a supported EQL layout or a drizzle-kit output, and both behaved this way before this branch, so this is documentation, not a regression. Verified the nested-block-comment guard is not vacuous: replacing the depth-tracking loop in isInsideCommentOrString with a plain first-`*/` scan makes the strengthened test fail (rewritten stays empty, but skipped now surfaces a source-unknown entry), confirming it exercises actual nesting behaviour rather than only checking `rewritten`. --- .../src/__tests__/rewrite-migrations.test.ts | 26 +++++++++++++- .../cli/src/commands/db/rewrite-migrations.ts | 34 ++++++++++++++++--- .../src/__tests__/rewrite-migrations.test.ts | 26 +++++++++++++- packages/wizard/src/lib/rewrite-migrations.ts | 34 ++++++++++++++++--- 4 files changed, 108 insertions(+), 12 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index c830b6001..80f235c1e 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -484,6 +484,29 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].statement).toBe(statement) }) + // A statement the STRICT matcher already matched but skipped (here: + // source-unknown) is left on disk unchanged, so it still contains + // `SET DATA TYPE` and the broad near-miss scan finds it again. Before the + // preamble regex stripped a leading block comment, that second pass reported + // a DIFFERENT statement string (comment glued to the front) than the strict + // pass's `match.trim()`, so the dedup key never matched and the same + // statement came back twice: once correctly as `source-unknown`, once as + // `unrecognised-form` — contradictory advice (look for a hand-authored + // `USING` clause) for a statement that has none. + it('reports a block-comment-prefixed statement once, not twice', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0040_block-preamble.sql') + fs.writeFileSync(filePath, `/* note */\n${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { const statement = 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' @@ -592,9 +615,10 @@ describe('rewriteEncryptedAlterColumns', () => { const filePath = path.join(tmpDir, '0030_nested.sql') fs.writeFileSync(filePath, original) - const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) }) diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index e82e5cedf..8042db3a8 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -96,8 +96,9 @@ const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi /** - * Blank lines and `--` line comments (including drizzle-kit's - * `--> statement-breakpoint`) at the head of a {@link NEAR_MISS_RE} match. + * Blank lines, a leading `/* … *\/` block comment, and `--` line comments + * (including drizzle-kit's `--> statement-breakpoint`) at the head of a + * {@link NEAR_MISS_RE} match. * * That regex opens with a lazy `[^;]*?`, whose only left boundary is the * previous `;` — or the start of the file when there is no preceding statement. @@ -106,10 +107,21 @@ const NEAR_MISS_RE = * with that whole block glued to its front. Strip the preamble so the statement * we quote back reads as the offending statement alone. * - * Only line comments are handled — `/* … *\/` block comments are not something - * drizzle-kit emits, and a stray one costs cosmetics, not correctness. + * The leading block comment is NOT cosmetic: a statement the strict + * {@link ALTER_COLUMN_TO_ENCRYPTED_RE} matched but skipped (`already-encrypted` + * or `source-unknown`) is left on disk unchanged, so it still contains + * `SET DATA TYPE` and the broad scan below finds it again. Without stripping + * the block comment here, that second pass reports a DIFFERENT statement string + * (comment glued to the front) than the strict pass's `match.trim()`, so + * {@link rewriteEncryptedAlterColumns}'s dedup key never matches and the same + * statement is reported twice — once with the correct reason, once as + * `unrecognised-form`, whose guidance (look for a hand-authored `USING`) is + * wrong for a statement the strict matcher already matched. Stripping the + * comment here makes both passes agree on the statement text so the second + * report collapses into the first. */ -const STATEMENT_PREAMBLE_RE = /^(?:[^\S\n]*(?:--[^\n]*)?\n)+/ +const STATEMENT_PREAMBLE_RE = + /^(?:\/\*[\s\S]*?\*\/\s*)?(?:[^\S\n]*(?:--[^\n]*)?\n)*/ /** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ function trimStatementPreamble(statement: string): string { @@ -296,6 +308,18 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * the fail-closed rule needs no type classification and no SQL parsing, only * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. * + * **The residue claim depends on {@link MANGLED_TYPE_FORMS} covering every + * encrypted shape.** "By residue, plaintext" is only as good as the encrypted + * side's coverage: a declaration {@link ENCRYPTED_TYPE_REF} fails to recognise + * falls to the plaintext residue and gets rewritten. Two forms do this: a + * domain installed into a non-`public` schema (`"email" "app"."eql_v3_text_search"`, + * since the mangled forms only special-case the literal `public` schema), and + * an array of the domain (`ADD COLUMN "email" public.eql_v3_text_search[]`, + * since {@link ENCRYPTED_TYPE_REF}'s trailing delimiter lookahead does not + * include `[`). Neither is a layout EQL installs into or a shape drizzle-kit + * emits, and both behaved identically before this branch — so this is a + * documentation gap, not a regression this branch introduced. + * * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, * `OVERLAPS`, `ON`, …) still lets a bare mention through, e.g. diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 9edc41bb3..5c737da73 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -440,6 +440,29 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].statement).toBe(statement) }) + // A statement the STRICT matcher already matched but skipped (here: + // source-unknown) is left on disk unchanged, so it still contains + // `SET DATA TYPE` and the broad near-miss scan finds it again. Before the + // preamble regex stripped a leading block comment, that second pass reported + // a DIFFERENT statement string (comment glued to the front) than the strict + // pass's `match.trim()`, so the dedup key never matched and the same + // statement came back twice: once correctly as `source-unknown`, once as + // `unrecognised-form` — contradictory advice (look for a hand-authored + // `USING` clause) for a statement that has none. + it('reports a block-comment-prefixed statement once, not twice', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0040_block-preamble.sql') + fs.writeFileSync(filePath, `/* note */\n${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { const statement = 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' @@ -522,9 +545,10 @@ describe('rewriteEncryptedAlterColumns', () => { const filePath = path.join(tmpDir, '0030_nested.sql') fs.writeFileSync(filePath, original) - const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) }) diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 4fb879f89..abb6afab0 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -104,8 +104,9 @@ const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi /** - * Blank lines and `--` line comments (including drizzle-kit's - * `--> statement-breakpoint`) at the head of a {@link NEAR_MISS_RE} match. + * Blank lines, a leading `/* … *\/` block comment, and `--` line comments + * (including drizzle-kit's `--> statement-breakpoint`) at the head of a + * {@link NEAR_MISS_RE} match. * * That regex opens with a lazy `[^;]*?`, whose only left boundary is the * previous `;` — or the start of the file when there is no preceding statement. @@ -114,10 +115,21 @@ const NEAR_MISS_RE = * with that whole block glued to its front. Strip the preamble so the statement * we quote back reads as the offending statement alone. * - * Only line comments are handled — `/* … *\/` block comments are not something - * drizzle-kit emits, and a stray one costs cosmetics, not correctness. + * The leading block comment is NOT cosmetic: a statement the strict + * {@link ALTER_COLUMN_TO_ENCRYPTED_RE} matched but skipped (`already-encrypted` + * or `source-unknown`) is left on disk unchanged, so it still contains + * `SET DATA TYPE` and the broad scan below finds it again. Without stripping + * the block comment here, that second pass reports a DIFFERENT statement string + * (comment glued to the front) than the strict pass's `match.trim()`, so + * {@link rewriteEncryptedAlterColumns}'s dedup key never matches and the same + * statement is reported twice — once with the correct reason, once as + * `unrecognised-form`, whose guidance (look for a hand-authored `USING`) is + * wrong for a statement the strict matcher already matched. Stripping the + * comment here makes both passes agree on the statement text so the second + * report collapses into the first. */ -const STATEMENT_PREAMBLE_RE = /^(?:[^\S\n]*(?:--[^\n]*)?\n)+/ +const STATEMENT_PREAMBLE_RE = + /^(?:\/\*[\s\S]*?\*\/\s*)?(?:[^\S\n]*(?:--[^\n]*)?\n)*/ /** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ function trimStatementPreamble(statement: string): string { @@ -304,6 +316,18 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * the fail-closed rule needs no type classification and no SQL parsing, only * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. * + * **The residue claim depends on {@link MANGLED_TYPE_FORMS} covering every + * encrypted shape.** "By residue, plaintext" is only as good as the encrypted + * side's coverage: a declaration {@link ENCRYPTED_TYPE_REF} fails to recognise + * falls to the plaintext residue and gets rewritten. Two forms do this: a + * domain installed into a non-`public` schema (`"email" "app"."eql_v3_text_search"`, + * since the mangled forms only special-case the literal `public` schema), and + * an array of the domain (`ADD COLUMN "email" public.eql_v3_text_search[]`, + * since {@link ENCRYPTED_TYPE_REF}'s trailing delimiter lookahead does not + * include `[`). Neither is a layout EQL installs into or a shape drizzle-kit + * emits, and both behaved identically before this branch — so this is a + * documentation gap, not a regression this branch introduced. + * * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, * `OVERLAPS`, `ON`, …) still lets a bare mention through, e.g. From 9ce6c534137a2c2f45fab5ac65e91d92d65564ca Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:40:43 +1000 Subject: [PATCH 053/123] fix(wizard): stop claiming data destruction when the sweep only flagged, not rewrote MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `unsafe = sweep.rewritten > 0 || sweep.skipped > 0` warned that the migration "DESTROYS data on any table that already holds rows" whenever anything was rewritten OR merely flagged. For a sweep that rewrote nothing and only flagged a source-unknown statement, that's false — nothing on disk changed, and the raw ALTER simply fails its cast at migrate time instead. This branch makes the pure-skip case the common one, so the overclaim needed its own message arm, matching the care already taken for the `unverified` (failed-sweep) case. Split `unsafe` into `destructive` (rewritten > 0) and `flaggedOnly` (nothing rewritten, but something was flagged), and give `flaggedOnly` a message that tells the truth: statements were flagged for review, nothing was rewritten, and the migration will fail at migrate time until they're resolved. The prompt still defaults to No in both cases. Also strengthens `defaults to No, and says why, when a file was rewritten` to assert the on-disk effect (the swept file contains DROP COLUMN, not SET DATA TYPE) rather than only the prompt defaulting to No and mentioning "DESTROYS data" — both of which the skipped-statement branch also produces, so the test could not tell a genuine rewrite from its `source-unknown` sibling. Confirmed by deleting the fixture's 0000_declare.sql (which makes the ALTER a source-unknown skip instead of a rewrite): the test now fails at the message assertion, since the flagged-only message added above no longer contains "DESTROYS data". Restored the fixture afterwards. --- .../wizard/src/__tests__/post-agent.test.ts | 19 ++++++++++++ packages/wizard/src/lib/post-agent.ts | 29 ++++++++++++------- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index afbba81c5..8e7656d9f 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -178,6 +178,20 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] expect(options?.initialValue).toBe(false) expect(String(options?.message)).toContain('DESTROYS data') + + // Assert the REWRITE actually happened, not just that the prompt defaulted + // to No — both this test and its `source-unknown` sibling below produce + // `initialValue: false` and a message containing "DESTROYS data" is the + // only thing that used to distinguish them, and that came from the same + // skipped-statement branch too. Without the 0000_declare.sql fixture the + // ALTER is skipped as source-unknown rather than rewritten, so pin the + // on-disk effect a genuine rewrite leaves behind. + const swept = fs.readFileSync( + path.join(cwd, 'drizzle', '0001_encrypt.sql'), + 'utf-8', + ) + expect(swept).toContain('DROP COLUMN') + expect(swept).not.toContain('SET DATA TYPE') }) it('defaults to No when a statement was flagged rather than rewritten', async () => { @@ -191,6 +205,11 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] expect(options?.initialValue).toBe(false) + // Nothing was rewritten here — the statement was left on disk and merely + // flagged — so the prompt must not claim data destruction the way the + // genuinely-rewritten case above does. + expect(String(options?.message)).not.toContain('DESTROYS data') + expect(String(options?.message)).toContain('flagged for review') }) // A directory whose sweep threw contributes 0 to both totals, so a failed diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index e4d4e2154..25c37b7da 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -82,12 +82,17 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { // scaffolds, and legacy eql_v2_encrypted. CIP-2991 + CIP-2994 + #693. const sweep = await rewriteEncryptedMigrations(cwd) - // A rewritten file is a DROP+ADD in disguise, and a flagged statement is one - // the sweep could not make safe at all. Either way the next keystroke can - // destroy data, so the prompt says so and defaults to NO — an - // `initialValue: true` immediately under a "do NOT run the migration" - // warning invites exactly the mistake the warning is about. - const unsafe = sweep.rewritten > 0 || sweep.skipped > 0 + // A rewritten file is a DROP+ADD in disguise — the next migrate destroys + // data on any table that already holds rows. A flagged statement never got + // that treatment: it is left on disk untouched, so nothing is destroyed by + // migrating, but a raw ALTER to an encrypted domain has no cast in + // Postgres and fails at migrate time until a human resolves it. Both + // default the prompt to NO — an `initialValue: true` immediately under + // either warning invites exactly the mistake the warning is about — but + // they need different words: claiming "DESTROYS data" for a migration + // that destroyed nothing is its own kind of wrong guidance. + const destructive = sweep.rewritten > 0 + const flaggedOnly = !destructive && sweep.skipped > 0 // A directory whose sweep threw contributes 0 to both totals, so on its own // it is indistinguishable from a clean sweep — except that it means the @@ -110,12 +115,14 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { } const shouldMigrate = await p.confirm({ - message: unsafe + message: destructive ? `Run the migration now? (${runner} drizzle-kit migrate) — see the warnings above: this migration DESTROYS data on any table that already holds rows` - : unverified - ? `Run the migration now? (${runner} drizzle-kit migrate) — the sweep could not check ${unverifiedCount} (${unverifiedList}); review those migrations before migrating, or you may apply broken/unsafe SQL` - : `Run the migration now? (${runner} drizzle-kit migrate)`, - initialValue: !unsafe && !unverified, + : flaggedOnly + ? `Run the migration now? (${runner} drizzle-kit migrate) — statement(s) were flagged for review above rather than rewritten; nothing was destroyed, but the raw ALTER will fail at migrate time until they're resolved` + : unverified + ? `Run the migration now? (${runner} drizzle-kit migrate) — the sweep could not check ${unverifiedCount} (${unverifiedList}); review those migrations before migrating, or you may apply broken/unsafe SQL` + : `Run the migration now? (${runner} drizzle-kit migrate)`, + initialValue: !destructive && !flaggedOnly && !unverified, }) if (!p.isCancel(shouldMigrate) && shouldMigrate) { From 0fde34405ffdc6b7f511102a79dd831b12c63e07 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 23:37:25 +1000 Subject: [PATCH 054/123] fix(cli,wizard): fold the block-comment strip into the preamble loop STATEMENT_PREAMBLE_RE anchored its block-comment group to `^` ahead of the line-comment loop, so it only stripped a leading block comment when that comment sat at the very start of the FILE. In any file with a preceding statement, NEAR_MISS_RE's match starts at that statement's `;` -- i.e. with a newline, not the comment -- so the anchored group could never match past it. The same statement was then reported twice: once correctly (already- encrypted or source-unknown), and once as unrecognised-form with the block comment glued to its front and guidance pointing at a hand-authored USING clause that isn't there. Fold the block comment into the repeating loop as its own alternative instead of a group ahead of it, so it can match after any number of leading newlines/line comments, in any interleaving. Hardens the existing regression test to use a preceding statement (the one shape the broken regex happened to handle, which is why it passed before), and adds coverage for an indented block comment on the ALTER's own line and for the already-encrypted reason. Each asserts the full skipped array, so both count and reason are pinned. Also documents the one known residue: a nested closed block comment ahead of a live ALTER still double-reports, because the alternative's lazy `*?` stops at the first `*/`. Accepted, not fixed. Verified by reverting the regex to the anchored form: the three new/ hardened tests fail with the described double-report in both packages, confirming they are load-bearing. --- .../src/__tests__/rewrite-migrations.test.ts | 65 ++++++++++++++++++- .../cli/src/commands/db/rewrite-migrations.ts | 36 +++++++--- .../src/__tests__/rewrite-migrations.test.ts | 65 ++++++++++++++++++- packages/wizard/src/lib/rewrite-migrations.ts | 36 +++++++--- 4 files changed, 184 insertions(+), 18 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 80f235c1e..275ee5852 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -493,11 +493,26 @@ describe('rewriteEncryptedAlterColumns', () => { // statement came back twice: once correctly as `source-unknown`, once as // `unrecognised-form` — contradictory advice (look for a hand-authored // `USING` clause) for a statement that has none. + // + // The block comment must NOT sit at the very start of the file — that is + // the one shape a previous, narrower version of the preamble regex happened + // to handle. A realistic migration file has a preceding statement, so + // NEAR_MISS_RE's match starts at THAT statement's `;`, dragging the newline + // before the comment in too; a regex that only strips a comment anchored to + // the very start of the match fails here. it('reports a block-comment-prefixed statement once, not twice', async () => { const alterSql = 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' const filePath = path.join(tmpDir, '0040_block-preamble.sql') - fs.writeFileSync(filePath, `/* note */\n${alterSql}\n`) + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '/* note */', + alterSql, + '', + ].join('\n'), + ) const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) @@ -507,6 +522,54 @@ describe('rewriteEncryptedAlterColumns', () => { ]) }) + // Same bug, but the comment sits on the ALTER's own line rather than its + // own — the preceding statement's `;` still starts the near-miss match + // before the (indented) comment, not at it. + it('reports an indented block-comment-prefixed statement once, not twice', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0041_block-preamble-indented.sql') + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + ` /* note */ ${alterSql}`, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // Same bug again, this time on the OTHER correct reason a near-miss can + // carry: the column is already encrypted, not merely undeclared. + it('reports a block-comment-prefixed statement once, not twice, for an already-encrypted column', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0042_block-preamble-encrypted.sql') + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" "public"."eql_v3_text_eq");', + '/* note */', + alterSql, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { const statement = 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 8042db3a8..cf3d0e0be 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -96,18 +96,30 @@ const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi /** - * Blank lines, a leading `/* … *\/` block comment, and `--` line comments - * (including drizzle-kit's `--> statement-breakpoint`) at the head of a + * Strips any run of blank lines, `--` line comments (including drizzle-kit's + * `--> statement-breakpoint`), and `/* … *\/` block comments — in any order, + * repeated as many times as they occur — from the head of a * {@link NEAR_MISS_RE} match. * * That regex opens with a lazy `[^;]*?`, whose only left boundary is the * previous `;` — or the start of the file when there is no preceding statement. * So the raw match drags in every comment and blank line since then, and a - * near-miss in a file that opens with a comment block gets reported to the user - * with that whole block glued to its front. Strip the preamble so the statement - * we quote back reads as the offending statement alone. - * - * The leading block comment is NOT cosmetic: a statement the strict + * near-miss preceded by a comment block gets reported to the user with that + * whole block glued to its front. Strip the preamble so the statement we quote + * back reads as the offending statement alone. + * + * The block comment is matched as its OWN loop alternative rather than a + * single group anchored ahead of the line-comment loop, because the latter + * only works when the block comment sits at the very start of the file: in + * the far more common case — a preceding statement earlier in the same file — + * the match starts at THAT statement's `;`, so it opens with the newline + * after it, not with the comment. An anchored `^(?:/\*…\*\/\s*)?` can't match + * past that newline to reach the comment, so it silently matches nothing and + * leaves the comment attached to the reported statement. Folding the block + * comment into the repeating loop lets it match after any number of leading + * newlines/line-comments, in any interleaving. + * + * The block comment strip is NOT cosmetic: a statement the strict * {@link ALTER_COLUMN_TO_ENCRYPTED_RE} matched but skipped (`already-encrypted` * or `source-unknown`) is left on disk unchanged, so it still contains * `SET DATA TYPE` and the broad scan below finds it again. Without stripping @@ -119,9 +131,17 @@ const NEAR_MISS_RE = * wrong for a statement the strict matcher already matched. Stripping the * comment here makes both passes agree on the statement text so the second * report collapses into the first. + * + * Known residue, accepted rather than fixed: a NESTED closed block comment + * ahead of a live ALTER (`/* outer /* inner *\/ still *\/`) still + * double-reports. The block-comment alternative's `*?` is lazy, so it stops + * at the FIRST `*\/` — consuming only `/* outer /* inner *\/` — and leaves + * `` still *\/`` glued to the front of the next iteration, which the + * line-comment alternative doesn't recognise either. That residual text rides + * along into the `unrecognised-form` report. */ const STATEMENT_PREAMBLE_RE = - /^(?:\/\*[\s\S]*?\*\/\s*)?(?:[^\S\n]*(?:--[^\n]*)?\n)*/ + /^(?:\s*\/\*[\s\S]*?\*\/|[^\S\n]*(?:--[^\n]*)?\n)*/ /** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ function trimStatementPreamble(statement: string): string { diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 5c737da73..01804dae6 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -449,11 +449,26 @@ describe('rewriteEncryptedAlterColumns', () => { // statement came back twice: once correctly as `source-unknown`, once as // `unrecognised-form` — contradictory advice (look for a hand-authored // `USING` clause) for a statement that has none. + // + // The block comment must NOT sit at the very start of the file — that is + // the one shape a previous, narrower version of the preamble regex happened + // to handle. A realistic migration file has a preceding statement, so + // NEAR_MISS_RE's match starts at THAT statement's `;`, dragging the newline + // before the comment in too; a regex that only strips a comment anchored to + // the very start of the match fails here. it('reports a block-comment-prefixed statement once, not twice', async () => { const alterSql = 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' const filePath = path.join(tmpDir, '0040_block-preamble.sql') - fs.writeFileSync(filePath, `/* note */\n${alterSql}\n`) + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '/* note */', + alterSql, + '', + ].join('\n'), + ) const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) @@ -463,6 +478,54 @@ describe('rewriteEncryptedAlterColumns', () => { ]) }) + // Same bug, but the comment sits on the ALTER's own line rather than its + // own — the preceding statement's `;` still starts the near-miss match + // before the (indented) comment, not at it. + it('reports an indented block-comment-prefixed statement once, not twice', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0041_block-preamble-indented.sql') + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + ` /* note */ ${alterSql}`, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'source-unknown' }, + ]) + }) + + // Same bug again, this time on the OTHER correct reason a near-miss can + // carry: the column is already encrypted, not merely undeclared. + it('reports a block-comment-prefixed statement once, not twice, for an already-encrypted column', async () => { + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0042_block-preamble-encrypted.sql') + fs.writeFileSync( + filePath, + [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY, "email" "public"."eql_v3_text_eq");', + '/* note */', + alterSql, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: filePath, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + it('reports a near-miss without a preceding statement-breakpoint marker', async () => { const statement = 'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;' diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index abb6afab0..0255df6b5 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -104,18 +104,30 @@ const NEAR_MISS_RE = /[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi /** - * Blank lines, a leading `/* … *\/` block comment, and `--` line comments - * (including drizzle-kit's `--> statement-breakpoint`) at the head of a + * Strips any run of blank lines, `--` line comments (including drizzle-kit's + * `--> statement-breakpoint`), and `/* … *\/` block comments — in any order, + * repeated as many times as they occur — from the head of a * {@link NEAR_MISS_RE} match. * * That regex opens with a lazy `[^;]*?`, whose only left boundary is the * previous `;` — or the start of the file when there is no preceding statement. * So the raw match drags in every comment and blank line since then, and a - * near-miss in a file that opens with a comment block gets reported to the user - * with that whole block glued to its front. Strip the preamble so the statement - * we quote back reads as the offending statement alone. - * - * The leading block comment is NOT cosmetic: a statement the strict + * near-miss preceded by a comment block gets reported to the user with that + * whole block glued to its front. Strip the preamble so the statement we quote + * back reads as the offending statement alone. + * + * The block comment is matched as its OWN loop alternative rather than a + * single group anchored ahead of the line-comment loop, because the latter + * only works when the block comment sits at the very start of the file: in + * the far more common case — a preceding statement earlier in the same file — + * the match starts at THAT statement's `;`, so it opens with the newline + * after it, not with the comment. An anchored `^(?:/\*…\*\/\s*)?` can't match + * past that newline to reach the comment, so it silently matches nothing and + * leaves the comment attached to the reported statement. Folding the block + * comment into the repeating loop lets it match after any number of leading + * newlines/line-comments, in any interleaving. + * + * The block comment strip is NOT cosmetic: a statement the strict * {@link ALTER_COLUMN_TO_ENCRYPTED_RE} matched but skipped (`already-encrypted` * or `source-unknown`) is left on disk unchanged, so it still contains * `SET DATA TYPE` and the broad scan below finds it again. Without stripping @@ -127,9 +139,17 @@ const NEAR_MISS_RE = * wrong for a statement the strict matcher already matched. Stripping the * comment here makes both passes agree on the statement text so the second * report collapses into the first. + * + * Known residue, accepted rather than fixed: a NESTED closed block comment + * ahead of a live ALTER (`/* outer /* inner *\/ still *\/`) still + * double-reports. The block-comment alternative's `*?` is lazy, so it stops + * at the FIRST `*\/` — consuming only `/* outer /* inner *\/` — and leaves + * `` still *\/`` glued to the front of the next iteration, which the + * line-comment alternative doesn't recognise either. That residual text rides + * along into the `unrecognised-form` report. */ const STATEMENT_PREAMBLE_RE = - /^(?:\/\*[\s\S]*?\*\/\s*)?(?:[^\S\n]*(?:--[^\n]*)?\n)*/ + /^(?:\s*\/\*[\s\S]*?\*\/|[^\S\n]*(?:--[^\n]*)?\n)*/ /** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */ function trimStatementPreamble(statement: string): string { From d96020e4cc3ab8475f73534b383564e26e78f2fb Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:12:59 +1000 Subject: [PATCH 055/123] docs(specs): design for the Encryption signature fixes and typecheck gates Covers A-4, A-6, A-7 and C-1 from the EQL v2 removal verification, and records what is deliberately excluded: the single generic signature and ClientFor machinery belong with the v2 removal (#637), not here. --- ...on-signature-and-typecheck-gates-design.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md diff --git a/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md b/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md new file mode 100644 index 000000000..5776ed1d7 --- /dev/null +++ b/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md @@ -0,0 +1,275 @@ +# `Encryption` signature fixes and typecheck gates + +**Date:** 2026-07-24 +**Branch:** `fix/encryption-signature-and-typecheck-gates` (off `8b3f0b15`) +**Addresses:** A-4, A-6, A-7, C-1 from `.work/2026-07-24-eql-v2-removal-verification.md` +**Issue:** #778 (review remediation for #772) + +## Why these four, and why not more + +A-6 and A-7 exist because `Encryption` is overloaded, and it is overloaded because there +are two kinds of client: the typed EQL v3 one and the nominal one that EQL v2 and loose +introspection-derived schemas need. Two overloads means two discriminators that can +disagree — overload resolution reads the `config` argument, the runtime reads the +`schemas` argument (`encryption/index.ts:947`). Remove EQL v2 and there is one client, +one signature, and both defects delete themselves. + +So this change deliberately does **not** attempt the type-level repair the verification +doc explores (a single generic signature returning `ClientFor`, a discriminated-union +`WireConfig`, `ConfigFor`). That machinery exists only to make two clients coexist +safely, and it would be deleted by the v2 removal that motivates it. It is issue #637 and +it belongs with PRs 9–12. + +What is left is the subset that is either independent of the v2 removal (A-4, C-1) or +cheap and non-throwaway (A-6, A-7). + +## A-4 — non-tuple schema arrays are rejected + +### Problem + +`Encryption({ schemas })` accepts only an array *literal*. Every indirect form fails with +`TS2769`: + +```ts +export const all: AnyV3Table[] = [users, orders] // shared module +await Encryption({ schemas: all }) // TS2769 +``` + +Also broken: `ReadonlyArray` (the type `prisma-next` exposes publicly), +push-built arrays, spreads, and unannotated `const all = [...]`. + +The constraint at `encryption/index.ts:872-877` is a non-empty *tuple*: + +```ts +export function Encryption< + const S extends readonly [AnyV3Table, ...AnyV3Table[]], +>(config: { schemas: S; config?: V3ClientConfig }): Promise> +``` + +It was narrowed to a tuple by `7e0092f3` for one reason: `readonly AnyV3Table[]` admits +`readonly []`, so `Encryption({ schemas: [] })` type-checked and then threw at runtime. + +This is a live break on a released surface. `prisma-next` already had to work around it +with a destructure-and-respread through three coupled edits in `from-stack-v3.ts`; any +customer with schema-building indirection hits the same wall. + +### Design + +Widen the type parameter to the array and move the non-emptiness check to the property, so +`[]` is rejected without the tuple constraining every other form: + +```ts +type NonEmptyV3 = S['length'] extends 0 ? never : S + +export function Encryption(config: { + schemas: NonEmptyV3 + config?: V3ClientConfig +}): Promise> +``` + +`schemas` must resolve through `NonEmptyV3` rather than being wrapped in a conditional +alias applied to the whole config — wrapping defeats `const` inference and degrades the +tuple to an array, losing per-column typing on the literal path. + +The runtime throw at `encryption/index.ts:891` stays as the backstop for JavaScript callers. + +### Acceptance + +A `.test-d.ts` probe pinning both directions: + +- compile: inline literal, shared `AnyV3Table[]`, `ReadonlyArray`, push-built, + spread, unannotated `const`; +- still error: `{ schemas: [] }`, wrong plaintext type for a column's domain, a table that + is not a member of the registered tuple. + +## A-6 — `ReturnType` resolves to the nominal client + +### Problem + +TypeScript's `ReturnType` reads the *last* overload, which is the nominal one. So +`Awaited>` yields `EncryptionClient` even for an all-v3 +schema set, and assigning the real client to it fails: + +``` +Type 'TypedEncryptionClient<…>' is missing the following properties +from type 'EncryptionClient': client, encryptConfig, init +``` + +Overload order cannot fix this — whichever signature is last wins, so one of the two forms +is always mis-resolved. Reordering additionally destroys the typed client, because a v3 +table structurally satisfies `BuildableTable` and so matches the nominal overload. + +This is a regression from the released surface, not a pre-existing wart: all 15 instances +were introduced by `d7ff8471`, which turned a single-signature `EncryptionV3` into an alias +of an overloaded `Encryption`. `packages/bench/src/drizzle/setup.ts:38-45` already carries a +hand-rolled workaround. + +15 sites, none visible to CI: + +| Package | Sites | +|---|---| +| `packages/stack` | `__tests__/dynamodb/encrypted-dynamodb-v3.test.ts` (×3), `__tests__/encrypt-lock-context-guards.test.ts`, `__tests__/encrypt-query-searchable-json.test.ts`, `__tests__/encrypt-query-stevec.test.ts`, `integration/shared/{matrix-crypto,matrix-sql,schema-pg,schema-v3-client}.integration.test.ts` | +| `packages/stack-drizzle` | `integration/{adapter,json-adapter}.ts`, `integration/{lock-context,null-persistence,relational}.integration.test.ts` | + +### Design + +No signature change. `EncryptionClientFor` (`encryption/v3.ts:399-402`) already exists and +is the correct idiom — the prior-art survey in the verification doc found that no library +dispatches a schema-dependent result type through `ReturnType`; they all expose a named +extraction type (`z.infer`, `typeof x.infer`, hono's `Client`). Convert the call sites to +it and document it as *the* way to name the client. + +**`EncryptionClientFor` must be widened in step with A-4.** It carries the same narrow tuple +guard: + +```ts +S extends readonly [AnyV3Table, ...AnyV3Table[]] ? TypedEncryptionClient : EncryptionClient +``` + +Left alone, `EncryptionClientFor` falls through to `EncryptionClient` — +so the type A-6 tells callers to use would silently hand back the nominal client for exactly +the non-tuple schemas A-4 just enabled. It becomes: + +```ts +export type EncryptionClientFor = + S extends readonly AnyV3Table[] + ? S['length'] extends 0 + ? EncryptionClient + : TypedEncryptionClient + : EncryptionClient +``` + +The `readonly []` arm must be checked *inside* the v3 branch and before the tuple is used: +`never extends X` is true, so an empty tuple otherwise satisfies "all elements are v3". + +Erased sites that pass `[schema as never]` (the generic `stack-drizzle` integration adapters) +are declared `EncryptionClientFor`, which resolves to +`TypedEncryptionClient` and accepts `TypedEncryptionClient` +by method bivariance. + +`encryption-overloads.test-d.ts:84-88` currently asserts the defect as expected behaviour and +is green in CI. It is rewritten to assert `EncryptionClientFor` resolves correctly instead. + +## A-7 — the type says nominal, the runtime hands back typed + +### Problem + +Overload resolution matches on the `config` argument; the runtime picks its client by +inspecting the `schemas` (`encryption/index.ts:947`, `isV3Only && eqlVersion === 3`). They +disagree whenever a config is hoisted into a `ClientConfig`-typed variable: + +```ts +const cfg: ClientConfig = {} // not assignable to V3ClientConfig +const c = await Encryption({ schemas: [users], config: cfg }) +// type: EncryptionClient runtime: TypedEncryptionClient +c.init(...) // TypeError: init is not a function +``` + +`ClientConfig.eqlVersion` is `2 | 3`; the v3 overload requires `eqlVersion?: 3`. So the +variable form selects the nominal overload while the runtime still returns the typed client. +This bites whenever the variable's runtime `eqlVersion` is anything but `2` — `{}`, +`{ keyset }`, `{ authStrategy }`: the common case. + +Measured blast radius: `init` is the **only** member on `EncryptionClient.prototype` absent +from the typed client at runtime. The other ten are all present. + +### Design + +Add an `init` passthrough to the object `typedClient()` returns, declared `@internal` on +`TypedEncryptionClient` so `satisfies` still checks the shape: + +```ts +init: (config) => client.init(config), +``` + +This reduces the runtime gap to zero. What remains is a silent capability downgrade — the +type says nominal, so the caller loses the typed surface with no diagnostic. No type-level +design closes that: the runtime inspects values while the type inspects an erasable static +type. It ends when v2 removal collapses the two clients into one. + +### Acceptance + +A unit test reproducing the exact shape (config declared `ClientConfig`, v3 schemas, +`EncryptionClient.prototype.init` stubbed so no credentials are needed) that fails with +`TypeError: client.init is not a function` before the change and passes after. + +## C-1 — typecheck gates + +### Problem + +The root `typecheck` gate for `examples/*` already exists (`.github/workflows/tests.yml:155-161`, +added by `5fab1cf6`) — the verification doc refutes the original framing. The remaining gap is +the surfaces nothing typechecks at all. + +Measured after a full `pnpm run build` (most apparent failures were unbuilt workspace deps +resolving to `dist/*.d.ts`, not real errors): + +| Surface | Errors | Has script | +|---|---|---| +| `packages/bench` | 0 | no | +| `packages/migrate` | 0 | no | +| `packages/prisma-next` | 0 | `typecheck` | +| `packages/test-kit` | 0 | `test:types` | +| `packages/wizard` | 0 | `typecheck` | +| `examples/basic` | 0 | `typecheck` (gated) | +| `examples/prisma` | 0 | `typecheck` (never invoked) | +| `e2e` | 0 | no | +| `packages/nextjs` | 2 | no | +| `packages/stack-supabase` | 11 | `test:types` (narrower config) | +| `packages/cli` | 21 | no | +| `packages/stack-drizzle` | 69 | `test:types` (narrower config) | +| `packages/stack` | 168 | `test:types` (narrower config) | + +The three `test:types` scripts run `vitest --typecheck.only` against a `tsconfig.typecheck.json` +whose `include` is `__tests__/**/*.test-d.ts` — so the package's real `tsconfig.json`, which +covers `src`, `__tests__/**/*.test.ts` and `integration/**`, is never checked. + +### Design + +Gate what is green, and record what is not with its count rather than silently skipping it. + +1. Add `typecheck` scripts (`tsc --noEmit -p tsconfig.json`) to `bench`, `migrate`, `nextjs`, + `e2e`; keep the existing ones on `prisma-next`, `wizard`, `examples/*`. +2. One CI job running the eight green surfaces plus `nextjs`, after `pnpm run build` (they + resolve workspace deps through `dist/*.d.ts`). +3. Fix `packages/nextjs`'s 2 errors: `vi.Mock` is used as a *type* in + `__tests__/nextjs.test.ts:68,81`; it needs `import type { Mock } from 'vitest'`. +4. Add `"outputs": ["dist/**"]` to `turbo.json`'s `build` task. It currently declares none, + so a cached build restores nothing and the `examples/basic` gate — which typechecks + against `packages/stack/dist/*.d.ts` — holds today only because fresh CI runners have no + cache. +5. Record `stack-supabase` (11), `cli` (21), `stack-drizzle` (69), `stack` (168) as + documented follow-ups. + +### Deliberately out of scope + +Making `stack`, `stack-drizzle`, `stack-supabase` and `cli` green. 51 of `stack-drizzle`'s 69 +and all 11 of `stack-supabase`'s are one root cause — `spec.indexes.unique` / `.ore` / `.ope` +against `V3_MATRIX` in `packages/test-kit`'s catalog, where `typedEntries` collapses the spec +to a union whose members do not all carry those keys. A single fix in `test-kit` likely clears +~62 of the ~80 errors across those two packages. That is the highest-leverage next step and it +is its own change. + +## Interactions and ordering + +A-4 and A-6 must land together: widening `Encryption` without widening `EncryptionClientFor` +leaves the documented idiom resolving to the wrong client. + +A-7 is independent but shares the same files, and it removes `init` from A-6's `TS2739` +message (leaving only the private `client` / `encryptConfig` fields), so it lands first to +keep the A-6 diffs legible. + +C-1 is independent of all three. It is sequenced last because the A-6 conversions remove 15 +of the errors in the two packages it reports counts for. + +## Other deliverables + +- Changeset: `@cipherstash/stack` **minor** — A-4 widens a released signature and A-7 adds a + member to `TypedEncryptionClient`. `@cipherstash/stack-drizzle` patch for the integration + adapter conversions. +- Delete the migration paragraph in `.changeset/stack-audit-on-decrypt.md` telling callers to + narrow `AnyV3Table[]` to `readonly [AnyV3Table, ...AnyV3Table[]]` — A-4 makes that advice + obsolete, and it was never correct for `ReadonlyArray` anyway. +- Skills sweep: `skills/stash-encryption/SKILL.md` for any `ReturnType` + guidance and for schema-array examples that the widening now permits. +- `packages/stack/README.md` and `packages/prisma-next` docs for the same. From c94a5cc46f7af91b4c2666f079070ee5842fcfde Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:16:48 +1000 Subject: [PATCH 056/123] fix(stack): give the typed client an init passthrough (A-7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Encryption` picks its return value with two discriminators that can disagree — overload resolution reads the `config` argument, the runtime reads the `schemas` argument. Hoisting a config into a `ClientConfig`-typed variable splits them: `ClientConfig.eqlVersion` is `2 | 3`, which the v3 overload's `eqlVersion?: 3` rejects, so the call resolves to the nominal client while the runtime still returns the typed one. `init` was the only member of `EncryptionClient` absent from the typed client, so anything holding that client through its declared type hit `TypeError: client.init is not a function`. Delegate it, and assert the parity so a future member cannot reopen the hole. The remaining half — the silent loss of the typed surface — is not closable at the type level (the runtime inspects values, the type inspects an erasable static type). It ends with the EQL v2 removal (#637). --- .../typed-client-nominal-parity.test.ts | 82 +++++++++++++++++++ packages/stack/src/encryption/v3.ts | 16 ++++ 2 files changed, 98 insertions(+) create mode 100644 packages/stack/__tests__/typed-client-nominal-parity.test.ts diff --git a/packages/stack/__tests__/typed-client-nominal-parity.test.ts b/packages/stack/__tests__/typed-client-nominal-parity.test.ts new file mode 100644 index 000000000..dc0bcc1c2 --- /dev/null +++ b/packages/stack/__tests__/typed-client-nominal-parity.test.ts @@ -0,0 +1,82 @@ +/** + * Runtime parity between the two clients `Encryption` can return. + * + * `Encryption` decides its return value with two discriminators that can + * disagree: overload resolution reads the `config` argument, while the runtime + * reads the `schemas` argument (`encryption/index.ts`, the `isV3Only && + * eqlVersion === 3` branch). Hoisting a config into a `ClientConfig`-typed + * variable is enough to split them — `ClientConfig.eqlVersion` is `2 | 3`, which + * is not assignable to the v3 overload's `eqlVersion?: 3`, so the call selects + * the NOMINAL overload while the runtime still hands back the TYPED client. + * + * No type-level design closes that: the runtime inspects values, the type + * inspects a static type that can be widened away. It ends when the EQL v2 + * removal collapses the two clients into one (#637). Until then the mismatch + * must not be able to crash, which means every member of `EncryptionClient` has + * to exist on the typed client at runtime. + */ +import { describe, expect, it, vi } from 'vitest' +import { Encryption, EncryptionClient } from '@/encryption' +import { encryptedTable, types } from '@/encryption/v3' +import type { ClientConfig } from '@/types' + +const users = encryptedTable('users', { email: types.TextSearch('email') }) + +/** + * Build a client without credentials by stubbing `init` to resolve to itself — + * `Encryption` only needs a successful `Result` to reach its return branch. + */ +async function buildWithStubbedInit(config: ClientConfig) { + const spy = vi + .spyOn(EncryptionClient.prototype, 'init') + .mockImplementation(async function (this: EncryptionClient) { + return { data: this } + } as never) + try { + return await Encryption({ schemas: [users], config }) + } finally { + spy.mockRestore() + } +} + +describe('typed client / nominal client runtime parity', () => { + it('a ClientConfig-typed variable types as nominal but returns the typed client', async () => { + const config: ClientConfig = {} + const client = await buildWithStubbedInit(config) + + // The static type here is `EncryptionClient`. The runtime disagrees. + expect('encryptQuery' in client).toBe(true) + expect(client).not.toBeInstanceOf(EncryptionClient) + }) + + it('exposes every EncryptionClient member, so the mismatch cannot crash', async () => { + const config: ClientConfig = {} + const client = await buildWithStubbedInit(config) + + const nominalMembers = Object.getOwnPropertyNames( + EncryptionClient.prototype, + ).filter((name) => name !== 'constructor') + + // `init` was the only member missing, which turned the type/runtime + // mismatch above into `TypeError: client.init is not a function` for + // anything holding the client through its declared `EncryptionClient` type. + const missing = nominalMembers.filter( + (name) => typeof (client as Record)[name] !== 'function', + ) + expect(missing).toEqual([]) + }) + + it('delegates init to the underlying client', async () => { + const config: ClientConfig = {} + const client = await buildWithStubbedInit(config) + + const spy = vi + .spyOn(EncryptionClient.prototype, 'init') + .mockResolvedValue({ data: {} } as never) + + await (client as unknown as EncryptionClient).init({} as never) + expect(spy).toHaveBeenCalledTimes(1) + + spy.mockRestore() + }) +}) diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 3ba2f19ce..3fd949157 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -150,6 +150,21 @@ export interface TypedEncryptionClient { ): BulkEncryptOperation bulkDecrypt(payloads: BulkDecryptPayload): BulkDecryptOperation getEncryptConfig(): ReturnType + + /** + * Re-initialize the underlying client. + * + * @internal Present for runtime parity with {@link EncryptionClient}, not as + * part of the typed authoring surface. `Encryption` picks its return value by + * inspecting the *schemas* while overload resolution inspects the *config*, so + * the two can disagree: a config hoisted into a `ClientConfig`-typed variable + * selects the nominal overload but still yields this client at runtime. Every + * other `EncryptionClient` method already existed here; without `init` that + * mismatch turned into `TypeError: client.init is not a function`. + */ + init( + config: Parameters[0], + ): ReturnType } /** @@ -367,6 +382,7 @@ export function typedClient( bulkEncrypt: (plaintexts, opts) => client.bulkEncrypt(plaintexts, opts), bulkDecrypt: (payloads) => client.bulkDecrypt(payloads), getEncryptConfig: () => client.getEncryptConfig(), + init: (config) => client.init(config), } satisfies TypedEncryptionClient } From fc13052e8a1b5dd6b33ee95a866e716a9a69d473 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:17:59 +1000 Subject: [PATCH 057/123] fix(stack)!: accept schema arrays that are not literals (A-4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Encryption({ schemas })` only accepted an array LITERAL. Every indirect form failed with TS2769: a shared `export const all: AnyV3Table[]`, the `ReadonlyArray` prisma-next exposes publicly, anything push-built or spread, an unannotated `const`. The cause was how the empty-schema hole was closed — by constraining the type parameter to a non-empty TUPLE, which also excluded every non-literal. Move non-emptiness onto the `schemas` property via `NonEmptyV3` and let the parameter be the array. `Encryption({ schemas: [] })` is still a compile error, and the literal path keeps its per-column plaintext typing — the guard has to sit on the property, since wrapping the config in a conditional alias defeats `const` inference. `EncryptionClientFor` carried the same tuple guard and is widened in step. Left alone it fell through to the nominal client for a loose `AnyV3Table[]` — handing the wrong type to exactly the callers this enables. Prisma-next's destructure-and-respread workaround can now be removed. --- .../__tests__/encryption-overloads.test-d.ts | 84 ++++++++++++++++--- packages/stack/src/encryption/index.ts | 32 +++++-- packages/stack/src/encryption/v3.ts | 47 +++++++---- 3 files changed, 128 insertions(+), 35 deletions(-) diff --git a/packages/stack/__tests__/encryption-overloads.test-d.ts b/packages/stack/__tests__/encryption-overloads.test-d.ts index e3afd5929..82cc1f978 100644 --- a/packages/stack/__tests__/encryption-overloads.test-d.ts +++ b/packages/stack/__tests__/encryption-overloads.test-d.ts @@ -16,7 +16,7 @@ import { encryptedTable, type TypedEncryptionClient, } from '@/encryption/v3' -import { types } from '@/eql/v3' +import { type AnyV3Table, types } from '@/eql/v3' import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' const users = encryptedTable('users', { @@ -49,6 +49,49 @@ describe('overload selection', () => { Encryption({ schemas: [] }) }) + // A-4: closing S-6 with a non-empty TUPLE constraint rejected every schema + // array that is not a literal, which is most real code — a shared module + // export, anything built from introspection, anything `readonly`. The + // non-emptiness check moved to the `schemas` property so these compile again + // while `[]` above still does not. One case per form that was broken. + it('accepts schema arrays that are not literals', async () => { + const shared: AnyV3Table[] = [users] + expectTypeOf(await Encryption({ schemas: shared })).toEqualTypeOf< + TypedEncryptionClient + >() + + // `ReadonlyArray` is the form `@cipherstash/prisma-next` exposes publicly. + const frozen: ReadonlyArray = [users] + expectTypeOf(await Encryption({ schemas: frozen })).toEqualTypeOf< + TypedEncryptionClient + >() + + const built: AnyV3Table[] = [] + built.push(users) + expectTypeOf(await Encryption({ schemas: built })).toEqualTypeOf< + TypedEncryptionClient + >() + + expectTypeOf(await Encryption({ schemas: [...shared] })).toEqualTypeOf< + TypedEncryptionClient + >() + }) + + // The widening must not cost the literal path its precision — that typing is + // the entire reason the typed client exists. `const` inference has to survive. + it('keeps per-column typing on the literal path', async () => { + const client = await Encryption({ schemas: [users] }) + + expectTypeOf(client.encrypt).toBeCallableWith('a@b.com', { + table: users, + column: users.email, + }) + // @ts-expect-error - `email` is a text domain, not a number + client.encrypt(123, { table: users, column: users.email }) + // @ts-expect-error - `createdAt` is a timestamp domain, not a string + client.encrypt('2020-01-01', { table: users, column: users.createdAt }) + }) + // S-4: forcing v2 wire over v3 schemas returns the NOMINAL client at runtime // (the typed client cannot author v3 columns in v2 mode). The types used to // claim the typed client, so `decryptModel(row, table, lockContext)` compiled @@ -77,16 +120,6 @@ describe('overload selection', () => { }) describe('naming the client type', () => { - // S-2: `ReturnType` reads the LAST overload, so this idiom resolves to the - // nominal client no matter what schemas you pass. Pinned rather than fixed — - // overload order cannot satisfy both forms — so the surprise is documented and - // cannot change silently. - it('ReturnType resolves to the NOMINAL client', () => { - expectTypeOf< - Awaited> - >().toEqualTypeOf() - }) - it('EncryptionClientFor names the typed client for a v3 tuple', async () => { const client: EncryptionClientFor = await Encryption({ schemas: [users] }) @@ -95,10 +128,39 @@ describe('naming the client type', () => { >() }) + // A-6: the form generic code needs — an integration adapter that builds its + // table per test family cannot name a tuple. This must track `Encryption`'s + // own constraint: while `EncryptionClientFor` still required a non-empty + // TUPLE it fell through to the nominal client here, silently handing the + // wrong type to exactly the callers the widening above exists to serve. + it('EncryptionClientFor names the typed client for a loose v3 array', () => { + expectTypeOf>().toEqualTypeOf< + TypedEncryptionClient + >() + }) + it('EncryptionClientFor falls back to the nominal client', () => { expectTypeOf< EncryptionClientFor >().toEqualTypeOf() + + // `never extends X` is true, so an empty tuple satisfies "every element is + // a v3 table" — the emptiness arm has to be checked inside the v3 branch. + expectTypeOf< + EncryptionClientFor + >().toEqualTypeOf() + }) + + // S-2: `ReturnType` reads the LAST overload, so this idiom resolves to the + // nominal client no matter what schemas you pass. Overload order cannot + // satisfy both forms — putting the nominal signature first mis-resolves v3 + // schemas instead, because a v3 table structurally satisfies `BuildableTable`. + // `EncryptionClientFor` above is the supported idiom; this pins the trap so it + // cannot start silently resolving differently. + it('ReturnType resolves to the NOMINAL client', () => { + expectTypeOf< + Awaited> + >().toEqualTypeOf() }) }) diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index 5a47a930a..39b8ff907 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -862,17 +862,35 @@ export function __resetStrategyDeprecationWarningForTests(): void { * @see {@link ClientConfig.authStrategy} for the auth strategy field. * @see {@link EncryptionClient} for available methods after initialization. */ +/** + * The schema-tuple guard for {@link Encryption}'s v3 overload. + * + * Resolves to `S` for any non-empty array of v3 tables and to `never` for + * `readonly []`, so `Encryption({ schemas: [] })` stays a compile error while + * every non-literal form (a shared `AnyV3Table[]`, a `ReadonlyArray`, a + * push-built or spread array) still selects this overload. + */ +type NonEmptyV3 = S['length'] extends 0 + ? never + : S + // Overload 1 — v3-typed: an array literal of concrete EQL v3 tables (from // `@cipherstash/stack/v3`) yields the strongly-typed {@link TypedEncryptionClient}, // the collapse of the former `EncryptionV3`. The wire format is forced to v3. // -// The schema tuple is constrained NON-EMPTY: `readonly AnyV3Table[]` admits -// `readonly []`, so `Encryption({ schemas: [] })` type-checked and then threw at -// runtime. The nominal overload has always required at least one table. -export function Encryption< - const S extends readonly [AnyV3Table, ...AnyV3Table[]], ->(config: { - schemas: S +// `S` is the ARRAY, not a non-empty tuple. Constraining the type parameter to +// `readonly [AnyV3Table, ...AnyV3Table[]]` — which is how the `[]` case was +// first closed — rejected every form that is not an array literal: a shared +// `export const all: AnyV3Table[]`, the `ReadonlyArray` prisma-next exposes, +// anything push-built or spread. Non-emptiness is enforced on the PROPERTY via +// {@link NonEmptyV3} instead, which leaves `Encryption({ schemas: [] })` a +// compile error without constraining the rest. +// +// `NonEmptyV3` must sit on `schemas` and nowhere else: wrapping the whole +// config in a conditional alias defeats `const` inference, degrading the tuple +// to an array and erasing per-column plaintext typing on the literal path. +export function Encryption(config: { + schemas: NonEmptyV3 config?: V3ClientConfig }): Promise> // Overload 2 — nominal: loose/dynamic schemas (introspection-derived, e.g. diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 3fd949157..248aaad41 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -389,32 +389,45 @@ export function typedClient( /** * The client type {@link Encryption} resolves to for the schema tuple `S`. * - * **Use this instead of `Awaited>`.** `Encryption` - * is overloaded, and TypeScript's `ReturnType` reads the LAST overload — the - * nominal one — so that expression yields `EncryptionClient` even for an all-v3 - * schema set, and assigning the real (typed) client to it is an error: - * - * ``` - * Type 'TypedEncryptionClient<…>' is missing the following properties - * from type 'EncryptionClient': client, encryptConfig, init - * ``` - * - * Overload order cannot fix that — whichever signature is last wins, so one of - * the two forms is always mis-resolved. Name the schema tuple instead: + * This is **the** way to name the client — reach for it whenever you need to + * declare a variable, field or return type before the `await` that produces it: * * ```typescript * const users = encryptedTable("users", { email: types.TextSearch("email") }) + * * let client: EncryptionClientFor * client = await Encryption({ schemas: [users] }) * ``` * - * The equivalent inline workaround — inferring through a single-signature - * helper, `Awaited>` — also works, and is what - * `packages/bench` does. + * For code that is generic over its schemas — integration adapters that build a + * table per test family, say — name the loose array and keep the typed surface: + * + * ```typescript + * let client: EncryptionClientFor + * ``` + * + * **Do not use `Awaited>`.** `Encryption` is + * overloaded, and TypeScript's `ReturnType` reads the LAST overload — the + * nominal one — so that expression yields `EncryptionClient` even for an all-v3 + * schema set, and assigning the real client to it is an error: + * + * ``` + * Type 'TypedEncryptionClient<…>' is missing the following properties + * from type 'EncryptionClient': client, encryptConfig + * ``` + * + * Overload order cannot fix that — whichever signature is last wins, so one of + * the two forms is always mis-resolved, and putting the nominal signature first + * mis-resolves v3 schemas instead (a v3 table structurally satisfies + * `BuildableTable`). A named extraction type is what every comparable library + * does for the same reason: `z.infer`, arktype's `typeof T.infer`, hono's + * `Client`. */ export type EncryptionClientFor = - S extends readonly [AnyV3Table, ...AnyV3Table[]] - ? TypedEncryptionClient + S extends readonly AnyV3Table[] + ? S['length'] extends 0 + ? EncryptionClient + : TypedEncryptionClient : EncryptionClient /** From 031eab7ffc0f9f58bf44e0202252e3e10ede1be2 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:34:28 +1000 Subject: [PATCH 058/123] fix(stack,stack-drizzle,bench): name the client with EncryptionClientFor (A-6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ReturnType` reads the LAST overload, which is the nominal one, so `Awaited>` yielded `EncryptionClient` even for an all-v3 schema set and every assignment of the real client to it failed with TS2739. 15 sites carried that error — 10 in stack, 5 in stack-drizzle — and none was visible to CI, because the only typecheck step in either package is scoped to `__tests__/**/*.test-d.ts`. Overload order cannot fix it (putting the nominal signature first mis-resolves v3 schemas instead, since a v3 table structurally satisfies BuildableTable), so name the client instead — which is what every comparable library does: z.infer, arktype's typeof T.infer, hono's Client. Sweeps the idiom out entirely, not just the erroring sites: nominal callers now say `EncryptionClient`, v3 callers `EncryptionClientFor`. packages/bench drops the single-signature helper it wrapped Encryption in for exactly this reason. Typing these clients correctly unmasked call sites the wrong type had been hiding. Most were `as never` casts that are simply no longer needed, and dropping them means the calls are now genuinely checked. Two are not: encrypt-query-stevec and encrypt-query-searchable-json exercise `encryptQuery` query types the typed client mis-models — it derives the plaintext from the column's domain, so every query type on a types.Json() column is typed JsonDocument, but the SteVec types take a JSONPath string, a { path, value } pair, or a bare scalar. Those two suites hold the client through the nominal surface with the gap named at the cast, rather than casting at every call and hiding it. packages/stack tsconfig: 168 -> 147 errors, none new. packages/stack-drizzle tsconfig: 69 -> 63 errors, none new. --- packages/bench/src/drizzle/setup.ts | 20 +++++----------- .../__tests__/operators.test-d.ts | 4 ++-- packages/stack-drizzle/integration/adapter.ts | 8 +++++-- .../stack-drizzle/integration/json-adapter.ts | 8 +++++-- .../lock-context.integration.test.ts | 8 +++++-- .../null-persistence.integration.test.ts | 8 +++++-- .../relational.integration.test.ts | 8 +++++-- packages/stack/__tests__/audit.test.ts | 3 ++- .../stack/__tests__/backward-compat.test.ts | 3 ++- .../stack/__tests__/basic-protect.test.ts | 3 ++- packages/stack/__tests__/bulk-protect.test.ts | 3 ++- .../decrypt-audit-forwarding.test.ts | 3 ++- .../dynamodb/encrypted-dynamodb-v3.test.ts | 23 +++++++++---------- .../encrypt-lock-context-guards.test.ts | 15 ++++++++---- .../encrypt-query-searchable-json.test.ts | 14 ++++++++--- .../__tests__/encrypt-query-stevec.test.ts | 15 +++++++++--- packages/stack/__tests__/json-protect.test.ts | 3 ++- .../__tests__/lock-context-wiring.test.ts | 3 ++- .../stack/__tests__/number-protect.test.ts | 3 ++- packages/stack/__tests__/protect-ops.test.ts | 3 ++- .../v3-matrix/matrix-lock-context.test.ts | 3 ++- .../matrix-identity.integration.test.ts | 5 ++-- .../shared/json-crypto.integration.test.ts | 3 ++- .../shared/match-bloom.integration.test.ts | 4 +++- .../shared/matrix-bulk.integration.test.ts | 3 ++- .../shared/matrix-crypto.integration.test.ts | 9 ++++++-- .../shared/matrix-sql.integration.test.ts | 17 ++++++++++---- .../shared/schema-pg.integration.test.ts | 8 ++++--- .../schema-v3-client.integration.test.ts | 4 ++-- 29 files changed, 140 insertions(+), 74 deletions(-) diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index 16ba56d2e..87da15873 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -1,4 +1,4 @@ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { type EncryptionClientFor, EncryptionV3 } from '@cipherstash/stack/v3' import { extractEncryptionSchema, types } from '@cipherstash/stack-drizzle' import { drizzle } from 'drizzle-orm/node-postgres' import { pgTable, serial } from 'drizzle-orm/pg-core' @@ -34,19 +34,9 @@ export type BenchPlaintextRow = { enc_jsonb: { idx: number; group: number } } -/** - * Build the typed EQL v3 client this bench drives. Wrapped in a - * single-signature helper because `EncryptionV3` is now overloaded (typed v3 - * vs. nominal) — `ReturnType` resolves to the *last* - * (nominal) overload, so we infer the return type through this helper instead. - */ -function makeEncryptionClient() { - return EncryptionV3({ schemas: [encryptionBenchTable] }) -} - /** The typed EQL v3 client this bench drives. */ -export type BenchEncryptionClient = Awaited< - ReturnType +export type BenchEncryptionClient = EncryptionClientFor< + readonly [typeof encryptionBenchTable] > export type BenchHandle = { @@ -69,7 +59,9 @@ export async function buildBench(): Promise { const db = drizzle(pool) - const encryptionClient = await makeEncryptionClient() + const encryptionClient = await EncryptionV3({ + schemas: [encryptionBenchTable], + }) return { pgClient, pool, db, encryptionClient } } diff --git a/packages/stack-drizzle/__tests__/operators.test-d.ts b/packages/stack-drizzle/__tests__/operators.test-d.ts index b5e1be918..600a62d63 100644 --- a/packages/stack-drizzle/__tests__/operators.test-d.ts +++ b/packages/stack-drizzle/__tests__/operators.test-d.ts @@ -4,7 +4,7 @@ import type { EncryptionClient } from '@cipherstash/stack/encryption' import type { EncryptionError } from '@cipherstash/stack/errors' import type { LockContext } from '@cipherstash/stack/identity' import type { EncryptedQueryResult } from '@cipherstash/stack/types' -import type { EncryptionV3 } from '@cipherstash/stack/v3' +import type { AnyV3Table, EncryptionClientFor } from '@cipherstash/stack/v3' import { describe, expectTypeOf, it } from 'vitest' import { createEncryptionOperators } from '../src/index.js' @@ -21,7 +21,7 @@ import { createEncryptionOperators } from '../src/index.js' * existing typecheck scope without dragging the loose-typed runtime suites in. */ describe('createEncryptionOperators - client parameter (M1)', () => { - type V3Client = Awaited> + type V3Client = EncryptionClientFor // A query operation resolving `Result` — the surface the factory drives. type QueryOp = { diff --git a/packages/stack-drizzle/integration/adapter.ts b/packages/stack-drizzle/integration/adapter.ts index a2e1ea848..635572de3 100644 --- a/packages/stack-drizzle/integration/adapter.ts +++ b/packages/stack-drizzle/integration/adapter.ts @@ -1,4 +1,8 @@ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { + type AnyV3Table, + type EncryptionClientFor, + EncryptionV3, +} from '@cipherstash/stack/v3' import { databaseUrl, type IntegrationAdapter, @@ -57,7 +61,7 @@ type AnyTable = any export function makeDrizzleAdapter(): IntegrationAdapter { let sqlClient: postgres.Sql let db: ReturnType - let client: Awaited> + let client: EncryptionClientFor let ops: ReturnType let table: AnyTable let schema: ReturnType diff --git a/packages/stack-drizzle/integration/json-adapter.ts b/packages/stack-drizzle/integration/json-adapter.ts index 2ed37577b..44743a1d3 100644 --- a/packages/stack-drizzle/integration/json-adapter.ts +++ b/packages/stack-drizzle/integration/json-adapter.ts @@ -1,4 +1,8 @@ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { + type AnyV3Table, + type EncryptionClientFor, + EncryptionV3, +} from '@cipherstash/stack/v3' import { databaseUrl, type JsonIntegrationAdapter, @@ -25,7 +29,7 @@ export function makeDrizzleJsonAdapter(): JsonIntegrationAdapter { let db: ReturnType let tableName: string let table: AnyTable - let client: Awaited> + let client: EncryptionClientFor let ops: ReturnType const rowsFor = async ( diff --git a/packages/stack-drizzle/integration/lock-context.integration.test.ts b/packages/stack-drizzle/integration/lock-context.integration.test.ts index 8167c0edf..489a78100 100644 --- a/packages/stack-drizzle/integration/lock-context.integration.test.ts +++ b/packages/stack-drizzle/integration/lock-context.integration.test.ts @@ -33,7 +33,11 @@ * control. */ import { OidcFederationStrategy } from '@cipherstash/stack' -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { + type AnyV3Table, + type EncryptionClientFor, + EncryptionV3, +} from '@cipherstash/stack/v3' import { databaseUrl, unwrapResult, V3_MATRIX } from '@cipherstash/test-kit' import { clerkJwtProvider } from '@cipherstash/test-kit/integration-clerk' import { and, asc as drizzleAsc, eq as drizzleEq, type SQL } from 'drizzle-orm' @@ -71,7 +75,7 @@ const schema = extractEncryptionSchema(secretTable) type SelectRow = { rowKey: string } -let client: Awaited> +let client: EncryptionClientFor let ops: ReturnType let db: ReturnType diff --git a/packages/stack-drizzle/integration/null-persistence.integration.test.ts b/packages/stack-drizzle/integration/null-persistence.integration.test.ts index f97c1d29f..6a0e0f418 100644 --- a/packages/stack-drizzle/integration/null-persistence.integration.test.ts +++ b/packages/stack-drizzle/integration/null-persistence.integration.test.ts @@ -13,7 +13,11 @@ * as SQL NULL, and the present cell still decrypts to its plaintext. */ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { + type AnyV3Table, + type EncryptionClientFor, + EncryptionV3, +} from '@cipherstash/stack/v3' import { databaseUrl, V3_MATRIX } from '@cipherstash/test-kit' import { and, asc as drizzleAsc, eq as drizzleEq, type SQL } from 'drizzle-orm' import { integer, pgTable, text } from 'drizzle-orm/pg-core' @@ -84,7 +88,7 @@ const schema = extractEncryptionSchema(nullableTable) type SelectRow = { rowKey: string } -let client: Awaited> +let client: EncryptionClientFor let ops: ReturnType let db: ReturnType diff --git a/packages/stack-drizzle/integration/relational.integration.test.ts b/packages/stack-drizzle/integration/relational.integration.test.ts index 8e6eecb38..5fac50151 100644 --- a/packages/stack-drizzle/integration/relational.integration.test.ts +++ b/packages/stack-drizzle/integration/relational.integration.test.ts @@ -19,7 +19,11 @@ * `stash eql install`. This suite throws rather than skips when unconfigured. */ -import { EncryptionV3 } from '@cipherstash/stack/v3' +import { + type AnyV3Table, + type EncryptionClientFor, + EncryptionV3, +} from '@cipherstash/stack/v3' import { type DomainSpec, databaseUrl, @@ -136,7 +140,7 @@ type RowKey = (typeof ROWS)[number] type MatrixPlainRow = Record type SelectRow = { rowKey: string } type Db = ReturnType -type Client = Awaited> +type Client = EncryptionClientFor type Ops = ReturnType type ComparisonOperator = 'gt' | 'gte' | 'lt' | 'lte' diff --git a/packages/stack/__tests__/audit.test.ts b/packages/stack/__tests__/audit.test.ts index bb14447bd..edf2b1df2 100644 --- a/packages/stack/__tests__/audit.test.ts +++ b/packages/stack/__tests__/audit.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { LockContext } from '@/identity' import { Encryption } from '@/index' import { encryptedColumn, encryptedTable } from '@/schema' @@ -20,7 +21,7 @@ type User = { number?: number } -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/backward-compat.test.ts b/packages/stack/__tests__/backward-compat.test.ts index 915108cc7..cdf5c930c 100644 --- a/packages/stack/__tests__/backward-compat.test.ts +++ b/packages/stack/__tests__/backward-compat.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { Encryption } from '@/index' import { encryptedColumn, encryptedTable } from '@/schema' @@ -8,7 +9,7 @@ const users = encryptedTable('users', { }) describe('k-field discriminator (EQL v2.3)', () => { - let protectClient: Awaited> + let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ schemas: [users] }) diff --git a/packages/stack/__tests__/basic-protect.test.ts b/packages/stack/__tests__/basic-protect.test.ts index a889328b0..af687818c 100644 --- a/packages/stack/__tests__/basic-protect.test.ts +++ b/packages/stack/__tests__/basic-protect.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { Encryption } from '@/index' import { encryptedColumn, encryptedTable } from '@/schema' @@ -9,7 +10,7 @@ const users = encryptedTable('users', { json: encryptedColumn('json').dataType('json'), }) -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/bulk-protect.test.ts b/packages/stack/__tests__/bulk-protect.test.ts index 45342f91b..10d615c7c 100644 --- a/packages/stack/__tests__/bulk-protect.test.ts +++ b/packages/stack/__tests__/bulk-protect.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { LockContext } from '@/identity' import { Encryption } from '@/index' import { encryptedColumn, encryptedTable } from '@/schema' @@ -19,7 +20,7 @@ type User = { number?: number } -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts index 3953d0a8b..f69519e81 100644 --- a/packages/stack/__tests__/decrypt-audit-forwarding.test.ts +++ b/packages/stack/__tests__/decrypt-audit-forwarding.test.ts @@ -40,6 +40,7 @@ vi.mock('@cipherstash/protect-ffi', () => ({ })) import * as ffi from '@cipherstash/protect-ffi' +import type { EncryptionClientFor } from '@/encryption/v3' // Imported after the mock so the v3 table builder is available; `Encryption` // returns the typed client for an all-v3 schema set. import { encryptedTable, types } from '@/encryption/v3' @@ -66,7 +67,7 @@ const lastDecryptOpts = () => (ffi.decryptBulk as any).mock.calls.at(-1)[1] const lastCiphertextLockContext = () => lastDecryptOpts().ciphertexts[0]?.lockContext -let client: Awaited>> +let client: EncryptionClientFor let prevWorkspaceCrn: string | undefined beforeEach(async () => { diff --git a/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts b/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts index 4f1f42096..1e2b5fd94 100644 --- a/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts +++ b/packages/stack/__tests__/dynamodb/encrypted-dynamodb-v3.test.ts @@ -21,9 +21,8 @@ import { beforeAll, describe, expect, it } from 'vitest' import { encryptedDynamoDB } from '@/dynamodb' import { toItemWithEqlPayloads } from '@/dynamodb/helpers' import type { EncryptedDynamoDBInstance } from '@/dynamodb/types' -import type { EncryptionClient } from '@/encryption' -import { EncryptionV3 } from '@/encryption/v3' -import type { JsonValue } from '@/eql/v3' +import { type EncryptionClientFor, EncryptionV3 } from '@/encryption/v3' +import type { AnyV3Table, JsonValue } from '@/eql/v3' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' @@ -66,7 +65,7 @@ type User = { /** The typed client from `EncryptionV3` — the documented v3 entry point. */ let typedDynamo: EncryptedDynamoDBInstance /** The nominal chainable client, forced into v3 mode. */ -let nominalClient: EncryptionClient +let nominalClient: EncryptionClientFor let nominalDynamo: EncryptedDynamoDBInstance beforeAll(async () => { @@ -362,7 +361,7 @@ describe('nested attributes with a v3 table', liveSuiteOptions, () => { } let nestedDynamo: EncryptedDynamoDBInstance - let nestedClient: EncryptionClient + let nestedClient: EncryptionClientFor beforeAll(async () => { nestedClient = await Encryption({ @@ -432,8 +431,8 @@ describe('nested attributes with a v3 table', liveSuiteOptions, () => { if (stored.failure) throw new Error(stored.failure.message) const term = await nestedClient.encryptQuery(ssn, { - table: nested as never, - column: nested['profile.ssn'] as never, + table: nested, + column: nested['profile.ssn'], }) if (term.failure) throw new Error(term.failure.message) @@ -477,7 +476,7 @@ describe( }) let renamedDynamo: EncryptedDynamoDBInstance - let renamedClient: EncryptionClient + let renamedClient: EncryptionClientFor beforeAll(async () => { renamedClient = await Encryption({ @@ -517,8 +516,8 @@ describe( expect(decrypted.data).toEqual(original) const term = await renamedClient.encryptQuery('c@d.com', { - table: renamed as never, - column: renamed.emailAddress as never, + table: renamed, + column: renamed.emailAddress, }) if (term.failure) throw new Error(term.failure.message) @@ -580,8 +579,8 @@ describe( if (stored.failure) throw new Error(stored.failure.message) const term = await nominalClient.encryptQuery(email, { - table: users as never, - column: users.email as never, + table: users, + column: users.email, }) if (term.failure) throw new Error(term.failure.message) diff --git a/packages/stack/__tests__/encrypt-lock-context-guards.test.ts b/packages/stack/__tests__/encrypt-lock-context-guards.test.ts index 981845eb6..2162b359a 100644 --- a/packages/stack/__tests__/encrypt-lock-context-guards.test.ts +++ b/packages/stack/__tests__/encrypt-lock-context-guards.test.ts @@ -19,6 +19,8 @@ */ import { beforeEach, describe, expect, it, vi } from 'vitest' +import type { EncryptionClient } from '@/encryption' +import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' import { LockContext } from '@/identity' import { Encryption } from '@/index' @@ -53,8 +55,8 @@ const usersV3 = encryptedTableV3('users_v3', { // biome-ignore lint/suspicious/noExplicitAny: test helper reads the Result union const failure = (result: any) => result.failure -let clientV2: Awaited> -let clientV3: Awaited> +let clientV2: EncryptionClient +let clientV3: EncryptionClientFor beforeEach(async () => { vi.clearAllMocks() @@ -66,8 +68,13 @@ beforeEach(async () => { clientV3 = await Encryption({ schemas: [usersV3] }) }) -const clientFor = (variant: 'v2' | 'v3') => - variant === 'v2' ? clientV2 : clientV3 +// The two clients have different `encrypt` signatures — `clientV3` pins the +// plaintext to each column's domain — so the shared `describe.each` call below +// cannot resolve against their union. The guards under test are runtime value +// checks that predate both surfaces and behave identically on each, so the +// dispatch (and only the dispatch) erases to the nominal shape. +const clientFor = (variant: 'v2' | 'v3'): EncryptionClient => + variant === 'v2' ? clientV2 : (clientV3 as unknown as EncryptionClient) describe.each([ ['v2', { column: users.score, table: users }], diff --git a/packages/stack/__tests__/encrypt-query-searchable-json.test.ts b/packages/stack/__tests__/encrypt-query-searchable-json.test.ts index bc99fcb4f..d65e791ed 100644 --- a/packages/stack/__tests__/encrypt-query-searchable-json.test.ts +++ b/packages/stack/__tests__/encrypt-query-searchable-json.test.ts @@ -1,11 +1,10 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' import { createMockLockContext, expectFailure, unwrapResult } from './fixtures' -type EncryptionClient = Awaited> - const documents = encryptedTable('documents', { metadata: types.Json('metadata'), }) @@ -21,10 +20,19 @@ function expectContainment(value: unknown): void { } describe('encryptQuery with searchableJson', () => { + // NOTE: this suite holds the client through the NOMINAL surface on purpose. + // The typed client derives `encryptQuery`'s plaintext from the column's domain, + // so every query type on a `types.Json()` column is typed `JsonDocument` — but + // the searchable-JSON query types take a JSONPath string, a `{ path, value }` + // pair, or a bare scalar. This file exercises exactly those, so typing it + // against the typed client would mean casting away the argument type at every + // call and hiding the gap. Cast once, here, where it is visible and explained. let client: EncryptionClient beforeAll(async () => { - client = await Encryption({ schemas: [documents] }) + client = (await Encryption({ + schemas: [documents], + })) as unknown as EncryptionClient }) it('infers a selector hash for string plaintext', async () => { diff --git a/packages/stack/__tests__/encrypt-query-stevec.test.ts b/packages/stack/__tests__/encrypt-query-stevec.test.ts index 54d9f1396..894592878 100644 --- a/packages/stack/__tests__/encrypt-query-stevec.test.ts +++ b/packages/stack/__tests__/encrypt-query-stevec.test.ts @@ -1,11 +1,10 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' import { expectFailure, unwrapResult } from './fixtures' -type EncryptionClient = Awaited> - const documents = encryptedTable('documents', { metadata: types.Json('metadata'), }) @@ -27,10 +26,20 @@ function expectOrderingTerm(value: unknown): void { } describe('encryptQuery with protect-ffi 0.30 SteVec operations', () => { + // NOTE: this suite holds the client through the NOMINAL surface on purpose. + // The typed client derives `encryptQuery`'s plaintext from the column's domain, + // so every query type on a `types.Json()` column is typed `JsonDocument` — but + // the SteVec query types take a JSONPath string (`steVecSelector`), a + // `{ path, value }` pair (`steVecValueSelector`) or a bare scalar + // (`steVecTerm`). This file exercises exactly those, so typing it against the + // typed client would mean casting away the argument type at every call and + // hiding the gap. Cast once, here, where it is visible and explained. let client: EncryptionClient beforeAll(async () => { - client = await Encryption({ schemas: [documents, plain] }) + client = (await Encryption({ + schemas: [documents, plain], + })) as unknown as EncryptionClient }) it('returns a bare selector hash for a JSONPath', async () => { diff --git a/packages/stack/__tests__/json-protect.test.ts b/packages/stack/__tests__/json-protect.test.ts index ea55f4213..34e3eb2b1 100644 --- a/packages/stack/__tests__/json-protect.test.ts +++ b/packages/stack/__tests__/json-protect.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { LockContext } from '@/identity' import { Encryption } from '@/index' import { encryptedColumn, encryptedField, encryptedTable } from '@/schema' @@ -33,7 +34,7 @@ type User = { } } -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/lock-context-wiring.test.ts b/packages/stack/__tests__/lock-context-wiring.test.ts index cf4b967cb..9a578c014 100644 --- a/packages/stack/__tests__/lock-context-wiring.test.ts +++ b/packages/stack/__tests__/lock-context-wiring.test.ts @@ -44,6 +44,7 @@ vi.mock('@cipherstash/protect-ffi', () => ({ })) import * as ffi from '@cipherstash/protect-ffi' +import type { EncryptionClient } from '@/encryption' const users = encryptedTable('users', { email: encryptedColumn('email').equality(), @@ -78,7 +79,7 @@ function unwrap(result: any) { // biome-ignore lint/suspicious/noExplicitAny: reading recorded mock args const lastOpts = (fn: any) => fn.mock.calls.at(-1)[1] -let client: Awaited> +let client: EncryptionClient beforeEach(async () => { vi.clearAllMocks() diff --git a/packages/stack/__tests__/number-protect.test.ts b/packages/stack/__tests__/number-protect.test.ts index 82222377a..8bc084982 100644 --- a/packages/stack/__tests__/number-protect.test.ts +++ b/packages/stack/__tests__/number-protect.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it, test } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { LockContext } from '@/identity' import { Encryption } from '@/index' import { encryptedColumn, encryptedField, encryptedTable } from '@/schema' @@ -29,7 +30,7 @@ type User = { } } -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/protect-ops.test.ts b/packages/stack/__tests__/protect-ops.test.ts index df9c40da4..c92f4595e 100644 --- a/packages/stack/__tests__/protect-ops.test.ts +++ b/packages/stack/__tests__/protect-ops.test.ts @@ -1,5 +1,6 @@ import 'dotenv/config' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClient } from '@/encryption' import { LockContext } from '@/identity' import { Encryption } from '@/index' import { encryptedColumn, encryptedTable } from '@/schema' @@ -18,7 +19,7 @@ type User = { number?: number } -let protectClient: Awaited> +let protectClient: EncryptionClient beforeAll(async () => { protectClient = await Encryption({ diff --git a/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts b/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts index e7456b947..0024902d6 100644 --- a/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts +++ b/packages/stack/__tests__/v3-matrix/matrix-lock-context.test.ts @@ -37,6 +37,7 @@ vi.mock('@cipherstash/protect-ffi', () => ({ })) import * as ffi from '@cipherstash/protect-ffi' +import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable, types } from '@/encryption/v3' const users = encryptedTable('users', { @@ -70,7 +71,7 @@ const lastOpts = (fn: any) => fn.mock.calls.at(-1)[1] // `Encryption` returns the typed client directly for an all-v3 schema set (the // collapse of `EncryptionV3`), so there is no separate `typedClient` wrap here. -let typed: Awaited>> +let typed: EncryptionClientFor let prevWorkspaceCrn: string | undefined beforeEach(async () => { diff --git a/packages/stack/integration/identity/matrix-identity.integration.test.ts b/packages/stack/integration/identity/matrix-identity.integration.test.ts index 5009a4a74..6f4bfa709 100644 --- a/packages/stack/integration/identity/matrix-identity.integration.test.ts +++ b/packages/stack/integration/identity/matrix-identity.integration.test.ts @@ -18,6 +18,7 @@ import { OidcFederationStrategy } from '@cipherstash/auth' import { unwrapResult } from '@cipherstash/test-kit' import { clerkJwtProvider } from '@cipherstash/test-kit/integration-clerk' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClientFor } from '@/encryption/v3' import { EncryptionV3, encryptedTable, types } from '@/encryption/v3' const users = encryptedTable('v3_identity_live_users', { @@ -35,7 +36,7 @@ const INFRA_FAULT = /ECONNREFUSED|ECONNRESET|ETIMEDOUT|ENOTFOUND|EAI_AGAIN|socket hang up|timed? ?out|network error/i describe('v3 typed client identity-aware operations (live)', () => { - let client: Awaited>> + let client: EncryptionClientFor beforeAll(async () => { const crn = process.env.CS_WORKSPACE_CRN @@ -114,7 +115,7 @@ describe('v3 typed client identity-aware operations (live)', () => { // test on a Clerk outage or a rotated `CLERK_MACHINE_TOKEN` — for behaviour it // never exercises. describe('v3 typed client audit metadata (live)', () => { - let client: Awaited>> + let client: EncryptionClientFor beforeAll(async () => { client = await EncryptionV3({ schemas: [users] }) diff --git a/packages/stack/integration/shared/json-crypto.integration.test.ts b/packages/stack/integration/shared/json-crypto.integration.test.ts index dd66d412b..0a54d708a 100644 --- a/packages/stack/integration/shared/json-crypto.integration.test.ts +++ b/packages/stack/integration/shared/json-crypto.integration.test.ts @@ -7,6 +7,7 @@ */ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClientFor } from '@/encryption/v3' import { EncryptionV3, encryptedTable, types } from '@/encryption/v3' const docs = encryptedTable('v3_json_docs', { @@ -14,7 +15,7 @@ const docs = encryptedTable('v3_json_docs', { }) describe('v3 typed client — encrypted JSONB round-trip', () => { - let client: Awaited>> + let client: EncryptionClientFor beforeAll(async () => { client = await EncryptionV3({ schemas: [docs] }) diff --git a/packages/stack/integration/shared/match-bloom.integration.test.ts b/packages/stack/integration/shared/match-bloom.integration.test.ts index 95fb70115..7b2543627 100644 --- a/packages/stack/integration/shared/match-bloom.integration.test.ts +++ b/packages/stack/integration/shared/match-bloom.integration.test.ts @@ -1,5 +1,7 @@ import { expect, it } from 'vitest' +import type { EncryptionClientFor } from '@/encryption/v3' import { EncryptionV3 } from '@/encryption/v3' +import type { AnyV3Table } from '@/eql/v3' import { encryptedTable, types } from '@/eql/v3' /** @@ -36,7 +38,7 @@ const docs = encryptedTable('match_bloom_probe', { const HAYSTACK = 'ada@example.com' async function bloomOf( - client: Awaited>, + client: EncryptionClientFor, value: string, ) { const result = await client.encrypt(value, { column: docs.bio, table: docs }) diff --git a/packages/stack/integration/shared/matrix-bulk.integration.test.ts b/packages/stack/integration/shared/matrix-bulk.integration.test.ts index 0ef71beea..e9123ae54 100644 --- a/packages/stack/integration/shared/matrix-bulk.integration.test.ts +++ b/packages/stack/integration/shared/matrix-bulk.integration.test.ts @@ -7,6 +7,7 @@ */ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' +import type { EncryptionClientFor } from '@/encryption/v3' import { EncryptionV3, encryptedTable, types } from '@/encryption/v3' const people = encryptedTable('v3_bulk_people', { @@ -15,7 +16,7 @@ const people = encryptedTable('v3_bulk_people', { }) describe('v3 typed client bulk-at-scale (live)', () => { - let client: Awaited>> + let client: EncryptionClientFor beforeAll(async () => { client = await EncryptionV3({ schemas: [people] }) diff --git a/packages/stack/integration/shared/matrix-crypto.integration.test.ts b/packages/stack/integration/shared/matrix-crypto.integration.test.ts index 03d9fcd0d..8358d4c2d 100644 --- a/packages/stack/integration/shared/matrix-crypto.integration.test.ts +++ b/packages/stack/integration/shared/matrix-crypto.integration.test.ts @@ -26,7 +26,12 @@ import { V3_MATRIX, } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' -import { EncryptionV3, encryptedTable } from '@/encryption/v3' +import { + type EncryptionClientFor, + EncryptionV3, + encryptedTable, +} from '@/encryption/v3' +import type { AnyV3Table } from '@/eql/v3' // `as const satisfies Record<...>` gives `V3_MATRIX` a narrower type than // `Record` (rows that omit the optional @@ -70,7 +75,7 @@ const errorCases = domains.flatMap(([t, spec]) => ) describe('v3 matrix live round-trip (all domains × samples)', () => { - let client: Awaited> + let client: EncryptionClientFor let encrypted: Array> let decrypted: Array> diff --git a/packages/stack/integration/shared/matrix-sql.integration.test.ts b/packages/stack/integration/shared/matrix-sql.integration.test.ts index 3532cdcbc..2fb113887 100644 --- a/packages/stack/integration/shared/matrix-sql.integration.test.ts +++ b/packages/stack/integration/shared/matrix-sql.integration.test.ts @@ -67,7 +67,12 @@ import { } from '@cipherstash/test-kit' import postgres from 'postgres' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { EncryptionV3, encryptedTable } from '@/encryption/v3' +import { + type EncryptionClientFor, + EncryptionV3, + encryptedTable, +} from '@/encryption/v3' +import type { AnyV3Table } from '@/eql/v3' // Previously force-skipped (CI run 28569708268, PR #540): `beforeAll` crashed // with `PostgresError: invalid input syntax for type json` on the dynamic @@ -203,7 +208,7 @@ const comparePlaintext = (a: unknown, b: unknown): number => { type Row = { id: number } -let client: Awaited> +let client: EncryptionClientFor let idA: number let idB: number // Separate run id for the multi-row ordering rows. Kept DISTINCT from @@ -399,7 +404,9 @@ describe('v3 matrix live Postgres coverage (all covered domains)', () => { const col = slug(eqlType) const column = (table as unknown as Record)[col] as never const encrypted = unwrapResult( - await client.encrypt('', { + // The matrix table is built from a dynamic column map, so `column` is + // already erased above and the plaintext cannot be derived from it. + await client.encrypt('' as never, { table: table as never, column, }), @@ -416,7 +423,9 @@ describe('v3 matrix live Postgres coverage (all covered domains)', () => { const col = slug(eqlType) const column = (table as unknown as Record)[col] as never const encrypted = unwrapResult( - await client.encrypt('', { + // The matrix table is built from a dynamic column map, so `column` is + // already erased above and the plaintext cannot be derived from it. + await client.encrypt('' as never, { table: table as never, column, }), diff --git a/packages/stack/integration/shared/schema-pg.integration.test.ts b/packages/stack/integration/shared/schema-pg.integration.test.ts index a8c8992de..152b94801 100644 --- a/packages/stack/integration/shared/schema-pg.integration.test.ts +++ b/packages/stack/integration/shared/schema-pg.integration.test.ts @@ -1,7 +1,7 @@ import { databaseUrl, unwrapResult } from '@cipherstash/test-kit' import postgres from 'postgres' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' -import type { EncryptionClient } from '@/encryption' +import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' import type { Encrypted } from '@/types' @@ -28,7 +28,9 @@ type InsertedRow = { type EncryptionPayload = postgres.JSONValue -let protectClient: EncryptionClient +let protectClient: EncryptionClientFor< + readonly [typeof table, typeof typedTable] +> async function encryptValue(value: string): Promise { return unwrapResult( @@ -48,7 +50,7 @@ async function encryptValue(value: string): Promise { // term, so which SQL function you call selects which term is compared. async function encryptOperand( value: unknown, - opts: Parameters[1], + opts: Parameters[1], ): Promise { return unwrapResult( await protectClient.encrypt(value as never, opts), diff --git a/packages/stack/integration/shared/schema-v3-client.integration.test.ts b/packages/stack/integration/shared/schema-v3-client.integration.test.ts index a3b31b3a5..80e60e647 100644 --- a/packages/stack/integration/shared/schema-v3-client.integration.test.ts +++ b/packages/stack/integration/shared/schema-v3-client.integration.test.ts @@ -1,6 +1,6 @@ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' -import type { EncryptionClient } from '@/encryption' +import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable, types } from '@/eql/v3' import { Encryption } from '@/index' @@ -19,7 +19,7 @@ const users = encryptedTable('schema_v3_client_users', { }) describe('eql_v3 client integration', () => { - let protectClient: EncryptionClient + let protectClient: EncryptionClientFor beforeAll(async () => { protectClient = await Encryption({ schemas: [users] }) From ff93a62c54d5123999946c751f62a582da191767 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:36:55 +1000 Subject: [PATCH 059/123] ci: typecheck the surfaces that compiled nowhere, and cache dist (C-1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four packages had a `tsconfig.json` covering their src and tests that no CI step ever ran, and were already clean — gate them before they drift: migrate, nextjs, examples/prisma and e2e. `packages/nextjs` needed two fixes first: `vi.Mock` was used as a TYPE, which needs `import type { Mock }`. Also declare `outputs: ["dist/**"]` on turbo's `build`. It declared none, so a cache hit replayed logs and restored no files — and the examples/basic gate added by 5fab1cf6 typechecks against `packages/stack/dist/*.d.ts`. It held only because fresh CI runners have no cache. Verified by deleting packages/stack/dist and re-running the gate: cache hit, dist restored, still green. Deliberately NOT gated, with counts recorded in the workflow so the gap is visible rather than silent: @cipherstash/stack (147), stack-drizzle (63), stash (21), stack-supabase (11). Their `test:types` scripts are scoped to `__tests__/**/*.test-d.ts`, so src, the runtime suites and integration/** compile nowhere. Most of the drizzle and supabase count is a single root cause in @cipherstash/test-kit's V3_MATRIX. --- .github/workflows/tests.yml | 25 ++++++++++++++++++++++++ e2e/package.json | 3 ++- packages/migrate/package.json | 1 + packages/nextjs/__tests__/nextjs.test.ts | 6 +++--- packages/nextjs/package.json | 1 + turbo.json | 1 + 6 files changed, 33 insertions(+), 4 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 3a72ef269..95f18246a 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -160,6 +160,31 @@ jobs: - name: Typecheck (examples/basic — guards the v3 stack/stack-drizzle importers) run: pnpm exec turbo run typecheck --filter @cipherstash/basic-example + # The rest of the surfaces that were compiling nowhere. Each of these has + # a `tsconfig.json` that covers its `src` and tests, and each was already + # clean — so they are gated now, before they drift. They resolve their + # workspace dependencies through `dist/*.d.ts`, hence the turbo filters + # (`^build` builds the dependencies first). + # + # NOT gated yet, and deliberately: `@cipherstash/stack` (147 errors under + # its own tsconfig), `@cipherstash/stack-drizzle` (63), `stash` (21) and + # `@cipherstash/stack-supabase` (11). Their `test:types` scripts only + # cover `__tests__/**/*.test-d.ts`, so `src`, the runtime test suites and + # `integration/**` compile nowhere. Most of the drizzle and supabase count + # is one root cause — `V3_MATRIX`'s `indexes` union in + # `@cipherstash/test-kit` — see #778. + - name: Typecheck (migrate) + run: pnpm exec turbo run typecheck --filter @cipherstash/migrate + + - name: Typecheck (nextjs) + run: pnpm exec turbo run typecheck --filter @cipherstash/nextjs + + - name: Typecheck (examples/prisma — guards the prisma-next importers) + run: pnpm exec turbo run typecheck --filter @cipherstash/prisma-next-example + + - name: Typecheck (e2e) + run: pnpm exec turbo run typecheck --filter @cipherstash/e2e + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/e2e/package.json b/e2e/package.json index 06cf30efb..dd10ac340 100644 --- a/e2e/package.json +++ b/e2e/package.json @@ -5,7 +5,8 @@ "description": "End-to-end tests that exercise built CipherStash binaries and cross-package behaviour.", "type": "module", "scripts": { - "test:e2e": "vitest run" + "test:e2e": "vitest run", + "typecheck": "tsc --noEmit -p tsconfig.json" }, "dependencies": { "stash": "workspace:*", diff --git a/packages/migrate/package.json b/packages/migrate/package.json index b14343caa..c96f2f9db 100644 --- a/packages/migrate/package.json +++ b/packages/migrate/package.json @@ -42,6 +42,7 @@ }, "scripts": { "build": "tsup", + "typecheck": "tsc --noEmit -p tsconfig.json", "dev": "tsup --watch", "test": "vitest run", "lint": "biome check ." diff --git a/packages/nextjs/__tests__/nextjs.test.ts b/packages/nextjs/__tests__/nextjs.test.ts index c15f05b52..e03df244c 100644 --- a/packages/nextjs/__tests__/nextjs.test.ts +++ b/packages/nextjs/__tests__/nextjs.test.ts @@ -1,6 +1,6 @@ import { type NextRequest, NextResponse } from 'next/server' // cts.test.ts -import { afterEach, describe, expect, it, vi } from 'vitest' +import { afterEach, describe, expect, it, type Mock, vi } from 'vitest' // --------------------------------------------- // 1) Mock next/headers before importing it @@ -65,7 +65,7 @@ describe('getCtsToken', () => { accessToken: 'fake_token', expiry: 999999, } - ;(cookies as unknown as vi.Mock).mockReturnValueOnce({ + ;(cookies as unknown as Mock).mockReturnValueOnce({ get: vi.fn().mockReturnValue({ value: JSON.stringify(mockCookieValue) }), }) @@ -78,7 +78,7 @@ describe('getCtsToken', () => { }) it('should return null if the cookie is not present', async () => { - ;(cookies as unknown as vi.Mock).mockReturnValueOnce({ + ;(cookies as unknown as Mock).mockReturnValueOnce({ get: vi.fn().mockReturnValue(undefined), }) diff --git a/packages/nextjs/package.json b/packages/nextjs/package.json index 2d7deebad..f4c512f65 100644 --- a/packages/nextjs/package.json +++ b/packages/nextjs/package.json @@ -33,6 +33,7 @@ }, "scripts": { "build": "tsup", + "typecheck": "tsc --noEmit -p tsconfig.json", "dev": "tsup --watch", "release": "tsup" }, diff --git a/turbo.json b/turbo.json index 6ed7b4a76..aa5ff0a07 100644 --- a/turbo.json +++ b/turbo.json @@ -3,6 +3,7 @@ "tasks": { "build": { "dependsOn": ["^build"], + "outputs": ["dist/**"], "inputs": ["$TURBO_DEFAULT$", ".env*"], "env": ["STASH_POSTHOG_KEY"] }, From 310bb19c0ebb9195381f0f6a1766d97fa7a128be Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:39:31 +1000 Subject: [PATCH 060/123] docs(changesets,skills,prisma-next): record the signature changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Two new changesets: the schema-array widening (stack minor, prisma-next patch) and the typed-client init parity (stack patch). - stack-audit-on-decrypt.md dropped the paragraph telling callers to narrow AnyV3Table[] to readonly [AnyV3Table, ...AnyV3Table[]]. That advice is obsolete, and it never worked for ReadonlyArray — the type prisma-next exposes publicly — since readonly also failed the old tuple constraint. - prisma-next drops its destructure-and-respread workaround. Its comment claimed 'Encryption requires a non-empty schema tuple', which is no longer true; the runtime emptiness check and its error message stay. - skills/stash-encryption documents EncryptionClientFor as the way to name the client, and that schemas need not be an array literal. --- .changeset/encryption-schema-arrays.md | 36 +++++++++++++++++++ .changeset/skills-encryption-client-naming.md | 8 +++++ .changeset/stack-audit-on-decrypt.md | 10 +++--- .changeset/typed-client-init-parity.md | 25 +++++++++++++ .../prisma-next/src/stack/from-stack-v3.ts | 22 ++++-------- skills/stash-encryption/SKILL.md | 17 +++++++++ 6 files changed, 96 insertions(+), 22 deletions(-) create mode 100644 .changeset/encryption-schema-arrays.md create mode 100644 .changeset/skills-encryption-client-naming.md create mode 100644 .changeset/typed-client-init-parity.md diff --git a/.changeset/encryption-schema-arrays.md b/.changeset/encryption-schema-arrays.md new file mode 100644 index 000000000..2fa9082bb --- /dev/null +++ b/.changeset/encryption-schema-arrays.md @@ -0,0 +1,36 @@ +--- +'@cipherstash/stack': minor +'@cipherstash/prisma-next': patch +--- + +`Encryption({ schemas })` now accepts any non-empty array of EQL v3 tables, not +only an array literal. + +Previously the v3 signature required a non-empty *tuple*, so every indirect form +was rejected at compile time even though it worked at runtime: + +```ts +// all of these used to fail with TS2769 +export const schemas: AnyV3Table[] = [users, orders] +await Encryption({ schemas }) + +const readonlySchemas: ReadonlyArray = [users, orders] +await Encryption({ schemas: readonlySchemas }) + +const built: AnyV3Table[] = [] +built.push(users) +await Encryption({ schemas: built }) +``` + +They all compile now. `Encryption({ schemas: [] })` is still a compile error, and +an array literal still gets full per-column typing — passing the wrong plaintext +for a column's domain, or a table the client was not built with, is still +rejected. + +`EncryptionClientFor` is widened to match, so it names the typed client for a +loose `readonly AnyV3Table[]` as well as for a tuple. Code that is generic over +its schemas — an adapter that builds a table per request, say — can now write +`EncryptionClientFor` and keep the typed surface. + +If you narrowed a schema array to `readonly [AnyV3Table, ...AnyV3Table[]]` to +satisfy the old signature, that narrowing is no longer needed. diff --git a/.changeset/skills-encryption-client-naming.md b/.changeset/skills-encryption-client-naming.md new file mode 100644 index 000000000..4960a2212 --- /dev/null +++ b/.changeset/skills-encryption-client-naming.md @@ -0,0 +1,8 @@ +--- +'stash': patch +--- + +`skills/stash-encryption` now documents how to name the client's type +(`EncryptionClientFor`, not `Awaited>`, which +always resolves to the untyped nominal client) and states that `schemas` accepts +any non-empty array of v3 tables rather than only an array literal. diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md index de4b34cdf..a50a1f0cb 100644 --- a/.changeset/stack-audit-on-decrypt.md +++ b/.changeset/stack-audit-on-decrypt.md @@ -34,12 +34,10 @@ return type changes, and the two-argument form reconstructs `Date` columns from `cast_as` instead of leaving them as ISO strings. Code that read those columns as strings needs updating. -The v3 overload takes a non-empty tuple of tables and a `V3ClientConfig` — -`ClientConfig` without the deprecated `eqlVersion` escape hatch. So -`Encryption({ schemas: [] })` no longer type-checks (it used to compile and then -throw), and `config: { eqlVersion: 2 }` selects the nominal overload, which is -the client you actually get back. Callers passing a plain `AnyV3Table[]` rather -than an array literal must narrow it to `readonly [AnyV3Table, ...AnyV3Table[]]`. +The v3 overload takes a `V3ClientConfig` — `ClientConfig` without the deprecated +`eqlVersion` escape hatch. So `Encryption({ schemas: [] })` no longer type-checks +(it used to compile and then throw), and `config: { eqlVersion: 2 }` selects the +nominal overload, which is the client you actually get back. `Awaited>` names the nominal client whatever you pass, because `ReturnType` reads the last overload; use the exported `EncryptionClientFor` to name the client for a schema tuple. diff --git a/.changeset/typed-client-init-parity.md b/.changeset/typed-client-init-parity.md new file mode 100644 index 000000000..7f835a6fa --- /dev/null +++ b/.changeset/typed-client-init-parity.md @@ -0,0 +1,25 @@ +--- +'@cipherstash/stack': patch +--- + +The typed EQL v3 client no longer throws `TypeError: init is not a function`. + +`Encryption` selects its overload from the `config` argument but selects the +client it actually returns by inspecting the `schemas`, so the two can disagree. +Hoisting a config into a `ClientConfig`-typed variable is enough to split them — +`ClientConfig.eqlVersion` is `2 | 3`, which the v3 overload's `eqlVersion?: 3` +rejects — so the call types as the nominal `EncryptionClient` while the runtime +still returns the typed client: + +```ts +const config: ClientConfig = { keyset } +const client = await Encryption({ schemas: [users], config }) +// type: EncryptionClient · runtime: TypedEncryptionClient +``` + +`init` was the only member of `EncryptionClient` missing from the typed client, +so any call through the declared type crashed. It is now delegated. + +The type still under-reports in this case: you lose the typed surface with no +diagnostic. Pass the config inline, or type the variable as `V3ClientConfig`, to +keep it. diff --git a/packages/prisma-next/src/stack/from-stack-v3.ts b/packages/prisma-next/src/stack/from-stack-v3.ts index 2f569b550..bcc012cfc 100644 --- a/packages/prisma-next/src/stack/from-stack-v3.ts +++ b/packages/prisma-next/src/stack/from-stack-v3.ts @@ -89,11 +89,8 @@ export async function cipherstashFromStack( ) } - // Destructured rather than length-checked so the non-emptiness survives into - // the type layer: `Encryption` requires a non-empty schema tuple (an empty one - // used to type-check and then throw at runtime). - const [firstDerived, ...restDerived] = deriveStackSchemasV3(opts.contractJson) - if (firstDerived === undefined) { + const derived = deriveStackSchemasV3(opts.contractJson) + if (derived.length === 0) { throw new Error( 'cipherstashFromStack: no v3 cipherstash columns found in contract.json. ' + 'Declare at least one v3 `cipherstash.*()` column (e.g. `cipherstash.TextSearch()`) in prisma/schema.prisma ' + @@ -101,10 +98,7 @@ export async function cipherstashFromStack( ) } - const schemas = resolveV3Schemas( - [firstDerived, ...restDerived], - opts.schemasV3, - ) + const schemas = resolveV3Schemas(derived, opts.schemasV3) const encryptionClient = await EncryptionV3({ schemas, @@ -144,18 +138,15 @@ function collectNonV3CipherstashCodecIds( return [...ids].sort() } -/** A schema list carrying its non-emptiness in the type, as `Encryption` wants. */ -type NonEmptyV3Schemas = readonly [AnyV3Table, ...AnyV3Table[]] - /** * Validate contract-declared tables against their overrides (exact * domain identity) and append override-only tables — same merge * semantics as the v2 `resolveSchemas`. */ function resolveV3Schemas( - derived: NonEmptyV3Schemas, + derived: ReadonlyArray, override: ReadonlyArray | undefined, -): NonEmptyV3Schemas { +): ReadonlyArray { if (override === undefined || override.length === 0) return derived const derivedByName = new Map(derived.map((t) => [t.tableName, t])) @@ -168,8 +159,7 @@ function resolveV3Schemas( } return [ - derived[0], - ...derived.slice(1), + ...derived, ...override.filter((t) => !derivedByName.has(t.tableName)), ] } diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 3c780c1cb..1ddab9b84 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -312,6 +312,23 @@ const client = await Encryption({ schemas: [users] }) - `EncryptionV3` (from `@cipherstash/stack/v3`) is a **deprecated, type-identical alias** of `Encryption`, kept for backwards compatibility. New code should use `Encryption`. - `typedClient(client, ...schemas)` (exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface, if you built one via a lower-level path. - **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. +- `schemas` takes any non-empty array of v3 tables — a shared `export const schemas: AnyV3Table[]`, a `ReadonlyArray`, one built at runtime. It does not have to be an array literal. An empty array is a compile error. +- **To name the client's type, use `EncryptionClientFor`** (from `@cipherstash/stack/v3`), not `Awaited>`. `Encryption` is overloaded and `ReturnType` reads the last overload, so that idiom always resolves to the untyped nominal client: + +```typescript +import { Encryption, type EncryptionClientFor, encryptedTable, types } from "@cipherstash/stack/v3" +import type { AnyV3Table } from "@cipherstash/stack/eql/v3" + +const users = encryptedTable("users", { email: types.TextSearch("email") }) + +// A named schema tuple keeps per-column typing. +let client: EncryptionClientFor +client = await Encryption({ schemas: [users] }) + +// Code that is generic over its schemas keeps the typed surface too. +function withClient(c: EncryptionClientFor) { /* … */ } +``` + ```typescript // Error handling From 9aa020e2b98649d06f842498c65359f949d3f390 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 21:41:49 +1000 Subject: [PATCH 061/123] ci: exempt bench from the dist outputs cache key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `packages/bench`'s build is `tsc --noEmit` — a typecheck, not a bundle — so the repo-wide `outputs: ["dist/**"]` added alongside it warned 'no output files found' on every run. --- turbo.json | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/turbo.json b/turbo.json index aa5ff0a07..f7c6b2dd0 100644 --- a/turbo.json +++ b/turbo.json @@ -20,6 +20,12 @@ "inputs": ["$TURBO_DEFAULT$", ".env*"], "cache": false }, + // `packages/bench`'s "build" is `tsc --noEmit` — a typecheck, not a bundle — + // so it emits nothing and the repo-wide `dist/**` would warn on every run. + "@cipherstash/bench#build": { + "dependsOn": ["^build"], + "outputs": [] + }, "typecheck": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$"] From d84ab1bcb75f86efd33d4bb3ddf58f1d112da01c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:07:06 +1000 Subject: [PATCH 062/123] fix(cli,skills,stack-supabase): correct v2 cutover read guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit review of this branch. Two guidance defects that would have shipped into customer repos: - `skills/stash-encryption` claimed reads of the promoted column return "decrypted ciphertext transparently" after `stash encrypt cutover`. Only CipherStash Proxy decrypts on the wire; SDK/ORM reads still return ciphertext, which is what the very next table row says. An agent following the table would return raw `eql_v2_encrypted` composites to end users. - `stash init`'s setup prompt told agents to declare an EQL v2 cutover target with a `types.*` domain. `types.*` is the v3 domain family; a v2 column is an `eql_v2_encrypted` composite. v2 is a read path now, so the branch points at the deprecated `@cipherstash/stack/schema` builders, decrypting through `@cipherstash/stack`. Also: the `EncryptedSingleQueryBuilder` doc comment said a method omitted from the interface "is absent at runtime too". `single()` returns `this` (query-builder.ts:479), so every filter and transform remains a property of the object — the interface narrows the static API only. Skipped three markdownlint findings (MD040 aside): this repo runs no markdownlint, and MD028 on the supabase skill and the code-span whitespace in the rewriter changeset are both correct as written under CommonMark. --- .changeset/skills-v3-lifecycle-honesty.md | 11 +++++++++++ ...encryption-signature-and-typecheck-gates-design.md | 2 +- packages/cli/src/commands/init/lib/setup-prompt.ts | 2 +- packages/stack-supabase/src/types.ts | 6 ++++-- skills/stash-encryption/SKILL.md | 2 +- 5 files changed, 18 insertions(+), 5 deletions(-) diff --git a/.changeset/skills-v3-lifecycle-honesty.md b/.changeset/skills-v3-lifecycle-honesty.md index 5808d215e..350dc68d8 100644 --- a/.changeset/skills-v3-lifecycle-honesty.md +++ b/.changeset/skills-v3-lifecycle-honesty.md @@ -22,6 +22,17 @@ case, expect the wrong column to be dropped. ship — on a v3-only database `db push` reports "Nothing to do." and exits 0, and `db activate` errors. The call-outs are now scoped to the EQL v2 + Proxy path. +- The `stash encrypt cutover` row claimed application reads of the promoted + column "return decrypted ciphertext transparently". That holds only through + CipherStash Proxy, and it contradicted the next row, which requires the + decrypt path — an agent following the table would return raw + `eql_v2_encrypted` composites to end users. SDK/ORM reads are now stated to + need the explicit decrypt path. +- `stash init`'s setup prompt told agents to declare an **EQL v2** cutover + target with a `types.*` domain. Those are EQL v3 only; a v2 column is an + `eql_v2_encrypted` composite. The v2 branch now points at the deprecated + `@cipherstash/stack/schema` builders as a read-only path, decrypting through + `@cipherstash/stack`. Skills ship inside the `stash` tarball and are copied into user projects at `stash init`, so this guidance was being installed into customer repos. diff --git a/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md b/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md index 5776ed1d7..c8c1cdb93 100644 --- a/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md +++ b/docs/superpowers/specs/2026-07-24-encryption-signature-and-typecheck-gates-design.md @@ -90,7 +90,7 @@ TypeScript's `ReturnType` reads the *last* overload, which is the nominal one. S `Awaited>` yields `EncryptionClient` even for an all-v3 schema set, and assigning the real client to it fails: -``` +```text Type 'TypedEncryptionClient<…>' is missing the following properties from type 'EncryptionClient': client, encryptConfig, init ``` diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index dd5ba0de7..c0e22ba21 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -303,7 +303,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '#### Backfill and switch — after dual-writes are live', '', `3. **Backfill.** Run \`${cli} encrypt backfill --table --column \`. The CLI prompts the user (or accepts \`--confirm-dual-writes-deployed\` non-interactively) to confirm dual-writes are live, then chunks through the existing rows. Resumable; checkpoints to \`cs_migrations\` after every chunk. SIGINT-safe.`, - `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`). The adapters no longer author v2 — \`@cipherstash/stack-drizzle\` removed \`encryptedType\` — so declare the column with a \`types.*\` v3 domain and reach v2 rows through \`@cipherstash/stack\`'s decrypt path.`, + `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`). Do **not** declare a v2 column with a \`types.*\` domain — those are EQL v3 only. The adapters no longer author v2 (\`@cipherstash/stack-drizzle\` removed \`encryptedType\`), so a v2 column is a read path: declare it with the deprecated \`@cipherstash/stack/schema\` builders and decrypt through \`@cipherstash/stack\`.`, '5. **Wire the read path through the encryption client.** The read column now holds ciphertext. Read code paths must decrypt before returning the value to callers — `decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, or the equivalent `decrypt`/`bulkDecryptModels` calls. Without this step, your read paths return raw encrypted payloads to end users. The integration skill has the exact API.', '6. **Remove the dual-write code.** The plaintext column (still `` on v3; renamed `_plaintext` on v2) is no longer authoritative. Delete the dual-write logic from the persistence layer.', `7. **Drop.** Run \`${cli} encrypt drop --table --column \`. Generates a migration that removes the now-unused plaintext column (on v3 it first verifies no rows are still plaintext-only). Apply with the project's normal migration tooling.`, diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index 6755b8b56..e19f685e9 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -464,8 +464,10 @@ export interface TypedEncryptedSupabaseInstance { * * What IS carried is exactly: `then` (via `PromiseLike`), `abortSignal`, * `throwOnError`, `returns`, `withLockContext` and `audit`. That is a - * hand-written list, not a passthrough — `single()`/`maybeSingle()` return the - * same builder instance, so a method absent here is absent at runtime too. + * hand-written list, not a passthrough. It narrows the static API only: + * `single()`/`maybeSingle()` return the same builder instance, so a method + * omitted here is still a property of that object at runtime — the type is what + * stops you reaching it, not the implementation. * * It is therefore NOT parity with postgrest-js, which carries a different set: * its `single()` returns a `PostgrestBuilder`, carrying diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 1ddab9b84..185ee801b 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -875,7 +875,7 @@ Once dual-writes are recorded as live in `cs_migrations`: |---|---| | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | | Schema rename (**v2 only**) | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. **v3:** there is no rename — leave `_encrypted` under its own name and point the schema/queries at that name instead. | -| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. | +| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Reads of `` through **CipherStash Proxy** are decrypted on the wire; SDK/ORM reads still return ciphertext and need the decrypt path in the next row. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. | | Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | | Remove dual-write code | The plaintext column is no longer authoritative — **v2:** it is now `_plaintext` (the cutover renamed it); **v3:** it is still the original ``, since nothing was renamed. Either way, delete the dual-write logic once reads are served from the encrypted column. | | `stash encrypt drop` | Emits a migration that drops the plaintext column — and *which* column that is depends on the generation. **v2** (precondition: phase `cut-over`): a plain `ALTER TABLE … DROP COLUMN "_plaintext"`. **v3** (precondition: phase `backfilled`): it drops the **original ``** — there is no `_plaintext` — and the generated SQL is a `DO` block that takes `ACCESS EXCLUSIVE` on the table, re-counts rows with `` set and `_encrypted` NULL *at apply time*, and raises instead of dropping if any remain. Apply with the project's normal migration tooling. | From eb75e0944d36282abab47729c22be6fe11a9329c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:12:48 +1000 Subject: [PATCH 063/123] docs(changesets): drop the redundant lifecycle bullets The existing changeset already states that the bundled skills described the v2 lifecycle as the default and that the guidance ships into customer repos. Enumerating each individual correction adds nothing for a reader of the changelog. --- .changeset/skills-v3-lifecycle-honesty.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/.changeset/skills-v3-lifecycle-honesty.md b/.changeset/skills-v3-lifecycle-honesty.md index 350dc68d8..5808d215e 100644 --- a/.changeset/skills-v3-lifecycle-honesty.md +++ b/.changeset/skills-v3-lifecycle-honesty.md @@ -22,17 +22,6 @@ case, expect the wrong column to be dropped. ship — on a v3-only database `db push` reports "Nothing to do." and exits 0, and `db activate` errors. The call-outs are now scoped to the EQL v2 + Proxy path. -- The `stash encrypt cutover` row claimed application reads of the promoted - column "return decrypted ciphertext transparently". That holds only through - CipherStash Proxy, and it contradicted the next row, which requires the - decrypt path — an agent following the table would return raw - `eql_v2_encrypted` composites to end users. SDK/ORM reads are now stated to - need the explicit decrypt path. -- `stash init`'s setup prompt told agents to declare an **EQL v2** cutover - target with a `types.*` domain. Those are EQL v3 only; a v2 column is an - `eql_v2_encrypted` composite. The v2 branch now points at the deprecated - `@cipherstash/stack/schema` builders as a read-only path, decrypting through - `@cipherstash/stack`. Skills ship inside the `stash` tarball and are copied into user projects at `stash init`, so this guidance was being installed into customer repos. From 856dcc8a89ec1adab7d2779d1375974fa03ea85c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:19:21 +1000 Subject: [PATCH 064/123] fix(stack): make typed encryptQuery callable against the built package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `client.encryptQuery(value, { table, column })` did not typecheck for ANY column when imported from the published package — every plaintext resolved to `never`. Reproduced identically against the rc.4 tarball on npm, released main @ b48494ce, origin/remove-v2 and this branch, so it predates the v2 removal and shipped in every release carrying the v3 typed client. `PlaintextForColumn` / `QueryTypesForColumn` recover the domain via `C extends EncryptedV3Column`, which needs `D` bare in the instance type. Its only bare site was `private readonly definition: D`, and tsc strips private member types on declaration emit — leaving only `D['eqlType']`, `D['capabilities']` and `QueryableFlag`, none of which TypeScript can invert. So `infer D` fell back to the V3DomainDefinition constraint: QueryTypesForColumn -> never -> QueryableColumnsOf -> never -> plaintext never. `encrypt` escaped it by resolving through `ColumnsOf`. Fix is a type-only `declare readonly __domain?: D`: nothing emitted at runtime, no constructor or call-site change, but it survives declaration emit and restores the inference site. Adds `packages/stack/dist-types` and a `test:types:dist` script that typechecks the EMITTED declarations rather than source, wired into CI. That gap is why this survived the whole rc series — every other type test imports `@/…`. Verified red without the fix (three errors, including the original `never`) and green with it. Found while converting the A-6 call sites: the same tests that would have caught it were holding the client through the broken `ReturnType` idiom. --- .changeset/typed-encrypt-query-dist.md | 34 ++++++++++++ .github/workflows/tests.yml | 8 +++ packages/stack/dist-types/README.md | 15 ++++++ packages/stack/dist-types/encrypt-query.ts | 63 ++++++++++++++++++++++ packages/stack/dist-types/tsconfig.json | 14 +++++ packages/stack/package.json | 1 + packages/stack/src/eql/v3/columns.ts | 26 +++++++++ turbo.json | 6 +++ 8 files changed, 167 insertions(+) create mode 100644 .changeset/typed-encrypt-query-dist.md create mode 100644 packages/stack/dist-types/README.md create mode 100644 packages/stack/dist-types/encrypt-query.ts create mode 100644 packages/stack/dist-types/tsconfig.json diff --git a/.changeset/typed-encrypt-query-dist.md b/.changeset/typed-encrypt-query-dist.md new file mode 100644 index 000000000..1fce605e7 --- /dev/null +++ b/.changeset/typed-encrypt-query-dist.md @@ -0,0 +1,34 @@ +--- +'@cipherstash/stack': patch +--- + +Fixed: `encryptQuery` on the typed EQL v3 client did not typecheck against the +published package — for any column. + +```ts +const users = encryptedTable('users', { email: types.TextSearch('email') }) +const client = await Encryption({ schemas: [users] }) + +client.encrypt('a@b.com', { table: users, column: users.email }) // fine +client.encryptQuery('a@b.com', { table: users, column: users.email }) +// TS2345: Argument of type '"a@b.com"' is not assignable to parameter of type 'never' +``` + +`PlaintextForColumn` and `QueryTypesForColumn` recover a column's domain with +`C extends EncryptedV3Column`, which needs `D` to appear bare somewhere +in the instance type. Its only bare occurrence was a **private** field, and +`tsc` strips the types of private members on declaration emit — so in the +shipped `.d.ts` the inference fell back to the `V3DomainDefinition` constraint. +`QueryTypesForColumn` collapsed to `never`, which made `QueryableColumnsOf` +`never`, which typed every query plaintext `never`. `encrypt` was unaffected +because it resolves through `ColumnsOf`. + +`EncryptedV3Column` now carries a type-only `declare readonly __domain?: D`. +Nothing is emitted at runtime and no call site changes; the declaration survives +emit and restores the inference site. + +This affected every published release with the v3 typed client, including +`1.0.0-rc.4` — the searchable-query recipes in the `stash-encryption` skill did +not compile in a customer's repo. It was invisible in CI because the type tests +import source rather than `dist/`; a new `test:types:dist` suite now typechecks +the emitted declarations and is wired into CI. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 95f18246a..0f27e8780 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -185,6 +185,14 @@ jobs: - name: Typecheck (e2e) run: pnpm exec turbo run typecheck --filter @cipherstash/e2e + # Everything else typechecks against SOURCE. This one reads the emitted + # `.d.ts`, which is what customers consume — and where typed `encryptQuery` + # sat broken for every column through the whole rc series, because `tsc` + # strips private member types and that was the only place the column's + # domain parameter appeared bare. + - name: Typecheck (stack — emitted declarations, not source) + run: pnpm exec turbo run test:types:dist --filter @cipherstash/stack + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/packages/stack/dist-types/README.md b/packages/stack/dist-types/README.md new file mode 100644 index 000000000..69e727c9c --- /dev/null +++ b/packages/stack/dist-types/README.md @@ -0,0 +1,15 @@ +# Declaration-emit contract + +These files typecheck against `../dist`, not `../src`. + +Every other type test in this package imports `@/…`, which resolves to source. +That is the right default — it keeps the tests fast and independent of a build — +but it means nothing checked what customers actually consume, and a defect lived +in the published `.d.ts` for the whole rc series because of it: +`EncryptedV3Column`'s domain parameter `D` was recoverable in source and not in +the emitted declarations, so typed `encryptQuery` was uncallable for every column +against the published package. + +Anything whose correctness depends on what `tsc` *emits* — rather than on what +the source says — belongs here. Requires a build first; wired into CI as +`test:types:dist`. diff --git a/packages/stack/dist-types/encrypt-query.ts b/packages/stack/dist-types/encrypt-query.ts new file mode 100644 index 000000000..c90fd7527 --- /dev/null +++ b/packages/stack/dist-types/encrypt-query.ts @@ -0,0 +1,63 @@ +/** + * Typed `encryptQuery` must be callable against the BUILT package. + * + * `PlaintextForColumn` / `QueryTypesForColumn` recover a column's domain with + * `C extends EncryptedV3Column`, which needs `D` to appear bare in the + * instance type. Its only bare site in source is a PRIVATE field, and `tsc` + * strips private member types on declaration emit — so in the published `.d.ts` + * the inference fell back to the `V3DomainDefinition` constraint and every + * derived helper collapsed. `QueryTypesForColumn` became `never`, which made + * `QueryableColumnsOf` `never`, which typed the plaintext `never`: + * + * client.encryptQuery('a@b.com', { table: users, column: users.email }) + * // TS2345: Argument of type '"a@b.com"' is not assignable to type 'never' + * + * `encrypt` was unaffected (it resolves through `ColumnsOf`), so a smoke test + * of the built types would have missed it. Assert the query path explicitly. + */ + +import { Encryption, encryptedTable, types } from '../dist/encryption/v3.js' +import type { + EncryptedTextSearchColumn, + PlaintextForColumn, + QueryTypesForColumn, +} from '../dist/eql/v3/index.js' + +// The two helpers, on the concrete column class, straight from the emitted types. +const plaintext: string = + null as unknown as PlaintextForColumn +const queryTypes: 'equality' | 'orderAndRange' | 'freeTextSearch' = + null as unknown as QueryTypesForColumn +void plaintext +void queryTypes + +const users = encryptedTable('users', { + email: types.TextSearch('email'), + age: types.IntegerOrd('age'), +}) + +export async function callable() { + const client = await Encryption({ schemas: [users] }) + + client.encrypt('a@b.com', { table: users, column: users.email }) + client.encryptQuery('a@b.com', { + table: users, + column: users.email, + queryType: 'freeTextSearch', + }) + client.encryptQuery(30, { + table: users, + column: users.age, + queryType: 'orderAndRange', + }) + + // The narrowing must survive emit too, or the gate above passes vacuously. + // @ts-expect-error - `email` is a text domain, not a number + client.encryptQuery(123, { table: users, column: users.email }) + // @ts-expect-error - `searchableJson` is not a capability of a text_search column + client.encryptQuery('x', { + table: users, + column: users.email, + queryType: 'searchableJson', + }) +} diff --git a/packages/stack/dist-types/tsconfig.json b/packages/stack/dist-types/tsconfig.json new file mode 100644 index 000000000..4ec0246f1 --- /dev/null +++ b/packages/stack/dist-types/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "lib": ["ES2022", "DOM"], + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "bundler", + "moduleDetection": "force", + "customConditions": ["node"], + "strict": true, + "skipLibCheck": true, + "noEmit": true + }, + "include": ["**/*.ts"] +} diff --git a/packages/stack/package.json b/packages/stack/package.json index 32a167b60..24580eb22 100644 --- a/packages/stack/package.json +++ b/packages/stack/package.json @@ -198,6 +198,7 @@ "db:eql-v3:install": "tsx scripts/install-eql-v3.ts", "test": "vitest run", "test:types": "vitest --run --typecheck.only", + "test:types:dist": "tsc --noEmit -p dist-types/tsconfig.json", "release": "tsup", "test:integration": "vitest run --config integration/vitest.config.ts" }, diff --git a/packages/stack/src/eql/v3/columns.ts b/packages/stack/src/eql/v3/columns.ts index 30c46ce81..d6094110c 100644 --- a/packages/stack/src/eql/v3/columns.ts +++ b/packages/stack/src/eql/v3/columns.ts @@ -432,6 +432,32 @@ function isQueryableCapabilities(capabilities: QueryCapabilities): boolean { * nominality is what keeps plaintext inference precise. */ export class EncryptedV3Column { + /** + * Phantom carrier for `D`, and the reason this class is usable from the + * BUILT package at all. + * + * `PlaintextForColumn` / `QueryTypesForColumn` recover the domain with + * `C extends EncryptedV3Column`. That inference needs `D` to appear + * BARE somewhere in the instance type. In source the only such site is + * `private readonly definition: D` — but `tsc` strips the types of private + * members on declaration emit, so the shipped `.d.ts` says + * `private readonly definition;` and the site disappears. What is left is + * `getEqlType(): D['eqlType']`, `getQueryCapabilities(): D['capabilities']` + * and `isQueryable(): QueryableFlag` — all uninvertible, so `infer D` fell + * back to the `V3DomainDefinition` constraint and every helper collapsed: + * `QueryTypesForColumn` to `never`, `PlaintextForColumn` to the union of + * every domain's plaintext. Typed `encryptQuery` was therefore uncallable for + * ANY column against the published package. + * + * `declare` means type-only: no field is emitted, no runtime cost, no + * constructor change — but the declaration survives emit and gives `infer D` + * the bare site it needs. Optional so no call site has to supply it. + * + * @internal Not for consumption; read the domain via `getEqlType()` / + * `getQueryCapabilities()`. + */ + declare readonly __domain?: D + constructor( private readonly columnName: string, private readonly definition: D, diff --git a/turbo.json b/turbo.json index f7c6b2dd0..ff8873f3c 100644 --- a/turbo.json +++ b/turbo.json @@ -26,6 +26,12 @@ "dependsOn": ["^build"], "outputs": [] }, + // Typechecks `packages/stack/dist-types` against the BUILT declarations, so + // it must run after `build` — unlike `test:types`, which reads source. + "test:types:dist": { + "dependsOn": ["build"], + "inputs": ["$TURBO_DEFAULT$"] + }, "typecheck": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$"] From 7faf16cf359f447a70de828620b6d1666fc91d09 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Fri, 24 Jul 2026 22:22:41 +1000 Subject: [PATCH 065/123] fix(stack): scope the dist type gate correctly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to the gate added in the previous commit: - Exclude `dist-types` from `packages/stack/tsconfig.json`. That config maps `@/*` and the `@cipherstash/stack/*` subpaths to SOURCE, so checking these files there resolved their imports to the very thing they exist to avoid and reported two failures that said nothing about what ships. - Move the `@ts-expect-error` directives onto the exact lines the errors are reported at. A directive only covers the following line, and the formatter is free to split a call across several — which it did, silently turning one negative into 'unused directive' and letting the other through as a real error. Re-proved red/green by deleting the phantom field and rebuilding: 4 errors without it, including the original `never`; clean with it. --- packages/stack/dist-types/encrypt-query.ts | 14 ++++++++++---- packages/stack/tsconfig.json | 7 ++++++- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/packages/stack/dist-types/encrypt-query.ts b/packages/stack/dist-types/encrypt-query.ts index c90fd7527..a8b0ac83d 100644 --- a/packages/stack/dist-types/encrypt-query.ts +++ b/packages/stack/dist-types/encrypt-query.ts @@ -51,13 +51,19 @@ export async function callable() { queryType: 'orderAndRange', }) - // The narrowing must survive emit too, or the gate above passes vacuously. - // @ts-expect-error - `email` is a text domain, not a number - client.encryptQuery(123, { table: users, column: users.email }) - // @ts-expect-error - `searchableJson` is not a capability of a text_search column + // The narrowing must survive emit too, or the assertions above pass + // vacuously. Each directive sits on the exact line the error is reported at — + // `@ts-expect-error` only covers the following line, and the formatter is free + // to split these calls across several. + client.encryptQuery( + // @ts-expect-error - `email` is a text domain, so the plaintext is a string + 123, + { table: users, column: users.email }, + ) client.encryptQuery('x', { table: users, column: users.email, + // @ts-expect-error - `searchableJson` is not a capability of text_search queryType: 'searchableJson', }) } diff --git a/packages/stack/tsconfig.json b/packages/stack/tsconfig.json index ddc4ca84c..7b8c05f90 100644 --- a/packages/stack/tsconfig.json +++ b/packages/stack/tsconfig.json @@ -58,5 +58,10 @@ // depend on `build`, so CI can run type tests against an older dist). "@cipherstash/stack/errors": ["./src/errors/index.ts"] } - } + }, + // `dist-types/` deliberately typechecks against the EMITTED declarations, so + // it has its own tsconfig with none of the `@/*` source paths above. Checking + // it here too would resolve its imports differently and report failures that + // say nothing about what ships. Run it with `pnpm test:types:dist`. + "exclude": ["dist-types"] } From 36f998862f63f4dbd123db9f223316d8cc27ac15 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:23:56 +1000 Subject: [PATCH 066/123] fix(stack): refuse EQL v2 wire over an all-v3 schema set (A-8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EncryptionV3 used to force `eqlVersion: 3` over whatever the caller passed, and its comment said that override was the whole point: a v2-mode client cannot author v3 concrete-domain columns. Collapsing EncryptionV3 into a deprecated alias of Encryption deleted the override, and nothing else caught the combination — resolveEqlVersion throws for a mixed set and for legacy v2 searchableJson, then returns an explicit version unchanged. So `EncryptionV3({ schemas: [v3Table], config: { eqlVersion: 2 } })` went from silently auto-corrected-and-working to silently building a v2-wire client, with no diagnostic anywhere. Verified against the FFI boundary: newClient received eqlVersion 2, and the encryptConfig and the args reaching ffi.encrypt were byte-identical to the v3 case — only the flag differed. The failure surfaces later, as an eql_v3_* domain CHECK rejecting the write, or as v2 wire landing in a v3 column wherever the check is looser. Throw instead, keyed on `v3Count === schemas.length` so the hatch survives exactly where it is used: minting v2 wire from a V2 schema set, which is what integration/shared/v2-decrypt-compat.integration.test.ts does. Nothing in the repo mints v2 wire from a v3 table — the unit test that asserted that shape was pure, with a rationale ("for minting v2 wire during a migration") no consumer ever exercised. Retargeted, along with the two init-strategy tests that pinned the same behaviour with comments already hedging about it. Also widens stack-drizzle's typecheck gate to the two integration adapters that this branch made zero-error (A-6). The remaining 63 errors are still ungated and still recorded in the workflow; most are one root cause in test-kit's V3_MATRIX (#778). Verified the widened gate actually fails on an injected error rather than silently including nothing. --- .changeset/reject-v2-wire-over-v3-schemas.md | 30 +++++++++++++ .../stack-drizzle/tsconfig.typecheck.json | 15 ++++++- .../__tests__/encryption-overloads.test-d.ts | 16 ++++--- .../stack/__tests__/init-strategy.test.ts | 44 +++++++++++++------ .../__tests__/resolve-eql-version.test.ts | 37 +++++++++++++++- packages/stack/src/encryption/index.ts | 22 ++++++++++ 6 files changed, 143 insertions(+), 21 deletions(-) create mode 100644 .changeset/reject-v2-wire-over-v3-schemas.md diff --git a/.changeset/reject-v2-wire-over-v3-schemas.md b/.changeset/reject-v2-wire-over-v3-schemas.md new file mode 100644 index 000000000..a4b90ccfe --- /dev/null +++ b/.changeset/reject-v2-wire-over-v3-schemas.md @@ -0,0 +1,30 @@ +--- +'@cipherstash/stack': minor +--- + +`Encryption({ schemas, config: { eqlVersion: 2 } })` now throws when every table +in `schemas` is an EQL v3 table, instead of building a client that writes EQL v2 +payloads into `eql_v3_*` columns. + +`EncryptionV3` used to prevent this by forcing `eqlVersion: 3` over whatever the +caller passed — the override's comment said that was why it existed. Collapsing +`EncryptionV3` into a deprecated alias of `Encryption` removed the override, and +nothing else rejected the combination: `resolveEqlVersion` already threw for a +mixed v2/v3 schema set and for legacy v2 `searchableJson()`, but returned an +explicit version unchanged. So a caller upgrading from +`EncryptionV3({ schemas: [v3Table], config: { eqlVersion: 2 } })` — previously +auto-corrected, and working — silently got a v2-wire client instead, with no +diagnostic at any layer. The type surface agreed with the runtime (both say +"nominal client"), so nothing disagreed loudly enough to notice, and the failure +surfaced later as an `eql_v3_*` domain CHECK rejecting the write, or as v2 wire +landing in a v3 column wherever the check is looser. + +The escape hatch itself is unchanged where it is actually used: an explicit +`eqlVersion: 2` over an **EQL v2** schema set still emits v2 wire, which is how +v2 payloads are minted for the read-compatibility suite. Mixed sets still throw +the existing mixing error. Reading v2 payloads is unaffected — `decrypt` and +`decryptModel` continue to read both generations regardless of the client's wire +version. + +If you hit the new error, drop `config.eqlVersion` to emit v3, or build the +client from the EQL v2 schema you actually intend to write. diff --git a/packages/stack-drizzle/tsconfig.typecheck.json b/packages/stack-drizzle/tsconfig.typecheck.json index 9d06679f0..16d312f2f 100644 --- a/packages/stack-drizzle/tsconfig.typecheck.json +++ b/packages/stack-drizzle/tsconfig.typecheck.json @@ -1,4 +1,17 @@ { + // The typecheck gate CI runs (`test:types` -> vitest typecheck) covers the + // `.test-d.ts` files plus the two `integration/**` adapters that #772's + // review found were erroring in a file nothing compiled. + // + // `src/`, the runtime `__tests__/*.test.ts` and the rest of `integration/**` + // are still ungated: 63 errors under the full `tsconfig.json`, most of them + // one root cause — `V3_MATRIX`'s `indexes` union in `@cipherstash/test-kit` + // (#778). Widen this `include` as that count comes down; do not widen it to + // the whole project until it is zero, or the gate is useless. "extends": "./tsconfig.json", - "include": ["__tests__/**/*.test-d.ts"] + "include": [ + "__tests__/**/*.test-d.ts", + "integration/adapter.ts", + "integration/json-adapter.ts" + ] } diff --git a/packages/stack/__tests__/encryption-overloads.test-d.ts b/packages/stack/__tests__/encryption-overloads.test-d.ts index 82cc1f978..7bbd1cff6 100644 --- a/packages/stack/__tests__/encryption-overloads.test-d.ts +++ b/packages/stack/__tests__/encryption-overloads.test-d.ts @@ -92,11 +92,17 @@ describe('overload selection', () => { client.encrypt('2020-01-01', { table: users, column: users.createdAt }) }) - // S-4: forcing v2 wire over v3 schemas returns the NOMINAL client at runtime - // (the typed client cannot author v3 columns in v2 mode). The types used to - // claim the typed client, so `decryptModel(row, table, lockContext)` compiled - // and then silently dropped `table` and `lockContext`. - it('forcing eqlVersion 2 over v3 schemas yields the nominal client', async () => { + // S-4: `eqlVersion: 2` selects the NOMINAL overload, because the typed client + // cannot author v3 columns in v2 mode. The types used to claim the typed + // client, so `decryptModel(row, table, lockContext)` compiled and then + // silently dropped `table` and `lockContext`. + // + // Over an all-v3 schema set this combination is now REJECTED AT RUNTIME + // (#772 review, finding 8) — v2 wire into `eql_v3_*` columns is a + // contradiction, and `EncryptionV3` used to force `eqlVersion: 3` precisely + // to stop it. This assertion therefore records overload resolution only; the + // call itself throws. `init-strategy.test.ts` pins the throw. + it('forcing eqlVersion 2 selects the nominal overload (and is refused at runtime for v3 schemas)', async () => { const client = await Encryption({ schemas: [users], config: { eqlVersion: 2 }, diff --git a/packages/stack/__tests__/init-strategy.test.ts b/packages/stack/__tests__/init-strategy.test.ts index 1b29b4359..b9357193d 100644 --- a/packages/stack/__tests__/init-strategy.test.ts +++ b/packages/stack/__tests__/init-strategy.test.ts @@ -228,8 +228,23 @@ describe('Encryption config.eqlVersion', () => { expect(lastNewClientOpts().eqlVersion).toBe(3) }) - it('lets an explicit eqlVersion 2 override v3 auto-detection (migration escape hatch)', async () => { - await Encryption({ schemas: [v3Table()], config: { eqlVersion: 2 } }) + // #772 review, finding 8. This used to be honoured, and the resulting client + // wrote `eql_v2_encrypted` payloads into `eql_v3_*` columns with no + // diagnostic at any layer — the type surface types it as the nominal client, + // which matches what the runtime returns, so nothing disagreed loudly enough + // to notice. + it('rejects an explicit eqlVersion 2 over a v3 schema set before constructing the FFI client', async () => { + await expect( + Encryption({ schemas: [v3Table()], config: { eqlVersion: 2 } }), + ).rejects.toThrow(/entirely EQL v3/) + + expect(ffi.newClient).not.toHaveBeenCalled() + }) + + // The escape hatch survives where it is actually used: minting v2 wire from + // a v2 schema set, which is how the v2-read-compat fixtures are produced. + it('still honours an explicit eqlVersion 2 for a v2 schema set', async () => { + await Encryption({ schemas: [users], config: { eqlVersion: 2 } }) expect(lastNewClientOpts().eqlVersion).toBe(2) }) @@ -254,17 +269,20 @@ describe('Encryption config.eqlVersion', () => { expect(lastNewClientOpts().eqlVersion).toBe(3) }) - it('EncryptionV3 is a deprecated alias of Encryption — it honours an explicit eqlVersion identically', async () => { - // Post-collapse `EncryptionV3` IS `Encryption` (a deprecated alias), so it no - // longer independently pins the wire format: an explicit `eqlVersion: 2` over - // a v3 schema set is honoured exactly as `Encryption` honours it (the - // migration escape hatch above). A v2-mode client still cannot encrypt v3 - // concrete-type columns — this only pins the wire flag reaching the FFI. - await EncryptionV3({ - schemas: [v3Table() as never], - config: { eqlVersion: 2 }, - }) + it('EncryptionV3 rejects an explicit eqlVersion 2 identically', async () => { + // Post-collapse `EncryptionV3` IS `Encryption` (a deprecated alias), so it + // no longer independently pins the wire format. The invariant it used to + // enforce by forcing `eqlVersion: 3` now lives in `resolveEqlVersion`, so + // the outcome for a caller upgrading from the old `EncryptionV3` is the + // same in the case that matters: the contradiction is refused rather than + // silently building a v2-wire client over v3 concrete domains. + await expect( + EncryptionV3({ + schemas: [v3Table() as never], + config: { eqlVersion: 2 }, + }), + ).rejects.toThrow(/entirely EQL v3/) - expect(lastNewClientOpts().eqlVersion).toBe(2) + expect(ffi.newClient).not.toHaveBeenCalled() }) }) diff --git a/packages/stack/__tests__/resolve-eql-version.test.ts b/packages/stack/__tests__/resolve-eql-version.test.ts index e2e93f0ce..2ec1d0c5c 100644 --- a/packages/stack/__tests__/resolve-eql-version.test.ts +++ b/packages/stack/__tests__/resolve-eql-version.test.ts @@ -86,14 +86,47 @@ describe('resolveEqlVersion — legacy v2 searchable JSON', () => { }) describe('resolveEqlVersion — explicit config.eqlVersion', () => { - it('honours an explicit 2 over v3 schemas, for minting v2 wire during a migration', () => { - expect(resolveEqlVersion([usersV3], 2)).toBe(2) + // #772 review, finding 8. `EncryptionV3` used to force `eqlVersion: 3` over + // whatever the caller passed, with a comment saying it existed to stop + // exactly this. It is now a bare alias of `Encryption`, so the override is + // gone and nothing rejected the combination — the client built v2-wire over + // v3 concrete domains, which writes an `eql_v2_encrypted` payload into an + // `eql_v3_*` column. `resolveEqlVersion` already throws for a mixed set and + // for v2 SteVec; this was the one contradiction it let through. + it('rejects an explicit 2 over an all-v3 schema set', () => { + expect(() => resolveEqlVersion([usersV3], 2)).toThrow( + /cannot emit EQL v2 wire for a schema set that is entirely EQL v3/, + ) + }) + + it('rejects an explicit 2 over several v3 tables', () => { + expect(() => resolveEqlVersion([usersV3, ordersV3], 2)).toThrow( + /entirely EQL v3/, + ) + }) + + // The escape hatch itself survives, and this is the shape that actually uses + // it: `integration/shared/v2-decrypt-compat.integration.test.ts` mints its v2 + // fixtures from a v2 schema set. Nothing mints v2 wire from a v3 table. + it('still honours an explicit 2 over a v2 schema set', () => { + expect(resolveEqlVersion([usersV2], 2)).toBe(2) }) it('honours an explicit 3 over a v2 schema set', () => { expect(resolveEqlVersion([usersV2], 3)).toBe(3) }) + it('honours an explicit 3 over a v3 schema set', () => { + expect(resolveEqlVersion([usersV3], 3)).toBe(3) + }) + + // An empty set never reaches here through `Encryption` — the caller throws + // first — but the function is exported, so pin that it does not invent a + // wire version for a set with no evidence in it either way. + it('does not reject an explicit 2 for an empty schema set', () => { + expect(resolveEqlVersion([], 2)).toBe(2) + }) + it('does not let an explicit version rescue a mixed schema set', () => { // Mixing is unfixable by declaration: the two generations target different // column types, so no single wire format serves both. diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index 39b8ff907..a0a3f0c4e 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -124,6 +124,28 @@ export function resolveEqlVersion( ) } + // An explicit 2 over a set that is ENTIRELY v3 is a contradiction, and a + // silent one: the FFI client emits `eql_v2_encrypted` payloads for columns + // whose Postgres domain is `eql_v3_*`, so the write either trips the domain + // CHECK or lands v2 wire wherever the check is looser. + // + // `EncryptionV3` used to prevent this by forcing `eqlVersion: 3` over + // whatever the caller passed — its comment said so explicitly. It is now a + // bare alias of `Encryption`, so a caller upgrading from + // `EncryptionV3({ schemas: [v3Table], config: { eqlVersion: 2 } })` — which + // was silently corrected and worked — now gets a v2-wire client with no + // diagnostic at any layer (#772 review, finding 8). + // + // The escape hatch itself stays: an explicit 2 over a V2 schema set is how + // `integration/shared/v2-decrypt-compat.integration.test.ts` mints the + // fixtures that prove v2 payloads still decrypt. That is the only shape that + // uses it — nothing mints v2 wire from a v3 table. + if (explicit === 2 && schemas.length > 0 && v3Count === schemas.length) { + throw new Error( + "[encryption]: cannot emit EQL v2 wire for a schema set that is entirely EQL v3 — the payloads would not match the columns' eql_v3_* domains. Drop `config.eqlVersion` to emit v3, or build the client from the EQL v2 schema you actually want to write.", + ) + } + if (explicit !== undefined) { return explicit } From 5d304ec13f0be09e57d2d3107ca0e889e0f857b7 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 10:42:47 +1000 Subject: [PATCH 067/123] fix(stack): close the init wire-version back door, unbreak generic schemas, stop shipping stale skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review remediation for #782. Six agents cross-checked the review findings; three of them were overturned and three defects the PR itself introduced were found in the process. Each fix has a regression test proven to fail without it. Defects introduced by this PR ----------------------------- `init` on the typed v3 client was a way around the guard A-8 had just added. `EncryptionClient.init` takes its own `eqlVersion`, forwards `undefined` to the FFI — whose default is EQL v2 — and never consults `resolveEqlVersion`. So the A-7 passthrough let a typed v3 client be re-initialised into v2 wire while keeping its v3 surface: `eql_v2_encrypted` payloads into `eql_v3_*` columns, the same contradiction refused at construction, one method call later. The existing parity test could not see it — it stubs `init` out and asserts only that it was called. `init` now pins `eqlVersion: 3`, refuses an explicit `2` with a failure `Result` (its declared shape), and resolves to the typed client so `client = (await client.init(cfg)).data` does not silently drop to nominal. `NonEmptyV3` on the `schemas` property broke every caller generic over its schemas — a shape that compiled before A-4, and the one the `EncryptionClientFor` docs point generic code at. A type parameter is assignable to a deferred conditional only if it is assignable to both branches, and one branch is `never`. Deleting the guard is not the fix (overload 1 then accepts `[]`); the v3 signature is split in two, with the tuple overload carrying non-emptiness in its constraint, where a generic `S` can satisfy it. `outputs: ["dist/**"]` could publish stale skills. `cli` and `wizard` copy the repo-root `skills/` into `dist/skills`, outside their own input scope, so a skills-only edit never invalidated the build — and once outputs were declared, a cache hit began actively restoring the previous copy. Since `stash init` installs those into customer repos, an edited skill could ship with its pre-edit text while the source on disk was correct and CI green. Reproduced end to end before fixing. Review findings --------------- Nine comments described the `eqlVersion: 2` escape hatch as still permitted, including two documenting live guards: the `isV3Only && eqlVersion === 3` branch in `Encryption`, and the `V3ClientConfig` JSDoc the prisma-next texts paraphrase. `skills/stash-encryption` claimed an empty array is a compile error. That holds only when the argument's type has a statically known length of 0 — a spread of an empty array is a literal at the call site and still compiles. Corrected, and the runtime half now has coverage it never had. Four typecheck gates compiled their own `dist/` alongside source, so they compiled a different program in CI than locally, and `migrate`'s file set depended on CI step order. `wizard` and `examples/basic` had the same defect. The declaration gate reached `dist/` by relative path under bundler resolution, so it never consulted the `exports` map: the `.d.cts` set and `wasm-inline.d.ts` — each with its own inlined copy of the column class — were ungated. The wasm entry is the documented target for Workers, Deno, Bun and Supabase Edge. `__domain` was reachable from consumer code and appeared in autocomplete on every column, typed `D | undefined` and always `undefined`; there is no `stripInternal`, so `@internal` was documentation, not enforcement. Now keyed by a non-exported `unique symbol` — as inferrable, unnameable. New tests --------- - `scripts/lint-typecheck-scope.mjs` + self-tests + CI step: a gated tsconfig must scope away its build output. - `dist-types/node16/`: `.cts`/`.mts`/wasm probes resolving by package name under node16, so the conditions a customer walks are the ones under test. - `turbo-skills-inputs.test.mjs`: a build that copies `skills/` must declare it. - `empty-schemas-boundary.test.ts`, `typed-client-init-wire-version.test.ts`. Left alone ---------- `eqlVersion: 3` over an all-v2 schema set is still accepted, writing v3 payloads into `eql_v2_encrypted` columns — the mirror of what A-8 refuses. It predates this branch and `resolve-eql-version.test.ts:115` pins it deliberately, so closing it is a product decision rather than review remediation. --- .../cli-v2-cutover-prompt-correction.md | 13 ++ .changeset/domain-carrier-not-public.md | 18 ++ .changeset/encryption-schema-arrays.md | 16 ++ .changeset/prisma-next-v3-client-config.md | 6 +- .changeset/skills-ship-current-copy.md | 18 ++ .changeset/stack-audit-on-decrypt.md | 2 +- .changeset/typecheck-gates-scope-source.md | 20 +++ .changeset/typed-client-init-parity.md | 14 +- .github/workflows/tests.yml | 8 + examples/basic/tsconfig.json | 7 +- package.json | 1 + packages/migrate/tsconfig.json | 7 +- packages/nextjs/package.json | 2 +- packages/nextjs/tsconfig.json | 7 +- .../prisma-next/src/stack/from-stack-v3.ts | 4 +- .../__tests__/supabase-v3-factory.test.ts | 6 +- .../dynamodb/client-compat.test-d.ts | 19 +- .../__tests__/empty-schemas-boundary.test.ts | 100 +++++++++++ .../__tests__/encryption-overloads.test-d.ts | 54 ++++++ .../typed-client-init-wire-version.test.ts | 96 ++++++++++ .../stack/dist-types/node16/encrypt-query.cts | 67 +++++++ .../stack/dist-types/node16/encrypt-query.mts | 66 +++++++ .../stack/dist-types/node16/tsconfig.json | 26 +++ .../stack/dist-types/node16/wasm-inline.mts | 33 ++++ packages/stack/dist-types/tsconfig.json | 4 + packages/stack/package.json | 2 +- packages/stack/src/dynamodb/index.ts | 8 +- packages/stack/src/encryption/index.ts | 42 +++-- packages/stack/src/encryption/v3.ts | 52 +++++- packages/stack/src/eql/v3/columns.ts | 15 +- packages/stack/src/types.ts | 8 +- packages/wizard/tsconfig.json | 7 +- .../excludes-dist/package.json | 4 + .../excludes-dist/tsconfig.json | 4 + .../has-include/package.json | 4 + .../has-include/tsconfig.json | 4 + .../jsonc-paths/package.json | 4 + .../jsonc-paths/tsconfig.json | 12 ++ .../lint-typecheck-scope/no-gate/package.json | 4 + .../no-gate/tsconfig.json | 3 + .../unscoped-too/package.json | 4 + .../unscoped-too/tsconfig.json | 4 + .../unscoped/package.json | 4 + .../unscoped/tsconfig.json | 3 + .../__tests__/lint-typecheck-scope.test.mjs | 87 +++++++++ .../__tests__/turbo-skills-inputs.test.mjs | 119 +++++++++++++ scripts/lint-typecheck-scope.mjs | 165 ++++++++++++++++++ skills/stash-encryption/SKILL.md | 6 +- turbo.json | 19 ++ 49 files changed, 1146 insertions(+), 52 deletions(-) create mode 100644 .changeset/cli-v2-cutover-prompt-correction.md create mode 100644 .changeset/domain-carrier-not-public.md create mode 100644 .changeset/skills-ship-current-copy.md create mode 100644 .changeset/typecheck-gates-scope-source.md create mode 100644 packages/stack/__tests__/empty-schemas-boundary.test.ts create mode 100644 packages/stack/__tests__/typed-client-init-wire-version.test.ts create mode 100644 packages/stack/dist-types/node16/encrypt-query.cts create mode 100644 packages/stack/dist-types/node16/encrypt-query.mts create mode 100644 packages/stack/dist-types/node16/tsconfig.json create mode 100644 packages/stack/dist-types/node16/wasm-inline.mts create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/tsconfig.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/has-include/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/has-include/tsconfig.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/tsconfig.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/tsconfig.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/tsconfig.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/package.json create mode 100644 scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/tsconfig.json create mode 100644 scripts/__tests__/lint-typecheck-scope.test.mjs create mode 100644 scripts/__tests__/turbo-skills-inputs.test.mjs create mode 100644 scripts/lint-typecheck-scope.mjs diff --git a/.changeset/cli-v2-cutover-prompt-correction.md b/.changeset/cli-v2-cutover-prompt-correction.md new file mode 100644 index 000000000..eb917fe2d --- /dev/null +++ b/.changeset/cli-v2-cutover-prompt-correction.md @@ -0,0 +1,13 @@ +--- +'stash': patch +--- + +`stash init`'s setup prompt no longer tells agents to declare an EQL v2 cutover +column with a `types.*` domain. + +The `types.*` factories are EQL v3 only — a v2 column is an `eql_v2_encrypted` +composite — so an agent following the old step-4 guidance would author a schema +that cannot describe the column it just cut over to. The prompt now says +explicitly not to use `types.*` for a v2 column, and points at the deprecated +`@cipherstash/stack/schema` builders with decryption through +`@cipherstash/stack`, which is the actual read path for legacy v2 rows. diff --git a/.changeset/domain-carrier-not-public.md b/.changeset/domain-carrier-not-public.md new file mode 100644 index 000000000..202284dbb --- /dev/null +++ b/.changeset/domain-carrier-not-public.md @@ -0,0 +1,18 @@ +--- +'@cipherstash/stack': patch +--- + +The phantom domain carrier on encrypted v3 columns is no longer part of the +public surface. + +Recovering a column's domain after declaration emit requires the type parameter +to appear bare somewhere in the instance type, which is what makes typed +`encryptQuery` callable against the published package. That carrier was a +plainly named `__domain` property, and this build has no `stripInternal`, so +`@internal` was documentation rather than enforcement: it appeared in +autocomplete on every column and could be read from consumer code, typed +`D | undefined` while being `undefined` at runtime in every case. + +It is now keyed by a `unique symbol` that is not exported, so it is unnameable +outside the package while remaining exactly as inferrable. Nothing is emitted at +runtime and no call site changes. diff --git a/.changeset/encryption-schema-arrays.md b/.changeset/encryption-schema-arrays.md index 2fa9082bb..f5a125303 100644 --- a/.changeset/encryption-schema-arrays.md +++ b/.changeset/encryption-schema-arrays.md @@ -34,3 +34,19 @@ its schemas — an adapter that builds a table per request, say — can now writ If you narrowed a schema array to `readonly [AnyV3Table, ...AnyV3Table[]]` to satisfy the old signature, that narrowing is no longer needed. + +A wrapper that is itself generic over its schemas keeps working too: + +```ts +async function makeClient( + schemas: S, +) { + return await Encryption({ schemas }) +} +``` + +Note the non-empty tuple constraint. A wrapper generic over a loose +`readonly AnyV3Table[]` is still rejected — that type admits `readonly []`, so +the wrapper cannot promise what `Encryption` requires. Constrain it as above, or +take `EncryptionClientFor` as a parameter instead of +building the client inside the generic function. diff --git a/.changeset/prisma-next-v3-client-config.md b/.changeset/prisma-next-v3-client-config.md index 8d17b9a54..000bb7d41 100644 --- a/.changeset/prisma-next-v3-client-config.md +++ b/.changeset/prisma-next-v3-client-config.md @@ -16,9 +16,9 @@ const cipherstash = await cipherstashFromStack({ ``` The option never did what it looked like it did. This package is EQL v3 only, -and `eqlVersion: 2` selects `Encryption`'s nominal (untyped) overload at -runtime — not the `TypedEncryptionClient` that `cipherstashFromStack` returns as -`encryptionClient`. The field disagreed with the client you got back. +and `cipherstashFromStack` always builds from an all-v3 schema set, over which +`eqlVersion: 2` is refused at setup — v2 payloads cannot satisfy the columns' +`eql_v3_*` domains. The field promised a client it could never return. **Migration:** drop the `eqlVersion` field. Every other `ClientConfig` option (`workspaceCrn`, `clientId`, `clientKey`, `accessKey`, `authStrategy`, logging, diff --git a/.changeset/skills-ship-current-copy.md b/.changeset/skills-ship-current-copy.md new file mode 100644 index 000000000..3aa96a880 --- /dev/null +++ b/.changeset/skills-ship-current-copy.md @@ -0,0 +1,18 @@ +--- +'stash': patch +'@cipherstash/wizard': patch +--- + +Fixed: a change to `skills/` could ship a stale copy of that skill. + +Both CLIs copy the repo-root `skills/` into their bundle at build time +(`dist/skills`), which `stash init` then installs into a customer's +`.claude/skills/` or `.codex/skills/`. That directory sits outside the package, +so it was not part of the build's declared inputs — a skills-only edit did not +invalidate the cached build. Once the build began declaring its output +directory, a cache hit stopped being merely stale and started actively restoring +the previous `dist/skills` over the tree, so an edited skill could be published +with the pre-edit text while the source file on disk was correct and CI green. + +The two builds now declare the skills directory as an input, and a test pins +that coupling so it cannot come undone. diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md index a50a1f0cb..48b32a60e 100644 --- a/.changeset/stack-audit-on-decrypt.md +++ b/.changeset/stack-audit-on-decrypt.md @@ -37,7 +37,7 @@ strings needs updating. The v3 overload takes a `V3ClientConfig` — `ClientConfig` without the deprecated `eqlVersion` escape hatch. So `Encryption({ schemas: [] })` no longer type-checks (it used to compile and then throw), and `config: { eqlVersion: 2 }` selects the -nominal overload, which is the client you actually get back. +nominal overload — though over an all-v3 schema set that call now throws at setup. `Awaited>` names the nominal client whatever you pass, because `ReturnType` reads the last overload; use the exported `EncryptionClientFor` to name the client for a schema tuple. diff --git a/.changeset/typecheck-gates-scope-source.md b/.changeset/typecheck-gates-scope-source.md new file mode 100644 index 000000000..33879a9b5 --- /dev/null +++ b/.changeset/typecheck-gates-scope-source.md @@ -0,0 +1,20 @@ +--- +'@cipherstash/stack': patch +--- + +The declaration gate now covers all three emitted type artifacts, not just one. + +`packages/stack/dist-types` typechecks what `tsc` emits rather than what the +source says — the gap that let typed `encryptQuery` ship uncallable for every +column through the whole rc series. It resolved `../dist/*.js` by relative path +under bundler resolution, which never consults the `exports` map, so it only ever +exercised the ESM `.d.ts` set. `tsup` emits the CJS `.d.cts` set in a separate +pass and `wasm-inline.d.ts` in a third, each with its own inlined copy of the +column class whose domain parameter the bug was about. + +A second suite now resolves `@cipherstash/stack/*` **by package name** under +`moduleResolution: node16`, with the file extension selecting the condition +(`.cts` → `require` → `.d.cts`, `.mts` → `import` → `.d.ts`), plus a probe for +the `wasm-inline` entry — the documented target for Workers, Deno, Bun and +Supabase Edge, and previously the one artifact nothing typechecked. Removing the +phantom domain carrier makes all three fail, so none of them passes vacuously. diff --git a/.changeset/typed-client-init-parity.md b/.changeset/typed-client-init-parity.md index 7f835a6fa..ecc41b3b5 100644 --- a/.changeset/typed-client-init-parity.md +++ b/.changeset/typed-client-init-parity.md @@ -18,7 +18,19 @@ const client = await Encryption({ schemas: [users], config }) ``` `init` was the only member of `EncryptionClient` missing from the typed client, -so any call through the declared type crashed. It is now delegated. +so any call through the declared type crashed. It is now present — and pins the +wire format rather than delegating verbatim. + +That distinction matters. `EncryptionClient.init` takes its own `eqlVersion` and +forwards `undefined` to the FFI, whose default is EQL **v2**, and it never +consults the check `Encryption` runs at construction. A bare passthrough would +therefore have let a typed v3 client be re-initialised into v2 wire while keeping +its v3 surface — writing `eql_v2_encrypted` payloads into `eql_v3_*` columns, the +same contradiction refused at construction, one method call later. The typed +`init` pins `eqlVersion: 3`, refuses an explicit `2` with a failure `Result`, and +resolves to the **typed** client, so the common +`client = (await client.init(cfg)).data` does not silently swap the typed surface +for the nominal one. The type still under-reports in this case: you lose the typed surface with no diagnostic. Pass the config inline, or type the variable as `V3ClientConfig`, to diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0f27e8780..a94a885b9 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -196,6 +196,14 @@ jobs: - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners + # The gates above only mean something if they compile source. A tsconfig + # with no `include` globs `**/*`, which sweeps the package's own `dist/` + # into the program — and whether `dist/` exists during a gate depends on + # what an earlier step built transitively, so the gate compiles a + # different program in CI than it does locally. + - name: Lint — typecheck gates compile source, not build output + run: pnpm run lint:typecheck-scope + # Deleting or renaming a package leaves `packages/` references # dangling in docs and CI config. Nothing else catches it (#760 review). - name: Lint — no references to deleted package directories diff --git a/examples/basic/tsconfig.json b/examples/basic/tsconfig.json index 238655f2c..28110041f 100644 --- a/examples/basic/tsconfig.json +++ b/examples/basic/tsconfig.json @@ -23,5 +23,10 @@ "noUnusedLocals": false, "noUnusedParameters": false, "noPropertyAccessFromIndexSignature": false - } + }, + // The `typecheck` gate must compile source, not build output. TypeScript's + // default `exclude` covers `node_modules` but NOT `dist`, and specifying + // `exclude` replaces that default — hence both. Enforced by + // `pnpm run lint:typecheck-scope`. + "exclude": ["dist", "node_modules"] } diff --git a/package.json b/package.json index 13cfd258b..f25fe0c29 100644 --- a/package.json +++ b/package.json @@ -31,6 +31,7 @@ "code:check": "biome check", "lint:package-paths": "node scripts/lint-no-dead-package-paths.mjs", "lint:runners": "node scripts/lint-no-hardcoded-runners.mjs", + "lint:typecheck-scope": "node scripts/lint-typecheck-scope.mjs", "lint:workflow-cache": "node scripts/lint-no-workflow-caching.mjs", "release": "pnpm run build && changeset publish", "test": "turbo test --filter './packages/*'", diff --git a/packages/migrate/tsconfig.json b/packages/migrate/tsconfig.json index 982c07850..f479fb74d 100644 --- a/packages/migrate/tsconfig.json +++ b/packages/migrate/tsconfig.json @@ -19,5 +19,10 @@ "paths": { "@/*": ["./src/*"] } - } + }, + // The `typecheck` gate must compile source, not build output. TypeScript's + // default `exclude` covers `node_modules` but NOT `dist`, and specifying + // `exclude` replaces that default — hence both. Enforced by + // `pnpm run lint:typecheck-scope`. + "exclude": ["dist", "node_modules"] } diff --git a/packages/nextjs/package.json b/packages/nextjs/package.json index f4c512f65..df6e8a461 100644 --- a/packages/nextjs/package.json +++ b/packages/nextjs/package.json @@ -33,7 +33,7 @@ }, "scripts": { "build": "tsup", - "typecheck": "tsc --noEmit -p tsconfig.json", + "typecheck": "tsc --noEmit -p tsconfig.json", "dev": "tsup --watch", "release": "tsup" }, diff --git a/packages/nextjs/tsconfig.json b/packages/nextjs/tsconfig.json index 6da81c927..ad991f9c5 100644 --- a/packages/nextjs/tsconfig.json +++ b/packages/nextjs/tsconfig.json @@ -24,5 +24,10 @@ "noUnusedLocals": false, "noUnusedParameters": false, "noPropertyAccessFromIndexSignature": false - } + }, + // The `typecheck` gate must compile source, not build output. TypeScript's + // default `exclude` covers `node_modules` but NOT `dist`, and specifying + // `exclude` replaces that default — hence both. Enforced by + // `pnpm run lint:typecheck-scope`. + "exclude": ["dist", "node_modules"] } diff --git a/packages/prisma-next/src/stack/from-stack-v3.ts b/packages/prisma-next/src/stack/from-stack-v3.ts index bcc012cfc..071446b4d 100644 --- a/packages/prisma-next/src/stack/from-stack-v3.ts +++ b/packages/prisma-next/src/stack/from-stack-v3.ts @@ -59,8 +59,8 @@ export interface CipherstashFromStackV3Options { * Pass-through to `EncryptionV3({ config })` (keyset overrides, logging, …). * * `V3ClientConfig`, not `ClientConfig`: this package is EQL v3 only, and the - * legacy `eqlVersion: 2` escape hatch returns the nominal (untyped) client at - * runtime, which is not what this entry point hands back. + * legacy `eqlVersion: 2` escape hatch throws at setup over the all-v3 schema + * set this entry point derives. */ readonly encryptionConfig?: V3ClientConfig } diff --git a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts index b9cebc982..083af15e7 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts @@ -259,9 +259,9 @@ describe('encryptedSupabaseV3 factory', () => { }) }) - // `eqlVersion` is forced, not defaulted. A caller who passes `eqlVersion: 2` - // against v3 domains would otherwise get a v2 encryption client and fail at - // runtime with a 23514 CHECK violation, far from the cause. + // `eqlVersion` is forced, not defaulted. Without the force a caller's + // `eqlVersion: 2` would now make `Encryption` throw at setup, against the + // all-v3 schema set introspection synthesizes. it('forces eqlVersion 3 over a caller-supplied config, passing other keys through', async () => { await encryptedSupabaseV3(fakeClient, { databaseUrl: 'postgres://x', diff --git a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts index 971b641a2..b8995a67f 100644 --- a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts +++ b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts @@ -15,7 +15,7 @@ import { describe, expectTypeOf, it } from 'vitest' import type { EncryptedAttributes, EncryptedDynamoDBInstance } from '@/dynamodb' import { encryptedDynamoDB } from '@/dynamodb' import type { EncryptionClient } from '@/encryption' -import type { EncryptionV3 } from '@/encryption/v3' +import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' @@ -42,12 +42,17 @@ const searchV3 = encryptedTableV3('search_v3', { type V3Model = { pk: string; email?: string; age?: number; role?: string } -// The two client shapes. `EncryptionV3` returns a `TypedEncryptionClient` -// parameterised by its own schema tuple; `Encryption` returns the nominal -// `EncryptionClient`. Both must be accepted by the factory WITHOUT a cast. -declare const typedClient: Awaited< - ReturnType> -> +// The two client shapes. A v3 schema tuple yields a `TypedEncryptionClient` +// parameterised by it; a v2 set yields the nominal `EncryptionClient`. Both must +// be accepted by the factory WITHOUT a cast. +// +// Named with `EncryptionClientFor`, not `Awaited>` +// — `ReturnType` reads the LAST overload, so that idiom resolves to the nominal +// client whatever schemas you pass. Supplying an explicit type argument happens +// to dodge it today, but only because it filters the non-generic nominal +// overload out of the instantiation; give that overload a type parameter and +// this assertion would silently start checking the wrong client. +declare const typedClient: EncryptionClientFor declare const nominalClient: EncryptionClient describe('encryptedDynamoDB accepts both client shapes without a cast', () => { diff --git a/packages/stack/__tests__/empty-schemas-boundary.test.ts b/packages/stack/__tests__/empty-schemas-boundary.test.ts new file mode 100644 index 000000000..7d67ab2b2 --- /dev/null +++ b/packages/stack/__tests__/empty-schemas-boundary.test.ts @@ -0,0 +1,100 @@ +/** + * Where the compile-time non-emptiness guard stops, and the runtime one starts. + * + * A-4 widened `Encryption`'s v3 overload from a non-empty TUPLE to any array, + * moving the guard onto the `schemas` property via + * `NonEmptyV3 = S['length'] extends 0 ? never : S`. That guard fires only + * when the argument's TYPE has a statically known length of `0`. Once the type + * widens to `AnyV3Table[]` — a shared module export, a push-built array, a + * `.filter()` result, or a spread of any of those — `S['length']` is `number`, + * `number extends 0` is false, and the call compiles no matter how many tables + * are actually in it. + * + * So the widening deliberately traded a compile error for a runtime one on + * exactly the forms it exists to enable. `skills/stash-encryption` documents + * that boundary, and this pins the runtime half of it: the guard must reject an + * empty set BEFORE any FFI client is constructed, on every entry point. + * + * The compile-time half is pinned in `encryption-overloads.test-d.ts`. + */ + +import { beforeEach, describe, expect, it, vi } from 'vitest' +import type { AnyV3Table } from '@/eql/v3' +import { Encryption } from '@/index' +import { encryptedColumn, encryptedTable } from '@/schema' + +vi.mock('@cipherstash/protect-ffi', () => ({ + newClient: vi.fn(async () => ({ __mock: 'client' })), +})) + +import * as ffi from '@cipherstash/protect-ffi' + +const EMPTY_SCHEMAS = /At least one encryptedTable must be provided/ + +beforeEach(() => { + vi.clearAllMocks() +}) + +describe('empty schema sets are refused at runtime', () => { + // The forms below all COMPILE — that is the point. Each is a shape the A-4 + // widening admits, and each is empty at runtime. + it('rejects a shared array annotated AnyV3Table[] that is empty', async () => { + const shared: AnyV3Table[] = [] + + await expect(Encryption({ schemas: shared })).rejects.toThrow(EMPTY_SCHEMAS) + }) + + it('rejects a ReadonlyArray that is empty', async () => { + const frozen: ReadonlyArray = [] + + await expect(Encryption({ schemas: frozen })).rejects.toThrow(EMPTY_SCHEMAS) + }) + + it('rejects a spread of an empty array — a literal at the call site', async () => { + // Spreading erases the tuple length, so even an array literal here carries + // no compile-time non-emptiness. This is the form most likely to be read as + // "a literal, therefore checked". + const src: AnyV3Table[] = [] + + await expect(Encryption({ schemas: [...src] })).rejects.toThrow( + EMPTY_SCHEMAS, + ) + }) + + it('rejects an array emptied by a filter', async () => { + const all: AnyV3Table[] = [] + + await expect( + Encryption({ schemas: all.filter(() => false) }), + ).rejects.toThrow(EMPTY_SCHEMAS) + }) + + // The v2 builders reach the same guard — it predates the v3 typed client and + // is not part of the overload machinery. + it('rejects an empty v2 schema set', async () => { + const v2: Array> = [] + + await expect(Encryption({ schemas: v2 })).rejects.toThrow(EMPTY_SCHEMAS) + }) + + it('refuses before constructing the FFI client, so no credentials are needed', async () => { + const shared: AnyV3Table[] = [] + + await expect(Encryption({ schemas: shared })).rejects.toThrow(EMPTY_SCHEMAS) + expect(ffi.newClient).not.toHaveBeenCalled() + }) + + // Control: the same call shape with one table gets past the guard, proving the + // assertions above fail on emptiness rather than on the array form itself. + it('accepts a non-empty array of the same widened type', async () => { + const shared: AnyV3Table[] = [] + shared.push( + encryptedTable('users', { + email: encryptedColumn('email'), + }) as unknown as AnyV3Table, + ) + + await expect(Encryption({ schemas: shared })).resolves.toBeDefined() + expect(ffi.newClient).toHaveBeenCalledTimes(1) + }) +}) diff --git a/packages/stack/__tests__/encryption-overloads.test-d.ts b/packages/stack/__tests__/encryption-overloads.test-d.ts index 7bbd1cff6..aae0fc789 100644 --- a/packages/stack/__tests__/encryption-overloads.test-d.ts +++ b/packages/stack/__tests__/encryption-overloads.test-d.ts @@ -49,6 +49,60 @@ describe('overload selection', () => { Encryption({ schemas: [] }) }) + // ...but only when the ARGUMENT'S TYPE has a statically known length of 0. + // `NonEmptyV3` keys off `S['length'] extends 0`, so once the type widens to + // `AnyV3Table[]` the length is `number` and emptiness stops being visible — + // including for a literal at the call site, because a spread erases the tuple. + // These compile ON PURPOSE: rejecting them is what the A-4 tuple constraint + // did, and it broke every non-literal caller. The runtime guard is what + // catches them; `empty-schemas-boundary.test.ts` pins that half, and + // `skills/stash-encryption` documents the split for users. + // A generic passthrough is the shape `EncryptionClientFor` and the skill both + // advertise ("code that is generic over its schemas"), and it is the one form + // a WIDENING change must not break. It compiled before A-4 and must still. + // + // A deferred conditional on the property is what broke it: `S` is assignable + // to `NonEmptyV3` only if it is assignable to BOTH branches, and one branch + // is `never`. The emptiness rejection does not need it — overload 2's + // `AtLeastOneCsTable` rejects `[]` on its own. + it('accepts schemas from a generic wrapper function', async () => { + async function makeTypedClient< + S extends readonly [AnyV3Table, ...AnyV3Table[]], + >(schemas: S) { + return await Encryption({ schemas }) + } + + expectTypeOf(await makeTypedClient([users])).toEqualTypeOf< + TypedEncryptionClient + >() + + // A wrapper generic over a LOOSE `readonly AnyV3Table[]` is still rejected, + // here and on the pre-A-4 signature alike. That is the honest answer rather + // than a gap: such an `S` admits `readonly []`, so the wrapper cannot + // promise what `Encryption` requires. Constrain it to a non-empty tuple, as + // above, and it compiles. + async function makeLooseClient( + schemas: S, + ) { + // @ts-expect-error - a loose `S` cannot prove it is non-empty + return await Encryption({ schemas }) + } + void makeLooseClient + }) + + it('accepts arrays whose type cannot prove non-emptiness', () => { + const shared: AnyV3Table[] = [] + expectTypeOf(Encryption).toBeCallableWith({ schemas: shared }) + + const frozen: ReadonlyArray = [] + expectTypeOf(Encryption).toBeCallableWith({ schemas: frozen }) + + expectTypeOf(Encryption).toBeCallableWith({ schemas: [...shared] }) + expectTypeOf(Encryption).toBeCallableWith({ + schemas: shared.filter(() => false), + }) + }) + // A-4: closing S-6 with a non-empty TUPLE constraint rejected every schema // array that is not a literal, which is most real code — a shared module // export, anything built from introspection, anything `readonly`. The diff --git a/packages/stack/__tests__/typed-client-init-wire-version.test.ts b/packages/stack/__tests__/typed-client-init-wire-version.test.ts new file mode 100644 index 000000000..e535cc001 --- /dev/null +++ b/packages/stack/__tests__/typed-client-init-wire-version.test.ts @@ -0,0 +1,96 @@ +/** + * The typed client's `init` passthrough must not reopen the door A-8 closed. + * + * A-7 added `init` to the typed EQL v3 client so that a caller holding it + * through its declared `EncryptionClient` type could not hit + * `TypeError: init is not a function`. But `EncryptionClient.init` takes its own + * `eqlVersion?: 2 | 3`, and forwards `undefined` straight to `newClient`, where + * the FFI's default is EQL **v2**. `resolveEqlVersion` — which refuses exactly + * that combination — is only ever consulted by `Encryption`, never by `init`. + * + * So a bare passthrough lets a v3 client be silently re-initialised into v2 wire + * while keeping the typed v3 surface: the same contradiction A-8 rejects at + * construction, reachable one method call later. + * + * const client = await Encryption({ schemas: [users] }) // eqlVersion: 3 + * await client.init({ encryptConfig }) // eqlVersion: undefined -> v2 + * + * `typed-client-nominal-parity.test.ts` cannot catch this: it stubs `init` out + * entirely and asserts only that it was called. + */ + +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('@cipherstash/protect-ffi', () => ({ + newClient: vi.fn(async () => ({ __mock: 'client' })), +})) + +import * as ffi from '@cipherstash/protect-ffi' +import type { EncryptionClient } from '@/encryption' +import { encryptedTable, types } from '@/encryption/v3' +import { Encryption } from '@/index' +import { buildEncryptConfig } from '@/schema' + +const users = encryptedTable('users', { email: types.TextSearch('email') }) + +// biome-ignore lint/suspicious/noExplicitAny: reading recorded mock args +const newClientCalls = () => (ffi.newClient as any).mock.calls +const lastEqlVersion = () => newClientCalls().at(-1)[0].eqlVersion + +beforeEach(() => { + vi.clearAllMocks() +}) + +describe('typed client init keeps the v3 wire format', () => { + it('constructs at eqlVersion 3', async () => { + await Encryption({ schemas: [users] }) + + expect(lastEqlVersion()).toBe(3) + }) + + it('stays at eqlVersion 3 when re-initialised without one', async () => { + const client = await Encryption({ schemas: [users] }) + expect(lastEqlVersion()).toBe(3) + + await (client as unknown as EncryptionClient).init({ + encryptConfig: buildEncryptConfig(users), + }) + + // Without pinning, this is `undefined` — the FFI's v2 default — leaving a + // typed v3 surface over a client emitting v2 payloads into eql_v3_* columns. + expect(newClientCalls()).toHaveLength(2) + expect(lastEqlVersion()).toBe(3) + }) + + it('refuses an explicit eqlVersion 2 on re-init, as construction does', async () => { + const client = await Encryption({ schemas: [users] }) + + // `Encryption` throws for this combination, but `init` declares + // `Promise>` — the Result shape is contract, so this refuses with + // a failure rather than rejecting. Either way it never reaches the FFI. + const result = await (client as unknown as EncryptionClient).init({ + encryptConfig: buildEncryptConfig(users), + eqlVersion: 2, + }) + + expect(result.failure?.message).toMatch(/eqlVersion 2|eql_v3_\* domains/) + expect(newClientCalls()).toHaveLength(1) + }) + + it('returns the TYPED client, not the bare nominal one', async () => { + const client = await Encryption({ schemas: [users] }) + + const result = await (client as unknown as EncryptionClient).init({ + encryptConfig: buildEncryptConfig(users), + }) + + // `EncryptionClient.init` resolves `{ data: this }` — the nominal client. + // Reassigning from it (`client = (await client.init(c)).data`) is the + // natural idiom, and it must not silently drop the typed surface. + if (result.failure) throw new Error(result.failure.message) + expect( + typeof (result.data as { encryptQuery?: unknown }).encryptQuery, + ).toBe('function') + expect(result.data).toBe(client) + }) +}) diff --git a/packages/stack/dist-types/node16/encrypt-query.cts b/packages/stack/dist-types/node16/encrypt-query.cts new file mode 100644 index 000000000..32b48e7b1 --- /dev/null +++ b/packages/stack/dist-types/node16/encrypt-query.cts @@ -0,0 +1,67 @@ +/** + * The CommonJS declaration surface — `require` → `dist/**\/*.d.cts`. + * + * The sibling `../encrypt-query.ts` gate reaches into `../dist/*.js` by relative + * path under `moduleResolution: bundler`, so it never consults the `exports` + * map and only ever sees the ESM `.d.ts` set. tsup emits the `.d.cts` set in a + * separate pass, and `package.json` routes `require` at it. A defect confined to + * that pass would ship to every CJS consumer with nothing here to catch it. + * + * This file is `.cts`, so TypeScript treats it as CommonJS and resolves + * `@cipherstash/stack/v3` through the `require` condition. Importing BY PACKAGE + * NAME (Node self-reference, which works because `package.json` has both a + * `name` and an `exports` map) is the point: it exercises the conditions a + * customer's resolver walks, not a path we happen to know. + */ + +import type { + EncryptedTextSearchColumn, + PlaintextForColumn, + QueryTypesForColumn, +} from '@cipherstash/stack/eql/v3' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +// The two helpers, on the concrete column class, straight from the emitted +// `.d.cts`. These collapse to `never` / a wide union if the domain parameter is +// not recoverable after declaration emit. +const plaintext: string = + null as unknown as PlaintextForColumn +const queryTypes: 'equality' | 'orderAndRange' | 'freeTextSearch' = + null as unknown as QueryTypesForColumn +void plaintext +void queryTypes + +const users = encryptedTable('users', { + email: types.TextSearch('email'), + age: types.IntegerOrd('age'), +}) + +export async function callable() { + const client = await Encryption({ schemas: [users] }) + + client.encrypt('a@b.com', { table: users, column: users.email }) + client.encryptQuery('a@b.com', { + table: users, + column: users.email, + queryType: 'freeTextSearch', + }) + client.encryptQuery(30, { + table: users, + column: users.age, + queryType: 'orderAndRange', + }) + + // The narrowing must survive emit too, or the assertions above pass + // vacuously. Each directive sits on the exact line the error is reported at. + client.encryptQuery( + // @ts-expect-error - `email` is a text domain, so the plaintext is a string + 123, + { table: users, column: users.email }, + ) + client.encryptQuery('x', { + table: users, + column: users.email, + // @ts-expect-error - `searchableJson` is not a capability of text_search + queryType: 'searchableJson', + }) +} diff --git a/packages/stack/dist-types/node16/encrypt-query.mts b/packages/stack/dist-types/node16/encrypt-query.mts new file mode 100644 index 000000000..ba8ac246f --- /dev/null +++ b/packages/stack/dist-types/node16/encrypt-query.mts @@ -0,0 +1,66 @@ +/** + * The ESM declaration surface under NODE16 resolution — `import` → `*.d.ts`. + * + * `../encrypt-query.ts` already covers the `.d.ts` set, but through a relative + * path under `moduleResolution: bundler`. That combination cannot catch a broken + * `exports` map, a missing `types` condition, or a subpath that resolves under a + * bundler but not under Node's own algorithm — the resolution mode most server + * consumers actually use. + * + * Same assertions as the `.cts` twin; the only difference is which condition the + * resolver takes, which is exactly what is under test. + */ + +import type { + EncryptedTextSearchColumn, + PlaintextForColumn, + QueryTypesForColumn, +} from '@cipherstash/stack/eql/v3' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +const plaintext: string = + null as unknown as PlaintextForColumn +const queryTypes: 'equality' | 'orderAndRange' | 'freeTextSearch' = + null as unknown as QueryTypesForColumn +void plaintext +void queryTypes + +const users = encryptedTable('users', { + email: types.TextSearch('email'), + age: types.IntegerOrd('age'), +}) + +// The domain carrier that makes the inference above possible must not be part +// of the public surface. There is no `stripInternal` in this build, so a plainly +// named property would be reachable here — typed `D | undefined`, and always +// `undefined` at runtime. Keyed by a non-exported `unique symbol`, it is not. +// @ts-expect-error - the phantom domain carrier is not nameable by consumers +void users.email.__domain + +export async function callable() { + const client = await Encryption({ schemas: [users] }) + + client.encrypt('a@b.com', { table: users, column: users.email }) + client.encryptQuery('a@b.com', { + table: users, + column: users.email, + queryType: 'freeTextSearch', + }) + client.encryptQuery(30, { + table: users, + column: users.age, + queryType: 'orderAndRange', + }) + + client.encryptQuery( + // @ts-expect-error - `email` is a text domain, so the plaintext is a string + 123, + { table: users, column: users.email }, + ) + client.encryptQuery('x', { + table: users, + column: users.email, + // @ts-expect-error - `searchableJson` is not a capability of text_search + queryType: 'searchableJson', + }) +} diff --git a/packages/stack/dist-types/node16/tsconfig.json b/packages/stack/dist-types/node16/tsconfig.json new file mode 100644 index 000000000..034b0ae3f --- /dev/null +++ b/packages/stack/dist-types/node16/tsconfig.json @@ -0,0 +1,26 @@ +{ + // The sibling gate resolves `../dist/*.js` by RELATIVE path under + // `moduleResolution: bundler`, which never consults the `exports` map. This + // one resolves `@cipherstash/stack/*` BY PACKAGE NAME under `node16` — the + // way a customer does — so the conditions in `exports` select the artifact. + // + // The package is `"type": "module"`, so the file extension picks the mode: + // `.cts` is CommonJS and takes the `require` branch (`*.d.cts`), `.mts` is + // ESM and takes the `import` branch (`*.d.ts`). Both must be probed: tsup + // emits the two declaration sets in separate passes, and a third for + // `wasm-inline`. + // + // No `customConditions` here: `node` is implicit under node16, and this + // package's own `exports` map does not branch on it. + "compilerOptions": { + "lib": ["ES2022", "DOM"], + "target": "ES2022", + "module": "node16", + "moduleResolution": "node16", + "strict": true, + "skipLibCheck": true, + "noEmit": true, + "types": [] + }, + "include": ["**/*.cts", "**/*.mts"] +} diff --git a/packages/stack/dist-types/node16/wasm-inline.mts b/packages/stack/dist-types/node16/wasm-inline.mts new file mode 100644 index 000000000..947e2b4ca --- /dev/null +++ b/packages/stack/dist-types/node16/wasm-inline.mts @@ -0,0 +1,33 @@ +/** + * The THIRD declaration artifact: `dist/wasm-inline.d.ts`. + * + * `tsup.config.ts` runs a second, independent DTS pass for the wasm-inline + * entry, which inlines its own copy of `EncryptedV3Column` and the helpers that + * invert its domain parameter. Neither the bundler gate nor the `.cts`/`.mts` + * probes above reach it — they resolve `./v3` and `./eql/v3`, which come from + * the first pass. So the entry documented for Workers, Deno, Bun and Supabase + * Edge — the runtimes with the least margin for a broken type — was the one + * artifact nothing typechecked. + * + * `./wasm-inline` is ESM-only in the `exports` map (no `require` branch, by + * design: the inlined WASM blob cannot be `require`d), hence `.mts` and no + * `.cts` twin. + * + * `encryptQuery` is deliberately NOT column-narrowed on this entry + * (`WasmEncryptQueryOptions.column` is wide), so this asserts the helpers + * directly rather than through a call — they are what carries the domain, and + * what collapses if the phantom carrier is lost on emit. + */ + +import type { + EncryptedTextSearchColumn, + PlaintextForColumn, + QueryTypesForColumn, +} from '@cipherstash/stack/wasm-inline' + +const plaintext: string = + null as unknown as PlaintextForColumn +const queryTypes: 'equality' | 'orderAndRange' | 'freeTextSearch' = + null as unknown as QueryTypesForColumn +void plaintext +void queryTypes diff --git a/packages/stack/dist-types/tsconfig.json b/packages/stack/dist-types/tsconfig.json index 4ec0246f1..4e52e3b05 100644 --- a/packages/stack/dist-types/tsconfig.json +++ b/packages/stack/dist-types/tsconfig.json @@ -10,5 +10,9 @@ "skipLibCheck": true, "noEmit": true }, + // `node16/` is the same contract checked under Node's own resolution, by + // package name rather than relative path. It needs `module: node16`, so it + // carries its own tsconfig and must not be swept up by this one. + "exclude": ["node16"], "include": ["**/*.ts"] } diff --git a/packages/stack/package.json b/packages/stack/package.json index 24580eb22..1f3b5b0c5 100644 --- a/packages/stack/package.json +++ b/packages/stack/package.json @@ -198,7 +198,7 @@ "db:eql-v3:install": "tsx scripts/install-eql-v3.ts", "test": "vitest run", "test:types": "vitest --run --typecheck.only", - "test:types:dist": "tsc --noEmit -p dist-types/tsconfig.json", + "test:types:dist": "tsc --noEmit -p dist-types/tsconfig.json && tsc --noEmit -p dist-types/node16/tsconfig.json", "release": "tsup", "test:integration": "vitest run --config integration/vitest.config.ts" }, diff --git a/packages/stack/src/dynamodb/index.ts b/packages/stack/src/dynamodb/index.ts index a950c8f7c..a19f1a691 100644 --- a/packages/stack/src/dynamodb/index.ts +++ b/packages/stack/src/dynamodb/index.ts @@ -31,10 +31,10 @@ import type { * hand: the operation methods below, when a table is supplied. `encryptedDynamoDB` * itself receives no table, so it cannot check earlier. * - * RESIDUAL GAP: a client explicitly forced to `eqlVersion: 2` over a v3 schema - * set (a deliberate migration path) DOES register the table yet emits v2 wire; - * that is not detectable without a wire-version accessor the client does not - * provide, so it is out of scope here. + * The one case this could not detect — a client forced to `eqlVersion: 2` while + * registering a v3 table, emitting v2 wire for an `eql_v3_*` domain — is now + * refused by `resolveEqlVersion` at client construction, so no such client can + * reach here. */ function assertClientTableVersionMatch( encryptionClient: EncryptedDynamoDBConfig['encryptionClient'], diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index a0a3f0c4e..079ac2b28 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -89,9 +89,10 @@ export const noClientError = () => * columns and the v3 tables target `eql_v3` domains, so no single wire * format serves both. Split them across two clients. * - * An explicit `config.eqlVersion` bypasses version detection (the wire format - * is then unambiguous — e.g. writing v2 wire from a v3 schema set during a - * migration), but mixed schemas and legacy v2 SteVec schemas still throw. + * An explicit `config.eqlVersion` bypasses DETECTION, not validation: mixed + * schemas and legacy v2 SteVec schemas still throw, and so does an explicit `2` + * over an all-v3 set. What survives is `2` over a v2 schema set — minting v2 + * wire during a migration. * * @internal exported for unit-test coverage of the detection matrix. */ @@ -911,6 +912,27 @@ type NonEmptyV3 = S['length'] extends 0 // `NonEmptyV3` must sit on `schemas` and nowhere else: wrapping the whole // config in a conditional alias defeats `const` inference, degrading the tuple // to an array and erasing per-column plaintext typing on the literal path. +// +// It also cannot be the ONLY v3 signature, which is why there are two. A type +// parameter is assignable to a deferred conditional only if it is assignable to +// BOTH branches, and one branch here is `never` — so `NonEmptyV3` rejects +// every caller that is itself generic over its schemas: +// +// async function make(schemas: S) { +// return await Encryption({ schemas }) // TS2769 against `NonEmptyV3` +// } +// +// That shape compiled before A-4 and is the one the `EncryptionClientFor` docs +// point generic code at, so it must keep compiling. The overload below carries +// the non-emptiness in its CONSTRAINT instead of a conditional, which a generic +// `S` can satisfy; the widened one after it then only has to serve arrays that +// are concrete at the call site, where the conditional resolves eagerly. +export function Encryption< + const S extends readonly [AnyV3Table, ...AnyV3Table[]], +>(config: { + schemas: S + config?: V3ClientConfig +}): Promise> export function Encryption(config: { schemas: NonEmptyV3 config?: V3ClientConfig @@ -963,9 +985,10 @@ export async function Encryption( const isV3Only = schemas.every(hasBuildColumnKeyMap) // Resolve the wire format: an explicit `config.eqlVersion` wins (the retained - // migration escape hatch — e.g. deliberately writing v2 wire from a v3 schema - // set), otherwise it is auto-detected (v3 tables → 3, v2 tables → the FFI's v2 - // default). A mixed v2 + v3 schema set throws inside `resolveEqlVersion`. + // migration escape hatch — `2` over a v2 schema set), otherwise it is + // auto-detected (v3 tables → 3, v2 tables → the FFI's v2 default). A mixed + // v2 + v3 set, and an explicit `2` over an all-v3 set, throw inside + // `resolveEqlVersion`. const eqlVersion = resolveEqlVersion(schemas, clientConfig?.eqlVersion) const result = await client.init({ @@ -980,10 +1003,9 @@ export async function Encryption( } // Return the typed client only when the client is genuinely in EQL v3 mode: an - // all-v3 schema set that resolved to the v3 wire format. A caller that forced - // `eqlVersion: 2` over v3 schemas gets the nominal client for that deliberate, - // low-level v2-wire migration path (the typed client cannot encrypt v3 columns - // in v2 mode anyway). + // all-v3 schema set that resolved to the v3 wire format. The `eqlVersion === 3` + // conjunct is now belt-and-braces: `resolveEqlVersion` refuses an explicit `2` + // over an all-v3 set, so an all-v3 set cannot reach here in v2 mode. if (isV3Only && eqlVersion === 3) { // biome-ignore lint/plugin: the runtime `isV3Only` guard (every schema has // buildColumnKeyMap) proves these are AnyV3Table — the compiler can't see it. diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 248aaad41..cc79cb306 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -1,3 +1,4 @@ +import type { Result } from '@byteslice/result' import type { AnyV3Table, ColumnsOf, @@ -152,7 +153,7 @@ export interface TypedEncryptionClient { getEncryptConfig(): ReturnType /** - * Re-initialize the underlying client. + * Re-initialize the underlying client, staying on EQL v3. * * @internal Present for runtime parity with {@link EncryptionClient}, not as * part of the typed authoring surface. `Encryption` picks its return value by @@ -161,10 +162,25 @@ export interface TypedEncryptionClient { * selects the nominal overload but still yields this client at runtime. Every * other `EncryptionClient` method already existed here; without `init` that * mismatch turned into `TypeError: client.init is not a function`. + * + * This is NOT a bare passthrough, because `EncryptionClient.init` would + * otherwise be a way around the guard `Encryption` applies at construction. + * `init` takes its own `eqlVersion`, forwards `undefined` to the FFI (whose + * default is **v2**), and never consults `resolveEqlVersion` — so delegating + * verbatim would let a typed v3 client be re-initialized into v2 wire while + * keeping this v3 surface, the exact contradiction `resolveEqlVersion` + * refuses. The wire version is pinned to `3` and an explicit `2` is rejected. + * + * It also resolves to the TYPED client rather than the underlying nominal one, + * so `client = (await client.init(cfg)).data` — the natural idiom, since + * `EncryptionClient.init` resolves `{ data: this }` — keeps the typed surface + * instead of silently degrading to the nominal one. */ init( - config: Parameters[0], - ): ReturnType + config: Omit[0], 'eqlVersion'> & { + eqlVersion?: 3 + }, + ): Promise, EncryptionError>> } /** @@ -368,7 +384,11 @@ export function typedClient( ) as never } - return { + // Annotated rather than `satisfies`, because `init` resolves to `typed` + // itself and a self-referencing initializer has no inferrable type. The + // annotation checks the same shape the `satisfies` did, and the function's + // declared return type is this type anyway, so nothing is widened. + const typed: TypedEncryptionClient = { encrypt: (plaintext, opts) => client.encrypt(plaintext as never, opts as never), encryptQuery, @@ -382,8 +402,28 @@ export function typedClient( bulkEncrypt: (plaintexts, opts) => client.bulkEncrypt(plaintexts, opts), bulkDecrypt: (payloads) => client.bulkDecrypt(payloads), getEncryptConfig: () => client.getEncryptConfig(), - init: (config) => client.init(config), - } satisfies TypedEncryptionClient + // See the interface member for why this is not `client.init(config)`: + // `init` bypasses `resolveEqlVersion` and defaults the FFI to v2 wire, so a + // verbatim delegation would reopen the contradiction A-8 closes. Pin v3, + // refuse an explicit 2, and resolve to the typed client so the surface + // survives the round trip. + init: async (config) => { + if (config.eqlVersion !== undefined && config.eqlVersion !== 3) { + return { + failure: { + type: EncryptionErrorTypes.EncryptionError, + message: + "[eql/v3]: cannot re-initialize a typed EQL v3 client at eqlVersion 2 — the payloads would not match the columns' eql_v3_* domains. Build a separate client from the EQL v2 schema you want to write.", + }, + } + } + + const result = await client.init({ ...config, eqlVersion: 3 }) + return result.failure ? result : { data: typed } + }, + } + + return typed } /** diff --git a/packages/stack/src/eql/v3/columns.ts b/packages/stack/src/eql/v3/columns.ts index d6094110c..faf4a0560 100644 --- a/packages/stack/src/eql/v3/columns.ts +++ b/packages/stack/src/eql/v3/columns.ts @@ -431,6 +431,12 @@ function isQueryableCapabilities(capabilities: QueryCapabilities): boolean { * `EncryptedDateColumn`, both storage-only) are NOT mutually assignable. This * nominality is what keeps plaintext inference precise. */ +/** + * Key for {@link EncryptedV3Column}'s phantom domain carrier. Declared, never + * defined — it exists only in the type layer, and is deliberately not exported. + */ +declare const domainCarrier: unique symbol + export class EncryptedV3Column { /** * Phantom carrier for `D`, and the reason this class is usable from the @@ -453,10 +459,17 @@ export class EncryptedV3Column { * constructor change — but the declaration survives emit and gives `infer D` * the bare site it needs. Optional so no call site has to supply it. * + * Keyed by a non-exported `unique symbol` rather than a plain name. There is + * no `stripInternal` in this build, so `@internal` is documentation only — a + * `__domain` property would be reachable from customer code and show up in + * autocomplete on every column, typed `D | undefined` while being `undefined` + * always. The symbol is not exported, so the carrier is unnameable outside + * this module while remaining exactly as inferrable. + * * @internal Not for consumption; read the domain via `getEqlType()` / * `getQueryCapabilities()`. */ - declare readonly __domain?: D + declare readonly [domainCarrier]?: D constructor( private readonly columnName: string, diff --git a/packages/stack/src/types.ts b/packages/stack/src/types.ts index 7cca68207..7195c4f7c 100644 --- a/packages/stack/src/types.ts +++ b/packages/stack/src/types.ts @@ -208,10 +208,10 @@ export type ClientConfig = { * minus the legacy `eqlVersion: 2` escape hatch. * * `Encryption` accepts this (not the full `ClientConfig`) alongside an all-v3 - * schema set. Forcing v2 wire over v3 schemas returns the NOMINAL client at - * runtime, so admitting `2` there typed the call as `TypedEncryptionClient` - * while handing back a client that silently ignores the typed client's extra - * `decryptModel` arguments. Adapters that are v3-only (`@cipherstash/prisma-next`, + * schema set. Forcing v2 wire over v3 schemas THROWS at setup — v2 payloads + * cannot satisfy an `eql_v3_*` domain — so admitting `2` there typed the call + * as `TypedEncryptionClient` for a call that returns no client at all. + * Adapters that are v3-only (`@cipherstash/prisma-next`, * `@cipherstash/stack-drizzle`) should take this type for their pass-through * config for the same reason. */ diff --git a/packages/wizard/tsconfig.json b/packages/wizard/tsconfig.json index d88a9b847..7cc1059cb 100644 --- a/packages/wizard/tsconfig.json +++ b/packages/wizard/tsconfig.json @@ -30,5 +30,10 @@ "paths": { "@/*": ["./src/*"] } - } + }, + // The `typecheck` gate must compile source, not build output. TypeScript's + // default `exclude` covers `node_modules` but NOT `dist`, and specifying + // `exclude` replaces that default — hence both. Enforced by + // `pnpm run lint:typecheck-scope`. + "exclude": ["dist", "node_modules"] } diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/package.json new file mode 100644 index 000000000..c35a0d5ac --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/excludes-dist", + "scripts": { "typecheck": "tsc --noEmit -p tsconfig.json" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/tsconfig.json new file mode 100644 index 000000000..0aecccec4 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/excludes-dist/tsconfig.json @@ -0,0 +1,4 @@ +{ + "compilerOptions": { "strict": true, "noEmit": true }, + "exclude": ["dist", "node_modules"] +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/package.json new file mode 100644 index 000000000..46a9e7c53 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/has-include", + "scripts": { "typecheck": "tsc --noEmit -p tsconfig.json" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/tsconfig.json new file mode 100644 index 000000000..e0ba56aa8 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/has-include/tsconfig.json @@ -0,0 +1,4 @@ +{ + "compilerOptions": { "strict": true, "noEmit": true }, + "include": ["src/**/*.ts"] +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/package.json new file mode 100644 index 000000000..f6c511632 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/jsonc-paths", + "scripts": { "typecheck": "tsc --noEmit -p tsconfig.json" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/tsconfig.json new file mode 100644 index 000000000..f384a6d17 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/jsonc-paths/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + /* A block comment, and below a path mapping whose KEY contains `/*` — a + naive strip eats from inside that string to the next comment close. */ + "paths": { + "@/*": ["./src/*"] + }, + "strict": true + }, + // A trailing comma follows, which `JSON.parse` also rejects. + "exclude": ["dist", "node_modules"] +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/package.json new file mode 100644 index 000000000..24edb2530 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/no-gate", + "scripts": { "build": "tsup" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/tsconfig.json new file mode 100644 index 000000000..98865a3ea --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/no-gate/tsconfig.json @@ -0,0 +1,3 @@ +{ + "compilerOptions": { "strict": true, "noEmit": true } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/package.json new file mode 100644 index 000000000..f1126d4b3 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/unscoped-too", + "scripts": { "typecheck": "tsc --project tsconfig.json --noEmit" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/tsconfig.json new file mode 100644 index 000000000..636f8a894 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped-too/tsconfig.json @@ -0,0 +1,4 @@ +{ + "compilerOptions": { "strict": true, "noEmit": true }, + "exclude": ["node_modules"] +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/package.json b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/package.json new file mode 100644 index 000000000..abd692cf5 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/package.json @@ -0,0 +1,4 @@ +{ + "name": "@fixture/unscoped", + "scripts": { "typecheck": "tsc --noEmit -p tsconfig.json" } +} diff --git a/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/tsconfig.json b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/tsconfig.json new file mode 100644 index 000000000..98865a3ea --- /dev/null +++ b/scripts/__tests__/fixtures/lint-typecheck-scope/unscoped/tsconfig.json @@ -0,0 +1,3 @@ +{ + "compilerOptions": { "strict": true, "noEmit": true } +} diff --git a/scripts/__tests__/lint-typecheck-scope.test.mjs b/scripts/__tests__/lint-typecheck-scope.test.mjs new file mode 100644 index 000000000..093f9cf0a --- /dev/null +++ b/scripts/__tests__/lint-typecheck-scope.test.mjs @@ -0,0 +1,87 @@ +import { execFileSync } from 'node:child_process' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const SCRIPT = resolve( + fileURLToPath(import.meta.url), + '../../lint-typecheck-scope.mjs', +) + +function run(...targets) { + try { + execFileSync('node', [SCRIPT, ...targets], { encoding: 'utf8' }) + return { exitCode: 0, output: '' } + } catch (err) { + return { + exitCode: err.status, + output: String(err.stdout) + String(err.stderr), + } + } +} + +const fx = (name) => `scripts/__tests__/fixtures/lint-typecheck-scope/${name}` + +describe('lint-typecheck-scope', () => { + it('passes on the repo as it stands', () => { + const r = run() + expect(r.output).toBe('') + expect(r.exitCode).toBe(0) + }) + + it('fails a gated package whose tsconfig scopes nothing', () => { + const r = run(fx('unscoped')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/@fixture\/unscoped/) + expect(r.output).toMatch(/compiles its own build output/) + }) + + it('names the offending tsconfig and the gate command', () => { + const r = run(fx('unscoped')) + expect(r.output).toMatch(/unscoped\/tsconfig\.json/) + expect(r.output).toMatch(/tsc --noEmit -p tsconfig\.json/) + }) + + it('passes when `exclude` covers dist', () => { + expect(run(fx('excludes-dist')).exitCode).toBe(0) + }) + + it('passes when an explicit `include` scopes the program', () => { + expect(run(fx('has-include')).exitCode).toBe(0) + }) + + it('ignores a package with no typecheck gate', () => { + // An unscoped tsconfig nothing runs is an editor setting, not a CI + // contract — flagging it would be noise. + expect(run(fx('no-gate')).exitCode).toBe(0) + }) + + it('parses a tsconfig with comments, block comments and a `/*` inside a path key', () => { + // Regression: the first cut stripped block comments with a regex, which ate + // from the `/*` inside the `"@/*"` mapping key to the next comment close and + // reported four real tsconfigs as unparseable. + const r = run(fx('jsonc-paths')) + expect(r.output).not.toMatch(/could not be parsed/) + expect(r.exitCode).toBe(0) + }) + + it('counts only the offenders among the targets it was given', () => { + const r = run(fx('unscoped'), fx('no-gate'), fx('excludes-dist')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/Found 1 typecheck gate\(s\)/) + }) + + it('reports every offender, not just the first', () => { + const r = run(fx('unscoped'), fx('unscoped-too')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/Found 2 typecheck gate\(s\)/) + expect(r.output).toMatch(/@fixture\/unscoped\b/) + expect(r.output).toMatch(/@fixture\/unscoped-too/) + }) + + it('explains the fix, including that `exclude` replaces the default', () => { + const r = run(fx('unscoped')) + expect(r.output).toMatch(/"exclude": \["dist", "node_modules"\]/) + expect(r.output).toMatch(/REPLACES the default/) + }) +}) diff --git a/scripts/__tests__/turbo-skills-inputs.test.mjs b/scripts/__tests__/turbo-skills-inputs.test.mjs new file mode 100644 index 000000000..7706ca767 --- /dev/null +++ b/scripts/__tests__/turbo-skills-inputs.test.mjs @@ -0,0 +1,119 @@ +/** + * A build that copies `skills/` must declare `skills/` as an input. + * + * `packages/cli` and `packages/wizard` both `cpSync('../../skills', + * 'dist/skills')` — they consume a directory outside their own package, which + * turbo's `$TURBO_DEFAULT$` does not cover. On its own that only meant a stale + * cache entry stayed valid; once `build` declared `outputs: ["dist/**"]`, a + * cache hit began actively RESTORING the previous `dist/skills` over the tree. + * + * `skills/` ships inside the `stash` and `@cipherstash/wizard` tarballs and + * `stash init` copies it into customer repos, so the failure mode is publishing + * guidance that was edited but never rebuilt — invisible, because the build is + * green and the source file on disk is correct. + * + * Verified by hand at the time of writing: edit a `SKILL.md`, run + * `turbo run build --filter=stash`, observe FULL TURBO and a `dist/skills` copy + * without the edit. This pins the fix so it cannot silently come undone. + */ + +import { existsSync, readdirSync, readFileSync } from 'node:fs' +import { join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + +/** Strip comments so `JSON.parse` accepts turbo.json (it is JSONC). */ +function readJsonc(path) { + const raw = readFileSync(path, 'utf8') + let out = '' + let inString = false + let escaped = false + for (let i = 0; i < raw.length; i++) { + const ch = raw[i] + if (inString) { + out += ch + if (escaped) escaped = false + else if (ch === '\\') escaped = true + else if (ch === '"') inString = false + continue + } + if (ch === '"') { + inString = true + out += ch + continue + } + if (ch === '/' && raw[i + 1] === '/') { + while (i < raw.length && raw[i] !== '\n') i++ + out += '\n' + continue + } + if (ch === '/' && raw[i + 1] === '*') { + i += 2 + while (i < raw.length && !(raw[i] === '*' && raw[i + 1] === '/')) i++ + i++ + continue + } + out += ch + } + return JSON.parse(out.replace(/,(\s*[}\]])/g, '$1')) +} + +/** Packages whose tsup config copies the repo-root `skills/` into `dist/`. */ +function packagesCopyingSkills() { + const pkgsDir = join(REPO_ROOT, 'packages') + const found = [] + for (const entry of readdirSync(pkgsDir, { withFileTypes: true })) { + if (!entry.isDirectory()) continue + const tsup = join(pkgsDir, entry.name, 'tsup.config.ts') + const pkgJson = join(pkgsDir, entry.name, 'package.json') + if (!existsSync(tsup) || !existsSync(pkgJson)) continue + if ( + !/cpSync\(\s*['"]\.\.\/\.\.\/skills['"]/.test(readFileSync(tsup, 'utf8')) + ) + continue + found.push(JSON.parse(readFileSync(pkgJson, 'utf8')).name) + } + return found +} + +describe('turbo build inputs cover the skills directory', () => { + const turbo = readJsonc(join(REPO_ROOT, 'turbo.json')) + const copiers = packagesCopyingSkills() + + it('finds the packages that copy skills into their bundle', () => { + // If this drops to zero the rest of the suite silently passes, so pin it. + expect(copiers).toEqual( + expect.arrayContaining(['stash', '@cipherstash/wizard']), + ) + }) + + it.each( + packagesCopyingSkills().map((name) => [name]), + )('%s#build declares skills as an input', (name) => { + const task = turbo.tasks[`${name}#build`] + + expect( + task, + `turbo.json has no "${name}#build" task, so it inherits the generic ` + + '`build` inputs, which do not include the repo-root skills/ directory', + ).toBeDefined() + + const inputs = task.inputs ?? [] + expect( + inputs.some((i) => i.includes('skills')), + `"${name}#build".inputs must name the repo-root skills/ directory ` + + '(e.g. "$TURBO_ROOT$/skills/**") or a skills-only edit will not ' + + 'invalidate the build, and `outputs: ["dist/**"]` will restore a stale copy', + ).toBe(true) + }) + + it('keeps the generic build outputs on the overrides', () => { + // An override replaces the generic task wholesale — dropping `outputs` + // would stop the cache restoring dist at all for these two packages. + for (const name of copiers) { + expect(turbo.tasks[`${name}#build`].outputs).toEqual(['dist/**']) + } + }) +}) diff --git a/scripts/lint-typecheck-scope.mjs b/scripts/lint-typecheck-scope.mjs new file mode 100644 index 000000000..51198fb02 --- /dev/null +++ b/scripts/lint-typecheck-scope.mjs @@ -0,0 +1,165 @@ +/** + * A `typecheck` gate must compile SOURCE, and only source. + * + * TypeScript's default `exclude` is `["node_modules", "bower_components", + * "jspm_packages"]` plus `outDir` — it does NOT cover `dist/`. So a tsconfig + * with no `include`, no `exclude` and no `outDir` globs `**\/*`, which sweeps the + * package's own build output into the program alongside its source. + * + * That makes a CI gate non-deterministic in a way that is invisible when it + * passes. `turbo run typecheck --filter ` only builds the package's + * DEPENDENCIES (`dependsOn: ["^build"]`), not the package itself, so whether + * `dist/` exists during the gate depends on what an unrelated earlier CI step + * happened to build transitively. Reorder the steps and the gate silently + * compiles a different set of files. Locally, after a full build, it compiles a + * different set again. + * + * `skipLibCheck` hides most of the consequences — a stale `.d.ts` importing a + * deleted package still exits 0 — so this cannot be left to "CI is green". + * + * Fix by scoping the tsconfig: either an explicit `include` naming the source + * roots, or an `exclude` listing `dist`. Note that specifying `exclude` REPLACES + * the default list, so `node_modules` must be re-listed alongside `dist`. + */ + +import { existsSync, readdirSync, readFileSync } from 'node:fs' +import { join, relative, resolve } from 'node:path' + +const REPO_ROOT = resolve(import.meta.dirname, '..') + +// Roots holding workspace members, mirroring `pnpm-workspace.yaml`. Override +// with argv[2..] for tests / ad-hoc checks (each arg is a package directory). +const WORKSPACE_ROOTS = ['packages', 'examples'] + +/** Every directory that looks like a workspace member. */ +function discoverPackages() { + const found = [] + for (const root of WORKSPACE_ROOTS) { + const abs = resolve(REPO_ROOT, root) + if (!existsSync(abs)) continue + for (const entry of readdirSync(abs, { withFileTypes: true })) { + if (!entry.isDirectory()) continue + found.push(join(abs, entry.name)) + } + } + const e2e = resolve(REPO_ROOT, 'e2e') + if (existsSync(e2e)) found.push(e2e) + return found +} + +const targets = process.argv.slice(2).length + ? process.argv.slice(2).map((t) => resolve(REPO_ROOT, t)) + : discoverPackages() + +/** + * Strip comments and trailing commas so `JSON.parse` accepts a tsconfig. + * + * Scans character by character tracking string state rather than pattern + * matching: these tsconfigs map `"@/*": ["../stack/src/*"]`, and a regex for + * block comments happily eats from the `/*` inside that key to the next `*\/`, + * silently destroying the file it was meant to read. + */ +function readJsonc(path) { + const raw = readFileSync(path, 'utf8') + let out = '' + let inString = false + let escaped = false + + for (let i = 0; i < raw.length; i++) { + const ch = raw[i] + + if (inString) { + out += ch + if (escaped) escaped = false + else if (ch === '\\') escaped = true + else if (ch === '"') inString = false + continue + } + + if (ch === '"') { + inString = true + out += ch + continue + } + if (ch === '/' && raw[i + 1] === '/') { + while (i < raw.length && raw[i] !== '\n') i++ + out += '\n' + continue + } + if (ch === '/' && raw[i + 1] === '*') { + i += 2 + while (i < raw.length && !(raw[i] === '*' && raw[i + 1] === '/')) i++ + i++ + continue + } + out += ch + } + + return JSON.parse(out.replace(/,(\s*[}\]])/g, '$1')) +} + +const offenders = [] + +for (const pkgDir of targets) { + const pkgJsonPath = join(pkgDir, 'package.json') + const tsconfigPath = join(pkgDir, 'tsconfig.json') + if (!existsSync(pkgJsonPath) || !existsSync(tsconfigPath)) continue + + let pkgJson + let tsconfig + try { + pkgJson = JSON.parse(readFileSync(pkgJsonPath, 'utf8')) + tsconfig = readJsonc(tsconfigPath) + } catch (err) { + offenders.push( + `${relative(REPO_ROOT, tsconfigPath)}: could not be parsed — ${err.message}`, + ) + continue + } + + // Only packages wired as a gate. A tsconfig nothing runs is an editor + // setting, not a CI contract. + const scripts = pkgJson.scripts ?? {} + const gate = + scripts.typecheck ?? + (scripts.build === 'tsc --noEmit' ? scripts.build : undefined) + if (gate === undefined) continue + + // An explicit `include` scopes the program by itself. + if (Array.isArray(tsconfig.include) && tsconfig.include.length > 0) continue + + // Otherwise `exclude` has to carry it. `outDir` also excludes by default, but + // these gates all set `noEmit`, so relying on it would be a trap. + const exclude = Array.isArray(tsconfig.exclude) ? tsconfig.exclude : [] + const excludesDist = exclude.some( + (e) => typeof e === 'string' && /(^|\/)dist(\/|$|\*)/.test(e), + ) + if (excludesDist) continue + + offenders.push( + `${relative(REPO_ROOT, tsconfigPath)}: \`${pkgJson.name}\` runs a typecheck gate ` + + `(\`${gate}\`) but the tsconfig declares neither an \`include\` nor an ` + + '`exclude` covering `dist`, so the gate compiles its own build output', + ) +} + +if (offenders.length > 0) { + console.error( + `Found ${offenders.length} typecheck gate(s) that compile build output as well as source:\n`, + ) + for (const o of offenders) console.error(` ${o}`) + console.error( + "\nTypeScript's default `exclude` does not cover `dist/`, so a tsconfig with\n" + + "no `include` globs `**/*` and sweeps the package's own emitted `.d.ts`\n" + + 'and `.js` into the gate. Whether `dist/` exists during CI depends on what\n' + + 'an earlier, unrelated step built transitively — so the gate compiles a\n' + + 'different program in CI than it does locally, and silently changes if the\n' + + 'steps are reordered.\n\n' + + 'Add to the tsconfig:\n\n' + + ' "exclude": ["dist", "node_modules"]\n\n' + + '(`exclude` REPLACES the default list, so `node_modules` must be re-listed.)\n' + + 'Or give it an explicit `include` naming the source roots, as `e2e` and\n' + + '`examples/prisma` do.', + ) + process.exit(1) +} diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 185ee801b..d766079df 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -312,12 +312,11 @@ const client = await Encryption({ schemas: [users] }) - `EncryptionV3` (from `@cipherstash/stack/v3`) is a **deprecated, type-identical alias** of `Encryption`, kept for backwards compatibility. New code should use `Encryption`. - `typedClient(client, ...schemas)` (exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface, if you built one via a lower-level path. - **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. -- `schemas` takes any non-empty array of v3 tables — a shared `export const schemas: AnyV3Table[]`, a `ReadonlyArray`, one built at runtime. It does not have to be an array literal. An empty array is a compile error. +- `schemas` takes any non-empty array of v3 tables — a shared `export const schemas: AnyV3Table[]`, a `ReadonlyArray`, one built at runtime. It does not have to be an array literal. Writing `Encryption({ schemas: [] })` is a compile error, but an array typed `AnyV3Table[]` that is empty at runtime compiles and throws on init instead. - **To name the client's type, use `EncryptionClientFor`** (from `@cipherstash/stack/v3`), not `Awaited>`. `Encryption` is overloaded and `ReturnType` reads the last overload, so that idiom always resolves to the untyped nominal client: ```typescript -import { Encryption, type EncryptionClientFor, encryptedTable, types } from "@cipherstash/stack/v3" -import type { AnyV3Table } from "@cipherstash/stack/eql/v3" +import { type AnyV3Table, Encryption, type EncryptionClientFor, encryptedTable, types } from "@cipherstash/stack/v3" const users = encryptedTable("users", { email: types.TextSearch("email") }) @@ -329,7 +328,6 @@ client = await Encryption({ schemas: [users] }) function withClient(c: EncryptionClientFor) { /* … */ } ``` - ```typescript // Error handling try { diff --git a/turbo.json b/turbo.json index ff8873f3c..f86fa0a04 100644 --- a/turbo.json +++ b/turbo.json @@ -26,6 +26,25 @@ "dependsOn": ["^build"], "outputs": [] }, + // These two builds `cpSync('../../skills', 'dist/skills')` — they consume a + // directory OUTSIDE their own package, which `$TURBO_DEFAULT$` does not + // cover. Without naming it here a skills-only edit does not invalidate the + // build, and since `build` now declares `outputs: ["dist/**"]`, the cache + // hit actively RESTORES the previous `dist/skills`. `skills/` ships inside + // the `stash` and `@cipherstash/wizard` tarballs and is copied into + // customer repos by `stash init`, so that silently publishes stale guidance. + "stash#build": { + "dependsOn": ["^build"], + "outputs": ["dist/**"], + "inputs": ["$TURBO_DEFAULT$", "$TURBO_ROOT$/skills/**", ".env*"], + "env": ["STASH_POSTHOG_KEY"] + }, + "@cipherstash/wizard#build": { + "dependsOn": ["^build"], + "outputs": ["dist/**"], + "inputs": ["$TURBO_DEFAULT$", "$TURBO_ROOT$/skills/**", ".env*"], + "env": ["STASH_POSTHOG_KEY"] + }, // Typechecks `packages/stack/dist-types` against the BUILT declarations, so // it must run after `build` — unlike `test:types`, which reads source. "test:types:dist": { From 427088469480aed6bf2e80bb1c04ec98dea08266 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 10:35:21 +1000 Subject: [PATCH 068/123] fix(cli,wizard): detect non-public-schema and array EQL domains as encrypted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The corpus index recognised an encrypted column only in the mangled forms drizzle-kit emits and the literal `public` schema. A domain installed into another schema ("app"."eql_v3_text_search") or an array of a domain (public.eql_v3_text_search[]) fell to the plaintext residue, so an ALTER on such a column was rewritten into ADD+DROP+RENAME — dropping ciphertext the corpus itself showed was encrypted. ENCRYPTED_TYPE_REF now admits any quoted schema and a trailing '[', closing both. It feeds only the encrypted index, never the rewrite matcher, so the worst case is a flagged statement, not a new rewrite. Applied to both copies of the sweep; changeset guarantee updated to match. --- .changeset/rewriter-never-drops-ciphertext.md | 5 +- .../src/__tests__/rewrite-migrations.test.ts | 48 +++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 29 +++++++---- .../src/__tests__/rewrite-migrations.test.ts | 48 +++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 29 +++++++---- 5 files changed, 138 insertions(+), 21 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index 66ad0f209..ec82f4b60 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -22,7 +22,10 @@ either kind makes the rest of the file inert rather than live. The sweep also refuses to rewrite a column the migration corpus already gives an encrypted type, so changing a column's encrypted domain no longer drops a column -full of ciphertext. Skipped statements report why they were left alone. +full of ciphertext. Skipped statements report why they were left alone. This +recognises the encrypted forms drizzle-kit emits, a domain installed into a +non-`public` schema, and an array of a domain — so a corpus that shows a column +as encrypted in any of those shapes is flagged, not rewritten. The sweep is now fail-closed about the columns it does not recognise at all. Previously a column missing from the corpus index was assumed to be plaintext diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 275ee5852..d6e357ea3 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -865,6 +865,54 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].reason).toBe('already-encrypted') }) + // An EQL domain installed into a NON-`public` schema is still ciphertext. + // The mangled forms special-case the literal `public`, so without the + // extra alternative this column falls to the plaintext residue and the + // ALTER drops a column full of ciphertext. + it('refuses to rewrite a domain change on a column encrypted in a non-public schema', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "app"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // An ARRAY of an EQL domain is still ciphertext. The trailing delimiter + // lookahead must admit `[` or the column falls to the plaintext residue. + it('refuses to rewrite a domain change on a column encrypted as a domain array', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" public.eql_v3_text_eq[];\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The // column it renamed onto is encrypted, so a later domain change on it is // just as destructive. diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index cf3d0e0be..61acc6009 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -256,8 +256,17 @@ const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` /** * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a * delimiter so a bare domain cannot match a prefix of a longer identifier. + * + * Two shapes {@link MANGLED_TYPE_FORMS} does not enumerate are folded in here so + * the encrypted index recognises them and the ADD+DROP+RENAME cannot drop their + * ciphertext: a domain in ANY quoted schema (`"app"."eql_v3_text_search"`, not + * just the literal `public` the mangled forms special-case), and a `[` in the + * trailing lookahead so an ARRAY of the domain (`public.eql_v3_text_search[]`) + * ends at a delimiter too. Both feed only the encrypted index — never the + * rewrite matcher — so the worst case of admitting one is a flagged statement, + * the same safe asymmetry the fail-closed rule already relies on. */ -const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS}|"[^"]+"\."(?:${ENCRYPTED_DOMAIN})")(?=[\s,;)[]|$)` /** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ const ADD_ENCRYPTED_COLUMN_RE = new RegExp( @@ -328,17 +337,17 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * the fail-closed rule needs no type classification and no SQL parsing, only * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. * - * **The residue claim depends on {@link MANGLED_TYPE_FORMS} covering every + * **The residue claim depends on {@link ENCRYPTED_TYPE_REF} recognising every * encrypted shape.** "By residue, plaintext" is only as good as the encrypted * side's coverage: a declaration {@link ENCRYPTED_TYPE_REF} fails to recognise - * falls to the plaintext residue and gets rewritten. Two forms do this: a - * domain installed into a non-`public` schema (`"email" "app"."eql_v3_text_search"`, - * since the mangled forms only special-case the literal `public` schema), and - * an array of the domain (`ADD COLUMN "email" public.eql_v3_text_search[]`, - * since {@link ENCRYPTED_TYPE_REF}'s trailing delimiter lookahead does not - * include `[`). Neither is a layout EQL installs into or a shape drizzle-kit - * emits, and both behaved identically before this branch — so this is a - * documentation gap, not a regression this branch introduced. + * falls to the plaintext residue and gets rewritten. Two such shapes are now + * covered explicitly by {@link ENCRYPTED_TYPE_REF} — a domain in a non-`public` + * schema (`"email" "app"."eql_v3_text_search"`) and an array of the domain + * (`"email" public.eql_v3_text_search[]`) — because both name ciphertext the + * corpus can see, so rewriting them is the exact drop this rule exists to + * prevent. The dependency itself remains: any encrypted shape a future EQL + * install introduces must be added to {@link ENCRYPTED_TYPE_REF} or it too will + * fall to the plaintext residue. * * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 01804dae6..fb9e08deb 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -795,6 +795,54 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].reason).toBe('already-encrypted') }) + // An EQL domain installed into a NON-`public` schema is still ciphertext. + // The mangled forms special-case the literal `public`, so without the + // extra alternative this column falls to the plaintext residue and the + // ALTER drops a column full of ciphertext. + it('refuses to rewrite a domain change on a column encrypted in a non-public schema', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "app"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // An ARRAY of an EQL domain is still ciphertext. The trailing delimiter + // lookahead must admit `[` or the column falls to the plaintext residue. + it('refuses to rewrite a domain change on a column encrypted as a domain array', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" public.eql_v3_text_eq[];\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The // column it renamed onto is encrypted, so a later domain change on it is // just as destructive. diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 0255df6b5..a8afed6e5 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -264,8 +264,17 @@ const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` /** * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a * delimiter so a bare domain cannot match a prefix of a longer identifier. + * + * Two shapes {@link MANGLED_TYPE_FORMS} does not enumerate are folded in here so + * the encrypted index recognises them and the ADD+DROP+RENAME cannot drop their + * ciphertext: a domain in ANY quoted schema (`"app"."eql_v3_text_search"`, not + * just the literal `public` the mangled forms special-case), and a `[` in the + * trailing lookahead so an ARRAY of the domain (`public.eql_v3_text_search[]`) + * ends at a delimiter too. Both feed only the encrypted index — never the + * rewrite matcher — so the worst case of admitting one is a flagged statement, + * the same safe asymmetry the fail-closed rule already relies on. */ -const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS}|"[^"]+"\."(?:${ENCRYPTED_DOMAIN})")(?=[\s,;)[]|$)` /** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ const ADD_ENCRYPTED_COLUMN_RE = new RegExp( @@ -336,17 +345,17 @@ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( * the fail-closed rule needs no type classification and no SQL parsing, only * the known encrypted list that {@link ENCRYPTED_TYPE_REF} already provides. * - * **The residue claim depends on {@link MANGLED_TYPE_FORMS} covering every + * **The residue claim depends on {@link ENCRYPTED_TYPE_REF} recognising every * encrypted shape.** "By residue, plaintext" is only as good as the encrypted * side's coverage: a declaration {@link ENCRYPTED_TYPE_REF} fails to recognise - * falls to the plaintext residue and gets rewritten. Two forms do this: a - * domain installed into a non-`public` schema (`"email" "app"."eql_v3_text_search"`, - * since the mangled forms only special-case the literal `public` schema), and - * an array of the domain (`ADD COLUMN "email" public.eql_v3_text_search[]`, - * since {@link ENCRYPTED_TYPE_REF}'s trailing delimiter lookahead does not - * include `[`). Neither is a layout EQL installs into or a shape drizzle-kit - * emits, and both behaved identically before this branch — so this is a - * documentation gap, not a regression this branch introduced. + * falls to the plaintext residue and gets rewritten. Two such shapes are now + * covered explicitly by {@link ENCRYPTED_TYPE_REF} — a domain in a non-`public` + * schema (`"email" "app"."eql_v3_text_search"`) and an array of the domain + * (`"email" public.eql_v3_text_search[]`) — because both name ciphertext the + * corpus can see, so rewriting them is the exact drop this rule exists to + * prevent. The dependency itself remains: any encrypted shape a future EQL + * install introduces must be added to {@link ENCRYPTED_TYPE_REF} or it too will + * fall to the plaintext residue. * * **Residue, accepted.** The lookahead is a fixed keyword list, not a parser: * a predicate keyword it does not enumerate (`SIMILAR`, `ISNULL`, `NOTNULL`, From af16e7bbd08c4862e0ac5991be8ab74324eece1a Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 10:43:10 +1000 Subject: [PATCH 069/123] test(cli,wizard): pin describeSkipReason's three reason mappings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit describeSkipReason had zero coverage repo-wide — replacing its body with a constant left every suite green. Its output is the guidance a user acts on when a statement is skipped, and source-unknown's is new. Pin each reason to a distinctive substring so a mis-mapped case is caught. --- .../src/__tests__/rewrite-migrations.test.ts | 42 ++++++++++++++++++- .../src/__tests__/rewrite-migrations.test.ts | 38 +++++++++++++++++ 2 files changed, 79 insertions(+), 1 deletion(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index d6e357ea3..f014cfec6 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -2,7 +2,10 @@ import fs from 'node:fs' import os from 'node:os' import path from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { rewriteEncryptedAlterColumns } from '../commands/db/rewrite-migrations.js' +import { + describeSkipReason, + rewriteEncryptedAlterColumns, +} from '../commands/db/rewrite-migrations.js' describe('rewriteEncryptedAlterColumns', () => { let tmpDir: string @@ -1379,3 +1382,40 @@ describe('rewriteEncryptedAlterColumns', () => { }) }) }) + +// The three reasons drive very different user action — re-encrypt through the +// staged lifecycle, fix a hand-authored cast, or go check the database. A +// switch with no `default` arm means a missing case fails the build, but a +// mis-MAPPED case (wiring `source-unknown` to the `already-encrypted` string) +// compiles fine and ships wrong remediation into a data-loss decision. Pin the +// mapping so that swap is caught. +describe('describeSkipReason', () => { + it('describes already-encrypted as a re-encrypt-through-the-lifecycle action', () => { + const text = describeSkipReason('already-encrypted') + expect(text).toContain('ALREADY encrypted') + expect(text).toContain('DROP the ciphertext') + expect(text).toContain('`stash encrypt` lifecycle') + }) + + it('describes unrecognised-form as a hand-authored / unknown cast', () => { + const text = describeSkipReason('unrecognised-form') + expect(text).toContain('SET DATA TYPE ... USING') + expect(text).toContain('fails at migrate time') + }) + + it('describes source-unknown as a go-check-the-database action', () => { + const text = describeSkipReason('source-unknown') + expect(text).toContain('could not find where this column was declared') + expect(text).toContain("Check the column's current type in the database") + }) + + it('gives each reason a distinct description', () => { + const reasons = [ + 'already-encrypted', + 'unrecognised-form', + 'source-unknown', + ] as const + const described = reasons.map(describeSkipReason) + expect(new Set(described).size).toBe(reasons.length) + }) +}) diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index fb9e08deb..22d106929 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -3,6 +3,7 @@ import os from 'node:os' import path from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { + describeSkipReason, rewriteEncryptedAlterColumns, sweepMigrationDirs, } from '../lib/rewrite-migrations.js' @@ -1452,3 +1453,40 @@ describe('sweepMigrationDirs', () => { expect(updated).not.toContain('DROP COLUMN') }) }) + +// The three reasons drive very different user action — re-encrypt through the +// staged lifecycle, fix a hand-authored cast, or go check the database. A +// switch with no `default` arm means a missing case fails the build, but a +// mis-MAPPED case (wiring `source-unknown` to the `already-encrypted` string) +// compiles fine and ships wrong remediation into a data-loss decision. Pin the +// mapping so that swap is caught. +describe('describeSkipReason', () => { + it('describes already-encrypted as a re-encrypt-through-the-lifecycle action', () => { + const text = describeSkipReason('already-encrypted') + expect(text).toContain('ALREADY encrypted') + expect(text).toContain('DROP the ciphertext') + expect(text).toContain('`stash encrypt` lifecycle') + }) + + it('describes unrecognised-form as a hand-authored / unknown cast', () => { + const text = describeSkipReason('unrecognised-form') + expect(text).toContain('SET DATA TYPE ... USING') + expect(text).toContain('fails at migrate time') + }) + + it('describes source-unknown as a go-check-the-database action', () => { + const text = describeSkipReason('source-unknown') + expect(text).toContain('could not find where this column was declared') + expect(text).toContain("Check the column's current type in the database") + }) + + it('gives each reason a distinct description', () => { + const reasons = [ + 'already-encrypted', + 'unrecognised-form', + 'source-unknown', + ] as const + const described = reasons.map(describeSkipReason) + expect(new Set(described).size).toBe(reasons.length) + }) +}) From 71d2d1855d75c689c312f0ca6bb354e7fceacd11 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 10:52:27 +1000 Subject: [PATCH 070/123] test(cli,wizard): cover the sweep's command and multi-directory layers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sweep's unit tests are thorough; everything above them was thin. - The clack log mock in migration.test.ts and generate-drizzle-migration.test.ts omitted `step`, so the per-statement report threw and landed in the command's catch — no test ever saw the report block. Add `step`. - eql migration: assert source-unknown guidance reaches the user and the closing warning fires for an undeclared column. - db install (generateDrizzleMigration): its step-5 sweep block had zero tests — every other test's out dir held only the skipped generated file. Add a genuine rewrite and a source-unknown skip. - wizard post-agent: harden the clean default-to-Yes arm to assert it carries no warning phrase, and add two multi-directory tests that put the actionable migration in a non-first candidate, so shrinking DRIZZLE_OUT_DIRS or short-circuiting the aggregation loop fails. --- .../generate-drizzle-migration.test.ts | 75 ++++++++++++++++- .../commands/eql/__tests__/migration.test.ts | 49 ++++++++++- .../wizard/src/__tests__/post-agent.test.ts | 82 +++++++++++++++++++ 3 files changed, 204 insertions(+), 2 deletions(-) diff --git a/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts b/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts index 3bfc03804..cf4f085e8 100644 --- a/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts +++ b/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts @@ -17,7 +17,15 @@ import { messages } from '../../../messages.js' // through. The spinner instance doubles as the `s` argument. const clack = vi.hoisted(() => ({ spinnerInstance: { start: vi.fn(), stop: vi.fn() }, - log: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), success: vi.fn() }, + // `step` is on the real clack `log`; the sweep's per-statement report calls + // it, so it must be present or the report throws into the catch unseen. + log: { + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + success: vi.fn(), + step: vi.fn(), + }, intro: vi.fn(), note: vi.fn(), outro: vi.fn(), @@ -165,6 +173,71 @@ describe('generateDrizzleMigration', () => { expect(written).toContain('cs_migrations') }) + // Step 5 of the generator sweeps sibling migrations. This block had no test at + // all: every other test's out dir contains only the file the generator wrote + // (which is `skip`ped), so the sweep never had anything to act on. A sibling + // ALTER on a plaintext column the corpus declares must be rewritten. + it('rewrites a sibling ALTER on a declared plaintext column', async () => { + const out = join(tmp, 'drizzle') + mkdirSync(out, { recursive: true }) + writeFileSync( + join(out, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + const sibling = join(out, '0001_encrypt-email.sql') + writeFileSync( + sibling, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n', + ) + spawnMock.mockImplementation(() => { + writeFileSync(join(out, '0002_install-eql.sql'), '') + return { status: 0, stdout: '', stderr: '' } + }) + + await generateDrizzleMigration(spinner, { out }) + + const rewritten = readFileSync(sibling, 'utf-8') + expect(rewritten).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(rewritten).not.toContain('SET DATA TYPE') + const info = clack.log.info.mock.calls.map((c) => String(c[0])) + expect(info.some((msg) => msg.includes('Rewrote 1 migration file'))).toBe( + true, + ) + }) + + // A sibling ALTER on a column the corpus never declares is fail-closed to + // `source-unknown`: left on disk and reported with its remediation. Exercises + // the skip branch of the step-5 report, which no db-install test reached. + it('reports source-unknown for a sibling ALTER on an undeclared column', async () => { + const out = join(tmp, 'drizzle') + mkdirSync(out, { recursive: true }) + const sibling = join(out, '0001_encrypt-email.sql') + const unsafeAlter = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n' + writeFileSync(sibling, unsafeAlter) + spawnMock.mockImplementation(() => { + writeFileSync(join(out, '0002_install-eql.sql'), '') + return { status: 0, stdout: '', stderr: '' } + }) + + await generateDrizzleMigration(spinner, { out }) + + // Never rewritten — its current type is unknown and could be ciphertext. + expect(readFileSync(sibling, 'utf-8')).toBe(unsafeAlter) + const stepped = clack.log.step.mock.calls.map((c) => String(c[0])) + expect( + stepped.some((msg) => + msg.includes("Check the column's current type in the database"), + ), + ).toBe(true) + const warned = clack.log.warn.mock.calls.map((c) => String(c[0])) + expect(warned.some((msg) => msg.includes('did not fully complete'))).toBe( + true, + ) + }) + it('includes --out in the dry-run preview', async () => { const out = join(tmp, 'custom-out') await generateDrizzleMigration(spinner, { dryRun: true, out }) diff --git a/packages/cli/src/commands/eql/__tests__/migration.test.ts b/packages/cli/src/commands/eql/__tests__/migration.test.ts index 9f0fa1f90..c0193b665 100644 --- a/packages/cli/src/commands/eql/__tests__/migration.test.ts +++ b/packages/cli/src/commands/eql/__tests__/migration.test.ts @@ -18,7 +18,16 @@ import { buildEqlV3MigrationSql, eqlMigrationCommand } from '../migration.js' // reports through. const clack = vi.hoisted(() => ({ spinnerInstance: { start: vi.fn(), stop: vi.fn() }, - log: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), success: vi.fn() }, + // `step` is on the real clack `log`; omitting it made the sweep's + // per-statement report throw and land in the command's catch block, so no + // test ever saw the report. Keep it here. + log: { + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + success: vi.fn(), + step: vi.fn(), + }, intro: vi.fn(), note: vi.fn(), outro: vi.fn(), @@ -313,6 +322,44 @@ describe('eqlMigrationCommand — Drizzle', () => { ) }) + // A column the corpus never declares is fail-closed to `source-unknown`: left + // on disk, and reported so the user goes and checks its type. This is the most + // common skip on a real (e.g. squashed) corpus, and it renders through the + // `p.log.step` path that a missing mock method used to make throw — so assert + // the guidance text actually reaches the user, not just that a warning fired. + it('reports source-unknown guidance for an undeclared column and warns at the close', async () => { + const out = join(tmp, 'drizzle') + mkdirSync(out, { recursive: true }) + // No CREATE TABLE / ADD COLUMN anywhere declares "users"."email". + const sibling = join(out, '0001_encrypt-email.sql') + const unsafeAlter = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n' + writeFileSync(sibling, unsafeAlter) + spawnMock.mockImplementation(() => { + writeFileSync(join(out, '0002_install-eql.sql'), '') + return { status: 0, stdout: '', stderr: '' } + }) + + await eqlMigrationCommand({ drizzle: true, out }) + + // Left exactly as written — a source-unknown statement is never rewritten. + expect(readFileSync(sibling, 'utf-8')).toBe(unsafeAlter) + // The per-statement report reached the user with the source-unknown + // remediation (the whole point of the reason), not a crash into the catch. + const stepped = clack.log.step.mock.calls.map((c) => String(c[0])) + expect(stepped.some((msg) => msg.includes(sibling))).toBe(true) + expect( + stepped.some((msg) => + msg.includes("Check the column's current type in the database"), + ), + ).toBe(true) + // And the closing note warns the sweep did not fully complete. + const warned = clack.log.warn.mock.calls.map((c) => String(c[0])) + expect(warned.some((msg) => msg.includes('did not fully complete'))).toBe( + true, + ) + }) + it('aborts (exit 1) when drizzle-kit exits non-zero', async () => { spawnMock.mockReturnValue({ status: 1, stdout: '', stderr: 'boom' }) await expect( diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index 8e7656d9f..f8f87a791 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -155,6 +155,15 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] expect(options?.initialValue).toBe(true) + // The clean arm is the last of a 4-way nested ternary; the negative + // assertions that guard the other arms' wording never run against this one. + // A mis-nested arm here would ship a destruction/flag/unchecked warning to a + // user with nothing to sweep, and asserting only `initialValue` would miss + // it — so pin that the clean message carries none of those phrases. + const message = String(options?.message) + expect(message).not.toContain('DESTROYS data') + expect(message).not.toContain('flagged for review') + expect(message).not.toContain('could not check') }) it('defaults to No, and says why, when a file was rewritten', async () => { @@ -247,4 +256,77 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { expect(options?.initialValue).toBe(false) expect(String(options?.message)).toContain('could not check 1 directory') }) + + // The wizard ships scanning drizzle/, migrations/ and src/db/migrations/ and + // indexes each SEPARATELY — the per-directory index is the mechanism the + // fail-closed rule exists to make safe. Every test above uses only drizzle/, + // so shrinking the shipped constant to ['drizzle'], or short-circuiting the + // aggregation loop after the first directory, leaves them all green. These two + // put the actionable content in a NON-first directory so those regressions + // fail. + it('sweeps a candidate directory other than the first', async () => { + // drizzle/ exists but has nothing to do; the rewrite lives in migrations/. + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_init.sql'), + 'CREATE TABLE "widgets" ("id" integer PRIMARY KEY);\n', + ) + fs.mkdirSync(path.join(cwd, 'migrations')) + fs.writeFileSync( + path.join(cwd, 'migrations', '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + fs.writeFileSync( + path.join(cwd, 'migrations', '0001_encrypt.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + await runDrizzle() + + // The migrations/ ALTER was rewritten — proof the second directory was + // swept, not just drizzle/. + const swept = fs.readFileSync( + path.join(cwd, 'migrations', '0001_encrypt.sql'), + 'utf-8', + ) + expect(swept).toContain('DROP COLUMN') + expect(swept).not.toContain('SET DATA TYPE') + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('DESTROYS data') + }) + + it('aggregates a rewrite in one directory with a flag in another', async () => { + // drizzle/ declares email, so its ALTER is rewritten. + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0001_encrypt.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + // migrations/ never declares its column, so its ALTER is source-unknown. + fs.mkdirSync(path.join(cwd, 'migrations')) + const flagged = path.join(cwd, 'migrations', '0001_encrypt.sql') + const flaggedSql = + 'ALTER TABLE "orders" ALTER COLUMN "total" SET DATA TYPE eql_v3_text_search;\n' + fs.writeFileSync(flagged, flaggedSql) + + await runDrizzle() + + // drizzle/ was rewritten... + const rewritten = fs.readFileSync( + path.join(cwd, 'drizzle', '0001_encrypt.sql'), + 'utf-8', + ) + expect(rewritten).toContain('DROP COLUMN') + // ...and migrations/ was left untouched, flagged rather than rewritten. + expect(fs.readFileSync(flagged, 'utf-8')).toBe(flaggedSql) + // A rewrite anywhere makes the prompt destructive and defaults it to No. + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('DESTROYS data') + }) }) From 3ab3925596a6f57f66acd03ec7760e5d9bcb2058 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 12:06:15 +1000 Subject: [PATCH 071/123] fix(cli,wizard): index the rewrite's own target, and resolve the implicit schema MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two ways a corpus could still lose ciphertext to the ADD+DROP+RENAME, both found by the #772 review. Neither is caught by the fail-closed `declared` rule, because in both the column really is declared — as plaintext, by the original CREATE TABLE. Finding 3 — chained conversions. `indexColumnDeclarations` reads CREATE TABLE, ADD COLUMN and RENAME, but never the strict matcher's own target. A corpus carrying an earlier `SET DATA TYPE eql_v2_encrypted` (a stack version that predates this sweep) left the column looking plaintext, so a later domain change dropped its ciphertext. Record the conversion as the sweep walks, in file-sorted order — not in the corpus-wide index, which has no order and would flag the legitimate first conversion too. Finding 4 — `"users"` and `"public"."users"` are the same table; Postgres resolves the unqualified name through `search_path`. drizzle-kit emits unqualified, while hand-written SQL and this sweep's own renderSafeAlter output are qualified, so a corpus mixing the two split one column across two keys: the already-encrypted guard missed, and the ALTER dropped ciphertext. Collapse the implicit schema in `columnKey`. Only that one — `"app"."users"` stays distinct, and there is no case folding, since TABLE_REF matches quoted identifiers only. Both fixes land in both copies of the rewriter. 7 tests each: the two destructive cases, the wrong-reason case finding 4 also produced (`source-unknown` for a column declared two files up), and four guards for the ways a naive fix breaks — the first conversion in a chain, a chain across different columns, a commented-out earlier ALTER, and a real non-public schema. --- .../src/__tests__/rewrite-migrations.test.ts | 182 ++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 42 +++- .../src/__tests__/rewrite-migrations.test.ts | 182 ++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 42 +++- 4 files changed, 444 insertions(+), 4 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index f014cfec6..8f999f851 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -1108,6 +1108,188 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toHaveLength(1) expect(skipped[0].reason).toBe('already-encrypted') }) + + // #772 review, finding 3. The index reads CREATE TABLE, ADD COLUMN and + // RENAME, but never the strict matcher's OWN target. So a corpus where an + // earlier migration already converted the column — realistic wherever a + // previous stack version's ALTER ran before this sweep existed — leaves + // that column looking plaintext, and the SECOND conversion rewrites a + // column that now holds ciphertext. `declared` cannot catch it: the column + // IS declared, as plaintext, by the original CREATE TABLE. + it('rewrites only the first conversion when two ALTERs chain on one column', async () => { + declarePlaintext('"users"', 'email') + const first = path.join(tmpDir, '0002_v2.sql') + fs.writeFileSync( + first, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;\n', + ) + const secondSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const second = path.join(tmpDir, '0003_v3.sql') + fs.writeFileSync(second, `${secondSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + // The first conversion is the legitimate plaintext -> encrypted one and + // must still happen — flagging it too would be the naive fix. + expect(rewritten).toEqual([first]) + expect(fs.readFileSync(first, 'utf-8')).toContain('DROP COLUMN') + // The second targets ciphertext. Left byte-identical on disk. + expect(fs.readFileSync(second, 'utf-8')).toBe(`${secondSql}\n`) + expect(skipped).toEqual([ + { file: second, statement: secondSql, reason: 'already-encrypted' }, + ]) + }) + + it('flags the second of two chained ALTERs inside a single file', async () => { + declarePlaintext('"users"', 'email') + const secondSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0002_chain.sql') + fs.writeFileSync( + filePath, + [ + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;', + secondSql, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + // Exactly one conversion was applied, and the v3 statement survives verbatim. + expect(updated.match(/DROP COLUMN/g)?.length).toBe(1) + expect(updated).toContain(secondSql) + expect(skipped).toEqual([ + { file: filePath, statement: secondSql, reason: 'already-encrypted' }, + ]) + }) + + // A chain on DIFFERENT columns is not a chain — each is its own first + // conversion and both must be rewritten. + it('rewrites both when chained ALTERs target different columns', async () => { + declarePlaintext('"users"', 'email', 'name') + const first = path.join(tmpDir, '0002_email.sql') + fs.writeFileSync( + first, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + const second = path.join(tmpDir, '0003_name.sql') + fs.writeFileSync( + second, + 'ALTER TABLE "users" ALTER COLUMN "name" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([first, second]) + expect(skipped).toEqual([]) + }) + + // A commented-out first conversion never ran, so the column is still + // plaintext and the live second conversion is the legitimate one. + it('does not treat a commented-out earlier ALTER as having encrypted the column', async () => { + declarePlaintext('"users"', 'email') + fs.writeFileSync( + path.join(tmpDir, '0002_v2.sql'), + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;\n', + ) + const live = path.join(tmpDir, '0003_v3.sql') + fs.writeFileSync( + live, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([live]) + expect(fs.readFileSync(live, 'utf-8')).toContain('DROP COLUMN') + }) + + // #772 review, finding 4. `columnKey` keys on the schema exactly as written, + // but Postgres resolves an unqualified name through `search_path` — so + // `"users"` and `"public"."users"` are the SAME table. drizzle-kit emits + // unqualified; hand-written SQL and this sweep's own output are qualified. + // A corpus mixing the two split the index across two keys. + it('treats "public"."users" and "users" as the same table when checking encryption', async () => { + // 0000 declares the column unqualified — drizzle-kit's default output. + fs.writeFileSync( + path.join(tmpDir, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + // 0001 is a previous sweep's output, schema-qualified. The RENAME leaves + // `email` holding ciphertext, recorded under the QUALIFIED key. + fs.writeFileSync( + path.join(tmpDir, '0001_encrypt.sql'), + [ + 'ALTER TABLE "public"."users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + 'ALTER TABLE "public"."users" DROP COLUMN "email";', + 'ALTER TABLE "public"."users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + // 0002 changes the domain, written unqualified. + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // The mirror direction: declared/encrypted unqualified, altered qualified. + // Before the fix this reported `source-unknown` — a WRONG reason, telling + // the user the corpus never declares a column it declares two files up. + it('resolves an unqualified declaration against a schema-qualified ALTER', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" ("email" "public"."eql_v3_text_eq");\n', + ) + const alterSql = + 'ALTER TABLE "public"."users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // Only the IMPLICIT schema collapses. A real non-public schema is a + // genuinely different table and must not be conflated with the bare name. + it('does not conflate a non-public schema with the unqualified table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_app.sql'), + 'CREATE TABLE "app"."users" ("email" "public"."eql_v3_text_eq");\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_public.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + const alter = path.join(tmpDir, '0002_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + // public.users.email is plaintext — app.users.email being encrypted is + // about a different table entirely. + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) }) it('handles multiple ALTER statements in one file', async () => { diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 61acc6009..2d027f0d9 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -389,9 +389,29 @@ function tableOf( : { schema: first, table: second } } -/** Identity of a column across the corpus, for {@link indexColumnDeclarations}. */ +/** + * Identity of a column across the corpus, for {@link indexColumnDeclarations}. + * + * `public` and no schema at all are the SAME table: Postgres resolves an + * unqualified name through `search_path`, which starts at `public`. The two + * spellings coexist in one corpus routinely — drizzle-kit emits unqualified, + * while hand-written SQL and this sweep's own {@link renderSafeAlter} output + * are qualified. Keying on the literal text split one column across two keys, + * so a column already encrypted under one spelling looked plaintext when + * altered under the other and the ADD+DROP+RENAME dropped its ciphertext + * (#772 review, finding 4). + * + * Only the IMPLICIT schema collapses. `"app"."users"` is a genuinely different + * table from `"users"` and keeps its own key. No case folding either: + * `TABLE_REF` matches quoted identifiers only, and those are case-sensitive in + * Postgres — `"PUBLIC"` really is a different schema from `"public"`. + */ function columnKey(table: string, column: string, schema?: string): string { - return JSON.stringify([schema ?? '', table, column]) + return JSON.stringify([ + schema === 'public' ? '' : (schema ?? ''), + table, + column, + ]) } /** What the migration corpus says about the columns it mentions. */ @@ -657,6 +677,24 @@ export async function rewriteEncryptedAlterColumns( // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. if (!domain) return match + + // This statement converts the column, so from here on in the corpus it + // holds CIPHERTEXT. `indexColumnDeclarations` cannot know that: it + // reads CREATE TABLE, ADD COLUMN and RENAME, never the strict matcher's + // own target. Without this, a corpus carrying an earlier conversion + // (`... SET DATA TYPE eql_v2_encrypted` from a stack version that + // predates this sweep) leaves the column looking plaintext, and the + // NEXT domain change drops the ciphertext — `declared` cannot catch it, + // because the column really is declared, as plaintext, by the original + // CREATE TABLE. + // + // Recorded here rather than in the corpus-wide index on purpose: the + // index has no order, so it would flag THIS conversion — the legitimate + // plaintext -> encrypted one — as already-encrypted too. Files are + // walked in sorted order and matches within a file in source order, so + // "already converted" means "converted by a statement that runs before + // this one". + encryptedColumns.add(columnKey(table, column, schema)) return renderSafeAlter(table, column, domain, schema) }, ) diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 22d106929..3a1b189c7 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -1036,6 +1036,188 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toHaveLength(1) expect(skipped[0].reason).toBe('already-encrypted') }) + + // #772 review, finding 3. The index reads CREATE TABLE, ADD COLUMN and + // RENAME, but never the strict matcher's OWN target. So a corpus where an + // earlier migration already converted the column — realistic wherever a + // previous stack version's ALTER ran before this sweep existed — leaves + // that column looking plaintext, and the SECOND conversion rewrites a + // column that now holds ciphertext. `declared` cannot catch it: the column + // IS declared, as plaintext, by the original CREATE TABLE. + it('rewrites only the first conversion when two ALTERs chain on one column', async () => { + declarePlaintext('"users"', 'email') + const first = path.join(tmpDir, '0002_v2.sql') + fs.writeFileSync( + first, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;\n', + ) + const secondSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const second = path.join(tmpDir, '0003_v3.sql') + fs.writeFileSync(second, `${secondSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + // The first conversion is the legitimate plaintext -> encrypted one and + // must still happen — flagging it too would be the naive fix. + expect(rewritten).toEqual([first]) + expect(fs.readFileSync(first, 'utf-8')).toContain('DROP COLUMN') + // The second targets ciphertext. Left byte-identical on disk. + expect(fs.readFileSync(second, 'utf-8')).toBe(`${secondSql}\n`) + expect(skipped).toEqual([ + { file: second, statement: secondSql, reason: 'already-encrypted' }, + ]) + }) + + it('flags the second of two chained ALTERs inside a single file', async () => { + declarePlaintext('"users"', 'email') + const secondSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const filePath = path.join(tmpDir, '0002_chain.sql') + fs.writeFileSync( + filePath, + [ + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;', + secondSql, + '', + ].join('\n'), + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + const updated = fs.readFileSync(filePath, 'utf-8') + // Exactly one conversion was applied, and the v3 statement survives verbatim. + expect(updated.match(/DROP COLUMN/g)?.length).toBe(1) + expect(updated).toContain(secondSql) + expect(skipped).toEqual([ + { file: filePath, statement: secondSql, reason: 'already-encrypted' }, + ]) + }) + + // A chain on DIFFERENT columns is not a chain — each is its own first + // conversion and both must be rewritten. + it('rewrites both when chained ALTERs target different columns', async () => { + declarePlaintext('"users"', 'email', 'name') + const first = path.join(tmpDir, '0002_email.sql') + fs.writeFileSync( + first, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + const second = path.join(tmpDir, '0003_name.sql') + fs.writeFileSync( + second, + 'ALTER TABLE "users" ALTER COLUMN "name" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([first, second]) + expect(skipped).toEqual([]) + }) + + // A commented-out first conversion never ran, so the column is still + // plaintext and the live second conversion is the legitimate one. + it('does not treat a commented-out earlier ALTER as having encrypted the column', async () => { + declarePlaintext('"users"', 'email') + fs.writeFileSync( + path.join(tmpDir, '0002_v2.sql'), + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v2_encrypted;\n', + ) + const live = path.join(tmpDir, '0003_v3.sql') + fs.writeFileSync( + live, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([live]) + expect(fs.readFileSync(live, 'utf-8')).toContain('DROP COLUMN') + }) + + // #772 review, finding 4. `columnKey` keys on the schema exactly as written, + // but Postgres resolves an unqualified name through `search_path` — so + // `"users"` and `"public"."users"` are the SAME table. drizzle-kit emits + // unqualified; hand-written SQL and this sweep's own output are qualified. + // A corpus mixing the two split the index across two keys. + it('treats "public"."users" and "users" as the same table when checking encryption', async () => { + // 0000 declares the column unqualified — drizzle-kit's default output. + fs.writeFileSync( + path.join(tmpDir, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + // 0001 is a previous sweep's output, schema-qualified. The RENAME leaves + // `email` holding ciphertext, recorded under the QUALIFIED key. + fs.writeFileSync( + path.join(tmpDir, '0001_encrypt.sql'), + [ + 'ALTER TABLE "public"."users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + 'ALTER TABLE "public"."users" DROP COLUMN "email";', + 'ALTER TABLE "public"."users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + // 0002 changes the domain, written unqualified. + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // The mirror direction: declared/encrypted unqualified, altered qualified. + // Before the fix this reported `source-unknown` — a WRONG reason, telling + // the user the corpus never declares a column it declares two files up. + it('resolves an unqualified declaration against a schema-qualified ALTER', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" ("email" "public"."eql_v3_text_eq");\n', + ) + const alterSql = + 'ALTER TABLE "public"."users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // Only the IMPLICIT schema collapses. A real non-public schema is a + // genuinely different table and must not be conflated with the bare name. + it('does not conflate a non-public schema with the unqualified table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_app.sql'), + 'CREATE TABLE "app"."users" ("email" "public"."eql_v3_text_eq");\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_public.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + const alter = path.join(tmpDir, '0002_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + // public.users.email is plaintext — app.users.email being encrypted is + // about a different table entirely. + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) }) it('handles multiple ALTER statements in one file', async () => { diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index a8afed6e5..6cea3886b 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -397,9 +397,29 @@ function tableOf( : { schema: first, table: second } } -/** Identity of a column across the corpus, for {@link indexColumnDeclarations}. */ +/** + * Identity of a column across the corpus, for {@link indexColumnDeclarations}. + * + * `public` and no schema at all are the SAME table: Postgres resolves an + * unqualified name through `search_path`, which starts at `public`. The two + * spellings coexist in one corpus routinely — drizzle-kit emits unqualified, + * while hand-written SQL and this sweep's own {@link renderSafeAlter} output + * are qualified. Keying on the literal text split one column across two keys, + * so a column already encrypted under one spelling looked plaintext when + * altered under the other and the ADD+DROP+RENAME dropped its ciphertext + * (#772 review, finding 4). + * + * Only the IMPLICIT schema collapses. `"app"."users"` is a genuinely different + * table from `"users"` and keeps its own key. No case folding either: + * `TABLE_REF` matches quoted identifiers only, and those are case-sensitive in + * Postgres — `"PUBLIC"` really is a different schema from `"public"`. + */ function columnKey(table: string, column: string, schema?: string): string { - return JSON.stringify([schema ?? '', table, column]) + return JSON.stringify([ + schema === 'public' ? '' : (schema ?? ''), + table, + column, + ]) } /** What the migration corpus says about the columns it mentions. */ @@ -665,6 +685,24 @@ export async function rewriteEncryptedAlterColumns( // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. if (!domain) return match + + // This statement converts the column, so from here on in the corpus it + // holds CIPHERTEXT. `indexColumnDeclarations` cannot know that: it + // reads CREATE TABLE, ADD COLUMN and RENAME, never the strict matcher's + // own target. Without this, a corpus carrying an earlier conversion + // (`... SET DATA TYPE eql_v2_encrypted` from a stack version that + // predates this sweep) leaves the column looking plaintext, and the + // NEXT domain change drops the ciphertext — `declared` cannot catch it, + // because the column really is declared, as plaintext, by the original + // CREATE TABLE. + // + // Recorded here rather than in the corpus-wide index on purpose: the + // index has no order, so it would flag THIS conversion — the legitimate + // plaintext -> encrypted one — as already-encrypted too. Files are + // walked in sorted order and matches within a file in source order, so + // "already converted" means "converted by a statement that runs before + // this one". + encryptedColumns.add(columnKey(table, column, schema)) return renderSafeAlter(table, column, domain, schema) }, ) From cc84183385d80aec793d82e7e364ea0815244909 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:05:59 +1000 Subject: [PATCH 072/123] fix(cli,wizard): track dollar-quoted bodies and E'' escapes when scanning for comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quote parity is a whole-file property. Anything that makes the scanner disagree with Postgres about where a literal ENDS shifts every token after it — so a commented-out ALTER downstream reads as live and is rewritten into a live DROP COLUMN. `isInsideCommentOrString` left two such gaps. A `$$ … $$` body was untracked on the stated grounds that mis-reading one can only make us SKIP a rewrite; that holds for an unterminated literal but not for an over-terminated one, and an odd apostrophe count inside the body (`$$ don't $$` — a function body, a comment, any prose) ends a literal EARLY. And `endOfQuoted` knew only the doubled-delimiter escape, so `E'a\'b'` read as closing at the escaped quote. Skip dollar-quoted bodies whole, and give `endOfQuoted` a backslashEscapes mode selected when the opening quote is an `E''` prefix (guarded so an identifier merely ending in `e` does not trigger it). Both copies. Four tests each. The two destructive shapes, a tagged `$fn$` variant, and a guard that a live ALTER *following* a dollar-quoted body is still rewritten — that one failed before this change too, in the over-skip direction the old docblock had assumed was the only risk. --- .../src/__tests__/rewrite-migrations.test.ts | 92 +++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 73 +++++++++++++-- .../src/__tests__/rewrite-migrations.test.ts | 92 +++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 73 +++++++++++++-- 4 files changed, 318 insertions(+), 12 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 8f999f851..c8090b216 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -819,6 +819,98 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) }) + + // #772 review, finding 1. Quote parity is a whole-file property: anything + // that makes the scanner disagree with Postgres about where a literal ENDS + // shifts every following token, so a commented-out ALTER downstream reads + // as live and is rewritten into a live DROP COLUMN. + // + // The two shapes below both do that. A `$$ … $$` body was documented as + // safe on the grounds that mis-reading one can only make us SKIP — that + // holds for an unterminated literal, but an odd apostrophe count inside the + // body ends a literal EARLY, which is the opposite direction. + it('leaves a commented-out ALTER alone after a dollar-quoted body with an odd apostrophe count', async () => { + declarePlaintext('"users"', 'email') + const original = [ + "CREATE FUNCTION note() RETURNS text AS $$ don't $$ LANGUAGE sql;", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_dollar-quoted.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves a commented-out ALTER alone after a tagged dollar-quoted body', async () => { + declarePlaintext('"users"', 'email') + const original = [ + "CREATE FUNCTION note() RETURNS text AS $fn$ don't $fn$ LANGUAGE sql;", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_tagged-dollar.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + // A backslash-escaped quote inside an E'' string does not close it. Reading + // it as a close shifts parity for the rest of the file the same way. + it('leaves a commented-out ALTER alone after an E-string with a backslash-escaped quote', async () => { + declarePlaintext('"users"', 'email') + const original = [ + 'CREATE TABLE "notes" ("body" text);', + '--> statement-breakpoint', + "INSERT INTO \"notes\" VALUES (E'a\\'b');", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_estring.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // The dollar-quote skip must not swallow live SQL: a `$$` body sitting + // BEFORE a genuine plaintext ALTER still leaves that ALTER rewritable. + it('still rewrites a live ALTER that follows a dollar-quoted body', async () => { + declarePlaintext('"users"', 'email') + const filePath = path.join(tmpDir, '0033_dollar-then-live.sql') + fs.writeFileSync( + filePath, + [ + "CREATE FUNCTION note() RETURNS text AS $$ don't $$ LANGUAGE sql;", + '--> statement-breakpoint', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain('DROP COLUMN') + }) }) // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 2d027f0d9..dae19ab83 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -176,9 +176,15 @@ function trimStatementPreamble(statement: string): string { * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in * an identifier) is an escape and does not close the token. * - * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a - * comment/literal here, which can only make us skip a rewrite (the statement - * then fails loudly at migrate time), never perform a destructive one. + * Dollar-quoted bodies (`$$ … $$`, `$fn$ … $fn$`) are skipped whole, and an + * `E''` literal gives backslash its escape meaning. Both are quote-PARITY + * concerns, not merely "we might skip a rewrite": an apostrophe inside an + * untracked `$$` body, or a `\'` read as a close, ends a literal EARLY, and + * every token after it is then misread — including a commented-out ALTER, which + * reads as live and gets rewritten into a live DROP COLUMN. That is the same + * failure mode as the apostrophe-in-identifier case below, and it is why the + * earlier reasoning here ("can only make us skip") was wrong: it holds for an + * UNDER-terminated literal, not an over-terminated one (#772 review, finding 1). */ function isInsideCommentOrString(sql: string, index: number): boolean { let i = 0 @@ -206,6 +212,20 @@ function isInsideCommentOrString(sql: string, index: number): boolean { } if (j > index) return true i = j + } else if (sql[i] === '$') { + // A `$tag$ … $tag$` body is opaque: its content is not SQL, so a `'` or + // `--` inside it means nothing. Skipping the whole body is what keeps the + // rest of the file's quote parity aligned with Postgres. + const delimiter = dollarQuoteDelimiter(sql, i) + if (delimiter === undefined) { + i += 1 + } else { + const close = sql.indexOf(delimiter, i + delimiter.length) + // Unterminated: inert to the end of the file, like the cases below. + const end = close === -1 ? sql.length : close + delimiter.length + if (end > index) return true + i = end + } } else if (sql[i] === '"') { // A quoted identifier is live SQL, but its body is not: consuming it here // — before the `'` branch below — is what stops an apostrophe inside one @@ -219,7 +239,7 @@ function isInsideCommentOrString(sql: string, index: number): boolean { if (end > index) return true i = end } else if (sql[i] === "'") { - const end = endOfQuoted(sql, i, "'") + const end = endOfQuoted(sql, i, "'", isEscapeStringOpener(sql, i)) // Unterminated, or the literal runs past `index`: `index` is inside a // string literal, which is every bit as inert as a comment. if (end > index) return true @@ -231,15 +251,56 @@ function isInsideCommentOrString(sql: string, index: number): boolean { return false } +/** + * The `$tag$` delimiter opening at `open`, or `undefined` when what sits there + * is just a `$`. The tag is empty (`$$`) or an SQL identifier (`$fn$`); a digit + * cannot start one, so a positional parameter (`$1`) is never mistaken for a + * dollar-quote opener. + */ +function dollarQuoteDelimiter(sql: string, open: number): string | undefined { + DOLLAR_QUOTE_OPEN_RE.lastIndex = open + return DOLLAR_QUOTE_OPEN_RE.exec(sql)?.[0] +} + +/** Sticky, so the scan never has to slice the file to test one position. */ +const DOLLAR_QUOTE_OPEN_RE = /\$(?:[A-Za-z_][A-Za-z0-9_]*)?\$/y + +/** + * Whether the `'` at `open` starts an `E''` escape-string, in which a backslash + * escapes the following character. Plain literals give backslash no special + * meaning under the default `standard_conforming_strings = on`. + * + * The character before the `E` must not be part of an identifier, so a column + * or type name that merely ends in `e` does not turn the literal after it into + * an escape-string. + */ +function isEscapeStringOpener(sql: string, open: number): boolean { + const prefix = sql[open - 1] + if (prefix !== 'E' && prefix !== 'e') return false + return !/[A-Za-z0-9_]/.test(sql[open - 2] ?? '') +} + /** * The index just past the `quote`-delimited token that opens at `open`, or * `sql.length` when it is never closed. A doubled delimiter inside the token is * an escaped one (`''`, `""`) and does not end it. + * + * With `backslashEscapes`, a `\` also escapes the next character — the `E''` + * form. Getting this wrong ends the literal EARLY, which shifts quote parity + * for the whole rest of the file and can make a commented-out ALTER downstream + * read as live (#772 review, finding 1). */ -function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { +function endOfQuoted( + sql: string, + open: number, + quote: "'" | '"', + backslashEscapes = false, +): number { let i = open + 1 while (i < sql.length) { - if (sql[i] !== quote) { + if (backslashEscapes && sql[i] === '\\') { + i += 2 + } else if (sql[i] !== quote) { i += 1 } else if (sql[i + 1] === quote) { i += 2 diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 3a1b189c7..4d9bfea8a 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -747,6 +747,98 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) }) + + // #772 review, finding 1. Quote parity is a whole-file property: anything + // that makes the scanner disagree with Postgres about where a literal ENDS + // shifts every following token, so a commented-out ALTER downstream reads + // as live and is rewritten into a live DROP COLUMN. + // + // The two shapes below both do that. A `$$ … $$` body was documented as + // safe on the grounds that mis-reading one can only make us SKIP — that + // holds for an unterminated literal, but an odd apostrophe count inside the + // body ends a literal EARLY, which is the opposite direction. + it('leaves a commented-out ALTER alone after a dollar-quoted body with an odd apostrophe count', async () => { + declarePlaintext('"users"', 'email') + const original = [ + "CREATE FUNCTION note() RETURNS text AS $$ don't $$ LANGUAGE sql;", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_dollar-quoted.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves a commented-out ALTER alone after a tagged dollar-quoted body', async () => { + declarePlaintext('"users"', 'email') + const original = [ + "CREATE FUNCTION note() RETURNS text AS $fn$ don't $fn$ LANGUAGE sql;", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_tagged-dollar.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + // A backslash-escaped quote inside an E'' string does not close it. Reading + // it as a close shifts parity for the rest of the file the same way. + it('leaves a commented-out ALTER alone after an E-string with a backslash-escaped quote', async () => { + declarePlaintext('"users"', 'email') + const original = [ + 'CREATE TABLE "notes" ("body" text);', + '--> statement-breakpoint', + "INSERT INTO \"notes\" VALUES (E'a\\'b');", + '--> statement-breakpoint', + '-- it\'s ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_estring.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // The dollar-quote skip must not swallow live SQL: a `$$` body sitting + // BEFORE a genuine plaintext ALTER still leaves that ALTER rewritable. + it('still rewrites a live ALTER that follows a dollar-quoted body', async () => { + declarePlaintext('"users"', 'email') + const filePath = path.join(tmpDir, '0033_dollar-then-live.sql') + fs.writeFileSync( + filePath, + [ + "CREATE FUNCTION note() RETURNS text AS $$ don't $$ LANGUAGE sql;", + '--> statement-breakpoint', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain('DROP COLUMN') + }) }) // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 6cea3886b..e0cec019d 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -184,9 +184,15 @@ function trimStatementPreamble(statement: string): string { * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in * an identifier) is an escape and does not close the token. * - * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a - * comment/literal here, which can only make us skip a rewrite (the statement - * then fails loudly at migrate time), never perform a destructive one. + * Dollar-quoted bodies (`$$ … $$`, `$fn$ … $fn$`) are skipped whole, and an + * `E''` literal gives backslash its escape meaning. Both are quote-PARITY + * concerns, not merely "we might skip a rewrite": an apostrophe inside an + * untracked `$$` body, or a `\'` read as a close, ends a literal EARLY, and + * every token after it is then misread — including a commented-out ALTER, which + * reads as live and gets rewritten into a live DROP COLUMN. That is the same + * failure mode as the apostrophe-in-identifier case below, and it is why the + * earlier reasoning here ("can only make us skip") was wrong: it holds for an + * UNDER-terminated literal, not an over-terminated one (#772 review, finding 1). */ function isInsideCommentOrString(sql: string, index: number): boolean { let i = 0 @@ -214,6 +220,20 @@ function isInsideCommentOrString(sql: string, index: number): boolean { } if (j > index) return true i = j + } else if (sql[i] === '$') { + // A `$tag$ … $tag$` body is opaque: its content is not SQL, so a `'` or + // `--` inside it means nothing. Skipping the whole body is what keeps the + // rest of the file's quote parity aligned with Postgres. + const delimiter = dollarQuoteDelimiter(sql, i) + if (delimiter === undefined) { + i += 1 + } else { + const close = sql.indexOf(delimiter, i + delimiter.length) + // Unterminated: inert to the end of the file, like the cases below. + const end = close === -1 ? sql.length : close + delimiter.length + if (end > index) return true + i = end + } } else if (sql[i] === '"') { // A quoted identifier is live SQL, but its body is not: consuming it here // — before the `'` branch below — is what stops an apostrophe inside one @@ -227,7 +247,7 @@ function isInsideCommentOrString(sql: string, index: number): boolean { if (end > index) return true i = end } else if (sql[i] === "'") { - const end = endOfQuoted(sql, i, "'") + const end = endOfQuoted(sql, i, "'", isEscapeStringOpener(sql, i)) // Unterminated, or the literal runs past `index`: `index` is inside a // string literal, which is every bit as inert as a comment. if (end > index) return true @@ -239,15 +259,56 @@ function isInsideCommentOrString(sql: string, index: number): boolean { return false } +/** + * The `$tag$` delimiter opening at `open`, or `undefined` when what sits there + * is just a `$`. The tag is empty (`$$`) or an SQL identifier (`$fn$`); a digit + * cannot start one, so a positional parameter (`$1`) is never mistaken for a + * dollar-quote opener. + */ +function dollarQuoteDelimiter(sql: string, open: number): string | undefined { + DOLLAR_QUOTE_OPEN_RE.lastIndex = open + return DOLLAR_QUOTE_OPEN_RE.exec(sql)?.[0] +} + +/** Sticky, so the scan never has to slice the file to test one position. */ +const DOLLAR_QUOTE_OPEN_RE = /\$(?:[A-Za-z_][A-Za-z0-9_]*)?\$/y + +/** + * Whether the `'` at `open` starts an `E''` escape-string, in which a backslash + * escapes the following character. Plain literals give backslash no special + * meaning under the default `standard_conforming_strings = on`. + * + * The character before the `E` must not be part of an identifier, so a column + * or type name that merely ends in `e` does not turn the literal after it into + * an escape-string. + */ +function isEscapeStringOpener(sql: string, open: number): boolean { + const prefix = sql[open - 1] + if (prefix !== 'E' && prefix !== 'e') return false + return !/[A-Za-z0-9_]/.test(sql[open - 2] ?? '') +} + /** * The index just past the `quote`-delimited token that opens at `open`, or * `sql.length` when it is never closed. A doubled delimiter inside the token is * an escaped one (`''`, `""`) and does not end it. + * + * With `backslashEscapes`, a `\` also escapes the next character — the `E''` + * form. Getting this wrong ends the literal EARLY, which shifts quote parity + * for the whole rest of the file and can make a commented-out ALTER downstream + * read as live (#772 review, finding 1). */ -function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { +function endOfQuoted( + sql: string, + open: number, + quote: "'" | '"', + backslashEscapes = false, +): number { let i = open + 1 while (i < sql.length) { - if (sql[i] !== quote) { + if (backslashEscapes && sql[i] === '\\') { + i += 2 + } else if (sql[i] !== quote) { i += 1 } else if (sql[i + 1] === quote) { i += 2 From fe5750b6852d4ae1b99780c5b049348914603d00 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:08:41 +1000 Subject: [PATCH 073/123] fix(cli,wizard): find the CREATE TABLE body terminator instead of guessing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CREATE_TABLE_RE carried a lazy `([\s\S]*?)\)\s*;` body, so it stopped at the first `);` anywhere in the file. That can sit inside a `--` comment or a string DEFAULT — `"id" text, -- pk (uuid);` is legal and unremarkable — and the body was then truncated at that point. Every column declared after it disappeared from BOTH indexes. For an encrypted column that means the already-encrypted guard never sees it. On its own that yields a wrong reason (`source-unknown` for a column declared two lines up). Composed with a `declared` record from elsewhere in the corpus — a table dropped and recreated in a later migration — the fail-closed rule is satisfied too, and the ADD+DROP+RENAME drops a column full of ciphertext. Match only `CREATE TABLE … (` and locate the closing `)` by scanning candidate `)\s*;` positions for the first one `isInsideCommentOrString` says is live. Nested parens (`numeric(10,2)`, `CHECK (x > 0)`) are not followed by `;`, so requiring the terminator still excludes them. An unclosed statement is skipped rather than indexed to end-of-file. Three tests each: the comment truncation, the DEFAULT-string truncation, and the composed case that actually destroys data. --- .../src/__tests__/rewrite-migrations.test.ts | 87 +++++++++++++++++++ .../cli/src/commands/db/rewrite-migrations.ts | 50 +++++++++-- .../src/__tests__/rewrite-migrations.test.ts | 87 +++++++++++++++++++ packages/wizard/src/lib/rewrite-migrations.ts | 50 +++++++++-- 4 files changed, 264 insertions(+), 10 deletions(-) diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index c8090b216..4a0d231bd 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -1358,6 +1358,93 @@ describe('rewriteEncryptedAlterColumns', () => { ]) }) + // #772 review, finding 2. CREATE_TABLE_RE's body is lazy up to the first + // `)\s*;`, which can sit inside a `--` comment or a string DEFAULT. The + // body is then truncated and every column declared after that point is + // lost from BOTH indexes — so an encrypted column reads as undeclared. + it('indexes columns declared after a ");" inside a comment in the CREATE TABLE body', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" text, -- pk (uuid);', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('indexes columns declared after a ");" inside a string DEFAULT', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"note" text DEFAULT \'see (ticket);\',', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // The destructive composition: the truncated body loses the encrypted + // column, but a LATER migration re-declares the table so `declared` is + // satisfied from elsewhere — the fail-closed rule passes and the ALTER + // drops a column full of ciphertext. + it('does not drop ciphertext when a truncated CREATE TABLE body hides the encrypted column', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_recreate.sql'), + [ + 'DROP TABLE "users";', + '--> statement-breakpoint', + 'CREATE TABLE "users" (', + '\t"id" text, -- pk (uuid);', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + // Only the IMPLICIT schema collapses. A real non-public schema is a // genuinely different table and must not be conflated with the bare name. it('does not conflate a non-public schema with the unqualified table', async () => { diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index dae19ab83..3852853f7 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -341,12 +341,47 @@ const RENAME_COLUMN_RE = new RegExp( 'gi', ) -/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ -const CREATE_TABLE_RE = new RegExp( - String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, +/** + * `CREATE TABLE … (` — $1/$2 table. The body's END is found by + * {@link endOfCreateTableBody} rather than by the matcher. + * + * This used to carry a lazy `([\s\S]*?)\)\s*;` body, which stopped at the FIRST + * `);` in the file — and that can sit inside a `--` comment or a string DEFAULT + * (`"id" text, -- pk (uuid);`). The body was then truncated and every column + * declared after that point vanished from both indexes, including an ENCRYPTED + * one — which is how a later domain change on a ciphertext column got past the + * already-encrypted guard (#772 review, finding 2). + */ +const CREATE_TABLE_HEAD_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(`, 'gi', ) +/** Candidate body terminators, filtered by {@link isInsideCommentOrString}. */ +const CREATE_TABLE_BODY_END_RE = /\)\s*;/g + +/** + * The offset of the `)` closing the CREATE TABLE body that opens at + * `bodyStart`, or `undefined` when the statement is never closed. + * + * Parens nested inside the body (`numeric(10,2)`, `CHECK (x > 0)`) are not + * followed by `;`, so requiring the terminator keeps them from matching. + */ +function endOfCreateTableBody( + sql: string, + bodyStart: number, +): number | undefined { + CREATE_TABLE_BODY_END_RE.lastIndex = bodyStart + for ( + let end = CREATE_TABLE_BODY_END_RE.exec(sql); + end !== null; + end = CREATE_TABLE_BODY_END_RE.exec(sql) + ) { + if (!isInsideCommentOrString(sql, end.index)) return end.index + } + return undefined +} + /** `"col" ` inside a CREATE TABLE body — $1 column. */ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, @@ -519,10 +554,15 @@ function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { const declared = new Set() for (const sql of contents) { - for (const created of sql.matchAll(CREATE_TABLE_RE)) { + for (const created of sql.matchAll(CREATE_TABLE_HEAD_RE)) { if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) - const body = created[3] + const bodyStart = created.index + created[0].length + const bodyEnd = endOfCreateTableBody(sql, bodyStart) + // Never closed — treat the statement as absent rather than index a body + // that runs to the end of the file. + if (bodyEnd === undefined) continue + const body = sql.slice(bodyStart, bodyEnd) // The body is scanned as its own document: a `--` earlier in it comments // out the rest of that line, and a CREATE inside a block comment or a diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 4d9bfea8a..5ebfbf360 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -1286,6 +1286,93 @@ describe('rewriteEncryptedAlterColumns', () => { ]) }) + // #772 review, finding 2. CREATE_TABLE_RE's body is lazy up to the first + // `)\s*;`, which can sit inside a `--` comment or a string DEFAULT. The + // body is then truncated and every column declared after that point is + // lost from BOTH indexes — so an encrypted column reads as undeclared. + it('indexes columns declared after a ");" inside a comment in the CREATE TABLE body', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"id" text, -- pk (uuid);', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('indexes columns declared after a ");" inside a string DEFAULT', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + [ + 'CREATE TABLE "users" (', + '\t"note" text DEFAULT \'see (ticket);\',', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + // The destructive composition: the truncated body loses the encrypted + // column, but a LATER migration re-declares the table so `declared` is + // satisfied from elsewhere — the fail-closed rule passes and the ALTER + // drops a column full of ciphertext. + it('does not drop ciphertext when a truncated CREATE TABLE body hides the encrypted column', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_declare.sql'), + 'CREATE TABLE "users" ("email" text);\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_recreate.sql'), + [ + 'DROP TABLE "users";', + '--> statement-breakpoint', + 'CREATE TABLE "users" (', + '\t"id" text, -- pk (uuid);', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0002_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + // Only the IMPLICIT schema collapses. A real non-public schema is a // genuinely different table and must not be conflated with the bare name. it('does not conflate a non-public schema with the unqualified table', async () => { diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index e0cec019d..1f1d3118a 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -349,12 +349,47 @@ const RENAME_COLUMN_RE = new RegExp( 'gi', ) -/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ -const CREATE_TABLE_RE = new RegExp( - String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, +/** + * `CREATE TABLE … (` — $1/$2 table. The body's END is found by + * {@link endOfCreateTableBody} rather than by the matcher. + * + * This used to carry a lazy `([\s\S]*?)\)\s*;` body, which stopped at the FIRST + * `);` in the file — and that can sit inside a `--` comment or a string DEFAULT + * (`"id" text, -- pk (uuid);`). The body was then truncated and every column + * declared after that point vanished from both indexes, including an ENCRYPTED + * one — which is how a later domain change on a ciphertext column got past the + * already-encrypted guard (#772 review, finding 2). + */ +const CREATE_TABLE_HEAD_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(`, 'gi', ) +/** Candidate body terminators, filtered by {@link isInsideCommentOrString}. */ +const CREATE_TABLE_BODY_END_RE = /\)\s*;/g + +/** + * The offset of the `)` closing the CREATE TABLE body that opens at + * `bodyStart`, or `undefined` when the statement is never closed. + * + * Parens nested inside the body (`numeric(10,2)`, `CHECK (x > 0)`) are not + * followed by `;`, so requiring the terminator keeps them from matching. + */ +function endOfCreateTableBody( + sql: string, + bodyStart: number, +): number | undefined { + CREATE_TABLE_BODY_END_RE.lastIndex = bodyStart + for ( + let end = CREATE_TABLE_BODY_END_RE.exec(sql); + end !== null; + end = CREATE_TABLE_BODY_END_RE.exec(sql) + ) { + if (!isInsideCommentOrString(sql, end.index)) return end.index + } + return undefined +} + /** `"col" ` inside a CREATE TABLE body — $1 column. */ const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, @@ -527,10 +562,15 @@ function indexColumnDeclarations(contents: readonly string[]): ColumnIndex { const declared = new Set() for (const sql of contents) { - for (const created of sql.matchAll(CREATE_TABLE_RE)) { + for (const created of sql.matchAll(CREATE_TABLE_HEAD_RE)) { if (isInsideCommentOrString(sql, created.index)) continue const { schema, table } = tableOf(created[1], created[2]) - const body = created[3] + const bodyStart = created.index + created[0].length + const bodyEnd = endOfCreateTableBody(sql, bodyStart) + // Never closed — treat the statement as absent rather than index a body + // that runs to the end of the file. + if (bodyEnd === undefined) continue + const body = sql.slice(bodyStart, bodyEnd) // The body is scanned as its own document: a `--` earlier in it comments // out the rest of that line, and a CREATE inside a block comment or a From b954d913657bcba4e489520a9a7e45f4cca7f596 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:11:35 +1000 Subject: [PATCH 074/123] test(scripts): compare the two rewriter copies so a one-sided fix fails CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rewriter lives twice — packages/wizard/src/lib and packages/cli/src/commands/db — at ~99% similarity, and neither package depends on the other. Four #772 review findings each needed the identical fix in both places; a fix landing in one still passes that package's own suite, and this code emits DROP COLUMN. Extraction is not available this cycle: packages/utils has no package.json, @cipherstash/test-kit is private and build-less, and both bin entries set skipNodeModulesBundle, so a shared workspace package would survive into dist as an unresolvable bare specifier. That leaves publishing a new package into a fixed mid-RC release group or adding noExternal to two CLI bundles — both worse than the drift they would prevent, for now. So compare instead. Everything outside an explicit `#region wizard-only` marker must match byte-for-byte modulo comments, imports and the tool name in the emitted header. Verified the guard actually fires: perturbing either copy fails with a message naming both files and pointing at the region marker. --- .changeset/rewriter-never-drops-ciphertext.md | 28 +++++ packages/wizard/src/lib/rewrite-migrations.ts | 6 + .../rewriter-copies-in-sync.test.mjs | 111 ++++++++++++++++++ 3 files changed, 145 insertions(+) create mode 100644 scripts/__tests__/rewriter-copies-in-sync.test.mjs diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index ec82f4b60..0815b61de 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -52,3 +52,31 @@ treated as empty, and the wizard's `Run the migration now?` prompt defaults to N whenever the sweep rewrote anything, flagged anything, or could not check a directory at all — naming the directories that went unchecked, and making no claim about data destruction for a directory nothing is known about. + +Four further ways the sweep could still reach a ciphertext column are closed: + +- **Chained conversions.** The corpus index read `CREATE TABLE`, `ADD COLUMN` + and `RENAME`, but never the sweep's own target. A directory where an earlier + migration already ran `SET DATA TYPE eql_v2_encrypted` — generated by a stack + version predating this sweep — left the column looking plaintext, so a later + domain change dropped its ciphertext. Conversions are now recorded as the + sweep walks the corpus in order, so the first conversion still applies and + only the ones after it are flagged. +- **Schema qualification.** `"users"` and `"public"."users"` are the same table + — Postgres resolves the unqualified name through `search_path` — but they were + indexed under different keys. drizzle-kit emits unqualified while hand-written + SQL and this sweep's own output are qualified, so a corpus mixing the two hid + an encrypted column from the guard. A non-`public` schema stays distinct. +- **Quote parity.** A `$$ … $$` body containing an odd number of apostrophes, or + an `E'a\'b'` literal, ended a string literal earlier than Postgres does. Every + token after it was then misread — including a commented-out `ALTER`, which + read as live and was rewritten into a live `DROP COLUMN`. Dollar-quoted bodies + are skipped whole and `E''` backslash escapes are honoured. +- **Truncated `CREATE TABLE` bodies.** The body was matched up to the first + `);`, which can sit inside a `--` comment or a string `DEFAULT`. Columns + declared after that point vanished from the index entirely. The closing paren + is now located by skipping candidates that are inside a comment or literal. + +The two copies of this rewriter — one in `stash`, one in `@cipherstash/wizard` — +are now compared by a repo test, so a fix can no longer land in one and silently +miss the other. diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 1f1d3118a..93ea0e364 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -834,6 +834,11 @@ export async function rewriteEncryptedAlterColumns( return { rewritten, skipped } } +// #region wizard-only — deliberately has no counterpart in +// packages/cli/src/commands/db/rewrite-migrations.ts. Everything OUTSIDE this +// region is mirrored there byte-for-byte and is checked by +// scripts/__tests__/rewriter-copies-in-sync.test.mjs. Only the wizard sweeps +// several candidate directories; both CLI call sites pass one explicit --out. /** One candidate migration directory's outcome from {@link sweepMigrationDirs}. */ export interface DirRewriteResult extends RewriteResult { /** The candidate directory as supplied (relative to `cwd`), for reporting. */ @@ -884,6 +889,7 @@ export async function sweepMigrationDirs( return results } +// #endregion wizard-only /** * The rewrite is identical for v2 and v3, and the ADD+DROP+RENAME sequence is diff --git a/scripts/__tests__/rewriter-copies-in-sync.test.mjs b/scripts/__tests__/rewriter-copies-in-sync.test.mjs new file mode 100644 index 000000000..213b1ffc5 --- /dev/null +++ b/scripts/__tests__/rewriter-copies-in-sync.test.mjs @@ -0,0 +1,111 @@ +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + +/** + * The destructive ALTER-COLUMN rewriter exists twice: `@cipherstash/wizard` + * runs it after its agent edits a schema, and `stash eql migration --drizzle` + * runs it over an explicit `--out`. Both are published, neither depends on the + * other, and `packages/utils` is not a package while `@cipherstash/test-kit` is + * private and build-less — so there is nowhere to put shared runtime code that + * both npm tarballs could resolve. Extracting one means either publishing a new + * package into a fixed release group or adding `noExternal` to two CLI bundles. + * + * Until that happens the two files are near-clones, and drift between them is + * the failure mode: four separate #772 review findings each needed the same fix + * applied in both places, and a fix landing in only one still passes that + * package's suite. Nothing else in CI compares them. + * + * So: everything outside the wizard's `#region wizard-only` must match the CLI + * copy exactly, modulo comments and the tool name in the emitted header. + */ +const WIZARD = 'packages/wizard/src/lib/rewrite-migrations.ts' +const CLI = 'packages/cli/src/commands/db/rewrite-migrations.ts' + +const REGION_OPEN = '// #region wizard-only' +const REGION_CLOSE = '// #endregion wizard-only' + +/** Drop the wizard-only region, delimited by the markers in the source. */ +function stripWizardOnly(source) { + const open = source.indexOf(REGION_OPEN) + if (open === -1) return source + const close = source.indexOf(REGION_CLOSE, open) + if (close === -1) return source + return source.slice(0, open) + source.slice(close + REGION_CLOSE.length) +} + +/** + * Comments, imports and blank lines removed; the emitted header's tool name + * canonicalised. Both files are Biome-formatted identically, so a pure comment + * line is reliably one starting with `//`, `/*` or `*` — no line of code in + * either file starts that way (the regex literals begin `/[`). + */ +function comparableCode(source) { + return source + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.length > 0) + .filter( + (line) => + !line.startsWith('//') && + !line.startsWith('*') && + !line.startsWith('/*'), + ) + .filter((line) => !line.startsWith('import ')) + .map((line) => + line.replace(/'-- Rewritten by [^:]+:/, "'-- Rewritten by :"), + ) +} + +const read = (file) => readFileSync(resolve(REPO_ROOT, file), 'utf8') + +describe('the wizard and cli rewriter copies stay in sync', () => { + const wizardSource = read(WIZARD) + const cliSource = read(CLI) + + it('marks the wizard-only region so the comparison knows what to exclude', () => { + expect(wizardSource).toContain(REGION_OPEN) + expect(wizardSource).toContain(REGION_CLOSE) + // The CLI copy has no counterpart, so it must not carry the markers. + expect(cliSource).not.toContain(REGION_OPEN) + }) + + it('finds real code in both (guards against a normaliser that strips everything)', () => { + expect( + comparableCode(stripWizardOnly(wizardSource)).length, + ).toBeGreaterThan(200) + expect(comparableCode(cliSource).length).toBeGreaterThan(200) + }) + + it('has identical shared logic', () => { + const wizard = comparableCode(stripWizardOnly(wizardSource)) + const cli = comparableCode(cliSource) + expect( + wizard.join('\n'), + `${WIZARD} and ${CLI} have drifted. Every fix to this rewriter must land ` + + "in BOTH copies — a one-sided fix still passes that package's own suite, " + + 'and this rewriter emits DROP COLUMN. If the difference is intentional and ' + + `wizard-only, move it inside the ${REGION_OPEN} region.`, + ).toBe(cli.join('\n')) + }) + + it('keeps sweepMigrationDirs as the only wizard-only export', () => { + // A second wizard-only export would mean shared logic had started to + // diverge under cover of the region marker. + const region = wizardSource.slice( + wizardSource.indexOf(REGION_OPEN), + wizardSource.indexOf(REGION_CLOSE), + ) + const exported = [ + ...region.matchAll( + /^export (?:async function|interface|const|type) (\w+)/gm, + ), + ] + .map((match) => match[1]) + .sort() + expect(exported).toEqual(['DirRewriteResult', 'sweepMigrationDirs']) + }) +}) From 3d0de782b6bfaa9a0f7b784fc25f6fbcc46fa76d Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:16:23 +1000 Subject: [PATCH 075/123] fix(wizard): sweep only drizzle-kit output directories MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wizard cannot discover a project's configured drizzle-kit `out`, so it tries drizzle/, migrations/ and src/db/migrations/. Two of those are generic names — Knex, node-pg-migrate, Flyway and hand-rolled psql all use them — and this sweep emits DROP COLUMN. So a project whose out is drizzle/ but which also keeps a hand-maintained migrations/ had that second directory rewritten into ADD+DROP+RENAME, in a directory the wizard was never pointed at. #783's fail-closed rule does not help: it indexes per directory, and a real migration history declares its own columns, so the guard passes and the rewrite proceeds. Worse, the emitted `--> statement-breakpoint` is a valid SQL line comment, so the mangled file stays runnable by whichever tool actually owns it. Gate on meta/_journal.json, which drizzle-kit writes on first generate and maintains after, and which none of the others produce. A directory holding .sql files but no journal is reported rather than silently passed over — a genuine drizzle output whose meta/ went missing must not just look clean. Both CLI call sites are unaffected; they pass one explicit --out and never sweep. The multi-directory tests added by the previous commit keep their intent — their fixtures are now journal-bearing, so shrinking DRIZZLE_OUT_DIRS or short-circuiting the aggregation loop still fails them — and a new pair covers the foreign-directory case at both the sweep and command layers. Also adds the afterEach that post-agent.test.ts never had: all five tests in that describe leaked a temp tree per run. --- .changeset/rewriter-never-drops-ciphertext.md | 14 ++++ .../wizard/src/__tests__/post-agent.test.ts | 81 ++++++++++++++++-- .../src/__tests__/rewrite-migrations.test.ts | 84 ++++++++++++++++++- packages/wizard/src/lib/post-agent.ts | 14 +++- packages/wizard/src/lib/rewrite-migrations.ts | 51 +++++++++++ 5 files changed, 233 insertions(+), 11 deletions(-) diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md index 0815b61de..e32890e4c 100644 --- a/.changeset/rewriter-never-drops-ciphertext.md +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -80,3 +80,17 @@ Four further ways the sweep could still reach a ciphertext column are closed: The two copies of this rewriter — one in `stash`, one in `@cipherstash/wizard` — are now compared by a repo test, so a fix can no longer land in one and silently miss the other. + +**The wizard now sweeps only drizzle-kit output directories.** It cannot +discover a project's configured `out`, so it tries `drizzle/`, `migrations/` and +`src/db/migrations/` — but the last two are generic names that Knex, +node-pg-migrate, Flyway and hand-rolled psql also use. A project whose drizzle +`out` is `drizzle/` and which also keeps a hand-maintained `migrations/` had +that second directory rewritten into ADD+DROP+RENAME, in a directory the wizard +was never pointed at. The fail-closed rule is no defence there: a real migration +history declares its own columns, so the rewrite proceeds. A candidate is now +swept only if it carries the `meta/_journal.json` drizzle-kit maintains. A +directory that holds `.sql` files but no journal is reported rather than passed +over in silence, so a genuine drizzle output whose `meta/` went missing is +visible instead of looking clean. `stash eql migration` and `stash eql install` +are unaffected — both already take a single explicit `--out`. diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index f8f87a791..e104be5a3 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -1,7 +1,7 @@ import fs from 'node:fs' import os from 'node:os' import path from 'node:path' -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { runPostAgentSteps } from '../lib/post-agent.js' import type { DetectedPackageManager } from '../lib/types.js' @@ -139,13 +139,35 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { } as never, }) + /** + * Make `dir` a drizzle-kit OUTPUT directory. The sweep now requires the + * `meta/_journal.json` drizzle-kit maintains, because `migrations/` and + * `src/db/migrations/` are generic names other tools use and this sweep + * emits `DROP COLUMN`. + */ + const makeDrizzleOut = (dir: string): string => { + const abs = path.join(cwd, dir) + fs.mkdirSync(path.join(abs, 'meta'), { recursive: true }) + fs.writeFileSync( + path.join(abs, 'meta', '_journal.json'), + JSON.stringify({ version: '7', dialect: 'postgresql', entries: [] }), + ) + return abs + } + beforeEach(() => { cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'wizard-post-agent-')) vi.mocked(p.confirm).mockClear() }) + afterEach(() => { + // Every test here writes a real temp tree; without this they accumulate in + // os.tmpdir() for the life of the machine. + fs.rmSync(cwd, { recursive: true, force: true }) + }) + it('defaults to Yes when the sweep changed nothing', async () => { - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') fs.writeFileSync( path.join(cwd, 'drizzle', '0000_init.sql'), 'CREATE TABLE "users" ("id" integer PRIMARY KEY);\n', @@ -167,7 +189,7 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { }) it('defaults to No, and says why, when a file was rewritten', async () => { - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') // The sweep is fail-closed: it rewrites a column only when the corpus // positively declares it (and it isn't already encrypted). A real drizzle // corpus carries this declaration in an earlier migration — supply it so @@ -204,7 +226,7 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { }) it('defaults to No when a statement was flagged rather than rewritten', async () => { - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') fs.writeFileSync( path.join(cwd, 'drizzle', '0001_using.sql'), 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', @@ -226,7 +248,7 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { // Yes over migrations nobody checked. Unknown is not the same as safe. it('defaults to No when a directory could not be swept at all', async () => { // A directory named `*.sql` makes readFile throw EISDIR mid-sweep. - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') fs.mkdirSync(path.join(cwd, 'drizzle', '0001_alter.sql')) await runDrizzle() @@ -266,12 +288,12 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { // fail. it('sweeps a candidate directory other than the first', async () => { // drizzle/ exists but has nothing to do; the rewrite lives in migrations/. - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') fs.writeFileSync( path.join(cwd, 'drizzle', '0000_init.sql'), 'CREATE TABLE "widgets" ("id" integer PRIMARY KEY);\n', ) - fs.mkdirSync(path.join(cwd, 'migrations')) + makeDrizzleOut('migrations') fs.writeFileSync( path.join(cwd, 'migrations', '0000_declare.sql'), 'CREATE TABLE "users" ("email" text);\n', @@ -296,9 +318,50 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { expect(String(options?.message)).toContain('DESTROYS data') }) + // The other half of the test above. `migrations/` is swept when it is a + // drizzle output directory — and must NOT be when it belongs to Knex, + // node-pg-migrate, Flyway or hand-rolled psql, all of which use that name. + // The wizard was never pointed at such a directory, and this sweep emits + // DROP COLUMN (#772 review, finding 5). + it('leaves a migrations/ directory that is not a drizzle output untouched', async () => { + makeDrizzleOut('drizzle') + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_init.sql'), + 'CREATE TABLE "widgets" ("id" integer PRIMARY KEY);\n', + ) + // A foreign migration history: self-contained, so the fail-closed + // `declared` rule would happily pass it. + fs.mkdirSync(path.join(cwd, 'migrations'), { recursive: true }) + const foreign = path.join(cwd, 'migrations', '002_encrypt.sql') + const foreignSql = [ + 'CREATE TABLE "patients" ("ssn" text);', + 'ALTER TABLE "patients" ALTER COLUMN "ssn" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + fs.writeFileSync(foreign, foreignSql) + + // Only `confirm` is mocked at module level; spy on the log so the + // "passed over" notice can be asserted. + const info = vi.spyOn(p.log, 'info').mockImplementation(() => {}) + + await runDrizzle() + + expect(fs.readFileSync(foreign, 'utf-8')).toBe(foreignSql) + expect(fs.readFileSync(foreign, 'utf-8')).not.toContain('DROP COLUMN') + // Nothing was rewritten or flagged, so the prompt stays on its clean arm. + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(true) + // But the user is told the directory was passed over, so a genuine drizzle + // output whose meta/ went missing does not just look clean. + const logged = info.mock.calls.flat().join('\n') + expect(logged).toContain('migrations/') + expect(logged).toContain('meta/_journal.json') + info.mockRestore() + }) + it('aggregates a rewrite in one directory with a flag in another', async () => { // drizzle/ declares email, so its ALTER is rewritten. - fs.mkdirSync(path.join(cwd, 'drizzle')) + makeDrizzleOut('drizzle') fs.writeFileSync( path.join(cwd, 'drizzle', '0000_declare.sql'), 'CREATE TABLE "users" ("email" text);\n', @@ -308,7 +371,7 @@ describe('drizzle migrate prompt after a destructive rewrite', () => { 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', ) // migrations/ never declares its column, so its ALTER is source-unknown. - fs.mkdirSync(path.join(cwd, 'migrations')) + makeDrizzleOut('migrations') const flagged = path.join(cwd, 'migrations', '0001_encrypt.sql') const flaggedSql = 'ALTER TABLE "orders" ALTER COLUMN "total" SET DATA TYPE eql_v3_text_search;\n' diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 5ebfbf360..1914febcf 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -1684,8 +1684,27 @@ describe('sweepMigrationDirs', () => { '', ].join('\n') - /** Create `dir` under the sandbox, optionally seeding `name` with `sql`. */ + /** + * Create a drizzle-kit OUTPUT directory under the sandbox, optionally seeding + * `name` with `sql`. + * + * The `meta/_journal.json` is what makes it drizzle-kit's: the sweep now + * requires it, because `migrations/` and `src/db/migrations/` are generic + * names that Knex, Flyway, node-pg-migrate and raw psql also use, and this + * sweep emits `DROP COLUMN`. Use {@link seedForeignDir} for the other case. + */ const seedDir = (dir: string, name?: string, sql?: string): string => { + const abs = seedForeignDir(dir, name, sql) + fs.mkdirSync(path.join(abs, 'meta'), { recursive: true }) + fs.writeFileSync( + path.join(abs, 'meta', '_journal.json'), + JSON.stringify({ version: '7', dialect: 'postgresql', entries: [] }), + ) + return abs + } + + /** A migration directory belonging to some OTHER tool — no drizzle journal. */ + const seedForeignDir = (dir: string, name?: string, sql?: string): string => { const abs = path.join(tmpDir, dir) fs.mkdirSync(abs, { recursive: true }) if (name) fs.writeFileSync(path.join(abs, name), sql ?? ALTER) @@ -1813,6 +1832,69 @@ describe('sweepMigrationDirs', () => { expect(updated).toBe(`${alterSql}\n`) expect(updated).not.toContain('DROP COLUMN') }) + + // #772 review, finding 5. `migrations/` and `src/db/migrations/` are generic + // names — Knex, node-pg-migrate, Flyway and hand-rolled psql all use them. + // Sweeping every candidate that merely EXISTS meant a project whose drizzle + // `out` is `drizzle/` but which also keeps a hand-maintained `migrations/` + // had that second directory rewritten into ADD+DROP+RENAME, in a directory + // this tool was never pointed at. The fail-closed `declared` rule does not + // help: a real migration history declares its own columns. + // + // drizzle-kit always writes `meta/_journal.json` into its output directory; + // none of the other tools do. That is the discriminator. + it('does not sweep a directory that is not a drizzle-kit output', async () => { + const foreign = seedForeignDir('migrations', '0001_alter.sql') + const alter = path.join(foreign, '0001_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['migrations']) + + expect(results.find((r) => r.dir === 'migrations')?.rewritten).toEqual([]) + const updated = fs.readFileSync(alter, 'utf-8') + expect(updated).toBe(ALTER) + expect(updated).not.toContain('DROP COLUMN') + }) + + // Silence would be its own failure: a genuine drizzle directory whose meta/ + // was deleted must not simply do nothing without saying so. + it('reports a skipped directory that holds SQL but no drizzle journal', async () => { + seedForeignDir('migrations', '0001_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['migrations']) + + expect(results.find((r) => r.dir === 'migrations')?.notDrizzleOutput).toBe( + true, + ) + }) + + // An empty foreign directory is not worth mentioning — there is nothing in it + // the sweep could have repaired. + it('does not report a journal-less directory that holds no SQL', async () => { + seedForeignDir('migrations') + + const results = await sweepMigrationDirs(tmpDir, ['migrations']) + + expect(results.find((r) => r.dir === 'migrations')?.notDrizzleOutput).toBe( + undefined, + ) + }) + + // The blast-radius fix must not shrink the legitimate reach: a real drizzle + // output directory that is NOT the first candidate is still swept. + it('still sweeps a drizzle output directory that is not the first candidate', async () => { + seedDir( + 'drizzle', + '0000_noop.sql', + 'CREATE TABLE "widgets" ("id" integer);\n', + ) + const migrations = seedDir('migrations', '0001_alter.sql') + + const results = await sweepMigrationDirs(tmpDir, ['drizzle', 'migrations']) + + expect(results.find((r) => r.dir === 'migrations')?.rewritten).toEqual([ + path.join(migrations, '0001_alter.sql'), + ]) + }) }) // The three reasons drive very different user action — re-encrypt through the diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index 25c37b7da..df8f7ec0c 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -170,10 +170,22 @@ async function rewriteEncryptedMigrations(cwd: string): Promise<{ const results = await sweepMigrationDirs(cwd, DRIZZLE_OUT_DIRS) const totals = { rewritten: 0, skipped: 0, failedDirs: [] as string[] } - for (const { dir, rewritten, skipped, error } of results) { + for (const { dir, rewritten, skipped, error, notDrizzleOutput } of results) { totals.rewritten += rewritten.length totals.skipped += skipped.length + // Not a failure and not a risk — the directory belongs to some other tool, + // so `drizzle-kit migrate` will not run it and the prompt below is + // unaffected. Said out loud anyway, so a user whose drizzle output really + // does live here (meta/ deleted, or a hand-assembled directory) can see why + // nothing was repaired instead of assuming it was clean. + if (notDrizzleOutput) { + p.log.info( + `Left ${dir}/ alone — it holds .sql files but no drizzle-kit journal (meta/_journal.json), so it is not a drizzle output directory. If it IS your drizzle \`out\`, run \`drizzle-kit generate\` once to create the journal, then re-run the wizard.`, + ) + continue + } + // Presence, not truthiness: `error` is `err.message` for a thrown `Error`, // and `new Error()` has an empty message. Testing `if (error)` would put a // blank-message failure back on the fail-open path this whole branch exists diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 93ea0e364..74393eac6 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -845,6 +845,34 @@ export interface DirRewriteResult extends RewriteResult { dir: string /** Set when this directory's sweep threw; the sweep continues regardless. */ error?: string + /** + * Set when the directory holds `.sql` files but no drizzle-kit journal, so + * nothing was swept. Reported rather than silent: a genuine drizzle output + * directory whose `meta/` was deleted would otherwise look clean. + */ + notDrizzleOutput?: true +} + +/** + * Whether `abs` is a drizzle-kit output directory. + * + * drizzle-kit writes `meta/_journal.json` there on the first `generate` and + * maintains it for every migration after; nothing else in the candidate list's + * ecosystem does. Knex, node-pg-migrate and Flyway all track state in the + * database, not in a sibling file. + */ +function isDrizzleOutputDir(abs: string): boolean { + return existsSync(join(abs, 'meta', '_journal.json')) +} + +/** Whether the directory holds anything this sweep could have acted on. */ +async function holdsSqlFiles(abs: string): Promise { + try { + const entries = await readdir(abs) + return entries.some((entry) => entry.endsWith('.sql')) + } catch { + return false + } } /** @@ -867,6 +895,15 @@ export interface DirRewriteResult extends RewriteResult { * * A directory that throws mid-sweep is reported via `error` rather than * aborting — one unreadable candidate must not strand the others. + * + * **Why only drizzle-kit output directories:** the candidate list is a guess, + * and two of its three entries are generic names that Knex, node-pg-migrate, + * Flyway and hand-rolled psql also use. This sweep emits `DROP COLUMN`, so + * rewriting a directory it was never pointed at is the worst thing it can do — + * and the fail-closed `declared` rule is no defence there, because a real + * migration history declares its own columns. Requiring the `meta/_journal.json` + * that drizzle-kit maintains keeps the reach to directories drizzle-kit owns + * (#772 review, finding 5). */ export async function sweepMigrationDirs( cwd: string, @@ -878,6 +915,20 @@ export async function sweepMigrationDirs( const abs = resolve(cwd, dir) if (!existsSync(abs)) continue + if (!isDrizzleOutputDir(abs)) { + // Only worth surfacing when there was something here to act on; an empty + // or non-SQL directory is noise. + if (await holdsSqlFiles(abs)) { + results.push({ + dir, + rewritten: [], + skipped: [], + notDrizzleOutput: true, + }) + } + continue + } + try { const { rewritten, skipped } = await rewriteEncryptedAlterColumns(abs) results.push({ dir, rewritten, skipped }) From 184995a22463faa6149636bd773757cfd72e242e Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:37:17 +1000 Subject: [PATCH 076/123] fix(bench): key seed rows by JS property, and check it without a database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit extractEncryptionSchema keys the encrypted-table column map by the Drizzle table's JS PROPERTY (encText), not the DB column name (enc_text) — so buildColumnKeyMap() is {encText: 'enc_text', ...} and resolveEncryptColumnMap matches models on the property names. The v2 -> v3 port left the seed keyed by DB name, so no field matched: bulkEncryptModels returned every row untouched, with no failure, and the insert then carried PLAINTEXT into eql_v3_* columns. The remap at seed.ts:66 was moving plaintext, not ciphertext, and its comment had become false. Not a data-safety bug — verified against a real Postgres with the pinned EQL 3.0.2 bundle, all three domains reject the insert with 23514. It is a bench that cannot run at all for anyone with credentials. Nothing caught it because nothing could. `tsc --noEmit` passes: BenchPlaintextRow was hand-written and agreed with itself. harness.test.ts asserts a row count and never that a column is encrypted. And CI runs only `test:local db-only`, which never seeds — every suite that touches the seed needs credentials. So the check that fails on this is deliberately cheap: __unit__/seed-keys.test.ts compares the row's keys against buildColumnKeyMap()'s, under a second vitest config with no globalSetup, so it needs neither a database nor credentials and cannot be skipped for want of either. BenchPlaintextRow is now the property space, and the identity remap is gone. --- .github/workflows/tests-bench.yml | 10 +++++ packages/bench/__unit__/seed-keys.test.ts | 50 +++++++++++++++++++++ packages/bench/package.json | 53 ++++++++++++----------- packages/bench/src/drizzle/setup.ts | 18 ++++++-- packages/bench/src/harness/seed.ts | 23 +++++----- packages/bench/vitest.unit.config.ts | 18 ++++++++ 6 files changed, 131 insertions(+), 41 deletions(-) create mode 100644 packages/bench/__unit__/seed-keys.test.ts create mode 100644 packages/bench/vitest.unit.config.ts diff --git a/.github/workflows/tests-bench.yml b/.github/workflows/tests-bench.yml index 5b8da9735..71ff378a6 100644 --- a/.github/workflows/tests-bench.yml +++ b/.github/workflows/tests-bench.yml @@ -88,3 +88,13 @@ jobs: - name: Run bench smoke tests working-directory: packages/bench run: pnpm test:local db-only + + # Separate config, no globalSetup: these need neither a database nor + # credentials, so they cannot be skipped for want of either. The seed + # keying they check was wrong for the whole v2 -> v3 port precisely + # because every suite that would have caught it needs credentials, and + # `tsc --noEmit` passes on a hand-written row type that agrees with + # itself (#772 review, finding 12). + - name: Run bench unit checks (no database, no credentials) + working-directory: packages/bench + run: pnpm test:unit diff --git a/packages/bench/__unit__/seed-keys.test.ts b/packages/bench/__unit__/seed-keys.test.ts new file mode 100644 index 000000000..82a356e11 --- /dev/null +++ b/packages/bench/__unit__/seed-keys.test.ts @@ -0,0 +1,50 @@ +/** + * The seed's row keys must be the keys model encryption MATCHES on. + * + * `extractEncryptionSchema` keys the encrypted-table column map by the Drizzle + * table's **JS property** (`encText`), while the column's DB name (`enc_text`) + * is what the builder carries. So `resolveEncryptColumnMap().columnPaths` — the + * list every model operation matches a row's fields against — is the property + * names. A row keyed by DB name matches nothing: `bulkEncryptModels` returns it + * untouched, with no failure, and the insert then puts PLAINTEXT into columns + * typed `eql_v3_*`. + * + * That is exactly what the v2 -> v3 port left behind, and nothing caught it: + * `tsc --noEmit` passes because `BenchPlaintextRow` was hand-written and agreed + * with itself, and CI only runs the `db-only` filter, which never seeds + * (#772 review, finding 12). + * + * Credential-free by construction — this compares two key sets and never + * reaches ZeroKMS. + */ +import { describe, expect, it } from 'vitest' +import { encryptionBenchTable } from '../src/drizzle/setup.js' +import { makePlaintextRow } from '../src/harness/seed.js' + +describe('bench seed rows are keyed for model encryption', () => { + // `resolveEncryptColumnMap` is internal, but it derives `columnPaths` as + // exactly `Object.keys(table.buildColumnKeyMap())` — read the same source. + const columnPaths = Object.keys(encryptionBenchTable.buildColumnKeyMap()) + + it('finds the encrypted columns (guards against an empty comparison)', () => { + expect(columnPaths.length).toBeGreaterThan(0) + }) + + it('emits a field for every encrypted column, under the matched key', () => { + const rowKeys = Object.keys(makePlaintextRow(0)) + + // Every encrypted column must be present in the row, or that column is + // silently never encrypted. + expect(rowKeys).toEqual(expect.arrayContaining([...columnPaths])) + }) + + it('emits no field the matcher would pass through as plaintext', () => { + const rowKeys = Object.keys(makePlaintextRow(0)) + const unmatched = rowKeys.filter((key) => !columnPaths.includes(key)) + + expect( + unmatched, + `these seed fields match no encrypted column, so they would be inserted as plaintext into an eql_v3_* column: ${unmatched.join(', ')}`, + ).toEqual([]) + }) +}) diff --git a/packages/bench/package.json b/packages/bench/package.json index 4134c57ce..550087366 100644 --- a/packages/bench/package.json +++ b/packages/bench/package.json @@ -1,28 +1,29 @@ { - "name": "@cipherstash/bench", - "version": "0.0.5-rc.4", - "private": true, - "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", - "type": "module", - "scripts": { - "build": "tsc --noEmit", - "db:setup": "tsx src/cli/setup.ts", - "db:reset": "tsx src/cli/reset.ts", - "test:local": "vitest run", - "bench:local": "vitest bench --run" - }, - "dependencies": { - "@cipherstash/stack": "workspace:*", - "@cipherstash/stack-drizzle": "workspace:*", - "drizzle-orm": "0.45.2", - "pg": "^8.22.0" - }, - "devDependencies": { - "@cipherstash/test-kit": "workspace:*", - "@types/node": "^22.20.1", - "@types/pg": "^8.20.0", - "tsx": "catalog:repo", - "typescript": "catalog:repo", - "vitest": "catalog:repo" - } + "name": "@cipherstash/bench", + "version": "0.0.5-rc.4", + "private": true, + "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", + "type": "module", + "scripts": { + "build": "tsc --noEmit", + "test:unit": "vitest run --config vitest.unit.config.ts", + "db:setup": "tsx src/cli/setup.ts", + "db:reset": "tsx src/cli/reset.ts", + "test:local": "vitest run", + "bench:local": "vitest bench --run" + }, + "dependencies": { + "@cipherstash/stack": "workspace:*", + "@cipherstash/stack-drizzle": "workspace:*", + "drizzle-orm": "0.45.2", + "pg": "^8.22.0" + }, + "devDependencies": { + "@cipherstash/test-kit": "workspace:*", + "@types/node": "^22.20.1", + "@types/pg": "^8.20.0", + "tsx": "catalog:repo", + "typescript": "catalog:repo", + "vitest": "catalog:repo" + } } diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index 87da15873..7978cd733 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -28,10 +28,22 @@ export const benchTable = pgTable('bench', { */ export const encryptionBenchTable = extractEncryptionSchema(benchTable) +/** + * A seed row, keyed by the Drizzle table's **JS property** names. + * + * That is what model encryption matches on: `extractEncryptionSchema` keys the + * encrypted-table column map by property (`encText`), not by DB column name + * (`enc_text`). A row keyed by DB name matches nothing — `bulkEncryptModels` + * returns it untouched, with no failure, and the plaintext then goes into an + * `eql_v3_*` column (#772 review, finding 12). + * + * Derived from the table so the two cannot drift again; + * `__unit__/seed-keys.test.ts` checks the row actually fills it. + */ export type BenchPlaintextRow = { - enc_text: string - enc_int: number - enc_jsonb: { idx: number; group: number } + encText: string + encInt: number + encJsonb: { idx: number; group: number } } /** The typed EQL v3 client this bench drives. */ diff --git a/packages/bench/src/harness/seed.ts b/packages/bench/src/harness/seed.ts index e9fc828b6..a145c6f6e 100644 --- a/packages/bench/src/harness/seed.ts +++ b/packages/bench/src/harness/seed.ts @@ -24,11 +24,12 @@ export function getTargetRows(): number { return n } -function makePlaintextRow(idx: number): BenchPlaintextRow { +/** Exported so `__unit__/seed-keys.test.ts` can check the row keys. */ +export function makePlaintextRow(idx: number): BenchPlaintextRow { return { - enc_text: `value-${String(idx).padStart(7, '0')}`, - enc_int: idx, - enc_jsonb: { idx, group: idx % 100 }, + encText: `value-${String(idx).padStart(7, '0')}`, + encInt: idx, + encJsonb: { idx, group: idx % 100 }, } } @@ -63,14 +64,12 @@ export async function seed( ) } - // bulkEncryptModels returns rows keyed by the encryptedTable column names - // (snake_case here) with encrypted EQL v3 envelopes as values. Drizzle's - // `benchTable` uses camelCase TS field names — remap before insert. - const encRows = encResult.data.map((r) => ({ - encText: r.enc_text, - encInt: r.enc_int, - encJsonb: r.enc_jsonb, - })) + // bulkEncryptModels returns rows under the SAME keys it matched on — the + // Drizzle table's JS property names — with EQL v3 envelopes as values. Those + // are the keys `db.insert()` wants, so there is nothing to remap. (The + // remap that used to sit here rewrote enc_text -> encText, which only + // appeared to work: nothing was ever encrypted, so it was moving plaintext.) + const encRows = encResult.data for (let i = 0; i < encRows.length; i += INSERT_BATCH) { const batch = encRows.slice(i, i + INSERT_BATCH) diff --git a/packages/bench/vitest.unit.config.ts b/packages/bench/vitest.unit.config.ts new file mode 100644 index 000000000..fa01c8c55 --- /dev/null +++ b/packages/bench/vitest.unit.config.ts @@ -0,0 +1,18 @@ +import { defineConfig } from 'vitest/config' + +/** + * Bench checks that need neither a database nor credentials. + * + * The main config's `globalSetup` installs EQL v3 through the built CLI, so + * every suite under it requires `turbo run build --filter stash` and a live + * Postgres. That is right for the benchmarks, and wrong for a check that only + * compares two key sets — and "it was too expensive to run in CI" is exactly + * how the seed came to insert plaintext into `eql_v3_*` columns unnoticed + * (#772 review, finding 12). + */ +export default defineConfig({ + test: { + include: ['__unit__/**/*.test.ts'], + server: { deps: { inline: [/packages\/test-kit/] } }, + }, +}) From a9d430b3cd4a62f1d086ffc33fef580061cbf4f8 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:40:16 +1000 Subject: [PATCH 077/123] fix(stack): refuse the wasm-inline client for DynamoDB EQL v2 reads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The adapter's v2 read path calls decryptModel(item) with NO table, on purpose: a v2 table means nothing to a v3 client's reconstructor map, and the native clients derive the table from the payloads. WasmEncryptionClient cannot — its decrypt requires the table and resolves date fields from a per-table map, so the omitted argument reached requireTable(undefined) and threw `TypeError: Cannot read properties of undefined (reading 'tableName')` from deep inside the client. That client ships in this package, is the documented entry for Deno, Workers and Supabase Edge Functions, and satisfies DynamoDBEncryptionClient structurally — so encryptedDynamoDB accepted it with no cast and the pairing only failed at the first read, with a message pointing nowhere near the cause. Reject it where the message can name the combination. The signal is a declared capability on the client (`requiresTableForDecrypt`) rather than sniffing arity or constructor name. v3 tables are always passed the table, so they still work. Also corrects three comments this package carried claiming audit metadata is forwarded "regardless of client shape" and that "every client this package ships carries .audit() on decrypt". The wasm-inline client's decrypt is a plain async method; its audit metadata is dropped, and the debug message that fires told the user to build a client with Encryption({ schemas }) — which is exactly what the wasm entry's own factory is. --- .changeset/dynamodb-wasm-v2-read.md | 27 +++++++ .../dynamodb/v2-table-forwarding.test.ts | 78 +++++++++++++++++++ packages/stack/src/dynamodb/helpers.ts | 22 +++--- packages/stack/src/dynamodb/index.ts | 19 ++++- packages/stack/src/dynamodb/types.ts | 10 ++- packages/stack/src/wasm-inline.ts | 15 ++++ 6 files changed, 158 insertions(+), 13 deletions(-) create mode 100644 .changeset/dynamodb-wasm-v2-read.md diff --git a/.changeset/dynamodb-wasm-v2-read.md b/.changeset/dynamodb-wasm-v2-read.md new file mode 100644 index 000000000..1e52ae725 --- /dev/null +++ b/.changeset/dynamodb-wasm-v2-read.md @@ -0,0 +1,27 @@ +--- +'@cipherstash/stack': patch +--- + +`encryptedDynamoDB` now refuses a `@cipherstash/stack/wasm-inline` client paired +with a legacy EQL v2 table, instead of failing at the first read with a +misleading error. + +The adapter's v2 read path deliberately calls `decryptModel(item)` with **no** +table — a v2 table means nothing to a v3 client's reconstructor map, and the +native clients derive the table from the payloads anyway. `WasmEncryptionClient` +cannot do that: its decrypt requires the table and resolves date fields from a +per-table map, so the omitted argument surfaced as +`TypeError: Cannot read properties of undefined (reading 'tableName')` thrown +from deep inside the client — on the documented entry for Deno, Cloudflare +Workers and Supabase Edge Functions, which satisfies the adapter's client type +structurally and so was accepted with no cast. + +The pairing is now rejected at the call site, with a message naming both the +combination and the fix. EQL v3 tables are unaffected: they are always passed +the table, so the wasm-inline client keeps working there. + +Three comments in this package claimed audit metadata was forwarded "regardless +of client shape" and that "every client this package ships carries `.audit()` on +decrypt". Neither was true of the wasm-inline client, whose decrypt returns a +plain promise — the metadata is dropped, observably (it is logged at debug), and +the comments and the debug message now say so. diff --git a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts index d8f1bc872..c6ab542ec 100644 --- a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts +++ b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts @@ -106,3 +106,81 @@ describe('bulkDecryptModels table forwarding', () => { expect(calls[0]?.table).toBe(usersV3) }) }) + +/** + * #772 review, finding 10. + * + * The table-less v2 decrypt above is correct for the native clients, which + * derive the table from the payloads. `WasmEncryptionClient` cannot: its + * decrypt requires the table and resolves date fields from a per-table map, so + * the omitted argument reached `requireTable(undefined)` and threw a TypeError + * about reading `tableName` — a message pointing nowhere near the cause, on the + * documented entry for Deno / Workers / Supabase Edge Functions, which + * satisfies `DynamoDBEncryptionClient` structurally and so is accepted with no + * cast. + */ +describe('a client whose decrypt requires the table', () => { + /** The shape `WasmEncryptionClient` presents: declared capability, no `.audit()`. */ + function wasmShapedClient(knownTables: string[]) { + const calls: { method: string; argCount: number }[] = [] + const record = + (method: string) => + (...args: unknown[]) => { + calls.push({ method, argCount: args.length }) + // Mirrors requireTable: throws rather than returning a Result. + if (args[1] === undefined) { + throw new TypeError( + "Cannot read properties of undefined (reading 'tableName')", + ) + } + return Promise.resolve({ data: {} }) + } + const client = { + requiresTableForDecrypt: true, + getEncryptConfig: () => ({ + v: 1, + tables: Object.fromEntries(knownTables.map((t) => [t, {}])), + }), + encryptModel: record('encryptModel'), + bulkEncryptModels: record('bulkEncryptModels'), + decryptModel: record('decryptModel'), + bulkDecryptModels: record('bulkDecryptModels'), + } + return { calls, client } + } + + it('is refused for an EQL v2 table, naming the entry to use instead', () => { + const { calls, client } = wasmShapedClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + // Synchronous: the guard runs when the operation is built, so the failure + // lands at the call site rather than as a rejected promise later. + expect(() => dynamo.decryptModel({ pk: 'a' }, usersV2)).toThrow( + /wasm-inline client cannot read legacy EQL v2 items/, + ) + // Refused before the client is touched, so the user never sees the + // TypeError about `tableName`. + expect(calls).toHaveLength(0) + }) + + it('is refused on the bulk v2 path too', () => { + const { client } = wasmShapedClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + expect(() => dynamo.bulkDecryptModels([{ pk: 'a' }], usersV2)).toThrow( + /wasm-inline client cannot read legacy EQL v2 items/, + ) + }) + + // v3 tables ARE forwarded the table, so this client works there — the guard + // must not turn into a blanket rejection of the wasm entry. + it('is accepted for an EQL v3 table, which is always given the table', async () => { + const { calls, client } = wasmShapedClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.decryptModel({ pk: 'a' }, usersV3) + + expect(calls).toHaveLength(1) + expect(calls[0]?.argCount).toBe(2) + }) +}) diff --git a/packages/stack/src/dynamodb/helpers.ts b/packages/stack/src/dynamodb/helpers.ts index 38c62d099..186e36490 100644 --- a/packages/stack/src/dynamodb/helpers.ts +++ b/packages/stack/src/dynamodb/helpers.ts @@ -86,11 +86,16 @@ export function handleError( /** * Resolve a decrypt call against either client shape. * - * Both the nominal `EncryptionClient` and the typed client return a chainable + * The nominal `EncryptionClient` and the typed client both return a chainable * operation carrying `.audit()` on decrypt (the typed client's is a - * `MappedDecryptOperation`). Chain the audit metadata onto it; the branch that - * awaits a bare promise remains only for a non-conforming custom client that - * exposes no `.audit()`. Audit metadata is forwarded regardless of client shape. + * `MappedDecryptOperation`). Chain the audit metadata onto it. + * + * NOT every client this package accepts does that. `WasmEncryptionClient` + * (`@cipherstash/stack/wasm-inline` — the documented entry for Deno, Workers + * and Supabase Edge Functions) returns a bare promise from decrypt, so it takes + * the branch below and its audit metadata is dropped. It ships in this package + * and satisfies `DynamoDBEncryptionClient` structurally, so it is accepted + * without a cast (#772 review, finding 10). */ export async function resolveDecryptResult( operation: unknown, @@ -103,12 +108,11 @@ export async function resolveDecryptResult( } if (typeof chainable?.audit !== 'function' && auditData.metadata) { - // Every client this package ships carries `.audit()` on decrypt, so this - // only fires for a custom client whose decrypt returns something else — - // there is then nowhere to put the metadata. Make the drop observable - // rather than silent. + // Reached by the wasm-inline client (bare promise, no `.audit()`) and by + // any custom client whose decrypt returns something else. There is nowhere + // to put the metadata, so make the drop observable rather than silent. logger.debug( - "DynamoDB: decrypt audit metadata ignored — this client's decrypt does not return a chainable operation with .audit(). Audited decrypts need a client built with Encryption({ schemas }).", + "DynamoDB: decrypt audit metadata ignored — this client's decrypt does not return a chainable operation with .audit(). Audited decrypts need a client from the default @cipherstash/stack entry; the wasm-inline client's decrypt returns a plain promise.", ) } diff --git a/packages/stack/src/dynamodb/index.ts b/packages/stack/src/dynamodb/index.ts index a19f1a691..8367f4094 100644 --- a/packages/stack/src/dynamodb/index.ts +++ b/packages/stack/src/dynamodb/index.ts @@ -41,7 +41,24 @@ function assertClientTableVersionMatch( table: AnyEncryptedTable, ): void { // Only v3 tables carry the strict wire-format requirement this guards. - if (!isV3Table(table)) return + if (!isV3Table(table)) { + // The v2 read path calls `decryptModel(item)` with NO table on purpose — + // a v2 table means nothing to a v3 client's reconstructor map. That is + // fine for the native clients, which derive the table from the payloads, + // and impossible for the WASM client, whose decrypt requires the table and + // otherwise throws a TypeError about `tableName` from deep inside + // `requireTable`. Refuse the pairing here, where the message can name it + // (#772 review, finding 10). + if ( + (encryptionClient as { requiresTableForDecrypt?: boolean }) + .requiresTableForDecrypt + ) { + throw new Error( + `encryptedDynamoDB: the @cipherstash/stack/wasm-inline client cannot read legacy EQL v2 items. Its decrypt requires the table, and a v2 table carries none of the information it needs — so "${table.tableName}" would fail at the first read. Use the default @cipherstash/stack entry for tables that still hold EQL v2 items, or migrate the table to an EQL v3 schema (types.* domains) and pass that.`, + ) + } + return + } const getEncryptConfig = ( encryptionClient as { diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index cfe37bb68..d521f1405 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -37,12 +37,16 @@ export type AnyEncryptedTable = * nominal `TypedEncryptionClient` parameter would reject a client built for * a narrower schema tuple. * - * Both clients now return a chainable operation on the decrypt paths — the + * Both NATIVE clients return a chainable operation on the decrypt paths — the * nominal client's `DecryptModelOperation` and the typed wrapper's * `MappedDecryptOperation` each carry `.audit()` (the typed wrapper also takes * the table as a second argument). The operation classes handle both; see - * `DecryptModelOperation` and `resolveDecryptResult`. Audit metadata on decrypt - * is therefore forwarded regardless of which client shape is supplied. + * `DecryptModelOperation` and `resolveDecryptResult`. + * + * The wasm-inline client does not: its decrypt is a plain `async` method, so + * audit metadata is dropped (observably — `resolveDecryptResult` logs it) and + * its EQL v2 read path is refused outright by `assertClientTableVersionMatch`, + * because that path relies on calling decrypt WITHOUT a table. */ export type DynamoDBEncryptionClient = { encryptModel(input: never, table: never): unknown diff --git a/packages/stack/src/wasm-inline.ts b/packages/stack/src/wasm-inline.ts index 0b7e89663..05a3a97b7 100644 --- a/packages/stack/src/wasm-inline.ts +++ b/packages/stack/src/wasm-inline.ts @@ -663,6 +663,21 @@ const INTERNAL_CONSTRUCT = Symbol('cs-wasm-client') * prevent callers from wrapping arbitrary objects in this type. */ export class WasmEncryptionClient { + /** + * This client's `decryptModel` / `bulkDecryptModels` REQUIRE the table — they + * resolve date fields from a per-table map and throw without it. The native + * clients derive the table from the payloads instead, so callers that hold a + * client structurally cannot tell the two apart. + * + * `encryptedDynamoDB` is the caller that cares: its legacy EQL v2 read path + * deliberately omits the table (a v2 table means nothing to a v3 + * reconstructor map), which reached `requireTable` with `undefined` and threw + * a TypeError about `tableName` pointing nowhere near the cause. Declared + * rather than sniffed so the check is a stated capability, not a guess about + * arity or constructor name (#772 review, finding 10). + */ + readonly requiresTableForDecrypt = true + /** @internal */ private readonly client: unknown From 966978a8781665e398db4241b626047f6fdc0034 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 13:27:34 +1000 Subject: [PATCH 078/123] fix(stack): stop the wasm-inline client failing every DynamoDB v3 write (#788 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the three review findings on #788, plus a real defect found while verifying the second one. The wasm+v2 guard message is now operation-neutral. `assertClientTableVersionMatch` runs on all four operations, so a plain-JS caller reaching `encryptModel` with a v2 table got "cannot read legacy EQL v2 items ... would fail at the first read" — naming an operation that never ran. The pairing is wrong in both directions anyway (`Encryption()` on that entry rejects a v2 schema), so the message now says so instead of describing a read. Verifying that finding surfaced the larger one: `encryptModel` / `bulkEncryptModels` chained `.audit()` onto the client's result unconditionally. The native clients return a thenable operation carrying it; the wasm-inline client's encrypt returns a bare `Promise` and has no `.audit()` anywhere. So EVERY EQL v3 write through this adapter on that entry failed with `client.encryptModel(...).audit is not a function` — surfaced as a `DYNAMODB_ENCRYPTION_ERROR`, indistinguishable from a genuine encryption fault. This PR's own changeset claimed the opposite ("EQL v3 tables are unaffected ... the wasm-inline client keeps working there"); it was true of decrypt only. `resolveEncryptResult` mirrors the existing `resolveDecryptResult`: tolerate the bare promise, drop audit metadata observably rather than crashing, and fail closed on a malformed result so an unencrypted item can never pass through as a success. The changeset and the `types.ts` docblock are corrected to match. Regression tests are credential-free. The chainable half matters most — the native clients' encrypt audit trail had no coverage outside a live-ZeroKMS suite, so nothing would have caught breaking it. skills/stash-dynamodb documented v2 decrypt as unconditionally supported and never mentioned that wasm-inline refuses v2 tables, on the documented entry for the runtimes most often paired with DynamoDB. It also claimed audit metadata forwards "regardless of client shape" — the exact phrase this PR removed from the source as untrue. Both corrected, plus the wasm-inline row in skills/stash-encryption, which never said the entry is v3-only. packages/bench/package.json is back to 2-space indent, keeping only the substantive `test:unit` addition: 53 changed lines down to 1. The repo's Biome config is `indentStyle: "space"` and the file was spaces on main, so the churn was neither required nor conformant — Biome ignores `**/package.json` entirely, verified by direct invocation. --- .changeset/dynamodb-skill-wasm-caveat.md | 17 +++ .changeset/dynamodb-wasm-v2-read.md | 16 ++- packages/bench/package.json | 54 ++++---- .../dynamodb/resolve-decrypt.test.ts | 120 ++++++++++++++++-- .../dynamodb/v2-table-forwarding.test.ts | 109 +++++++++++++++- packages/stack/src/dynamodb/helpers.ts | 58 +++++++++ packages/stack/src/dynamodb/index.ts | 10 +- .../operations/bulk-encrypt-models.ts | 13 +- .../src/dynamodb/operations/encrypt-model.ts | 11 +- packages/stack/src/dynamodb/types.ts | 19 ++- skills/stash-dynamodb/SKILL.md | 18 ++- skills/stash-encryption/SKILL.md | 4 +- 12 files changed, 391 insertions(+), 58 deletions(-) create mode 100644 .changeset/dynamodb-skill-wasm-caveat.md diff --git a/.changeset/dynamodb-skill-wasm-caveat.md b/.changeset/dynamodb-skill-wasm-caveat.md new file mode 100644 index 000000000..21d3f80ba --- /dev/null +++ b/.changeset/dynamodb-skill-wasm-caveat.md @@ -0,0 +1,17 @@ +--- +'stash': patch +--- + +The `stash-dynamodb` and `stash-encryption` skills documented EQL v2 decrypt as +unconditionally supported, without noting that `@cipherstash/stack/wasm-inline` +is EQL v3 only. Since that is the documented entry for Deno, Bun, Cloudflare +Workers and Supabase Edge Functions — runtimes commonly paired with DynamoDB — a +reader following the skill hit a runtime refusal with no forewarning. Both skills +now state that legacy v2 items are readable on the native `@cipherstash/stack` +entry only. + +The `stash-dynamodb` API reference also claimed audit metadata forwards to +ZeroKMS "regardless of client shape". It does not: the wasm-inline client's +operations return a plain promise with no `.audit()`, so its audit metadata is +dropped (logged at debug). The reference now says so, and says the operation +still succeeds. diff --git a/.changeset/dynamodb-wasm-v2-read.md b/.changeset/dynamodb-wasm-v2-read.md index 1e52ae725..b75673051 100644 --- a/.changeset/dynamodb-wasm-v2-read.md +++ b/.changeset/dynamodb-wasm-v2-read.md @@ -17,8 +17,20 @@ Workers and Supabase Edge Functions, which satisfies the adapter's client type structurally and so was accepted with no cast. The pairing is now rejected at the call site, with a message naming both the -combination and the fix. EQL v3 tables are unaffected: they are always passed -the table, so the wasm-inline client keeps working there. +combination and the fix. The message is operation-neutral: the guard runs on all +four operations, so a plain-JS caller reaching the write path with a v2 table +gets a message that does not claim a read it never attempted. + +`encryptModel` / `bulkEncryptModels` now also tolerate a client whose encrypt +returns a plain promise. They chained `.audit()` onto the result +unconditionally, which the wasm-inline client does not carry — so **every EQL v3 +write through this adapter on that entry** failed with +`client.encryptModel(...).audit is not a function`, surfaced as a +`DYNAMODB_ENCRYPTION_ERROR` and so indistinguishable from a real encryption +fault. The read path already handled this; the write path now matches, via the +same fail-closed check that rejects a malformed result rather than passing an +unencrypted item through as a success. Audit metadata still has nowhere to go on +that client, so it is dropped and logged at debug — the write itself succeeds. Three comments in this package claimed audit metadata was forwarded "regardless of client shape" and that "every client this package ships carries `.audit()` on diff --git a/packages/bench/package.json b/packages/bench/package.json index 550087366..46ef39c61 100644 --- a/packages/bench/package.json +++ b/packages/bench/package.json @@ -1,29 +1,29 @@ { - "name": "@cipherstash/bench", - "version": "0.0.5-rc.4", - "private": true, - "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", - "type": "module", - "scripts": { - "build": "tsc --noEmit", - "test:unit": "vitest run --config vitest.unit.config.ts", - "db:setup": "tsx src/cli/setup.ts", - "db:reset": "tsx src/cli/reset.ts", - "test:local": "vitest run", - "bench:local": "vitest bench --run" - }, - "dependencies": { - "@cipherstash/stack": "workspace:*", - "@cipherstash/stack-drizzle": "workspace:*", - "drizzle-orm": "0.45.2", - "pg": "^8.22.0" - }, - "devDependencies": { - "@cipherstash/test-kit": "workspace:*", - "@types/node": "^22.20.1", - "@types/pg": "^8.20.0", - "tsx": "catalog:repo", - "typescript": "catalog:repo", - "vitest": "catalog:repo" - } + "name": "@cipherstash/bench", + "version": "0.0.5-rc.4", + "private": true, + "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", + "type": "module", + "scripts": { + "build": "tsc --noEmit", + "test:unit": "vitest run --config vitest.unit.config.ts", + "db:setup": "tsx src/cli/setup.ts", + "db:reset": "tsx src/cli/reset.ts", + "test:local": "vitest run", + "bench:local": "vitest bench --run" + }, + "dependencies": { + "@cipherstash/stack": "workspace:*", + "@cipherstash/stack-drizzle": "workspace:*", + "drizzle-orm": "0.45.2", + "pg": "^8.22.0" + }, + "devDependencies": { + "@cipherstash/test-kit": "workspace:*", + "@types/node": "^22.20.1", + "@types/pg": "^8.20.0", + "tsx": "catalog:repo", + "typescript": "catalog:repo", + "vitest": "catalog:repo" + } } diff --git a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts index 92044eb78..f3ce936ae 100644 --- a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts +++ b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts @@ -1,15 +1,28 @@ /** - * Pure unit tests for the two client-shape helpers behind the DynamoDB adapter's - * decrypt path. Both shipped clients — nominal `EncryptionClient` and the typed - * EQL v3 client (whose decrypt returns a `MappedDecryptOperation`) — are - * chainable and carry `.audit()`; the bare-promise branch remains only for a - * non-conforming custom client. Every branch was previously reachable only - * through a live ZeroKMS decrypt; these move that assurance onto the pure CI - * lane. No credentials, no network. + * Pure unit tests for the client-shape helpers behind the DynamoDB adapter's + * decrypt AND encrypt paths. + * + * On decrypt, both native clients — nominal `EncryptionClient` and the typed EQL + * v3 client (whose decrypt returns a `MappedDecryptOperation`) — are chainable + * and carry `.audit()`; the bare-promise branch is taken by the wasm-inline + * client and by a non-conforming custom one. + * + * The same split exists on encrypt, and only the decrypt half handled it: the + * encrypt operations chained `.audit()` unconditionally, so the wasm-inline + * client failed every write (#788 review follow-up). `resolveEncryptResult` is + * the mirror, and the chainable half of its coverage matters most — the native + * clients' encrypt audit trail has no other credential-free test. + * + * Every branch was previously reachable only through live ZeroKMS; these move + * that assurance onto the pure CI lane. No credentials, no network. */ import type { Result } from '@byteslice/result' import { afterEach, describe, expect, it, vi } from 'vitest' -import { resolveDecryptResult, throwPreservingCode } from '@/dynamodb/helpers' +import { + resolveDecryptResult, + resolveEncryptResult, + throwPreservingCode, +} from '@/dynamodb/helpers' import { EncryptionOperation } from '@/encryption/operations/base-operation' import { MappedDecryptOperation } from '@/encryption/operations/mapped-decrypt' import { type EncryptionError, EncryptionErrorTypes } from '@/errors' @@ -178,6 +191,97 @@ describe('resolveDecryptResult', () => { }) }) +/** + * The write-path mirror (#788 review follow-up). The encrypt operations used + * to chain `.audit()` unconditionally, so a client returning a bare promise + * (the wasm-inline entry) failed every encrypt with + * `.audit is not a function`. These pin BOTH directions: the chainable path + * must still forward metadata — the native clients' audit trail depends on it, + * and its only other coverage is a live-credential suite — and the bare path + * must resolve instead of throwing. + */ +describe('resolveEncryptResult', () => { + it('chains .audit and forwards metadata when present (native clients)', async () => { + let seen: unknown + let calls = 0 + const operation = { + audit(config: { metadata?: Record }) { + calls += 1 + seen = config.metadata + return Promise.resolve({ data: { encrypted: true } }) + }, + } + + const result = await resolveEncryptResult( + operation, + { metadata: { sub: 'u1' } }, + 'encryptModel', + ) + + expect(result).toEqual({ data: { encrypted: true } }) + expect(seen).toEqual({ sub: 'u1' }) + expect(calls).toBe(1) + }) + + it('awaits a bare promise instead of throwing (wasm-inline encrypt)', async () => { + const result = await resolveEncryptResult( + Promise.resolve({ data: { encrypted: true } }), + { metadata: { dropped: true } }, + 'encryptModel', + ) + + expect(result).toEqual({ data: { encrypted: true } }) + }) + + it('propagates a failure result unchanged', async () => { + const failure = { failure: { message: 'boom', code: 'X' } } + + await expect( + resolveEncryptResult(Promise.resolve(failure), {}, 'bulkEncryptModels'), + ).resolves.toEqual(failure) + }) + + it('returns a failure — not a fake success — for a malformed result', async () => { + // Fail closed. A bare value cast straight through would hand the caller an + // "encrypted" item that was never encrypted. + for (const malformed of [{}, 42, undefined]) { + const result = await resolveEncryptResult( + Promise.resolve(malformed), + {}, + 'encryptModel', + ) + + expect(result.failure).toBeDefined() + expect(result.data).toBeUndefined() + } + }) + + it('names the operation in the dropped-metadata log, and stays silent without metadata', async () => { + const spy = vi.spyOn(logger, 'debug').mockImplementation(() => {}) + + try { + await resolveEncryptResult( + Promise.resolve({ data: {} }), + { metadata: { m: 1 } }, + 'bulkEncryptModels', + ) + expect(spy).toHaveBeenCalledWith( + expect.stringContaining('bulkEncryptModels audit metadata ignored'), + ) + + spy.mockClear() + await resolveEncryptResult( + Promise.resolve({ data: {} }), + {}, + 'encryptModel', + ) + expect(spy).not.toHaveBeenCalled() + } finally { + spy.mockRestore() + } + }) +}) + describe('throwPreservingCode', () => { it('rethrows as an Error carrying the FFI code', () => { try { diff --git a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts index c6ab542ec..ce9e87cdd 100644 --- a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts +++ b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts @@ -133,7 +133,13 @@ describe('a client whose decrypt requires the table', () => { "Cannot read properties of undefined (reading 'tableName')", ) } - return Promise.resolve({ data: {} }) + // A bare promise — NOT a thenable operation with `.audit()`. That is + // the whole point of this stub: it is the shape `WasmEncryptionClient` + // returns from `wasmResult`. The bulk methods resolve to an array, + // index-aligned with their input, as the real client does. + return Promise.resolve( + method.startsWith('bulk') ? { data: [{}] } : { data: {} }, + ) } const client = { requiresTableForDecrypt: true, @@ -156,7 +162,7 @@ describe('a client whose decrypt requires the table', () => { // Synchronous: the guard runs when the operation is built, so the failure // lands at the call site rather than as a rejected promise later. expect(() => dynamo.decryptModel({ pk: 'a' }, usersV2)).toThrow( - /wasm-inline client cannot read legacy EQL v2 items/, + /wasm-inline client cannot be paired with the legacy EQL v2 table/, ) // Refused before the client is touched, so the user never sees the // TypeError about `tableName`. @@ -168,10 +174,41 @@ describe('a client whose decrypt requires the table', () => { const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) expect(() => dynamo.bulkDecryptModels([{ pk: 'a' }], usersV2)).toThrow( - /wasm-inline client cannot read legacy EQL v2 items/, + /wasm-inline client cannot be paired with the legacy EQL v2 table/, ) }) + /** + * #788 review, minor finding. + * + * The guard runs on all four operations, not just the two read ones, so a + * plain-JS caller reaching the write path with a v2 table hits the SAME + * message. It must therefore not be phrased for reads only — "would fail at + * the first read" names an operation that never ran. + * + * Typed callers cannot get here (the write overloads are `AnyV3Table`-only, + * pinned by `client-compat.test-d.ts`), so this is about the message a JS + * caller or a cast lands on, not about reachable behaviour changing. + */ + it('is refused on the v2 WRITE path, with a message that does not claim a read', () => { + const { calls, client } = wasmShapedClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + for (const call of [ + () => dynamo.encryptModel({ pk: 'a' } as never, usersV2 as never), + () => dynamo.bulkEncryptModels([{ pk: 'a' }] as never, usersV2 as never), + ]) { + expect(call).toThrow( + /wasm-inline client cannot be paired with the legacy EQL v2 table/, + ) + // The read-path phrasing must not survive on a write. + expect(call).not.toThrow(/would fail at the first read/) + expect(call).not.toThrow(/cannot read legacy EQL v2 items/) + } + + expect(calls).toHaveLength(0) + }) + // v3 tables ARE forwarded the table, so this client works there — the guard // must not turn into a blanket rejection of the wasm entry. it('is accepted for an EQL v3 table, which is always given the table', async () => { @@ -183,4 +220,70 @@ describe('a client whose decrypt requires the table', () => { expect(calls).toHaveLength(1) expect(calls[0]?.argCount).toBe(2) }) + + /** + * #788 review follow-up: the same "v3 tables are unaffected" promise, on the + * WRITE path. + * + * The encrypt operations chained `.audit()` onto the client's result + * unconditionally. The native clients return a thenable operation carrying + * it; `WasmEncryptionClient.encryptModel` returns a plain + * `Promise` from `wasmResult` (it has no `.audit()` anywhere), + * so every v3 encrypt through this adapter died with + * `client.encryptModel(...).audit is not a function` — surfaced as a + * `DYNAMODB_ENCRYPTION_ERROR` failure, not a crash, so it read as a genuine + * encryption fault. The decrypt path already tolerates the bare promise via + * `resolveDecryptResult`; the write path must match. + */ + it('encrypts an EQL v3 table even though its encrypt returns a bare promise', async () => { + const { calls, client } = wasmShapedClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + const single = await dynamo.encryptModel({ pk: 'a' } as never, usersV3) + expect(single.failure).toBeUndefined() + + const bulk = await dynamo.bulkEncryptModels([{ pk: 'a' }] as never, usersV3) + expect(bulk.failure).toBeUndefined() + + expect(calls.map((c) => c.method)).toEqual([ + 'encryptModel', + 'bulkEncryptModels', + ]) + }) + + /** + * Audit metadata has nowhere to go on this client shape, so it is dropped — + * but the encrypt must still succeed rather than failing the whole write. + * Mirrors the decrypt path's documented behaviour. + */ + it('drops audit metadata on that client rather than failing the encrypt', async () => { + const { client } = wasmShapedClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + const result = await dynamo + .encryptModel({ pk: 'a' } as never, usersV3) + .audit({ metadata: { requestId: 'r-1' } }) + + expect(result.failure).toBeUndefined() + }) + + /** + * The tolerance must not swallow a malformed result into a fake success — + * the same guard `resolveDecryptResult` applies on read. + */ + it('rejects a bare encrypt result that is not { data } or { failure }', async () => { + const client = { + requiresTableForDecrypt: true, + getEncryptConfig: () => ({ v: 1, tables: { users_v3: {} } }), + encryptModel: () => Promise.resolve('not-a-result'), + bulkEncryptModels: () => Promise.resolve('not-a-result'), + decryptModel: () => Promise.resolve({ data: {} }), + bulkDecryptModels: () => Promise.resolve({ data: {} }), + } + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + const result = await dynamo.encryptModel({ pk: 'a' } as never, usersV3) + + expect(result.failure?.message).toMatch(/malformed result/) + }) }) diff --git a/packages/stack/src/dynamodb/helpers.ts b/packages/stack/src/dynamodb/helpers.ts index 186e36490..538091d31 100644 --- a/packages/stack/src/dynamodb/helpers.ts +++ b/packages/stack/src/dynamodb/helpers.ts @@ -144,6 +144,64 @@ export async function resolveDecryptResult( type DecryptFailure = { message: string; code?: string } +/** + * Resolve an encrypt call against either client shape — the write-path mirror + * of {@link resolveDecryptResult}. + * + * Both paths face the same split, and only the read one handled it. The native + * clients return a thenable operation carrying `.audit()`; the WASM client's + * `encryptModel` / `bulkEncryptModels` return a plain `Promise` + * from `wasmResult`, with no `.audit()` on this entry at all. Chaining it + * unconditionally threw `client.encryptModel(...).audit is not a function`, + * which `withResult` caught and reported as a `DYNAMODB_ENCRYPTION_ERROR` — + * so every v3 write through this adapter on the wasm entry looked like a + * genuine encryption fault (#788 review follow-up). + * + * Audit metadata still has nowhere to go on that shape, so it is dropped, and + * the drop is logged rather than silent — exactly as on decrypt. + */ +export async function resolveEncryptResult( + operation: unknown, + auditData: { metadata?: Record }, + context: 'encryptModel' | 'bulkEncryptModels', +): Promise< + { data: T; failure?: never } | { data?: never; failure: DecryptFailure } +> { + const chainable = operation as { + audit?: (data: { metadata?: Record }) => unknown + } + + if (typeof chainable?.audit !== 'function' && auditData.metadata) { + logger.debug( + `DynamoDB: ${context} audit metadata ignored — this client's encrypt does not return a chainable operation with .audit(). Audited encrypts need a client from the default @cipherstash/stack entry; the wasm-inline client's encrypt returns a plain promise.`, + ) + } + + const resolved = + typeof chainable?.audit === 'function' + ? await chainable.audit(auditData) + : await operation + + // Same fail-closed check the read path applies: a bare value has neither + // `data` nor `failure`, and casting it through would surface a fake success + // carrying `undefined` — here, an "encrypted" item that was never encrypted. + if ( + resolved === null || + typeof resolved !== 'object' || + (!('data' in resolved) && !('failure' in resolved)) + ) { + return { + failure: { + message: `DynamoDB: ${context} returned a malformed result — expected { data } or { failure }.`, + }, + } + } + + return resolved as + | { data: T; failure?: never } + | { data?: never; failure: DecryptFailure } +} + /** * Rethrow a Result failure as an `Error` that preserves the FFI error code. * `withResult`'s `ensureError` wraps non-Error objects, which would otherwise diff --git a/packages/stack/src/dynamodb/index.ts b/packages/stack/src/dynamodb/index.ts index 8367f4094..bc4c5194f 100644 --- a/packages/stack/src/dynamodb/index.ts +++ b/packages/stack/src/dynamodb/index.ts @@ -49,12 +49,20 @@ function assertClientTableVersionMatch( // otherwise throws a TypeError about `tableName` from deep inside // `requireTable`. Refuse the pairing here, where the message can name it // (#772 review, finding 10). + // + // The message is deliberately operation-NEUTRAL. This guard runs on all + // four operations, so a plain-JS caller (the write overloads are + // `AnyV3Table`-only, so TypeScript never reaches this) hits it on encrypt + // too — where "would fail at the first read" names an operation that never + // ran. The pairing is wrong in both directions anyway: `Encryption()` on + // that entry rejects a v2 schema outright, so the client can never hold + // this table (#788 review). if ( (encryptionClient as { requiresTableForDecrypt?: boolean }) .requiresTableForDecrypt ) { throw new Error( - `encryptedDynamoDB: the @cipherstash/stack/wasm-inline client cannot read legacy EQL v2 items. Its decrypt requires the table, and a v2 table carries none of the information it needs — so "${table.tableName}" would fail at the first read. Use the default @cipherstash/stack entry for tables that still hold EQL v2 items, or migrate the table to an EQL v3 schema (types.* domains) and pass that.`, + `encryptedDynamoDB: the @cipherstash/stack/wasm-inline client cannot be paired with the legacy EQL v2 table "${table.tableName}". That entry is EQL v3 only — Encryption() there rejects a v2 schema, and its model operations require a table it was initialized with — so both encrypt and decrypt would fail on this table. Use the default @cipherstash/stack entry for tables that still hold EQL v2 items, or migrate the table to an EQL v3 schema (types.* domains) and pass that.`, ) } return diff --git a/packages/stack/src/dynamodb/operations/bulk-encrypt-models.ts b/packages/stack/src/dynamodb/operations/bulk-encrypt-models.ts index eafee15aa..063d83545 100644 --- a/packages/stack/src/dynamodb/operations/bulk-encrypt-models.ts +++ b/packages/stack/src/dynamodb/operations/bulk-encrypt-models.ts @@ -5,6 +5,7 @@ import { deepClone, handleError, isV3Table, + resolveEncryptResult, throwPreservingCode, toEncryptedDynamoItem, } from '../helpers' @@ -43,12 +44,16 @@ export class BulkEncryptModelsOperation< return await withResult( async () => { const client = this.encryptionClient as CallableEncryptionClient - const encryptResult = await client - .bulkEncryptModels( + const encryptResult = await resolveEncryptResult< + Record[] + >( + client.bulkEncryptModels( this.items.map((item) => deepClone(item)), this.table, - ) - .audit(this.getAuditData()) + ), + this.getAuditData(), + 'bulkEncryptModels', + ) if (encryptResult.failure) { throwPreservingCode(encryptResult.failure) diff --git a/packages/stack/src/dynamodb/operations/encrypt-model.ts b/packages/stack/src/dynamodb/operations/encrypt-model.ts index 56a56ea47..18c5840a5 100644 --- a/packages/stack/src/dynamodb/operations/encrypt-model.ts +++ b/packages/stack/src/dynamodb/operations/encrypt-model.ts @@ -5,6 +5,7 @@ import { deepClone, handleError, isV3Table, + resolveEncryptResult, throwPreservingCode, toEncryptedDynamoItem, } from '../helpers' @@ -43,9 +44,13 @@ export class EncryptModelOperation< return await withResult( async () => { const client = this.encryptionClient as CallableEncryptionClient - const encryptResult = await client - .encryptModel(deepClone(this.item), this.table) - .audit(this.getAuditData()) + const encryptResult = await resolveEncryptResult< + Record + >( + client.encryptModel(deepClone(this.item), this.table), + this.getAuditData(), + 'encryptModel', + ) if (encryptResult.failure) { throwPreservingCode(encryptResult.failure) diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index d521f1405..df3533f9e 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -37,16 +37,23 @@ export type AnyEncryptedTable = * nominal `TypedEncryptionClient` parameter would reject a client built for * a narrower schema tuple. * - * Both NATIVE clients return a chainable operation on the decrypt paths — the - * nominal client's `DecryptModelOperation` and the typed wrapper's + * Both NATIVE clients return a chainable operation on every path — the nominal + * client's `DecryptModelOperation` and the typed wrapper's * `MappedDecryptOperation` each carry `.audit()` (the typed wrapper also takes * the table as a second argument). The operation classes handle both; see * `DecryptModelOperation` and `resolveDecryptResult`. * - * The wasm-inline client does not: its decrypt is a plain `async` method, so - * audit metadata is dropped (observably — `resolveDecryptResult` logs it) and - * its EQL v2 read path is refused outright by `assertClientTableVersionMatch`, - * because that path relies on calling decrypt WITHOUT a table. + * The wasm-inline client does not, on EITHER path: its encrypt and decrypt are + * plain `async` methods returning a bare `Promise`, so audit + * metadata is dropped (observably — `resolveDecryptResult` and + * `resolveEncryptResult` log it). Chaining `.audit()` unconditionally is + * therefore a bug, not just a lost audit record; the encrypt path made exactly + * that mistake and failed every v3 write on this entry (#788 review follow-up). + * + * Its EQL v2 path is refused outright by `assertClientTableVersionMatch` — the + * v2 read relies on calling decrypt WITHOUT a table, and that entry's + * `Encryption()` rejects a v2 schema anyway, so the pairing is wrong in both + * directions. */ export type DynamoDBEncryptionClient = { encryptModel(input: never, table: never): unknown diff --git a/skills/stash-dynamodb/SKILL.md b/skills/stash-dynamodb/SKILL.md index af53d7fca..d6ba5ed19 100644 --- a/skills/stash-dynamodb/SKILL.md +++ b/skills/stash-dynamodb/SKILL.md @@ -52,9 +52,15 @@ Non-encrypted attributes pass through unchanged. On decryption, the `__source` a | Schema | `encryptedTable` + `types.*` | `encryptedTable` + `encryptedColumn` | | Client | `Encryption({ schemas })` (or the deprecated `EncryptionV3`) | `Encryption({ schemas })` | | encrypt | ✅ | ❌ removed — write is v3 only | -| decrypt | ✅ | ✅ reads stored v2 items | +| decrypt | ✅ | ✅ reads stored v2 items — **native entry only** | | Nested fields | Flat dotted path (`"profile.ssn"`) | Nested group + `encryptedField` | +> **v2 reads need the native entry.** A client from `@cipherstash/stack/wasm-inline` +> (the documented entry for Deno, Bun, Cloudflare Workers and Supabase Edge Functions) +> is EQL v3 only: `encryptedDynamoDB` throws at the call site if you pass it a v2 +> table. Read legacy v2 items with a client from the default `@cipherstash/stack` +> entry. EQL v3 tables work on both entries. + There is no infrastructure migration between the versions — DynamoDB has no EQL extension to install and no schema to alter — and there is no automatic *data* migration either. The two use **different wire formats**; a stored v2 item still decrypts through a v2 table, but new writes are v3. To fully move a table to v3, re-encrypt every item with a v3 schema. The client must be built for the table. Build the client with the same v3 table you hand to `encryptedDynamoDB` — `Encryption({ schemas: [users] })` returns the typed v3 client for a concrete v3 schema set. Passing a v3 table to a client that never registered it (a client built for a different schema set) throws a clear error naming the table on the first operation, instead of failing later with an opaque FFI deserialization error. @@ -181,6 +187,14 @@ A v2 table is **read-only** through this adapter now: pass it to `decryptModel` `bulkEncryptModels` no longer accept a v2 table — author new writes with an EQL v3 table. +**Not on the WASM entry.** This whole section requires a client from the default +`@cipherstash/stack` entry. `@cipherstash/stack/wasm-inline` is EQL v3 only — +`Encryption()` there rejects a v2 schema outright, and `encryptedDynamoDB` refuses +the pairing at the call site with a message naming the fix. So on Deno, Bun, +Workers and Supabase Edge Functions, legacy v2 items are not readable through this +adapter; re-encrypt them to a v3 schema, or read them from a Node process using the +native entry. + ```typescript import { encryptedTable, encryptedColumn, encryptedField } from "@cipherstash/stack/schema" import { Encryption } from "@cipherstash/stack" @@ -497,7 +511,7 @@ Let `T` be inferred from the argument; do not pass explicit type arguments on th | `decryptModel` | `(item: Record, v2Table)` | `T` | | `bulkDecryptModels` | `(items: Record[], v2Table)` | `T[]` | -All operations are thenable (awaitable) and support `.audit({ metadata })` chaining — including the decrypt methods, whose metadata now forwards to ZeroKMS regardless of client shape (see the Setup note). +All operations are thenable (awaitable) and support `.audit({ metadata })` chaining. On the default `@cipherstash/stack` entry the metadata forwards to ZeroKMS on every operation, encrypt and decrypt alike (see the Setup note). The `@cipherstash/stack/wasm-inline` client has no `.audit()` — its operations return a plain promise — so audit metadata is **dropped** there (logged at debug level). The operation itself still succeeds; only the audit record is lost. Use the native entry when audit trails matter. Types exported from `@cipherstash/stack/dynamodb`: `EncryptedDynamoDBInstance`, `EncryptedDynamoDBConfig`, `EncryptedDynamoDBError`, `AnyEncryptedTable`, `DynamoDBEncryptionClient`, `EncryptedAttributes`, `DecryptedAttributes`, `AuditConfig`. diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index d766079df..036b3f19f 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -190,8 +190,8 @@ The SDK never logs plaintext data. | `@cipherstash/stack/types` | All TypeScript types | | `@cipherstash/stack-drizzle` | Drizzle ORM integration for EQL v3 schemas — the package root, EQL v3 only (see the `stash-drizzle` skill) | | `@cipherstash/stack-supabase` | `encryptedSupabase` wrapper for Supabase — EQL v3 only (see the `stash-supabase` skill) | -| `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. | -| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items. See the `stash-dynamodb` skill | +| `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. **EQL v3 only** — `Encryption()` here rejects a v2 schema, and its operations return plain Results with no `.audit()` or `.withLockContext()` chaining. | +| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items, on the native entry only. See the `stash-dynamodb` skill | | `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | ## Schema Definition From 3484411f5e359f69949388d9f61ec27d9d171723 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 14:25:02 +1000 Subject: [PATCH 079/123] fix(stack,bench): stop the client type asserting the .audit() it just disproved (#788 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the #788 review of this branch. The largest item is that the fix landed at runtime but the TYPE still declared the invariant it disproves. `CallableEncryptionClient.encryptModel` / `bulkEncryptModels` returned `ChainableEncryptOperation`, declaring an `.audit()` that the wasm-inline entry does not have — precisely the assumption that let the write path chain it unconditionally and fail every EQL v3 write there. The decrypt members already returned `unknown` with a docblock explaining why; encrypt never got the same treatment. Both are now `unknown`, `ChainableEncryptOperation` is deleted (it had no other reference), and the docblock covers all four members. This is not cosmetic: re-introducing `client.encryptModel(...).audit(...)` now fails to compile with TS2571, verified by reverting the call site under tsc. The regression is statically unreachable rather than merely fixed. The type is internal — absent from every emitted .d.ts — so there is no API change and no changeset. The test I had described as load-bearing was the weakest one. It stubbed `audit()` returning a promise; the native contract is `audit(): this` on a thenable whose `then()` calls `execute()`, so metadata is read back off the operation at execution time rather than passed forward. That stub passes even if `audit()` stops recording entirely. It now subclasses the real `EncryptionOperation`, mirroring what the decrypt half of the file already did, and asserts the metadata reaches `execute()` and that the operation runs exactly once. A native-failure case covers the other arm. Also strengthened, all verified by mutation (each kills exactly 2 tests): - the audit-drop test asserted only that the result succeeded — it now asserts the drop is logged and names the entry to switch to, plus a mirror proving a chainable client does NOT trip the drop path - both malformed-result loops gained `null` and `[]`. `null` makes the `resolved === null` clause load-bearing (without it `'data' in null` throws); `[]` is `typeof 'object'` so it must be rejected on the key checks `DecryptFailure` is renamed `ResultFailure` — it was already the encrypt helper's failure type, structurally identical, wrong name. bench: the `BenchPlaintextRow` docblock claimed it was "derived from the table so the two cannot drift again". It is hand-written. Deriving it was investigated and is actively worse: `InferInsertModel` describes the ENCRYPTED column and degrades to optional `any`, and `extractEncryptionSchema` returns the widened `AnyV3Table` whose index-signature column map admits both `encTxt: 'x'` and `encText: 12345`, which the literal rejects. The comment now says so and names the unit test as the real guard. The credential-free bench step moved ahead of `docker compose up`: leaving it last meant a database failure would skip the one check whose entire rationale is that it cannot be skipped for want of a database. Its `server.deps.inline` for test-kit was inert — nothing on that path imports it — and is gone; verified still passing with DATABASE_URL and all four CS_* vars unset. --- .github/workflows/tests-bench.yml | 22 ++--- packages/bench/src/drizzle/setup.ts | 12 ++- packages/bench/vitest.unit.config.ts | 1 - .../dynamodb/client-compat.test-d.ts | 35 ++++++++ .../dynamodb/resolve-decrypt.test.ts | 84 ++++++++++++++----- .../dynamodb/v2-table-forwarding.test.ts | 81 +++++++++++++++--- packages/stack/src/dynamodb/helpers.ts | 16 ++-- packages/stack/src/dynamodb/types.ts | 30 ++++--- 8 files changed, 218 insertions(+), 63 deletions(-) diff --git a/.github/workflows/tests-bench.yml b/.github/workflows/tests-bench.yml index 71ff378a6..488b03ba4 100644 --- a/.github/workflows/tests-bench.yml +++ b/.github/workflows/tests-bench.yml @@ -68,6 +68,18 @@ jobs: - name: Build stack + adapter packages run: pnpm exec turbo run build --filter @cipherstash/stack --filter @cipherstash/stack-drizzle + # Deliberately BEFORE Postgres. Separate config, no globalSetup: these + # need neither a database nor credentials, so they must not be reachable + # only after the steps that do — a docker failure below would otherwise + # skip the one check whose whole rationale is that it cannot be skipped. + # The seed keying they check was wrong for the entire v2 -> v3 port + # precisely because every suite that would have caught it needs + # credentials, and `tsc --noEmit` passes on a hand-written row type that + # agrees with itself (#772 review, finding 12). + - name: Run bench unit checks (no database, no credentials) + working-directory: packages/bench + run: pnpm test:unit + # Service-scoped `postgres`: local/docker-compose.yml also defines a # PostgREST that bench never talks to. # @@ -88,13 +100,3 @@ jobs: - name: Run bench smoke tests working-directory: packages/bench run: pnpm test:local db-only - - # Separate config, no globalSetup: these need neither a database nor - # credentials, so they cannot be skipped for want of either. The seed - # keying they check was wrong for the whole v2 -> v3 port precisely - # because every suite that would have caught it needs credentials, and - # `tsc --noEmit` passes on a hand-written row type that agrees with - # itself (#772 review, finding 12). - - name: Run bench unit checks (no database, no credentials) - working-directory: packages/bench - run: pnpm test:unit diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index 7978cd733..9aa08bae5 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -37,8 +37,16 @@ export const encryptionBenchTable = extractEncryptionSchema(benchTable) * returns it untouched, with no failure, and the plaintext then goes into an * `eql_v3_*` column (#772 review, finding 12). * - * Derived from the table so the two cannot drift again; - * `__unit__/seed-keys.test.ts` checks the row actually fills it. + * HAND-WRITTEN — it cannot be derived from `benchTable` without getting weaker, + * so `__unit__/seed-keys.test.ts` is the real drift guard, not the type. + * Drizzle's `InferInsertModel` describes the ENCRYPTED column (the custom type's + * `data` is the EQL envelope, not the plaintext) and degrades these three to + * optional `any`. Deriving from `encryptionBenchTable` is no better: + * `extractEncryptionSchema` returns the widened `AnyV3Table`, whose column map + * is an index signature — a type that admits `encTxt: 'x'` and `encText: 12345` + * alike, both of which the literal below rejects. + * + * Update it by hand when `benchTable` changes; the unit test will tell you. */ export type BenchPlaintextRow = { encText: string diff --git a/packages/bench/vitest.unit.config.ts b/packages/bench/vitest.unit.config.ts index fa01c8c55..6e8695ad8 100644 --- a/packages/bench/vitest.unit.config.ts +++ b/packages/bench/vitest.unit.config.ts @@ -13,6 +13,5 @@ import { defineConfig } from 'vitest/config' export default defineConfig({ test: { include: ['__unit__/**/*.test.ts'], - server: { deps: { inline: [/packages\/test-kit/] } }, }, }) diff --git a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts index b8995a67f..715b9b665 100644 --- a/packages/stack/__tests__/dynamodb/client-compat.test-d.ts +++ b/packages/stack/__tests__/dynamodb/client-compat.test-d.ts @@ -14,6 +14,7 @@ import { describe, expectTypeOf, it } from 'vitest' import type { EncryptedAttributes, EncryptedDynamoDBInstance } from '@/dynamodb' import { encryptedDynamoDB } from '@/dynamodb' +import type { CallableEncryptionClient } from '@/dynamodb/types' import type { EncryptionClient } from '@/encryption' import type { EncryptionClientFor } from '@/encryption/v3' import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' @@ -388,3 +389,37 @@ describe('operations chain and resolve', () => { expectTypeOf(result.data.email__source).toEqualTypeOf() }) }) + +/** + * The INTERNAL client view, `CallableEncryptionClient` — the shape the operation + * classes cast to. Distinct from everything above, which is the PUBLIC instance. + * + * All four members must return `unknown`. The shipped clients disagree about + * what an operation is: the nominal client returns a chainable + * `EncryptModelOperation`, the typed client a `MappedDecryptOperation`, and the + * wasm-inline client a bare `Promise` carrying no `.audit()` at all. + * Declaring a chainable shape here asserts an `.audit()` that entry does not + * have — and that assertion is not academic: it is what let the write path chain + * `.audit()` unconditionally and fail EVERY EQL v3 write on the wasm entry + * (#788 review follow-up). + * + * `unknown` forces every call site through `resolveEncryptResult` / + * `resolveDecryptResult`, which normalise all three shapes and fail closed. The + * encrypt members were the last two still declaring the lie. + */ +describe('the internal callable client view is shape-agnostic', () => { + it('returns unknown from all four members, so no shape is presumed', () => { + expectTypeOf< + ReturnType + >().toBeUnknown() + expectTypeOf< + ReturnType + >().toBeUnknown() + expectTypeOf< + ReturnType + >().toBeUnknown() + expectTypeOf< + ReturnType + >().toBeUnknown() + }) +}) diff --git a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts index f3ce936ae..77f31bf4e 100644 --- a/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts +++ b/packages/stack/__tests__/dynamodb/resolve-decrypt.test.ts @@ -75,11 +75,13 @@ describe('resolveDecryptResult', () => { // A non-conforming client that resolves to a bare value (or `{}`) has // neither `data` nor `failure`. Casting it straight through would surface a // fake success carrying `undefined`; the shape must be rejected instead. - for (const malformed of [{}, 42, undefined]) { + // `null` and `[]` pin the two clauses that are otherwise untested: without + // the `resolved === null` guard, `'data' in null` throws a TypeError, and an + // array passes `typeof === 'object'` so it must be rejected on the key checks. + for (const malformed of [{}, 42, undefined, null, []]) { const result = await resolveDecryptResult(Promise.resolve(malformed), {}) - expect(result.failure).toBeDefined() - expect(typeof result.failure?.message).toBe('string') + expect(result.failure?.message).toMatch(/malformed result/) expect(result.data).toBeUndefined() } }) @@ -201,26 +203,67 @@ describe('resolveDecryptResult', () => { * must resolve instead of throwing. */ describe('resolveEncryptResult', () => { - it('chains .audit and forwards metadata when present (native clients)', async () => { - let seen: unknown - let calls = 0 - const operation = { - audit(config: { metadata?: Record }) { - calls += 1 - seen = config.metadata - return Promise.resolve({ data: { encrypted: true } }) - }, + /** + * The NATIVE shape, not a hand-rolled lookalike. + * + * `EncryptionOperation.audit()` returns `this` and the operation is a thenable + * whose `then()` calls `execute()` — so the metadata is read back out of the + * operation at execution time, not passed forward as an argument. A stub whose + * `audit()` returns a promise satisfies the helper while testing none of that: + * break `audit()` so it stops recording, or break `then()` so it never + * executes, and such a stub still passes while every native encrypt loses its + * audit trail. Subclass the real base class instead, exactly as the decrypt + * half of this file does with `MappedDecryptOperation`. + */ + it('forwards metadata into a real native operation and executes it once', async () => { + let seenMetadata: Record | undefined + let executions = 0 + + class NativeEncrypt extends EncryptionOperation<{ encrypted: boolean }> { + override async execute(): Promise< + Result<{ encrypted: boolean }, EncryptionError> + > { + executions += 1 + seenMetadata = this.getAuditData().metadata + return { data: { encrypted: true } } + } } const result = await resolveEncryptResult( - operation, + new NativeEncrypt(), { metadata: { sub: 'u1' } }, 'encryptModel', ) expect(result).toEqual({ data: { encrypted: true } }) - expect(seen).toEqual({ sub: 'u1' }) - expect(calls).toBe(1) + // The metadata reached `execute()` — the only place it can reach ZeroKMS. + expect(seenMetadata).toEqual({ sub: 'u1' }) + // Awaiting a thenable that `.audit()` returned as `this` must not run it twice. + expect(executions).toBe(1) + }) + + it('propagates a native operation failure without unwrapping it', async () => { + class FailingEncrypt extends EncryptionOperation<{ encrypted: boolean }> { + override async execute(): Promise< + Result<{ encrypted: boolean }, EncryptionError> + > { + return { + failure: { + type: EncryptionErrorTypes.EncryptionError, + message: 'ffi exploded', + }, + } + } + } + + const result = await resolveEncryptResult( + new FailingEncrypt(), + { metadata: { sub: 'u1' } }, + 'encryptModel', + ) + + expect(result.failure?.message).toBe('ffi exploded') + expect(result.data).toBeUndefined() }) it('awaits a bare promise instead of throwing (wasm-inline encrypt)', async () => { @@ -242,16 +285,19 @@ describe('resolveEncryptResult', () => { }) it('returns a failure — not a fake success — for a malformed result', async () => { - // Fail closed. A bare value cast straight through would hand the caller an - // "encrypted" item that was never encrypted. - for (const malformed of [{}, 42, undefined]) { + // Fail closed on every non-Result shape. `null` and `[]` are here + // deliberately: `null` is what makes the `resolved === null` clause + // load-bearing (without it, `'data' in null` throws a TypeError rather than + // returning the intended message), and `typeof [] === 'object'` means an + // array reaches the property checks and must still be rejected. + for (const malformed of [{}, 42, undefined, null, []]) { const result = await resolveEncryptResult( Promise.resolve(malformed), {}, 'encryptModel', ) - expect(result.failure).toBeDefined() + expect(result.failure?.message).toMatch(/malformed result/) expect(result.data).toBeUndefined() } }) diff --git a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts index ce9e87cdd..d13ad10b7 100644 --- a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts +++ b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts @@ -18,10 +18,18 @@ * the adapter makes rather than on a live decrypt. That matters because the only * other coverage of this path is an integration suite requiring live ZeroKMS. */ -import { describe, expect, it } from 'vitest' +import { afterEach, describe, expect, it, vi } from 'vitest' import { encryptedDynamoDB } from '@/dynamodb' import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' +import { logger } from '@/utils/logger' + +// The audit-drop tests spy on the shared `logger` singleton, which other suites +// in this directory also patch. Each restores its own spy in a `finally`; this +// is the safety net so a patched method can never survive a test boundary. +afterEach(() => { + vi.restoreAllMocks() +}) const usersV2 = encryptedTableV2('users_v2', { email: encryptedColumn('email').equality(), @@ -253,18 +261,71 @@ describe('a client whose decrypt requires the table', () => { /** * Audit metadata has nowhere to go on this client shape, so it is dropped — - * but the encrypt must still succeed rather than failing the whole write. - * Mirrors the decrypt path's documented behaviour. + * but the encrypt must still succeed rather than failing the whole write, and + * the drop must be OBSERVABLE rather than silent. Asserting only that the + * result succeeded would pass even with the debug log deleted, and that log is + * the half that makes a missing audit record diagnosable. */ - it('drops audit metadata on that client rather than failing the encrypt', async () => { - const { client } = wasmShapedClient(['users_v3']) - const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + it('drops audit metadata observably, without failing the encrypt', async () => { + const spy = vi.spyOn(logger, 'debug').mockImplementation(() => {}) + + try { + const { client } = wasmShapedClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + const result = await dynamo + .encryptModel({ pk: 'a' } as never, usersV3) + .audit({ metadata: { requestId: 'r-1' } }) + + expect(result.failure).toBeUndefined() + expect(spy).toHaveBeenCalledWith( + expect.stringContaining('encryptModel audit metadata ignored'), + ) + // It must name the entry to switch to, not merely report a loss. + expect(spy.mock.calls.at(-1)?.[0]).toMatch(/@cipherstash\/stack/) + } finally { + spy.mockRestore() + } + }) + + /** + * The mirror: a client that DOES carry `.audit()` must not trip the drop path. + * Without this, a "fix" that logged unconditionally — or that stopped chaining + * `.audit()` at all — would leave every test green while silently discarding + * the native clients' audit trail. + */ + it('does not report a drop when the client can carry the metadata', async () => { + const spy = vi.spyOn(logger, 'debug').mockImplementation(() => {}) + + try { + let seen: unknown + const chainable = { + audit(config: { metadata?: Record }) { + seen = config.metadata + return Promise.resolve({ data: {} }) + }, + } + const client = { + getEncryptConfig: () => ({ v: 1, tables: { users_v3: {} } }), + encryptModel: () => chainable, + bulkEncryptModels: () => chainable, + decryptModel: () => Promise.resolve({ data: {} }), + bulkDecryptModels: () => Promise.resolve({ data: {} }), + } + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) - const result = await dynamo - .encryptModel({ pk: 'a' } as never, usersV3) - .audit({ metadata: { requestId: 'r-1' } }) + const result = await dynamo + .encryptModel({ pk: 'a' } as never, usersV3) + .audit({ metadata: { requestId: 'r-1' } }) - expect(result.failure).toBeUndefined() + expect(result.failure).toBeUndefined() + expect(seen).toEqual({ requestId: 'r-1' }) + expect(spy).not.toHaveBeenCalledWith( + expect.stringContaining('audit metadata ignored'), + ) + } finally { + spy.mockRestore() + } }) /** diff --git a/packages/stack/src/dynamodb/helpers.ts b/packages/stack/src/dynamodb/helpers.ts index 538091d31..4ee8efa59 100644 --- a/packages/stack/src/dynamodb/helpers.ts +++ b/packages/stack/src/dynamodb/helpers.ts @@ -101,7 +101,7 @@ export async function resolveDecryptResult( operation: unknown, auditData: { metadata?: Record }, ): Promise< - { data: T; failure?: never } | { data?: never; failure: DecryptFailure } + { data: T; failure?: never } | { data?: never; failure: ResultFailure } > { const chainable = operation as { audit?: (data: { metadata?: Record }) => unknown @@ -139,10 +139,16 @@ export async function resolveDecryptResult( return resolved as | { data: T; failure?: never } - | { data?: never; failure: DecryptFailure } + | { data?: never; failure: ResultFailure } } -type DecryptFailure = { message: string; code?: string } +/** + * The failure member both resolvers hand back — shared, because the encrypt and + * decrypt paths return structurally identical failures and `throwPreservingCode` + * consumes either. `code` is the FFI error code, preserved so `handleError` can + * read it back off the rethrown Error (the error-code contract in AGENTS.md). + */ +type ResultFailure = { message: string; code?: string } /** * Resolve an encrypt call against either client shape — the write-path mirror @@ -165,7 +171,7 @@ export async function resolveEncryptResult( auditData: { metadata?: Record }, context: 'encryptModel' | 'bulkEncryptModels', ): Promise< - { data: T; failure?: never } | { data?: never; failure: DecryptFailure } + { data: T; failure?: never } | { data?: never; failure: ResultFailure } > { const chainable = operation as { audit?: (data: { metadata?: Record }) => unknown @@ -199,7 +205,7 @@ export async function resolveEncryptResult( return resolved as | { data: T; failure?: never } - | { data?: never; failure: DecryptFailure } + | { data?: never; failure: ResultFailure } } /** diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index df3533f9e..83aac0414 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -62,15 +62,6 @@ export type DynamoDBEncryptionClient = { bulkDecryptModels(input: never, table: never): unknown } -type ChainableEncryptOperation = { - audit(data: { - metadata?: Record - }): PromiseLike< - | { data: T; failure?: never } - | { data?: never; failure: { message: string; code?: string } } - > -} - /** * @internal Callable view of {@link DynamoDBEncryptionClient}. * @@ -80,21 +71,28 @@ type ChainableEncryptOperation = { * satisfy. The operation classes therefore cast to this shape at the call site * — the same split the Drizzle v3 operators use. * - * `decryptModel` is intentionally untyped in its return: both shipped clients - * return a chainable operation, but different classes of one (the nominal - * client's `DecryptModelOperation`, the typed client's `MappedDecryptOperation`), - * and a custom client may return something else entirely. See - * `resolveDecryptResult`, which normalises all three. + * The returns are intentionally untyped on ALL FOUR members. The clients + * disagree about what an operation even is: the nominal client returns a + * chainable `EncryptModelOperation` / `DecryptModelOperation`, the typed client + * a `MappedDecryptOperation`, and the wasm-inline client a bare + * `Promise` with no `.audit()` anywhere on it. + * + * Declaring a chainable shape here asserts an `.audit()` that the wasm entry + * does not have, and that assertion was not academic — it is exactly what let + * the write path chain `.audit()` unconditionally and fail EVERY EQL v3 write on + * that entry (#788 review follow-up). `unknown` forces each call site through + * `resolveEncryptResult` / `resolveDecryptResult`, which normalise all three + * shapes and fail closed on anything else. */ export type CallableEncryptionClient = { encryptModel( input: Record, table: AnyEncryptedTable, - ): ChainableEncryptOperation> + ): unknown bulkEncryptModels( input: Record[], table: AnyEncryptedTable, - ): ChainableEncryptOperation[]> + ): unknown decryptModel( input: Record, table?: AnyEncryptedTable, From 36e515ae4fd9dbfb9fc59e47fa1247207036e1e1 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:29:40 +1000 Subject: [PATCH 080/123] fix(cli): stop encrypt cutover/drop acting on a guessed encrypted column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On a mixed table — a legacy v2 pair the classifier no longer sees, plus one unrelated EQL v3 column — pickEncryptedColumn's sole-EQL-column rule claims the v3 column for the v2 plaintext. Three separate defects turned that guess into wrong outcomes. cutover had no `via` gate at all. Its v3 branch returns from inside try with exitCode untouched, so the finally never fires process.exit: it printed "point your application at email_enc" and exited 0 while the v2 rename never ran. drop.ts:106 already gated on `via === 'sole'` for exactly this reason. The manifest hint was discarded. backfill records the true pairing, so the answer was on disk, but resolveColumnLifecycle dropped a hint that failed to resolve and re-picked without it — reaching the sole rule. Recording the pairing changed nothing. Split the two reasons a hint fails: a column that is GONE is stale and still falls through to convention; a column that EXISTS but is not EQL v3 (the legacy eql_v2_encrypted case) is reported by name. drop's remedy prescribed the guess — `--encrypted-column `. Recording it makes the next resolution `via: 'hint'`, which walks past the gate, and the coverage check passes vacuously because an unrelated backfilled column is non-NULL on every row. The message was the instruction manual for generating a live DROP COLUMN on the plaintext at exit 0. Tested at the layer that had none: resolve-eql.test.ts covered only explainUnresolved, and encrypt-v3.test.ts stubs resolveColumnLifecycle outright, so neither could see the hint discard. The new tests keep pickEncryptedColumn real and replace only the two I/O boundaries. --- .changeset/encrypt-lifecycle-mixed-table.md | 39 +++++ .../encrypt/__tests__/encrypt-v3.test.ts | 44 +++++- packages/cli/src/commands/encrypt/cutover.ts | 18 ++- packages/cli/src/commands/encrypt/drop.ts | 12 +- .../encrypt/lib/__tests__/resolve-eql.test.ts | 149 +++++++++++++++++- .../src/commands/encrypt/lib/resolve-eql.ts | 64 +++++++- 6 files changed, 315 insertions(+), 11 deletions(-) create mode 100644 .changeset/encrypt-lifecycle-mixed-table.md diff --git a/.changeset/encrypt-lifecycle-mixed-table.md b/.changeset/encrypt-lifecycle-mixed-table.md new file mode 100644 index 000000000..78d41244e --- /dev/null +++ b/.changeset/encrypt-lifecycle-mixed-table.md @@ -0,0 +1,39 @@ +--- +'stash': patch +--- + +`stash encrypt cutover` and `stash encrypt drop` no longer act on — or report +success for — an encrypted column they only guessed at. + +On a table holding both a legacy EQL v2 pair (`ssn` / `ssn_encrypted`) and one +unrelated EQL v3 column, the v2 ciphertext column is not classified as an EQL +column at all, so column resolution fell through to the "this is the table's +only EQL column" rule and claimed the unrelated v3 column. Three consequences, +all now fixed: + +- **`cutover` reported success for work it never did.** Its EQL v3 branch had no + guard on how the column was resolved, and returned without setting an exit + code — so it printed "point your application at `email_enc`" and exited 0 + while the v2 rename never ran. A scripted rollout read that as complete. It + now refuses, and exits 1, exactly as `drop` already did. + +- **The recorded pairing was discarded.** `encrypt backfill` writes the true + `encryptedColumn` to `.cipherstash/migrations.json`, so the answer was already + on disk — but a hint that failed to resolve was dropped entirely and the + re-resolution reached the guess. A hint naming a column that still exists but + is not an EQL v3 column is now reported as what it is (most often a legacy + `eql_v2_encrypted` counterpart) instead of being replaced by a guess. A + genuinely stale hint — one naming a column that is gone — still falls back to + the naming convention as before. + +- **`drop`'s refusal message prescribed the guess.** It told the user to re-run + `backfill --encrypted-column `. Following it recorded the + guess as fact, so the next run resolved "by hint", walked past the refusal, + and passed the coverage check vacuously — an unrelated but legitimately + backfilled column is non-NULL on every row — then generated a live + `DROP COLUMN` on the plaintext and exited 0. The message now asks for the + column that actually encrypts the named one, and says explicitly not to record + the guess. + +Pure-v2 and pure-v3 tables are unaffected, as are tables with two or more EQL v3 +columns (resolution already failed closed there). diff --git a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts index 45def1ef2..a0d583294 100644 --- a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts @@ -216,6 +216,36 @@ describe('encrypt cutover — EQL version awareness', () => { ) }) + // #772 review, finding 7. `drop` gated on `via === 'sole'`; `cutover` did + // not, and its v3 branch `return`s without setting exitCode — so a mixed + // table (a v2 pair the classifier no longer sees, plus one unrelated v3 + // column) produced a success-shaped message and exit 0 while the v2 rename + // never ran. A scripted rollout read that as done. + it("refuses a by-elimination ('sole') match rather than reporting success", async () => { + lifecycleMock.mockResolvedValue( + resolved( + { column: 'email_enc', domain: 'eql_v3_text_search', version: 3 }, + 'sole', + ), + ) + migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' }) + const exitSpy = spyExit() + + await cutoverCommand({ table: 'users', column: 'ssn' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('nothing confirms it encrypts "ssn"'), + ) + // The old message told the user to point their application at the guessed + // column, as though the lifecycle were complete. + expect(p.log.info).not.toHaveBeenCalledWith( + expect.stringContaining('point your application at email_enc'), + ) + expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + it('fails closed when EQL columns exist but none is identifiable', async () => { lifecycleMock.mockResolvedValue({ info: null, @@ -372,9 +402,21 @@ describe('encrypt drop — EQL version awareness', () => { expect(p.log.error).toHaveBeenCalledWith( expect.stringContaining('nothing confirms it encrypts "email"'), ) - expect(p.log.error).toHaveBeenCalledWith( + // The remedy must not prescribe the GUESS. Recording `secret_blob` makes + // the next resolution `via: 'hint'`, which walks past this very gate — and + // the coverage check then passes vacuously, because an unrelated but + // legitimately-backfilled column is non-NULL on every row. Following that + // advice generated a live DROP COLUMN on the plaintext at exit 0 + // (#772 review, finding 7). + expect(p.log.error).not.toHaveBeenCalledWith( expect.stringContaining('--encrypted-column secret_blob'), ) + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('--encrypted-column '), + ) + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('do not record secret_blob'), + ) expect(migrateMocks.countUnencrypted).not.toHaveBeenCalled() expect(writeFileMock).not.toHaveBeenCalled() expect(exitSpy).toHaveBeenCalledWith(1) diff --git a/packages/cli/src/commands/encrypt/cutover.ts b/packages/cli/src/commands/encrypt/cutover.ts index 676c4e244..5b9906134 100644 --- a/packages/cli/src/commands/encrypt/cutover.ts +++ b/packages/cli/src/commands/encrypt/cutover.ts @@ -70,7 +70,7 @@ export async function cutoverCommand(options: CutoverCommandOptions) { // DOMAIN TYPES (manifest name as a hint; the `_encrypted` naming is // a convention, never relied upon) before any phase/config checks so v3 // users get the real answer, not a confusing precondition error. - const { info, candidates } = await resolveColumnLifecycle( + const { info, candidates, unresolvedHint } = await resolveColumnLifecycle( client, options.table, options.column, @@ -86,6 +86,7 @@ export async function cutoverCommand(options: CutoverCommandOptions) { options.table, options.column, candidates, + unresolvedHint, ) if (!info && unresolved) { p.log.error(unresolved) @@ -96,6 +97,21 @@ export async function cutoverCommand(options: CutoverCommandOptions) { if (info?.version === 3) { const encryptedColumn = info.column + + // `via: 'sole'` means only that this is the table's ONE EQL v3 column — + // nothing ties it to the plaintext column the user named. On a mixed + // table (a v2 pair the classifier no longer sees, plus one unrelated v3 + // column) that guess is simply wrong, and reporting "nothing to do for + // EQL v3" for it told a scripted rollout the cut-over had succeeded when + // the v2 rename never ran. `drop.ts` already refuses a `'sole'` match for + // the same reason (#772 review, finding 7). + if (info.via === 'sole') { + p.log.error( + `${options.table}.${encryptedColumn} (${info.domain}) is the table's only EQL v3 column, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, this release no longer manages that lifecycle. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \`.`, + ) + exitCode = 1 + return + } if (state?.phase === 'dropped') { // Terminal phase — the lifecycle already finished. Not an error and // not "finish the backfill": there is nothing left to backfill. diff --git a/packages/cli/src/commands/encrypt/drop.ts b/packages/cli/src/commands/encrypt/drop.ts index 9beb6953a..ae5181e4f 100644 --- a/packages/cli/src/commands/encrypt/drop.ts +++ b/packages/cli/src/commands/encrypt/drop.ts @@ -74,7 +74,7 @@ export async function dropCommand(options: DropCommandOptions) { // column, droppable straight after `backfilled`. The version and the // encrypted column's name are resolved from the DOMAIN TYPES (manifest // name as a hint) — the `_encrypted` naming is a convention only. - const { info, candidates } = await resolveColumnLifecycle( + const { info, candidates, unresolvedHint } = await resolveColumnLifecycle( client, options.table, options.column, @@ -91,6 +91,7 @@ export async function dropCommand(options: DropCommandOptions) { options.table, options.column, candidates, + unresolvedHint, ) if (!info && unresolved) { p.log.error(unresolved) @@ -103,9 +104,16 @@ export async function dropCommand(options: DropCommandOptions) { // that destroys the only copy of this data. Dropping is the single // irreversible step in the lifecycle, so it demands a positively // asserted pairing (manifest hint or the naming convention). + // + // The remedy must NOT name `info.column`. That is the guess itself, and + // recording it turns the next resolution into `via: 'hint'`, which walks + // straight past this gate — while the coverage check below passes + // vacuously, because a legitimately-backfilled unrelated column is + // non-NULL on every row. Following the old message verbatim generated a + // live `DROP COLUMN` on the plaintext at exit 0 (#772 review, finding 7). if (info?.via === 'sole') { p.log.error( - `${options.table}.${info.column} (${info.domain}) is the table's only encrypted column, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Record the pairing and retry: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column ${info.column}\` (which writes it to the manifest), or set "encryptedColumn": "${info.column}" for this column in .cipherstash/migrations.json.`, + `${options.table}.${info.column} (${info.domain}) is the table's only encrypted column, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, this release no longer manages that lifecycle — do not record ${info.column}.`, ) exitCode = 1 return diff --git a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts index 1e5bc6cae..0310891c5 100644 --- a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts +++ b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts @@ -13,8 +13,40 @@ */ import type { EncryptedColumnInfo } from '@cipherstash/migrate' -import { describe, expect, it } from 'vitest' -import { explainUnresolved } from '../resolve-eql.js' +import type pg from 'pg' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +// Only the two I/O boundaries are replaced. `pickEncryptedColumn` stays real — +// it IS the resolution rule under test, and stubbing it (as `encrypt-v3.test.ts` +// stubs `resolveColumnLifecycle`) is why the hint-discard below had no coverage. +const readManifest = vi.hoisted(() => + vi.fn(async () => null as { tables: Record } | null), +) +const listEncryptedColumns = vi.hoisted(() => + vi.fn(async () => [] as EncryptedColumnInfo[]), +) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + readManifest, + listEncryptedColumns, +})) + +const { explainUnresolved, resolveColumnLifecycle } = await import( + '../resolve-eql.js' +) + +/** + * A `pg` double answering only the `pg_attribute` existence probe, from a set + * of column names the table is pretended to have. That probe is the whole + * point: it is what separates "the hint is stale" from "the hint names a real + * column that simply is not EQL v3". + */ +const clientWithColumns = (...columns: string[]): pg.ClientBase => + ({ + query: async (_sql: string, params: unknown[]) => ({ + rows: [{ exists: columns.includes(String(params[1])) }], + }), + }) as unknown as pg.ClientBase const v3 = ( column: string, @@ -57,3 +89,116 @@ describe('explainUnresolved', () => { expect(message).toContain('Cannot identify which encrypted column') }) }) + +/** + * #772 review, finding 7 — the mixed-table dead end. + * + * `classifyEqlDomain` recognises `eql_v3_*` only, so on a table holding a v2 + * pair (`ssn` / `ssn_encrypted`) plus one unrelated v3 column, the v2 ciphertext + * column is not a candidate at all. `pickEncryptedColumn`'s sole-EQL-column rule + * then claims the unrelated v3 column for `ssn`. + * + * `encrypt backfill` records the true pairing in the manifest, so the answer is + * on disk — but the hint was thrown away whenever it failed to resolve, and the + * re-pick without it reached the sole rule. Recording the pairing changed + * nothing, and `drop`'s remedy told the user to record the guess instead. + */ +describe('resolveColumnLifecycle — a recorded hint that is not a v3 candidate', () => { + const V3_OTHER: EncryptedColumnInfo = { + column: 'email_enc', + domain: 'eql_v3_text_search', + version: 3, + } + + beforeEach(() => { + vi.clearAllMocks() + readManifest.mockResolvedValue(null) + listEncryptedColumns.mockResolvedValue([]) + }) + + it('does not fall back to the sole-column guess when the manifest names a counterpart', async () => { + // The v2 pair is invisible to the classifier; only email_enc is a candidate. + listEncryptedColumns.mockResolvedValue([V3_OTHER]) + readManifest.mockResolvedValue({ + tables: { users: [{ column: 'ssn', encryptedColumn: 'ssn_encrypted' }] }, + }) + + const { info, unresolvedHint } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'ssn_encrypted', 'email_enc'), + 'users', + 'ssn', + ) + + expect(info).toBeNull() + expect(unresolvedHint).toBe('ssn_encrypted') + }) + + it('explains the recorded counterpart by name rather than listing candidates', async () => { + const message = explainUnresolved( + 'users', + 'ssn', + [V3_OTHER], + 'ssn_encrypted', + ) + + expect(message).toContain('ssn_encrypted') + expect(message).toContain('not an EQL v3 column') + // Naming the guess here is what sent users to `--encrypted-column email_enc`. + expect(message).not.toContain('--encrypted-column email_enc') + }) + + // The fallback the comment actually describes — a hint naming a column that + // is simply gone — must still resolve through convention. + it('still falls back to convention when the hint is genuinely stale', async () => { + listEncryptedColumns.mockResolvedValue([ + { column: 'ssn_encrypted', domain: 'eql_v3_text_eq', version: 3 }, + ]) + readManifest.mockResolvedValue({ + tables: { users: [{ column: 'ssn', encryptedColumn: 'ssn_old' }] }, + }) + + // `ssn_old` is gone from the table entirely — the genuinely stale case. + const { info } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'ssn_encrypted'), + 'users', + 'ssn', + ) + + expect(info?.column).toBe('ssn_encrypted') + expect(info?.via).toBe('convention') + }) + + it('resolves through the hint when it does name a candidate', async () => { + listEncryptedColumns.mockResolvedValue([ + { column: 'ssn_enc', domain: 'eql_v3_text_eq', version: 3 }, + ]) + readManifest.mockResolvedValue({ + tables: { users: [{ column: 'ssn', encryptedColumn: 'ssn_enc' }] }, + }) + + const { info, unresolvedHint } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'ssn_enc'), + 'users', + 'ssn', + ) + + expect(info?.column).toBe('ssn_enc') + expect(info?.via).toBe('hint') + expect(unresolvedHint).toBeUndefined() + }) + + // No manifest entry at all is the ordinary case; the sole rule still applies + // and `drop`'s own `via === 'sole'` gate is what refuses it. + it('leaves the sole-column rule alone when nothing was recorded', async () => { + listEncryptedColumns.mockResolvedValue([V3_OTHER]) + + const { info, unresolvedHint } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'email_enc'), + 'users', + 'ssn', + ) + + expect(info?.via).toBe('sole') + expect(unresolvedHint).toBeUndefined() + }) +}) diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index b4e595a10..df8f765b2 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -23,6 +23,17 @@ export interface ResolvedLifecycle { * identifiable — callers name them instead of erroring blind. */ candidates: EncryptedColumnInfo[] + /** + * The manifest's recorded `encryptedColumn` when it named a column that is + * NOT among `candidates` — i.e. the pairing is on record but the column it + * names is not an EQL v3 column (typically legacy `eql_v2_encrypted`, which + * `classifyEqlDomain` no longer recognises). + * + * Distinct from a stale hint. It means the answer IS known and disagrees + * with anything resolution could otherwise guess, so callers must fail closed + * naming it rather than fall through to `via: 'sole'` (#772 review, finding 7). + */ + unresolvedHint?: string } /** @@ -59,11 +70,45 @@ export async function resolveColumnLifecycle( )?.encryptedColumn const candidates = await listEncryptedColumns(client, table) - let info = hint ? pickEncryptedColumn(candidates, column, hint) : null - // A stale hint (column since renamed/retyped) must not mask a resolvable - // counterpart — fall back to convention + sole-EQL-column resolution. - if (!info) info = pickEncryptedColumn(candidates, column) - return { info, candidates } + const hinted = hint ? pickEncryptedColumn(candidates, column, hint) : null + if (hinted) return { info: hinted, candidates } + + // The hint named a column that is not a candidate. Two very different + // reasons, and they must not share an outcome: + // + // - the column no longer exists (renamed, dropped) — a genuinely STALE hint, + // which must not mask a resolvable counterpart, so fall through; + // - the column exists but is not an EQL v3 column — the usual shape being a + // legacy `eql_v2_encrypted` counterpart, which `classifyEqlDomain` stopped + // recognising. Here the pairing IS known and falling through would discard + // it in favour of a guess: on a mixed table the sole-EQL-column rule then + // claims an unrelated v3 column, `cutover` reports success for a rename it + // never performed, and `drop`'s remedy tells the user to record the guess + // (#772 review, finding 7). + if (hint && (await columnExists(client, table, hint))) { + return { info: null, candidates, unresolvedHint: hint } + } + + return { info: pickEncryptedColumn(candidates, column), candidates } +} + +/** Whether `column` exists on `table` at all, whatever its type. */ +async function columnExists( + client: pg.ClientBase, + table: string, + column: string, +): Promise { + const { rows } = await client.query<{ exists: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM pg_attribute + WHERE attrelid = to_regclass($1) + AND attname = $2 + AND attnum > 0 + AND NOT attisdropped + ) AS exists`, + [table, column], + ) + return rows[0]?.exists === true } /** @@ -87,7 +132,16 @@ export function explainUnresolved( table: string, column: string, candidates: readonly EncryptedColumnInfo[], + unresolvedHint?: string, ): string | null { + // The recorded pairing points at a real column that is not an EQL v3 column + // — almost always a legacy `eql_v2_encrypted` counterpart. Say exactly that: + // listing the v3 candidates here would invite the user to record one of them, + // which is how the guess used to get laundered into a `via: 'hint'` match. + if (unresolvedHint !== undefined) { + return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column, which this command no longer manages. Finish the EQL v2 lifecycle for this column with a stash release that still supports it, or drop the recorded pairing from .cipherstash/migrations.json if it is wrong.` + } + if (candidates.length === 0) return null const listed = candidates .map((c) => ` - ${c.column} (${c.domain})`) From 31b9e699bc2a2145a2213cdbeefbdc91c8cdfcaf Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:34:07 +1000 Subject: [PATCH 081/123] fix(cli): make stash init's scaffolded client compile, and gate it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both placeholder templates emitted `await Encryption({ schemas: [] })`. An empty schema set is a hard TS2769 against both overloads — deliberately, per S-6 — so every `stash init` left a project failing its first tsc, in the one file the CLI tells the user not to hand-edit. The old scaffold called EncryptionV3, whose `readonly AnyV3Table[]` bound accepted `[]`; collapsing it into an alias of Encryption tightened that away. Relaxing the constraint is not an option (it exists to catch a real mistake), and `stash init` has no table names in scope by design — it stopped introspecting, and `build-schema.ts` sets `schemas: []` on its own state. So the scaffold declares a sentinel table instead, which keeps the file compiling and keeps the "you haven't declared anything yet" signal: loadEncryptConfig exits 1 when `__stash_placeholder__` is the only table left, naming the file. The gap that let this ship is the more important half. packages/cli has no typecheck step (21 pre-existing errors), utils-codegen*.test.ts only `toContain`-matches fragments, and build-schema.test.ts mocks generatePlaceholderClient to '// placeholder' — so nothing anywhere compiled, parsed or executed the generated output. Both templates are now committed as `.generated.ts` fixtures compiled by a scoped tsconfig in CI, and pinned byte-for-byte to the generator by a unit test. Verified the gate reproduces the original TS2769 when the fixture is reverted. The `.generated.ts` suffix is load-bearing: biome.json already excludes it, so formatting cannot rewrite template output and break the byte comparison. --- .changeset/init-scaffold-compiles.md | 24 ++++++ .github/workflows/tests.yml | 9 +++ .../scaffold/drizzle.generated.ts | 65 ++++++++++++++++ .../scaffold/generic.generated.ts | 60 +++++++++++++++ packages/cli/package.json | 5 +- .../placeholder-client-fixture.test.ts | 74 +++++++++++++++++++ packages/cli/src/commands/init/utils.ts | 36 ++++++--- packages/cli/src/config/index.ts | 22 ++++++ packages/cli/tsconfig.scaffold.json | 25 +++++++ skills/stash-cli/SKILL.md | 2 +- 10 files changed, 309 insertions(+), 13 deletions(-) create mode 100644 .changeset/init-scaffold-compiles.md create mode 100644 packages/cli/__fixtures__/scaffold/drizzle.generated.ts create mode 100644 packages/cli/__fixtures__/scaffold/generic.generated.ts create mode 100644 packages/cli/src/commands/init/__tests__/placeholder-client-fixture.test.ts create mode 100644 packages/cli/tsconfig.scaffold.json diff --git a/.changeset/init-scaffold-compiles.md b/.changeset/init-scaffold-compiles.md new file mode 100644 index 000000000..84587fa34 --- /dev/null +++ b/.changeset/init-scaffold-compiles.md @@ -0,0 +1,24 @@ +--- +'stash': patch +--- + +The client file `stash init` writes now compiles. + +Both placeholder templates emitted `await Encryption({ schemas: [] })`, and +`Encryption` requires at least one table — an empty schema set is a deliberate +compile error, so it cannot be relaxed. Every `stash init` therefore left a +project whose first `tsc` or `next build` failed, in a file the CLI had just +told the user not to hand-edit. (The previous scaffold called `EncryptionV3`, +whose looser bound accepted `[]`; collapsing that into an alias of `Encryption` +tightened it.) + +The scaffold now declares a single sentinel table, `__stash_placeholder__`, so +the file typechecks as written. `stash encrypt` commands refuse to run while +that table is still the only one declared, and say so — rather than failing +later with a confusing "table not found". + +Nothing in the repo compiled this output before: `packages/cli` has no +typecheck step, the codegen tests only string-match fragments of the template, +and the step test stubs the generator out entirely. Both templates are now +committed as fixtures that CI typechecks, pinned byte-for-byte to the generator +so they cannot drift. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a94a885b9..644d3cadb 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -193,6 +193,15 @@ jobs: - name: Typecheck (stack — emitted declarations, not source) run: pnpm exec turbo run test:types:dist --filter @cipherstash/stack + # `stash init` writes a client file into the user's project and tells them + # not to hand-edit it. Nothing compiled that file, so tightening + # `Encryption` to require a non-empty schema set left every `stash init` + # emitting a project that fails its first `tsc` — with CI green (#772 + # review). The fixtures are pinned byte-for-byte to the generator by + # `placeholder-client-fixture.test.ts`. + - name: Typecheck (stash init's scaffolded client) + run: pnpm --filter stash run typecheck:scaffold + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts new file mode 100644 index 000000000..a0f9b77f0 --- /dev/null +++ b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts @@ -0,0 +1,65 @@ +/** + * CipherStash encryption client — placeholder. + * + * `stash init` wrote this file. It is intentionally NOT a real Drizzle + * schema. Your existing schema files (typically under `src/db/`) remain + * authoritative — your agent will edit those directly when you encrypt a + * column, then update the `Encryption({ schemas: [...] })` call below + * to reference the encrypted tables you declared there. + * + * Until that happens, the encryption client is initialised with a single + * placeholder table so that this file compiles, and `stash encrypt` + * commands refuse to run and point back here. + * + * This project uses EQL v3. Encrypted columns are concrete Postgres domains + * built with the `types.*` factories from `@cipherstash/stack-drizzle`. + * Each domain's query capabilities are FIXED by the type you pick — there is + * no capability config object. Choose the factory whose capabilities you need: + * types.Text / types.Integer / … storage only (encrypt/decrypt, no queries) + * types.TextEq / types.IntegerEq equality (eq, inArray) + * types.IntegerOrd / types.DateOrd equality + order/range (gt/lt/between/sort) + * types.TextMatch free-text match only + * types.TextSearch equality + order/range + free-text + * types.Json encrypted-JSONB containment + selectors + * + * --- Pattern reference (copy into your real schema, do NOT use as-is) --- + * + * Encrypted twin column for an existing populated column (path 3 — lifecycle): + * + * import { pgTable, integer, text } from 'drizzle-orm/pg-core' + * import { types } from '@cipherstash/stack-drizzle' + * + * export const users = pgTable('users', { + * id: integer('id').primaryKey().generatedAlwaysAsIdentity(), + * email: text('email').notNull(), // existing plaintext, unchanged for now + * email_encrypted: types.TextSearch('email_encrypted'), // encrypted twin, NULLABLE — never .notNull() + * }) + * + * Net-new encrypted column (path 1 — declare encrypted from the start): + * + * export const orders = pgTable('orders', { + * id: integer('id').primaryKey().generatedAlwaysAsIdentity(), + * billing_address: types.TextEq('billing_address'), + * }) + * + * Once you have encrypted tables declared, harvest them and pass to Encryption(): + * + * import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' + * import { Encryption } from '@cipherstash/stack/v3' + * import { users, orders } from './db/schema' + * + * export const encryptionClient = await Encryption({ + * schemas: [extractEncryptionSchema(users), extractEncryptionSchema(orders)], + * }) + */ +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +// REPLACE THIS. It exists only so this file compiles before you have declared +// any encrypted tables — `Encryption` requires at least one. Swap it for your +// real tables (see the patterns above); `stash encrypt` refuses to run while +// the placeholder is still here. +export const placeholderTable = encryptedTable('__stash_placeholder__', { + replace_me: types.Text('replace_me'), +}) + +export const encryptionClient = await Encryption({ schemas: [placeholderTable] }) diff --git a/packages/cli/__fixtures__/scaffold/generic.generated.ts b/packages/cli/__fixtures__/scaffold/generic.generated.ts new file mode 100644 index 000000000..4f5831847 --- /dev/null +++ b/packages/cli/__fixtures__/scaffold/generic.generated.ts @@ -0,0 +1,60 @@ +/** + * CipherStash encryption client — placeholder. + * + * `stash init` wrote this file. It is intentionally NOT a real schema + * definition. Your existing schema files remain authoritative — your + * agent will declare encrypted columns there and update the + * `Encryption({ schemas: [...] })` call below to reference them. + * + * Until that happens, the encryption client is initialised with a single + * placeholder table so that this file compiles, and `stash encrypt` + * commands refuse to run and point back here. + * + * This project uses EQL v3. Encrypted columns are concrete Postgres domains + * built with the `types.*` factories from `@cipherstash/stack/eql/v3` + * (also re-exported from `@cipherstash/stack/v3`). Each domain's query + * capabilities are FIXED by the type you pick — there is no chainable + * capability tuner. Choose the factory whose capabilities you need: + * types.Text / types.Integer / … storage only (encrypt/decrypt, no queries) + * types.TextEq / types.IntegerEq equality + * types.IntegerOrd / types.DateOrd equality + order/range + * types.TextMatch free-text match only + * types.TextSearch equality + order/range + free-text + * types.Json encrypted-JSONB containment + selectors + * + * --- Pattern reference (copy into your real schema, do NOT use as-is) --- + * + * Encrypted twin column for an existing populated column (path 3 — lifecycle): + * + * import { encryptedTable, types } from '@cipherstash/stack/v3' + * + * export const users = encryptedTable('users', { + * email_encrypted: types.TextSearch('email_encrypted'), + * }) + * + * Net-new encrypted column (path 1 — declare encrypted from the start): + * + * export const orders = encryptedTable('orders', { + * billing_address: types.TextEq('billing_address'), + * }) + * + * Once you have encrypted tables declared, pass them to Encryption(): + * + * import { Encryption } from '@cipherstash/stack/v3' + * import { users, orders } from './db/schema' + * + * export const encryptionClient = await Encryption({ + * schemas: [users, orders], + * }) + */ +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +// REPLACE THIS. It exists only so this file compiles before you have declared +// any encrypted tables — `Encryption` requires at least one. Swap it for your +// real tables (see the patterns above); `stash encrypt` refuses to run while +// the placeholder is still here. +export const placeholderTable = encryptedTable('__stash_placeholder__', { + replace_me: types.Text('replace_me'), +}) + +export const encryptionClient = await Encryption({ schemas: [placeholderTable] }) diff --git a/packages/cli/package.json b/packages/cli/package.json index 8f1c36b90..5b81308aa 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,7 +1,7 @@ { "name": "stash", "version": "1.0.0-rc.4", - "description": "CipherStash CLI — the one stash command for auth, init, encryption schema, database setup, and secrets.", + "description": "CipherStash CLI \u2014 the one stash command for auth, init, encryption schema, database setup, and secrets.", "repository": { "type": "git", "url": "git+https://github.com/cipherstash/stack.git", @@ -41,6 +41,7 @@ "postbuild": "chmod +x ./dist/bin/stash.js", "dev": "tsup --watch", "test": "vitest run", + "typecheck:scaffold": "tsc -p tsconfig.scaffold.json", "test:e2e": "vitest run --config vitest.integration.config.ts", "lint": "biome check ." }, @@ -56,7 +57,7 @@ "posthog-node": "^5.41.0", "zod": "^3.25.76" }, - "//optionalDependencies": "@cipherstash/auth ships per-platform native bindings as optional peerDependencies. pnpm does not auto-install platform-matched optional peer deps, so we declare them here as optionalDependencies — pnpm then picks the binary matching the host's os/cpu (from each sub-package's own package.json) and ignores the rest. All seven names share a single catalog entry to keep them in lockstep.", + "//optionalDependencies": "@cipherstash/auth ships per-platform native bindings as optional peerDependencies. pnpm does not auto-install platform-matched optional peer deps, so we declare them here as optionalDependencies \u2014 pnpm then picks the binary matching the host's os/cpu (from each sub-package's own package.json) and ignores the rest. All seven names share a single catalog entry to keep them in lockstep.", "optionalDependencies": { "@cipherstash/auth-darwin-arm64": "catalog:repo", "@cipherstash/auth-darwin-x64": "catalog:repo", diff --git a/packages/cli/src/commands/init/__tests__/placeholder-client-fixture.test.ts b/packages/cli/src/commands/init/__tests__/placeholder-client-fixture.test.ts new file mode 100644 index 000000000..c3ddbf548 --- /dev/null +++ b/packages/cli/src/commands/init/__tests__/placeholder-client-fixture.test.ts @@ -0,0 +1,74 @@ +/** + * Binds `__fixtures__/scaffold/*.ts` to what `generatePlaceholderClient` + * actually emits. + * + * The fixtures are the only thing in the repo that COMPILES the file + * `stash init` writes into a user's project — `tsconfig.scaffold.json` and the + * CI step that runs it exist for that. But a typechecked fixture is worthless + * if the generator can drift away from it, and the existing codegen tests only + * `toContain`-match fragments while `build-schema.test.ts` stubs the generator + * out entirely. That combination is how `Encryption({ schemas: [] })` shipped: + * every test was green and no compiler ever saw the output (#772 review, + * finding 6). + * + * So: byte-for-byte, both directions. Change a template, regenerate the + * fixture; the compiler then has an opinion about it. + */ +import { readFileSync } from 'node:fs' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' +import { PLACEHOLDER_TABLE_NAME } from '@/config/index.js' +import { generatePlaceholderClient } from '../utils.js' + +const FIXTURE_DIR = path.resolve( + fileURLToPath(import.meta.url), + '../../../../../__fixtures__/scaffold', +) + +const fixture = (name: string) => + readFileSync(path.join(FIXTURE_DIR, name), 'utf-8') + +describe('the scaffolded client fixtures match the generator', () => { + // Named `.generated.ts` so biome.json's existing exclusion leaves them alone: + // they are template OUTPUT, and reformatting them would break the byte-for-byte + // comparison below (which is the only thing tying the compiler to the generator). + it.each([ + ['generic.generated.ts', 'postgresql'], + ['drizzle.generated.ts', 'drizzle'], + ] as const)('%s', (file, integration) => { + expect(fixture(file)).toBe(generatePlaceholderClient(integration)) + }) + + // `supabase` shares the generic template; pin that so a future split does not + // silently leave the supabase path ungated. + it('supabase reuses the generic template', () => { + expect(generatePlaceholderClient('supabase')).toBe( + fixture('generic.generated.ts'), + ) + }) +}) + +describe('the scaffold compiles because it declares a table', () => { + it.each([ + 'generic.generated.ts', + 'drizzle.generated.ts', + ])('%s passes a non-empty schema set', (file) => { + const body = fixture(file) + // The empty form is a hard TS2769 against both overloads — `Encryption` + // requires at least one table by design (S-6), so the scaffold cannot go + // back to `schemas: []` without breaking every project it is written into. + expect(body).not.toContain('Encryption({ schemas: [] })') + expect(body).toContain('schemas: [placeholderTable]') + }) + + it.each([ + 'generic.generated.ts', + 'drizzle.generated.ts', + ])('%s uses the sentinel name the config loader refuses', (file) => { + // `loadEncryptConfig` exits 1 when this is the only table left, so the + // two must agree — otherwise the user gets a confusing "table not found" + // from whichever command runs next instead of "you never replaced this". + expect(fixture(file)).toContain(`'${PLACEHOLDER_TABLE_NAME}'`) + }) +}) diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index fc0e6aec1..282a0320b 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -384,9 +384,9 @@ const DRIZZLE_PLACEHOLDER = `/** * column, then update the \`Encryption({ schemas: [...] })\` call below * to reference the encrypted tables you declared there. * - * Until that happens, the encryption client is initialised with no - * schemas, and \`stash encrypt\` commands will surface a clear error - * pointing at this file. + * Until that happens, the encryption client is initialised with a single + * placeholder table so that this file compiles, and \`stash encrypt\` + * commands refuse to run and point back here. * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack-drizzle\`. @@ -429,9 +429,17 @@ const DRIZZLE_PLACEHOLDER = `/** * schemas: [extractEncryptionSchema(users), extractEncryptionSchema(orders)], * }) */ -import { Encryption } from '@cipherstash/stack/v3' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +// REPLACE THIS. It exists only so this file compiles before you have declared +// any encrypted tables — \`Encryption\` requires at least one. Swap it for your +// real tables (see the patterns above); \`stash encrypt\` refuses to run while +// the placeholder is still here. +export const placeholderTable = encryptedTable('__stash_placeholder__', { + replace_me: types.Text('replace_me'), +}) -export const encryptionClient = await Encryption({ schemas: [] }) +export const encryptionClient = await Encryption({ schemas: [placeholderTable] }) ` const GENERIC_PLACEHOLDER = `/** @@ -442,9 +450,9 @@ const GENERIC_PLACEHOLDER = `/** * agent will declare encrypted columns there and update the * \`Encryption({ schemas: [...] })\` call below to reference them. * - * Until that happens, the encryption client is initialised with no - * schemas, and \`stash encrypt\` commands will surface a clear error - * pointing at this file. + * Until that happens, the encryption client is initialised with a single + * placeholder table so that this file compiles, and \`stash encrypt\` + * commands refuse to run and point back here. * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack/eql/v3\` @@ -483,7 +491,15 @@ const GENERIC_PLACEHOLDER = `/** * schemas: [users, orders], * }) */ -import { Encryption } from '@cipherstash/stack/v3' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' + +// REPLACE THIS. It exists only so this file compiles before you have declared +// any encrypted tables — \`Encryption\` requires at least one. Swap it for your +// real tables (see the patterns above); \`stash encrypt\` refuses to run while +// the placeholder is still here. +export const placeholderTable = encryptedTable('__stash_placeholder__', { + replace_me: types.Text('replace_me'), +}) -export const encryptionClient = await Encryption({ schemas: [] }) +export const encryptionClient = await Encryption({ schemas: [placeholderTable] }) ` diff --git a/packages/cli/src/config/index.ts b/packages/cli/src/config/index.ts index 7103d7baa..c304f7d58 100644 --- a/packages/cli/src/config/index.ts +++ b/packages/cli/src/config/index.ts @@ -238,5 +238,27 @@ export async function loadEncryptConfig( ) process.exit(1) } + + // `stash init` scaffolds a client holding one placeholder table, because + // `Encryption` requires a non-empty schema set and the scaffold has no real + // tables to name yet. Reaching here with only that table means the user never + // replaced it — which used to surface as a confusing "table not found" from + // whichever command ran next. + const tables = Object.keys(config.tables ?? {}) + if (tables.length === 1 && tables[0] === PLACEHOLDER_TABLE_NAME) { + console.error( + `Error: ${encryptClientPath} still contains the placeholder table \`${PLACEHOLDER_TABLE_NAME}\` that \`stash init\` wrote.\n\nDeclare your encrypted columns and pass those tables to Encryption({ schemas: [...] }) in that file, then re-run this command.`, + ) + process.exit(1) + } + return config } + +/** + * The table name `stash init`'s scaffold uses so the file it writes compiles. + * + * Kept in sync with the templates in `commands/init/utils.ts` by + * `__tests__/placeholder-client-fixture.test.ts`, which also typechecks them. + */ +export const PLACEHOLDER_TABLE_NAME = '__stash_placeholder__' diff --git a/packages/cli/tsconfig.scaffold.json b/packages/cli/tsconfig.scaffold.json new file mode 100644 index 000000000..1a11f6286 --- /dev/null +++ b/packages/cli/tsconfig.scaffold.json @@ -0,0 +1,25 @@ +{ + // Compiles the exact files `stash init` writes into a user's project. + // + // Nothing else in the repo did. `packages/cli` has no typecheck script (21 + // pre-existing errors), the codegen tests only string-match the templates, + // and `build-schema.test.ts` stubs the generator out entirely — so when + // `Encryption` was tightened to require a non-empty schema set, both + // placeholders started emitting a file that fails `tsc` on first build, in a + // file the CLI had just told the user not to hand-edit, and CI stayed green + // (#772 review, finding 6). + // + // Deliberately scoped to the fixtures so it gates the scaffold without + // waiting on the rest of the package to compile. + "extends": "../../tsconfig.json", + "compilerOptions": { + "noEmit": true, + "module": "ESNext", + "moduleResolution": "bundler", + "target": "ESNext", + "lib": ["ESNext"], + "strict": true, + "skipLibCheck": true + }, + "include": ["__fixtures__/scaffold/*.ts"] +} diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index aa1c03c27..49873523e 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -221,7 +221,7 @@ Seven mechanical steps, no agent handoff. It prompts only when it can't pick a s 1. **Authenticate** — silent when a valid token exists. 2. **Resolve database** — per the resolution order above; verifies the connection. 3. **Resolve proxy choice** — CipherStash Proxy or direct SDK access (the default). Stored as `usesProxy` in `context.json`. Set by `--proxy` / `--no-proxy`; non-TTY without a flag defaults to SDK. -4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. +4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. The placeholder declares one sentinel table, `__stash_placeholder__`, because `Encryption` requires at least one — replace it with the tables you actually encrypt. Until you do, `stash encrypt` commands exit 1 and point back at that file. 5. **Install dependencies** — one combined prompt for `@cipherstash/stack` and `stash`. Skipped when both are present. 6. **Install EQL** — always EQL v3. **Drizzle** projects generate a v3 install migration (the same output as `eql migration --drizzle`, including the `cs_migrations` tracking schema) so the install lands in your migration history — apply it with `drizzle-kit migrate`; requires `drizzle-kit` to be installed and configured. **Prisma Next** is skipped (it installs EQL via `prisma-next migrate`). Everything else runs `eql install` directly against the resolved database, and is skipped when EQL is already installed. 7. **Gather context** — detects available coding agents and writes `.cipherstash/context.json`. From 04e0981d266ea36842152241585e7b74940f03d9 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 09:51:31 +1000 Subject: [PATCH 082/123] fix(cli): keep pure-v2 tables on the v2 ladder (#787 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `unresolvedHint` fail-closed added for #772 finding 7 had no `candidates.length > 0` guard, so it fired on pure EQL v2 tables too. `encrypt backfill` records `encryptedColumn` in migrations.json unconditionally, v2 included. On a pure-v2 table `listEncryptedColumns` returns [] (the classifier recognises `eql_v3_*` only), so the hint failed to resolve, `columnExists` found the real `eql_v2_encrypted` column, and `cutover` / `drop` exited 1 — telling users to downgrade for a lifecycle this same build still fully implements in cutover.ts / drop.ts. Gate the fail-closed on a non-empty candidate list. Finding 7's protection is unchanged: the mixed table it targets always has candidates. Order `explainUnresolved`'s empty-candidates fall-through ahead of the hint branch so both agree for direct callers. Also drop the "this release no longer manages that lifecycle" claim from the cutover/drop `via: 'sole'` messages — the build does still implement it; the command simply resolves EQL v3 counterparts only. Tests cover the previously untested shape: candidates [] + a recorded hint. Review follow-ups: - Extend the placeholder-table guard to `encrypt backfill` via loadEncryptionContext; correct the changeset/skill wording to name the commands that actually refuse (cutover/drop never read the client file). - Alias `@cipherstash/migrate` to source in the CLI unit vitest config, so `pnpm --filter stash test` no longer needs a prior workspace build (verified: 888 tests pass with packages/migrate/dist removed). - Revert the unrelated em-dash re-encoding in packages/cli/package.json. --- .changeset/encrypt-lifecycle-mixed-table.md | 9 +- .changeset/init-scaffold-compiles.md | 9 +- packages/cli/package.json | 4 +- .../__tests__/context-placeholder.test.ts | 89 +++++++++++++++++++ packages/cli/src/commands/encrypt/context.ts | 20 ++++- packages/cli/src/commands/encrypt/cutover.ts | 2 +- packages/cli/src/commands/encrypt/drop.ts | 2 +- .../encrypt/lib/__tests__/resolve-eql.test.ts | 56 ++++++++++-- .../src/commands/encrypt/lib/resolve-eql.ts | 77 +++++++++++----- packages/cli/vitest.config.ts | 7 ++ skills/stash-cli/SKILL.md | 2 +- 11 files changed, 237 insertions(+), 40 deletions(-) create mode 100644 packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts diff --git a/.changeset/encrypt-lifecycle-mixed-table.md b/.changeset/encrypt-lifecycle-mixed-table.md index 78d41244e..95bebb711 100644 --- a/.changeset/encrypt-lifecycle-mixed-table.md +++ b/.changeset/encrypt-lifecycle-mixed-table.md @@ -35,5 +35,10 @@ all now fixed: column that actually encrypts the named one, and says explicitly not to record the guess. -Pure-v2 and pure-v3 tables are unaffected, as are tables with two or more EQL v3 -columns (resolution already failed closed there). +The new refusal is scoped to the mixed table it was written for: it applies only +when the table actually holds EQL v3 columns that a guess could wrongly claim. A +pure-v2 table has none, so it still falls through to the EQL v2 lifecycle exactly +as before — including when `encrypt backfill` recorded an `encryptedColumn` for +it, which it does for v2 columns too. Pure-v2 and pure-v3 tables are therefore +unaffected, as are tables with two or more EQL v3 columns (resolution already +failed closed there). diff --git a/.changeset/init-scaffold-compiles.md b/.changeset/init-scaffold-compiles.md index 84587fa34..f20ae5d4d 100644 --- a/.changeset/init-scaffold-compiles.md +++ b/.changeset/init-scaffold-compiles.md @@ -13,9 +13,12 @@ whose looser bound accepted `[]`; collapsing that into an alias of `Encryption` tightened it.) The scaffold now declares a single sentinel table, `__stash_placeholder__`, so -the file typechecks as written. `stash encrypt` commands refuse to run while -that table is still the only one declared, and say so — rather than failing -later with a confusing "table not found". +the file typechecks as written. Every command that reads the encryption client +— `stash db push`, `stash db validate`, and `stash encrypt backfill` — refuses +to run while that table is still the only one declared, and names it, rather +than failing later with a confusing "table not found". (`stash encrypt cutover` +and `stash encrypt drop` do not read the client file at all; they resolve +against the database.) Nothing in the repo compiled this output before: `packages/cli` has no typecheck step, the codegen tests only string-match fragments of the template, diff --git a/packages/cli/package.json b/packages/cli/package.json index 5b81308aa..5cafc5d7f 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,7 +1,7 @@ { "name": "stash", "version": "1.0.0-rc.4", - "description": "CipherStash CLI \u2014 the one stash command for auth, init, encryption schema, database setup, and secrets.", + "description": "CipherStash CLI — the one stash command for auth, init, encryption schema, database setup, and secrets.", "repository": { "type": "git", "url": "git+https://github.com/cipherstash/stack.git", @@ -57,7 +57,7 @@ "posthog-node": "^5.41.0", "zod": "^3.25.76" }, - "//optionalDependencies": "@cipherstash/auth ships per-platform native bindings as optional peerDependencies. pnpm does not auto-install platform-matched optional peer deps, so we declare them here as optionalDependencies \u2014 pnpm then picks the binary matching the host's os/cpu (from each sub-package's own package.json) and ignores the rest. All seven names share a single catalog entry to keep them in lockstep.", + "//optionalDependencies": "@cipherstash/auth ships per-platform native bindings as optional peerDependencies. pnpm does not auto-install platform-matched optional peer deps, so we declare them here as optionalDependencies — pnpm then picks the binary matching the host's os/cpu (from each sub-package's own package.json) and ignores the rest. All seven names share a single catalog entry to keep them in lockstep.", "optionalDependencies": { "@cipherstash/auth-darwin-arm64": "catalog:repo", "@cipherstash/auth-darwin-x64": "catalog:repo", diff --git a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts new file mode 100644 index 000000000..ad32cded8 --- /dev/null +++ b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts @@ -0,0 +1,89 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { PLACEHOLDER_TABLE_NAME } from '@/config/index.js' + +/** + * `stash encrypt` does not load its client through `loadEncryptConfig`, so the + * placeholder guard that command group needs lives in `loadEncryptionContext`. + * Both must refuse the un-replaced scaffold; otherwise `requireTable` reports + * `Table "users" was not found … Available: __stash_placeholder__`, which names + * the symptom instead of the cause (#787 review). + * + * Runs against the real jiti runtime — the guard reads the tables harvested + * from an actually-evaluated client module, which is the part worth pinning. + */ +describe('loadEncryptionContext — the un-replaced init scaffold', () => { + let tmpDir: string + let originalCwd: () => string + + const writeProject = (clientBody: string) => { + fs.writeFileSync( + path.join(tmpDir, 'stash.config.ts'), + `export default { + databaseUrl: 'postgresql://u:p@127.0.0.1:5432/db', + client: './client.ts', + }`, + ) + fs.writeFileSync(path.join(tmpDir, 'client.ts'), clientBody) + process.cwd = () => tmpDir + } + + /** + * The duck-typed shapes `loadEncryptionContext` harvests: any export with a + * `getEncryptConfig()` method is the client, and any with `tableName` + + * `build()` is a table. Hand-rolled rather than imported from + * `@cipherstash/stack` so the test needs no native module. + */ + const table = (name: string) => + `{ tableName: '${name}', build: () => ({ tableName: '${name}', columns: {} }) }` + + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-encrypt-ctx-')) + originalCwd = process.cwd + }) + + afterEach(() => { + process.cwd = originalCwd + vi.restoreAllMocks() + if (tmpDir && fs.existsSync(tmpDir)) { + fs.rmSync(tmpDir, { recursive: true, force: true }) + } + }) + + it('exits 1 naming the sentinel when it is the only table declared', async () => { + writeProject( + `export const encryptionClient = { getEncryptConfig: () => ({}) } + export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)}`, + ) + const error = vi.spyOn(console, 'error').mockImplementation(() => {}) + vi.spyOn(process, 'exit').mockImplementation(() => { + throw new Error('process.exit') + }) + + const { loadEncryptionContext } = await import('../context.js') + await expect(loadEncryptionContext()).rejects.toThrow('process.exit') + + const message = error.mock.calls.flat().join('\n') + expect(message).toContain(PLACEHOLDER_TABLE_NAME) + // Names the cause, not `requireTable`'s "table not found" symptom. + expect(message).toContain('still contains the placeholder table') + expect(message).not.toContain('was not found in the encryption client') + }) + + it('allows the sentinel through once a real table sits alongside it', async () => { + // Only the SOLE-placeholder case is the un-replaced scaffold. A user who + // has added real tables must not be blocked by a leftover sentinel export. + writeProject( + `export const encryptionClient = { getEncryptConfig: () => ({}) } + export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)} + export const users = ${table('users')}`, + ) + + const { loadEncryptionContext } = await import('../context.js') + const ctx = await loadEncryptionContext() + + expect(ctx.tables.has('users')).toBe(true) + }) +}) diff --git a/packages/cli/src/commands/encrypt/context.ts b/packages/cli/src/commands/encrypt/context.ts index 1975e2d96..e37dd8a31 100644 --- a/packages/cli/src/commands/encrypt/context.ts +++ b/packages/cli/src/commands/encrypt/context.ts @@ -1,7 +1,11 @@ import fs from 'node:fs' import path from 'node:path' import type { EncryptionClient } from '@cipherstash/stack/encryption' -import { loadStashConfig, type ResolvedStashConfig } from '@/config/index.js' +import { + loadStashConfig, + PLACEHOLDER_TABLE_NAME, + type ResolvedStashConfig, +} from '@/config/index.js' /** * Structural shape of `@cipherstash/stack`'s `EncryptedTable` class. @@ -138,6 +142,20 @@ export async function loadEncryptionContext(): Promise { process.exit(1) } + // Same guard `loadEncryptConfig` applies for `stash db push` / `db validate`, + // repeated here because `stash encrypt` does not go through that loader. The + // scaffold `stash init` writes declares one sentinel table so the file + // compiles; reaching here with only that table means it was never replaced. + // Without this, `requireTable` reported `Table "users" was not found … + // Available: __stash_placeholder__`, which names the symptom and not the + // cause (#787 review). + if (tables.size === 1 && tables.has(PLACEHOLDER_TABLE_NAME)) { + console.error( + `Error: ${stashConfig.client} still contains the placeholder table \`${PLACEHOLDER_TABLE_NAME}\` that \`stash init\` wrote.\n\nDeclare your encrypted columns and pass those tables to Encryption({ schemas: [...] }) in that file, then re-run this command.`, + ) + process.exit(1) + } + return { stashConfig, client, tables } } diff --git a/packages/cli/src/commands/encrypt/cutover.ts b/packages/cli/src/commands/encrypt/cutover.ts index 5b9906134..7a5cd7f1c 100644 --- a/packages/cli/src/commands/encrypt/cutover.ts +++ b/packages/cli/src/commands/encrypt/cutover.ts @@ -107,7 +107,7 @@ export async function cutoverCommand(options: CutoverCommandOptions) { // the same reason (#772 review, finding 7). if (info.via === 'sole') { p.log.error( - `${options.table}.${encryptedColumn} (${info.domain}) is the table's only EQL v3 column, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, this release no longer manages that lifecycle. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \`.`, + `${options.table}.${encryptedColumn} (${info.domain}) is the table's only EQL v3 column, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only) — drive that column's v2 lifecycle against its own encrypted column directly. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \`.`, ) exitCode = 1 return diff --git a/packages/cli/src/commands/encrypt/drop.ts b/packages/cli/src/commands/encrypt/drop.ts index ae5181e4f..49b0bff8c 100644 --- a/packages/cli/src/commands/encrypt/drop.ts +++ b/packages/cli/src/commands/encrypt/drop.ts @@ -113,7 +113,7 @@ export async function dropCommand(options: DropCommandOptions) { // live `DROP COLUMN` on the plaintext at exit 0 (#772 review, finding 7). if (info?.via === 'sole') { p.log.error( - `${options.table}.${info.column} (${info.domain}) is the table's only encrypted column, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, this release no longer manages that lifecycle — do not record ${info.column}.`, + `${options.table}.${info.column} (${info.domain}) is the table's only encrypted column, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only) — drive that column's v2 lifecycle against its own encrypted column directly, and do not record ${info.column}.`, ) exitCode = 1 return diff --git a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts index 0310891c5..9c45d78b7 100644 --- a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts +++ b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts @@ -4,12 +4,17 @@ * * `listEncryptedColumns` can no longer emit `version: 2` — a legacy * `eql_v2_encrypted` column is not classified as an EQL column at all, so it - * never reaches this function as a candidate. The post-cutover v2 state (the - * ciphertext renamed onto the plaintext column's own name) therefore arrives - * here as an EMPTY candidate list, which the first guard already falls through - * on. These tests exist so removing the now-unreachable `version === 2` branch - * is provably behaviour-preserving, and so a future v2 sweep cannot delete the - * empty-list guard the v2 lifecycle actually depends on. + * never reaches this function as a candidate. Every pure-v2 table therefore + * arrives here as an EMPTY candidate list, both pre-cutover (`` / + * `_encrypted`) and post-cutover (the ciphertext renamed onto the + * plaintext column's own name), and the first guard falls through on it. + * + * These tests exist so removing the now-unreachable `version === 2` branch is + * provably behaviour-preserving, and so a future v2 sweep cannot delete the + * empty-list guard the v2 lifecycle actually depends on. That guard has to hold + * even when the manifest recorded an `encryptedColumn` — `backfill` records one + * for v2 columns too — which is the `candidates.length > 0` gate on the + * fail-closed path (#787 review). */ import type { EncryptedColumnInfo } from '@cipherstash/migrate' @@ -64,6 +69,15 @@ describe('explainUnresolved', () => { expect(explainUnresolved('users', 'email', [])).toBeNull() }) + it('still falls through when no EQL v3 columns exist BUT a hint was recorded', () => { + // The pure-v2 table. `encrypt backfill` records `encryptedColumn` for v2 + // columns too, so a hint is present on every table backfilled with this + // release — it must not flip the empty-candidate fall-through into a + // refusal, because `cutover` / `drop` in this same build still implement + // the v2 ladder this falls through to (#787 review). + expect(explainUnresolved('users', 'ssn', [], 'ssn_encrypted')).toBeNull() + }) + it('fails closed, naming every candidate, when none is identifiable', () => { const message = explainUnresolved('users', 'email', [ v3('a_enc'), @@ -133,6 +147,36 @@ describe('resolveColumnLifecycle — a recorded hint that is not a v3 candidate' expect(unresolvedHint).toBe('ssn_encrypted') }) + // The pure-v2 shape, and the regression the `candidates.length > 0` gate + // exists to prevent (#787 review). `backfill` records `encryptedColumn` + // unconditionally — v2 included — so EVERY pure-v2 table backfilled with this + // release carries a hint naming a real, existing, non-v3 column. Without the + // gate, `columnExists` returned true, `unresolvedHint` was set, and + // `cutover` / `drop` refused a lifecycle this same build still performs. + it('does not fail closed on a pure-v2 table, where no v3 column can be mis-claimed', async () => { + // No v3 columns at all: just the `ssn` / `ssn_encrypted` v2 pair, which the + // classifier does not see. + listEncryptedColumns.mockResolvedValue([]) + readManifest.mockResolvedValue({ + tables: { users: [{ column: 'ssn', encryptedColumn: 'ssn_encrypted' }] }, + }) + + const { info, candidates, unresolvedHint } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'ssn_encrypted'), + 'users', + 'ssn', + ) + + expect(info).toBeNull() + expect(candidates).toEqual([]) + // The fall-through signal: no hint reported, so `explainUnresolved` returns + // null and the caller reaches its own v2 preconditions. + expect(unresolvedHint).toBeUndefined() + expect( + explainUnresolved('users', 'ssn', candidates, unresolvedHint), + ).toBeNull() + }) + it('explains the recorded counterpart by name rather than listing candidates', async () => { const message = explainUnresolved( 'users', diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index df8f765b2..6c4834d0b 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -24,14 +24,23 @@ export interface ResolvedLifecycle { */ candidates: EncryptedColumnInfo[] /** - * The manifest's recorded `encryptedColumn` when it named a column that is - * NOT among `candidates` — i.e. the pairing is on record but the column it - * names is not an EQL v3 column (typically legacy `eql_v2_encrypted`, which - * `classifyEqlDomain` no longer recognises). + * The manifest's recorded `encryptedColumn` when it named a real column that + * is NOT among `candidates`, AND `candidates` is non-empty — i.e. the pairing + * is on record, the column it names is not an EQL v3 column (typically legacy + * `eql_v2_encrypted`, which `classifyEqlDomain` no longer recognises), and + * there are v3 columns here that a guess could wrongly claim. * * Distinct from a stale hint. It means the answer IS known and disagrees * with anything resolution could otherwise guess, so callers must fail closed * naming it rather than fall through to `via: 'sole'` (#772 review, finding 7). + * + * Deliberately NOT set when `candidates` is empty. That is the pure-v2 table + * — a `` / `_encrypted` pair and nothing else — where there is no + * v3 column to mis-claim and so nothing to protect against. The v2 lifecycle + * is still implemented in `cutover.ts` / `drop.ts`, and those commands must + * keep reaching it; failing closed here would refuse a lifecycle this same + * build still performs, and tell the user to downgrade to reach it (#787 + * review). */ unresolvedHint?: string } @@ -73,19 +82,29 @@ export async function resolveColumnLifecycle( const hinted = hint ? pickEncryptedColumn(candidates, column, hint) : null if (hinted) return { info: hinted, candidates } - // The hint named a column that is not a candidate. Two very different + // The hint named a column that is not a candidate. Three very different // reasons, and they must not share an outcome: // // - the column no longer exists (renamed, dropped) — a genuinely STALE hint, // which must not mask a resolvable counterpart, so fall through; - // - the column exists but is not an EQL v3 column — the usual shape being a - // legacy `eql_v2_encrypted` counterpart, which `classifyEqlDomain` stopped - // recognising. Here the pairing IS known and falling through would discard - // it in favour of a guess: on a mixed table the sole-EQL-column rule then - // claims an unrelated v3 column, `cutover` reports success for a rename it - // never performed, and `drop`'s remedy tells the user to record the guess - // (#772 review, finding 7). - if (hint && (await columnExists(client, table, hint))) { + // - the column exists, is not an EQL v3 column, and the table HAS v3 columns + // — the mixed table. The usual shape is a legacy `eql_v2_encrypted` + // counterpart, which `classifyEqlDomain` stopped recognising. Here the + // pairing IS known and falling through would discard it in favour of a + // guess: the sole-EQL-column rule claims an unrelated v3 column, `cutover` + // reports success for a rename it never performed, and `drop`'s remedy + // tells the user to record the guess (#772 review, finding 7). Fail closed; + // - the column exists, is not an EQL v3 column, and there are NO v3 columns + // on the table — the pure-v2 table. Nothing here can be mis-claimed, and + // `cutover` / `drop` still implement the v2 ladder, so fall through to it + // exactly as before. Gating on `candidates.length` is what keeps the + // protection above scoped to the mixed table it was written for (#787 + // review). + if ( + hint && + candidates.length > 0 && + (await columnExists(client, table, hint)) + ) { return { info: null, candidates, unresolvedHint: hint } } @@ -115,12 +134,16 @@ async function columnExists( * Explain a failed resolution (`info === null`) to the user, or return * `null` when the failure is fine to fall through to the v2 lifecycle. * - * The one fall-through case is "no EQL columns at all", which the v2 + * The one fall-through case is "no EQL v3 columns at all", which the v2 * phase/config preconditions turn into an accurate error ("not backfilled", * "no pending config", …). Since `classifyEqlDomain` recognises `eql_v3_*` - * only, that case now also covers the post-cutover v2 state — `` was - * renamed onto the ciphertext, and its `eql_v2_encrypted` domain is no longer - * classified, so the column never appears as a candidate. + * only, that case covers every pure-v2 table — both the pre-cutover pair + * (`` / `_encrypted`) and the post-cutover state where `` was + * renamed onto the ciphertext. Neither column is ever a candidate, so a pure-v2 + * table reaches the v2 ladder here regardless of what the manifest recorded; + * a recorded `encryptedColumn` must NOT turn that into a refusal, because + * `cutover.ts` / `drop.ts` in this same build still implement that ladder + * (#787 review). * * A non-empty candidate list therefore means EQL v3 columns exist but none is * identifiable — the caller must fail closed with this message rather than @@ -134,15 +157,23 @@ export function explainUnresolved( candidates: readonly EncryptedColumnInfo[], unresolvedHint?: string, ): string | null { - // The recorded pairing points at a real column that is not an EQL v3 column - // — almost always a legacy `eql_v2_encrypted` counterpart. Say exactly that: - // listing the v3 candidates here would invite the user to record one of them, - // which is how the guess used to get laundered into a `via: 'hint'` match. + // "No EQL v3 columns at all" always falls through, even with a recorded hint. + // That is the pure-v2 table, and the v2 ladder in `cutover.ts` / `drop.ts` + // still handles it — the caller's own preconditions produce the accurate + // error. Ordered ahead of the hint branch deliberately: `resolveColumnLifecycle` + // already declines to set `unresolvedHint` on an empty candidate list, and this + // keeps the two agreeing for direct callers of this function (#787 review). + if (candidates.length === 0) return null + + // The recorded pairing points at a real column that is not an EQL v3 column, + // on a table that does have v3 columns — almost always a legacy + // `eql_v2_encrypted` counterpart alongside an unrelated v3 one. Say exactly + // that: listing the v3 candidates here would invite the user to record one of + // them, which is how the guess used to get laundered into a `via: 'hint'` match. if (unresolvedHint !== undefined) { - return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column, which this command no longer manages. Finish the EQL v2 lifecycle for this column with a stash release that still supports it, or drop the recorded pairing from .cipherstash/migrations.json if it is wrong.` + return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column. ${table} also holds EQL v3 columns, and none of them is a confirmed counterpart for ${column}, so this command cannot tell which lifecycle applies and will not guess.\n\nIf that pairing is wrong, correct or remove "encryptedColumn" for ${column} in .cipherstash/migrations.json and re-run. If it is right, ${column} is on the EQL v2 lifecycle: drive it against ${unresolvedHint} directly rather than through this command, which resolves EQL v3 counterparts only.` } - if (candidates.length === 0) return null const listed = candidates .map((c) => ` - ${c.column} (${c.domain})`) .join('\n') diff --git a/packages/cli/vitest.config.ts b/packages/cli/vitest.config.ts index 53df05899..195a6a156 100644 --- a/packages/cli/vitest.config.ts +++ b/packages/cli/vitest.config.ts @@ -9,6 +9,13 @@ export default defineConfig({ resolve: { alias: { '@/': `${resolve(__dirname, './src')}/`, + // `@cipherstash/migrate` publishes `./dist` only, so importing it — or + // spreading `importOriginal()` inside a partial `vi.mock` — makes the + // UNIT suite require a prior workspace build. CI only hides that because + // turbo runs `^build` first; on a clean checkout `pnpm --filter stash + // test` would fail to resolve it. Resolving to source keeps the unit + // config self-contained, as `AGENTS.md` says it is (#787 review). + '@cipherstash/migrate': resolve(__dirname, '../migrate/src/index.ts'), }, }, }) diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 49873523e..e2541e40f 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -221,7 +221,7 @@ Seven mechanical steps, no agent handoff. It prompts only when it can't pick a s 1. **Authenticate** — silent when a valid token exists. 2. **Resolve database** — per the resolution order above; verifies the connection. 3. **Resolve proxy choice** — CipherStash Proxy or direct SDK access (the default). Stored as `usesProxy` in `context.json`. Set by `--proxy` / `--no-proxy`; non-TTY without a flag defaults to SDK. -4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. The placeholder declares one sentinel table, `__stash_placeholder__`, because `Encryption` requires at least one — replace it with the tables you actually encrypt. Until you do, `stash encrypt` commands exit 1 and point back at that file. +4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. The placeholder declares one sentinel table, `__stash_placeholder__`, because `Encryption` requires at least one — replace it with the tables you actually encrypt. Until you do, the commands that read the client file — `db push`, `db validate`, and `encrypt backfill` — exit 1 and point back at it. 5. **Install dependencies** — one combined prompt for `@cipherstash/stack` and `stash`. Skipped when both are present. 6. **Install EQL** — always EQL v3. **Drizzle** projects generate a v3 install migration (the same output as `eql migration --drizzle`, including the `cs_migrations` tracking schema) so the install lands in your migration history — apply it with `drizzle-kit migrate`; requires `drizzle-kit` to be installed and configured. **Prisma Next** is skipped (it installs EQL via `prisma-next migrate`). Everything else runs `eql install` directly against the resolved database, and is skipped when EQL is already installed. 7. **Gather context** — detects available coding agents and writes `.cipherstash/context.json`. From 1b8cac2316ec6ec9163435990864db8dcccaae7b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 10:52:38 +1000 Subject: [PATCH 083/123] fix(cli,migrate): close gaps found verifying the #787 review fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to 1d144128, from an adversarial cross-check of that commit. resolve-eql's private `columnExists` used a bare `to_regclass($1)`. That form parses and case-folds unquoted identifiers, so on a Prisma-style "User" table the probe reported the column missing, the recorded pairing was treated as stale, and the #772 fail-closed silently did not fire — falling through to the sole/convention rules and resolving the guess it exists to prevent. migrate already documents this exact anti-pattern (REGCLASS_SQL) and its test asserts `not.toMatch(/to_regclass\(\$1\)/)`. Moved to a shared, case-exact `columnExists` export and deleted the CLI copy; the test double is now case-exact too, so it cannot hide a regression. The placeholder guard read the harvested export map while the `db push` / `db validate` guard it mirrors reads `getEncryptConfig().tables`. Those disagree in both directions on one file: `schemas: [placeholderTable]` minus the `export` keyword fell through to "table not found … Available: (none)" — the error the guard replaces — and a stale placeholder export beside real tables wrongly fired it. Now reads the same source. cutover's `via:'sole'` refusal was nested inside `version === 3`; drop's is top-level. Equivalent today, but cutover's v2 ladder does an irreversible rename plus config promotion, so a restored v2 classification or a v4 family would let cutover rename on a guess drop refuses. Hoisted to match. Text that was false: - the scaffold `stash init` writes into every customer project (and both fixtures) claimed `stash encrypt` commands refuse to run; only backfill does - skills/stash-cli listed the client as loaded by `schema build` and `encrypt *`, and omitted `db push` - skills/stash-cli and skills/stash-encryption still said cutover on a backfilled v3 column exits 0. Since `.cipherstash/` is gitignored, a clone or CI runner hits the new `sole` refusal on a pure-v3 table with an unconventional column name — the changeset's "pure-v3 unaffected" was wrong too - both `sole` messages said "the table's only EQL column"; pickEncryptedColumn excludes the plaintext column first, so it fires with two - the remedy said to drive the v2 lifecycle "directly"; there is no CLI route, so it now says to run the eql_v2 SQL yourself - vitest.config.ts and packages/cli/AGENTS.md claimed the unit suite is self-contained. It is not: @cipherstash/stack is still reached via migrate/src/backfill.ts. The alias removed one of two couplings typecheck:scaffold ran `tsc` directly, bypassing turbo, so it needed a prior build and passed only by accident of step ordering behind steps that read as independently droppable. Now a turbo task with dependsOn ^build. Added scripts/__tests__/cli-vitest-alias.test.mjs — the CLI's alias map sits outside the vitest.shared.ts guard and must, since stackSourceAlias's '@/' would clobber the CLI's own. Runs in test:scripts, ahead of the build-dependent suite. Verified: frozen-lockfile install clean; 891 CLI unit, 41 migrate, 93 scripts, 76 pty e2e; code:check 0 errors; scaffold gate green through turbo with packages/stack/dist absent. --- .changeset/encrypt-lifecycle-mixed-table.md | 29 +++++++--- .changeset/migrate-column-exists.md | 21 ++++++++ .github/workflows/tests.yml | 8 ++- packages/cli/AGENTS.md | 15 +++++- .../scaffold/drizzle.generated.ts | 10 ++-- .../scaffold/generic.generated.ts | 10 ++-- .../__tests__/context-placeholder.test.ts | 54 ++++++++++++++++++- packages/cli/src/commands/encrypt/context.ts | 15 +++++- packages/cli/src/commands/encrypt/cutover.ts | 37 ++++++++----- packages/cli/src/commands/encrypt/drop.ts | 2 +- .../encrypt/lib/__tests__/resolve-eql.test.ts | 30 ++++++++++- .../src/commands/encrypt/lib/resolve-eql.ts | 22 +------- packages/cli/src/commands/init/utils.ts | 20 ++++--- packages/cli/vitest.config.ts | 18 +++++-- .../migrate/src/__tests__/version.test.ts | 34 ++++++++++++ packages/migrate/src/index.ts | 1 + packages/migrate/src/version.ts | 32 +++++++++++ scripts/__tests__/cli-vitest-alias.test.mjs | 45 ++++++++++++++++ skills/stash-cli/SKILL.md | 4 +- skills/stash-encryption/SKILL.md | 4 +- turbo.json | 4 ++ 21 files changed, 343 insertions(+), 72 deletions(-) create mode 100644 .changeset/migrate-column-exists.md create mode 100644 scripts/__tests__/cli-vitest-alias.test.mjs diff --git a/.changeset/encrypt-lifecycle-mixed-table.md b/.changeset/encrypt-lifecycle-mixed-table.md index 95bebb711..3b3dab41a 100644 --- a/.changeset/encrypt-lifecycle-mixed-table.md +++ b/.changeset/encrypt-lifecycle-mixed-table.md @@ -35,10 +35,25 @@ all now fixed: column that actually encrypts the named one, and says explicitly not to record the guess. -The new refusal is scoped to the mixed table it was written for: it applies only -when the table actually holds EQL v3 columns that a guess could wrongly claim. A -pure-v2 table has none, so it still falls through to the EQL v2 lifecycle exactly -as before — including when `encrypt backfill` recorded an `encryptedColumn` for -it, which it does for v2 columns too. Pure-v2 and pure-v3 tables are therefore -unaffected, as are tables with two or more EQL v3 columns (resolution already -failed closed there). +The `unresolvedHint` refusal is scoped to tables that actually hold EQL v3 +columns a guess could wrongly claim. A **pure-v2** table has none, so it still +falls through to the EQL v2 lifecycle exactly as before — including when +`encrypt backfill` recorded an `encryptedColumn` for it, which it does for v2 +columns too. + +Two cases DO newly exit 1, both deliberately: + +- Any table with at least one EQL v3 column where the manifest records an + `encryptedColumn` that exists but is not one of them — not only the + v2-pair-plus-one-v3 shape. The recorded pairing is authoritative and + disagrees with every candidate, so guessing past it is the bug. + +- A column whose encrypted counterpart could only be identified **by + elimination** — no recorded `encryptedColumn`, no `_encrypted` name + match, one EQL column left once the plaintext column itself is excluded. + `cutover` now refuses this as `drop` already did. Note `.cipherstash/` is + gitignored, so `migrations.json` is machine-local: a fresh clone or CI runner + can hit this on a **pure-v3** table whose encrypted column is named + unconventionally, where `cutover` previously exited 0 with "not applicable". + Re-run `stash encrypt backfill --table T --column C --encrypted-column ` + to record the pairing. diff --git a/.changeset/migrate-column-exists.md b/.changeset/migrate-column-exists.md new file mode 100644 index 000000000..d88867430 --- /dev/null +++ b/.changeset/migrate-column-exists.md @@ -0,0 +1,21 @@ +--- +'@cipherstash/migrate': minor +'stash': patch +--- + +Add `columnExists(client, tableName, columnName)` — a case-exact "does this +column exist at all?" catalog probe, distinct from `detectColumnEqlVersion`'s +"and is it an EQL column?". + +Callers need that difference to tell a STALE column reference (it is gone) from +a live one the domain classifier simply does not recognise — most often a legacy +`eql_v2_encrypted` counterpart. + +`stash encrypt cutover` / `drop` had a private copy of this probe built on a bare +`to_regclass($1)`. That form *parses* its argument and case-folds unquoted +identifiers, so on a Prisma-style `"User"` table it resolved `user`, reported the +column missing, and treated a valid recorded pairing as stale — silently skipping +the fail-closed that stops those commands acting on a guessed encrypted column. +The shared implementation quotes with `format('%I')` first, like every other +catalog probe in this package, so the lookup is case-exact while still honouring +`search_path` for unqualified names. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 644d3cadb..4ece7ea51 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -199,8 +199,14 @@ jobs: # emitting a project that fails its first `tsc` — with CI green (#772 # review). The fixtures are pinned byte-for-byte to the generator by # `placeholder-client-fixture.test.ts`. + # Through turbo, not `pnpm run` — the fixtures import `@cipherstash/stack/v3`, + # so this needs that package BUILT. Invoked directly it passed only because + # earlier steps in this job happen to build it via their own `^build`; drop + # or reorder those (they read as guards for other packages, so they look + # independently removable) and this fails `TS2307`, which reads as "the + # scaffold is broken" rather than "you forgot to build" (#787 review). - name: Typecheck (stash init's scaffolded client) - run: pnpm --filter stash run typecheck:scaffold + run: pnpm exec turbo run typecheck:scaffold --filter stash - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index 73795c3bc..ced2efbff 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -6,11 +6,22 @@ This package has **two** Vitest configs. Run the right one for the change. | Command | Config | Scope | Needs build? | | --- | --- | --- | --- | -| `pnpm --filter stash test` | `vitest.config.ts` | Unit tests under `src/__tests__/**` and `src/**/__tests__/**` | No | +| `pnpm --filter stash test` | `vitest.config.ts` | Unit tests under `src/__tests__/**` and `src/**/__tests__/**` | **Partly** — needs `@cipherstash/stack` built (see below). Turbo's `^build` supplies it in CI. | | `pnpm --filter stash test:e2e` | `vitest.integration.config.ts` | E2E tests under `tests/e2e/**.e2e.test.ts` driving the built `dist/bin/stash.js` through a real pty (`node-pty`) | **Yes** — run `pnpm --filter stash build` first, or use the turbo `test:e2e` task which depends on `build`. | The unit config explicitly excludes `tests/e2e/**` so the default `pnpm test` -stays fast and self-contained. +stays fast. + +It is **not** fully self-contained, despite running standalone in CI. Some `src` +modules import workspace packages that publish `./dist` only, so an unbuilt +workspace fails at collection with `Failed to resolve entry for package …` +rather than at an assertion. `vitest.config.ts` aliases `@cipherstash/migrate` +to its source to remove one such coupling; `@cipherstash/stack` remains, reached +via `packages/migrate/src/backfill.ts` and a direct import in +`init/lib/introspect.test.ts`. Deleting `packages/stack/dist` fails 10 files. +Closing it needs `vitest.shared.ts`'s `stackSourceAlias`, which cannot be spread +into this config — its `'@/'` entry points at `packages/stack/src` and would +clobber this package's own `'@/'` (#787 review). ## When to add or update an E2E test diff --git a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts index a0f9b77f0..755815874 100644 --- a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts +++ b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts @@ -8,8 +8,10 @@ * to reference the encrypted tables you declared there. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and `stash encrypt` - * commands refuse to run and point back here. + * placeholder table so that this file compiles, and `stash db push`, + * `stash db validate` and `stash encrypt backfill` refuse to run and point + * back here. (`stash encrypt cutover` / `drop` resolve against the database + * and never read this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the `types.*` factories from `@cipherstash/stack-drizzle`. @@ -56,8 +58,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — `Encryption` requires at least one. Swap it for your -// real tables (see the patterns above); `stash encrypt` refuses to run while -// the placeholder is still here. +// real tables (see the patterns above); `stash db push`, `stash db validate` +// and `stash encrypt backfill` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/__fixtures__/scaffold/generic.generated.ts b/packages/cli/__fixtures__/scaffold/generic.generated.ts index 4f5831847..9f0187adc 100644 --- a/packages/cli/__fixtures__/scaffold/generic.generated.ts +++ b/packages/cli/__fixtures__/scaffold/generic.generated.ts @@ -7,8 +7,10 @@ * `Encryption({ schemas: [...] })` call below to reference them. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and `stash encrypt` - * commands refuse to run and point back here. + * placeholder table so that this file compiles, and `stash db push`, + * `stash db validate` and `stash encrypt backfill` refuse to run and point + * back here. (`stash encrypt cutover` / `drop` resolve against the database + * and never read this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the `types.*` factories from `@cipherstash/stack/eql/v3` @@ -51,8 +53,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — `Encryption` requires at least one. Swap it for your -// real tables (see the patterns above); `stash encrypt` refuses to run while -// the placeholder is still here. +// real tables (see the patterns above); `stash db push`, `stash db validate` +// and `stash encrypt backfill` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts index ad32cded8..f1dacb93a 100644 --- a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts @@ -53,8 +53,12 @@ describe('loadEncryptionContext — the un-replaced init scaffold', () => { }) it('exits 1 naming the sentinel when it is the only table declared', async () => { + // The scaffold's real shape: the sentinel is both exported AND the sole + // entry in the built encrypt config, because it was passed to `Encryption`. writeProject( - `export const encryptionClient = { getEncryptConfig: () => ({}) } + `export const encryptionClient = { + getEncryptConfig: () => ({ tables: { '${PLACEHOLDER_TABLE_NAME}': {} } }), + } export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)}`, ) const error = vi.spyOn(console, 'error').mockImplementation(() => {}) @@ -72,6 +76,54 @@ describe('loadEncryptionContext — the un-replaced init scaffold', () => { expect(message).not.toContain('was not found in the encryption client') }) + /** + * #787 review. The guard originally read the harvested EXPORT map, while the + * `db push` / `db validate` guard it mirrors reads `getEncryptConfig().tables`. + * Those disagree in both directions on the same client file, so the two + * commands gave different answers for identical input. + */ + it('fires when the placeholder is passed to Encryption but never exported', async () => { + // The false NEGATIVE. `schemas: [placeholderTable]` with a bare `const` — + // the scaffold minus one `export` keyword. Reading exports, the guard saw + // no tables and fell through to `requireTable`'s "table not found … + // Available: (none)" — the very error this guard exists to replace. + writeProject( + `export const encryptionClient = { + getEncryptConfig: () => ({ tables: { '${PLACEHOLDER_TABLE_NAME}': {} } }), + }`, + ) + const error = vi.spyOn(console, 'error').mockImplementation(() => {}) + vi.spyOn(process, 'exit').mockImplementation(() => { + throw new Error('process.exit') + }) + + const { loadEncryptionContext } = await import('../context.js') + await expect(loadEncryptionContext()).rejects.toThrow('process.exit') + + const message = error.mock.calls.flat().join('\n') + expect(message).toContain('still contains the placeholder table') + expect(message).not.toContain('was not found in the encryption client') + }) + + it('stays silent when a stale placeholder export sits beside real configured tables', async () => { + // The false POSITIVE, and the mirror of the case above: the user replaced + // the schema set but left the sentinel `export` behind (or imports their + // real tables without re-exporting them). Reading exports, the guard fired + // and told them to declare columns they had already declared. `db push` + // passes on this same file. + writeProject( + `export const encryptionClient = { + getEncryptConfig: () => ({ tables: { users: {} } }), + } + export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)}`, + ) + + const { loadEncryptionContext } = await import('../context.js') + const ctx = await loadEncryptionContext() + + expect(ctx.tables.has(PLACEHOLDER_TABLE_NAME)).toBe(true) + }) + it('allows the sentinel through once a real table sits alongside it', async () => { // Only the SOLE-placeholder case is the un-replaced scaffold. A user who // has added real tables must not be blocked by a leftover sentinel export. diff --git a/packages/cli/src/commands/encrypt/context.ts b/packages/cli/src/commands/encrypt/context.ts index e37dd8a31..0eab28db1 100644 --- a/packages/cli/src/commands/encrypt/context.ts +++ b/packages/cli/src/commands/encrypt/context.ts @@ -142,14 +142,25 @@ export async function loadEncryptionContext(): Promise { process.exit(1) } - // Same guard `loadEncryptConfig` applies for `stash db push` / `db validate`, + // The guard `loadEncryptConfig` applies for `stash db push` / `db validate`, // repeated here because `stash encrypt` does not go through that loader. The // scaffold `stash init` writes declares one sentinel table so the file // compiles; reaching here with only that table means it was never replaced. // Without this, `requireTable` reported `Table "users" was not found … // Available: __stash_placeholder__`, which names the symptom and not the // cause (#787 review). - if (tables.size === 1 && tables.has(PLACEHOLDER_TABLE_NAME)) { + // + // Read from `getEncryptConfig()` — the SAME source `loadEncryptConfig` uses — + // not from the harvested export map, or the two commands disagree on one + // file. `schemas: [placeholderTable]` with the `export` keyword dropped is + // still the un-replaced scaffold but exports nothing; conversely a stale + // `export const placeholderTable` beside real tables that are imported + // rather than re-exported is NOT (#787 review). + const configuredTables = Object.keys(client.getEncryptConfig()?.tables ?? {}) + if ( + configuredTables.length === 1 && + configuredTables[0] === PLACEHOLDER_TABLE_NAME + ) { console.error( `Error: ${stashConfig.client} still contains the placeholder table \`${PLACEHOLDER_TABLE_NAME}\` that \`stash init\` wrote.\n\nDeclare your encrypted columns and pass those tables to Encryption({ schemas: [...] }) in that file, then re-run this command.`, ) diff --git a/packages/cli/src/commands/encrypt/cutover.ts b/packages/cli/src/commands/encrypt/cutover.ts index 7a5cd7f1c..1080e9524 100644 --- a/packages/cli/src/commands/encrypt/cutover.ts +++ b/packages/cli/src/commands/encrypt/cutover.ts @@ -95,23 +95,32 @@ export async function cutoverCommand(options: CutoverCommandOptions) { } const state = await progress(client, options.table, options.column) + // `via: 'sole'` means only that this is the table's one remaining EQL + // candidate once the plaintext column itself is excluded — nothing ties it + // to the plaintext column the user named. On a mixed table (a v2 pair the + // classifier no longer sees, plus one unrelated v3 column) that guess is + // simply wrong, and reporting "nothing to do for EQL v3" for it told a + // scripted rollout the cut-over had succeeded when the v2 rename never ran + // (#772 review, finding 7). + // + // Deliberately at TOP LEVEL, not inside the `version === 3` branch, so it + // mirrors `drop.ts` exactly. Equivalent today — `classifyEqlDomain` + // recognises `eql_v3_*` only, so a non-null `info` is always version 3 — + // but the v2 ladder below performs an irreversible rename plus config + // promotion. Were v2 classification restored, or a v4 family added, a + // nested guard would let `cutover` rename on a guess that `drop` refuses + // (#787 review). + if (info?.via === 'sole') { + p.log.error( + `${options.table}.${info.column} (${info.domain}) is the only EQL column left on ${options.table} once "${options.column}" itself is excluded, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only): complete that column's v2 lifecycle yourself with the eql_v2 SQL — \`SELECT eql_v2.rename_encrypted_columns();\` plus the config promotion — since no stash command can drive it here. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \`.`, + ) + exitCode = 1 + return + } + if (info?.version === 3) { const encryptedColumn = info.column - // `via: 'sole'` means only that this is the table's ONE EQL v3 column — - // nothing ties it to the plaintext column the user named. On a mixed - // table (a v2 pair the classifier no longer sees, plus one unrelated v3 - // column) that guess is simply wrong, and reporting "nothing to do for - // EQL v3" for it told a scripted rollout the cut-over had succeeded when - // the v2 rename never ran. `drop.ts` already refuses a `'sole'` match for - // the same reason (#772 review, finding 7). - if (info.via === 'sole') { - p.log.error( - `${options.table}.${encryptedColumn} (${info.domain}) is the table's only EQL v3 column, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only) — drive that column's v2 lifecycle against its own encrypted column directly. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \`.`, - ) - exitCode = 1 - return - } if (state?.phase === 'dropped') { // Terminal phase — the lifecycle already finished. Not an error and // not "finish the backfill": there is nothing left to backfill. diff --git a/packages/cli/src/commands/encrypt/drop.ts b/packages/cli/src/commands/encrypt/drop.ts index 49b0bff8c..d016a0c63 100644 --- a/packages/cli/src/commands/encrypt/drop.ts +++ b/packages/cli/src/commands/encrypt/drop.ts @@ -113,7 +113,7 @@ export async function dropCommand(options: DropCommandOptions) { // live `DROP COLUMN` on the plaintext at exit 0 (#772 review, finding 7). if (info?.via === 'sole') { p.log.error( - `${options.table}.${info.column} (${info.domain}) is the table's only encrypted column, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only) — drive that column's v2 lifecycle against its own encrypted column directly, and do not record ${info.column}.`, + `${options.table}.${info.column} (${info.domain}) is the only EQL column left on ${options.table} once "${options.column}" itself is excluded, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column \` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only): complete that column's v2 lifecycle yourself with the eql_v2 SQL, since no stash command can drive it here — and do not record ${info.column}.`, ) exitCode = 1 return diff --git a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts index 9c45d78b7..154c5e563 100644 --- a/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts +++ b/packages/cli/src/commands/encrypt/lib/__tests__/resolve-eql.test.ts @@ -48,8 +48,13 @@ const { explainUnresolved, resolveColumnLifecycle } = await import( */ const clientWithColumns = (...columns: string[]): pg.ClientBase => ({ + // `columnExists` binds [table, schema, column] — it splits schema-qualified + // names and quotes with `format('%I')` so the lookup is case-EXACT. This + // double matches case-exactly for the same reason: a case-insensitive + // fixture would keep passing if the probe regressed to a bare + // `to_regclass($1)`, which case-folds (#787 review). query: async (_sql: string, params: unknown[]) => ({ - rows: [{ exists: columns.includes(String(params[1])) }], + rows: [{ exists: columns.includes(String(params[2])) }], }), }) as unknown as pg.ClientBase @@ -177,6 +182,29 @@ describe('resolveColumnLifecycle — a recorded hint that is not a v3 candidate' ).toBeNull() }) + // #787 review. `columnExists` used to be a local copy using a bare + // `to_regclass($1)`, which PARSES and case-folds its argument — so on a + // Prisma-style `"User"` table the probe reported "missing", the hint was + // treated as stale, and the fail-closed above silently did not fire. It fell + // through to the sole/convention rules and resolved a guess, which is exactly + // what #772 finding 7 exists to prevent. Now delegated to + // `@cipherstash/migrate`'s shared, case-exact probe. + it('fires the fail-closed on a mixed-case table name too', async () => { + listEncryptedColumns.mockResolvedValue([V3_OTHER]) + readManifest.mockResolvedValue({ + tables: { User: [{ column: 'ssn', encryptedColumn: 'ssn_encrypted' }] }, + }) + + const { info, unresolvedHint } = await resolveColumnLifecycle( + clientWithColumns('ssn', 'ssn_encrypted', 'email_enc'), + 'User', + 'ssn', + ) + + expect(info).toBeNull() + expect(unresolvedHint).toBe('ssn_encrypted') + }) + it('explains the recorded counterpart by name rather than listing candidates', async () => { const message = explainUnresolved( 'users', diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index 6c4834d0b..d2b0ac889 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -1,4 +1,5 @@ import { + columnExists, type EncryptedColumnInfo, listEncryptedColumns, pickEncryptedColumn, @@ -111,25 +112,6 @@ export async function resolveColumnLifecycle( return { info: pickEncryptedColumn(candidates, column), candidates } } -/** Whether `column` exists on `table` at all, whatever its type. */ -async function columnExists( - client: pg.ClientBase, - table: string, - column: string, -): Promise { - const { rows } = await client.query<{ exists: boolean }>( - `SELECT EXISTS ( - SELECT 1 FROM pg_attribute - WHERE attrelid = to_regclass($1) - AND attname = $2 - AND attnum > 0 - AND NOT attisdropped - ) AS exists`, - [table, column], - ) - return rows[0]?.exists === true -} - /** * Explain a failed resolution (`info === null`) to the user, or return * `null` when the failure is fine to fall through to the v2 lifecycle. @@ -171,7 +153,7 @@ export function explainUnresolved( // that: listing the v3 candidates here would invite the user to record one of // them, which is how the guess used to get laundered into a `via: 'hint'` match. if (unresolvedHint !== undefined) { - return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column. ${table} also holds EQL v3 columns, and none of them is a confirmed counterpart for ${column}, so this command cannot tell which lifecycle applies and will not guess.\n\nIf that pairing is wrong, correct or remove "encryptedColumn" for ${column} in .cipherstash/migrations.json and re-run. If it is right, ${column} is on the EQL v2 lifecycle: drive it against ${unresolvedHint} directly rather than through this command, which resolves EQL v3 counterparts only.` + return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column. ${table} also holds EQL v3 columns, and none of them is a confirmed counterpart for ${column}, so this command cannot tell which lifecycle applies and will not guess.\n\nIf that pairing is wrong, correct or remove "encryptedColumn" for ${column} in .cipherstash/migrations.json and re-run. If it is right, ${column} is on the EQL v2 lifecycle, which no stash command can drive on this table — complete it yourself against ${unresolvedHint} with the eql_v2 SQL.` } const listed = candidates diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 282a0320b..9750af055 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -385,8 +385,10 @@ const DRIZZLE_PLACEHOLDER = `/** * to reference the encrypted tables you declared there. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and \`stash encrypt\` - * commands refuse to run and point back here. + * placeholder table so that this file compiles, and \`stash db push\`, + * \`stash db validate\` and \`stash encrypt backfill\` refuse to run and point + * back here. (\`stash encrypt cutover\` / \`drop\` resolve against the database + * and never read this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack-drizzle\`. @@ -433,8 +435,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — \`Encryption\` requires at least one. Swap it for your -// real tables (see the patterns above); \`stash encrypt\` refuses to run while -// the placeholder is still here. +// real tables (see the patterns above); \`stash db push\`, \`stash db validate\` +// and \`stash encrypt backfill\` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) @@ -451,8 +453,10 @@ const GENERIC_PLACEHOLDER = `/** * \`Encryption({ schemas: [...] })\` call below to reference them. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and \`stash encrypt\` - * commands refuse to run and point back here. + * placeholder table so that this file compiles, and \`stash db push\`, + * \`stash db validate\` and \`stash encrypt backfill\` refuse to run and point + * back here. (\`stash encrypt cutover\` / \`drop\` resolve against the database + * and never read this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack/eql/v3\` @@ -495,8 +499,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — \`Encryption\` requires at least one. Swap it for your -// real tables (see the patterns above); \`stash encrypt\` refuses to run while -// the placeholder is still here. +// real tables (see the patterns above); \`stash db push\`, \`stash db validate\` +// and \`stash encrypt backfill\` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/vitest.config.ts b/packages/cli/vitest.config.ts index 195a6a156..6f2cf7f76 100644 --- a/packages/cli/vitest.config.ts +++ b/packages/cli/vitest.config.ts @@ -12,9 +12,21 @@ export default defineConfig({ // `@cipherstash/migrate` publishes `./dist` only, so importing it — or // spreading `importOriginal()` inside a partial `vi.mock` — makes the // UNIT suite require a prior workspace build. CI only hides that because - // turbo runs `^build` first; on a clean checkout `pnpm --filter stash - // test` would fail to resolve it. Resolving to source keeps the unit - // config self-contained, as `AGENTS.md` says it is (#787 review). + // turbo runs `^build` first; without this alias and without a build, 10 + // files / 177 tests fail to collect with `Failed to resolve entry for + // package "@cipherstash/migrate"` — the transitive `src` importers + // (`status/index.ts`, `db/install.ts`, `encrypt/lib/db-readers.ts`), not + // just the one mocked test. A full-factory `vi.mock` does not help: + // Vite's import analysis fails on the source module's import statement + // before mocking runs. + // + // This removes the `@cipherstash/migrate` build coupling ONLY. The suite + // is still not self-contained: `packages/migrate/src/backfill.ts` imports + // `@cipherstash/stack`, so removing `packages/stack/dist` still fails 10 + // files. Closing that needs `stackSourceAlias`, which cannot be spread + // here — its `'@/'` entry (pointing at `packages/stack/src`) would + // clobber this package's own `'@/'`. That is why `vitest.shared.ts` is + // not imported by this config (#787 review). '@cipherstash/migrate': resolve(__dirname, '../migrate/src/index.ts'), }, }, diff --git a/packages/migrate/src/__tests__/version.test.ts b/packages/migrate/src/__tests__/version.test.ts index 96e5cfbb6..140b12738 100644 --- a/packages/migrate/src/__tests__/version.test.ts +++ b/packages/migrate/src/__tests__/version.test.ts @@ -2,6 +2,7 @@ import type { ClientBase } from 'pg' import { describe, expect, it, vi } from 'vitest' import { classifyEqlDomain, + columnExists, detectColumnEqlVersion, type EncryptedColumnInfo, listEncryptedColumns, @@ -202,3 +203,36 @@ describe('resolveEncryptedColumn', () => { }) }) }) + +describe('columnExists', () => { + it('returns true when the catalog reports the column', async () => { + const { client } = mockClient([{ exists: true }]) + expect(await columnExists(client, 'users', 'ssn_encrypted')).toBe(true) + }) + + it('returns false when it does not', async () => { + const { client } = mockClient([{ exists: false }]) + expect(await columnExists(client, 'users', 'gone')).toBe(false) + }) + + it('preserves identifier case — no bare to_regclass($1)', async () => { + // Same guard as `detectColumnEqlVersion` above. A bare `to_regclass($1)` + // parses its argument and case-folds it, so a Prisma-style `"User"` table + // silently resolves to `user` and the probe reports "column missing" for a + // column that plainly exists. In `resolveColumnLifecycle` that turns the + // #772 fail-closed into a silent no-op (#787 review). + const { client, query } = mockClient([{ exists: true }]) + await columnExists(client, 'User', 'ssn_encrypted') + const [sql, values] = query.mock.calls[0] as [string, unknown[]] + expect(sql).toContain("format('%I'") + expect(sql).not.toMatch(/to_regclass\(\$1\)/) + expect(values).toEqual(['User', null, 'ssn_encrypted']) + }) + + it('splits schema-qualified names on the first dot, like qualifyTable', async () => { + const { client, query } = mockClient([{ exists: true }]) + await columnExists(client, 'app.users', 'ssn_encrypted') + const [, values] = query.mock.calls[0] as [string, unknown[]] + expect(values).toEqual(['users', 'app', 'ssn_encrypted']) + }) +}) diff --git a/packages/migrate/src/index.ts b/packages/migrate/src/index.ts index 65bd0db15..026b5f6e5 100644 --- a/packages/migrate/src/index.ts +++ b/packages/migrate/src/index.ts @@ -73,6 +73,7 @@ export { } from './state.js' export { classifyEqlDomain, + columnExists, detectColumnEqlVersion, type EncryptedColumnInfo, type EncryptedColumnResolution, diff --git a/packages/migrate/src/version.ts b/packages/migrate/src/version.ts index 60e1d9d41..88c496170 100644 --- a/packages/migrate/src/version.ts +++ b/packages/migrate/src/version.ts @@ -105,6 +105,38 @@ export async function detectColumnEqlVersion( return domain === undefined ? null : classifyEqlDomain(domain) } +/** + * Whether `columnName` exists on `tableName` at all, whatever its type. + * + * Distinct from {@link detectColumnEqlVersion}, which answers "and is it an EQL + * column?". Callers need the difference to tell a STALE reference (the column + * is gone) from a live one the classifier simply does not recognise — most + * often a legacy `eql_v2_encrypted` counterpart. + * + * Lives here rather than in the CLI so it shares {@link REGCLASS_SQL} with the + * other catalog probes: a hand-rolled `to_regclass($1)` case-folds unquoted + * identifiers and silently reports "missing" for a Prisma-style `"User"` table + * (#787 review). + */ +export async function columnExists( + client: ClientBase, + tableName: string, + columnName: string, +): Promise { + const { schema, table } = splitTableName(tableName) + const { rows } = await client.query<{ exists: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM pg_attribute + WHERE attrelid = ${REGCLASS_SQL} + AND attname = $3 + AND attnum > 0 + AND NOT attisdropped + ) AS exists`, + [table, schema, columnName], + ) + return rows[0]?.exists === true +} + /** * Every EQL-domain column on a table, classified. The EQL types are * self-describing, so this is the ground truth for "which columns on this diff --git a/scripts/__tests__/cli-vitest-alias.test.mjs b/scripts/__tests__/cli-vitest-alias.test.mjs new file mode 100644 index 000000000..881ca11af --- /dev/null +++ b/scripts/__tests__/cli-vitest-alias.test.mjs @@ -0,0 +1,45 @@ +import { existsSync } from 'node:fs' +import { describe, expect, it } from 'vitest' +import cliVitestConfig from '../../packages/cli/vitest.config.ts' + +/** + * `packages/cli/vitest.config.ts` carries its own alias map, outside the one + * `vitest-shared-alias.test.mjs` guards. It has to: closing the CLI suite's + * remaining build coupling would need `stackSourceAlias`, whose `'@/'` entry + * points at `packages/stack/src` and would clobber the CLI's own `'@/'`. So the + * map is package-local and needs its own guard, for the same reason — an alias + * pointing at a moved file fails LATE, with a "failed to resolve import" naming + * a path nobody wrote. + * + * `@cipherstash/migrate` is pinned by name because it is load-bearing, not + * cosmetic: without it, 10 files / 177 tests fail to COLLECT on an unbuilt + * workspace (the transitive `src` importers, not just the one mocked test). + * `pnpm run test:scripts` runs ahead of the build-dependent suite in CI, so + * this fires before that failure would (#787 review). + */ + +const aliases = cliVitestConfig.resolve.alias + +/** Directory aliases are written with a trailing slash (`'@/'`). */ +const targets = Object.entries(aliases).map(([specifier, target]) => [ + specifier, + target.endsWith('/') ? target.slice(0, -1) : target, +]) + +describe('packages/cli vitest alias map', () => { + it('still aliases @cipherstash/migrate to source', () => { + // migrate publishes `./dist` only. Drop this entry — or re-introduce an + // `importOriginal()` spread that needs the built package — and the unit + // suite silently regains a dependency on a prior `turbo run build`. + expect(Object.keys(aliases)).toContain('@cipherstash/migrate') + expect(aliases['@cipherstash/migrate']).toMatch( + /packages[/\\]migrate[/\\]src[/\\]index\.ts$/, + ) + }) + + it.each( + targets, + )('%s resolves to a file that exists', (_specifier, target) => { + expect(existsSync(target), `missing alias target: ${target}`).toBe(true) + }) +}) diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index e2541e40f..849763fb9 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -199,7 +199,7 @@ export default defineConfig({ | Option | Required | Default | Purpose | |---|---|---|---| | `databaseUrl` | yes | — | PostgreSQL connection string | -| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db validate`, `schema build`, `encrypt *` | +| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db push`, `db validate`, `encrypt backfill` (`schema build` only *writes* here; `encrypt cutover`/`drop` resolve against the database) | Resolved by walking up from `process.cwd()`, like `tsconfig.json`. `stash init` scaffolds it; `stash eql install` offers to. @@ -473,7 +473,7 @@ Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgr stash encrypt cutover --table users --column email ``` -**EQL v2 only** — v3 has no cut-over: the application switches to the encrypted column by name. Running this command on a **backfilled** v3 column reports "not applicable" (exit 0) with the next step; before backfill completes it exits 1 and says to finish the backfill (never "switch now" onto a half-populated column). For v2, the preconditions are: the column is in the `backfilled` phase, **and** a pending EQL configuration exists (on a v3-only database — no `eql_v2_configuration` table — it explains that and exits 1). +**EQL v2 only** — v3 has no cut-over: the application switches to the encrypted column by name. Running this command on a **backfilled** v3 column reports "not applicable" (exit 0) with the next step — *provided the encrypted column can be identified*. If it was resolved only by elimination (the table's one remaining EQL column, with no `encryptedColumn` recorded in `.cipherstash/migrations.json` and no `_encrypted` name match), it refuses and exits 1 rather than report an outcome for a column it guessed. `.cipherstash/` is gitignored, so a clone or CI runner without that manifest can hit this on a table whose encrypted column is named unconventionally; re-run `stash encrypt backfill … --encrypted-column ` to record the pairing; before backfill completes it exits 1 and says to finish the backfill (never "switch now" onto a half-populated column). For v2, the preconditions are: the column is in the `backfilled` phase, **and** a pending EQL configuration exists (on a v3-only database — no `eql_v2_configuration` table — it explains that and exits 1). In one transaction it renames `` → `_plaintext` and `_encrypted` → ``, advances the pending config to `encrypting`, activates it, and appends a `cut_over` event. With a Proxy URL configured (`--proxy-url` or `CIPHERSTASH_PROXY_URL`) it then calls `eql_v2.reload_config()` so Proxy picks up the new shape. diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 036b3f19f..010e63d4d 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -822,7 +822,7 @@ try { ## Rolling Encryption Out to Production -> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). +> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column — *unless* the table also holds EQL v3 columns and `.cipherstash/migrations.json` records an `encryptedColumn` that is one of the unclassified ones. That combination is ambiguous, so `cutover`/`drop` fail closed and name the recorded column instead of guessing. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. @@ -873,7 +873,7 @@ Once dual-writes are recorded as live in `cs_migrations`: |---|---| | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | | Schema rename (**v2 only**) | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. **v3:** there is no rename — leave `_encrypted` under its own name and point the schema/queries at that name instead. | -| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Reads of `` through **CipherStash Proxy** are decrypted on the wire; SDK/ORM reads still return ciphertext and need the decrypt path in the next row. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. | +| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Reads of `` through **CipherStash Proxy** are decrypted on the wire; SDK/ORM reads still return ciphertext and need the decrypt path in the next row. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. Exception: if the encrypted column was identified only by elimination (no recorded `encryptedColumn`, no `_encrypted` name match), it exits 1 rather than report an outcome for a guess — record the pairing with `stash encrypt backfill … --encrypted-column `. | | Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | | Remove dual-write code | The plaintext column is no longer authoritative — **v2:** it is now `_plaintext` (the cutover renamed it); **v3:** it is still the original ``, since nothing was renamed. Either way, delete the dual-write logic once reads are served from the encrypted column. | | `stash encrypt drop` | Emits a migration that drops the plaintext column — and *which* column that is depends on the generation. **v2** (precondition: phase `cut-over`): a plain `ALTER TABLE … DROP COLUMN "_plaintext"`. **v3** (precondition: phase `backfilled`): it drops the **original ``** — there is no `_plaintext` — and the generated SQL is a `DO` block that takes `ACCESS EXCLUSIVE` on the table, re-counts rows with `` set and `_encrypted` NULL *at apply time*, and raises instead of dropping if any remain. Apply with the project's normal migration tooling. | diff --git a/turbo.json b/turbo.json index f86fa0a04..0acfb8b38 100644 --- a/turbo.json +++ b/turbo.json @@ -55,6 +55,10 @@ "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$"] }, + "typecheck:scaffold": { + "dependsOn": ["^build"], + "inputs": ["$TURBO_DEFAULT$"] + }, "test:e2e": { "dependsOn": ["^build", "build"], "inputs": ["$TURBO_DEFAULT$", ".env*"], From 1e837e562855d6b587fd86a5cce4490bd11a994b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 11:06:55 +1000 Subject: [PATCH 084/123] test(ci): pin the turbo routing that the #787 review's minor finding exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reviewer's two findings were both already fixed at bbade2c4 — this adds the guards that would have caught them, plus two accuracy fixes. `typecheck:scaffold` declares `dependsOn: ["^build"]`, so turbo has to be the thing invoking it. #787 first added it as a bare `pnpm --filter stash run typecheck:scaffold` and CI went green anyway: an earlier step in the same job happened to build the workspace via its own `^build`. Those earlier steps read as independent guards for other packages, so they look freely removable — delete one and the scaffold step fails `TS2307`, which reads as "the scaffold is broken" rather than "you skipped the build". Nothing caught that. scripts/__tests__/workflow-turbo-build-deps.test.mjs derives the protected set from turbo.json (every task declaring `^build`) rather than hardcoding names, and asserts tests.yml routes them through turbo. Mutation-tested against three regressions: reverting the step to the bare form, deleting the step outright, and adding a new bare build-dependent step — each fails with a message naming the step and the fix. Two pre-existing bare `typecheck` invocations (prisma-next, wizard) carry the same latent trap; they are recorded in KNOWN_BARE rather than silently skipped, and a fourth test fails if an entry goes stale so the list gets worked down instead of accumulating. turbo.json is JSONC, so reading it needs the string-aware comment stripper #782 wrote for `turbo-skills-inputs.test.mjs` — a naive regex eats the `//` in `"$schema": "https://turbo.build/schema.json"`. Rather than hand-copy 30 lines of parser into an adjacent file, that function moves to `scripts/__tests__/lib/read-jsonc.mjs` and both guards import it. Two accuracy fixes: - vitest.config.ts claimed the suite's residual `@cipherstash/stack` coupling runs through `migrate/src/backfill.ts`. That is only one of two routes: `init/lib/__tests__/introspect.test.ts:1` imports `@cipherstash/stack/eql/v3` directly, never touching migrate. As written, someone could decouple backfill.ts and expect the suite to go standalone. It would not. - cli-vitest-alias.test.mjs called `.endsWith()` on every alias value, so Vite's array form (`[{find, replacement}]`) would throw an opaque TypeError from a helper — reading as "the guard is broken" rather than "the config changed shape". It now fails cleanly, naming the offender, and says to update the guard rather than delete it. No changeset: repo tooling and a comment, no published surface touched. --- packages/cli/vitest.config.ts | 17 +- scripts/__tests__/cli-vitest-alias.test.mjs | 28 ++- scripts/__tests__/lib/read-jsonc.mjs | 49 ++++++ .../__tests__/turbo-skills-inputs.test.mjs | 37 +--- .../workflow-turbo-build-deps.test.mjs | 166 ++++++++++++++++++ 5 files changed, 251 insertions(+), 46 deletions(-) create mode 100644 scripts/__tests__/lib/read-jsonc.mjs create mode 100644 scripts/__tests__/workflow-turbo-build-deps.test.mjs diff --git a/packages/cli/vitest.config.ts b/packages/cli/vitest.config.ts index 6f2cf7f76..76b3903f4 100644 --- a/packages/cli/vitest.config.ts +++ b/packages/cli/vitest.config.ts @@ -21,12 +21,17 @@ export default defineConfig({ // before mocking runs. // // This removes the `@cipherstash/migrate` build coupling ONLY. The suite - // is still not self-contained: `packages/migrate/src/backfill.ts` imports - // `@cipherstash/stack`, so removing `packages/stack/dist` still fails 10 - // files. Closing that needs `stackSourceAlias`, which cannot be spread - // here — its `'@/'` entry (pointing at `packages/stack/src`) would - // clobber this package's own `'@/'`. That is why `vitest.shared.ts` is - // not imported by this config (#787 review). + // is still not self-contained: removing `packages/stack/dist` still + // fails 10 files, via TWO independent routes — + // 1. `packages/migrate/src/backfill.ts` imports `@cipherstash/stack`, + // so this alias reaches it transitively; and + // 2. `init/lib/__tests__/introspect.test.ts` imports + // `@cipherstash/stack/eql/v3` directly, never touching migrate. + // Route 2 means decoupling `backfill.ts` would NOT make the suite + // standalone. Closing both needs `stackSourceAlias`, which cannot be + // spread here — its `'@/'` entry (pointing at `packages/stack/src`) + // would clobber this package's own `'@/'`. That is why + // `vitest.shared.ts` is not imported by this config (#787 review). '@cipherstash/migrate': resolve(__dirname, '../migrate/src/index.ts'), }, }, diff --git a/scripts/__tests__/cli-vitest-alias.test.mjs b/scripts/__tests__/cli-vitest-alias.test.mjs index 881ca11af..5d93db5e7 100644 --- a/scripts/__tests__/cli-vitest-alias.test.mjs +++ b/scripts/__tests__/cli-vitest-alias.test.mjs @@ -20,13 +20,33 @@ import cliVitestConfig from '../../packages/cli/vitest.config.ts' const aliases = cliVitestConfig.resolve.alias +/** + * Vite also accepts the array form (`[{ find, replacement }]`). This guard + * reads the object-of-strings form; on the array form every check below would + * throw `TypeError: target.endsWith is not a function` from a helper, which + * reads as "the guard is broken" rather than "the config changed shape". + * Fail here instead, naming the offender. + */ +const nonStringTargets = Object.entries(aliases).filter( + ([, target]) => typeof target !== 'string', +) + /** Directory aliases are written with a trailing slash (`'@/'`). */ -const targets = Object.entries(aliases).map(([specifier, target]) => [ - specifier, - target.endsWith('/') ? target.slice(0, -1) : target, -]) +const targets = Object.entries(aliases) + .filter(([, target]) => typeof target === 'string') + .map(([specifier, target]) => [ + specifier, + target.endsWith('/') ? target.slice(0, -1) : target, + ]) describe('packages/cli vitest alias map', () => { + it('is the object-of-strings form this guard can read', () => { + expect( + nonStringTargets.map(([specifier]) => specifier), + 'packages/cli/vitest.config.ts switched away from the object-of-strings alias form. Update this guard to walk the new shape — do not delete it.', + ).toEqual([]) + }) + it('still aliases @cipherstash/migrate to source', () => { // migrate publishes `./dist` only. Drop this entry — or re-introduce an // `importOriginal()` spread that needs the built package — and the unit diff --git a/scripts/__tests__/lib/read-jsonc.mjs b/scripts/__tests__/lib/read-jsonc.mjs new file mode 100644 index 000000000..3ecff8b80 --- /dev/null +++ b/scripts/__tests__/lib/read-jsonc.mjs @@ -0,0 +1,49 @@ +import { readFileSync } from 'node:fs' + +/** + * Parse a JSONC file (`turbo.json` and friends allow comments and trailing + * commas; `JSON.parse` does not). + * + * String-aware by necessity, not fussiness: `turbo.json` opens with + * `"$schema": "https://turbo.build/schema.json"`, so a regex that strips `//` + * without tracking string state eats the URL and yields invalid JSON — or, + * worse, silently valid JSON with the wrong contents. + * + * Extracted from `turbo-skills-inputs.test.mjs` when a second guard needed it. + * Two hand-rolled copies of this would drift, and a subtly wrong copy fails as + * a confusing `SyntaxError` at a byte offset rather than as a clear defect. + */ +export function readJsonc(path) { + const raw = readFileSync(path, 'utf8') + let out = '' + let inString = false + let escaped = false + for (let i = 0; i < raw.length; i++) { + const ch = raw[i] + if (inString) { + out += ch + if (escaped) escaped = false + else if (ch === '\\') escaped = true + else if (ch === '"') inString = false + continue + } + if (ch === '"') { + inString = true + out += ch + continue + } + if (ch === '/' && raw[i + 1] === '/') { + while (i < raw.length && raw[i] !== '\n') i++ + out += '\n' + continue + } + if (ch === '/' && raw[i + 1] === '*') { + i += 2 + while (i < raw.length && !(raw[i] === '*' && raw[i + 1] === '/')) i++ + i++ + continue + } + out += ch + } + return JSON.parse(out.replace(/,(\s*[}\]])/g, '$1')) +} diff --git a/scripts/__tests__/turbo-skills-inputs.test.mjs b/scripts/__tests__/turbo-skills-inputs.test.mjs index 7706ca767..a2413a121 100644 --- a/scripts/__tests__/turbo-skills-inputs.test.mjs +++ b/scripts/__tests__/turbo-skills-inputs.test.mjs @@ -21,45 +21,10 @@ import { existsSync, readdirSync, readFileSync } from 'node:fs' import { join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { describe, expect, it } from 'vitest' +import { readJsonc } from './lib/read-jsonc.mjs' const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') -/** Strip comments so `JSON.parse` accepts turbo.json (it is JSONC). */ -function readJsonc(path) { - const raw = readFileSync(path, 'utf8') - let out = '' - let inString = false - let escaped = false - for (let i = 0; i < raw.length; i++) { - const ch = raw[i] - if (inString) { - out += ch - if (escaped) escaped = false - else if (ch === '\\') escaped = true - else if (ch === '"') inString = false - continue - } - if (ch === '"') { - inString = true - out += ch - continue - } - if (ch === '/' && raw[i + 1] === '/') { - while (i < raw.length && raw[i] !== '\n') i++ - out += '\n' - continue - } - if (ch === '/' && raw[i + 1] === '*') { - i += 2 - while (i < raw.length && !(raw[i] === '*' && raw[i + 1] === '/')) i++ - i++ - continue - } - out += ch - } - return JSON.parse(out.replace(/,(\s*[}\]])/g, '$1')) -} - /** Packages whose tsup config copies the repo-root `skills/` into `dist/`. */ function packagesCopyingSkills() { const pkgsDir = join(REPO_ROOT, 'packages') diff --git a/scripts/__tests__/workflow-turbo-build-deps.test.mjs b/scripts/__tests__/workflow-turbo-build-deps.test.mjs new file mode 100644 index 000000000..1de2124d1 --- /dev/null +++ b/scripts/__tests__/workflow-turbo-build-deps.test.mjs @@ -0,0 +1,166 @@ +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import yaml from 'js-yaml' +import { describe, expect, it } from 'vitest' +import { readJsonc } from './lib/read-jsonc.mjs' + +/** + * A turbo task declaring `dependsOn: ["^build"]` gets its workspace + * dependencies built before it runs — but ONLY when turbo is the one invoking + * it. Run the same task as a bare package script (`pnpm --filter run + * `) and the dependency graph is skipped silently. + * + * That failure mode is invisible in CI, because a bare step usually still + * passes: some earlier step in the same job built the workspace via its own + * `^build`. The step is then load-bearing on an ordering nobody declared. + * Reorder or delete that earlier step — and they read as independent guards + * for other packages, so they look freely removable — and the bare step fails + * with a module-resolution error (`TS2307`, `Failed to resolve entry for + * package`) that reads as "the code is broken" rather than "you skipped the + * build". + * + * This is not hypothetical: #787 added `typecheck:scaffold` as a bare + * `pnpm --filter stash run typecheck:scaffold` and it went green for exactly + * that reason. The fixtures it typechecks import `@cipherstash/stack/v3`, so + * the step needs that package built; it was routed through turbo in review. + * This test pins that routing so it cannot be quietly "simplified" back. + */ + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') +const WORKFLOW = '.github/workflows/tests.yml' + +/** + * Bare invocations that predate this guard. Each is the same latent trap: it + * passes today only because an earlier step in its job builds the workspace. + * They are recorded rather than ignored so the list can be worked down — do + * not add to it. Route new steps through `turbo run` instead. + */ +const KNOWN_BARE = new Set([ + 'pnpm --filter @cipherstash/prisma-next run typecheck', + 'pnpm --filter @cipherstash/wizard run typecheck', +]) + +const turboJson = readJsonc(resolve(REPO_ROOT, 'turbo.json')) +const rootScripts = + JSON.parse(readFileSync(resolve(REPO_ROOT, 'package.json'), 'utf8')) + .scripts ?? {} + +/** Turbo tasks that build their workspace dependencies first. */ +const buildDependentTasks = new Set( + Object.entries(turboJson.tasks ?? {}) + .filter(([, task]) => (task?.dependsOn ?? []).includes('^build')) + .map(([name]) => name), +) + +/** pnpm's own verbs — never package scripts. */ +const PNPM_VERBS = new Set([ + 'install', + 'exec', + 'dlx', + 'add', + 'why', + 'store', + 'run', +]) + +/** + * The task a command line runs, and whether turbo is the thing running it. + * Returns null when the line runs no recognisable task. + * + * Two shapes matter: + * `pnpm exec turbo run --filter ` / `pnpm turbo ` → routed + * `pnpm [--filter ] [run] ` → bare + */ +function invokedTask(line) { + const turbo = line.match(/\bturbo\b\s+(?:run\s+)?([\w:.-]+)/) + if (turbo) return { task: turbo[1], routed: true } + + const pnpm = line.match( + /\bpnpm\b(?:\s+(?:--filter|-F)\s+\S+|\s+--if-present)*\s+(?:run\s+)?([\w:.-]+)/, + ) + if (!pnpm) return null + const task = pnpm[1] + if (PNPM_VERBS.has(task)) return null + return { task, routed: false } +} + +/** Root scripts may delegate to turbo themselves (`"test": "turbo test ..."`). */ +const rootScriptDelegatesToTurbo = (task) => + typeof rootScripts[task] === 'string' && /\bturbo\b/.test(rootScripts[task]) + +function workflowRunLines(path) { + const doc = yaml.load(readFileSync(resolve(REPO_ROOT, path), 'utf8')) + const lines = [] + for (const [jobName, job] of Object.entries(doc?.jobs ?? {})) { + for (const step of job?.steps ?? []) { + if (typeof step?.run !== 'string') continue + for (const raw of step.run.split('\n')) { + const line = raw.trim() + if (line) lines.push({ jobName, stepName: step.name, line }) + } + } + } + return lines +} + +describe('tests.yml routes build-dependent turbo tasks through turbo', () => { + it('turbo.json declares the tasks this guard protects', () => { + // If `^build` is dropped from these, the guard below silently stops + // checking anything — assert the premise rather than trusting it. + expect(buildDependentTasks).toContain('typecheck:scaffold') + expect(buildDependentTasks).toContain('typecheck') + expect(buildDependentTasks).toContain('build') + }) + + it('typecheck:scaffold is invoked via turbo, never bare', () => { + const invocations = workflowRunLines(WORKFLOW).filter( + ({ line }) => invokedTask(line)?.task === 'typecheck:scaffold', + ) + + // A bare `pnpm --filter stash run typecheck:scaffold` matches + // `invokedTask` but not `routedThroughTurbo`, so it would fail here. + // Deleting the step entirely is caught by this length assertion. + expect( + invocations.length, + `${WORKFLOW} must run typecheck:scaffold — the scaffold fixtures are only typechecked there`, + ).toBeGreaterThan(0) + + for (const { jobName, stepName, line } of invocations) { + expect( + invokedTask(line).routed, + `${WORKFLOW} job "${jobName}" step "${stepName}" runs typecheck:scaffold without turbo:\n ${line}\nThe scaffold fixtures import @cipherstash/stack/v3, so this needs that package BUILT. Use \`pnpm exec turbo run typecheck:scaffold --filter stash\`.`, + ).toBe(true) + } + }) + + it('no new bare invocation of a build-dependent turbo task', () => { + const offenders = workflowRunLines(WORKFLOW) + .filter(({ line }) => { + const invoked = invokedTask(line) + if (!invoked || invoked.routed) return false + if (!buildDependentTasks.has(invoked.task)) return false + if (rootScriptDelegatesToTurbo(invoked.task)) return false + return !KNOWN_BARE.has(line) + }) + .map(({ jobName, stepName, line }) => `${jobName} / ${stepName}: ${line}`) + + expect( + offenders, + `These steps run a turbo task declaring \`dependsOn: ["^build"]\` without turbo, so nothing guarantees their workspace dependencies are built. They pass only while an earlier step in the same job happens to build them. Route them through \`pnpm exec turbo run --filter \`.`, + ).toEqual([]) + }) + + it('the grandfathered bare invocations still exist as written', () => { + // KNOWN_BARE entries are matched by exact string. If a step is reworded or + // fixed, its entry goes stale and silently exempts nothing — or worse, + // masks a different line later. Fail so the list gets pruned. + const lines = new Set(workflowRunLines(WORKFLOW).map(({ line }) => line)) + for (const bare of KNOWN_BARE) { + expect( + lines.has(bare), + `KNOWN_BARE entry is stale — no step in ${WORKFLOW} runs:\n ${bare}\nRemove it from the allowlist.`, + ).toBe(true) + } + }) +}) From 659423a67496eaf45d7133eafa1b9980cc81589f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 14:27:33 +1000 Subject: [PATCH 085/123] test(cli,ci): close the coverage and guard gaps found auditing #787 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the #787 review. Five gaps, none raised by the reviewer, all found by auditing what the PR's own tests actually pin. A guard clause that no test held. `context-placeholder.test.ts` named a case it did not construct: its fixture returned `getEncryptConfig: () => ({})`, so `configuredTables` was empty and the guard short-circuited on the arity check before the sentinel-name comparison ever ran. Deleting `configuredTables.length === 1` left all 891 tests green — measured, not argued. The fixture now declares the sentinel FIRST alongside a real table, because the guard indexes `[0]`, and both tests spy `process.exit` so a regression fails as an assertion instead of killing the worker. One guard, not two. The same refusal was hand-copied into `loadEncryptConfig` and `loadEncryptionContext`, sharing only the constant — and it had already drifted: on a client whose `getEncryptConfig()` returns nothing, `db push` named the cause while `encrypt backfill` fell through to `Table "..." was not found`, the symptom-not-cause message the guard exists to replace. Both now call `requireUsableEncryptConfig`, pinned by a parity test asserting the two seams emit byte-identical text. The resolver/command seam, tested. `encrypt-v3.test.ts` stubs `resolveColumnLifecycle` and hand-writes its return value; `resolve-eql.test.ts` proves the pure-v2 shape produces that value. Nothing ran the real producer into the real consumer. The new composition test mocks no resolution at all — only pg, fs, prompts and the effectful migrate helpers — so the v2 verdict is derived from a real manifest hint and a real catalog read. Mutation-checked: the two #787 defences are independent, so removing either alone is masked by the other; removing both fails this test and nothing else. Its sibling types are now imported rather than re-declared, so the shape cannot drift silently. The workflow guard, applied to all workflows. It scanned only `tests.yml` while enforcing a rule its own docblock generalises. Widened to all 13, which surfaced six real bare invocations of `^build`-dependent tasks — five `test:integration`, plus a `test:e2e` that `rootScriptDelegatesToTurbo` was wrongly exempting (a `--filter` invocation runs the PACKAGE's script, so the root script's delegation says nothing about it). All six routed through turbo with `--env-mode=loose`: turbo defaults to strict and would otherwise withhold the step env these suites need, which `passThroughEnv` could only fix by enumerating ten variables across workflows this change cannot execute. Also corrects `AGENTS.md`'s path to `introspect.test.ts` and states the `stackSourceAlias` collision symmetrically — spreading it either way breaks one package's `'@/'`. Its "fails 10 files" figure was re-measured by deleting `packages/stack/dist`: exactly 10. `stash` unit suite 896/896, scripts 112/112, typecheck:scaffold clean, Biome 0 errors. --- .changeset/encrypt-client-guard-parity.md | 20 ++ .github/workflows/integration-drizzle.yml | 10 +- .github/workflows/integration-prisma-next.yml | 2 +- .github/workflows/integration-supabase.yml | 4 +- .github/workflows/prisma-next-e2e.yml | 2 +- packages/cli/AGENTS.md | 10 +- .../placeholder-guard-parity.test.ts | 121 +++++++++++ .../__tests__/context-placeholder.test.ts | 33 ++- .../encrypt/__tests__/encrypt-v3.test.ts | 16 +- ...-lifecycle-composition.integration.test.ts | 202 ++++++++++++++++++ packages/cli/src/commands/encrypt/context.ts | 33 +-- packages/cli/src/config/index.ts | 34 ++- .../workflow-turbo-build-deps.test.mjs | 85 ++++++-- 13 files changed, 508 insertions(+), 64 deletions(-) create mode 100644 .changeset/encrypt-client-guard-parity.md create mode 100644 packages/cli/src/__tests__/placeholder-guard-parity.test.ts create mode 100644 packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts diff --git a/.changeset/encrypt-client-guard-parity.md b/.changeset/encrypt-client-guard-parity.md new file mode 100644 index 000000000..63c3a2664 --- /dev/null +++ b/.changeset/encrypt-client-guard-parity.md @@ -0,0 +1,20 @@ +--- +'stash': patch +--- + +`stash encrypt backfill` now names the cause when the encryption client has no +initialized encrypt config, instead of reporting a missing table. + +The guard that refuses an unusable client file existed twice — once in +`loadEncryptConfig` (`stash db push` / `db validate`) and once, hand-copied, in +`loadEncryptionContext` (`stash encrypt backfill`). The copies had already +drifted: for a client whose `getEncryptConfig()` returns nothing, `db push` +exited 1 with `Encryption client in has no initialized encrypt config`, +while `encrypt backfill` fell through to `Table "users" was not found in the +encryption client exports. Available: (none)` — naming the symptom rather than +the cause, which is precisely the failure the guard was added to eliminate. + +Both loaders now call one shared guard, so a single file cannot produce two +different diagnoses, and the refusals are pinned at both public entry points. +The un-replaced `stash init` scaffold is unaffected — it was already refused by +both, with the same message. diff --git a/.github/workflows/integration-drizzle.yml b/.github/workflows/integration-drizzle.yml index 7dd8310ba..c4b3e7669 100644 --- a/.github/workflows/integration-drizzle.yml +++ b/.github/workflows/integration-drizzle.yml @@ -153,8 +153,14 @@ jobs: # # Step env, not a `.env` file: `dotenv/config` does not override an # already-set `process.env`, so these win and no secret is written to disk. + # Through turbo, not a bare `pnpm --filter`: `test:integration` declares + # `dependsOn: ["^build", "build"]`, and the only build in this job is the + # setup action's `--filter stash`, which reaches these packages purely by + # coincidence of stash's own dependency graph. `--env-mode=loose` because + # turbo defaults to strict and would otherwise withhold the step env below + # from the test process (#787 review follow-up). - name: Drizzle v3 integration suites - run: pnpm --filter @cipherstash/stack-drizzle run test:integration + run: pnpm exec turbo run test:integration --filter @cipherstash/stack-drizzle --env-mode=loose env: DATABASE_URL: ${{ steps.db.outputs.database-url }} PGRST_URL: ${{ steps.db.outputs.pgrest-url }} @@ -164,7 +170,7 @@ jobs: # `isInstalled` short-circuits against the DB the first invocation already # provisioned — a fast no-op check, not a second schema apply. - name: Shared core + identity + wasm integration suites - run: pnpm --filter @cipherstash/stack run test:integration + run: pnpm exec turbo run test:integration --filter @cipherstash/stack --env-mode=loose env: DATABASE_URL: ${{ steps.db.outputs.database-url }} PGRST_URL: ${{ steps.db.outputs.pgrest-url }} diff --git a/.github/workflows/integration-prisma-next.yml b/.github/workflows/integration-prisma-next.yml index acfb99b64..6b7e19134 100644 --- a/.github/workflows/integration-prisma-next.yml +++ b/.github/workflows/integration-prisma-next.yml @@ -114,7 +114,7 @@ jobs: # `stash eql install --eql-version 3`, so an installer regression fails # here rather than hiding behind a test-only SQL apply. - name: prisma-next v3 family suites - run: pnpm --filter @cipherstash/prisma-next run test:integration + run: pnpm exec turbo run test:integration --filter @cipherstash/prisma-next --env-mode=loose env: # Step env, not a `.env` file: `dotenv/config` does not override an # already-set `process.env`, so these win and no secret hits disk. diff --git a/.github/workflows/integration-supabase.yml b/.github/workflows/integration-supabase.yml index 98b281655..f5eff3093 100644 --- a/.github/workflows/integration-supabase.yml +++ b/.github/workflows/integration-supabase.yml @@ -129,7 +129,7 @@ jobs: # Step env, not a `.env` file: `dotenv/config` does not override an # already-set `process.env`, so these win and no secret is written to disk. - name: Supabase v3 integration suites - run: pnpm --filter @cipherstash/stack-supabase run test:integration + run: pnpm exec turbo run test:integration --filter @cipherstash/stack-supabase --env-mode=loose env: DATABASE_URL: ${{ steps.db.outputs.database-url }} PGRST_URL: ${{ steps.db.outputs.pgrest-url }} @@ -139,7 +139,7 @@ jobs: # `isInstalled` short-circuits against the DB the first invocation already # provisioned — so this is a fast no-op check, not a second schema apply. - name: Shared core integration suites (against Supabase Postgres) - run: pnpm --filter @cipherstash/stack run test:integration + run: pnpm exec turbo run test:integration --filter @cipherstash/stack --env-mode=loose env: DATABASE_URL: ${{ steps.db.outputs.database-url }} PGRST_URL: ${{ steps.db.outputs.pgrest-url }} diff --git a/.github/workflows/prisma-next-e2e.yml b/.github/workflows/prisma-next-e2e.yml index 39be3949a..4bccc8dd4 100644 --- a/.github/workflows/prisma-next-e2e.yml +++ b/.github/workflows/prisma-next-e2e.yml @@ -121,7 +121,7 @@ jobs: done - name: Run E2E suite - run: pnpm --filter @cipherstash/prisma-next-example test:e2e + run: pnpm exec turbo run test:e2e --filter @cipherstash/prisma-next-example --env-mode=loose - name: Stop E2E Postgres container if: always() diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index ced2efbff..aa65e4d00 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -18,10 +18,12 @@ workspace fails at collection with `Failed to resolve entry for package …` rather than at an assertion. `vitest.config.ts` aliases `@cipherstash/migrate` to its source to remove one such coupling; `@cipherstash/stack` remains, reached via `packages/migrate/src/backfill.ts` and a direct import in -`init/lib/introspect.test.ts`. Deleting `packages/stack/dist` fails 10 files. -Closing it needs `vitest.shared.ts`'s `stackSourceAlias`, which cannot be spread -into this config — its `'@/'` entry points at `packages/stack/src` and would -clobber this package's own `'@/'` (#787 review). +`init/lib/__tests__/introspect.test.ts`. Deleting `packages/stack/dist` fails 10 +files. Closing it needs `vitest.shared.ts`'s `stackSourceAlias`, which cannot be +spread into this config: its `'@/'` points at `packages/stack/src` while this +package's points at `packages/cli/src`, and a flat alias map admits only one — +spread it after and stack's entry clobbers the CLI's, spread it before and the +CLI's breaks stack's own source imports (#787 review). ## When to add or update an E2E test diff --git a/packages/cli/src/__tests__/placeholder-guard-parity.test.ts b/packages/cli/src/__tests__/placeholder-guard-parity.test.ts new file mode 100644 index 000000000..80563e9c2 --- /dev/null +++ b/packages/cli/src/__tests__/placeholder-guard-parity.test.ts @@ -0,0 +1,121 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { PLACEHOLDER_TABLE_NAME } from '@/config/index.js' + +/** + * `stash db push` / `db validate` reach the user's encryption client through + * `loadEncryptConfig`; `stash encrypt backfill` reaches the same file through + * `loadEncryptionContext`. Both must refuse the un-replaced `stash init` + * scaffold, and — because it is one file and one mistake — both must say the + * same thing about it. + * + * The two guards were hand-copied rather than shared, so nothing held them + * together: the message text could drift on one side, and the nullish-config + * case HAD already drifted (#787 review follow-up). This pins the agreement at + * both public seams rather than testing the shared helper directly, which + * would prove only that a function calls itself. + * + * Real jiti, real temp project, real filesystem — the only doubles are + * `process.exit` and `console.error`. + */ +describe('the un-replaced init scaffold, refused identically by both loaders', () => { + let tmpDir: string + let originalCwd: () => string + + const writeProject = (clientBody: string) => { + fs.writeFileSync( + path.join(tmpDir, 'stash.config.ts'), + `export default { + databaseUrl: 'postgresql://u:p@127.0.0.1:5432/db', + client: './client.ts', + }`, + ) + fs.writeFileSync(path.join(tmpDir, 'client.ts'), clientBody) + process.cwd = () => tmpDir + } + + /** The duck-typed table shape `loadEncryptionContext` harvests. */ + const table = (name: string) => + `{ tableName: '${name}', build: () => ({ tableName: '${name}', columns: {} }) }` + + /** Run one loader, capturing whatever it wrote to stderr before exiting. */ + const captureRefusal = async (load: () => Promise) => { + const error = vi.spyOn(console, 'error').mockImplementation(() => {}) + const exit = vi.spyOn(process, 'exit').mockImplementation(() => { + throw new Error('process.exit') + }) + let exited = false + try { + await load() + } catch (err) { + exited = (err as Error).message === 'process.exit' + if (!exited) throw err + } + const message = error.mock.calls.flat().join('\n') + error.mockRestore() + exit.mockRestore() + return { exited, message } + } + + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-guard-parity-')) + originalCwd = process.cwd + }) + + afterEach(() => { + process.cwd = originalCwd + vi.restoreAllMocks() + if (tmpDir && fs.existsSync(tmpDir)) { + fs.rmSync(tmpDir, { recursive: true, force: true }) + } + }) + + it('gives db push and encrypt backfill the same refusal for the same file', async () => { + writeProject( + `export const encryptionClient = { + getEncryptConfig: () => ({ tables: { '${PLACEHOLDER_TABLE_NAME}': {} } }), + } + export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)}`, + ) + + const { loadEncryptConfig } = await import('@/config/index.js') + const dbPush = await captureRefusal(() => loadEncryptConfig('./client.ts')) + + const { loadEncryptionContext } = await import( + '../commands/encrypt/context.js' + ) + const backfill = await captureRefusal(() => loadEncryptionContext()) + + expect(dbPush.exited).toBe(true) + expect(backfill.exited).toBe(true) + expect(dbPush.message).toContain('still contains the placeholder table') + expect(backfill.message).toEqual(dbPush.message) + }) + + it('names the cause, not the symptom, when the client has no encrypt config', async () => { + // A client whose `getEncryptConfig()` returns nothing is the same class of + // un-finished setup. `db push` already named it; `encrypt backfill` used to + // fall through to `requireTable`'s `Table "users" was not found … + // Available: (none)` — the symptom-not-cause message this guard exists to + // replace (#787 review follow-up). + writeProject( + `export const encryptionClient = { getEncryptConfig: () => undefined } + export const users = ${table('users')}`, + ) + + const { loadEncryptConfig } = await import('@/config/index.js') + const dbPush = await captureRefusal(() => loadEncryptConfig('./client.ts')) + + const { loadEncryptionContext } = await import( + '../commands/encrypt/context.js' + ) + const backfill = await captureRefusal(() => loadEncryptionContext()) + + expect(dbPush.exited).toBe(true) + expect(backfill.exited).toBe(true) + expect(backfill.message).toContain('no initialized encrypt config') + expect(backfill.message).not.toContain('was not found') + }) +}) diff --git a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts index f1dacb93a..9a0ad2440 100644 --- a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts @@ -118,24 +118,53 @@ describe('loadEncryptionContext — the un-replaced init scaffold', () => { export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)}`, ) + const error = vi.spyOn(console, 'error').mockImplementation(() => {}) + const exit = vi.spyOn(process, 'exit').mockImplementation(() => { + throw new Error('process.exit') + }) + const { loadEncryptionContext } = await import('../context.js') const ctx = await loadEncryptionContext() + // The silence this test is named for. Without the `exit` spy the guard + // firing here would kill the worker instead of failing an assertion. + expect(exit).not.toHaveBeenCalled() + expect(error.mock.calls.flat().join('\n')).not.toContain( + 'still contains the placeholder table', + ) expect(ctx.tables.has(PLACEHOLDER_TABLE_NAME)).toBe(true) }) it('allows the sentinel through once a real table sits alongside it', async () => { // Only the SOLE-placeholder case is the un-replaced scaffold. A user who - // has added real tables must not be blocked by a leftover sentinel export. + // has added real tables must not be blocked by a leftover sentinel. + // + // The sentinel is declared FIRST in the config deliberately: the guard + // reads `configuredTables[0]`, so a placeholder-last fixture passes even + // with the arity check deleted. Ordered this way, dropping + // `configuredTables.length === 1` makes this test fail — which is the only + // thing that pins that clause anywhere in the suite (#787 review follow-up). writeProject( - `export const encryptionClient = { getEncryptConfig: () => ({}) } + `export const encryptionClient = { + getEncryptConfig: () => ({ + tables: { '${PLACEHOLDER_TABLE_NAME}': {}, users: {} }, + }), + } export const placeholderTable = ${table(PLACEHOLDER_TABLE_NAME)} export const users = ${table('users')}`, ) + const error = vi.spyOn(console, 'error').mockImplementation(() => {}) + const exit = vi.spyOn(process, 'exit').mockImplementation(() => { + throw new Error('process.exit') + }) const { loadEncryptionContext } = await import('../context.js') const ctx = await loadEncryptionContext() + expect(exit).not.toHaveBeenCalled() + expect(error.mock.calls.flat().join('\n')).not.toContain( + 'still contains the placeholder table', + ) expect(ctx.tables.has('users')).toBe(true) }) }) diff --git a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts index a0d583294..2b8255e79 100644 --- a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts @@ -1,4 +1,9 @@ +import type { + EncryptedColumnInfo, + ResolvedEncryptedColumn, +} from '@cipherstash/migrate' import { beforeEach, describe, expect, it, vi } from 'vitest' +import type { ResolvedLifecycle } from '../lib/resolve-eql.js' // The EQL-v3 lifecycle branches (#648/#649): v3 has no cut-over rename — // `encrypt cutover` must short-circuit with guidance instead of running the @@ -23,9 +28,14 @@ vi.mock('pg', () => ({ }, })) -type ColumnInfo = { column: string; domain: string; version: 2 | 3 } -type ResolvedInfo = ColumnInfo & { via: 'hint' | 'convention' | 'sole' } -type Lifecycle = { info: ResolvedInfo | null; candidates: ColumnInfo[] } +// Imported, never re-declared. These stubs stand in for the real +// `resolveColumnLifecycle`, so a structural copy would let a renamed or added +// field compile on both sides while the commands read something the resolver no +// longer returns — the exact seam `v2-lifecycle-composition.integration.test.ts` +// exercises at runtime, pinned here at compile time (#787 review follow-up). +type ColumnInfo = EncryptedColumnInfo +type ResolvedInfo = ResolvedEncryptedColumn +type Lifecycle = ResolvedLifecycle const migrateMocks = vi.hoisted(() => ({ listEncryptedColumns: vi.fn(async (): Promise => []), diff --git a/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts b/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts new file mode 100644 index 000000000..15ea2b01d --- /dev/null +++ b/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts @@ -0,0 +1,202 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +/** + * The seam between lifecycle RESOLUTION and the commands that act on it. + * + * `encrypt-v3.test.ts` stubs `resolveColumnLifecycle` and hand-writes the value + * it returns; `resolve-eql.test.ts` separately proves a pure-v2 table produces + * that value. Both can stay green while the shape passed between them changes, + * because neither runs the real producer into the real consumer — and the + * commands' v2/v3 branch is chosen entirely from that shape. + * + * So here `resolve-eql.js` is NOT mocked at all: `resolveColumnLifecycle`, + * `explainUnresolved`, `pickEncryptedColumn`, `listEncryptedColumns` and + * `columnExists` all run for real, and the pure-v2 verdict is derived from a + * real manifest hint plus a real catalog read rather than asserted into + * existence. Only genuine boundaries are faked: `pg`, the filesystem, prompts, + * and the effectful migrate helpers that write to the database (#787 review + * follow-up). + */ + +const queryMock = vi.hoisted(() => + vi.fn(async (_sql: string, _params?: unknown[]) => ({ + rows: [] as unknown[], + })), +) +vi.mock('pg', () => ({ + default: { + Client: class { + connect = vi.fn(async () => {}) + end = vi.fn(async () => {}) + query = queryMock + }, + }, +})) + +/** + * Spread the real module and override only the functions that TOUCH something — + * the manifest file and the database. Everything resolution actually reasons + * with (`listEncryptedColumns`, `columnExists`, `pickEncryptedColumn`, + * `classifyEqlDomain`) stays real and runs against `queryMock` below. + */ +const migrateMocks = vi.hoisted(() => ({ + readManifest: vi.fn(async () => null as unknown), + countUnencrypted: vi.fn(async () => 0), + progress: vi.fn( + async () => ({ phase: 'cut-over' }) as { phase: string } | null, + ), + appendEvent: vi.fn(async () => {}), + setManifestTargetPhase: vi.fn(async () => {}), + renameEncryptedColumns: vi.fn(async () => {}), + migrateConfig: vi.fn(async () => {}), + activateConfig: vi.fn(async () => {}), + reloadConfig: vi.fn(async () => {}), +})) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + ...migrateMocks, +})) + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + confirm: vi.fn(async () => true), + isCancel: vi.fn(() => false), + log: { + info: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + step: vi.fn(), + }, + note: vi.fn(), +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('@/commands/db/detect.js', () => ({ + detectDrizzle: vi.fn(() => false), +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: vi.fn(() => 'npm'), + runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), +})) +vi.mock('../drizzle-helper.js', () => ({ + scaffoldDrizzleMigration: vi.fn(async ({ name }: { name: string }) => ({ + path: `drizzle/${name}.sql`, + })), +})) +const writeFileMock = vi.hoisted(() => vi.fn()) +vi.mock('node:fs', () => ({ + default: { mkdirSync: vi.fn(), writeFileSync: writeFileMock }, +})) + +import * as p from '@clack/prompts' +import { cutoverCommand } from '../cutover.js' +import { dropCommand } from '../drop.js' + +function spyExit() { + return vi + .spyOn(process, 'exit') + .mockImplementation((() => undefined) as never) +} + +/** Columns the faked catalog reports as EQL-domain typed. Empty = pure v2. */ +let eqlColumns: { column: string; domain_name: string }[] = [] +/** Columns the faked catalog reports as existing at all. */ +let existingColumns: string[] = [] + +/** + * Route on the catalog statements these paths issue. Each is matched on text + * unique to it, and ORDER MATTERS: three of them end in `AS exists`, and + * `columnExists` and `listEncryptedColumns` both embed the same shared + * `to_regclass` expression. Matching loosely silently answers the wrong probe — + * routing the v2 pending-config check into `columnExists` made cutover report + * "no pending EQL configuration" instead of running the ladder. + */ +function fakeCatalog() { + queryMock.mockImplementation(async (sql: string, params?: unknown[]) => { + if (typeof sql !== 'string') return { rows: [] } + // The v2 config machine exists… + if (sql.includes("to_regclass('public.eql_v2_configuration')")) { + return { rows: [{ exists: 'eql_v2_configuration' }] } + } + // …and holds a pending row to promote. + if (sql.includes("state = 'pending'")) return { rows: [{ exists: true }] } + if (sql.includes('typname AS domain_name')) return { rows: eqlColumns } + if (sql.includes('pg_attribute') && sql.includes('AS exists')) { + return { + rows: [{ exists: existingColumns.includes(String(params?.[2])) }], + } + } + return { rows: [] } + }) +} + +beforeEach(() => { + vi.clearAllMocks() + // The hint `encrypt backfill` records for EVERY column, v2 included. It is + // what used to be discarded into a guess, and what must now be ignored + // harmlessly on a table with no v3 columns to mis-claim. + migrateMocks.readManifest.mockResolvedValue({ + tables: { + users: [{ column: 'ssn', encryptedColumn: 'ssn_encrypted' }], + }, + }) + migrateMocks.progress.mockResolvedValue({ phase: 'cut-over' }) + eqlColumns = [] + existingColumns = ['ssn', 'ssn_encrypted'] + fakeCatalog() +}) + +describe('the pure-v2 lifecycle, resolved for real end to end', () => { + it('generates the v2 plaintext drop for a table whose encrypted column is legacy v2', async () => { + const exitSpy = spyExit() + + await dropCommand({ table: 'users', column: 'ssn' }) + + expect(exitSpy).not.toHaveBeenCalled() + + // Reached the v2 ladder: `_plaintext`, not the v3 target ``. + const sql = writeFileMock.mock.calls[0]?.[1] as string + expect(sql).toContain('DROP COLUMN "ssn_plaintext"') + // The v3-only coverage gate must not run on a v2 column. + expect(migrateMocks.countUnencrypted).not.toHaveBeenCalled() + }) + + it('runs the v2 rename and config promotion for a table with no EQL v3 columns', async () => { + migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' }) + + const exitSpy = spyExit() + + await cutoverCommand({ table: 'users', column: 'ssn' }) + + expect(exitSpy).not.toHaveBeenCalled() + expect(migrateMocks.renameEncryptedColumns).toHaveBeenCalled() + expect(migrateMocks.migrateConfig).toHaveBeenCalled() + expect(migrateMocks.activateConfig).toHaveBeenCalled() + expect(p.log.error).not.toHaveBeenCalled() + }) + + it('refuses both commands when the table also holds an unidentifiable EQL v3 column', async () => { + // The mixed table from #772 finding 7. Same manifest hint, same v2 pair — + // the ONLY difference is an unrelated v3 column, which is what makes the + // recorded pairing meaningful and a fall-through a guess. Crossing this + // boundary is what neither existing half of the coverage can see. + eqlColumns = [{ column: 'email_enc', domain_name: 'eql_v3_text_search' }] + const exitSpy = spyExit() + + await dropCommand({ table: 'users', column: 'ssn' }) + await cutoverCommand({ table: 'users', column: 'ssn' }) + + // Refusing is the point: both must exit non-zero rather than act on + // `email_enc`, the unrelated v3 column the sole-EQL-column rule would + // otherwise claim. + expect(exitSpy).toHaveBeenCalledWith(1) + expect(writeFileMock).not.toHaveBeenCalled() + expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() + const errors = vi.mocked(p.log.error).mock.calls.flat().join('\n') + expect(errors).toContain('is not an EQL v3 column') + expect(errors).toContain('ssn_encrypted') + }) +}) diff --git a/packages/cli/src/commands/encrypt/context.ts b/packages/cli/src/commands/encrypt/context.ts index 0eab28db1..682328eed 100644 --- a/packages/cli/src/commands/encrypt/context.ts +++ b/packages/cli/src/commands/encrypt/context.ts @@ -3,8 +3,8 @@ import path from 'node:path' import type { EncryptionClient } from '@cipherstash/stack/encryption' import { loadStashConfig, - PLACEHOLDER_TABLE_NAME, type ResolvedStashConfig, + requireUsableEncryptConfig, } from '@/config/index.js' /** @@ -142,30 +142,13 @@ export async function loadEncryptionContext(): Promise { process.exit(1) } - // The guard `loadEncryptConfig` applies for `stash db push` / `db validate`, - // repeated here because `stash encrypt` does not go through that loader. The - // scaffold `stash init` writes declares one sentinel table so the file - // compiles; reaching here with only that table means it was never replaced. - // Without this, `requireTable` reported `Table "users" was not found … - // Available: __stash_placeholder__`, which names the symptom and not the - // cause (#787 review). - // - // Read from `getEncryptConfig()` — the SAME source `loadEncryptConfig` uses — - // not from the harvested export map, or the two commands disagree on one - // file. `schemas: [placeholderTable]` with the `export` keyword dropped is - // still the un-replaced scaffold but exports nothing; conversely a stale - // `export const placeholderTable` beside real tables that are imported - // rather than re-exported is NOT (#787 review). - const configuredTables = Object.keys(client.getEncryptConfig()?.tables ?? {}) - if ( - configuredTables.length === 1 && - configuredTables[0] === PLACEHOLDER_TABLE_NAME - ) { - console.error( - `Error: ${stashConfig.client} still contains the placeholder table \`${PLACEHOLDER_TABLE_NAME}\` that \`stash init\` wrote.\n\nDeclare your encrypted columns and pass those tables to Encryption({ schemas: [...] }) in that file, then re-run this command.`, - ) - process.exit(1) - } + // The same refusal `stash db push` / `db validate` get from + // `loadEncryptConfig`, applied here because `stash encrypt` does not go + // through that loader. Called, not re-implemented: it guards one file, so the + // two commands must say one thing about it. Without this, `requireTable` + // reported `Table "users" was not found … Available: __stash_placeholder__`, + // naming the symptom and not the cause (#787 review). + requireUsableEncryptConfig(client.getEncryptConfig(), stashConfig.client) return { stashConfig, client, tables } } diff --git a/packages/cli/src/config/index.ts b/packages/cli/src/config/index.ts index c304f7d58..4442c43e0 100644 --- a/packages/cli/src/config/index.ts +++ b/packages/cli/src/config/index.ts @@ -231,7 +231,31 @@ export async function loadEncryptConfig( process.exit(1) } - const config = encryptClient.getEncryptConfig() + return requireUsableEncryptConfig( + encryptClient.getEncryptConfig(), + encryptClientPath, + ) +} + +/** + * Refuse an encryption client that cannot drive any command yet, naming the + * cause. + * + * Shared rather than duplicated because it guards ONE file reached by two + * loaders — `loadEncryptConfig` for `stash db push` / `db validate`, and + * `loadEncryptionContext` for `stash encrypt backfill`. When the copies were + * separate they had already drifted on the nullish-config case, so one command + * named the cause while the other fell through to `requireTable`'s `Table + * "users" was not found … Available: (none)` — the symptom-not-cause message + * this guard exists to replace (#787 review follow-up). + * + * Both refusals are hard exits: there is no partially-usable state here, and + * every caller would otherwise have to re-derive that. + */ +export function requireUsableEncryptConfig( + config: EncryptConfig | undefined, + encryptClientPath: string, +): EncryptConfig { if (!config) { console.error( `Error: Encryption client in ${encryptClientPath} has no initialized encrypt config.`, @@ -242,8 +266,12 @@ export async function loadEncryptConfig( // `stash init` scaffolds a client holding one placeholder table, because // `Encryption` requires a non-empty schema set and the scaffold has no real // tables to name yet. Reaching here with only that table means the user never - // replaced it — which used to surface as a confusing "table not found" from - // whichever command ran next. + // replaced it. + // + // Read from the built encrypt config, never from the module's export map: a + // scaffold whose `export` keyword was dropped is still un-replaced, while a + // stale `export const placeholderTable` beside real tables that are imported + // rather than re-exported is not (#787 review). const tables = Object.keys(config.tables ?? {}) if (tables.length === 1 && tables[0] === PLACEHOLDER_TABLE_NAME) { console.error( diff --git a/scripts/__tests__/workflow-turbo-build-deps.test.mjs b/scripts/__tests__/workflow-turbo-build-deps.test.mjs index 1de2124d1..82386574d 100644 --- a/scripts/__tests__/workflow-turbo-build-deps.test.mjs +++ b/scripts/__tests__/workflow-turbo-build-deps.test.mjs @@ -1,4 +1,4 @@ -import { readFileSync } from 'node:fs' +import { readdirSync, readFileSync } from 'node:fs' import { resolve } from 'node:path' import { fileURLToPath } from 'node:url' import yaml from 'js-yaml' @@ -28,7 +28,26 @@ import { readJsonc } from './lib/read-jsonc.mjs' */ const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') -const WORKFLOW = '.github/workflows/tests.yml' + +/** + * The workflow that must carry the `typecheck:scaffold` step specifically. + * Scoped, because "that step exists" is a claim about this file, not about + * workflows in general. + */ +const SCAFFOLD_WORKFLOW = '.github/workflows/tests.yml' + +/** + * Every workflow, not just `tests.yml`. A bare build-dependent step is the same + * latent trap wherever it runs, and the integration workflows are where it is + * least visible: they build only `stash` (via the shared `integration-setup` + * action) and reach every other package through that one task's `^build`. + * Narrowing the guard to one file left five real bare invocations unchecked + * (#787 review follow-up). + */ +const WORKFLOWS = readdirSync(resolve(REPO_ROOT, '.github/workflows')) + .filter((file) => /\.ya?ml$/.test(file)) + .map((file) => `.github/workflows/${file}`) + .sort() /** * Bare invocations that predate this guard. Each is the same latent trap: it @@ -74,7 +93,7 @@ const PNPM_VERBS = new Set([ */ function invokedTask(line) { const turbo = line.match(/\bturbo\b\s+(?:run\s+)?([\w:.-]+)/) - if (turbo) return { task: turbo[1], routed: true } + if (turbo) return { task: turbo[1], routed: true, filtered: false } const pnpm = line.match( /\bpnpm\b(?:\s+(?:--filter|-F)\s+\S+|\s+--if-present)*\s+(?:run\s+)?([\w:.-]+)/, @@ -82,10 +101,19 @@ function invokedTask(line) { if (!pnpm) return null const task = pnpm[1] if (PNPM_VERBS.has(task)) return null - return { task, routed: false } + return { task, routed: false, filtered: /\s(?:--filter|-F)\s/.test(line) } } -/** Root scripts may delegate to turbo themselves (`"test": "turbo test ..."`). */ +/** + * Root scripts may delegate to turbo themselves (`"test": "turbo test ..."`). + * + * Only ever consult this for an UNFILTERED invocation. `pnpm --filter + * ` runs that package's script, so the root script's delegation says + * nothing about it — reading it either way exempted a genuinely bare + * `pnpm --filter @cipherstash/prisma-next-example test:e2e` purely because the + * root happens to define `"test:e2e": "turbo run test:e2e"` (#787 review + * follow-up). + */ const rootScriptDelegatesToTurbo = (task) => typeof rootScripts[task] === 'string' && /\bturbo\b/.test(rootScripts[task]) @@ -104,7 +132,7 @@ function workflowRunLines(path) { return lines } -describe('tests.yml routes build-dependent turbo tasks through turbo', () => { +describe('workflows route build-dependent turbo tasks through turbo', () => { it('turbo.json declares the tasks this guard protects', () => { // If `^build` is dropped from these, the guard below silently stops // checking anything — assert the premise rather than trusting it. @@ -114,7 +142,7 @@ describe('tests.yml routes build-dependent turbo tasks through turbo', () => { }) it('typecheck:scaffold is invoked via turbo, never bare', () => { - const invocations = workflowRunLines(WORKFLOW).filter( + const invocations = workflowRunLines(SCAFFOLD_WORKFLOW).filter( ({ line }) => invokedTask(line)?.task === 'typecheck:scaffold', ) @@ -123,27 +151,35 @@ describe('tests.yml routes build-dependent turbo tasks through turbo', () => { // Deleting the step entirely is caught by this length assertion. expect( invocations.length, - `${WORKFLOW} must run typecheck:scaffold — the scaffold fixtures are only typechecked there`, + `${SCAFFOLD_WORKFLOW} must run typecheck:scaffold — the scaffold fixtures are only typechecked there`, ).toBeGreaterThan(0) for (const { jobName, stepName, line } of invocations) { expect( invokedTask(line).routed, - `${WORKFLOW} job "${jobName}" step "${stepName}" runs typecheck:scaffold without turbo:\n ${line}\nThe scaffold fixtures import @cipherstash/stack/v3, so this needs that package BUILT. Use \`pnpm exec turbo run typecheck:scaffold --filter stash\`.`, + `${SCAFFOLD_WORKFLOW} job "${jobName}" step "${stepName}" runs typecheck:scaffold without turbo:\n ${line}\nThe scaffold fixtures import @cipherstash/stack/v3, so this needs that package BUILT. Use \`pnpm exec turbo run typecheck:scaffold --filter stash\`.`, ).toBe(true) } }) - it('no new bare invocation of a build-dependent turbo task', () => { - const offenders = workflowRunLines(WORKFLOW) - .filter(({ line }) => { - const invoked = invokedTask(line) - if (!invoked || invoked.routed) return false - if (!buildDependentTasks.has(invoked.task)) return false - if (rootScriptDelegatesToTurbo(invoked.task)) return false - return !KNOWN_BARE.has(line) - }) - .map(({ jobName, stepName, line }) => `${jobName} / ${stepName}: ${line}`) + it('no new bare invocation of a build-dependent turbo task in any workflow', () => { + const offenders = WORKFLOWS.flatMap((file) => + workflowRunLines(file) + .filter(({ line }) => { + const invoked = invokedTask(line) + if (!invoked || invoked.routed) return false + if (!buildDependentTasks.has(invoked.task)) return false + // A filtered invocation runs the package's own script, so the root + // script's turbo delegation is irrelevant to it. + if (!invoked.filtered && rootScriptDelegatesToTurbo(invoked.task)) + return false + return !KNOWN_BARE.has(line) + }) + .map( + ({ jobName, stepName, line }) => + `${file} / ${jobName} / ${stepName}: ${line}`, + ), + ) expect( offenders, @@ -155,11 +191,18 @@ describe('tests.yml routes build-dependent turbo tasks through turbo', () => { // KNOWN_BARE entries are matched by exact string. If a step is reworded or // fixed, its entry goes stale and silently exempts nothing — or worse, // masks a different line later. Fail so the list gets pruned. - const lines = new Set(workflowRunLines(WORKFLOW).map(({ line }) => line)) + // Searched across every workflow, not just tests.yml: the allowlist now + // exempts lines found anywhere, so a narrower staleness check would let an + // entry keep exempting a step it no longer describes. + const lines = new Set( + WORKFLOWS.flatMap((file) => workflowRunLines(file)).map( + ({ line }) => line, + ), + ) for (const bare of KNOWN_BARE) { expect( lines.has(bare), - `KNOWN_BARE entry is stale — no step in ${WORKFLOW} runs:\n ${bare}\nRemove it from the allowlist.`, + `KNOWN_BARE entry is stale — no workflow step runs:\n ${bare}\nRemove it from the allowlist.`, ).toBe(true) } }) From ace2a4f0acf5bc8f94b4fa341b828a910c35aa8c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 14:43:55 +1000 Subject: [PATCH 086/123] fix(skills,stack,scripts): correct shipped decrypt guidance, and make the package-path linter sound MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finding 14 — decryptModel/bulkDecryptModels return an AuditableDecryptModelOperation: thenable, with .withLockContext() and .audit(). Three sites in skills/stash-encryption and four in packages/stack/README.md said the opposite ("a plain Promise>", "no .withLockContext() to chain"), steering agents away from the very .audit() chain the audit-on-decrypt changeset advertises. The skill contradicted its own reference table, which was already right — 8b744309 fixed that table and nothing else, despite a commit message claiming otherwise. Both files ship: the skill inside the stash tarball and thence into customer repos via installSkills(), the README inside the @cipherstash/stack tarball. The equivalent statement about the WASM entry is CORRECT and deliberately untouched — that client really does return a bare promise. Also `protectOps.eq` in the setup prompt stash init writes for coding agents. One occurrence repo-wide, and no such export exists; the real API is createEncryptionOperators(client), conventionally `ops`. Finding 15 — the package-path linter had a false positive and a false negative, both fixed with the fixture-driven self-tests the suite already uses: - The name capture had no right anchor, so a sentence-final `packages/stack.` swallowed the period and reported a LIVE package as dead. Uppercase was also excluded from the class, so a capitalised name was never checked at all. - livePackages came from readdirSync, i.e. the working tree. Deleting a package leaves dist/ and node_modules/ behind, so every reference to it kept passing — the exact case the linter was commissioned to catch, silently unenforced on any checkout that had built the package. Now derived from `git ls-files`. Deliberately not "has a package.json": packages/utils has none and is live. - scripts/ was not scanned, so the linters never checked themselves. Adding it immediately surfaced a dead allowlist entry in lint-no-hardcoded-runners for a path that never existed in git history. Self-test fixtures stay exempt — they must name dead packages. Severity note for the record: CI was never affected (fresh checkout, caching disabled), and main carries no required status checks — this was a local developer papercut plus a real soundness hole, not a broken build. Finally, resolves two changesets that contradicted each other in the same release: dynamodb-eql-v3 claimed "EQL v2 tables continue to work unchanged" and "no existing caller needs to change" while stack-dynamodb-v2-write-removal announced the v2 encrypt overloads as removed. The code sides with removal; the claims are now scoped to the decrypt path. --- .changeset/decrypt-chaining-docs.md | 27 +++++++++++ .changeset/dynamodb-eql-v3.md | 13 +++-- .../cli/src/commands/init/lib/setup-prompt.ts | 2 +- packages/stack/README.md | 12 +++-- .../sentence-final.md | 7 +++ .../lint-no-dead-package-paths/uppercase.md | 3 ++ .../lint-no-dead-package-paths.test.mjs | 28 +++++++++++ scripts/lint-no-dead-package-paths.mjs | 47 ++++++++++++++++--- scripts/lint-no-hardcoded-runners.mjs | 1 - skills/stash-encryption/SKILL.md | 6 +-- 10 files changed, 125 insertions(+), 21 deletions(-) create mode 100644 .changeset/decrypt-chaining-docs.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/sentence-final.md create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/uppercase.md diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md new file mode 100644 index 000000000..f2a40d9f7 --- /dev/null +++ b/.changeset/decrypt-chaining-docs.md @@ -0,0 +1,27 @@ +--- +'stash': patch +'@cipherstash/stack': patch +--- + +Correct the shipped documentation for `decryptModel` / `bulkDecryptModels`. + +Three places in `skills/stash-encryption` and four in `packages/stack/README.md` +said these return "a plain `Promise>` (not a chainable operation)" +and that there is therefore "no `.withLockContext()` to chain". They return an +`AuditableDecryptModelOperation`, which is thenable and carries both +`.withLockContext()` and `.audit()` — the same `.audit()` chain the +audit-on-decrypt work advertises. The skill contradicted itself: its own +reference table already listed the correct return type. + +The skill ships inside the `stash` tarball and `installSkills()` copies it into +customer repos, so this was steering agents away from an API that exists. The +README ships in the `@cipherstash/stack` tarball. + +The equivalent statement about the **WASM entry** is correct and unchanged — +`@cipherstash/stack/wasm-inline` really does return a plain promise from decrypt, +with no lock-context argument. + +Also fixes the setup prompt `stash init` writes for coding agents, which +referenced `protectOps.eq` — an API that does not exist anywhere in the repo. +The operators come from `createEncryptionOperators(client)`, conventionally +bound to `ops`. diff --git a/.changeset/dynamodb-eql-v3.md b/.changeset/dynamodb-eql-v3.md index c6303c997..3763c2ee5 100644 --- a/.changeset/dynamodb-eql-v3.md +++ b/.changeset/dynamodb-eql-v3.md @@ -11,10 +11,13 @@ Pass a table built with `encryptedTable` + the `types.*` domains from the typed client from `EncryptionV3` and the nominal client from `Encryption({ config: { eqlVersion: 3 } })` are accepted. -EQL v2 tables continue to work unchanged — this is additive, and no existing -caller needs to change. The table decides which wire format is used, so a -DynamoDB table populated under one version must keep being read with that -version. +EQL v2 tables continue to be **readable** — `decryptModel` / +`bulkDecryptModels` still accept one, so existing items stay accessible. Writing +through a v2 table is a separate matter: the encrypt overloads for it are +removed in this same release (see the DynamoDB v2 write-removal entry), so a +caller that still encrypts through a v2 table does need to change. The table +decides which wire format is used, so a DynamoDB table populated under one +version must keep being read with that version. This fixes a latent bug that made v3 unusable: the write path detected an encrypted value by its `k: 'ct'` tag, but EQL v3 scalars carry no `k` @@ -71,4 +74,4 @@ type, where a declared column `email` becomes `email__source` (plus `email`. `decryptModel` / `bulkDecryptModels` invert it via `DecryptedAttributes`. `AnyEncryptedTable`, `DynamoDBEncryptionClient` and `AuditConfig` are now exported from `@cipherstash/stack/dynamodb` so these signatures can be named. -The EQL v2 overloads are unchanged. +The EQL v2 **decrypt** overloads are unchanged; the v2 encrypt overloads are removed in this release. diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index c0e22ba21..12b149aae 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -280,7 +280,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories from `@cipherstash/stack-drizzle` for Drizzle, and the `types.*` factories from `@cipherstash/stack/eql/v3` (via `encryptedTable`, passed as `schemas`) for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, - '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`protectOps.eq`, etc. — see the integration skill).', + '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`ops.eq`, from `createEncryptionOperators(client)` — see the integration skill).', '6. Verify with a round-trip: insert a record, select it back, confirm the value decrypts and the search ops work.', '', '### Migrate an existing column to encrypted', diff --git a/packages/stack/README.md b/packages/stack/README.md index ae45f1ea6..9e5710eda 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -528,8 +528,10 @@ same claim must be supplied to encrypt and decrypt. Lock contexts work with all operations: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncryptModels`, `bulkDecryptModels`, `bulkEncrypt`, `bulkDecrypt`, `encryptQuery`. `.withLockContext()` also accepts a `LockContext` instance. -On the typed client, `decryptModel` / `bulkDecryptModels` take the lock -context as an optional third argument instead of chaining. +On the typed client, `decryptModel` / `bulkDecryptModels` additionally accept +the lock context as an optional third argument. Use that or `.withLockContext()`, +not both — chaining onto a decrypt that already took a positional lock context +throws. > **Deprecated: `LockContext.identify()`.** Per-operation CTS tokens were removed > in `protect-ffi` 0.25; the token `identify()` fetches is no longer used by @@ -707,14 +709,14 @@ Method signatures are derived from your schemas: plaintext arguments are pinned `returnType` controls the encrypted query term's shape: `'eql'` (default, the EQL JSON payload for the ORM adapters), `'composite-literal'` (a Postgres composite string for `.eq()`/string-based APIs), or `'escaped-composite-literal'` (the same, escaped for embedding). Most users take the default; the adapters set it as needed. | `encryptQuery` | `(terms: ScalarQueryTerm[])` | `BatchEncryptQueryOperation` (thenable) | | `encryptModel` | `(model, table)` | `EncryptModelOperation` (thenable) | -| `decryptModel` | `(encryptedModel, table, lockContext?)` | `Promise>` | +| `decryptModel` | `(encryptedModel, table, lockContext?)` | `AuditableDecryptModelOperation` (thenable) | | `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation` (thenable) | -| `bulkDecryptModels` | `(encryptedModels, table, lockContext?)` | `Promise>` | +| `bulkDecryptModels` | `(encryptedModels, table, lockContext?)` | `AuditableDecryptModelOperation` (thenable) | | `bulkEncrypt` | `(plaintexts, { column, table })` | `BulkEncryptOperation` (thenable) | | `bulkDecrypt` | `(encryptedPayloads)` | `BulkDecryptOperation` (thenable) | | `getEncryptConfig` | `()` | The resolved encrypt config | -The thenable operations support `.withLockContext(lockContext)` for identity-aware encryption. `decryptModel` / `bulkDecryptModels` return a plain `Promise` instead — pass the lock context as the optional third argument. `decrypt` of a single value cannot be strongly typed (a lone ciphertext carries no column identity), and `encryptQuery` rejects storage-only columns at compile time. +The thenable operations support `.withLockContext(lockContext)` for identity-aware encryption, and `decryptModel` / `bulkDecryptModels` also support `.audit({ metadata })`. Those two additionally accept the lock context as an optional third argument — use one form or the other. `decrypt` of a single value cannot be strongly typed (a lone ciphertext carries no column identity), and `encryptQuery` rejects storage-only columns at compile time. ### `LockContext` (legacy) diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/sentence-final.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/sentence-final.md new file mode 100644 index 000000000..7ae81cfb4 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/sentence-final.md @@ -0,0 +1,7 @@ +# Sentence-final package paths + +The client lives in packages/stack. +A hyphen at a wrap: packages/migrate- +An underscore: packages/wizard_ +An ellipsis: packages/cli... +Parenthesised (packages/stack) and comma'd packages/cli, both fine. diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/uppercase.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/uppercase.md new file mode 100644 index 000000000..a2060a9c9 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/uppercase.md @@ -0,0 +1,3 @@ +# Uppercase package names must still be checked + +See packages/Foo/does/not/exist for details. diff --git a/scripts/__tests__/lint-no-dead-package-paths.test.mjs b/scripts/__tests__/lint-no-dead-package-paths.test.mjs index bc9ea927f..c0dacca7b 100644 --- a/scripts/__tests__/lint-no-dead-package-paths.test.mjs +++ b/scripts/__tests__/lint-no-dead-package-paths.test.mjs @@ -43,6 +43,34 @@ describe('lint-no-dead-package-paths', () => { expect(r.output).toMatch(/packages\/protect/) }) + // #772 review, finding 15. The name capture had no right anchor, so a + // sentence-final `packages/stack.` swallowed the period and the linter + // reported a LIVE package as dead — failing the build with a message naming a + // directory that plainly exists. Never fired in 400 commits only because the + // repo's backtick convention happened to dodge it. + it('does not flag a live package followed by sentence punctuation', () => { + const r = run(fx('sentence-final.md')) + expect(r.output).toBe('') + expect(r.exitCode).toBe(0) + }) + + // The character class excluded uppercase, so `packages/Foo` was never checked + // at all — a silent hole rather than a false alarm. + it('checks a package name containing uppercase', () => { + const r = run(fx('uppercase.md')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/Foo/) + }) + + // The linters carry package paths of their own; `scripts/` was not scanned, + // so a `packages/drizzle` allowlist entry for a package that never existed in + // git history sat there unnoticed. Its own fixtures must stay exempt. + it('scans scripts/ but not its fixtures', () => { + const r = run() + expect(r.exitCode).toBe(0) + expect(r.output).toBe('') + }) + it('names the file and line of each offender', () => { const r = run(fx('dead-ref.md')) expect(r.output).toMatch(/dead-ref\.md:3/) diff --git a/scripts/lint-no-dead-package-paths.mjs b/scripts/lint-no-dead-package-paths.mjs index 8ca468d27..47f7497a9 100644 --- a/scripts/lint-no-dead-package-paths.mjs +++ b/scripts/lint-no-dead-package-paths.mjs @@ -1,3 +1,4 @@ +import { execFileSync } from 'node:child_process' import { readdirSync, readFileSync, statSync } from 'node:fs' import { join, relative, resolve } from 'node:path' @@ -24,6 +25,10 @@ const TARGETS = process.argv.slice(2).length 'skills', 'e2e/README.md', 'packages/cli/AGENTS.md', + // The linters themselves carry package paths — an allowlist entry for a + // deleted package sat here unnoticed because `scripts/` was not scanned. + // `__tests__` is excluded below: its fixtures MUST name dead packages. + 'scripts', ] const SKIP_DIRS = new Set([ @@ -32,20 +37,50 @@ const SKIP_DIRS = new Set([ 'plans', 'superpowers', '.git', + // Fixtures for this linter's own self-tests deliberately reference deleted + // packages; scanning them would make the suite unrunnable. + '__tests__', ]) const SKIP_FILES = new Set(['CHANGELOG.md']) const TEXT_EXT = /\.(md|ya?ml|json|mjs|ts|txt)$/ // `packages/` where `` is a real directory name. The character // class excludes `*`, so workspace globs (`packages/*`, `./packages/*`) are -// left alone, and `+` is greedy so `packages/stack-forge` is never excused by -// the live `packages/stack`. -const PACKAGE_REF = /packages\/([a-z0-9][a-z0-9._-]*)/g +// left alone, and it is greedy so a longer directory name is never excused by +// a live package whose name is a prefix of it. +// +// The name must END on an alphanumeric. Without that anchor a sentence-final +// `packages/stack.` — or a hyphen at a line wrap, or a trailing underscore — +// captured the punctuation too and reported a LIVE package as dead, failing +// the build with a message naming a directory that plainly exists. Uppercase +// is admitted so a capitalised directory name is checked rather than silently +// skipped; no package uses one today, which is exactly why nothing noticed +// (#772 review, finding 15). +const PACKAGE_REF = /packages\/([a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?)/g +// Live packages come from what git TRACKS, not from what is on disk. +// +// `readdirSync` was wrong in the direction that matters: deleting a package +// leaves its `dist/` and `node_modules/` behind, so the directory still exists +// and every reference to the deleted package passed. That is the exact failure +// this linter was written to catch, and it silently stopped catching it on any +// checkout where the package had previously been built — two packages deleted +// by this very stack are sitting on `main` right now as exactly such shells +// (#772 review, finding 15). +// +// Note this deliberately does NOT require a `package.json`: `packages/utils` has +// none (it is two loose files consumed by relative path from `packages/nextjs`) +// yet is tracked, live, and referenced from AGENTS.md. const livePackages = new Set( - readdirSync(resolve(REPO_ROOT, 'packages'), { withFileTypes: true }) - .filter((e) => e.isDirectory()) - .map((e) => e.name), + execFileSync('git', ['ls-files', '-z', 'packages'], { + cwd: REPO_ROOT, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + }) + .split('\0') + .filter(Boolean) + .map((file) => file.split('/')[1]) + .filter(Boolean), ) function* walk(abs) { diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index e8bbc8b49..29c453fe8 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -10,7 +10,6 @@ const ALLOWLISTED_PATHS = new Set([ 'packages/wizard/src/lib/detect.ts', // npm row of the PM table 'packages/cli/src/commands/init/utils.ts', // runnerCommand `case 'npm'` 'packages/cli/src/commands/init/lib/setup-prompt.ts', // execCommand `case 'npm':` switch - 'packages/drizzle/src/bin/runner.ts', // Pre-allowlisted: helper for Task 13 'scripts/lint-no-hardcoded-runners.mjs', // this script's own docs ]) diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 010e63d4d..6865ebd2f 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -390,7 +390,7 @@ if (!enc.failure) { Typed-client model notes: -- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a plain `Promise>` (not a chainable operation) — pass a lock context as the optional third argument instead of chaining `.withLockContext()`. +- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a chainable `AuditableDecryptModelOperation` — await it for the `Result`, or chain `.audit({ metadata })` / `.withLockContext(lockContext)` first. A lock context may instead be passed as the optional third argument; use one form or the other, not both (chaining `.withLockContext()` onto a decrypt that already took a positional lock context throws). - `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. - Nullable schema fields stay nullable through the round trip. @@ -698,7 +698,7 @@ Every operation returns a `Result`. Narrow on `.failure` before touching `.data` `identityClaim` is an array of JWT claim *names*, not values: `["sub"]` (the default) or `["sub", "org_id"]`. ZeroKMS resolves each claim's value from the JWT the strategy federated. **The same claim must be supplied to encrypt and decrypt** — it is baked into the data key's tag, so decrypting without it fails with `Failed to retrieve key`. -Lock contexts work with every operation: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. On the typed client, `decryptModel` and `bulkDecryptModels` take the lock context as their optional **third argument** (they return a plain `Promise>`, so there is no `.withLockContext()` to chain): +Lock contexts work with every operation: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. On the typed client, `decryptModel` and `bulkDecryptModels` additionally accept the lock context as an optional **third argument**, which is the form shown here — `.withLockContext()` chains on them too, but use one or the other, not both: ```typescript const dec = await client.decryptModel(enc.data, users, IDENTITY) @@ -747,7 +747,7 @@ const result = await client .audit({ metadata: { action: "create" } }) // optional: audit trail ``` -(The typed client's `decryptModel` / `bulkDecryptModels` are the exception: they return a plain `Promise>` and take the lock context as an argument — see "Model Operations".) +(The typed client's `decryptModel` / `bulkDecryptModels` may also take the lock context as a positional argument instead of chaining — see "Model Operations".) ## Error Handling From 79f3a536ca014f29498435d309f48e89ffce8ff4 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sat, 25 Jul 2026 19:50:20 +1000 Subject: [PATCH 087/123] fix(cli,scripts): per-integration query guidance, and linters that catch their own rot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the #772 review findings, addressing what the review of that review turned up. `stash init`'s setup prompt named `createEncryptionOperators` unconditionally. That symbol is exported by `@cipherstash/stack-drizzle` alone, so a Supabase or Prisma Next project was sent after a package that is not in its dependency tree — the previous `protectOps.eq` was wrong for everyone, this was wrong for three integrations out of four. Step 5 now branches through `queryOperatorGuidance()`, following the `migrationCommands()` pattern already in the file: `ops.eq` for Drizzle, the `encryptedSupabase` wrapper's own filters for Supabase, the `eql*` column operators for Prisma Next, and `client.encryptQuery(...)` for plain Postgres — which is also pointed at `stash-encryption`, since it is the default integration and installs no integration skill, making "see the integration skill" a dangling pointer. The package-path linter reported an untracked-but-present package as "does not exist" — finding 15's false alarm pointed the other way, at a directory sitting right there on disk. The live set now unions `git ls-files --others --exclude-standard`. `--directory` is deliberately not passed: it collapses an all-ignored directory to one entry and would resurrect exactly the `dist/`-and- `node_modules/` shells the linter exists to catch (verified both ways). git failing now exits 2 with an actionable message instead of a raw ENOENT stack trace and exit 1, which was indistinguishable from a genuine lint failure, and an empty live set refuses to run rather than flagging every reference at once. The `scans scripts/ but not its fixtures` test asserted the repo was clean — byte-for-byte what the suite's first test already asserted, and passing whether or not `scripts` was in TARGETS. It now plants an offender in the scanned directory. Verified by mutation: it dies when `scripts` is dropped, as the untracked-package test dies without the union and the git-failure test dies when the exit code is flipped. The runners linter now requires each allowlist entry to exist and to still contain an unexcused `npx` literal. It immediately found a second stale entry the sibling linter structurally cannot see, since it only matches the `packages/` shape: `setup-prompt.ts` has had no `npx` since its switch moved to `utils.ts`. Removed. Also corrects two claims: the test comment saying `packages/drizzle` never existed in git history (it was added speculatively by c6715608, became load-bearing 31 minutes later with 9d259e6e, and rotted when 413ca396 deleted the package), and the `AnyEncryptedTable` doc comment still promising v3 is "purely additive and no existing caller has to change" — true of decrypt, false of encrypt since the v2 write overloads were removed. --- .changeset/decrypt-chaining-docs.md | 8 ++- .changeset/dynamodb-eql-v3.md | 10 +-- .../init/lib/__tests__/setup-prompt.test.ts | 39 +++++++++++ .../cli/src/commands/init/lib/setup-prompt.ts | 26 ++++++- packages/stack/src/dynamodb/types.ts | 11 ++- .../untracked-package.md | 3 + .../lint-no-dead-package-paths.test.mjs | 69 +++++++++++++++++-- .../lint-no-hardcoded-runners.test.mjs | 57 ++++++++++++++- scripts/lint-no-dead-package-paths.mjs | 64 ++++++++++++++--- scripts/lint-no-hardcoded-runners.mjs | 47 ++++++++++++- 10 files changed, 304 insertions(+), 30 deletions(-) create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/untracked-package.md diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md index f2a40d9f7..3e81a244d 100644 --- a/.changeset/decrypt-chaining-docs.md +++ b/.changeset/decrypt-chaining-docs.md @@ -23,5 +23,9 @@ with no lock-context argument. Also fixes the setup prompt `stash init` writes for coding agents, which referenced `protectOps.eq` — an API that does not exist anywhere in the repo. -The operators come from `createEncryptionOperators(client)`, conventionally -bound to `ops`. +The step now names the query API each integration can actually import: +`createEncryptionOperators(client)` (conventionally `ops`) for Drizzle, the +`encryptedSupabase` wrapper's own filters for Supabase, the `eql*` column +operators for Prisma Next, and `client.encryptQuery(...)` for a plain Postgres +project — which is also pointed at `stash-encryption`, since it is installed +with no integration skill. diff --git a/.changeset/dynamodb-eql-v3.md b/.changeset/dynamodb-eql-v3.md index 3763c2ee5..54b960c92 100644 --- a/.changeset/dynamodb-eql-v3.md +++ b/.changeset/dynamodb-eql-v3.md @@ -13,11 +13,11 @@ the typed client from `EncryptionV3` and the nominal client from EQL v2 tables continue to be **readable** — `decryptModel` / `bulkDecryptModels` still accept one, so existing items stay accessible. Writing -through a v2 table is a separate matter: the encrypt overloads for it are -removed in this same release (see the DynamoDB v2 write-removal entry), so a -caller that still encrypts through a v2 table does need to change. The table -decides which wire format is used, so a DynamoDB table populated under one -version must keep being read with that version. +through a v2 table is a separate matter: `encryptModel` / `bulkEncryptModels` +narrowed to EQL v3 tables in this same release, so a caller that still encrypts +through a v2 table does need to change. The table decides which wire format is +used, so a DynamoDB table populated under one version must keep being read with +that version. This fixes a latent bug that made v3 unusable: the write path detected an encrypted value by its `k: 'ct'` tag, but EQL v3 scalars carry no `k` diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index b1b684910..4a76304cc 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -105,6 +105,45 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { expect(out).toContain('pnpm exec drizzle-kit migrate') }) + // The "wire the column through" step names the query API by hand. It used to + // name `protectOps.eq`, which exists nowhere; naming `createEncryptionOperators` + // unconditionally is the same mistake one step smaller — that symbol is + // exported only by `@cipherstash/stack-drizzle`, which a Supabase or Prisma + // project does not even have in its dependency tree. Each integration gets + // the API it can actually import. + describe('query-operator guidance is per-integration', () => { + const render = (integration: SetupPromptContext['integration']) => + renderSetupPrompt({ ...baseCtx, integration }) + + it('names the Drizzle operators for drizzle', () => { + const out = render('drizzle') + expect(out).toContain('createEncryptionOperators(client)') + expect(out).toContain('ops.eq') + }) + + it('names the wrapper builder for supabase, not the Drizzle operators', () => { + const out = render('supabase') + expect(out).not.toContain('createEncryptionOperators') + expect(out).toContain('encryptedSupabase') + }) + + it('names the eql* column operators for prisma-next', () => { + const out = render('prisma-next') + expect(out).not.toContain('createEncryptionOperators') + expect(out).toContain('eqlEq') + }) + + // `postgresql` is the default when nothing is detected, and it gets no + // integration skill at all (install-skills.ts) — so "see the integration + // skill" is a dangling pointer for it too. + it('names the core encryptQuery path for plain postgresql', () => { + const out = render('postgresql') + expect(out).not.toContain('createEncryptionOperators') + expect(out).toContain('encryptQuery') + expect(out).toContain('stash-encryption') + }) + }) + it('emits supabase migration commands for supabase integration', () => { const out = renderSetupPrompt({ ...baseCtx, diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 12b149aae..5a22d42a6 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -68,6 +68,30 @@ function migrationCommands( return undefined } +/** + * How this integration filters on an encrypted column. Named per-integration + * rather than generically because the APIs are not interchangeable and only one + * of them is importable from any given project: `createEncryptionOperators` is + * exported by `@cipherstash/stack-drizzle` alone, so naming it for a Supabase + * or Prisma project sends the agent after a package that is not installed. + * + * `postgresql` is the fallback integration and is installed with no integration + * skill (see `SKILL_MAP` in `install-skills.ts`), so it is pointed at + * `stash-encryption` instead of "the integration skill". + */ +function queryOperatorGuidance(integration: Integration): string { + if (integration === 'supabase') { + return 'query paths filter through the `encryptedSupabase` wrapper (`es.from("users").select(...).eq("email", value)`) — it encrypts filter operands for encrypted columns automatically; see the integration skill' + } + if (integration === 'prisma-next') { + return 'query paths use the `eql*` operators on the column inside `.where()` (`u.email.eqlEq(value)`, `eqlMatch`, `eqlGt`, …) — see the integration skill' + } + if (integration === 'postgresql') { + return 'query paths encrypt the search term first with `client.encryptQuery(value, { table, column, queryType })` and compare against the encrypted column — see the `stash-encryption` skill (a plain Postgres project gets no integration skill)' + } + return 'query paths use the right operator (`ops.eq`, from `createEncryptionOperators(client)`) — see the integration skill' +} + function bullet(line: string): string { return `- ${line}` } @@ -280,7 +304,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories from `@cipherstash/stack-drizzle` for Drizzle, and the `types.*` factories from `@cipherstash/stack/eql/v3` (via `encryptedTable`, passed as `schemas`) for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, - '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`ops.eq`, from `createEncryptionOperators(client)` — see the integration skill).', + `5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, ${queryOperatorGuidance(ctx.integration)}.`, '6. Verify with a round-trip: insert a record, select it back, confirm the value decrypts and the search ops work.', '', '### Migrate an existing column to encrypted', diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index 83aac0414..c84af7b2c 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -21,9 +21,14 @@ import type { EncryptModelOperation } from './operations/encrypt-model' * `encryptedColumn`/`encryptedField` from `@cipherstash/stack/schema`) or an * EQL v3 one (`encryptedTable` + `types.*` from `@cipherstash/stack/eql/v3`). * - * Both are supported deliberately. DynamoDB shares none of the v2 Postgres - * machinery — there is no EQL extension to install and no migration to run — - * so accepting v3 is purely additive and no existing caller has to change. + * This union is the adapter's widest input type — the erased view the internal + * `CallableEncryptionClient` is declared against. It is NOT the public contract: + * the surface split the two versions apart. `encryptModel` / + * `bulkEncryptModels` narrowed to `AnyV3Table` (the v2 write overloads were + * removed, so a v2 encrypt call site does have to change); `decryptModel` / + * `bulkDecryptModels` still take either, so items stored under v2 stay + * readable. See {@link EncryptedDynamoDBInstance} for the overloads that decide + * this per method. */ export type AnyEncryptedTable = | EncryptedTable diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/untracked-package.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/untracked-package.md new file mode 100644 index 000000000..4f808752e --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/untracked-package.md @@ -0,0 +1,3 @@ +# A package that exists on disk but is not yet tracked + +The new thing lives in packages/lint-untracked-probe, scaffolded but not staged. diff --git a/scripts/__tests__/lint-no-dead-package-paths.test.mjs b/scripts/__tests__/lint-no-dead-package-paths.test.mjs index c0dacca7b..89e855dd6 100644 --- a/scripts/__tests__/lint-no-dead-package-paths.test.mjs +++ b/scripts/__tests__/lint-no-dead-package-paths.test.mjs @@ -1,4 +1,5 @@ import { execFileSync } from 'node:child_process' +import { mkdirSync, rmSync, writeFileSync } from 'node:fs' import { resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { describe, expect, it } from 'vitest' @@ -8,9 +9,18 @@ const SCRIPT = resolve( '../../lint-no-dead-package-paths.mjs', ) +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + function run(...targets) { + return runWith({}, ...targets) +} + +function runWith(opts, ...targets) { try { - execFileSync('node', [SCRIPT, ...targets], { encoding: 'utf8' }) + execFileSync(process.execPath, [SCRIPT, ...targets], { + encoding: 'utf8', + ...opts, + }) return { exitCode: 0, output: '' } } catch (err) { return { @@ -63,12 +73,59 @@ describe('lint-no-dead-package-paths', () => { }) // The linters carry package paths of their own; `scripts/` was not scanned, - // so a `packages/drizzle` allowlist entry for a package that never existed in - // git history sat there unnoticed. Its own fixtures must stay exempt. + // so lint-no-hardcoded-runners' `packages/drizzle/src/bin/runner.ts` allowlist + // entry — added speculatively by c6715608, load-bearing 31 minutes later once + // 9d259e6e created the file — sat dead for the two months after 413ca396 + // deleted the package. Its sibling entry for `packages/protect` was removed + // with its package; this one was simply missed. + // + // Asserting the repo is clean would NOT pin this: the suite's first test + // already does that, and both pass whether or not `scripts` is in TARGETS. + // Plant an offender in the scanned directory instead. it('scans scripts/ but not its fixtures', () => { - const r = run() - expect(r.exitCode).toBe(0) - expect(r.output).toBe('') + const probe = resolve(REPO_ROOT, 'scripts/dead-path-probe.md') + try { + writeFileSync(probe, 'A reference to `packages/protect`, long gone.\n') + const r = run() + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/scripts\/dead-path-probe\.md:1/) + // `__tests__` stays skipped — the fixtures name dead packages on purpose, + // and so do the comments in this very file. + expect(r.output).not.toMatch(/__tests__/) + } finally { + rmSync(probe, { force: true }) + } + }) + + // A package scaffolded a minute ago is live, but nothing about it is tracked + // yet. Deriving the live set from `git ls-files` alone reported it as "does + // not exist" — the same species of false alarm as the sentence-final one + // above, pointed the other way, at a directory sitting right there on disk. + it('treats a package that exists but is not yet tracked as live', () => { + const pkg = resolve(REPO_ROOT, 'packages/lint-untracked-probe') + try { + mkdirSync(resolve(pkg, 'src'), { recursive: true }) + writeFileSync(resolve(pkg, 'src/index.ts'), 'export const x = 1\n') + const r = run(fx('untracked-package.md')) + expect(r.output).toBe('') + expect(r.exitCode).toBe(0) + } finally { + rmSync(pkg, { recursive: true, force: true }) + } + }) + + // The live set is derived by shelling out to git, so git failing is a mode + // this linter has to own. Exiting 1 with a raw ENOENT stack trace would be + // indistinguishable from a genuine lint failure; exit 2 says "the linter + // could not run", not "your docs are wrong". + it('exits 2 with an actionable message when git is unavailable', () => { + const r = runWith({ + env: { PATH: resolve(REPO_ROOT, 'scripts/nonexistent') }, + }) + expect(r.exitCode).toBe(2) + expect(r.output).toMatch(/git/) + expect(r.output).toMatch(/checkout/) + expect(r.output).not.toMatch(/at ModuleJob/) }) it('names the file and line of each offender', () => { diff --git a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs index 7bb1202dc..96531ea0e 100644 --- a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs +++ b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs @@ -1,4 +1,5 @@ import { execFileSync } from 'node:child_process' +import { readFileSync, rmSync, writeFileSync } from 'node:fs' import { resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { describe, expect, it } from 'vitest' @@ -8,9 +9,9 @@ const SCRIPT = resolve( '../../lint-no-hardcoded-runners.mjs', ) -function run(target) { +function runScript(script, ...targets) { try { - execFileSync('node', [SCRIPT, target], { encoding: 'utf8' }) + execFileSync(process.execPath, [script, ...targets], { encoding: 'utf8' }) return { exitCode: 0, output: '' } } catch (err) { return { @@ -20,6 +21,10 @@ function run(target) { } } +function run(target) { + return runScript(SCRIPT, target) +} + describe('lint-no-hardcoded-runners', () => { const fx = (name) => resolve(fileURLToPath(import.meta.url), `../fixtures/${name}`) @@ -70,3 +75,51 @@ describe('lint-no-hardcoded-runners', () => { expect(run(fx('identifier.ts')).exitCode).toBe(0) }) }) + +// An allowlist entry is a standing exemption. When the file it names is deleted +// — or stops carrying the `npx` literal it was excused for — the entry becomes +// silent dead weight, and the next reader takes it as evidence that the file +// still needs an exemption. `packages/drizzle/src/bin/runner.ts` sat here for +// the two months after 413ca396 deleted its package, and surfaced only because +// a sibling linter happened to start scanning `scripts/` (#772 review, finding +// 15). That sibling can only ever catch the `packages/` shape; this check +// covers every entry, including a stale path inside a live package. +describe('lint-no-hardcoded-runners — allowlist hygiene', () => { + // A copy alongside the original so `REPO_ROOT` still resolves to the repo. + const PROBE = resolve( + fileURLToPath(import.meta.url), + '../../allowlist-probe.mjs', + ) + + function runWithExtraEntry(entry) { + const src = readFileSync(SCRIPT, 'utf8').replace( + 'const ALLOWLISTED_PATHS = new Set([', + `const ALLOWLISTED_PATHS = new Set([\n '${entry}',`, + ) + try { + writeFileSync(PROBE, src) + return runScript(PROBE) + } finally { + rmSync(PROBE, { force: true }) + } + } + + it('rejects an entry whose file no longer exists', () => { + const r = runWithExtraEntry('scripts/deleted-helper.mjs') + expect(r.exitCode).toBe(2) + expect(r.output).toMatch(/scripts\/deleted-helper\.mjs/) + expect(r.output).toMatch(/no such file/) + }) + + it('rejects an entry whose file no longer needs the exemption', () => { + const r = runWithExtraEntry('scripts/vitest.config.mjs') + expect(r.exitCode).toBe(2) + expect(r.output).toMatch(/scripts\/vitest\.config\.mjs/) + expect(r.output).toMatch(/no longer contains/) + }) + + it('accepts the allowlist as it stands', () => { + const r = runScript(SCRIPT) + expect(r.exitCode).toBe(0) + }) +}) diff --git a/scripts/lint-no-dead-package-paths.mjs b/scripts/lint-no-dead-package-paths.mjs index 47f7497a9..9fe1a4984 100644 --- a/scripts/lint-no-dead-package-paths.mjs +++ b/scripts/lint-no-dead-package-paths.mjs @@ -37,8 +37,9 @@ const SKIP_DIRS = new Set([ 'plans', 'superpowers', '.git', - // Fixtures for this linter's own self-tests deliberately reference deleted - // packages; scanning them would make the suite unrunnable. + // This linter's own self-tests deliberately reference deleted packages — + // in the fixtures, and in the prose comments of the test files themselves. + // Scanning them would make the suite unrunnable. '__tests__', ]) const SKIP_FILES = new Set(['CHANGELOG.md']) @@ -58,7 +59,7 @@ const TEXT_EXT = /\.(md|ya?ml|json|mjs|ts|txt)$/ // (#772 review, finding 15). const PACKAGE_REF = /packages\/([a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?)/g -// Live packages come from what git TRACKS, not from what is on disk. +// Live packages come from git, not from what is on disk. // // `readdirSync` was wrong in the direction that matters: deleting a package // leaves its `dist/` and `node_modules/` behind, so the directory still exists @@ -71,18 +72,61 @@ const PACKAGE_REF = /packages\/([a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?)/g // Note this deliberately does NOT require a `package.json`: `packages/utils` has // none (it is two loose files consumed by relative path from `packages/nextjs`) // yet is tracked, live, and referenced from AGENTS.md. +// +// Shelling out to git is a dependency this linter has to own: git missing, or a +// tree with no `.git`, must not read as "every package is dead". +function gitPackagePaths(...args) { + try { + return execFileSync('git', args, { + cwd: REPO_ROOT, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + }) + .split('\0') + .filter(Boolean) + } catch (err) { + const detail = String(err.stderr || err.message || '').trim() + console.error( + `Could not list packages via \`git ${args.join(' ')}\`:\n\n ${detail}\n\n` + + 'This linter derives the live package set from git, so it cannot run\n' + + 'without git on PATH or outside a git checkout.', + ) + // Exit 2, not 1: the linter failed to run. Exit 1 means it ran and found + // dead references, which is a different thing to go and fix. + process.exit(2) + } +} + const livePackages = new Set( - execFileSync('git', ['ls-files', '-z', 'packages'], { - cwd: REPO_ROOT, - encoding: 'utf8', - maxBuffer: 64 * 1024 * 1024, - }) - .split('\0') - .filter(Boolean) + [ + ...gitPackagePaths('ls-files', '-z', 'packages'), + // Untracked but not ignored. A package scaffolded a minute ago is live + // even though nothing about it is staged yet, and reporting it as "does + // not exist" is the sentence-final false alarm all over again, pointed the + // other way. `--directory` is deliberately NOT passed: it collapses an + // all-ignored directory to a single entry, which would resurrect exactly + // the `dist/`-and-`node_modules/` shells this linter exists to catch. + ...gitPackagePaths( + 'ls-files', + '--others', + '--exclude-standard', + '-z', + 'packages', + ), + ] .map((file) => file.split('/')[1]) .filter(Boolean), ) +if (livePackages.size === 0) { + console.error( + 'git reported no packages at all under `packages/`. Refusing to run —\n' + + 'every reference would be flagged. Check that `packages/` is present and\n' + + 'not wholly ignored.', + ) + process.exit(2) +} + function* walk(abs) { const stat = statSync(abs) if (stat.isFile()) { diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index 29c453fe8..95b216570 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -9,7 +9,6 @@ const REPO_ROOT = resolve(import.meta.dirname, '..') const ALLOWLISTED_PATHS = new Set([ 'packages/wizard/src/lib/detect.ts', // npm row of the PM table 'packages/cli/src/commands/init/utils.ts', // runnerCommand `case 'npm'` - 'packages/cli/src/commands/init/lib/setup-prompt.ts', // execCommand `case 'npm':` switch 'scripts/lint-no-hardcoded-runners.mjs', // this script's own docs ]) @@ -68,6 +67,52 @@ function isAllowedRunnerSwitch(line) { return /\bcase\s+['"]npm['"]/.test(line) || /name:\s*['"]npm['"]/.test(line) } +// An allowlist entry is a standing exemption, so it has to keep earning its +// place. Two ways it rots: the file is deleted (which is how the legacy Drizzle +// package's `src/bin/runner.ts` entry outlived its package by two months), or +// the file survives but the `npx` literal it was excused for moves elsewhere — +// leaving an entry that reads as evidence the exemption is still needed. Check +// both before scanning anything. +const staleAllowlist = [] +for (const rel of ALLOWLISTED_PATHS) { + let source + try { + source = readFileSync(resolve(REPO_ROOT, rel), 'utf8') + } catch { + staleAllowlist.push(`${rel}: no such file`) + continue + } + const stillNeeded = source + .split('\n') + .some( + (line) => + NPX_TOKEN.test(line) && + !isCommentLine(line) && + !isAllowedFallback(line) && + !isAllowedRunnerSwitch(line), + ) + if (!stillNeeded) { + staleAllowlist.push( + `${rel}: no longer contains an unexcused \`npx\` literal`, + ) + } +} + +if (staleAllowlist.length > 0) { + console.error( + `Found ${staleAllowlist.length} stale allowlist entr(ies) in this script:\n`, + ) + for (const s of staleAllowlist) console.error(` ${s}`) + console.error( + '\nDrop the entry. An exemption for a file that no longer exists, or that\n' + + 'no longer contains the literal it was excused for, is dead weight that\n' + + 'reads as deliberate.', + ) + // Exit 2, not 1: the linter's own configuration is wrong, which is a + // different thing to fix than an `npx` literal in the codebase. + process.exit(2) +} + const offenders = [] for (const target of TARGETS) { const abs = resolve(REPO_ROOT, target) From 0344e27ef000134ca03fa7750a3c217349486e6e Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 15:24:39 +1000 Subject: [PATCH 088/123] fix(cli,scripts): close the review nits, and the same defect where it repeats MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remediation for the two review rounds on #789, plus what verifying them turned up in the same code. **The linter fix the review asked for twice.** `livePackages` moving from `readdirSync` to `git ls-files` was the highest-value change in the PR and had no test that pinned it. It still doesn't fail under the revert anyone would actually write: keeping the git call and unioning the filesystem back in passes every existing test while fully restoring the false negative. The new fixture builds the discriminating case — a package deleted from git whose gitignored `dist/` and `node_modules/` shells survive on disk — and is the only test that fails under both a naive `readdirSync` revert and that hybrid. The `git status` guard is scoped to the probe, not to `packages/`, so an unrelated uncommitted file doesn't turn it into an assertion about nothing. **A missing target no longer passes in silence.** A linter whose entire job is catching dead paths in configuration skipped dead paths in its own: rename a target and it dropped out of coverage forever, green. Its sibling already exits 2 for a stale allowlist entry. Both now do, and both report a target outside the repo by name instead of as a `../../../../..` chain climbing out of the root — the review's cosmetic nit, which turned out to sit in both files. **Every step that names an integration-specific API now branches.** Step 5 was fixed for this; steps 1, 2 and the read-path step were not. A Prisma Next project was sent at `types.*` / `encryptedTable` — the client `stash schema build` explicitly refuses to scaffold for it — and a plain-Postgres project was pointed three times at "the integration skill" it never gets installed. `encryptQuery` is shown taking the schema objects rather than an object-shorthand that read as three required strings. `queryOperatorGuidance` is a switch with a neutral default rather than an if-chain ending in Drizzle's answer: tsup transpiles without type-checking and this package has no typecheck script, so nothing would have caught a fifth integration inheriting it. One test asserted nothing: it scoped to a `#### Encryption cutover` heading that does not exist, and `substring(-1)` returns the whole document. --- .changeset/decrypt-chaining-docs.md | 30 ++++-- .../init/lib/__tests__/setup-prompt.test.ts | 79 +++++++++++++++- .../cli/src/commands/init/lib/setup-prompt.ts | 92 ++++++++++++++++--- .../lint-no-dead-package-paths/dead-shell.md | 4 + .../lint-no-dead-package-paths.test.mjs | 83 ++++++++++++++++- .../lint-no-hardcoded-runners.test.mjs | 31 ++++++- scripts/lint-no-dead-package-paths.mjs | 28 +++++- scripts/lint-no-hardcoded-runners.mjs | 22 ++++- 8 files changed, 338 insertions(+), 31 deletions(-) create mode 100644 scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-shell.md diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md index 3e81a244d..5345a1e9c 100644 --- a/.changeset/decrypt-chaining-docs.md +++ b/.changeset/decrypt-chaining-docs.md @@ -23,9 +23,27 @@ with no lock-context argument. Also fixes the setup prompt `stash init` writes for coding agents, which referenced `protectOps.eq` — an API that does not exist anywhere in the repo. -The step now names the query API each integration can actually import: -`createEncryptionOperators(client)` (conventionally `ops`) for Drizzle, the -`encryptedSupabase` wrapper's own filters for Supabase, the `eql*` column -operators for Prisma Next, and `client.encryptQuery(...)` for a plain Postgres -project — which is also pointed at `stash-encryption`, since it is installed -with no integration skill. +Every step naming an integration-specific API now branches on the project's +actual integration, instead of naming Drizzle's and Supabase's and leaving the +other two to guess: + +- **Query paths.** `createEncryptionOperators(client)` (conventionally `ops`) + for Drizzle, the `encryptedSupabase` wrapper's own filters for Supabase, the + `eql*` column operators for Prisma Next, and `client.encryptQuery(...)` for a + plain Postgres project. +- **Schema authoring.** The `types.*` column factories for Drizzle, the + `eql_v3_encrypted` domain in migration SQL for Supabase, the `cipherstash.*` + field constructors in `schema.prisma` for Prisma Next, and `encryptedTable` + for plain Postgres. Prisma Next was previously sent at `types.*` / + `encryptedTable` — the client `stash schema build` explicitly refuses to + scaffold for that integration. +- **Read paths.** `decryptModel(row, usersSchema)` where that applies, and the + wrapper's transparent decryption where it does not. +- **Skill pointers.** A plain Postgres project installs no integration-specific + skill, so each "see the integration skill" was a pointer at a file that was + never written. Those now point at `stash-encryption`, which it does get. + +`client.encryptQuery` is also shown taking the schema objects themselves +(`{ table: usersSchema, column: usersSchema.email }`) rather than an +object-shorthand that read as three required strings — `queryType` is inferred +from the column's configured indexes. diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 4a76304cc..8ea972d9f 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -142,6 +142,78 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { expect(out).toContain('encryptQuery') expect(out).toContain('stash-encryption') }) + + // `EncryptQueryOptions` takes the schema OBJECTS — `usersSchema` and + // `usersSchema.email` — not their names as strings, and `queryType` is + // inferred from the column's indexes when omitted. The object-shorthand + // `{ table, column, queryType }` reads as three required string fields: + // the same species of plausible-but-wrong API as the `protectOps.eq` this + // PR exists to have removed. + it('shows encryptQuery taking schema objects, not string names', () => { + const out = render('postgresql') + expect(out).toContain('column: usersSchema.email') + expect(out).not.toContain('{ table, column, queryType }') + }) + + // `packages/cli` builds with tsup, which transpiles without type-checking, + // and has no `typecheck` script — so `Record` buys nothing + // at build time here. An if-chain ending in a bare Drizzle `return` hands a + // future fifth integration the exact string this helper exists to stop it + // getting. Degrade to neutral guidance, as `skillsFor()` degrades to + // BASE_SKILLS. + it('degrades to neutral guidance for an unrecognised integration', () => { + const out = renderSetupPrompt({ + ...baseCtx, + integration: 'mystery-orm' as SetupPromptContext['integration'], + }) + expect(out).not.toContain('createEncryptionOperators') + expect(out).not.toContain('encryptedSupabase') + expect(out).toContain('encryptQuery') + }) + }) + + // Step 2 named the Drizzle and Supabase schema APIs unconditionally — the + // identical defect step 5 above was just fixed for. A prisma-next project was + // sent at `types.*` / `encryptedTable`, which `stash schema build` explicitly + // refuses to scaffold for that integration; a plain-Postgres project was + // given no named path at all. + describe('schema-authoring guidance is per-integration', () => { + const render = (integration: SetupPromptContext['integration']) => + renderSetupPrompt({ ...baseCtx, integration }) + + it('names the Drizzle column factories for drizzle', () => { + const out = render('drizzle') + expect(out).toContain('`types.*` column factories') + expect(out).toContain('@cipherstash/stack-drizzle') + }) + + it('names the eql_v3 domain for supabase', () => { + expect(render('supabase')).toContain('eql_v3_encrypted') + }) + + it('names the cipherstash.* field constructors for prisma-next', () => { + const out = render('prisma-next') + expect(out).toContain('cipherstash.TextSearch()') + expect(out).toContain('prisma/schema.prisma') + // The `types.*` client is precisely what `stash schema build` refuses to + // emit for prisma-next. + expect(out).not.toContain('`types.*` column factories') + }) + + it('names encryptedTable for plain postgresql', () => { + const out = render('postgresql') + expect(out).toContain('encryptedTable') + expect(out).toContain('@cipherstash/stack/v3') + }) + }) + + // `postgresql` installs no integration-specific skill (SKILL_MAP), so every + // unconditional "see the integration skill" is a pointer at a file that was + // never written. Step 5's was fixed; two more survived elsewhere. + it('never points plain postgresql at "the integration skill"', () => { + const out = renderSetupPrompt({ ...baseCtx, integration: 'postgresql' }) + expect(out).not.toMatch(/the integration skill/i) + expect(out).toContain('stash-encryption') }) it('emits supabase migration commands for supabase integration', () => { @@ -477,7 +549,12 @@ describe('renderSetupPrompt — no db push recommendations', () => { expect(out).toMatch(/1\.\s*\*\*Schema-add/) expect(out).toMatch(/2\.\s*\*\*Dual-write/) // Cutover is still covered, just without a db push workaround note. - const cutoverSection = out.substring(out.indexOf('#### Encryption cutover')) + // The heading has to be one that exists: `indexOf` returning -1 makes + // `substring(-1)` the whole document, so this scoped assertion was + // silently asserting nothing at all. + const heading = '#### Backfill and switch' + expect(out).toContain(heading) + const cutoverSection = out.substring(out.indexOf(heading)) expect(cutoverSection).toMatch(/encrypt cutover/) }) diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 5a22d42a6..03d9a7e3b 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -68,6 +68,46 @@ function migrationCommands( return undefined } +/** + * Where this integration's rules were actually written. + * + * `postgresql` is the fallback integration and installs no integration-specific + * skill (see `SKILL_MAP` in `install-skills.ts`), so "the integration skill" is + * a pointer at a file that was never created. Point it at `stash-encryption`, + * which it does get. + */ +function integrationSkillRef(integration: Integration): string { + switch (integration) { + case 'drizzle': + case 'supabase': + case 'prisma-next': + return 'the integration skill' + default: + return 'the `stash-encryption` skill' + } +} + +/** + * How this integration declares an encrypted column in the schema. + * + * Per-integration for the same reason the query operators are: the APIs are not + * interchangeable, and `stash schema build` (`utils.ts`) refuses outright to + * scaffold a `types.*` client for prisma-next — that integration authors its + * columns with `cipherstash.*` constructors in `schema.prisma` instead. + */ +function schemaAuthoringGuidance(integration: Integration): string { + switch (integration) { + case 'drizzle': + return 'Declare it with the `types.*` column factories from `@cipherstash/stack-drizzle` on the existing `pgTable`' + case 'supabase': + return 'Declare the column with the `eql_v3_encrypted` domain in the migration SQL (`encryptedSupabase` derives its encryption config by introspecting those domains)' + case 'prisma-next': + return 'Declare the field with the `cipherstash.*` constructors in `prisma/schema.prisma` (`cipherstash.TextSearch()`, `cipherstash.DoubleOrd()`, …), with the `cipherstash` extension pack wired up per `@cipherstash/prisma-next/control`' + default: + return 'Declare the table with `encryptedTable` and the `types.*` domain factories from `@cipherstash/stack/v3`, then pass it to `Encryption({ schemas })`' + } +} + /** * How this integration filters on an encrypted column. Named per-integration * rather than generically because the APIs are not interchangeable and only one @@ -75,21 +115,45 @@ function migrationCommands( * exported by `@cipherstash/stack-drizzle` alone, so naming it for a Supabase * or Prisma project sends the agent after a package that is not installed. * - * `postgresql` is the fallback integration and is installed with no integration - * skill (see `SKILL_MAP` in `install-skills.ts`), so it is pointed at - * `stash-encryption` instead of "the integration skill". + * A `switch` with a neutral `default`, not an if-chain ending in the Drizzle + * string: `packages/cli` is built by tsup, which transpiles without + * type-checking, and the package has no `typecheck` script — so nothing would + * catch a fifth `Integration` variant silently inheriting Drizzle's answer. + * `skillsFor()` in `install-skills.ts` degrades the same way, for the same + * reason. */ function queryOperatorGuidance(integration: Integration): string { - if (integration === 'supabase') { - return 'query paths filter through the `encryptedSupabase` wrapper (`es.from("users").select(...).eq("email", value)`) — it encrypts filter operands for encrypted columns automatically; see the integration skill' + switch (integration) { + case 'supabase': + return 'query paths filter through the `encryptedSupabase` wrapper (`es.from("users").select(...).eq("email", value)`) — it encrypts filter operands for encrypted columns automatically; see the integration skill' + case 'prisma-next': + return 'query paths use the `eql*` operators on the column inside `.where()` (`u.email.eqlEq(value)`, `eqlMatch`, `eqlGt`, …) — see the integration skill' + case 'drizzle': + return 'query paths use the right operator (`ops.eq`, from `createEncryptionOperators(client)`) — see the integration skill' + default: + // `table` and `column` are the schema OBJECTS, not their names as + // strings, and `queryType` is inferred from the column's configured + // indexes unless it is passed to override the inference. + return 'query paths encrypt the search term first — `client.encryptQuery(value, { table: usersSchema, column: usersSchema.email })`, passing the schema objects themselves rather than their names — and compare that against the encrypted column; see the `stash-encryption` skill (a plain Postgres project gets no integration-specific skill)' } - if (integration === 'prisma-next') { - return 'query paths use the `eql*` operators on the column inside `.where()` (`u.email.eqlEq(value)`, `eqlMatch`, `eqlGt`, …) — see the integration skill' - } - if (integration === 'postgresql') { - return 'query paths encrypt the search term first with `client.encryptQuery(value, { table, column, queryType })` and compare against the encrypted column — see the `stash-encryption` skill (a plain Postgres project gets no integration skill)' +} + +/** + * How this integration turns ciphertext back into values on the read path. + * + * Named per-integration for the third time in this file, and for the third + * time because the answer is not portable: `decryptModel` is the typed + * client's, transparent decryption is the Supabase wrapper's. + */ +function readPathGuidance(integration: Integration): string { + switch (integration) { + case 'supabase': + return 'selects through the `encryptedSupabase` wrapper decrypt transparently' + case 'prisma-next': + return 'the encrypted fields decrypt through the Prisma Next client' + default: + return 'call `decryptModel(row, usersSchema)` — or `bulkDecryptModels` for a set — before returning the value to callers' } - return 'query paths use the right operator (`ops.eq`, from `createEncryptionOperators(client)`) — see the integration skill' } function bullet(line: string): string { @@ -298,10 +362,10 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '', '### Add a new encrypted column', '', - 'Use when the column **does not yet exist** in the database (no plaintext predecessor to preserve). This is normal Drizzle / Supabase work plus the encryption client patterns from the integration skill.', + `Use when the column **does not yet exist** in the database (no plaintext predecessor to preserve). This is normal schema work in the project's own ORM or migration tooling, plus the encryption client patterns from ${integrationSkillRef(ctx.integration)}.`, '', "1. **If this is the first encrypted column in the project, configure the bundler exclusion first.** `@cipherstash/stack` cannot be bundled (it wraps a native FFI module). Next.js: add `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` to `next.config.*`. Webpack: `externals`. esbuild: `external`. Vite SSR: `ssr.external`. Without this, the encryption client crashes at runtime with `Cannot find module '@cipherstash/protect-ffi-*'`. See the `stash-encryption` skill's Installation section for the full snippets.", - "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories from `@cipherstash/stack-drizzle` for Drizzle, and the `types.*` factories from `@cipherstash/stack/eql/v3` (via `encryptedTable`, passed as `schemas`) for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", + `2. Edit the user's real schema file (\`src/db/schema.ts\` or wherever they keep it) to declare the new encrypted column. ${schemaAuthoringGuidance(ctx.integration)} — the patterns are in ${integrationSkillRef(ctx.integration)}. Encrypted columns must be **nullable \`jsonb\`** at creation time (the \`eql_v3_*\` domains are over \`jsonb\`). Never \`.notNull()\`.`, `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, `5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, ${queryOperatorGuidance(ctx.integration)}.`, @@ -328,7 +392,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '', `3. **Backfill.** Run \`${cli} encrypt backfill --table --column \`. The CLI prompts the user (or accepts \`--confirm-dual-writes-deployed\` non-interactively) to confirm dual-writes are live, then chunks through the existing rows. Resumable; checkpoints to \`cs_migrations\` after every chunk. SIGINT-safe.`, `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`). Do **not** declare a v2 column with a \`types.*\` domain — those are EQL v3 only. The adapters no longer author v2 (\`@cipherstash/stack-drizzle\` removed \`encryptedType\`), so a v2 column is a read path: declare it with the deprecated \`@cipherstash/stack/schema\` builders and decrypt through \`@cipherstash/stack\`.`, - '5. **Wire the read path through the encryption client.** The read column now holds ciphertext. Read code paths must decrypt before returning the value to callers — `decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, or the equivalent `decrypt`/`bulkDecryptModels` calls. Without this step, your read paths return raw encrypted payloads to end users. The integration skill has the exact API.', + `5. **Wire the read path through the encryption client.** The read column now holds ciphertext — ${readPathGuidance(ctx.integration)}. Without this step, your read paths return raw encrypted payloads to end users. See ${integrationSkillRef(ctx.integration)} for the exact API.`, '6. **Remove the dual-write code.** The plaintext column (still `` on v3; renamed `_plaintext` on v2) is no longer authoritative. Delete the dual-write logic from the persistence layer.', `7. **Drop.** Run \`${cli} encrypt drop --table --column \`. Generates a migration that removes the now-unused plaintext column (on v3 it first verifies no rows are still plaintext-only). Apply with the project's normal migration tooling.`, '', diff --git a/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-shell.md b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-shell.md new file mode 100644 index 000000000..9b25c6550 --- /dev/null +++ b/scripts/__tests__/fixtures/lint-no-dead-package-paths/dead-shell.md @@ -0,0 +1,4 @@ +# A deleted package whose build output survives on disk + +The old thing lived in packages/lint-dead-shell-probe, deleted from git but +still sitting there as a `dist/` shell on any checkout that once built it. diff --git a/scripts/__tests__/lint-no-dead-package-paths.test.mjs b/scripts/__tests__/lint-no-dead-package-paths.test.mjs index 89e855dd6..92f9d7aff 100644 --- a/scripts/__tests__/lint-no-dead-package-paths.test.mjs +++ b/scripts/__tests__/lint-no-dead-package-paths.test.mjs @@ -1,6 +1,7 @@ import { execFileSync } from 'node:child_process' -import { mkdirSync, rmSync, writeFileSync } from 'node:fs' -import { resolve } from 'node:path' +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { describe, expect, it } from 'vitest' @@ -114,6 +115,49 @@ describe('lint-no-dead-package-paths', () => { } }) + // The whole reason `livePackages` comes from git rather than from + // `readdirSync`, pinned in the discriminating direction (#772 review, + // finding 15). + // + // Deleting a package leaves its gitignored `dist/` and `node_modules/` + // behind, so the directory is still on disk and any filesystem-derived live + // set calls it live — silently excusing every reference to it. The test + // above, `treats a package that exists but is not yet tracked as live`, does + // NOT pin this: it builds a divergence that `readdirSync` and git agree on, + // so it passes under a revert. This one is the only test that fails under + // both a plain `readdirSync` revert and the more plausible + // `readdirSync`-unioned-with-git hybrid. + // + // The shell's contents MUST be gitignored. If they aren't, the + // `--others --exclude-standard` arm legitimately counts the package as live + // and this test would assert the opposite of what it claims — so guard on a + // clean `git status` and fail loudly rather than silently degrading. + it('flags a deleted package whose gitignored dist/ shell survives on disk', () => { + const shell = resolve(REPO_ROOT, 'packages/lint-dead-shell-probe') + try { + mkdirSync(resolve(shell, 'dist'), { recursive: true }) + writeFileSync(resolve(shell, 'dist/index.js'), 'exports.x = 1\n') + mkdirSync(resolve(shell, 'node_modules'), { recursive: true }) + writeFileSync(resolve(shell, 'node_modules/.keep'), '') + + // Scoped to the probe alone: asserting the whole `packages/` tree is + // clean would couple this to whatever else is uncommitted in the + // working copy, which is nobody's business but the developer's. + const visible = execFileSync( + 'git', + ['status', '--porcelain', 'packages/lint-dead-shell-probe'], + { cwd: REPO_ROOT, encoding: 'utf8' }, + ) + expect(visible).toBe('') + + const r = run(fx('dead-shell.md')) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/lint-dead-shell-probe/) + } finally { + rmSync(shell, { recursive: true, force: true }) + } + }) + // The live set is derived by shelling out to git, so git failing is a mode // this linter has to own. Exiting 1 with a raw ENOENT stack trace would be // indistinguishable from a genuine lint failure; exit 2 says "the linter @@ -128,6 +172,41 @@ describe('lint-no-dead-package-paths', () => { expect(r.output).not.toMatch(/at ModuleJob/) }) + // A linter whose whole job is catching dead paths in configuration silently + // ignored dead paths in its OWN configuration: a target that did not exist + // was skipped, so a renamed or moved entry dropped out of coverage forever + // with a green build. Its sibling `lint-no-hardcoded-runners.mjs` exits 2 on + // a stale allowlist entry — same rot, opposite handling, same PR. + // + // Exit 2, not 1: the linter could not check what it was asked to check, + // which is not the same as "your docs are wrong". + it('exits 2 when a target does not exist rather than skipping it', () => { + const r = run(fx('no-such-fixture.md')) + expect(r.exitCode).toBe(2) + expect(r.output).toMatch(/no-such-fixture\.md/) + expect(r.output).not.toMatch(/at ModuleJob/) + }) + + // `relative(REPO_ROOT, file)` renders a target outside the repo as a + // `../../../../..` chain climbing out of the root — technically resolvable, + // unreadable in practice, and it buries the one thing the line is for. + // Unreachable via the default target list, which is repo-relative + // throughout; reachable the moment anyone passes an argv override. + it('renders an absolute path for an offender outside the repo root', () => { + const dir = mkdtempSync(join(tmpdir(), 'lint-dead-package-paths-')) + try { + const file = join(dir, 'outside.md') + writeFileSync(file, 'Points at packages/lint-dead-shell-probe, gone.\n') + const r = run(file) + expect(r.exitCode).toBe(1) + expect(r.output).toMatch(/packages\/lint-dead-shell-probe/) + expect(r.output).toContain(file) + expect(r.output).not.toMatch(/\.\.\//) + } finally { + rmSync(dir, { recursive: true, force: true }) + } + }) + it('names the file and line of each offender', () => { const r = run(fx('dead-ref.md')) expect(r.output).toMatch(/dead-ref\.md:3/) diff --git a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs index 96531ea0e..709ac63ed 100644 --- a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs +++ b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs @@ -1,6 +1,7 @@ import { execFileSync } from 'node:child_process' -import { readFileSync, rmSync, writeFileSync } from 'node:fs' -import { resolve } from 'node:path' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { describe, expect, it } from 'vitest' @@ -40,6 +41,32 @@ describe('lint-no-hardcoded-runners', () => { expect(r.output).toMatch(/\bnpx\b/) }) + // `statSync` on a missing target threw uncaught: exit 1 plus a raw stack + // trace, indistinguishable from "found a hardcoded npx" to anything reading + // the exit code. Exit 2 means the linter could not run. + it('exits 2 when a target does not exist', () => { + const r = run(fx('no-such-fixture.ts')) + expect(r.exitCode).toBe(2) + expect(r.output).toContain('no-such-fixture.ts') + expect(r.output).not.toMatch(/at ModuleJob/) + }) + + // Matches the sibling linter: a target outside the repo rendered as a + // `../../../../..` chain out of the root instead of naming the file. + it('renders an absolute path for an offender outside the repo root', () => { + const dir = mkdtempSync(join(tmpdir(), 'lint-hardcoded-runners-')) + try { + const file = join(dir, 'outside.ts') + writeFileSync(file, "export const cmd = 'npx drizzle-kit generate'\n") + const r = run(file) + expect(r.exitCode).toBe(1) + expect(r.output).toContain(file) + expect(r.output).not.toMatch(/\.\.\//) + } finally { + rmSync(dir, { recursive: true, force: true }) + } + }) + it("ignores `?? 'npx'` fallback expressions", () => { expect(run(fx('allowed-fallback.ts')).exitCode).toBe(0) }) diff --git a/scripts/lint-no-dead-package-paths.mjs b/scripts/lint-no-dead-package-paths.mjs index 9fe1a4984..321b30e04 100644 --- a/scripts/lint-no-dead-package-paths.mjs +++ b/scripts/lint-no-dead-package-paths.mjs @@ -152,19 +152,39 @@ for (const target of TARGETS) { } catch { exists = false } - // A default target that doesn't exist is not an error — the list covers - // optional files (CONTRIBUTE.md, per-package AGENTS.md). - if (!exists) continue + // A target that isn't there used to be skipped in silence, on the theory + // that the default list covered optional files. It doesn't — all of TARGETS + // is tracked and present — and the silence was the same rot this linter + // exists to catch, one level up: rename a target and it drops out of + // coverage forever with a green build. `lint-no-hardcoded-runners.mjs` + // already exits 2 on a stale allowlist entry; this is that rule applied to + // this linter's own configuration. + if (!exists) { + console.error( + `Target \`${target}\` does not exist.\n\n` + + 'Either it was renamed or removed — in which case update TARGETS in\n' + + 'this script, since a target that is silently skipped is coverage\n' + + 'quietly lost — or it was mistyped on the command line.', + ) + // Exit 2, not 1: the linter could not check what it was asked to check. + process.exit(2) + } for (const file of walk(abs)) { + // A target outside the repo (only reachable via an argv override — every + // default is repo-relative) renders as a `../../../../..` chain climbing + // out of the root, which buries the filename it exists to point at. const rel = relative(REPO_ROOT, file) + const shown = rel.startsWith('..') ? file : rel if (SKIP_FILES.has(rel.split('/').pop())) continue const lines = readFileSync(file, 'utf8').split('\n') lines.forEach((line, idx) => { PACKAGE_REF.lastIndex = 0 for (const m of line.matchAll(PACKAGE_REF)) { if (livePackages.has(m[1])) continue - offenders.push(`${rel}:${idx + 1}: \`packages/${m[1]}\` does not exist`) + offenders.push( + `${shown}:${idx + 1}: \`packages/${m[1]}\` does not exist`, + ) } }) } diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index 95b216570..3070963c8 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -116,10 +116,28 @@ if (staleAllowlist.length > 0) { const offenders = [] for (const target of TARGETS) { const abs = resolve(REPO_ROOT, target) - const stat = statSync(abs) + let stat + try { + stat = statSync(abs) + } catch { + // Was an uncaught throw: exit 1 plus a raw stack trace, which reads to + // anything checking the exit code exactly like a genuine `npx` finding. + // Exit 2 says the linter could not run — the same contract the allowlist + // self-check above already uses. + console.error( + `Target \`${target}\` does not exist.\n\n` + + 'Update the scan roots in this script if it was renamed or removed,\n' + + 'or check the path passed on the command line.', + ) + process.exit(2) + } const files = stat.isDirectory() ? walk(abs) : [abs] for await (const file of files) { + // `rel` stays repo-relative for the allowlist lookup; only the rendering + // falls back to the absolute path, for a target outside the repo root that + // would otherwise print a `../../../../..` chain instead of a filename. const rel = relative(REPO_ROOT, file) + const shown = rel.startsWith('..') ? file : rel if (ALLOWLISTED_PATHS.has(rel)) continue if (/\.(test|spec)\.(ts|tsx|mts|cts)$/.test(file)) continue const lines = readFileSync(file, 'utf8').split('\n') @@ -129,7 +147,7 @@ for (const target of TARGETS) { if (isCommentLine(line)) return if (isAllowedFallback(line)) return if (isAllowedRunnerSwitch(line)) return - offenders.push(`${rel}:${idx + 1}: ${line.trim()}`) + offenders.push(`${shown}:${idx + 1}: ${line.trim()}`) }) } } From dc6dab6c37dd551fca1c4c9ef91343555ed8a25e Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 17:09:47 +1000 Subject: [PATCH 089/123] fix(cli): stop the plan templates drafting a v2-only cutover on a v3 default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `stash plan` writes one of three templates. Two of them — the cutover plan and the `--complete-rollout` plan — described the EQL v2 rename swap as the only way to switch reads: "a single transaction renames `` → `_plaintext` and `_encrypted` → ``", with the steps after it referring to `_plaintext` as though the rename had certainly happened. On EQL v3, which is the default, none of that is true. `stash encrypt cutover` refuses a v3 column outright — "Cut-over is not applicable to EQL v3 columns … there is no rename step" (`encrypt/cutover.ts:119`) — and refuses entirely on a v3-only database, where the `eql_v2_configuration` relation it needs does not exist (`:142`). So on a default install the agent was being asked to draft a plan around a command that will decline to run, and to name a `_plaintext` column that will never exist. The implement prompt in this same file already splits the two correctly. This copies that split into the templates that were missing it. The EQL version is per-column — one database can hold both — so the templates tell the agent to establish it per column (`stash encrypt status`) rather than deciding once for the whole plan; a context-level flag would have been wrong for a mixed database. Also: the "this column is already encrypted" stop-and-ask keyed on the `eql_v2_encrypted` udt alone. v3 columns carry `eql_v3_*` domains, so on the default path the check could never fire. The agent reads the schema itself, so naming both is sufficient here. `introspect.ts:88` has the same v2-only assumption in code (`isEqlEncrypted: row.udt_name === 'eql_v2_encrypted'`). Left alone deliberately: it decides which columns `stash init` pre-selects as already-managed, so widening it changes setup behaviour and wants its own tests rather than riding along here. --- .changeset/decrypt-chaining-docs.md | 14 +++++ .../init/lib/__tests__/setup-prompt.test.ts | 60 +++++++++++++++++++ .../cli/src/commands/init/lib/setup-prompt.ts | 25 ++++---- 3 files changed, 85 insertions(+), 14 deletions(-) diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md index 5345a1e9c..e955fd495 100644 --- a/.changeset/decrypt-chaining-docs.md +++ b/.changeset/decrypt-chaining-docs.md @@ -47,3 +47,17 @@ other two to guess: (`{ table: usersSchema, column: usersSchema.email }`) rather than an object-shorthand that read as three required strings — `queryType` is inferred from the column's configured indexes. + +The cutover and complete-rollout **plan templates** now split EQL v3 from v2. +Both described the v2 rename swap (`` → `_plaintext`, twin → ``) +as the only cutover path, so on the default v3 install `stash plan` drafted a +plan built around `stash encrypt cutover` — a command that refuses v3 columns +outright ("there is no rename step") and refuses entirely on a v3-only +database, where `eql_v2_configuration` does not exist. The implement prompt +already carried this split; the plan templates did not. The version is +per-column, so the templates tell the agent to establish it per column rather +than deciding once for the whole plan. + +The "already encrypted" stop-and-ask now recognises `eql_v3_*` domains +alongside the legacy `eql_v2_encrypted` udt, so it can fire on the default +path at all. diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 8ea972d9f..ee1c302b7 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -566,6 +566,66 @@ describe('renderSetupPrompt — no db push recommendations', () => { }) }) +// The rename swap is an EQL v2 operation. `stash encrypt cutover` refuses a v3 +// column outright ("Cut-over is not applicable to EQL v3 columns … there is no +// rename step", `encrypt/cutover.ts`) and refuses entirely on a v3-only +// install, where `eql_v2_configuration` does not exist. v3 is the default, so +// a template presenting the rename as the only path drafts a plan around a +// command that will decline to run. The implement prompt already splits the +// two; these templates did not. +describe('renderSetupPrompt — plan templates split EQL v3 from v2', () => { + const plan = (planStep: 'cutover' | 'complete') => + renderSetupPrompt({ ...baseCtx, mode: 'plan', planStep }) + + for (const planStep of ['cutover', 'complete'] as const) { + describe(`${planStep} plan`, () => { + it('names both versions and states v3 has no rename', () => { + const out = plan(planStep) + expect(out).toContain('EQL v3') + expect(out).toContain('EQL v2') + expect(out).toMatch(/no rename/i) + }) + + it('does not present the rename swap unconditionally', () => { + const out = plan(planStep) + // Every mention of the rename swap has to sit next to the v2 label. + for (const line of out.split('\n')) { + if (/_plaintext/.test(line)) { + expect(line).toMatch(/v2|v3/) + } + } + }) + + it('tells the agent how to determine the version per column', () => { + // The version is per-column — a database can hold both — so this + // cannot be decided once for the whole plan. + expect(plan(planStep)).toMatch(/encrypt status|encrypt backfill/) + }) + }) + } + + // The stop-and-ask trigger for "already encrypted" named the v2 UDT alone. + // v3 is the default and its columns carry `eql_v3_*` domains, so on the + // default path the check silently never fired. The agent reads the schema + // itself, so this half is fixable here; `introspect.ts`'s `isEqlEncrypted` + // has the same gap and is a separate behavioural change. + it('recognises v3 domains in the already-encrypted stop-and-ask', () => { + for (const mode of ['implement', 'plan'] as const) { + const out = renderSetupPrompt({ ...baseCtx, mode }) + expect(out).toMatch(/eql_v3_/) + } + }) + + it('keeps the cutover invocation scoped to v2', () => { + const out = plan('cutover') + const cutoverLine = out + .split('\n') + .find((l) => l.includes('encrypt cutover --table')) + expect(cutoverLine).toBeDefined() + expect(cutoverLine).toMatch(/v2/) + }) +}) + describe('renderSetupPrompt — honours what the handoff actually wrote', () => { for (const mode of ['implement', 'plan'] as const) { it(`claude-code with no skills points at neither a skills dir nor AGENTS.md (${mode})`, () => { diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 03d9a7e3b..ac97a3687 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -414,7 +414,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { "The user asks to convert a populated column in place. Explain why it doesn't work and offer the migrate-existing-column flow instead.", ), bullet( - "A column the user names is already encrypted (`eql_v2_encrypted` udt) but with a different EQL config than they've described. This is the post-cutover re-encryption case (`stash encrypt update`, not yet shipped) — surface it instead of guessing.", + "A column the user names is already encrypted — an `eql_v3_*` domain (`eql_v3_text_search`, `eql_v3_integer_ord`, …) on the default path, or the legacy `eql_v2_encrypted` udt — but with a different EQL config than they've described. This is the post-cutover re-encryption case (`stash encrypt update`, not yet shipped) — surface it instead of guessing.", ), bullet( 'The schema migration would change the data type of a column the user has already filled.', @@ -516,7 +516,7 @@ function planSharedStopAndAsk(): string[] { "The user asks to convert a populated column in place. Explain why it doesn't work and offer the migrate-existing-column flow instead.", ), bullet( - "A column the user names is already encrypted (`eql_v2_encrypted` udt) but with a different EQL config than they've described. This is the post-cutover re-encryption case (`stash encrypt update`, not yet shipped) — surface it in the plan as a flagged risk.", + "A column the user names is already encrypted — an `eql_v3_*` domain (`eql_v3_text_search`, `eql_v3_integer_ord`, …) on the default path, or the legacy `eql_v2_encrypted` udt — but with a different EQL config than they've described. This is the post-cutover re-encryption case (`stash encrypt update`, not yet shipped) — surface it in the plan as a flagged risk.", ), bullet( 'You discover existing partial CipherStash setup that disagrees with what the user is describing — someone else may have run `stash init` earlier with different choices. Note this in the plan and ask the user to clarify before writing prescriptive steps.', @@ -644,19 +644,16 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { '**Backfill.** Encrypt the historical rows that pre-date the rollout deploy. Resumable; chunked; SIGINT-safe.', ), bullet( - '**Schema rename.** Update the schema declaration so the original column points at the encrypted type.', + `**Switch reads to the encrypted column.** This step depends on the column's EQL version, so establish it per column first — \`${cli} encrypt backfill\` prints it and \`${cli} encrypt status\` shows it. **EQL v3 (the default):** there is no rename. Update the schema declaration and the queries to read and write \`_encrypted\` under its own name. **EQL v2 (legacy data only):** declare the encrypted column under its final name (drop the twin suffix), then a single \`${cli} encrypt cutover\` transaction renames \`\` → \`_plaintext\` and \`_encrypted\` → \`\`.`, ), bullet( - '**Cutover.** A single transaction renames `` → `_plaintext` and `_encrypted` → ``.', + '**Read path.** Reads of the encrypted column return ciphertext until the read path decrypts via the encryption client. The plan must specify what changes per read site.', ), bullet( - '**Read path.** Application reads of `` now return ciphertext until the read path decrypts via the encryption client. The plan must specify what changes per read site.', + '**Remove dual-writes.** The plaintext column — still `` on v3, renamed `_plaintext` on v2 — is no longer authoritative. Delete the dual-write code paths.', ), bullet( - '**Remove dual-writes.** The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write code paths.', - ), - bullet( - "**Drop plaintext.** `stash encrypt drop` emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling.", + `**Drop plaintext.** \`${cli} encrypt drop\` emits a migration that removes the now-unused plaintext column; on v3 it first verifies no rows are still plaintext-only. Apply with the project's normal migration tooling.`, ), '', '## Your task: produce the cutover plan file', @@ -679,12 +676,12 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { ' encrypt backfill` invocation with concrete `--table` / `--column` values.', ), bullet( - 'The schema-edit step, with the exact rename pattern (drop `_encrypted` suffix on the encrypted column, switch the original column declaration off `text`/`varchar` and onto the encrypted type).', + 'The schema-edit step, stated per column against its EQL version. **v3:** point the declaration and queries at `_encrypted` under its own name — there is no rename, and no `_encrypted` suffix to drop. **v2:** drop the `_encrypted` suffix and switch the original column declaration off `text`/`varchar` and onto the encrypted type.', ), bullet( - 'The cutover invocation per column: `' + + 'For EQL v2 columns only, the cutover invocation per column: `' + cli + - ' encrypt cutover --table --column `.', + ' encrypt cutover --table --column `. It refuses v3 columns — there is nothing to rename — so do not schedule it for them.', ), bullet( 'Read-path code changes: every site that reads `` from this table must decrypt via the encryption client. Enumerate the sites you can find via grep so the user can verify nothing was missed.', @@ -696,7 +693,7 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { ' encrypt drop --table --column `, plus the schema-migration apply step that follows.', ), bullet( - `Risks specific to cutover: row-count for the backfill (use \`${cli} eql status\` to estimate if helpful), tables under heavy write load (cutover holds a brief lock on the rename), application code that constructs SQL by string (those reads won't transparently decrypt).`, + `Risks specific to cutover: row-count for the backfill (use \`${cli} eql status\` to estimate if helpful), tables under heavy write load (on v2, cutover holds a brief lock for the rename), application code that constructs SQL by string (those reads won't transparently decrypt).`, ), bullet( "Open questions for the user — anything you can't determine from the schema, context.json, or the skills.", @@ -741,7 +738,7 @@ function renderCompletePlanPrompt(ctx: SetupPromptContext): string { '**Add new encrypted columns** — declared encrypted from the start; single-deploy.', ), bullet( - '**Migrate existing columns** — schema-add → dual-write code → backfill → schema rename → cutover → read-path switch → remove dual-write code → drop plaintext. No deploy gate between rollout and cutover steps because there is no deployed application to gate on.', + `**Migrate existing columns** — schema-add → dual-write code → backfill → switch reads to the encrypted column → remove dual-write code → drop plaintext. The switch step depends on the column's EQL version (\`${cli} encrypt status\` shows it): on **EQL v3**, the default, there is no rename — point the schema and queries at \`_encrypted\` by name; on **EQL v2** only, \`${cli} encrypt cutover\` renames the twin into \`\` first. No deploy gate between rollout and cutover steps because there is no deployed application to gate on.`, ), '', '## Your task: produce the complete-rollout plan file', From ed2cc718fad5497841c907c3f0c2ea268c33043c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Mon, 27 Jul 2026 17:51:46 +1000 Subject: [PATCH 090/123] fix(cli): detect already-encrypted columns on EQL v3 during init MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `introspectDatabase` marked a column as CipherStash-managed only when its udt was exactly `eql_v2_encrypted`: isEqlEncrypted: row.udt_name === 'eql_v2_encrypted' v3 is the default generation and its columns carry per-domain types — `eql_v3_text_search`, `eql_v3_integer_ord`, and so on — so on the default path this was false for every encrypted column. The consequences are all in the column picker: the table hint never said "N already encrypted", the pre-selection notice never fired, and each encrypted column was displayed with its plaintext `dataType` and left unticked. An encrypted column presented as plaintext invites the user to encrypt it a second time, which is the direction of wrongness that costs data rather than time. `packages/wizard` already had the correct predicate, with the reasoning written down (`isEqlEncryptedDomain`, `wizard-tools.ts:167`). This mirrors it rather than inventing a second answer, and exports it so it is testable — the old inline comparison sat inside a function that needs a live pg connection, which is why nothing covered it. Deliberately not `classifyEqlDomain` from `@cipherstash/migrate`, despite the dependency already being present: that answers "which generation authors this" and returns null for `eql_v2_encrypted`, since v2 is no longer authorable. The question here is "is this encrypted at all", and v2 answers yes. The per-column hint was also the hardcoded string `eql_v2_encrypted` for any encrypted column, which mislabels a v3 column with a domain it does not have. It now reports the column's actual udt, and the pre-selection notice lists the domains it found instead of naming one it may not have seen. Checked the rest of the CLI for the same assumption: every other site either already handles both generations (`rewrite-migrations.ts` matches `eql_v2_encrypted|eql_v3_[a-z0-9_]+`, `db-readers.ts`, `backfill.ts`) or is legitimately v2-only (`cutover.ts`, the installer's CREATE TYPE permission messages). Introspection was the sole outlier. --- .changeset/decrypt-chaining-docs.md | 9 +++ .../init/lib/__tests__/introspect.test.ts | 71 +++++++++++++++++++ .../cli/src/commands/init/lib/introspect.ts | 38 ++++++++-- 3 files changed, 111 insertions(+), 7 deletions(-) diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md index e955fd495..19bcbb8ea 100644 --- a/.changeset/decrypt-chaining-docs.md +++ b/.changeset/decrypt-chaining-docs.md @@ -61,3 +61,12 @@ than deciding once for the whole plan. The "already encrypted" stop-and-ask now recognises `eql_v3_*` domains alongside the legacy `eql_v2_encrypted` udt, so it can fire on the default path at all. + +**`stash init` now detects already-encrypted columns on EQL v3.** Database +introspection marked a column as CipherStash-managed only when its udt was +exactly `eql_v2_encrypted`. v3 columns carry per-domain types +(`eql_v3_text_search`, `eql_v3_integer_ord`, …), so on the default path every +encrypted column was reported as plaintext — shown with its `dataType` and left +unticked in the column picker, inviting a re-run to encrypt it a second time. +The picker also labelled any encrypted column with the literal string +`eql_v2_encrypted`; it now shows the column's real domain. diff --git a/packages/cli/src/commands/init/lib/__tests__/introspect.test.ts b/packages/cli/src/commands/init/lib/__tests__/introspect.test.ts index 40a011e95..1efca50e3 100644 --- a/packages/cli/src/commands/init/lib/__tests__/introspect.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/introspect.test.ts @@ -5,6 +5,7 @@ import type { DataType } from '../../types.js' import { candidateDomains, defaultDomain, + isEqlEncryptedDomain, pgTypeToDataType, selectTableColumns, } from '../introspect.js' @@ -103,6 +104,44 @@ describe('defaultDomain', () => { }) }) +// The marker used to be `udt_name === 'eql_v2_encrypted'` alone. v3 is the +// default generation and its columns carry `eql_v3_*` domains, so on the +// default path an already-encrypted column was reported as plaintext: shown +// with its `dataType` hint and left unticked. `packages/wizard` already made +// this call, and its comment names the consequence — misreporting encrypted +// columns as plaintext lets an agent clobber real ciphertext. +describe('isEqlEncryptedDomain', () => { + it('recognises every eql_v3_* domain', () => { + for (const udt of [ + 'eql_v3_encrypted', + 'eql_v3_text_search', + 'eql_v3_integer_ord', + 'eql_v3_date_ord', + 'eql_v3_json', + ]) { + expect(isEqlEncryptedDomain(udt)).toBe(true) + } + }) + + it('still recognises the legacy v2 udt', () => { + expect(isEqlEncryptedDomain('eql_v2_encrypted')).toBe(true) + }) + + it('does not claim plaintext types', () => { + for (const udt of ['text', 'int4', 'jsonb', 'bool', 'timestamptz']) { + expect(isEqlEncryptedDomain(udt)).toBe(false) + } + }) + + // The trailing underscore is load-bearing: a bare `startsWith('eql_v3')` + // would also claim a hypothetical future `eql_v30_*` generation. Same + // reasoning as `classifyEqlDomain` in `@cipherstash/migrate`. + it('does not claim a future generation by prefix', () => { + expect(isEqlEncryptedDomain('eql_v30_text_search')).toBe(false) + expect(isEqlEncryptedDomain('eql_v3')).toBe(false) + }) +}) + describe('selectTableColumns', () => { beforeEach(() => { selectMock.mockReset() @@ -214,6 +253,38 @@ describe('selectTableColumns', () => { expect(schema?.columns).toEqual([{ name: 'ssn', domain: 'TextSearch' }]) }) + // The per-column hint was the literal string 'eql_v2_encrypted' for any + // encrypted column, so a v3 column was labelled with a domain it does not + // have. Report the column's actual udt. + it('hints an encrypted column with its real domain, not a hardcoded v2 udt', async () => { + const withV3 = [ + { + tableName: 'accounts', + columns: [ + { + columnName: 'ssn', + dataType: 'jsonb', + udtName: 'eql_v3_text_search', + isEqlEncrypted: true, + }, + ], + }, + ] + selectMock + .mockResolvedValueOnce('accounts') + .mockResolvedValueOnce('TextSearch') + multiselectMock.mockResolvedValueOnce(['ssn']) + + await selectTableColumns(withV3) + + const opts = multiselectMock.mock.calls[0][0].options + expect(opts).toEqual([ + expect.objectContaining({ value: 'ssn', hint: 'eql_v3_text_search' }), + ]) + // And it starts ticked, so a re-run does not offer to re-encrypt it. + expect(multiselectMock.mock.calls[0][0].initialValues).toEqual(['ssn']) + }) + it('pre-selects the widest searchable domain as the per-column default', async () => { // Asserts the ARGS to the domain prompt, not just its return: the picker // must pass initialValue: defaultDomain(options) so the widest searchable diff --git a/packages/cli/src/commands/init/lib/introspect.ts b/packages/cli/src/commands/init/lib/introspect.ts index 2e4a4c732..31b7bb398 100644 --- a/packages/cli/src/commands/init/lib/introspect.ts +++ b/packages/cli/src/commands/init/lib/introspect.ts @@ -42,12 +42,36 @@ export function pgTypeToDataType(udtName: string): DataType { } } +/** + * Is this column already managed by CipherStash? + * + * Both generations count. v3 is the sole generation this workspace authors, + * and its columns carry per-domain types (`eql_v3_text_search`, + * `eql_v3_integer_ord`, …) rather than v2's single `eql_v2_encrypted` udt — + * so keying on the v2 name alone, as this did, reported every column on the + * default path as plaintext. That is the dangerous direction to be wrong in: + * an encrypted column shown as plaintext invites the caller to encrypt it a + * second time. `packages/wizard` already carries this predicate; the two + * should agree. + * + * Deliberately not `classifyEqlDomain` from `@cipherstash/migrate`, despite + * the dependency being present: that answers "which generation authors this", + * and returns `null` for `eql_v2_encrypted` because v2 is no longer authorable. + * The question here is "is this encrypted at all", which v2 answers yes to. + * + * The trailing underscore matters — a bare `eql_v3` prefix would also claim a + * hypothetical future `eql_v30_*`. + */ +export function isEqlEncryptedDomain(udtName: string): boolean { + return udtName === 'eql_v2_encrypted' || udtName.startsWith('eql_v3_') +} + /** * Read every base table in the `public` schema along with its columns. * - * The `eql_v2_encrypted` UDT marker tells us a column is already managed by - * CipherStash — useful for re-runs against a partially set up DB so we can - * pre-select those columns rather than asking the user to reconfirm. + * The EQL domain markers tell us a column is already managed by CipherStash — + * useful for re-runs against a partially set up DB so we can pre-select those + * columns rather than asking the user to reconfirm. */ export async function introspectDatabase( databaseUrl: string, @@ -85,7 +109,7 @@ export async function introspectDatabase( columnName: row.column_name, dataType: row.data_type, udtName: row.udt_name, - isEqlEncrypted: row.udt_name === 'eql_v2_encrypted', + isEqlEncrypted: isEqlEncryptedDomain(row.udt_name), }) tableMap.set(row.table_name, cols) } @@ -203,7 +227,7 @@ export function defaultDomain( * Returns `undefined` if the user cancels at any prompt — callers should * propagate the cancellation rather than treating it as "no columns selected". * - * Pre-selects columns that are already `eql_v2_encrypted` so re-running on a + * Pre-selects columns that already carry an EQL domain so re-running on a * partially encrypted DB is a no-op by default. */ export async function selectTableColumns( @@ -230,7 +254,7 @@ export async function selectTableColumns( if (eqlColumns.length > 0) { p.log.info( - `Detected ${eqlColumns.length} column${eqlColumns.length !== 1 ? 's' : ''} with eql_v2_encrypted type — pre-selected for you.`, + `Detected ${eqlColumns.length} already-encrypted column${eqlColumns.length !== 1 ? 's' : ''} (${[...new Set(eqlColumns.map((c) => c.udtName))].join(', ')}) — pre-selected for you.`, ) } @@ -239,7 +263,7 @@ export async function selectTableColumns( options: table.columns.map((col) => ({ value: col.columnName, label: col.columnName, - hint: col.isEqlEncrypted ? 'eql_v2_encrypted' : col.dataType, + hint: col.isEqlEncrypted ? col.udtName : col.dataType, })), required: true, initialValues: eqlColumns.map((c) => c.columnName), From ea74846f22133b84fed7de50cf5a50751fbfd162 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sun, 26 Jul 2026 10:19:06 +1000 Subject: [PATCH 091/123] fix(stack): make adapter-kit importable without a process global --- .changeset/stack-logger-edge-safe.md | 11 +++ .../__tests__/helpers/process-free-realm.mjs | 73 +++++++++++++++++++ .../__tests__/logger-edge-safety.test.ts | 47 ++++++++++++ packages/stack/src/utils/logger/index.ts | 7 +- packages/stack/turbo.json | 9 +++ 5 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 .changeset/stack-logger-edge-safe.md create mode 100644 packages/stack/__tests__/helpers/process-free-realm.mjs create mode 100644 packages/stack/__tests__/logger-edge-safety.test.ts create mode 100644 packages/stack/turbo.json diff --git a/.changeset/stack-logger-edge-safe.md b/.changeset/stack-logger-edge-safe.md new file mode 100644 index 000000000..c84c0d97e --- /dev/null +++ b/.changeset/stack-logger-edge-safe.md @@ -0,0 +1,11 @@ +--- +'@cipherstash/stack': patch +--- + +`@cipherstash/stack/adapter-kit` is now importable in a runtime with no `process` global. + +The shared logger read `process.env.STASH_STACK_LOG` unguarded while initialising at module scope, so importing adapter-kit — which re-exports that logger — threw `ReferenceError: process is not defined` before any user code ran. The environment read is now guarded. + +This is not a single-adapter fix. `@cipherstash/stack-supabase`, `@cipherstash/stack-drizzle` and `@cipherstash/prisma-next` all value-import `@cipherstash/stack/adapter-kit`, so edge users of all three hit the same import-time throw, and all three are fixed by this release. + +**No behaviour change on Node.** `STASH_STACK_LOG`, its accepted values (`debug` / `info` / `error`), its `error` default, and the point at which the logger is configured are all unchanged. diff --git a/packages/stack/__tests__/helpers/process-free-realm.mjs b/packages/stack/__tests__/helpers/process-free-realm.mjs new file mode 100644 index 000000000..4ec23c2c8 --- /dev/null +++ b/packages/stack/__tests__/helpers/process-free-realm.mjs @@ -0,0 +1,73 @@ +/** + * Evaluate an emitted ESM file graph in a realm with NO `process` global — + * i.e. a Worker or a Deno isolate. + * + * Uses `node:vm`'s `SourceTextModule` with a linker that COMPILES each relative + * import into the same sandbox context, rather than `import()`ing it in the host + * realm (which would hand the module the host's `process` and prove nothing). + * Bare specifiers are rejected: the graphs this is pointed at have none, and a + * new one appearing is itself a finding. + * + * Run with `node --experimental-vm-modules`; callers spawn it that way, as a + * CHILD PROCESS, so no vitest configuration or flag is involved. + * + * Usage: node --experimental-vm-modules process-free-realm.mjs + * Exits 0 and prints `OK exports`, or exits 1 and prints the failure. + */ +import { readFileSync } from 'node:fs' +import { dirname, resolve as resolvePath } from 'node:path' +import { fileURLToPath, pathToFileURL } from 'node:url' +import vm from 'node:vm' + +async function loadInProcessFreeRealm(entryPath) { + // Deliberately minimal: `console` for diagnostics and the handful of globals + // a Worker genuinely has. NO `process`, NO `require`, NO `Buffer`. + const context = vm.createContext({ + console, + TextEncoder, + TextDecoder, + URL, + WebAssembly, + atob, + btoa, + }) + const cache = new Map() + + const compile = (file) => { + const cached = cache.get(file) + if (cached) return cached + const mod = new vm.SourceTextModule(readFileSync(file, 'utf8'), { + context, + identifier: pathToFileURL(file).href, + initializeImportMeta(meta) { + meta.url = pathToFileURL(file).href + }, + }) + cache.set(file, mod) + return mod + } + + const link = (specifier, referencing) => { + if (!specifier.startsWith('.') && !specifier.startsWith('/')) { + throw new Error( + `unexpected bare specifier "${specifier}" in ${referencing.identifier} — this graph is supposed to be self-contained`, + ) + } + const base = dirname(fileURLToPath(referencing.identifier)) + return compile(resolvePath(base, specifier)) + } + + const entry = compile(entryPath) + await entry.link(link) + await entry.evaluate() + return entry.namespace +} + +const [, , entryPath] = process.argv +try { + const namespace = await loadInProcessFreeRealm(entryPath) + console.log(`OK ${Object.keys(namespace).length} exports`) +} catch (error) { + console.error(`${error.constructor.name}: ${error.message}`) + process.exit(1) +} diff --git a/packages/stack/__tests__/logger-edge-safety.test.ts b/packages/stack/__tests__/logger-edge-safety.test.ts new file mode 100644 index 000000000..a28ba0f2a --- /dev/null +++ b/packages/stack/__tests__/logger-edge-safety.test.ts @@ -0,0 +1,47 @@ +/** + * `@cipherstash/stack/adapter-kit` re-exports this package's `logger` + * (`src/adapter-kit.ts:60`), and three first-party adapters value-import + * adapter-kit: `packages/stack-supabase/src/column-map.ts:1`, + * `packages/stack-drizzle/src/column.ts:1`, + * `packages/prisma-next/src/exports/column-types.ts:19`. A realm with no + * `process` binding turns an unguarded module-scope `process.env` read in the + * logger into a `ReferenceError` at import time on exactly the runtimes those + * builds exist to serve. + * + * This asserts the fix rather than a proxy for it: delete the `typeof process` + * guard from `src/utils/logger/index.ts`, rebuild, and this test fails. + * + * It reads `dist/`, so it SKIPS when the package has not been built — run + * `pnpm --filter @cipherstash/stack build` first for it to mean anything. The + * portable-entry plan's `bundling-isolation.test.ts` spawns the same harness + * against the WASM entry. + */ +import { execFile } from 'node:child_process' +import { existsSync } from 'node:fs' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' +import { describe, expect, it } from 'vitest' + +const execFileAsync = promisify(execFile) +const testsDir = fileURLToPath(new URL('.', import.meta.url)) +const harness = resolve(testsDir, 'helpers/process-free-realm.mjs') +const emittedEntry = resolve(testsDir, '../dist/adapter-kit.js') + +describe.skipIf(!existsSync(emittedEntry))( + 'the emitted adapter-kit seam imports without a process global', + () => { + it('evaluates dist/adapter-kit.js in a process-free realm', async () => { + // Rejects on a non-zero exit, so the harness's own failure line + // (`ReferenceError: process is not defined`) surfaces as the assertion + // failure. + const { stdout } = await execFileAsync(process.execPath, [ + '--experimental-vm-modules', + harness, + emittedEntry, + ]) + + expect(stdout.trim()).toMatch(/^OK \d+ exports$/) + }, 30_000) + }, +) diff --git a/packages/stack/src/utils/logger/index.ts b/packages/stack/src/utils/logger/index.ts index cf84ab9ad..8e6e8e611 100644 --- a/packages/stack/src/utils/logger/index.ts +++ b/packages/stack/src/utils/logger/index.ts @@ -14,7 +14,12 @@ export type LogLevel = 'debug' | 'info' | 'error' const validLevels: readonly LogLevel[] = ['debug', 'info', 'error'] as const function levelFromEnv(): LogLevel { - const env = process.env.STASH_STACK_LOG + // `process` is absent in a Worker or Deno isolate. This module is reachable + // from `@cipherstash/stack/adapter-kit` (`src/adapter-kit.ts:60`), which the + // Supabase, Drizzle and Prisma Next adapters all value-import — an unguarded + // read here is a ReferenceError at import time on those runtimes. + const env = + typeof process === 'undefined' ? undefined : process.env.STASH_STACK_LOG if (env && validLevels.includes(env as LogLevel)) return env as LogLevel return 'error' } diff --git a/packages/stack/turbo.json b/packages/stack/turbo.json new file mode 100644 index 000000000..5ff716600 --- /dev/null +++ b/packages/stack/turbo.json @@ -0,0 +1,9 @@ +{ + "$schema": "https://turbo.build/schema.json", + "extends": ["//"], + "tasks": { + "test": { + "dependsOn": ["build"] + } + } +} From 41a04a6ca901c32005f573cf58b4a289c29de768 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Sun, 26 Jul 2026 10:19:06 +1000 Subject: [PATCH 092/123] fix(stack-supabase): recognise v3 columns structurally, not by class identity --- .changeset/supabase-structural-v3-columns.md | 9 +++ .../__tests__/helpers/supabase-mock.ts | 40 +++++++++++ .../__tests__/supabase-schema-builder.test.ts | 66 +++++++++++++++++++ .../__tests__/supabase-v3-wire.test.ts | 36 +++++++++- packages/stack-supabase/src/column-map.ts | 58 ++++++++++++++-- 5 files changed, 202 insertions(+), 7 deletions(-) create mode 100644 .changeset/supabase-structural-v3-columns.md diff --git a/.changeset/supabase-structural-v3-columns.md b/.changeset/supabase-structural-v3-columns.md new file mode 100644 index 000000000..3d50513a4 --- /dev/null +++ b/.changeset/supabase-structural-v3-columns.md @@ -0,0 +1,9 @@ +--- +'@cipherstash/stack-supabase': patch +--- + +Fix: a table authored with `encryptedTable`/`types` imported from `@cipherstash/stack/wasm-inline` was treated as having **no encrypted columns**, so filter operands were sent to PostgREST as plaintext. + +`ColumnMap` gated on `builder instanceof EncryptedV3Column`, and the published bundles contain two separately-emitted copies of that class (`dist/adapter-kit.js` and `dist/wasm-inline.js` are separate esbuild runs). The check is now structural, so both copies are recognised. Tables authored from `@cipherstash/stack/eql/v3` were never affected — they resolve to the same copy the adapter imports. + +The failure was silent: `::jsonb` casts and result decryption go through a different path and kept working. diff --git a/packages/stack-supabase/__tests__/helpers/supabase-mock.ts b/packages/stack-supabase/__tests__/helpers/supabase-mock.ts index 26d8576a4..866a93283 100644 --- a/packages/stack-supabase/__tests__/helpers/supabase-mock.ts +++ b/packages/stack-supabase/__tests__/helpers/supabase-mock.ts @@ -212,3 +212,43 @@ export function createMockSupabase(resultData: unknown = []) { return { client, calls, callsFor } } + +/** + * A table whose column builders are structurally EQL v3 but are NOT instances + * of the `EncryptedV3Column` this package imports — which is exactly how a + * table authored from `@cipherstash/stack/wasm-inline` presents, because tsup + * emits that class twice (see `isV3ColumnLike` in `src/column-map.ts`). + * + * Object literals, not the real classes: reproducing the split with the real + * ones needs a built `dist/`, and `vitest.shared.ts:4-14` keeps `pnpm test` + * free of that. The dist-level version lives in the portable-entry plan. + */ +export function wasmAuthoredV3Table(tableName: string, columnNames: string[]) { + const columnBuilders = Object.fromEntries( + columnNames.map((name) => [ + name, + { + getName: () => name, + getEqlType: () => 'public.eql_v3_text_eq', + getQueryCapabilities: () => ({ + equality: true, + orderAndRange: false, + freeTextSearch: false, + }), + build: () => ({ cast_as: 'text', indexes: {} }), + }, + ]), + ) + return { + tableName, + columnBuilders, + buildColumnKeyMap: () => + Object.fromEntries(columnNames.map((name) => [name, name])), + build: () => ({ + tableName, + columns: Object.fromEntries( + columnNames.map((name) => [name, columnBuilders[name].build()]), + ), + }), + } +} diff --git a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts index caa0192b6..43f955048 100644 --- a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts @@ -1,8 +1,10 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' +import { ColumnMap } from '../src/column-map' import type { IntrospectionResult } from '../src/introspect' import { groupUnmodelledRows } from '../src/introspect' import { mergeDeclaredTables, synthesizeTables } from '../src/schema-builder' +import { wasmAuthoredV3Table } from './helpers/supabase-mock' const introspection: IntrospectionResult = [ { @@ -242,3 +244,67 @@ describe('groupUnmodelledRows', () => { expect(groupUnmodelledRows([]).size).toBe(0) }) }) + +describe('ColumnMap recognises v3 columns structurally, not by class identity', () => { + // tsup emits `EncryptedV3Column` TWICE — once into the chunk + // `dist/adapter-kit.js` imports, once inline in `dist/wasm-inline.js` (a + // separate esbuild run). A table authored from `@cipherstash/stack/wasm-inline` + // therefore failed `builder instanceof EncryptedV3Column` for EVERY column, + // leaving `v3Columns` empty — so the filter collector skipped every term and + // the RAW PLAINTEXT operand went into the PostgREST query string, while + // `::jsonb` casts and decryption kept working. + // + // These two assert the MECHANISM (`v3Columns` is populated / not + // over-populated). The HARM — what PostgREST actually receives — is asserted + // in `supabase-v3-wire.test.ts` by Step 2, because a check that merely + // probed `getName` would satisfy the two below. + it('accepts a builder that merely has the v3 column surface', () => { + const table = wasmAuthoredV3Table('users', ['email']) + + const columns = new ColumnMap('users', table as never, null) + + expect(columns.isEncryptedV3Column('email')).toBe(true) + expect(columns.encryptedColumnNames).toContain('email') + }) + + it('still rejects a v2 column builder', () => { + // v2 columns have `build()` and `getName()` (`packages/stack/src/schema/ + // index.ts:257,264`) but neither `getEqlType()` nor + // `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). Four probes, not + // two, is what keeps the predicate honest. + const v2 = { getName: () => 'email', build: () => ({}) } + const table = { + tableName: 'users', + columnBuilders: { email: v2 }, + buildColumnKeyMap: () => ({ email: 'email' }), + build: () => ({ tableName: 'users', columns: {} }), + } + + const columns = new ColumnMap('users', table as never, null) + + expect(columns.isEncryptedV3Column('email')).toBe(false) + expect(columns.encryptedColumnNames).toEqual([]) + }) +}) + +describe('every types.* domain satisfies the structural v3 probe', () => { + // `isV3ColumnLike` is module-private, so the property is stated through the + // public consequence: whatever the catalog grows to, ColumnMap must see the + // column as encrypted. A domain whose builder lost one of the four methods + // would be silently treated as PLAINTEXT — the PF2 failure again, from a + // different direction. Enumerated, not hardcoded (40 domains today), so a new + // one is covered the day it is added. + it('recognises a column built by any factory in the catalog', () => { + // Deterministic iteration over the WHOLE catalog, not a probabilistic + // sample: the guarantee this pins is "every domain", so every domain must + // actually run. `fc.constantFrom` would leave that to chance across its + // default run count. + for (const domain of Object.keys(types) as (keyof typeof types)[]) { + const table = encryptedTable('t', { c: types[domain]('c') }) + + const columns = new ColumnMap('t', table as never, null) + + expect(columns.isEncryptedV3Column('c')).toBe(true) + } + }) +}) diff --git a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts index 6b33b7c66..b7c483ed1 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts @@ -12,7 +12,10 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' import { describe, expect, it } from 'vitest' import { EncryptedQueryBuilderImpl as EncryptedQueryBuilderV3Impl } from '../src/query-builder' import { createWirePostgrest } from './helpers/postgrest-wire' -import { createMockEncryptionClient } from './helpers/supabase-mock' +import { + createMockEncryptionClient, + wasmAuthoredV3Table, +} from './helpers/supabase-mock' const users = encryptedTable('users', { email: types.TextSearch('email'), @@ -215,3 +218,34 @@ describe('plaintext not(col, contains, …) emits a parseable containment litera expect(wire.operandFor('note')).toBe('not.cs.{vip}') }) }) + +describe('a structurally-v3 table still encrypts the filter operand', () => { + // The regression this plan exists to stop, at the layer where it hurt: with + // the `instanceof` gate, a table that is structurally v3 but not an instance + // of THIS package's copy of `EncryptedV3Column` sent `eq.` to + // PostgREST. + it('emits an envelope, not the bare plaintext', async () => { + const wire = createWirePostgrest([]) + const builder = new EncryptedQueryBuilderV3Impl( + 'users', + wasmAuthoredV3Table('users', ['email']) as never, + createMockEncryptionClient(), + wire.client, + ['id', 'email'], + ) + + await builder.select('id').eq('email', 'ada@example.com') + + const operand = wire.operandFor('email') + expect(operand.startsWith('eq.')).toBe(true) + const value = operand.slice('eq.'.length) + + // NOT `expect(operand).not.toContain('ada@example.com')`: the encryption + // double deliberately carries the plaintext in the envelope's `pt` field so + // its fake decrypt can undo it (`helpers/supabase-mock.ts`). The contract + // being pinned is that the operand is an ENVELOPE rather than the raw + // value — unfixed, `value` is literally `ada@example.com`. + expect(value).not.toBe('ada@example.com') + expect(JSON.parse(value)).toMatchObject({ c: 'ct:ada@example.com' }) + }) +}) diff --git a/packages/stack-supabase/src/column-map.ts b/packages/stack-supabase/src/column-map.ts index 8920d87d1..4c6b932a6 100644 --- a/packages/stack-supabase/src/column-map.ts +++ b/packages/stack-supabase/src/column-map.ts @@ -1,13 +1,14 @@ -import { EncryptedV3Column } from '@cipherstash/stack/adapter-kit' import type { AnyV3Table } from '@cipherstash/stack/eql/v3' import type { ColumnSchema } from '@cipherstash/stack/schema' import type { BuildableQueryColumn } from '@cipherstash/stack/types' import type { DbName } from './types' /** - * The subset of a v3 column builder the dialect relies on. Structural rather - * than the concrete class union so the runtime `instanceof EncryptedV3Column` - * gate and this type stay independent. + * The subset of a v3 column builder the dialect relies on. + * + * This is BOTH the type and the runtime gate: `isV3ColumnLike` below probes + * exactly these members. It must not become an `instanceof` check — see that + * function's comment. */ export type V3ColumnLike = { getName(): string @@ -22,6 +23,45 @@ export type V3ColumnLike = { build(): ColumnSchema } +/** + * Whether a column builder is an EQL v3 column, checked STRUCTURALLY. + * + * NOT `instanceof EncryptedV3Column`. tsup emits that class twice — once into + * the chunk `dist/adapter-kit.js` imports, and once inline in + * `dist/wasm-inline.js`, a separate esbuild run + * (`packages/stack/tsup.config.ts:43-52`). A table authored with + * `encryptedTable`/`types` from `@cipherstash/stack/wasm-inline` is built from + * the second copy, so an `instanceof` against the first returned `false` for + * every column: `v3Columns` came out empty and the adapter treated encrypted + * columns as plaintext — filter operands reached PostgREST in the clear, while + * `::jsonb` casts and decryption kept working (they read `buildColumnKeyMap()` + * and the encrypt config, not this map). + * + * Mirrors `hasBuildColumnKeyMap` (`packages/stack/src/types.ts:276-283`), the + * repo's canonical answer to the same problem, used identically at + * `wasm-inline.ts:1361` — including its spelling: `'k' in obj && typeof (obj as + * { k?: unknown }).k === 'function'`, one narrowed probe per member, rather + * than one blanket `as Record<string, unknown>` over the whole object. + * + * Four probes, not two: a v2 column builder has `build()` and `getName()` + * (`packages/stack/src/schema/index.ts:257,264`) but neither `getEqlType()` nor + * `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). + */ +function isV3ColumnLike(builder: unknown): builder is V3ColumnLike { + if (typeof builder !== 'object' || builder === null) return false + return ( + 'getName' in builder && + typeof (builder as { getName?: unknown }).getName === 'function' && + 'getEqlType' in builder && + typeof (builder as { getEqlType?: unknown }).getEqlType === 'function' && + 'getQueryCapabilities' in builder && + typeof (builder as { getQueryCapabilities?: unknown }) + .getQueryCapabilities === 'function' && + 'build' in builder && + typeof (builder as { build?: unknown }).build === 'function' + ) +} + /** * Reject a declared property name that is also a DIFFERENT physical column. * @@ -102,8 +142,8 @@ export class ColumnMap { // otherwise resolve truthy for a plaintext column of that name. this.v3Columns = Object.create(null) as Record<string, V3ColumnLike> for (const [property, builder] of Object.entries(table.columnBuilders)) { - if (builder instanceof EncryptedV3Column) { - const col = builder as unknown as V3ColumnLike + if (isV3ColumnLike(builder)) { + const col = builder this.v3Columns[property] = col this.v3Columns[col.getName()] = col } @@ -213,6 +253,12 @@ export class ColumnMap { /** The encrypted builders as the term collector's column lookup. */ queryColumnMap(): Record<string, BuildableQueryColumn> { + // `V3ColumnLike` omits `isQueryable(): true`, which `BuildableV3QueryableColumn` + // requires — and `v3Columns` intentionally holds storage-only columns, for which + // `isQueryable()` is `false`. The collector consults `getQueryCapabilities()` + // before using an entry (`query-encrypt.ts:513`), so the widening is safe; + // narrowing the type would mean narrowing the map. + // biome-ignore lint/plugin: storage-only v3 columns lack `isQueryable(): true`; widening is safe (see above). return this.v3Columns as unknown as Record<string, BuildableQueryColumn> } } From 1118efdd86f4240d2bee1a3ad946ae603b2eb832 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Sun, 26 Jul 2026 11:40:28 +1000 Subject: [PATCH 093/123] fix(stack-supabase): fail closed on unrecognised v3 column builders MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses review on #799. - ColumnMap now throws at construction when a builder in an AnyV3Table's columnBuilders fails the structural v3 probe, instead of silently omitting it. An omitted column dropped out of v3Columns and its filter operands went to PostgREST as plaintext — the exact leak this work closes, from the other direction. Regression tests prove construction throws and no PostgREST request is issued. - Harden the logger env guard against a process defined without env. --- .changeset/supabase-structural-v3-columns.md | 2 ++ .../__tests__/supabase-schema-builder.test.ts | 15 ++++++--- .../__tests__/supabase-v3-wire.test.ts | 31 +++++++++++++++++++ packages/stack-supabase/src/column-map.ts | 19 +++++++++--- packages/stack/src/utils/logger/index.ts | 8 +++-- 5 files changed, 64 insertions(+), 11 deletions(-) diff --git a/.changeset/supabase-structural-v3-columns.md b/.changeset/supabase-structural-v3-columns.md index 3d50513a4..27d017118 100644 --- a/.changeset/supabase-structural-v3-columns.md +++ b/.changeset/supabase-structural-v3-columns.md @@ -7,3 +7,5 @@ Fix: a table authored with `encryptedTable`/`types` imported from `@cipherstash/ `ColumnMap` gated on `builder instanceof EncryptedV3Column`, and the published bundles contain two separately-emitted copies of that class (`dist/adapter-kit.js` and `dist/wasm-inline.js` are separate esbuild runs). The check is now structural, so both copies are recognised. Tables authored from `@cipherstash/stack/eql/v3` were never affected — they resolve to the same copy the adapter imports. The failure was silent: `::jsonb` casts and result decryption go through a different path and kept working. + +The recognition now also fails closed: a column builder that does not present the v3 surface makes `encryptedSupabase` throw at construction rather than silently omitting the column — an omitted column would send its filter operands to PostgREST as plaintext. diff --git a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts index 43f955048..8d9ac2803 100644 --- a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts @@ -267,11 +267,17 @@ describe('ColumnMap recognises v3 columns structurally, not by class identity', expect(columns.encryptedColumnNames).toContain('email') }) - it('still rejects a v2 column builder', () => { + it('throws on a builder missing the v3 column surface', () => { // v2 columns have `build()` and `getName()` (`packages/stack/src/schema/ // index.ts:257,264`) but neither `getEqlType()` nor // `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). Four probes, not // two, is what keeps the predicate honest. + // + // `columnBuilders` on an `AnyV3Table` must hold ONLY encrypted v3 columns, + // so a builder that fails the probe is malformed input. Silently skipping it + // is not a safe default: the column would drop out of `v3Columns` and its + // filter operands would go to PostgREST as PLAINTEXT. Fail closed at + // construction instead. const v2 = { getName: () => 'email', build: () => ({}) } const table = { tableName: 'users', @@ -280,10 +286,9 @@ describe('ColumnMap recognises v3 columns structurally, not by class identity', build: () => ({ tableName: 'users', columns: {} }), } - const columns = new ColumnMap('users', table as never, null) - - expect(columns.isEncryptedV3Column('email')).toBe(false) - expect(columns.encryptedColumnNames).toEqual([]) + expect(() => new ColumnMap('users', table as never, null)).toThrow( + /\[supabase v3\]/, + ) }) }) diff --git a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts index b7c483ed1..edd5fe6f3 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts @@ -249,3 +249,34 @@ describe('a structurally-v3 table still encrypts the filter operand', () => { expect(JSON.parse(value)).toMatchObject({ c: 'ct:ada@example.com' }) }) }) + +describe('an unrecognised column builder fails closed', () => { + // The mirror image of the leak above: a builder that does NOT present the v3 + // surface must never be silently demoted to plaintext passthrough, because + // then its filter operands would reach PostgREST in the clear. `ColumnMap` is + // built eagerly in the query-builder constructor, so construction throws + // BEFORE any request — this pins that no query string is ever emitted. + it('throws at construction, so no PostgREST request is issued', () => { + const wire = createWirePostgrest([]) + const malformed = { + tableName: 'users', + columnBuilders: { + email: { getName: () => 'email', build: () => ({}) }, + }, + buildColumnKeyMap: () => ({ email: 'email' }), + build: () => ({ tableName: 'users', columns: {} }), + } + + expect( + () => + new EncryptedQueryBuilderV3Impl( + 'users', + malformed as never, + createMockEncryptionClient(), + wire.client, + ['id', 'email'], + ), + ).toThrow(/\[supabase v3\]/) + expect(wire.urls).toEqual([]) + }) +}) diff --git a/packages/stack-supabase/src/column-map.ts b/packages/stack-supabase/src/column-map.ts index 4c6b932a6..81c85dd26 100644 --- a/packages/stack-supabase/src/column-map.ts +++ b/packages/stack-supabase/src/column-map.ts @@ -142,11 +142,22 @@ export class ColumnMap { // otherwise resolve truthy for a plaintext column of that name. this.v3Columns = Object.create(null) as Record<string, V3ColumnLike> for (const [property, builder] of Object.entries(table.columnBuilders)) { - if (isV3ColumnLike(builder)) { - const col = builder - this.v3Columns[property] = col - this.v3Columns[col.getName()] = col + // FAIL CLOSED. `columnBuilders` is typed `EncryptedV3TableColumn` + // (`eql/v3/table.ts:18-25`), so every entry is meant to be an encrypted v3 + // column. A builder that fails the structural probe is malformed input — + // and silently skipping it is the one thing we must not do here: the + // column would drop out of `v3Columns`, `isEncryptedColumn()` would return + // false for it, and its filter operands would go to PostgREST as + // PLAINTEXT. Refuse to construct instead, mirroring `build()`'s + // fail-loudly-on-malformed stance (`eql/v3/table.ts:47-51`). + if (!isV3ColumnLike(builder)) { + throw new Error( + `[supabase v3]: column "${property}" on table "${tableName}" is not a recognised EQL v3 column builder. Its filter operands would otherwise be sent to PostgREST unencrypted, so construction is refused. Author the table with \`encryptedTable\`/\`types\` from \`@cipherstash/stack/eql/v3\` or \`@cipherstash/stack/wasm-inline\`.`, + ) } + const col = builder + this.v3Columns[property] = col + this.v3Columns[col.getName()] = col } this.encryptedColumnNames = Object.keys(this.v3Columns) diff --git a/packages/stack/src/utils/logger/index.ts b/packages/stack/src/utils/logger/index.ts index 8e6e8e611..0d54ead7c 100644 --- a/packages/stack/src/utils/logger/index.ts +++ b/packages/stack/src/utils/logger/index.ts @@ -17,9 +17,13 @@ function levelFromEnv(): LogLevel { // `process` is absent in a Worker or Deno isolate. This module is reachable // from `@cipherstash/stack/adapter-kit` (`src/adapter-kit.ts:60`), which the // Supabase, Drizzle and Prisma Next adapters all value-import — an unguarded - // read here is a ReferenceError at import time on those runtimes. + // read here is a ReferenceError at import time on those runtimes. Guard + // `process.env` too: some partial polyfills define `process` without `env`, + // where `process.env.STASH_STACK_LOG` would throw just the same. const env = - typeof process === 'undefined' ? undefined : process.env.STASH_STACK_LOG + typeof process === 'undefined' || !process.env + ? undefined + : process.env.STASH_STACK_LOG if (env && validLevels.includes(env as LogLevel)) return env as LogLevel return 'error' } From f9c785eb2fbcf400e01e6ac2a3fa0a3e409e0ba5 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Mon, 27 Jul 2026 15:14:22 +1000 Subject: [PATCH 094/123] test(stack-supabase): pin the fail-closed message, not the [supabase v3] prefix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both assertions on the unrecognised-builder path matched `/\[supabase v3\]/`, which 32 messages across this package satisfy — two of them thrown by `ColumnMap` itself. Neither test could tell which error it caught. Demonstrated rather than assumed: making `assertNoPropertyDbNameCollision` throw unconditionally, so the fail-closed probe is bypassed entirely, left both tests GREEN. (Deleting the probe outright does turn them red — construction then succeeds and nothing throws. It is the bypass, not the deletion, the loose matcher was blind to.) `supabase-v3-wire.test.ts` is the more serious of the two: it guards the harm — no query string emitted — not just the mechanism. Both now pin the identity clause and both interpolations. Stops short of the advisory tail, which carries six unescaped `/` and would make the matcher a syntax error rather than a stricter test. Re-running the same bypass mutant now fails both. Also corrects a misattributed citation the assertion's comment repeats: the v2 `build()`/`getName()` at `schema/index.ts:257,264` are `EncryptedField`'s. `EncryptedColumn` opens at :275, so its own are at :442,449. --- .../__tests__/supabase-schema-builder.test.ts | 32 ++++++++++++------- .../__tests__/supabase-v3-wire.test.ts | 8 ++++- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts index 8d9ac2803..94f1a7d04 100644 --- a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts @@ -268,10 +268,10 @@ describe('ColumnMap recognises v3 columns structurally, not by class identity', }) it('throws on a builder missing the v3 column surface', () => { - // v2 columns have `build()` and `getName()` (`packages/stack/src/schema/ - // index.ts:257,264`) but neither `getEqlType()` nor - // `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). Four probes, not - // two, is what keeps the predicate honest. + // v2 columns have `build()` and `getName()` (`EncryptedColumn`, + // `packages/stack/src/schema/index.ts:442,449`) but neither `getEqlType()` + // nor `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). Four probes, + // not two, is what keeps the predicate honest. // // `columnBuilders` on an `AnyV3Table` must hold ONLY encrypted v3 columns, // so a builder that fails the probe is malformed input. Silently skipping it @@ -286,19 +286,29 @@ describe('ColumnMap recognises v3 columns structurally, not by class identity', build: () => ({ tableName: 'users', columns: {} }), } + // Pin the SPECIFIC message, not just the `[supabase v3]` prefix: 32 errors + // across this package share that prefix, two of them thrown by `ColumnMap` + // itself. A prefix-only matcher stays green whenever a DIFFERENT one of + // those fires first — measured: with `assertNoPropertyDbNameCollision` + // throwing unconditionally, so the fail-closed probe below is never + // reached, this test still passed. It could not tell which error it caught. + // (Deleting the probe outright does turn it red — construction then + // succeeds and nothing throws. It is the bypass, not the deletion, that the + // loose matcher was blind to.) expect(() => new ColumnMap('users', table as never, null)).toThrow( - /\[supabase v3\]/, + /\[supabase v3\]: column "email" on table "users" is not a recognised EQL v3 column builder/, ) }) }) describe('every types.* domain satisfies the structural v3 probe', () => { - // `isV3ColumnLike` is module-private, so the property is stated through the - // public consequence: whatever the catalog grows to, ColumnMap must see the - // column as encrypted. A domain whose builder lost one of the four methods - // would be silently treated as PLAINTEXT — the PF2 failure again, from a - // different direction. Enumerated, not hardcoded (40 domains today), so a new - // one is covered the day it is added. + // Stated through the public consequence rather than against the predicate: + // whatever the catalog grows to, ColumnMap must see the column as encrypted. + // A domain whose builder lost one of the four methods would be silently + // treated as PLAINTEXT — the PF2 failure again, from a different direction. + // Enumerated, not hardcoded (40 domains today), so a new one is covered the + // day it is added. The predicate's own shape is pinned separately, one probe + // at a time, in `column-map-predicate.test.ts`. it('recognises a column built by any factory in the catalog', () => { // Deterministic iteration over the WHOLE catalog, not a probabilistic // sample: the guarantee this pins is "every domain", so every domain must diff --git a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts index edd5fe6f3..3ffc94d92 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-wire.test.ts @@ -276,7 +276,13 @@ describe('an unrecognised column builder fails closed', () => { wire.client, ['id', 'email'], ), - ).toThrow(/\[supabase v3\]/) + // Prefix-only would not discriminate — see the matching note in + // `supabase-schema-builder.test.ts`. This test guards the HARM, so it is + // the one that most needs to fail if the fail-closed probe stops firing + // and some other `[supabase v3]` error surfaces in its place. + ).toThrow( + /\[supabase v3\]: column "email" on table "users" is not a recognised EQL v3 column builder/, + ) expect(wire.urls).toEqual([]) }) }) From 556072e7213e2e12a9b3cee46251fd27019657b5 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Mon, 27 Jul 2026 15:14:22 +1000 Subject: [PATCH 095/123] test(stack-supabase): unit-test the structural v3 probe, one member at a time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `isV3ColumnLike` had no test that could distinguish a four-probe gate from a two-probe one. Every builder reaching it either satisfied all four probes (the 40 catalog domains, the wasm-authored double) or missed two at once (the v2 stub) — so no fixture was ever one member away from passing. Measured with a mutation harness: deleting any single conjunct, weakening any single `typeof` to `true`, or dropping the object/null guard entirely left the whole 471-test suite green. Nine surviving mutants of those enumerated. (The `'x' in builder` half of each conjunct is an equivalent mutant — for any non-exotic object the `typeof` half subsumes it — so those four are correctly left alive.) The new cases are each one member apart from a conforming builder, so each fails if and only if its own probe goes. All nine die, and precisely: deleting a conjunct fails exactly its two tests, weakening a `typeof` exactly its one, and the two halves of the object/null guard are split so a mutant that drops one half names which half it dropped (1 test vs 4). Also covered: prototype-borne members (the real `Encrypted*Column` classes carry all four on the prototype, so `in` must not become `Object.hasOwn`), and a real `encryptedColumn().equality()` v2 builder rather than a hand-rolled stub — the case the four-probe design exists for. Neither kills a mutant the others miss; both are kept because they turn a 335-test avalanche and an opaque `true !== false` into one precise, self-describing failure. `isV3ColumnLike` is exported to allow this. `column-map.ts` is not re-exported from `src/index.ts` and the package publishes only `.`, so the published surface is unchanged: the built `dist/index.d.ts` and `.d.cts` contain no mention of it, and both bundles' runtime exports remain exactly `encryptedSupabase, encryptedSupabaseV3`. --- .../__tests__/column-map-predicate.test.ts | 133 ++++++++++++++++++ packages/stack-supabase/src/column-map.ts | 13 +- 2 files changed, 143 insertions(+), 3 deletions(-) create mode 100644 packages/stack-supabase/__tests__/column-map-predicate.test.ts diff --git a/packages/stack-supabase/__tests__/column-map-predicate.test.ts b/packages/stack-supabase/__tests__/column-map-predicate.test.ts new file mode 100644 index 000000000..2575e993a --- /dev/null +++ b/packages/stack-supabase/__tests__/column-map-predicate.test.ts @@ -0,0 +1,133 @@ +import { encryptedColumn } from '@cipherstash/stack/schema' +import { describe, expect, it } from 'vitest' +import { isV3ColumnLike } from '../src/column-map' + +/** + * Unit coverage for the structural v3 gate. + * + * `ColumnMap`'s tests pin the CONSEQUENCE — construction refuses, and no query + * string reaches PostgREST. They do not pin the predicate's shape: every fixture + * that reaches it either satisfies all four probes or (the one negative case) + * misses two at once, so no test there distinguishes a four-probe gate from a + * three- or two-probe one. Deleting any single probe, or weakening any single + * `typeof` to `true`, left the whole package suite green. + * + * These cases are written to be one-probe-apart from a passing builder, so each + * fails if and only if its own probe is removed. + */ + +type Probe = 'getName' | 'getEqlType' | 'getQueryCapabilities' | 'build' + +const PROBES: Probe[] = [ + 'getName', + 'getEqlType', + 'getQueryCapabilities', + 'build', +] + +/** A builder presenting exactly the surface the predicate probes for. */ +const conforming = (): Record<Probe, unknown> => ({ + getName: () => 'email', + getEqlType: () => 'public.eql_v3_text_search', + getQueryCapabilities: () => ({ + equality: true, + orderAndRange: false, + freeTextSearch: true, + }), + build: () => ({}), +}) + +const withoutProbe = (probe: Probe): Record<string, unknown> => { + const builder = conforming() + delete builder[probe] + return builder +} + +const withNonFunctionProbe = (probe: Probe): Record<string, unknown> => ({ + ...conforming(), + [probe]: 'not a function', +}) + +describe('isV3ColumnLike', () => { + it('accepts a builder presenting all four members', () => { + expect(isV3ColumnLike(conforming())).toBe(true) + }) + + // The mutation-killing core. Each fixture is one member away from the + // conforming builder above, so it can only be rejected by that member's own + // probe. A fixture missing two members (the shape the ColumnMap tests use) + // would be rejected by either, and so kills neither. + it.each(PROBES)('rejects a builder missing only %s()', (probe) => { + expect(isV3ColumnLike(withoutProbe(probe))).toBe(false) + }) + + // `'x' in builder` alone is satisfied by any present key. These pin the + // `typeof … === 'function'` half of each conjunct, which the `in` half cannot + // stand in for. + it.each(PROBES)('rejects a builder whose %s is not a function', (probe) => { + expect(isV3ColumnLike(withNonFunctionProbe(probe))).toBe(false) + }) + + // `typeof null === 'object'`, so `null` slips past the first half of the + // guard and only the explicit `builder === null` half rejects it. That half + // is load-bearing on its own: without it `'getName' in null` throws a raw + // TypeError instead of returning false, and ColumnMap's fail-closed error is + // replaced by an unrelated crash. + it('rejects null', () => { + expect(isV3ColumnLike(null)).toBe(false) + }) + + // These four go the other way — each is rejected by the `typeof builder !== + // 'object'` half. Split from `null` above so a mutant that drops only one + // half of the guard names which half it dropped. + it.each([ + ['undefined', undefined], + ['a number', 42], + ['a string', 'email'], + ['a boolean', true], + ])('rejects %s', (_label, builder) => { + expect(isV3ColumnLike(builder)).toBe(false) + }) + + // The real `Encrypted*Column` classes carry all four methods on the + // PROTOTYPE — their own properties are just `columnName`/`definition`. So the + // probes must use `in`, which walks the chain, and must not become + // `Object.hasOwn`, which does not. + it('accepts a builder whose members live on the prototype', () => { + class PrototypeColumn { + getName() { + return 'email' + } + getEqlType() { + return 'public.eql_v3_text_search' + } + getQueryCapabilities() { + return { equality: true, orderAndRange: false, freeTextSearch: true } + } + build() { + return {} + } + } + + const builder = new PrototypeColumn() + + expect(Object.hasOwn(builder, 'getName')).toBe(false) + expect(isV3ColumnLike(builder)).toBe(true) + }) + + // The case the four-probe design exists for, stated with the real class + // rather than a hand-rolled stub: a v2 column builder genuinely has `build()` + // and `getName()` and genuinely lacks the other two. A two-probe gate would + // accept it and its filter operands would go to PostgREST in the clear. + it('rejects a real EQL v2 column builder', () => { + const v2 = encryptedColumn('email').equality() + + // Spelled out so that if `EncryptedColumn` ever grows one of these, the + // failure names the cause rather than just reporting `true !== false`. + expect(typeof v2.getName).toBe('function') + expect(typeof v2.build).toBe('function') + expect('getEqlType' in v2).toBe(false) + expect('getQueryCapabilities' in v2).toBe(false) + expect(isV3ColumnLike(v2)).toBe(false) + }) +}) diff --git a/packages/stack-supabase/src/column-map.ts b/packages/stack-supabase/src/column-map.ts index 81c85dd26..1bfb0a8c6 100644 --- a/packages/stack-supabase/src/column-map.ts +++ b/packages/stack-supabase/src/column-map.ts @@ -44,10 +44,17 @@ export type V3ColumnLike = { * than one blanket `as Record<string, unknown>` over the whole object. * * Four probes, not two: a v2 column builder has `build()` and `getName()` - * (`packages/stack/src/schema/index.ts:257,264`) but neither `getEqlType()` nor - * `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). + * (`EncryptedColumn`, `packages/stack/src/schema/index.ts:442,449`) but neither + * `getEqlType()` nor `getQueryCapabilities()` (`eql/v3/columns.ts:445,450`). + * + * Exported for `__tests__/column-map-predicate.test.ts` only — `column-map.ts` + * is not re-exported from `src/index.ts` and the package publishes just `.`, so + * this does not widen the published surface (`V3ColumnLike` above is exported + * on the same terms). Testing it through `ColumnMap` cannot distinguish a + * four-probe gate from a two-probe one: every builder that reaches the + * constructor either passes every probe or fails two at once. */ -function isV3ColumnLike(builder: unknown): builder is V3ColumnLike { +export function isV3ColumnLike(builder: unknown): builder is V3ColumnLike { if (typeof builder !== 'object' || builder === null) return false return ( 'getName' in builder && From 001a527dda067cfb06b73f411283cab8a510ba2f Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Mon, 27 Jul 2026 15:14:22 +1000 Subject: [PATCH 096/123] docs(stack): a new bare specifier is a signal, not automatically a defect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The process-free realm harness rejects every bare specifier, which is stricter than the resolution any real runtime performs. It passes today only because everything the adapter-kit graph reaches is bundled via tsup `noExternal` — load-bearing, not incidental: the chunk adapter-kit imports carries inlined `evlog`, so without that entry it would emit a bare import. `@cipherstash/auth` and `@cipherstash/protect-ffi` stay external and are simply unreachable from that graph; if either became reachable, the gate would fail with a message reading like a self-containment defect. The header and the throw now say what the constraint rests on and what to do about it. No behaviour change — verified by pointing the harness at `dist/index.js`, whose bare `@cipherstash/auth` import produces the reworded error. `turbo.json`'s override also replaced the root `dependsOn` rather than extending it, resolving `test` to `["build"]` and dropping the inherited `["^build"]`. This is defence-in-depth rather than a live bug: ordering survives transitively via `build`'s own `^build`, as `packages/prisma-next` demonstrates with the identical override and real workspace deps. Made explicit anyway, matching how the root spells `test:e2e`; prisma-next is left alone deliberately, since the same reasoning says it is not broken either. Lastly, `logger-edge-safety.test.ts` described a `bundling-isolation.test.ts` that spawns this harness against the WASM entry as though it existed. It does not yet — reworded to future tense, and the turbo/bare-invocation skip asymmetry noted where a reader will hit it. --- packages/stack/__tests__/helpers/process-free-realm.mjs | 9 ++++++--- packages/stack/__tests__/logger-edge-safety.test.ts | 7 ++++--- packages/stack/turbo.json | 2 +- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/packages/stack/__tests__/helpers/process-free-realm.mjs b/packages/stack/__tests__/helpers/process-free-realm.mjs index 4ec23c2c8..cf1a8be8d 100644 --- a/packages/stack/__tests__/helpers/process-free-realm.mjs +++ b/packages/stack/__tests__/helpers/process-free-realm.mjs @@ -5,8 +5,11 @@ * Uses `node:vm`'s `SourceTextModule` with a linker that COMPILES each relative * import into the same sandbox context, rather than `import()`ing it in the host * realm (which would hand the module the host's `process` and prove nothing). - * Bare specifiers are rejected: the graphs this is pointed at have none, and a - * new one appearing is itself a finding. + * Bare specifiers are rejected. The graphs this is pointed at have none today + * only because everything they reach is bundled (`noExternal` in + * `packages/stack/tsup.config.ts`) — a property of the build config, not of + * module resolution. So a new one is a SIGNAL, not automatically a defect: if + * it is a legitimate external the target runtime resolves, allow it here. * * Run with `node --experimental-vm-modules`; callers spawn it that way, as a * CHILD PROCESS, so no vitest configuration or flag is involved. @@ -50,7 +53,7 @@ async function loadInProcessFreeRealm(entryPath) { const link = (specifier, referencing) => { if (!specifier.startsWith('.') && !specifier.startsWith('/')) { throw new Error( - `unexpected bare specifier "${specifier}" in ${referencing.identifier} — this graph is supposed to be self-contained`, + `unexpected bare specifier "${specifier}" in ${referencing.identifier} — this graph is bundled (tsup \`noExternal\`) and should have none. If this is a legitimate new external, allow it in this harness.`, ) } const base = dirname(fileURLToPath(referencing.identifier)) diff --git a/packages/stack/__tests__/logger-edge-safety.test.ts b/packages/stack/__tests__/logger-edge-safety.test.ts index a28ba0f2a..87e607f40 100644 --- a/packages/stack/__tests__/logger-edge-safety.test.ts +++ b/packages/stack/__tests__/logger-edge-safety.test.ts @@ -12,9 +12,10 @@ * guard from `src/utils/logger/index.ts`, rebuild, and this test fails. * * It reads `dist/`, so it SKIPS when the package has not been built — run - * `pnpm --filter @cipherstash/stack build` first for it to mean anything. The - * portable-entry plan's `bundling-isolation.test.ts` spawns the same harness - * against the WASM entry. + * `pnpm --filter @cipherstash/stack build` first for it to mean anything. + * (`turbo.json` wires `test` to `build`, so the turbo path cannot skip it; a + * bare `pnpm --filter … test` on a clean checkout still can.) The + * portable-entry plan will point the same harness at the WASM entry. */ import { execFile } from 'node:child_process' import { existsSync } from 'node:fs' diff --git a/packages/stack/turbo.json b/packages/stack/turbo.json index 5ff716600..8649e7366 100644 --- a/packages/stack/turbo.json +++ b/packages/stack/turbo.json @@ -3,7 +3,7 @@ "extends": ["//"], "tasks": { "test": { - "dependsOn": ["build"] + "dependsOn": ["^build", "build"] } } } From 8d31708e30ba107979467f65fe7e0c7ba5c259be Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Mon, 27 Jul 2026 17:15:29 +1000 Subject: [PATCH 097/123] fix(stack,stack-supabase): diagnose an EQL v2 table instead of crashing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A v2 `EncryptedTable` is structurally identical to a v3 one — same `tableName`, same `columnBuilders` — and only `buildColumnKeyMap()` tells them apart. So nothing that inspected shape caught the swap, and TypeScript only helps callers who are actually type-checking. Two paths, two raw TypeErrors, both naming an internal method rather than the version mismatch behind it: - `encryptedSupabase({ schemas })` sailed past the record-key check and died in `verifyDeclaredSchemas` as `builder.getEqlType is not a function`. This is the reachable one — the realistic mistake is migrating from v2 and passing the old `schemas` through. - Constructing the query builder directly died one layer down in `ColumnMap`, on the constructor's very first statement, as `table.buildColumnKeyMap is not a function`. Not reachable from the factory today (`mergeDeclaredTables` rebuilds a fresh v3 table), so this half is defence-in-depth at the seam — symmetric with the column-level fail-closed guard already there, which cannot cover it because it runs after the constructor has already crashed. Both now name the table and state the fix. Each guard is load-bearing: removing either puts its original TypeError back. The check routes through `hasBuildColumnKeyMap` rather than a second hand-written spelling, per that function's own doctrine — "a second hand-written spelling of the check is how a v2 envelope eventually gets built for a v3 table once the marker drifts". It is re-exported from `@cipherstash/stack/adapter-kit`, the first-party adapter seam, and deliberately NOT promoted to `./types`: deciding which wire version a table targets is adapter plumbing, not end-user API. Verified edge-safe — `dist/adapter-kit.js` still evaluates in a process-free realm with no bare specifiers (`OK 14 exports`). `skills/stash-supabase` gains the error under "Legacy: EQL v2", where a migrating reader will hit it. --- .changeset/supabase-v2-table-diagnosis.md | 25 +++++++++++++++++ .../__tests__/supabase-schema-builder.test.ts | 23 ++++++++++++++++ .../__tests__/supabase-v3-factory.test.ts | 27 +++++++++++++++++++ packages/stack-supabase/src/column-map.ts | 17 ++++++++++++ packages/stack-supabase/src/index.ts | 13 +++++++++ packages/stack/src/adapter-kit.ts | 12 ++++++--- skills/stash-supabase/SKILL.md | 11 ++++++++ 7 files changed, 124 insertions(+), 4 deletions(-) create mode 100644 .changeset/supabase-v2-table-diagnosis.md diff --git a/.changeset/supabase-v2-table-diagnosis.md b/.changeset/supabase-v2-table-diagnosis.md new file mode 100644 index 000000000..1c6cfb160 --- /dev/null +++ b/.changeset/supabase-v2-table-diagnosis.md @@ -0,0 +1,25 @@ +--- +'@cipherstash/stack-supabase': patch +'@cipherstash/stack': minor +'stash': patch +--- + +Diagnose an EQL v2 table by name instead of crashing with a raw `TypeError`. + +A v2 `EncryptedTable` is structurally identical to a v3 one — same `tableName`, +same `columnBuilders` — and only `buildColumnKeyMap()` tells them apart. Passing +one to `encryptedSupabase({ schemas })` therefore sailed past every check that +looked at shape and died deep inside verification as `builder.getEqlType is not +a function`, naming an internal method rather than the version mismatch that +caused it. Constructing the query builder directly failed the same way one layer +down, as `table.buildColumnKeyMap is not a function`. + +Both paths now fail closed with the table named and the fix stated. The check +routes through `hasBuildColumnKeyMap`, the canonical v2/v3 discriminator, rather +than a second hand-written spelling of it. + +`@cipherstash/stack` re-exports `hasBuildColumnKeyMap` from +`@cipherstash/stack/adapter-kit` so first-party adapters can make that routing +decision without reaching into internals. It is deliberately not on `./types`: +deciding which wire version a table targets is adapter plumbing, not end-user +API. diff --git a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts index 94f1a7d04..185700364 100644 --- a/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-schema-builder.test.ts @@ -1,4 +1,8 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' +import { + encryptedColumn, + encryptedTable as v2EncryptedTable, +} from '@cipherstash/stack/schema' import { describe, expect, it } from 'vitest' import { ColumnMap } from '../src/column-map' import type { IntrospectionResult } from '../src/introspect' @@ -299,6 +303,25 @@ describe('ColumnMap recognises v3 columns structurally, not by class identity', /\[supabase v3\]: column "email" on table "users" is not a recognised EQL v3 column builder/, ) }) + + it('throws a diagnosis, not a raw TypeError, on a whole v2 table', () => { + // A v2 `EncryptedTable` is structurally identical to a v3 one at the table + // level — same `tableName`, same `columnBuilders`. The only discriminator + // is `buildColumnKeyMap()`, which the constructor calls UNGUARDED as its + // first statement, so a v2 table died with `table.buildColumnKeyMap is not + // a function` — naming an internal method, not the version mismatch that + // caused it. + // + // The column-level probe above cannot catch this: it runs later, and by + // then the constructor has already crashed. + const v2Table = v2EncryptedTable('users', { + email: encryptedColumn('email').equality(), + }) + + expect(() => new ColumnMap('users', v2Table as never, null)).toThrow( + /\[supabase v3\]: table "users" is an EQL v2 table/, + ) + }) }) describe('every types.* domain satisfies the structural v3 probe', () => { diff --git a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts index 083af15e7..93bab05f9 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts @@ -1,4 +1,8 @@ import { encryptedTable, types } from '@cipherstash/stack/eql/v3' +import { + encryptedColumn, + encryptedTable as v2EncryptedTable, +} from '@cipherstash/stack/schema' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import type { SupabaseClientLike } from '../src/index.js' import { encryptedSupabaseV3 } from '../src/index.js' @@ -101,6 +105,29 @@ describe('encryptedSupabaseV3 factory', () => { expect(encryptionMock).not.toHaveBeenCalled() }) + it('rejects a v2 table in schemas before introspection results are used', async () => { + // The realistic caller mistake this guards: migrating from v2 and passing + // the old `schemas` through. A v2 table has `tableName` and + // `columnBuilders` just like a v3 one, so it sails past the record-key + // check and dies deeper in — previously at `verify.ts`, as + // `builder.getEqlType is not a function`, which names an internal method + // rather than the version mismatch. + const users = v2EncryptedTable('users', { + email: encryptedColumn('email').equality(), + }) + + await expect( + encryptedSupabaseV3(fakeClient, { + databaseUrl: 'postgres://x', + schemas: { users } as never, + }), + ).rejects.toThrow( + /\[supabase v3\]: schemas entry "users" is an EQL v2 table/, + ) + // The client must never be built from a schema set we could not validate. + expect(encryptionMock).not.toHaveBeenCalled() + }) + it('passes only non-empty tables to Encryption', async () => { introspectMock.mockResolvedValue( introspectionOf({ diff --git a/packages/stack-supabase/src/column-map.ts b/packages/stack-supabase/src/column-map.ts index 1bfb0a8c6..c3974b0fa 100644 --- a/packages/stack-supabase/src/column-map.ts +++ b/packages/stack-supabase/src/column-map.ts @@ -1,3 +1,4 @@ +import { hasBuildColumnKeyMap } from '@cipherstash/stack/adapter-kit' import type { AnyV3Table } from '@cipherstash/stack/eql/v3' import type { ColumnSchema } from '@cipherstash/stack/schema' import type { BuildableQueryColumn } from '@cipherstash/stack/types' @@ -134,6 +135,22 @@ export class ColumnMap { table: AnyV3Table, allColumns: string[] | null, ) { + // FAIL CLOSED at the table level, for the same reason the column loop does + // below. `buildColumnKeyMap()` is the canonical v2/v3 discriminator + // (`packages/stack/src/types.ts:276`), and it is also the very first thing + // this constructor calls — so a v2 table died on the next line with + // `table.buildColumnKeyMap is not a function`, naming an internal method + // instead of the version mismatch. The column-level probe cannot cover + // this: it runs after the constructor has already crashed. + // + // Routed through `hasBuildColumnKeyMap` rather than hand-spelled, per that + // function's own doctrine — a second spelling is how the marker drifts. + if (!hasBuildColumnKeyMap(table)) { + throw new Error( + `[supabase v3]: table "${tableName}" is an EQL v2 table — it has no buildColumnKeyMap(), the marker every v3 table carries. This adapter is EQL v3 only. Author the table with \`encryptedTable\`/\`types\` from \`@cipherstash/stack/eql/v3\` or \`@cipherstash/stack/wasm-inline\`.`, + ) + } + this.propToDb = table.buildColumnKeyMap() this.columnSchemas = table.build().columns diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index 4710e6f52..a05c7b81f 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -1,4 +1,5 @@ import { Encryption } from '@cipherstash/stack' +import { hasBuildColumnKeyMap } from '@cipherstash/stack/adapter-kit' import type { UnmodelledColumn } from './introspect' import { eqlRequiresQueryDomains, introspect } from './introspect' import { EncryptedQueryBuilderImpl } from './query-builder' @@ -169,6 +170,18 @@ export async function encryptedSupabase( `[supabase v3]: schemas key "${key}" does not match its table name "${table.tableName}" — the record key must equal the table's name`, ) } + // A v2 `EncryptedTable` carries `tableName` and `columnBuilders` exactly + // as a v3 one does, so nothing above this distinguishes them and the + // types only catch it for callers who are actually type-checking. Left + // unguarded it reached `verifyDeclaredSchemas`, which calls + // `builder.getEqlType()` — absent on a v2 column — and died as + // `builder.getEqlType is not a function`: an internal method name, from + // a caller's perspective unrelated to the mistake they made. + if (!hasBuildColumnKeyMap(table)) { + throw new Error( + `[supabase v3]: schemas entry "${key}" is an EQL v2 table — it has no buildColumnKeyMap(), the marker every v3 table carries. This adapter is EQL v3 only. Author the table with \`encryptedTable\`/\`types\` from \`@cipherstash/stack/eql/v3\`.`, + ) + } assertTableIsModelled(key, unmodelled) } verifyDeclaredSchemas(options.schemas, tables) diff --git a/packages/stack/src/adapter-kit.ts b/packages/stack/src/adapter-kit.ts index 2d1fad591..a46706d8d 100644 --- a/packages/stack/src/adapter-kit.ts +++ b/packages/stack/src/adapter-kit.ts @@ -26,7 +26,6 @@ export { // Audit config carried by chainable operations. export type { AuditConfig } from './encryption/operations/base-operation.js' - // v3 column model + the date-like cast set the Supabase builder uses to // reconstruct `Date` values from PostgREST select aliases. export { @@ -34,7 +33,6 @@ export { DATE_LIKE_CASTS, EncryptedV3Column, } from './eql/v3/columns.js' - // Domain registry: the Supabase adapter classifies introspected columns by their // Postgres domain and rebuilds each column's encryption config from it. export { @@ -42,7 +40,6 @@ export { factoryForDomain, stripDomainSchema, } from './eql/v3/domain-registry.js' - // Shared JSONPath-selector path handling (parse/validate, needle-document // reconstruction, scalar-leaf guard) for encrypted-JSON querying — used by the // Drizzle selector operators (#651) and the Supabase selector filters (#650) so @@ -53,8 +50,15 @@ export { reconstructSelectorDocument, unsupportedLeafReason, } from './eql/v3/selector-path.js' - // Shared match-index guard (short-needle rejection), reused by both adapters. export { matchNeedleError } from './schema/match-defaults.js' +// The canonical EQL v2/v3 table discriminator. Added because the adapters must +// make the same routing decision core does, and that function's own doctrine is +// that every site routes through it — "a second hand-written spelling of the +// check is how a v2 envelope eventually gets built for a v3 table once the +// marker drifts" (`types.ts:268-275`). Re-exported here rather than promoted to +// `./types`: deciding which wire version a table targets is adapter plumbing, +// not something an end user should be reaching for. +export { hasBuildColumnKeyMap } from './types.js' // Shared structured logger (adapters log rejections through the same instance). export { logger } from './utils/logger/index.js' diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 2cb45f9a8..4bcec8b35 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -778,6 +778,17 @@ and queries EQL v3 only, via the introspecting `encryptedSupabase(url, key)` / `encryptedSupabase(client, options)` factory described above. There is no longer a code path in this package that emits or reads `eql_v2_encrypted` columns. +Passing a v2 table in `schemas` is rejected by name: + +``` +[supabase v3]: schemas entry "users" is an EQL v2 table — it has no +buildColumnKeyMap(), the marker every v3 table carries. +``` + +A v2 `encryptedTable` is structurally identical to a v3 one apart from that +marker, so TypeScript alone will not always catch the swap — re-author the table +with `encryptedTable`/`types` from `@cipherstash/stack/eql/v3`. + Existing v2 deployments have two options: - **Migrate to EQL v3** (recommended): add an `eql_v3_*` twin column and run the From 17393b97af6589011dbf566e284025891a80da9f Mon Sep 17 00:00:00 2001 From: Dan Draper <dan@cipherstash.com> Date: Tue, 28 Jul 2026 10:21:57 +1000 Subject: [PATCH 098/123] =?UTF-8?q?feat(cli):=20add=20the=20stash-postgres?= =?UTF-8?q?=20and=20stash-edge=20skills=20=E2=80=94=20raw-SQL=20predicates?= =?UTF-8?q?=20and=20the=20WASM=20entry=20(#777)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(cli): add the stash-sql and stash-edge skills — raw-SQL predicates and the WASM entry Closes #754. No shipped skill covered the integrations that don't use an ORM: hand-written SQL over `pg` / `postgres-js`, and the WASM entry on Deno / Supabase Edge Functions / Workers. Grepping the skills `stash init` installs for `postgres-js|::jsonb::eql|sql.json|query_text_search` returned one hit, in an unrelated code comment — so an integration on that path had to recover the binding surface from `dist/*.d.ts`, the Postgres catalog, and experiment. Two skills rather than one, per the scope question in the issue: the raw-SQL predicate cookbook serves a supported plain-Node path (`hono-pg`) with no edge or WASM involvement, so scoping it under an edge-named skill would leave that surface uncovered. skills/stash-sql — the predicate matrix (which of `=`, `<>`, `<`, `>=`, `@@`, `@>` each column domain accepts, against which `eql_v3.query_*` operand), storage-vs-query payload shapes, per-driver parameter binding, and recipes for equality / free-text / range / ORDER BY / JSON containment / field selectors. skills/stash-edge — import specifier per runtime, the four mandatory `CS_*` variables and minting them with `stash env`, how the WASM client surface differs from the native typed client, and why a schema module can't be shared across the two entries. Both carry the credential-identity rule, also folded into stash-cli (under `env` and `encrypt backfill`, where it bites) and stash-supabase: EQL index terms derive from the ZeroKMS client key, so rows written under one credential and queried under another decrypt correctly and never match a query. Three claims were corrected against a live EQL v3 3.0.2 install rather than carried over from the issue: - The binding rule is driver-specific. On postgres-js a bare object fails to INSERT and `JSON.stringify(...)::jsonb` fails the domain CHECK; only `sql.json()` works in both positions. On `pg` all three forms work. - A bare-`jsonb` operand does not fall through to native jsonb semantics — EQL defines `jsonb` overloads that coerce to the *storage* domain, which requires the ciphertext key `c`, so a query term without the domain cast raises a CHECK violation rather than silently matching nothing. - The schema type incompatibility reproduces in both directions, not one. Also fixes the wasm-inline module JSDoc, which passed `OidcFederationStrategy.create(...)`'s Result straight to `config.authStrategy` without unwrapping — the same JSDoc the raw-SQL surface was being reverse-engineered from. SKILL_MAP (CLI + wizard) installs both for `postgresql` and `supabase`; Drizzle and Prisma Next get cross-links from their own skills instead. * refactor(skills): rename stash-sql to stash-postgres The skill is Postgres-specific throughout — encrypted columns are Postgres domains over jsonb, the driver rules cover pg and postgres-js, and the drift check reads information_schema. Nothing in it generalises to another SQL dialect, so the name overclaimed. Renames the directory and frontmatter name, updates the SKILL_MAP entries in both the CLI and wizard installers, the setup-prompt purpose line, the cross-links in the five sibling skills, and the AGENTS.md skill list and change→skill map. The skill has not shipped yet (no CHANGELOG entry — only the pending changeset), so no installed .claude/skills/ directory carries the old name and the existing changeset is amended in place rather than adding a second one. The new name also matches the CLI's `postgresql` integration key, which is the no-ORM path that installs it. * docs(skills): scope stash-postgres against Proxy and cite EQL upstream The skill covered client-side encryption over a direct connection without ever saying so. That is precisely the fork `stash init` asks about and stores as `usesProxy`: a CipherStash Proxy user writes plaintext SQL and Proxy encrypts on the wire, so every rule in the skill — mint a term with encryptQuery, cast to eql_v3.query_*, bind with sql.json — is wrong for them, and nothing said which world they were in. Adds a scope callout up front, and notes that Proxy's config lifecycle (stash db push into eql_v2_configuration) is the EQL v2 one, while this skill is v3 where there is nothing to push. The operator matrix also read as if the client library defined it. It does not: the domains, operators, CHECKs, and extractors all come from the EQL bundle developed at cipherstash/encrypt-query-language and shipped as @cipherstash/eql. A reader hitting an operator the matrix does not list had nowhere authoritative to check and no idea where to file it. Adds a provenance section with the version check (SELECT eql_v3.version()), ties the "operator does not exist" troubleshooting entry to it, and adds upstream links for EQL and Proxy to the Reference list. Also fixes the frontmatter description, which now states the direct-connection assumption so skill selection reflects it. * docs(skills): drop the usesProxy pointer from the stash-postgres Proxy callout The callout told readers to check `usesProxy` in `.cipherstash/context.json` to learn which path they were on. That flag and field are being removed (the CLI never used the value for anything beyond gating a wizard `stash db push` step), so the pointer would have shipped stale. The Proxy scoping itself stands — it is the substance of the callout. * docs(skills): separate the database domain from its language mappings "Assume `users.email` is a `types.TextEq` column" named the schema builder as though it were the column's type. It isn't: `types.TextEq` is a TypeScript factory in this repo, while the column is `public.eql_v3_text_eq`, a Postgres domain over jsonb. The distinction is the one the whole section turns on, so opening by conflating them was the wrong footing. Now opens with the database type and states that the domain is the authority — what the CHECK enforces and what decides the operator set — then lists the three things that map onto it: the `types.*` schema factory, the `TextEq` / `TextEqQuery` wire types from `@cipherstash/eql`, and the `eql_v3.query_*` operand domain. Records where the TypeScript types come from: generated with the JSON Schemas from the Rust `eql-bindings` crate, with the SQL bundle built from the same commit, so the wire shape and the domain CHECK cannot drift. Verified against @cipherstash/eql 3.0.2 — `TextEq` is `{ v, i, c, hm }` and `TextEqQuery` is `{ v, i, hm }`, so the generated pair is exactly the storage-vs-query split this section goes on to explain. * docs(skills): send agents to EQL for the current type surface The domain and operator tables read as authoritative. They are not — they are a snapshot of a versioned surface defined in `encrypt-query-language`, and nothing in CI catches them drifting when the `@cipherstash/eql` pin moves. Marks them as a snapshot at the point of use, and adds a ranked list of places to confirm current types: the EQL skill first (it ships beside the bundle it documents, so it tracks the installed version), then the generated `@cipherstash/eql` TypeScript types, then the install SQL's CREATE OPERATOR statements, then `SELECT eql_v3.version()` against the database. The middle two need only `node_modules` — the `stash` CLI depends on `@cipherstash/eql` at an exact pin — so the check costs nothing and needs no connection. The EQL skill does not exist yet (cipherstash/encrypt-query-language#422), so it is referenced by role rather than by a name that may not survive review, and every other rung of the ladder works today without it. * docs(skills): stop teaching native-entry bundling in the edge skill `stash-edge` named `@cipherstash/protect-ffi` five times. Three of those were configuration for the runtime this skill exists to steer readers away from: the entry-choice table's Node row explained how to externalise it in Next, and the Workers section named it again to say the config was irrelevant. A reader here is not configuring a Node server, so that is the wrong skill to carry it — the bundling guide already owns it, and is now linked instead. Two mentions stay, for different reasons: - The `When to Use` trigger keeps the name. Deploying the default entry to an edge runtime fails with `Cannot find module '@cipherstash/protect-ffi'`, and that string in a deploy log is how an agent should route itself here. - The Deno note keeps `--allow-ffi`, which is Deno's own permission flag rather than the package — a diagnostic that the wrong entry resolved. The intro now says "a Node-API native module" without naming the package; the contrast it draws does not need it. * docs(skills): correct the lock-context explanation in stash-edge DEPENDS ON #797 — do not merge #777 until the wasm-inline `.withLockContext()` lands. This describes the post-#797 surface. The skill said "Identity-bound encryption is configured, not chained" and presented `config.authStrategy` as the replacement for `.withLockContext()`. That conflates two orthogonal mechanisms. An auth strategy decides who the client is; a lock context decides which key the value is encrypted under. You need both, and a strategy alone writes data that is not identity-bound. What actually changed on the native entry was authentication: per-operation CTS tokens were removed in protect-ffi 0.25, so `LockContext.identify()` is deprecated and the strategy handles token acquisition. `.withLockContext()` never went anywhere. The WASM entry picked up the authentication half and not the key-binding half, and the skill then described the gap as though the first subsumed the second. Rewrites the section as two numbered steps, adds the encrypt-side example, and states the symmetry rule — the same claim must be supplied on decrypt, and a mismatch surfaces as a failed decrypt rather than a key error. The code example assumes the chainable form, matching native. If #797 lands a per-call option instead, this example must change before #777 merges. * docs(skills): address CodeRabbit review on #777 Five of six findings applied; the sixth skipped with a reason. - `stash-drizzle`: `db.execute(sql\`…\`)` used backslash-escaped backticks inside an inline code span. Escapes do not work inside code spans, so the backslashes rendered literally. Switched to a double-backtick span, which is the actual mechanism for embedding a backtick. - `stash-edge`, `stash-postgres`: language tags on three unlabelled fences — a TypeScript error, the domain-name mapping, and a Postgres CHECK-violation message. All `text`. - `stash-postgres` frontmatter: the operator list omitted `<=` and `>`, which the predicate matrix includes. The description is the skill-selection surface, so an incomplete list there is worse than verbose. - `stash-postgres` Query Recipes: the recipes dereference `.data` without guarding `.failure`, and agents copy from them. Rather than adding the guard to six short recipes, the section now states the omission up front and shows the guard once — unguarded, the failure surfaces as a domain CHECK violation rather than as the encryption error it is, which is the confusing outcome this skill exists to prevent. Skipped: adding a "the docs site needs a corresponding update" note to the changeset. Changesets become CHANGELOG entries, which are customer-facing; an internal follow-up action does not belong there. Tracked separately. Two other unlabelled fences exist in `stash-encryption` and `stash-cli` but predate this branch and are untouched by it. * docs(skills): reconcile the WASM entry surface after the rebase Rebasing onto current `remove-v2` pulled in a `stash-encryption` subpath row asserting the WASM entry has "no `.audit()` or `.withLockContext()` chaining", which contradicts `stash-edge` — the two ship in the same tarball, so an agent reading both gets opposite answers about the same entry. `.withLockContext()` stays described as chainable (this branch documents the post-#797 surface, per 9a575095), so the `stash-encryption` row now states only the `.audit()` difference and defers identity-bound encryption to `stash-edge`. The `.audit()` claim is the one that survives independently: the WASM operations return plain Results rather than thenable operations (`wasm-inline.ts:664`). The changeset carried the same stale "no `.withLockContext()`" phrasing and is corrected with it. Also documents a real native/WASM divergence the difference table was missing, raised in review: the model decrypt helpers. Both entries take the table as the second argument, but the native typed client *also* carries a one-arg `decryptModel(model)` overload — the read path for rows whose table isn't in the schema set, legacy EQL v2 above all (`encryption/v3.ts:121-134`). The WASM entry has no such overload; `requiresTableForDecrypt` is declared true and the call throws without a table rather than returning a `{ failure }` (`wasm-inline.ts:674-685`). A wrapper written against the one-arg form therefore compiles on one entry and breaks on the other. Still blocked on #797 — `.withLockContext()` does not exist on the wasm-inline client at any published `protect-ffi` (0.30.0 is the pin and npm `latest`; its WASM typings carry no lock-context declarations). protectjs-ffi#143 has since landed the FFI half, as a per-call `lockContext` option field rather than a chainable method, so the chainable form here remains a bet on how #797 wraps it — the review condition 9a575095 set out. * docs(skills,stack): document the wasm entry as having no lock context Unblocks this PR from #797 by describing the surface that exists today rather than the one #797 will create. `.withLockContext()` is absent from the wasm-inline client on every branch in this repo, and no published protect-ffi adds it — the FFI exposes `lockContext` as an option field, never a chainable, so the chainable form was always going to be a stack-side wrapper that has not been written. What is kept from 9a575095 is the part that was right and is the reason #793 exists: an auth strategy and a lock context are orthogonal. A strategy decides who the client is; a lock context decides which key the value is encrypted under. The earlier text presented the first as a replacement for the second. That claim is now stated as the mistake it is, in both the skill and the source comment it came from, instead of being silently deleted. The two consequences are silent, so both are now written down rather than left to surface as a failed decrypt: - Values written from the edge entry are encrypted under the workspace key even when the client is authenticated as an end user. - The edge entry cannot read anything the native entry wrote under a lock context, since decrypt requires the same context. A read split between the two entries, on top of the schema nominal-typing incompatibility. `wasm-inline.ts` also records that the binding already accepts a lock context on both paths, so closing #797 is plumbing rather than a new capability — and that it wants a live round-trip test first, since a wrong or missing claim surfaces as a failed decrypt rather than a key error. The `stash-encryption` subpath row regains the `.withLockContext()` fact the rebase reconciliation had removed, now that the two skills agree again. --- .changeset/stash-postgres-edge-skills.md | 75 +++ AGENTS.md | 4 +- .../init/lib/__tests__/install-skills.test.ts | 22 + .../src/commands/init/lib/install-skills.ts | 17 +- .../cli/src/commands/init/lib/setup-prompt.ts | 4 + packages/stack/src/wasm-inline.ts | 40 +- packages/wizard/src/lib/install-skills.ts | 12 +- skills/stash-cli/SKILL.md | 10 + skills/stash-drizzle/SKILL.md | 2 + skills/stash-edge/SKILL.md | 458 +++++++++++++++++ skills/stash-encryption/SKILL.md | 7 +- skills/stash-indexing/SKILL.md | 4 +- skills/stash-postgres/SKILL.md | 479 ++++++++++++++++++ skills/stash-prisma-next/SKILL.md | 5 +- skills/stash-supabase/SKILL.md | 8 + 15 files changed, 1133 insertions(+), 14 deletions(-) create mode 100644 .changeset/stash-postgres-edge-skills.md create mode 100644 skills/stash-edge/SKILL.md create mode 100644 skills/stash-postgres/SKILL.md diff --git a/.changeset/stash-postgres-edge-skills.md b/.changeset/stash-postgres-edge-skills.md new file mode 100644 index 000000000..424c2774b --- /dev/null +++ b/.changeset/stash-postgres-edge-skills.md @@ -0,0 +1,75 @@ +--- +'stash': minor +'@cipherstash/wizard': minor +'@cipherstash/stack': patch +--- + +Two new bundled agent skills for the integrations that don't use an ORM — +`stash-postgres` and `stash-edge` (#754). + +Everything a raw-SQL or edge integration needed was reachable only from +`dist/*.d.ts` JSDoc, the Postgres catalog, or experiment: grepping the skills +`stash init` installs for `postgres-js|::jsonb::eql|sql.json|query_text_search` +returned a single hit, in an unrelated code comment. + +**`stash-postgres`** — hand-written SQL over `pg` / `postgres-js`, no ORM. The +column-domain-to-query-domain operator matrix (which of `=`, `<>`, `<`, `>=`, +`@@`, `@>` each encrypted domain accepts, and against which `eql_v3.query_*` +operand), the storage-vs-query payload distinction, per-driver parameter +binding, recipes for equality / free-text / range / `ORDER BY` / JSON +containment / JSON field selectors, and the `information_schema` drift check. +Two failure modes get their mechanism spelled out: pre-stringifying a payload +on postgres-js double-encodes it into a jsonb *string* scalar, tripping the +domain CHECK with a message naming neither JSON nor encoding; and leaving an +operand as bare `jsonb` silently selects a different operator overload — one +that coerces to the *storage* domain and so rejects the ciphertext-free query +term. It also scopes itself against the two things a hand-written-SQL reader +is otherwise left to infer: **CipherStash Proxy** (where you write plaintext +SQL and none of the skill applies — the `usesProxy` fork `stash init` already +asked about), and the provenance of the operator surface itself (the EQL +bundle from `cipherstash/encrypt-query-language`, version-checkable with +`SELECT eql_v3.version()`, and where operator gaps should be filed). Its +domain and operator tables are explicitly marked as a snapshot of a versioned +surface, with a ranked list of authorities to confirm current types against — +the EQL skill first, then the generated `@cipherstash/eql` types and install +SQL, both of which need only `node_modules` and no database. + +**`stash-edge`** — the `@cipherstash/stack/wasm-inline` entry for Deno, +Supabase Edge Functions, Cloudflare Workers, and Bun. Import specifier per +runtime, the four mandatory `CS_*` variables and minting them with +`stash env`, how the WASM client surface differs from the native typed client +(no `.audit()`, no `.withLockContext()`, per-item bulk shape, a required +`table` argument on `decryptModel` / `bulkDecryptModels`, ESM-only), and the +auth-strategy `Result` that must be unwrapped before it reaches +`config.authStrategy`. + +It also separates the two mechanisms behind identity-bound encryption, which +are routinely conflated — and which the source comment on the entry itself got +wrong. An auth strategy decides *who the client is*; a lock context decides +*which key the value is encrypted under*. Only the first exists on this entry, +so an `authStrategy` alone still writes values encrypted under the workspace +key, and the entry cannot read what the native one wrote under a lock context. +That is a silent read split between the two entries, and the skill says so +rather than leaving it to be discovered as a failed decrypt. + +Both carry **the credential-identity rule**, a silent data-loss footgun now +also stated in `stash-cli` (under `env` and `encrypt backfill`) and +`stash-supabase`: EQL index terms derive from the ZeroKMS client key, so rows +written under one credential and queried under another decrypt correctly and +never match a query, with no error. + +`stash-encryption` now states that the two entries' schema types **do not +interchange** — their column classes carry private fields, so TypeScript +compares them nominally and rejects a shared schema module in both directions. +It works at runtime, which makes a type assertion the tempting fix; the +guidance is to author the schema against exactly one entry instead. + +`stash init` / `stash impl` handoffs and the `@cipherstash/wizard` skills +prompt install both skills for the `postgresql` and `supabase` integrations. +Drizzle and Prisma Next get cross-links from their own skills instead, since +those integrations emit correctly-typed operands themselves. + +Also fixes the `@cipherstash/stack/wasm-inline` module JSDoc, which showed +`OidcFederationStrategy.create(...)`'s `Result` being passed straight to +`config.authStrategy` without unwrapping — the same JSDoc the raw-SQL surface +was being reverse-engineered from. diff --git a/AGENTS.md b/AGENTS.md index 943dbb981..2ce2f337b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,7 +85,7 @@ If these variables are missing, tests that require live encryption will fail or - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). -- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-indexing`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`) +- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-indexing`, `stash-postgres`, `stash-edge`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`) ## Agent Skills — these ship to customers @@ -113,6 +113,8 @@ nothing type-checks them, and the damage lands in a customer's repo, not ours. | Drizzle / Supabase / Prisma Next / DynamoDB integrations | `skills/stash-drizzle`, `skills/stash-supabase`, `skills/stash-prisma-next`, `skills/stash-dynamodb` | | The rollout/cutover lifecycle (`packages/migrate`, `stash encrypt *`) | `skills/stash-encryption` and `skills/stash-cli` | | The `@cipherstash/eql` pin, `eql install`/`eql migration` behaviour, or index-related SQL guidance | `skills/stash-indexing` | +| The EQL operator/domain surface (`eql_v3.query_*` casts, predicate forms) | `skills/stash-postgres` | +| `packages/stack/src/wasm-inline.ts`, the WASM entry's exports, or `stash env` | `skills/stash-edge` | | pnpm config, CI workflows, dependency policy | `skills/stash-supply-chain-security` | | The durable agent rules themselves | `packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md` | diff --git a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts index e4a2e3aa1..052496cc0 100644 --- a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts @@ -82,6 +82,28 @@ describe('SKILL_MAP', () => { expect(SKILL_MAP.postgresql).not.toContain('stash-supabase') expect(SKILL_MAP.postgresql).not.toContain('stash-prisma-next') }) + + // #754: the no-ORM path had no source for the raw-SQL binding surface or + // the WASM entry — an integration on it had to reverse-engineer both from + // `dist/*.d.ts` and the Postgres catalog. `postgresql` is that path; + // Supabase shares it (Edge Functions are the flagship WASM-entry use, and + // its migrations/RPC are hand-written SQL). + it.each([ + 'postgresql', + 'supabase', + ] as const)('%s includes the raw-SQL and edge skills', (integration) => { + expect(SKILL_MAP[integration]).toContain('stash-postgres') + expect(SKILL_MAP[integration]).toContain('stash-edge') + }) + + // The ORM integrations emit correctly-typed operands themselves, so they + // get cross-links from their own skills rather than the full install. + it.each([ + 'drizzle', + 'prisma-next', + ] as const)('%s does not install the raw-SQL skill', (integration) => { + expect(SKILL_MAP[integration]).not.toContain('stash-postgres') + }) }) describe('skillsFor', () => { diff --git a/packages/cli/src/commands/init/lib/install-skills.ts b/packages/cli/src/commands/init/lib/install-skills.ts index acd158180..8353ce1b1 100644 --- a/packages/cli/src/commands/init/lib/install-skills.ts +++ b/packages/cli/src/commands/init/lib/install-skills.ts @@ -12,10 +12,16 @@ import { findBundledDir } from './bundled-paths.js' */ export const SKILL_MAP: Record<Integration, readonly string[]> = { drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'], + // Supabase gets the raw-SQL and edge skills on top of its own: Edge + // Functions are the flagship use of the WASM entry, and Supabase projects + // write hand-written SQL in migrations and RPC even when the app itself + // goes through PostgREST (#754). supabase: [ 'stash-encryption', 'stash-supabase', 'stash-indexing', + 'stash-postgres', + 'stash-edge', 'stash-cli', ], 'prisma-next': [ @@ -24,7 +30,16 @@ export const SKILL_MAP: Record<Integration, readonly string[]> = { 'stash-indexing', 'stash-cli', ], - postgresql: ['stash-encryption', 'stash-indexing', 'stash-cli'], + // The no-ORM path: `stash-postgres` (binding + predicate forms) and `stash-edge` + // (WASM entry, CS_* credentials) are the two skills this integration has no + // other source for — everything else assumes an ORM emits the operands. + postgresql: [ + 'stash-encryption', + 'stash-indexing', + 'stash-postgres', + 'stash-edge', + 'stash-cli', + ], } /** The skills every integration gets — the safe fallback for an unmapped one. */ diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index ac97a3687..3deefd3f3 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -198,6 +198,10 @@ const SKILL_PURPOSES: Record<string, string> = { 'Prisma Next-specific patterns: `cipherstash.*` field constructors, migration flow, encrypted query operators', 'stash-indexing': 'index recipes for encrypted columns — the `eql_v3` extractor functional indexes, Supabase/managed-Postgres constraints, EXPLAIN verification', + 'stash-postgres': + 'hand-written Postgres SQL over `pg` / `postgres-js`: the encrypted predicate matrix, `eql_v3.query_*` operand casts, per-driver parameter binding', + 'stash-edge': + 'the `@cipherstash/stack/wasm-inline` entry for Deno / Supabase Edge Functions / Workers: imports, `CS_*` credentials, the credential-identity rule', 'stash-dynamodb': 'DynamoDB encryption: per-item encrypt/decrypt, HMAC attribute keys, audit logging', 'stash-cli': diff --git a/packages/stack/src/wasm-inline.ts b/packages/stack/src/wasm-inline.ts index 05a3a97b7..2b2d8dd6d 100644 --- a/packages/stack/src/wasm-inline.ts +++ b/packages/stack/src/wasm-inline.ts @@ -64,17 +64,24 @@ * import { OidcFederationStrategy } from "@cipherstash/stack/wasm-inline" * import { cookieStore } from "@cipherstash/auth/cookies" * - * const authStrategy = OidcFederationStrategy.create( + * // `create` returns a Result — UNWRAP it. `config.authStrategy` takes the + * // strategy itself; handing it the Result fails later and opaquely. + * const strategy = OidcFederationStrategy.create( * "crn:ap-southeast-2.aws:my-workspace-id", () => getClerkSessionToken(req), * { store: cookieStore({ request: req, responseHeaders }) }, * ) - * const client = await Encryption({ schemas, config: { authStrategy, clientId, clientKey } }) + * if (strategy.failure) throw new Error(strategy.failure.error.message) + * + * const client = await Encryption({ + * schemas, config: { authStrategy: strategy.data, clientId, clientKey }, + * }) * ``` * * For service-to-service / CI use with a custom token store, build an * `AccessKeyStrategy.create(workspaceCrn, accessKey, { store })` the same - * way (it derives the region from the CRN). Both strategies are - * re-exported from this entry. + * way — same Result-unwrapping, and it derives the region from the CRN. Both + * strategies are re-exported from this entry. An auth strategy and + * `config.accessKey` are mutually exclusive. */ import { withResult } from '@byteslice/result' @@ -655,9 +662,28 @@ const INTERNAL_CONSTRUCT = Symbol('cs-wasm-client') * ZeroKMS round trip regardless of how many fields or models it covers. What * still differs from the native surface is deliberate and local: failures come * back as this entry's `{ failure }` Results (rather than a thenable operation - * with `.audit()`), and there is no `.withLockContext()` — identity-bound - * encryption on the edge is configured at client construction via - * `config.authStrategy` instead (#663 context). + * with `.audit()`). + * + * There is also no `.withLockContext()` here, and that one is a **gap, not a + * design decision** (#797). An earlier version of this comment said + * identity-bound encryption is "configured at client construction via + * `config.authStrategy` instead" (#663 context). That is wrong, and the + * conflation is worth naming because it is the one people arrive with: + * an auth strategy decides WHO THE CLIENT IS; a lock context decides WHICH + * KEY THE VALUE IS ENCRYPTED UNDER. They are orthogonal, and only the first + * exists on this entry. + * + * The consequences are silent, which is why they are stated here rather than + * left to a failed decrypt: values written from this entry are encrypted under + * the workspace key even when the client is authenticated as an end user, and + * this entry cannot read anything the native entry wrote under a lock context, + * since decrypt requires the same context. `skills/stash-edge` documents both. + * + * The binding accepts a lock context on both paths — `lockContext` is an + * option field on the single calls and per-payload-item on the bulk ones — so + * closing this is plumbing rather than a new capability. It is not plumbed + * here yet, and shipping it would want a live round-trip test first: a wrong + * or missing claim surfaces as a failed decrypt, not a key error. * * Construct via {@link Encryption} — the constructor is private to * prevent callers from wrapping arbitrary objects in this type. diff --git a/packages/wizard/src/lib/install-skills.ts b/packages/wizard/src/lib/install-skills.ts index 474bc2bdc..b3f5d2f96 100644 --- a/packages/wizard/src/lib/install-skills.ts +++ b/packages/wizard/src/lib/install-skills.ts @@ -13,14 +13,24 @@ import type { Integration } from './types.js' */ const SKILL_MAP: Record<Integration, readonly string[]> = { drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'], + // `stash-postgres` / `stash-edge` mirror the CLI's SKILL_MAP — see the comments + // there for why Supabase and the generic (no-ORM) path get them (#754). supabase: [ 'stash-encryption', 'stash-supabase', 'stash-indexing', + 'stash-postgres', + 'stash-edge', 'stash-cli', ], prisma: ['stash-encryption', 'stash-indexing', 'stash-cli'], - generic: ['stash-encryption', 'stash-indexing', 'stash-cli'], + generic: [ + 'stash-encryption', + 'stash-indexing', + 'stash-postgres', + 'stash-edge', + 'stash-cli', + ], } /** diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 849763fb9..4a279964b 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -457,6 +457,8 @@ Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgr **Dual-write precondition.** The application must already write both `<col>` and `<col>_encrypted` on every insert and update. Otherwise rows written *during* the backfill land in plaintext only, silently. The first run prompts (interactive) or requires `--confirm-dual-writes-deployed` (non-interactive), then records `dual_writing`. Resumes don't re-prompt. +**Credential precondition — run the backfill with the *application's* credentials.** Backfill encrypts through whichever `CS_*` credentials are in its environment, and EQL index terms derive from the ZeroKMS client key. Backfilling from a laptop on the local device profile, then querying from an app using credentials minted by `stash env`, produces rows that decrypt correctly and **never match a query** — with no error. Export the target environment's `CS_*` values in the shell running the backfill. See [`env`](#env) and `stash-edge` § The Credential-Identity Rule. + | Flag | Description | |---|---| | `--table` / `--column` | Required | @@ -516,6 +518,14 @@ CS_CLIENT_KEY=<hex> CS_CLIENT_ACCESS_KEY=CSAK… ``` +> **Every writer of a searchable column must use these same credentials** — +> including `stash encrypt backfill`, seed scripts, and admin tools — or their +> rows decrypt but never match a query. EQL index terms derive from the ZeroKMS +> client key, so a row written under one credential and queried under another +> decrypts correctly and silently fails every search. Mint one credential per +> environment and export it for **every** process that writes that +> environment's data. See `stash-edge` § The Credential-Identity Rule. + Things to know: - **The access key is shown exactly once** — CTS cannot re-reveal it. Pipe the diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 50ae36fd2..d9d03b22e 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -417,6 +417,8 @@ import { index } from "drizzle-orm/pg-core" Run `ANALYZE <table>` after the migration applies — an expression index gathers no statistics at `CREATE INDEX` time. For when to create indexes during a rollout (after backfill, before switching reads), engagement rules, and `EXPLAIN` verification, see the `stash-indexing` skill. +> **Dropping to raw SQL?** ``db.execute(sql`…`)`` bypasses the operators this integration emits, so you own the operand casts yourself — an encrypted predicate needs its needle cast to the column's `eql_v3.query_*` domain, and the driver's parameter-binding rules differ between `pg` and `postgres-js`. The `stash-postgres` skill is the reference for both. + ## Migrating an Existing Column to Encrypted The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. diff --git a/skills/stash-edge/SKILL.md b/skills/stash-edge/SKILL.md new file mode 100644 index 000000000..12ca34fc3 --- /dev/null +++ b/skills/stash-edge/SKILL.md @@ -0,0 +1,458 @@ +--- +name: stash-edge +description: Run CipherStash encryption on edge and non-Node runtimes with the `@cipherstash/stack/wasm-inline` entry — Deno, Supabase Edge Functions, Cloudflare Workers, and Bun. Covers the import specifier per runtime, the four mandatory `CS_*` variables and minting them with `stash env`, the credential-identity rule (rows written under different credentials decrypt but never match a query), how the WASM client surface differs from the native typed client, and why an EQL v3 schema module cannot be shared across the two entries. Use when adding encryption to a Supabase Edge Function, a Worker, or a Deno service; when a native module fails to load in a deployed runtime; when wiring `CS_*` secrets into an edge deploy; or when encrypted search returns zero rows on the edge but works locally. +--- + +# Encryption on the Edge (WASM entry) + +`@cipherstash/stack` has two runtime entries. The default one binds a +Node-API native module and must be loaded by Node's own `require`. +**`@cipherstash/stack/wasm-inline` is the entry for +everywhere else** — it carries the WASM build of the same engine as a base64 +blob inside the JS, so there is no native binding, no separate `.wasm` fetch, +and nothing for a bundler to externalise. + +This skill covers that entry and the deployment shape around it. It is EQL v3 +throughout. For the SQL that actually queries the encrypted columns — the +predicate forms and driver binding rules — see `stash-postgres`; edge functions +almost always talk to Postgres over a raw driver, so the two are usually read +together. + +## When to Use This Skill + +- Adding encryption to a Supabase Edge Function, Cloudflare Worker, Deno + service, or Bun app. +- A deployed runtime fails to load the native module (`protect-ffi`), or a + bundler chokes trying to include it. +- Wiring `CS_*` credentials into an edge deploy, or minting them at all. +- Encrypted search works locally but returns **zero rows** in the deployed + function — see [The Credential-Identity Rule](#the-credential-identity-rule-a-silent-data-footgun). +- A schema module shared with Node tooling fails to typecheck against the + edge client. + +## Choosing the Entry + +| Runtime | Entry | Why | +|---|---|---| +| Node server, Next.js server code | `@cipherstash/stack` (+ `/v3`) | Native NAPI is faster; the native module must be excluded from bundling — see the [bundling guide](https://cipherstash.com/docs/stack/deploy/bundling) | +| Supabase Edge Functions | `@cipherstash/stack/wasm-inline` | Deno, V8-only, no native modules | +| Cloudflare Workers | `@cipherstash/stack/wasm-inline` | V8 isolate, no native modules | +| Deno (any) | `@cipherstash/stack/wasm-inline` | No NAPI under Deno's default permissions | +| Bun | `@cipherstash/stack/wasm-inline` | Works, and avoids native-module resolution differences | +| Anywhere bundling server code | `@cipherstash/stack/wasm-inline` | Bundles cleanly; nothing to externalise | + +**The WASM entry is ESM-only.** Its `exports` map has an `import` condition +and no `require` — deliberately, since the runtimes it targets are ESM. A CJS +`require('@cipherstash/stack/wasm-inline')` will not resolve. Node consumers +that need it must be ESM (`"type": "module"` or `.mjs`). + +## Importing It + +### Supabase Edge Functions / Deno — `npm:` specifier + +The Edge runtime resolves `npm:` specifiers at function start; there is no +build step. + +```ts +import { + Encryption, encryptedTable, types, isEncrypted, +} from 'npm:@cipherstash/stack@1.0.0-rc.4/wasm-inline' +``` + +**Pin an exact version.** Deno caches by specifier, so an unpinned import +drifts between deploys. And while the package is on a prerelease line, a +caret range does not do what it looks like: `@^1.0.0` will **not** match +`1.0.0-rc.4`, because semver ranges exclude prereleases unless the range +itself names one. Use the exact version, or a prerelease-bearing range +(`@^1.0.0-rc.4`). Check what is current with `npm view @cipherstash/stack dist-tags`. + +### Deno with an import map + +For a project with a `deno.json`, map the specifier once and import the bare +name everywhere: + +```jsonc +{ + "imports": { + "@cipherstash/stack/wasm-inline": "npm:@cipherstash/stack@1.0.0-rc.4/wasm-inline" + } +} +``` + +```ts +import { Encryption, encryptedTable, types } from '@cipherstash/stack/wasm-inline' +``` + +> **No `--allow-ffi` needed.** The whole point of this entry is that nothing +> native loads. If a Deno process running this entry ever demands an FFI +> permission, something has resolved the native entry instead — check the +> import path before granting anything. + +### Cloudflare Workers / Bun / bundlers — normal install + +```bash +npm install @cipherstash/stack +``` + +```ts +import { Encryption, encryptedTable, types } from '@cipherstash/stack/wasm-inline' +``` + +No `externals`, no `nodeExternals`, no `serverExternalPackages` entry. If a +build config already externalises the native module for the default entry, +that config does not apply here and can be left alone. + +## Credentials + +The edge client takes **all four** `CS_*` values explicitly. There is no +credential discovery: `~/.cipherstash` does not exist in a Worker or an Edge +Function container, and there is no device-code login to fall back on. + +```ts +const client = await Encryption({ + schemas: [users], + config: { + workspaceCrn: Deno.env.get('CS_WORKSPACE_CRN')!, + accessKey: Deno.env.get('CS_CLIENT_ACCESS_KEY')!, + clientId: Deno.env.get('CS_CLIENT_ID')!, + clientKey: Deno.env.get('CS_CLIENT_KEY')!, + }, +}) +``` + +Read them from the platform's environment accessor — `Deno.env.get(...)` on +Deno/Supabase, the `env` binding argument on Workers, `process.env` on Bun. + +### Minting them: `stash env` + +```bash +stash env --name my-app-prod # print the four vars to stdout +stash env --name my-app-prod --write # write .env.production.local (mode 0600) +stash env --name edge-dev --write .env.local +``` + +This creates a fresh ZeroKMS client **and** a CipherStash access key from your +local `stash auth login` session. Things that matter here: + +- **The access key is shown exactly once.** Pipe it straight into the secret + store; it cannot be re-revealed. +- **Stdout is pipe-clean** — only the dotenv block goes to stdout, so + `stash env --name x > prod.env` and pipes into secret-store CLIs are safe. +- Each run mints a **new** credential, and duplicate names are rejected. Use a + distinct `--name` per environment. +- `CS_CLIENT_KEY` and `CS_CLIENT_ACCESS_KEY` are secrets. Never commit them; + put placeholder names in `.env.example` instead. + +### Getting them into the runtime + +```bash +# Supabase — local +supabase functions serve --env-file .env.local my-function + +# Supabase — deployed +stash env --name my-app-prod --write .env.production.local +supabase secrets set --env-file .env.production.local + +# Cloudflare Workers +wrangler secret put CS_CLIENT_KEY # repeat per variable + +# Vercel / other platforms +vercel env add CS_CLIENT_KEY production +``` + +## The Credential-Identity Rule (a silent data footgun) + +> **Every writer of a searchable column must use the same credentials as every +> reader — including `stash encrypt backfill`, seed scripts, and admin tools. +> Rows written under different credentials decrypt correctly but never match a +> query.** + +EQL's searchable-encryption index terms (the `hm`, `op`, `bf` fields in the +stored payload) derive from the **ZeroKMS client key**, not from the workspace +or keyset. Two clients in the same workspace with different `CS_CLIENT_ID` / +`CS_CLIENT_KEY` pairs therefore produce **different terms for the same +plaintext**. + +The consequences are asymmetric, which is what makes this hard to spot: + +- **Decryption still works.** The data key is wrapped through ZeroKMS against + the workspace, so any authorised client in the workspace can decrypt the + row. Round-trip tests pass. +- **Search silently fails.** An equality or match predicate compares the + query term against the stored term. Different client keys, different terms, + no match — and no error. The query returns zero rows exactly as though the + data were absent. + +The classic way to hit it: run `stash encrypt backfill` from a laptop (using +the local device-profile credentials), then query those rows from an Edge +Function using `CS_*` values minted by `stash env`. Every row decrypts. No +search ever matches. + +**What to do:** + +- Mint one credential per *environment*, and use it for **every** process that + touches that environment's data — the app, the backfill, seed scripts, admin + jobs, and one-off scripts alike. +- Before running `stash encrypt backfill` against an environment, export that + environment's `CS_*` values into the shell running it. +- If rows have already been written under the wrong credentials, re-encrypt + them with the correct client: read (decryption still works), then write back + through a client built with the target credentials. + +**Diagnosing it:** if `decrypt` returns the right plaintext but an equality +query on the same row returns nothing, compare the stored term against a +freshly minted one for the same plaintext. Matching plaintext with differing +`hm` values is this bug, not an indexing problem. + +## The Client Surface + +`Encryption` from the WASM entry. **Both entries name the factory +`Encryption`**, so the import path is the only thing that distinguishes them — +which makes a stray `import { Encryption } from '@cipherstash/stack'` easy to +miss and confusing to debug, because the two clients take different config and +different bulk shapes. Check the specifier first whenever an edge client +behaves unexpectedly. + +Every fallible method returns the same `{ data } | { failure }` Result +contract as the native client; unwrap before use. + +```ts +const enc = await client.encrypt('alice@example.com', { table: users, column: users.email }) +if (enc.failure) throw new Error(enc.failure.message) + +const dec = await client.decrypt(enc.data) +if (dec.failure) throw new Error(dec.failure.message) +``` + +Available: `encrypt`, `decrypt`, `isEncrypted`, `encryptQuery`, +`encryptQueryBulk`, `bulkEncrypt`, `bulkDecrypt`, `encryptModel`, +`decryptModel`, `bulkEncryptModels`, `bulkDecryptModels`. + +### How it differs from the native typed client + +| | Native (`@cipherstash/stack`) | WASM (`@cipherstash/stack/wasm-inline`) | +|---|---|---| +| Factory | `Encryption({ schemas })` | `Encryption({ schemas, config })` — same name, different module | +| Schema authoring | `encryptedTable` / `types` from `@cipherstash/stack/v3` | the entry's own re-exports (see below) | +| Config | discovered from env / `~/.cipherstash` | all four `CS_*` passed explicitly | +| Typing | signatures derived from the schema | schema-aware, but not the full typed client | +| `.audit()` | chainable on operations | **not available** | +| `.withLockContext()` | chainable on operations | **not available** — see below | +| `bulkEncrypt` shape | `(plaintexts, { table, column })`, `{ id, plaintext }` envelopes | per-item `{ plaintext, table, column }`, plain index-aligned array | +| `decryptModel` / `bulkDecryptModels` | `(model, table, lockContext?)`, **plus** a table-less `(model)` overload for legacy rows | `(model, table)` only — the table is **required** | +| Module format | ESM + CJS | **ESM only** | + +**Authentication and key binding are two different things**, and conflating +them is the standard mistake. Identity-bound encryption needs both: + +1. **Authenticate as the user** — build an `OidcFederationStrategy` (or + `AccessKeyStrategy` for service-to-service) and pass it as + `config.authStrategy`. The client then acts as that user for its lifetime. + **Available on this entry**, and shown below. +2. **Bind the data key to a claim** — chain `.withLockContext({ identityClaim })` + on the operation. *This* is what changes key derivation. **Not available on + this entry** ([#797](https://github.com/cipherstash/stack/issues/797)). + +> [!IMPORTANT] +> **An auth strategy alone does not produce identity-bound data.** It decides +> *who the client is*; a lock context decides *which key the value is encrypted +> under*. Only the first exists here, so on this entry today: +> +> - Values you write are encrypted under the **workspace key**, not the user's +> — even with a per-user `authStrategy`. +> - You **cannot read** anything the native entry wrote under a lock context, +> because decrypt needs the same context. That is a silent split in what the +> two entries can read, on top of the schema incompatibility below. +> +> If a value must be bound to an end-user claim, encrypt and decrypt it on the +> native entry. Don't reach for `as any` to force a lock context through here — +> there is nothing on the other side to receive it. + +The strategy replaced the old per-operation token ceremony +(`LockContext.identify()`, deprecated) — it did **not** replace the lock +context, and the native entry still chains `.withLockContext()` for that. + +```ts +import { Encryption, OidcFederationStrategy } from '@cipherstash/stack/wasm-inline' + +// `create` returns a Result — unwrap it. Passing the Result itself as +// `authStrategy` is the easy mistake, and it fails opaquely later. +const strategy = OidcFederationStrategy.create( + workspaceCrn, // 'crn:<region>:<workspace-id>' + () => getUserJwt(req), // called on every re-federation — Clerk, Supabase Auth, … +) +if (strategy.failure) throw new Error(strategy.failure.error.message) + +const client = await Encryption({ + schemas: [users], + config: { authStrategy: strategy.data, clientId, clientKey }, +}) + +// Authenticated as the end user — but the value is still encrypted under the +// workspace key. There is no `.withLockContext()` on this entry to bind it. +const enc = await client.encrypt('alice@example.com', { + table: users, + column: users.email, +}) +if (enc.failure) throw new Error(enc.failure.message) +``` + +**On the native entry, where lock contexts do exist, the same claim must be +supplied on decrypt.** A value encrypted under a lock context and decrypted +without one — or under a different claim — does not come back. This is the +single most common identity-aware encryption bug, and it does not surface as a +key error; it surfaces as a failed decrypt. It is also why an edge function +cannot read what a lock-context-using Node service wrote. + +`AccessKeyStrategy.create(workspaceCrn, accessKey)` has the same +Result-returning shape, for service-to-service use with a custom token store. +When you pass an auth strategy, do **not** also pass `config.accessKey` — they +are mutually exclusive and the client rejects the combination. + +Construct a client **per request** when using a user-scoped strategy — a +module-level client would bind whichever user happened to arrive first. + +### The bulk shape differs — don't copy the native form + +```ts +// WASM entry: each entry carries its own table and column. +const out = await client.bulkEncrypt([ + { plaintext: 'a@example.com', table: users, column: users.email }, + { plaintext: 'b@example.com', table: users, column: users.email }, +]) +if (out.failure) throw new Error(out.failure.message) +// out.data is index-aligned; a null/undefined plaintext yields null at that index. +``` + +The model helpers (`encryptModel` / `decryptModel` and their bulk forms) *are* +present on this entry, and the traversal is the same one the native entry runs +— declared columns encrypted by JS property name, everything else passing +through, one ZeroKMS round trip per call. + +**The decrypt side has no table-less form.** Both entries take the table as the +second argument, and on both that is the form to prefer. What the native client +*also* offers is a one-arg `decryptModel(model)` overload — the read path for +rows whose table isn't in the schema set, legacy EQL v2 above all, at the cost +of `Date` reconstruction and a precise plaintext shape. This entry has no such +overload: `decryptModel(model, table)` and `bulkDecryptModels(models, table)` +**require** the table, because they resolve date fields from a per-table map +built at client construction. Omitting it throws rather than returning a +`{ failure }`, and a table the client was not initialized with is a defined +failure: + +```ts +const rows = await client.bulkDecryptModels(encryptedRows, users) +if (rows.failure) throw new Error(rows.failure.message) +``` + +A wrapper written against the native signature will therefore compile against +one entry and break on the other — one more reason to author against exactly +one entry (see below). + +## Schema Modules Do Not Cross Entries + +A schema authored with `@cipherstash/stack/v3` **will not typecheck** +against the WASM entry's `Encryption`, and the reverse fails too: + +```text +Type 'EncryptedTextSearchColumn' is not assignable to type 'AnyEncryptedV3Column'. + Types have separate declarations of a private property 'columnName'. +``` + +The two entries ship independent type bundles, and the column classes carry +private fields — which TypeScript compares **nominally**. The declarations are +identical in shape but not the same declaration, so assignment is rejected in +both directions. + +It works fine at runtime, which is the trap: the tempting fix is +`as never` / `as any` on the schema, which silences a real signal and will +keep silencing it after a genuine schema mismatch appears. + +**Author the schema module against exactly one entry, and use that entry's +client with it.** For a project whose encryption runs on the edge, that means +importing `encryptedTable` and `types` from `@cipherstash/stack/wasm-inline` +in the shared schema module: + +```ts +// schema.ts — the single source of truth for this project's schema +import { encryptedTable, types } from '@cipherstash/stack/wasm-inline' + +export const users = encryptedTable('users', { + email: types.TextSearch('email'), + ssn: types.TextEq('ssn'), +}) +``` + +Node-side code that imports this module must then also build its client from +`@cipherstash/stack/wasm-inline` (which runs on Node perfectly well, just with +the WASM engine rather than the native one) and must be ESM. + +If a project genuinely needs the native client on the server *and* the WASM +client on the edge, keep two schema modules and treat their agreement as +something to test, not something the type system will enforce for you. Column +names and domains must match exactly — they are what the database and the +stored payload's `i` identifier are keyed by. + +## Querying from the Edge + +Edge functions rarely have an ORM, so encrypted search is usually hand-written +SQL over `pg` or `postgres-js`. Mint the search needle with `encryptQuery`, +then bind it as a typed parameter: + +```ts +const term = await client.encryptQuery('alice@example.com', { + table: users, column: users.email, queryType: 'equality', +}) +if (term.failure) throw new Error(term.failure.message) + +// postgres-js — bind the unwrapped term, not the Result +const rows = await sql` + SELECT * FROM users + WHERE email = ${sql.json(term.data)}::jsonb::eql_v3.query_text_eq` +``` + +The predicate forms, the per-driver binding rules, and the query-domain names +are the subject of `stash-postgres` — read it before writing the first query. The +two rules that bite immediately: the operand must be **cast to the column's +`eql_v3.query_*` domain**, and on `postgres-js` payloads must be bound with +`sql.json(...)`, never pre-stringified. + +## Troubleshooting + +**`Dynamic require of "..." is not supported` / a native `.node` file in the bundle** +— the native entry got imported. Check every import path resolves to +`@cipherstash/stack/wasm-inline`, including transitive ones from your own +shared modules. + +**`require(...) is not a function` / the specifier won't resolve in CJS** — this +entry is ESM-only. Move the consumer to ESM. + +**Missing `CS_*` at runtime** — the secret store was never populated, or the +function was served without `--env-file`. Validate all four at handler entry +and return an actionable error rather than letting client construction fail +opaquely; the example in `examples/supabase-worker` does exactly this. + +**Encryption works, search returns zero rows** — the credential-identity rule +above. Second most likely: a missing index (see `stash-indexing`) makes it +slow, not empty, so empty results point at credentials or an untyped operand +(`stash-postgres`). + +**Search needle rejected** — free-text needles must be at least 3 characters; +shorter ones tokenize to nothing. + +**Cold-start latency** — the inlined WASM module is compiled on first use. +Construct the client at module scope when the auth strategy is not +user-scoped, so it is reused across invocations on a warm isolate. + +## Reference + +- `stash-postgres` — the raw-SQL predicate cookbook and driver binding rules. +- `stash-encryption` — schema authoring, the `types.*` domain catalog, and the + rollout/cutover lifecycle. +- `stash-cli` — `stash env`, `stash eql install`, `stash encrypt backfill`. +- `stash-indexing` — indexes on encrypted columns (the DDL is the same + wherever the app runs). +- `stash-supabase` — the PostgREST wrapper, for Supabase apps that are not + writing raw SQL. +- Working example: `examples/supabase-worker` in the `cipherstash/stack` repo. +- Bundling guide: https://cipherstash.com/docs/stack/deploy/bundling diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 6865ebd2f..e8016b013 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -190,7 +190,7 @@ The SDK never logs plaintext data. | `@cipherstash/stack/types` | All TypeScript types | | `@cipherstash/stack-drizzle` | Drizzle ORM integration for EQL v3 schemas — the package root, EQL v3 only (see the `stash-drizzle` skill) | | `@cipherstash/stack-supabase` | `encryptedSupabase` wrapper for Supabase — EQL v3 only (see the `stash-supabase` skill) | -| `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. **EQL v3 only** — `Encryption()` here rejects a v2 schema, and its operations return plain Results with no `.audit()` or `.withLockContext()` chaining. | +| `@cipherstash/stack/wasm-inline` | The **edge** entry — Deno, Bun, Cloudflare Workers, Supabase Edge Functions. Its own `Encryption` factory plus its own copy of the v3 authoring surface, `EncryptionErrorTypes`, and the WASM build of protect-ffi inlined into the bundle. No native binding, so no bundler externalisation needed. **EQL v3 only** — `Encryption()` here rejects a v2 schema, and its operations return plain Results with no `.audit()` or `.withLockContext()` chaining, so **values written here cannot be identity-bound** and it cannot read what the native entry wrote under a lock context. **ESM-only, and its schema types do not interchange with the other entries'** — see the `stash-edge` skill. | | `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — encrypt/write is **EQL v3 only** (`types.*`); decrypt still reads existing v2 items, on the native entry only. See the `stash-dynamodb` skill | | `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | @@ -446,6 +446,9 @@ for (const item of decrypted.data) { > [!IMPORTANT] > The `client` below is a **different client** from the one used everywhere else in this skill. The edge entry has its own `Encryption` factory — the native `Encryption` client's `bulkEncrypt` takes `(plaintexts, { table, column })` and will fail at runtime if given the per-item shape below. Construct the WASM client explicitly: +> [!IMPORTANT] +> **The schema is not shareable between entries either.** Note that `encryptedTable` and `types` are imported *from the WASM entry* below, not from `@cipherstash/stack/eql/v3`. The entries ship independent type bundles whose column classes carry private fields, so TypeScript compares them **nominally**: a schema authored on one entry is rejected by the other's client, in both directions (`Types have separate declarations of a private property 'columnName'`). It works at runtime, which makes `as any` the tempting fix — don't. Author the shared schema module against exactly one entry and build that entry's client from it. See the `stash-edge` skill. + ```typescript // Deno / Workers / Supabase Edge Functions — note the import path import { Encryption, encryptedTable, types } from "@cipherstash/stack/wasm-inline" @@ -582,7 +585,7 @@ All values in the array must be non-null. ### On the Wire: Operators and Ordering -Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. These same extractor expressions are also what you index — the functional-index recipes are in the `stash-indexing` skill. +Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. These same extractor expressions are also what you index — the functional-index recipes are in the `stash-indexing` skill. Writing the SQL by hand instead (no ORM, `pg` / `postgres-js`)? The predicate matrix, the `eql_v3.query_*` operand casts, and the per-driver parameter-binding rules are in the `stash-postgres` skill; running on Deno / Workers / Supabase Edge Functions, the `stash-edge` skill. ## Encrypted JSON (`types.Json`) diff --git a/skills/stash-indexing/SKILL.md b/skills/stash-indexing/SKILL.md index 5e7b46cc9..d7f7d981e 100644 --- a/skills/stash-indexing/SKILL.md +++ b/skills/stash-indexing/SKILL.md @@ -257,7 +257,7 @@ Index not being used: - **Drizzle** — `encryptedIndexes(t)` from `@cipherstash/stack-drizzle` derives the recommended indexes for every encrypted column in the table, or declare individual expression indexes in the schema DSL. See `stash-drizzle` § Indexing Encrypted Columns. - **Prisma Next** — Prisma's schema language cannot express functional indexes; the DDL goes in a migration in the adapter's flow. See `stash-prisma-next`. - **Supabase** — a `supabase/migrations/` file; no superuser needed (see above). See `stash-supabase`. -- **Raw SQL / plain PostgreSQL** — the recipes in this skill, in whatever migration tool owns the schema. Never ad-hoc in production. +- **Raw SQL / plain PostgreSQL** — the recipes in this skill, in whatever migration tool owns the schema. Never ad-hoc in production. The predicates those indexes serve are in `stash-postgres`. ## When to Create Indexes During an Encryption Rollout @@ -271,3 +271,5 @@ Index not being used: - `stash-encryption` — the `types.*` domain catalog, wire-format operators and ordering, and the rollout/cutover lifecycle. - `stash-cli` — `stash eql install`, `stash db validate` (its "No indexes on an encrypted column" Info finding is resolved by this skill), `stash encrypt backfill` / `cutover`. - `stash-drizzle`, `stash-supabase`, `stash-prisma-next` — per-integration query patterns; index DDL placement per the section above. +- `stash-postgres` — the hand-written predicate forms these indexes serve (`pg` / `postgres-js`, no ORM). +- `stash-edge` — the WASM entry, for apps whose queries run on Deno / Workers / Supabase Edge Functions. diff --git a/skills/stash-postgres/SKILL.md b/skills/stash-postgres/SKILL.md new file mode 100644 index 000000000..8e3260b32 --- /dev/null +++ b/skills/stash-postgres/SKILL.md @@ -0,0 +1,479 @@ +--- +name: stash-postgres +description: Query EQL v3 encrypted columns from hand-written Postgres SQL over `pg` (node-postgres) or `postgres` (postgres-js) — no ORM. Covers the column-domain-to-query-domain operator matrix (which of `=`, `<>`, `<`, `<=`, `>`, `>=`, `@@`, `@>` each encrypted domain accepts), minting search needles with `encryptQuery`, the per-driver parameter-binding rules for encrypted payloads, and the double-encoding failure that trips the domain CHECK with a message naming neither JSON nor encoding. Use when writing INSERT/SELECT against an encrypted column without an ORM, when a predicate returns zero rows or raises "operator does not exist", or when a domain CHECK constraint rejects an encrypted value on write. Assumes a direct Postgres connection with client-side encryption — CipherStash Proxy encrypts on the wire and needs none of this. +--- + +# Raw Postgres SQL Against Encrypted Columns (EQL v3) + +An EQL v3 encrypted column is a **Postgres domain over `jsonb`** (`public.eql_v3_text_search`, +`public.eql_v3_bigint_ord`, …). Reading and writing it from raw SQL is two rules: + +1. **Writing** — bind the `Encrypted` payload your client produced as a + *JSON object* parameter. How you do that differs per driver, and getting it + wrong trips a domain CHECK with an unhelpful message. +2. **Querying** — never send a plaintext. Mint a **query term** with + `encryptQuery` and cast it to the column's matching `eql_v3.query_*` + domain. That cast is what selects the right operator overload: leave the + operand as bare `jsonb` and you get a *different* overload, one that + expects a full storage envelope. + +This covers the `pg` and `postgres-js` drivers with no ORM — plain Node +services, Hono, edge functions. If you use Drizzle, Prisma Next, or the +Supabase client, those integrations emit correct operands for you: see +`stash-drizzle`, `stash-prisma-next`, `stash-supabase` instead. + +> **Using CipherStash Proxy? None of this applies.** This skill assumes the +> app connects to Postgres **directly** and encrypts **client-side**: Stack +> mints the payloads and the query terms, and your SQL carries them. Through +> [CipherStash Proxy](https://github.com/cipherstash/proxy) the split is the +> opposite — you write *plaintext* SQL and Proxy encrypts on write and +> decrypts on read, so there is no `encryptQuery`, no `eql_v3.query_*` cast, +> and no payload to bind. +> +> Proxy's schema lifecycle — `stash db push` into `eql_v2_configuration`, +> promoted at cutover — belongs to EQL **v2**. This skill is EQL v3, where a +> column's encryption config lives in its own domain and there is no +> configuration table to push. The `stash` CLI does not track a +> Proxy-versus-SDK choice; it targets the direct-connection path this skill +> describes. + +## When to Use This Skill + +- Writing `INSERT` / `UPDATE` / `SELECT` against an encrypted column by hand. +- A predicate returns zero rows, or errors with `operator does not exist`. +- A write fails with `value for domain eql_v3_… violates check constraint`. +- Choosing the right operator for a column's domain. +- Ordering, ranging, or searching inside an encrypted JSON document. + +## The Two Halves + +Assume `users.email` is declared in the database as `public.eql_v3_text_eq` — +a Postgres **domain over `jsonb`**. The domain is the authority: it is what +the column actually *is*, what the CHECK constraint enforces, and what decides +which operators the column admits. + +Everything else is a mapping onto that domain: + +- **`types.TextEq('email')`** — the schema factory from + `@cipherstash/stack/eql/v3` that *declares* the column. A TypeScript builder, + not the column's type. +- **`TextEq` / `TextEqQuery`** — the wire shapes, exported as TypeScript types + by `@cipherstash/eql`. These and the JSON Schemas are generated from the Rust + `eql-bindings` crate, and the SQL bundle is built from the same commit, so + the payload shape and the domain CHECK cannot drift apart. Each type's doc + comment names the domain it maps to and the operators that domain admits. +- **`eql_v3.query_text_eq`** — the *query* domain, also a database type, that a + search needle is cast to. + +That `TextEq` / `TextEqQuery` pairing is precisely the split this section is +about. The domain names below follow the column, so a +`public.eql_v3_text_search` column would use `eql_v3.query_text_search` in +exactly the same places. + +```ts +// 1. Encrypt for storage — the payload includes the ciphertext (`c`). +const enc = await client.encrypt('alice@example.com', { table: users, column: users.email }) +if (enc.failure) throw new Error(enc.failure.message) +await sql`INSERT INTO users (email) VALUES (${sql.json(enc.data)})` + +// 2. Mint a search needle — CIPHERTEXT-FREE, terms only. +const term = await client.encryptQuery('alice@example.com', { + table: users, column: users.email, queryType: 'equality', +}) +if (term.failure) throw new Error(term.failure.message) +const rows = await sql` + SELECT * FROM users WHERE email = ${sql.json(term.data)}::jsonb::eql_v3.query_text_eq` +``` + +Storage payloads and query terms are **different shapes with different +domains**. A storage payload carries `c` (the ciphertext); a query term +deliberately omits it and the `eql_v3.query_*` CHECKs *require* its absence. +Binding a storage payload where a query term belongs fails the CHECK, and vice +versa. + +`queryType` is one of `'equality'`, `'freeTextSearch'`, `'orderAndRange'`, +`'searchableJson'`. Omit it only for single-index columns (`types.TextEq`); +be explicit on multi-index domains like `types.TextSearch`. + +## Naming: Column Domain → Query Domain + +Strip `public.`, insert `query_`, move to the `eql_v3` schema: + +```text +public.eql_v3_text_eq → eql_v3.query_text_eq +public.eql_v3_text_search → eql_v3.query_text_search +public.eql_v3_bigint_ord → eql_v3.query_bigint_ord +public.eql_v3_timestamp_ord → eql_v3.query_timestamp_ord +``` + +**One irregular case:** `types.Json` builds `public.eql_v3_json_search`, but +its query domain is `eql_v3.query_json` — not `query_json_search`. + +| Schema factory | Column domain (`public.`) | Query domain (`eql_v3.`) | +|---|---|---| +| `types.TextEq` | `eql_v3_text_eq` | `query_text_eq` | +| `types.TextMatch` | `eql_v3_text_match` | `query_text_match` | +| `types.TextOrd` | `eql_v3_text_ord` | `query_text_ord` | +| `types.TextSearch` | `eql_v3_text_search` | `query_text_search` | +| `types.<N>Eq` | `eql_v3_<n>_eq` | `query_<n>_eq` | +| `types.<N>Ord` | `eql_v3_<n>_ord` | `query_<n>_ord` | +| `types.<N>OrdOre` | `eql_v3_<n>_ord_ore` | `query_<n>_ord_ore` | +| `types.Json` | `eql_v3_json_search` | **`query_json`** | +| `types.Text`, `types.<N>`, `types.Boolean` | `eql_v3_text` / `eql_v3_<n>` / `eql_v3_boolean` | **none — storage only** | + +`<N>` ranges over `Integer`, `Smallint`, `Bigint`, `Numeric`, `Real`, +`Double`, `Date`, `Timestamp`. The storage-only domains carry no query terms +by design — there is no query domain and nothing to search server-side. + +## The Predicate Matrix + +Which operators each column domain accepts against its query domain. Anything +not listed does not exist as an encrypted operator. + +> **Confirm types against EQL before relying on them.** This table — and every +> other domain/operator table in this skill — is a snapshot of a *versioned* +> surface that is defined elsewhere. Do not treat it as the last word. Consult +> EQL for the precise current types, in the order given under **Where this +> surface is defined** below; the first two checks need nothing but +> `node_modules`. + +| Column domain | Operators | Query domain operand | +|---|---|---| +| `eql_v3_<n>_eq`, `eql_v3_text_eq` | `=` `<>` | `query_<n>_eq` / `query_text_eq` | +| `eql_v3_<n>_ord`, `eql_v3_text_ord` | `=` `<>` `<` `<=` `>` `>=` | `query_<n>_ord` / `query_text_ord` | +| `eql_v3_<n>_ord_ore`, `eql_v3_text_ord_ore` | `=` `<>` `<` `<=` `>` `>=` | `query_<n>_ord_ore` / `query_text_ord_ore` | +| `eql_v3_text_match` | `@@` | `query_text_match` | +| `eql_v3_text_search` | `=` `<>` `<` `<=` `>` `>=` `@@` | `query_text_search` | +| `eql_v3_json_search` | `@>` | `query_json` | +| `eql_v3_json_entry` (from `col -> 'selector'`) | `=` `<>` `<` `<=` `>` `>=` | any `query_<n>_ord` / `query_text_ord` / `query_text_search` | + +Note what is **absent**: there is no `<` on an `_eq` domain, and no `@@` +outside the match-capable text domains. Asking for one raises +`operator does not exist` — which is the good failure. The bad failure is +leaving the operand as bare `jsonb` (see [Traps](#traps)). + +Every operator has a function twin, useful when an operator is awkward to +emit: `eql_v3.eq(col, term)`, `eql_v3.matches(col, term)`, and the comparison +functions. `col = term` and `eql_v3.eq(col, term)` are equivalent. + +### Where this surface is defined + +Nothing in the matrix above is authored by the client library. The domains, +operators, CHECKs, and extractor functions all come from the **EQL SQL +bundle** that `stash eql install` applies — published as the `@cipherstash/eql` +package and developed at +[`cipherstash/encrypt-query-language`](https://github.com/cipherstash/encrypt-query-language). +The CLI pins an exact version, so a database is only ever on one bundle. + +That makes every table in this skill a snapshot of a *versioned* surface. +**Go to EQL for the precise current types rather than trusting these tables +alone** — in this order: + +1. **The EQL skill**, when it is installed. It ships from + `encrypt-query-language` alongside the bundle it documents, so it tracks the + version you actually have, and it is authoritative for domain names, query + domains, operators, and payload shapes. +2. **The generated TypeScript types** in `@cipherstash/eql`. Every per-domain + type's doc comment names its domain and that domain's operators — the + `TextEqQuery` type, for instance, is documented as the + `eql_v3.query_text_eq` equality query operand admitting `=` and `<>`. They + are generated from the same Rust `eql-bindings` commit as the SQL bundle, + so the two cannot disagree. +3. **The install SQL**, shipped at + `@cipherstash/eql/dist/sql/cipherstash-encrypt.sql` (under `node_modules`, + wherever your package manager resolves it). Its `CREATE OPERATOR` + statements are the last word on which overloads exist — each names its + `LEFTARG`, `RIGHTARG`, and implementing `FUNCTION`. +4. **The database itself**, which is the runtime truth and the right check + when a query is failing right now: + + ```sql + SELECT eql_v3.version(); -- which bundle is actually installed + ``` + +The `stash` CLI depends on `@cipherstash/eql` and pins an exact version, so +2 and 3 are available in any project that has the CLI installed, with no +database connection required. + +If an operator is absent from all of these, that is an upstream question, not +a client-library one — the operator set lives in `encrypt-query-language`. + +## Binding Parameters: The Driver Rules + +**This differs between drivers.** Both encrypted payloads and query terms are +plain JS objects, and the two drivers disagree about how to put a JS object +into a `jsonb`-backed domain. + +### `postgres` (postgres-js) — always `sql.json(...)` + +| Binding form | `INSERT` into a domain column | Query operand with `::jsonb::eql_v3.query_*` | +|---|---|---| +| `${sql.json(payload)}` | ✅ | ✅ | +| `${payload}` (bare object) | ❌ `invalid input syntax for type json` | ✅ | +| `${JSON.stringify(payload)}::jsonb` | ❌ CHECK violation | ❌ CHECK violation | + +**Use `sql.json(...)` in both positions** — it is the only form that works in +both, so there is no reason to track which position you are in. + +```ts +await sql`INSERT INTO users (email) VALUES (${sql.json(enc.data)})` +await sql`SELECT * FROM users + WHERE email = ${sql.json(term.data)}::jsonb::eql_v3.query_text_eq` +``` + +### `pg` (node-postgres) — pass the object + +node-postgres serialises a JS object to JSON exactly once, so all three forms +happen to work. Pass the object and let the driver do it: + +```ts +await client.query('INSERT INTO users (email) VALUES ($1)', [enc.data]) +await client.query( + 'SELECT * FROM users WHERE email = $1::eql_v3.query_text_eq', [term.data]) +``` + +Do not pre-stringify even though `pg` tolerates it — it is the one habit that +silently breaks if the project ever moves to `postgres-js`. + +### The double-encoding failure, precisely + +`${JSON.stringify(payload)}::jsonb` on postgres-js produces: + +```text +value for domain eql_v3_text_search violates check constraint "eql_v3_text_search_check" +``` + +The message names neither JSON nor encoding, which is why this one costs an +afternoon. What happened: the explicit `::jsonb` makes postgres-js infer a +`jsonb` parameter, so it JSON-encodes the value — which was *already* a JSON +string. The result is a jsonb **string scalar**, not an object: + +```sql +SELECT jsonb_typeof($1::jsonb) -- 'string', not 'object' +``` + +Every EQL domain CHECK opens with `jsonb_typeof(VALUE) = 'object'`, so it +fails on the very first clause. Diagnose any CHECK-violation-on-write by +running `jsonb_typeof` on the parameter; `'string'` means double-encoded. + +## Query Recipes + +Assume `sql` is a postgres-js tag; for `pg` use numbered placeholders as above. + +**The recipes below omit the `Result` guard for brevity — your code must not.** +`encryptQuery` returns `{ data } | { failure }`, so reading `.data` without +first checking `.failure` binds `undefined` into the query, which fails as a +domain CHECK violation rather than as the encryption error it actually is. +Every recipe should be read as though it were written: + +```ts +const term = await client.encryptQuery(/* … */) +if (term.failure) throw new Error(term.failure.message) +``` + +### Equality + +```ts +const term = await client.encryptQuery(email, { + table: users, column: users.email, queryType: 'equality', +}) +await sql`SELECT * FROM users + WHERE email = ${sql.json(term.data)}::jsonb::eql_v3.query_text_eq` +``` + +On a `types.TextSearch` column the cast is `::eql_v3.query_text_search` — the +query domain always matches the *column's* domain, not the query type. + +### Free-text match + +```ts +// `bio` is a types.TextSearch column here; on a types.TextMatch column the +// cast is ::eql_v3.query_text_match. +const term = await client.encryptQuery('needle', { + table: users, column: users.bio, queryType: 'freeTextSearch', +}) +await sql`SELECT * FROM users + WHERE bio @@ ${sql.json(term.data)}::jsonb::eql_v3.query_text_search` +``` + +Match is **one-sided**: a hit may be a false positive, a miss never is. Filter +client-side after decryption if exactness matters — and never build a negated +match (`NOT (bio @@ …)`), which would drop true rows. Needles must be at least +3 characters; shorter ones tokenize to nothing and are rejected. + +### Range and ordering + +```ts +const term = await client.encryptQuery(new Date('2026-01-01'), { + table: events, column: events.createdAt, queryType: 'orderAndRange', +}) +await sql`SELECT * FROM events + WHERE created_at >= ${sql.json(term.data)}::jsonb::eql_v3.query_timestamp_ord + ORDER BY eql_v3.ord_term(created_at) DESC + LIMIT 20` +``` + +**`ORDER BY` must use the extractor form.** `ORDER BY created_at` sorts the +raw encrypted payload — which is neither meaningful nor index-backed. Sorting +on `eql_v3.ord_term(col)` is both. Ordering is available on `_ord`, +`_ord_ore`, and `text_search` columns; use `ord_term_ore` for `_ord_ore`. + +### Encrypted JSON — containment + +```ts +const needle = await client.encryptQuery({ role: 'admin' }, { + table: users, column: users.prefs, queryType: 'searchableJson', +}) +await sql`SELECT * FROM users + WHERE prefs @> ${sql.json(needle.data)}::jsonb::eql_v3.query_json` +``` + +An **object** value produces a containment needle. Note the containment needle +is a bare `{ sv: [...] }` shape with no version field — unlike the scalar +terms, which are full v3 envelopes. Bind it the same way regardless. + +### Encrypted JSON — field selector + +A **string** value produces a JSONPath selector, and v3 has no encrypted-selector +envelope: `encryptQuery` returns the **bare selector-hash string**. Bind it as +the plain text argument of `->` / `->>`, with no domain cast: + +```ts +const sel = await client.encryptQuery('$.role', { + table: users, column: users.prefs, queryType: 'searchableJson', +}) +await sql`SELECT prefs -> ${sel.data} FROM users` +``` + +The extracted value is an `eql_v3_json_entry`, which accepts the ordering +operators — so a field inside an encrypted document can be ranged and ordered: + +```ts +await sql`SELECT * FROM orders + WHERE data -> ${sel.data} >= ${sql.json(term.data)}::jsonb::eql_v3.query_integer_ord + ORDER BY eql_v3.ord_term(data -> ${sel.data})` +``` + +Field-level `=` between extracted entries is **not** supported (an extracted +entry carries no value selector) — use document containment for exact field +equality. + +## Reading Rows Back + +`SELECT` returns the stored payload as an object; hand it straight to +`decrypt` — do not `JSON.parse` it, and do not cast it to `::jsonb` in the +query (see the projection trap below). + +```ts +const [row] = await sql`SELECT id, email FROM users WHERE id = ${id}` +const dec = await client.decrypt(row.email) +if (dec.failure) throw new Error(dec.failure.message) +``` + +For whole rows, `decryptModel` / `bulkDecryptModels` walk the schema and +decrypt every declared column in one ZeroKMS round trip. They match by **JS +property name**, so a raw `SELECT` returning snake_case DB column names will +not match a schema keyed by camelCase properties — alias in the query +(`SELECT last_login AS "lastLogin"`) or decrypt the columns individually. + +## Traps + +**A bare `::jsonb` operand picks a different operator, not a missing one.** +EQL also defines overloads with `jsonb` on the right — and they coerce that +operand to the **storage** domain: + +```sql +-- what `col = $1::jsonb` actually resolves to: +eql_v3.eq_term(a) = eql_v3.eq_term(b::public.eql_v3_text_search) +``` + +The storage domain's CHECK requires the ciphertext key `c`, which query terms +deliberately omit — so binding a query term without the domain cast raises a +CHECK violation rather than doing what you meant. Those overloads exist so you +can compare against a *full storage envelope* (an already-encrypted value). +For a needle from `encryptQuery`, always cast to the `eql_v3.query_*` domain. + +**Cast to the `query_*` domain, not the column domain.** `$1::public.eql_v3_text_eq` +fails for the same reason — the column domain's CHECK requires `c`. + +**The `value::jsonb` projection trap.** `SELECT email::jsonb … ORDER BY email` +folds the cast into the scan and sorts on `(email)::jsonb`, matching no index. +Project the column raw. + +**`GROUP BY` / `DISTINCT` on the raw column** hashes the whole encrypted +payload (1–2 KB per row) and spills. Group on the extractor — +`GROUP BY eql_v3.eq_term(email)` — which is small and deterministic. + +**Predicates are not indexes.** Everything here works without an index and +sequential-scans. Adding the functional index over the extractor is a separate +step — see `stash-indexing`. + +**Every writer needs the same credentials.** Index terms derive from the +ZeroKMS client key, so rows written by a client with different `CS_*` +credentials decrypt correctly but never match a query — silently. This +includes `stash encrypt backfill` and seed scripts. See `stash-edge` § +The Credential-Identity Rule. + +## Troubleshooting + +**`operator does not exist: public.eql_v3_… = eql_v3.query_…`** — the domain +pair has no such operator. Check the [matrix](#the-predicate-matrix): the +column's domain may not support that predicate (e.g. `<` on an `_eq` column), +or the query domain does not match the column's domain. If the matrix says +the operator *should* exist, check the installed bundle version +(`SELECT eql_v3.version()`) — an older EQL install can predate an overload +documented here. See [Where this surface is defined](#where-this-surface-is-defined). + +**`value for domain … violates check constraint`** on write — double-encoded +payload; run `SELECT jsonb_typeof($1::jsonb)` and see +[above](#the-double-encoding-failure-precisely). On a *query* operand, the +same error usually means a storage payload (with `c`) was bound where a query +term belongs. + +**Zero rows, no error** — in order of likelihood: (1) the rows were written +under different `CS_*` credentials (see the trap above — this is the common +one, and it is completely silent); (2) the column's domain does not carry the +term the predicate needs (a `types.Text` column carries none); (3) a +free-text needle under 3 characters tokenized to nothing. A *missing* domain +cast raises an error rather than returning zero rows, so it is not a candidate +here. + +**Slow but correct** — no index. See `stash-indexing`; confirm with +`EXPLAIN (COSTS OFF)` that the plan shows an `Index Cond` on the extractor +rather than a `Seq Scan`. + +**Checking what a column actually is**, when the schema and the database may +have drifted: + +```sql +SELECT eql_v3.version(); -- which bundle defines the operators available + +SELECT column_name, domain_schema, domain_name + FROM information_schema.columns + WHERE table_name = 'users' AND domain_name LIKE 'eql_v3%'; +``` + +## Reference + +- `stash-encryption` — schema authoring, the `types.*` catalog, `encryptQuery` + and the client API, the rollout/cutover lifecycle. +- `stash-indexing` — functional indexes over the term extractors, and the + `EXPLAIN` checklist. +- `stash-edge` — the WASM entry, `CS_*` credentials, and the + credential-identity rule. +- `stash-cli` — `stash eql install`, `stash db validate`, `stash encrypt backfill`. + +Upstream: + +- **The EQL skill**, shipped from + [`cipherstash/encrypt-query-language`](https://github.com/cipherstash/encrypt-query-language) + — the authority for the current domain, operator, and payload-shape surface. + Consult it in preference to the tables here whenever it is installed; those + tables are a snapshot, it tracks the bundle. +- [`cipherstash/encrypt-query-language`](https://github.com/cipherstash/encrypt-query-language) + — EQL itself: the definition of every domain, operator, CHECK, and extractor + named in this skill, shipped as `@cipherstash/eql`. Operator gaps and + domain-level bugs belong there, not against the client library. +- [`cipherstash/proxy`](https://github.com/cipherstash/proxy) — CipherStash + Proxy, the alternative to this entire skill: plaintext SQL, encryption on + the wire. diff --git a/skills/stash-prisma-next/SKILL.md b/skills/stash-prisma-next/SKILL.md index 632af947c..fef869875 100644 --- a/skills/stash-prisma-next/SKILL.md +++ b/skills/stash-prisma-next/SKILL.md @@ -178,7 +178,10 @@ The `ANALYZE` is part of the recipe — an expression index has no statistics until it runs. Works as a non-superuser role (Supabase included); only the ORE-flavour (`_ord_ore`) ordering opclass is superuser-gated. For the full model — which domains take which index, engagement rules, `EXPLAIN` -verification, rollout timing — see the `stash-indexing` skill. +verification, rollout timing — see the `stash-indexing` skill. For encrypted +predicates written as raw SQL rather than through the `cipherstash:*` +operators — operand casts to `eql_v3.query_*`, per-driver parameter binding — +see the `stash-postgres` skill. In a migration, the recipes ride a raw-SQL operation (`rawSql` from `@prisma-next/postgres/migration`) in the migration's `operations`: diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 4bcec8b35..deacdf19c 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -56,6 +56,14 @@ this is also how **Supabase Edge Functions** get credentials in local dev — `stash env --name edge-dev --write` and pass `--env-file`, or `supabase secrets set` them for deploys. +> **One credential per environment, used by everything that writes.** EQL index +> terms derive from the ZeroKMS client key, so rows written by a client with +> different `CS_*` values decrypt correctly but never match a query — silently. +> That includes `stash encrypt backfill`, seed scripts, and Edge Functions. +> Encryption *inside* an Edge Function (Deno, no native modules) uses the +> `@cipherstash/stack/wasm-inline` entry — see the `stash-edge` skill; SQL +> written by hand in a migration or RPC is covered by `stash-postgres`. + ### 1. Install EQL v3 on the database ```bash From 5aed44772a5e35abc18cd086ca0571a65d38a464 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Tue, 28 Jul 2026 11:42:28 +1000 Subject: [PATCH 099/123] fix(scripts): stop the runners linter exiting 1 when a directory vanishes mid-walk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `lint-no-hardcoded-runners.mjs` walks the packages tree with a recursive `readdir`. The repo's own script self-tests create and delete probe package directories in that same tree while other suites run, and vitest ran those suites in parallel — so the walk could enumerate a probe directory and find it gone before recursing into it. Unhandled, that rejects with ENOENT and Node exits 1: the same code the linter uses for "found a hardcoded npx". CI then failed naming nothing, on a tree the identical script had just passed one step earlier, and a re-run went green. The script already draws this distinction for a missing target — "Exit 2 means the linter could not run" — but only at the top level, not one frame down. Observed on three branches before this one (30316404472, 30317589094), which is why it reads as a flake rather than a defect. Two changes: - `walk` treats ENOENT as "nothing here to lint" and skips. A directory that no longer exists cannot contain a violation, and crashing with the violation exit code is the actual bug. - `scripts/vitest.config.mjs` sets `fileParallelism: false`. Several of these suites mutate the real repo tree, so cross-file parallelism is unsound here regardless of this particular pair. Eleven small files; it costs a second. Verified by reproducing the race locally — churning a probe directory while running the linter failed 6/30 before, 0/40 after. One wrinkle worth recording: the first draft of these comments named the probe directories with their `packages/` prefix, which the sibling `lint-no-dead-package-paths` linter correctly flagged as references to directories that do not exist. The names are now written bare. --- scripts/lint-no-hardcoded-runners.mjs | 21 ++++++++++++++++++++- scripts/vitest.config.mjs | 24 +++++++++++++++++++++++- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index 3070963c8..26c6a2548 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -28,7 +28,26 @@ const TARGETS = process.argv.slice(2).length const NPX_TOKEN = /['"`]npx\b|(?:^|[^a-zA-Z0-9_$])npx\s+[@\w-]/ async function* walk(dir) { - const entries = await readdir(dir, { withFileTypes: true }) + // A directory can vanish between the parent's `readdir` and this call. The + // repo's own script self-tests create and delete probe packages under + // `packages/` (`lint-untracked-probe`, `lint-dead-shell-probe` in + // `lint-no-dead-package-paths.test.mjs`) while other suites run, so a walk of + // `packages/` can enumerate one and find it gone a moment later. + // + // Left unhandled this rejects with ENOENT, and Node exits **1** — the same + // code as "found a hardcoded npx". Every caller then reads a crash as a + // violation: CI fails naming a file that is perfectly clean, and re-running + // it passes. The missing-target case deliberately exits 2 for exactly this + // reason ("Exit 2 means the linter could not run"); this is the same + // distinction, one level down. There is nothing to lint in a directory that + // no longer exists, so skip it. + let entries + try { + entries = await readdir(dir, { withFileTypes: true }) + } catch (err) { + if (err?.code === 'ENOENT') return + throw err + } for (const entry of entries) { const full = join(dir, entry.name) if (entry.isDirectory()) { diff --git a/scripts/vitest.config.mjs b/scripts/vitest.config.mjs index 2a918fc98..5db344161 100644 --- a/scripts/vitest.config.mjs +++ b/scripts/vitest.config.mjs @@ -1,4 +1,26 @@ import { defineConfig } from 'vitest/config' + +// `fileParallelism: false` is load-bearing, not a performance knob. +// +// Several of these suites mutate the real repo tree while they run: +// `lint-no-dead-package-paths.test.mjs` creates and deletes probe package +// directories inside the packages tree (`lint-untracked-probe`, +// `lint-dead-shell-probe`), and `lint-no-hardcoded-runners.test.mjs` walks that +// same tree and writes an `allowlist-probe.mjs` into `scripts/`. Run in +// parallel, one suite's scratch state lands inside another's scan — which is +// how a clean tree produced an `ENOENT: scandir` on a probe directory and a CI +// failure that pointed at nothing. +// +// Note the probe names are deliberately written WITHOUT their directory prefix: +// the sibling `lint-no-dead-package-paths` linter flags that shape when the +// directory does not exist, and these only exist mid-test. +// +// These are eleven small files; serialising them costs a second and removes +// the whole class of cross-suite filesystem race. export default defineConfig({ - test: { include: ['scripts/__tests__/**/*.test.mjs'], pool: 'forks' }, + test: { + include: ['scripts/__tests__/**/*.test.mjs'], + pool: 'forks', + fileParallelism: false, + }, }) From 19cff11b85b8814b4098a654246b206da30d809f Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Tue, 28 Jul 2026 20:07:30 +1000 Subject: [PATCH 100/123] feat(cli): remove the EQL v2 migration leaf --- .../cli-v2-cutover-prompt-correction.md | 2 + .changeset/encrypt-lifecycle-mixed-table.md | 2 + .changeset/eql-v3-cli-install.md | 2 + .changeset/eql-v3-sole-docs.md | 2 + .changeset/init-drizzle-eql-v3.md | 7 +- .changeset/init-scaffold-compiles.md | 2 + .changeset/migrate-eql-v3.md | 2 + .changeset/remove-cli-migrate-v2-leaf.md | 8 + .changeset/skills-v3-lifecycle-honesty.md | 2 + .changeset/stash-cli-eql-v3-default.md | 4 + .changeset/stash-cli-skill-refresh.md | 2 + packages/cli/README.md | 97 +- .../scaffold/drizzle.generated.ts | 11 +- .../scaffold/generic.generated.ts | 11 +- packages/cli/scripts/e2e-encrypt.sh | 9 +- packages/cli/scripts/fixtures/seed-users.sql | 4 +- packages/cli/src/__tests__/installer.test.ts | 460 +- .../placeholder-guard-parity.test.ts | 22 +- .../src/__tests__/supabase-migration.test.ts | 479 -- .../cli/src/__tests__/v2-retirement.test.ts | 48 + packages/cli/src/bin/main.ts | 68 +- packages/cli/src/cli/__tests__/help.test.ts | 5 +- packages/cli/src/cli/registry.ts | 105 +- .../find-generated-migration.test.ts | 8 +- .../generate-drizzle-migration.test.ts | 370 - packages/cli/src/commands/db/activate.ts | 73 - .../cli/src/commands/db/config-scaffold.ts | 6 +- packages/cli/src/commands/db/install.ts | 936 +- packages/cli/src/commands/db/push.ts | 177 - .../cli/src/commands/db/rewrite-migrations.ts | 20 +- packages/cli/src/commands/db/status.ts | 3 +- .../cli/src/commands/db/supabase-migration.ts | 133 - packages/cli/src/commands/db/upgrade.ts | 91 +- .../__tests__/context-placeholder.test.ts | 4 +- .../encrypt/__tests__/encrypt-v3.test.ts | 255 +- ...-lifecycle-composition.integration.test.ts | 202 - packages/cli/src/commands/encrypt/backfill.ts | 52 +- packages/cli/src/commands/encrypt/context.ts | 2 +- packages/cli/src/commands/encrypt/cutover.ts | 331 - .../src/commands/encrypt/drizzle-helper.ts | 6 +- packages/cli/src/commands/encrypt/drop.ts | 142 +- .../src/commands/encrypt/lib/resolve-eql.ts | 9 +- packages/cli/src/commands/eql/migration.ts | 53 +- packages/cli/src/commands/impl/index.ts | 1 - .../init/__tests__/init-command.test.ts | 3 - .../commands/init/doctrine/AGENTS-doctrine.md | 2 +- packages/cli/src/commands/init/index.ts | 9 - .../init/lib/__tests__/setup-prompt.test.ts | 59 +- .../cli/src/commands/init/lib/introspect.ts | 3 +- .../cli/src/commands/init/lib/read-context.ts | 8 +- .../cli/src/commands/init/lib/setup-prompt.ts | 25 +- .../src/commands/init/lib/write-context.ts | 3 - .../init/providers/__tests__/drizzle.test.ts | 8 +- .../src/commands/init/providers/drizzle.ts | 2 +- .../init/steps/__tests__/install-eql.test.ts | 7 +- .../__tests__/resolve-proxy-choice.test.ts | 157 - .../src/commands/init/steps/install-eql.ts | 15 +- .../init/steps/resolve-proxy-choice.ts | 63 - packages/cli/src/commands/init/types.ts | 2 - packages/cli/src/commands/init/utils.ts | 22 +- packages/cli/src/commands/plan/index.ts | 1 - .../commands/status/__tests__/status.test.ts | 51 +- packages/cli/src/commands/status/quest.ts | 19 +- packages/cli/src/config/index.ts | 2 +- packages/cli/src/index.ts | 6 +- packages/cli/src/installer/grants.ts | 10 +- packages/cli/src/installer/index.ts | 460 +- ...cipherstash-encrypt-no-operator-family.sql | 7434 ---------------- .../src/sql/cipherstash-encrypt-supabase.sql | 7434 ---------------- packages/cli/src/sql/cipherstash-encrypt.sql | 7644 ----------------- .../cli/tests/e2e/command-help.e2e.test.ts | 3 +- packages/cli/tests/e2e/smoke.e2e.test.ts | 11 +- .../cli/tests/e2e/v2-retirement.e2e.test.ts | 39 + packages/cli/tsup.config.ts | 2 - packages/migrate/README.md | 56 +- .../__tests__/backfill-v3.integration.test.ts | 2 +- .../migrate/src/__tests__/manifest.test.ts | 40 +- .../src/__tests__/v2-retirement.test.ts | 19 + packages/migrate/src/backfill.ts | 4 +- packages/migrate/src/cursor.ts | 5 +- packages/migrate/src/eql.ts | 166 - packages/migrate/src/index.ts | 19 +- packages/migrate/src/install.ts | 10 +- packages/migrate/src/manifest.ts | 9 +- packages/migrate/src/state.ts | 4 +- packages/migrate/src/version.ts | 13 +- skills/stash-cli/SKILL.md | 88 +- skills/stash-drizzle/SKILL.md | 24 +- skills/stash-encryption/SKILL.md | 127 +- skills/stash-indexing/SKILL.md | 4 +- skills/stash-postgres/SKILL.md | 11 +- skills/stash-supabase/SKILL.md | 67 +- 92 files changed, 903 insertions(+), 27497 deletions(-) create mode 100644 .changeset/remove-cli-migrate-v2-leaf.md delete mode 100644 packages/cli/src/__tests__/supabase-migration.test.ts create mode 100644 packages/cli/src/__tests__/v2-retirement.test.ts delete mode 100644 packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts delete mode 100644 packages/cli/src/commands/db/activate.ts delete mode 100644 packages/cli/src/commands/db/push.ts delete mode 100644 packages/cli/src/commands/db/supabase-migration.ts delete mode 100644 packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts delete mode 100644 packages/cli/src/commands/encrypt/cutover.ts delete mode 100644 packages/cli/src/commands/init/steps/__tests__/resolve-proxy-choice.test.ts delete mode 100644 packages/cli/src/commands/init/steps/resolve-proxy-choice.ts delete mode 100644 packages/cli/src/sql/cipherstash-encrypt-no-operator-family.sql delete mode 100644 packages/cli/src/sql/cipherstash-encrypt-supabase.sql delete mode 100644 packages/cli/src/sql/cipherstash-encrypt.sql create mode 100644 packages/cli/tests/e2e/v2-retirement.e2e.test.ts create mode 100644 packages/migrate/src/__tests__/v2-retirement.test.ts delete mode 100644 packages/migrate/src/eql.ts diff --git a/.changeset/cli-v2-cutover-prompt-correction.md b/.changeset/cli-v2-cutover-prompt-correction.md index eb917fe2d..4a90e2f6b 100644 --- a/.changeset/cli-v2-cutover-prompt-correction.md +++ b/.changeset/cli-v2-cutover-prompt-correction.md @@ -11,3 +11,5 @@ that cannot describe the column it just cut over to. The prompt now says explicitly not to use `types.*` for a v2 column, and points at the deprecated `@cipherstash/stack/schema` builders with decryption through `@cipherstash/stack`, which is the actual read path for legacy v2 rows. + +Superseded later in this release: `stash encrypt cutover` and all v2 mutation guidance are removed; legacy v2 remains read-only. diff --git a/.changeset/encrypt-lifecycle-mixed-table.md b/.changeset/encrypt-lifecycle-mixed-table.md index 3b3dab41a..f6b6273bd 100644 --- a/.changeset/encrypt-lifecycle-mixed-table.md +++ b/.changeset/encrypt-lifecycle-mixed-table.md @@ -57,3 +57,5 @@ Two cases DO newly exit 1, both deliberately: unconventionally, where `cutover` previously exited 0 with "not applicable". Re-run `stash encrypt backfill --table T --column C --encrypted-column <name>` to record the pairing. + +Superseded later in this release: no v2 lifecycle can be driven by `stash encrypt`; mixed and pure-v2 state now fail with migration/recovery guidance. diff --git a/.changeset/eql-v3-cli-install.md b/.changeset/eql-v3-cli-install.md index 68f88e443..264f449d9 100644 --- a/.changeset/eql-v3-cli-install.md +++ b/.changeset/eql-v3-cli-install.md @@ -11,3 +11,5 @@ stripped). v3 currently supports the direct install path only — `--drizzle`/`--migration`/`--migrations-dir`/`--latest` are rejected — and the installer keys `isInstalled`/version checks and Supabase grants to the `eql_v3` schema. + +Superseded later in this release: `--eql-version` and the v2 installer are removed; installs and upgrades are v3-only. diff --git a/.changeset/eql-v3-sole-docs.md b/.changeset/eql-v3-sole-docs.md index 540d34a1f..cb21d891e 100644 --- a/.changeset/eql-v3-sole-docs.md +++ b/.changeset/eql-v3-sole-docs.md @@ -24,3 +24,5 @@ than a Legacy section, because EQL v2 is still reachable there: Also corrects the legacy `@cipherstash/drizzle` README's pointer to the removed `@cipherstash/stack/drizzle` subpath (now the separate `@cipherstash/stack-drizzle` package). + +Superseded later in this release: CLI/migrate v2 mutation guidance is removed; only legacy ciphertext reads and status diagnostics remain. diff --git a/.changeset/init-drizzle-eql-v3.md b/.changeset/init-drizzle-eql-v3.md index fac150347..4ae941236 100644 --- a/.changeset/init-drizzle-eql-v3.md +++ b/.changeset/init-drizzle-eql-v3.md @@ -21,6 +21,7 @@ The generated migration also carries the `cs_migrations` tracking schema, so one isn't installed or configured, init now reports EQL as not installed and points at `stash eql migration --drizzle` rather than aborting the run. -The v2 Drizzle path remains available for existing deployments via an explicit -`stash eql install --drizzle --eql-version 2`; that command's error message now -points at the v3 alternative instead of only suggesting `--eql-version 2`. +The final CLI installation and mutation surface is v3-only: the explicit v2 +Drizzle install path is removed. Legacy v2 remains readable and visible in +diagnostics. Generate a checked-in install migration with `stash eql migration +--drizzle`. diff --git a/.changeset/init-scaffold-compiles.md b/.changeset/init-scaffold-compiles.md index f20ae5d4d..acb2a14ff 100644 --- a/.changeset/init-scaffold-compiles.md +++ b/.changeset/init-scaffold-compiles.md @@ -25,3 +25,5 @@ typecheck step, the codegen tests only string-match fragments of the template, and the step test stubs the generator out entirely. Both templates are now committed as fixtures that CI typechecks, pinned byte-for-byte to the generator so they cannot drift. + +Superseded later in this release: the generated guidance no longer references removed `db push` or `encrypt cutover` commands. diff --git a/.changeset/migrate-eql-v3.md b/.changeset/migrate-eql-v3.md index d1e055117..6060e6c9b 100644 --- a/.changeset/migrate-eql-v3.md +++ b/.changeset/migrate-eql-v3.md @@ -58,3 +58,5 @@ right lifecycle, no new flags: The `stash-cli` and `stash-encryption` skills and the `@cipherstash/migrate` README document the two lifecycles (v2: backfill → cutover → drop; v3: backfill → switch-by-name → drop). + +Superseded later in this release: `@cipherstash/migrate` and the CLI now author and mutate v3 only; legacy v2 manifest fields remain readable. diff --git a/.changeset/remove-cli-migrate-v2-leaf.md b/.changeset/remove-cli-migrate-v2-leaf.md new file mode 100644 index 000000000..b88011fa3 --- /dev/null +++ b/.changeset/remove-cli-migrate-v2-leaf.md @@ -0,0 +1,8 @@ +--- +'stash': major +'@cipherstash/migrate': major +--- + +Remove the remaining EQL v2 installation and rollout surface. CLI installs, +upgrades, backfills, and drops now mutate EQL v3 state only, while legacy v2 +status diagnostics and migration-manifest compatibility remain read-only. diff --git a/.changeset/skills-v3-lifecycle-honesty.md b/.changeset/skills-v3-lifecycle-honesty.md index 5808d215e..2c29c1d0e 100644 --- a/.changeset/skills-v3-lifecycle-honesty.md +++ b/.changeset/skills-v3-lifecycle-honesty.md @@ -25,3 +25,5 @@ case, expect the wrong column to be dropped. Skills ship inside the `stash` tarball and are copied into user projects at `stash init`, so this guidance was being installed into customer repos. + +Superseded later in this release: the bundled skills no longer document v2/Proxy mutation commands because those CLI paths are removed. diff --git a/.changeset/stash-cli-eql-v3-default.md b/.changeset/stash-cli-eql-v3-default.md index b19eecc57..b6eb6da19 100644 --- a/.changeset/stash-cli-eql-v3-default.md +++ b/.changeset/stash-cli-eql-v3-default.md @@ -8,3 +8,7 @@ Default EQL to v3 and stop the CLI recommending `stash db push` (#585). - **`stash db push` is no longer recommended in CLI output.** `db push` writes the `public.eql_v2_configuration` table, which is a v2 + CipherStash Proxy artifact — EQL v3 has no configuration table (config lives in each column's `eql_v3.*` type) and nothing in the v3 stack reads it. The push recommendations are removed from `eql status`, the help banner, and the init/plan/cutover guidance. `db push` (and `db activate`) remain available for EQL v2 + Proxy users; they're now labelled as such. - **`eql status` is v3-aware.** On a v3-only database it reports that encrypt config lives in the column types instead of hitting a "table not found" dead-end that told users to run `db push` (which neither creates that table nor applies to v3). - **`stash db push` guards a v3-only database** with a clear "not needed under EQL v3" message instead of a raw `relation "public.eql_v2_configuration" does not exist` error. + +**Superseded later in this release:** the CLI no longer installs or mutates EQL +v2 state; `db push`, `db activate`, the Proxy choice, and the v2 install flags +described above are removed. Legacy state remains visible through status only. diff --git a/.changeset/stash-cli-skill-refresh.md b/.changeset/stash-cli-skill-refresh.md index 7a65d45c8..a2e62330b 100644 --- a/.changeset/stash-cli-skill-refresh.md +++ b/.changeset/stash-cli-skill-refresh.md @@ -29,3 +29,5 @@ handoff time, so a stale skill becomes stale guidance in the user's project. and `--region` flags; corrects six programmatic API signatures; fixes the README's claim that `stash init` ends in an agent-handoff menu (that belongs to `stash plan` / `stash impl`); and marks `stash env` as the non-functional stub it currently is. + +Superseded later in this release: the v2/Proxy commands and flags listed above are removed from both the CLI and bundled skill. diff --git a/packages/cli/README.md b/packages/cli/README.md index 853e1b21b..64d70e5e7 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -3,7 +3,7 @@ [![npm version](https://img.shields.io/npm/v/stash.svg?style=for-the-badge&labelColor=000000)](https://www.npmjs.com/package/stash) [![License: MIT](https://img.shields.io/npm/l/stash.svg?style=for-the-badge&labelColor=000000)](https://github.com/cipherstash/stack/blob/main/LICENSE.md) -The single CLI for CipherStash. It handles authentication, project initialization, EQL database lifecycle (install, upgrade, validate, push, migrate), and schema building. Install it as a devDependency alongside the runtime SDK `@cipherstash/stack`. +The single CLI for CipherStash. It handles authentication, project initialization, EQL v3 installation/upgrades, validation, migration rollout, and schema building. --- @@ -15,7 +15,7 @@ npx stash auth login # authenticate with CipherStash npx stash init # scaffold, introspect, install EQL ``` -`stash init` does the scaffold-once work as one flow: authenticate, resolve `DATABASE_URL`, choose Proxy or direct SDK access, introspect your database and scaffold an encryption client, install dependencies, install the EQL extension, and write `.cipherstash/context.json`. It stops there, at a clean save-point, and offers to continue into `stash plan`. +`stash init` authenticates, resolves `DATABASE_URL`, introspects the database, scaffolds an encryption client, installs dependencies and EQL v3, and writes `.cipherstash/context.json`. The agent handoff belongs to the next two commands — `stash plan` drafts a reviewable `.cipherstash/plan.md`, and `stash impl` executes it. Both present the same four targets: @@ -42,9 +42,7 @@ npx stash auth login └── npx stash status ← where am I? ``` -`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, the encryption rollout and cutover commands, and a `stash wizard` subcommand that thin-wraps [`@cipherstash/wizard`](https://www.npmjs.com/package/@cipherstash/wizard). The wizard package itself is a separate npm install — kept out of the `stash` bundle so the agent SDK doesn't bloat the CLI. - -> `stash db push` is **not** part of the default flow. It registers the encryption config in `public.eql_v2_configuration`, which only [CipherStash Proxy](https://github.com/cipherstash/proxy) reads. SDK users (Drizzle, Supabase, plain PostgreSQL) keep that config in application code and can skip it. +`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, and the staged EQL v3 encryption rollout. --- @@ -68,7 +66,7 @@ export default defineConfig({ The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory. -Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `db push`, `db validate`, `eql status`, `db test-connection`, `schema build`. `eql install` will scaffold `stash.config.ts` for you if it's missing. +Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `db validate`, `eql status`, `db test-connection`, and `schema build`. --- @@ -92,19 +90,19 @@ What `init` does, in order: 1. **Authenticate** — re-uses an existing token if found, otherwise opens the browser device-code flow. 2. **Resolve `DATABASE_URL`** — flag → env → `supabase status` → interactive prompt → hard-fail. The same resolver `eql install` uses. -3. **Generate the encryption client** — connects to your database, lists tables, and prompts you to multi-select which columns to encrypt. Writes `./src/encryption/index.ts` with the right shape for the detected ORM (Drizzle / Supabase / plain Postgres). Falls back to a placeholder if the database has no tables yet. +3. **Generate the encryption client placeholder** — detects the integration and writes `./src/encryption/index.ts` without selecting columns or replacing the project's authoritative schema files. The subsequent `stash plan` / `stash impl` workflow edits the real schema. 4. **Install dependencies** — `@cipherstash/stack` (runtime) and `stash` (dev), with a confirmation prompt. 5. **Install EQL** — runs `stash eql install` against the resolved URL after a y/N confirm. -6. **Hand off** — four-option menu (Claude Code / Codex / CipherStash Agent / write `AGENTS.md`). See the Quickstart section above for what each option writes and spawns. +6. **Checkpoint** — writes `.cipherstash/context.json` and exits. Continue with `stash plan`, then `stash impl`, as described in Quickstart. -The full pipeline state — integration, columns, env-key names, paths, versions — is captured in `.cipherstash/context.json`. The action plan at `.cipherstash/setup-prompt.md` tells whichever agent picks up next what's already done and what's left. +The full pipeline state — integration, columns, env-key names, paths, versions — is captured in `.cipherstash/context.json`. The action plan at `.cipherstash/setup-prompt.md` records what's already done and what `stash plan` / `stash impl` should do next. `CIPHERSTASH_WIZARD_URL` overrides the gateway endpoint for the rulebook fetch. Useful for local-dev against a wizard gateway running on `localhost`. -**Running `init` non-interactively** (CI, agents, pipes): every prompt has an escape hatch, so `init` never blocks waiting on a TTY. Provide the region up front (`--region` / `STASH_REGION`) if you aren't already logged in, the database URL (`--database-url` / `DATABASE_URL`), the proxy choice (`--proxy` / `--no-proxy`), and — for the closing agent handoff — nothing is required (init exits at a clean checkpoint and points you at `stash plan --target …`). When a required value is missing in a non-TTY context the command exits non-zero with an actionable message rather than hanging. +**Running `init` non-interactively** (CI, agents, pipes): every prompt has an escape hatch, so `init` never blocks waiting on a TTY. Provide the region up front (`--region` / `STASH_REGION`) if you aren't already logged in and set `DATABASE_URL`. Init exits at a clean checkpoint and points you at `stash plan --target …`; run `stash impl` after planning. When a required value is missing in a non-TTY context the command exits non-zero with an actionable message rather than hanging. ```bash -STASH_REGION=us-east-1 DATABASE_URL=postgres://… npx stash init --no-proxy +STASH_REGION=us-east-1 DATABASE_URL=postgres://… npx stash init ``` --- @@ -180,7 +178,7 @@ Any flags after `wizard` are forwarded verbatim to the wizard package. On the fi Configure your database and install CipherStash EQL extensions in a single command. Run this after `npx stash init`. (`npx stash db install` is a deprecated alias — it still works but prints a warning.) -When `stash.config.ts` is missing, the command auto-detects your encryption client file (or asks for the path) and writes the config before installing. Supabase and Drizzle are detected from your `DATABASE_URL` and project files, so the matching flags default on. Install uses bundled SQL for offline, deterministic runs. +When `stash.config.ts` is missing, the command offers to scaffold it. Installation is direct and EQL v3 only, using the bundle pinned by `@cipherstash/eql`. ```bash npx stash eql install [options] @@ -190,22 +188,18 @@ npx stash eql install [options] |------|-------------| | `--force` | Reinstall even if EQL is already installed | | `--dry-run` | Show what would happen without making changes | -| `--supabase` | Supabase-compatible install (no operator families + grants Supabase roles) | -| `--exclude-operator-family` | Skip operator family creation | -| `--drizzle` | Generate a Drizzle migration instead of direct install | -| `--latest` | Fetch the latest EQL from GitHub | -| `--name <value>` | Migration name (Drizzle mode, default: `install-eql`) | -| `--out <value>` | Drizzle output directory (default: `drizzle`) | +| `--supabase` | Supabase-compatible install with grants for built-in roles | +| `--database-url <url>` | One-shot target; leaves project files untouched | -The `--supabase` flag uses a Supabase-specific SQL variant and grants `USAGE`, table, routine, and sequence permissions on the `eql_v2` schema to the `anon`, `authenticated`, and `service_role` roles. +`--supabase` grants the built-in roles access to both `eql_v3` and `eql_v3_internal`. Removed v2 options fail explicitly; `--eql-version 2` points dump-recovery users to the upstream EQL 2.3.1 SQL release. -> **Good to know:** Without operator families, `ORDER BY` on encrypted columns is not supported. Sort application-side after decrypting results as a workaround. This applies to both `--supabase` and `--exclude-operator-family` installs. +> **Good to know:** The pinned EQL v3 bundle self-adapts when the install role cannot create its optional ORE operator family. In that case it disables the `*OrdOre` domains; use the ordinary `*Ord` domains for ordering. --- ### `npx stash eql upgrade` -Upgrade an existing EQL installation to the version bundled with the package (or the latest from GitHub). +Upgrade an existing EQL v3 installation to the package-pinned version. ```bash npx stash eql upgrade [options] @@ -215,40 +209,11 @@ npx stash eql upgrade [options] |------|-------------| | `--dry-run` | Show what would happen without making changes | | `--supabase` | Use Supabase-compatible upgrade | -| `--exclude-operator-family` | Skip operator family creation | -| `--latest` | Fetch the latest EQL from GitHub | The install SQL is idempotent and safe to re-run. If EQL is not installed, the command suggests running `npx stash eql install` instead. --- -### `npx stash db push` - -Push your encryption schema to the database. **Only required when using CipherStash Proxy.** If you use the SDK directly with Drizzle, Supabase, or plain PostgreSQL, skip this step. - -```bash -npx stash db push [--dry-run] -``` - -| Flag | Description | -|------|-------------| -| `--dry-run` | Load and validate the schema, print as JSON. No database changes. | - -When pushing, the CLI loads the encryption client from `stash.config.ts`, runs schema validation (warns but does not block), maps SDK types to EQL types, and upserts the config row in `eql_v2_configuration`. - -**SDK to EQL type mapping:** - -| SDK `dataType()` | EQL `cast_as` | -|------------------|---------------| -| `string` / `text` | `text` | -| `number` | `double` | -| `bigint` | `big_int` | -| `boolean` | `boolean` | -| `date` | `date` | -| `json` | `jsonb` | - ---- - ### `npx stash db validate` Validate your encryption schema for common misconfigurations. @@ -264,7 +229,7 @@ npx stash db validate [--supabase] [--exclude-operator-family] | No indexes on an encrypted column | Info | | `searchableJson` without `dataType("json")` | Error | -The command exits with code 1 on errors (not on warnings or info). Validation also runs automatically before `db push`. +The command exits with code 1 on errors (not on warnings or info). --- @@ -288,7 +253,7 @@ Show the current state of EQL in your database. npx stash eql status ``` -Reports EQL installation status and version, database permission status, and whether an active encrypt config exists in `eql_v2_configuration` (relevant only for CipherStash Proxy). +Reports EQL installation status and version, database permission status, and read-only diagnostics for legacy EQL v2/Proxy configuration state. --- @@ -320,22 +285,22 @@ Reads `databaseUrl` from `stash.config.ts`. ## Drizzle migration mode -Use `--drizzle` with `npx stash eql install` to add EQL installation to your Drizzle migration history instead of applying it directly. `--drizzle` is auto-detected when your project has `drizzle-orm`, `drizzle-kit`, or a `drizzle.config.*` file, so you usually don't need to pass it explicitly. +Use `eql migration --drizzle` to add EQL v3 installation to Drizzle migration history instead of applying it directly. ```bash -npx stash eql install --drizzle +npx stash eql migration --drizzle npx drizzle-kit migrate ``` How it works: 1. Runs `npx drizzle-kit generate --custom --name=<name>` to create an empty migration. -2. Loads the bundled EQL SQL (or fetches from GitHub with `--latest`). +2. Loads the pinned EQL v3 SQL. 3. Writes the EQL SQL into the generated migration file. With a custom name or output directory: ```bash -npx stash eql install --drizzle --name setup-eql --out ./migrations +npx stash eql migration --drizzle --name setup-eql --out ./migrations npx drizzle-kit migrate ``` @@ -348,7 +313,7 @@ npx drizzle-kit migrate Before installing EQL, the CLI verifies that the connected role has: - `CREATE` on the database (for `CREATE SCHEMA` and `CREATE EXTENSION`). -- `CREATE` on the `public` schema (for `CREATE TYPE public.eql_v2_encrypted`). +- `CREATE` on the `public` schema (for the `public.eql_v3_*` domains). - `SUPERUSER` or extension owner privileges (for `CREATE EXTENSION pgcrypto`, if not already installed). If permissions are insufficient, the CLI exits with a message listing what is missing. @@ -363,7 +328,6 @@ import { loadStashConfig, EQLInstaller, loadBundledEqlSql, - downloadEqlSql, } from 'stash' ``` @@ -415,11 +379,11 @@ if (!(await installer.isInstalled())) { | Method | Returns | Description | |--------|---------|-------------| | `checkPermissions()` | `Promise<PermissionCheckResult>` | Check required database permissions | -| `isInstalled()` | `Promise<boolean>` | Check if the `eql_v2` schema exists | +| `isInstalled()` | `Promise<boolean>` | Check if the EQL v3 schemas exist | | `getInstalledVersion()` | `Promise<string \| null>` | Get the installed EQL version | | `install(options?)` | `Promise<void>` | Execute the EQL install SQL in a transaction | -Install options: `excludeOperatorFamily`, `supabase`, `latest` (all boolean). +Install options: `supabase`. ### `loadBundledEqlSql` @@ -429,19 +393,6 @@ Load the bundled EQL install SQL as a string: import { loadBundledEqlSql } from 'stash' const sql = loadBundledEqlSql() -const sql = loadBundledEqlSql({ supabase: true }) -const sql = loadBundledEqlSql({ excludeOperatorFamily: true }) -``` - -### `downloadEqlSql` - -Download the latest EQL install SQL from GitHub: - -```typescript -import { downloadEqlSql } from 'stash' - -const sql = await downloadEqlSql() // standard -const sql = await downloadEqlSql(true) // no operator family variant ``` --- diff --git a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts index 755815874..51571571b 100644 --- a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts +++ b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts @@ -8,10 +8,9 @@ * to reference the encrypted tables you declared there. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and `stash db push`, - * `stash db validate` and `stash encrypt backfill` refuse to run and point - * back here. (`stash encrypt cutover` / `drop` resolve against the database - * and never read this file.) + * placeholder table so that this file compiles, and `stash db validate` and + * `stash encrypt backfill` refuse to run and point back here. (`stash + * encrypt drop` resolves against the database and never reads this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the `types.*` factories from `@cipherstash/stack-drizzle`. @@ -58,8 +57,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — `Encryption` requires at least one. Swap it for your -// real tables (see the patterns above); `stash db push`, `stash db validate` -// and `stash encrypt backfill` refuse to run while the placeholder is still here. +// real tables (see the patterns above); `stash db validate` and `stash +// encrypt backfill` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/__fixtures__/scaffold/generic.generated.ts b/packages/cli/__fixtures__/scaffold/generic.generated.ts index 9f0187adc..97b7c4c22 100644 --- a/packages/cli/__fixtures__/scaffold/generic.generated.ts +++ b/packages/cli/__fixtures__/scaffold/generic.generated.ts @@ -7,10 +7,9 @@ * `Encryption({ schemas: [...] })` call below to reference them. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and `stash db push`, - * `stash db validate` and `stash encrypt backfill` refuse to run and point - * back here. (`stash encrypt cutover` / `drop` resolve against the database - * and never read this file.) + * placeholder table so that this file compiles, and `stash db validate` and + * `stash encrypt backfill` refuse to run and point back here. (`stash + * encrypt drop` resolves against the database and never reads this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the `types.*` factories from `@cipherstash/stack/eql/v3` @@ -53,8 +52,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — `Encryption` requires at least one. Swap it for your -// real tables (see the patterns above); `stash db push`, `stash db validate` -// and `stash encrypt backfill` refuse to run while the placeholder is still here. +// real tables (see the patterns above); `stash db validate` and `stash +// encrypt backfill` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/scripts/e2e-encrypt.sh b/packages/cli/scripts/e2e-encrypt.sh index 6e310f505..0e027a80b 100755 --- a/packages/cli/scripts/e2e-encrypt.sh +++ b/packages/cli/scripts/e2e-encrypt.sh @@ -26,11 +26,11 @@ psql -h "$HOST" -d postgres -c "CREATE DATABASE ${DB}" >/dev/null export DATABASE_URL echo "==> 1. Install EQL + cs_migrations" -"$STASH" db install --force +"$STASH" eql install --force echo "==> 2. Seed 5000 plaintext users" psql "$DATABASE_URL" -f "$FIXTURES/seed-users.sql" >/dev/null -psql "$DATABASE_URL" -c "ALTER TABLE users ADD COLUMN email_encrypted eql_v2_encrypted" >/dev/null +psql "$DATABASE_URL" -c "ALTER TABLE users ADD COLUMN email_encrypted public.eql_v3_text_search" >/dev/null echo "==> 3. Backfill with interrupt/resume (dual-writes confirmed via flag for non-interactive run)" "$STASH" encrypt backfill --table users --column email --chunk-size 500 --confirm-dual-writes-deployed & @@ -50,10 +50,9 @@ echo "OK: all 5000 rows encrypted" echo "==> 4. Status" "$STASH" encrypt status -echo "==> 5. Cutover" -"$STASH" encrypt cutover --table users --column email +echo "==> 5. Switch application reads to email_encrypted (manual deploy step)" -echo "==> 6. Drop" +echo "==> 6. Generate plaintext drop migration" "$STASH" encrypt drop --table users --column email --migrations-dir "$(pwd)/drizzle" echo "==> Done." diff --git a/packages/cli/scripts/fixtures/seed-users.sql b/packages/cli/scripts/fixtures/seed-users.sql index 1b114e038..90352b934 100644 --- a/packages/cli/scripts/fixtures/seed-users.sql +++ b/packages/cli/scripts/fixtures/seed-users.sql @@ -1,7 +1,7 @@ -- Seed a users table with plaintext emails for e2e backfill testing. -- --- The encrypted target column must be created separately (drizzle-kit / --- stash db push route), after which the backfill encrypts `email` → `email_encrypted`. +-- The EQL v3 encrypted target column must be created separately (for example, +-- by Drizzle Kit), after which backfill encrypts `email` → `email_encrypted`. DROP TABLE IF EXISTS users CASCADE; diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 5ee0bd32e..52c23852b 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -4,381 +4,133 @@ const mockConnect = vi.fn() const mockQuery = vi.fn() const mockEnd = vi.fn() -vi.mock('pg', () => { - const Client = vi.fn(() => ({ - connect: mockConnect, - query: mockQuery, - end: mockEnd, - })) - return { default: { Client } } -}) +vi.mock('pg', () => ({ + default: { + Client: vi.fn(() => ({ + connect: mockConnect, + query: mockQuery, + end: mockEnd, + })), + }, +})) describe('EQLInstaller', () => { - beforeEach(() => { - vi.clearAllMocks() - }) - - afterEach(() => { - vi.restoreAllMocks() - }) - - describe('checkPermissions', () => { - it('returns ok when role is superuser', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ - rows: [{ rolsuper: true, rolcreatedb: true }], - rowCount: 1, - }) - mockEnd.mockResolvedValue(undefined) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - const result = await installer.checkPermissions() - expect(result.ok).toBe(true) - expect(result.missing).toEqual([]) + beforeEach(() => vi.clearAllMocks()) + afterEach(() => vi.restoreAllMocks()) + + it('reports sufficient permissions for a superuser', async () => { + mockConnect.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ + rows: [{ rolsuper: true, rolcreatedb: true }], + rowCount: 1, }) - - it('returns missing permissions when role lacks privileges', async () => { - mockConnect.mockResolvedValue(undefined) - mockEnd.mockResolvedValue(undefined) - - let queryCall = 0 - mockQuery.mockImplementation(() => { - queryCall++ - switch (queryCall) { - // pg_roles query — not superuser - case 1: - return { - rows: [{ rolsuper: false, rolcreatedb: false }], - rowCount: 1, - } - // has_database_privilege — no CREATE - case 2: - return { rows: [{ has_create: false }], rowCount: 1 } - // has_schema_privilege — no CREATE on public - case 3: - return { rows: [{ has_create: false }], rowCount: 1 } - // pgcrypto check — not installed - case 4: - return { rows: [], rowCount: 0 } - default: - return { rows: [], rowCount: 0 } - } - }) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - const result = await installer.checkPermissions() - expect(result.ok).toBe(false) - expect(result.missing).toHaveLength(3) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.checkPermissions()).resolves.toEqual({ + ok: true, + missing: [], + isSuperuser: true, }) }) - describe('isInstalled', () => { - it('returns false when schema does not exist', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - const result = await installer.isInstalled() - expect(result).toBe(false) - }) - - it('returns true when schema exists', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ - rows: [{ found: 1 }], + it('accepts database-local CREATE privilege for installing pgcrypto', async () => { + mockConnect.mockResolvedValue(undefined) + mockQuery + .mockResolvedValueOnce({ + rows: [{ rolsuper: false, rolcreatedb: false }], rowCount: 1, }) - mockEnd.mockResolvedValue(undefined) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - const result = await installer.isInstalled({ eqlVersion: 2 }) - expect(result).toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [['eql_v2']]) + .mockResolvedValueOnce({ rows: [{ has_create: true }], rowCount: 1 }) + .mockResolvedValueOnce({ rows: [{ has_create: true }], rowCount: 1 }) + .mockResolvedValueOnce({ rows: [], rowCount: 0 }) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.checkPermissions()).resolves.toEqual({ + ok: true, + missing: [], + isSuperuser: false, }) }) - describe('install', () => { - it('uses bundled SQL and executes in a transaction', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) - - const fetchSpy = vi.spyOn(globalThis, 'fetch') - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await installer.install({ eqlVersion: 2 }) - - // Should NOT call fetch — uses bundled SQL - expect(fetchSpy).not.toHaveBeenCalled() - expect(mockQuery).toHaveBeenCalledWith('BEGIN') - // The second query should be the bundled SQL (a large string) - const sqlCall = mockQuery.mock.calls.find( - (call: string[]) => - typeof call[0] === 'string' && - call[0] !== 'BEGIN' && - call[0] !== 'COMMIT', - ) - expect(sqlCall).toBeDefined() - expect(sqlCall[0]).toContain('eql_v2') - expect(mockQuery).toHaveBeenCalledWith('COMMIT') - }) - - it('fetches from GitHub when latest: true', async () => { - const installSql = 'CREATE SCHEMA eql_v2;' - - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) - - const fetchSpy = vi - .spyOn(globalThis, 'fetch') - .mockResolvedValue(new Response(installSql, { status: 200 })) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await installer.install({ eqlVersion: 2, latest: true }) - - expect(fetchSpy).toHaveBeenCalledWith( - expect.stringContaining('cipherstash-encrypt.sql'), - ) - expect(mockQuery).toHaveBeenCalledWith('BEGIN') - expect(mockQuery).toHaveBeenCalledWith(installSql) - expect(mockQuery).toHaveBeenCalledWith('COMMIT') - }) - - it('grants Supabase permissions as a single SUPABASE_PERMISSIONS_SQL query', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) + it('requires both EQL v3 schemas for the current install', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - const { EQLInstaller, SUPABASE_PERMISSIONS_SQL } = await import( - '@/installer/index.ts' - ) - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await installer.install({ eqlVersion: 2, supabase: true }) - - // Capture every query string that isn't a transaction control verb. - const otherCalls = mockQuery.mock.calls - .map((call: unknown[]) => call[0]) - .filter( - (sql: unknown): sql is string => - typeof sql === 'string' && - sql !== 'BEGIN' && - sql !== 'COMMIT' && - sql !== 'ROLLBACK', - ) - - // Two non-transaction queries: bundled EQL SQL, then permissions SQL. - expect(otherCalls).toHaveLength(2) - expect(otherCalls[1]).toBe(SUPABASE_PERMISSIONS_SQL) - // Permissions SQL must mention each role + the eql_v2 schema. - expect(SUPABASE_PERMISSIONS_SQL).toContain('eql_v2') - expect(SUPABASE_PERMISSIONS_SQL).toContain('anon') - expect(SUPABASE_PERMISSIONS_SQL).toContain('authenticated') - expect(SUPABASE_PERMISSIONS_SQL).toContain('service_role') - }) - - it('installs the v3 bundle and grants eql_v3 permissions with eqlVersion: 3 + supabase', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) - - const { EQLInstaller, SUPABASE_PERMISSIONS_SQL_V3 } = await import( - '@/installer/index.ts' - ) - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await installer.install({ eqlVersion: 3, supabase: true }) - - const otherCalls = mockQuery.mock.calls - .map((call: unknown[]) => call[0]) - .filter( - (sql: unknown): sql is string => - typeof sql === 'string' && - sql !== 'BEGIN' && - sql !== 'COMMIT' && - sql !== 'ROLLBACK', - ) - - expect(otherCalls).toHaveLength(2) - // Since eql-3.0.0 there is ONE v3 bundle for every target: the - // operator-class statements run inside a DO block that self-skips on - // insufficient_privilege, and the bundle disables the ORE-backed - // domains when the opclass is absent (CIP-3468). The supabase install - // therefore executes the SAME artifact as the direct install. - expect(otherCalls[0]).toContain('eql_v3') - expect(otherCalls[0]).toContain('CREATE OPERATOR CLASS') - expect(otherCalls[0]).toContain('insufficient_privilege') - // The grants are keyed to eql_v3, not eql_v2. The installed block must be - // the SAME string the Supabase migration file embeds — the installer used - // to rebuild it from the schema name alone, letting the two drift. - expect(otherCalls[1]).toBe(SUPABASE_PERMISSIONS_SQL_V3) - expect(SUPABASE_PERMISSIONS_SQL_V3).toContain('eql_v3') - expect(SUPABASE_PERMISSIONS_SQL_V3).not.toContain('eql_v2') - - // `eql_v3.eq_term`/`ord_term`/`match_term` are SECURITY INVOKER and - // qualify `eql_v3_internal.*` in their bodies, so without USAGE on that - // schema every encrypted filter fails for anon/authenticated with - // "permission denied for schema eql_v3_internal". See - // `supabaseInternalPermissionsSql`, and the live proof in - // packages/stack/__tests__/supabase-v3-grants-pg.test.ts. - expect(SUPABASE_PERMISSIONS_SQL_V3).toContain( - 'GRANT USAGE ON SCHEMA eql_v3_internal TO anon, authenticated, service_role;', - ) - expect(SUPABASE_PERMISSIONS_SQL_V3).toContain( - 'GRANT EXECUTE ON ALL ROUTINES IN SCHEMA eql_v3_internal TO anon, authenticated, service_role;', - ) - }) - - // `eql_v2` has no internal schema; the v3-only addition must not leak into - // the v2 block, where it would fail with "schema does not exist". - it('does not grant an internal schema in the v2 permissions block', async () => { - const { SUPABASE_PERMISSIONS_SQL } = await import('@/installer/index.ts') - - expect(SUPABASE_PERMISSIONS_SQL).not.toContain('_internal') - }) - - it('installs the full v3 bundle (with operator classes) without supabase', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await installer.install({ eqlVersion: 3 }) + mockQuery.mockResolvedValue({ rows: [{ found: 2 }], rowCount: 1 }) + await expect(installer.isInstalled()).resolves.toBe(true) + expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [ + ['eql_v3', 'eql_v3_internal'], + ]) - const sqlCall = mockQuery.mock.calls.find( - (call: string[]) => - typeof call[0] === 'string' && - call[0] !== 'BEGIN' && - call[0] !== 'COMMIT', - ) - expect(sqlCall).toBeDefined() - expect(sqlCall?.[0]).toContain('eql_v3') - expect(sqlCall?.[0]).toContain('CREATE OPERATOR CLASS') - }) - - it('rejects latest: true for eqlVersion: 3', async () => { - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await expect( - installer.install({ eqlVersion: 3, latest: true }), - ).rejects.toThrow('not supported for EQL v3') - }) - - it('requires BOTH eql_v3 and eql_v3_internal for isInstalled({ eqlVersion: 3 })', async () => { - mockConnect.mockResolvedValue(undefined) - mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + await expect(installer.isInstalled()).resolves.toBe(false) + }) - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) + it('retains read-only EQL v2 installation detection for status', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - // Current-generation install: both schemas present - mockQuery.mockResolvedValue({ rows: [{ found: 2 }], rowCount: 1 }) - await expect(installer.isInstalled({ eqlVersion: 3 })).resolves.toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [ - ['eql_v3', 'eql_v3_internal'], - ]) + await expect(installer.isInstalled({ eqlVersion: 2 })).resolves.toBe(true) + expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [['eql_v2']]) + }) - // STALE pre-alpha.2 install: eql_v3 exists but eql_v3_internal does not - // — must report NOT installed so an install run replaces it instead of - // a stale surface silently accepting wrong-generation wire data. - mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) - await expect(installer.isInstalled({ eqlVersion: 3 })).resolves.toBe( - false, - ) - }) + it('installs only the pinned EQL v3 bundle', async () => { + mockConnect.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await installer.install() + + const sqlCall = mockQuery.mock.calls.find( + ([sql]) => + typeof sql === 'string' && + !['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql), + ) + expect(sqlCall?.[0]).toContain('eql_v3') + expect(sqlCall?.[0]).not.toContain('CREATE SCHEMA eql_v2') + expect(mockQuery).toHaveBeenCalledWith('COMMIT') + }) - it('grants eql_v3 AND eql_v3_internal for the v3 supabase install', async () => { - mockConnect.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) - mockEnd.mockResolvedValue(undefined) + it('grants both EQL v3 schemas to Supabase roles', async () => { + mockConnect.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller, SUPABASE_PERMISSIONS_SQL_V3 } = await import( + '@/installer/index.ts' + ) + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - const { EQLInstaller, SUPABASE_PERMISSIONS_SQL_V3 } = await import( - '@/installer/index.ts' - ) - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) + await installer.install({ supabase: true }) - await installer.install({ eqlVersion: 3, supabase: true }) + expect(mockQuery).toHaveBeenCalledWith(SUPABASE_PERMISSIONS_SQL_V3) + expect(SUPABASE_PERMISSIONS_SQL_V3).toContain('eql_v3_internal') + expect(SUPABASE_PERMISSIONS_SQL_V3).not.toContain('eql_v2') + }) - const otherCalls = mockQuery.mock.calls - .map((call: unknown[]) => call[0]) - .filter( - (sql: unknown): sql is string => - typeof sql === 'string' && - sql !== 'BEGIN' && - sql !== 'COMMIT' && - sql !== 'ROLLBACK', - ) - expect(otherCalls[1]).toBe(SUPABASE_PERMISSIONS_SQL_V3) - // The eql_v3 operators call SECURITY INVOKER functions that live in - // eql_v3_internal — the roles need grants on BOTH schemas. - expect(SUPABASE_PERMISSIONS_SQL_V3).toContain( - 'GRANT USAGE ON SCHEMA eql_v3 ', - ) - expect(SUPABASE_PERMISSIONS_SQL_V3).toContain( - 'GRANT USAGE ON SCHEMA eql_v3_internal ', - ) + it('rolls back when the install SQL fails', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (!['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql)) { + return Promise.reject(new Error('permission denied')) + } + return Promise.resolve({ rows: [], rowCount: 0 }) }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - it('rolls back on SQL execution failure', async () => { - mockConnect.mockResolvedValue(undefined) - mockEnd.mockResolvedValue(undefined) - - mockQuery.mockImplementation((sql: string) => { - // BEGIN succeeds, any SQL containing eql_v2 (the bundled install) fails - if (sql !== 'BEGIN' && sql !== 'COMMIT' && sql !== 'ROLLBACK') { - return Promise.reject(new Error('permission denied')) - } - return Promise.resolve({ rows: [], rowCount: 0 }) - }) - - const { EQLInstaller } = await import('@/installer/index.ts') - const installer = new EQLInstaller({ - databaseUrl: 'postgresql://localhost:5432/test', - }) - - await expect(installer.install()).rejects.toThrow('Failed to install EQL') - expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') - }) + await expect(installer.install()).rejects.toThrow('Failed to install EQL') + expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') }) }) diff --git a/packages/cli/src/__tests__/placeholder-guard-parity.test.ts b/packages/cli/src/__tests__/placeholder-guard-parity.test.ts index 80563e9c2..6b57d14d0 100644 --- a/packages/cli/src/__tests__/placeholder-guard-parity.test.ts +++ b/packages/cli/src/__tests__/placeholder-guard-parity.test.ts @@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { PLACEHOLDER_TABLE_NAME } from '@/config/index.js' /** - * `stash db push` / `db validate` reach the user's encryption client through + * `stash db validate` reaches the user's encryption client through * `loadEncryptConfig`; `stash encrypt backfill` reaches the same file through * `loadEncryptionContext`. Both must refuse the un-replaced `stash init` * scaffold, and — because it is one file and one mistake — both must say the @@ -72,7 +72,7 @@ describe('the un-replaced init scaffold, refused identically by both loaders', ( } }) - it('gives db push and encrypt backfill the same refusal for the same file', async () => { + it('gives db validate and encrypt backfill the same refusal for the same file', async () => { writeProject( `export const encryptionClient = { getEncryptConfig: () => ({ tables: { '${PLACEHOLDER_TABLE_NAME}': {} } }), @@ -81,22 +81,24 @@ describe('the un-replaced init scaffold, refused identically by both loaders', ( ) const { loadEncryptConfig } = await import('@/config/index.js') - const dbPush = await captureRefusal(() => loadEncryptConfig('./client.ts')) + const dbValidate = await captureRefusal(() => + loadEncryptConfig('./client.ts'), + ) const { loadEncryptionContext } = await import( '../commands/encrypt/context.js' ) const backfill = await captureRefusal(() => loadEncryptionContext()) - expect(dbPush.exited).toBe(true) + expect(dbValidate.exited).toBe(true) expect(backfill.exited).toBe(true) - expect(dbPush.message).toContain('still contains the placeholder table') - expect(backfill.message).toEqual(dbPush.message) + expect(dbValidate.message).toContain('still contains the placeholder table') + expect(backfill.message).toEqual(dbValidate.message) }) it('names the cause, not the symptom, when the client has no encrypt config', async () => { // A client whose `getEncryptConfig()` returns nothing is the same class of - // un-finished setup. `db push` already named it; `encrypt backfill` used to + // unfinished setup. `db validate` already named it; backfill used to // fall through to `requireTable`'s `Table "users" was not found … // Available: (none)` — the symptom-not-cause message this guard exists to // replace (#787 review follow-up). @@ -106,14 +108,16 @@ describe('the un-replaced init scaffold, refused identically by both loaders', ( ) const { loadEncryptConfig } = await import('@/config/index.js') - const dbPush = await captureRefusal(() => loadEncryptConfig('./client.ts')) + const dbValidate = await captureRefusal(() => + loadEncryptConfig('./client.ts'), + ) const { loadEncryptionContext } = await import( '../commands/encrypt/context.js' ) const backfill = await captureRefusal(() => loadEncryptionContext()) - expect(dbPush.exited).toBe(true) + expect(dbValidate.exited).toBe(true) expect(backfill.exited).toBe(true) expect(backfill.message).toContain('no initialized encrypt config') expect(backfill.message).not.toContain('was not found') diff --git a/packages/cli/src/__tests__/supabase-migration.test.ts b/packages/cli/src/__tests__/supabase-migration.test.ts deleted file mode 100644 index bf2a57f6c..000000000 --- a/packages/cli/src/__tests__/supabase-migration.test.ts +++ /dev/null @@ -1,479 +0,0 @@ -import fs from 'node:fs' -import os from 'node:os' -import path from 'node:path' -import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { detectSupabaseProject } from '../commands/db/detect.js' -import { - chooseSupabaseInstallMode, - routeInstallPathForEqlVersion, - validateInstallFlags, -} from '../commands/db/install.js' -import { - SUPABASE_EQL_MIGRATION_FILENAME, - writeSupabaseEqlMigration, -} from '../commands/db/supabase-migration.js' -import { - DEFAULT_EQL_VERSION, - resolveEqlVersion, - SUPABASE_PERMISSIONS_SQL, -} from '../installer/index.js' - -/** - * Generate the migration header for testing purposes. - * Mirrors the production function but imported for testing. - */ -function migrationHeader(runner: string): string { - return `-- CipherStash EQL — installed by \`${runner} stash eql install --supabase --migration\`. --- --- This migration installs the CipherStash Encrypt Query Language (EQL) types, --- functions, and operators into the \`eql_v2\` schema, then grants Supabase's --- \`anon\`, \`authenticated\`, and \`service_role\` roles the access they need. --- --- The all-zero \`YYYYMMDDHHMMSS\` prefix is intentional: Supabase orders --- migrations lexically, so this file runs before any user migration that --- references the \`eql_v2_encrypted\` type. Do not rename it. --- --- To upgrade EQL, re-run the install command — it will refuse to overwrite --- this file unless you pass --force. --- --- Docs: https://cipherstash.com/docs/stack/cipherstash/supabase -` -} - -describe('detectSupabaseProject', () => { - let tmpDir: string - - beforeEach(() => { - tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-supa-detect-')) - }) - - afterEach(() => { - fs.rmSync(tmpDir, { recursive: true, force: true }) - }) - - it('detects only config.toml', () => { - fs.mkdirSync(path.join(tmpDir, 'supabase'), { recursive: true }) - fs.writeFileSync(path.join(tmpDir, 'supabase', 'config.toml'), '') - - const info = detectSupabaseProject(tmpDir) - expect(info.hasConfigToml).toBe(true) - expect(info.hasMigrationsDir).toBe(false) - expect(info.migrationsDir).toBe( - path.resolve(tmpDir, 'supabase', 'migrations'), - ) - }) - - it('detects only the migrations directory', () => { - fs.mkdirSync(path.join(tmpDir, 'supabase', 'migrations'), { - recursive: true, - }) - - const info = detectSupabaseProject(tmpDir) - expect(info.hasConfigToml).toBe(false) - expect(info.hasMigrationsDir).toBe(true) - }) - - it('detects both config.toml and the migrations directory', () => { - fs.mkdirSync(path.join(tmpDir, 'supabase', 'migrations'), { - recursive: true, - }) - fs.writeFileSync(path.join(tmpDir, 'supabase', 'config.toml'), '') - - const info = detectSupabaseProject(tmpDir) - expect(info.hasConfigToml).toBe(true) - expect(info.hasMigrationsDir).toBe(true) - }) - - it('returns false flags when neither marker is present', () => { - const info = detectSupabaseProject(tmpDir) - expect(info.hasConfigToml).toBe(false) - expect(info.hasMigrationsDir).toBe(false) - }) - - it('honors a custom override path (relative + absolute)', () => { - const customRel = 'db/migrations' - fs.mkdirSync(path.join(tmpDir, customRel), { recursive: true }) - - const relInfo = detectSupabaseProject(tmpDir, customRel) - expect(relInfo.migrationsDir).toBe(path.resolve(tmpDir, customRel)) - expect(relInfo.hasMigrationsDir).toBe(true) - - const absPath = path.resolve(tmpDir, customRel) - const absInfo = detectSupabaseProject(tmpDir, absPath) - expect(absInfo.migrationsDir).toBe(absPath) - expect(absInfo.hasMigrationsDir).toBe(true) - }) - - it('treats a file at the migrations path as a missing directory', () => { - fs.mkdirSync(path.join(tmpDir, 'supabase'), { recursive: true }) - fs.writeFileSync(path.join(tmpDir, 'supabase', 'migrations'), 'not a dir') - - const info = detectSupabaseProject(tmpDir) - expect(info.hasMigrationsDir).toBe(false) - }) -}) - -describe('writeSupabaseEqlMigration', () => { - let tmpDir: string - - beforeEach(() => { - tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-supa-write-')) - }) - - afterEach(() => { - fs.rmSync(tmpDir, { recursive: true, force: true }) - }) - - it('writes the file at the well-known filename', async () => { - const migrationsDir = path.join(tmpDir, 'supabase', 'migrations') - - const result = await writeSupabaseEqlMigration({ migrationsDir }) - - expect(path.basename(result.path)).toBe(SUPABASE_EQL_MIGRATION_FILENAME) - expect(result.overwritten).toBe(false) - expect(fs.existsSync(result.path)).toBe(true) - }) - - it('writes EQL SQL plus the SUPABASE_PERMISSIONS_SQL block', async () => { - const migrationsDir = path.join(tmpDir, 'supabase', 'migrations') - const result = await writeSupabaseEqlMigration({ migrationsDir }) - - const contents = fs.readFileSync(result.path, 'utf-8') - // Header comment block includes the detected runner instruction - expect(contents).toMatch( - /-- CipherStash EQL — installed by `(npx|bunx|pnpm dlx|yarn dlx) stash eql install --supabase --migration`/, - ) - expect(contents).toContain('CipherStash') - // EQL SQL body — the bundled supabase variant defines eql_v2. The - // migration-file flow is v2-only, so the body must be the v2 generation: - // assert the v2 composite type is present and no v3 schema leaks in. (A - // bare `toContain('eql_v2')` isn't enough — the header + permissions - // mention eql_v2 even if the body were v3.) - expect(contents).toContain('eql_v2_encrypted') - expect(contents).not.toContain('eql_v3') - // Permissions block (single source of truth). - expect(contents).toContain(SUPABASE_PERMISSIONS_SQL.trim()) - }) - - it('creates the migrations directory if missing', async () => { - const migrationsDir = path.join(tmpDir, 'supabase', 'migrations') - expect(fs.existsSync(migrationsDir)).toBe(false) - - const result = await writeSupabaseEqlMigration({ migrationsDir }) - - expect(fs.statSync(migrationsDir).isDirectory()).toBe(true) - expect(fs.existsSync(result.path)).toBe(true) - }) - - it('throws when the file already exists and force is false', async () => { - const migrationsDir = path.join(tmpDir, 'supabase', 'migrations') - fs.mkdirSync(migrationsDir, { recursive: true }) - const existingPath = path.join( - migrationsDir, - SUPABASE_EQL_MIGRATION_FILENAME, - ) - fs.writeFileSync(existingPath, '-- existing') - - await expect(writeSupabaseEqlMigration({ migrationsDir })).rejects.toThrow( - /already exists/, - ) - - // Existing content untouched - expect(fs.readFileSync(existingPath, 'utf-8')).toBe('-- existing') - }) - - it('overwrites when force is true', async () => { - const migrationsDir = path.join(tmpDir, 'supabase', 'migrations') - fs.mkdirSync(migrationsDir, { recursive: true }) - const existingPath = path.join( - migrationsDir, - SUPABASE_EQL_MIGRATION_FILENAME, - ) - fs.writeFileSync(existingPath, '-- existing') - - const result = await writeSupabaseEqlMigration({ - migrationsDir, - force: true, - }) - - expect(result.overwritten).toBe(true) - expect(fs.readFileSync(result.path, 'utf-8')).not.toBe('-- existing') - expect(fs.readFileSync(result.path, 'utf-8')).toContain('eql_v2') - }) - - it('sorts before realistic Supabase-style migration filenames', () => { - const filenames = [ - SUPABASE_EQL_MIGRATION_FILENAME, - '20251015120000_users.sql', - '99999999999999_other.sql', - ] - const sorted = [...filenames].sort() - expect(sorted[0]).toBe(SUPABASE_EQL_MIGRATION_FILENAME) - }) -}) - -describe('resolveEqlVersion', () => { - it('defaults a missing version to DEFAULT_EQL_VERSION (v3)', () => { - expect(DEFAULT_EQL_VERSION).toBe(3) - expect(resolveEqlVersion(undefined)).toBe(3) - }) - - it('resolves explicit strings to their generation', () => { - expect(resolveEqlVersion('2')).toBe(2) - expect(resolveEqlVersion('3')).toBe(3) - }) -}) - -describe('validateInstallFlags', () => { - it('returns null for an empty options object', () => { - expect(validateInstallFlags({})).toBeNull() - }) - - it('returns null when --supabase + --migration is pinned to --eql-version 2', () => { - // --migration is a v2-only path, so under the v3 default it needs an - // explicit `--eql-version 2` alongside --supabase. - expect( - validateInstallFlags({ - supabase: true, - migration: true, - eqlVersion: '2', - }), - ).toBeNull() - }) - - it('returns null when --supabase is paired with --direct', () => { - expect(validateInstallFlags({ supabase: true, direct: true })).toBeNull() - }) - - it('rejects --migration without --supabase', () => { - const err = validateInstallFlags({ migration: true }) - expect(err).toMatch(/--migration/) - expect(err).toMatch(/--supabase/) - }) - - it('rejects --direct without --supabase', () => { - const err = validateInstallFlags({ direct: true }) - expect(err).toMatch(/--direct/) - expect(err).toMatch(/--supabase/) - }) - - it('rejects --migrations-dir without --supabase', () => { - const err = validateInstallFlags({ migrationsDir: 'db/migrations' }) - expect(err).toMatch(/--migrations-dir/) - expect(err).toMatch(/--supabase/) - }) - - it('rejects --migration AND --direct together', () => { - const err = validateInstallFlags({ - supabase: true, - migration: true, - direct: true, - }) - expect(err).toMatch(/mutually exclusive/i) - }) - - it('accepts --eql-version 2 and 3', () => { - expect(validateInstallFlags({ eqlVersion: '2' })).toBeNull() - expect(validateInstallFlags({ eqlVersion: '3' })).toBeNull() - expect( - validateInstallFlags({ eqlVersion: '3', supabase: true, direct: true }), - ).toBeNull() - }) - - it('rejects an unknown --eql-version value', () => { - expect(validateInstallFlags({ eqlVersion: '4' })).toMatch(/--eql-version/) - }) - - it('rejects --eql-version 3 with --drizzle, --migration, --latest, or --migrations-dir', () => { - expect(validateInstallFlags({ eqlVersion: '3', drizzle: true })).toMatch( - /--drizzle/, - ) - expect( - validateInstallFlags({ - eqlVersion: '3', - supabase: true, - migration: true, - }), - ).toMatch(/--migration/) - expect(validateInstallFlags({ eqlVersion: '3', latest: true })).toMatch( - /--latest/, - ) - // --migrations-dir only feeds the v2 Supabase migration-file path; the v3 - // direct install would silently ignore it (flagged in review of #547). - expect( - validateInstallFlags({ - eqlVersion: '3', - supabase: true, - migrationsDir: 'db/migrations', - }), - ).toMatch(/--migrations-dir/) - }) - - it('defaults to v3: a plain install with no version is valid', () => { - expect(validateInstallFlags({})).toBeNull() - expect(validateInstallFlags({ supabase: true })).toBeNull() - }) - - it('v2-only flags without an explicit --eql-version 2 are rejected under the v3 default', () => { - // No eqlVersion passed → resolves to the v3 default, which can't do these. - for (const opts of [ - { drizzle: true }, - { latest: true }, - { supabase: true, migration: true }, - { supabase: true, migrationsDir: 'db/migrations' }, - ]) { - const err = validateInstallFlags(opts) - expect(err).toMatch(/--eql-version 2/) - } - }) - - it('accepts the v2-only paths when --eql-version 2 is explicit', () => { - expect(validateInstallFlags({ eqlVersion: '2', drizzle: true })).toBeNull() - expect(validateInstallFlags({ eqlVersion: '2', latest: true })).toBeNull() - expect( - validateInstallFlags({ - eqlVersion: '2', - supabase: true, - migration: true, - }), - ).toBeNull() - }) - - it('points --drizzle at the v3 migration command instead of only at --eql-version 2', () => { - // Steering users to `--eql-version 2` is how new projects ended up on the - // legacy generation (#691). Both the default-branch wording and the explicit - // `--eql-version 3` wording must carry the pointer. Assert the distinctive - // substring, not a bare `/--drizzle/` — that already appears in the base - // message body and would pass even if the pointer were dropped. - expect(validateInstallFlags({ drizzle: true })).toMatch( - /stash eql migration --drizzle/, - ) - expect(validateInstallFlags({ eqlVersion: '3', drizzle: true })).toMatch( - /stash eql migration --drizzle/, - ) - }) - - it('does NOT offer the Drizzle alternative for the other v2-only flags', () => { - // There is no `stash eql migration` route for these — suggesting one would - // send the user to a command that cannot help them. Guards the pointer being - // gated on `v2OnlyFlag === '--drizzle'`. - for (const opts of [ - { latest: true }, - { supabase: true, migration: true }, - { supabase: true, migrationsDir: 'db/migrations' }, - ]) { - expect(validateInstallFlags(opts)).not.toMatch(/stash eql migration/) - expect(validateInstallFlags({ ...opts, eqlVersion: '3' })).not.toMatch( - /stash eql migration/, - ) - } - }) -}) - -describe('routeInstallPathForEqlVersion', () => { - it('passes v2 routing through untouched', () => { - expect( - routeInstallPathForEqlVersion(2, { supabase: false, drizzle: true }), - ).toEqual({ drizzle: true, useSupabaseInstallModeSelection: false }) - expect( - routeInstallPathForEqlVersion(2, { supabase: true, drizzle: false }), - ).toEqual({ drizzle: false, useSupabaseInstallModeSelection: true }) - }) - - it('falls back from auto-detected drizzle to direct for v3, with a notice', () => { - const routing = routeInstallPathForEqlVersion(3, { - supabase: false, - drizzle: true, - }) - expect(routing.drizzle).toBe(false) - expect(routing.useSupabaseInstallModeSelection).toBe(false) - expect(routing.notice).toMatch(/Drizzle/) - }) - - it('skips the supabase migration-vs-direct mode selection for v3', () => { - const routing = routeInstallPathForEqlVersion(3, { - supabase: true, - drizzle: false, - }) - expect(routing.drizzle).toBe(false) - expect(routing.useSupabaseInstallModeSelection).toBe(false) - expect(routing.notice).toBeUndefined() - }) - - it('does NOT auto-imply --supabase from --migration', () => { - // Even with --supabase: false explicitly, --migration must error. - const err = validateInstallFlags({ supabase: false, migration: true }) - expect(err).not.toBeNull() - }) -}) - -describe('migrationHeader', () => { - it('renders the header with the provided runner for npx', () => { - const header = migrationHeader('npx') - expect(header).toContain( - '-- CipherStash EQL — installed by `npx stash eql install --supabase --migration`.', - ) - }) - - it('renders the header with the provided runner for bunx', () => { - const header = migrationHeader('bunx') - expect(header).toContain('bunx stash eql install') - }) - - it('renders the header with the provided runner for pnpm dlx', () => { - const header = migrationHeader('pnpm dlx') - expect(header).toContain('pnpm dlx stash eql install') - }) - - it('includes all expected documentation lines', () => { - const header = migrationHeader('npx') - expect(header).toContain('eql_v2_encrypted') - expect(header).toContain( - 'https://cipherstash.com/docs/stack/cipherstash/supabase', - ) - }) -}) - -describe('chooseSupabaseInstallMode', () => { - const projectWith = { - hasMigrationsDir: true, - hasConfigToml: true, - migrationsDir: '/tmp/x', - } - const projectWithout = { - hasMigrationsDir: false, - hasConfigToml: false, - migrationsDir: '/tmp/x', - } - - it('honors explicit --migration regardless of TTY or detection', () => { - expect( - chooseSupabaseInstallMode({ migration: true }, projectWithout, true), - ).toBe('migration') - expect( - chooseSupabaseInstallMode({ migration: true }, projectWithout, false), - ).toBe('migration') - }) - - it('honors explicit --direct regardless of TTY or detection', () => { - expect(chooseSupabaseInstallMode({ direct: true }, projectWith, true)).toBe( - 'direct', - ) - expect( - chooseSupabaseInstallMode({ direct: true }, projectWith, false), - ).toBe('direct') - }) - - it('returns null in TTY mode when neither sub-flag is set (caller should prompt)', () => { - expect(chooseSupabaseInstallMode({}, projectWith, true)).toBeNull() - expect(chooseSupabaseInstallMode({}, projectWithout, true)).toBeNull() - }) - - it('non-interactive: defaults to migration when supabase/migrations exists', () => { - expect(chooseSupabaseInstallMode({}, projectWith, false)).toBe('migration') - }) - - it('non-interactive: defaults to direct when supabase/migrations is missing', () => { - expect(chooseSupabaseInstallMode({}, projectWithout, false)).toBe('direct') - }) -}) diff --git a/packages/cli/src/__tests__/v2-retirement.test.ts b/packages/cli/src/__tests__/v2-retirement.test.ts new file mode 100644 index 000000000..32140668b --- /dev/null +++ b/packages/cli/src/__tests__/v2-retirement.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from 'vitest' +import { registry } from '../cli/registry.js' +import { validateInstallFlags } from '../commands/db/install.js' + +const commands = registry.flatMap((group) => group.commands) + +describe('EQL v2 CLI retirement', () => { + it('removes Proxy configuration and v2 cutover commands from the manifest', () => { + const names = commands.map((command) => command.name) + + expect(names).not.toContain('db push') + expect(names).not.toContain('db activate') + expect(names).not.toContain('encrypt cutover') + }) + + it('removes generation selection and release downloads from install and upgrade', () => { + for (const name of ['eql install', 'eql upgrade']) { + const command = commands.find((candidate) => candidate.name === name) + const flags = command?.flags?.map((flag) => flag.name) ?? [] + + expect(flags).not.toContain('--eql-version') + expect(flags).not.toContain('--latest') + expect(flags).not.toContain('--exclude-operator-family') + } + }) + + it('removes the Proxy choice from init', () => { + const init = commands.find((command) => command.name === 'init') + const flags = init?.flags?.map((flag) => flag.name) ?? [] + + expect(flags).not.toContain('--proxy') + expect(flags).not.toContain('--no-proxy') + }) + + it('points legacy install requests at the upstream EQL 2.3.1 release', () => { + const releaseUrl = + 'https://github.com/cipherstash/encrypt-query-language/releases/tag/eql-2.3.1' + + expect(validateInstallFlags({ eqlVersion: '2' })).toContain(releaseUrl) + expect(validateInstallFlags({ latest: true })).toContain(releaseUrl) + }) + + it('rejects the obsolete operator-family flag because v3 self-adapts', () => { + expect(validateInstallFlags({ excludeOperatorFamily: true })).toMatch( + /self-adapts/, + ) + }) +}) diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index 1b164fa5b..94576492d 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -21,6 +21,7 @@ import { fileURLToPath } from 'node:url' import * as p from '@clack/prompts' import { CliExit } from '../cli/exit.js' import { renderCommandHelp } from '../cli/help.js' +import { validateInstallFlags } from '../commands/db/install.js' // Commands that depend on @cipherstash/stack are lazy-loaded in the switch below. import { authCommand, @@ -112,8 +113,6 @@ Commands: eql upgrade Upgrade EQL extensions to the latest version eql status Show EQL installation status - db push (EQL v2 + Proxy) Push encryption schema to eql_v2_configuration - db activate (EQL v2 + Proxy) Promote pending → active without renames db validate Validate encryption schema db migrate Run pending encrypt config migrations db test-connection Test database connectivity @@ -123,7 +122,6 @@ Commands: encrypt status Show per-column migration status (phase, progress, drift) encrypt plan Diff intent (.cipherstash/migrations.json) vs observed state encrypt backfill Resumably encrypt plaintext into the encrypted column - encrypt cutover Rename swap encrypted → primary column (EQL v2 only) encrypt drop Generate a migration to drop the plaintext column env Mint deployment credentials and print them as env vars @@ -190,19 +188,11 @@ async function runInstall( flags: Record<string, boolean>, values: Record<string, string>, ) { + rejectRetiredEqlFlags(flags, values) await installCommand({ force: flags.force, dryRun: flags['dry-run'], supabase: flags.supabase, - excludeOperatorFamily: flags['exclude-operator-family'], - drizzle: flags.drizzle, - latest: flags.latest, - name: values.name, - out: values.out, - migration: flags.migration, - direct: flags.direct, - migrationsDir: values['migrations-dir'], - eqlVersion: values['eql-version'], databaseUrl: values['database-url'], // An explicit `--database-url` is a one-shot install against that DB — leave // the project untouched. Otherwise offer to scaffold a config for later. @@ -214,16 +204,35 @@ async function runUpgrade( flags: Record<string, boolean>, values: Record<string, string>, ) { + rejectRetiredEqlFlags(flags, values) await upgradeCommand({ dryRun: flags['dry-run'], supabase: flags.supabase, - excludeOperatorFamily: flags['exclude-operator-family'], - latest: flags.latest, - eqlVersion: values['eql-version'], databaseUrl: values['database-url'], }) } +function rejectRetiredEqlFlags( + flags: Record<string, boolean>, + values: Record<string, string>, +): void { + const error = validateInstallFlags({ + eqlVersion: values['eql-version'], + latest: flags.latest, + drizzle: flags.drizzle, + name: values.name, + out: values.out, + migration: flags.migration, + direct: flags.direct, + migrationsDir: values['migrations-dir'], + excludeOperatorFamily: flags['exclude-operator-family'], + }) + if (error) { + p.log.error(error) + throw new CliExit(1) + } +} + async function runEqlCommand( sub: string | undefined, flags: Record<string, boolean>, @@ -282,19 +291,12 @@ async function runDbCommand( p.log.warn(messages.db.aliasDeprecated(STASH, 'upgrade')) await runUpgrade(flags, values) break - case 'push': { - const { pushCommand } = await requireStack( - () => import('../commands/db/push.js'), - ) - await pushCommand({ dryRun: flags['dry-run'], databaseUrl }) - break - } + case 'push': case 'activate': { - const { activateCommand } = await requireStack( - () => import('../commands/db/activate.js'), + p.log.error( + `stash db ${sub} was removed with the EQL v2 CipherStash Proxy configuration lifecycle. EQL v3 stores query configuration in its column domains and has nothing to push or activate.`, ) - await activateCommand({ databaseUrl }) - break + throw new CliExit(1) } case 'validate': { const { validateCommand } = await requireStack( @@ -366,18 +368,10 @@ async function runEncryptCommand( break } case 'cutover': { - const table = requireValue(values, 'table') - const column = requireValue(values, 'column') - const { cutoverCommand } = await requireStack( - () => import('../commands/encrypt/cutover.js'), + p.log.error( + '`stash encrypt cutover` was the EQL v2 rename/config-promotion command and has been removed. For EQL v3, finish the backfill, switch the application to the encrypted column by name, then run `stash encrypt drop` for the plaintext column.', ) - await cutoverCommand({ - table, - column, - proxyUrl: values['proxy-url'], - migrationsDir: values['migrations-dir'], - }) - break + throw new CliExit(1) } case 'drop': { const table = requireValue(values, 'table') diff --git a/packages/cli/src/cli/__tests__/help.test.ts b/packages/cli/src/cli/__tests__/help.test.ts index 5e737bca5..de0c48906 100644 --- a/packages/cli/src/cli/__tests__/help.test.ts +++ b/packages/cli/src/cli/__tests__/help.test.ts @@ -15,9 +15,8 @@ describe('renderCommandHelp', () => { ) expect(out).toContain('Options:') expect(out).toContain('--force') - // Value-taking flag renders its placeholder + default annotation. - expect(out).toContain('--eql-version <2|3>') - expect(out).toContain('(default: 3)') + // Value-taking flags render their placeholder. + expect(out).toContain('--database-url <url>') }) it('surfaces a flag env var alongside its description', () => { diff --git a/packages/cli/src/cli/registry.ts b/packages/cli/src/cli/registry.ts index d287dffde..dcbe319c0 100644 --- a/packages/cli/src/cli/registry.ts +++ b/packages/cli/src/cli/registry.ts @@ -130,15 +130,6 @@ export const registry: CommandGroup[] = [ description: 'Use Prisma Next-specific setup flow (EQL bundle installed via prisma-next migrate).', }, - { - name: '--proxy', - description: 'Query encrypted data via CipherStash Proxy.', - }, - { - name: '--no-proxy', - description: 'Query encrypted data directly via the SDK.', - default: 'true', - }, { ...REGION_FLAG, description: @@ -327,53 +318,6 @@ export const registry: CommandGroup[] = [ description: 'Use Supabase-compatible mode (auto-detected from DATABASE_URL).', }, - { - name: '--drizzle', - description: - 'Generate a Drizzle migration instead of direct install (auto-detected from project).', - }, - { - name: '--migration', - description: - 'Write a Supabase migration file instead of running SQL directly (requires --supabase).', - }, - { - name: '--direct', - description: - 'Run the SQL directly against the database (requires --supabase; mutually exclusive with --migration).', - }, - { - name: '--migrations-dir', - value: '<path>', - description: - 'Override the Supabase migrations directory (requires --supabase).', - default: 'supabase/migrations', - }, - EXCLUDE_OPERATOR_FAMILY_FLAG, - { - name: '--eql-version', - value: '<2|3>', - description: - 'EQL generation to target. v3 (native eql_v3.* domain schema) is the default; pass `2` for the legacy composite type. --drizzle, --migration, --migrations-dir, and --latest are v2-only and require `--eql-version 2`.', - default: '3', - }, - { - name: '--latest', - description: - 'Fetch the latest EQL from GitHub (v2 only — requires --eql-version 2).', - }, - { - name: '--name', - value: '<name>', - description: - 'With --drizzle: name for the generated migration (defaults to a scaffold name).', - }, - { - name: '--out', - value: '<path>', - description: - 'With --drizzle: directory to write the generated migration into.', - }, DATABASE_URL_FLAG, ], }, @@ -419,24 +363,7 @@ export const registry: CommandGroup[] = [ { name: 'eql upgrade', summary: 'Upgrade EQL extensions to the latest version', - flags: [ - DRY_RUN_FLAG, - SUPABASE_COMPAT_FLAG, - EXCLUDE_OPERATOR_FAMILY_FLAG, - { - name: '--eql-version', - value: '<2|3>', - description: - 'EQL generation to target. v3 is the default; pass `2` for the legacy composite type. --latest is v2-only and requires `--eql-version 2`.', - default: '3', - }, - { - name: '--latest', - description: - 'Fetch the latest EQL from GitHub (v2 only — requires --eql-version 2).', - }, - DATABASE_URL_FLAG, - ], + flags: [DRY_RUN_FLAG, SUPABASE_COMPAT_FLAG, DATABASE_URL_FLAG], }, { name: 'eql status', @@ -448,18 +375,6 @@ export const registry: CommandGroup[] = [ { title: 'Database', commands: [ - { - name: 'db push', - summary: - 'Push encryption schema (writes pending if active config already exists)', - flags: [DRY_RUN_FLAG, DATABASE_URL_FLAG], - }, - { - name: 'db activate', - summary: - 'Promote pending → active without renames (use after additive db push)', - flags: [DATABASE_URL_FLAG], - }, { name: 'db validate', summary: 'Validate encryption schema', @@ -544,24 +459,6 @@ export const registry: CommandGroup[] = [ }, ], }, - { - name: 'encrypt cutover', - summary: 'Rename swap encrypted → primary column (EQL v2 only)', - flags: [ - TABLE_FLAG, - COLUMN_FLAG, - { - name: '--proxy-url', - value: '<url>', - description: 'Proxy URL to verify against.', - }, - { - name: '--migrations-dir', - value: '<path>', - description: 'Directory to write the rename migration into.', - }, - ], - }, { name: 'encrypt drop', summary: 'Generate a migration to drop the plaintext column', diff --git a/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts b/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts index 8674bdf9f..a5cfe479f 100644 --- a/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts +++ b/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts @@ -2,13 +2,11 @@ import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { findGeneratedMigration } from '../install.js' +import { findGeneratedMigration } from '../../eql/migration.js' /** - * `findGeneratedMigration` was promoted to public API by the `eql migration` - * command and now has two consumers (`db install --drizzle` and - * `eql migration --drizzle`), so its branches are pinned directly — a change for - * one consumer must not silently break the other. + * `findGeneratedMigration` locates the custom migration scaffolded by + * `eql migration --drizzle`; pin its failure and ordering branches directly. */ describe('findGeneratedMigration', () => { let dir: string diff --git a/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts b/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts deleted file mode 100644 index cf4f085e8..000000000 --- a/packages/cli/src/commands/db/__tests__/generate-drizzle-migration.test.ts +++ /dev/null @@ -1,370 +0,0 @@ -import { - existsSync, - mkdirSync, - mkdtempSync, - readFileSync, - rmSync, - writeFileSync, -} from 'node:fs' -import { tmpdir } from 'node:os' -import { join, resolve } from 'node:path' -import * as p from '@clack/prompts' -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { CliExit } from '../../../cli/exit.js' -import { messages } from '../../../messages.js' - -// clack is chrome — silence it and spy on the channels the generator reports -// through. The spinner instance doubles as the `s` argument. -const clack = vi.hoisted(() => ({ - spinnerInstance: { start: vi.fn(), stop: vi.fn() }, - // `step` is on the real clack `log`; the sweep's per-statement report calls - // it, so it must be present or the report throws into the catch unseen. - log: { - info: vi.fn(), - warn: vi.fn(), - error: vi.fn(), - success: vi.fn(), - step: vi.fn(), - }, - intro: vi.fn(), - note: vi.fn(), - outro: vi.fn(), -})) -vi.mock('@clack/prompts', () => ({ - spinner: vi.fn(() => clack.spinnerInstance), - log: clack.log, - intro: clack.intro, - note: clack.note, - outro: clack.outro, -})) - -// Only the child process is faked — everything else (fs, bundled SQL) is real. -const spawnMock = vi.hoisted(() => vi.fn()) -vi.mock('node:child_process', () => ({ spawnSync: spawnMock })) - -// node:fs stays REAL by default — the generator's own writes, the bundled-SQL -// reads, and cleanup all need it. We only need a seam on `writeFileSync` to -// simulate the SQL-bundle write failing, so route it through a spy that -// delegates to the real implementation (reset in beforeEach) and is overridden -// only in the write-failure test. -const fsWrite = vi.hoisted(() => ({ - real: (() => { - throw new Error('fsWrite.real not initialised') - }) as typeof import('node:fs').writeFileSync, - spy: vi.fn(), -})) -vi.mock('node:fs', async (importOriginal) => { - const actual = await importOriginal<typeof import('node:fs')>() - fsWrite.real = actual.writeFileSync - return { ...actual, default: actual, writeFileSync: fsWrite.spy } -}) - -// The `--latest` path calls `downloadEqlSql`; stub just that export so we can -// make the network fetch reject. Everything else in the installer module -// (including the real `loadBundledEqlSql` the other tests rely on) stays real. -const downloadMock = vi.hoisted(() => vi.fn()) -vi.mock('@/installer/index.js', async (importOriginal) => ({ - ...(await importOriginal<typeof import('@/installer/index.js')>()), - downloadEqlSql: downloadMock, -})) - -// Pin the package manager so the argv assertion below is exact. Detection reads -// the lockfile in cwd and npm_config_user_agent, both of which vary by how the -// suite was launched. The runner MAPPING stays real — `pnpm` + `['exec', …]` is -// part of what's being asserted, so mocking it would defeat the test. -vi.mock('@/commands/init/utils.js', async (importOriginal) => ({ - ...(await importOriginal<typeof import('@/commands/init/utils.js')>()), - detectPackageManager: () => 'pnpm', -})) - -const { generateDrizzleMigration } = await import('../install.js') - -const spinner = p.spinner() - -beforeEach(() => { - // Default: `writeFileSync` behaves exactly like the real thing. Tests that - // need it to fail override the implementation for their own scope. - fsWrite.spy.mockImplementation(fsWrite.real) -}) - -afterEach(() => { - vi.clearAllMocks() -}) - -/** - * The v2 (`eql install --drizzle`) generator. Both regressions pinned here are - * invocation-level: an unvalidated `--name` reaching a shell string, and - * `--out` being computed for the search but never handed to drizzle-kit. - */ -describe('generateDrizzleMigration', () => { - let tmp: string - beforeEach(() => { - tmp = mkdtempSync(join(tmpdir(), 'stash-v2-drizzle-migration-')) - }) - afterEach(() => { - rmSync(tmp, { recursive: true, force: true }) - }) - - it('rejects a migration name with unsafe characters before spawning', async () => { - await expect( - generateDrizzleMigration(spinner, { - name: 'x; rm -rf ~', - out: join(tmp, 'drizzle'), - }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.log.error).toHaveBeenCalledWith(messages.eql.migrationBadName) - expect(spawnMock).not.toHaveBeenCalled() - }) - - it.each([ - ['command substitution', 'a$(whoami)'], - ['backticks', 'a`id`'], - ['a space', 'add eql'], - ['a path separator', '../escape'], - // `''` is not nullish, so it slips past `options.name ?? DEFAULT` and hits - // the regex, where `+` rejects it — `--name ''` aborts, it does NOT fall - // back to `install-eql`. The one input where "empty" and "absent" diverge. - ['an empty string', ''], - ])('rejects %s in --name', async (_label, name) => { - await expect( - generateDrizzleMigration(spinner, { name, out: join(tmp, 'drizzle') }), - ).rejects.toBeInstanceOf(CliExit) - expect(spawnMock).not.toHaveBeenCalled() - }) - - it('rejects an unsafe name in a dry run too (validation precedes the preview)', async () => { - await expect( - generateDrizzleMigration(spinner, { name: 'x; ls', dryRun: true }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.note).not.toHaveBeenCalled() - }) - - it('passes --name and --out to drizzle-kit as argv (no shell) and writes the SQL', async () => { - const out = join(tmp, 'db', 'migrations') - mkdirSync(out, { recursive: true }) - // Stand in for drizzle-kit scaffolding an empty custom migration. - spawnMock.mockImplementation(() => { - writeFileSync(join(out, '0000_add-eql.sql'), '') - return { status: 0, stdout: '', stderr: '' } - }) - - await generateDrizzleMigration(spinner, { name: 'add-eql', out }) - - expect(spawnMock).toHaveBeenCalledTimes(1) - const [command, argv] = spawnMock.mock.calls[0] - // The whole argv, exactly — not `toContain` checks, which would still pass - // if the runner prefix (`exec`) were dropped and drizzle-kit ran under the - // wrong resolver. DEFECT 1: name and out are discrete inert tokens in an - // array, never interpolated into a shell string. DEFECT 2: `--out` is - // actually passed, so drizzle-kit writes where step 2 then looks. The - // project-local `exec` form (not `dlx`) is asserted so a regression back to - // download-and-run — which resolves a different drizzle.config.ts — fails. - expect(command).toBe('pnpm') - expect(argv).toEqual([ - 'exec', - 'drizzle-kit', - 'generate', - '--custom', - '--name=add-eql', - `--out=${out}`, - ]) - - const written = readFileSync(join(out, '0000_add-eql.sql'), 'utf-8') - expect(written).toContain('cs_migrations') - }) - - // Step 5 of the generator sweeps sibling migrations. This block had no test at - // all: every other test's out dir contains only the file the generator wrote - // (which is `skip`ped), so the sweep never had anything to act on. A sibling - // ALTER on a plaintext column the corpus declares must be rewritten. - it('rewrites a sibling ALTER on a declared plaintext column', async () => { - const out = join(tmp, 'drizzle') - mkdirSync(out, { recursive: true }) - writeFileSync( - join(out, '0000_declare.sql'), - 'CREATE TABLE "users" ("email" text);\n', - ) - const sibling = join(out, '0001_encrypt-email.sql') - writeFileSync( - sibling, - 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n', - ) - spawnMock.mockImplementation(() => { - writeFileSync(join(out, '0002_install-eql.sql'), '') - return { status: 0, stdout: '', stderr: '' } - }) - - await generateDrizzleMigration(spinner, { out }) - - const rewritten = readFileSync(sibling, 'utf-8') - expect(rewritten).toContain( - 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', - ) - expect(rewritten).not.toContain('SET DATA TYPE') - const info = clack.log.info.mock.calls.map((c) => String(c[0])) - expect(info.some((msg) => msg.includes('Rewrote 1 migration file'))).toBe( - true, - ) - }) - - // A sibling ALTER on a column the corpus never declares is fail-closed to - // `source-unknown`: left on disk and reported with its remediation. Exercises - // the skip branch of the step-5 report, which no db-install test reached. - it('reports source-unknown for a sibling ALTER on an undeclared column', async () => { - const out = join(tmp, 'drizzle') - mkdirSync(out, { recursive: true }) - const sibling = join(out, '0001_encrypt-email.sql') - const unsafeAlter = - 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE "public"."eql_v3_text_search";\n' - writeFileSync(sibling, unsafeAlter) - spawnMock.mockImplementation(() => { - writeFileSync(join(out, '0002_install-eql.sql'), '') - return { status: 0, stdout: '', stderr: '' } - }) - - await generateDrizzleMigration(spinner, { out }) - - // Never rewritten — its current type is unknown and could be ciphertext. - expect(readFileSync(sibling, 'utf-8')).toBe(unsafeAlter) - const stepped = clack.log.step.mock.calls.map((c) => String(c[0])) - expect( - stepped.some((msg) => - msg.includes("Check the column's current type in the database"), - ), - ).toBe(true) - const warned = clack.log.warn.mock.calls.map((c) => String(c[0])) - expect(warned.some((msg) => msg.includes('did not fully complete'))).toBe( - true, - ) - }) - - it('includes --out in the dry-run preview', async () => { - const out = join(tmp, 'custom-out') - await generateDrizzleMigration(spinner, { dryRun: true, out }) - expect(spawnMock).not.toHaveBeenCalled() - expect(clack.note).toHaveBeenCalledWith( - expect.stringContaining(`--out=${out}`), - 'Dry Run', - ) - }) - - it('aborts with CliExit when drizzle-kit exits non-zero', async () => { - spawnMock.mockReturnValue({ status: 1, stdout: '', stderr: 'boom' }) - await expect( - generateDrizzleMigration(spinner, { out: join(tmp, 'drizzle') }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.log.error).toHaveBeenCalledWith('boom') - }) - - it('reports the spawn error when drizzle-kit cannot be launched', async () => { - // spawnSync's ENOENT shape: null status, no captured stderr, `error` set. - // `result.stderr?.trim()` is undefined, so the message falls through to the - // second arm (`result.error?.message`) — the realistic "drizzle-kit isn't - // installed" case. If the `?.` on stderr were dropped this shape would - // throw a TypeError instead of reporting. - spawnMock.mockReturnValue({ - status: null, - stdout: null, - stderr: null, - error: Object.assign(new Error('spawnSync pnpm ENOENT'), { - code: 'ENOENT', - }), - }) - - await expect( - generateDrizzleMigration(spinner, { out: join(tmp, 'drizzle') }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.log.error).toHaveBeenCalledWith('spawnSync pnpm ENOENT') - expect(clack.log.info).toHaveBeenCalledWith( - 'Make sure drizzle-kit is installed: npm install -D drizzle-kit', - ) - }) - - it('falls back to the exit status when there is no stderr or spawn error', async () => { - // Third arm of the fallback chain: non-zero status, empty stderr, no - // `error`. The user still gets a message instead of a blank error line. - spawnMock.mockReturnValue({ status: 2, stdout: '', stderr: '' }) - - await expect( - generateDrizzleMigration(spinner, { out: join(tmp, 'drizzle') }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.log.error).toHaveBeenCalledWith( - 'drizzle-kit exited with status 2.', - ) - }) - - it('defaults --out to an absolute drizzle/ when the flag is omitted', async () => { - // Widest blast radius in the diff: because `--out` is always appended, a - // flag-less invocation has its drizzle.config.ts `out` overridden by - // `<cwd>/drizzle`. The dry-run preview reaches that arm without touching - // the filesystem or spawning. - await generateDrizzleMigration(spinner, { dryRun: true }) - expect(clack.note).toHaveBeenCalledWith( - expect.stringContaining(`--out=${resolve('drizzle')}`), - 'Dry Run', - ) - }) - - it('points at --out when the generated migration is not where we looked', async () => { - const out = join(tmp, 'drizzle') - mkdirSync(out, { recursive: true }) - // drizzle-kit "succeeds" but wrote to its drizzle.config.ts `out`, not ours, - // so step 2 finds nothing and aborts with the remediation hint (defect 2). - spawnMock.mockReturnValue({ status: 0, stdout: '', stderr: '' }) - - await expect( - generateDrizzleMigration(spinner, { out }), - ).rejects.toBeInstanceOf(CliExit) - expect(clack.log.info).toHaveBeenCalledWith( - 'If your drizzle.config.ts writes elsewhere, pass --out <dir> so it matches.', - ) - }) - - it('removes the scaffolded migration when writing the SQL fails', async () => { - const out = join(tmp, 'drizzle') - mkdirSync(out, { recursive: true }) - const scaffolded = join(out, '0000_install-eql.sql') - // drizzle-kit scaffolds an empty custom migration where we look. - spawnMock.mockImplementation(() => { - fsWrite.real(scaffolded, '') - return { status: 0, stdout: '', stderr: '' } - }) - // Throw only on the big bundle write, letting the empty scaffold through. - fsWrite.spy.mockImplementation(((path, data, ...rest) => { - if (typeof data === 'string' && data.includes('cs_migrations')) { - throw new Error('EACCES: permission denied') - } - return fsWrite.real( - path as string, - data as string, - ...(rest as unknown[]), - ) - }) as typeof import('node:fs').writeFileSync) - - await expect( - generateDrizzleMigration(spinner, { out }), - ).rejects.toBeInstanceOf(CliExit) - // Without cleanup, an empty scaffolded migration survives and - // `drizzle-kit migrate` applies and records it — a silent no-op migration. - expect(existsSync(scaffolded)).toBe(false) - expect(clack.log.error).toHaveBeenCalledWith('EACCES: permission denied') - }) - - it('cleans up the scaffold when --latest cannot fetch the SQL', async () => { - const out = join(tmp, 'drizzle') - mkdirSync(out, { recursive: true }) - const scaffolded = join(out, '0000_install-eql.sql') - spawnMock.mockImplementation(() => { - fsWrite.real(scaffolded, '') - return { status: 0, stdout: '', stderr: '' } - }) - // A network failure (rate limit / offline) is the likeliest real trip here. - downloadMock.mockRejectedValueOnce(new Error('403 rate limited')) - - await expect( - generateDrizzleMigration(spinner, { out, latest: true }), - ).rejects.toBeInstanceOf(CliExit) - expect(existsSync(scaffolded)).toBe(false) - expect(clack.log.error).toHaveBeenCalledWith('403 rate limited') - }) -}) diff --git a/packages/cli/src/commands/db/activate.ts b/packages/cli/src/commands/db/activate.ts deleted file mode 100644 index 86e19af72..000000000 --- a/packages/cli/src/commands/db/activate.ts +++ /dev/null @@ -1,73 +0,0 @@ -import { activateConfig, migrateConfig } from '@cipherstash/migrate' -import * as p from '@clack/prompts' -import pg from 'pg' -import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' -import { loadStashConfig } from '@/config/index.js' - -/** - * `stash db activate` — promote the pending EQL configuration to active - * **without** renaming any columns. - * - * Used after `stash db push` when the new config is purely additive - * (e.g. registering a brand-new encrypted column on a project that - * already has an active config) — there are no `<col>_encrypted` twins - * to rename, so the cut-over rename step is unnecessary. - * - * For path 3 (existing populated column → encrypted via lifecycle), use - * `stash encrypt cutover` instead. Cutover does the same activation but - * also runs the physical rename. - * - * Mechanics: chains `eql_v2.migrate_config()` (pending → encrypting) and - * `eql_v2.activate_config()` (encrypting → active) inside a single - * transaction. Errors out clearly when there is no pending config to - * activate. - */ -export interface ActivateCommandOptions { - databaseUrl?: string -} - -export async function activateCommand( - options: ActivateCommandOptions, -): Promise<void> { - p.intro(runnerCommand(detectPackageManager(), 'stash db activate')) - - const stashConfig = await loadStashConfig({ - databaseUrlFlag: options.databaseUrl, - }) - const client = new pg.Client({ connectionString: stashConfig.databaseUrl }) - let exitCode = 0 - - try { - await client.connect() - - const pending = await client.query<{ exists: boolean }>( - "SELECT EXISTS(SELECT 1 FROM public.eql_v2_configuration WHERE state = 'pending') AS exists", - ) - if (pending.rows[0]?.exists !== true) { - p.log.error( - 'No pending EQL configuration to activate. This applies to the EQL v2 + CipherStash Proxy config lifecycle — register a pending change before activating.', - ) - exitCode = 1 - return - } - - await client.query('BEGIN') - try { - await migrateConfig(client) - await activateConfig(client) - await client.query('COMMIT') - } catch (err) { - await client.query('ROLLBACK').catch(() => {}) - throw err - } - - p.log.success('Pending configuration promoted to active.') - p.outro('Done.') - } catch (error) { - p.log.error(error instanceof Error ? error.message : 'Activation failed.') - exitCode = 1 - } finally { - await client.end() - } - if (exitCode) process.exit(exitCode) -} diff --git a/packages/cli/src/commands/db/config-scaffold.ts b/packages/cli/src/commands/db/config-scaffold.ts index 8bec16da2..b8b09f6d1 100644 --- a/packages/cli/src/commands/db/config-scaffold.ts +++ b/packages/cli/src/commands/db/config-scaffold.ts @@ -96,8 +96,8 @@ function writeStashConfig(configPath: string, clientPath: string): string { } /** - * Create a `stash.config.ts` for the rest of the workflow (`db push` / - * `schema build` / `encrypt *` load the encryption client through it). + * Create a `stash.config.ts` for the rest of the workflow (`db validate` and + * `encrypt *` load the encryption client through it). * `eql install` itself doesn't need one — it resolves the database URL * directly — so this is a setup convenience, never a blocker. * @@ -136,7 +136,7 @@ export async function offerStashConfig( if (!isInteractive()) return null const create = await p.confirm({ - message: `Create a ${CONFIG_FILENAME}? (needed later for db push / schema build / encrypt)`, + message: `Create a ${CONFIG_FILENAME}? (needed later for db validate / encrypt)`, initialValue: true, }) if (p.isCancel(create) || !create) { diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index 60cfb0aae..b17c9061a 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -1,154 +1,65 @@ -import { spawnSync } from 'node:child_process' -import { existsSync, unlinkSync, writeFileSync } from 'node:fs' -import { readdir } from 'node:fs/promises' -import { join, resolve } from 'node:path' -import { - installMigrationsSchema, - MIGRATIONS_SCHEMA_SQL, -} from '@cipherstash/migrate' +import { installMigrationsSchema } from '@cipherstash/migrate' import * as p from '@clack/prompts' import pg from 'pg' -import { CliExit } from '@/cli/exit.js' -import { - detectPackageManager, - execArgv, - execCommand, - runnerCommand, -} from '@/commands/init/utils.js' import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' -import { isInteractive } from '@/config/tty.js' -import { - downloadEqlSql, - EQLInstaller, - loadBundledEqlSql, - resolveEqlVersion, -} from '@/installer/index.js' +import { EQLInstaller } from '@/installer/index.js' import { messages } from '@/messages.js' +import { detectPackageManager, runnerCommand } from '../init/utils.js' import { ensureEncryptionClient } from './client-scaffold.js' import { offerStashConfig } from './config-scaffold.js' -import { - detectDrizzle, - detectPrismaNext, - detectSupabase, - detectSupabaseProject, - type SupabaseProjectInfo, -} from './detect.js' -import { - describeSkipReason, - rewriteEncryptedAlterColumns, -} from './rewrite-migrations.js' -import { - SUPABASE_EQL_MIGRATION_FILENAME, - writeSupabaseEqlMigration, -} from './supabase-migration.js' +import { detectPrismaNext, detectSupabase } from './detect.js' -const DEFAULT_MIGRATION_NAME = 'install-eql' -const DEFAULT_DRIZZLE_OUT = 'drizzle' - -/** - * File-system-safe migration name: what drizzle-kit accepts, and shell-inert. - * - * Lives here rather than in `commands/eql/migration.ts` (the v3 generator) only - * because that module already imports from this one — keeping the constant on - * this side of the existing edge avoids an import cycle. Both generators share - * it; there must not be a second copy. - */ export const SAFE_MIGRATION_NAME = /^[\w-]+$/ +export const EQL_V2_RELEASE_URL = + 'https://github.com/cipherstash/encrypt-query-language/releases/tag/eql-2.3.1' + export interface InstallOptions { force?: boolean dryRun?: boolean - /** - * `undefined` means "auto-detect" (via {@link detectSupabase}). An explicit - * `true`/`false` from the user is preserved and skips detection. - */ - excludeOperatorFamily?: boolean supabase?: boolean - drizzle?: boolean - latest?: boolean - name?: string - out?: string - /** - * Write the EQL install SQL into a Supabase migration file instead of - * running it directly against the database. Requires `--supabase`. - */ - migration?: boolean - /** - * Run the EQL install SQL directly against the database (current behavior). - * Requires `--supabase`. Mutually exclusive with `--migration`. - */ - direct?: boolean - /** - * Override the directory the Supabase migration file is written into. - * Defaults to `<cwd>/supabase/migrations`. - */ - migrationsDir?: string - /** - * Connection string passed via `--database-url`. Used for this run only — - * never persisted. See `src/config/database-url.ts`. - */ databaseUrl?: string - /** - * How to handle a missing `stash.config.ts` — the caller owns this intent - * rather than it being inferred from whether a URL was supplied: - * - `'ensure'` — create it without asking (`stash init`, where the user has - * already committed to setting the project up). - * - `'offer'` — offer to create it (plain `stash eql install`). Default. - * - `'skip'` — never scaffold (a one-shot `eql install --database-url ...`). - */ scaffoldConfig?: 'ensure' | 'offer' | 'skip' - /** - * EQL generation to install: `'3'` (default, native `eql_v3.*` domain - * schema) or `'2'` (composite `eql_v2_encrypted`). v3 currently supports the - * direct install path only — `--drizzle`, `--migration`, `--migrations-dir`, - * and `--latest` require an explicit `'2'`. - */ - eqlVersion?: string } -/** Resolved install mode for the Supabase non-Drizzle branch. */ -export type SupabaseInstallMode = 'migration' | 'direct' +/** Recognise removed argv before any I/O instead of silently ignoring it. */ +export function validateInstallFlags( + options: Record<string, unknown>, +): string | null { + if (options.eqlVersion === '2' || options.latest === true) { + return `EQL v2 installation has been removed from stash. To recover or restore an existing EQL v2 database dump, download the EQL 2.3.1 SQL from ${EQL_V2_RELEASE_URL}. New installs use EQL v3.` + } + if (options.eqlVersion !== undefined) { + return '`--eql-version` has been removed. EQL v3 is now the only installable generation; remove the flag.' + } + if ( + options.drizzle === true || + options.name !== undefined || + options.out !== undefined + ) { + return '`eql install --drizzle` has been removed. Generate an EQL v3 Drizzle migration with `stash eql migration --drizzle` (and pass --name/--out there).' + } + if (options.migration === true || options.migrationsDir !== undefined) { + return '`eql install --migration` has been removed. Use `stash eql migration --drizzle` to keep the EQL v3 install in migration history, adding `--supabase` when needed.' + } + if (options.direct === true) { + return '`--direct` has been removed because `stash eql install` is now always a direct EQL v3 install.' + } + if (options.excludeOperatorFamily === true) { + return '`--exclude-operator-family` has been removed. The pinned EQL v3 bundle self-adapts when the database role cannot create its optional operator family.' + } + return null +} -/** - * Resolve the database URL + encryption-client path for the install. - * - * A pre-existing `stash.config.ts` is authoritative — later workflow commands - * (db push / schema build / encrypt) load the client through it — so we load - * it. Without one, `eql install` doesn't need a config: resolve the URL - * directly (flag → env → supabase → prompt). Decoupling install from the config - * is what lets `npx stash eql install --database-url ...` run in a bare project - * without the `stash` / `@cipherstash/stack` dependencies the config would - * otherwise import (#579). - * - * Whether a missing config gets scaffolded is the caller's explicit intent - * ({@link InstallOptions.scaffoldConfig}), not inferred from whether a URL was - * supplied — `stash init` passes a resolved URL but still wants a config, while - * a one-shot `--database-url` run wants the project left untouched. When no - * config is created, `clientPath` comes back `null` so the caller skips the - * client scaffold too. - * - * A one-shot run (`mode === 'skip'`, set when `--database-url` is passed alone) - * bypasses config loading entirely: it must leave the project untouched, so it - * neither honours nor scaffolds a config or client. This also means the flag - * always wins — loading a config here could pick up a parent-directory config - * with a hand-edited literal `databaseUrl` that silently overrides the flag and - * installs EQL against the wrong database. - */ async function resolveInstallContext( options: InstallOptions, s: ReturnType<typeof p.spinner>, ): Promise<{ databaseUrl: string; clientPath: string | null }> { const mode = options.scaffoldConfig ?? 'offer' - - // A one-shot `--database-url` install leaves the project untouched: don't load - // an existing config (see doc comment re: parent-dir literal override) and - // don't scaffold. Resolve the URL directly and return a null clientPath. const configPath = mode === 'skip' ? null : findConfigFile(process.cwd()) if (configPath) { s.start('Loading stash.config.ts...') - // Pass the path we already located so loadStashConfig doesn't re-walk the - // filesystem to find it. const config = await loadStashConfig( { databaseUrlFlag: options.databaseUrl, @@ -157,11 +68,6 @@ async function resolveInstallContext( configPath, ) s.stop('Configuration loaded.') - - // A config with a hand-edited literal `databaseUrl` bypasses the resolver, - // so a `--database-url` flag would be silently ignored. Surface it rather - // than installing against a different database than the user asked for — - // especially since `findConfigFile` walks up into parent directories. if ( options.databaseUrl !== undefined && config.databaseUrl !== options.databaseUrl.trim() @@ -177,45 +83,19 @@ async function resolveInstallContext( databaseUrlFlag: options.databaseUrl, supabase: options.supabase, }) - - // A dry run must not write scaffold files, so it never enters the scaffold - // path (nor does an explicit `skip`). if (mode === 'skip' || options.dryRun) { return { databaseUrl, clientPath: null } } - const clientPath = await offerStashConfig({ ensure: mode === 'ensure' }) return { databaseUrl, clientPath } } -/** - * What `installCommand` actually did — so callers (notably `stash init`'s - * completion gate) can tell "EQL is now in the database" from "a migration - * file was written but nothing was applied yet". - * - * - `installed` / `already-installed`: EQL is present in the target database. - * - `migration-generated`: a migration file was written (Drizzle, or the - * Supabase `--migration` mode); the user still has to APPLY it - * (`drizzle-kit migrate` / `supabase db push`) before EQL exists in the DB. - * - `dry-run`: nothing was changed. - * - * Terminal error paths call `process.exit(1)` and never return an outcome. - */ -export type InstallOutcome = - | 'installed' - | 'already-installed' - | 'migration-generated' - | 'dry-run' +export type InstallOutcome = 'installed' | 'already-installed' | 'dry-run' export async function installCommand( options: InstallOptions, ): Promise<InstallOutcome> { p.intro(runnerCommand(detectPackageManager(), 'stash eql install')) - - // Validate mutually-exclusive / supabase-required flags BEFORE doing any - // I/O. `--migration` and `--direct` only make sense in the Supabase flow; - // they must NOT implicitly enable `--supabase`. (Strong product preference - // — auto-enabling here has bitten users before.) const flagError = validateInstallFlags(options) if (flagError) { p.log.error(flagError) @@ -223,10 +103,6 @@ export async function installCommand( process.exit(1) } - // Prisma Next owns EQL installation via its own migration system, so the - // standalone installer is the wrong tool here. Refuse (before any DB I/O) - // unless --force. Fires fast so a user who typed the wrong command gets a - // pointer, not a half-applied install. const prismaNextBlock = prismaNextInstallGuard(process.cwd(), options) if (prismaNextBlock) { p.log.error(prismaNextBlock) @@ -235,143 +111,52 @@ export async function installCommand( } const s = p.spinner() - - // `eql install` only needs a database URL to install EQL. It does NOT require - // a stash.config.ts: an existing config is authoritative (later workflow - // commands rely on it), but without one we resolve the URL directly — so a - // standalone `npx stash eql install --database-url ...` works with zero - // project dependencies. A one-shot `--database-url` run leaves the project - // untouched; otherwise we offer to scaffold a config for the rest of the - // workflow (CIP-2986 / #579). const { databaseUrl, clientPath } = await resolveInstallContext(options, s) - - // Safety net: if the user ran `eql install` without first running `init`, - // scaffold the encryption client file so clientPath points somewhere real. - // No-op when the file already exists. Skipped for a one-shot `--database-url` - // install (clientPath === null) or a dry run, which leave the project - // untouched — an existing config still yields a clientPath, so guard on dryRun - // too. if (clientPath && !options.dryRun) { ensureEncryptionClient(clientPath, process.cwd(), databaseUrl) } - // Auto-detect provider hints when the user didn't explicitly pass flags. - // CIP-2985. - const resolved = resolveProviderOptions(options, databaseUrl) - - const eqlVersion: 2 | 3 = resolveEqlVersion(options.eqlVersion) - - // v3 supports the direct install path only. Explicit --drizzle/--migration - // are rejected up-front by validateInstallFlags; auto-DETECTED drizzle or - // migration modes fall back to direct here rather than erroring. - const routing = routeInstallPathForEqlVersion(eqlVersion, resolved) - if (routing.notice) { - p.log.info(routing.notice) - } - - if (routing.drizzle) { - await generateDrizzleMigration(s, { - name: options.name, - out: options.out, - dryRun: options.dryRun, - latest: options.latest, - supabase: resolved.supabase, - excludeOperatorFamily: resolved.excludeOperatorFamily, - }) - // A Drizzle migration file was written — NOT applied. EQL only lands in - // the database once the user runs `drizzle-kit migrate`. - return 'migration-generated' - } - - // Supabase non-Drizzle path: pick between writing a migration file and - // running SQL directly. Detection of `supabase/migrations/` only seeds the - // prompt default — it never enables `--supabase`. Direct install is the - // historical default and remains the fallback when nothing else applies. - // v3 skips the mode selection entirely: direct install only for now. - if (routing.useSupabaseInstallModeSelection) { - const projectInfo = detectSupabaseProject( - process.cwd(), - options.migrationsDir, + const supabase = + options.supabase === undefined + ? detectSupabase(databaseUrl) + : options.supabase + if (options.supabase === undefined && supabase) { + p.log.info( + 'Detected Supabase database from DATABASE_URL — enabling --supabase.', ) - const mode = await resolveSupabaseInstallMode(options, projectInfo) - - if (mode === 'migration') { - // CIP: --latest in the migration path is not yet implemented. Loading - // the bundled SQL works today; downloading from GitHub adds an extra - // moving part we'd rather defer until someone needs it. - if (options.latest) { - p.log.error( - '`eql install --supabase --migration --latest` is not yet supported. Please open an issue at https://github.com/cipherstash/stack/issues if you need this.', - ) - p.outro('Installation aborted.') - process.exit(1) - } - - await writeSupabaseMigrationFile(s, { - projectInfo, - force: options.force, - dryRun: options.dryRun, - }) - // Migration file written, not applied — the user runs it via their - // Supabase migration workflow (`supabase db push`). - return 'migration-generated' - } - // mode === 'direct' — fall through to existing direct-install behavior. } if (options.dryRun) { p.log.info('Dry run — no changes will be made.') - const source = options.latest - ? 'Would download EQL install script from GitHub' - : `Would use bundled EQL${eqlVersion === 3 ? ' v3' : ''} install script` - p.note(`${source}\nWould execute the SQL against the database`, 'Dry Run') + p.note( + 'Would use the pinned EQL v3 install SQL\nWould execute the SQL against the database', + 'Dry Run', + ) p.outro('Dry run complete.') return 'dry-run' } - const installer = new EQLInstaller({ - databaseUrl, - }) - + const installer = new EQLInstaller({ databaseUrl }) s.start('Checking database permissions...') const permissions = await installer.checkPermissions() - - // CIP-2989: if the role is not a superuser and neither --supabase nor - // --exclude-operator-family was passed, auto-fall back to the - // no-operator-family (OPE) install variant. This is the same thing an - // experienced user would do manually; doing it automatically avoids the - // "what flag do I need?" failure mode on Supabase/Neon/RDS. - let excludeOperatorFamily = resolved.excludeOperatorFamily - if ( - !permissions.isSuperuser && - !resolved.supabase && - options.excludeOperatorFamily === undefined - ) { - excludeOperatorFamily = true - s.stop( - 'Role lacks superuser — falling back to the no-operator-family (OPE) install.', - ) - } else if (!permissions.ok) { + if (!permissions.ok) { s.stop('Insufficient database permissions.') p.log.error('The connected database role is missing required permissions:') - for (const missing of permissions.missing) { - p.log.warn(` - ${missing}`) - } - p.note( - 'EQL installation requires a role with CREATE SCHEMA,\nCREATE TYPE, and CREATE EXTENSION privileges.\n\nConnect with a superuser or admin role, or ask your\ndatabase administrator to grant the required permissions.', - 'Required Permissions', - ) + for (const missing of permissions.missing) p.log.warn(` - ${missing}`) p.outro('Installation aborted.') process.exit(1) + } else if (!permissions.isSuperuser && !supabase) { + s.stop( + 'Database permissions verified. The EQL v3 bundle will self-skip optional operator classes this role cannot create.', + ) } else { s.stop('Database permissions verified.') } if (!options.force) { s.start('Checking if EQL is already installed...') - const installed = await installer.isInstalled({ eqlVersion }) + const installed = await installer.isInstalled() s.stop(installed ? 'EQL is already installed.' : 'EQL is not installed.') - if (installed) { p.log.info('Use --force to re-run the install script.') p.outro('Nothing to do.') @@ -379,21 +164,10 @@ export async function installCommand( } } - const source = options.latest ? 'from GitHub (latest)' : 'bundled' - s.start( - `Installing EQL ${eqlVersion === 3 ? 'v3 ' : ''}extensions (${source})...`, - ) - await installer.install({ - excludeOperatorFamily, - supabase: resolved.supabase, - latest: options.latest, - eqlVersion, - }) + s.start('Installing EQL v3 extensions (pinned bundle)...') + await installer.install({ supabase }) s.stop('EQL extensions installed.') - - if (resolved.supabase) { - p.log.success('Supabase role permissions granted.') - } + if (supabase) p.log.success('Supabase role permissions granted.') s.start('Installing cs_migrations tracking schema...') const migrationsDb = new pg.Client({ connectionString: databaseUrl }) @@ -417,45 +191,17 @@ export async function installCommand( return 'installed' } -/** - * Merge explicit CLI flags with auto-detected hints. - * - * Rules: - * - `--supabase` explicitly passed wins. - * - `--supabase` not passed → if the database URL looks like Supabase, enable it. - * - `--drizzle` explicitly passed wins. - * - `--drizzle` not passed → if drizzle-orm/drizzle-kit/drizzle.config.* exists, enable it. - * - `--exclude-operator-family` explicitly passed wins. - */ -function resolveProviderOptions( - options: InstallOptions, - databaseUrl: string, -): { - supabase: boolean - drizzle: boolean - excludeOperatorFamily: boolean -} { - const supabase = - options.supabase === undefined - ? detectSupabase(databaseUrl) - : options.supabase - if (options.supabase === undefined && supabase) { - p.log.info( - 'Detected Supabase database from DATABASE_URL — enabling --supabase.', - ) - } - - const drizzle = - options.drizzle === undefined - ? detectDrizzle(process.cwd()) - : options.drizzle - if (options.drizzle === undefined && drizzle) { - p.log.info('Detected Drizzle in this project — enabling --drizzle.') - } - - const excludeOperatorFamily = options.excludeOperatorFamily ?? false - - return { supabase, drizzle, excludeOperatorFamily } +export function prismaNextInstallGuard( + cwd: string, + options: Pick<InstallOptions, 'force'>, +): string | null { + if (options.force || !detectPrismaNext(cwd)) return null + return ( + `${messages.eql.prismaNextDetected} (found prisma-next.config.* or @cipherstash/prisma-next). ` + + 'Prisma Next installs the EQL bundle through its own migration system — run ' + + '`prisma-next migrate` instead of `stash eql install`. ' + + 'Pass --force to run the standalone installer against this database anyway.' + ) } export function printNextSteps(): void { @@ -479,547 +225,3 @@ export function printNextSteps(): void { 'What next', ) } - -/** - * Generate a Drizzle migration that installs CipherStash EQL. - * - * Uses `drizzle-kit generate --custom` to scaffold an empty migration, - * then loads the EQL install SQL (bundled by default, or from GitHub with - * `--latest`) and writes it into the file. - * - * Exported for unit tests; not part of the CLI's public surface. Failures - * throw {@link CliExit} rather than calling `process.exit` so the telemetry - * `finally` in `main.ts` still runs (and the branches stay testable). - */ -export async function generateDrizzleMigration( - s: ReturnType<typeof p.spinner>, - options: { - name?: string - out?: string - dryRun?: boolean - latest?: boolean - supabase?: boolean - excludeOperatorFamily?: boolean - }, -) { - const migrationName = options.name ?? DEFAULT_MIGRATION_NAME - if (!SAFE_MIGRATION_NAME.test(migrationName)) { - p.log.error(messages.eql.migrationBadName) - p.outro('Migration aborted.') - throw new CliExit(1) - } - const outDir = resolve(options.out ?? DEFAULT_DRIZZLE_OUT) - - // Run the PROJECT-LOCAL drizzle-kit (`pnpm exec` / `npx --no-install`), not - // the download-and-run form (`pnpm dlx`) — it must resolve THIS project's - // drizzle.config.ts and schema, and `--no-install` fails loudly instead of - // surprise-downloading a possibly-different drizzle-kit major. Matches the v3 - // generator (`commands/eql/migration.ts`). Invoke via spawnSync with an argv - // array (no shell), so a `--name` carrying spaces or shell metacharacters is - // one inert token, never word-split or executed. `--out` is always passed so - // drizzle-kit WRITES where we then LOOK — otherwise a project whose - // drizzle.config.ts points elsewhere would have drizzle-kit write there while - // we search `drizzle/` and fail in step 2. - const pm = detectPackageManager() - const { command, prefixArgs } = execArgv(pm) - const drizzleArgs = [ - ...prefixArgs, - 'drizzle-kit', - 'generate', - '--custom', - `--name=${migrationName}`, - `--out=${outDir}`, - ] - const drizzleCmd = `${execCommand(pm)} ${drizzleArgs.slice(prefixArgs.length).join(' ')}` - - if (options.dryRun) { - p.log.info('Dry run — no changes will be made.') - const source = options.latest - ? 'Would download EQL install SQL from GitHub' - : 'Would use bundled EQL install SQL' - p.note( - `Would run: ${drizzleCmd}\n${source}\nWould write SQL to migration file in ${outDir}`, - 'Dry Run', - ) - p.outro('Dry run complete.') - return - } - - let generatedMigrationPath: string | undefined - - // Step 1: Generate a custom Drizzle migration - s.start('Generating custom Drizzle migration...') - - const result = spawnSync(command, drizzleArgs, { - stdio: 'pipe', - encoding: 'utf-8', - }) - if (result.status !== 0) { - s.stop('Failed to generate migration.') - const stderr = result.stderr?.trim() - p.log.error( - stderr || - result.error?.message || - `drizzle-kit exited with status ${result.status ?? 'unknown'}.`, - ) - p.log.info('Make sure drizzle-kit is installed: npm install -D drizzle-kit') - p.outro('Migration aborted.') - throw new CliExit(1) - } - s.stop('Custom Drizzle migration generated.') - - // Step 2: Find the generated migration file - s.start('Locating generated migration file...') - - try { - generatedMigrationPath = await findGeneratedMigration(outDir, migrationName) - s.stop(`Found migration: ${generatedMigrationPath}`) - } catch (error) { - s.stop('Failed to locate migration file.') - p.log.error(error instanceof Error ? error.message : String(error)) - p.log.info( - 'If your drizzle.config.ts writes elsewhere, pass --out <dir> so it matches.', - ) - p.outro('Migration aborted.') - throw new CliExit(1) - } - - // Step 3: Load the EQL SQL (bundled or from GitHub). Thread supabase / - // excludeOperatorFamily through so the user's flag reaches the SQL - // selection — previously this path ignored both (CIP-2988). - let eqlSql: string - const sqlOptions = { - supabase: options.supabase ?? false, - excludeOperatorFamily: options.excludeOperatorFamily ?? false, - } - - if (options.latest) { - s.start('Downloading EQL install script from GitHub (latest)...') - try { - eqlSql = await downloadEqlSql(sqlOptions) - s.stop('EQL install script downloaded.') - } catch (error) { - s.stop('Failed to download EQL install script.') - p.log.error(error instanceof Error ? error.message : String(error)) - cleanupMigrationFile(generatedMigrationPath) - p.outro('Migration aborted.') - throw new CliExit(1) - } - } else { - s.start('Loading bundled EQL install script...') - try { - eqlSql = loadBundledEqlSql(sqlOptions) - s.stop('Bundled EQL install script loaded.') - } catch (error) { - s.stop('Failed to load bundled EQL install script.') - p.log.error(error instanceof Error ? error.message : String(error)) - cleanupMigrationFile(generatedMigrationPath) - p.outro('Migration aborted.') - throw new CliExit(1) - } - } - - // Step 4: Write the EQL SQL (and cs_migrations tracking schema) into - // the migration file. Bundling both means `drizzle-kit migrate` rolls - // everything needed for `stash encrypt ...` out to each environment - // in one go, rather than requiring an out-of-band `stash eql install`. - s.start('Writing EQL SQL into migration file...') - - const migrationContents = `${eqlSql}\n\n-- CipherStash encryption-migration tracking schema.\n-- Tracks per-column phase + backfill progress for \`stash encrypt\`.\n${MIGRATIONS_SCHEMA_SQL.trim()}\n` - - try { - writeFileSync(generatedMigrationPath, migrationContents, 'utf-8') - s.stop('EQL SQL written to migration file.') - } catch (error) { - s.stop('Failed to write migration file.') - p.log.error(error instanceof Error ? error.message : String(error)) - cleanupMigrationFile(generatedMigrationPath) - p.outro('Migration aborted.') - throw new CliExit(1) - } - - // Step 5: Sweep for sibling migrations that drizzle-kit may have emitted - // with `ALTER COLUMN ... SET DATA TYPE eql_v2_encrypted`. These fail in - // Postgres because there's no implicit cast from text/numeric to the - // encrypted type. Rewrite them into a runnable ADD+DROP+RENAME sequence. - // That sequence is equivalent to DROP+ADD — safe on an EMPTY table, but - // data-destroying on a populated one — so the rewritten file carries a - // comment steering populated tables to the staged `stash encrypt` path. - // CIP-2991 + CIP-2994. - let sweepIncomplete = false - try { - const { rewritten, skipped } = await rewriteEncryptedAlterColumns(outDir, { - skip: generatedMigrationPath, - }) - if (rewritten.length > 0) { - p.log.info( - `Rewrote ${rewritten.length} migration file(s) into a runnable ADD+DROP+RENAME for encrypted columns (safe on empty tables; see each file's header before running against populated data):`, - ) - for (const file of rewritten) p.log.step(` - ${file}`) - } - if (skipped.length > 0) { - sweepIncomplete = true - p.log.warn( - `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep left alone. Review and fix them before running your migrations:`, - ) - for (const { file, statement, reason } of skipped) { - p.log.step(` - ${file}: ${statement}`) - p.log.step(` ${describeSkipReason(reason)}`) - } - } - } catch (error) { - sweepIncomplete = true - p.log.warn( - `Could not rewrite ALTER COLUMN migrations: ${error instanceof Error ? error.message : String(error)}`, - ) - } - - p.log.success(`Migration created: ${generatedMigrationPath}`) - if (sweepIncomplete) { - p.log.warn( - `The ALTER COLUMN sweep did not fully complete — review the sibling migrations in ${outDir} before running drizzle-kit migrate, or you may apply broken/unsafe SQL.`, - ) - } - p.note( - `Run your Drizzle migrations to install EQL:\n\n ${execCommand(detectPackageManager())} drizzle-kit migrate`, - 'Next Steps', - ) - printNextSteps() - p.outro('Done!') -} - -/** - * Validate flag combinations that we can detect without doing any I/O. - * - * Rules: - * - `--migration` and `--direct` are mutually exclusive. - * - `--migration`, `--direct`, and `--migrations-dir` each REQUIRE - * `--supabase`. They do NOT auto-imply it. - * - * Returns a user-facing error message, or `null` when the flags are valid. - */ -/** - * Route the install between the drizzle / supabase-migration / direct paths - * for the requested EQL generation. Pure — no I/O, no prompts — so the v3 - * fallback behaviour is unit-testable. - * - * v3 supports the direct path only: auto-detected drizzle falls back to - * direct with a user-facing notice (explicit `--drizzle`/`--migration` are - * already rejected by {@link validateInstallFlags}), and the Supabase - * migration-vs-direct mode selection is skipped entirely. - */ -export function routeInstallPathForEqlVersion( - eqlVersion: 2 | 3, - resolved: { supabase: boolean; drizzle: boolean }, -): { - drizzle: boolean - useSupabaseInstallModeSelection: boolean - notice?: string -} { - if (eqlVersion === 3) { - return { - drizzle: false, - useSupabaseInstallModeSelection: false, - notice: resolved.drizzle - ? 'EQL v3 does not support the Drizzle migration path yet — installing directly.' - : undefined, - } - } - return { - drizzle: resolved.drizzle, - useSupabaseInstallModeSelection: resolved.supabase, - } -} - -/** - * `stash eql install` is the wrong tool in a Prisma Next project: Prisma Next - * contributes a `migrations/cipherstash/` control space that installs the EQL - * bundle as part of `prisma-next migrate`, in the same ledger as the - * app schema. Running the standalone installer applies EQL out-of-band from - * that ledger. `stash init --prisma-next` already skips the installer; this - * guards the manual-invocation path too. - * - * Returns the guidance string when the install should be blocked, else null. - * `--force` overrides (an escape hatch for a deliberate standalone install). - * Pure + cwd-injected so it unit-tests without a real project. - */ -export function prismaNextInstallGuard( - cwd: string, - options: Pick<InstallOptions, 'force'>, -): string | null { - if (options.force) return null - if (!detectPrismaNext(cwd)) return null - return ( - `${messages.eql.prismaNextDetected} (found prisma-next.config.* or @cipherstash/prisma-next). ` + - 'Prisma Next installs the EQL bundle through its own migration system — run ' + - '`prisma-next migrate` instead of `stash eql install`. ' + - 'Pass --force to run the standalone installer against this database anyway.' - ) -} - -export function validateInstallFlags(options: InstallOptions): string | null { - if (options.migration && options.direct) { - return '`--migration` and `--direct` are mutually exclusive. Pick one.' - } - - if ( - options.eqlVersion !== undefined && - options.eqlVersion !== '2' && - options.eqlVersion !== '3' - ) { - return `Unknown \`--eql-version ${options.eqlVersion}\`. Supported values: 2, 3.` - } - - // `--migration` / `--direct` / `--migrations-dir` require `--supabase`. Check - // this before the version gate below so a bare `--migration` still points at - // the missing `--supabase` (its more fundamental prerequisite) rather than - // the version. - const subFlag = - options.migration === true - ? '--migration' - : options.direct === true - ? '--direct' - : options.migrationsDir !== undefined - ? '--migrations-dir' - : null - - if (subFlag !== null && options.supabase !== true) { - return `\`${subFlag}\` requires \`--supabase\`. Re-run with \`eql install --supabase ${subFlag}\`.` - } - - // v3 is the default and installs via the direct path only. The Drizzle / - // Supabase-migration / `--latest` paths are v2-only, so they require an - // explicit `--eql-version 2` — otherwise they'd silently resolve to a v3 - // direct install that ignores what the user asked for. `--migrations-dir` - // only feeds the Supabase v2 migration-file path, so it's in the same bucket. - const resolvedVersion = resolveEqlVersion(options.eqlVersion) - if (resolvedVersion === 3) { - const v2OnlyFlag = options.drizzle - ? '--drizzle' - : options.migration - ? '--migration' - : options.latest - ? '--latest' - : options.migrationsDir !== undefined - ? '--migrations-dir' - : null - if (v2OnlyFlag) { - // `--drizzle` has a v3 answer now (`stash eql migration --drizzle`, #691), - // so point there rather than at `--eql-version 2` — steering users to v2 - // is how new projects ended up on the legacy generation. - const v3Alternative = - v2OnlyFlag === '--drizzle' - ? ' For a v3 install as a Drizzle migration, use `stash eql migration --drizzle`.' - : '' - return options.eqlVersion === '3' - ? `\`--eql-version 3\` does not support \`${v2OnlyFlag}\` yet — v3 currently installs via the direct path only.${v3Alternative}` - : `\`${v2OnlyFlag}\` requires EQL v2. Re-run with \`--eql-version 2 ${v2OnlyFlag}\` (v3 is the default and installs via the direct path only).${v3Alternative}` - } - } - - return null -} - -/** - * Pick the Supabase install mode purely from inputs. No I/O, no prompts — - * easy to unit-test and to reason about. - * - * - Explicit `--migration` or `--direct` always wins. - * - Otherwise, when stdin isn't a TTY, default to `migration` if the - * `supabase/migrations/` directory exists and `direct` otherwise. This is - * the same heuristic the prompt uses for its default — keeps interactive - * and non-interactive runs aligned. - * - When stdin IS a TTY and neither flag is set, returns `null` to signal - * that the caller should prompt. - */ -export function chooseSupabaseInstallMode( - options: Pick<InstallOptions, 'migration' | 'direct'>, - projectInfo: SupabaseProjectInfo, - isTTY: boolean, -): SupabaseInstallMode | null { - if (options.migration) return 'migration' - if (options.direct) return 'direct' - if (!isTTY) return projectInfo.hasMigrationsDir ? 'migration' : 'direct' - return null -} - -/** - * Resolve the install mode, prompting the user when stdin is a TTY and - * neither sub-flag was passed. Pure logic lives in - * {@link chooseSupabaseInstallMode}; this is the I/O wrapper. - */ -async function resolveSupabaseInstallMode( - options: InstallOptions, - projectInfo: SupabaseProjectInfo, -): Promise<SupabaseInstallMode> { - const interactive = isInteractive() - const decided = chooseSupabaseInstallMode(options, projectInfo, interactive) - - if (decided !== null) { - if ( - !interactive && - options.migration === undefined && - options.direct === undefined - ) { - // Make non-interactive choices visible — surprise auto-decisions are a - // common debugging headache. - p.log.info( - projectInfo.hasMigrationsDir - ? `Detected ${projectInfo.migrationsDir} — defaulting to --migration in non-interactive mode.` - : 'No supabase/migrations directory found — defaulting to --direct in non-interactive mode.', - ) - } - return decided - } - - const defaultMode: SupabaseInstallMode = projectInfo.hasMigrationsDir - ? 'migration' - : 'direct' - - const choice = await p.select<SupabaseInstallMode>({ - message: 'How should EQL be installed?', - initialValue: defaultMode, - options: [ - { - value: 'migration', - label: 'Write a Supabase migration file', - hint: projectInfo.hasMigrationsDir - ? 'recommended — works with `supabase db reset`' - : 'creates supabase/migrations/ if missing', - }, - { - value: 'direct', - label: 'Run the SQL directly against the database', - hint: 'fastest, but `supabase db reset` will not re-install EQL', - }, - ], - }) - - if (p.isCancel(choice)) { - p.cancel('Installation cancelled.') - process.exit(0) - } - - return choice -} - -/** - * Write the `00000000000000_cipherstash_eql.sql` migration to the project's - * Supabase migrations directory. Mirrors the structure of the Drizzle - * migration helper for parity in the user-facing flow. - */ -async function writeSupabaseMigrationFile( - s: ReturnType<typeof p.spinner>, - opts: { - projectInfo: SupabaseProjectInfo - force?: boolean - dryRun?: boolean - }, -): Promise<void> { - const { projectInfo, force, dryRun } = opts - const targetPath = join( - projectInfo.migrationsDir, - SUPABASE_EQL_MIGRATION_FILENAME, - ) - - if (dryRun) { - p.log.info('Dry run — no changes will be made.') - p.note( - [ - `Would write Supabase migration to:\n ${targetPath}`, - '', - 'Apply with one of:', - ' supabase db reset # local', - ' supabase migration up # remote (or push)', - ].join('\n'), - 'Dry Run', - ) - p.outro('Dry run complete.') - return - } - - s.start('Writing CipherStash EQL migration...') - let result: { path: string; overwritten: boolean } - try { - result = await writeSupabaseEqlMigration({ - migrationsDir: projectInfo.migrationsDir, - force, - }) - } catch (error) { - s.stop('Failed to write Supabase migration.') - const message = error instanceof Error ? error.message : String(error) - p.log.error(message) - if (!force && message.includes('already exists')) { - p.log.info( - 'Re-run with --force to overwrite the existing migration file.', - ) - } - p.outro('Installation aborted.') - process.exit(1) - } - - s.stop( - result.overwritten - ? `Overwrote ${result.path}` - : `Migration created: ${result.path}`, - ) - - p.note( - [ - 'Apply the migration to install EQL:', - '', - ' supabase db reset # local — re-runs all migrations', - ' supabase migration up # remote — applies pending migrations', - '', - 'EQL is NOT installed yet. The SQL only runs when Supabase applies the migration.', - ].join('\n'), - 'Next Steps', - ) - printNextSteps() - p.outro('Done!') -} - -/** - * Find the most recently generated migration file matching the given name. - * Drizzle-kit generates flat SQL files like `0000_install-eql.sql`. - */ -export async function findGeneratedMigration( - outDir: string, - migrationName: string, -): Promise<string> { - if (!existsSync(outDir)) { - throw new Error( - `Drizzle output directory not found: ${outDir}\nMake sure drizzle-kit is configured correctly.`, - ) - } - - const entries = await readdir(outDir) - - const matchingFiles = entries - .filter((entry) => entry.endsWith('.sql') && entry.includes(migrationName)) - .sort() - - if (matchingFiles.length === 0) { - throw new Error( - `Could not find a migration matching "${migrationName}" in ${outDir}`, - ) - } - - return join(outDir, matchingFiles[matchingFiles.length - 1]) -} - -/** - * Attempt to clean up a generated migration file on failure. - */ -export function cleanupMigrationFile(filePath: string | undefined): void { - if (!filePath) return - - try { - if (existsSync(filePath)) { - unlinkSync(filePath) - p.log.info(`Cleaned up migration file: ${filePath}`) - } - } catch { - p.log.warn(`Could not clean up migration file: ${filePath}`) - } -} diff --git a/packages/cli/src/commands/db/push.ts b/packages/cli/src/commands/db/push.ts deleted file mode 100644 index 21602be01..000000000 --- a/packages/cli/src/commands/db/push.ts +++ /dev/null @@ -1,177 +0,0 @@ -import { discardPendingConfig } from '@cipherstash/migrate' -import type { CastAs, EncryptConfig } from '@cipherstash/stack/schema' -import { toEqlCastAs } from '@cipherstash/stack/schema' -import * as p from '@clack/prompts' -import pg from 'pg' -import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' -import { loadEncryptConfig, loadStashConfig } from '@/config/index.js' -import { validateEncryptConfig } from './validate.js' - -/** - * Transform an EncryptConfig so that all `cast_as` values use EQL-compatible - * types (e.g. `'number'` → `'double'`, `'string'` → `'text'`, `'json'` → `'jsonb'`). - */ -function toEqlConfig(config: EncryptConfig): Record<string, unknown> { - const tables: Record<string, Record<string, unknown>> = {} - - for (const [tableName, columns] of Object.entries(config.tables)) { - const eqlColumns: Record<string, unknown> = {} - for (const [columnName, column] of Object.entries(columns)) { - eqlColumns[columnName] = { - ...column, - cast_as: toEqlCastAs(column.cast_as as CastAs), - } - } - tables[tableName] = eqlColumns - } - - return { v: config.v, tables } -} - -export async function pushCommand(options: { - dryRun?: boolean - databaseUrl?: string -}) { - p.intro(runnerCommand(detectPackageManager(), 'stash db push')) - p.log.info( - 'This command pushes the encryption schema to the database for use with CipherStash Proxy.\nIf you are using the SDK directly (Drizzle, Supabase, or plain PostgreSQL), this step is not required.', - ) - - const s = p.spinner() - - s.start('Loading stash.config.ts...') - const config = await loadStashConfig({ databaseUrlFlag: options.databaseUrl }) - s.stop('Configuration loaded.') - - s.start(`Loading encrypt client from ${config.client}...`) - const encryptConfig = await loadEncryptConfig(config.client) - s.stop('Encrypt client loaded and validated.') - - // Run validation as a pre-push check (warn but don't block) - if (encryptConfig) { - const issues = validateEncryptConfig(encryptConfig, {}) - if (issues.length > 0) { - p.log.warn('Schema validation found issues:') - for (const issue of issues) { - const logFn = - issue.severity === 'error' - ? p.log.error - : issue.severity === 'warning' - ? p.log.warn - : p.log.info - logFn(`${issue.table}.${issue.column}: ${issue.message}`) - } - console.log() - } - } - - // Transform SDK types to EQL types for the database - const eqlConfig = toEqlConfig(encryptConfig) - - if (options.dryRun) { - p.log.info('Dry run — no changes will be pushed.') - p.note(JSON.stringify(eqlConfig, null, 2), 'Encryption Schema') - p.outro('Dry run complete.') - return - } - - const client = new pg.Client({ connectionString: config.databaseUrl }) - let exitCode = 0 - - try { - s.start('Connecting to Postgres...') - await client.connect() - s.stop('Connected to Postgres.') - - // `db push` writes `public.eql_v2_configuration` — a v2 + CipherStash Proxy - // artifact. EQL v3 has no configuration table (config lives in each - // column's `eql_v3.*` domain type) and nothing reads it, so if the table is - // absent — a v3-only database, or a database with no EQL installed at all — - // there's nothing to push. Check the table directly (one query on the - // connection we already hold) and exit cleanly instead of failing later - // with a raw `relation "public.eql_v2_configuration" does not exist`. - const tableResult = await client.query<{ reg: string | null }>( - "SELECT to_regclass('public.eql_v2_configuration') AS reg", - ) - if (tableResult.rows[0]?.reg == null) { - p.log.info( - "`stash db push` writes the EQL v2 configuration table, which this database doesn't have. It only applies to EQL v2 with CipherStash Proxy — under EQL v3 (the default) encryption config lives in each column's `eql_v3.*` type, so there's nothing to push. For a v2 + Proxy setup, run `stash eql install --eql-version 2` first.", - ) - p.outro('Nothing to do.') - return - } - - s.start('Checking public.eql_v2_configuration state...') - const activeResult = await client.query<{ exists: boolean }>( - "SELECT EXISTS(SELECT 1 FROM public.eql_v2_configuration WHERE state = 'active') AS exists", - ) - const hasActive = activeResult.rows[0]?.exists === true - s.stop( - hasActive - ? 'Active configuration found.' - : 'No active configuration yet (first push).', - ) - - if (!hasActive) { - // First push: nothing to rename, no risk of contention with a live - // Proxy reading the active config. Insert directly as `active`. - s.start('Writing initial active configuration...') - await client.query( - "INSERT INTO public.eql_v2_configuration (state, data) VALUES ('active', $1)", - [eqlConfig], - ) - s.stop('Active configuration written.') - p.outro('Push complete. Encryption is live.') - return - } - - // Active config already exists. Write the new config as `pending` so - // the EQL state machine (pending → encrypting → active) can mediate - // the change — the same flow Proxy uses for hot-reloads. The user - // promotes pending → active by running either `stash encrypt cutover` - // (when columns need renaming, e.g. `<col>_encrypted` → `<col>`) or - // `stash db activate` (when the change is purely additive). - s.start('Replacing pending configuration...') - await client.query('BEGIN') - try { - await discardPendingConfig(client) - await client.query( - "INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', $1)", - [eqlConfig], - ) - await client.query('COMMIT') - } catch (err) { - await client.query('ROLLBACK').catch(() => {}) - throw err - } - s.stop('Pending configuration written.') - - p.note( - [ - 'A pending configuration is registered but not yet active. The current', - 'active configuration continues to serve reads until you finalise it:', - '', - ' stash encrypt cutover --table T --column C', - ' Renames `<col>_encrypted` → `<col>` (and `<col>` → `<col>_plaintext`),', - ' then promotes pending → active. Use this when the new config replaces', - ' a column you migrated via `stash encrypt backfill`.', - '', - ' stash db activate', - ' Promotes pending → active without renaming. Use this when the new', - ' config purely adds columns or changes index ops on already-active', - ' columns (no `<col>_encrypted` twin to swap in).', - ].join('\n'), - 'Next step', - ) - p.outro('Push complete (pending).') - } catch (error) { - s.stop('Failed.') - p.log.error( - error instanceof Error ? error.message : 'Failed to push configuration.', - ) - exitCode = 1 - } finally { - await client.end() - } - if (exitCode) process.exit(exitCode) -} diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 3852853f7..2e0c33c8b 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -675,10 +675,11 @@ export interface RewriteResult { * plaintext is destroyed. The commented UPDATE is a placeholder that can never * become real SQL (the encrypted value is the EQL envelope produced by ZeroKMS * via the client — there is no expression Postgres can evaluate to fill it), so - * a populated table must instead use the staged `stash encrypt` lifecycle - * (add → backfill via `@cipherstash/stack`'s `encryptModel` → cutover → drop), - * which keeps both columns alive across deploys. Each rewritten file carries a - * header comment saying exactly this. + * a populated table must instead use the staged EQL v3 lifecycle (add an + * encrypted twin → dual-write → backfill via `@cipherstash/stack`'s + * `encryptModel` → switch the application to the encrypted column by name → + * drop plaintext), which keeps both columns alive across deploys. Each + * rewritten file carries a header comment saying exactly this. * * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements * left for a human — ones outside the strict matcher (a hand-authored @@ -840,9 +841,10 @@ export async function rewriteEncryptedAlterColumns( * composite, but this is equally true on both surfaces.) * * So the guidance does NOT tell the user to backfill and run this migration — - * that would still lose data on cutover. It points a populated table at the - * staged `stash encrypt` lifecycle (add → backfill → cutover → drop), which - * keeps both columns alive across deploys. + * that would still lose data. It points a populated table at the staged EQL v3 + * lifecycle (add an encrypted twin → dual-write → backfill → switch the + * application to the encrypted column by name → drop plaintext), which keeps + * both columns alive across deploys. */ function renderSafeAlter( table: string, @@ -859,8 +861,8 @@ function renderSafeAlter( `-- ${domain}. This ADD+DROP+RENAME equals DROP+ADD and is safe ONLY if`, `-- ${qualifiedTable} is empty. On a populated table it DESTROYS existing "${column}"`, '-- data (the new column starts NULL) — do NOT run it there. Use the staged', - "-- `stash encrypt` path instead: add -> backfill via @cipherstash/stack's", - '-- encryptModel in application code -> cutover -> drop.', + '-- EQL v3 path instead: add an encrypted twin -> dual-write -> backfill via', + '-- encryptModel -> switch the app to the encrypted column -> drop plaintext.', '-- NOTE: constraints, defaults, and indexes on the original column are NOT', '-- carried over by this ADD/DROP/RENAME — re-add any NOT NULL, DEFAULT,', '-- UNIQUE, or index definitions manually.', diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 20fe19864..0b4291f0c 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -99,8 +99,7 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { // install creates it and Proxy reads it. EQL v3 has no configuration table — // encryption config lives in each column's `eql_v3.*` domain type — so on a // v3-only install there's nothing to probe. Gate the check on v2 being - // installed; this also removes the old dead-end that told v3-only users to - // run `db push` (which neither creates that table nor applies to v3). + // installed; this also avoids probing a table that does not apply to v3. if (!installedV2) { p.log.info( "Encrypt config: carried in each column's `eql_v3.*` type (EQL v3 has no Proxy config table).", diff --git a/packages/cli/src/commands/db/supabase-migration.ts b/packages/cli/src/commands/db/supabase-migration.ts deleted file mode 100644 index c541a8a7e..000000000 --- a/packages/cli/src/commands/db/supabase-migration.ts +++ /dev/null @@ -1,133 +0,0 @@ -import { existsSync } from 'node:fs' -import { mkdir, writeFile } from 'node:fs/promises' -import { join } from 'node:path' -import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' -import { - loadBundledEqlSql, - SUPABASE_PERMISSIONS_SQL, -} from '@/installer/index.js' - -/** - * Filename of the Supabase migration that installs CipherStash EQL. - * - * Supabase orders migrations lexically by the `YYYYMMDDHHMMSS_` prefix; an - * all-zero prefix is guaranteed to sort before any real timestamp, so this - * file always runs first on `supabase db reset` and `supabase migration up`. - * That ordering is the whole point of this code path — without it, user - * migrations referencing `eql_v2_encrypted` blow up because the EQL types - * aren't installed yet. - */ -export const SUPABASE_EQL_MIGRATION_FILENAME = - '00000000000000_cipherstash_eql.sql' - -/** - * Header comment block prepended to the generated migration. Explains *why* - * this file exists for future maintainers reading their own migrations - * directory. The runner is resolved at call time based on the detected - * package manager. - */ -function migrationHeader(runner: string): string { - return `-- CipherStash EQL — installed by \`${runner} stash eql install --supabase --migration\`. --- --- This migration installs the CipherStash Encrypt Query Language (EQL) types, --- functions, and operators into the \`eql_v2\` schema, then grants Supabase's --- \`anon\`, \`authenticated\`, and \`service_role\` roles the access they need. --- --- The all-zero \`YYYYMMDDHHMMSS\` prefix is intentional: Supabase orders --- migrations lexically, so this file runs before any user migration that --- references the \`eql_v2_encrypted\` type. Do not rename it. --- --- To upgrade EQL, re-run the install command — it will refuse to overwrite --- this file unless you pass --force. --- --- Docs: https://cipherstash.com/docs/stack/cipherstash/supabase -` -} - -export interface WriteSupabaseEqlMigrationOptions { - /** - * Absolute path to the directory the migration file should be written into. - * Created (recursively) if it doesn't already exist. - */ - migrationsDir: string - /** - * Overwrite an existing migration file at this path. When `false` (default) - * an existing file causes the function to throw. - */ - force?: boolean - /** - * Whether to use the no-operator-family EQL bundle. Supabase always wants - * this — we expose the flag for symmetry with the runtime install path and - * to leave room for future provider variants. - */ - excludeOperatorFamily?: boolean -} - -export interface WriteSupabaseEqlMigrationResult { - /** Absolute path to the migration file that was written. */ - path: string - /** Whether an existing file at `path` was overwritten. */ - overwritten: boolean -} - -/** - * Generate the `<migrationsDir>/00000000000000_cipherstash_eql.sql` migration. - * - * The file body is, in order: - * 1. Migration header (generated from {@link migrationHeader}) — explains why the file exists. - * 2. The bundled `cipherstash-encrypt-supabase.sql` install script. - * 3. {@link SUPABASE_PERMISSIONS_SQL} — the same grants the runtime install - * path issues. One source of truth for both code paths. - * - * @throws if the target file already exists and `force` is `false`. - */ -export async function writeSupabaseEqlMigration( - options: WriteSupabaseEqlMigrationOptions, -): Promise<WriteSupabaseEqlMigrationResult> { - const { - migrationsDir, - force = false, - excludeOperatorFamily = false, - } = options - - const targetPath = join(migrationsDir, SUPABASE_EQL_MIGRATION_FILENAME) - const alreadyExists = existsSync(targetPath) - - if (alreadyExists && !force) { - throw new Error( - `Refusing to overwrite ${targetPath}: file already exists. Re-run with --force to overwrite.`, - ) - } - - // The Supabase migration-file flow is v2-only (v3 has no migration-file - // path), so pin `eqlVersion: 2` explicitly — otherwise it would follow the - // v3 default and emit `cipherstash-encrypt-v3-supabase.sql` alongside the v2 - // `SUPABASE_PERMISSIONS_SQL` below, producing a mismatched migration. We also - // pass both supabase/no-operator-family flags so intent is explicit and - // `loadBundledEqlSql` resolves the supabase file even if selection rules ever - // change. - const eqlSql = loadBundledEqlSql({ - eqlVersion: 2, - supabase: true, - excludeOperatorFamily: excludeOperatorFamily || true, - }) - - const pm = detectPackageManager() - const runner = runnerCommand(pm, '').trim() - const header = migrationHeader(runner) - - const body = [ - header, - '', - eqlSql.trimEnd(), - '', - '-- Grant access to Supabase roles (anon, authenticated, service_role).', - SUPABASE_PERMISSIONS_SQL.trimEnd(), - '', - ].join('\n') - - await mkdir(migrationsDir, { recursive: true }) - await writeFile(targetPath, body, 'utf-8') - - return { path: targetPath, overwritten: alreadyExists } -} diff --git a/packages/cli/src/commands/db/upgrade.ts b/packages/cli/src/commands/db/upgrade.ts index b62926778..bd0514aa0 100644 --- a/packages/cli/src/commands/db/upgrade.ts +++ b/packages/cli/src/commands/db/upgrade.ts @@ -1,45 +1,15 @@ import * as p from '@clack/prompts' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' -import { EQLInstaller, resolveEqlVersion } from '@/installer/index.js' +import { EQLInstaller } from '@/installer/index.js' export async function upgradeCommand(options: { dryRun?: boolean supabase?: boolean - excludeOperatorFamily?: boolean - latest?: boolean databaseUrl?: string - /** EQL generation to upgrade: `'3'` (default) or `'2'`. */ - eqlVersion?: string }) { const pm = detectPackageManager() p.intro(runnerCommand(pm, 'stash eql upgrade')) - - if ( - options.eqlVersion !== undefined && - options.eqlVersion !== '2' && - options.eqlVersion !== '3' - ) { - p.log.error( - `Unknown \`--eql-version ${options.eqlVersion}\`. Supported values: 2, 3.`, - ) - p.outro('Upgrade aborted.') - process.exit(1) - } - const eqlVersion: 2 | 3 = resolveEqlVersion(options.eqlVersion) - - if (eqlVersion === 3 && options.latest) { - // `--latest` is v2-only (no public v3 release artifacts exist yet). Since - // v3 is the default, tell the user how to reach the v2 latest upgrade. - p.log.error( - options.eqlVersion === '3' - ? '`--eql-version 3` does not support `--latest` — no public v3 release artifacts exist yet. Use the bundled upgrade.' - : '`--latest` requires EQL v2 — no public v3 release artifacts exist yet. Re-run with `--eql-version 2 --latest`, or drop `--latest` for the bundled v3 upgrade.', - ) - p.outro('Upgrade aborted.') - process.exit(1) - } - const s = p.spinner() s.start('Loading stash.config.ts...') @@ -49,73 +19,40 @@ export async function upgradeCommand(options: { }) s.stop('Configuration loaded.') - const installer = new EQLInstaller({ - databaseUrl: config.databaseUrl, - }) - - s.start('Checking current EQL installation...') - const installed = await installer.isInstalled({ eqlVersion }) - + const installer = new EQLInstaller({ databaseUrl: config.databaseUrl }) + s.start('Checking current EQL v3 installation...') + const installed = await installer.isInstalled() if (!installed) { - s.stop(`EQL v${eqlVersion} is not installed.`) - // A version mismatch is the likely cause — point at the generation that - // IS installed rather than a bare "run install". - const otherVersion: 2 | 3 = eqlVersion === 3 ? 2 : 3 - const otherInstalled = await installer - .isInstalled({ eqlVersion: otherVersion }) - .catch(() => false) - if (otherInstalled) { - p.log.warn( - `EQL v${eqlVersion} is not installed, but EQL v${otherVersion} is. Re-run with \`--eql-version ${otherVersion}\`, or install v${eqlVersion} with "${runnerCommand(pm, `stash eql install --eql-version ${eqlVersion}`)}".`, - ) - } else { - p.log.warn( - `EQL is not currently installed. Run "${runnerCommand(pm, 'stash eql install')}" first.`, - ) - } + s.stop('EQL v3 is not installed.') + p.log.warn( + `EQL v3 is not currently installed. Run "${runnerCommand(pm, 'stash eql install')}" first.`, + ) p.outro('Upgrade aborted.') process.exit(1) } - const previousVersion = await installer.getInstalledVersion({ eqlVersion }) + const previousVersion = await installer.getInstalledVersion() s.stop(`Current version: ${previousVersion ?? 'unknown'}`) - if (options.dryRun) { p.log.info('Dry run — no changes will be made.') - const source = options.latest - ? 'Would download EQL install script from GitHub (latest)' - : 'Would re-run bundled EQL install script' p.note( - `Current version: ${previousVersion ?? 'unknown'}\n${source}\nWould execute the SQL against the database`, + `Current version: ${previousVersion ?? 'unknown'}\nWould re-run the pinned EQL v3 install SQL against the database`, 'Dry Run', ) p.outro('Dry run complete.') return } - const source = options.latest ? 'from GitHub (latest)' : 'bundled' - s.start( - `Upgrading EQL ${eqlVersion === 3 ? 'v3 ' : ''}extensions (${source})...`, - ) - await installer.install({ - excludeOperatorFamily: options.excludeOperatorFamily, - supabase: options.supabase, - latest: options.latest, - eqlVersion, - }) + s.start('Upgrading EQL v3 extensions (pinned bundle)...') + await installer.install({ supabase: options.supabase }) s.stop('EQL extensions upgraded.') - - if (options.supabase) { - p.log.success('Supabase role permissions granted.') - } + if (options.supabase) p.log.success('Supabase role permissions granted.') s.start('Verifying new version...') - const newVersion = await installer.getInstalledVersion({ eqlVersion }) + const newVersion = await installer.getInstalledVersion() s.stop(`New version: ${newVersion ?? 'unknown'}`) - if (previousVersion && newVersion && previousVersion === newVersion) { p.log.info('Version unchanged — EQL was already up to date.') } - p.outro('Done!') } diff --git a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts index 9a0ad2440..71d0d4af7 100644 --- a/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/context-placeholder.test.ts @@ -78,7 +78,7 @@ describe('loadEncryptionContext — the un-replaced init scaffold', () => { /** * #787 review. The guard originally read the harvested EXPORT map, while the - * `db push` / `db validate` guard it mirrors reads `getEncryptConfig().tables`. + * The `db validate` guard it mirrors reads `getEncryptConfig().tables`. * Those disagree in both directions on the same client file, so the two * commands gave different answers for identical input. */ @@ -109,7 +109,7 @@ describe('loadEncryptionContext — the un-replaced init scaffold', () => { // The false POSITIVE, and the mirror of the case above: the user replaced // the schema set but left the sentinel `export` behind (or imports their // real tables without re-exporting them). Reading exports, the guard fired - // and told them to declare columns they had already declared. `db push` + // and told them to declare columns they had already declared. `db validate` // passes on this same file. writeProject( `export const encryptionClient = { diff --git a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts index 2b8255e79..2b447264c 100644 --- a/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts +++ b/packages/cli/src/commands/encrypt/__tests__/encrypt-v3.test.ts @@ -5,10 +5,8 @@ import type { import { beforeEach, describe, expect, it, vi } from 'vitest' import type { ResolvedLifecycle } from '../lib/resolve-eql.js' -// The EQL-v3 lifecycle branches (#648/#649): v3 has no cut-over rename — -// `encrypt cutover` must short-circuit with guidance instead of running the -// v2 config machine, and `encrypt drop` must target the ORIGINAL plaintext -// column (there is no `<col>_plaintext`) gated on `backfilled` AND a live +// The EQL-v3 drop lifecycle (#649): `encrypt drop` must target the ORIGINAL +// plaintext column (there is no `<col>_plaintext`) gated on `backfilled` AND a live // coverage check. Version + encrypted-column NAME come from the domain types // via `resolveColumnLifecycle` — the `<col>_encrypted` naming is a // convention, never relied upon — and both commands FAIL CLOSED when @@ -47,10 +45,12 @@ const migrateMocks = vi.hoisted(() => ({ ), appendEvent: vi.fn(async () => {}), setManifestTargetPhase: vi.fn(async () => {}), - renameEncryptedColumns: vi.fn(async () => {}), - migrateConfig: vi.fn(async () => {}), - activateConfig: vi.fn(async () => {}), - reloadConfig: vi.fn(async () => {}), + quoteIdent: (identifier: string) => `"${identifier.replace(/"/g, '""')}"`, + qualifyTable: (table: string) => + table + .split('.') + .map((part) => `"${part.replace(/"/g, '""')}"`) + .join('.'), })) vi.mock('@cipherstash/migrate', () => migrateMocks) @@ -108,7 +108,6 @@ vi.mock('node:fs', () => ({ })) import * as p from '@clack/prompts' -import { cutoverCommand } from '../cutover.js' import { dropCommand } from '../drop.js' const V3_INFO: ColumnInfo = { @@ -121,12 +120,6 @@ const V3_CUSTOM_INFO: ColumnInfo = { domain: 'eql_v3_text_search', version: 3, } -const V2_INFO: ColumnInfo = { - column: 'email_encrypted', - domain: 'eql_v2_encrypted', - version: 2, -} - function resolved( info: ColumnInfo, via: ResolvedInfo['via'] = 'convention', @@ -134,182 +127,12 @@ function resolved( return { info: { ...info, via }, candidates: [info] } } -/** v2 config machine present + a pending config row. */ -function mockV2ConfigQueries() { - queryMock.mockImplementation(async (sql: string) => { - if (typeof sql !== 'string') return { rows: [] } - if (sql.includes('to_regclass')) { - return { rows: [{ exists: 'eql_v2_configuration' }] } - } - if (sql.includes('eql_v2_configuration')) { - return { rows: [{ exists: true }] } - } - return { rows: [] } - }) -} - function spyExit() { return vi .spyOn(process, 'exit') .mockImplementation((() => undefined) as never) } -describe('encrypt cutover — EQL version awareness', () => { - beforeEach(() => { - vi.clearAllMocks() - queryMock.mockResolvedValue({ rows: [] }) - migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' }) - lifecycleMock.mockResolvedValue(resolved(V3_INFO)) - }) - - it('short-circuits on a backfilled v3 column: guidance, no rename, no config machine, exit 0', async () => { - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.info).toHaveBeenCalledWith( - expect.stringContaining('not applicable to EQL v3'), - ) - expect(p.log.info).toHaveBeenCalledWith( - expect.stringContaining('stash encrypt drop'), - ) - expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() - expect(migrateMocks.migrateConfig).not.toHaveBeenCalled() - expect(migrateMocks.activateConfig).not.toHaveBeenCalled() - expect(migrateMocks.appendEvent).not.toHaveBeenCalled() - expect(exitSpy).not.toHaveBeenCalled() - exitSpy.mockRestore() - }) - - it('v3 mid-backfill: does NOT tell the user to switch; exits 1', async () => { - migrateMocks.progress.mockResolvedValue({ phase: 'dual-writing' }) - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.error).toHaveBeenCalledWith( - expect.stringContaining("hasn't finished backfilling"), - ) - // The switch-now guidance must not appear before backfill completes — - // following it mid-backfill reads NULLs for unbackfilled rows. - expect(p.log.info).not.toHaveBeenCalledWith( - expect.stringContaining('point your application'), - ) - expect(exitSpy).toHaveBeenCalledWith(1) - exitSpy.mockRestore() - }) - - it('v3 already dropped: terminal phase is "nothing to do", not "finish the backfill"; exit 0', async () => { - migrateMocks.progress.mockResolvedValue({ phase: 'dropped' }) - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.info).toHaveBeenCalledWith( - expect.stringContaining('already completed'), - ) - expect(p.log.error).not.toHaveBeenCalled() - expect(exitSpy).not.toHaveBeenCalled() - exitSpy.mockRestore() - }) - - it('uses the RESOLVED encrypted column name, not the naming convention', async () => { - lifecycleMock.mockResolvedValue(resolved(V3_CUSTOM_INFO, 'hint')) - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.info).toHaveBeenCalledWith( - expect.stringContaining('email_enc'), - ) - expect(p.log.info).not.toHaveBeenCalledWith( - expect.stringContaining('email_encrypted'), - ) - }) - - // #772 review, finding 7. `drop` gated on `via === 'sole'`; `cutover` did - // not, and its v3 branch `return`s without setting exitCode — so a mixed - // table (a v2 pair the classifier no longer sees, plus one unrelated v3 - // column) produced a success-shaped message and exit 0 while the v2 rename - // never ran. A scripted rollout read that as done. - it("refuses a by-elimination ('sole') match rather than reporting success", async () => { - lifecycleMock.mockResolvedValue( - resolved( - { column: 'email_enc', domain: 'eql_v3_text_search', version: 3 }, - 'sole', - ), - ) - migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' }) - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'ssn' }) - - expect(p.log.error).toHaveBeenCalledWith( - expect.stringContaining('nothing confirms it encrypts "ssn"'), - ) - // The old message told the user to point their application at the guessed - // column, as though the lifecycle were complete. - expect(p.log.info).not.toHaveBeenCalledWith( - expect.stringContaining('point your application at email_enc'), - ) - expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() - expect(exitSpy).toHaveBeenCalledWith(1) - exitSpy.mockRestore() - }) - - it('fails closed when EQL columns exist but none is identifiable', async () => { - lifecycleMock.mockResolvedValue({ - info: null, - candidates: [ - { column: 'a_enc', domain: 'eql_v3_text_eq', version: 3 }, - { column: 'b_enc', domain: 'eql_v3_text_eq', version: 3 }, - ], - }) - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.error).toHaveBeenCalledWith( - expect.stringContaining('Cannot identify which encrypted column'), - ) - expect(p.log.error).toHaveBeenCalledWith(expect.stringContaining('a_enc')) - expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() - expect(exitSpy).toHaveBeenCalledWith(1) - exitSpy.mockRestore() - }) - - it('still runs the v2 flow for a v2 column (regression pin)', async () => { - lifecycleMock.mockResolvedValue(resolved(V2_INFO)) - mockV2ConfigQueries() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(migrateMocks.renameEncryptedColumns).toHaveBeenCalled() - expect(migrateMocks.migrateConfig).toHaveBeenCalled() - expect(migrateMocks.activateConfig).toHaveBeenCalled() - }) - - it('explains a v3-only database instead of a raw relation-does-not-exist error', async () => { - // Detection missed (e.g. no EQL columns visible) → v2 path — but the - // config table doesn't exist on this database. - lifecycleMock.mockResolvedValue({ info: null, candidates: [] }) - queryMock.mockImplementation(async (sql: string) => - typeof sql === 'string' && sql.includes('to_regclass') - ? { rows: [{ exists: null }] } - : { rows: [] }, - ) - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'email' }) - - expect(p.log.error).toHaveBeenCalledWith( - expect.stringContaining('no EQL v2 configuration table'), - ) - expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() - expect(exitSpy).toHaveBeenCalledWith(1) - exitSpy.mockRestore() - }) -}) - describe('encrypt drop — EQL version awareness', () => { beforeEach(() => { vi.clearAllMocks() @@ -358,6 +181,32 @@ describe('encrypt drop — EQL version awareness', () => { expect(sql).toMatch(/EXECUTE 'ALTER TABLE "users" DROP COLUMN "email"'/) }) + it('quotes unusual identifiers and sanitizes the migration filename', async () => { + lifecycleMock.mockResolvedValue( + resolved( + { + column: 'email"encrypted', + domain: 'eql_v3_text_search', + version: 3, + }, + 'hint', + ), + ) + migrateMocks.progress.mockResolvedValueOnce({ phase: 'backfilled' }) + + await dropCommand({ + table: 'tenant.weird"table', + column: '../email"', + }) + + const [filePath, sql] = writeFileMock.mock.calls[0] as [string, string] + expect(sql).toContain('LOCK TABLE "tenant"."weird""table"') + expect(sql).toContain('"../email""" IS NOT NULL') + expect(sql).toContain('"email""encrypted" IS NULL') + expect(filePath).toMatch(/_drop_tenant_weird_table_email\.sql$/) + expect(filePath).not.toContain('../email') + }) + it('v3: refuses to generate when rows are still plaintext-only', async () => { migrateMocks.progress.mockResolvedValueOnce({ phase: 'backfilled' }) migrateMocks.countUnencrypted.mockResolvedValueOnce(7) @@ -467,35 +316,19 @@ describe('encrypt drop — EQL version awareness', () => { exitSpy.mockRestore() }) - it('v2: unchanged — requires cut-over, no coverage gate, drops <col>_plaintext (regression pin)', async () => { - lifecycleMock.mockResolvedValue(resolved(V2_INFO)) - migrateMocks.progress.mockResolvedValueOnce({ phase: 'cut-over' }) - - await dropCommand({ table: 'users', column: 'email' }) - - const sql = writeFileMock.mock.calls[0]?.[1] as string - expect(sql).toContain('DROP COLUMN "email_plaintext"') - expect(migrateMocks.countUnencrypted).not.toHaveBeenCalled() - }) - - it('v2 post-cutover: `email` itself carrying the v2 domain is NOT ambiguity — proceeds down the v2 path', async () => { - // After cutover renamed the ciphertext onto `email`, no counterpart is - // resolvable BY DESIGN. The fail-closed guard must recognize this state - // rather than blocking the one drop the lifecycle actually wants. - // - // `candidates` is EMPTY, not `[{ column: 'email', version: 2 }]`: the - // classifier no longer recognises `eql_v2_encrypted`, so a post-cutover v2 - // column drops out of `listEncryptedColumns` entirely. The state reaches - // `explainUnresolved` as "no EQL columns at all", which is exactly the - // case it already falls through on. This pins that the v2 lifecycle still - // works through the narrower classifier. + it('rejects an unclassified legacy column instead of taking a v2 drop path', async () => { lifecycleMock.mockResolvedValue({ info: null, candidates: [] }) - migrateMocks.progress.mockResolvedValueOnce({ phase: 'cut-over' }) + const exitSpy = spyExit() await dropCommand({ table: 'users', column: 'email' }) - const sql = writeFileMock.mock.calls[0]?.[1] as string - expect(sql).toContain('DROP COLUMN "email_plaintext"') - expect(migrateMocks.countUnencrypted).not.toHaveBeenCalled() + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining( + 'Legacy EQL v2 drop/cut-over automation has been removed', + ), + ) + expect(writeFileMock).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() }) }) diff --git a/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts b/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts deleted file mode 100644 index 15ea2b01d..000000000 --- a/packages/cli/src/commands/encrypt/__tests__/v2-lifecycle-composition.integration.test.ts +++ /dev/null @@ -1,202 +0,0 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' - -/** - * The seam between lifecycle RESOLUTION and the commands that act on it. - * - * `encrypt-v3.test.ts` stubs `resolveColumnLifecycle` and hand-writes the value - * it returns; `resolve-eql.test.ts` separately proves a pure-v2 table produces - * that value. Both can stay green while the shape passed between them changes, - * because neither runs the real producer into the real consumer — and the - * commands' v2/v3 branch is chosen entirely from that shape. - * - * So here `resolve-eql.js` is NOT mocked at all: `resolveColumnLifecycle`, - * `explainUnresolved`, `pickEncryptedColumn`, `listEncryptedColumns` and - * `columnExists` all run for real, and the pure-v2 verdict is derived from a - * real manifest hint plus a real catalog read rather than asserted into - * existence. Only genuine boundaries are faked: `pg`, the filesystem, prompts, - * and the effectful migrate helpers that write to the database (#787 review - * follow-up). - */ - -const queryMock = vi.hoisted(() => - vi.fn(async (_sql: string, _params?: unknown[]) => ({ - rows: [] as unknown[], - })), -) -vi.mock('pg', () => ({ - default: { - Client: class { - connect = vi.fn(async () => {}) - end = vi.fn(async () => {}) - query = queryMock - }, - }, -})) - -/** - * Spread the real module and override only the functions that TOUCH something — - * the manifest file and the database. Everything resolution actually reasons - * with (`listEncryptedColumns`, `columnExists`, `pickEncryptedColumn`, - * `classifyEqlDomain`) stays real and runs against `queryMock` below. - */ -const migrateMocks = vi.hoisted(() => ({ - readManifest: vi.fn(async () => null as unknown), - countUnencrypted: vi.fn(async () => 0), - progress: vi.fn( - async () => ({ phase: 'cut-over' }) as { phase: string } | null, - ), - appendEvent: vi.fn(async () => {}), - setManifestTargetPhase: vi.fn(async () => {}), - renameEncryptedColumns: vi.fn(async () => {}), - migrateConfig: vi.fn(async () => {}), - activateConfig: vi.fn(async () => {}), - reloadConfig: vi.fn(async () => {}), -})) -vi.mock('@cipherstash/migrate', async (importOriginal) => ({ - ...(await importOriginal<typeof import('@cipherstash/migrate')>()), - ...migrateMocks, -})) - -vi.mock('@clack/prompts', () => ({ - intro: vi.fn(), - outro: vi.fn(), - confirm: vi.fn(async () => true), - isCancel: vi.fn(() => false), - log: { - info: vi.fn(), - success: vi.fn(), - warn: vi.fn(), - error: vi.fn(), - step: vi.fn(), - }, - note: vi.fn(), -})) -vi.mock('@/config/index.js', () => ({ - loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), -})) -vi.mock('@/commands/db/detect.js', () => ({ - detectDrizzle: vi.fn(() => false), -})) -vi.mock('@/commands/init/utils.js', () => ({ - detectPackageManager: vi.fn(() => 'npm'), - runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), -})) -vi.mock('../drizzle-helper.js', () => ({ - scaffoldDrizzleMigration: vi.fn(async ({ name }: { name: string }) => ({ - path: `drizzle/${name}.sql`, - })), -})) -const writeFileMock = vi.hoisted(() => vi.fn()) -vi.mock('node:fs', () => ({ - default: { mkdirSync: vi.fn(), writeFileSync: writeFileMock }, -})) - -import * as p from '@clack/prompts' -import { cutoverCommand } from '../cutover.js' -import { dropCommand } from '../drop.js' - -function spyExit() { - return vi - .spyOn(process, 'exit') - .mockImplementation((() => undefined) as never) -} - -/** Columns the faked catalog reports as EQL-domain typed. Empty = pure v2. */ -let eqlColumns: { column: string; domain_name: string }[] = [] -/** Columns the faked catalog reports as existing at all. */ -let existingColumns: string[] = [] - -/** - * Route on the catalog statements these paths issue. Each is matched on text - * unique to it, and ORDER MATTERS: three of them end in `AS exists`, and - * `columnExists` and `listEncryptedColumns` both embed the same shared - * `to_regclass` expression. Matching loosely silently answers the wrong probe — - * routing the v2 pending-config check into `columnExists` made cutover report - * "no pending EQL configuration" instead of running the ladder. - */ -function fakeCatalog() { - queryMock.mockImplementation(async (sql: string, params?: unknown[]) => { - if (typeof sql !== 'string') return { rows: [] } - // The v2 config machine exists… - if (sql.includes("to_regclass('public.eql_v2_configuration')")) { - return { rows: [{ exists: 'eql_v2_configuration' }] } - } - // …and holds a pending row to promote. - if (sql.includes("state = 'pending'")) return { rows: [{ exists: true }] } - if (sql.includes('typname AS domain_name')) return { rows: eqlColumns } - if (sql.includes('pg_attribute') && sql.includes('AS exists')) { - return { - rows: [{ exists: existingColumns.includes(String(params?.[2])) }], - } - } - return { rows: [] } - }) -} - -beforeEach(() => { - vi.clearAllMocks() - // The hint `encrypt backfill` records for EVERY column, v2 included. It is - // what used to be discarded into a guess, and what must now be ignored - // harmlessly on a table with no v3 columns to mis-claim. - migrateMocks.readManifest.mockResolvedValue({ - tables: { - users: [{ column: 'ssn', encryptedColumn: 'ssn_encrypted' }], - }, - }) - migrateMocks.progress.mockResolvedValue({ phase: 'cut-over' }) - eqlColumns = [] - existingColumns = ['ssn', 'ssn_encrypted'] - fakeCatalog() -}) - -describe('the pure-v2 lifecycle, resolved for real end to end', () => { - it('generates the v2 plaintext drop for a table whose encrypted column is legacy v2', async () => { - const exitSpy = spyExit() - - await dropCommand({ table: 'users', column: 'ssn' }) - - expect(exitSpy).not.toHaveBeenCalled() - - // Reached the v2 ladder: `<col>_plaintext`, not the v3 target `<col>`. - const sql = writeFileMock.mock.calls[0]?.[1] as string - expect(sql).toContain('DROP COLUMN "ssn_plaintext"') - // The v3-only coverage gate must not run on a v2 column. - expect(migrateMocks.countUnencrypted).not.toHaveBeenCalled() - }) - - it('runs the v2 rename and config promotion for a table with no EQL v3 columns', async () => { - migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' }) - - const exitSpy = spyExit() - - await cutoverCommand({ table: 'users', column: 'ssn' }) - - expect(exitSpy).not.toHaveBeenCalled() - expect(migrateMocks.renameEncryptedColumns).toHaveBeenCalled() - expect(migrateMocks.migrateConfig).toHaveBeenCalled() - expect(migrateMocks.activateConfig).toHaveBeenCalled() - expect(p.log.error).not.toHaveBeenCalled() - }) - - it('refuses both commands when the table also holds an unidentifiable EQL v3 column', async () => { - // The mixed table from #772 finding 7. Same manifest hint, same v2 pair — - // the ONLY difference is an unrelated v3 column, which is what makes the - // recorded pairing meaningful and a fall-through a guess. Crossing this - // boundary is what neither existing half of the coverage can see. - eqlColumns = [{ column: 'email_enc', domain_name: 'eql_v3_text_search' }] - const exitSpy = spyExit() - - await dropCommand({ table: 'users', column: 'ssn' }) - await cutoverCommand({ table: 'users', column: 'ssn' }) - - // Refusing is the point: both must exit non-zero rather than act on - // `email_enc`, the unrelated v3 column the sole-EQL-column rule would - // otherwise claim. - expect(exitSpy).toHaveBeenCalledWith(1) - expect(writeFileMock).not.toHaveBeenCalled() - expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled() - const errors = vi.mocked(p.log.error).mock.calls.flat().join('\n') - expect(errors).toContain('is not an EQL v3 column') - expect(errors).toContain('ssn_encrypted') - }) -}) diff --git a/packages/cli/src/commands/encrypt/backfill.ts b/packages/cli/src/commands/encrypt/backfill.ts index 1698d8825..4da4b63eb 100644 --- a/packages/cli/src/commands/encrypt/backfill.ts +++ b/packages/cli/src/commands/encrypt/backfill.ts @@ -1,7 +1,6 @@ import { appendEvent, detectColumnEqlVersion, - type EqlVersion, type ManifestColumn, progress, runBackfill, @@ -138,24 +137,22 @@ export async function backfillCommand(options: BackfillCommandOptions) { const encryptedColumn = options.encryptedColumn ?? `${options.column}_encrypted` - // v2 or v3 changes the rest of the LIFECYCLE (v3 has no cut-over — the - // ladder is backfill → switch-by-name → drop), so detect it up front, - // record it in the manifest, and tell the user which path they're on. - // `null` means the target column doesn't exist, isn't an EQL domain, or is - // a legacy `eql_v2_encrypted` column (no longer classified — v3 is the sole - // authored generation). Every one of those falls through to the v2 ladder, - // which is the correct default for the v2 case and lets the existing checks - // below produce their specific errors for the other two. + // Backfill authors ciphertext, so it accepts only the current EQL v3 + // domains. Legacy v2 remains visible to status/read diagnostics but is no + // longer a writable rollout target. const eqlVersion = await detectColumnEqlVersion( db, options.table, encryptedColumn, ) - if (eqlVersion) { - p.log.info( - `${options.table}.${encryptedColumn} is EQL v${eqlVersion}${eqlVersion === 3 ? ' — lifecycle is backfill → switch the app to the encrypted column by name → drop (no cut-over rename).' : ''}`, + if (eqlVersion !== 3) { + throw new BackfillConfigError( + `${options.table}.${encryptedColumn} is not an EQL v3 domain. stash no longer backfills legacy EQL v2 columns; migrate the schema to an eql_v3_* domain first.`, ) } + p.log.info( + `${options.table}.${encryptedColumn} is EQL v3 — lifecycle is backfill → switch the app to the encrypted column by name → drop (no cut-over rename).`, + ) // Phase guard: backfill requires the application to already be writing // to both columns, otherwise rows inserted *during* the backfill land @@ -277,12 +274,10 @@ export async function backfillCommand(options: BackfillCommandOptions) { return } - if (eqlVersion === 3) { - p.note( - `EQL v3 has no cut-over. Next:\n 1. Point your application at ${encryptedColumn} (schema + queries), deploy, verify reads.\n 2. Generate the plaintext drop: stash encrypt drop --table ${options.table} --column ${plaintextColumn}`, - 'Next steps (EQL v3)', - ) - } + p.note( + `EQL v3 has no cut-over. Next:\n 1. Point your application at ${encryptedColumn} (schema + queries), deploy, verify reads.\n 2. Generate the plaintext drop: stash encrypt drop --table ${options.table} --column ${plaintextColumn}`, + 'Next steps (EQL v3)', + ) p.outro( `Backfill complete. ${result.rowsProcessed.toLocaleString()} rows encrypted.`, ) @@ -529,11 +524,11 @@ function buildManifestEntry( plaintextColumn: string, encryptedColumn: string, pkColumn: string | undefined, - eqlVersion: EqlVersion | null, + eqlVersion: 3, ): ManifestColumn { // SDK `cast_as` ('string', 'number', …) and EQL `castAs` ('text', // 'double', …) are different vocabularies; translate via the same - // helper `stash db push` uses so the two stay aligned. + // shared schema translation helper so SDK and EQL vocabularies stay aligned. const castAs: ManifestColumn['castAs'] = column?.cast_as !== undefined ? translateCastAs( @@ -551,25 +546,14 @@ function buildManifestEntry( column: plaintextColumn, castAs, indexes, - // Recorded so later commands (cutover/drop/status) don't have to guess + // Recorded so later commands (drop/status) don't have to guess // the name from the `<column>_encrypted` convention — the name is a // convention only, never relied upon. encryptedColumn, - // v2's ladder ends with the rename cut-over; v3 has none — its end - // state is the plaintext column dropped. An unclassified column (null, - // which now includes a legacy v2 domain) takes the v2 ladder. - targetPhase: eqlVersion === 3 ? 'dropped' : 'cut-over', + targetPhase: 'dropped', + eqlVersion, } if (pkColumn) entry.pkColumn = pkColumn - // Absent means the classifier returned null: the column isn't there, isn't - // an EQL domain, or carries the legacy `eql_v2_encrypted` domain (which - // `classifyEqlDomain` no longer recognises — v3 is the sole authored - // generation). Readers fall back to the live domain type, which yields null - // for those same three cases, so `encrypt status` reports no version rather - // than guessing one. Manifests written before v2 classification was dropped - // still carry `eqlVersion: 2`, so existing v2 columns keep their version; - // only a v2 column backfilled from here on records none. - if (eqlVersion) entry.eqlVersion = eqlVersion return entry } diff --git a/packages/cli/src/commands/encrypt/context.ts b/packages/cli/src/commands/encrypt/context.ts index 682328eed..264bc2099 100644 --- a/packages/cli/src/commands/encrypt/context.ts +++ b/packages/cli/src/commands/encrypt/context.ts @@ -142,7 +142,7 @@ export async function loadEncryptionContext(): Promise<EncryptionContext> { process.exit(1) } - // The same refusal `stash db push` / `db validate` get from + // The same refusal `stash db validate` gets from // `loadEncryptConfig`, applied here because `stash encrypt` does not go // through that loader. Called, not re-implemented: it guards one file, so the // two commands must say one thing about it. Without this, `requireTable` diff --git a/packages/cli/src/commands/encrypt/cutover.ts b/packages/cli/src/commands/encrypt/cutover.ts deleted file mode 100644 index 1080e9524..000000000 --- a/packages/cli/src/commands/encrypt/cutover.ts +++ /dev/null @@ -1,331 +0,0 @@ -import { - activateConfig, - appendEvent, - migrateConfig, - progress, - reloadConfig, - renameEncryptedColumns, -} from '@cipherstash/migrate' -import * as p from '@clack/prompts' -import pg from 'pg' -import { detectDrizzle } from '@/commands/db/detect.js' -import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' -import { loadStashConfig } from '@/config/index.js' -import { scaffoldDrizzleMigration } from './drizzle-helper.js' -import { explainUnresolved, resolveColumnLifecycle } from './lib/resolve-eql.js' - -/** - * Options accepted by `stash encrypt cutover`. Swaps the plaintext and - * encrypted columns via `eql_v2.rename_encrypted_columns()` so that apps - * reading `<column>` transparently receive the encrypted column - * (decrypted on read by Proxy or client-side by Stack). - */ -export interface CutoverCommandOptions { - /** Physical table name, e.g. `users`. Supports `schema.table`. */ - table: string - /** - * Physical plaintext column that is being cut over, e.g. `email`. Used - * only for the state-transition check and event log; the actual rename - * affects every column in the active EQL config in a single call. - */ - column: string - /** - * Optional Postgres URL of a CipherStash Proxy. When set, the command - * connects to the Proxy after the rename and runs `eql_v2.reload_config()` - * so Proxy picks up the renamed columns immediately rather than waiting - * for its 60-second refresh. When unset, prints a warning to that effect - * and returns — the Proxy will refresh on its own. - * - * Also readable from `CIPHERSTASH_PROXY_URL` in the environment. - */ - proxyUrl?: string - /** - * Drizzle migrations directory (passed to `drizzle-kit generate - * --custom`). Defaults to `./drizzle`. Only used when the project is - * Drizzle — non-Drizzle projects skip the snapshot-resync step. - */ - migrationsDir?: string -} - -/** - * CLI handler for `stash encrypt cutover`. Verifies the target column is - * in phase `backfilled`, runs `eql_v2.rename_encrypted_columns()` inside - * a transaction, appends a `cut_over` event, and optionally triggers a - * Proxy config reload. Exits with code `1` if preconditions are not met. - */ -export async function cutoverCommand(options: CutoverCommandOptions) { - p.intro(runnerCommand(detectPackageManager(), 'stash encrypt cutover')) - - const config = await loadStashConfig() - const client = new pg.Client({ connectionString: config.databaseUrl }) - let exitCode = 0 - - try { - await client.connect() - - // Cut-over is an EQL v2 concept: v2 hides the swap behind - // `eql_v2.rename_encrypted_columns()` + a Proxy config promotion. A v3 - // column has neither — the application switches to the encrypted column - // BY NAME, and the plaintext column is dropped later. Resolve from the - // DOMAIN TYPES (manifest name as a hint; the `<col>_encrypted` naming is - // a convention, never relied upon) before any phase/config checks so v3 - // users get the real answer, not a confusing precondition error. - const { info, candidates, unresolvedHint } = await resolveColumnLifecycle( - client, - options.table, - options.column, - ) - // Fail closed on ambiguity: `info === null` with EQL columns present - // means we can't tell WHICH lifecycle applies — running the v2 config - // machine against (possibly) v3 columns would only produce a misleading - // downstream error. (A table with no EQL v3 columns still falls through to - // the v2 preconditions below — which now also covers the post-cutover v2 - // same-name state, since an `eql_v2_encrypted` column is no longer - // classified and so never appears as a candidate.) - const unresolved = explainUnresolved( - options.table, - options.column, - candidates, - unresolvedHint, - ) - if (!info && unresolved) { - p.log.error(unresolved) - exitCode = 1 - return - } - const state = await progress(client, options.table, options.column) - - // `via: 'sole'` means only that this is the table's one remaining EQL - // candidate once the plaintext column itself is excluded — nothing ties it - // to the plaintext column the user named. On a mixed table (a v2 pair the - // classifier no longer sees, plus one unrelated v3 column) that guess is - // simply wrong, and reporting "nothing to do for EQL v3" for it told a - // scripted rollout the cut-over had succeeded when the v2 rename never ran - // (#772 review, finding 7). - // - // Deliberately at TOP LEVEL, not inside the `version === 3` branch, so it - // mirrors `drop.ts` exactly. Equivalent today — `classifyEqlDomain` - // recognises `eql_v3_*` only, so a non-null `info` is always version 3 — - // but the v2 ladder below performs an irreversible rename plus config - // promotion. Were v2 classification restored, or a v4 family added, a - // nested guard would let `cutover` rename on a guess that `drop` refuses - // (#787 review). - if (info?.via === 'sole') { - p.log.error( - `${options.table}.${info.column} (${info.domain}) is the only EQL column left on ${options.table} once "${options.column}" itself is excluded, but nothing confirms it encrypts "${options.column}" — refusing to report a cut-over outcome on that guess. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only): complete that column's v2 lifecycle yourself with the eql_v2 SQL — \`SELECT eql_v2.rename_encrypted_columns();\` plus the config promotion — since no stash command can drive it here. Otherwise record the pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column <the column that actually encrypts ${options.column}>\`.`, - ) - exitCode = 1 - return - } - - if (info?.version === 3) { - const encryptedColumn = info.column - - if (state?.phase === 'dropped') { - // Terminal phase — the lifecycle already finished. Not an error and - // not "finish the backfill": there is nothing left to backfill. - p.log.info( - `${options.table}.${options.column} has already completed the EQL v3 lifecycle (plaintext dropped). Nothing to cut over.`, - ) - p.outro('Nothing to do for EQL v3.') - return - } - if (state?.phase !== 'backfilled') { - // Not a "nothing to do" — the user isn't ready for ANY next step - // yet. Exit 1 so scripted pipelines gating on cutover don't read - // an incomplete backfill as success. - p.log.error( - `Cut-over is not applicable to EQL v3 columns, and ${options.table}.${options.column} hasn't finished backfilling (phase '${state?.phase ?? '—'}'). Finish the backfill first:\n stash encrypt backfill --table ${options.table} --column ${options.column}`, - ) - exitCode = 1 - return - } - p.log.info( - `Cut-over is not applicable to EQL v3 columns. ${options.table}.${encryptedColumn} is EQL v3 (${info.domain}): there is no rename step — point your application at ${encryptedColumn} (update your schema/queries), verify reads, then generate the plaintext drop with:\n stash encrypt drop --table ${options.table} --column ${options.column}`, - ) - p.outro('Nothing to do for EQL v3.') - return - } - - if (state?.phase !== 'backfilled') { - p.log.error( - `Cannot cut over: ${options.table}.${options.column} is in phase '${state?.phase ?? '—'}'. Must be 'backfilled'.`, - ) - exitCode = 1 - return - } - - // Guard the v2 config machinery's existence before querying it: on a - // v3-only database (v3 is the default install) there is no - // `eql_v2_configuration` relation, and an unguarded query surfaces as a - // raw "relation does not exist" error instead of an explanation. - const configTable = await client.query<{ exists: string | null }>( - "SELECT to_regclass('public.eql_v2_configuration')::text AS exists", - ) - if (configTable.rows[0]?.exists == null) { - p.log.error( - `This database has no EQL v2 configuration table — it looks like an EQL v3-only install. Cut-over only applies to EQL v2 columns; for v3, point your application at the encrypted column by name, then generate the plaintext drop with:\n stash encrypt drop --table ${options.table} --column ${options.column}`, - ) - exitCode = 1 - return - } - - // Verify a pending EQL config exists. cutover assumes the user has - // already run `stash db push` against a schema that switches the - // column from `<col>_encrypted` (or whatever twin name) to `<col>` — - // db push writes that as pending, and cutover transitions - // pending → encrypting → active alongside the physical rename. - const pending = await client.query<{ exists: boolean }>( - "SELECT EXISTS(SELECT 1 FROM public.eql_v2_configuration WHERE state = 'pending') AS exists", - ) - if (pending.rows[0]?.exists !== true) { - p.log.error( - 'No pending EQL configuration to cut over. Cutover operates on the EQL v2 + CipherStash Proxy config lifecycle — update your schema to point at the encrypted column (drop the `_encrypted` suffix) and register the pending change before cutting over.', - ) - exitCode = 1 - return - } - - // Full lifecycle in one transaction: - // 1. rename_encrypted_columns — physical column rename - // 2. migrate_config — pending → encrypting - // 3. activate_config — encrypting → active (and prior active → inactive) - // Each step is a side-effect-free function from the user's POV - // (everything happens inside the txn). Rollback on any error leaves - // the system in its pre-cutover state. - await client.query('BEGIN') - try { - await renameEncryptedColumns(client) - await migrateConfig(client) - await activateConfig(client) - await appendEvent(client, { - tableName: options.table, - columnName: options.column, - event: 'cut_over', - phase: 'cut-over', - details: { renamed: true }, - }) - await client.query('COMMIT') - } catch (err) { - await client.query('ROLLBACK').catch(() => {}) - throw err - } - - p.log.success( - `Renamed ${options.column} → ${options.column}_plaintext and ${options.column}_encrypted → ${options.column}; pending config promoted to active.`, - ) - - // Drizzle snapshot resync. The rename above ran outside drizzle-kit's - // authority — the snapshot at `<out>/meta/<idx>_snapshot.json` still - // describes the pre-rename column shape. If we don't acknowledge the - // change in Drizzle's metadata, the next `drizzle-kit generate` will - // produce a confused diff trying to re-create the old layout. - // - // Scaffolding a custom migration with idempotent rename SQL solves - // both problems: it adds the journal entry + snapshot diff that - // Drizzle expects, and the SQL itself is a no-op on the source DB - // (the pre-rename column doesn't exist any more) but applies - // correctly when migrating a fresh database. - if (detectDrizzle(process.cwd())) { - try { - const renameSql = buildRenameMigrationSql(options.table, options.column) - const result = await scaffoldDrizzleMigration({ - name: `cutover_${options.table}_${options.column}`, - outDir: options.migrationsDir ?? 'drizzle', - sql: renameSql, - }) - p.log.success( - `Drizzle snapshot updated: ${result.path} (idempotent — no-op on this DB, applies on a fresh restore).`, - ) - } catch (err) { - const reason = err instanceof Error ? err.message : String(err) - p.log.warn( - `Could not scaffold the Drizzle rename migration: ${reason}\nDrizzle's snapshot may be out of sync with the live schema. Run \`drizzle-kit pull\` to resync, or scaffold the rename migration manually.`, - ) - } - } - - // Proxy reload runs *after* the cutover transaction has committed. - // Any error from here on is post-commit cosmetic — the rename and - // config promotion are durable. Catch the proxy reload separately so - // a transient Proxy connectivity blip doesn't make the outer catch - // exit(1), which would falsely tell automation the cutover failed - // and encourage unsafe retries. - const proxyUrl = options.proxyUrl ?? process.env.CIPHERSTASH_PROXY_URL - if (proxyUrl) { - const proxy = new pg.Client({ connectionString: proxyUrl }) - try { - await proxy.connect() - try { - await reloadConfig(proxy) - p.log.success('Proxy config reloaded.') - } catch (err) { - const reason = err instanceof Error ? err.message : String(err) - p.log.warn( - `Proxy config reload failed (${reason}). The cutover itself is committed and durable; Proxy will pick up the new config on its next 60s refresh.`, - ) - } - } finally { - await proxy.end() - } - } else { - p.log.warn( - 'CIPHERSTASH_PROXY_URL not set; Proxy users must wait up to 60s for config refresh.', - ) - } - - p.outro( - 'Cut-over complete. Your app reads the encrypted column transparently.', - ) - } catch (error) { - p.log.error(error instanceof Error ? error.message : 'Cut-over failed.') - exitCode = 1 - } finally { - await client.end() - // In `finally` (not after the try/catch) deliberately: the precondition - // guards above `return` from inside `try`, which skips any code placed - // after the block — so a trailing `if (exitCode) process.exit(...)` - // was unreachable on exactly the failure paths it existed for, and - // guard failures exited 0. - if (exitCode) process.exit(exitCode) - } -} - -/** - * Build the SQL body for the post-cutover Drizzle migration. Wrapped in a - * `DO` block that checks whether `<col>_encrypted` still exists — on the - * source database the rename already ran (so the column is gone and the - * block does nothing), but on a fresh restore the rename hasn't run yet - * (so the block performs the swap). Same migration file, both behaviours, - * idempotent. - * - * Splits `schema.table` into separate identifiers so the generated SQL is - * correct regardless of whether the user passed `users` or `public.users`. - */ -function buildRenameMigrationSql(table: string, column: string): string { - const dot = table.indexOf('.') - const [schema, tableName] = - dot >= 0 ? [table.slice(0, dot), table.slice(dot + 1)] : [null, table] - - const qualified = schema ? `"${schema}"."${tableName}"` : `"${tableName}"` - const schemaPredicate = schema - ? `table_schema = '${schema.replace(/'/g, "''")}' AND ` - : '' - - return `-- Generated by stash encrypt cutover. --- Records the rename that eql_v2.rename_encrypted_columns() performed --- so Drizzle's snapshot stays in sync. Idempotent: a no-op on the DB --- where cutover already ran; applies on a fresh restore. -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE ${schemaPredicate}table_name = '${tableName.replace(/'/g, "''")}' - AND column_name = '${column}_encrypted' - ) THEN - ALTER TABLE ${qualified} RENAME COLUMN "${column}" TO "${column}_plaintext"; - ALTER TABLE ${qualified} RENAME COLUMN "${column}_encrypted" TO "${column}"; - END IF; -END $$; -` -} diff --git a/packages/cli/src/commands/encrypt/drizzle-helper.ts b/packages/cli/src/commands/encrypt/drizzle-helper.ts index fb775cafa..f820d6c46 100644 --- a/packages/cli/src/commands/encrypt/drizzle-helper.ts +++ b/packages/cli/src/commands/encrypt/drizzle-helper.ts @@ -10,10 +10,8 @@ import { detectPackageManager, runnerArgv } from '@/commands/init/utils.js' * does — `drizzle-kit generate --custom` creates the file and records the * journal entry / snapshot, then we overwrite the empty body with our SQL. * - * Used by `encrypt drop` (to ship the plaintext-column drop migration in a - * shape `drizzle-kit migrate` actually picks up) and `encrypt cutover` (to - * record the live rename so Drizzle's snapshot reflects post-cutover - * reality). + * Used by `encrypt drop` to ship the plaintext-column drop migration in a + * shape `drizzle-kit migrate` actually picks up. * * Throws if `drizzle-kit` isn't on PATH or the generated file can't be * located afterwards. Callers should fall back to the self-named-file diff --git a/packages/cli/src/commands/encrypt/drop.ts b/packages/cli/src/commands/encrypt/drop.ts index d016a0c63..ec7871a35 100644 --- a/packages/cli/src/commands/encrypt/drop.ts +++ b/packages/cli/src/commands/encrypt/drop.ts @@ -4,6 +4,8 @@ import { appendEvent, countUnencrypted, progress, + qualifyTable, + quoteIdent, setManifestTargetPhase, } from '@cipherstash/migrate' import * as p from '@clack/prompts' @@ -16,9 +18,7 @@ import { explainUnresolved, resolveColumnLifecycle } from './lib/resolve-eql.js' /** * Options accepted by `stash encrypt drop`. Generates a migration file that - * drops the now-unused plaintext column — for EQL v2 that is - * `<col>_plaintext` (the name cutover's rename left it with); for EQL v3 - * (no rename) it is the original `<col>` itself. Does *not* apply the + * drops the now-unused original plaintext column for EQL v3. Does *not* apply the * migration — the user runs their usual migration tool (drizzle-kit, * prisma, psql) to actually execute it. */ @@ -27,9 +27,7 @@ export interface DropCommandOptions { table: string /** * Physical column — the original plaintext name. What gets dropped - * depends on the column's EQL version: v2 drops `<column>_plaintext` - * (post-cutover leftover); v3 drops `<column>` itself, gated on a live - * ciphertext-coverage check. + * is `<column>` itself, gated on a live ciphertext-coverage check. */ column: string /** @@ -41,9 +39,8 @@ export interface DropCommandOptions { } /** - * CLI handler for `stash encrypt drop`. Version-aware preconditions: - * EQL v2 requires phase `cut-over`; EQL v3 (which has no cut-over) requires - * `backfilled` plus a live coverage check — and the generated v3 migration + * CLI handler for `stash encrypt drop`. EQL v3 requires `backfilled` plus a + * live coverage check — and the generated migration * re-verifies coverage at APPLY time, since rows can be written between * generation and application. * @@ -67,13 +64,8 @@ export async function dropCommand(options: DropCommandOptions) { try { await client.connect() - // The plaintext column's name depends on the EQL version's lifecycle: - // v2's cut-over renames `<col>` → `<col>_plaintext` (so that's what we - // drop, after `cut-over`); v3 has no rename — the app switched to the - // encrypted column by name, so the original `<col>` IS the plaintext - // column, droppable straight after `backfilled`. The version and the - // encrypted column's name are resolved from the DOMAIN TYPES (manifest - // name as a hint) — the `<col>_encrypted` naming is a convention only. + // EQL v3 has no rename: the app switches to the encrypted column by name, + // so the original `<col>` remains the plaintext column to drop. const { info, candidates, unresolvedHint } = await resolveColumnLifecycle( client, options.table, @@ -82,11 +74,7 @@ export async function dropCommand(options: DropCommandOptions) { // Fail closed on ambiguity: with EQL columns present but no identifiable // counterpart, guessing a lifecycle here could validate coverage against // the wrong ciphertext and generate an irreversible drop of the wrong - // data. (The post-cutover v2 state — `<col>` itself carries the v2 - // domain, counterpart legitimately unresolvable — falls through to the - // v2 path: the classifier recognises `eql_v3_*` only, so that column is - // not a candidate and the table reads as having no EQL columns. Live - // truth from the DB wins over the manifest's cached version throughout.) + // data. const unresolved = explainUnresolved( options.table, options.column, @@ -98,6 +86,13 @@ export async function dropCommand(options: DropCommandOptions) { exitCode = 1 return } + if (!info) { + p.log.error( + `Cannot identify an EQL v3 encrypted column for ${options.table}.${options.column}. Legacy EQL v2 drop/cut-over automation has been removed; migrate the schema to EQL v3 before continuing.`, + ) + exitCode = 1 + return + } // A `via: 'sole'` match only proves the column is the table's ONE EQL // column — it may encrypt a DIFFERENT field, in which case the coverage // gate below would count the wrong ciphertext and wave through a drop @@ -113,62 +108,55 @@ export async function dropCommand(options: DropCommandOptions) { // live `DROP COLUMN` on the plaintext at exit 0 (#772 review, finding 7). if (info?.via === 'sole') { p.log.error( - `${options.table}.${info.column} (${info.domain}) is the only EQL column left on ${options.table} once "${options.column}" itself is excluded, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column <name>\` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. If "${options.column}" pairs with a legacy eql_v2_encrypted column, resolution cannot see it (this command resolves EQL v3 counterparts only): complete that column's v2 lifecycle yourself with the eql_v2 SQL, since no stash command can drive it here — and do not record ${info.column}.`, + `${options.table}.${info.column} (${info.domain}) is the only EQL column left on ${options.table} once "${options.column}" itself is excluded, but nothing confirms it encrypts "${options.column}" — refusing to generate an irreversible drop on that guess. Identify the EQL v3 column that actually encrypts "${options.column}" and record that pairing: re-run \`stash encrypt backfill --table ${options.table} --column ${options.column} --encrypted-column <name>\` (which writes it to the manifest), or set "encryptedColumn" for this column in .cipherstash/migrations.json. Legacy EQL v2 drop/cut-over automation has been removed — do not record ${info.column} as a workaround.`, ) exitCode = 1 return } - const isV3 = info?.version === 3 - const encryptedColumn = info?.column ?? `${options.column}_encrypted` - const requiredPhase = isV3 ? 'backfilled' : 'cut-over' - const plaintextToDrop = isV3 - ? options.column - : `${options.column}_plaintext` + const encryptedColumn = info.column + const requiredPhase = 'backfilled' + const plaintextToDrop = options.column const state = await progress(client, options.table, options.column) if (state?.phase !== requiredPhase) { p.log.error( - `Cannot generate drop migration: ${options.table}.${options.column} is in phase '${state?.phase ?? '—'}'. Must be '${requiredPhase}'${isV3 ? ' (EQL v3 has no cut-over — backfill, switch the app to the encrypted column, then drop)' : ''}.`, + `Cannot generate drop migration: ${options.table}.${options.column} is in phase '${state?.phase ?? '—'}'. Must be '${requiredPhase}' (EQL v3 has no cut-over — backfill, switch the app to the encrypted column, then drop).`, ) exitCode = 1 return } - if (isV3) { - // The phase gate above proves a backfill FINISHED at some point; it - // says nothing about rows written since (a bulk import or a service - // that isn't dual-writing leaves plaintext-only rows). Dropping the - // original column is the one irreversible step in the v3 ladder, so - // verify live coverage before generating the migration. - const unencrypted = await countUnencrypted( - client, - options.table, - options.column, - encryptedColumn, - ) - if (unencrypted > 0) { - p.log.error( - `Refusing to generate the drop migration: ${unencrypted} row(s) in ${options.table} have "${options.column}" set but "${encryptedColumn}" NULL — dropping "${options.column}" would permanently destroy that data. Likely rows written without dual-writes since the backfill. Re-run:\n stash encrypt backfill --table ${options.table} --column ${options.column}\nthen generate the drop again.`, - ) - exitCode = 1 - return - } - p.log.success( - `Verified: no rows with "${options.column}" set and "${encryptedColumn}" NULL.`, - ) - p.log.info( - `${options.table}.${encryptedColumn} is EQL v3 (${info?.domain}) — the drop targets the original plaintext column "${options.column}" (v3 has no rename, so there is no "${options.column}_plaintext"). Make sure your application reads/writes ${encryptedColumn} before applying this migration.`, + // The phase gate above proves a backfill FINISHED at some point; it + // says nothing about rows written since (a bulk import or a service + // that isn't dual-writing leaves plaintext-only rows). Dropping the + // original column is the one irreversible step in the v3 ladder, so + // verify live coverage before generating the migration. + const unencrypted = await countUnencrypted( + client, + options.table, + options.column, + encryptedColumn, + ) + if (unencrypted > 0) { + p.log.error( + `Refusing to generate the drop migration: ${unencrypted} row(s) in ${options.table} have "${options.column}" set but "${encryptedColumn}" NULL — dropping "${options.column}" would permanently destroy that data. Likely rows written without dual-writes since the backfill. Re-run:\n stash encrypt backfill --table ${options.table} --column ${options.column}\nthen generate the drop again.`, ) + exitCode = 1 + return } + p.log.success( + `Verified: no rows with "${options.column}" set and "${encryptedColumn}" NULL.`, + ) + p.log.info( + `${options.table}.${encryptedColumn} is EQL v3 (${info.domain}) — the drop targets the original plaintext column "${options.column}" (v3 has no rename, so there is no "${options.column}_plaintext"). Make sure your application reads/writes ${encryptedColumn} before applying this migration.`, + ) - const dot = options.table.indexOf('.') - const qualifiedTable = - dot >= 0 - ? `"${options.table.slice(0, dot)}"."${options.table.slice(dot + 1)}"` - : `"${options.table}"` - const dropSql = isV3 - ? buildV3DropSql(qualifiedTable, options.column, encryptedColumn) - : `-- Generated by stash encrypt drop\n-- Drops the plaintext column now that ${options.table}.${options.column} is encrypted.\n\nALTER TABLE ${qualifiedTable} DROP COLUMN "${plaintextToDrop}";\n` + const dropSql = buildV3DropSql( + options.table, + options.column, + encryptedColumn, + ) + const migrationStem = buildMigrationStem(options.table, plaintextToDrop) const cwd = process.cwd() const migrationsDir = options.migrationsDir ?? 'drizzle' @@ -181,7 +169,7 @@ export async function dropCommand(options: DropCommandOptions) { // Without this, the file ships but `drizzle-kit migrate` never picks it // up because the journal doesn't reference it. const result = await scaffoldDrizzleMigration({ - name: `drop_${options.table}_${plaintextToDrop}`, + name: migrationStem, outDir: migrationsDir, sql: dropSql, }) @@ -196,7 +184,7 @@ export async function dropCommand(options: DropCommandOptions) { .toISOString() .replace(/[-:.TZ]/g, '') .slice(0, 14) - const fileName = `${ts}_drop_${options.table}_${plaintextToDrop}.sql` + const fileName = `${ts}_${migrationStem}.sql` filePath = path.join(dirAbs, fileName) fs.writeFileSync(filePath, dropSql, 'utf-8') nextStep = `Review the migration, then apply with your migration tool:\n - prisma migrate deploy\n - psql -f ${fileName}` @@ -246,17 +234,20 @@ export async function dropCommand(options: DropCommandOptions) { * DO block so check-and-drop stay atomic even under migration runners that * don't wrap files in a transaction (plain `psql -f`). * - * Identifiers arrive pre-validated (they resolved against the live catalog - * above); embedded string literals escape single quotes defensively. + * Identifiers resolved against the live catalog above, but still require SQL + * quoting because valid PostgreSQL identifiers may contain quotes or dots. */ function buildV3DropSql( - qualifiedTable: string, + table: string, plaintextColumn: string, encryptedColumn: string, ): string { const lit = (s: string) => s.replace(/'/g, "''") + const qualifiedTable = qualifyTable(table) + const quotedPlaintext = quoteIdent(plaintextColumn) + const quotedEncrypted = quoteIdent(encryptedColumn) return `-- Generated by stash encrypt drop --- Drops the plaintext column now that ${qualifiedTable}."${encryptedColumn}" is encrypted. +-- Drops the plaintext column now that ${qualifiedTable}.${quotedEncrypted} is encrypted. -- Coverage is re-verified here, at apply time: the check stash ran at -- generation time cannot see rows written after it. @@ -270,16 +261,27 @@ BEGIN SELECT count(*) INTO unencrypted FROM ${qualifiedTable} - WHERE "${plaintextColumn}" IS NOT NULL - AND "${encryptedColumn}" IS NULL; + WHERE ${quotedPlaintext} IS NOT NULL + AND ${quotedEncrypted} IS NULL; IF unencrypted > 0 THEN RAISE EXCEPTION 'stash encrypt drop: refusing to drop %.% — % row(s) have plaintext set but % NULL. Dropping now would permanently destroy that data. Re-run: stash encrypt backfill, then regenerate this migration.', '${lit(qualifiedTable)}', '${lit(plaintextColumn)}', unencrypted, '${lit(encryptedColumn)}'; END IF; - EXECUTE 'ALTER TABLE ${lit(qualifiedTable)} DROP COLUMN "${lit(plaintextColumn)}"'; + EXECUTE 'ALTER TABLE ${lit(qualifiedTable)} DROP COLUMN ${lit(quotedPlaintext)}'; END $stash_drop$; ` } + +/** Filesystem- and drizzle-kit-safe name derived from untrusted identifiers. */ +function buildMigrationStem(...identifiers: string[]): string { + const sanitized = identifiers + .map((identifier) => + identifier.replace(/[^A-Za-z0-9_-]+/g, '_').replace(/^_+|_+$/g, ''), + ) + .filter(Boolean) + .join('_') + return `drop_${sanitized || 'column'}` +} diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index d2b0ac889..6f7f513bd 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -96,11 +96,8 @@ export async function resolveColumnLifecycle( // reports success for a rename it never performed, and `drop`'s remedy // tells the user to record the guess (#772 review, finding 7). Fail closed; // - the column exists, is not an EQL v3 column, and there are NO v3 columns - // on the table — the pure-v2 table. Nothing here can be mis-claimed, and - // `cutover` / `drop` still implement the v2 ladder, so fall through to it - // exactly as before. Gating on `candidates.length` is what keeps the - // protection above scoped to the mixed table it was written for (#787 - // review). + // on the table — the pure-v2 table. Nothing here can be mis-claimed, so + // fall through and let the caller emit its explicit v2-retirement error. if ( hint && candidates.length > 0 && @@ -153,7 +150,7 @@ export function explainUnresolved( // that: listing the v3 candidates here would invite the user to record one of // them, which is how the guess used to get laundered into a `via: 'hint'` match. if (unresolvedHint !== undefined) { - return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column. ${table} also holds EQL v3 columns, and none of them is a confirmed counterpart for ${column}, so this command cannot tell which lifecycle applies and will not guess.\n\nIf that pairing is wrong, correct or remove "encryptedColumn" for ${column} in .cipherstash/migrations.json and re-run. If it is right, ${column} is on the EQL v2 lifecycle, which no stash command can drive on this table — complete it yourself against ${unresolvedHint} with the eql_v2 SQL.` + return `${table}.${column} is recorded as pairing with "${unresolvedHint}", but ${unresolvedHint} is not an EQL v3 column — it is most likely a legacy eql_v2_encrypted column. ${table} also holds EQL v3 columns, and none of them is a confirmed counterpart for ${column}, so this command cannot tell which lifecycle applies and will not guess.\n\nIf that pairing is wrong, correct or remove "encryptedColumn" for ${column} in .cipherstash/migrations.json and re-run. If it is right, v2 mutation automation has been removed. For dump recovery, use the upstream EQL 2.3.1 SQL from https://github.com/cipherstash/encrypt-query-language/releases/tag/eql-2.3.1.` } const listed = candidates diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index c2adf39c5..e211a6367 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -1,15 +1,11 @@ import { spawnSync } from 'node:child_process' -import { writeFileSync } from 'node:fs' -import { resolve } from 'node:path' +import { existsSync, unlinkSync, writeFileSync } from 'node:fs' +import { readdir } from 'node:fs/promises' +import { join, resolve } from 'node:path' import { MIGRATIONS_SCHEMA_SQL } from '@cipherstash/migrate' import * as p from '@clack/prompts' import { CliExit } from '@/cli/exit.js' -import { - cleanupMigrationFile, - findGeneratedMigration, - printNextSteps, - SAFE_MIGRATION_NAME, -} from '@/commands/db/install.js' +import { printNextSteps, SAFE_MIGRATION_NAME } from '@/commands/db/install.js' import { describeSkipReason, rewriteEncryptedAlterColumns, @@ -25,6 +21,39 @@ import { messages } from '@/messages.js' const DEFAULT_MIGRATION_NAME = 'install-eql' const DEFAULT_DRIZZLE_OUT = 'drizzle' +/** Find the most recently generated Drizzle migration matching the name. */ +export async function findGeneratedMigration( + outDir: string, + migrationName: string, +): Promise<string> { + if (!existsSync(outDir)) { + throw new Error( + `Drizzle output directory not found: ${outDir}\nMake sure drizzle-kit is configured correctly.`, + ) + } + const matchingFiles = (await readdir(outDir)) + .filter((entry) => entry.endsWith('.sql') && entry.includes(migrationName)) + .sort() + if (matchingFiles.length === 0) { + throw new Error( + `Could not find a migration matching "${migrationName}" in ${outDir}`, + ) + } + return join(outDir, matchingFiles[matchingFiles.length - 1]) +} + +function cleanupMigrationFile(filePath: string | undefined): void { + if (!filePath) return + try { + if (existsSync(filePath)) { + unlinkSync(filePath) + p.log.info(`Cleaned up migration file: ${filePath}`) + } + } catch { + p.log.warn(`Could not clean up migration file: ${filePath}`) + } +} + export interface EqlMigrationOptions { /** Emit a Drizzle custom migration. */ drizzle?: boolean @@ -52,9 +81,9 @@ export interface EqlMigrationOptions { * Assemble the EQL **v3** install SQL for a generated migration. * * One source of truth: the SQL is the CLI's bundled v3 install script - * (`loadBundledEqlSql({ eqlVersion: 3 })`) — the same bundle `stash eql install` + * (`loadBundledEqlSql()`) — the same bundle `stash eql install` * applies directly. On `--supabase` the v3 role grants are appended - * (`supabaseGrantsFor(3)` → USAGE/EXECUTE on `eql_v3` + `eql_v3_internal` for + * (`supabaseGrantsFor()` → USAGE/EXECUTE on `eql_v3` + `eql_v3_internal` for * `anon`/`authenticated`/`service_role`), matching `stash eql install --supabase`. * Apps that connect directly as `postgres` don't need the grants, but they're * idempotent and harmless, and required when the same tables are reached via @@ -66,9 +95,9 @@ export interface EqlMigrationOptions { * install`. */ export function buildEqlV3MigrationSql(opts: { supabase: boolean }): string { - const eqlSql = loadBundledEqlSql({ eqlVersion: 3 }) + const eqlSql = loadBundledEqlSql() const grants = opts.supabase - ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${supabaseGrantsFor(3).trim()}` + ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${supabaseGrantsFor().trim()}` : '' return `${eqlSql.trim()}${grants}\n\n-- CipherStash encryption-migration tracking schema.\n-- Tracks per-column phase + backfill progress for \`stash encrypt\`.\n${MIGRATIONS_SCHEMA_SQL.trim()}\n` } diff --git a/packages/cli/src/commands/impl/index.ts b/packages/cli/src/commands/impl/index.ts index 044d82e6f..44fa65f28 100644 --- a/packages/cli/src/commands/impl/index.ts +++ b/packages/cli/src/commands/impl/index.ts @@ -40,7 +40,6 @@ function buildStateFromContext( eqlInstalled: true, agents, mode: 'implement', - usesProxy: ctx.usesProxy ?? false, } } diff --git a/packages/cli/src/commands/init/__tests__/init-command.test.ts b/packages/cli/src/commands/init/__tests__/init-command.test.ts index 44c5b3a65..cf9344eee 100644 --- a/packages/cli/src/commands/init/__tests__/init-command.test.ts +++ b/packages/cli/src/commands/init/__tests__/init-command.test.ts @@ -23,9 +23,6 @@ vi.mock('../steps/authenticate.js', () => ({ vi.mock('../steps/resolve-database.js', () => ({ resolveDatabaseStep: { id: 'resolve-database', ...passthrough }, })) -vi.mock('../steps/resolve-proxy-choice.js', () => ({ - resolveProxyChoiceStep: { id: 'resolve-proxy-choice', ...passthrough }, -})) vi.mock('../steps/build-schema.js', () => ({ buildSchemaStep: { id: 'build-schema', ...passthrough }, })) diff --git a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md index 7e6400669..8d271e939 100644 --- a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md +++ b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md @@ -18,7 +18,7 @@ This document is the **durable rule book** for any agent working on this codebas 3. **Never read or echo secrets.** Env key *names* (`CS_WORKSPACE_CRN`, `CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, `DATABASE_URL`) are fine to reference in code and docs. Their *values* are not. New env keys go in `.env.example` with placeholders; instruct the user to add the real value locally. Never read, `cat`, `grep`, or echo `~/.cipherstash/secretkey.json` (the development key), `~/.cipherstash/auth.json` (OAuth token and JWTs), anything under `~/.cipherstash/workspaces/`, or a value-bearing env file (`.env`, `.env.local`, `.env.production`, …). `.env.example` is the exception — it holds placeholders, not values, and you are expected to edit it. The CLI reads the credentials itself; no command needs you to open them. If a command fails on authentication, re-run `stash auth login` rather than inspecting the profile. 4. **Never invent CipherStash APIs.** If you don't know how a function is called, read the relevant skill (see below) — don't guess. The TypeScript types in `@cipherstash/stack` are the source of truth for what's callable. 5. **Never run database introspection yourself.** Don't run `psql`, `\d`, `pg_dump`, `supabase db dump`, or `drizzle-kit introspect`. The CLI already did this; the result is in `context.json`. If you need fresh introspection, ask the user to re-run `stash init`. -6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The EQL schemas and functions installed by the CLI — `eql_v3`/`public.eql_v3_*` today, `eql_v2`/`eql_v2_*` on a legacy install (missing function ⇒ `stash eql upgrade`, not a hand-edit). +6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The EQL schemas and functions installed by the CLI — `eql_v3`/`public.eql_v3_*` today, `eql_v2`/`eql_v2_*` on a legacy install. `stash eql upgrade` upgrades v3 only; legacy v2 state is diagnostic/read-only in current tooling. 7. **`@cipherstash/stack` must be excluded from any bundler.** The package wraps a native FFI module (`@cipherstash/protect-ffi`) that cannot be bundled. The moment you `npm install @cipherstash/stack` in a project with a bundler, configure the exclusion *before* writing any code that imports it. Concretely: Next.js needs `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` in `next.config.{js,ts,mjs}`; webpack needs `externals` entries; esbuild needs `external`; Vite SSR needs `ssr.external`. Skipping this surfaces as `Cannot find module '@cipherstash/protect-ffi-*'` at runtime, often after the user has shipped to production. If you're declaring an encrypted column for the first time in a project, treat configuring this exclusion as part of the same change. ## Migrations — three phases, always reversible diff --git a/packages/cli/src/commands/init/index.ts b/packages/cli/src/commands/init/index.ts index 4fec87bfc..11c52960b 100644 --- a/packages/cli/src/commands/init/index.ts +++ b/packages/cli/src/commands/init/index.ts @@ -14,7 +14,6 @@ import { gatherContextStep } from './steps/gather-context.js' import { installDepsStep } from './steps/install-deps.js' import { installEqlStep } from './steps/install-eql.js' import { resolveDatabaseStep } from './steps/resolve-database.js' -import { resolveProxyChoiceStep } from './steps/resolve-proxy-choice.js' import type { InitProvider, InitState } from './types.js' import { CancelledError } from './types.js' import { detectPackageManager, runnerCommand } from './utils.js' @@ -39,7 +38,6 @@ const PROVIDER_MAP: Record<string, () => InitProvider> = { const STEPS = [ authenticateStep, resolveDatabaseStep, - resolveProxyChoiceStep, buildSchemaStep, installDepsStep, installEqlStep, @@ -87,13 +85,6 @@ export async function initCommand( state.regionFlag = values.region } - // Parse --proxy and --no-proxy flags; --proxy wins if both are set - if (flags.proxy) { - state.usesProxy = true - } else if (flags['no-proxy']) { - state.usesProxy = false - } - try { for (const step of STEPS) { state = await step.run(state, provider) diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index ee1c302b7..512ad68a4 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -50,14 +50,13 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { it('frames the migrate-existing flow as rollout + backfill-and-switch with a deploy gate', () => { // The whole point of the rewrite. No "phase" jargon; explicit deploy - // gate banner; named sections. The switch step is EQL-version-aware: - // v3 (the default) has no rename — the app points at the encrypted - // column by name; cutover is the v2 rename path. + // gate banner; named sections. The v3 switch has no rename — the app + // points at the encrypted column by name. const out = renderSetupPrompt(baseCtx) expect(out).toMatch(/encryption rollout/i) expect(out).toMatch(/backfill and switch/i) - expect(out).toMatch(/EQL v3 \(the default\)/) - expect(out).toMatch(/encrypt cutover/) + expect(out).toMatch(/EQL v3/) + expect(out).not.toMatch(/encrypt cutover/) expect(out).toMatch(/deploy gate/i) expect(out).not.toMatch(/phase 1|phase 2|four-deploy/i) }) @@ -93,7 +92,7 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { it('names the lifecycle CLI commands inline in the migrate-existing flow', () => { const out = renderSetupPrompt(baseCtx) expect(out).toContain('pnpm dlx stash encrypt backfill') - expect(out).toContain('pnpm dlx stash encrypt cutover') + expect(out).not.toContain('pnpm dlx stash encrypt cutover') expect(out).toContain('pnpm dlx stash encrypt drop') expect(out).toContain('--confirm-dual-writes-deployed') expect(out).toContain('--force') @@ -358,7 +357,7 @@ describe('renderSetupPrompt — plan mode (rollout, default)', () => { const out = renderSetupPrompt(planCtx) expect(out).toContain('## What you must NOT do') expect(out).toMatch(/encrypt backfill/) - expect(out).toMatch(/encrypt cutover/) + expect(out).not.toMatch(/encrypt cutover/) expect(out).toMatch(/encrypt drop/) }) @@ -539,7 +538,7 @@ describe('renderSetupPrompt — no db push recommendations', () => { // `db push` / `eql_v2_configuration` is a v2 + CipherStash Proxy artifact and // is redundant under EQL v3 (the default). The setup prompt no longer steers // the agent toward it in any mode. - it('omits db push from the add-new-column and cutover flows (implement mode)', () => { + it('omits db push and the removed cutover command (implement mode)', () => { const out = renderSetupPrompt(baseCtx) expect(out).not.toMatch(/db push/) expect(out).not.toMatch(/Register the encryption config/) @@ -548,14 +547,10 @@ describe('renderSetupPrompt — no db push recommendations', () => { // The rollout path is schema-add → dual-write, with no push step between. expect(out).toMatch(/1\.\s*\*\*Schema-add/) expect(out).toMatch(/2\.\s*\*\*Dual-write/) - // Cutover is still covered, just without a db push workaround note. - // The heading has to be one that exists: `indexOf` returning -1 makes - // `substring(-1)` the whole document, so this scoped assertion was - // silently asserting nothing at all. const heading = '#### Backfill and switch' expect(out).toContain(heading) const cutoverSection = out.substring(out.indexOf(heading)) - expect(cutoverSection).toMatch(/encrypt cutover/) + expect(cutoverSection).not.toMatch(/encrypt cutover/) }) it('omits db push from every plan-mode template', () => { @@ -566,40 +561,17 @@ describe('renderSetupPrompt — no db push recommendations', () => { }) }) -// The rename swap is an EQL v2 operation. `stash encrypt cutover` refuses a v3 -// column outright ("Cut-over is not applicable to EQL v3 columns … there is no -// rename step", `encrypt/cutover.ts`) and refuses entirely on a v3-only -// install, where `eql_v2_configuration` does not exist. v3 is the default, so -// a template presenting the rename as the only path drafts a plan around a -// command that will decline to run. The implement prompt already splits the -// two; these templates did not. -describe('renderSetupPrompt — plan templates split EQL v3 from v2', () => { +describe('renderSetupPrompt — plan templates are EQL v3-only', () => { const plan = (planStep: 'cutover' | 'complete') => renderSetupPrompt({ ...baseCtx, mode: 'plan', planStep }) for (const planStep of ['cutover', 'complete'] as const) { describe(`${planStep} plan`, () => { - it('names both versions and states v3 has no rename', () => { + it('states v3 has no rename and never schedules v2 mutation', () => { const out = plan(planStep) expect(out).toContain('EQL v3') - expect(out).toContain('EQL v2') expect(out).toMatch(/no rename/i) - }) - - it('does not present the rename swap unconditionally', () => { - const out = plan(planStep) - // Every mention of the rename swap has to sit next to the v2 label. - for (const line of out.split('\n')) { - if (/_plaintext/.test(line)) { - expect(line).toMatch(/v2|v3/) - } - } - }) - - it('tells the agent how to determine the version per column', () => { - // The version is per-column — a database can hold both — so this - // cannot be decided once for the whole plan. - expect(plan(planStep)).toMatch(/encrypt status|encrypt backfill/) + expect(out).not.toMatch(/encrypt cutover/) }) }) } @@ -616,13 +588,8 @@ describe('renderSetupPrompt — plan templates split EQL v3 from v2', () => { } }) - it('keeps the cutover invocation scoped to v2', () => { - const out = plan('cutover') - const cutoverLine = out - .split('\n') - .find((l) => l.includes('encrypt cutover --table')) - expect(cutoverLine).toBeDefined() - expect(cutoverLine).toMatch(/v2/) + it('does not emit the removed cutover invocation', () => { + expect(plan('cutover')).not.toMatch(/encrypt cutover/) }) }) diff --git a/packages/cli/src/commands/init/lib/introspect.ts b/packages/cli/src/commands/init/lib/introspect.ts index 31b7bb398..fb2d6c3d1 100644 --- a/packages/cli/src/commands/init/lib/introspect.ts +++ b/packages/cli/src/commands/init/lib/introspect.ts @@ -56,7 +56,8 @@ export function pgTypeToDataType(udtName: string): DataType { * * Deliberately not `classifyEqlDomain` from `@cipherstash/migrate`, despite * the dependency being present: that answers "which generation authors this", - * and returns `null` for `eql_v2_encrypted` because v2 is no longer authorable. + * Legacy `eql_v2_encrypted` remains recognisable for read-only diagnostics even + * though v2 is no longer authorable. * The question here is "is this encrypted at all", which v2 answers yes to. * * The trailing underscore matters — a bare `eql_v3` prefix would also claim a diff --git a/packages/cli/src/commands/init/lib/read-context.ts b/packages/cli/src/commands/init/lib/read-context.ts index 9b7550917..4d6fa2efb 100644 --- a/packages/cli/src/commands/init/lib/read-context.ts +++ b/packages/cli/src/commands/init/lib/read-context.ts @@ -38,13 +38,7 @@ export function readContextFile(cwd: string): ContextFile | undefined { try { const parsed = JSON.parse(readFileSync(path, 'utf-8')) as unknown if (!isContextFile(parsed)) return undefined - // Normalize usesProxy to a strict boolean: older files lack it and a - // hand-edited file could hold any JSON value, so coerce to `true` only - // when it is exactly `true` and `false` otherwise. - return { - ...parsed, - usesProxy: parsed.usesProxy === true, - } + return parsed } catch { return undefined } diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 3deefd3f3..5e190612e 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -379,7 +379,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '', "Use when the column **already exists** in the user's database and contains live data that must be preserved.", '', - "Why it's staged: there is no atomic way to replace a populated column with an encrypted one without corrupting data. Instead, taking encryption to production happens in two passes around a deploy gate. The first pass — the **encryption rollout** — adds the encrypted twin column and the dual-write code; the user deploys that to production so every new write produces both plaintext and ciphertext. The second pass backfills historical rows and switches reads to the encrypted column: on EQL v3 (the default) the application points at the encrypted column **by name** and the old plaintext column is dropped — there is no rename step; on EQL v2 a **cutover** renames the encrypted twin into the original column name first.", + "Why it's staged: there is no atomic way to replace a populated column with an encrypted one without corrupting data. Instead, taking encryption to production happens in two passes around a deploy gate. The first pass — the **encryption rollout** — adds the encrypted twin column and the dual-write code; the user deploys that to production so every new write produces both plaintext and ciphertext. The second pass backfills historical rows, switches reads to the EQL v3 encrypted column by name, and drops the old plaintext column. There is no rename step.", '', '#### Encryption rollout — what lands before the deploy', '', @@ -390,14 +390,14 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '', ` \`${cli} status\``, '', - `to confirm where they are, then \`${cli} plan\` to draft the cutover. Do not run \`${cli} encrypt backfill\`, \`${cli} encrypt cutover\`, or \`${cli} encrypt drop\` until that has happened — \`${cli} impl\` will refuse to run cutover-step plans without a recorded \`dual_writing\` event.`, + `to confirm where they are, then \`${cli} plan\` to draft the remaining rollout. Do not run \`${cli} encrypt backfill\` or \`${cli} encrypt drop\` until that has happened — \`${cli} impl\` will refuse to run cutover-step plans without a recorded \`dual_writing\` event.`, '', '#### Backfill and switch — after dual-writes are live', '', `3. **Backfill.** Run \`${cli} encrypt backfill --table <T> --column <c>\`. The CLI prompts the user (or accepts \`--confirm-dual-writes-deployed\` non-interactively) to confirm dual-writes are live, then chunks through the existing rows. Resumable; checkpoints to \`cs_migrations\` after every chunk. SIGINT-safe.`, - `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table <T> --column <c>\` runs the rename in one transaction (\`<col>\` → \`<col>_plaintext\`, twin → \`<col>\`). Do **not** declare a v2 column with a \`types.*\` domain — those are EQL v3 only. The adapters no longer author v2 (\`@cipherstash/stack-drizzle\` removed \`encryptedType\`), so a v2 column is a read path: declare it with the deprecated \`@cipherstash/stack/schema\` builders and decrypt through \`@cipherstash/stack\`.`, + `4. **Switch reads to the encrypted column.** Update the schema and queries to read/write the EQL v3 encrypted column by its own name, and wire decryption through the encryption client. There is no rename step. \`${cli} encrypt backfill\` rejects legacy EQL v2 rollout state because v2 mutation automation has been removed.`, `5. **Wire the read path through the encryption client.** The read column now holds ciphertext — ${readPathGuidance(ctx.integration)}. Without this step, your read paths return raw encrypted payloads to end users. See ${integrationSkillRef(ctx.integration)} for the exact API.`, - '6. **Remove the dual-write code.** The plaintext column (still `<col>` on v3; renamed `<col>_plaintext` on v2) is no longer authoritative. Delete the dual-write logic from the persistence layer.', + '6. **Remove the dual-write code.** The original plaintext column is no longer authoritative. Delete the dual-write logic from the persistence layer.', `7. **Drop.** Run \`${cli} encrypt drop --table <T> --column <c>\`. Generates a migration that removes the now-unused plaintext column (on v3 it first verifies no rows are still plaintext-only). Apply with the project's normal migration tooling.`, '', 'Recovery: if the user reports that backfill ran *before* the dual-write code was actually live, drift is expected (rows written during the backfill window land in plaintext only). Re-run with `--force` to encrypt every plaintext row regardless of current state.', @@ -498,7 +498,7 @@ function planSharedNotDoBlock(ctx: SetupPromptContext): string[] { 'Edit schema files, application code, or migration files. The plan describes future changes — it does not perform them.', ), bullet( - `Run \`${cli} encrypt backfill\`, \`${cli} encrypt cutover\`, \`${cli} encrypt drop\`, or any other state-mutating command.`, + `Run \`${cli} encrypt backfill\`, \`${cli} encrypt drop\`, or any other state-mutating command.`, ), bullet( 'Run schema migrations (`drizzle-kit migrate`, `supabase migration up`, `prisma migrate`, etc.).', @@ -648,13 +648,13 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { '**Backfill.** Encrypt the historical rows that pre-date the rollout deploy. Resumable; chunked; SIGINT-safe.', ), bullet( - `**Switch reads to the encrypted column.** This step depends on the column's EQL version, so establish it per column first — \`${cli} encrypt backfill\` prints it and \`${cli} encrypt status\` shows it. **EQL v3 (the default):** there is no rename. Update the schema declaration and the queries to read and write \`<col>_encrypted\` under its own name. **EQL v2 (legacy data only):** declare the encrypted column under its final name (drop the twin suffix), then a single \`${cli} encrypt cutover\` transaction renames \`<col>\` → \`<col>_plaintext\` and \`<col>_encrypted\` → \`<col>\`.`, + `**Switch reads to the encrypted column.** Update the schema declaration and queries to read and write the EQL v3 \`<col>_encrypted\` column under its own name. There is no rename step; legacy EQL v2 rollout state must be migrated before mutation commands can continue.`, ), bullet( '**Read path.** Reads of the encrypted column return ciphertext until the read path decrypts via the encryption client. The plan must specify what changes per read site.', ), bullet( - '**Remove dual-writes.** The plaintext column — still `<col>` on v3, renamed `<col>_plaintext` on v2 — is no longer authoritative. Delete the dual-write code paths.', + '**Remove dual-writes.** The original plaintext column is no longer authoritative. Delete the dual-write code paths.', ), bullet( `**Drop plaintext.** \`${cli} encrypt drop\` emits a migration that removes the now-unused plaintext column; on v3 it first verifies no rows are still plaintext-only. Apply with the project's normal migration tooling.`, @@ -680,12 +680,7 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { ' encrypt backfill` invocation with concrete `--table` / `--column` values.', ), bullet( - 'The schema-edit step, stated per column against its EQL version. **v3:** point the declaration and queries at `<col>_encrypted` under its own name — there is no rename, and no `_encrypted` suffix to drop. **v2:** drop the `_encrypted` suffix and switch the original column declaration off `text`/`varchar` and onto the encrypted type.', - ), - bullet( - 'For EQL v2 columns only, the cutover invocation per column: `' + - cli + - ' encrypt cutover --table <T> --column <c>`. It refuses v3 columns — there is nothing to rename — so do not schedule it for them.', + 'The schema-edit step: point the declaration and queries at the EQL v3 `<col>_encrypted` column under its own name. There is no rename and no `_encrypted` suffix to drop.', ), bullet( 'Read-path code changes: every site that reads `<col>` from this table must decrypt via the encryption client. Enumerate the sites you can find via grep so the user can verify nothing was missed.', @@ -697,7 +692,7 @@ function renderCutoverPlanPrompt(ctx: SetupPromptContext): string { ' encrypt drop --table <T> --column <c>`, plus the schema-migration apply step that follows.', ), bullet( - `Risks specific to cutover: row-count for the backfill (use \`${cli} eql status\` to estimate if helpful), tables under heavy write load (on v2, cutover holds a brief lock for the rename), application code that constructs SQL by string (those reads won't transparently decrypt).`, + `Risks specific to cutover: row-count for the backfill (use \`${cli} eql status\` to estimate if helpful), tables under heavy write load, and application code that constructs SQL by string (those reads won't transparently decrypt).`, ), bullet( "Open questions for the user — anything you can't determine from the schema, context.json, or the skills.", @@ -742,7 +737,7 @@ function renderCompletePlanPrompt(ctx: SetupPromptContext): string { '**Add new encrypted columns** — declared encrypted from the start; single-deploy.', ), bullet( - `**Migrate existing columns** — schema-add → dual-write code → backfill → switch reads to the encrypted column → remove dual-write code → drop plaintext. The switch step depends on the column's EQL version (\`${cli} encrypt status\` shows it): on **EQL v3**, the default, there is no rename — point the schema and queries at \`<col>_encrypted\` by name; on **EQL v2** only, \`${cli} encrypt cutover\` renames the twin into \`<col>\` first. No deploy gate between rollout and cutover steps because there is no deployed application to gate on.`, + `**Migrate existing columns** — schema-add → dual-write code → backfill → switch reads to the EQL v3 encrypted column by name → remove dual-write code → drop plaintext. There is no rename step. No deploy gate between rollout and cutover steps because there is no deployed application to gate on.`, ), '', '## Your task: produce the complete-rollout plan file', diff --git a/packages/cli/src/commands/init/lib/write-context.ts b/packages/cli/src/commands/init/lib/write-context.ts index ba5bd3e95..25514ad2d 100644 --- a/packages/cli/src/commands/init/lib/write-context.ts +++ b/packages/cli/src/commands/init/lib/write-context.ts @@ -66,8 +66,6 @@ export interface ContextFile { * handoffs use. Absent when the file was written by `stash init` or * `stash impl` rather than `stash plan`. */ planStep?: PlanStep - /** Whether the user queries encrypted data via CipherStash Proxy. Captured in stash init. SDK users default to false. */ - usesProxy?: boolean generatedAt: string } @@ -134,7 +132,6 @@ export function buildContextFile(state: InitState): ContextFile { installedSkills: [], inlinedSkills: [], planStep: state.planStep, - usesProxy: state.usesProxy, generatedAt: new Date().toISOString(), } } diff --git a/packages/cli/src/commands/init/providers/__tests__/drizzle.test.ts b/packages/cli/src/commands/init/providers/__tests__/drizzle.test.ts index ef6ccfe8a..0040deb71 100644 --- a/packages/cli/src/commands/init/providers/__tests__/drizzle.test.ts +++ b/packages/cli/src/commands/init/providers/__tests__/drizzle.test.ts @@ -7,14 +7,14 @@ describe('createDrizzleProvider getNextSteps', () => { it('uses npx when package manager is npm', () => { const steps = provider.getNextSteps({}, 'npm') expect(steps[0]).toBe( - 'Set up your database: npx stash eql install --drizzle', + 'Set up your database: npx stash eql migration --drizzle', ) }) it('uses bunx when package manager is bun', () => { const steps = provider.getNextSteps({}, 'bun') expect(steps[0]).toBe( - 'Set up your database: bunx stash eql install --drizzle', + 'Set up your database: bunx stash eql migration --drizzle', ) expect(steps[1]).toContain('bunx stash wizard') for (const s of steps) expect(s).not.toMatch(/\bnpx\b/) @@ -23,14 +23,14 @@ describe('createDrizzleProvider getNextSteps', () => { it('uses pnpm dlx when package manager is pnpm', () => { const steps = provider.getNextSteps({}, 'pnpm') expect(steps[0]).toBe( - 'Set up your database: pnpm dlx stash eql install --drizzle', + 'Set up your database: pnpm dlx stash eql migration --drizzle', ) }) it('uses yarn dlx when package manager is yarn', () => { const steps = provider.getNextSteps({}, 'yarn') expect(steps[0]).toBe( - 'Set up your database: yarn dlx stash eql install --drizzle', + 'Set up your database: yarn dlx stash eql migration --drizzle', ) }) }) diff --git a/packages/cli/src/commands/init/providers/drizzle.ts b/packages/cli/src/commands/init/providers/drizzle.ts index ac79167f2..3516491cc 100644 --- a/packages/cli/src/commands/init/providers/drizzle.ts +++ b/packages/cli/src/commands/init/providers/drizzle.ts @@ -7,7 +7,7 @@ export function createDrizzleProvider(): InitProvider { introMessage: 'Setting up CipherStash for your Drizzle project...', getNextSteps(state: InitState, pm: PackageManager): string[] { const cli = runnerCommand(pm, 'stash') - const steps = [`Set up your database: ${cli} eql install --drizzle`] + const steps = [`Set up your database: ${cli} eql migration --drizzle`] const manualEdit = state.clientFilePath ? `edit ${state.clientFilePath} directly` diff --git a/packages/cli/src/commands/init/steps/__tests__/install-eql.test.ts b/packages/cli/src/commands/init/steps/__tests__/install-eql.test.ts index 7a8f373e9..023e91e8f 100644 --- a/packages/cli/src/commands/init/steps/__tests__/install-eql.test.ts +++ b/packages/cli/src/commands/init/steps/__tests__/install-eql.test.ts @@ -106,10 +106,9 @@ describe('installEqlStep', () => { it('never pins EQL v2 for the non-Drizzle paths', async () => { await installEqlStep.run(baseState, provider) - // `undefined` means "take the v3 default" in resolveEqlVersion. - expect( - vi.mocked(installCommand).mock.calls[0][0].eqlVersion, - ).toBeUndefined() + expect(vi.mocked(installCommand).mock.calls[0][0]).not.toHaveProperty( + 'eqlVersion', + ) }) describe('Drizzle', () => { diff --git a/packages/cli/src/commands/init/steps/__tests__/resolve-proxy-choice.test.ts b/packages/cli/src/commands/init/steps/__tests__/resolve-proxy-choice.test.ts deleted file mode 100644 index 9dd72eb2c..000000000 --- a/packages/cli/src/commands/init/steps/__tests__/resolve-proxy-choice.test.ts +++ /dev/null @@ -1,157 +0,0 @@ -import * as p from '@clack/prompts' -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import type { InitProvider, InitState } from '../../types.js' -import { resolveProxyChoiceStep } from '../resolve-proxy-choice.js' - -vi.mock('@clack/prompts', () => ({ - select: vi.fn(), - isCancel: vi.fn(() => false), - log: { info: vi.fn(), success: vi.fn(), warn: vi.fn(), error: vi.fn() }, -})) - -const provider = {} as InitProvider -const baseState = {} as InitState - -const originalStdinIsTTY = process.stdin.isTTY -const originalStdoutIsTTY = process.stdout.isTTY - -/** - * Force BOTH streams. A CI runner that allocates a TTY has stdin *and* stdout - * as TTYs — that is the configuration this gate used to hang in, and setting - * stdin alone would let the old `process.stdout.isTTY` gate pass these tests - * for the wrong reason (vitest's own stdout is not a TTY). - */ -function setTty(value: boolean) { - Object.defineProperty(process.stdin, 'isTTY', { value, configurable: true }) - Object.defineProperty(process.stdout, 'isTTY', { value, configurable: true }) -} - -function restoreTty() { - Object.defineProperty(process.stdin, 'isTTY', { - value: originalStdinIsTTY, - configurable: true, - }) - Object.defineProperty(process.stdout, 'isTTY', { - value: originalStdoutIsTTY, - configurable: true, - }) -} - -beforeEach(() => { - vi.clearAllMocks() -}) - -afterEach(() => { - restoreTty() - vi.unstubAllEnvs() -}) - -describe('resolveProxyChoiceStep — flag short-circuit', () => { - it('returns state untouched when --proxy/--no-proxy already set it', async () => { - setTty(true) - vi.stubEnv('CI', '') - - const state = { ...baseState, usesProxy: true } - await expect(resolveProxyChoiceStep.run(state, provider)).resolves.toEqual( - state, - ) - expect(vi.mocked(p.select)).not.toHaveBeenCalled() - }) - - it('returns state untouched when --no-proxy set usesProxy to false', async () => { - // The guard is `state.usesProxy !== undefined`; `false` is the value that - // distinguishes it from a truthiness check. Rewriting it as - // `if (state.usesProxy)` would silently re-prompt for --no-proxy (and log - // the misleading "No --proxy flag set" notice non-interactively). - setTty(true) - vi.stubEnv('CI', '') - - const state = { ...baseState, usesProxy: false } - await expect(resolveProxyChoiceStep.run(state, provider)).resolves.toEqual( - state, - ) - expect(vi.mocked(p.select)).not.toHaveBeenCalled() - // An explicit --no-proxy is not "no flag set" — don't log the default notice. - expect(vi.mocked(p.log.info)).not.toHaveBeenCalled() - }) -}) - -describe('resolveProxyChoiceStep — CI detection with a TTY attached', () => { - // Regression: this gate was `process.stdout.isTTY`, which consulted CI not - // at all. On a CI runner with an allocated TTY the clack `select` rendered - // and blocked on /dev/tty forever — a hang, not an error. `isInteractive()` - // gates on stdin and treats CI=1/TRUE/true alike. - beforeEach(() => { - setTty(true) - }) - - for (const ciValue of ['1', 'TRUE', 'true']) { - it(`defaults to SDK-only under CI=${ciValue} without prompting`, async () => { - vi.stubEnv('CI', ciValue) - - const result = await resolveProxyChoiceStep.run(baseState, provider) - - // Non-interactive proceeds with the safe default rather than failing: - // SDK-only is what the flagless interactive prompt defaults to anyway. - expect(vi.mocked(p.select)).not.toHaveBeenCalled() - expect(result.usesProxy).toBe(false) - expect(vi.mocked(p.log.info)).toHaveBeenCalledWith( - expect.stringContaining('defaulting to SDK-only mode'), - ) - }) - } - - it('prompts when CI is unset and stdin is a TTY', async () => { - vi.stubEnv('CI', '') - vi.mocked(p.select).mockResolvedValueOnce(true) - - const result = await resolveProxyChoiceStep.run(baseState, provider) - - expect(vi.mocked(p.select)).toHaveBeenCalledTimes(1) - expect(result.usesProxy).toBe(true) - }) - - it('falls back to SDK-only when the interactive prompt is cancelled', async () => { - vi.stubEnv('CI', '') - vi.mocked(p.select).mockResolvedValueOnce(Symbol('cancel')) - vi.mocked(p.isCancel).mockReturnValueOnce(true) - - const result = await resolveProxyChoiceStep.run(baseState, provider) - - expect(result.usesProxy).toBe(false) - }) -}) - -describe('resolveProxyChoiceStep — no TTY', () => { - it('defaults to SDK-only when stdin is not a TTY, CI unset', async () => { - setTty(false) - vi.stubEnv('CI', '') - - const result = await resolveProxyChoiceStep.run(baseState, provider) - - expect(vi.mocked(p.select)).not.toHaveBeenCalled() - expect(result.usesProxy).toBe(false) - }) - - it('does not prompt when stdin is piped even though stdout is a TTY', async () => { - // setTty() can't express this — it moves both streams together. The fix - // gates on stdin, so a redirected stdin (`stash init < /dev/null` from a - // terminal) must not reach the prompt even while stdout is still a TTY. - // This is the case that verifies the changeset's "wrong stream" claim; - // only the CI axis distinguishes old from new anywhere else. - Object.defineProperty(process.stdin, 'isTTY', { - value: false, - configurable: true, - }) - Object.defineProperty(process.stdout, 'isTTY', { - value: true, - configurable: true, - }) - vi.stubEnv('CI', '') - - const result = await resolveProxyChoiceStep.run(baseState, provider) - - expect(vi.mocked(p.select)).not.toHaveBeenCalled() - expect(result.usesProxy).toBe(false) - }) -}) diff --git a/packages/cli/src/commands/init/steps/install-eql.ts b/packages/cli/src/commands/init/steps/install-eql.ts index 31c9e5b9f..e8e5003a8 100644 --- a/packages/cli/src/commands/init/steps/install-eql.ts +++ b/packages/cli/src/commands/init/steps/install-eql.ts @@ -83,7 +83,7 @@ export const installEqlStep: InitStep = { // installCommand scaffolds stash.config.ts (which `import`s from `stash`) // for the rest of the workflow. `stash` must be installed or the config the - // user relies on next (db push / schema build / encrypt) can't load. Detect + // user relies on next (db validate / encrypt) can't load. Detect // the precondition and bail with a clear message instead. install-deps is // what installs the package, so a "no" there leaves us here. if (!isPackageInstalled('stash')) { @@ -147,13 +147,10 @@ export const installEqlStep: InitStep = { return { ...state, eqlInstalled: false, eqlMigrationPending: true } } - let outcome: Awaited<ReturnType<typeof installCommand>> try { - outcome = await installCommand({ + await installCommand({ supabase: supabase || undefined, databaseUrl: state.databaseUrl, - // No `eqlVersion` — take the v3 default. (The Drizzle branch above - // returns before this point.) // init passes a resolved URL to avoid re-prompting, but still wants a // config scaffolded — this is NOT a one-shot `--database-url` run. scaffoldConfig: 'ensure', @@ -174,14 +171,6 @@ export const installEqlStep: InitStep = { return { ...state, eqlInstalled: false } } - // Supabase `--migration` mode only WRITES a migration file — EQL isn't in - // the database until the user applies it. (Drizzle is handled above.) - // Report that honestly rather than claiming the extension is installed; - // the init summary turns this into "migration generated, apply it". - if (outcome === 'migration-generated') { - return { ...state, eqlInstalled: false, eqlMigrationPending: true } - } - // 'installed' | 'already-installed' — the extension is present in the DB. // ('dry-run' never happens from init; it doesn't pass dryRun.) return { ...state, eqlInstalled: true } diff --git a/packages/cli/src/commands/init/steps/resolve-proxy-choice.ts b/packages/cli/src/commands/init/steps/resolve-proxy-choice.ts deleted file mode 100644 index 1aaef5537..000000000 --- a/packages/cli/src/commands/init/steps/resolve-proxy-choice.ts +++ /dev/null @@ -1,63 +0,0 @@ -import * as p from '@clack/prompts' -import { isInteractive } from '../../../config/tty.js' -import type { InitProvider, InitState, InitStep } from '../types.js' - -/** - * Resolve whether the user queries encrypted data via CipherStash Proxy or - * directly via the SDK. Captured as a flag (--proxy / --no-proxy) or an - * interactive prompt, and stored on state.usesProxy. - * - * The prompt is non-blocking: cancellation falls back to false (SDK-only, - * the default). In non-interactive contexts without a flag, defaults to false - * and logs an info message. - */ -export const resolveProxyChoiceStep: InitStep = { - id: 'resolve-proxy-choice', - name: 'Resolve proxy choice', - async run(state: InitState, _provider: InitProvider): Promise<InitState> { - // If the flag was already set by the user, use it - if (state.usesProxy !== undefined) { - return state - } - - // Prompt only when it's safe to: stdin is a real TTY and we're not in CI. - // Via the shared `isInteractive()` (config/tty.ts) so this gate matches - // every other prompt gate. Keying off `process.stdout.isTTY` was wrong on - // both counts — it ignored CI entirely (a runner with an allocated TTY - // blocked here forever), and a redirected stdin still hangs clack `select`, - // which reads from /dev/tty. - if (isInteractive()) { - const choice = await p.select({ - message: - 'Are you planning to query encrypted data via CipherStash Proxy, or directly via the SDK?', - options: [ - { - value: false, - label: 'Directly via the SDK (most common)', - }, - { - value: true, - label: 'CipherStash Proxy', - }, - ], - initialValue: false, - }) - - // Non-blocking: if cancelled, default to false - if (p.isCancel(choice)) { - p.log.info( - 'Cancelled proxy choice; defaulting to SDK-only mode (no `stash db push` in default flows).', - ) - return { ...state, usesProxy: false } - } - - return { ...state, usesProxy: choice } - } - - // Non-interactive: default to false and log - p.log.info( - 'No --proxy flag set; defaulting to SDK-only mode (no `stash db push` in default flows).', - ) - return { ...state, usesProxy: false } - }, -} diff --git a/packages/cli/src/commands/init/types.ts b/packages/cli/src/commands/init/types.ts index d1317ef0b..738955b26 100644 --- a/packages/cli/src/commands/init/types.ts +++ b/packages/cli/src/commands/init/types.ts @@ -108,8 +108,6 @@ export interface InitState { * on-disk plan-summary block instead. Defaults to `'rollout'` when the * CLI has nothing else to go on (fresh project, no DB connectivity). */ planStep?: PlanStep - /** Whether the user queries encrypted data via CipherStash Proxy. Captured in stash init. SDK users default to false; setting true makes prompts/skills include `stash db push` steps. */ - usesProxy?: boolean } /** diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 9750af055..a1b134fe2 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -385,10 +385,9 @@ const DRIZZLE_PLACEHOLDER = `/** * to reference the encrypted tables you declared there. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and \`stash db push\`, - * \`stash db validate\` and \`stash encrypt backfill\` refuse to run and point - * back here. (\`stash encrypt cutover\` / \`drop\` resolve against the database - * and never read this file.) + * placeholder table so that this file compiles, and \`stash db validate\` and + * \`stash encrypt backfill\` refuse to run and point back here. (\`stash + * encrypt drop\` resolves against the database and never reads this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack-drizzle\`. @@ -435,8 +434,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — \`Encryption\` requires at least one. Swap it for your -// real tables (see the patterns above); \`stash db push\`, \`stash db validate\` -// and \`stash encrypt backfill\` refuse to run while the placeholder is still here. +// real tables (see the patterns above); \`stash db validate\` and \`stash +// encrypt backfill\` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) @@ -453,10 +452,9 @@ const GENERIC_PLACEHOLDER = `/** * \`Encryption({ schemas: [...] })\` call below to reference them. * * Until that happens, the encryption client is initialised with a single - * placeholder table so that this file compiles, and \`stash db push\`, - * \`stash db validate\` and \`stash encrypt backfill\` refuse to run and point - * back here. (\`stash encrypt cutover\` / \`drop\` resolve against the database - * and never read this file.) + * placeholder table so that this file compiles, and \`stash db validate\` and + * \`stash encrypt backfill\` refuse to run and point back here. (\`stash + * encrypt drop\` resolves against the database and never reads this file.) * * This project uses EQL v3. Encrypted columns are concrete Postgres domains * built with the \`types.*\` factories from \`@cipherstash/stack/eql/v3\` @@ -499,8 +497,8 @@ import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' // REPLACE THIS. It exists only so this file compiles before you have declared // any encrypted tables — \`Encryption\` requires at least one. Swap it for your -// real tables (see the patterns above); \`stash db push\`, \`stash db validate\` -// and \`stash encrypt backfill\` refuse to run while the placeholder is still here. +// real tables (see the patterns above); \`stash db validate\` and \`stash +// encrypt backfill\` refuse to run while the placeholder is still here. export const placeholderTable = encryptedTable('__stash_placeholder__', { replace_me: types.Text('replace_me'), }) diff --git a/packages/cli/src/commands/plan/index.ts b/packages/cli/src/commands/plan/index.ts index 094afd685..18182d9a4 100644 --- a/packages/cli/src/commands/plan/index.ts +++ b/packages/cli/src/commands/plan/index.ts @@ -68,7 +68,6 @@ function buildStateFromContext( agents, mode: 'plan', planStep, - usesProxy: ctx.usesProxy ?? false, } } diff --git a/packages/cli/src/commands/status/__tests__/status.test.ts b/packages/cli/src/commands/status/__tests__/status.test.ts index 343acc230..8546e9c72 100644 --- a/packages/cli/src/commands/status/__tests__/status.test.ts +++ b/packages/cli/src/commands/status/__tests__/status.test.ts @@ -193,17 +193,21 @@ describe('buildColumnQuest — migrate path', () => { // Regression guard: the hint must be prefixed with whatever `cli` the // caller passed in (e.g. `pnpm dlx stash`), never a hard-coded // `npx stash` string. - const backfill = buildColumnQuest(obs({ phase: 'dual-writing' }), CLI) + const backfill = buildColumnQuest( + obs({ phase: 'dual-writing', eqlVersion: 3 }), + CLI, + ) expect(backfill.nextMove).toContain(`${CLI} encrypt backfill`) expect(backfill.nextMove).toContain('--table users') expect(backfill.nextMove).toContain('--column email') expect(backfill.nextMove).not.toContain('npx stash') const cutover = buildColumnQuest(obs({ phase: 'backfilled' }), CLI) - expect(cutover.nextMove).toContain(`${CLI} encrypt cutover`) + expect(cutover.nextMove).toMatch(/Legacy EQL v2 rollout state/) + expect(cutover.nextMove).not.toContain('encrypt cutover') const drop = buildColumnQuest(obs({ phase: 'cut-over' }), CLI) - expect(drop.nextMove).toContain(`${CLI} encrypt drop`) + expect(drop.nextMove).toMatch(/Mutation commands for v2 have been removed/) }) it('falls back to physical-column existence as a schema-add signal', () => { @@ -231,7 +235,7 @@ describe('buildColumnQuest — new path', () => { expect(quest.objectives[0].status).toBe('active') }) - it('EQL pending: 1/2, activate active, hint uses caller-supplied runner', () => { + it('EQL pending: 1/2, retains diagnostics without recommending activation', () => { const quest = buildColumnQuest( { table: 'orders', @@ -243,7 +247,8 @@ describe('buildColumnQuest — new path', () => { ) expect(quest.progress).toEqual({ done: 1, total: 2 }) expect(quest.objectives[1].status).toBe('active') - expect(quest.nextMove).toContain(`${CLI} db activate`) + expect(quest.nextMove).toMatch(/Automatic activation has been removed/) + expect(quest.nextMove).not.toContain('db activate') }) it('EQL active: 2/2, complete', () => { @@ -347,7 +352,12 @@ describe('renderQuestLogTTY', () => { planExists: false, observedFromDb: true, observations: [ - { table: 'users', column: 'email', phase: 'dual-writing' }, + { + table: 'users', + column: 'email', + phase: 'dual-writing', + eqlVersion: 3, + }, ], cli: CLI, }) @@ -355,7 +365,7 @@ describe('renderQuestLogTTY', () => { expect(out).toContain('CipherStash Quest Log') expect(out).toContain('ACTIVE QUEST') expect(out).toContain('Encrypt users.email') - expect(out).toMatch(/2\/5 objectives/) + expect(out).toMatch(/2\/4 objectives/) expect(out).toContain('▓') expect(out).toContain('░') expect(out).toMatch(/← you are here/) @@ -419,14 +429,19 @@ describe('renderQuestLogPlain', () => { planExists: false, observedFromDb: true, observations: [ - { table: 'users', column: 'email', phase: 'dual-writing' }, + { + table: 'users', + column: 'email', + phase: 'dual-writing', + eqlVersion: 3, + }, ], cli: CLI, }) const out = renderQuestLogPlain(log, CLI) expect(out).not.toMatch(/⚔️|🏆|🔒|💡|▓|░/) expect(out).toContain('Encrypt users.email') - expect(out).toMatch(/Progress: 2\/5/) + expect(out).toMatch(/Progress: 2\/4/) expect(out).toContain('Next move:') expect(out).toContain(`${CLI} encrypt backfill`) expect(out).not.toContain('npx stash') @@ -487,8 +502,13 @@ describe('renderQuestLogJSON', () => { planExists: false, observedFromDb: true, observations: [ - { table: 'users', column: 'email', phase: 'dual-writing' }, - { table: 'users', column: 'ssn', phase: 'dropped' }, + { + table: 'users', + column: 'email', + phase: 'dual-writing', + eqlVersion: 3, + }, + { table: 'users', column: 'ssn', phase: 'dropped', eqlVersion: 3 }, ], cli: CLI, }) @@ -506,7 +526,7 @@ describe('renderQuestLogJSON', () => { expect(active.table).toBe('users') expect(active.column).toBe('email') expect(active.path).toBe('migrate') - expect(active.progress).toEqual({ done: 2, total: 5 }) + expect(active.progress).toEqual({ done: 2, total: 4 }) expect(active.complete).toBe(false) expect(active.nextMove).toContain(`${CLI} encrypt backfill`) expect(Array.isArray(active.objectives)).toBe(true) @@ -557,7 +577,12 @@ describe('nextMoveHint', () => { planExists: false, observedFromDb: true, observations: [ - { table: 'users', column: 'email', phase: 'dual-writing' }, + { + table: 'users', + column: 'email', + phase: 'dual-writing', + eqlVersion: 3, + }, ], cli: CLI, }) diff --git a/packages/cli/src/commands/status/quest.ts b/packages/cli/src/commands/status/quest.ts index 2c5ceca3d..c69d7a709 100644 --- a/packages/cli/src/commands/status/quest.ts +++ b/packages/cli/src/commands/status/quest.ts @@ -249,7 +249,7 @@ function nextMoveFor( if (doneCount === 0) { return 'Declare the encrypted column in your schema and run the migration.' } - return `Promote the pending EQL config — \`${cli} db activate\`.` + return 'Legacy EQL v2 pending configuration detected. Automatic activation has been removed; migrate the column definition to EQL v3 before continuing.' } // Migrate. The v3 ladder has no cut-over — after backfill the app @@ -269,20 +269,9 @@ function nextMoveFor( } } - switch (doneCount) { - case 0: - return 'Add the encrypted twin column (`<col>_encrypted`) and run the migration.' - case 1: - return `Wire dual-write code on every persistence path, deploy to production, then run \`${cli} encrypt backfill\` (it confirms dual-writes and records the event).` - case 2: - return `Run \`${cli} encrypt backfill --table ${obs.table} --column ${obs.column}\` to encrypt historical rows.` - case 3: - return `Run \`${cli} encrypt cutover --table ${obs.table} --column ${obs.column}\` to rename the encrypted twin into place and switch reads.` - case 4: - return `Run \`${cli} encrypt drop --table ${obs.table} --column ${obs.column}\` to remove the plaintext column.` - default: - return '' - } + return doneCount >= 5 + ? '' + : 'Legacy EQL v2 rollout state detected. Mutation commands for v2 have been removed; migrate the deployment to EQL v3 before continuing.' } /** diff --git a/packages/cli/src/config/index.ts b/packages/cli/src/config/index.ts index 4442c43e0..2932be644 100644 --- a/packages/cli/src/config/index.ts +++ b/packages/cli/src/config/index.ts @@ -242,7 +242,7 @@ export async function loadEncryptConfig( * cause. * * Shared rather than duplicated because it guards ONE file reached by two - * loaders — `loadEncryptConfig` for `stash db push` / `db validate`, and + * loaders — `loadEncryptConfig` for `stash db validate`, and * `loadEncryptionContext` for `stash encrypt backfill`. When the copies were * separate they had already drifted on the nullish-config case, so one command * named the cause while the other fell through to `requireTable`'s `Table diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index bcc9316cd..fafff0b57 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -6,8 +6,4 @@ export { resolveDatabaseUrl } from './config/database-url.ts' export type { StashConfig } from './config/index.ts' export { defineConfig, loadStashConfig } from './config/index.ts' export type { PermissionCheckResult } from './installer/index.ts' -export { - downloadEqlSql, - EQLInstaller, - loadBundledEqlSql, -} from './installer/index.ts' +export { EQLInstaller, loadBundledEqlSql } from './installer/index.ts' diff --git a/packages/cli/src/installer/grants.ts b/packages/cli/src/installer/grants.ts index ee9ec00af..9a808a3c2 100644 --- a/packages/cli/src/installer/grants.ts +++ b/packages/cli/src/installer/grants.ts @@ -10,9 +10,6 @@ * the public entry point. */ -/** EQL v2's operator schema. It has no internal schema. */ -export const EQL_SCHEMA_NAME = 'eql_v2' - /** * EQL v3 installs its operator functions into `eql_v3` (constructors live in * `eql_v3_internal`; the scalar type domains live in `public`). The `eql_v3` @@ -31,8 +28,7 @@ export const EQL_V3_INTERNAL_SCHEMA_NAME = 'eql_v3_internal' * are required. Returned as a single multi-statement string so it can be * executed in one `client.query()` (Postgres accepts multi-statement strings) * AND embedded directly into a Supabase migration file. One source of truth - * for both the runtime install path and the generated migration file, shared - * by the v2 (`eql_v2`) and v3 (`eql_v3`) installs. + * for both the runtime install path and the generated migration file. */ export function supabasePermissionsSql(schemaName: string): string { return `GRANT USAGE ON SCHEMA ${schemaName} TO anon, authenticated, service_role; @@ -60,7 +56,6 @@ ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA ${schemaName} GRANT USAGE O * only real barrier; EXECUTE is granted too so an install into a database that * has revoked EXECUTE from PUBLIC still works. * - * `eql_v2` has no internal schema, so this applies to v3 only. */ export function supabaseInternalPermissionsSql(schemaName: string): string { return `GRANT USAGE ON SCHEMA ${schemaName} TO anon, authenticated, service_role; @@ -69,9 +64,6 @@ ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA ${schemaName} GRANT EXECUTE ` } -/** The v2 (`eql_v2`) Supabase grants block. See {@link supabasePermissionsSql}. */ -export const SUPABASE_PERMISSIONS_SQL = supabasePermissionsSql(EQL_SCHEMA_NAME) - /** * The v3 Supabase grants block: `eql_v3` (the public surface) AND * `eql_v3_internal` (which its function bodies reach into). See diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 90d6ad7f7..6da7ea045 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -1,149 +1,44 @@ -import { existsSync, readFileSync } from 'node:fs' -import { dirname, join, resolve } from 'node:path' import { readInstallSql } from '@cipherstash/eql/sql' import pg from 'pg' import { - EQL_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME, - SUPABASE_PERMISSIONS_SQL, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' -/** - * The EQL **v3** install SQL, read from `@cipherstash/eql` at runtime — the - * single source of truth (the same `readInstallSql()` the stack and prisma-next - * consume). We deliberately do NOT vendor a copy into the repo: - * - a ~44k-line plpgsql artifact in the tree made GitHub classify the whole - * repo as plpgsql (CIP-3518); - * - a vendored copy silently drifts from the pinned package on every bump. - * Reading it here means a `@cipherstash/eql` version bump flows straight through. - * There is no Supabase / no-operator-family variant for v3: the bundle installs - * everywhere from one artifact (its superuser-only operator-class statements run - * inside a `DO` block that catches `insufficient_privilege` and skips). - */ -function readV3InstallSql(): string { - try { - return readInstallSql() - } catch (error) { - throw new Error( - 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', - { cause: error }, - ) - } -} - -// EQL release, pinned to match the EQL payload format this package emits. -// Bump in lockstep with @cipherstash/protect-ffi. -const EQL_VERSION = 'eql-2.3.1' -const EQL_INSTALL_URL = `https://github.com/cipherstash/encrypt-query-language/releases/download/${EQL_VERSION}/cipherstash-encrypt.sql` -const EQL_INSTALL_NO_OPERATOR_FAMILY_URL = `https://github.com/cipherstash/encrypt-query-language/releases/download/${EQL_VERSION}/cipherstash-encrypt-supabase.sql` - -// The grants SQL lives in `./grants.ts` (import-free) so the live proof in -// `@cipherstash/stack` can assert against the exact shipped strings without a -// package cycle. Re-exported here: this module stays the public entry point. export { - EQL_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME, - SUPABASE_PERMISSIONS_SQL, SUPABASE_PERMISSIONS_SQL_V3, supabaseInternalPermissionsSql, supabasePermissionsSql, } from './grants.js' -/** - * Which EQL generation to install / inspect. `2` is the composite - * `eql_v2_encrypted` type; `3` is the native concrete-domain schema - * (`public.*` domains, `eql_v3` operators). - */ +/** EQL generations recognised by read-only installation diagnostics. */ export type EqlVersion = 2 | 3 -/** - * The generation `stash eql install` / `eql upgrade` target when the user - * doesn't pass `--eql-version`. v3 (native `eql_v3.*` domain types) is the - * current default; v2 is opt-in via `--eql-version 2` and is required for the - * Drizzle / Supabase-migration / `--latest` paths, which v3 doesn't support - * yet. - */ -export const DEFAULT_EQL_VERSION: EqlVersion = 3 - -/** - * Resolve the `--eql-version` CLI string (`'2'` / `'3'` / `undefined`) to a - * concrete {@link EqlVersion}, applying {@link DEFAULT_EQL_VERSION} for anything - * that isn't an explicit `'2'`. The single authority for "which generation does - * a bare invocation target", shared by `eql install` / `eql upgrade` so the - * default lives in one place rather than being re-inlined as `=== '2' ? 2 : 3`. - * Assumes the value has already been validated (see `validateInstallFlags`). - */ -export function resolveEqlVersion(eqlVersion?: string): EqlVersion { - return eqlVersion === '2' ? 2 : DEFAULT_EQL_VERSION -} - -function schemaNameFor(eqlVersion: EqlVersion): string { - return eqlVersion === 3 ? EQL_V3_SCHEMA_NAME : EQL_SCHEMA_NAME -} +const EQL_V2_SCHEMA_NAME = 'eql_v2' -/** - * The grants block for an EQL generation — the ONE source of truth for both the - * runtime install path ({@link EQLInstaller.grantSupabasePermissions}) and the - * generated Supabase migration file. Previously the installer rebuilt the block - * from `supabasePermissionsSql(schemaNameFor(...))`, so a v3-only addition (the - * `eql_v3_internal` grants) reached the migration file and NOT the database the - * CLI installs into. - */ -export function supabaseGrantsFor(eqlVersion: EqlVersion): string { - return eqlVersion === 3 - ? SUPABASE_PERMISSIONS_SQL_V3 - : SUPABASE_PERMISSIONS_SQL -} - -/** - * Get the directory of the current file, supporting both ESM and CJS. - */ -function getCurrentDir(): string { - // ESM: import.meta.url is available - if (typeof import.meta?.url === 'string' && import.meta.url) { - return dirname(new URL(import.meta.url).pathname) +/** The pinned EQL v3 install SQL. */ +export function loadBundledEqlSql(): string { + try { + return readInstallSql() + } catch (error) { + throw new Error( + 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', + { cause: error }, + ) } - // CJS: __dirname is available - return __dirname } -/** - * Resolve the path to a bundled SQL file shipped with the package. - * - * tsup bundles everything flat: - * - Library: dist/index.js → SQL at dist/sql/ - * - CLI: dist/bin/stash.js → SQL at dist/sql/ - * - * We walk up from the current file until we find the sql/ directory. - */ -function bundledSqlPath(filename: string): string { - const thisDir = getCurrentDir() - - // Try sql/ as a sibling first (library path: dist/ -> dist/sql/) - const sibling = join(thisDir, 'sql', filename) - if (existsSync(sibling)) return sibling - - // Try one level up (CLI path: dist/bin/ -> dist/sql/) - const parent = join(thisDir, '..', 'sql', filename) - if (existsSync(parent)) return resolve(parent) - - // Fallback: return the sibling path and let the caller handle the error - return sibling +/** Supabase grants for the sole installable generation, EQL v3. */ +export function supabaseGrantsFor(): string { + return SUPABASE_PERMISSIONS_SQL_V3 } export interface PermissionCheckResult { ok: boolean missing: string[] - /** - * Whether the connected role is a Postgres superuser. Managed Postgres - * providers (Supabase, Neon, RDS, etc.) do not grant superuser, which means - * `CREATE OPERATOR FAMILY` / `CREATE OPERATOR CLASS` in the EQL install - * script will fail. Callers use this to auto-fall back to the - * no-operator-family install variant (OPE index only) instead of aborting. - */ isSuperuser: boolean } @@ -154,42 +49,20 @@ export class EQLInstaller { this.databaseUrl = options.databaseUrl } - /** - * Check whether the connected database role has the permissions required - * to install EQL. - * - * EQL installation requires: - * - SUPERUSER or CREATEDB — for `CREATE EXTENSION IF NOT EXISTS pgcrypto` - * - CREATE on the current database — for `CREATE SCHEMA eql_v2` - * - CREATE on the public schema — for `CREATE TYPE public.eql_v2_encrypted` - */ async checkPermissions(): Promise<PermissionCheckResult> { const client = new pg.Client({ connectionString: this.databaseUrl }) - try { await client.connect() - const missing: string[] = [] - - // Check if the role is a superuser (can do everything) const roleResult = await client.query(` - SELECT - rolsuper, - rolcreatedb + SELECT rolsuper, rolcreatedb FROM pg_roles WHERE rolname = current_user `) - const role = roleResult.rows[0] const isSuperuser = role?.rolsuper === true + if (isSuperuser) return { ok: true, missing: [], isSuperuser: true } - if (isSuperuser) { - return { ok: true, missing: [], isSuperuser: true } - } - - // Not a superuser — check individual permissions - - // CREATE on the current database (needed for CREATE SCHEMA, CREATE EXTENSION) const dbCreateResult = await client.query(` SELECT has_database_privilege(current_user, current_database(), 'CREATE') AS has_create `) @@ -199,28 +72,25 @@ export class EQLInstaller { ) } - // CREATE on the public schema (needed for CREATE TYPE public.eql_v2_encrypted) const schemaCreateResult = await client.query(` SELECT has_schema_privilege(current_user, 'public', 'CREATE') AS has_create `) if (!schemaCreateResult.rows[0]?.has_create) { missing.push( - 'CREATE on public schema (required for CREATE TYPE public.eql_v2_encrypted)', + 'CREATE on public schema (required for CREATE DOMAIN public.eql_v3_*)', ) } - // Check if pgcrypto is already installed — if not, we need CREATE EXTENSION privilege const pgcryptoResult = await client.query(` SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto' `) - if (pgcryptoResult.rowCount === 0 || pgcryptoResult.rowCount === null) { - // pgcrypto not installed — need to be able to create extensions - // This typically requires superuser or the role must be the extension owner - if (!role?.rolcreatedb) { - missing.push( - 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', - ) - } + if ( + (pgcryptoResult.rowCount === 0 || pgcryptoResult.rowCount === null) && + !dbCreateResult.rows[0]?.has_create + ) { + missing.push( + 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', + ) } return { ok: missing.length === 0, missing, isSuperuser: false } @@ -234,32 +104,19 @@ export class EQLInstaller { } } - /** - * Check whether the EQL extension is installed by looking for its schema - * (`eql_v3` by default, `eql_v2` when `eqlVersion: 2`). - */ + /** Generation-aware read-only detection retained for legacy diagnostics. */ async isInstalled(options?: { eqlVersion?: EqlVersion }): Promise<boolean> { const client = new pg.Client({ connectionString: this.databaseUrl }) - const eqlVersion = options?.eqlVersion ?? DEFAULT_EQL_VERSION - + const requiredSchemas = + (options?.eqlVersion ?? 3) === 3 + ? [EQL_V3_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME] + : [EQL_V2_SCHEMA_NAME] try { await client.connect() - - // v3 is generation-aware: a pre-alpha.2 install has eql_v3 but no - // eql_v3_internal (and pins v:2 envelopes in its domain CHECKs) — treat - // it as NOT installed so an install run replaces it (the bundle opens by - // dropping both schemas) instead of a stale surface silently accepting - // wrong-generation wire data. - const requiredSchemas = - eqlVersion === 3 - ? [EQL_V3_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME] - : [EQL_SCHEMA_NAME] - const result = await client.query( 'SELECT count(*)::int AS found FROM information_schema.schemata WHERE schema_name = ANY($1)', [requiredSchemas], ) - return result.rows[0]?.found === requiredSchemas.length } catch (error) { const detail = error instanceof Error ? error.message : String(error) @@ -271,45 +128,32 @@ export class EQLInstaller { } } - /** - * Return the installed EQL version, or `null` if EQL is not installed. - * - * This is best-effort: if the schema exists but no version metadata is - * available, `'unknown'` is returned. - */ + /** Read-only version diagnostics for current and legacy installs. */ async getInstalledVersion(options?: { eqlVersion?: EqlVersion }): Promise<string | null> { - const schemaName = schemaNameFor(options?.eqlVersion ?? DEFAULT_EQL_VERSION) + const schemaName = + (options?.eqlVersion ?? 3) === 3 ? EQL_V3_SCHEMA_NAME : EQL_V2_SCHEMA_NAME const client = new pg.Client({ connectionString: this.databaseUrl }) - try { await client.connect() - const schemaResult = await client.query( 'SELECT schema_name FROM information_schema.schemata WHERE schema_name = $1', [schemaName], ) - if (schemaResult.rowCount === null || schemaResult.rowCount === 0) { return null } - - // Attempt to read a version from the schema — the EQL extension may - // expose a `version()` function or a `version` table. If neither exists - // we fall back to 'unknown'. try { const versionResult = await client.query( `SELECT ${schemaName}.version() AS version`, ) - - if (versionResult.rows.length > 0 && versionResult.rows[0].version) { + if (versionResult.rows[0]?.version) { return String(versionResult.rows[0].version) } } catch { - // version() function does not exist — that's fine + // Older installs may not expose version(). } - return 'unknown' } catch (error) { const detail = error instanceof Error ? error.message : String(error) @@ -321,48 +165,9 @@ export class EQLInstaller { } } - /** - * Install the CipherStash EQL PostgreSQL extension. - * - * By default, uses the SQL bundled with this package. Pass `latest: true` - * to fetch the latest version from GitHub instead. - * - * This method is intentionally "silent" — it does not produce any console - * output. The calling CLI command is responsible for all user-facing output. - */ - async install(options?: { - excludeOperatorFamily?: boolean - supabase?: boolean - latest?: boolean - eqlVersion?: EqlVersion - }): Promise<void> { - const { - supabase = false, - latest = false, - eqlVersion = DEFAULT_EQL_VERSION, - } = options ?? {} - const excludeOperatorFamily = options?.excludeOperatorFamily || supabase - - if (latest && eqlVersion === 3) { - // The v3 bundle is read from the pinned `@cipherstash/eql` package (see - // `readV3InstallSql`), not fetched from a GitHub release, so `--latest` - // (which downloads a release asset) has no v3 target. - // Gating --latest behind --eql-version 2 is tracked in #585. - throw new Error( - '`--latest` is not supported for EQL v3 yet: no public v3 release artifacts exist. Use the bundled install.', - ) - } - - const sql = latest - ? await this.downloadInstallScript(excludeOperatorFamily) - : this.loadBundledInstallScript({ - excludeOperatorFamily, - supabase, - eqlVersion, - }) - + /** Install the pinned EQL v3 bundle. */ + async install(options?: { supabase?: boolean }): Promise<void> { const client = new pg.Client({ connectionString: this.databaseUrl }) - try { await client.connect() } catch (error) { @@ -374,200 +179,17 @@ export class EQLInstaller { try { await client.query('BEGIN') - await client.query(sql) - - if (supabase) { - await this.grantSupabasePermissions(client, eqlVersion) + await client.query(loadBundledEqlSql()) + if (options?.supabase) { + await client.query(SUPABASE_PERMISSIONS_SQL_V3) } - await client.query('COMMIT') } catch (error) { - await client.query('ROLLBACK').catch(() => { - // Swallow rollback errors — the original error is more important. - }) - + await client.query('ROLLBACK').catch(() => {}) const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to install EQL: ${detail}`, { - cause: error, - }) + throw new Error(`Failed to install EQL: ${detail}`, { cause: error }) } finally { await client.end() } } - - /** - * Grant Supabase roles access to the installed EQL schema. - * - * Supabase uses dedicated roles (anon, authenticated, service_role) that - * don't own the schema, so explicit grants are required. Issues the - * {@link supabaseGrantsFor} block as a single multi-statement query — - * Postgres accepts that and it keeps the SQL identical to what we'd write - * into a Supabase migration file. - */ - private async grantSupabasePermissions( - client: pg.Client, - eqlVersion: EqlVersion, - ): Promise<void> { - await client.query(supabaseGrantsFor(eqlVersion)) - } - - /** - * Load the EQL SQL install script bundled with this package. - */ - private loadBundledInstallScript(options: { - excludeOperatorFamily: boolean - supabase: boolean - eqlVersion: EqlVersion - }): string { - // v3 is read from `@cipherstash/eql` at runtime; only v2 is vendored. - if (options.eqlVersion === 3) return readV3InstallSql() - - const filename = resolveBundledFilename(options) - - try { - return readFileSync(bundledSqlPath(filename), 'utf-8') - } catch (error) { - throw new Error( - `Failed to load bundled EQL install script (${filename}). The package may be corrupted — try reinstalling stash.`, - { cause: error }, - ) - } - } - - /** - * Download the EQL SQL install script from GitHub. - */ - private async downloadInstallScript( - excludeOperatorFamily: boolean, - ): Promise<string> { - const url = excludeOperatorFamily - ? EQL_INSTALL_NO_OPERATOR_FAMILY_URL - : EQL_INSTALL_URL - - let response: Response - - try { - response = await fetch(url) - } catch (error) { - throw new Error('Failed to download EQL install script from GitHub.', { - cause: error, - }) - } - - if (!response.ok) { - throw new Error( - `Failed to download EQL install script from GitHub. HTTP ${response.status}: ${response.statusText}`, - ) - } - - return response.text() - } -} - -/** - * Determine which bundled SQL file to use based on install options. - * - * - `supabase: true` → Supabase-specific variant - * - `excludeOperatorFamily: true` → no operator family variant - * - default → standard install - * - * EQL v3 (eql-3.0.0+) ships ONE artifact for every target: its only - * superuser-requiring statements (the ore_block_256 operator class/family) - * self-skip on `insufficient_privilege`, and the bundle then disables the - * ORE-backed encrypted domains it cannot support (CIP-3468). So `supabase` - * and `excludeOperatorFamily` change nothing for v3 — the bundle adapts at - * install time instead of at file-selection time. - */ -function resolveBundledFilename(options: { - excludeOperatorFamily: boolean - supabase: boolean - eqlVersion?: EqlVersion -}): string { - // v3 is not vendored — it's read from `@cipherstash/eql` at runtime (see - // `readV3InstallSql`), so callers route it away before reaching here. - if ((options.eqlVersion ?? DEFAULT_EQL_VERSION) === 3) { - throw new Error( - 'resolveBundledFilename is v2-only; v3 SQL comes from `@cipherstash/eql`.', - ) - } - if (options.supabase) return 'cipherstash-encrypt-supabase.sql' - if (options.excludeOperatorFamily) - return 'cipherstash-encrypt-no-operator-family.sql' - return 'cipherstash-encrypt.sql' -} - -/** - * Load the bundled EQL install SQL. Used by the Drizzle migration path. - */ -export function loadBundledEqlSql( - options: { - excludeOperatorFamily?: boolean - supabase?: boolean - eqlVersion?: EqlVersion - } = {}, -): string { - const eqlVersion = options.eqlVersion ?? DEFAULT_EQL_VERSION - // v3 is read from `@cipherstash/eql` at runtime; only v2 is vendored. - if (eqlVersion === 3) return readV3InstallSql() - - const filename = resolveBundledFilename({ - excludeOperatorFamily: options.excludeOperatorFamily ?? false, - supabase: options.supabase ?? false, - eqlVersion, - }) - - try { - return readFileSync(bundledSqlPath(filename), 'utf-8') - } catch (error) { - throw new Error( - `Failed to load bundled EQL install script (${filename}). The package may be corrupted — try reinstalling stash.`, - { cause: error }, - ) - } -} - -/** - * Download the latest EQL install SQL from GitHub. Used by the Drizzle migration path - * when `--latest` is passed. - * - * Supabase uses the same GitHub asset as the no-operator-family variant — - * treating either flag as "no operator families" keeps the intent explicit - * even though the underlying URL is the same. - */ -export async function downloadEqlSql( - options: - | { excludeOperatorFamily?: boolean; supabase?: boolean } - | boolean = false, -): Promise<string> { - const normalized = - typeof options === 'boolean' - ? { excludeOperatorFamily: options, supabase: false } - : { - excludeOperatorFamily: options.excludeOperatorFamily ?? false, - supabase: options.supabase ?? false, - } - - const useNoOperatorFamilyUrl = - normalized.excludeOperatorFamily || normalized.supabase - const url = useNoOperatorFamilyUrl - ? EQL_INSTALL_NO_OPERATOR_FAMILY_URL - : EQL_INSTALL_URL - - let response: Response - - try { - response = await fetch(url) - } catch (error) { - throw new Error('Failed to download EQL install script from GitHub.', { - cause: error, - }) - } - - if (!response.ok) { - throw new Error( - `Failed to download EQL install script. HTTP ${response.status}: ${response.statusText}`, - ) - } - - return response.text() } diff --git a/packages/cli/src/sql/cipherstash-encrypt-no-operator-family.sql b/packages/cli/src/sql/cipherstash-encrypt-no-operator-family.sql deleted file mode 100644 index e3a0e4a94..000000000 --- a/packages/cli/src/sql/cipherstash-encrypt-no-operator-family.sql +++ /dev/null @@ -1,7434 +0,0 @@ ---! @file schema.sql ---! @brief EQL v2 schema creation ---! ---! Creates the eql_v2 schema which contains all Encrypt Query Language ---! functions, types, and tables. Drops existing schema if present to ---! support clean reinstallation. ---! ---! @warning DROP SCHEMA CASCADE will remove all objects in the schema ---! @note All EQL objects (functions, types, tables) reside in eql_v2 schema - ---! @brief Drop existing EQL v2 schema ---! @warning CASCADE will drop all dependent objects -DROP SCHEMA IF EXISTS eql_v2 CASCADE; - ---! @brief Create EQL v2 schema ---! @note All EQL functions and types will be created in this schema -CREATE SCHEMA eql_v2; - ---! @brief HMAC-SHA256 index term type ---! ---! Domain type representing HMAC-SHA256 hash values. ---! Used for exact-match encrypted searches via the 'unique' index type. ---! The hash is stored in the 'hm' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.hmac_256 AS text; - ---! @file src/ste_vec/types.sql ---! @brief Domain type for individual STE-vec entries ---! ---! Defines `eql_v2.ste_vec_entry` as a DOMAIN over `jsonb` constrained to the ---! shape of a single element inside an `sv` array — a JSON object that ---! carries at minimum a selector field (`s`). This is the type returned by ---! the `->` operator on `eql_v2_encrypted` (a single sv element extracted by ---! selector) and the type accepted by sv-element extractors such as ---! `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and ---! `eql_v2.hmac_256(eql_v2.ste_vec_entry)`. ---! ---! Why a separate type. Before #219, the `(eql_v2_encrypted)` overloads of ---! sv-element extractors read fields like `oc` off the root `data` jsonb, ---! which is misleading: a root `EncryptedPayload` or `SteVecPayload` (the ---! shapes that an actual `eql_v2_encrypted` column value carries) never has ---! `oc` at the root. The previous pattern only worked because the `->` ---! operator merged ste-vec entry fields into a fake root-shaped payload ---! before the extractor ran. This domain type makes the distinction ---! explicit: `eql_v2_encrypted` is the root shape; `eql_v2.ste_vec_entry` ---! is the per-entry shape; extractors are typed accordingly. ---! ---! @note The CHECK constraint reflects the cipherstash-suite emission ---! contract: ---! - `s` (selector — column-name HMAC) and `c` (ciphertext) are ---! emitted on every sv element. ---! - Each sv element carries **exactly one** of `hm` (HMAC-256, for ---! hash-equality queries) or `oc` (CLLW ORE, for ordered queries) ---! — they are mutually exclusive. A given selector / field is ---! configured for one mode or the other; the crypto layer emits ---! the corresponding term and only that term. ---! Other fields (`a` for array marker, etc.) are allowed but not ---! required. ---! ---! @see src/operators/->.sql ---! @see src/ore_cllw/functions.sql ---! @see src/hmac_256/functions.sql -CREATE DOMAIN eql_v2.ste_vec_entry AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 's' - AND VALUE ? 'c' - AND (VALUE ? 'hm') <> (VALUE ? 'oc') - ); - - ---! @brief Domain type for an STE-vec containment needle ---! ---! `eql_v2.stevec_query` is a query-shaped sv payload: a top-level ---! `{"sv": [...]}` object whose elements carry selector + index ---! terms but **never** a ciphertext (`c`) field. Containment (`@>`) ---! against an `eql_v2_encrypted` column is structurally typed ---! through this domain so the call site reads as "match against an ---! sv query", not "compare two encrypted values". ---! ---! Compared to `eql_v2.ste_vec_entry` (single sv element with `s`, ---! `c`, and `hm` XOR `oc`), `stevec_query` is the wrapping ---! `{"sv": [...]}` payload: it forbids `c` on every element but ---! otherwise keeps the same per-element contract — each element must ---! carry a selector `s` and exactly one deterministic term (`hm` XOR ---! `oc`). This mirrors the `SteVecQueryElement` JSON schema and stops ---! selector-only needles (e.g. `{"sv":[{"s":"x"}]}`) from casting and ---! then matching every row through the bare `jsonb @>` implementation. ---! The implementation of `ste_vec_contains` ignores `c` either way, ---! but typing the needle as `stevec_query` documents the contract at ---! the API surface. ---! ---! @note Constructing a `stevec_query` literal from inline JSON works ---! via the standard DOMAIN cast: ---! `'{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query` ---! Casting an `eql_v2_encrypted` value strips `c` fields from ---! each sv element — see `eql_v2.to_stevec_query`. ---! ---! @see eql_v2.to_stevec_query ---! @see src/operators/@>.sql -CREATE DOMAIN eql_v2.stevec_query AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 'sv' - AND jsonb_typeof(VALUE -> 'sv') = 'array' - -- No element may carry a ciphertext (`c`) — this is a query, not a value. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.c))'::jsonpath) - -- Every element must carry a selector (`s`) ... - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.s))'::jsonpath) - -- ... and exactly one deterministic term — `hm` XOR `oc` — matching - -- the `ste_vec_entry` emission contract and the `SteVecQueryElement` - -- JSON schema. Rejects selector-only needles that would otherwise - -- cast and then match every row via the bare `jsonb @>` body. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.hm) && exists(@.oc))'::jsonpath) - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.hm) && !exists(@.oc))'::jsonpath) - ); - - ---! @brief Convert an `eql_v2_encrypted` to a `stevec_query` needle ---! ---! Normalises each sv element down to the matching-relevant fields: ---! `s` (selector) plus exactly one of `hm` / `oc`. Other fields ---! (`c` ciphertext, `a` array marker, `i`/`v` envelope metadata, anything ---! else cipherstash-client might emit) are stripped. This is the ---! canonical needle shape for `@>` containment — matching the contract ---! that containment compares by selector + deterministic term and ---! ignores everything else. ---! ---! Designed for use as a functional GIN index expression: a single ---! `GIN (eql_v2.to_stevec_query(col)::jsonb jsonb_path_ops)` index ---! covers containment queries against any selector (both hm-bearing ---! and oc-bearing — XOR-aware), and the typed `@>` overloads inline ---! to a native `jsonb @>` on the same expression so the planner ---! engages Bitmap Index Scan structurally. ---! ---! @param e eql_v2_encrypted Source encrypted payload ---! @return eql_v2.stevec_query Query-shaped needle, sv elements ---! normalised to `{s, hm}` or `{s, oc}`. ---! ---! @example ---! -- Functional GIN index — canonical containment recipe ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! ---! -- Cross-row containment ---! SELECT a.* ---! FROM docs a, docs b ---! WHERE a.encrypted_doc @> b.encrypted_doc::eql_v2.stevec_query ---! AND b.id = 42; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2.to_stevec_query(e eql_v2_encrypted) - RETURNS eql_v2.stevec_query - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT jsonb_build_object( - 'sv', - coalesce( - (SELECT jsonb_agg( - jsonb_strip_nulls( - jsonb_build_object( - 's', elem -> 's', - 'hm', elem -> 'hm', - 'oc', elem -> 'oc' - ) - ) - ) - FROM jsonb_array_elements((e).data -> 'sv') AS elem), - '[]'::jsonb - ) - )::eql_v2.stevec_query -$$; - -CREATE CAST (eql_v2_encrypted AS eql_v2.stevec_query) - WITH FUNCTION eql_v2.to_stevec_query - AS ASSIGNMENT; - ---! @brief Composite type for encrypted column data ---! ---! Core type used for all encrypted columns in EQL. Stores encrypted data as JSONB ---! with the following structure: ---! - `c`: ciphertext (base64-encoded encrypted value) ---! - `i`: index terms (searchable metadata for encrypted searches) ---! - `k`: key ID (identifier for encryption key) ---! - `m`: metadata (additional encryption metadata) ---! ---! Created in public schema to persist independently of eql_v2 schema lifecycle. ---! Customer data columns use this type, so it must not be dropped if data exists. ---! ---! @note DO NOT DROP this type unless absolutely certain no encrypted data uses it ---! @see eql_v2.ciphertext ---! @see eql_v2.meta_data ---! @see eql_v2.add_column -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_encrypted') THEN - CREATE TYPE public.eql_v2_encrypted AS ( - data jsonb - ); - END IF; - END -$$; - - - - - - - - - - ---! @brief Bloom filter index term type ---! ---! Domain type representing Bloom filter bit arrays stored as smallint arrays. ---! Used for pattern-match encrypted searches via the 'match' index type. ---! The filter is stored in the 'bf' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2."~~" ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.bloom_filter AS smallint[]; - - - ---! @brief ORE block term type for Order-Revealing Encryption ---! ---! Composite type representing a single ORE (Order-Revealing Encryption) block term. ---! Stores encrypted data as bytea that enables range comparisons without decryption. ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE TYPE eql_v2.ore_block_u64_8_256_term AS ( - bytes bytea -); - - ---! @brief ORE block index term type for range queries ---! ---! Composite type containing an array of ORE block terms. Used for encrypted ---! range queries via the 'ore' index type. The array is stored in the 'ob' field ---! of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_block_u64_8_256_terms ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_block_u64_8_256 AS ( - terms eql_v2.ore_block_u64_8_256_term[] -); - ---! @brief Extract HMAC-SHA256 index term from JSONB payload ---! ---! Extracts the HMAC-SHA256 hash value from the 'hm' field of an encrypted ---! data payload. Inlinable single-statement SQL — the planner can fold this ---! into the calling query so functional hash indexes built on ---! `eql_v2.hmac_256(col)` engage structurally. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @note Returns NULL when the payload lacks `hm`. Callers that need to ---! surface misconfiguration loudly should use ---! `eql_v2.hash_encrypted` (`GROUP BY` / `DISTINCT` / hash joins) ---! which raises with a clear message when `hm` is missing. ---! ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare_hmac_256 ---! @see eql_v2.hash_encrypted -CREATE FUNCTION eql_v2.hmac_256(val jsonb) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (val ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if JSONB payload contains HMAC-SHA256 index term ---! ---! Tests whether the encrypted data payload includes an 'hm' field, ---! indicating an HMAC-SHA256 hash is available for exact-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'hm' field is present and non-null ---! ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.has_hmac_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'hm' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains HMAC-SHA256 index term ---! ---! Tests whether an encrypted column value includes an HMAC-SHA256 hash ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if HMAC-SHA256 hash is present ---! ---! @see eql_v2.has_hmac_256(jsonb) -CREATE FUNCTION eql_v2.has_hmac_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_hmac_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract HMAC-SHA256 index term from encrypted column value ---! ---! Extracts the HMAC-SHA256 hash from an encrypted column value. Inlinable ---! single-statement SQL — see the jsonb overload for the rationale. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @see eql_v2.hmac_256(jsonb) -CREATE FUNCTION eql_v2.hmac_256(val eql_v2_encrypted) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ((val).data ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Extract HMAC-SHA256 index term from a ste_vec entry ---! ---! Extracts the HMAC from the `hm` field of an `sv` element extracted via ---! the `->` operator. Inlinable. The recipe for field-level equality on ---! encrypted JSON is: ---! ---! @example ---! -- Functional hash index ---! CREATE INDEX ON users USING hash (eql_v2.hmac_256(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.hmac_256 HMAC value, or NULL when `hm` is absent ---! ---! @see eql_v2.has_hmac_256 ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.hmac_256(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (entry ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if a ste_vec entry contains an HMAC-SHA256 index term ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `hm` field is present and non-null -CREATE FUNCTION eql_v2.has_hmac_256(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'hm' IS NOT NULL -$$; - - --- AUTOMATICALLY GENERATED FILE - ---! @file common.sql ---! @brief Common utility functions ---! ---! Provides general-purpose utility functions used across EQL: ---! - Constant-time bytea comparison for security ---! - JSONB to bytea array conversion ---! - Logging helpers for debugging and testing - - ---! @brief Constant-time comparison of bytea values ---! @internal ---! ---! Compares two bytea values in constant time to prevent timing attacks. ---! Always checks all bytes even after finding differences, maintaining ---! consistent execution time regardless of where differences occur. ---! ---! @param a bytea First value to compare ---! @param b bytea Second value to compare ---! @return boolean True if values are equal ---! ---! @note Returns false immediately if lengths differ (length is not secret) ---! @note Used for secure comparison of cryptographic values -CREATE FUNCTION eql_v2.bytea_eq(a bytea, b bytea) RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result boolean; - differing bytea; -BEGIN - - -- Check if the bytea values are the same length - IF LENGTH(a) != LENGTH(b) THEN - RETURN false; - END IF; - - -- Compare each byte in the bytea values - result := true; - FOR i IN 1..LENGTH(a) LOOP - IF SUBSTRING(a FROM i FOR 1) != SUBSTRING(b FROM i FOR 1) THEN - result := result AND false; - END IF; - END LOOP; - - RETURN result; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Convert JSONB hex array to bytea array ---! @internal ---! ---! Converts a JSONB array of hex-encoded strings into a PostgreSQL bytea array. ---! Used for deserializing binary data (like ORE terms) from JSONB storage. ---! ---! @param jsonb JSONB array of hex-encoded strings ---! @return bytea[] Array of decoded binary values ---! ---! @note Returns NULL if input is JSON null ---! @note Each array element is hex-decoded to bytea -CREATE FUNCTION eql_v2.jsonb_array_to_bytea_array(val jsonb) -RETURNS bytea[] - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms_arr bytea[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(decode(value::text, 'hex')::bytea) - INTO terms_arr - FROM jsonb_array_elements_text(val) AS value; - - RETURN terms_arr; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message for debugging ---! ---! Convenience function to emit log messages during testing and debugging. ---! Uses RAISE NOTICE to output messages to PostgreSQL logs. ---! ---! @param text Message to log ---! ---! @note Primarily used in tests and development ---! @see eql_v2.log(text, text) for contextual logging -CREATE FUNCTION eql_v2.log(s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] %', s; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message with context ---! ---! Overload of log function that includes context label for better ---! log organization during testing. ---! ---! @param ctx text Context label (e.g., test name, module name) ---! @param s text Message to log ---! ---! @note Format: "[LOG] {ctx} {message}" ---! @see eql_v2.log(text) -CREATE FUNCTION eql_v2.log(ctx text, s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] % %', ctx, s; -END; -$$ LANGUAGE plpgsql; - ---! @brief CLLW ORE index term type for STE-vec range queries ---! ---! Composite type for CLLW (Copyless Logarithmic Width) Order-Revealing ---! Encryption. The ciphertext is stored in the `oc` field of encrypted data ---! payloads (Standard-mode `ste_vec` elements). Used by `eql_v2.compare` and ---! the range operators (`<`, `<=`, `>`, `>=`) when the payload carries an ---! `oc` term. ---! ---! The wire-format `oc` value is a hex string with a leading domain-tag byte ---! (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext. The ---! decoded `bytes` field on this composite carries the full byte string ---! including the tag — the comparator is variable-length capable, so numeric ---! and string values within the same column are ordered correctly: the ---! domain tag separates the two ranges (numeric < string) and the ---! within-domain comparison falls through to the CLLW per-byte protocol. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_cllw ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_cllw AS ( - bytes bytea -); - ---! @file crypto.sql ---! @brief PostgreSQL pgcrypto extension enablement ---! ---! Enables the pgcrypto extension which provides cryptographic functions ---! used by EQL for hashing and other cryptographic operations. ---! ---! Installs pgcrypto into the `extensions` schema (Supabase convention) to ---! avoid the `extension_in_public` lint. Every EQL function that uses ---! pgcrypto has `pg_catalog, extensions, public` on its `search_path`, so a ---! pre-existing install in `public` keeps working — and a pre-existing ---! install anywhere else will be rejected at install time rather than ---! failing later inside an encrypted comparison. ---! ---! @note pgcrypto provides functions like digest(), hmac(), gen_random_bytes() ---! @note If pgcrypto is already installed in `public`, EQL works but emits ---! a NOTICE recommending `ALTER EXTENSION pgcrypto SET SCHEMA extensions`. ---! @note If pgcrypto is already installed in any other schema, install ---! fails. Relocate it first with `ALTER EXTENSION pgcrypto SET SCHEMA ---! extensions` (or move it into `public` if compatibility with other ---! consumers requires it). - ---! @brief Create extensions schema (Supabase convention) -CREATE SCHEMA IF NOT EXISTS extensions; - ---! @brief Enable pgcrypto extension and validate its schema -DO $$ -DECLARE - pgcrypto_schema name; -BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') THEN - CREATE EXTENSION pgcrypto WITH SCHEMA extensions; - END IF; - - SELECT n.nspname INTO pgcrypto_schema - FROM pg_extension e - JOIN pg_namespace n ON n.oid = e.extnamespace - WHERE e.extname = 'pgcrypto'; - - IF pgcrypto_schema = 'extensions' THEN - -- expected location, nothing to say - NULL; - ELSIF pgcrypto_schema = 'public' THEN - RAISE NOTICE - 'pgcrypto is installed in the `public` schema. EQL works against this layout, ' - 'but Supabase splinter will flag it as `extension_in_public`. Move it with: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions'; - ELSE - RAISE EXCEPTION - 'pgcrypto is installed in schema `%`, which is not on the EQL function search_path ' - '(pg_catalog, extensions, public). EQL cryptographic operations would fail at ' - 'runtime. Relocate the extension before installing EQL: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions', - pgcrypto_schema; - END IF; -END $$; - ---! @brief Extract ciphertext from encrypted JSONB value ---! ---! Extracts the ciphertext (c field) from a raw JSONB encrypted value. ---! The ciphertext is the base64-encoded encrypted data. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if 'c' field is not present in JSONB ---! ---! @example ---! -- Extract ciphertext from JSONB literal ---! SELECT eql_v2.ciphertext('{"c":"AQIDBA==","i":{"unique":"..."}}'::jsonb); ---! ---! @see eql_v2.ciphertext(eql_v2_encrypted) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'c' THEN - RETURN val->>'c'; - END IF; - RAISE 'Expected a ciphertext (c) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract ciphertext from encrypted column value ---! ---! Extracts the ciphertext from an encrypted column value. Convenience ---! overload that unwraps eql_v2_encrypted type and delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if encrypted value is malformed ---! ---! @example ---! -- Extract ciphertext from encrypted column ---! SELECT eql_v2.ciphertext(encrypted_email) FROM users; ---! ---! @see eql_v2.ciphertext(jsonb) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.ciphertext(val.data); -$$; - ---! @brief State transition function for grouped_value aggregate ---! @internal ---! ---! Returns the first non-null value encountered. Used as state function ---! for the grouped_value aggregate to select first value in each group. ---! ---! @param $1 JSONB Accumulated state (first non-null value found) ---! @param $2 JSONB New value from current row ---! @return JSONB First non-null value (state or new value) ---! ---! @see eql_v2.grouped_value -CREATE FUNCTION eql_v2._first_grouped_value(jsonb, jsonb) -RETURNS jsonb -AS $$ - SELECT COALESCE($1, $2); -$$ LANGUAGE sql IMMUTABLE; - ---! @brief Return first non-null encrypted value in a group ---! ---! Aggregate function that returns the first non-null encrypted value ---! encountered within a GROUP BY clause. Useful for deduplication or ---! selecting representative values from grouped encrypted data. ---! ---! @param input JSONB Encrypted values to aggregate ---! @return JSONB First non-null encrypted value in group ---! ---! @example ---! -- Get first email per user group ---! SELECT user_id, eql_v2.grouped_value(encrypted_email) ---! FROM user_emails ---! GROUP BY user_id; ---! ---! -- Deduplicate encrypted values ---! SELECT DISTINCT ON (user_id) ---! user_id, ---! eql_v2.grouped_value(encrypted_ssn) as primary_ssn ---! FROM user_records ---! GROUP BY user_id; ---! ---! @see eql_v2._first_grouped_value -CREATE AGGREGATE eql_v2.grouped_value(jsonb) ( - SFUNC = eql_v2._first_grouped_value, - STYPE = jsonb -); - ---! @brief Add validation constraint to encrypted column ---! ---! Adds a CHECK constraint to ensure column values conform to encrypted data ---! structure. Constraint uses eql_v2.check_encrypted to validate format. ---! Called automatically by eql_v2.add_column. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to constrain ---! @return Void ---! ---! @example ---! -- Manually add constraint (normally done by add_column) ---! SELECT eql_v2.add_encrypted_constraint('users', 'encrypted_email'); ---! ---! -- Resulting constraint: ---! -- ALTER TABLE users ADD CONSTRAINT eql_v2_encrypted_check_encrypted_email ---! -- CHECK (eql_v2.check_encrypted(encrypted_email)); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_encrypted_constraint -CREATE FUNCTION eql_v2.add_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I ADD CONSTRAINT eql_v2_encrypted_constraint_%I_%I CHECK (eql_v2.check_encrypted(%I))', table_name, table_name, column_name, column_name); - EXCEPTION - WHEN duplicate_table THEN - WHEN duplicate_object THEN - RAISE NOTICE 'Constraint `eql_v2_encrypted_constraint_%_%` already exists, skipping', table_name, column_name; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove validation constraint from encrypted column ---! ---! Removes the CHECK constraint that validates encrypted data structure. ---! Called automatically by eql_v2.remove_column. Uses IF EXISTS to avoid ---! errors if constraint doesn't exist. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to unconstrain ---! @return Void ---! ---! @example ---! -- Manually remove constraint (normally done by remove_column) ---! SELECT eql_v2.remove_encrypted_constraint('users', 'encrypted_email'); ---! ---! @see eql_v2.remove_column ---! @see eql_v2.add_encrypted_constraint -CREATE FUNCTION eql_v2.remove_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I DROP CONSTRAINT IF EXISTS eql_v2_encrypted_constraint_%I_%I', table_name, table_name, column_name); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract metadata from encrypted JSONB value ---! ---! Extracts index terms (i) and version (v) from a raw JSONB encrypted value. ---! Returns metadata object containing searchable index terms without ciphertext. ---! ---! @param jsonb containing encrypted EQL payload ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Extract metadata to inspect index terms ---! SELECT eql_v2.meta_data('{"c":"...","i":{"unique":"abc123"},"v":1}'::jsonb); ---! -- Returns: {"i":{"unique":"abc123"},"v":1} ---! ---! @see eql_v2.meta_data(eql_v2_encrypted) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val jsonb) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT jsonb_build_object('i', val->'i', 'v', val->'v'); -$$; - ---! @brief Extract metadata from encrypted column value ---! ---! Extracts index terms and version from an encrypted column value. ---! Convenience overload that unwraps eql_v2_encrypted type and ---! delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Inspect index terms for encrypted column ---! SELECT user_id, eql_v2.meta_data(encrypted_email) as email_metadata ---! FROM users; ---! ---! @see eql_v2.meta_data(jsonb) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.meta_data(val.data); -$$; - - - - ---! @brief Convert JSONB to encrypted type ---! ---! Wraps a JSONB encrypted payload into the eql_v2_encrypted composite type. ---! Used internally for type conversions and operator implementations. ---! ---! @param jsonb JSONB encrypted payload with structure: {"c": "...", "i": {...}, "k": "...", "v": "2"} ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note This is primarily used for implicit casts in operator expressions ---! @see eql_v2.to_jsonb -CREATE FUNCTION eql_v2.to_encrypted(data jsonb) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT ROW(data)::public.eql_v2_encrypted; -$$; - - ---! @brief Implicit cast from JSONB to encrypted type ---! ---! Enables PostgreSQL to automatically convert JSONB values to eql_v2_encrypted ---! in assignment contexts and comparison operations. ---! ---! @see eql_v2.to_encrypted(jsonb) -CREATE CAST (jsonb AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(jsonb) AS ASSIGNMENT; - - ---! @brief Convert text to encrypted type ---! ---! Parses a text representation of encrypted JSONB payload and wraps it ---! in the eql_v2_encrypted composite type. ---! ---! @param text Text representation of JSONB encrypted payload ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note Delegates to eql_v2.to_encrypted(jsonb) after parsing text as JSON ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_encrypted(data text) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.to_encrypted(data::jsonb); -$$; - - ---! @brief Implicit cast from text to encrypted type ---! ---! Enables PostgreSQL to automatically convert text JSON strings to eql_v2_encrypted ---! in assignment contexts. ---! ---! @see eql_v2.to_encrypted(text) -CREATE CAST (text AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(text) AS ASSIGNMENT; - - - ---! @brief Convert encrypted type to JSONB ---! ---! Extracts the underlying JSONB payload from an eql_v2_encrypted composite type. ---! Useful for debugging or when raw encrypted payload access is needed. ---! ---! @param e eql_v2_encrypted Encrypted value to unwrap ---! @return jsonb Raw JSONB encrypted payload ---! ---! @note Returns the raw encrypted structure including ciphertext and index terms ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_jsonb(e public.eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT e.data; -$$; - ---! @brief Implicit cast from encrypted type to JSONB ---! ---! Enables PostgreSQL to automatically extract the JSONB payload from ---! eql_v2_encrypted values in assignment contexts. ---! ---! @see eql_v2.to_jsonb(eql_v2_encrypted) -CREATE CAST (public.eql_v2_encrypted AS jsonb) - WITH FUNCTION eql_v2.to_jsonb(public.eql_v2_encrypted) AS ASSIGNMENT; - - - - - ---! @brief Compare two encrypted values using HMAC-SHA256 index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their HMAC-SHA256 hash index terms. Used internally by the equality operator (=) ---! for exact-match queries without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Comparison uses underlying text type ordering of HMAC-SHA256 hashes ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2."=" -CREATE FUNCTION eql_v2.compare_hmac_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.hmac_256; - b_term eql_v2.hmac_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_hmac_256(a) THEN - a_term = eql_v2.hmac_256(a); - END IF; - - IF eql_v2.has_hmac_256(b) THEN - b_term = eql_v2.hmac_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - -- Using the underlying text type comparison - IF a_term = b_term THEN - RETURN 0; - END IF; - - IF a_term < b_term THEN - RETURN -1; - END IF; - - IF a_term > b_term THEN - RETURN 1; - END IF; - - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract CLLW ORE index term from a ste_vec entry ---! ---! Returns the CLLW ORE ciphertext from the `oc` field of an `sv` element. ---! `oc` is **only ever present on a `SteVecElement`** in the v2.3 payload ---! shape — never at the root of an `eql_v2_encrypted` column value — so the ---! type signature accepts `eql_v2.ste_vec_entry` directly. Callers must ---! extract first: `eql_v2.ore_cllw(col -> '<selector>')`. ---! ---! Inlinable single-statement SQL — the planner folds the body into the ---! calling query so the extractor disappears at planning time. Functional ---! btree index match on this extractor requires the `eql_v2.ore_cllw_ops` ---! opclass (installed automatically by the main / protect variants; absent ---! in the supabase variant). ---! ---! **Missing-`oc` semantics**: when the `oc` field is absent, returns a ---! SQL-level NULL (not a composite with NULL bytes). Btree's standard ---! NULL handling then filters those rows from range queries: they don't ---! match `WHERE ore_cllw(col) <op> $1`, they sort at the NULLS LAST end ---! of `ORDER BY ore_cllw(col)`, and they never reach the comparator. ---! This avoids the btree FUNCTION 1 contract violation that ---! `(bytes => NULL)` would otherwise cause (`compare_ore_cllw_term` ---! must return non-NULL int for non-NULL composite inputs). ---! ---! Callers needing a loud RAISE on missing `oc` should check ---! `eql_v2.has_ore_cllw(entry)` first. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. ---! ---! @see eql_v2.has_ore_cllw ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN entry ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(entry ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Extract CLLW ORE index term from raw jsonb (RHS parameter helper) ---! ---! Companion overload for `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` that ---! accepts a raw `jsonb` value. Intended for the right-hand side of ---! comparisons where the caller binds a literal/parameter jsonb representing ---! a single ste_vec entry: `... < eql_v2.ore_cllw($1::jsonb)`. The (jsonb) ---! form skips the domain CHECK constraint so it works for ad-hoc test inputs ---! and for the GenericComparison case in `eql_v2.compare_ore_cllw_term`. ---! ---! Returns SQL-level NULL when the input lacks `oc`, matching the ---! `(ste_vec_entry)` overload's missing-`oc` semantics so a `WHERE ---! ore_cllw(col) < ore_cllw($1::jsonb)` with a malformed query needle ---! evaluates to no rows rather than indexing a NULL-bytes composite. ---! ---! @param val jsonb An object carrying an `oc` field ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. -CREATE FUNCTION eql_v2.ore_cllw(val jsonb) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN val ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(val ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Check if a ste_vec entry contains a CLLW ORE index term ---! ---! Tests whether the entry includes an `oc` field. Inlinable. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `oc` field is present and non-null ---! ---! @see eql_v2.ore_cllw -CREATE FUNCTION eql_v2.has_ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'oc' IS NOT NULL -$$; - - ---! @brief Check if a raw jsonb value contains a CLLW ORE index term ---! ---! Companion to `eql_v2.has_ore_cllw(ste_vec_entry)` for raw jsonb inputs. ---! ---! @param val jsonb An object that may carry an `oc` field ---! @return Boolean True if `oc` field is present and non-null -CREATE FUNCTION eql_v2.has_ore_cllw(val jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT val ->> 'oc' IS NOT NULL -$$; - - ---! @brief CLLW per-byte comparison helper ---! @internal ---! ---! Byte-by-byte comparison implementing the CLLW order-revealing protocol. ---! Used by `eql_v2.compare_ore_cllw_term` for the within-prefix step. The ---! protocol: identify the index of the first differing byte across both ---! inputs; if `(y_byte + 1) == x_byte` modulo 256 at that index, then x > y; ---! otherwise x < y. Equal inputs return 0. ---! ---! Inputs MUST be the same length. The caller (`compare_ore_cllw_term`) ---! guarantees this by passing equal-length prefixes. ---! ---! @par Soft constant-time intent ---! Plpgsql is not a constant-time environment — the interpreter, `SUBSTRING`, ---! `get_byte`, and the SQL bytea representation all leak timing in ways we ---! can't control from here. Still, the loop deliberately walks every byte ---! (no `EXIT` on first difference) and the rotation check uses a bitmask ---! (`& 255`) instead of `% 256` so that what little timing structure plpgsql ---! does expose is independent of the position and value of the differing ---! byte. This is hardening intent, not a guarantee. ---! ---! Stays `LANGUAGE plpgsql` — the per-byte loop can't be expressed as a ---! single inlinable SQL expression. This is the architectural reason ORE ---! CLLW needs a custom operator class for index match, where OPE does not. ---! ---! @param a Bytea First CLLW ciphertext slice ---! @param b Bytea Second CLLW ciphertext slice ---! @return Integer -1, 0, or 1 ---! @throws Exception if inputs are different lengths ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.compare_ore_cllw_term_bytes(a bytea, b bytea) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - i INT; - first_diff INT := 0; -BEGIN - - len_a := LENGTH(a); - len_b := LENGTH(b); - - IF len_a != len_b THEN - RAISE EXCEPTION 'ore_cllw index terms are not the same length'; - END IF; - - -- Walk every byte, even after a difference is found. Record only the - -- index of the first difference (1-based; 0 means "no difference"). - -- Avoids an early `EXIT` whose presence is itself a timing signal. - FOR i IN 1..len_a LOOP - IF first_diff = 0 AND get_byte(a, i - 1) != get_byte(b, i - 1) THEN - first_diff := i; - END IF; - END LOOP; - - IF first_diff = 0 THEN - RETURN 0; - END IF; - - -- Bitmask instead of `% 256` — the modulo's operand is a power of two - -- so the two are arithmetically equivalent, but `& 255` is a single - -- machine instruction with no division-related timing variance. - IF ((get_byte(b, first_diff - 1) + 1) & 255) = get_byte(a, first_diff - 1) THEN - RETURN 1; - ELSE - RETURN -1; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Variable-length CLLW ORE term comparison ---! @internal ---! ---! Three-way comparison of two CLLW ORE ciphertext terms of potentially ---! different lengths. Compares the shared prefix via the CLLW per-byte ---! protocol; on equal prefixes, the shorter input sorts first. ---! ---! Handles both numeric (Standard-mode 65-byte CLLW outputs from the u64 ---! variant) and string (variable-length CLLW outputs) by virtue of the ---! domain-tag byte being the first byte of `bytes`. A numeric/string pair ---! differs at byte 0 (`0x00` vs `0x01`), which the CLLW rule resolves ---! correctly to numeric < string. ---! ---! Stays `LANGUAGE plpgsql` because it dispatches to ---! `compare_ore_cllw_term_bytes`, which can't be inlined. ---! ---! @par Null handling — btree FUNCTION 1 contract ---! PostgreSQL's btree filters NULL composites at the row level, so this ---! function should never be called with `a IS NULL` or `b IS NULL` under ---! normal operation. The leading IS-NULL guard returns NULL defensively ---! to cover edge cases (e.g., a non-index `ORDER BY` or `WHERE` path ---! that bypasses the opclass). ---! ---! A composite that is non-NULL but whose `bytes` field is NULL is a ---! contract violation: btree expects FUNCTION 1 to return a non-NULL ---! integer for non-NULL composite inputs. The extractor overloads of ---! `eql_v2.ore_cllw` are designed to return SQL NULL (not `ROW(NULL)`) ---! when the source payload lacks `oc`, so a NULL-bytes composite should ---! only arise from a hand-crafted literal or a future field addition to ---! the composite type. Raise loudly to surface the bug instead of ---! producing silent misordering downstream. ---! ---! @param a eql_v2.ore_cllw First term ---! @param b eql_v2.ore_cllw Second term ---! @return Integer -1, 0, or 1; NULL if either composite is NULL ---! @throws Exception if either composite has a NULL `bytes` field ---! ---! @see eql_v2.compare_ore_cllw_term_bytes ---! @see eql_v2.compare_ore_cllw -CREATE FUNCTION eql_v2.compare_ore_cllw_term(a eql_v2.ore_cllw, b eql_v2.ore_cllw) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - common_len INT; - cmp_result INT; -BEGIN - -- Composite-level NULL: btree's null-handling layer filters these at - -- the row level under normal operation. Returning NULL covers - -- non-index code paths that might still reach here. - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- Non-NULL composite with NULL bytes is a contract violation: btree's - -- FUNCTION 1 must return non-NULL int for non-NULL composite inputs. - -- The extractors return SQL NULL (not ROW(NULL)) on missing `oc`, so - -- reaching here means a hand-crafted literal or a regression in the - -- extractor body. Raise loudly rather than silently misorder. - IF a.bytes IS NULL OR b.bytes IS NULL THEN - RAISE EXCEPTION 'eql_v2.compare_ore_cllw_term: composite has NULL bytes field — extractor invariant violated. Check that the index expression uses eql_v2.ore_cllw(...) and not a hand-crafted ROW(NULL).'; - END IF; - - len_a := LENGTH(a.bytes); - len_b := LENGTH(b.bytes); - - IF len_a = 0 AND len_b = 0 THEN - RETURN 0; - ELSIF len_a = 0 THEN - RETURN -1; - ELSIF len_b = 0 THEN - RETURN 1; - END IF; - - IF len_a < len_b THEN - common_len := len_a; - ELSE - common_len := len_b; - END IF; - - cmp_result := eql_v2.compare_ore_cllw_term_bytes( - SUBSTRING(a.bytes FROM 1 FOR common_len), - SUBSTRING(b.bytes FROM 1 FOR common_len) - ); - - IF cmp_result = -1 THEN - RETURN -1; - ELSIF cmp_result = 1 THEN - RETURN 1; - END IF; - - -- Equal prefixes: shorter sorts first - IF len_a < len_b THEN - RETURN -1; - ELSIF len_a > len_b THEN - RETURN 1; - ELSE - RETURN 0; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Convert JSONB array to ORE block composite type ---! @internal ---! ---! Converts a JSONB array of hex-encoded ORE terms from the CipherStash Proxy ---! payload into the PostgreSQL composite type used for ORE operations. ---! ---! @param val JSONB Array of hex-encoded ORE block terms ---! @return eql_v2.ore_block_u64_8_256 ORE block composite type, or NULL if input is null ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_to_ore_block_u64_8_256(val jsonb) -RETURNS eql_v2.ore_block_u64_8_256 - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms eql_v2.ore_block_u64_8_256_term[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(ROW(b)::eql_v2.ore_block_u64_8_256_term) - INTO terms - FROM unnest(eql_v2.jsonb_array_to_bytea_array(val)) AS b; - - RETURN ROW(terms)::eql_v2.ore_block_u64_8_256; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from JSONB payload ---! ---! Extracts the ORE block array from the 'ob' field of an encrypted ---! data payload. Used internally for range query comparisons. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! @throws Exception if 'ob' field is missing when ore index is expected ---! ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256 -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val jsonb) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(val) THEN - RETURN eql_v2.jsonb_array_to_ore_block_u64_8_256(val->'ob'); - END IF; - RAISE 'Expected an ore index (ob) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from encrypted column value ---! ---! Extracts the ORE block from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains ORE block index term ---! ---! Tests whether the encrypted data payload includes an 'ob' field, ---! indicating an ORE block is available for range queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'ob' field is present and non-null ---! ---! @see eql_v2.ore_block_u64_8_256 -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'ob' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains ORE block index term ---! ---! Tests whether an encrypted column value includes an ORE block ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if ORE block is present ---! ---! @see eql_v2.has_ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Compare two ORE block terms using cryptographic comparison ---! @internal ---! ---! Performs a three-way comparison (returns -1/0/1) of individual ORE block terms ---! using the ORE cryptographic protocol. Compares PRP and PRF blocks to determine ---! ordering without decryption. ---! ---! @param a eql_v2.ore_block_u64_8_256_term First ORE term to compare ---! @param b eql_v2.ore_block_u64_8_256_term Second ORE term to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! @throws Exception if ciphertexts are different lengths ---! ---! @note Uses AES-ECB encryption for bit comparisons per ORE protocol ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_term(a eql_v2.ore_block_u64_8_256_term, b eql_v2.ore_block_u64_8_256_term) - RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - eq boolean := true; - unequal_block smallint := 0; - hash_key bytea; - data_block bytea; - encrypt_block bytea; - target_block bytea; - - left_block_size CONSTANT smallint := 16; - right_block_size CONSTANT smallint := 32; - right_offset CONSTANT smallint := 136; -- 8 * 17 - - indicator smallint := 0; - BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF bit_length(a.bytes) != bit_length(b.bytes) THEN - RAISE EXCEPTION 'Ciphertexts are different lengths'; - END IF; - - FOR block IN 0..7 LOOP - -- Compare each PRP (byte from the first 8 bytes) and PRF block (8 byte - -- chunks of the rest of the value). - -- NOTE: - -- * Substr is ordinally indexed (hence 1 and not 0, and 9 and not 8). - -- * We are not worrying about timing attacks here; don't fret about - -- the OR or !=. - IF - substr(a.bytes, 1 + block, 1) != substr(b.bytes, 1 + block, 1) - OR substr(a.bytes, 9 + left_block_size * block, left_block_size) != substr(b.bytes, 9 + left_block_size * BLOCK, left_block_size) - THEN - -- set the first unequal block we find - IF eq THEN - unequal_block := block; - END IF; - eq = false; - END IF; - END LOOP; - - IF eq THEN - RETURN 0::integer; - END IF; - - -- Hash key is the IV from the right CT of b - hash_key := substr(b.bytes, right_offset + 1, 16); - - -- first right block is at right offset + nonce_size (ordinally indexed) - target_block := substr(b.bytes, right_offset + 17 + (unequal_block * right_block_size), right_block_size); - - data_block := substr(a.bytes, 9 + (left_block_size * unequal_block), left_block_size); - - encrypt_block := encrypt(data_block::bytea, hash_key::bytea, 'aes-ecb'); - - indicator := ( - get_bit( - encrypt_block, - 0 - ) + get_bit(target_block, get_byte(a.bytes, unequal_block))) % 2; - - IF indicator = 1 THEN - RETURN 1::integer; - ELSE - RETURN -1::integer; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Compare arrays of ORE block terms recursively ---! @internal ---! ---! Recursively compares arrays of ORE block terms element-by-element. ---! Empty arrays are considered less than non-empty arrays. If the first elements ---! are equal, recursively compares remaining elements. ---! ---! @param a eql_v2.ore_block_u64_8_256_term[] First array of ORE terms ---! @param b eql_v2.ore_block_u64_8_256_term[] Second array of ORE terms ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b, NULL if either array is NULL ---! ---! @note Empty arrays sort before non-empty arrays ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256_term[], b eql_v2.ore_block_u64_8_256_term[]) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - cmp_result integer; - BEGIN - - -- NULLs are NULL - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- empty a and b - IF cardinality(a) = 0 AND cardinality(b) = 0 THEN - RETURN 0; - END IF; - - -- empty a and some b - IF (cardinality(a) = 0) AND cardinality(b) > 0 THEN - RETURN -1; - END IF; - - -- some a and empty b - IF cardinality(a) > 0 AND (cardinality(b) = 0) THEN - RETURN 1; - END IF; - - cmp_result := eql_v2.compare_ore_block_u64_8_256_term(a[1], b[1]); - - IF cmp_result = 0 THEN - -- Removes the first element in the array, and calls this fn again to compare the next element/s in the array. - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a[2:array_length(a,1)], b[2:array_length(b,1)]); - END IF; - - RETURN cmp_result; - END -$$ LANGUAGE plpgsql; - - ---! @brief Compare ORE block composite types ---! @internal ---! ---! Wrapper function that extracts term arrays from ORE block composite types ---! and delegates to the array comparison function. ---! ---! @param a eql_v2.ore_block_u64_8_256 First ORE block ---! @param b eql_v2.ore_block_u64_8_256 Second ORE block ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms(eql_v2.ore_block_u64_8_256_term[], eql_v2.ore_block_u64_8_256_term[]) -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a.terms, b.terms); - END -$$ LANGUAGE plpgsql; - - ---! @brief Extract STE vector index from JSONB payload ---! ---! Extracts the STE (Searchable Symmetric Encryption) vector from the 'sv' field ---! of an encrypted data payload. Returns an array of encrypted values used for ---! containment queries (@>, <@). If no 'sv' field exists, wraps the entire payload ---! as a single-element array. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(eql_v2_encrypted) ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.ste_vec(val jsonb) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv jsonb; - ary public.eql_v2_encrypted[]; - BEGIN - - IF val ? 'sv' THEN - sv := val->'sv'; - ELSE - sv := jsonb_build_array(val); - END IF; - - SELECT array_agg(eql_v2.to_encrypted(elem)) - INTO ary - FROM jsonb_array_elements(sv) AS elem; - - RETURN ary; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract STE vector index from encrypted column value ---! ---! Extracts the STE vector from an encrypted column value by accessing its ---! underlying JSONB data field. Used for containment query operations. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(jsonb) -CREATE FUNCTION eql_v2.ste_vec(val eql_v2_encrypted) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.ste_vec(val.data)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if JSONB payload is a single-element STE vector ---! ---! Tests whether the encrypted data payload contains an 'sv' field with exactly ---! one element. Single-element STE vectors can be treated as regular encrypted values. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'sv' field exists with exactly one element ---! ---! @see eql_v2.to_ste_vec_value -CREATE FUNCTION eql_v2.is_ste_vec_value(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'sv' THEN - RETURN jsonb_array_length(val->'sv') = 1; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if encrypted column value is a single-element STE vector ---! ---! Tests whether an encrypted column value is a single-element STE vector ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is a single-element STE vector ---! ---! @see eql_v2.is_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_value(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.is_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value ---! ---! Extracts the single element from a single-element STE vector and returns it ---! as a regular encrypted value, preserving metadata. If the input is not a ---! single-element STE vector, returns it unchanged. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.is_ste_vec_value -CREATE FUNCTION eql_v2.to_ste_vec_value(val jsonb) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - meta jsonb; - sv jsonb; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_value(val) THEN - meta := eql_v2.meta_data(val); - sv := val->'sv'; - sv := sv[0]; - - RETURN eql_v2.to_encrypted(meta || sv); - END IF; - - RETURN eql_v2.to_encrypted(val); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value (encrypted type) ---! ---! Converts an encrypted column value to a regular encrypted value by unwrapping ---! if it's a single-element STE vector. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.to_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.to_ste_vec_value(val eql_v2_encrypted) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.to_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract selector value from JSONB payload ---! ---! Extracts the selector ('s') field from an encrypted data payload. ---! Selectors are used to match STE vector elements during containment queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text The selector value ---! @throws Exception if 's' field is missing ---! ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.selector(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF val ? 's' THEN - RETURN val->>'s'; - END IF; - RAISE 'Expected a selector index (s) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from encrypted column value ---! @internal ---! ---! Internal convenience: unwraps the encrypted composite and delegates ---! to `eql_v2.selector(jsonb)`. Exists so the encrypted-selector ---! overloads of `eql_v2."->"` / `eql_v2."->>"` / `eql_v2.jsonb_path_*` ---! can dispatch without each having to spell out `(val).data` first. ---! Not part of the public API — callers should use ---! `eql_v2.selector(jsonb)` or `eql_v2.selector(eql_v2.ste_vec_entry)`. ---! ---! @param eql_v2_encrypted Encrypted column value (single-element form) ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) ---! @see eql_v2.selector(eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2._selector(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.selector(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from a ste_vec entry ---! ---! Direct overload on the domain type. The DOMAIN's CHECK constraint ---! already guarantees `s` is present, so this is a simple field access. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) -CREATE FUNCTION eql_v2.selector(entry eql_v2.ste_vec_entry) - RETURNS text - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 's' -$$; - - - ---! @brief Check if JSONB payload is marked as an STE vector array ---! ---! Tests whether the encrypted data payload has the 'a' (array) flag set to true, ---! indicating it represents an array for STE vector operations. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'a' field is present and true ---! ---! @see eql_v2.ste_vec -CREATE FUNCTION eql_v2.is_ste_vec_array(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'a' THEN - RETURN (val->>'a')::boolean; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value is marked as an STE vector array ---! ---! Tests whether an encrypted column value has the array flag set by checking ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is marked as an STE vector array ---! ---! @see eql_v2.is_ste_vec_array(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_array(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.is_ste_vec_array(val.data)); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract full encrypted JSONB elements as array ---! ---! Extracts all JSONB elements from the STE vector including non-deterministic fields. ---! Use jsonb_array() instead for GIN indexing and containment queries. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT CASE - WHEN val ? 'sv' THEN - ARRAY(SELECT elem FROM jsonb_array_elements(val->'sv') AS elem) - ELSE - ARRAY[val] - END; -$$; - - ---! @brief Extract full encrypted JSONB elements as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array_from_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array_from_array_elements(val.data); -$$; - - ---! @brief Extract deterministic fields as array for GIN indexing ---! ---! Extracts only deterministic search term fields (`s`, `hm`, `oc`, `op`) ---! from each STE vector element. Excludes non-deterministic ciphertext for ---! correct containment comparison using PostgreSQL's native `@>` operator. ---! ---! Field set: selector (`s`), HMAC equality (`hm`), ORE CLLW (`oc`, ---! Standard-mode), OPE CLLW (`op`, Compat-mode). The pre-2.3 fields ---! (`b3` / `ocf` / `ocv` / `opf` / `opv`) are no longer emitted — see U-004 ---! and U-006 in `docs/upgrading/v2.3.md`. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @note Use this for GIN indexes and containment queries ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_array(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT ARRAY( - SELECT jsonb_object_agg(kv.key, kv.value) - FROM jsonb_array_elements( - CASE WHEN val ? 'sv' THEN val->'sv' ELSE jsonb_build_array(val) END - ) AS elem, - LATERAL jsonb_each(elem) AS kv(key, value) - WHERE kv.key IN ('s', 'hm', 'oc', 'op') - GROUP BY elem - ); -$$; - - ---! @brief Extract deterministic fields as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @see eql_v2.jsonb_array(jsonb) -CREATE FUNCTION eql_v2.jsonb_array(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(val.data); -$$; - - ---! @brief GIN-indexable JSONB containment check ---! ---! Checks if encrypted value 'a' contains all JSONB elements from 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! This function is designed for use with a GIN index on jsonb_array(column). ---! When combined with such an index, PostgreSQL can efficiently search large tables. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b eql_v2_encrypted Value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @example ---! -- Create GIN index for efficient containment queries ---! CREATE INDEX idx ON mytable USING GIN (eql_v2.jsonb_array(encrypted_col)); ---! ---! -- Query using the helper function ---! SELECT * FROM mytable WHERE eql_v2.jsonb_contains(encrypted_col, search_value); ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (encrypted, jsonb) ---! ---! Checks if encrypted value 'a' contains all JSONB elements from jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b jsonb JSONB value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (jsonb, encrypted) ---! ---! Checks if jsonb value 'a' contains all JSONB elements from encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Container JSONB value ---! @param b eql_v2_encrypted Encrypted value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check ---! ---! Checks if all JSONB elements from 'a' are contained in 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b eql_v2_encrypted Container value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (encrypted, jsonb) ---! ---! Checks if all JSONB elements from encrypted value 'a' are contained in jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b jsonb Container JSONB value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (jsonb, encrypted) ---! ---! Checks if all JSONB elements from jsonb value 'a' are contained in encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Value to check ---! @param b eql_v2_encrypted Container encrypted value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief Check if STE vector array contains a specific encrypted element ---! ---! Tests whether any element in the STE vector array 'a' contains the encrypted value 'b'. ---! Matching requires both the selector and encrypted value to be equal. ---! Used internally by ste_vec_contains(encrypted, encrypted) for array containment checks. ---! ---! @param eql_v2_encrypted[] STE vector array to search within ---! @param eql_v2_encrypted Encrypted element to search for ---! @return Boolean True if b is found in any element of a ---! ---! @note Compares both selector and encrypted value for match ---! ---! @see eql_v2.selector ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.ste_vec_contains(a public.eql_v2_encrypted[], b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - _a public.eql_v2_encrypted; - BEGIN - - result := false; - - FOR idx IN 1..array_length(a, 1) LOOP - _a := a[idx]; - -- Element-level match for ste_vec entries. - -- - -- Per the v2.3 sv-element contract (encoded in - -- `docs/reference/schema/eql-payload-v2.3.schema.json` and the - -- `eql_v2.ste_vec_entry` DOMAIN), each entry carries **exactly - -- one** of: - -- - `hm` — HMAC-256 for boolean leaves and for the placeholder - -- entries that represent array / object roots. - -- - `oc` — CLLW ORE for string and number leaves. - -- Both terms are deterministic for the same plaintext at the same - -- selector under the same workspace, so either one serves as the - -- equality discriminator. A selector configures the leaf's role - -- (eq / ordered), and the role determines which term is emitted — - -- two sv entries with the same selector therefore always carry - -- the same term type. - -- - -- The selector check is a fast-path gate so we don't compare - -- terms across mismatched fields. Once selectors match, exactly - -- one of the two CASE branches fires (XOR contract above). - -- - -- The `ELSE false` arm covers the malformed case (entry carries - -- neither term, or only one side has the term for a given role). - -- That's a data error rather than a normal containment result, - -- but returning false is safer than raising mid-array-scan. - result := result OR ( - eql_v2._selector(_a) = eql_v2._selector(b) AND - CASE - WHEN eql_v2.has_hmac_256(_a) AND eql_v2.has_hmac_256(b) THEN - eql_v2.compare_hmac_256(_a, b) = 0 - WHEN eql_v2.has_ore_cllw((_a).data) AND eql_v2.has_ore_cllw((b).data) THEN - eql_v2.compare_ore_cllw_term( - eql_v2.ore_cllw((_a).data), - eql_v2.ore_cllw((b).data) - ) = 0 - ELSE false - END - ); - - -- Short-circuit once a match is found. Without this we still walk - -- the rest of the sv array, which on a 100-element document means - -- 99 wasted selector + extractor calls per row. - EXIT WHEN result; - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted value 'a' contains all elements of encrypted value 'b' ---! ---! Performs STE vector containment comparison between two encrypted values. ---! Returns true if all elements in b's STE vector are found in a's STE vector. ---! Used internally by the @> containment operator for searchable encryption. ---! ---! @param a eql_v2_encrypted First encrypted value (container) ---! @param b eql_v2_encrypted Second encrypted value (elements to find) ---! @return Boolean True if all elements of b are contained in a ---! ---! @note Empty b is always contained in any a ---! @note Each element of b must match both selector and value in a ---! ---! @see eql_v2.ste_vec ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted[], eql_v2_encrypted) ---! @see eql_v2."@>" -CREATE FUNCTION eql_v2.ste_vec_contains(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - sv_a public.eql_v2_encrypted[]; - sv_b public.eql_v2_encrypted[]; - _b public.eql_v2_encrypted; - BEGIN - - -- jsonb arrays of ste_vec encrypted values - sv_a := eql_v2.ste_vec(a); - sv_b := eql_v2.ste_vec(b); - - -- an empty b is always contained in a - IF array_length(sv_b, 1) IS NULL THEN - RETURN true; - END IF; - - IF array_length(sv_a, 1) IS NULL THEN - RETURN false; - END IF; - - result := true; - - -- for each element of b check if it is in a - FOR idx IN 1..array_length(sv_b, 1) LOOP - _b := sv_b[idx]; - result := result AND eql_v2.ste_vec_contains(sv_a, _b); - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; ---! @file config/types.sql ---! @brief Configuration state type definition ---! ---! Defines the ENUM type for tracking encryption configuration lifecycle states. ---! The configuration table uses this type to manage transitions between states ---! during setup, activation, and encryption operations. ---! ---! @note CREATE TYPE does not support IF NOT EXISTS, so wrapped in DO block ---! @note Configuration data stored as JSONB directly, not as DOMAIN ---! @see config/tables.sql - - ---! @brief Configuration lifecycle state ---! ---! Defines valid states for encryption configurations in the eql_v2_configuration table. ---! Configurations transition through these states during setup and activation. ---! ---! @note Only one configuration can be in 'active', 'pending', or 'encrypting' state at once ---! @see config/indexes.sql for uniqueness enforcement ---! @see config/tables.sql for usage in eql_v2_configuration table -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_configuration_state') THEN - CREATE TYPE public.eql_v2_configuration_state AS ENUM ('active', 'inactive', 'encrypting', 'pending'); - END IF; - END -$$; - - - ---! @brief Extract Bloom filter index term from JSONB payload ---! ---! Extracts the Bloom filter array from the 'bf' field of an encrypted ---! data payload. Used internally for pattern-match queries (LIKE operator). ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! @throws Exception if 'bf' field is missing when bloom_filter index is expected ---! ---! @see eql_v2.has_bloom_filter ---! @see eql_v2."~~" -CREATE FUNCTION eql_v2.bloom_filter(val jsonb) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_bloom_filter(val) THEN - RETURN ARRAY(SELECT jsonb_array_elements(val->'bf'))::eql_v2.bloom_filter; - END IF; - - RAISE 'Expected a match index (bf) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract Bloom filter index term from encrypted column value ---! ---! Extracts the Bloom filter from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! ---! @see eql_v2.bloom_filter(jsonb) -CREATE FUNCTION eql_v2.bloom_filter(val eql_v2_encrypted) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.bloom_filter(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains Bloom filter index term ---! ---! Tests whether the encrypted data payload includes a 'bf' field, ---! indicating a Bloom filter is available for pattern-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'bf' field is present and non-null ---! ---! @see eql_v2.bloom_filter -CREATE FUNCTION eql_v2.has_bloom_filter(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'bf' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains Bloom filter index term ---! ---! Tests whether an encrypted column value includes a Bloom filter ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if Bloom filter is present ---! ---! @see eql_v2.has_bloom_filter(jsonb) -CREATE FUNCTION eql_v2.has_bloom_filter(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_bloom_filter(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @file src/ste_vec/eq_term.sql ---! @brief XOR-aware equality term extractor for `eql_v2.ste_vec_entry` ---! ---! Returns the bytea representation of whichever deterministic term ---! the sv entry carries — `hm` (HMAC-256) for bool leaves / array ---! roots / object roots, or `oc` (CLLW ORE) for string / number ---! leaves. The two byte distributions are disjoint by construction ---! (different keys, different protocols), so byte equality on the ---! coalesce is unambiguous: equal terms imply equal plaintexts under ---! the same selector, and unequal terms imply different plaintexts ---! (or different protocols, which can't happen for a single ---! selector). ---! ---! This is the canonical equality extractor used by `=` and `<>` on ---! `eql_v2.ste_vec_entry` — see `src/operators/ste_vec_entry.sql`. ---! The recipe for field-level equality on encrypted JSON is: ---! ---! @example ---! -- Functional hash index covers both hm-bearing and oc-bearing selectors ---! CREATE INDEX ON users USING hash (eql_v2.eq_term(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return bytea Decoded `hm` or `oc` bytes (NULL if entry is NULL). ---! ---! @note The XOR contract (each sv entry carries exactly one of `hm` ---! or `oc` — enforced by the `ste_vec_entry` DOMAIN CHECK) means ---! the coalesce always picks the one present term. ---! ---! @see eql_v2.hmac_256(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.eq_term(entry eql_v2.ste_vec_entry) - RETURNS bytea - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT decode(coalesce(entry ->> 'hm', entry ->> 'oc'), 'hex') -$$; - - - ---! @file src/operators/compare.sql ---! @brief Three-way ordering on the root `eql_v2_encrypted` type ---! ---! Returns `-1` / `0` / `1` for two encrypted column values that carry ---! Block ORE (`ob`) terms at the root. Used by the btree operator class on ---! `eql_v2_encrypted` (FUNCTION 1), by the legacy `eql_v2.lt` / `lte` / ---! `gt` / `gte` helpers, and by `sort_compare`'s `strategy = 'compare'` ---! fallback path. ---! ---! **Strict Block-ORE-only contract.** Root-level `eql_v2_encrypted` values ---! only carry root-scope ORE terms (`ob`) per the v2.3 payload shape — the ---! `oc` field (CLLW ORE) is sv-element scope only and never appears on a ---! root payload. Equality on `eql_v2_encrypted` is hm-only and runs through ---! the inlined `=` / `<>` operators (post-#193) — it does *not* go through ---! this function. For sv-element ordering, use the typed ---! `eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry)` overload ---! (or the `<` / `<=` / `>` / `>=` operators on the same pair). ---! ---! @param a eql_v2_encrypted First encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @param b eql_v2_encrypted Second encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either value lacks an `ob` (Block ORE) term ---! ---! @see eql_v2.compare_ore_block_u64_8_256 ---! @see eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry) ---! @see eql_v2."=" -- hm-only equality, post-#193 inlining -CREATE FUNCTION eql_v2.compare(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF eql_v2.has_ore_block_u64_8_256(a) AND eql_v2.has_ore_block_u64_8_256(b) THEN - RETURN eql_v2.compare_ore_block_u64_8_256(a, b); - END IF; - - RAISE EXCEPTION - 'eql_v2.compare requires Block ORE (`ob`) on both root operands. For sv-element ordering, extract entries via `col -> ''<selector>''` and use eql_v2.compare on the resulting `eql_v2.ste_vec_entry` values (or their `<` / `<=` / `>` / `>=` operators). Equality is hmac-only via the `=` operator — this function is for ordering only.' - USING ERRCODE = 'feature_not_supported'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Three-way ordering on `eql_v2.ste_vec_entry` ---! ---! CLLW ORE three-way comparator on ste-vec entries. Returns `-1` / `0` / ---! `1` by extracting the `oc` term from each entry and delegating to ---! `eql_v2.compare_ore_cllw_term`. Use this when you need an `int` ordering ---! out of two extracted ste-vec entries — for the boolean-form operators ---! (`<` / `<=` / `>` / `>=`) on the same pair, see ---! `src/operators/ste_vec_entry.sql`. ---! ---! Note: the caller is responsible for extracting an `eql_v2.ste_vec_entry` ---! first; the `(eql_v2_encrypted, text)` form would be a natural extension ---! but is deliberately *not* added here so that callers stay aware of the ---! two-step shape (extract via `->`, then compare). ---! ---! @param a eql_v2.ste_vec_entry First entry ---! @param b eql_v2.ste_vec_entry Second entry ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either entry lacks an `oc` term ---! ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.compare(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF NOT (eql_v2.has_ore_cllw(a) AND eql_v2.has_ore_cllw(b)) THEN - RAISE EXCEPTION - 'eql_v2.compare(ste_vec_entry, ste_vec_entry) requires `oc` (CLLW ORE) on both entries.' - USING ERRCODE = 'feature_not_supported'; - END IF; - - RETURN eql_v2.compare_ore_cllw_term(eql_v2.ore_cllw(a), eql_v2.ore_cllw(b)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract ORE index term for ordering encrypted values ---! ---! Helper function that extracts the ore_block_u64_8_256 index term from an encrypted value ---! for use in ORDER BY clauses when comparison operators are not appropriate or available. ---! ---! @param eql_v2_encrypted Encrypted value to extract order term from ---! @return eql_v2.ore_block_u64_8_256 ORE index term for ordering ---! ---! @example ---! -- Order encrypted values without using comparison operators ---! SELECT * FROM users ORDER BY eql_v2.order_by(encrypted_age); ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.order_by(a eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(a); - END; -$$ LANGUAGE plpgsql; - ---! @brief Equality operator for ORE block types ---! @internal ---! ---! Implements the = operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_eq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 0 -$$; - - - ---! @brief Not equal operator for ORE block types ---! @internal ---! ---! Implements the <> operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are not equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_neq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) <> 0 -$$; - - - ---! @brief Less than operator for ORE block types ---! @internal ---! ---! Implements the < operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = -1 -$$; - - - ---! @brief Less than or equal operator for ORE block types ---! @internal ---! ---! Implements the <= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != 1 -$$; - - - ---! @brief Greater than operator for ORE block types ---! @internal ---! ---! Implements the > operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 1 -$$; - - - ---! @brief Greater than or equal operator for ORE block types ---! @internal ---! ---! Implements the >= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != -1 -$$; - - - ---! @brief = operator for ORE block types -CREATE OPERATOR = ( - FUNCTION=eql_v2.ore_block_u64_8_256_eq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - - ---! @brief <> operator for ORE block types -CREATE OPERATOR <> ( - FUNCTION=eql_v2.ore_block_u64_8_256_neq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief > operator for ORE block types -CREATE OPERATOR > ( - FUNCTION=eql_v2.ore_block_u64_8_256_gt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - - ---! @brief < operator for ORE block types -CREATE OPERATOR < ( - FUNCTION=eql_v2.ore_block_u64_8_256_lt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - - ---! @brief <= operator for ORE block types -CREATE OPERATOR <= ( - FUNCTION=eql_v2.ore_block_u64_8_256_lte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - - ---! @brief >= operator for ORE block types -CREATE OPERATOR >= ( - FUNCTION=eql_v2.ore_block_u64_8_256_gte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief Contains operator for encrypted values (@>) ---! ---! Implements the @> (contains) operator for testing if left encrypted value ---! contains the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2_encrypted Right operand (contained value) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Check if encrypted array contains value ---! SELECT * FROM documents ---! WHERE encrypted_tags @> '["security"]'::jsonb::eql_v2_encrypted; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.add_search_config --- Marked IMMUTABLE STRICT PARALLEL SAFE so the planner inlines the body --- and a functional GIN index on `eql_v2.ste_vec(col)` can match --- `WHERE col @> val`. The previous default-VOLATILE declaration prevented --- inlining and forced seq scan even on Supabase installs that have the --- ste_vec functional index in place. -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ste_vec_contains(a, b) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contains operator (@>) with an `eql_v2.stevec_query` needle ---! ---! Type-safe containment for the recommended recipe: the right-hand ---! side is an `stevec_query` (sv-shaped payload, no `c` fields). The ---! body inlines to a native `jsonb @>` over `eql_v2.to_stevec_query(a)::jsonb`, ---! so the planner can match a functional GIN index built on the same ---! expression — engaging Bitmap Index Scan for bare-form containment ---! across both `hm`-bearing and `oc`-bearing selectors with a single ---! index. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.stevec_query Right operand (query payload) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Functional GIN index (covers all selectors, hm and oc): ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! -- Bare-form predicate engages the index: ---! SELECT * FROM users ---! WHERE encrypted_doc @> '{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2.to_stevec_query -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.stevec_query) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Single-expression body so the planner can inline. The haystack - -- normalisation happens in `to_stevec_query`; the needle is trusted - -- to be clean (sv elements of shape `{s, hm-or-oc}` — the documented - -- stevec_query contract). For untrusted needles, callers should - -- normalise via the json-shape `{"sv":[{"s":"<sel>","hm":"<term>"}]}`. - SELECT eql_v2.to_stevec_query(a)::jsonb @> b::jsonb -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.stevec_query -); - - ---! @brief Contains operator (@>) with an `eql_v2.ste_vec_entry` needle ---! ---! Convenience overload for the common pattern "does this encrypted ---! payload include this specific sv entry?". Wraps the entry into a ---! single-element sv array (stripping `c`) and reduces to the same ---! `to_stevec_query(a)::jsonb @> needle::jsonb` form as the ---! `stevec_query` overload — so it engages the same functional GIN ---! index. Inlinable. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.ste_vec_entry Right operand (single entry) ---! @return Boolean True if a contains an sv entry matching `b` ---! ---! @example ---! -- Does this row's encrypted doc contain the same name as this other doc? ---! SELECT a.* FROM docs a, docs b ---! WHERE a.doc @> (b.doc -> '<name-sel>'); ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.ste_vec_entry) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.to_stevec_query(a)::jsonb - @> jsonb_build_object( - 'sv', - jsonb_build_array( - jsonb_strip_nulls( - jsonb_build_object( - 's', b -> 's', - 'hm', b -> 'hm', - 'oc', b -> 'oc' - ) - ) - ) - ) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.ste_vec_entry -); - ---! @file config/tables.sql ---! @brief Encryption configuration storage table ---! ---! Defines the main table for storing EQL v2 encryption configurations. ---! Each row represents a configuration specifying which tables/columns to encrypt ---! and what index types to use. Configurations progress through lifecycle states. ---! ---! @see config/types.sql for state ENUM definition ---! @see config/indexes.sql for state uniqueness constraints ---! @see config/constraints.sql for data validation - - ---! @brief Encryption configuration table ---! ---! Stores encryption configurations with their state and metadata. ---! The 'data' JSONB column contains the full configuration structure including ---! table/column mappings, index types, and casting rules. ---! ---! @note Only one configuration can be 'active', 'pending', or 'encrypting' at once ---! @note 'id' is auto-generated identity column ---! @note 'state' defaults to 'pending' for new configurations ---! @note 'data' validated by CHECK constraint (see config/constraints.sql) -CREATE TABLE IF NOT EXISTS public.eql_v2_configuration -( - id bigint GENERATED ALWAYS AS IDENTITY, - state eql_v2_configuration_state NOT NULL DEFAULT 'pending', - data jsonb, - created_at timestamptz not null default current_timestamp, - PRIMARY KEY(id) -); - - ---! @brief Initialize default configuration structure ---! @internal ---! ---! Creates a default configuration object if input is NULL. Used internally ---! by public configuration functions to ensure consistent structure. ---! ---! @param config JSONB Existing configuration or NULL ---! @return JSONB Configuration with default structure (version 1, empty tables) -CREATE FUNCTION eql_v2.config_default(config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF config IS NULL THEN - SELECT jsonb_build_object('v', 1, 'tables', jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add table to configuration if not present ---! @internal ---! ---! Ensures the specified table exists in the configuration structure. ---! Creates empty table entry if needed. Idempotent operation. ---! ---! @param table_name Text Name of table to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with table entry -CREATE FUNCTION eql_v2.config_add_table(table_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - tbl jsonb; - BEGIN - IF NOT config #> array['tables'] ? table_name THEN - SELECT jsonb_insert(config, array['tables', table_name], jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add column to table configuration if not present ---! @internal ---! ---! Ensures the specified column exists in the table's configuration structure. ---! Creates empty column entry with indexes object if needed. Idempotent operation. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with column entry -CREATE FUNCTION eql_v2.config_add_column(table_name text, column_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - col jsonb; - BEGIN - IF NOT config #> array['tables', table_name] ? column_name THEN - SELECT jsonb_build_object('indexes', jsonb_build_object()) into col; - SELECT jsonb_set(config, array['tables', table_name, column_name], col) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Set cast type for column in configuration ---! @internal ---! ---! Updates the cast_as field for a column, specifying the PostgreSQL type ---! that decrypted values should be cast to. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param cast_as Text PostgreSQL type for casting (e.g., 'text', 'int', 'jsonb') ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with cast_as set -CREATE FUNCTION eql_v2.config_add_cast(table_name text, column_name text, cast_as text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_set(config, array['tables', table_name, column_name, 'cast_as'], to_jsonb(cast_as)) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add search index to column configuration ---! @internal ---! ---! Inserts a search index entry (unique, match, ore, ste_vec) with its options ---! into the column's indexes object. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param index_name Text Type of index to add ---! @param opts JSONB Index-specific options ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with index added -CREATE FUNCTION eql_v2.config_add_index(table_name text, column_name text, index_name text, opts jsonb, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_insert(config, array['tables', table_name, column_name, 'indexes', index_name], opts) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Generate default options for match index ---! @internal ---! ---! Returns default configuration for match (LIKE) indexes: k=6, bf=2048, ---! ngram tokenizer with token_length=3, downcase filter, include_original=true. ---! ---! @return JSONB Default match index options -CREATE FUNCTION eql_v2.config_match_default() - RETURNS jsonb -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_build_object( - 'k', 6, - 'bf', 2048, - 'include_original', true, - 'tokenizer', json_build_object('kind', 'ngram', 'token_length', 3), - 'token_filters', json_build_array(json_build_object('kind', 'downcase'))); -END; --- AUTOMATICALLY GENERATED FILE --- Source is version-template.sql - -DROP FUNCTION IF EXISTS eql_v2.version(); - ---! @file version.sql ---! @brief EQL version reporting ---! ---! This file is auto-generated from version.template during build. ---! The version string placeholder is replaced with the actual release version. - ---! @brief Get EQL library version string ---! ---! Returns the version string for the installed EQL library. ---! This value is set at build time from the project version. ---! ---! @return text Version string (e.g., "2.1.0" or "DEV" for development builds) ---! ---! @note Auto-generated during build from version.template ---! ---! @example ---! -- Check installed EQL version ---! SELECT eql_v2.version(); ---! -- Returns: '2.1.0' -CREATE FUNCTION eql_v2.version() - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT 'eql-2.3.1'; -$$ LANGUAGE SQL; - - ---! @file src/ore_cllw/operators.sql ---! @brief Comparison operators on the `eql_v2.ore_cllw` composite type ---! ---! Same-type comparison operators backing the btree operator class on the ---! composite `eql_v2.ore_cllw` type. Each operator reduces to a single SELECT ---! over `eql_v2.compare_ore_cllw_term(a, b)`, which is the canonical CLLW ---! per-byte comparator (`y + 1 == x` mod 256). The operator wrappers are ---! inlinable `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` so the planner can ---! fold them into the calling query — that's what lets a functional btree ---! index on `eql_v2.ore_cllw(col)` engage for both `WHERE eql_v2.ore_cllw(col) ---! < eql_v2.ore_cllw($1)` and `ORDER BY eql_v2.ore_cllw(col)` shapes. ---! ---! The inner `eql_v2.compare_ore_cllw_term` is `LANGUAGE plpgsql` (it has a ---! per-byte loop) and is NOT inlined. That's fine for index *match* (the ---! planner only needs the outer operator function call to fold so the ---! predicate's expression tree matches the index's expression tree); only the ---! per-comparison cost is the plpgsql call overhead. That's the cost the ---! functional index avoids by walking the btree in order rather than calling ---! compare on every row. ---! ---! @note Deliberately no `HASHES` / `MERGES` flags on the operator ---! declarations. HASHES requires a registered hash function on the type ---! (the CLLW protocol gives ordering, not a sensible hashing); MERGES ---! requires an equivalent merge-joinable operator class on both sides. ---! ---! @see src/ore_cllw/operator_class.sql ---! @see src/ore_cllw/functions.sql - ---! @brief Equality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare equal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_eq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 0 -$$; - ---! @brief Inequality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare unequal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_neq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 0 -$$; - ---! @brief Less-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = -1 -$$; - ---! @brief Less-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 1 -$$; - ---! @brief Greater-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 1 -$$; - ---! @brief Greater-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> -1 -$$; - - -CREATE OPERATOR = ( - FUNCTION = eql_v2.ore_cllw_eq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel -); - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.ore_cllw_neq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - -CREATE OPERATOR < ( - FUNCTION = eql_v2.ore_cllw_lt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.ore_cllw_lte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - -CREATE OPERATOR > ( - FUNCTION = eql_v2.ore_cllw_gt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.ore_cllw_gte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - - ---! @brief Compare two encrypted values using ORE block index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their ORE block index terms. Used internally by range operators (<, <=, >, >=) ---! for order-revealing comparisons without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Uses ORE cryptographic protocol for secure comparisons ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2."<" ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.ore_block_u64_8_256; - b_term eql_v2.ore_block_u64_8_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - a_term := eql_v2.ore_block_u64_8_256(a); - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - b_term := eql_v2.ore_block_u64_8_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a_term.terms, b_term.terms); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Cast text to ORE block term ---! @internal ---! ---! Converts text to bytea and wraps in ore_block_u64_8_256_term type. ---! Used internally for ORE block extraction and manipulation. ---! ---! @param t Text Text value to convert ---! @return eql_v2.ore_block_u64_8_256_term ORE term containing bytea representation ---! ---! @see eql_v2.ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(t text) - RETURNS eql_v2.ore_block_u64_8_256_term - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN t::bytea; -END; - ---! @brief Implicit cast from text to ORE block term ---! ---! Defines an implicit cast allowing automatic conversion of text values ---! to ore_block_u64_8_256_term type for ORE operations. ---! ---! @see eql_v2.text_to_ore_block_u64_8_256_term -CREATE CAST (text AS eql_v2.ore_block_u64_8_256_term) - WITH FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(text) AS IMPLICIT; - ---! @brief Pattern matching helper using bloom filters ---! @internal ---! ---! Internal helper for LIKE-style pattern matching on encrypted values. ---! Uses bloom filter index terms to test substring containment without decryption. ---! Requires 'match' index configuration on the column. ---! ---! Marked IMMUTABLE so the planner inlines the body and a functional index on ---! `eql_v2.bloom_filter(col)` can match `WHERE eql_v2.like(col, val)`. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @see eql_v2."~~" ---! @see eql_v2.bloom_filter ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.like(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief Case-insensitive pattern matching helper ---! @internal ---! ---! Internal helper for ILIKE-style case-insensitive pattern matching. ---! Case sensitivity is controlled by index configuration (token_filters with downcase). ---! This function has same implementation as like() - actual case handling is in index terms. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @note Case sensitivity depends on match index token_filters configuration ---! @see eql_v2."~~" ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.ilike(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief LIKE operator for encrypted values (pattern matching) ---! ---! Implements the ~~ (LIKE) operator for substring/pattern matching on encrypted ---! text using bloom filter index terms. Enables WHERE col LIKE '%pattern%' queries ---! without decryption. Requires 'match' index configuration on the column. ---! ---! Pattern matching uses n-gram tokenization configured in match index. Token length ---! and filters affect matching behavior. ---! ---! @param a eql_v2_encrypted Haystack (encrypted text to search in) ---! @param b eql_v2_encrypted Needle (encrypted pattern to search for) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! -- Search for substring in encrypted email ---! SELECT * FROM users ---! WHERE encrypted_email ~~ '%@example.com%'::text::eql_v2_encrypted; ---! ---! -- Pattern matching on encrypted names ---! SELECT * FROM customers ---! WHERE encrypted_name ~~ 'John%'::text::eql_v2_encrypted; ---! ---! @brief SQL LIKE operator (~~ operator) for encrypted text pattern matching ---! ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b eql_v2_encrypted Right operand (encrypted pattern) ---! @return boolean True if pattern matches ---! ---! @note Requires match index: eql_v2.add_search_config(table, column, 'match') ---! @see eql_v2.like ---! @see eql_v2.add_search_config --- Inlinable: delegates to `eql_v2.like` which is itself an inlinable --- single-statement SQL function. Two levels of inlining produce --- `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)`, which matches a --- functional GIN index built on `eql_v2.bloom_filter(col)`. PostgREST --- and ORM `~~`/`~~*` queries engage the bloom-filter index without --- the caller wrapping the column themselves. -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b) -$$; - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Case-insensitive LIKE operator (~~*) ---! ---! Implements ~~* (ILIKE) operator for case-insensitive pattern matching. ---! Case handling depends on match index token_filters configuration (use downcase filter). ---! Same implementation as ~~, with case sensitivity controlled by index configuration. ---! ---! @param a eql_v2_encrypted Haystack ---! @param b eql_v2_encrypted Needle ---! @return Boolean True if a contains b (case-insensitive) ---! ---! @note Configure match index with downcase token filter for case-insensitivity ---! @see eql_v2."~~" -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for encrypted value and JSONB ---! ---! Overload of ~~ operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param eql_v2_encrypted Haystack (encrypted value) ---! @param b JSONB Needle (will be cast to eql_v2_encrypted) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE encrypted_email ~~ '%gmail%'::jsonb; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b::eql_v2_encrypted) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for JSONB and encrypted value ---! ---! Overload of ~~ operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param a JSONB Haystack (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Needle (encrypted pattern) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE 'test@example.com'::jsonb ~~ encrypted_pattern; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a::eql_v2_encrypted, b) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - --- ----------------------------------------------------------------------------- - ---! @file src/operators/ste_vec_entry.sql ---! @brief Comparison operators on `eql_v2.ste_vec_entry` ---! ---! Equality (`=`, `<>`) reduces to `eq_term(a) = eq_term(b)` — a bytea ---! comparison of `coalesce(hm, oc)`. Ordering (`<`, `<=`, `>`, `>=`) ---! reduces to `ore_cllw(a) <op> ore_cllw(b)`. Each backing function is ---! inlinable single-statement SQL, so the planner can fold the ---! operator body into the calling query — `WHERE col -> 'sel' = $1` ---! and `WHERE col -> 'sel' < $1` therefore match functional indexes ---! built on `eql_v2.eq_term(col -> 'sel')` / ---! `eql_v2.ore_cllw(col -> 'sel')` without per-query rewriting. ---! ---! XOR contract. Each sv entry carries exactly one of `hm` (bool ---! leaves, array / object roots) or `oc` (string / number leaves) — ---! enforced by the `ste_vec_entry` DOMAIN CHECK. Equality coalesces ---! across both protocols because both are deterministic and the byte ---! distributions are disjoint; ordering strictly uses `ore_cllw` ---! (range on hm-only entries is meaningless and produces silent NULL, ---! which the lint subsystem `src/lint/lints.sql` flags as a ---! configuration error). ---! ---! Same convention as the `eql_v2_encrypted` operators (#193 / #211): the ---! operator-class function-matching layer is what makes index match work ---! structurally, the backing functions just need to inline cleanly through ---! to the extractor calls. ---! ---! @see eql_v2.eq_term(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/=.sql ---! @see src/operators/<.sql - ---! @brief Equality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if both entries share the same deterministic ---! equality term (hm-or-oc, via `eq_term`). -CREATE FUNCTION eql_v2.eq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) = eql_v2.eq_term(b) -$$; - -CREATE OPERATOR = ( - FUNCTION = eql_v2.eq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief Inequality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if the entries' equality terms (hm-or-oc, via ---! `eq_term`) differ. -CREATE FUNCTION eql_v2.neq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) <> eql_v2.eq_term(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.neq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - - ---! @brief Less-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before `b`'s -CREATE FUNCTION eql_v2.lt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) < eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR < ( - FUNCTION = eql_v2.lt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - ---! @brief Less-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before or equal to `b`'s -CREATE FUNCTION eql_v2.lte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) <= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.lte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - ---! @brief Greater-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after `b`'s -CREATE FUNCTION eql_v2.gt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) > eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR > ( - FUNCTION = eql_v2.gt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - ---! @brief Greater-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after or equal to `b`'s -CREATE FUNCTION eql_v2.gte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) >= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.gte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @file operators/sort.sql ---! @brief Comparison-based sorting functions for encrypted values without operator classes ---! ---! Provides O(n log n) quicksort-based sorting using eql_v2.compare() for environments ---! where btree operator classes are unavailable (e.g., Supabase). This is significantly ---! faster than the O(n^2) correlated subquery workaround. ---! ---! When all input rows share an ORE term (`ob`) the sort path pre-extracts the ---! ORE order key once per row and compares those keys directly. Rows lacking ---! an ORE term entirely fall back to `eql_v2.compare()` per pair. - - ---! @internal ---! @brief Compare pre-extracted ORE order keys with encrypted NULL semantics ---! ---! Mirrors eql_v2.compare() for NULL handling, then delegates to the ---! ore_block_u64_8_256 comparator when both keys are present. ---! ---! @param a eql_v2.ore_block_u64_8_256 First order key ---! @param b eql_v2.ore_block_u64_8_256 Second order key ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b -CREATE FUNCTION eql_v2._compare_order_key( - a eql_v2.ore_block_u64_8_256, - b eql_v2.ore_block_u64_8_256 -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a, b); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare two elements from aligned arrays using the selected sort strategy ---! ---! @param vals eql_v2_encrypted[] Encrypted values (used when strategy = 'compare') ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (strategy = 'ore') ---! @param left_idx integer Index of the left element ---! @param right_idx integer Index of the right element ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if left < right, 0 if equal, 1 if left > right -CREATE FUNCTION eql_v2._compare_sort_elements( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - left_idx integer, - right_idx integer, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[left_idx], ore_keys[right_idx]); - END IF; - - RETURN eql_v2.compare(vals[left_idx], vals[right_idx]); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare an array element against a captured pivot using the selected strategy ---! ---! @param vals eql_v2_encrypted[] Array of encrypted values ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys ---! @param idx integer Index of the element to compare ---! @param pivot_val eql_v2_encrypted Pivot encrypted value (strategy = 'compare') ---! @param pivot_ore_key eql_v2.ore_block_u64_8_256 Pivot ORE key (strategy = 'ore') ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if element < pivot, 0 if equal, 1 if element > pivot -CREATE FUNCTION eql_v2._compare_sort_pivot( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - idx integer, - pivot_val eql_v2_encrypted, - pivot_ore_key eql_v2.ore_block_u64_8_256, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[idx], pivot_ore_key); - END IF; - - RETURN eql_v2.compare(vals[idx], pivot_val); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place insertion sort on parallel id/value/key arrays ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._insertion_sort( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - i integer; - j integer; - key_id bigint; - key_val eql_v2_encrypted; - sort_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - IF lo >= hi THEN - RETURN; - END IF; - - FOR i IN lo + 1..hi LOOP - key_id := ids[i]; - key_val := vals[i]; - sort_ore_key := ore_keys[i]; - j := i - 1; - - WHILE j >= lo LOOP - EXIT WHEN strategy = 'compare' - AND eql_v2.compare(vals[j], key_val) <= 0; - EXIT WHEN strategy = 'ore' - AND eql_v2._compare_order_key(ore_keys[j], sort_ore_key) <= 0; - - ids[j + 1] := ids[j]; - vals[j + 1] := vals[j]; - ore_keys[j + 1] := ore_keys[j]; - j := j - 1; - END LOOP; - - ids[j + 1] := key_id; - vals[j + 1] := key_val; - ore_keys[j + 1] := sort_ore_key; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place quicksort on parallel id/value/key arrays ---! ---! Sorts aligned arrays simultaneously using Hoare partition with median-of-three pivot ---! selection. The median-of-three strategy avoids O(n^2) degradation on already-sorted ---! input, which is common with sequential test data. ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values to compare (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._quicksort_sorter( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - insertion_threshold CONSTANT integer := 16; - pivot_val eql_v2_encrypted; - pivot_ore_key eql_v2.ore_block_u64_8_256; - mid integer; - i integer; - j integer; - left_hi integer; - right_lo integer; - tmp_id bigint; - tmp_val eql_v2_encrypted; - tmp_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - WHILE lo < hi LOOP - IF hi - lo <= insertion_threshold THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._insertion_sort(ids, vals, ore_keys, lo, hi, strategy) q; - RETURN; - END IF; - - -- Median-of-three pivot selection: sort lo, mid, hi then use mid as pivot - mid := lo + (hi - lo) / 2; - - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, mid, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[mid]; ids[mid] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[mid]; vals[mid] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[mid]; ore_keys[mid] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, hi, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, mid, hi, strategy) > 0 THEN - tmp_id := ids[mid]; ids[mid] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[mid]; vals[mid] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[mid]; ore_keys[mid] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - - pivot_val := vals[mid]; - pivot_ore_key := ore_keys[mid]; - i := lo; - j := hi; - - LOOP - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, i, - pivot_val, pivot_ore_key, strategy - ) < 0 LOOP - i := i + 1; - END LOOP; - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, j, - pivot_val, pivot_ore_key, strategy - ) > 0 LOOP - j := j - 1; - END LOOP; - - EXIT WHEN i >= j; - - tmp_id := ids[i]; ids[i] := ids[j]; ids[j] := tmp_id; - tmp_val := vals[i]; vals[i] := vals[j]; vals[j] := tmp_val; - tmp_ore_key := ore_keys[i]; ore_keys[i] := ore_keys[j]; ore_keys[j] := tmp_ore_key; - - i := i + 1; - j := j - 1; - END LOOP; - - left_hi := j; - right_lo := j + 1; - - IF left_hi - lo < hi - right_lo THEN - IF lo < left_hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, lo, left_hi, strategy) q; - END IF; - lo := right_lo; - ELSE - IF right_lo < hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, right_lo, hi, strategy) q; - END IF; - hi := left_hi; - END IF; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Emit aligned arrays as rows in ASC or DESC order ---! ---! @param ids bigint[] Array of sorted row identifiers ---! @param vals eql_v2_encrypted[] Array of sorted encrypted values ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Rows emitted in the requested order -CREATE FUNCTION eql_v2._emit_sorted_rows( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - i integer; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - IF upper(direction) = 'DESC' THEN - FOR i IN REVERSE n..1 LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - ELSE - FOR i IN 1..n LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Sort encrypted values using precomputed ORE keys when available ---! ---! Shared implementation for public sorting entrypoints. The `strategy` ---! parameter selects the comparison path: `'ore'` uses the aligned `ore_keys` ---! array; `'compare'` falls back to `eql_v2.compare()` on the encrypted values ---! directly. ---! ---! @param ids bigint[] Row identifiers aligned with `vals` ---! @param vals eql_v2_encrypted[] Encrypted values to sort ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (used when strategy = 'ore') ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param strategy text One of 'ore' or 'compare' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows -CREATE FUNCTION eql_v2._sort_compare_precomputed( - ids bigint[], - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - direction text DEFAULT 'ASC', - strategy text DEFAULT 'ore' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - m integer; - k integer; - sorted_ids bigint[]; - sorted_vals eql_v2_encrypted[]; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; -BEGIN - n := coalesce(array_length(ids, 1), 0); - m := coalesce(array_length(vals, 1), 0); - - IF n <> m THEN - RAISE EXCEPTION 'ids and vals must have the same length'; - END IF; - - IF strategy = 'ore' THEN - k := coalesce(array_length(ore_keys, 1), 0); - IF n <> k THEN - RAISE EXCEPTION 'ids and ore_keys must have the same length when strategy = ''ore'''; - END IF; - END IF; - - IF n = 0 THEN - RETURN; - END IF; - - IF n = 1 THEN - id := ids[1]; - val := vals[1]; - RETURN NEXT; - RETURN; - END IF; - - SELECT q.ids, q.vals, q.ore_keys - INTO sorted_ids, sorted_vals, sorted_ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, 1, n, strategy) q; - - RETURN QUERY - SELECT emitted.id, emitted.val - FROM eql_v2._emit_sorted_rows(sorted_ids, sorted_vals, direction) emitted; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values using comparison-based quicksort ---! ---! Sorts parallel arrays of identifiers and encrypted values using O(n log n) ---! quicksort with eql_v2.compare(). Returns sorted rows as a table, avoiding ---! the need for unnest() or other array manipulation by callers. ---! ---! When all input rows share an `ore` term the sort uses pre-extracted ORE ---! keys; otherwise it falls back to `eql_v2.compare()` per pair. ---! ---! This function is designed for environments without operator classes (e.g., Supabase) ---! where direct ORDER BY on encrypted columns is not available. ---! ---! @param ids bigint[] Array of row identifiers ---! @param vals eql_v2_encrypted[] Array of encrypted values (must be same length as ids) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @example ---! -- Sort all rows from an encrypted table ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore), ---! 'ASC' ---! ); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore WHERE id > 42), ---! (SELECT array_agg(e ORDER BY id) FROM ore WHERE id > 42), ---! 'DESC' ---! ); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore) ---! ) LIMIT 5; ---! ---! @see eql_v2.compare ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; - i integer; - use_ore boolean := true; - strategy text; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - FOR i IN 1..n LOOP - IF vals[i] IS NULL THEN - sorted_ore_keys[i] := NULL; - ELSE - IF use_ore THEN - IF eql_v2.has_ore_block_u64_8_256(vals[i]) THEN - sorted_ore_keys[i] := eql_v2.order_by(vals[i]); - ELSE - use_ore := false; - END IF; - END IF; - - EXIT WHEN NOT use_ore; - END IF; - END LOOP; - - IF use_ore THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - ids, vals, sorted_ore_keys, direction, strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a table using column and table references ---! ---! Convenience overload that accepts column names, a table name, and an optional ---! filter clause instead of pre-aggregated arrays. Internally constructs the ---! query and delegates to eql_v2.order_by_compare(). ---! ---! @param id_column text Name of the bigint identifier column ---! @param val_column text Name of the eql_v2_encrypted value column ---! @param tbl text Table name (may be schema-qualified) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param filter text Optional WHERE clause (without the WHERE keyword) ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note The id column must be castable to bigint. Uses dynamic SQL internally. ---! @warning The filter parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ascending (default) ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore'); ---! ---! -- Sort descending ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'DESC'); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'ASC', 'id > 42'); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore') LIMIT 10; ---! ---! @see eql_v2.sort_compare(bigint[], eql_v2_encrypted[], text) ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - id_column text, - val_column text, - tbl text, - direction text DEFAULT 'ASC', - filter text DEFAULT NULL -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - query text; - resolved_tbl regclass; -BEGIN - resolved_tbl := to_regclass(tbl); - - IF resolved_tbl IS NULL THEN - RAISE EXCEPTION 'table "%" does not exist', tbl; - END IF; - - query := format('SELECT %I, %I FROM %s', id_column, val_column, resolved_tbl); - - IF filter IS NOT NULL THEN - query := query || ' WHERE ' || filter; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2.order_by_compare(query, direction) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a query using comparison-based quicksort ---! ---! Convenience wrapper that accepts a SQL query string, executes it, collects the ---! results, and returns them sorted. For ORE-backed values this pre-extracts the ---! order key once per row and sorts on that key; other inputs fall back to ---! eql_v2.compare(). The query must return exactly two columns: a bigint ---! identifier and an eql_v2_encrypted value. ---! ---! @param query text SQL query returning (bigint, eql_v2_encrypted) columns ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note Uses dynamic SQL (EXECUTE) so cannot be IMMUTABLE or PARALLEL SAFE ---! @warning The query parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ---! SELECT * FROM eql_v2.order_by_compare('SELECT id, e FROM ore'); ---! ---! -- Sort with WHERE clause ---! SELECT * FROM eql_v2.order_by_compare( ---! 'SELECT id, e FROM ore WHERE id > 42', ---! 'DESC' ---! ); ---! ---! @see eql_v2.sort_compare ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.order_by_compare( - query text, - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - all_ids bigint[]; - all_vals eql_v2_encrypted[]; - all_ore_keys eql_v2.ore_block_u64_8_256[]; - all_have_ore_keys boolean; - strategy text; -BEGIN - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - EXECUTE format( - 'WITH input_rows AS ( - SELECT row_number() OVER () AS ord, - sub.id, - sub.val, - CASE - WHEN sub.val IS NULL THEN NULL - WHEN eql_v2.has_ore_block_u64_8_256(sub.val) THEN eql_v2.order_by(sub.val) - ELSE NULL - END AS ore_key, - CASE - WHEN sub.val IS NULL THEN TRUE - ELSE eql_v2.has_ore_block_u64_8_256(sub.val) - END AS has_ore_key - FROM (%s) sub(id, val) - ) - SELECT array_agg(id ORDER BY ord), - array_agg(val ORDER BY ord), - array_agg(ore_key ORDER BY ord), - coalesce(bool_and(has_ore_key), TRUE) - FROM input_rows', - query - ) INTO all_ids, all_vals, all_ore_keys, all_have_ore_keys; - - IF all_ids IS NULL THEN - RETURN; - END IF; - - IF all_have_ore_keys THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - all_ids, - all_vals, - all_ore_keys, - direction, - strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `>=` testing. ---! The `>=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a >= b (compare result >= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2.">=" -CREATE FUNCTION eql_v2.gte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) >= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than-or-equal operator for encrypted values ---! ---! Implements the >= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a >= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age >= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = jsonb, - RIGHTARG =eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief Greater-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for greater-than ---! testing. The `>` operator wrappers no longer go through this helper — ---! see the inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a > b (compare result = 1) ---! ---! @see eql_v2.compare ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.gt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = 1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than operator for encrypted values ---! ---! Implements the > operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is greater than b ---! ---! @example ---! SELECT * FROM events ---! WHERE encrypted_value > '100'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. Predicate --- `WHERE col > val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) > eql_v2.ore_block_u64_8_256(val)` --- and matches a functional ORE index built on the same expression. --- Breaking impact: columns with only `ore_cllw_*` or OPE terms now --- raise from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where they --- previously fell through `eql_v2.compare`. See U-005. -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION=eql_v2.">", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief Equality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `=` operator's body: reduces to ---! `hmac_256(a) = hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.eq`); equivalent to using the `=` operator ---! directly. ---! ---! Equality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `=` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms match ---! ---! @see eql_v2."=" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.eq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - ---! @brief Equality operator for encrypted values ---! ---! Implements the = operator for comparing two encrypted values using their ---! encrypted index terms (hmac_256). Enables WHERE clause comparisons ---! without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are equal ---! ---! @example ---! -- Compare encrypted columns ---! SELECT * FROM users WHERE encrypted_email = other_encrypted_email; ---! ---! -- Search using encrypted literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col = val` reduces to --- `WHERE eql_v2.hmac_256(col) = eql_v2.hmac_256(val)` and matches a --- functional hash index built on `eql_v2.hmac_256(col)`. Bare equality --- queries (including those issued by PostgREST and ORMs that don't --- wrap columns themselves) become fast on Supabase and any --- --exclude-operator-family install. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2.eq` walked `eql_v2.compare`, which fell back to ORE / Blake3 / --- literal comparison when HMAC wasn't present. Now `=` requires the --- column to have `equality` configured (i.e. carry an `hm` field). --- Calling `=` on an ORE-only column will return NULL where it --- previously returned a Boolean. This is intentional — it surfaces --- config errors loudly. See the predicate/extractor RFC for context. -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - ---! @brief Equality operator for encrypted value and JSONB ---! ---! Overload of = operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Useful for comparing ---! against JSONB literals or columns. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand (will be cast to eql_v2_encrypted) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare encrypted column to JSONB literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::jsonb; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Equality operator for JSONB and encrypted value ---! ---! Overload of = operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Enables commutative ---! equality comparisons. ---! ---! @param a JSONB Left operand (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare JSONB literal to encrypted column ---! SELECT * FROM users ---! WHERE '{"c":"...","i":{"unique":"..."}}'::jsonb = encrypted_email; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - ---! @brief Contained-by operator for encrypted values (<@) ---! ---! Implements the <@ (contained-by) operator for testing if left encrypted value ---! is contained by the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. Reverse of @> operator. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (contained value) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if a is contained by b ---! ---! @example ---! -- Check if value is contained in encrypted array ---! SELECT * FROM documents ---! WHERE '["security"]'::jsonb::eql_v2_encrypted <@ encrypted_tags; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.\"@>\" ---! @see eql_v2.add_search_config - --- Marked IMMUTABLE STRICT PARALLEL SAFE — see operators/@>.sql for rationale. -CREATE FUNCTION eql_v2."<@"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Contains with reversed arguments - SELECT eql_v2.ste_vec_contains(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.stevec_query` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.stevec_query)`. Mirrors the ---! typed needle convention: "is this query payload contained in that ---! encrypted document?". ---! ---! @param a eql_v2.stevec_query Left operand (query payload) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."<@"(a eql_v2.stevec_query, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.stevec_query, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.ste_vec_entry` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)`. Convenience ---! shape for "is this entry contained in that encrypted document?". ---! ---! @param a eql_v2.ste_vec_entry Left operand (single entry) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2."<@"(a eql_v2.ste_vec_entry, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.ste_vec_entry, - RIGHTARG=eql_v2_encrypted -); - ---! @brief Inequality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `<>` operator's body: reduces to ---! `hmac_256(a) <> hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.neq`); equivalent to using the `<>` operator ---! directly. ---! ---! Inequality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `<>` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms differ ---! ---! @see eql_v2."<>" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.neq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - ---! @brief Not-equal operator for encrypted values ---! ---! Implements the <> (not equal) operator for comparing encrypted values using their ---! encrypted index terms. Enables WHERE clause inequality comparisons without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are not equal ---! ---! @example ---! -- Find records with non-matching values ---! SELECT * FROM users ---! WHERE encrypted_email <> 'admin@example.com'::text::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2."=" --- Inlinable; mirrors `=` (see operators/=.sql for rationale). --- Returns NULL on ORE-only encrypted columns (no `hm` field) instead --- of falling back to a slower comparison path; surface the config --- error rather than hide it. -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for encrypted value and JSONB ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for JSONB and encrypted value ---! ---! @param jsonb Plain JSONB value ---! @param eql_v2_encrypted Encrypted value ---! @return boolean True if values are not equal ---! ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) <> eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - - - - ---! @brief Less-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `<=` testing. ---! The `<=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `<=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `<=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a <= b (compare result <= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2."<=" -CREATE FUNCTION eql_v2.lte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) <= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than-or-equal operator for encrypted values ---! ---! Implements the <= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a <= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age <= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for encrypted value and JSONB ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for JSONB and encrypted value ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief Less-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for less-than ---! testing. The `<` operator wrappers no longer call this helper — they ---! inline a direct `ore_block_u64_8_256` comparison instead (see the ---! inlinable bodies below). ---! ---! @warning Behaviour now diverges from the `<` operator: this helper ---! still walks `eql_v2.compare`'s priority list (ore_block → ore_cllw ---! → hm), whereas `<` goes straight to `ore_block_u64_8_256` and raises ---! on missing `ob`. Callers relying on the dispatcher fallback should ---! migrate to the extractor form: `eql_v2.ore_cllw(col) < ---! eql_v2.ore_cllw($1::jsonb)`. See U-005. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a < b (compare result = -1) ---! ---! @see eql_v2.compare ---! @see eql_v2."<" -CREATE FUNCTION eql_v2.lt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = -1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than operator for encrypted values ---! ---! Implements the < operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term (configured ---! via the `ore` index in the EQL schema). ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is less than b ---! ---! @example ---! -- Range query on encrypted timestamps ---! SELECT * FROM events ---! WHERE encrypted_timestamp < '2024-01-01'::timestamp::text::eql_v2_encrypted; ---! ---! -- Compare encrypted numeric columns ---! SELECT * FROM products WHERE encrypted_price < encrypted_discount_price; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col < val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) < eql_v2.ore_block_u64_8_256(val)` --- and matches a functional btree index built on --- `eql_v2.ore_block_u64_8_256(col)` (using the DEFAULT --- `eql_v2.ore_block_u64_8_256_operator_class`). Bare range queries --- (`WHERE col < $1`) engage the functional ORE index on Supabase and any --- install that doesn't ship `eql_v2.encrypted_operator_class`. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2."<"` walked `eql_v2.compare`, which dispatched through --- ore_block / ore_cllw_u64 / ore_cllw_var / ope. Now `<` requires the --- column to have `ore_block_u64_8_256` configured (i.e. carry an `ob` --- field). Calling `<` on a column with only `ore_cllw_*` or OPE terms --- now raises from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where it --- previously returned a Boolean. Loud failure surfaces config errors --- rather than silently producing zero rows — see U-005. -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for encrypted value and JSONB ---! ---! Overload of < operator accepting JSONB on the right side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides; the jsonb ---! extractor `eql_v2.ore_block_u64_8_256(jsonb)` reads `b->'ob'` directly. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE encrypted_age < '{"ob":[...]}'::jsonb; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for JSONB and encrypted value ---! ---! Overload of < operator accepting JSONB on the left side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides. ---! ---! @param a JSONB Left operand ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE '{"ob":[...]}'::jsonb < encrypted_date; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief JSONB field accessor operator alias (->>) ---! ---! Implements the ->> operator as an alias of -> for encrypted JSONB data. This mirrors ---! PostgreSQL semantics where ->> returns text via implicit casts. The underlying ---! implementation delegates to eql_v2."->" and allows PostgreSQL to coerce the result. ---! ---! Provides two overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! ---! @see eql_v2."->" ---! @see eql_v2.selector - ---! @brief ->> operator with text selector ---! @param eql_v2_encrypted Encrypted JSONB data ---! @param text Field name to extract ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @example ---! SELECT encrypted_json ->> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector text) - RETURNS text -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - found eql_v2_encrypted; - BEGIN - -- found = eql_v2."->"(e, selector); - -- RETURN eql_v2.ciphertext(found); - RETURN eql_v2."->"(e, selector); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - - - ---------------------------------------------------- - ---! @brief ->> operator with encrypted selector ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted field selector ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @see eql_v2."->>"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2."->>"(e, eql_v2._selector(selector)); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - ---! @brief JSONB field accessor operator for encrypted values (->) ---! ---! Implements the -> operator to access fields/elements from encrypted JSONB data. ---! Returns the matching sv entry as `eql_v2.ste_vec_entry` (or NULL on miss). ---! ---! Encrypted JSON is represented as an array of sv elements in the ---! StEVec format. Each element has a selector, ciphertext, and index ---! terms: `{"sv": [{"c": "...", "s": "...", "hm": "..."}, ...]}`. ---! ---! Provides three overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! - (eql_v2_encrypted, integer) - Array index selector (0-based) ---! ---! All three return `eql_v2.ste_vec_entry` and preserve the source ---! payload's root `i` / `v` envelope metadata in the returned entry ---! (the DOMAIN CHECK on `ste_vec_entry` doesn't forbid extra fields). ---! ---! @note Operator resolution: Assignment casts are considered (PostgreSQL standard behavior). ---! To use text selector, parameter may need explicit cast to text. ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2.selector ---! @see eql_v2."->>" - ---! @brief -> operator with text selector ---! ---! Returns the sv entry whose `s` selector equals @p selector, with ---! the source payload's `i` / `v` metadata merged in. Selectors are ---! deterministic per (path, key) within a document, so at most one ---! entry matches; `jsonb_path_query_first` returns the first match ---! and stops scanning. ---! ---! Inlinable single-statement SQL: the planner folds this body into ---! the calling query, so `WHERE col -> 'sel' = $1` reduces structurally ---! to `eql_v2.eq_term(col -> 'sel') = eql_v2.eq_term($1)` and matches ---! a functional index built on `eql_v2.eq_term(col -> 'sel')`. ---! ---! @param e eql_v2_encrypted Encrypted JSONB payload (root) ---! @param selector text Selector hash (the `s` field value) ---! @return eql_v2.ste_vec_entry Matching entry merged with root meta, ---! NULL if no element matches. ---! ---! @note The returned entry carries `i` / `v` from the root in addition ---! to the sv-element fields. This is intentional: per-entry ---! extractors (`eql_v2.eq_term`, `eql_v2.ore_cllw`, ...) read ---! only their own fields and ignore `i` / `v`; callers that need ---! the root envelope (e.g. for decryption) still see it. ---! ---! @example ---! SELECT encrypted_json -> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector text) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ( - eql_v2.meta_data(e) || - jsonb_path_query_first( - (e).data, - '$.sv[*] ? (@.s == $sel)'::jsonpath, - jsonb_build_object('sel', selector) - ) - )::eql_v2.ste_vec_entry -$$; - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - ---------------------------------------------------- - ---! @brief -> operator with encrypted selector ---! ---! Convenience overload: extracts the selector text from an encrypted ---! selector payload and delegates to the (text) form. Inlinable. ---! ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted selector payload ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @see eql_v2."->"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."->"(e, eql_v2._selector(selector)) -$$; - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---------------------------------------------------- - ---! @brief -> operator with integer array index ---! ---! Returns the sv entry at the given (0-based, JSONB-style) array ---! index, merged with the root payload's `i` / `v` metadata. Returns ---! NULL when the underlying value isn't an sv-array payload or when ---! the index is out of bounds. ---! ---! @param e eql_v2_encrypted Encrypted sv-array payload ---! @param selector integer Array index (0-based, JSONB convention) ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @note Array index is 0-based (JSONB standard) despite PostgreSQL arrays being 1-based ---! @example ---! SELECT encrypted_array -> 0 FROM table; ---! @see eql_v2.is_ste_vec_array -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector integer) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE - WHEN eql_v2.is_ste_vec_array(e) THEN - (eql_v2.meta_data(e) || ((e).data -> 'sv' -> selector))::eql_v2.ste_vec_entry - ELSE NULL - END -$$; - - - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=integer -); - - ---! @brief EQL lint: detect non-inlinable operator implementation functions ---! ---! Returns one row per violation found in the installed EQL surface. The ---! Postgres planner can only inline a function during index matching when: ---! ---! * `LANGUAGE sql` (plpgsql / C / etc. cannot be inlined) ---! * `IMMUTABLE` or `STABLE` volatility (VOLATILE cannot be inlined into ---! index expressions) ---! * No `SET` clauses (e.g. `SET search_path = ...`) ---! * Not `SECURITY DEFINER` ---! * Single-statement SELECT body ---! ---! @note The single-statement SELECT body condition is **not yet checked** by ---! this lint. A `LANGUAGE sql` function with a multi-statement body, a CTE, ---! or any pre-SELECT statement will pass all four implemented checks while ---! remaining non-inlinable. Implementing the check requires walking `prosrc` ---! (or `pg_get_functiondef`); tracked as a follow-up to #194. ---! ---! Operators on encrypted types (`eql_v2_encrypted`, `eql_v2.bloom_filter`, ---! `eql_v2.ore_*`, etc.) whose implementation functions fail any of these ---! rules silently fall back to seq scan when the documented functional ---! indexes (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, ---! `eql_v2.ste_vec(col)`) are in place. This lint surfaces every such case. ---! ---! Severity: ---! `error` — fixable, blocks index matching, ship-blocking. ---! `warning` — likely-fixable, may not block matching but signals intent. ---! `info` — observational; useful for review, not a defect on its own. ---! ---! Categories: ---! `inlinability_language` — implementation function isn't `LANGUAGE sql`. ---! `inlinability_volatility` — implementation function is VOLATILE. ---! `inlinability_set_clause` — implementation function has a `SET` clause. ---! `inlinability_secdef` — implementation function is `SECURITY DEFINER`. ---! `inlinability_transitive` — implementation function is itself inlinable ---! but its body invokes a non-inlinable function ---! (depth 1; the planner can't peek through ---! that boundary). ---! ---! @example ---! ``` ---! SELECT severity, category, object_name, message ---! FROM eql_v2.lints() ---! WHERE severity = 'error' ---! ORDER BY category, object_name; ---! ``` ---! ---! @return SETOF record (severity text, category text, object_name text, message text) -CREATE OR REPLACE FUNCTION eql_v2.lints() -RETURNS TABLE ( - severity text, - category text, - object_name text, - message text -) -LANGUAGE sql STABLE -AS $$ - WITH - -- All operators where at least one operand involves an EQL type. Limits - -- the scope of the lint to the operator surface customers actually hit - -- via SQL (`col = val`, `col LIKE '...'`, `col @> '...'` and friends). - eql_operators AS ( - SELECT - op.oid AS oprid, - op.oprname AS opname, - op.oprcode AS implfunc, - op.oprleft::regtype AS lhs, - op.oprright::regtype AS rhs, - op.oprcode::regprocedure AS impl_signature - FROM pg_operator op - WHERE EXISTS ( - SELECT 1 FROM pg_type t - WHERE t.oid IN (op.oprleft, op.oprright) - AND (t.typname LIKE 'eql_v2%' - OR t.typnamespace = 'eql_v2'::regnamespace) - ) - ), - - -- Cross-join with each operator's implementation function metadata. - -- One row per operator; columns describe the inlinability of the impl. - op_impl AS ( - SELECT - eo.opname, - eo.lhs, - eo.rhs, - eo.impl_signature::text AS impl_signature, - lang_l.lanname AS lang, - p.provolatile AS volatility, - p.proconfig AS config, - p.prosecdef AS secdef, - p.prosrc AS body - FROM eql_operators eo - JOIN pg_proc p ON p.oid = eo.implfunc - JOIN pg_language lang_l ON lang_l.oid = p.prolang - ) - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Direct inlinability checks: each row examines one operator's │ - -- │ implementation function and emits a violation if any rule is │ - -- │ broken. Multiple violations on the same function become │ - -- │ multiple rows (developers see every reason it doesn't inline). │ - -- └─────────────────────────────────────────────────────────────────┘ - - SELECT - 'error' AS severity, - 'inlinability_language' AS category, - format('operator %s(%s, %s) -> %s', - opname, lhs, rhs, impl_signature) AS object_name, - format( - 'Operator implementation function is `LANGUAGE %s`; only `LANGUAGE sql` functions can be inlined by the planner. Bare `col %s val` queries fall back to seq scan even when a matching functional index exists.', - lang, opname) AS message - FROM op_impl - WHERE lang <> 'sql' - - UNION ALL - - SELECT - 'error', - 'inlinability_volatility', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function is `VOLATILE`. The Postgres planner refuses to inline volatile functions into index expressions, so functional indexes never engage. Mark the function `IMMUTABLE` (or `STABLE` if it depends on session state).', - opname) - FROM op_impl - WHERE volatility = 'v' - - UNION ALL - - SELECT - 'error', - 'inlinability_set_clause', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function has a `SET` clause (e.g. `SET search_path = ...`). Per Postgres function-inlining rules, any `SET` clause blocks inlining. Use schema-qualified identifiers in the body and remove the `SET` clause to allow the planner to inline.') - FROM op_impl - WHERE config IS NOT NULL - - UNION ALL - - SELECT - 'error', - 'inlinability_secdef', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - 'Operator implementation function is `SECURITY DEFINER`. Such functions cannot be inlined; remove `SECURITY DEFINER` or use a non-inlinable wrapper layer.' - FROM op_impl - WHERE secdef - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Transitive inlinability: an operator implementation function │ - -- │ that's itself inlinable can still fail to inline if its body │ - -- │ calls a non-inlinable function. Walk one level via pg_depend. │ - -- │ │ - -- │ Postgres records function-to-function dependencies in │ - -- │ pg_depend with deptype 'n' (normal) when one function references│ - -- │ another in its body — but only at CREATE time and only for │ - -- │ direct calls. This is good enough for v1; deeper transitive │ - -- │ analysis is a follow-up. │ - -- └─────────────────────────────────────────────────────────────────┘ - - UNION ALL - - SELECT - 'error', - 'inlinability_transitive', - format('operator %s(%s, %s) -> %s', oi.opname, oi.lhs, oi.rhs, - oi.impl_signature), - format( - 'Operator implementation function is inlinable but invokes non-inlinable function `%s` (lang=%s, volatility=%s%s). The chain blocks at depth 1: the planner inlines the outer call but cannot reduce the inner call into an index expression.', - called.proname, - called_lang.lanname, - CASE called.provolatile - WHEN 'i' THEN 'IMMUTABLE' - WHEN 's' THEN 'STABLE' - WHEN 'v' THEN 'VOLATILE' - END, - CASE WHEN called.proconfig IS NOT NULL - THEN ', has SET clause' - ELSE '' END) - FROM op_impl oi - -- Only worth the transitive check if the outer function is otherwise - -- inlinable — otherwise the direct lints above already report it. - JOIN pg_proc outer_p ON outer_p.oid = oi.impl_signature::regprocedure - JOIN pg_depend d - ON d.classid = 'pg_proc'::regclass - AND d.objid = outer_p.oid - AND d.refclassid = 'pg_proc'::regclass - AND d.deptype = 'n' - JOIN pg_proc called ON called.oid = d.refobjid - JOIN pg_language called_lang ON called_lang.oid = called.prolang - WHERE oi.lang = 'sql' - AND oi.volatility IN ('i', 's') - AND oi.config IS NULL - AND NOT oi.secdef - AND called.oid <> outer_p.oid - AND ( - called_lang.lanname <> 'sql' - OR called.provolatile = 'v' - OR called.proconfig IS NOT NULL - OR called.prosecdef - ) - - ORDER BY 1, 2, 3; -$$; - -COMMENT ON FUNCTION eql_v2.lints() IS - 'EQL lint: returns one row per non-inlinable operator implementation. ' - 'Run `SELECT * FROM eql_v2.lints() WHERE severity = ''error''` for a ' - 'CI-gateable check that all operator implementations on EQL types are ' - 'eligible for planner inlining.'; - ---! @file jsonb/functions.sql ---! @brief JSONB path query and array manipulation functions for encrypted data ---! ---! These functions provide PostgreSQL-compatible operations on encrypted JSONB values ---! using Structured Transparent Encryption (STE). They support: ---! - Path-based queries to extract nested encrypted values ---! - Existence checks for encrypted fields ---! - Array operations (length, elements extraction) ---! - Field-level HMAC term extraction for equality / GROUP BY / DISTINCT ---! ---! @note STE stores encrypted JSONB as a vector of encrypted elements ('sv') with selectors ---! @note Functions suppress errors for missing fields, type mismatches (similar to PostgreSQL jsonpath) ---! @note `selector` parameters in this module are *encrypted-side* selector ---! hashes — the deterministic hash that the crypto layer (e.g. ---! `@cipherstash/protect`) emits in the `s` field of each `sv` element ---! (e.g. `'a7cea93975ed8c01f861ccb6bd082784'`). Plaintext JSONPaths ---! like `'$.address.city'` are never accepted at runtime; the proxy / ---! client rewrites them to selector hashes before the query reaches EQL. - - ---! @brief Query encrypted JSONB for elements matching selector ---! ---! Searches the Structured Transparent Encryption (STE) vector for elements matching ---! the given selector path. Returns all matching encrypted elements. If multiple ---! matches form an array, they are wrapped with array metadata. ---! ---! @param jsonb Encrypted JSONB payload containing STE vector ('sv') ---! @param text Path selector to match against encrypted elements ---! @return SETOF eql_v2_encrypted Matching encrypted elements (may return multiple rows) ---! ---! @note Returns empty set if selector is not found (does not throw exception) ---! @note Array elements use same selector; multiple matches wrapped with 'a' flag ---! @note Returns a set containing NULL if val is NULL; returns empty set if no matches found ---! @see eql_v2.jsonb_path_query_first ---! @see eql_v2.jsonb_path_exists -CREATE FUNCTION eql_v2.jsonb_path_query(val jsonb, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT - CASE - WHEN bool_or(eql_v2.is_ste_vec_array(elem)) THEN - (eql_v2.meta_data(val) || jsonb_build_object('sv', jsonb_agg(elem), 'a', 1))::eql_v2_encrypted - ELSE - (eql_v2.meta_data(val) || (array_agg(elem))[1])::eql_v2_encrypted - END - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - HAVING count(*) > 0 -$$; - - ---! @brief Query encrypted JSONB with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its plaintext value ---! before delegating to main jsonb_path_query implementation. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Query encrypted JSONB with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector, ---! extracting the JSONB payload before querying. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @example ---! -- Query encrypted JSONB for the sv element at a given selector hash ---! SELECT * FROM eql_v2.jsonb_path_query(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Check if selector path exists in encrypted JSONB ---! ---! Tests whether any encrypted elements match the given selector path. ---! More efficient than jsonb_path_query when only existence check is needed. ---! ---! @param jsonb Encrypted JSONB payload to check ---! @param text Path selector to test ---! @return boolean True if matching element exists, false otherwise ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val jsonb, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT 1 FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - ); -$$; - - ---! @brief Check existence with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before checking existence. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to check ---! @param selector eql_v2_encrypted Encrypted selector to test ---! @return boolean True if path exists ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Check existence with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to check ---! @param text Path selector to test ---! @return boolean True if path exists ---! ---! @example ---! -- Check if the encrypted document has an sv element at a given selector hash ---! SELECT eql_v2.jsonb_path_exists(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Get first element matching selector ---! ---! Returns only the first encrypted element matching the selector path, ---! or NULL if no match found. More efficient than jsonb_path_query when ---! only one result is needed. ---! ---! @param jsonb Encrypted JSONB payload to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @note Uses LIMIT 1 internally for efficiency ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val jsonb, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (eql_v2.meta_data(val) || elem)::eql_v2_encrypted - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - LIMIT 1 -$$; - - ---! @brief Get first element with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before querying for first match. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Get first element with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @example ---! -- Get the first matching sv element from an encrypted document ---! SELECT eql_v2.jsonb_path_query_first(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, selector); -$$; - - - ------------------------------------------------------------------------------------- - - ---! @brief Get length of encrypted JSONB array ---! ---! Returns the number of elements in an encrypted JSONB array by counting ---! elements in the STE vector ('sv'). The encrypted value must have the ---! array flag ('a') set to true. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return integer Number of elements in the array ---! @throws Exception 'cannot get array length of a non-array' if 'a' flag is missing or not true ---! ---! @note Array flag 'a' must be present and set to true value ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_length(val jsonb) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_array(val) THEN - sv := eql_v2.ste_vec(val); - RETURN array_length(sv, 1); - END IF; - - RAISE 'cannot get array length of a non-array'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get array length from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts the ---! JSONB payload before computing array length. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return integer Number of elements in the array ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get length of encrypted array ---! SELECT eql_v2.jsonb_array_length(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_length(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_length(val eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN ( - SELECT eql_v2.jsonb_array_length(val.data) - ); - END; -$$ LANGUAGE plpgsql; - - - - ---! @brief Extract elements from encrypted JSONB array ---! ---! Returns each element of an encrypted JSONB array as a separate row. ---! Each element is returned as an eql_v2_encrypted value with metadata ---! preserved from the parent array. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Each element inherits metadata (version, ident) from parent ---! @see eql_v2.jsonb_array_length ---! @see eql_v2.jsonb_array_elements_text -CREATE FUNCTION eql_v2.jsonb_array_elements(val jsonb) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - meta jsonb; - item jsonb; - BEGIN - - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - -- Column identifier and version - meta := eql_v2.meta_data(val); - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - item = sv[idx]; - RETURN NEXT (meta || item)::eql_v2_encrypted; - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract elements from encrypted array type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element as a separate row. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Expand encrypted array into rows ---! SELECT * FROM eql_v2.jsonb_array_elements(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements(val eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract encrypted array elements as ciphertext ---! ---! Returns each element of an encrypted JSONB array as its raw ciphertext ---! value (text representation). Unlike jsonb_array_elements, this returns ---! only the ciphertext 'c' field without metadata. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Returns ciphertext only, not full encrypted structure ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val jsonb) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - RETURN NEXT eql_v2.ciphertext(sv[idx]); - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract array elements as ciphertext from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element's ciphertext as text. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get ciphertext of each array element ---! SELECT * FROM eql_v2.jsonb_array_elements_text(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements_text(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val eql_v2_encrypted) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements_text(val.data); - END; -$$ LANGUAGE plpgsql; - - ------------------------------------------------------------------------------------- - --- `eql_v2.hmac_256_terms(eql_v2_encrypted)` was added under #205 as a --- GIN-indexable {s, hm} aggregate. It's been removed: under the XOR --- contract each sv element carries exactly one of `hm` (bool leaves, --- array / object roots) or `oc` (string / number leaves), and --- `hmac_256_terms` filters out everything without `hm` — so containment --- queries via this index could never match on string / number selectors. --- The canonical XOR-aware replacement is the typed --- `@>(eql_v2_encrypted, eql_v2.stevec_query)` overload, which inlines --- to `eql_v2.to_stevec_query(col)::jsonb @> needle::jsonb` and engages --- a functional GIN on `(eql_v2.to_stevec_query(col)::jsonb) jsonb_path_ops`. --- See U-007 / U-008 in `docs/upgrading/v2.3.md`. ---! @file encryptindex/functions.sql ---! @brief Configuration lifecycle and column encryption management ---! ---! Provides functions for managing encryption configuration transitions: ---! - Comparing configurations to identify changes ---! - Identifying columns needing encryption ---! - Creating and renaming encrypted columns during initial setup ---! - Tracking encryption progress ---! ---! These functions support the workflow of activating a pending configuration ---! and performing the initial encryption of plaintext columns. - - ---! @brief Compare two configurations and find differences ---! @internal ---! ---! Returns table/column pairs where configuration differs between two configs. ---! Used to identify which columns need encryption when activating a pending config. ---! ---! @param a jsonb First configuration to compare ---! @param b jsonb Second configuration to compare ---! @return TABLE(table_name text, column_name text) Columns with differing configuration ---! ---! @note Compares configuration structure, not just presence/absence ---! @see eql_v2.select_pending_columns -CREATE FUNCTION eql_v2.diff_config(a JSONB, b JSONB) - RETURNS TABLE(table_name TEXT, column_name TEXT) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - WITH table_keys AS ( - SELECT jsonb_object_keys(a->'tables') AS key - UNION - SELECT jsonb_object_keys(b->'tables') AS key - ), - column_keys AS ( - SELECT tk.key AS table_key, jsonb_object_keys(a->'tables'->tk.key) AS column_key - FROM table_keys tk - UNION - SELECT tk.key AS table_key, jsonb_object_keys(b->'tables'->tk.key) AS column_key - FROM table_keys tk - ) - SELECT - ck.table_key AS table_name, - ck.column_key AS column_name - FROM - column_keys ck - WHERE - (a->'tables'->ck.table_key->ck.column_key IS DISTINCT FROM b->'tables'->ck.table_key->ck.column_key); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get columns with pending configuration changes ---! ---! Compares 'pending' and 'active' configurations to identify columns that need ---! encryption or re-encryption. Returns columns where configuration differs. ---! ---! @return TABLE(table_name text, column_name text) Columns needing encryption ---! @throws Exception if no pending configuration exists ---! ---! @note Treats missing active config as empty config ---! @see eql_v2.diff_config ---! @see eql_v2.select_target_columns -CREATE FUNCTION eql_v2.select_pending_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - active JSONB; - pending JSONB; - config_id BIGINT; - BEGIN - SELECT data INTO active FROM eql_v2_configuration WHERE state = 'active'; - - -- set default config - IF active IS NULL THEN - active := '{}'; - END IF; - - SELECT id, data INTO config_id, pending FROM eql_v2_configuration WHERE state = 'pending'; - - -- set default config - IF config_id IS NULL THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - RETURN QUERY - SELECT d.table_name, d.column_name FROM eql_v2.diff_config(active, pending) as d; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Map pending columns to their encrypted target columns ---! ---! For each column with pending configuration, identifies the corresponding ---! encrypted column. During initial encryption, target is '{column_name}_encrypted'. ---! Returns NULL for target_column if encrypted column doesn't exist yet. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Column mappings ---! ---! @note Target column is NULL if no column exists matching either 'column_name' or 'column_name_encrypted' with type eql_v2_encrypted ---! @note The LEFT JOIN checks both original and '_encrypted' suffix variations with type verification ---! @see eql_v2.select_pending_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.select_target_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT - c.table_name, - c.column_name, - s.column_name as target_column - FROM - eql_v2.select_pending_columns() c - LEFT JOIN information_schema.columns s ON - s.table_name = c.table_name AND - (s.column_name = c.column_name OR s.column_name = c.column_name || '_encrypted') AND - s.udt_name = 'eql_v2_encrypted'; -$$ LANGUAGE sql; - - ---! @brief Check if database is ready for encryption ---! ---! Verifies that all columns with pending configuration have corresponding ---! encrypted target columns created. Returns true if encryption can proceed. ---! ---! @return boolean True if all pending columns have target encrypted columns ---! ---! @note Returns false if any pending column lacks encrypted column ---! @see eql_v2.select_target_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.ready_for_encryption() - RETURNS BOOLEAN - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT * - FROM eql_v2.select_target_columns() AS c - WHERE c.target_column IS NOT NULL); -$$ LANGUAGE sql; - - ---! @brief Create encrypted columns for initial encryption ---! ---! For each plaintext column with pending configuration that lacks an encrypted ---! target column, creates a new column '{column_name}_encrypted' of type ---! eql_v2_encrypted. This prepares the database schema for initial encryption. ---! ---! @return TABLE(table_name text, column_name text) Created encrypted columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE ADD COLUMN) - modifies database schema ---! @note Only creates columns that don't already exist ---! @see eql_v2.select_target_columns ---! @see eql_v2.rename_encrypted_columns -CREATE FUNCTION eql_v2.create_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name IN - SELECT c.table_name, (c.column_name || '_encrypted') FROM eql_v2.select_target_columns() AS c WHERE c.target_column IS NULL - LOOP - EXECUTE format('ALTER TABLE %I ADD column %I eql_v2_encrypted;', table_name, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Finalize initial encryption by renaming columns ---! ---! After initial encryption completes, renames columns to complete the transition: ---! - Plaintext column '{column_name}' → '{column_name}_plaintext' ---! - Encrypted column '{column_name}_encrypted' → '{column_name}' ---! ---! This makes the encrypted column the primary column with the original name. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Renamed columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE RENAME COLUMN) - modifies database schema ---! @note Only renames columns where target is '{column_name}_encrypted' ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.rename_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name, target_column IN - SELECT * FROM eql_v2.select_target_columns() as c WHERE c.target_column = c.column_name || '_encrypted' - LOOP - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, column_name, column_name || '_plaintext'); - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, target_column, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Count rows encrypted with active configuration ---! @internal ---! ---! Counts rows in a table where the encrypted column was encrypted using ---! the currently active configuration. Used to track encryption progress. ---! ---! @param table_name text Name of table to check ---! @param column_name text Name of encrypted column to check ---! @return bigint Count of rows encrypted with active configuration ---! ---! @note The 'v' field in encrypted payloads stores the payload version ("2"), not the configuration ID ---! @note Configuration tracking mechanism is implementation-specific -CREATE FUNCTION eql_v2.count_encrypted_with_active_config(table_name TEXT, column_name TEXT) - RETURNS BIGINT - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result BIGINT; -BEGIN - EXECUTE format( - 'SELECT COUNT(%I) FROM %s t WHERE %I->>%L = (SELECT id::TEXT FROM eql_v2_configuration WHERE state = %L)', - column_name, table_name, column_name, 'v', 'active' - ) - INTO result; - RETURN result; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Compute hash integer for encrypted value ---! ---! Produces a 32-bit integer hash suitable for PostgreSQL hash joins, GROUP BY, ---! DISTINCT, and hash aggregate operations. Used by the `eql_v2_encrypted` hash ---! operator class (`FUNCTION 1`). Inlinable single-statement SQL — the SQL ---! function machinery is much cheaper per row than plpgsql, which matters ---! because HashAggregate / hash-join call this once per input row. ---! ---! Returns `hashtext` of the root payload's `hm` term. This is the canonical ---! bucket for equality groups, since `=` on `eql_v2_encrypted` reduces to ---! `hmac_256(a) = hmac_256(b)` post-#193. ---! ---! @par Contract ---! Callers using `GROUP BY` / `DISTINCT` / hash joins on `eql_v2_encrypted` ---! MUST configure the column with a `unique` index so the crypto layer ---! emits `hm` — `hm` is assumed present. A missing `hm` is a misconfiguration ---! that surfaces upstream via [U-002](docs/upgrading/v2.3.md#u-002-equality-and-hashing-require-hmac). ---! ---! @param val eql_v2_encrypted Encrypted value to hash ---! @return integer 32-bit hash value derived from `hm` ---! ---! @note For grouping a value extracted from an encrypted JSON document, use ---! the field-level recipe directly: `GROUP BY eql_v2.eq_term(col -> '<selector>')` ---! (covers both hm-bearing and oc-bearing selectors via the XOR-aware ---! extractor — see `src/ste_vec/eq_term.sql`). That bypasses ---! `hash_encrypted` entirely. ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.hash_encrypted(val eql_v2_encrypted) - RETURNS integer - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT pg_catalog.hashtext(eql_v2.hmac_256(val)::text) -$$; - - ---! @brief Validate presence of ident field in encrypted payload ---! @internal ---! ---! Checks that the encrypted JSONB payload contains the required 'i' (ident) field. ---! The ident field tracks which table and column the encrypted value belongs to. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'i' field is present ---! @throws Exception if 'i' field is missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'i' THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ident (i) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate table and column fields in ident ---! @internal ---! ---! Checks that the 'i' (ident) field contains both 't' (table) and 'c' (column) ---! subfields, which identify the origin of the encrypted value. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if both 't' and 'c' subfields are present ---! @throws Exception if 't' or 'c' subfields are missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i_ct(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val->'i' ?& array['t', 'c']) THEN - RETURN true; - END IF; - RAISE 'Encrypted column ident (i) missing table (t) or column (c) fields: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Validate version field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload has version field 'v' set to '2', ---! the current EQL v2 payload version. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'v' field is present and equals '2' ---! @throws Exception if 'v' field is missing or not '2' ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_v(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - - IF val->>'v' <> '2' THEN - RAISE 'Expected encrypted column version (v) 2'; - RETURN false; - END IF; - - RETURN true; - END IF; - RAISE 'Encrypted column missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ciphertext field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload carries the required root-level ciphertext ---! envelope. The v2.3 payload schema admits two mutually exclusive top-level ---! shapes (`docs/reference/schema/eql-payload-v2.3.schema.json`): ---! ---! - `EncryptedPayload` (scalar) — carries `c` at the root. ---! - `SteVecPayload` (jsonb / structured) — carries `sv` at the root; the ---! root document ciphertext lives inside `sv[0].c`, so `c` is absent at ---! the root. ---! ---! Either shape satisfies this check. Per-element ciphertext validity on ---! `sv` entries is enforced separately by the `eql_v2.ste_vec_entry` DOMAIN. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if either 'c' or 'sv' is present at the root ---! @throws Exception if neither 'c' nor 'sv' is present ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_c(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'c') OR (val ? 'sv') THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ciphertext (c) or ste_vec (sv) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate complete encrypted payload structure ---! ---! Comprehensive validation function that checks all required fields in an ---! encrypted JSONB payload: version ('v'), ciphertext ('c'), ident ('i'), ---! and ident subfields ('t', 'c'). ---! ---! This function is used in CHECK constraints to ensure encrypted column ---! data integrity at the database level. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if all structure checks pass ---! @throws Exception if any required field is missing or invalid ---! ---! @example ---! -- Add validation constraint to encrypted column ---! ALTER TABLE users ADD CONSTRAINT check_email_encrypted ---! CHECK (eql_v2.check_encrypted(encrypted_email::jsonb)); ---! ---! @see eql_v2._encrypted_check_v ---! @see eql_v2._encrypted_check_c ---! @see eql_v2._encrypted_check_i ---! @see eql_v2._encrypted_check_i_ct -CREATE FUNCTION eql_v2.check_encrypted(val jsonb) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN ( - eql_v2._encrypted_check_v(val) AND - eql_v2._encrypted_check_c(val) AND - eql_v2._encrypted_check_i(val) AND - eql_v2._encrypted_check_i_ct(val) - ); -END; - - ---! @brief Validate encrypted composite type structure ---! ---! Validates an eql_v2_encrypted composite type by checking its underlying ---! JSONB payload. Delegates to eql_v2.check_encrypted(jsonb). ---! ---! @param eql_v2_encrypted Encrypted value to validate ---! @return Boolean True if structure is valid ---! @throws Exception if any required field is missing or invalid ---! ---! @see eql_v2.check_encrypted(jsonb) -CREATE FUNCTION eql_v2.check_encrypted(val eql_v2_encrypted) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN eql_v2.check_encrypted(val.data); -END; - - ---! @brief Fallback literal comparison for encrypted values ---! @internal ---! ---! Compares two encrypted values by their raw JSONB representation when no ---! suitable index terms are available. This ensures consistent ordering required ---! for btree correctness and prevents "lock BufferContent is not held" errors. ---! ---! Used as a last resort fallback in eql_v2.compare() when encrypted values ---! lack matching index terms (hmac_256, ore). ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note This compares the encrypted payloads directly, not the plaintext values ---! @note Ordering is consistent but not meaningful for range queries ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.compare_literal(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT CASE - WHEN a.data < b.data THEN -1 - WHEN a.data > b.data THEN 1 - ELSE 0 - END; -$$; - --- Aggregate functions for ORE - ---! @brief State transition function for min aggregate ---! @internal ---! ---! Returns the smaller of two encrypted values for use in MIN aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The smaller of the two values ---! ---! @see eql_v2.min(eql_v2_encrypted) -CREATE FUNCTION eql_v2.min(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a < b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find minimum encrypted value in a group ---! ---! Aggregate function that returns the minimum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Minimum value in the group ---! ---! @example ---! -- Find minimum age per department ---! SELECT department, eql_v2.min(encrypted_age) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.min(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.min(eql_v2_encrypted) -( - sfunc = eql_v2.min, - stype = eql_v2_encrypted -); - - ---! @brief State transition function for max aggregate ---! @internal ---! ---! Returns the larger of two encrypted values for use in MAX aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The larger of the two values ---! ---! @see eql_v2.max(eql_v2_encrypted) -CREATE FUNCTION eql_v2.max(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a > b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find maximum encrypted value in a group ---! ---! Aggregate function that returns the maximum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Maximum value in the group ---! ---! @example ---! -- Find maximum salary per department ---! SELECT department, eql_v2.max(encrypted_salary) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.max(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.max(eql_v2_encrypted) -( - sfunc = eql_v2.max, - stype = eql_v2_encrypted -); - - ---! @file config/indexes.sql ---! @brief Configuration state uniqueness indexes ---! ---! Creates partial unique indexes to enforce that only one configuration ---! can be in 'active', 'pending', or 'encrypting' state at any time. ---! Multiple 'inactive' configurations are allowed. ---! ---! @note Uses partial indexes (WHERE clauses) for efficiency ---! @note Prevents conflicting configurations from being active simultaneously ---! @see config/types.sql for state definitions - - ---! @brief Unique active configuration constraint ---! @note Only one configuration can be 'active' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'active'; - ---! @brief Unique pending configuration constraint ---! @note Only one configuration can be 'pending' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'pending'; - ---! @brief Unique encrypting configuration constraint ---! @note Only one configuration can be 'encrypting' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'encrypting'; - - ---! @brief Add a search index configuration for an encrypted column ---! ---! Configures a searchable encryption index (unique, match, ore, ope, or ste_vec) ---! on an encrypted column. Creates or updates the pending configuration, then ---! migrates and activates it unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to configure ---! @param index_name Text Type of index ('unique', 'match', 'ore', 'ope', 'ste_vec') ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB Index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if index already exists for this column ---! @throws Exception if cast_as is not a valid type ---! ---! @example ---! -- Add unique index for exact-match searches ---! SELECT eql_v2.add_search_config('users', 'email', 'unique'); ---! ---! -- Add match index for LIKE searches with custom token length ---! SELECT eql_v2.add_search_config('posts', 'content', 'match', 'text', ---! '{"token_filters": [{"kind": "downcase"}], "tokenizer": {"kind": "ngram", "token_length": 3}}' ---! ); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.add_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - o jsonb; - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if index exists - IF _config #> array['tables', table_name, column_name, 'indexes'] ? index_name THEN - RAISE EXCEPTION '% index exists for column: % %', index_name, table_name, column_name; - END IF; - - IF NOT cast_as = ANY('{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}') THEN - RAISE EXCEPTION '% is not a valid cast type', cast_as; - END IF; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- set default options for index if opts empty - IF index_name = 'match' AND opts = '{}' THEN - SELECT eql_v2.config_match_default() INTO opts; - END IF; - - SELECT eql_v2.config_add_index(table_name, column_name, index_name, opts, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a search index configuration from an encrypted column ---! ---! Removes a previously configured search index from an encrypted column. ---! Updates the pending configuration, then migrates and activates it ---! unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove match index from column ---! SELECT eql_v2.remove_search_config('posts', 'content', 'match'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.modify_search_config -CREATE FUNCTION eql_v2.remove_search_config(table_name text, column_name text, index_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the index does not exist - -- IF NOT _config->key ? index_name THEN - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No % index exists for column: % %', index_name, table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the index - SELECT _config #- array['tables', table_name, column_name, 'indexes', index_name] INTO _config; - - -- update the config and migrate (even if empty) - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Modify a search index configuration for an encrypted column ---! ---! Updates an existing search index configuration by removing and re-adding it ---! with new options. Convenience function that combines remove and add operations. ---! If index does not exist, it is added. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to modify ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB New index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! ---! @example ---! -- Change match index tokenizer settings ---! SELECT eql_v2.modify_search_config('posts', 'content', 'match', 'text', ---! '{"tokenizer": {"kind": "ngram", "token_length": 4}}' ---! ); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.modify_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - PERFORM eql_v2.remove_search_config(table_name, column_name, index_name, migrating); - RETURN eql_v2.add_search_config(table_name, column_name, index_name, cast_as, opts, migrating); - END; -$$ LANGUAGE plpgsql; - ---! @brief Migrate pending configuration to encrypting state ---! ---! Transitions the pending configuration to encrypting state, validating that ---! all configured columns have encrypted target columns ready. This is part of ---! the configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if migration succeeds ---! @throws Exception if encryption already in progress ---! @throws Exception if no pending configuration exists ---! @throws Exception if configured columns lack encrypted targets ---! ---! @example ---! -- Manually migrate configuration (normally done automatically) ---! SELECT eql_v2.migrate_config(); ---! ---! @see eql_v2.activate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.migrate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - RAISE EXCEPTION 'An encryption is already in progress'; - END IF; - - IF NOT EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - IF NOT eql_v2.ready_for_encryption() THEN - RAISE EXCEPTION 'Some pending columns do not have an encrypted target'; - END IF; - - UPDATE public.eql_v2_configuration SET state = 'encrypting' WHERE state = 'pending'; - RETURN true; - END; -$$ LANGUAGE plpgsql; - ---! @brief Activate encrypting configuration ---! ---! Transitions the encrypting configuration to active state, making it the ---! current operational configuration. Marks previous active configuration as ---! inactive. Final step in configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if activation succeeds ---! @throws Exception if no encrypting configuration exists to activate ---! ---! @example ---! -- Manually activate configuration (normally done automatically) ---! SELECT eql_v2.activate_config(); ---! ---! @see eql_v2.migrate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.activate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'encrypting'; - RETURN true; - ELSE - RAISE EXCEPTION 'No encrypting configuration exists to activate'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Discard pending configuration ---! ---! Deletes the pending configuration without applying changes. Use this to ---! abandon configuration changes before they are migrated and activated. ---! ---! @return Boolean True if discard succeeds ---! @throws Exception if no pending configuration exists to discard ---! ---! @example ---! -- Discard uncommitted configuration changes ---! SELECT eql_v2.discard(); ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.discard() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - DELETE FROM public.eql_v2_configuration WHERE state = 'pending'; - RETURN true; - ELSE - RAISE EXCEPTION 'No pending configuration exists to discard'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Configure a column for encryption ---! ---! Adds a column to the encryption configuration, making it eligible for ---! encrypted storage and search indexes. Creates or updates pending configuration, ---! adds encrypted constraint, then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to encrypt ---! @param cast_as Text PostgreSQL type to cast decrypted values (default: 'text') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if column already configured for encryption ---! ---! @example ---! -- Configure email column for encryption ---! SELECT eql_v2.add_column('users', 'email', 'text'); ---! ---! -- Configure age column with integer casting ---! SELECT eql_v2.add_column('users', 'age', 'int'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_column -CREATE FUNCTION eql_v2.add_column(table_name text, column_name text, cast_as text DEFAULT 'text', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - -- if index exists - IF _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'Config exists for column: % %', table_name, column_name; - END IF; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a column from encryption configuration ---! ---! Removes a column from the encryption configuration, including all associated ---! search indexes. Removes encrypted constraint, updates pending configuration, ---! then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove email column from encryption ---! SELECT eql_v2.remove_column('users', 'email'); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.remove_column(table_name text, column_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the column does not exist - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No configuration exists for column: % %', table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the column - SELECT _config #- array['tables', table_name, column_name] INTO _config; - - -- if table is now empty, remove the table - IF _config #> array['tables', table_name] = '{}' THEN - SELECT _config #- array['tables', table_name] INTO _config; - END IF; - - PERFORM eql_v2.remove_encrypted_constraint(table_name, column_name); - - -- update the config (even if empty) and activate - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - -- For empty configs, skip migration validation and directly activate - IF _config #> array['tables'] = '{}' THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'pending'; - ELSE - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - END IF; - - -- exeunt - RETURN _config; - - END; -$$ LANGUAGE plpgsql; - ---! @brief Reload configuration from CipherStash Proxy ---! ---! Placeholder function for reloading configuration from the CipherStash Proxy. ---! Currently returns NULL without side effects. ---! ---! @return Void ---! ---! @note This function may be used for configuration synchronization in future versions -CREATE FUNCTION eql_v2.reload_config() - RETURNS void -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN NULL; -END; - ---! @brief Query encryption configuration in tabular format ---! ---! Returns the active encryption configuration as a table for easier querying ---! and filtering. Shows all configured tables, columns, cast types, and indexes. ---! ---! @return TABLE Contains configuration state, relation name, column name, cast type, and indexes ---! ---! @example ---! -- View all encrypted columns ---! SELECT * FROM eql_v2.config(); ---! ---! -- Find all columns with match indexes ---! SELECT relation, col_name FROM eql_v2.config() ---! WHERE indexes ? 'match'; ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.config() RETURNS TABLE ( - state eql_v2_configuration_state, - relation text, - col_name text, - decrypts_as text, - indexes jsonb -) - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - RETURN QUERY - WITH tables AS ( - SELECT cfg.state, tables.key AS table, tables.value AS tbl_config - FROM public.eql_v2_configuration cfg, jsonb_each(data->'tables') tables - WHERE cfg.data->>'v' = '1' - ) - SELECT - tables.state, - tables.table, - column_config.key, - COALESCE(column_config.value->>'plaintext_type', column_config.value->>'cast_as'), - column_config.value->'indexes' - FROM tables, jsonb_each(tables.tbl_config) column_config; -END; -$$ LANGUAGE plpgsql; - ---! @file config/constraints.sql ---! @brief Configuration validation functions and constraints ---! ---! Provides CHECK constraint functions to validate encryption configuration structure. ---! Ensures configurations have required fields (version, tables) and valid values ---! for index types and cast types before being stored. ---! ---! @see config/tables.sql where constraints are applied - - ---! @brief Extract index type names from configuration ---! @internal ---! ---! Helper function that extracts all index type names from the configuration's ---! 'indexes' sections across all tables and columns. ---! ---! @param jsonb Configuration data to extract from ---! @return SETOF text Index type names (e.g., 'match', 'ore', 'unique', 'ste_vec') ---! ---! @note Used by config_check_indexes for validation ---! @see eql_v2.config_check_indexes -CREATE FUNCTION eql_v2.config_get_indexes(val jsonb) - RETURNS SETOF text - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_object_keys(jsonb_path_query(val,'$.tables.*.*.indexes')); -END; - - ---! @brief Validate index types in configuration ---! @internal ---! ---! Checks that all index types specified in the configuration are valid. ---! Valid index types are: match, ore, ope, unique, ste_vec. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all index types are valid ---! @throws Exception if any invalid index type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @see eql_v2.config_get_indexes -CREATE FUNCTION eql_v2.config_check_indexes(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF (SELECT EXISTS (SELECT eql_v2.config_get_indexes(val))) THEN - IF (SELECT bool_and(index = ANY('{match, ore, ope, unique, ste_vec}')) FROM eql_v2.config_get_indexes(val) AS index) THEN - RETURN true; - END IF; - RAISE 'Configuration has an invalid index (%). Index should be one of {match, ore, ope, unique, ste_vec}', val; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate cast types in configuration ---! @internal ---! ---! Checks that all 'cast_as' and 'plaintext_type' types specified in the configuration are valid. ---! Valid cast types are: text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all cast types are valid or no cast types specified ---! @throws Exception if any invalid cast type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Empty configurations (no cast_as/plaintext_type fields) are valid ---! @note Cast type names are EQL's internal representations, not PostgreSQL native types ---! @note 'plaintext_type' is accepted as a canonical alias for 'cast_as' -CREATE FUNCTION eql_v2.config_check_cast(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_types text[] := '{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}'; - BEGIN - -- Validate cast_as fields - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as'))) THEN - IF NOT (SELECT bool_and(cast_as = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as')) AS cast_as) casts) THEN - RAISE 'Configuration has an invalid cast_as (%). Cast should be one of %', val, _valid_types; - END IF; - END IF; - - -- Validate plaintext_type fields (canonical alias for cast_as) - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type'))) THEN - IF NOT (SELECT bool_and(pt = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type')) AS pt) types) THEN - RAISE 'Configuration has an invalid plaintext_type (%). Type should be one of %', val, _valid_types; - END IF; - END IF; - - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate tables field presence ---! @internal ---! ---! Ensures the configuration has a 'tables' field, which is required ---! to specify which database tables contain encrypted columns. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'tables' field exists ---! @throws Exception if 'tables' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_tables(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'tables') THEN - RETURN true; - END IF; - RAISE 'Configuration missing tables (tables) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate version field presence ---! @internal ---! ---! Ensures the configuration has a 'v' (version) field, which tracks ---! the configuration format version. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'v' field exists ---! @throws Exception if 'v' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_version(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - RETURN true; - END IF; - RAISE 'Configuration missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ste_vec index mode option ---! @internal ---! ---! Checks that the optional `mode` field on `ste_vec` index configurations is ---! one of the recognised values. Valid modes are: standard, compat. ---! Configurations without a `mode` field (the default) pass unconditionally. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if every ste_vec mode is valid, or none are set ---! @throws Exception if any ste_vec.mode value is not in the allowed set ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Mode is optional — only configurations that set it are validated -CREATE FUNCTION eql_v2.config_check_ste_vec_mode(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_modes text[] := '{standard, compat}'; - BEGIN - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode'))) THEN - IF NOT (SELECT bool_and(mode = ANY(_valid_modes)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode')) AS mode) modes) THEN - RAISE 'Configuration has an invalid ste_vec mode (%). Mode should be one of %', val, _valid_modes; - END IF; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Drop existing data validation constraint if present ---! @note Allows constraint to be recreated during upgrades -ALTER TABLE public.eql_v2_configuration DROP CONSTRAINT IF EXISTS eql_v2_configuration_data_check; - - ---! @brief Comprehensive configuration data validation ---! ---! CHECK constraint that validates all aspects of configuration data: ---! - Version field presence ---! - Tables field presence ---! - Valid cast_as types ---! - Valid index types ---! - Valid ste_vec mode (when set) ---! ---! @note Combines all config_check_* validation functions ---! @see eql_v2.config_check_version ---! @see eql_v2.config_check_tables ---! @see eql_v2.config_check_cast ---! @see eql_v2.config_check_indexes ---! @see eql_v2.config_check_ste_vec_mode -ALTER TABLE public.eql_v2_configuration - ADD CONSTRAINT eql_v2_configuration_data_check CHECK ( - eql_v2.config_check_version(data) AND - eql_v2.config_check_tables(data) AND - eql_v2.config_check_cast(data) AND - eql_v2.config_check_indexes(data) AND - eql_v2.config_check_ste_vec_mode(data) -); - - ---! @file pin_search_path.sql ---! @brief Post-install: pin search_path on every eql_v2.* function ---! ---! This file is appended verbatim by `tasks/build.sh` to the end of every ---! release variant (main, supabase, protect/stack), AFTER all `src/**/*.sql` ---! files have been concatenated. It lives outside `src/` so it stays out of ---! the dependency graph entirely — each variant has a different leaf set ---! (supabase excludes `**/*operator_class.sql`; protect excludes `src/config/*` ---! and `src/encryptindex/*`), and threading REQUIREs to be ordered last in ---! every variant simultaneously is fragile. ---! ---! Iterates over functions in the `eql_v2` schema and applies a fixed ---! `search_path` via `ALTER FUNCTION ... SET search_path = ...`. This is the ---! only way to satisfy Supabase splinter's `function_search_path_mutable` ---! lint, which checks `pg_proc.proconfig` directly. ---! ---! @note A SET clause disables PostgreSQL's SQL-function inlining (see ---! inline_function() in src/backend/optimizer/util/clauses.c). For most ---! eql_v2 helpers this is irrelevant. The exceptions are wrappers that ---! must inline to expose `eql_v2.jsonb_array(col) @> ...` to the planner ---! so the GIN index on `jsonb_array(e)` can be matched. Those are ---! deliberately skipped here and allowlisted in `tasks/test/splinter.sh`. ---! ---! @see tasks/test/splinter.sh ---! @see tasks/build.sh - -DO $$ -DECLARE - fn_oid oid; - inline_critical_oids oid[]; - enc_oid oid; - jsonb_oid oid; - text_oid oid; - entry_oid oid; -BEGIN - -- Resolve type oids without depending on caller search_path. The encrypted - -- composite type is created in `public`; jsonb / text are in `pg_catalog`; - -- the ste_vec_entry DOMAIN lives in `eql_v2`. - SELECT t.oid INTO enc_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'public' AND t.typname = 'eql_v2_encrypted'; - - IF enc_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type public.eql_v2_encrypted not found — ' - 'this script must run after all EQL src/**/*.sql files have been loaded'; - END IF; - - SELECT t.oid INTO jsonb_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'jsonb'; - - IF jsonb_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.jsonb not found'; - END IF; - - SELECT t.oid INTO text_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'text'; - - IF text_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.text not found'; - END IF; - - SELECT t.oid INTO entry_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'ste_vec_entry'; - - IF entry_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type eql_v2.ste_vec_entry not found'; - END IF; - - -- Wrappers that must remain inlinable for functional-index matching. - -- Verified empirically: with SET, EXPLAIN drops to Seq Scan; without, - -- it uses Bitmap Index Scan / Index Scan. - -- - -- Phase 1 operator inlining (#193): `=`, `<>`, `~~`, `~~*`, `@>`, `<@` - -- on `eql_v2_encrypted` and the cross-type (encrypted, jsonb) / - -- (jsonb, encrypted) overloads emitted by ORMs that bind parameters - -- as jsonb (Drizzle, PostgREST, encryptedSupabase). The implementation - -- functions reduce to `extractor(a) op extractor(b)` and must inline - -- to match the documented functional indexes - -- (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, - -- `eql_v2.ste_vec(col)`). - -- - -- For `~~` / `~~*` the planner must inline two layers — the operator - -- function `eql_v2."~~"` and the helper `eql_v2.like` / `eql_v2.ilike` - -- — to reach the canonical `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)` - -- form that the documented functional index matches. The helpers are - -- allowlisted alongside the operator wrappers below; pinning either - -- layer breaks the chain and reverts to Seq Scan. - -- - -- Note: pg_proc.proargtypes is an oidvector with 0-based bounds, so we - -- compare elements individually rather than using array equality (which - -- requires matching bounds, not just contents). - SELECT pg_catalog.array_agg(p.oid) INTO inline_critical_oids - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - AND ( - -- Same-type (encrypted, encrypted) operators that must inline. - -- `like`/`ilike` are the SQL helpers that `~~`/`~~*` delegate to; - -- both layers must inline to reach `bloom_filter(a) @> bloom_filter(b)`. - -- `<`, `<=`, `>`, `>=` inline to `ore_block_u64_8_256(a) op - -- ore_block_u64_8_256(b)`; they must reach the functional ORE index - -- expression `eql_v2.ore_block_u64_8_256(col)` for bare range - -- queries to engage Index Scan. - (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', '@>', '<@', - 'jsonb_contains', 'jsonb_contained_by', - 'like', 'ilike') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = enc_oid) - -- Cross-type (encrypted, jsonb). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = jsonb_oid) - -- Cross-type (jsonb, encrypted). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = jsonb_oid AND p.proargtypes[1] = enc_oid) - -- Root-level HMAC extractor (#205): all 1-arg overloads are now - -- inlinable SQL. Must stay unpinned so the planner can fold extractor - -- calls inside the inlined equality operator bodies into the calling - -- query, preserving the functional-index match. - OR (p.pronargs = 1 - AND p.proname = 'hmac_256' - AND (p.proargtypes[0] = enc_oid OR p.proargtypes[0] = jsonb_oid)) - -- Field-level JSONB extractors (#205): inlinable SQL replacements for - -- the previous plpgsql bodies. Inlining lets the planner fold the - -- `jsonb_array_elements(...) WHERE elem->>'s' = selector` body into - -- the calling query, eliminating per-row function call overhead on - -- large ste_vec scans. - OR (p.pronargs = 2 - AND p.proname IN ('jsonb_path_query', - 'jsonb_path_query_first', - 'jsonb_path_exists')) - -- Inner ORE-block comparison helpers backing the `<`, `<=`, `>`, `>=` - -- operators on `eql_v2.ore_block_u64_8_256`. The outer operators on - -- `eql_v2_encrypted` inline to `ore_block(a) <op> ore_block(b)`, and - -- PG only carries the inlined form through to index matching if the - -- inner operator function is also inlinable (no SET, IMMUTABLE). - -- Pinning these would prevent the planner from structurally matching - -- predicates against a functional `eql_v2.ore_block_u64_8_256(col)` - -- index. The inner functions are deterministic comparisons of - -- composite type bytes, declared IMMUTABLE STRICT PARALLEL SAFE. - OR (p.pronargs = 2 - AND p.proname IN ('ore_block_u64_8_256_eq', 'ore_block_u64_8_256_neq', - 'ore_block_u64_8_256_lt', 'ore_block_u64_8_256_lte', - 'ore_block_u64_8_256_gt', 'ore_block_u64_8_256_gte')) - -- Hash operator class FUNCTION 1: called once per row by HashAggregate, - -- hash joins, DISTINCT. Inlinable SQL avoids the per-row plpgsql - -- interpreter overhead — without this, `GROUP BY value` on - -- `eql_v2_encrypted` at 1M rows degrades super-linearly because the - -- plpgsql cost compounds with HashAggregate work_mem spillage. - OR (p.pronargs = 1 - AND p.proname = 'hash_encrypted' - AND p.proargtypes[0] = enc_oid) - -- Consolidated ORE-CLLW extractor (U-006). Inlinable SQL — pinning - -- would silently undo it and prevent the planner from folding - -- `eql_v2.ore_cllw(col)` calls into the calling query. The - -- `compare_ore_cllw_term` comparator stays plpgsql by design (per-byte - -- protocol can't be expressed as a single inlinable SELECT), so it is - -- NOT on this list. The (jsonb) form is a RHS-parameter helper for - -- comparisons against literal jsonb; the (eql_v2.ste_vec_entry) form - -- is the typed extractor for the result of `col -> '<selector>'`. - OR (p.pronargs = 1 - AND p.proname IN ('ore_cllw', 'has_ore_cllw') - AND (p.proargtypes[0] = jsonb_oid OR p.proargtypes[0] = entry_oid)) - -- Typed HMAC extractor on a ste_vec entry (#219 strict separation). - -- Same rationale as `ore_cllw(ste_vec_entry)` — must inline so - -- `eql_v2.hmac_256(col -> 'sel')` folds into the calling query and - -- matches a functional hash index built on the same expression. - OR (p.pronargs = 1 - AND p.proname IN ('hmac_256', 'has_hmac_256', 'selector') - AND p.proargtypes[0] = entry_oid) - -- `eql_v2.ste_vec_entry × eql_v2.ste_vec_entry` operators (#219). - -- Inline to `hmac_256(a) = hmac_256(b)` (equality) or - -- `ore_cllw(a) <op> ore_cllw(b)` (ordering); both chains must remain - -- unpinned for functional-index match through extractor form. - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - 'eq', 'neq', 'lt', 'lte', 'gt', 'gte') - AND p.proargtypes[0] = entry_oid AND p.proargtypes[1] = entry_oid) - -- Inner ORE-CLLW comparison helpers backing the `<`, `<=`, `=`, - -- `>=`, `>`, `<>` operators on `eql_v2.ore_cllw` (the composite - -- type, registered via `eql_v2.ore_cllw_ops` opclass — #221). Same - -- precedent as the `ore_block_u64_8_256_*` helpers above: PG only - -- carries the inlined operator wrapper through to functional-index - -- match if the inner backing function is also inlinable. Pinning - -- these would break the index match for `ORDER BY eql_v2.ore_cllw - -- (value -> '<selector>'::text)` and the matching `WHERE` form. - OR (p.pronargs = 2 - AND p.proname IN ('ore_cllw_eq', 'ore_cllw_neq', - 'ore_cllw_lt', 'ore_cllw_lte', - 'ore_cllw_gt', 'ore_cllw_gte')) - -- `->` selector lookup: inlinable SQL post the type flip - -- (returns `eql_v2.ste_vec_entry`). Must stay unpinned so the - -- planner can fold `col -> '<selector>'` into the calling query - -- — without this, the chained recipe - -- `WHERE col -> 'sel' = $1::ste_vec_entry` would not match a - -- functional hash index on `eql_v2.eq_term(col -> 'sel')`. - OR (p.proname = '->' - AND p.pronargs = 2 - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = text_oid - OR p.proargtypes[1] = enc_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'int4'))) - -- XOR-aware equality term extractor on a ste_vec entry. Must - -- inline so `eql_v2.eq_term(col -> 'sel')` folds into the - -- calling query and matches a functional hash index built on - -- the same expression. - OR (p.pronargs = 1 - AND p.proname = 'eq_term' - AND p.proargtypes[0] = entry_oid) - -- Type-safe `@>` / `<@` overloads with typed needles - -- (`stevec_query`, `ste_vec_entry`). Inline to the existing - -- `ste_vec_contains` machinery — must stay unpinned to engage - -- the GIN index on `eql_v2.ste_vec(col)` structurally for - -- bare-form containment. - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = entry_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[1] = enc_oid - AND (p.proargtypes[0] = entry_oid - OR p.proargtypes[0] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - ); - - FOR fn_oid IN - SELECT p.oid - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - -- Only normal functions ('f') and window functions ('w') accept - -- ALTER FUNCTION ... SET. Aggregates ('a') would be rejected by - -- ALTER ROUTINE/FUNCTION, and procedures ('p') would need ALTER - -- PROCEDURE. The 3 affected aggregates (min, max, grouped_value) - -- are allowlisted in splinter. - AND p.prokind IN ('f', 'w') - AND NOT EXISTS ( - SELECT 1 FROM pg_catalog.unnest(coalesce(p.proconfig, '{}'::text[])) c - WHERE c LIKE 'search_path=%' - ) - AND NOT (p.oid = ANY (coalesce(inline_critical_oids, '{}'::oid[]))) - LOOP - -- oid::regprocedure renders as `schema.name(argtype, argtype)` and is a - -- valid target for ALTER FUNCTION regardless of caller search_path. - EXECUTE pg_catalog.format( - 'ALTER FUNCTION %s SET search_path = pg_catalog, extensions, public', - fn_oid::regprocedure - ); - END LOOP; -END $$; diff --git a/packages/cli/src/sql/cipherstash-encrypt-supabase.sql b/packages/cli/src/sql/cipherstash-encrypt-supabase.sql deleted file mode 100644 index e3a0e4a94..000000000 --- a/packages/cli/src/sql/cipherstash-encrypt-supabase.sql +++ /dev/null @@ -1,7434 +0,0 @@ ---! @file schema.sql ---! @brief EQL v2 schema creation ---! ---! Creates the eql_v2 schema which contains all Encrypt Query Language ---! functions, types, and tables. Drops existing schema if present to ---! support clean reinstallation. ---! ---! @warning DROP SCHEMA CASCADE will remove all objects in the schema ---! @note All EQL objects (functions, types, tables) reside in eql_v2 schema - ---! @brief Drop existing EQL v2 schema ---! @warning CASCADE will drop all dependent objects -DROP SCHEMA IF EXISTS eql_v2 CASCADE; - ---! @brief Create EQL v2 schema ---! @note All EQL functions and types will be created in this schema -CREATE SCHEMA eql_v2; - ---! @brief HMAC-SHA256 index term type ---! ---! Domain type representing HMAC-SHA256 hash values. ---! Used for exact-match encrypted searches via the 'unique' index type. ---! The hash is stored in the 'hm' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.hmac_256 AS text; - ---! @file src/ste_vec/types.sql ---! @brief Domain type for individual STE-vec entries ---! ---! Defines `eql_v2.ste_vec_entry` as a DOMAIN over `jsonb` constrained to the ---! shape of a single element inside an `sv` array — a JSON object that ---! carries at minimum a selector field (`s`). This is the type returned by ---! the `->` operator on `eql_v2_encrypted` (a single sv element extracted by ---! selector) and the type accepted by sv-element extractors such as ---! `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and ---! `eql_v2.hmac_256(eql_v2.ste_vec_entry)`. ---! ---! Why a separate type. Before #219, the `(eql_v2_encrypted)` overloads of ---! sv-element extractors read fields like `oc` off the root `data` jsonb, ---! which is misleading: a root `EncryptedPayload` or `SteVecPayload` (the ---! shapes that an actual `eql_v2_encrypted` column value carries) never has ---! `oc` at the root. The previous pattern only worked because the `->` ---! operator merged ste-vec entry fields into a fake root-shaped payload ---! before the extractor ran. This domain type makes the distinction ---! explicit: `eql_v2_encrypted` is the root shape; `eql_v2.ste_vec_entry` ---! is the per-entry shape; extractors are typed accordingly. ---! ---! @note The CHECK constraint reflects the cipherstash-suite emission ---! contract: ---! - `s` (selector — column-name HMAC) and `c` (ciphertext) are ---! emitted on every sv element. ---! - Each sv element carries **exactly one** of `hm` (HMAC-256, for ---! hash-equality queries) or `oc` (CLLW ORE, for ordered queries) ---! — they are mutually exclusive. A given selector / field is ---! configured for one mode or the other; the crypto layer emits ---! the corresponding term and only that term. ---! Other fields (`a` for array marker, etc.) are allowed but not ---! required. ---! ---! @see src/operators/->.sql ---! @see src/ore_cllw/functions.sql ---! @see src/hmac_256/functions.sql -CREATE DOMAIN eql_v2.ste_vec_entry AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 's' - AND VALUE ? 'c' - AND (VALUE ? 'hm') <> (VALUE ? 'oc') - ); - - ---! @brief Domain type for an STE-vec containment needle ---! ---! `eql_v2.stevec_query` is a query-shaped sv payload: a top-level ---! `{"sv": [...]}` object whose elements carry selector + index ---! terms but **never** a ciphertext (`c`) field. Containment (`@>`) ---! against an `eql_v2_encrypted` column is structurally typed ---! through this domain so the call site reads as "match against an ---! sv query", not "compare two encrypted values". ---! ---! Compared to `eql_v2.ste_vec_entry` (single sv element with `s`, ---! `c`, and `hm` XOR `oc`), `stevec_query` is the wrapping ---! `{"sv": [...]}` payload: it forbids `c` on every element but ---! otherwise keeps the same per-element contract — each element must ---! carry a selector `s` and exactly one deterministic term (`hm` XOR ---! `oc`). This mirrors the `SteVecQueryElement` JSON schema and stops ---! selector-only needles (e.g. `{"sv":[{"s":"x"}]}`) from casting and ---! then matching every row through the bare `jsonb @>` implementation. ---! The implementation of `ste_vec_contains` ignores `c` either way, ---! but typing the needle as `stevec_query` documents the contract at ---! the API surface. ---! ---! @note Constructing a `stevec_query` literal from inline JSON works ---! via the standard DOMAIN cast: ---! `'{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query` ---! Casting an `eql_v2_encrypted` value strips `c` fields from ---! each sv element — see `eql_v2.to_stevec_query`. ---! ---! @see eql_v2.to_stevec_query ---! @see src/operators/@>.sql -CREATE DOMAIN eql_v2.stevec_query AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 'sv' - AND jsonb_typeof(VALUE -> 'sv') = 'array' - -- No element may carry a ciphertext (`c`) — this is a query, not a value. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.c))'::jsonpath) - -- Every element must carry a selector (`s`) ... - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.s))'::jsonpath) - -- ... and exactly one deterministic term — `hm` XOR `oc` — matching - -- the `ste_vec_entry` emission contract and the `SteVecQueryElement` - -- JSON schema. Rejects selector-only needles that would otherwise - -- cast and then match every row via the bare `jsonb @>` body. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.hm) && exists(@.oc))'::jsonpath) - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.hm) && !exists(@.oc))'::jsonpath) - ); - - ---! @brief Convert an `eql_v2_encrypted` to a `stevec_query` needle ---! ---! Normalises each sv element down to the matching-relevant fields: ---! `s` (selector) plus exactly one of `hm` / `oc`. Other fields ---! (`c` ciphertext, `a` array marker, `i`/`v` envelope metadata, anything ---! else cipherstash-client might emit) are stripped. This is the ---! canonical needle shape for `@>` containment — matching the contract ---! that containment compares by selector + deterministic term and ---! ignores everything else. ---! ---! Designed for use as a functional GIN index expression: a single ---! `GIN (eql_v2.to_stevec_query(col)::jsonb jsonb_path_ops)` index ---! covers containment queries against any selector (both hm-bearing ---! and oc-bearing — XOR-aware), and the typed `@>` overloads inline ---! to a native `jsonb @>` on the same expression so the planner ---! engages Bitmap Index Scan structurally. ---! ---! @param e eql_v2_encrypted Source encrypted payload ---! @return eql_v2.stevec_query Query-shaped needle, sv elements ---! normalised to `{s, hm}` or `{s, oc}`. ---! ---! @example ---! -- Functional GIN index — canonical containment recipe ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! ---! -- Cross-row containment ---! SELECT a.* ---! FROM docs a, docs b ---! WHERE a.encrypted_doc @> b.encrypted_doc::eql_v2.stevec_query ---! AND b.id = 42; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2.to_stevec_query(e eql_v2_encrypted) - RETURNS eql_v2.stevec_query - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT jsonb_build_object( - 'sv', - coalesce( - (SELECT jsonb_agg( - jsonb_strip_nulls( - jsonb_build_object( - 's', elem -> 's', - 'hm', elem -> 'hm', - 'oc', elem -> 'oc' - ) - ) - ) - FROM jsonb_array_elements((e).data -> 'sv') AS elem), - '[]'::jsonb - ) - )::eql_v2.stevec_query -$$; - -CREATE CAST (eql_v2_encrypted AS eql_v2.stevec_query) - WITH FUNCTION eql_v2.to_stevec_query - AS ASSIGNMENT; - ---! @brief Composite type for encrypted column data ---! ---! Core type used for all encrypted columns in EQL. Stores encrypted data as JSONB ---! with the following structure: ---! - `c`: ciphertext (base64-encoded encrypted value) ---! - `i`: index terms (searchable metadata for encrypted searches) ---! - `k`: key ID (identifier for encryption key) ---! - `m`: metadata (additional encryption metadata) ---! ---! Created in public schema to persist independently of eql_v2 schema lifecycle. ---! Customer data columns use this type, so it must not be dropped if data exists. ---! ---! @note DO NOT DROP this type unless absolutely certain no encrypted data uses it ---! @see eql_v2.ciphertext ---! @see eql_v2.meta_data ---! @see eql_v2.add_column -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_encrypted') THEN - CREATE TYPE public.eql_v2_encrypted AS ( - data jsonb - ); - END IF; - END -$$; - - - - - - - - - - ---! @brief Bloom filter index term type ---! ---! Domain type representing Bloom filter bit arrays stored as smallint arrays. ---! Used for pattern-match encrypted searches via the 'match' index type. ---! The filter is stored in the 'bf' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2."~~" ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.bloom_filter AS smallint[]; - - - ---! @brief ORE block term type for Order-Revealing Encryption ---! ---! Composite type representing a single ORE (Order-Revealing Encryption) block term. ---! Stores encrypted data as bytea that enables range comparisons without decryption. ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE TYPE eql_v2.ore_block_u64_8_256_term AS ( - bytes bytea -); - - ---! @brief ORE block index term type for range queries ---! ---! Composite type containing an array of ORE block terms. Used for encrypted ---! range queries via the 'ore' index type. The array is stored in the 'ob' field ---! of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_block_u64_8_256_terms ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_block_u64_8_256 AS ( - terms eql_v2.ore_block_u64_8_256_term[] -); - ---! @brief Extract HMAC-SHA256 index term from JSONB payload ---! ---! Extracts the HMAC-SHA256 hash value from the 'hm' field of an encrypted ---! data payload. Inlinable single-statement SQL — the planner can fold this ---! into the calling query so functional hash indexes built on ---! `eql_v2.hmac_256(col)` engage structurally. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @note Returns NULL when the payload lacks `hm`. Callers that need to ---! surface misconfiguration loudly should use ---! `eql_v2.hash_encrypted` (`GROUP BY` / `DISTINCT` / hash joins) ---! which raises with a clear message when `hm` is missing. ---! ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare_hmac_256 ---! @see eql_v2.hash_encrypted -CREATE FUNCTION eql_v2.hmac_256(val jsonb) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (val ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if JSONB payload contains HMAC-SHA256 index term ---! ---! Tests whether the encrypted data payload includes an 'hm' field, ---! indicating an HMAC-SHA256 hash is available for exact-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'hm' field is present and non-null ---! ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.has_hmac_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'hm' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains HMAC-SHA256 index term ---! ---! Tests whether an encrypted column value includes an HMAC-SHA256 hash ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if HMAC-SHA256 hash is present ---! ---! @see eql_v2.has_hmac_256(jsonb) -CREATE FUNCTION eql_v2.has_hmac_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_hmac_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract HMAC-SHA256 index term from encrypted column value ---! ---! Extracts the HMAC-SHA256 hash from an encrypted column value. Inlinable ---! single-statement SQL — see the jsonb overload for the rationale. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @see eql_v2.hmac_256(jsonb) -CREATE FUNCTION eql_v2.hmac_256(val eql_v2_encrypted) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ((val).data ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Extract HMAC-SHA256 index term from a ste_vec entry ---! ---! Extracts the HMAC from the `hm` field of an `sv` element extracted via ---! the `->` operator. Inlinable. The recipe for field-level equality on ---! encrypted JSON is: ---! ---! @example ---! -- Functional hash index ---! CREATE INDEX ON users USING hash (eql_v2.hmac_256(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.hmac_256 HMAC value, or NULL when `hm` is absent ---! ---! @see eql_v2.has_hmac_256 ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.hmac_256(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (entry ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if a ste_vec entry contains an HMAC-SHA256 index term ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `hm` field is present and non-null -CREATE FUNCTION eql_v2.has_hmac_256(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'hm' IS NOT NULL -$$; - - --- AUTOMATICALLY GENERATED FILE - ---! @file common.sql ---! @brief Common utility functions ---! ---! Provides general-purpose utility functions used across EQL: ---! - Constant-time bytea comparison for security ---! - JSONB to bytea array conversion ---! - Logging helpers for debugging and testing - - ---! @brief Constant-time comparison of bytea values ---! @internal ---! ---! Compares two bytea values in constant time to prevent timing attacks. ---! Always checks all bytes even after finding differences, maintaining ---! consistent execution time regardless of where differences occur. ---! ---! @param a bytea First value to compare ---! @param b bytea Second value to compare ---! @return boolean True if values are equal ---! ---! @note Returns false immediately if lengths differ (length is not secret) ---! @note Used for secure comparison of cryptographic values -CREATE FUNCTION eql_v2.bytea_eq(a bytea, b bytea) RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result boolean; - differing bytea; -BEGIN - - -- Check if the bytea values are the same length - IF LENGTH(a) != LENGTH(b) THEN - RETURN false; - END IF; - - -- Compare each byte in the bytea values - result := true; - FOR i IN 1..LENGTH(a) LOOP - IF SUBSTRING(a FROM i FOR 1) != SUBSTRING(b FROM i FOR 1) THEN - result := result AND false; - END IF; - END LOOP; - - RETURN result; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Convert JSONB hex array to bytea array ---! @internal ---! ---! Converts a JSONB array of hex-encoded strings into a PostgreSQL bytea array. ---! Used for deserializing binary data (like ORE terms) from JSONB storage. ---! ---! @param jsonb JSONB array of hex-encoded strings ---! @return bytea[] Array of decoded binary values ---! ---! @note Returns NULL if input is JSON null ---! @note Each array element is hex-decoded to bytea -CREATE FUNCTION eql_v2.jsonb_array_to_bytea_array(val jsonb) -RETURNS bytea[] - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms_arr bytea[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(decode(value::text, 'hex')::bytea) - INTO terms_arr - FROM jsonb_array_elements_text(val) AS value; - - RETURN terms_arr; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message for debugging ---! ---! Convenience function to emit log messages during testing and debugging. ---! Uses RAISE NOTICE to output messages to PostgreSQL logs. ---! ---! @param text Message to log ---! ---! @note Primarily used in tests and development ---! @see eql_v2.log(text, text) for contextual logging -CREATE FUNCTION eql_v2.log(s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] %', s; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message with context ---! ---! Overload of log function that includes context label for better ---! log organization during testing. ---! ---! @param ctx text Context label (e.g., test name, module name) ---! @param s text Message to log ---! ---! @note Format: "[LOG] {ctx} {message}" ---! @see eql_v2.log(text) -CREATE FUNCTION eql_v2.log(ctx text, s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] % %', ctx, s; -END; -$$ LANGUAGE plpgsql; - ---! @brief CLLW ORE index term type for STE-vec range queries ---! ---! Composite type for CLLW (Copyless Logarithmic Width) Order-Revealing ---! Encryption. The ciphertext is stored in the `oc` field of encrypted data ---! payloads (Standard-mode `ste_vec` elements). Used by `eql_v2.compare` and ---! the range operators (`<`, `<=`, `>`, `>=`) when the payload carries an ---! `oc` term. ---! ---! The wire-format `oc` value is a hex string with a leading domain-tag byte ---! (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext. The ---! decoded `bytes` field on this composite carries the full byte string ---! including the tag — the comparator is variable-length capable, so numeric ---! and string values within the same column are ordered correctly: the ---! domain tag separates the two ranges (numeric < string) and the ---! within-domain comparison falls through to the CLLW per-byte protocol. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_cllw ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_cllw AS ( - bytes bytea -); - ---! @file crypto.sql ---! @brief PostgreSQL pgcrypto extension enablement ---! ---! Enables the pgcrypto extension which provides cryptographic functions ---! used by EQL for hashing and other cryptographic operations. ---! ---! Installs pgcrypto into the `extensions` schema (Supabase convention) to ---! avoid the `extension_in_public` lint. Every EQL function that uses ---! pgcrypto has `pg_catalog, extensions, public` on its `search_path`, so a ---! pre-existing install in `public` keeps working — and a pre-existing ---! install anywhere else will be rejected at install time rather than ---! failing later inside an encrypted comparison. ---! ---! @note pgcrypto provides functions like digest(), hmac(), gen_random_bytes() ---! @note If pgcrypto is already installed in `public`, EQL works but emits ---! a NOTICE recommending `ALTER EXTENSION pgcrypto SET SCHEMA extensions`. ---! @note If pgcrypto is already installed in any other schema, install ---! fails. Relocate it first with `ALTER EXTENSION pgcrypto SET SCHEMA ---! extensions` (or move it into `public` if compatibility with other ---! consumers requires it). - ---! @brief Create extensions schema (Supabase convention) -CREATE SCHEMA IF NOT EXISTS extensions; - ---! @brief Enable pgcrypto extension and validate its schema -DO $$ -DECLARE - pgcrypto_schema name; -BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') THEN - CREATE EXTENSION pgcrypto WITH SCHEMA extensions; - END IF; - - SELECT n.nspname INTO pgcrypto_schema - FROM pg_extension e - JOIN pg_namespace n ON n.oid = e.extnamespace - WHERE e.extname = 'pgcrypto'; - - IF pgcrypto_schema = 'extensions' THEN - -- expected location, nothing to say - NULL; - ELSIF pgcrypto_schema = 'public' THEN - RAISE NOTICE - 'pgcrypto is installed in the `public` schema. EQL works against this layout, ' - 'but Supabase splinter will flag it as `extension_in_public`. Move it with: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions'; - ELSE - RAISE EXCEPTION - 'pgcrypto is installed in schema `%`, which is not on the EQL function search_path ' - '(pg_catalog, extensions, public). EQL cryptographic operations would fail at ' - 'runtime. Relocate the extension before installing EQL: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions', - pgcrypto_schema; - END IF; -END $$; - ---! @brief Extract ciphertext from encrypted JSONB value ---! ---! Extracts the ciphertext (c field) from a raw JSONB encrypted value. ---! The ciphertext is the base64-encoded encrypted data. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if 'c' field is not present in JSONB ---! ---! @example ---! -- Extract ciphertext from JSONB literal ---! SELECT eql_v2.ciphertext('{"c":"AQIDBA==","i":{"unique":"..."}}'::jsonb); ---! ---! @see eql_v2.ciphertext(eql_v2_encrypted) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'c' THEN - RETURN val->>'c'; - END IF; - RAISE 'Expected a ciphertext (c) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract ciphertext from encrypted column value ---! ---! Extracts the ciphertext from an encrypted column value. Convenience ---! overload that unwraps eql_v2_encrypted type and delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if encrypted value is malformed ---! ---! @example ---! -- Extract ciphertext from encrypted column ---! SELECT eql_v2.ciphertext(encrypted_email) FROM users; ---! ---! @see eql_v2.ciphertext(jsonb) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.ciphertext(val.data); -$$; - ---! @brief State transition function for grouped_value aggregate ---! @internal ---! ---! Returns the first non-null value encountered. Used as state function ---! for the grouped_value aggregate to select first value in each group. ---! ---! @param $1 JSONB Accumulated state (first non-null value found) ---! @param $2 JSONB New value from current row ---! @return JSONB First non-null value (state or new value) ---! ---! @see eql_v2.grouped_value -CREATE FUNCTION eql_v2._first_grouped_value(jsonb, jsonb) -RETURNS jsonb -AS $$ - SELECT COALESCE($1, $2); -$$ LANGUAGE sql IMMUTABLE; - ---! @brief Return first non-null encrypted value in a group ---! ---! Aggregate function that returns the first non-null encrypted value ---! encountered within a GROUP BY clause. Useful for deduplication or ---! selecting representative values from grouped encrypted data. ---! ---! @param input JSONB Encrypted values to aggregate ---! @return JSONB First non-null encrypted value in group ---! ---! @example ---! -- Get first email per user group ---! SELECT user_id, eql_v2.grouped_value(encrypted_email) ---! FROM user_emails ---! GROUP BY user_id; ---! ---! -- Deduplicate encrypted values ---! SELECT DISTINCT ON (user_id) ---! user_id, ---! eql_v2.grouped_value(encrypted_ssn) as primary_ssn ---! FROM user_records ---! GROUP BY user_id; ---! ---! @see eql_v2._first_grouped_value -CREATE AGGREGATE eql_v2.grouped_value(jsonb) ( - SFUNC = eql_v2._first_grouped_value, - STYPE = jsonb -); - ---! @brief Add validation constraint to encrypted column ---! ---! Adds a CHECK constraint to ensure column values conform to encrypted data ---! structure. Constraint uses eql_v2.check_encrypted to validate format. ---! Called automatically by eql_v2.add_column. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to constrain ---! @return Void ---! ---! @example ---! -- Manually add constraint (normally done by add_column) ---! SELECT eql_v2.add_encrypted_constraint('users', 'encrypted_email'); ---! ---! -- Resulting constraint: ---! -- ALTER TABLE users ADD CONSTRAINT eql_v2_encrypted_check_encrypted_email ---! -- CHECK (eql_v2.check_encrypted(encrypted_email)); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_encrypted_constraint -CREATE FUNCTION eql_v2.add_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I ADD CONSTRAINT eql_v2_encrypted_constraint_%I_%I CHECK (eql_v2.check_encrypted(%I))', table_name, table_name, column_name, column_name); - EXCEPTION - WHEN duplicate_table THEN - WHEN duplicate_object THEN - RAISE NOTICE 'Constraint `eql_v2_encrypted_constraint_%_%` already exists, skipping', table_name, column_name; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove validation constraint from encrypted column ---! ---! Removes the CHECK constraint that validates encrypted data structure. ---! Called automatically by eql_v2.remove_column. Uses IF EXISTS to avoid ---! errors if constraint doesn't exist. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to unconstrain ---! @return Void ---! ---! @example ---! -- Manually remove constraint (normally done by remove_column) ---! SELECT eql_v2.remove_encrypted_constraint('users', 'encrypted_email'); ---! ---! @see eql_v2.remove_column ---! @see eql_v2.add_encrypted_constraint -CREATE FUNCTION eql_v2.remove_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I DROP CONSTRAINT IF EXISTS eql_v2_encrypted_constraint_%I_%I', table_name, table_name, column_name); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract metadata from encrypted JSONB value ---! ---! Extracts index terms (i) and version (v) from a raw JSONB encrypted value. ---! Returns metadata object containing searchable index terms without ciphertext. ---! ---! @param jsonb containing encrypted EQL payload ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Extract metadata to inspect index terms ---! SELECT eql_v2.meta_data('{"c":"...","i":{"unique":"abc123"},"v":1}'::jsonb); ---! -- Returns: {"i":{"unique":"abc123"},"v":1} ---! ---! @see eql_v2.meta_data(eql_v2_encrypted) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val jsonb) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT jsonb_build_object('i', val->'i', 'v', val->'v'); -$$; - ---! @brief Extract metadata from encrypted column value ---! ---! Extracts index terms and version from an encrypted column value. ---! Convenience overload that unwraps eql_v2_encrypted type and ---! delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Inspect index terms for encrypted column ---! SELECT user_id, eql_v2.meta_data(encrypted_email) as email_metadata ---! FROM users; ---! ---! @see eql_v2.meta_data(jsonb) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.meta_data(val.data); -$$; - - - - ---! @brief Convert JSONB to encrypted type ---! ---! Wraps a JSONB encrypted payload into the eql_v2_encrypted composite type. ---! Used internally for type conversions and operator implementations. ---! ---! @param jsonb JSONB encrypted payload with structure: {"c": "...", "i": {...}, "k": "...", "v": "2"} ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note This is primarily used for implicit casts in operator expressions ---! @see eql_v2.to_jsonb -CREATE FUNCTION eql_v2.to_encrypted(data jsonb) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT ROW(data)::public.eql_v2_encrypted; -$$; - - ---! @brief Implicit cast from JSONB to encrypted type ---! ---! Enables PostgreSQL to automatically convert JSONB values to eql_v2_encrypted ---! in assignment contexts and comparison operations. ---! ---! @see eql_v2.to_encrypted(jsonb) -CREATE CAST (jsonb AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(jsonb) AS ASSIGNMENT; - - ---! @brief Convert text to encrypted type ---! ---! Parses a text representation of encrypted JSONB payload and wraps it ---! in the eql_v2_encrypted composite type. ---! ---! @param text Text representation of JSONB encrypted payload ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note Delegates to eql_v2.to_encrypted(jsonb) after parsing text as JSON ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_encrypted(data text) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.to_encrypted(data::jsonb); -$$; - - ---! @brief Implicit cast from text to encrypted type ---! ---! Enables PostgreSQL to automatically convert text JSON strings to eql_v2_encrypted ---! in assignment contexts. ---! ---! @see eql_v2.to_encrypted(text) -CREATE CAST (text AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(text) AS ASSIGNMENT; - - - ---! @brief Convert encrypted type to JSONB ---! ---! Extracts the underlying JSONB payload from an eql_v2_encrypted composite type. ---! Useful for debugging or when raw encrypted payload access is needed. ---! ---! @param e eql_v2_encrypted Encrypted value to unwrap ---! @return jsonb Raw JSONB encrypted payload ---! ---! @note Returns the raw encrypted structure including ciphertext and index terms ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_jsonb(e public.eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT e.data; -$$; - ---! @brief Implicit cast from encrypted type to JSONB ---! ---! Enables PostgreSQL to automatically extract the JSONB payload from ---! eql_v2_encrypted values in assignment contexts. ---! ---! @see eql_v2.to_jsonb(eql_v2_encrypted) -CREATE CAST (public.eql_v2_encrypted AS jsonb) - WITH FUNCTION eql_v2.to_jsonb(public.eql_v2_encrypted) AS ASSIGNMENT; - - - - - ---! @brief Compare two encrypted values using HMAC-SHA256 index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their HMAC-SHA256 hash index terms. Used internally by the equality operator (=) ---! for exact-match queries without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Comparison uses underlying text type ordering of HMAC-SHA256 hashes ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2."=" -CREATE FUNCTION eql_v2.compare_hmac_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.hmac_256; - b_term eql_v2.hmac_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_hmac_256(a) THEN - a_term = eql_v2.hmac_256(a); - END IF; - - IF eql_v2.has_hmac_256(b) THEN - b_term = eql_v2.hmac_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - -- Using the underlying text type comparison - IF a_term = b_term THEN - RETURN 0; - END IF; - - IF a_term < b_term THEN - RETURN -1; - END IF; - - IF a_term > b_term THEN - RETURN 1; - END IF; - - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract CLLW ORE index term from a ste_vec entry ---! ---! Returns the CLLW ORE ciphertext from the `oc` field of an `sv` element. ---! `oc` is **only ever present on a `SteVecElement`** in the v2.3 payload ---! shape — never at the root of an `eql_v2_encrypted` column value — so the ---! type signature accepts `eql_v2.ste_vec_entry` directly. Callers must ---! extract first: `eql_v2.ore_cllw(col -> '<selector>')`. ---! ---! Inlinable single-statement SQL — the planner folds the body into the ---! calling query so the extractor disappears at planning time. Functional ---! btree index match on this extractor requires the `eql_v2.ore_cllw_ops` ---! opclass (installed automatically by the main / protect variants; absent ---! in the supabase variant). ---! ---! **Missing-`oc` semantics**: when the `oc` field is absent, returns a ---! SQL-level NULL (not a composite with NULL bytes). Btree's standard ---! NULL handling then filters those rows from range queries: they don't ---! match `WHERE ore_cllw(col) <op> $1`, they sort at the NULLS LAST end ---! of `ORDER BY ore_cllw(col)`, and they never reach the comparator. ---! This avoids the btree FUNCTION 1 contract violation that ---! `(bytes => NULL)` would otherwise cause (`compare_ore_cllw_term` ---! must return non-NULL int for non-NULL composite inputs). ---! ---! Callers needing a loud RAISE on missing `oc` should check ---! `eql_v2.has_ore_cllw(entry)` first. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. ---! ---! @see eql_v2.has_ore_cllw ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN entry ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(entry ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Extract CLLW ORE index term from raw jsonb (RHS parameter helper) ---! ---! Companion overload for `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` that ---! accepts a raw `jsonb` value. Intended for the right-hand side of ---! comparisons where the caller binds a literal/parameter jsonb representing ---! a single ste_vec entry: `... < eql_v2.ore_cllw($1::jsonb)`. The (jsonb) ---! form skips the domain CHECK constraint so it works for ad-hoc test inputs ---! and for the GenericComparison case in `eql_v2.compare_ore_cllw_term`. ---! ---! Returns SQL-level NULL when the input lacks `oc`, matching the ---! `(ste_vec_entry)` overload's missing-`oc` semantics so a `WHERE ---! ore_cllw(col) < ore_cllw($1::jsonb)` with a malformed query needle ---! evaluates to no rows rather than indexing a NULL-bytes composite. ---! ---! @param val jsonb An object carrying an `oc` field ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. -CREATE FUNCTION eql_v2.ore_cllw(val jsonb) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN val ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(val ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Check if a ste_vec entry contains a CLLW ORE index term ---! ---! Tests whether the entry includes an `oc` field. Inlinable. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `oc` field is present and non-null ---! ---! @see eql_v2.ore_cllw -CREATE FUNCTION eql_v2.has_ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'oc' IS NOT NULL -$$; - - ---! @brief Check if a raw jsonb value contains a CLLW ORE index term ---! ---! Companion to `eql_v2.has_ore_cllw(ste_vec_entry)` for raw jsonb inputs. ---! ---! @param val jsonb An object that may carry an `oc` field ---! @return Boolean True if `oc` field is present and non-null -CREATE FUNCTION eql_v2.has_ore_cllw(val jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT val ->> 'oc' IS NOT NULL -$$; - - ---! @brief CLLW per-byte comparison helper ---! @internal ---! ---! Byte-by-byte comparison implementing the CLLW order-revealing protocol. ---! Used by `eql_v2.compare_ore_cllw_term` for the within-prefix step. The ---! protocol: identify the index of the first differing byte across both ---! inputs; if `(y_byte + 1) == x_byte` modulo 256 at that index, then x > y; ---! otherwise x < y. Equal inputs return 0. ---! ---! Inputs MUST be the same length. The caller (`compare_ore_cllw_term`) ---! guarantees this by passing equal-length prefixes. ---! ---! @par Soft constant-time intent ---! Plpgsql is not a constant-time environment — the interpreter, `SUBSTRING`, ---! `get_byte`, and the SQL bytea representation all leak timing in ways we ---! can't control from here. Still, the loop deliberately walks every byte ---! (no `EXIT` on first difference) and the rotation check uses a bitmask ---! (`& 255`) instead of `% 256` so that what little timing structure plpgsql ---! does expose is independent of the position and value of the differing ---! byte. This is hardening intent, not a guarantee. ---! ---! Stays `LANGUAGE plpgsql` — the per-byte loop can't be expressed as a ---! single inlinable SQL expression. This is the architectural reason ORE ---! CLLW needs a custom operator class for index match, where OPE does not. ---! ---! @param a Bytea First CLLW ciphertext slice ---! @param b Bytea Second CLLW ciphertext slice ---! @return Integer -1, 0, or 1 ---! @throws Exception if inputs are different lengths ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.compare_ore_cllw_term_bytes(a bytea, b bytea) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - i INT; - first_diff INT := 0; -BEGIN - - len_a := LENGTH(a); - len_b := LENGTH(b); - - IF len_a != len_b THEN - RAISE EXCEPTION 'ore_cllw index terms are not the same length'; - END IF; - - -- Walk every byte, even after a difference is found. Record only the - -- index of the first difference (1-based; 0 means "no difference"). - -- Avoids an early `EXIT` whose presence is itself a timing signal. - FOR i IN 1..len_a LOOP - IF first_diff = 0 AND get_byte(a, i - 1) != get_byte(b, i - 1) THEN - first_diff := i; - END IF; - END LOOP; - - IF first_diff = 0 THEN - RETURN 0; - END IF; - - -- Bitmask instead of `% 256` — the modulo's operand is a power of two - -- so the two are arithmetically equivalent, but `& 255` is a single - -- machine instruction with no division-related timing variance. - IF ((get_byte(b, first_diff - 1) + 1) & 255) = get_byte(a, first_diff - 1) THEN - RETURN 1; - ELSE - RETURN -1; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Variable-length CLLW ORE term comparison ---! @internal ---! ---! Three-way comparison of two CLLW ORE ciphertext terms of potentially ---! different lengths. Compares the shared prefix via the CLLW per-byte ---! protocol; on equal prefixes, the shorter input sorts first. ---! ---! Handles both numeric (Standard-mode 65-byte CLLW outputs from the u64 ---! variant) and string (variable-length CLLW outputs) by virtue of the ---! domain-tag byte being the first byte of `bytes`. A numeric/string pair ---! differs at byte 0 (`0x00` vs `0x01`), which the CLLW rule resolves ---! correctly to numeric < string. ---! ---! Stays `LANGUAGE plpgsql` because it dispatches to ---! `compare_ore_cllw_term_bytes`, which can't be inlined. ---! ---! @par Null handling — btree FUNCTION 1 contract ---! PostgreSQL's btree filters NULL composites at the row level, so this ---! function should never be called with `a IS NULL` or `b IS NULL` under ---! normal operation. The leading IS-NULL guard returns NULL defensively ---! to cover edge cases (e.g., a non-index `ORDER BY` or `WHERE` path ---! that bypasses the opclass). ---! ---! A composite that is non-NULL but whose `bytes` field is NULL is a ---! contract violation: btree expects FUNCTION 1 to return a non-NULL ---! integer for non-NULL composite inputs. The extractor overloads of ---! `eql_v2.ore_cllw` are designed to return SQL NULL (not `ROW(NULL)`) ---! when the source payload lacks `oc`, so a NULL-bytes composite should ---! only arise from a hand-crafted literal or a future field addition to ---! the composite type. Raise loudly to surface the bug instead of ---! producing silent misordering downstream. ---! ---! @param a eql_v2.ore_cllw First term ---! @param b eql_v2.ore_cllw Second term ---! @return Integer -1, 0, or 1; NULL if either composite is NULL ---! @throws Exception if either composite has a NULL `bytes` field ---! ---! @see eql_v2.compare_ore_cllw_term_bytes ---! @see eql_v2.compare_ore_cllw -CREATE FUNCTION eql_v2.compare_ore_cllw_term(a eql_v2.ore_cllw, b eql_v2.ore_cllw) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - common_len INT; - cmp_result INT; -BEGIN - -- Composite-level NULL: btree's null-handling layer filters these at - -- the row level under normal operation. Returning NULL covers - -- non-index code paths that might still reach here. - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- Non-NULL composite with NULL bytes is a contract violation: btree's - -- FUNCTION 1 must return non-NULL int for non-NULL composite inputs. - -- The extractors return SQL NULL (not ROW(NULL)) on missing `oc`, so - -- reaching here means a hand-crafted literal or a regression in the - -- extractor body. Raise loudly rather than silently misorder. - IF a.bytes IS NULL OR b.bytes IS NULL THEN - RAISE EXCEPTION 'eql_v2.compare_ore_cllw_term: composite has NULL bytes field — extractor invariant violated. Check that the index expression uses eql_v2.ore_cllw(...) and not a hand-crafted ROW(NULL).'; - END IF; - - len_a := LENGTH(a.bytes); - len_b := LENGTH(b.bytes); - - IF len_a = 0 AND len_b = 0 THEN - RETURN 0; - ELSIF len_a = 0 THEN - RETURN -1; - ELSIF len_b = 0 THEN - RETURN 1; - END IF; - - IF len_a < len_b THEN - common_len := len_a; - ELSE - common_len := len_b; - END IF; - - cmp_result := eql_v2.compare_ore_cllw_term_bytes( - SUBSTRING(a.bytes FROM 1 FOR common_len), - SUBSTRING(b.bytes FROM 1 FOR common_len) - ); - - IF cmp_result = -1 THEN - RETURN -1; - ELSIF cmp_result = 1 THEN - RETURN 1; - END IF; - - -- Equal prefixes: shorter sorts first - IF len_a < len_b THEN - RETURN -1; - ELSIF len_a > len_b THEN - RETURN 1; - ELSE - RETURN 0; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Convert JSONB array to ORE block composite type ---! @internal ---! ---! Converts a JSONB array of hex-encoded ORE terms from the CipherStash Proxy ---! payload into the PostgreSQL composite type used for ORE operations. ---! ---! @param val JSONB Array of hex-encoded ORE block terms ---! @return eql_v2.ore_block_u64_8_256 ORE block composite type, or NULL if input is null ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_to_ore_block_u64_8_256(val jsonb) -RETURNS eql_v2.ore_block_u64_8_256 - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms eql_v2.ore_block_u64_8_256_term[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(ROW(b)::eql_v2.ore_block_u64_8_256_term) - INTO terms - FROM unnest(eql_v2.jsonb_array_to_bytea_array(val)) AS b; - - RETURN ROW(terms)::eql_v2.ore_block_u64_8_256; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from JSONB payload ---! ---! Extracts the ORE block array from the 'ob' field of an encrypted ---! data payload. Used internally for range query comparisons. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! @throws Exception if 'ob' field is missing when ore index is expected ---! ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256 -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val jsonb) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(val) THEN - RETURN eql_v2.jsonb_array_to_ore_block_u64_8_256(val->'ob'); - END IF; - RAISE 'Expected an ore index (ob) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from encrypted column value ---! ---! Extracts the ORE block from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains ORE block index term ---! ---! Tests whether the encrypted data payload includes an 'ob' field, ---! indicating an ORE block is available for range queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'ob' field is present and non-null ---! ---! @see eql_v2.ore_block_u64_8_256 -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'ob' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains ORE block index term ---! ---! Tests whether an encrypted column value includes an ORE block ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if ORE block is present ---! ---! @see eql_v2.has_ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Compare two ORE block terms using cryptographic comparison ---! @internal ---! ---! Performs a three-way comparison (returns -1/0/1) of individual ORE block terms ---! using the ORE cryptographic protocol. Compares PRP and PRF blocks to determine ---! ordering without decryption. ---! ---! @param a eql_v2.ore_block_u64_8_256_term First ORE term to compare ---! @param b eql_v2.ore_block_u64_8_256_term Second ORE term to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! @throws Exception if ciphertexts are different lengths ---! ---! @note Uses AES-ECB encryption for bit comparisons per ORE protocol ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_term(a eql_v2.ore_block_u64_8_256_term, b eql_v2.ore_block_u64_8_256_term) - RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - eq boolean := true; - unequal_block smallint := 0; - hash_key bytea; - data_block bytea; - encrypt_block bytea; - target_block bytea; - - left_block_size CONSTANT smallint := 16; - right_block_size CONSTANT smallint := 32; - right_offset CONSTANT smallint := 136; -- 8 * 17 - - indicator smallint := 0; - BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF bit_length(a.bytes) != bit_length(b.bytes) THEN - RAISE EXCEPTION 'Ciphertexts are different lengths'; - END IF; - - FOR block IN 0..7 LOOP - -- Compare each PRP (byte from the first 8 bytes) and PRF block (8 byte - -- chunks of the rest of the value). - -- NOTE: - -- * Substr is ordinally indexed (hence 1 and not 0, and 9 and not 8). - -- * We are not worrying about timing attacks here; don't fret about - -- the OR or !=. - IF - substr(a.bytes, 1 + block, 1) != substr(b.bytes, 1 + block, 1) - OR substr(a.bytes, 9 + left_block_size * block, left_block_size) != substr(b.bytes, 9 + left_block_size * BLOCK, left_block_size) - THEN - -- set the first unequal block we find - IF eq THEN - unequal_block := block; - END IF; - eq = false; - END IF; - END LOOP; - - IF eq THEN - RETURN 0::integer; - END IF; - - -- Hash key is the IV from the right CT of b - hash_key := substr(b.bytes, right_offset + 1, 16); - - -- first right block is at right offset + nonce_size (ordinally indexed) - target_block := substr(b.bytes, right_offset + 17 + (unequal_block * right_block_size), right_block_size); - - data_block := substr(a.bytes, 9 + (left_block_size * unequal_block), left_block_size); - - encrypt_block := encrypt(data_block::bytea, hash_key::bytea, 'aes-ecb'); - - indicator := ( - get_bit( - encrypt_block, - 0 - ) + get_bit(target_block, get_byte(a.bytes, unequal_block))) % 2; - - IF indicator = 1 THEN - RETURN 1::integer; - ELSE - RETURN -1::integer; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Compare arrays of ORE block terms recursively ---! @internal ---! ---! Recursively compares arrays of ORE block terms element-by-element. ---! Empty arrays are considered less than non-empty arrays. If the first elements ---! are equal, recursively compares remaining elements. ---! ---! @param a eql_v2.ore_block_u64_8_256_term[] First array of ORE terms ---! @param b eql_v2.ore_block_u64_8_256_term[] Second array of ORE terms ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b, NULL if either array is NULL ---! ---! @note Empty arrays sort before non-empty arrays ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256_term[], b eql_v2.ore_block_u64_8_256_term[]) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - cmp_result integer; - BEGIN - - -- NULLs are NULL - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- empty a and b - IF cardinality(a) = 0 AND cardinality(b) = 0 THEN - RETURN 0; - END IF; - - -- empty a and some b - IF (cardinality(a) = 0) AND cardinality(b) > 0 THEN - RETURN -1; - END IF; - - -- some a and empty b - IF cardinality(a) > 0 AND (cardinality(b) = 0) THEN - RETURN 1; - END IF; - - cmp_result := eql_v2.compare_ore_block_u64_8_256_term(a[1], b[1]); - - IF cmp_result = 0 THEN - -- Removes the first element in the array, and calls this fn again to compare the next element/s in the array. - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a[2:array_length(a,1)], b[2:array_length(b,1)]); - END IF; - - RETURN cmp_result; - END -$$ LANGUAGE plpgsql; - - ---! @brief Compare ORE block composite types ---! @internal ---! ---! Wrapper function that extracts term arrays from ORE block composite types ---! and delegates to the array comparison function. ---! ---! @param a eql_v2.ore_block_u64_8_256 First ORE block ---! @param b eql_v2.ore_block_u64_8_256 Second ORE block ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms(eql_v2.ore_block_u64_8_256_term[], eql_v2.ore_block_u64_8_256_term[]) -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a.terms, b.terms); - END -$$ LANGUAGE plpgsql; - - ---! @brief Extract STE vector index from JSONB payload ---! ---! Extracts the STE (Searchable Symmetric Encryption) vector from the 'sv' field ---! of an encrypted data payload. Returns an array of encrypted values used for ---! containment queries (@>, <@). If no 'sv' field exists, wraps the entire payload ---! as a single-element array. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(eql_v2_encrypted) ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.ste_vec(val jsonb) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv jsonb; - ary public.eql_v2_encrypted[]; - BEGIN - - IF val ? 'sv' THEN - sv := val->'sv'; - ELSE - sv := jsonb_build_array(val); - END IF; - - SELECT array_agg(eql_v2.to_encrypted(elem)) - INTO ary - FROM jsonb_array_elements(sv) AS elem; - - RETURN ary; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract STE vector index from encrypted column value ---! ---! Extracts the STE vector from an encrypted column value by accessing its ---! underlying JSONB data field. Used for containment query operations. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(jsonb) -CREATE FUNCTION eql_v2.ste_vec(val eql_v2_encrypted) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.ste_vec(val.data)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if JSONB payload is a single-element STE vector ---! ---! Tests whether the encrypted data payload contains an 'sv' field with exactly ---! one element. Single-element STE vectors can be treated as regular encrypted values. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'sv' field exists with exactly one element ---! ---! @see eql_v2.to_ste_vec_value -CREATE FUNCTION eql_v2.is_ste_vec_value(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'sv' THEN - RETURN jsonb_array_length(val->'sv') = 1; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if encrypted column value is a single-element STE vector ---! ---! Tests whether an encrypted column value is a single-element STE vector ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is a single-element STE vector ---! ---! @see eql_v2.is_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_value(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.is_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value ---! ---! Extracts the single element from a single-element STE vector and returns it ---! as a regular encrypted value, preserving metadata. If the input is not a ---! single-element STE vector, returns it unchanged. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.is_ste_vec_value -CREATE FUNCTION eql_v2.to_ste_vec_value(val jsonb) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - meta jsonb; - sv jsonb; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_value(val) THEN - meta := eql_v2.meta_data(val); - sv := val->'sv'; - sv := sv[0]; - - RETURN eql_v2.to_encrypted(meta || sv); - END IF; - - RETURN eql_v2.to_encrypted(val); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value (encrypted type) ---! ---! Converts an encrypted column value to a regular encrypted value by unwrapping ---! if it's a single-element STE vector. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.to_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.to_ste_vec_value(val eql_v2_encrypted) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.to_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract selector value from JSONB payload ---! ---! Extracts the selector ('s') field from an encrypted data payload. ---! Selectors are used to match STE vector elements during containment queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text The selector value ---! @throws Exception if 's' field is missing ---! ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.selector(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF val ? 's' THEN - RETURN val->>'s'; - END IF; - RAISE 'Expected a selector index (s) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from encrypted column value ---! @internal ---! ---! Internal convenience: unwraps the encrypted composite and delegates ---! to `eql_v2.selector(jsonb)`. Exists so the encrypted-selector ---! overloads of `eql_v2."->"` / `eql_v2."->>"` / `eql_v2.jsonb_path_*` ---! can dispatch without each having to spell out `(val).data` first. ---! Not part of the public API — callers should use ---! `eql_v2.selector(jsonb)` or `eql_v2.selector(eql_v2.ste_vec_entry)`. ---! ---! @param eql_v2_encrypted Encrypted column value (single-element form) ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) ---! @see eql_v2.selector(eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2._selector(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.selector(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from a ste_vec entry ---! ---! Direct overload on the domain type. The DOMAIN's CHECK constraint ---! already guarantees `s` is present, so this is a simple field access. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) -CREATE FUNCTION eql_v2.selector(entry eql_v2.ste_vec_entry) - RETURNS text - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 's' -$$; - - - ---! @brief Check if JSONB payload is marked as an STE vector array ---! ---! Tests whether the encrypted data payload has the 'a' (array) flag set to true, ---! indicating it represents an array for STE vector operations. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'a' field is present and true ---! ---! @see eql_v2.ste_vec -CREATE FUNCTION eql_v2.is_ste_vec_array(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'a' THEN - RETURN (val->>'a')::boolean; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value is marked as an STE vector array ---! ---! Tests whether an encrypted column value has the array flag set by checking ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is marked as an STE vector array ---! ---! @see eql_v2.is_ste_vec_array(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_array(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.is_ste_vec_array(val.data)); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract full encrypted JSONB elements as array ---! ---! Extracts all JSONB elements from the STE vector including non-deterministic fields. ---! Use jsonb_array() instead for GIN indexing and containment queries. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT CASE - WHEN val ? 'sv' THEN - ARRAY(SELECT elem FROM jsonb_array_elements(val->'sv') AS elem) - ELSE - ARRAY[val] - END; -$$; - - ---! @brief Extract full encrypted JSONB elements as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array_from_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array_from_array_elements(val.data); -$$; - - ---! @brief Extract deterministic fields as array for GIN indexing ---! ---! Extracts only deterministic search term fields (`s`, `hm`, `oc`, `op`) ---! from each STE vector element. Excludes non-deterministic ciphertext for ---! correct containment comparison using PostgreSQL's native `@>` operator. ---! ---! Field set: selector (`s`), HMAC equality (`hm`), ORE CLLW (`oc`, ---! Standard-mode), OPE CLLW (`op`, Compat-mode). The pre-2.3 fields ---! (`b3` / `ocf` / `ocv` / `opf` / `opv`) are no longer emitted — see U-004 ---! and U-006 in `docs/upgrading/v2.3.md`. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @note Use this for GIN indexes and containment queries ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_array(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT ARRAY( - SELECT jsonb_object_agg(kv.key, kv.value) - FROM jsonb_array_elements( - CASE WHEN val ? 'sv' THEN val->'sv' ELSE jsonb_build_array(val) END - ) AS elem, - LATERAL jsonb_each(elem) AS kv(key, value) - WHERE kv.key IN ('s', 'hm', 'oc', 'op') - GROUP BY elem - ); -$$; - - ---! @brief Extract deterministic fields as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @see eql_v2.jsonb_array(jsonb) -CREATE FUNCTION eql_v2.jsonb_array(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(val.data); -$$; - - ---! @brief GIN-indexable JSONB containment check ---! ---! Checks if encrypted value 'a' contains all JSONB elements from 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! This function is designed for use with a GIN index on jsonb_array(column). ---! When combined with such an index, PostgreSQL can efficiently search large tables. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b eql_v2_encrypted Value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @example ---! -- Create GIN index for efficient containment queries ---! CREATE INDEX idx ON mytable USING GIN (eql_v2.jsonb_array(encrypted_col)); ---! ---! -- Query using the helper function ---! SELECT * FROM mytable WHERE eql_v2.jsonb_contains(encrypted_col, search_value); ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (encrypted, jsonb) ---! ---! Checks if encrypted value 'a' contains all JSONB elements from jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b jsonb JSONB value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (jsonb, encrypted) ---! ---! Checks if jsonb value 'a' contains all JSONB elements from encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Container JSONB value ---! @param b eql_v2_encrypted Encrypted value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check ---! ---! Checks if all JSONB elements from 'a' are contained in 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b eql_v2_encrypted Container value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (encrypted, jsonb) ---! ---! Checks if all JSONB elements from encrypted value 'a' are contained in jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b jsonb Container JSONB value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (jsonb, encrypted) ---! ---! Checks if all JSONB elements from jsonb value 'a' are contained in encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Value to check ---! @param b eql_v2_encrypted Container encrypted value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief Check if STE vector array contains a specific encrypted element ---! ---! Tests whether any element in the STE vector array 'a' contains the encrypted value 'b'. ---! Matching requires both the selector and encrypted value to be equal. ---! Used internally by ste_vec_contains(encrypted, encrypted) for array containment checks. ---! ---! @param eql_v2_encrypted[] STE vector array to search within ---! @param eql_v2_encrypted Encrypted element to search for ---! @return Boolean True if b is found in any element of a ---! ---! @note Compares both selector and encrypted value for match ---! ---! @see eql_v2.selector ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.ste_vec_contains(a public.eql_v2_encrypted[], b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - _a public.eql_v2_encrypted; - BEGIN - - result := false; - - FOR idx IN 1..array_length(a, 1) LOOP - _a := a[idx]; - -- Element-level match for ste_vec entries. - -- - -- Per the v2.3 sv-element contract (encoded in - -- `docs/reference/schema/eql-payload-v2.3.schema.json` and the - -- `eql_v2.ste_vec_entry` DOMAIN), each entry carries **exactly - -- one** of: - -- - `hm` — HMAC-256 for boolean leaves and for the placeholder - -- entries that represent array / object roots. - -- - `oc` — CLLW ORE for string and number leaves. - -- Both terms are deterministic for the same plaintext at the same - -- selector under the same workspace, so either one serves as the - -- equality discriminator. A selector configures the leaf's role - -- (eq / ordered), and the role determines which term is emitted — - -- two sv entries with the same selector therefore always carry - -- the same term type. - -- - -- The selector check is a fast-path gate so we don't compare - -- terms across mismatched fields. Once selectors match, exactly - -- one of the two CASE branches fires (XOR contract above). - -- - -- The `ELSE false` arm covers the malformed case (entry carries - -- neither term, or only one side has the term for a given role). - -- That's a data error rather than a normal containment result, - -- but returning false is safer than raising mid-array-scan. - result := result OR ( - eql_v2._selector(_a) = eql_v2._selector(b) AND - CASE - WHEN eql_v2.has_hmac_256(_a) AND eql_v2.has_hmac_256(b) THEN - eql_v2.compare_hmac_256(_a, b) = 0 - WHEN eql_v2.has_ore_cllw((_a).data) AND eql_v2.has_ore_cllw((b).data) THEN - eql_v2.compare_ore_cllw_term( - eql_v2.ore_cllw((_a).data), - eql_v2.ore_cllw((b).data) - ) = 0 - ELSE false - END - ); - - -- Short-circuit once a match is found. Without this we still walk - -- the rest of the sv array, which on a 100-element document means - -- 99 wasted selector + extractor calls per row. - EXIT WHEN result; - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted value 'a' contains all elements of encrypted value 'b' ---! ---! Performs STE vector containment comparison between two encrypted values. ---! Returns true if all elements in b's STE vector are found in a's STE vector. ---! Used internally by the @> containment operator for searchable encryption. ---! ---! @param a eql_v2_encrypted First encrypted value (container) ---! @param b eql_v2_encrypted Second encrypted value (elements to find) ---! @return Boolean True if all elements of b are contained in a ---! ---! @note Empty b is always contained in any a ---! @note Each element of b must match both selector and value in a ---! ---! @see eql_v2.ste_vec ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted[], eql_v2_encrypted) ---! @see eql_v2."@>" -CREATE FUNCTION eql_v2.ste_vec_contains(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - sv_a public.eql_v2_encrypted[]; - sv_b public.eql_v2_encrypted[]; - _b public.eql_v2_encrypted; - BEGIN - - -- jsonb arrays of ste_vec encrypted values - sv_a := eql_v2.ste_vec(a); - sv_b := eql_v2.ste_vec(b); - - -- an empty b is always contained in a - IF array_length(sv_b, 1) IS NULL THEN - RETURN true; - END IF; - - IF array_length(sv_a, 1) IS NULL THEN - RETURN false; - END IF; - - result := true; - - -- for each element of b check if it is in a - FOR idx IN 1..array_length(sv_b, 1) LOOP - _b := sv_b[idx]; - result := result AND eql_v2.ste_vec_contains(sv_a, _b); - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; ---! @file config/types.sql ---! @brief Configuration state type definition ---! ---! Defines the ENUM type for tracking encryption configuration lifecycle states. ---! The configuration table uses this type to manage transitions between states ---! during setup, activation, and encryption operations. ---! ---! @note CREATE TYPE does not support IF NOT EXISTS, so wrapped in DO block ---! @note Configuration data stored as JSONB directly, not as DOMAIN ---! @see config/tables.sql - - ---! @brief Configuration lifecycle state ---! ---! Defines valid states for encryption configurations in the eql_v2_configuration table. ---! Configurations transition through these states during setup and activation. ---! ---! @note Only one configuration can be in 'active', 'pending', or 'encrypting' state at once ---! @see config/indexes.sql for uniqueness enforcement ---! @see config/tables.sql for usage in eql_v2_configuration table -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_configuration_state') THEN - CREATE TYPE public.eql_v2_configuration_state AS ENUM ('active', 'inactive', 'encrypting', 'pending'); - END IF; - END -$$; - - - ---! @brief Extract Bloom filter index term from JSONB payload ---! ---! Extracts the Bloom filter array from the 'bf' field of an encrypted ---! data payload. Used internally for pattern-match queries (LIKE operator). ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! @throws Exception if 'bf' field is missing when bloom_filter index is expected ---! ---! @see eql_v2.has_bloom_filter ---! @see eql_v2."~~" -CREATE FUNCTION eql_v2.bloom_filter(val jsonb) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_bloom_filter(val) THEN - RETURN ARRAY(SELECT jsonb_array_elements(val->'bf'))::eql_v2.bloom_filter; - END IF; - - RAISE 'Expected a match index (bf) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract Bloom filter index term from encrypted column value ---! ---! Extracts the Bloom filter from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! ---! @see eql_v2.bloom_filter(jsonb) -CREATE FUNCTION eql_v2.bloom_filter(val eql_v2_encrypted) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.bloom_filter(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains Bloom filter index term ---! ---! Tests whether the encrypted data payload includes a 'bf' field, ---! indicating a Bloom filter is available for pattern-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'bf' field is present and non-null ---! ---! @see eql_v2.bloom_filter -CREATE FUNCTION eql_v2.has_bloom_filter(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'bf' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains Bloom filter index term ---! ---! Tests whether an encrypted column value includes a Bloom filter ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if Bloom filter is present ---! ---! @see eql_v2.has_bloom_filter(jsonb) -CREATE FUNCTION eql_v2.has_bloom_filter(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_bloom_filter(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @file src/ste_vec/eq_term.sql ---! @brief XOR-aware equality term extractor for `eql_v2.ste_vec_entry` ---! ---! Returns the bytea representation of whichever deterministic term ---! the sv entry carries — `hm` (HMAC-256) for bool leaves / array ---! roots / object roots, or `oc` (CLLW ORE) for string / number ---! leaves. The two byte distributions are disjoint by construction ---! (different keys, different protocols), so byte equality on the ---! coalesce is unambiguous: equal terms imply equal plaintexts under ---! the same selector, and unequal terms imply different plaintexts ---! (or different protocols, which can't happen for a single ---! selector). ---! ---! This is the canonical equality extractor used by `=` and `<>` on ---! `eql_v2.ste_vec_entry` — see `src/operators/ste_vec_entry.sql`. ---! The recipe for field-level equality on encrypted JSON is: ---! ---! @example ---! -- Functional hash index covers both hm-bearing and oc-bearing selectors ---! CREATE INDEX ON users USING hash (eql_v2.eq_term(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return bytea Decoded `hm` or `oc` bytes (NULL if entry is NULL). ---! ---! @note The XOR contract (each sv entry carries exactly one of `hm` ---! or `oc` — enforced by the `ste_vec_entry` DOMAIN CHECK) means ---! the coalesce always picks the one present term. ---! ---! @see eql_v2.hmac_256(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.eq_term(entry eql_v2.ste_vec_entry) - RETURNS bytea - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT decode(coalesce(entry ->> 'hm', entry ->> 'oc'), 'hex') -$$; - - - ---! @file src/operators/compare.sql ---! @brief Three-way ordering on the root `eql_v2_encrypted` type ---! ---! Returns `-1` / `0` / `1` for two encrypted column values that carry ---! Block ORE (`ob`) terms at the root. Used by the btree operator class on ---! `eql_v2_encrypted` (FUNCTION 1), by the legacy `eql_v2.lt` / `lte` / ---! `gt` / `gte` helpers, and by `sort_compare`'s `strategy = 'compare'` ---! fallback path. ---! ---! **Strict Block-ORE-only contract.** Root-level `eql_v2_encrypted` values ---! only carry root-scope ORE terms (`ob`) per the v2.3 payload shape — the ---! `oc` field (CLLW ORE) is sv-element scope only and never appears on a ---! root payload. Equality on `eql_v2_encrypted` is hm-only and runs through ---! the inlined `=` / `<>` operators (post-#193) — it does *not* go through ---! this function. For sv-element ordering, use the typed ---! `eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry)` overload ---! (or the `<` / `<=` / `>` / `>=` operators on the same pair). ---! ---! @param a eql_v2_encrypted First encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @param b eql_v2_encrypted Second encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either value lacks an `ob` (Block ORE) term ---! ---! @see eql_v2.compare_ore_block_u64_8_256 ---! @see eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry) ---! @see eql_v2."=" -- hm-only equality, post-#193 inlining -CREATE FUNCTION eql_v2.compare(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF eql_v2.has_ore_block_u64_8_256(a) AND eql_v2.has_ore_block_u64_8_256(b) THEN - RETURN eql_v2.compare_ore_block_u64_8_256(a, b); - END IF; - - RAISE EXCEPTION - 'eql_v2.compare requires Block ORE (`ob`) on both root operands. For sv-element ordering, extract entries via `col -> ''<selector>''` and use eql_v2.compare on the resulting `eql_v2.ste_vec_entry` values (or their `<` / `<=` / `>` / `>=` operators). Equality is hmac-only via the `=` operator — this function is for ordering only.' - USING ERRCODE = 'feature_not_supported'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Three-way ordering on `eql_v2.ste_vec_entry` ---! ---! CLLW ORE three-way comparator on ste-vec entries. Returns `-1` / `0` / ---! `1` by extracting the `oc` term from each entry and delegating to ---! `eql_v2.compare_ore_cllw_term`. Use this when you need an `int` ordering ---! out of two extracted ste-vec entries — for the boolean-form operators ---! (`<` / `<=` / `>` / `>=`) on the same pair, see ---! `src/operators/ste_vec_entry.sql`. ---! ---! Note: the caller is responsible for extracting an `eql_v2.ste_vec_entry` ---! first; the `(eql_v2_encrypted, text)` form would be a natural extension ---! but is deliberately *not* added here so that callers stay aware of the ---! two-step shape (extract via `->`, then compare). ---! ---! @param a eql_v2.ste_vec_entry First entry ---! @param b eql_v2.ste_vec_entry Second entry ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either entry lacks an `oc` term ---! ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.compare(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF NOT (eql_v2.has_ore_cllw(a) AND eql_v2.has_ore_cllw(b)) THEN - RAISE EXCEPTION - 'eql_v2.compare(ste_vec_entry, ste_vec_entry) requires `oc` (CLLW ORE) on both entries.' - USING ERRCODE = 'feature_not_supported'; - END IF; - - RETURN eql_v2.compare_ore_cllw_term(eql_v2.ore_cllw(a), eql_v2.ore_cllw(b)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract ORE index term for ordering encrypted values ---! ---! Helper function that extracts the ore_block_u64_8_256 index term from an encrypted value ---! for use in ORDER BY clauses when comparison operators are not appropriate or available. ---! ---! @param eql_v2_encrypted Encrypted value to extract order term from ---! @return eql_v2.ore_block_u64_8_256 ORE index term for ordering ---! ---! @example ---! -- Order encrypted values without using comparison operators ---! SELECT * FROM users ORDER BY eql_v2.order_by(encrypted_age); ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.order_by(a eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(a); - END; -$$ LANGUAGE plpgsql; - ---! @brief Equality operator for ORE block types ---! @internal ---! ---! Implements the = operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_eq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 0 -$$; - - - ---! @brief Not equal operator for ORE block types ---! @internal ---! ---! Implements the <> operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are not equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_neq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) <> 0 -$$; - - - ---! @brief Less than operator for ORE block types ---! @internal ---! ---! Implements the < operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = -1 -$$; - - - ---! @brief Less than or equal operator for ORE block types ---! @internal ---! ---! Implements the <= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != 1 -$$; - - - ---! @brief Greater than operator for ORE block types ---! @internal ---! ---! Implements the > operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 1 -$$; - - - ---! @brief Greater than or equal operator for ORE block types ---! @internal ---! ---! Implements the >= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != -1 -$$; - - - ---! @brief = operator for ORE block types -CREATE OPERATOR = ( - FUNCTION=eql_v2.ore_block_u64_8_256_eq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - - ---! @brief <> operator for ORE block types -CREATE OPERATOR <> ( - FUNCTION=eql_v2.ore_block_u64_8_256_neq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief > operator for ORE block types -CREATE OPERATOR > ( - FUNCTION=eql_v2.ore_block_u64_8_256_gt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - - ---! @brief < operator for ORE block types -CREATE OPERATOR < ( - FUNCTION=eql_v2.ore_block_u64_8_256_lt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - - ---! @brief <= operator for ORE block types -CREATE OPERATOR <= ( - FUNCTION=eql_v2.ore_block_u64_8_256_lte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - - ---! @brief >= operator for ORE block types -CREATE OPERATOR >= ( - FUNCTION=eql_v2.ore_block_u64_8_256_gte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief Contains operator for encrypted values (@>) ---! ---! Implements the @> (contains) operator for testing if left encrypted value ---! contains the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2_encrypted Right operand (contained value) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Check if encrypted array contains value ---! SELECT * FROM documents ---! WHERE encrypted_tags @> '["security"]'::jsonb::eql_v2_encrypted; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.add_search_config --- Marked IMMUTABLE STRICT PARALLEL SAFE so the planner inlines the body --- and a functional GIN index on `eql_v2.ste_vec(col)` can match --- `WHERE col @> val`. The previous default-VOLATILE declaration prevented --- inlining and forced seq scan even on Supabase installs that have the --- ste_vec functional index in place. -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ste_vec_contains(a, b) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contains operator (@>) with an `eql_v2.stevec_query` needle ---! ---! Type-safe containment for the recommended recipe: the right-hand ---! side is an `stevec_query` (sv-shaped payload, no `c` fields). The ---! body inlines to a native `jsonb @>` over `eql_v2.to_stevec_query(a)::jsonb`, ---! so the planner can match a functional GIN index built on the same ---! expression — engaging Bitmap Index Scan for bare-form containment ---! across both `hm`-bearing and `oc`-bearing selectors with a single ---! index. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.stevec_query Right operand (query payload) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Functional GIN index (covers all selectors, hm and oc): ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! -- Bare-form predicate engages the index: ---! SELECT * FROM users ---! WHERE encrypted_doc @> '{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2.to_stevec_query -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.stevec_query) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Single-expression body so the planner can inline. The haystack - -- normalisation happens in `to_stevec_query`; the needle is trusted - -- to be clean (sv elements of shape `{s, hm-or-oc}` — the documented - -- stevec_query contract). For untrusted needles, callers should - -- normalise via the json-shape `{"sv":[{"s":"<sel>","hm":"<term>"}]}`. - SELECT eql_v2.to_stevec_query(a)::jsonb @> b::jsonb -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.stevec_query -); - - ---! @brief Contains operator (@>) with an `eql_v2.ste_vec_entry` needle ---! ---! Convenience overload for the common pattern "does this encrypted ---! payload include this specific sv entry?". Wraps the entry into a ---! single-element sv array (stripping `c`) and reduces to the same ---! `to_stevec_query(a)::jsonb @> needle::jsonb` form as the ---! `stevec_query` overload — so it engages the same functional GIN ---! index. Inlinable. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.ste_vec_entry Right operand (single entry) ---! @return Boolean True if a contains an sv entry matching `b` ---! ---! @example ---! -- Does this row's encrypted doc contain the same name as this other doc? ---! SELECT a.* FROM docs a, docs b ---! WHERE a.doc @> (b.doc -> '<name-sel>'); ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.ste_vec_entry) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.to_stevec_query(a)::jsonb - @> jsonb_build_object( - 'sv', - jsonb_build_array( - jsonb_strip_nulls( - jsonb_build_object( - 's', b -> 's', - 'hm', b -> 'hm', - 'oc', b -> 'oc' - ) - ) - ) - ) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.ste_vec_entry -); - ---! @file config/tables.sql ---! @brief Encryption configuration storage table ---! ---! Defines the main table for storing EQL v2 encryption configurations. ---! Each row represents a configuration specifying which tables/columns to encrypt ---! and what index types to use. Configurations progress through lifecycle states. ---! ---! @see config/types.sql for state ENUM definition ---! @see config/indexes.sql for state uniqueness constraints ---! @see config/constraints.sql for data validation - - ---! @brief Encryption configuration table ---! ---! Stores encryption configurations with their state and metadata. ---! The 'data' JSONB column contains the full configuration structure including ---! table/column mappings, index types, and casting rules. ---! ---! @note Only one configuration can be 'active', 'pending', or 'encrypting' at once ---! @note 'id' is auto-generated identity column ---! @note 'state' defaults to 'pending' for new configurations ---! @note 'data' validated by CHECK constraint (see config/constraints.sql) -CREATE TABLE IF NOT EXISTS public.eql_v2_configuration -( - id bigint GENERATED ALWAYS AS IDENTITY, - state eql_v2_configuration_state NOT NULL DEFAULT 'pending', - data jsonb, - created_at timestamptz not null default current_timestamp, - PRIMARY KEY(id) -); - - ---! @brief Initialize default configuration structure ---! @internal ---! ---! Creates a default configuration object if input is NULL. Used internally ---! by public configuration functions to ensure consistent structure. ---! ---! @param config JSONB Existing configuration or NULL ---! @return JSONB Configuration with default structure (version 1, empty tables) -CREATE FUNCTION eql_v2.config_default(config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF config IS NULL THEN - SELECT jsonb_build_object('v', 1, 'tables', jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add table to configuration if not present ---! @internal ---! ---! Ensures the specified table exists in the configuration structure. ---! Creates empty table entry if needed. Idempotent operation. ---! ---! @param table_name Text Name of table to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with table entry -CREATE FUNCTION eql_v2.config_add_table(table_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - tbl jsonb; - BEGIN - IF NOT config #> array['tables'] ? table_name THEN - SELECT jsonb_insert(config, array['tables', table_name], jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add column to table configuration if not present ---! @internal ---! ---! Ensures the specified column exists in the table's configuration structure. ---! Creates empty column entry with indexes object if needed. Idempotent operation. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with column entry -CREATE FUNCTION eql_v2.config_add_column(table_name text, column_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - col jsonb; - BEGIN - IF NOT config #> array['tables', table_name] ? column_name THEN - SELECT jsonb_build_object('indexes', jsonb_build_object()) into col; - SELECT jsonb_set(config, array['tables', table_name, column_name], col) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Set cast type for column in configuration ---! @internal ---! ---! Updates the cast_as field for a column, specifying the PostgreSQL type ---! that decrypted values should be cast to. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param cast_as Text PostgreSQL type for casting (e.g., 'text', 'int', 'jsonb') ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with cast_as set -CREATE FUNCTION eql_v2.config_add_cast(table_name text, column_name text, cast_as text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_set(config, array['tables', table_name, column_name, 'cast_as'], to_jsonb(cast_as)) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add search index to column configuration ---! @internal ---! ---! Inserts a search index entry (unique, match, ore, ste_vec) with its options ---! into the column's indexes object. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param index_name Text Type of index to add ---! @param opts JSONB Index-specific options ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with index added -CREATE FUNCTION eql_v2.config_add_index(table_name text, column_name text, index_name text, opts jsonb, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_insert(config, array['tables', table_name, column_name, 'indexes', index_name], opts) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Generate default options for match index ---! @internal ---! ---! Returns default configuration for match (LIKE) indexes: k=6, bf=2048, ---! ngram tokenizer with token_length=3, downcase filter, include_original=true. ---! ---! @return JSONB Default match index options -CREATE FUNCTION eql_v2.config_match_default() - RETURNS jsonb -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_build_object( - 'k', 6, - 'bf', 2048, - 'include_original', true, - 'tokenizer', json_build_object('kind', 'ngram', 'token_length', 3), - 'token_filters', json_build_array(json_build_object('kind', 'downcase'))); -END; --- AUTOMATICALLY GENERATED FILE --- Source is version-template.sql - -DROP FUNCTION IF EXISTS eql_v2.version(); - ---! @file version.sql ---! @brief EQL version reporting ---! ---! This file is auto-generated from version.template during build. ---! The version string placeholder is replaced with the actual release version. - ---! @brief Get EQL library version string ---! ---! Returns the version string for the installed EQL library. ---! This value is set at build time from the project version. ---! ---! @return text Version string (e.g., "2.1.0" or "DEV" for development builds) ---! ---! @note Auto-generated during build from version.template ---! ---! @example ---! -- Check installed EQL version ---! SELECT eql_v2.version(); ---! -- Returns: '2.1.0' -CREATE FUNCTION eql_v2.version() - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT 'eql-2.3.1'; -$$ LANGUAGE SQL; - - ---! @file src/ore_cllw/operators.sql ---! @brief Comparison operators on the `eql_v2.ore_cllw` composite type ---! ---! Same-type comparison operators backing the btree operator class on the ---! composite `eql_v2.ore_cllw` type. Each operator reduces to a single SELECT ---! over `eql_v2.compare_ore_cllw_term(a, b)`, which is the canonical CLLW ---! per-byte comparator (`y + 1 == x` mod 256). The operator wrappers are ---! inlinable `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` so the planner can ---! fold them into the calling query — that's what lets a functional btree ---! index on `eql_v2.ore_cllw(col)` engage for both `WHERE eql_v2.ore_cllw(col) ---! < eql_v2.ore_cllw($1)` and `ORDER BY eql_v2.ore_cllw(col)` shapes. ---! ---! The inner `eql_v2.compare_ore_cllw_term` is `LANGUAGE plpgsql` (it has a ---! per-byte loop) and is NOT inlined. That's fine for index *match* (the ---! planner only needs the outer operator function call to fold so the ---! predicate's expression tree matches the index's expression tree); only the ---! per-comparison cost is the plpgsql call overhead. That's the cost the ---! functional index avoids by walking the btree in order rather than calling ---! compare on every row. ---! ---! @note Deliberately no `HASHES` / `MERGES` flags on the operator ---! declarations. HASHES requires a registered hash function on the type ---! (the CLLW protocol gives ordering, not a sensible hashing); MERGES ---! requires an equivalent merge-joinable operator class on both sides. ---! ---! @see src/ore_cllw/operator_class.sql ---! @see src/ore_cllw/functions.sql - ---! @brief Equality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare equal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_eq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 0 -$$; - ---! @brief Inequality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare unequal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_neq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 0 -$$; - ---! @brief Less-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = -1 -$$; - ---! @brief Less-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 1 -$$; - ---! @brief Greater-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 1 -$$; - ---! @brief Greater-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> -1 -$$; - - -CREATE OPERATOR = ( - FUNCTION = eql_v2.ore_cllw_eq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel -); - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.ore_cllw_neq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - -CREATE OPERATOR < ( - FUNCTION = eql_v2.ore_cllw_lt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.ore_cllw_lte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - -CREATE OPERATOR > ( - FUNCTION = eql_v2.ore_cllw_gt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.ore_cllw_gte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - - ---! @brief Compare two encrypted values using ORE block index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their ORE block index terms. Used internally by range operators (<, <=, >, >=) ---! for order-revealing comparisons without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Uses ORE cryptographic protocol for secure comparisons ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2."<" ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.ore_block_u64_8_256; - b_term eql_v2.ore_block_u64_8_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - a_term := eql_v2.ore_block_u64_8_256(a); - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - b_term := eql_v2.ore_block_u64_8_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a_term.terms, b_term.terms); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Cast text to ORE block term ---! @internal ---! ---! Converts text to bytea and wraps in ore_block_u64_8_256_term type. ---! Used internally for ORE block extraction and manipulation. ---! ---! @param t Text Text value to convert ---! @return eql_v2.ore_block_u64_8_256_term ORE term containing bytea representation ---! ---! @see eql_v2.ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(t text) - RETURNS eql_v2.ore_block_u64_8_256_term - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN t::bytea; -END; - ---! @brief Implicit cast from text to ORE block term ---! ---! Defines an implicit cast allowing automatic conversion of text values ---! to ore_block_u64_8_256_term type for ORE operations. ---! ---! @see eql_v2.text_to_ore_block_u64_8_256_term -CREATE CAST (text AS eql_v2.ore_block_u64_8_256_term) - WITH FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(text) AS IMPLICIT; - ---! @brief Pattern matching helper using bloom filters ---! @internal ---! ---! Internal helper for LIKE-style pattern matching on encrypted values. ---! Uses bloom filter index terms to test substring containment without decryption. ---! Requires 'match' index configuration on the column. ---! ---! Marked IMMUTABLE so the planner inlines the body and a functional index on ---! `eql_v2.bloom_filter(col)` can match `WHERE eql_v2.like(col, val)`. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @see eql_v2."~~" ---! @see eql_v2.bloom_filter ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.like(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief Case-insensitive pattern matching helper ---! @internal ---! ---! Internal helper for ILIKE-style case-insensitive pattern matching. ---! Case sensitivity is controlled by index configuration (token_filters with downcase). ---! This function has same implementation as like() - actual case handling is in index terms. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @note Case sensitivity depends on match index token_filters configuration ---! @see eql_v2."~~" ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.ilike(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief LIKE operator for encrypted values (pattern matching) ---! ---! Implements the ~~ (LIKE) operator for substring/pattern matching on encrypted ---! text using bloom filter index terms. Enables WHERE col LIKE '%pattern%' queries ---! without decryption. Requires 'match' index configuration on the column. ---! ---! Pattern matching uses n-gram tokenization configured in match index. Token length ---! and filters affect matching behavior. ---! ---! @param a eql_v2_encrypted Haystack (encrypted text to search in) ---! @param b eql_v2_encrypted Needle (encrypted pattern to search for) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! -- Search for substring in encrypted email ---! SELECT * FROM users ---! WHERE encrypted_email ~~ '%@example.com%'::text::eql_v2_encrypted; ---! ---! -- Pattern matching on encrypted names ---! SELECT * FROM customers ---! WHERE encrypted_name ~~ 'John%'::text::eql_v2_encrypted; ---! ---! @brief SQL LIKE operator (~~ operator) for encrypted text pattern matching ---! ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b eql_v2_encrypted Right operand (encrypted pattern) ---! @return boolean True if pattern matches ---! ---! @note Requires match index: eql_v2.add_search_config(table, column, 'match') ---! @see eql_v2.like ---! @see eql_v2.add_search_config --- Inlinable: delegates to `eql_v2.like` which is itself an inlinable --- single-statement SQL function. Two levels of inlining produce --- `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)`, which matches a --- functional GIN index built on `eql_v2.bloom_filter(col)`. PostgREST --- and ORM `~~`/`~~*` queries engage the bloom-filter index without --- the caller wrapping the column themselves. -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b) -$$; - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Case-insensitive LIKE operator (~~*) ---! ---! Implements ~~* (ILIKE) operator for case-insensitive pattern matching. ---! Case handling depends on match index token_filters configuration (use downcase filter). ---! Same implementation as ~~, with case sensitivity controlled by index configuration. ---! ---! @param a eql_v2_encrypted Haystack ---! @param b eql_v2_encrypted Needle ---! @return Boolean True if a contains b (case-insensitive) ---! ---! @note Configure match index with downcase token filter for case-insensitivity ---! @see eql_v2."~~" -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for encrypted value and JSONB ---! ---! Overload of ~~ operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param eql_v2_encrypted Haystack (encrypted value) ---! @param b JSONB Needle (will be cast to eql_v2_encrypted) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE encrypted_email ~~ '%gmail%'::jsonb; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b::eql_v2_encrypted) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for JSONB and encrypted value ---! ---! Overload of ~~ operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param a JSONB Haystack (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Needle (encrypted pattern) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE 'test@example.com'::jsonb ~~ encrypted_pattern; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a::eql_v2_encrypted, b) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - --- ----------------------------------------------------------------------------- - ---! @file src/operators/ste_vec_entry.sql ---! @brief Comparison operators on `eql_v2.ste_vec_entry` ---! ---! Equality (`=`, `<>`) reduces to `eq_term(a) = eq_term(b)` — a bytea ---! comparison of `coalesce(hm, oc)`. Ordering (`<`, `<=`, `>`, `>=`) ---! reduces to `ore_cllw(a) <op> ore_cllw(b)`. Each backing function is ---! inlinable single-statement SQL, so the planner can fold the ---! operator body into the calling query — `WHERE col -> 'sel' = $1` ---! and `WHERE col -> 'sel' < $1` therefore match functional indexes ---! built on `eql_v2.eq_term(col -> 'sel')` / ---! `eql_v2.ore_cllw(col -> 'sel')` without per-query rewriting. ---! ---! XOR contract. Each sv entry carries exactly one of `hm` (bool ---! leaves, array / object roots) or `oc` (string / number leaves) — ---! enforced by the `ste_vec_entry` DOMAIN CHECK. Equality coalesces ---! across both protocols because both are deterministic and the byte ---! distributions are disjoint; ordering strictly uses `ore_cllw` ---! (range on hm-only entries is meaningless and produces silent NULL, ---! which the lint subsystem `src/lint/lints.sql` flags as a ---! configuration error). ---! ---! Same convention as the `eql_v2_encrypted` operators (#193 / #211): the ---! operator-class function-matching layer is what makes index match work ---! structurally, the backing functions just need to inline cleanly through ---! to the extractor calls. ---! ---! @see eql_v2.eq_term(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/=.sql ---! @see src/operators/<.sql - ---! @brief Equality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if both entries share the same deterministic ---! equality term (hm-or-oc, via `eq_term`). -CREATE FUNCTION eql_v2.eq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) = eql_v2.eq_term(b) -$$; - -CREATE OPERATOR = ( - FUNCTION = eql_v2.eq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief Inequality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if the entries' equality terms (hm-or-oc, via ---! `eq_term`) differ. -CREATE FUNCTION eql_v2.neq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) <> eql_v2.eq_term(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.neq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - - ---! @brief Less-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before `b`'s -CREATE FUNCTION eql_v2.lt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) < eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR < ( - FUNCTION = eql_v2.lt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - ---! @brief Less-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before or equal to `b`'s -CREATE FUNCTION eql_v2.lte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) <= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.lte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - ---! @brief Greater-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after `b`'s -CREATE FUNCTION eql_v2.gt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) > eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR > ( - FUNCTION = eql_v2.gt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - ---! @brief Greater-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after or equal to `b`'s -CREATE FUNCTION eql_v2.gte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) >= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.gte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @file operators/sort.sql ---! @brief Comparison-based sorting functions for encrypted values without operator classes ---! ---! Provides O(n log n) quicksort-based sorting using eql_v2.compare() for environments ---! where btree operator classes are unavailable (e.g., Supabase). This is significantly ---! faster than the O(n^2) correlated subquery workaround. ---! ---! When all input rows share an ORE term (`ob`) the sort path pre-extracts the ---! ORE order key once per row and compares those keys directly. Rows lacking ---! an ORE term entirely fall back to `eql_v2.compare()` per pair. - - ---! @internal ---! @brief Compare pre-extracted ORE order keys with encrypted NULL semantics ---! ---! Mirrors eql_v2.compare() for NULL handling, then delegates to the ---! ore_block_u64_8_256 comparator when both keys are present. ---! ---! @param a eql_v2.ore_block_u64_8_256 First order key ---! @param b eql_v2.ore_block_u64_8_256 Second order key ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b -CREATE FUNCTION eql_v2._compare_order_key( - a eql_v2.ore_block_u64_8_256, - b eql_v2.ore_block_u64_8_256 -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a, b); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare two elements from aligned arrays using the selected sort strategy ---! ---! @param vals eql_v2_encrypted[] Encrypted values (used when strategy = 'compare') ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (strategy = 'ore') ---! @param left_idx integer Index of the left element ---! @param right_idx integer Index of the right element ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if left < right, 0 if equal, 1 if left > right -CREATE FUNCTION eql_v2._compare_sort_elements( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - left_idx integer, - right_idx integer, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[left_idx], ore_keys[right_idx]); - END IF; - - RETURN eql_v2.compare(vals[left_idx], vals[right_idx]); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare an array element against a captured pivot using the selected strategy ---! ---! @param vals eql_v2_encrypted[] Array of encrypted values ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys ---! @param idx integer Index of the element to compare ---! @param pivot_val eql_v2_encrypted Pivot encrypted value (strategy = 'compare') ---! @param pivot_ore_key eql_v2.ore_block_u64_8_256 Pivot ORE key (strategy = 'ore') ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if element < pivot, 0 if equal, 1 if element > pivot -CREATE FUNCTION eql_v2._compare_sort_pivot( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - idx integer, - pivot_val eql_v2_encrypted, - pivot_ore_key eql_v2.ore_block_u64_8_256, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[idx], pivot_ore_key); - END IF; - - RETURN eql_v2.compare(vals[idx], pivot_val); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place insertion sort on parallel id/value/key arrays ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._insertion_sort( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - i integer; - j integer; - key_id bigint; - key_val eql_v2_encrypted; - sort_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - IF lo >= hi THEN - RETURN; - END IF; - - FOR i IN lo + 1..hi LOOP - key_id := ids[i]; - key_val := vals[i]; - sort_ore_key := ore_keys[i]; - j := i - 1; - - WHILE j >= lo LOOP - EXIT WHEN strategy = 'compare' - AND eql_v2.compare(vals[j], key_val) <= 0; - EXIT WHEN strategy = 'ore' - AND eql_v2._compare_order_key(ore_keys[j], sort_ore_key) <= 0; - - ids[j + 1] := ids[j]; - vals[j + 1] := vals[j]; - ore_keys[j + 1] := ore_keys[j]; - j := j - 1; - END LOOP; - - ids[j + 1] := key_id; - vals[j + 1] := key_val; - ore_keys[j + 1] := sort_ore_key; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place quicksort on parallel id/value/key arrays ---! ---! Sorts aligned arrays simultaneously using Hoare partition with median-of-three pivot ---! selection. The median-of-three strategy avoids O(n^2) degradation on already-sorted ---! input, which is common with sequential test data. ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values to compare (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._quicksort_sorter( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - insertion_threshold CONSTANT integer := 16; - pivot_val eql_v2_encrypted; - pivot_ore_key eql_v2.ore_block_u64_8_256; - mid integer; - i integer; - j integer; - left_hi integer; - right_lo integer; - tmp_id bigint; - tmp_val eql_v2_encrypted; - tmp_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - WHILE lo < hi LOOP - IF hi - lo <= insertion_threshold THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._insertion_sort(ids, vals, ore_keys, lo, hi, strategy) q; - RETURN; - END IF; - - -- Median-of-three pivot selection: sort lo, mid, hi then use mid as pivot - mid := lo + (hi - lo) / 2; - - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, mid, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[mid]; ids[mid] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[mid]; vals[mid] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[mid]; ore_keys[mid] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, hi, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, mid, hi, strategy) > 0 THEN - tmp_id := ids[mid]; ids[mid] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[mid]; vals[mid] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[mid]; ore_keys[mid] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - - pivot_val := vals[mid]; - pivot_ore_key := ore_keys[mid]; - i := lo; - j := hi; - - LOOP - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, i, - pivot_val, pivot_ore_key, strategy - ) < 0 LOOP - i := i + 1; - END LOOP; - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, j, - pivot_val, pivot_ore_key, strategy - ) > 0 LOOP - j := j - 1; - END LOOP; - - EXIT WHEN i >= j; - - tmp_id := ids[i]; ids[i] := ids[j]; ids[j] := tmp_id; - tmp_val := vals[i]; vals[i] := vals[j]; vals[j] := tmp_val; - tmp_ore_key := ore_keys[i]; ore_keys[i] := ore_keys[j]; ore_keys[j] := tmp_ore_key; - - i := i + 1; - j := j - 1; - END LOOP; - - left_hi := j; - right_lo := j + 1; - - IF left_hi - lo < hi - right_lo THEN - IF lo < left_hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, lo, left_hi, strategy) q; - END IF; - lo := right_lo; - ELSE - IF right_lo < hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, right_lo, hi, strategy) q; - END IF; - hi := left_hi; - END IF; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Emit aligned arrays as rows in ASC or DESC order ---! ---! @param ids bigint[] Array of sorted row identifiers ---! @param vals eql_v2_encrypted[] Array of sorted encrypted values ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Rows emitted in the requested order -CREATE FUNCTION eql_v2._emit_sorted_rows( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - i integer; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - IF upper(direction) = 'DESC' THEN - FOR i IN REVERSE n..1 LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - ELSE - FOR i IN 1..n LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Sort encrypted values using precomputed ORE keys when available ---! ---! Shared implementation for public sorting entrypoints. The `strategy` ---! parameter selects the comparison path: `'ore'` uses the aligned `ore_keys` ---! array; `'compare'` falls back to `eql_v2.compare()` on the encrypted values ---! directly. ---! ---! @param ids bigint[] Row identifiers aligned with `vals` ---! @param vals eql_v2_encrypted[] Encrypted values to sort ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (used when strategy = 'ore') ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param strategy text One of 'ore' or 'compare' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows -CREATE FUNCTION eql_v2._sort_compare_precomputed( - ids bigint[], - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - direction text DEFAULT 'ASC', - strategy text DEFAULT 'ore' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - m integer; - k integer; - sorted_ids bigint[]; - sorted_vals eql_v2_encrypted[]; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; -BEGIN - n := coalesce(array_length(ids, 1), 0); - m := coalesce(array_length(vals, 1), 0); - - IF n <> m THEN - RAISE EXCEPTION 'ids and vals must have the same length'; - END IF; - - IF strategy = 'ore' THEN - k := coalesce(array_length(ore_keys, 1), 0); - IF n <> k THEN - RAISE EXCEPTION 'ids and ore_keys must have the same length when strategy = ''ore'''; - END IF; - END IF; - - IF n = 0 THEN - RETURN; - END IF; - - IF n = 1 THEN - id := ids[1]; - val := vals[1]; - RETURN NEXT; - RETURN; - END IF; - - SELECT q.ids, q.vals, q.ore_keys - INTO sorted_ids, sorted_vals, sorted_ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, 1, n, strategy) q; - - RETURN QUERY - SELECT emitted.id, emitted.val - FROM eql_v2._emit_sorted_rows(sorted_ids, sorted_vals, direction) emitted; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values using comparison-based quicksort ---! ---! Sorts parallel arrays of identifiers and encrypted values using O(n log n) ---! quicksort with eql_v2.compare(). Returns sorted rows as a table, avoiding ---! the need for unnest() or other array manipulation by callers. ---! ---! When all input rows share an `ore` term the sort uses pre-extracted ORE ---! keys; otherwise it falls back to `eql_v2.compare()` per pair. ---! ---! This function is designed for environments without operator classes (e.g., Supabase) ---! where direct ORDER BY on encrypted columns is not available. ---! ---! @param ids bigint[] Array of row identifiers ---! @param vals eql_v2_encrypted[] Array of encrypted values (must be same length as ids) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @example ---! -- Sort all rows from an encrypted table ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore), ---! 'ASC' ---! ); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore WHERE id > 42), ---! (SELECT array_agg(e ORDER BY id) FROM ore WHERE id > 42), ---! 'DESC' ---! ); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore) ---! ) LIMIT 5; ---! ---! @see eql_v2.compare ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; - i integer; - use_ore boolean := true; - strategy text; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - FOR i IN 1..n LOOP - IF vals[i] IS NULL THEN - sorted_ore_keys[i] := NULL; - ELSE - IF use_ore THEN - IF eql_v2.has_ore_block_u64_8_256(vals[i]) THEN - sorted_ore_keys[i] := eql_v2.order_by(vals[i]); - ELSE - use_ore := false; - END IF; - END IF; - - EXIT WHEN NOT use_ore; - END IF; - END LOOP; - - IF use_ore THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - ids, vals, sorted_ore_keys, direction, strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a table using column and table references ---! ---! Convenience overload that accepts column names, a table name, and an optional ---! filter clause instead of pre-aggregated arrays. Internally constructs the ---! query and delegates to eql_v2.order_by_compare(). ---! ---! @param id_column text Name of the bigint identifier column ---! @param val_column text Name of the eql_v2_encrypted value column ---! @param tbl text Table name (may be schema-qualified) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param filter text Optional WHERE clause (without the WHERE keyword) ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note The id column must be castable to bigint. Uses dynamic SQL internally. ---! @warning The filter parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ascending (default) ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore'); ---! ---! -- Sort descending ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'DESC'); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'ASC', 'id > 42'); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore') LIMIT 10; ---! ---! @see eql_v2.sort_compare(bigint[], eql_v2_encrypted[], text) ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - id_column text, - val_column text, - tbl text, - direction text DEFAULT 'ASC', - filter text DEFAULT NULL -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - query text; - resolved_tbl regclass; -BEGIN - resolved_tbl := to_regclass(tbl); - - IF resolved_tbl IS NULL THEN - RAISE EXCEPTION 'table "%" does not exist', tbl; - END IF; - - query := format('SELECT %I, %I FROM %s', id_column, val_column, resolved_tbl); - - IF filter IS NOT NULL THEN - query := query || ' WHERE ' || filter; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2.order_by_compare(query, direction) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a query using comparison-based quicksort ---! ---! Convenience wrapper that accepts a SQL query string, executes it, collects the ---! results, and returns them sorted. For ORE-backed values this pre-extracts the ---! order key once per row and sorts on that key; other inputs fall back to ---! eql_v2.compare(). The query must return exactly two columns: a bigint ---! identifier and an eql_v2_encrypted value. ---! ---! @param query text SQL query returning (bigint, eql_v2_encrypted) columns ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note Uses dynamic SQL (EXECUTE) so cannot be IMMUTABLE or PARALLEL SAFE ---! @warning The query parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ---! SELECT * FROM eql_v2.order_by_compare('SELECT id, e FROM ore'); ---! ---! -- Sort with WHERE clause ---! SELECT * FROM eql_v2.order_by_compare( ---! 'SELECT id, e FROM ore WHERE id > 42', ---! 'DESC' ---! ); ---! ---! @see eql_v2.sort_compare ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.order_by_compare( - query text, - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - all_ids bigint[]; - all_vals eql_v2_encrypted[]; - all_ore_keys eql_v2.ore_block_u64_8_256[]; - all_have_ore_keys boolean; - strategy text; -BEGIN - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - EXECUTE format( - 'WITH input_rows AS ( - SELECT row_number() OVER () AS ord, - sub.id, - sub.val, - CASE - WHEN sub.val IS NULL THEN NULL - WHEN eql_v2.has_ore_block_u64_8_256(sub.val) THEN eql_v2.order_by(sub.val) - ELSE NULL - END AS ore_key, - CASE - WHEN sub.val IS NULL THEN TRUE - ELSE eql_v2.has_ore_block_u64_8_256(sub.val) - END AS has_ore_key - FROM (%s) sub(id, val) - ) - SELECT array_agg(id ORDER BY ord), - array_agg(val ORDER BY ord), - array_agg(ore_key ORDER BY ord), - coalesce(bool_and(has_ore_key), TRUE) - FROM input_rows', - query - ) INTO all_ids, all_vals, all_ore_keys, all_have_ore_keys; - - IF all_ids IS NULL THEN - RETURN; - END IF; - - IF all_have_ore_keys THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - all_ids, - all_vals, - all_ore_keys, - direction, - strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `>=` testing. ---! The `>=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a >= b (compare result >= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2.">=" -CREATE FUNCTION eql_v2.gte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) >= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than-or-equal operator for encrypted values ---! ---! Implements the >= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a >= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age >= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = jsonb, - RIGHTARG =eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief Greater-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for greater-than ---! testing. The `>` operator wrappers no longer go through this helper — ---! see the inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a > b (compare result = 1) ---! ---! @see eql_v2.compare ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.gt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = 1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than operator for encrypted values ---! ---! Implements the > operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is greater than b ---! ---! @example ---! SELECT * FROM events ---! WHERE encrypted_value > '100'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. Predicate --- `WHERE col > val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) > eql_v2.ore_block_u64_8_256(val)` --- and matches a functional ORE index built on the same expression. --- Breaking impact: columns with only `ore_cllw_*` or OPE terms now --- raise from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where they --- previously fell through `eql_v2.compare`. See U-005. -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION=eql_v2.">", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief Equality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `=` operator's body: reduces to ---! `hmac_256(a) = hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.eq`); equivalent to using the `=` operator ---! directly. ---! ---! Equality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `=` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms match ---! ---! @see eql_v2."=" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.eq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - ---! @brief Equality operator for encrypted values ---! ---! Implements the = operator for comparing two encrypted values using their ---! encrypted index terms (hmac_256). Enables WHERE clause comparisons ---! without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are equal ---! ---! @example ---! -- Compare encrypted columns ---! SELECT * FROM users WHERE encrypted_email = other_encrypted_email; ---! ---! -- Search using encrypted literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col = val` reduces to --- `WHERE eql_v2.hmac_256(col) = eql_v2.hmac_256(val)` and matches a --- functional hash index built on `eql_v2.hmac_256(col)`. Bare equality --- queries (including those issued by PostgREST and ORMs that don't --- wrap columns themselves) become fast on Supabase and any --- --exclude-operator-family install. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2.eq` walked `eql_v2.compare`, which fell back to ORE / Blake3 / --- literal comparison when HMAC wasn't present. Now `=` requires the --- column to have `equality` configured (i.e. carry an `hm` field). --- Calling `=` on an ORE-only column will return NULL where it --- previously returned a Boolean. This is intentional — it surfaces --- config errors loudly. See the predicate/extractor RFC for context. -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - ---! @brief Equality operator for encrypted value and JSONB ---! ---! Overload of = operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Useful for comparing ---! against JSONB literals or columns. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand (will be cast to eql_v2_encrypted) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare encrypted column to JSONB literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::jsonb; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Equality operator for JSONB and encrypted value ---! ---! Overload of = operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Enables commutative ---! equality comparisons. ---! ---! @param a JSONB Left operand (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare JSONB literal to encrypted column ---! SELECT * FROM users ---! WHERE '{"c":"...","i":{"unique":"..."}}'::jsonb = encrypted_email; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - ---! @brief Contained-by operator for encrypted values (<@) ---! ---! Implements the <@ (contained-by) operator for testing if left encrypted value ---! is contained by the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. Reverse of @> operator. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (contained value) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if a is contained by b ---! ---! @example ---! -- Check if value is contained in encrypted array ---! SELECT * FROM documents ---! WHERE '["security"]'::jsonb::eql_v2_encrypted <@ encrypted_tags; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.\"@>\" ---! @see eql_v2.add_search_config - --- Marked IMMUTABLE STRICT PARALLEL SAFE — see operators/@>.sql for rationale. -CREATE FUNCTION eql_v2."<@"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Contains with reversed arguments - SELECT eql_v2.ste_vec_contains(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.stevec_query` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.stevec_query)`. Mirrors the ---! typed needle convention: "is this query payload contained in that ---! encrypted document?". ---! ---! @param a eql_v2.stevec_query Left operand (query payload) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."<@"(a eql_v2.stevec_query, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.stevec_query, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.ste_vec_entry` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)`. Convenience ---! shape for "is this entry contained in that encrypted document?". ---! ---! @param a eql_v2.ste_vec_entry Left operand (single entry) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2."<@"(a eql_v2.ste_vec_entry, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.ste_vec_entry, - RIGHTARG=eql_v2_encrypted -); - ---! @brief Inequality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `<>` operator's body: reduces to ---! `hmac_256(a) <> hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.neq`); equivalent to using the `<>` operator ---! directly. ---! ---! Inequality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `<>` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms differ ---! ---! @see eql_v2."<>" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.neq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - ---! @brief Not-equal operator for encrypted values ---! ---! Implements the <> (not equal) operator for comparing encrypted values using their ---! encrypted index terms. Enables WHERE clause inequality comparisons without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are not equal ---! ---! @example ---! -- Find records with non-matching values ---! SELECT * FROM users ---! WHERE encrypted_email <> 'admin@example.com'::text::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2."=" --- Inlinable; mirrors `=` (see operators/=.sql for rationale). --- Returns NULL on ORE-only encrypted columns (no `hm` field) instead --- of falling back to a slower comparison path; surface the config --- error rather than hide it. -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for encrypted value and JSONB ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for JSONB and encrypted value ---! ---! @param jsonb Plain JSONB value ---! @param eql_v2_encrypted Encrypted value ---! @return boolean True if values are not equal ---! ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) <> eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - - - - ---! @brief Less-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `<=` testing. ---! The `<=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `<=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `<=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a <= b (compare result <= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2."<=" -CREATE FUNCTION eql_v2.lte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) <= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than-or-equal operator for encrypted values ---! ---! Implements the <= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a <= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age <= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for encrypted value and JSONB ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for JSONB and encrypted value ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief Less-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for less-than ---! testing. The `<` operator wrappers no longer call this helper — they ---! inline a direct `ore_block_u64_8_256` comparison instead (see the ---! inlinable bodies below). ---! ---! @warning Behaviour now diverges from the `<` operator: this helper ---! still walks `eql_v2.compare`'s priority list (ore_block → ore_cllw ---! → hm), whereas `<` goes straight to `ore_block_u64_8_256` and raises ---! on missing `ob`. Callers relying on the dispatcher fallback should ---! migrate to the extractor form: `eql_v2.ore_cllw(col) < ---! eql_v2.ore_cllw($1::jsonb)`. See U-005. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a < b (compare result = -1) ---! ---! @see eql_v2.compare ---! @see eql_v2."<" -CREATE FUNCTION eql_v2.lt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = -1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than operator for encrypted values ---! ---! Implements the < operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term (configured ---! via the `ore` index in the EQL schema). ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is less than b ---! ---! @example ---! -- Range query on encrypted timestamps ---! SELECT * FROM events ---! WHERE encrypted_timestamp < '2024-01-01'::timestamp::text::eql_v2_encrypted; ---! ---! -- Compare encrypted numeric columns ---! SELECT * FROM products WHERE encrypted_price < encrypted_discount_price; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col < val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) < eql_v2.ore_block_u64_8_256(val)` --- and matches a functional btree index built on --- `eql_v2.ore_block_u64_8_256(col)` (using the DEFAULT --- `eql_v2.ore_block_u64_8_256_operator_class`). Bare range queries --- (`WHERE col < $1`) engage the functional ORE index on Supabase and any --- install that doesn't ship `eql_v2.encrypted_operator_class`. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2."<"` walked `eql_v2.compare`, which dispatched through --- ore_block / ore_cllw_u64 / ore_cllw_var / ope. Now `<` requires the --- column to have `ore_block_u64_8_256` configured (i.e. carry an `ob` --- field). Calling `<` on a column with only `ore_cllw_*` or OPE terms --- now raises from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where it --- previously returned a Boolean. Loud failure surfaces config errors --- rather than silently producing zero rows — see U-005. -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for encrypted value and JSONB ---! ---! Overload of < operator accepting JSONB on the right side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides; the jsonb ---! extractor `eql_v2.ore_block_u64_8_256(jsonb)` reads `b->'ob'` directly. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE encrypted_age < '{"ob":[...]}'::jsonb; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for JSONB and encrypted value ---! ---! Overload of < operator accepting JSONB on the left side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides. ---! ---! @param a JSONB Left operand ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE '{"ob":[...]}'::jsonb < encrypted_date; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief JSONB field accessor operator alias (->>) ---! ---! Implements the ->> operator as an alias of -> for encrypted JSONB data. This mirrors ---! PostgreSQL semantics where ->> returns text via implicit casts. The underlying ---! implementation delegates to eql_v2."->" and allows PostgreSQL to coerce the result. ---! ---! Provides two overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! ---! @see eql_v2."->" ---! @see eql_v2.selector - ---! @brief ->> operator with text selector ---! @param eql_v2_encrypted Encrypted JSONB data ---! @param text Field name to extract ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @example ---! SELECT encrypted_json ->> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector text) - RETURNS text -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - found eql_v2_encrypted; - BEGIN - -- found = eql_v2."->"(e, selector); - -- RETURN eql_v2.ciphertext(found); - RETURN eql_v2."->"(e, selector); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - - - ---------------------------------------------------- - ---! @brief ->> operator with encrypted selector ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted field selector ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @see eql_v2."->>"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2."->>"(e, eql_v2._selector(selector)); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - ---! @brief JSONB field accessor operator for encrypted values (->) ---! ---! Implements the -> operator to access fields/elements from encrypted JSONB data. ---! Returns the matching sv entry as `eql_v2.ste_vec_entry` (or NULL on miss). ---! ---! Encrypted JSON is represented as an array of sv elements in the ---! StEVec format. Each element has a selector, ciphertext, and index ---! terms: `{"sv": [{"c": "...", "s": "...", "hm": "..."}, ...]}`. ---! ---! Provides three overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! - (eql_v2_encrypted, integer) - Array index selector (0-based) ---! ---! All three return `eql_v2.ste_vec_entry` and preserve the source ---! payload's root `i` / `v` envelope metadata in the returned entry ---! (the DOMAIN CHECK on `ste_vec_entry` doesn't forbid extra fields). ---! ---! @note Operator resolution: Assignment casts are considered (PostgreSQL standard behavior). ---! To use text selector, parameter may need explicit cast to text. ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2.selector ---! @see eql_v2."->>" - ---! @brief -> operator with text selector ---! ---! Returns the sv entry whose `s` selector equals @p selector, with ---! the source payload's `i` / `v` metadata merged in. Selectors are ---! deterministic per (path, key) within a document, so at most one ---! entry matches; `jsonb_path_query_first` returns the first match ---! and stops scanning. ---! ---! Inlinable single-statement SQL: the planner folds this body into ---! the calling query, so `WHERE col -> 'sel' = $1` reduces structurally ---! to `eql_v2.eq_term(col -> 'sel') = eql_v2.eq_term($1)` and matches ---! a functional index built on `eql_v2.eq_term(col -> 'sel')`. ---! ---! @param e eql_v2_encrypted Encrypted JSONB payload (root) ---! @param selector text Selector hash (the `s` field value) ---! @return eql_v2.ste_vec_entry Matching entry merged with root meta, ---! NULL if no element matches. ---! ---! @note The returned entry carries `i` / `v` from the root in addition ---! to the sv-element fields. This is intentional: per-entry ---! extractors (`eql_v2.eq_term`, `eql_v2.ore_cllw`, ...) read ---! only their own fields and ignore `i` / `v`; callers that need ---! the root envelope (e.g. for decryption) still see it. ---! ---! @example ---! SELECT encrypted_json -> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector text) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ( - eql_v2.meta_data(e) || - jsonb_path_query_first( - (e).data, - '$.sv[*] ? (@.s == $sel)'::jsonpath, - jsonb_build_object('sel', selector) - ) - )::eql_v2.ste_vec_entry -$$; - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - ---------------------------------------------------- - ---! @brief -> operator with encrypted selector ---! ---! Convenience overload: extracts the selector text from an encrypted ---! selector payload and delegates to the (text) form. Inlinable. ---! ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted selector payload ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @see eql_v2."->"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."->"(e, eql_v2._selector(selector)) -$$; - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---------------------------------------------------- - ---! @brief -> operator with integer array index ---! ---! Returns the sv entry at the given (0-based, JSONB-style) array ---! index, merged with the root payload's `i` / `v` metadata. Returns ---! NULL when the underlying value isn't an sv-array payload or when ---! the index is out of bounds. ---! ---! @param e eql_v2_encrypted Encrypted sv-array payload ---! @param selector integer Array index (0-based, JSONB convention) ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @note Array index is 0-based (JSONB standard) despite PostgreSQL arrays being 1-based ---! @example ---! SELECT encrypted_array -> 0 FROM table; ---! @see eql_v2.is_ste_vec_array -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector integer) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE - WHEN eql_v2.is_ste_vec_array(e) THEN - (eql_v2.meta_data(e) || ((e).data -> 'sv' -> selector))::eql_v2.ste_vec_entry - ELSE NULL - END -$$; - - - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=integer -); - - ---! @brief EQL lint: detect non-inlinable operator implementation functions ---! ---! Returns one row per violation found in the installed EQL surface. The ---! Postgres planner can only inline a function during index matching when: ---! ---! * `LANGUAGE sql` (plpgsql / C / etc. cannot be inlined) ---! * `IMMUTABLE` or `STABLE` volatility (VOLATILE cannot be inlined into ---! index expressions) ---! * No `SET` clauses (e.g. `SET search_path = ...`) ---! * Not `SECURITY DEFINER` ---! * Single-statement SELECT body ---! ---! @note The single-statement SELECT body condition is **not yet checked** by ---! this lint. A `LANGUAGE sql` function with a multi-statement body, a CTE, ---! or any pre-SELECT statement will pass all four implemented checks while ---! remaining non-inlinable. Implementing the check requires walking `prosrc` ---! (or `pg_get_functiondef`); tracked as a follow-up to #194. ---! ---! Operators on encrypted types (`eql_v2_encrypted`, `eql_v2.bloom_filter`, ---! `eql_v2.ore_*`, etc.) whose implementation functions fail any of these ---! rules silently fall back to seq scan when the documented functional ---! indexes (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, ---! `eql_v2.ste_vec(col)`) are in place. This lint surfaces every such case. ---! ---! Severity: ---! `error` — fixable, blocks index matching, ship-blocking. ---! `warning` — likely-fixable, may not block matching but signals intent. ---! `info` — observational; useful for review, not a defect on its own. ---! ---! Categories: ---! `inlinability_language` — implementation function isn't `LANGUAGE sql`. ---! `inlinability_volatility` — implementation function is VOLATILE. ---! `inlinability_set_clause` — implementation function has a `SET` clause. ---! `inlinability_secdef` — implementation function is `SECURITY DEFINER`. ---! `inlinability_transitive` — implementation function is itself inlinable ---! but its body invokes a non-inlinable function ---! (depth 1; the planner can't peek through ---! that boundary). ---! ---! @example ---! ``` ---! SELECT severity, category, object_name, message ---! FROM eql_v2.lints() ---! WHERE severity = 'error' ---! ORDER BY category, object_name; ---! ``` ---! ---! @return SETOF record (severity text, category text, object_name text, message text) -CREATE OR REPLACE FUNCTION eql_v2.lints() -RETURNS TABLE ( - severity text, - category text, - object_name text, - message text -) -LANGUAGE sql STABLE -AS $$ - WITH - -- All operators where at least one operand involves an EQL type. Limits - -- the scope of the lint to the operator surface customers actually hit - -- via SQL (`col = val`, `col LIKE '...'`, `col @> '...'` and friends). - eql_operators AS ( - SELECT - op.oid AS oprid, - op.oprname AS opname, - op.oprcode AS implfunc, - op.oprleft::regtype AS lhs, - op.oprright::regtype AS rhs, - op.oprcode::regprocedure AS impl_signature - FROM pg_operator op - WHERE EXISTS ( - SELECT 1 FROM pg_type t - WHERE t.oid IN (op.oprleft, op.oprright) - AND (t.typname LIKE 'eql_v2%' - OR t.typnamespace = 'eql_v2'::regnamespace) - ) - ), - - -- Cross-join with each operator's implementation function metadata. - -- One row per operator; columns describe the inlinability of the impl. - op_impl AS ( - SELECT - eo.opname, - eo.lhs, - eo.rhs, - eo.impl_signature::text AS impl_signature, - lang_l.lanname AS lang, - p.provolatile AS volatility, - p.proconfig AS config, - p.prosecdef AS secdef, - p.prosrc AS body - FROM eql_operators eo - JOIN pg_proc p ON p.oid = eo.implfunc - JOIN pg_language lang_l ON lang_l.oid = p.prolang - ) - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Direct inlinability checks: each row examines one operator's │ - -- │ implementation function and emits a violation if any rule is │ - -- │ broken. Multiple violations on the same function become │ - -- │ multiple rows (developers see every reason it doesn't inline). │ - -- └─────────────────────────────────────────────────────────────────┘ - - SELECT - 'error' AS severity, - 'inlinability_language' AS category, - format('operator %s(%s, %s) -> %s', - opname, lhs, rhs, impl_signature) AS object_name, - format( - 'Operator implementation function is `LANGUAGE %s`; only `LANGUAGE sql` functions can be inlined by the planner. Bare `col %s val` queries fall back to seq scan even when a matching functional index exists.', - lang, opname) AS message - FROM op_impl - WHERE lang <> 'sql' - - UNION ALL - - SELECT - 'error', - 'inlinability_volatility', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function is `VOLATILE`. The Postgres planner refuses to inline volatile functions into index expressions, so functional indexes never engage. Mark the function `IMMUTABLE` (or `STABLE` if it depends on session state).', - opname) - FROM op_impl - WHERE volatility = 'v' - - UNION ALL - - SELECT - 'error', - 'inlinability_set_clause', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function has a `SET` clause (e.g. `SET search_path = ...`). Per Postgres function-inlining rules, any `SET` clause blocks inlining. Use schema-qualified identifiers in the body and remove the `SET` clause to allow the planner to inline.') - FROM op_impl - WHERE config IS NOT NULL - - UNION ALL - - SELECT - 'error', - 'inlinability_secdef', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - 'Operator implementation function is `SECURITY DEFINER`. Such functions cannot be inlined; remove `SECURITY DEFINER` or use a non-inlinable wrapper layer.' - FROM op_impl - WHERE secdef - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Transitive inlinability: an operator implementation function │ - -- │ that's itself inlinable can still fail to inline if its body │ - -- │ calls a non-inlinable function. Walk one level via pg_depend. │ - -- │ │ - -- │ Postgres records function-to-function dependencies in │ - -- │ pg_depend with deptype 'n' (normal) when one function references│ - -- │ another in its body — but only at CREATE time and only for │ - -- │ direct calls. This is good enough for v1; deeper transitive │ - -- │ analysis is a follow-up. │ - -- └─────────────────────────────────────────────────────────────────┘ - - UNION ALL - - SELECT - 'error', - 'inlinability_transitive', - format('operator %s(%s, %s) -> %s', oi.opname, oi.lhs, oi.rhs, - oi.impl_signature), - format( - 'Operator implementation function is inlinable but invokes non-inlinable function `%s` (lang=%s, volatility=%s%s). The chain blocks at depth 1: the planner inlines the outer call but cannot reduce the inner call into an index expression.', - called.proname, - called_lang.lanname, - CASE called.provolatile - WHEN 'i' THEN 'IMMUTABLE' - WHEN 's' THEN 'STABLE' - WHEN 'v' THEN 'VOLATILE' - END, - CASE WHEN called.proconfig IS NOT NULL - THEN ', has SET clause' - ELSE '' END) - FROM op_impl oi - -- Only worth the transitive check if the outer function is otherwise - -- inlinable — otherwise the direct lints above already report it. - JOIN pg_proc outer_p ON outer_p.oid = oi.impl_signature::regprocedure - JOIN pg_depend d - ON d.classid = 'pg_proc'::regclass - AND d.objid = outer_p.oid - AND d.refclassid = 'pg_proc'::regclass - AND d.deptype = 'n' - JOIN pg_proc called ON called.oid = d.refobjid - JOIN pg_language called_lang ON called_lang.oid = called.prolang - WHERE oi.lang = 'sql' - AND oi.volatility IN ('i', 's') - AND oi.config IS NULL - AND NOT oi.secdef - AND called.oid <> outer_p.oid - AND ( - called_lang.lanname <> 'sql' - OR called.provolatile = 'v' - OR called.proconfig IS NOT NULL - OR called.prosecdef - ) - - ORDER BY 1, 2, 3; -$$; - -COMMENT ON FUNCTION eql_v2.lints() IS - 'EQL lint: returns one row per non-inlinable operator implementation. ' - 'Run `SELECT * FROM eql_v2.lints() WHERE severity = ''error''` for a ' - 'CI-gateable check that all operator implementations on EQL types are ' - 'eligible for planner inlining.'; - ---! @file jsonb/functions.sql ---! @brief JSONB path query and array manipulation functions for encrypted data ---! ---! These functions provide PostgreSQL-compatible operations on encrypted JSONB values ---! using Structured Transparent Encryption (STE). They support: ---! - Path-based queries to extract nested encrypted values ---! - Existence checks for encrypted fields ---! - Array operations (length, elements extraction) ---! - Field-level HMAC term extraction for equality / GROUP BY / DISTINCT ---! ---! @note STE stores encrypted JSONB as a vector of encrypted elements ('sv') with selectors ---! @note Functions suppress errors for missing fields, type mismatches (similar to PostgreSQL jsonpath) ---! @note `selector` parameters in this module are *encrypted-side* selector ---! hashes — the deterministic hash that the crypto layer (e.g. ---! `@cipherstash/protect`) emits in the `s` field of each `sv` element ---! (e.g. `'a7cea93975ed8c01f861ccb6bd082784'`). Plaintext JSONPaths ---! like `'$.address.city'` are never accepted at runtime; the proxy / ---! client rewrites them to selector hashes before the query reaches EQL. - - ---! @brief Query encrypted JSONB for elements matching selector ---! ---! Searches the Structured Transparent Encryption (STE) vector for elements matching ---! the given selector path. Returns all matching encrypted elements. If multiple ---! matches form an array, they are wrapped with array metadata. ---! ---! @param jsonb Encrypted JSONB payload containing STE vector ('sv') ---! @param text Path selector to match against encrypted elements ---! @return SETOF eql_v2_encrypted Matching encrypted elements (may return multiple rows) ---! ---! @note Returns empty set if selector is not found (does not throw exception) ---! @note Array elements use same selector; multiple matches wrapped with 'a' flag ---! @note Returns a set containing NULL if val is NULL; returns empty set if no matches found ---! @see eql_v2.jsonb_path_query_first ---! @see eql_v2.jsonb_path_exists -CREATE FUNCTION eql_v2.jsonb_path_query(val jsonb, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT - CASE - WHEN bool_or(eql_v2.is_ste_vec_array(elem)) THEN - (eql_v2.meta_data(val) || jsonb_build_object('sv', jsonb_agg(elem), 'a', 1))::eql_v2_encrypted - ELSE - (eql_v2.meta_data(val) || (array_agg(elem))[1])::eql_v2_encrypted - END - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - HAVING count(*) > 0 -$$; - - ---! @brief Query encrypted JSONB with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its plaintext value ---! before delegating to main jsonb_path_query implementation. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Query encrypted JSONB with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector, ---! extracting the JSONB payload before querying. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @example ---! -- Query encrypted JSONB for the sv element at a given selector hash ---! SELECT * FROM eql_v2.jsonb_path_query(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Check if selector path exists in encrypted JSONB ---! ---! Tests whether any encrypted elements match the given selector path. ---! More efficient than jsonb_path_query when only existence check is needed. ---! ---! @param jsonb Encrypted JSONB payload to check ---! @param text Path selector to test ---! @return boolean True if matching element exists, false otherwise ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val jsonb, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT 1 FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - ); -$$; - - ---! @brief Check existence with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before checking existence. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to check ---! @param selector eql_v2_encrypted Encrypted selector to test ---! @return boolean True if path exists ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Check existence with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to check ---! @param text Path selector to test ---! @return boolean True if path exists ---! ---! @example ---! -- Check if the encrypted document has an sv element at a given selector hash ---! SELECT eql_v2.jsonb_path_exists(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Get first element matching selector ---! ---! Returns only the first encrypted element matching the selector path, ---! or NULL if no match found. More efficient than jsonb_path_query when ---! only one result is needed. ---! ---! @param jsonb Encrypted JSONB payload to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @note Uses LIMIT 1 internally for efficiency ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val jsonb, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (eql_v2.meta_data(val) || elem)::eql_v2_encrypted - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - LIMIT 1 -$$; - - ---! @brief Get first element with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before querying for first match. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Get first element with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @example ---! -- Get the first matching sv element from an encrypted document ---! SELECT eql_v2.jsonb_path_query_first(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, selector); -$$; - - - ------------------------------------------------------------------------------------- - - ---! @brief Get length of encrypted JSONB array ---! ---! Returns the number of elements in an encrypted JSONB array by counting ---! elements in the STE vector ('sv'). The encrypted value must have the ---! array flag ('a') set to true. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return integer Number of elements in the array ---! @throws Exception 'cannot get array length of a non-array' if 'a' flag is missing or not true ---! ---! @note Array flag 'a' must be present and set to true value ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_length(val jsonb) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_array(val) THEN - sv := eql_v2.ste_vec(val); - RETURN array_length(sv, 1); - END IF; - - RAISE 'cannot get array length of a non-array'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get array length from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts the ---! JSONB payload before computing array length. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return integer Number of elements in the array ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get length of encrypted array ---! SELECT eql_v2.jsonb_array_length(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_length(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_length(val eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN ( - SELECT eql_v2.jsonb_array_length(val.data) - ); - END; -$$ LANGUAGE plpgsql; - - - - ---! @brief Extract elements from encrypted JSONB array ---! ---! Returns each element of an encrypted JSONB array as a separate row. ---! Each element is returned as an eql_v2_encrypted value with metadata ---! preserved from the parent array. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Each element inherits metadata (version, ident) from parent ---! @see eql_v2.jsonb_array_length ---! @see eql_v2.jsonb_array_elements_text -CREATE FUNCTION eql_v2.jsonb_array_elements(val jsonb) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - meta jsonb; - item jsonb; - BEGIN - - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - -- Column identifier and version - meta := eql_v2.meta_data(val); - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - item = sv[idx]; - RETURN NEXT (meta || item)::eql_v2_encrypted; - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract elements from encrypted array type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element as a separate row. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Expand encrypted array into rows ---! SELECT * FROM eql_v2.jsonb_array_elements(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements(val eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract encrypted array elements as ciphertext ---! ---! Returns each element of an encrypted JSONB array as its raw ciphertext ---! value (text representation). Unlike jsonb_array_elements, this returns ---! only the ciphertext 'c' field without metadata. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Returns ciphertext only, not full encrypted structure ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val jsonb) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - RETURN NEXT eql_v2.ciphertext(sv[idx]); - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract array elements as ciphertext from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element's ciphertext as text. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get ciphertext of each array element ---! SELECT * FROM eql_v2.jsonb_array_elements_text(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements_text(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val eql_v2_encrypted) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements_text(val.data); - END; -$$ LANGUAGE plpgsql; - - ------------------------------------------------------------------------------------- - --- `eql_v2.hmac_256_terms(eql_v2_encrypted)` was added under #205 as a --- GIN-indexable {s, hm} aggregate. It's been removed: under the XOR --- contract each sv element carries exactly one of `hm` (bool leaves, --- array / object roots) or `oc` (string / number leaves), and --- `hmac_256_terms` filters out everything without `hm` — so containment --- queries via this index could never match on string / number selectors. --- The canonical XOR-aware replacement is the typed --- `@>(eql_v2_encrypted, eql_v2.stevec_query)` overload, which inlines --- to `eql_v2.to_stevec_query(col)::jsonb @> needle::jsonb` and engages --- a functional GIN on `(eql_v2.to_stevec_query(col)::jsonb) jsonb_path_ops`. --- See U-007 / U-008 in `docs/upgrading/v2.3.md`. ---! @file encryptindex/functions.sql ---! @brief Configuration lifecycle and column encryption management ---! ---! Provides functions for managing encryption configuration transitions: ---! - Comparing configurations to identify changes ---! - Identifying columns needing encryption ---! - Creating and renaming encrypted columns during initial setup ---! - Tracking encryption progress ---! ---! These functions support the workflow of activating a pending configuration ---! and performing the initial encryption of plaintext columns. - - ---! @brief Compare two configurations and find differences ---! @internal ---! ---! Returns table/column pairs where configuration differs between two configs. ---! Used to identify which columns need encryption when activating a pending config. ---! ---! @param a jsonb First configuration to compare ---! @param b jsonb Second configuration to compare ---! @return TABLE(table_name text, column_name text) Columns with differing configuration ---! ---! @note Compares configuration structure, not just presence/absence ---! @see eql_v2.select_pending_columns -CREATE FUNCTION eql_v2.diff_config(a JSONB, b JSONB) - RETURNS TABLE(table_name TEXT, column_name TEXT) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - WITH table_keys AS ( - SELECT jsonb_object_keys(a->'tables') AS key - UNION - SELECT jsonb_object_keys(b->'tables') AS key - ), - column_keys AS ( - SELECT tk.key AS table_key, jsonb_object_keys(a->'tables'->tk.key) AS column_key - FROM table_keys tk - UNION - SELECT tk.key AS table_key, jsonb_object_keys(b->'tables'->tk.key) AS column_key - FROM table_keys tk - ) - SELECT - ck.table_key AS table_name, - ck.column_key AS column_name - FROM - column_keys ck - WHERE - (a->'tables'->ck.table_key->ck.column_key IS DISTINCT FROM b->'tables'->ck.table_key->ck.column_key); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get columns with pending configuration changes ---! ---! Compares 'pending' and 'active' configurations to identify columns that need ---! encryption or re-encryption. Returns columns where configuration differs. ---! ---! @return TABLE(table_name text, column_name text) Columns needing encryption ---! @throws Exception if no pending configuration exists ---! ---! @note Treats missing active config as empty config ---! @see eql_v2.diff_config ---! @see eql_v2.select_target_columns -CREATE FUNCTION eql_v2.select_pending_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - active JSONB; - pending JSONB; - config_id BIGINT; - BEGIN - SELECT data INTO active FROM eql_v2_configuration WHERE state = 'active'; - - -- set default config - IF active IS NULL THEN - active := '{}'; - END IF; - - SELECT id, data INTO config_id, pending FROM eql_v2_configuration WHERE state = 'pending'; - - -- set default config - IF config_id IS NULL THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - RETURN QUERY - SELECT d.table_name, d.column_name FROM eql_v2.diff_config(active, pending) as d; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Map pending columns to their encrypted target columns ---! ---! For each column with pending configuration, identifies the corresponding ---! encrypted column. During initial encryption, target is '{column_name}_encrypted'. ---! Returns NULL for target_column if encrypted column doesn't exist yet. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Column mappings ---! ---! @note Target column is NULL if no column exists matching either 'column_name' or 'column_name_encrypted' with type eql_v2_encrypted ---! @note The LEFT JOIN checks both original and '_encrypted' suffix variations with type verification ---! @see eql_v2.select_pending_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.select_target_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT - c.table_name, - c.column_name, - s.column_name as target_column - FROM - eql_v2.select_pending_columns() c - LEFT JOIN information_schema.columns s ON - s.table_name = c.table_name AND - (s.column_name = c.column_name OR s.column_name = c.column_name || '_encrypted') AND - s.udt_name = 'eql_v2_encrypted'; -$$ LANGUAGE sql; - - ---! @brief Check if database is ready for encryption ---! ---! Verifies that all columns with pending configuration have corresponding ---! encrypted target columns created. Returns true if encryption can proceed. ---! ---! @return boolean True if all pending columns have target encrypted columns ---! ---! @note Returns false if any pending column lacks encrypted column ---! @see eql_v2.select_target_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.ready_for_encryption() - RETURNS BOOLEAN - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT * - FROM eql_v2.select_target_columns() AS c - WHERE c.target_column IS NOT NULL); -$$ LANGUAGE sql; - - ---! @brief Create encrypted columns for initial encryption ---! ---! For each plaintext column with pending configuration that lacks an encrypted ---! target column, creates a new column '{column_name}_encrypted' of type ---! eql_v2_encrypted. This prepares the database schema for initial encryption. ---! ---! @return TABLE(table_name text, column_name text) Created encrypted columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE ADD COLUMN) - modifies database schema ---! @note Only creates columns that don't already exist ---! @see eql_v2.select_target_columns ---! @see eql_v2.rename_encrypted_columns -CREATE FUNCTION eql_v2.create_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name IN - SELECT c.table_name, (c.column_name || '_encrypted') FROM eql_v2.select_target_columns() AS c WHERE c.target_column IS NULL - LOOP - EXECUTE format('ALTER TABLE %I ADD column %I eql_v2_encrypted;', table_name, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Finalize initial encryption by renaming columns ---! ---! After initial encryption completes, renames columns to complete the transition: ---! - Plaintext column '{column_name}' → '{column_name}_plaintext' ---! - Encrypted column '{column_name}_encrypted' → '{column_name}' ---! ---! This makes the encrypted column the primary column with the original name. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Renamed columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE RENAME COLUMN) - modifies database schema ---! @note Only renames columns where target is '{column_name}_encrypted' ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.rename_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name, target_column IN - SELECT * FROM eql_v2.select_target_columns() as c WHERE c.target_column = c.column_name || '_encrypted' - LOOP - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, column_name, column_name || '_plaintext'); - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, target_column, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Count rows encrypted with active configuration ---! @internal ---! ---! Counts rows in a table where the encrypted column was encrypted using ---! the currently active configuration. Used to track encryption progress. ---! ---! @param table_name text Name of table to check ---! @param column_name text Name of encrypted column to check ---! @return bigint Count of rows encrypted with active configuration ---! ---! @note The 'v' field in encrypted payloads stores the payload version ("2"), not the configuration ID ---! @note Configuration tracking mechanism is implementation-specific -CREATE FUNCTION eql_v2.count_encrypted_with_active_config(table_name TEXT, column_name TEXT) - RETURNS BIGINT - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result BIGINT; -BEGIN - EXECUTE format( - 'SELECT COUNT(%I) FROM %s t WHERE %I->>%L = (SELECT id::TEXT FROM eql_v2_configuration WHERE state = %L)', - column_name, table_name, column_name, 'v', 'active' - ) - INTO result; - RETURN result; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Compute hash integer for encrypted value ---! ---! Produces a 32-bit integer hash suitable for PostgreSQL hash joins, GROUP BY, ---! DISTINCT, and hash aggregate operations. Used by the `eql_v2_encrypted` hash ---! operator class (`FUNCTION 1`). Inlinable single-statement SQL — the SQL ---! function machinery is much cheaper per row than plpgsql, which matters ---! because HashAggregate / hash-join call this once per input row. ---! ---! Returns `hashtext` of the root payload's `hm` term. This is the canonical ---! bucket for equality groups, since `=` on `eql_v2_encrypted` reduces to ---! `hmac_256(a) = hmac_256(b)` post-#193. ---! ---! @par Contract ---! Callers using `GROUP BY` / `DISTINCT` / hash joins on `eql_v2_encrypted` ---! MUST configure the column with a `unique` index so the crypto layer ---! emits `hm` — `hm` is assumed present. A missing `hm` is a misconfiguration ---! that surfaces upstream via [U-002](docs/upgrading/v2.3.md#u-002-equality-and-hashing-require-hmac). ---! ---! @param val eql_v2_encrypted Encrypted value to hash ---! @return integer 32-bit hash value derived from `hm` ---! ---! @note For grouping a value extracted from an encrypted JSON document, use ---! the field-level recipe directly: `GROUP BY eql_v2.eq_term(col -> '<selector>')` ---! (covers both hm-bearing and oc-bearing selectors via the XOR-aware ---! extractor — see `src/ste_vec/eq_term.sql`). That bypasses ---! `hash_encrypted` entirely. ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.hash_encrypted(val eql_v2_encrypted) - RETURNS integer - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT pg_catalog.hashtext(eql_v2.hmac_256(val)::text) -$$; - - ---! @brief Validate presence of ident field in encrypted payload ---! @internal ---! ---! Checks that the encrypted JSONB payload contains the required 'i' (ident) field. ---! The ident field tracks which table and column the encrypted value belongs to. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'i' field is present ---! @throws Exception if 'i' field is missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'i' THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ident (i) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate table and column fields in ident ---! @internal ---! ---! Checks that the 'i' (ident) field contains both 't' (table) and 'c' (column) ---! subfields, which identify the origin of the encrypted value. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if both 't' and 'c' subfields are present ---! @throws Exception if 't' or 'c' subfields are missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i_ct(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val->'i' ?& array['t', 'c']) THEN - RETURN true; - END IF; - RAISE 'Encrypted column ident (i) missing table (t) or column (c) fields: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Validate version field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload has version field 'v' set to '2', ---! the current EQL v2 payload version. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'v' field is present and equals '2' ---! @throws Exception if 'v' field is missing or not '2' ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_v(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - - IF val->>'v' <> '2' THEN - RAISE 'Expected encrypted column version (v) 2'; - RETURN false; - END IF; - - RETURN true; - END IF; - RAISE 'Encrypted column missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ciphertext field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload carries the required root-level ciphertext ---! envelope. The v2.3 payload schema admits two mutually exclusive top-level ---! shapes (`docs/reference/schema/eql-payload-v2.3.schema.json`): ---! ---! - `EncryptedPayload` (scalar) — carries `c` at the root. ---! - `SteVecPayload` (jsonb / structured) — carries `sv` at the root; the ---! root document ciphertext lives inside `sv[0].c`, so `c` is absent at ---! the root. ---! ---! Either shape satisfies this check. Per-element ciphertext validity on ---! `sv` entries is enforced separately by the `eql_v2.ste_vec_entry` DOMAIN. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if either 'c' or 'sv' is present at the root ---! @throws Exception if neither 'c' nor 'sv' is present ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_c(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'c') OR (val ? 'sv') THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ciphertext (c) or ste_vec (sv) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate complete encrypted payload structure ---! ---! Comprehensive validation function that checks all required fields in an ---! encrypted JSONB payload: version ('v'), ciphertext ('c'), ident ('i'), ---! and ident subfields ('t', 'c'). ---! ---! This function is used in CHECK constraints to ensure encrypted column ---! data integrity at the database level. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if all structure checks pass ---! @throws Exception if any required field is missing or invalid ---! ---! @example ---! -- Add validation constraint to encrypted column ---! ALTER TABLE users ADD CONSTRAINT check_email_encrypted ---! CHECK (eql_v2.check_encrypted(encrypted_email::jsonb)); ---! ---! @see eql_v2._encrypted_check_v ---! @see eql_v2._encrypted_check_c ---! @see eql_v2._encrypted_check_i ---! @see eql_v2._encrypted_check_i_ct -CREATE FUNCTION eql_v2.check_encrypted(val jsonb) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN ( - eql_v2._encrypted_check_v(val) AND - eql_v2._encrypted_check_c(val) AND - eql_v2._encrypted_check_i(val) AND - eql_v2._encrypted_check_i_ct(val) - ); -END; - - ---! @brief Validate encrypted composite type structure ---! ---! Validates an eql_v2_encrypted composite type by checking its underlying ---! JSONB payload. Delegates to eql_v2.check_encrypted(jsonb). ---! ---! @param eql_v2_encrypted Encrypted value to validate ---! @return Boolean True if structure is valid ---! @throws Exception if any required field is missing or invalid ---! ---! @see eql_v2.check_encrypted(jsonb) -CREATE FUNCTION eql_v2.check_encrypted(val eql_v2_encrypted) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN eql_v2.check_encrypted(val.data); -END; - - ---! @brief Fallback literal comparison for encrypted values ---! @internal ---! ---! Compares two encrypted values by their raw JSONB representation when no ---! suitable index terms are available. This ensures consistent ordering required ---! for btree correctness and prevents "lock BufferContent is not held" errors. ---! ---! Used as a last resort fallback in eql_v2.compare() when encrypted values ---! lack matching index terms (hmac_256, ore). ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note This compares the encrypted payloads directly, not the plaintext values ---! @note Ordering is consistent but not meaningful for range queries ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.compare_literal(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT CASE - WHEN a.data < b.data THEN -1 - WHEN a.data > b.data THEN 1 - ELSE 0 - END; -$$; - --- Aggregate functions for ORE - ---! @brief State transition function for min aggregate ---! @internal ---! ---! Returns the smaller of two encrypted values for use in MIN aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The smaller of the two values ---! ---! @see eql_v2.min(eql_v2_encrypted) -CREATE FUNCTION eql_v2.min(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a < b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find minimum encrypted value in a group ---! ---! Aggregate function that returns the minimum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Minimum value in the group ---! ---! @example ---! -- Find minimum age per department ---! SELECT department, eql_v2.min(encrypted_age) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.min(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.min(eql_v2_encrypted) -( - sfunc = eql_v2.min, - stype = eql_v2_encrypted -); - - ---! @brief State transition function for max aggregate ---! @internal ---! ---! Returns the larger of two encrypted values for use in MAX aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The larger of the two values ---! ---! @see eql_v2.max(eql_v2_encrypted) -CREATE FUNCTION eql_v2.max(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a > b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find maximum encrypted value in a group ---! ---! Aggregate function that returns the maximum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Maximum value in the group ---! ---! @example ---! -- Find maximum salary per department ---! SELECT department, eql_v2.max(encrypted_salary) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.max(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.max(eql_v2_encrypted) -( - sfunc = eql_v2.max, - stype = eql_v2_encrypted -); - - ---! @file config/indexes.sql ---! @brief Configuration state uniqueness indexes ---! ---! Creates partial unique indexes to enforce that only one configuration ---! can be in 'active', 'pending', or 'encrypting' state at any time. ---! Multiple 'inactive' configurations are allowed. ---! ---! @note Uses partial indexes (WHERE clauses) for efficiency ---! @note Prevents conflicting configurations from being active simultaneously ---! @see config/types.sql for state definitions - - ---! @brief Unique active configuration constraint ---! @note Only one configuration can be 'active' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'active'; - ---! @brief Unique pending configuration constraint ---! @note Only one configuration can be 'pending' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'pending'; - ---! @brief Unique encrypting configuration constraint ---! @note Only one configuration can be 'encrypting' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'encrypting'; - - ---! @brief Add a search index configuration for an encrypted column ---! ---! Configures a searchable encryption index (unique, match, ore, ope, or ste_vec) ---! on an encrypted column. Creates or updates the pending configuration, then ---! migrates and activates it unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to configure ---! @param index_name Text Type of index ('unique', 'match', 'ore', 'ope', 'ste_vec') ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB Index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if index already exists for this column ---! @throws Exception if cast_as is not a valid type ---! ---! @example ---! -- Add unique index for exact-match searches ---! SELECT eql_v2.add_search_config('users', 'email', 'unique'); ---! ---! -- Add match index for LIKE searches with custom token length ---! SELECT eql_v2.add_search_config('posts', 'content', 'match', 'text', ---! '{"token_filters": [{"kind": "downcase"}], "tokenizer": {"kind": "ngram", "token_length": 3}}' ---! ); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.add_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - o jsonb; - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if index exists - IF _config #> array['tables', table_name, column_name, 'indexes'] ? index_name THEN - RAISE EXCEPTION '% index exists for column: % %', index_name, table_name, column_name; - END IF; - - IF NOT cast_as = ANY('{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}') THEN - RAISE EXCEPTION '% is not a valid cast type', cast_as; - END IF; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- set default options for index if opts empty - IF index_name = 'match' AND opts = '{}' THEN - SELECT eql_v2.config_match_default() INTO opts; - END IF; - - SELECT eql_v2.config_add_index(table_name, column_name, index_name, opts, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a search index configuration from an encrypted column ---! ---! Removes a previously configured search index from an encrypted column. ---! Updates the pending configuration, then migrates and activates it ---! unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove match index from column ---! SELECT eql_v2.remove_search_config('posts', 'content', 'match'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.modify_search_config -CREATE FUNCTION eql_v2.remove_search_config(table_name text, column_name text, index_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the index does not exist - -- IF NOT _config->key ? index_name THEN - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No % index exists for column: % %', index_name, table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the index - SELECT _config #- array['tables', table_name, column_name, 'indexes', index_name] INTO _config; - - -- update the config and migrate (even if empty) - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Modify a search index configuration for an encrypted column ---! ---! Updates an existing search index configuration by removing and re-adding it ---! with new options. Convenience function that combines remove and add operations. ---! If index does not exist, it is added. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to modify ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB New index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! ---! @example ---! -- Change match index tokenizer settings ---! SELECT eql_v2.modify_search_config('posts', 'content', 'match', 'text', ---! '{"tokenizer": {"kind": "ngram", "token_length": 4}}' ---! ); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.modify_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - PERFORM eql_v2.remove_search_config(table_name, column_name, index_name, migrating); - RETURN eql_v2.add_search_config(table_name, column_name, index_name, cast_as, opts, migrating); - END; -$$ LANGUAGE plpgsql; - ---! @brief Migrate pending configuration to encrypting state ---! ---! Transitions the pending configuration to encrypting state, validating that ---! all configured columns have encrypted target columns ready. This is part of ---! the configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if migration succeeds ---! @throws Exception if encryption already in progress ---! @throws Exception if no pending configuration exists ---! @throws Exception if configured columns lack encrypted targets ---! ---! @example ---! -- Manually migrate configuration (normally done automatically) ---! SELECT eql_v2.migrate_config(); ---! ---! @see eql_v2.activate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.migrate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - RAISE EXCEPTION 'An encryption is already in progress'; - END IF; - - IF NOT EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - IF NOT eql_v2.ready_for_encryption() THEN - RAISE EXCEPTION 'Some pending columns do not have an encrypted target'; - END IF; - - UPDATE public.eql_v2_configuration SET state = 'encrypting' WHERE state = 'pending'; - RETURN true; - END; -$$ LANGUAGE plpgsql; - ---! @brief Activate encrypting configuration ---! ---! Transitions the encrypting configuration to active state, making it the ---! current operational configuration. Marks previous active configuration as ---! inactive. Final step in configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if activation succeeds ---! @throws Exception if no encrypting configuration exists to activate ---! ---! @example ---! -- Manually activate configuration (normally done automatically) ---! SELECT eql_v2.activate_config(); ---! ---! @see eql_v2.migrate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.activate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'encrypting'; - RETURN true; - ELSE - RAISE EXCEPTION 'No encrypting configuration exists to activate'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Discard pending configuration ---! ---! Deletes the pending configuration without applying changes. Use this to ---! abandon configuration changes before they are migrated and activated. ---! ---! @return Boolean True if discard succeeds ---! @throws Exception if no pending configuration exists to discard ---! ---! @example ---! -- Discard uncommitted configuration changes ---! SELECT eql_v2.discard(); ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.discard() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - DELETE FROM public.eql_v2_configuration WHERE state = 'pending'; - RETURN true; - ELSE - RAISE EXCEPTION 'No pending configuration exists to discard'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Configure a column for encryption ---! ---! Adds a column to the encryption configuration, making it eligible for ---! encrypted storage and search indexes. Creates or updates pending configuration, ---! adds encrypted constraint, then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to encrypt ---! @param cast_as Text PostgreSQL type to cast decrypted values (default: 'text') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if column already configured for encryption ---! ---! @example ---! -- Configure email column for encryption ---! SELECT eql_v2.add_column('users', 'email', 'text'); ---! ---! -- Configure age column with integer casting ---! SELECT eql_v2.add_column('users', 'age', 'int'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_column -CREATE FUNCTION eql_v2.add_column(table_name text, column_name text, cast_as text DEFAULT 'text', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - -- if index exists - IF _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'Config exists for column: % %', table_name, column_name; - END IF; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a column from encryption configuration ---! ---! Removes a column from the encryption configuration, including all associated ---! search indexes. Removes encrypted constraint, updates pending configuration, ---! then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove email column from encryption ---! SELECT eql_v2.remove_column('users', 'email'); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.remove_column(table_name text, column_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the column does not exist - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No configuration exists for column: % %', table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the column - SELECT _config #- array['tables', table_name, column_name] INTO _config; - - -- if table is now empty, remove the table - IF _config #> array['tables', table_name] = '{}' THEN - SELECT _config #- array['tables', table_name] INTO _config; - END IF; - - PERFORM eql_v2.remove_encrypted_constraint(table_name, column_name); - - -- update the config (even if empty) and activate - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - -- For empty configs, skip migration validation and directly activate - IF _config #> array['tables'] = '{}' THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'pending'; - ELSE - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - END IF; - - -- exeunt - RETURN _config; - - END; -$$ LANGUAGE plpgsql; - ---! @brief Reload configuration from CipherStash Proxy ---! ---! Placeholder function for reloading configuration from the CipherStash Proxy. ---! Currently returns NULL without side effects. ---! ---! @return Void ---! ---! @note This function may be used for configuration synchronization in future versions -CREATE FUNCTION eql_v2.reload_config() - RETURNS void -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN NULL; -END; - ---! @brief Query encryption configuration in tabular format ---! ---! Returns the active encryption configuration as a table for easier querying ---! and filtering. Shows all configured tables, columns, cast types, and indexes. ---! ---! @return TABLE Contains configuration state, relation name, column name, cast type, and indexes ---! ---! @example ---! -- View all encrypted columns ---! SELECT * FROM eql_v2.config(); ---! ---! -- Find all columns with match indexes ---! SELECT relation, col_name FROM eql_v2.config() ---! WHERE indexes ? 'match'; ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.config() RETURNS TABLE ( - state eql_v2_configuration_state, - relation text, - col_name text, - decrypts_as text, - indexes jsonb -) - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - RETURN QUERY - WITH tables AS ( - SELECT cfg.state, tables.key AS table, tables.value AS tbl_config - FROM public.eql_v2_configuration cfg, jsonb_each(data->'tables') tables - WHERE cfg.data->>'v' = '1' - ) - SELECT - tables.state, - tables.table, - column_config.key, - COALESCE(column_config.value->>'plaintext_type', column_config.value->>'cast_as'), - column_config.value->'indexes' - FROM tables, jsonb_each(tables.tbl_config) column_config; -END; -$$ LANGUAGE plpgsql; - ---! @file config/constraints.sql ---! @brief Configuration validation functions and constraints ---! ---! Provides CHECK constraint functions to validate encryption configuration structure. ---! Ensures configurations have required fields (version, tables) and valid values ---! for index types and cast types before being stored. ---! ---! @see config/tables.sql where constraints are applied - - ---! @brief Extract index type names from configuration ---! @internal ---! ---! Helper function that extracts all index type names from the configuration's ---! 'indexes' sections across all tables and columns. ---! ---! @param jsonb Configuration data to extract from ---! @return SETOF text Index type names (e.g., 'match', 'ore', 'unique', 'ste_vec') ---! ---! @note Used by config_check_indexes for validation ---! @see eql_v2.config_check_indexes -CREATE FUNCTION eql_v2.config_get_indexes(val jsonb) - RETURNS SETOF text - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_object_keys(jsonb_path_query(val,'$.tables.*.*.indexes')); -END; - - ---! @brief Validate index types in configuration ---! @internal ---! ---! Checks that all index types specified in the configuration are valid. ---! Valid index types are: match, ore, ope, unique, ste_vec. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all index types are valid ---! @throws Exception if any invalid index type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @see eql_v2.config_get_indexes -CREATE FUNCTION eql_v2.config_check_indexes(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF (SELECT EXISTS (SELECT eql_v2.config_get_indexes(val))) THEN - IF (SELECT bool_and(index = ANY('{match, ore, ope, unique, ste_vec}')) FROM eql_v2.config_get_indexes(val) AS index) THEN - RETURN true; - END IF; - RAISE 'Configuration has an invalid index (%). Index should be one of {match, ore, ope, unique, ste_vec}', val; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate cast types in configuration ---! @internal ---! ---! Checks that all 'cast_as' and 'plaintext_type' types specified in the configuration are valid. ---! Valid cast types are: text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all cast types are valid or no cast types specified ---! @throws Exception if any invalid cast type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Empty configurations (no cast_as/plaintext_type fields) are valid ---! @note Cast type names are EQL's internal representations, not PostgreSQL native types ---! @note 'plaintext_type' is accepted as a canonical alias for 'cast_as' -CREATE FUNCTION eql_v2.config_check_cast(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_types text[] := '{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}'; - BEGIN - -- Validate cast_as fields - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as'))) THEN - IF NOT (SELECT bool_and(cast_as = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as')) AS cast_as) casts) THEN - RAISE 'Configuration has an invalid cast_as (%). Cast should be one of %', val, _valid_types; - END IF; - END IF; - - -- Validate plaintext_type fields (canonical alias for cast_as) - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type'))) THEN - IF NOT (SELECT bool_and(pt = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type')) AS pt) types) THEN - RAISE 'Configuration has an invalid plaintext_type (%). Type should be one of %', val, _valid_types; - END IF; - END IF; - - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate tables field presence ---! @internal ---! ---! Ensures the configuration has a 'tables' field, which is required ---! to specify which database tables contain encrypted columns. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'tables' field exists ---! @throws Exception if 'tables' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_tables(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'tables') THEN - RETURN true; - END IF; - RAISE 'Configuration missing tables (tables) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate version field presence ---! @internal ---! ---! Ensures the configuration has a 'v' (version) field, which tracks ---! the configuration format version. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'v' field exists ---! @throws Exception if 'v' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_version(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - RETURN true; - END IF; - RAISE 'Configuration missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ste_vec index mode option ---! @internal ---! ---! Checks that the optional `mode` field on `ste_vec` index configurations is ---! one of the recognised values. Valid modes are: standard, compat. ---! Configurations without a `mode` field (the default) pass unconditionally. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if every ste_vec mode is valid, or none are set ---! @throws Exception if any ste_vec.mode value is not in the allowed set ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Mode is optional — only configurations that set it are validated -CREATE FUNCTION eql_v2.config_check_ste_vec_mode(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_modes text[] := '{standard, compat}'; - BEGIN - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode'))) THEN - IF NOT (SELECT bool_and(mode = ANY(_valid_modes)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode')) AS mode) modes) THEN - RAISE 'Configuration has an invalid ste_vec mode (%). Mode should be one of %', val, _valid_modes; - END IF; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Drop existing data validation constraint if present ---! @note Allows constraint to be recreated during upgrades -ALTER TABLE public.eql_v2_configuration DROP CONSTRAINT IF EXISTS eql_v2_configuration_data_check; - - ---! @brief Comprehensive configuration data validation ---! ---! CHECK constraint that validates all aspects of configuration data: ---! - Version field presence ---! - Tables field presence ---! - Valid cast_as types ---! - Valid index types ---! - Valid ste_vec mode (when set) ---! ---! @note Combines all config_check_* validation functions ---! @see eql_v2.config_check_version ---! @see eql_v2.config_check_tables ---! @see eql_v2.config_check_cast ---! @see eql_v2.config_check_indexes ---! @see eql_v2.config_check_ste_vec_mode -ALTER TABLE public.eql_v2_configuration - ADD CONSTRAINT eql_v2_configuration_data_check CHECK ( - eql_v2.config_check_version(data) AND - eql_v2.config_check_tables(data) AND - eql_v2.config_check_cast(data) AND - eql_v2.config_check_indexes(data) AND - eql_v2.config_check_ste_vec_mode(data) -); - - ---! @file pin_search_path.sql ---! @brief Post-install: pin search_path on every eql_v2.* function ---! ---! This file is appended verbatim by `tasks/build.sh` to the end of every ---! release variant (main, supabase, protect/stack), AFTER all `src/**/*.sql` ---! files have been concatenated. It lives outside `src/` so it stays out of ---! the dependency graph entirely — each variant has a different leaf set ---! (supabase excludes `**/*operator_class.sql`; protect excludes `src/config/*` ---! and `src/encryptindex/*`), and threading REQUIREs to be ordered last in ---! every variant simultaneously is fragile. ---! ---! Iterates over functions in the `eql_v2` schema and applies a fixed ---! `search_path` via `ALTER FUNCTION ... SET search_path = ...`. This is the ---! only way to satisfy Supabase splinter's `function_search_path_mutable` ---! lint, which checks `pg_proc.proconfig` directly. ---! ---! @note A SET clause disables PostgreSQL's SQL-function inlining (see ---! inline_function() in src/backend/optimizer/util/clauses.c). For most ---! eql_v2 helpers this is irrelevant. The exceptions are wrappers that ---! must inline to expose `eql_v2.jsonb_array(col) @> ...` to the planner ---! so the GIN index on `jsonb_array(e)` can be matched. Those are ---! deliberately skipped here and allowlisted in `tasks/test/splinter.sh`. ---! ---! @see tasks/test/splinter.sh ---! @see tasks/build.sh - -DO $$ -DECLARE - fn_oid oid; - inline_critical_oids oid[]; - enc_oid oid; - jsonb_oid oid; - text_oid oid; - entry_oid oid; -BEGIN - -- Resolve type oids without depending on caller search_path. The encrypted - -- composite type is created in `public`; jsonb / text are in `pg_catalog`; - -- the ste_vec_entry DOMAIN lives in `eql_v2`. - SELECT t.oid INTO enc_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'public' AND t.typname = 'eql_v2_encrypted'; - - IF enc_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type public.eql_v2_encrypted not found — ' - 'this script must run after all EQL src/**/*.sql files have been loaded'; - END IF; - - SELECT t.oid INTO jsonb_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'jsonb'; - - IF jsonb_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.jsonb not found'; - END IF; - - SELECT t.oid INTO text_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'text'; - - IF text_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.text not found'; - END IF; - - SELECT t.oid INTO entry_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'ste_vec_entry'; - - IF entry_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type eql_v2.ste_vec_entry not found'; - END IF; - - -- Wrappers that must remain inlinable for functional-index matching. - -- Verified empirically: with SET, EXPLAIN drops to Seq Scan; without, - -- it uses Bitmap Index Scan / Index Scan. - -- - -- Phase 1 operator inlining (#193): `=`, `<>`, `~~`, `~~*`, `@>`, `<@` - -- on `eql_v2_encrypted` and the cross-type (encrypted, jsonb) / - -- (jsonb, encrypted) overloads emitted by ORMs that bind parameters - -- as jsonb (Drizzle, PostgREST, encryptedSupabase). The implementation - -- functions reduce to `extractor(a) op extractor(b)` and must inline - -- to match the documented functional indexes - -- (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, - -- `eql_v2.ste_vec(col)`). - -- - -- For `~~` / `~~*` the planner must inline two layers — the operator - -- function `eql_v2."~~"` and the helper `eql_v2.like` / `eql_v2.ilike` - -- — to reach the canonical `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)` - -- form that the documented functional index matches. The helpers are - -- allowlisted alongside the operator wrappers below; pinning either - -- layer breaks the chain and reverts to Seq Scan. - -- - -- Note: pg_proc.proargtypes is an oidvector with 0-based bounds, so we - -- compare elements individually rather than using array equality (which - -- requires matching bounds, not just contents). - SELECT pg_catalog.array_agg(p.oid) INTO inline_critical_oids - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - AND ( - -- Same-type (encrypted, encrypted) operators that must inline. - -- `like`/`ilike` are the SQL helpers that `~~`/`~~*` delegate to; - -- both layers must inline to reach `bloom_filter(a) @> bloom_filter(b)`. - -- `<`, `<=`, `>`, `>=` inline to `ore_block_u64_8_256(a) op - -- ore_block_u64_8_256(b)`; they must reach the functional ORE index - -- expression `eql_v2.ore_block_u64_8_256(col)` for bare range - -- queries to engage Index Scan. - (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', '@>', '<@', - 'jsonb_contains', 'jsonb_contained_by', - 'like', 'ilike') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = enc_oid) - -- Cross-type (encrypted, jsonb). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = jsonb_oid) - -- Cross-type (jsonb, encrypted). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = jsonb_oid AND p.proargtypes[1] = enc_oid) - -- Root-level HMAC extractor (#205): all 1-arg overloads are now - -- inlinable SQL. Must stay unpinned so the planner can fold extractor - -- calls inside the inlined equality operator bodies into the calling - -- query, preserving the functional-index match. - OR (p.pronargs = 1 - AND p.proname = 'hmac_256' - AND (p.proargtypes[0] = enc_oid OR p.proargtypes[0] = jsonb_oid)) - -- Field-level JSONB extractors (#205): inlinable SQL replacements for - -- the previous plpgsql bodies. Inlining lets the planner fold the - -- `jsonb_array_elements(...) WHERE elem->>'s' = selector` body into - -- the calling query, eliminating per-row function call overhead on - -- large ste_vec scans. - OR (p.pronargs = 2 - AND p.proname IN ('jsonb_path_query', - 'jsonb_path_query_first', - 'jsonb_path_exists')) - -- Inner ORE-block comparison helpers backing the `<`, `<=`, `>`, `>=` - -- operators on `eql_v2.ore_block_u64_8_256`. The outer operators on - -- `eql_v2_encrypted` inline to `ore_block(a) <op> ore_block(b)`, and - -- PG only carries the inlined form through to index matching if the - -- inner operator function is also inlinable (no SET, IMMUTABLE). - -- Pinning these would prevent the planner from structurally matching - -- predicates against a functional `eql_v2.ore_block_u64_8_256(col)` - -- index. The inner functions are deterministic comparisons of - -- composite type bytes, declared IMMUTABLE STRICT PARALLEL SAFE. - OR (p.pronargs = 2 - AND p.proname IN ('ore_block_u64_8_256_eq', 'ore_block_u64_8_256_neq', - 'ore_block_u64_8_256_lt', 'ore_block_u64_8_256_lte', - 'ore_block_u64_8_256_gt', 'ore_block_u64_8_256_gte')) - -- Hash operator class FUNCTION 1: called once per row by HashAggregate, - -- hash joins, DISTINCT. Inlinable SQL avoids the per-row plpgsql - -- interpreter overhead — without this, `GROUP BY value` on - -- `eql_v2_encrypted` at 1M rows degrades super-linearly because the - -- plpgsql cost compounds with HashAggregate work_mem spillage. - OR (p.pronargs = 1 - AND p.proname = 'hash_encrypted' - AND p.proargtypes[0] = enc_oid) - -- Consolidated ORE-CLLW extractor (U-006). Inlinable SQL — pinning - -- would silently undo it and prevent the planner from folding - -- `eql_v2.ore_cllw(col)` calls into the calling query. The - -- `compare_ore_cllw_term` comparator stays plpgsql by design (per-byte - -- protocol can't be expressed as a single inlinable SELECT), so it is - -- NOT on this list. The (jsonb) form is a RHS-parameter helper for - -- comparisons against literal jsonb; the (eql_v2.ste_vec_entry) form - -- is the typed extractor for the result of `col -> '<selector>'`. - OR (p.pronargs = 1 - AND p.proname IN ('ore_cllw', 'has_ore_cllw') - AND (p.proargtypes[0] = jsonb_oid OR p.proargtypes[0] = entry_oid)) - -- Typed HMAC extractor on a ste_vec entry (#219 strict separation). - -- Same rationale as `ore_cllw(ste_vec_entry)` — must inline so - -- `eql_v2.hmac_256(col -> 'sel')` folds into the calling query and - -- matches a functional hash index built on the same expression. - OR (p.pronargs = 1 - AND p.proname IN ('hmac_256', 'has_hmac_256', 'selector') - AND p.proargtypes[0] = entry_oid) - -- `eql_v2.ste_vec_entry × eql_v2.ste_vec_entry` operators (#219). - -- Inline to `hmac_256(a) = hmac_256(b)` (equality) or - -- `ore_cllw(a) <op> ore_cllw(b)` (ordering); both chains must remain - -- unpinned for functional-index match through extractor form. - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - 'eq', 'neq', 'lt', 'lte', 'gt', 'gte') - AND p.proargtypes[0] = entry_oid AND p.proargtypes[1] = entry_oid) - -- Inner ORE-CLLW comparison helpers backing the `<`, `<=`, `=`, - -- `>=`, `>`, `<>` operators on `eql_v2.ore_cllw` (the composite - -- type, registered via `eql_v2.ore_cllw_ops` opclass — #221). Same - -- precedent as the `ore_block_u64_8_256_*` helpers above: PG only - -- carries the inlined operator wrapper through to functional-index - -- match if the inner backing function is also inlinable. Pinning - -- these would break the index match for `ORDER BY eql_v2.ore_cllw - -- (value -> '<selector>'::text)` and the matching `WHERE` form. - OR (p.pronargs = 2 - AND p.proname IN ('ore_cllw_eq', 'ore_cllw_neq', - 'ore_cllw_lt', 'ore_cllw_lte', - 'ore_cllw_gt', 'ore_cllw_gte')) - -- `->` selector lookup: inlinable SQL post the type flip - -- (returns `eql_v2.ste_vec_entry`). Must stay unpinned so the - -- planner can fold `col -> '<selector>'` into the calling query - -- — without this, the chained recipe - -- `WHERE col -> 'sel' = $1::ste_vec_entry` would not match a - -- functional hash index on `eql_v2.eq_term(col -> 'sel')`. - OR (p.proname = '->' - AND p.pronargs = 2 - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = text_oid - OR p.proargtypes[1] = enc_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'int4'))) - -- XOR-aware equality term extractor on a ste_vec entry. Must - -- inline so `eql_v2.eq_term(col -> 'sel')` folds into the - -- calling query and matches a functional hash index built on - -- the same expression. - OR (p.pronargs = 1 - AND p.proname = 'eq_term' - AND p.proargtypes[0] = entry_oid) - -- Type-safe `@>` / `<@` overloads with typed needles - -- (`stevec_query`, `ste_vec_entry`). Inline to the existing - -- `ste_vec_contains` machinery — must stay unpinned to engage - -- the GIN index on `eql_v2.ste_vec(col)` structurally for - -- bare-form containment. - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = entry_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[1] = enc_oid - AND (p.proargtypes[0] = entry_oid - OR p.proargtypes[0] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - ); - - FOR fn_oid IN - SELECT p.oid - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - -- Only normal functions ('f') and window functions ('w') accept - -- ALTER FUNCTION ... SET. Aggregates ('a') would be rejected by - -- ALTER ROUTINE/FUNCTION, and procedures ('p') would need ALTER - -- PROCEDURE. The 3 affected aggregates (min, max, grouped_value) - -- are allowlisted in splinter. - AND p.prokind IN ('f', 'w') - AND NOT EXISTS ( - SELECT 1 FROM pg_catalog.unnest(coalesce(p.proconfig, '{}'::text[])) c - WHERE c LIKE 'search_path=%' - ) - AND NOT (p.oid = ANY (coalesce(inline_critical_oids, '{}'::oid[]))) - LOOP - -- oid::regprocedure renders as `schema.name(argtype, argtype)` and is a - -- valid target for ALTER FUNCTION regardless of caller search_path. - EXECUTE pg_catalog.format( - 'ALTER FUNCTION %s SET search_path = pg_catalog, extensions, public', - fn_oid::regprocedure - ); - END LOOP; -END $$; diff --git a/packages/cli/src/sql/cipherstash-encrypt.sql b/packages/cli/src/sql/cipherstash-encrypt.sql deleted file mode 100644 index 3cd4f3984..000000000 --- a/packages/cli/src/sql/cipherstash-encrypt.sql +++ /dev/null @@ -1,7644 +0,0 @@ ---! @file schema.sql ---! @brief EQL v2 schema creation ---! ---! Creates the eql_v2 schema which contains all Encrypt Query Language ---! functions, types, and tables. Drops existing schema if present to ---! support clean reinstallation. ---! ---! @warning DROP SCHEMA CASCADE will remove all objects in the schema ---! @note All EQL objects (functions, types, tables) reside in eql_v2 schema - ---! @brief Drop existing EQL v2 schema ---! @warning CASCADE will drop all dependent objects -DROP SCHEMA IF EXISTS eql_v2 CASCADE; - ---! @brief Create EQL v2 schema ---! @note All EQL functions and types will be created in this schema -CREATE SCHEMA eql_v2; - ---! @brief Composite type for encrypted column data ---! ---! Core type used for all encrypted columns in EQL. Stores encrypted data as JSONB ---! with the following structure: ---! - `c`: ciphertext (base64-encoded encrypted value) ---! - `i`: index terms (searchable metadata for encrypted searches) ---! - `k`: key ID (identifier for encryption key) ---! - `m`: metadata (additional encryption metadata) ---! ---! Created in public schema to persist independently of eql_v2 schema lifecycle. ---! Customer data columns use this type, so it must not be dropped if data exists. ---! ---! @note DO NOT DROP this type unless absolutely certain no encrypted data uses it ---! @see eql_v2.ciphertext ---! @see eql_v2.meta_data ---! @see eql_v2.add_column -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_encrypted') THEN - CREATE TYPE public.eql_v2_encrypted AS ( - data jsonb - ); - END IF; - END -$$; - - - - - - - - - - ---! @brief Bloom filter index term type ---! ---! Domain type representing Bloom filter bit arrays stored as smallint arrays. ---! Used for pattern-match encrypted searches via the 'match' index type. ---! The filter is stored in the 'bf' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2."~~" ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.bloom_filter AS smallint[]; - - - ---! @brief ORE block term type for Order-Revealing Encryption ---! ---! Composite type representing a single ORE (Order-Revealing Encryption) block term. ---! Stores encrypted data as bytea that enables range comparisons without decryption. ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE TYPE eql_v2.ore_block_u64_8_256_term AS ( - bytes bytea -); - - ---! @brief ORE block index term type for range queries ---! ---! Composite type containing an array of ORE block terms. Used for encrypted ---! range queries via the 'ore' index type. The array is stored in the 'ob' field ---! of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_block_u64_8_256_terms ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_block_u64_8_256 AS ( - terms eql_v2.ore_block_u64_8_256_term[] -); - ---! @brief HMAC-SHA256 index term type ---! ---! Domain type representing HMAC-SHA256 hash values. ---! Used for exact-match encrypted searches via the 'unique' index type. ---! The hash is stored in the 'hm' field of encrypted data payloads. ---! ---! @see eql_v2.add_search_config ---! @note This is a transient type used only during query execution -CREATE DOMAIN eql_v2.hmac_256 AS text; - ---! @file src/ste_vec/types.sql ---! @brief Domain type for individual STE-vec entries ---! ---! Defines `eql_v2.ste_vec_entry` as a DOMAIN over `jsonb` constrained to the ---! shape of a single element inside an `sv` array — a JSON object that ---! carries at minimum a selector field (`s`). This is the type returned by ---! the `->` operator on `eql_v2_encrypted` (a single sv element extracted by ---! selector) and the type accepted by sv-element extractors such as ---! `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and ---! `eql_v2.hmac_256(eql_v2.ste_vec_entry)`. ---! ---! Why a separate type. Before #219, the `(eql_v2_encrypted)` overloads of ---! sv-element extractors read fields like `oc` off the root `data` jsonb, ---! which is misleading: a root `EncryptedPayload` or `SteVecPayload` (the ---! shapes that an actual `eql_v2_encrypted` column value carries) never has ---! `oc` at the root. The previous pattern only worked because the `->` ---! operator merged ste-vec entry fields into a fake root-shaped payload ---! before the extractor ran. This domain type makes the distinction ---! explicit: `eql_v2_encrypted` is the root shape; `eql_v2.ste_vec_entry` ---! is the per-entry shape; extractors are typed accordingly. ---! ---! @note The CHECK constraint reflects the cipherstash-suite emission ---! contract: ---! - `s` (selector — column-name HMAC) and `c` (ciphertext) are ---! emitted on every sv element. ---! - Each sv element carries **exactly one** of `hm` (HMAC-256, for ---! hash-equality queries) or `oc` (CLLW ORE, for ordered queries) ---! — they are mutually exclusive. A given selector / field is ---! configured for one mode or the other; the crypto layer emits ---! the corresponding term and only that term. ---! Other fields (`a` for array marker, etc.) are allowed but not ---! required. ---! ---! @see src/operators/->.sql ---! @see src/ore_cllw/functions.sql ---! @see src/hmac_256/functions.sql -CREATE DOMAIN eql_v2.ste_vec_entry AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 's' - AND VALUE ? 'c' - AND (VALUE ? 'hm') <> (VALUE ? 'oc') - ); - - ---! @brief Domain type for an STE-vec containment needle ---! ---! `eql_v2.stevec_query` is a query-shaped sv payload: a top-level ---! `{"sv": [...]}` object whose elements carry selector + index ---! terms but **never** a ciphertext (`c`) field. Containment (`@>`) ---! against an `eql_v2_encrypted` column is structurally typed ---! through this domain so the call site reads as "match against an ---! sv query", not "compare two encrypted values". ---! ---! Compared to `eql_v2.ste_vec_entry` (single sv element with `s`, ---! `c`, and `hm` XOR `oc`), `stevec_query` is the wrapping ---! `{"sv": [...]}` payload: it forbids `c` on every element but ---! otherwise keeps the same per-element contract — each element must ---! carry a selector `s` and exactly one deterministic term (`hm` XOR ---! `oc`). This mirrors the `SteVecQueryElement` JSON schema and stops ---! selector-only needles (e.g. `{"sv":[{"s":"x"}]}`) from casting and ---! then matching every row through the bare `jsonb @>` implementation. ---! The implementation of `ste_vec_contains` ignores `c` either way, ---! but typing the needle as `stevec_query` documents the contract at ---! the API surface. ---! ---! @note Constructing a `stevec_query` literal from inline JSON works ---! via the standard DOMAIN cast: ---! `'{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query` ---! Casting an `eql_v2_encrypted` value strips `c` fields from ---! each sv element — see `eql_v2.to_stevec_query`. ---! ---! @see eql_v2.to_stevec_query ---! @see src/operators/@>.sql -CREATE DOMAIN eql_v2.stevec_query AS jsonb - CHECK ( - jsonb_typeof(VALUE) = 'object' - AND VALUE ? 'sv' - AND jsonb_typeof(VALUE -> 'sv') = 'array' - -- No element may carry a ciphertext (`c`) — this is a query, not a value. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.c))'::jsonpath) - -- Every element must carry a selector (`s`) ... - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.s))'::jsonpath) - -- ... and exactly one deterministic term — `hm` XOR `oc` — matching - -- the `ste_vec_entry` emission contract and the `SteVecQueryElement` - -- JSON schema. Rejects selector-only needles that would otherwise - -- cast and then match every row via the bare `jsonb @>` body. - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (exists(@.hm) && exists(@.oc))'::jsonpath) - AND NOT jsonb_path_exists(VALUE, '$.sv[*] ? (!exists(@.hm) && !exists(@.oc))'::jsonpath) - ); - - ---! @brief Convert an `eql_v2_encrypted` to a `stevec_query` needle ---! ---! Normalises each sv element down to the matching-relevant fields: ---! `s` (selector) plus exactly one of `hm` / `oc`. Other fields ---! (`c` ciphertext, `a` array marker, `i`/`v` envelope metadata, anything ---! else cipherstash-client might emit) are stripped. This is the ---! canonical needle shape for `@>` containment — matching the contract ---! that containment compares by selector + deterministic term and ---! ignores everything else. ---! ---! Designed for use as a functional GIN index expression: a single ---! `GIN (eql_v2.to_stevec_query(col)::jsonb jsonb_path_ops)` index ---! covers containment queries against any selector (both hm-bearing ---! and oc-bearing — XOR-aware), and the typed `@>` overloads inline ---! to a native `jsonb @>` on the same expression so the planner ---! engages Bitmap Index Scan structurally. ---! ---! @param e eql_v2_encrypted Source encrypted payload ---! @return eql_v2.stevec_query Query-shaped needle, sv elements ---! normalised to `{s, hm}` or `{s, oc}`. ---! ---! @example ---! -- Functional GIN index — canonical containment recipe ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! ---! -- Cross-row containment ---! SELECT a.* ---! FROM docs a, docs b ---! WHERE a.encrypted_doc @> b.encrypted_doc::eql_v2.stevec_query ---! AND b.id = 42; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2.to_stevec_query(e eql_v2_encrypted) - RETURNS eql_v2.stevec_query - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT jsonb_build_object( - 'sv', - coalesce( - (SELECT jsonb_agg( - jsonb_strip_nulls( - jsonb_build_object( - 's', elem -> 's', - 'hm', elem -> 'hm', - 'oc', elem -> 'oc' - ) - ) - ) - FROM jsonb_array_elements((e).data -> 'sv') AS elem), - '[]'::jsonb - ) - )::eql_v2.stevec_query -$$; - -CREATE CAST (eql_v2_encrypted AS eql_v2.stevec_query) - WITH FUNCTION eql_v2.to_stevec_query - AS ASSIGNMENT; - ---! @file crypto.sql ---! @brief PostgreSQL pgcrypto extension enablement ---! ---! Enables the pgcrypto extension which provides cryptographic functions ---! used by EQL for hashing and other cryptographic operations. ---! ---! Installs pgcrypto into the `extensions` schema (Supabase convention) to ---! avoid the `extension_in_public` lint. Every EQL function that uses ---! pgcrypto has `pg_catalog, extensions, public` on its `search_path`, so a ---! pre-existing install in `public` keeps working — and a pre-existing ---! install anywhere else will be rejected at install time rather than ---! failing later inside an encrypted comparison. ---! ---! @note pgcrypto provides functions like digest(), hmac(), gen_random_bytes() ---! @note If pgcrypto is already installed in `public`, EQL works but emits ---! a NOTICE recommending `ALTER EXTENSION pgcrypto SET SCHEMA extensions`. ---! @note If pgcrypto is already installed in any other schema, install ---! fails. Relocate it first with `ALTER EXTENSION pgcrypto SET SCHEMA ---! extensions` (or move it into `public` if compatibility with other ---! consumers requires it). - ---! @brief Create extensions schema (Supabase convention) -CREATE SCHEMA IF NOT EXISTS extensions; - ---! @brief Enable pgcrypto extension and validate its schema -DO $$ -DECLARE - pgcrypto_schema name; -BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') THEN - CREATE EXTENSION pgcrypto WITH SCHEMA extensions; - END IF; - - SELECT n.nspname INTO pgcrypto_schema - FROM pg_extension e - JOIN pg_namespace n ON n.oid = e.extnamespace - WHERE e.extname = 'pgcrypto'; - - IF pgcrypto_schema = 'extensions' THEN - -- expected location, nothing to say - NULL; - ELSIF pgcrypto_schema = 'public' THEN - RAISE NOTICE - 'pgcrypto is installed in the `public` schema. EQL works against this layout, ' - 'but Supabase splinter will flag it as `extension_in_public`. Move it with: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions'; - ELSE - RAISE EXCEPTION - 'pgcrypto is installed in schema `%`, which is not on the EQL function search_path ' - '(pg_catalog, extensions, public). EQL cryptographic operations would fail at ' - 'runtime. Relocate the extension before installing EQL: ' - 'ALTER EXTENSION pgcrypto SET SCHEMA extensions', - pgcrypto_schema; - END IF; -END $$; - ---! @brief Extract ciphertext from encrypted JSONB value ---! ---! Extracts the ciphertext (c field) from a raw JSONB encrypted value. ---! The ciphertext is the base64-encoded encrypted data. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if 'c' field is not present in JSONB ---! ---! @example ---! -- Extract ciphertext from JSONB literal ---! SELECT eql_v2.ciphertext('{"c":"AQIDBA==","i":{"unique":"..."}}'::jsonb); ---! ---! @see eql_v2.ciphertext(eql_v2_encrypted) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'c' THEN - RETURN val->>'c'; - END IF; - RAISE 'Expected a ciphertext (c) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract ciphertext from encrypted column value ---! ---! Extracts the ciphertext from an encrypted column value. Convenience ---! overload that unwraps eql_v2_encrypted type and delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Text Base64-encoded ciphertext string ---! @throws Exception if encrypted value is malformed ---! ---! @example ---! -- Extract ciphertext from encrypted column ---! SELECT eql_v2.ciphertext(encrypted_email) FROM users; ---! ---! @see eql_v2.ciphertext(jsonb) ---! @see eql_v2.meta_data -CREATE FUNCTION eql_v2.ciphertext(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.ciphertext(val.data); -$$; - ---! @brief State transition function for grouped_value aggregate ---! @internal ---! ---! Returns the first non-null value encountered. Used as state function ---! for the grouped_value aggregate to select first value in each group. ---! ---! @param $1 JSONB Accumulated state (first non-null value found) ---! @param $2 JSONB New value from current row ---! @return JSONB First non-null value (state or new value) ---! ---! @see eql_v2.grouped_value -CREATE FUNCTION eql_v2._first_grouped_value(jsonb, jsonb) -RETURNS jsonb -AS $$ - SELECT COALESCE($1, $2); -$$ LANGUAGE sql IMMUTABLE; - ---! @brief Return first non-null encrypted value in a group ---! ---! Aggregate function that returns the first non-null encrypted value ---! encountered within a GROUP BY clause. Useful for deduplication or ---! selecting representative values from grouped encrypted data. ---! ---! @param input JSONB Encrypted values to aggregate ---! @return JSONB First non-null encrypted value in group ---! ---! @example ---! -- Get first email per user group ---! SELECT user_id, eql_v2.grouped_value(encrypted_email) ---! FROM user_emails ---! GROUP BY user_id; ---! ---! -- Deduplicate encrypted values ---! SELECT DISTINCT ON (user_id) ---! user_id, ---! eql_v2.grouped_value(encrypted_ssn) as primary_ssn ---! FROM user_records ---! GROUP BY user_id; ---! ---! @see eql_v2._first_grouped_value -CREATE AGGREGATE eql_v2.grouped_value(jsonb) ( - SFUNC = eql_v2._first_grouped_value, - STYPE = jsonb -); - ---! @brief Add validation constraint to encrypted column ---! ---! Adds a CHECK constraint to ensure column values conform to encrypted data ---! structure. Constraint uses eql_v2.check_encrypted to validate format. ---! Called automatically by eql_v2.add_column. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to constrain ---! @return Void ---! ---! @example ---! -- Manually add constraint (normally done by add_column) ---! SELECT eql_v2.add_encrypted_constraint('users', 'encrypted_email'); ---! ---! -- Resulting constraint: ---! -- ALTER TABLE users ADD CONSTRAINT eql_v2_encrypted_check_encrypted_email ---! -- CHECK (eql_v2.check_encrypted(encrypted_email)); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_encrypted_constraint -CREATE FUNCTION eql_v2.add_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I ADD CONSTRAINT eql_v2_encrypted_constraint_%I_%I CHECK (eql_v2.check_encrypted(%I))', table_name, table_name, column_name, column_name); - EXCEPTION - WHEN duplicate_table THEN - WHEN duplicate_object THEN - RAISE NOTICE 'Constraint `eql_v2_encrypted_constraint_%_%` already exists, skipping', table_name, column_name; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove validation constraint from encrypted column ---! ---! Removes the CHECK constraint that validates encrypted data structure. ---! Called automatically by eql_v2.remove_column. Uses IF EXISTS to avoid ---! errors if constraint doesn't exist. ---! ---! @param table_name TEXT Name of table containing the column ---! @param column_name TEXT Name of column to unconstrain ---! @return Void ---! ---! @example ---! -- Manually remove constraint (normally done by remove_column) ---! SELECT eql_v2.remove_encrypted_constraint('users', 'encrypted_email'); ---! ---! @see eql_v2.remove_column ---! @see eql_v2.add_encrypted_constraint -CREATE FUNCTION eql_v2.remove_encrypted_constraint(table_name TEXT, column_name TEXT) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - EXECUTE format('ALTER TABLE %I DROP CONSTRAINT IF EXISTS eql_v2_encrypted_constraint_%I_%I', table_name, table_name, column_name); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract metadata from encrypted JSONB value ---! ---! Extracts index terms (i) and version (v) from a raw JSONB encrypted value. ---! Returns metadata object containing searchable index terms without ciphertext. ---! ---! @param jsonb containing encrypted EQL payload ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Extract metadata to inspect index terms ---! SELECT eql_v2.meta_data('{"c":"...","i":{"unique":"abc123"},"v":1}'::jsonb); ---! -- Returns: {"i":{"unique":"abc123"},"v":1} ---! ---! @see eql_v2.meta_data(eql_v2_encrypted) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val jsonb) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT jsonb_build_object('i', val->'i', 'v', val->'v'); -$$; - ---! @brief Extract metadata from encrypted column value ---! ---! Extracts index terms and version from an encrypted column value. ---! Convenience overload that unwraps eql_v2_encrypted type and ---! delegates to JSONB version. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return JSONB Metadata object with 'i' (index terms) and 'v' (version) fields ---! ---! @example ---! -- Inspect index terms for encrypted column ---! SELECT user_id, eql_v2.meta_data(encrypted_email) as email_metadata ---! FROM users; ---! ---! @see eql_v2.meta_data(jsonb) ---! @see eql_v2.ciphertext -CREATE FUNCTION eql_v2.meta_data(val eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.meta_data(val.data); -$$; - --- AUTOMATICALLY GENERATED FILE - ---! @file common.sql ---! @brief Common utility functions ---! ---! Provides general-purpose utility functions used across EQL: ---! - Constant-time bytea comparison for security ---! - JSONB to bytea array conversion ---! - Logging helpers for debugging and testing - - ---! @brief Constant-time comparison of bytea values ---! @internal ---! ---! Compares two bytea values in constant time to prevent timing attacks. ---! Always checks all bytes even after finding differences, maintaining ---! consistent execution time regardless of where differences occur. ---! ---! @param a bytea First value to compare ---! @param b bytea Second value to compare ---! @return boolean True if values are equal ---! ---! @note Returns false immediately if lengths differ (length is not secret) ---! @note Used for secure comparison of cryptographic values -CREATE FUNCTION eql_v2.bytea_eq(a bytea, b bytea) RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result boolean; - differing bytea; -BEGIN - - -- Check if the bytea values are the same length - IF LENGTH(a) != LENGTH(b) THEN - RETURN false; - END IF; - - -- Compare each byte in the bytea values - result := true; - FOR i IN 1..LENGTH(a) LOOP - IF SUBSTRING(a FROM i FOR 1) != SUBSTRING(b FROM i FOR 1) THEN - result := result AND false; - END IF; - END LOOP; - - RETURN result; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Convert JSONB hex array to bytea array ---! @internal ---! ---! Converts a JSONB array of hex-encoded strings into a PostgreSQL bytea array. ---! Used for deserializing binary data (like ORE terms) from JSONB storage. ---! ---! @param jsonb JSONB array of hex-encoded strings ---! @return bytea[] Array of decoded binary values ---! ---! @note Returns NULL if input is JSON null ---! @note Each array element is hex-decoded to bytea -CREATE FUNCTION eql_v2.jsonb_array_to_bytea_array(val jsonb) -RETURNS bytea[] - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms_arr bytea[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(decode(value::text, 'hex')::bytea) - INTO terms_arr - FROM jsonb_array_elements_text(val) AS value; - - RETURN terms_arr; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message for debugging ---! ---! Convenience function to emit log messages during testing and debugging. ---! Uses RAISE NOTICE to output messages to PostgreSQL logs. ---! ---! @param text Message to log ---! ---! @note Primarily used in tests and development ---! @see eql_v2.log(text, text) for contextual logging -CREATE FUNCTION eql_v2.log(s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] %', s; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Log message with context ---! ---! Overload of log function that includes context label for better ---! log organization during testing. ---! ---! @param ctx text Context label (e.g., test name, module name) ---! @param s text Message to log ---! ---! @note Format: "[LOG] {ctx} {message}" ---! @see eql_v2.log(text) -CREATE FUNCTION eql_v2.log(ctx text, s text) - RETURNS void - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RAISE NOTICE '[LOG] % %', ctx, s; -END; -$$ LANGUAGE plpgsql; - ---! @brief CLLW ORE index term type for STE-vec range queries ---! ---! Composite type for CLLW (Copyless Logarithmic Width) Order-Revealing ---! Encryption. The ciphertext is stored in the `oc` field of encrypted data ---! payloads (Standard-mode `ste_vec` elements). Used by `eql_v2.compare` and ---! the range operators (`<`, `<=`, `>`, `>=`) when the payload carries an ---! `oc` term. ---! ---! The wire-format `oc` value is a hex string with a leading domain-tag byte ---! (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext. The ---! decoded `bytes` field on this composite carries the full byte string ---! including the tag — the comparator is variable-length capable, so numeric ---! and string values within the same column are ordered correctly: the ---! domain tag separates the two ranges (numeric < string) and the ---! within-domain comparison falls through to the CLLW per-byte protocol. ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.compare_ore_cllw ---! @note This is a transient type used only during query execution -CREATE TYPE eql_v2.ore_cllw AS ( - bytes bytea -); - ---! @brief Extract HMAC-SHA256 index term from JSONB payload ---! ---! Extracts the HMAC-SHA256 hash value from the 'hm' field of an encrypted ---! data payload. Inlinable single-statement SQL — the planner can fold this ---! into the calling query so functional hash indexes built on ---! `eql_v2.hmac_256(col)` engage structurally. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @note Returns NULL when the payload lacks `hm`. Callers that need to ---! surface misconfiguration loudly should use ---! `eql_v2.hash_encrypted` (`GROUP BY` / `DISTINCT` / hash joins) ---! which raises with a clear message when `hm` is missing. ---! ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare_hmac_256 ---! @see eql_v2.hash_encrypted -CREATE FUNCTION eql_v2.hmac_256(val jsonb) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (val ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if JSONB payload contains HMAC-SHA256 index term ---! ---! Tests whether the encrypted data payload includes an 'hm' field, ---! indicating an HMAC-SHA256 hash is available for exact-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'hm' field is present and non-null ---! ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.has_hmac_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'hm' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains HMAC-SHA256 index term ---! ---! Tests whether an encrypted column value includes an HMAC-SHA256 hash ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if HMAC-SHA256 hash is present ---! ---! @see eql_v2.has_hmac_256(jsonb) -CREATE FUNCTION eql_v2.has_hmac_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_hmac_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract HMAC-SHA256 index term from encrypted column value ---! ---! Extracts the HMAC-SHA256 hash from an encrypted column value. Inlinable ---! single-statement SQL — see the jsonb overload for the rationale. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.hmac_256 HMAC-SHA256 hash value, or NULL when `hm` is absent ---! ---! @see eql_v2.hmac_256(jsonb) -CREATE FUNCTION eql_v2.hmac_256(val eql_v2_encrypted) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ((val).data ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Extract HMAC-SHA256 index term from a ste_vec entry ---! ---! Extracts the HMAC from the `hm` field of an `sv` element extracted via ---! the `->` operator. Inlinable. The recipe for field-level equality on ---! encrypted JSON is: ---! ---! @example ---! -- Functional hash index ---! CREATE INDEX ON users USING hash (eql_v2.hmac_256(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.hmac_256 HMAC value, or NULL when `hm` is absent ---! ---! @see eql_v2.has_hmac_256 ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.hmac_256(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.hmac_256 - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (entry ->> 'hm')::eql_v2.hmac_256 -$$; - - ---! @brief Check if a ste_vec entry contains an HMAC-SHA256 index term ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `hm` field is present and non-null -CREATE FUNCTION eql_v2.has_hmac_256(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'hm' IS NOT NULL -$$; - - - - ---! @brief Convert JSONB array to ORE block composite type ---! @internal ---! ---! Converts a JSONB array of hex-encoded ORE terms from the CipherStash Proxy ---! payload into the PostgreSQL composite type used for ORE operations. ---! ---! @param val JSONB Array of hex-encoded ORE block terms ---! @return eql_v2.ore_block_u64_8_256 ORE block composite type, or NULL if input is null ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_to_ore_block_u64_8_256(val jsonb) -RETURNS eql_v2.ore_block_u64_8_256 - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - terms eql_v2.ore_block_u64_8_256_term[]; -BEGIN - IF jsonb_typeof(val) = 'null' THEN - RETURN NULL; - END IF; - - SELECT array_agg(ROW(b)::eql_v2.ore_block_u64_8_256_term) - INTO terms - FROM unnest(eql_v2.jsonb_array_to_bytea_array(val)) AS b; - - RETURN ROW(terms)::eql_v2.ore_block_u64_8_256; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from JSONB payload ---! ---! Extracts the ORE block array from the 'ob' field of an encrypted ---! data payload. Used internally for range query comparisons. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! @throws Exception if 'ob' field is missing when ore index is expected ---! ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2.compare_ore_block_u64_8_256 -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val jsonb) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(val) THEN - RETURN eql_v2.jsonb_array_to_ore_block_u64_8_256(val->'ob'); - END IF; - RAISE 'Expected an ore index (ob) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract ORE block index term from encrypted column value ---! ---! Extracts the ORE block from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.ore_block_u64_8_256 ORE block index term ---! ---! @see eql_v2.ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains ORE block index term ---! ---! Tests whether the encrypted data payload includes an 'ob' field, ---! indicating an ORE block is available for range queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'ob' field is present and non-null ---! ---! @see eql_v2.ore_block_u64_8_256 -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'ob' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains ORE block index term ---! ---! Tests whether an encrypted column value includes an ORE block ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if ORE block is present ---! ---! @see eql_v2.has_ore_block_u64_8_256(jsonb) -CREATE FUNCTION eql_v2.has_ore_block_u64_8_256(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_ore_block_u64_8_256(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Compare two ORE block terms using cryptographic comparison ---! @internal ---! ---! Performs a three-way comparison (returns -1/0/1) of individual ORE block terms ---! using the ORE cryptographic protocol. Compares PRP and PRF blocks to determine ---! ordering without decryption. ---! ---! @param a eql_v2.ore_block_u64_8_256_term First ORE term to compare ---! @param b eql_v2.ore_block_u64_8_256_term Second ORE term to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! @throws Exception if ciphertexts are different lengths ---! ---! @note Uses AES-ECB encryption for bit comparisons per ORE protocol ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_term(a eql_v2.ore_block_u64_8_256_term, b eql_v2.ore_block_u64_8_256_term) - RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - eq boolean := true; - unequal_block smallint := 0; - hash_key bytea; - data_block bytea; - encrypt_block bytea; - target_block bytea; - - left_block_size CONSTANT smallint := 16; - right_block_size CONSTANT smallint := 32; - right_offset CONSTANT smallint := 136; -- 8 * 17 - - indicator smallint := 0; - BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF bit_length(a.bytes) != bit_length(b.bytes) THEN - RAISE EXCEPTION 'Ciphertexts are different lengths'; - END IF; - - FOR block IN 0..7 LOOP - -- Compare each PRP (byte from the first 8 bytes) and PRF block (8 byte - -- chunks of the rest of the value). - -- NOTE: - -- * Substr is ordinally indexed (hence 1 and not 0, and 9 and not 8). - -- * We are not worrying about timing attacks here; don't fret about - -- the OR or !=. - IF - substr(a.bytes, 1 + block, 1) != substr(b.bytes, 1 + block, 1) - OR substr(a.bytes, 9 + left_block_size * block, left_block_size) != substr(b.bytes, 9 + left_block_size * BLOCK, left_block_size) - THEN - -- set the first unequal block we find - IF eq THEN - unequal_block := block; - END IF; - eq = false; - END IF; - END LOOP; - - IF eq THEN - RETURN 0::integer; - END IF; - - -- Hash key is the IV from the right CT of b - hash_key := substr(b.bytes, right_offset + 1, 16); - - -- first right block is at right offset + nonce_size (ordinally indexed) - target_block := substr(b.bytes, right_offset + 17 + (unequal_block * right_block_size), right_block_size); - - data_block := substr(a.bytes, 9 + (left_block_size * unequal_block), left_block_size); - - encrypt_block := encrypt(data_block::bytea, hash_key::bytea, 'aes-ecb'); - - indicator := ( - get_bit( - encrypt_block, - 0 - ) + get_bit(target_block, get_byte(a.bytes, unequal_block))) % 2; - - IF indicator = 1 THEN - RETURN 1::integer; - ELSE - RETURN -1::integer; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Compare arrays of ORE block terms recursively ---! @internal ---! ---! Recursively compares arrays of ORE block terms element-by-element. ---! Empty arrays are considered less than non-empty arrays. If the first elements ---! are equal, recursively compares remaining elements. ---! ---! @param a eql_v2.ore_block_u64_8_256_term[] First array of ORE terms ---! @param b eql_v2.ore_block_u64_8_256_term[] Second array of ORE terms ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b, NULL if either array is NULL ---! ---! @note Empty arrays sort before non-empty arrays ---! @see eql_v2.compare_ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256_term[], b eql_v2.ore_block_u64_8_256_term[]) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - cmp_result integer; - BEGIN - - -- NULLs are NULL - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- empty a and b - IF cardinality(a) = 0 AND cardinality(b) = 0 THEN - RETURN 0; - END IF; - - -- empty a and some b - IF (cardinality(a) = 0) AND cardinality(b) > 0 THEN - RETURN -1; - END IF; - - -- some a and empty b - IF cardinality(a) > 0 AND (cardinality(b) = 0) THEN - RETURN 1; - END IF; - - cmp_result := eql_v2.compare_ore_block_u64_8_256_term(a[1], b[1]); - - IF cmp_result = 0 THEN - -- Removes the first element in the array, and calls this fn again to compare the next element/s in the array. - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a[2:array_length(a,1)], b[2:array_length(b,1)]); - END IF; - - RETURN cmp_result; - END -$$ LANGUAGE plpgsql; - - ---! @brief Compare ORE block composite types ---! @internal ---! ---! Wrapper function that extracts term arrays from ORE block composite types ---! and delegates to the array comparison function. ---! ---! @param a eql_v2.ore_block_u64_8_256 First ORE block ---! @param b eql_v2.ore_block_u64_8_256 Second ORE block ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms(eql_v2.ore_block_u64_8_256_term[], eql_v2.ore_block_u64_8_256_term[]) -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS integer - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a.terms, b.terms); - END -$$ LANGUAGE plpgsql; - - ---! @brief Extract CLLW ORE index term from a ste_vec entry ---! ---! Returns the CLLW ORE ciphertext from the `oc` field of an `sv` element. ---! `oc` is **only ever present on a `SteVecElement`** in the v2.3 payload ---! shape — never at the root of an `eql_v2_encrypted` column value — so the ---! type signature accepts `eql_v2.ste_vec_entry` directly. Callers must ---! extract first: `eql_v2.ore_cllw(col -> '<selector>')`. ---! ---! Inlinable single-statement SQL — the planner folds the body into the ---! calling query so the extractor disappears at planning time. Functional ---! btree index match on this extractor requires the `eql_v2.ore_cllw_ops` ---! opclass (installed automatically by the main / protect variants; absent ---! in the supabase variant). ---! ---! **Missing-`oc` semantics**: when the `oc` field is absent, returns a ---! SQL-level NULL (not a composite with NULL bytes). Btree's standard ---! NULL handling then filters those rows from range queries: they don't ---! match `WHERE ore_cllw(col) <op> $1`, they sort at the NULLS LAST end ---! of `ORDER BY ore_cllw(col)`, and they never reach the comparator. ---! This avoids the btree FUNCTION 1 contract violation that ---! `(bytes => NULL)` would otherwise cause (`compare_ore_cllw_term` ---! must return non-NULL int for non-NULL composite inputs). ---! ---! Callers needing a loud RAISE on missing `oc` should check ---! `eql_v2.has_ore_cllw(entry)` first. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. ---! ---! @see eql_v2.has_ore_cllw ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/->.sql -CREATE FUNCTION eql_v2.ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN entry ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(entry ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Extract CLLW ORE index term from raw jsonb (RHS parameter helper) ---! ---! Companion overload for `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` that ---! accepts a raw `jsonb` value. Intended for the right-hand side of ---! comparisons where the caller binds a literal/parameter jsonb representing ---! a single ste_vec entry: `... < eql_v2.ore_cllw($1::jsonb)`. The (jsonb) ---! form skips the domain CHECK constraint so it works for ad-hoc test inputs ---! and for the GenericComparison case in `eql_v2.compare_ore_cllw_term`. ---! ---! Returns SQL-level NULL when the input lacks `oc`, matching the ---! `(ste_vec_entry)` overload's missing-`oc` semantics so a `WHERE ---! ore_cllw(col) < ore_cllw($1::jsonb)` with a malformed query needle ---! evaluates to no rows rather than indexing a NULL-bytes composite. ---! ---! @param val jsonb An object carrying an `oc` field ---! @return eql_v2.ore_cllw Composite carrying the CLLW ciphertext, or ---! NULL when the `oc` field is absent. -CREATE FUNCTION eql_v2.ore_cllw(val jsonb) - RETURNS eql_v2.ore_cllw - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE WHEN val ->> 'oc' IS NULL THEN NULL - ELSE ROW(decode(val ->> 'oc', 'hex'))::eql_v2.ore_cllw - END -$$; - - ---! @brief Check if a ste_vec entry contains a CLLW ORE index term ---! ---! Tests whether the entry includes an `oc` field. Inlinable. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Boolean True if `oc` field is present and non-null ---! ---! @see eql_v2.ore_cllw -CREATE FUNCTION eql_v2.has_ore_cllw(entry eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 'oc' IS NOT NULL -$$; - - ---! @brief Check if a raw jsonb value contains a CLLW ORE index term ---! ---! Companion to `eql_v2.has_ore_cllw(ste_vec_entry)` for raw jsonb inputs. ---! ---! @param val jsonb An object that may carry an `oc` field ---! @return Boolean True if `oc` field is present and non-null -CREATE FUNCTION eql_v2.has_ore_cllw(val jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT val ->> 'oc' IS NOT NULL -$$; - - ---! @brief CLLW per-byte comparison helper ---! @internal ---! ---! Byte-by-byte comparison implementing the CLLW order-revealing protocol. ---! Used by `eql_v2.compare_ore_cllw_term` for the within-prefix step. The ---! protocol: identify the index of the first differing byte across both ---! inputs; if `(y_byte + 1) == x_byte` modulo 256 at that index, then x > y; ---! otherwise x < y. Equal inputs return 0. ---! ---! Inputs MUST be the same length. The caller (`compare_ore_cllw_term`) ---! guarantees this by passing equal-length prefixes. ---! ---! @par Soft constant-time intent ---! Plpgsql is not a constant-time environment — the interpreter, `SUBSTRING`, ---! `get_byte`, and the SQL bytea representation all leak timing in ways we ---! can't control from here. Still, the loop deliberately walks every byte ---! (no `EXIT` on first difference) and the rotation check uses a bitmask ---! (`& 255`) instead of `% 256` so that what little timing structure plpgsql ---! does expose is independent of the position and value of the differing ---! byte. This is hardening intent, not a guarantee. ---! ---! Stays `LANGUAGE plpgsql` — the per-byte loop can't be expressed as a ---! single inlinable SQL expression. This is the architectural reason ORE ---! CLLW needs a custom operator class for index match, where OPE does not. ---! ---! @param a Bytea First CLLW ciphertext slice ---! @param b Bytea Second CLLW ciphertext slice ---! @return Integer -1, 0, or 1 ---! @throws Exception if inputs are different lengths ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.compare_ore_cllw_term_bytes(a bytea, b bytea) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - i INT; - first_diff INT := 0; -BEGIN - - len_a := LENGTH(a); - len_b := LENGTH(b); - - IF len_a != len_b THEN - RAISE EXCEPTION 'ore_cllw index terms are not the same length'; - END IF; - - -- Walk every byte, even after a difference is found. Record only the - -- index of the first difference (1-based; 0 means "no difference"). - -- Avoids an early `EXIT` whose presence is itself a timing signal. - FOR i IN 1..len_a LOOP - IF first_diff = 0 AND get_byte(a, i - 1) != get_byte(b, i - 1) THEN - first_diff := i; - END IF; - END LOOP; - - IF first_diff = 0 THEN - RETURN 0; - END IF; - - -- Bitmask instead of `% 256` — the modulo's operand is a power of two - -- so the two are arithmetically equivalent, but `& 255` is a single - -- machine instruction with no division-related timing variance. - IF ((get_byte(b, first_diff - 1) + 1) & 255) = get_byte(a, first_diff - 1) THEN - RETURN 1; - ELSE - RETURN -1; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Variable-length CLLW ORE term comparison ---! @internal ---! ---! Three-way comparison of two CLLW ORE ciphertext terms of potentially ---! different lengths. Compares the shared prefix via the CLLW per-byte ---! protocol; on equal prefixes, the shorter input sorts first. ---! ---! Handles both numeric (Standard-mode 65-byte CLLW outputs from the u64 ---! variant) and string (variable-length CLLW outputs) by virtue of the ---! domain-tag byte being the first byte of `bytes`. A numeric/string pair ---! differs at byte 0 (`0x00` vs `0x01`), which the CLLW rule resolves ---! correctly to numeric < string. ---! ---! Stays `LANGUAGE plpgsql` because it dispatches to ---! `compare_ore_cllw_term_bytes`, which can't be inlined. ---! ---! @par Null handling — btree FUNCTION 1 contract ---! PostgreSQL's btree filters NULL composites at the row level, so this ---! function should never be called with `a IS NULL` or `b IS NULL` under ---! normal operation. The leading IS-NULL guard returns NULL defensively ---! to cover edge cases (e.g., a non-index `ORDER BY` or `WHERE` path ---! that bypasses the opclass). ---! ---! A composite that is non-NULL but whose `bytes` field is NULL is a ---! contract violation: btree expects FUNCTION 1 to return a non-NULL ---! integer for non-NULL composite inputs. The extractor overloads of ---! `eql_v2.ore_cllw` are designed to return SQL NULL (not `ROW(NULL)`) ---! when the source payload lacks `oc`, so a NULL-bytes composite should ---! only arise from a hand-crafted literal or a future field addition to ---! the composite type. Raise loudly to surface the bug instead of ---! producing silent misordering downstream. ---! ---! @param a eql_v2.ore_cllw First term ---! @param b eql_v2.ore_cllw Second term ---! @return Integer -1, 0, or 1; NULL if either composite is NULL ---! @throws Exception if either composite has a NULL `bytes` field ---! ---! @see eql_v2.compare_ore_cllw_term_bytes ---! @see eql_v2.compare_ore_cllw -CREATE FUNCTION eql_v2.compare_ore_cllw_term(a eql_v2.ore_cllw, b eql_v2.ore_cllw) -RETURNS int - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - len_a INT; - len_b INT; - common_len INT; - cmp_result INT; -BEGIN - -- Composite-level NULL: btree's null-handling layer filters these at - -- the row level under normal operation. Returning NULL covers - -- non-index code paths that might still reach here. - IF a IS NULL OR b IS NULL THEN - RETURN NULL; - END IF; - - -- Non-NULL composite with NULL bytes is a contract violation: btree's - -- FUNCTION 1 must return non-NULL int for non-NULL composite inputs. - -- The extractors return SQL NULL (not ROW(NULL)) on missing `oc`, so - -- reaching here means a hand-crafted literal or a regression in the - -- extractor body. Raise loudly rather than silently misorder. - IF a.bytes IS NULL OR b.bytes IS NULL THEN - RAISE EXCEPTION 'eql_v2.compare_ore_cllw_term: composite has NULL bytes field — extractor invariant violated. Check that the index expression uses eql_v2.ore_cllw(...) and not a hand-crafted ROW(NULL).'; - END IF; - - len_a := LENGTH(a.bytes); - len_b := LENGTH(b.bytes); - - IF len_a = 0 AND len_b = 0 THEN - RETURN 0; - ELSIF len_a = 0 THEN - RETURN -1; - ELSIF len_b = 0 THEN - RETURN 1; - END IF; - - IF len_a < len_b THEN - common_len := len_a; - ELSE - common_len := len_b; - END IF; - - cmp_result := eql_v2.compare_ore_cllw_term_bytes( - SUBSTRING(a.bytes FROM 1 FOR common_len), - SUBSTRING(b.bytes FROM 1 FOR common_len) - ); - - IF cmp_result = -1 THEN - RETURN -1; - ELSIF cmp_result = 1 THEN - RETURN 1; - END IF; - - -- Equal prefixes: shorter sorts first - IF len_a < len_b THEN - RETURN -1; - ELSIF len_a > len_b THEN - RETURN 1; - ELSE - RETURN 0; - END IF; -END; -$$ LANGUAGE plpgsql; - - - ---! @brief Convert JSONB to encrypted type ---! ---! Wraps a JSONB encrypted payload into the eql_v2_encrypted composite type. ---! Used internally for type conversions and operator implementations. ---! ---! @param jsonb JSONB encrypted payload with structure: {"c": "...", "i": {...}, "k": "...", "v": "2"} ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note This is primarily used for implicit casts in operator expressions ---! @see eql_v2.to_jsonb -CREATE FUNCTION eql_v2.to_encrypted(data jsonb) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT ROW(data)::public.eql_v2_encrypted; -$$; - - ---! @brief Implicit cast from JSONB to encrypted type ---! ---! Enables PostgreSQL to automatically convert JSONB values to eql_v2_encrypted ---! in assignment contexts and comparison operations. ---! ---! @see eql_v2.to_encrypted(jsonb) -CREATE CAST (jsonb AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(jsonb) AS ASSIGNMENT; - - ---! @brief Convert text to encrypted type ---! ---! Parses a text representation of encrypted JSONB payload and wraps it ---! in the eql_v2_encrypted composite type. ---! ---! @param text Text representation of JSONB encrypted payload ---! @return eql_v2_encrypted Encrypted value wrapped in composite type ---! ---! @note Delegates to eql_v2.to_encrypted(jsonb) after parsing text as JSON ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_encrypted(data text) - RETURNS public.eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT eql_v2.to_encrypted(data::jsonb); -$$; - - ---! @brief Implicit cast from text to encrypted type ---! ---! Enables PostgreSQL to automatically convert text JSON strings to eql_v2_encrypted ---! in assignment contexts. ---! ---! @see eql_v2.to_encrypted(text) -CREATE CAST (text AS public.eql_v2_encrypted) - WITH FUNCTION eql_v2.to_encrypted(text) AS ASSIGNMENT; - - - ---! @brief Convert encrypted type to JSONB ---! ---! Extracts the underlying JSONB payload from an eql_v2_encrypted composite type. ---! Useful for debugging or when raw encrypted payload access is needed. ---! ---! @param e eql_v2_encrypted Encrypted value to unwrap ---! @return jsonb Raw JSONB encrypted payload ---! ---! @note Returns the raw encrypted structure including ciphertext and index terms ---! @see eql_v2.to_encrypted(jsonb) -CREATE FUNCTION eql_v2.to_jsonb(e public.eql_v2_encrypted) - RETURNS jsonb - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT e.data; -$$; - ---! @brief Implicit cast from encrypted type to JSONB ---! ---! Enables PostgreSQL to automatically extract the JSONB payload from ---! eql_v2_encrypted values in assignment contexts. ---! ---! @see eql_v2.to_jsonb(eql_v2_encrypted) -CREATE CAST (public.eql_v2_encrypted AS jsonb) - WITH FUNCTION eql_v2.to_jsonb(public.eql_v2_encrypted) AS ASSIGNMENT; - - - - - ---! @brief Compare two encrypted values using HMAC-SHA256 index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their HMAC-SHA256 hash index terms. Used internally by the equality operator (=) ---! for exact-match queries without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Comparison uses underlying text type ordering of HMAC-SHA256 hashes ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2."=" -CREATE FUNCTION eql_v2.compare_hmac_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.hmac_256; - b_term eql_v2.hmac_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_hmac_256(a) THEN - a_term = eql_v2.hmac_256(a); - END IF; - - IF eql_v2.has_hmac_256(b) THEN - b_term = eql_v2.hmac_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - -- Using the underlying text type comparison - IF a_term = b_term THEN - RETURN 0; - END IF; - - IF a_term < b_term THEN - RETURN -1; - END IF; - - IF a_term > b_term THEN - RETURN 1; - END IF; - - END; -$$ LANGUAGE plpgsql; - - - ---! @file src/operators/compare.sql ---! @brief Three-way ordering on the root `eql_v2_encrypted` type ---! ---! Returns `-1` / `0` / `1` for two encrypted column values that carry ---! Block ORE (`ob`) terms at the root. Used by the btree operator class on ---! `eql_v2_encrypted` (FUNCTION 1), by the legacy `eql_v2.lt` / `lte` / ---! `gt` / `gte` helpers, and by `sort_compare`'s `strategy = 'compare'` ---! fallback path. ---! ---! **Strict Block-ORE-only contract.** Root-level `eql_v2_encrypted` values ---! only carry root-scope ORE terms (`ob`) per the v2.3 payload shape — the ---! `oc` field (CLLW ORE) is sv-element scope only and never appears on a ---! root payload. Equality on `eql_v2_encrypted` is hm-only and runs through ---! the inlined `=` / `<>` operators (post-#193) — it does *not* go through ---! this function. For sv-element ordering, use the typed ---! `eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry)` overload ---! (or the `<` / `<=` / `>` / `>=` operators on the same pair). ---! ---! @param a eql_v2_encrypted First encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @param b eql_v2_encrypted Second encrypted value (STRICT — NULL inputs short-circuit to NULL) ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either value lacks an `ob` (Block ORE) term ---! ---! @see eql_v2.compare_ore_block_u64_8_256 ---! @see eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry) ---! @see eql_v2."=" -- hm-only equality, post-#193 inlining -CREATE FUNCTION eql_v2.compare(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF eql_v2.has_ore_block_u64_8_256(a) AND eql_v2.has_ore_block_u64_8_256(b) THEN - RETURN eql_v2.compare_ore_block_u64_8_256(a, b); - END IF; - - RAISE EXCEPTION - 'eql_v2.compare requires Block ORE (`ob`) on both root operands. For sv-element ordering, extract entries via `col -> ''<selector>''` and use eql_v2.compare on the resulting `eql_v2.ste_vec_entry` values (or their `<` / `<=` / `>` / `>=` operators). Equality is hmac-only via the `=` operator — this function is for ordering only.' - USING ERRCODE = 'feature_not_supported'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Three-way ordering on `eql_v2.ste_vec_entry` ---! ---! CLLW ORE three-way comparator on ste-vec entries. Returns `-1` / `0` / ---! `1` by extracting the `oc` term from each entry and delegating to ---! `eql_v2.compare_ore_cllw_term`. Use this when you need an `int` ordering ---! out of two extracted ste-vec entries — for the boolean-form operators ---! (`<` / `<=` / `>` / `>=`) on the same pair, see ---! `src/operators/ste_vec_entry.sql`. ---! ---! Note: the caller is responsible for extracting an `eql_v2.ste_vec_entry` ---! first; the `(eql_v2_encrypted, text)` form would be a natural extension ---! but is deliberately *not* added here so that callers stay aware of the ---! two-step shape (extract via `->`, then compare). ---! ---! @param a eql_v2.ste_vec_entry First entry ---! @param b eql_v2.ste_vec_entry Second entry ---! @return integer -1, 0, or 1 ---! ---! @throws Exception when either entry lacks an `oc` term ---! ---! @see eql_v2.compare_ore_cllw_term ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.compare(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF NOT (eql_v2.has_ore_cllw(a) AND eql_v2.has_ore_cllw(b)) THEN - RAISE EXCEPTION - 'eql_v2.compare(ste_vec_entry, ste_vec_entry) requires `oc` (CLLW ORE) on both entries.' - USING ERRCODE = 'feature_not_supported'; - END IF; - - RETURN eql_v2.compare_ore_cllw_term(eql_v2.ore_cllw(a), eql_v2.ore_cllw(b)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Equality operator for ORE block types ---! @internal ---! ---! Implements the = operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_eq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 0 -$$; - - - ---! @brief Not equal operator for ORE block types ---! @internal ---! ---! Implements the <> operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if ORE blocks are not equal ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_neq(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) <> 0 -$$; - - - ---! @brief Less than operator for ORE block types ---! @internal ---! ---! Implements the < operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = -1 -$$; - - - ---! @brief Less than or equal operator for ORE block types ---! @internal ---! ---! Implements the <= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is less than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_lte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != 1 -$$; - - - ---! @brief Greater than operator for ORE block types ---! @internal ---! ---! Implements the > operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gt(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) = 1 -$$; - - - ---! @brief Greater than or equal operator for ORE block types ---! @internal ---! ---! Implements the >= operator for direct ORE block comparisons. ---! ---! @param a eql_v2.ore_block_u64_8_256 Left operand ---! @param b eql_v2.ore_block_u64_8_256 Right operand ---! @return Boolean True if left operand is greater than or equal to right operand ---! ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE FUNCTION eql_v2.ore_block_u64_8_256_gte(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256) -RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_block_u64_8_256_terms(a, b) != -1 -$$; - - - ---! @brief = operator for ORE block types -CREATE OPERATOR = ( - FUNCTION=eql_v2.ore_block_u64_8_256_eq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - - ---! @brief <> operator for ORE block types -CREATE OPERATOR <> ( - FUNCTION=eql_v2.ore_block_u64_8_256_neq, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief > operator for ORE block types -CREATE OPERATOR > ( - FUNCTION=eql_v2.ore_block_u64_8_256_gt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - - ---! @brief < operator for ORE block types -CREATE OPERATOR < ( - FUNCTION=eql_v2.ore_block_u64_8_256_lt, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - - ---! @brief <= operator for ORE block types -CREATE OPERATOR <= ( - FUNCTION=eql_v2.ore_block_u64_8_256_lte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - - ---! @brief >= operator for ORE block types -CREATE OPERATOR >= ( - FUNCTION=eql_v2.ore_block_u64_8_256_gte, - LEFTARG=eql_v2.ore_block_u64_8_256, - RIGHTARG=eql_v2.ore_block_u64_8_256, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - - ---! @brief Extract STE vector index from JSONB payload ---! ---! Extracts the STE (Searchable Symmetric Encryption) vector from the 'sv' field ---! of an encrypted data payload. Returns an array of encrypted values used for ---! containment queries (@>, <@). If no 'sv' field exists, wraps the entire payload ---! as a single-element array. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(eql_v2_encrypted) ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.ste_vec(val jsonb) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv jsonb; - ary public.eql_v2_encrypted[]; - BEGIN - - IF val ? 'sv' THEN - sv := val->'sv'; - ELSE - sv := jsonb_build_array(val); - END IF; - - SELECT array_agg(eql_v2.to_encrypted(elem)) - INTO ary - FROM jsonb_array_elements(sv) AS elem; - - RETURN ary; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract STE vector index from encrypted column value ---! ---! Extracts the STE vector from an encrypted column value by accessing its ---! underlying JSONB data field. Used for containment query operations. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted[] Array of encrypted STE vector elements ---! ---! @see eql_v2.ste_vec(jsonb) -CREATE FUNCTION eql_v2.ste_vec(val eql_v2_encrypted) - RETURNS public.eql_v2_encrypted[] - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.ste_vec(val.data)); - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if JSONB payload is a single-element STE vector ---! ---! Tests whether the encrypted data payload contains an 'sv' field with exactly ---! one element. Single-element STE vectors can be treated as regular encrypted values. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'sv' field exists with exactly one element ---! ---! @see eql_v2.to_ste_vec_value -CREATE FUNCTION eql_v2.is_ste_vec_value(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'sv' THEN - RETURN jsonb_array_length(val->'sv') = 1; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - ---! @brief Check if encrypted column value is a single-element STE vector ---! ---! Tests whether an encrypted column value is a single-element STE vector ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is a single-element STE vector ---! ---! @see eql_v2.is_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_value(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.is_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value ---! ---! Extracts the single element from a single-element STE vector and returns it ---! as a regular encrypted value, preserving metadata. If the input is not a ---! single-element STE vector, returns it unchanged. ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.is_ste_vec_value -CREATE FUNCTION eql_v2.to_ste_vec_value(val jsonb) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - meta jsonb; - sv jsonb; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_value(val) THEN - meta := eql_v2.meta_data(val); - sv := val->'sv'; - sv := sv[0]; - - RETURN eql_v2.to_encrypted(meta || sv); - END IF; - - RETURN eql_v2.to_encrypted(val); - END; -$$ LANGUAGE plpgsql; - ---! @brief Convert single-element STE vector to regular encrypted value (encrypted type) ---! ---! Converts an encrypted column value to a regular encrypted value by unwrapping ---! if it's a single-element STE vector. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2_encrypted Regular encrypted value (unwrapped if single-element STE vector) ---! ---! @see eql_v2.to_ste_vec_value(jsonb) -CREATE FUNCTION eql_v2.to_ste_vec_value(val eql_v2_encrypted) - RETURNS eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.to_ste_vec_value(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @brief Extract selector value from JSONB payload ---! ---! Extracts the selector ('s') field from an encrypted data payload. ---! Selectors are used to match STE vector elements during containment queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Text The selector value ---! @throws Exception if 's' field is missing ---! ---! @see eql_v2.ste_vec_contains -CREATE FUNCTION eql_v2.selector(val jsonb) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF val ? 's' THEN - RETURN val->>'s'; - END IF; - RAISE 'Expected a selector index (s) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from encrypted column value ---! @internal ---! ---! Internal convenience: unwraps the encrypted composite and delegates ---! to `eql_v2.selector(jsonb)`. Exists so the encrypted-selector ---! overloads of `eql_v2."->"` / `eql_v2."->>"` / `eql_v2.jsonb_path_*` ---! can dispatch without each having to spell out `(val).data` first. ---! Not part of the public API — callers should use ---! `eql_v2.selector(jsonb)` or `eql_v2.selector(eql_v2.ste_vec_entry)`. ---! ---! @param eql_v2_encrypted Encrypted column value (single-element form) ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) ---! @see eql_v2.selector(eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2._selector(val eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.selector(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract selector value from a ste_vec entry ---! ---! Direct overload on the domain type. The DOMAIN's CHECK constraint ---! already guarantees `s` is present, so this is a simple field access. ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry ---! @return Text The selector value ---! ---! @see eql_v2.selector(jsonb) -CREATE FUNCTION eql_v2.selector(entry eql_v2.ste_vec_entry) - RETURNS text - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT entry ->> 's' -$$; - - - ---! @brief Check if JSONB payload is marked as an STE vector array ---! ---! Tests whether the encrypted data payload has the 'a' (array) flag set to true, ---! indicating it represents an array for STE vector operations. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'a' field is present and true ---! ---! @see eql_v2.ste_vec -CREATE FUNCTION eql_v2.is_ste_vec_array(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'a' THEN - RETURN (val->>'a')::boolean; - END IF; - - RETURN false; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value is marked as an STE vector array ---! ---! Tests whether an encrypted column value has the array flag set by checking ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if value is marked as an STE vector array ---! ---! @see eql_v2.is_ste_vec_array(jsonb) -CREATE FUNCTION eql_v2.is_ste_vec_array(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.is_ste_vec_array(val.data)); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract full encrypted JSONB elements as array ---! ---! Extracts all JSONB elements from the STE vector including non-deterministic fields. ---! Use jsonb_array() instead for GIN indexing and containment queries. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT CASE - WHEN val ? 'sv' THEN - ARRAY(SELECT elem FROM jsonb_array_elements(val->'sv') AS elem) - ELSE - ARRAY[val] - END; -$$; - - ---! @brief Extract full encrypted JSONB elements as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of full JSONB elements ---! ---! @see eql_v2.jsonb_array_from_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_from_array_elements(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array_from_array_elements(val.data); -$$; - - ---! @brief Extract deterministic fields as array for GIN indexing ---! ---! Extracts only deterministic search term fields (`s`, `hm`, `oc`, `op`) ---! from each STE vector element. Excludes non-deterministic ciphertext for ---! correct containment comparison using PostgreSQL's native `@>` operator. ---! ---! Field set: selector (`s`), HMAC equality (`hm`), ORE CLLW (`oc`, ---! Standard-mode), OPE CLLW (`op`, Compat-mode). The pre-2.3 fields ---! (`b3` / `ocf` / `ocv` / `opf` / `opv`) are no longer emitted — see U-004 ---! and U-006 in `docs/upgrading/v2.3.md`. ---! ---! @param val jsonb containing encrypted EQL payload ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @note Use this for GIN indexes and containment queries ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_array(val jsonb) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT ARRAY( - SELECT jsonb_object_agg(kv.key, kv.value) - FROM jsonb_array_elements( - CASE WHEN val ? 'sv' THEN val->'sv' ELSE jsonb_build_array(val) END - ) AS elem, - LATERAL jsonb_each(elem) AS kv(key, value) - WHERE kv.key IN ('s', 'hm', 'oc', 'op') - GROUP BY elem - ); -$$; - - ---! @brief Extract deterministic fields as array from encrypted column ---! ---! @param val eql_v2_encrypted Encrypted column value ---! @return jsonb[] Array of JSONB elements with only deterministic fields ---! ---! @see eql_v2.jsonb_array(jsonb) -CREATE FUNCTION eql_v2.jsonb_array(val eql_v2_encrypted) -RETURNS jsonb[] -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(val.data); -$$; - - ---! @brief GIN-indexable JSONB containment check ---! ---! Checks if encrypted value 'a' contains all JSONB elements from 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! This function is designed for use with a GIN index on jsonb_array(column). ---! When combined with such an index, PostgreSQL can efficiently search large tables. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b eql_v2_encrypted Value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @example ---! -- Create GIN index for efficient containment queries ---! CREATE INDEX idx ON mytable USING GIN (eql_v2.jsonb_array(encrypted_col)); ---! ---! -- Query using the helper function ---! SELECT * FROM mytable WHERE eql_v2.jsonb_contains(encrypted_col, search_value); ---! ---! @see eql_v2.jsonb_array -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (encrypted, jsonb) ---! ---! Checks if encrypted value 'a' contains all JSONB elements from jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Container value (typically a table column) ---! @param b jsonb JSONB value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB containment check (jsonb, encrypted) ---! ---! Checks if jsonb value 'a' contains all JSONB elements from encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Container JSONB value ---! @param b eql_v2_encrypted Encrypted value to search for ---! @return Boolean True if a contains all elements of b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contains(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) @> eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check ---! ---! Checks if all JSONB elements from 'a' are contained in 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b eql_v2_encrypted Container value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contains -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (encrypted, jsonb) ---! ---! Checks if all JSONB elements from encrypted value 'a' are contained in jsonb value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a eql_v2_encrypted Value to check (typically a table column) ---! @param b jsonb Container JSONB value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a eql_v2_encrypted, b jsonb) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief GIN-indexable JSONB "is contained by" check (jsonb, encrypted) ---! ---! Checks if all JSONB elements from jsonb value 'a' are contained in encrypted value 'b'. ---! Uses jsonb[] arrays internally for native PostgreSQL GIN index support. ---! ---! @param a jsonb Value to check ---! @param b eql_v2_encrypted Container encrypted value ---! @return Boolean True if all elements of a are contained in b ---! ---! @see eql_v2.jsonb_array ---! @see eql_v2.jsonb_contained_by(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.jsonb_contained_by(a jsonb, b eql_v2_encrypted) -RETURNS boolean -IMMUTABLE STRICT PARALLEL SAFE -LANGUAGE SQL -AS $$ - SELECT eql_v2.jsonb_array(a) <@ eql_v2.jsonb_array(b); -$$; - - ---! @brief Check if STE vector array contains a specific encrypted element ---! ---! Tests whether any element in the STE vector array 'a' contains the encrypted value 'b'. ---! Matching requires both the selector and encrypted value to be equal. ---! Used internally by ste_vec_contains(encrypted, encrypted) for array containment checks. ---! ---! @param eql_v2_encrypted[] STE vector array to search within ---! @param eql_v2_encrypted Encrypted element to search for ---! @return Boolean True if b is found in any element of a ---! ---! @note Compares both selector and encrypted value for match ---! ---! @see eql_v2.selector ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.ste_vec_contains(a public.eql_v2_encrypted[], b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - _a public.eql_v2_encrypted; - BEGIN - - result := false; - - FOR idx IN 1..array_length(a, 1) LOOP - _a := a[idx]; - -- Element-level match for ste_vec entries. - -- - -- Per the v2.3 sv-element contract (encoded in - -- `docs/reference/schema/eql-payload-v2.3.schema.json` and the - -- `eql_v2.ste_vec_entry` DOMAIN), each entry carries **exactly - -- one** of: - -- - `hm` — HMAC-256 for boolean leaves and for the placeholder - -- entries that represent array / object roots. - -- - `oc` — CLLW ORE for string and number leaves. - -- Both terms are deterministic for the same plaintext at the same - -- selector under the same workspace, so either one serves as the - -- equality discriminator. A selector configures the leaf's role - -- (eq / ordered), and the role determines which term is emitted — - -- two sv entries with the same selector therefore always carry - -- the same term type. - -- - -- The selector check is a fast-path gate so we don't compare - -- terms across mismatched fields. Once selectors match, exactly - -- one of the two CASE branches fires (XOR contract above). - -- - -- The `ELSE false` arm covers the malformed case (entry carries - -- neither term, or only one side has the term for a given role). - -- That's a data error rather than a normal containment result, - -- but returning false is safer than raising mid-array-scan. - result := result OR ( - eql_v2._selector(_a) = eql_v2._selector(b) AND - CASE - WHEN eql_v2.has_hmac_256(_a) AND eql_v2.has_hmac_256(b) THEN - eql_v2.compare_hmac_256(_a, b) = 0 - WHEN eql_v2.has_ore_cllw((_a).data) AND eql_v2.has_ore_cllw((b).data) THEN - eql_v2.compare_ore_cllw_term( - eql_v2.ore_cllw((_a).data), - eql_v2.ore_cllw((b).data) - ) = 0 - ELSE false - END - ); - - -- Short-circuit once a match is found. Without this we still walk - -- the rest of the sv array, which on a 100-element document means - -- 99 wasted selector + extractor calls per row. - EXIT WHEN result; - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted value 'a' contains all elements of encrypted value 'b' ---! ---! Performs STE vector containment comparison between two encrypted values. ---! Returns true if all elements in b's STE vector are found in a's STE vector. ---! Used internally by the @> containment operator for searchable encryption. ---! ---! @param a eql_v2_encrypted First encrypted value (container) ---! @param b eql_v2_encrypted Second encrypted value (elements to find) ---! @return Boolean True if all elements of b are contained in a ---! ---! @note Empty b is always contained in any a ---! @note Each element of b must match both selector and value in a ---! ---! @see eql_v2.ste_vec ---! @see eql_v2.ste_vec_contains(eql_v2_encrypted[], eql_v2_encrypted) ---! @see eql_v2."@>" -CREATE FUNCTION eql_v2.ste_vec_contains(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - result boolean; - sv_a public.eql_v2_encrypted[]; - sv_b public.eql_v2_encrypted[]; - _b public.eql_v2_encrypted; - BEGIN - - -- jsonb arrays of ste_vec encrypted values - sv_a := eql_v2.ste_vec(a); - sv_b := eql_v2.ste_vec(b); - - -- an empty b is always contained in a - IF array_length(sv_b, 1) IS NULL THEN - RETURN true; - END IF; - - IF array_length(sv_a, 1) IS NULL THEN - RETURN false; - END IF; - - result := true; - - -- for each element of b check if it is in a - FOR idx IN 1..array_length(sv_b, 1) LOOP - _b := sv_b[idx]; - result := result AND eql_v2.ste_vec_contains(sv_a, _b); - END LOOP; - - RETURN result; - END; -$$ LANGUAGE plpgsql; ---! @file config/types.sql ---! @brief Configuration state type definition ---! ---! Defines the ENUM type for tracking encryption configuration lifecycle states. ---! The configuration table uses this type to manage transitions between states ---! during setup, activation, and encryption operations. ---! ---! @note CREATE TYPE does not support IF NOT EXISTS, so wrapped in DO block ---! @note Configuration data stored as JSONB directly, not as DOMAIN ---! @see config/tables.sql - - ---! @brief Configuration lifecycle state ---! ---! Defines valid states for encryption configurations in the eql_v2_configuration table. ---! Configurations transition through these states during setup and activation. ---! ---! @note Only one configuration can be in 'active', 'pending', or 'encrypting' state at once ---! @see config/indexes.sql for uniqueness enforcement ---! @see config/tables.sql for usage in eql_v2_configuration table -DO $$ - BEGIN - IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'eql_v2_configuration_state') THEN - CREATE TYPE public.eql_v2_configuration_state AS ENUM ('active', 'inactive', 'encrypting', 'pending'); - END IF; - END -$$; - - ---! @file src/ore_cllw/operators.sql ---! @brief Comparison operators on the `eql_v2.ore_cllw` composite type ---! ---! Same-type comparison operators backing the btree operator class on the ---! composite `eql_v2.ore_cllw` type. Each operator reduces to a single SELECT ---! over `eql_v2.compare_ore_cllw_term(a, b)`, which is the canonical CLLW ---! per-byte comparator (`y + 1 == x` mod 256). The operator wrappers are ---! inlinable `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` so the planner can ---! fold them into the calling query — that's what lets a functional btree ---! index on `eql_v2.ore_cllw(col)` engage for both `WHERE eql_v2.ore_cllw(col) ---! < eql_v2.ore_cllw($1)` and `ORDER BY eql_v2.ore_cllw(col)` shapes. ---! ---! The inner `eql_v2.compare_ore_cllw_term` is `LANGUAGE plpgsql` (it has a ---! per-byte loop) and is NOT inlined. That's fine for index *match* (the ---! planner only needs the outer operator function call to fold so the ---! predicate's expression tree matches the index's expression tree); only the ---! per-comparison cost is the plpgsql call overhead. That's the cost the ---! functional index avoids by walking the btree in order rather than calling ---! compare on every row. ---! ---! @note Deliberately no `HASHES` / `MERGES` flags on the operator ---! declarations. HASHES requires a registered hash function on the type ---! (the CLLW protocol gives ordering, not a sensible hashing); MERGES ---! requires an equivalent merge-joinable operator class on both sides. ---! ---! @see src/ore_cllw/operator_class.sql ---! @see src/ore_cllw/functions.sql - ---! @brief Equality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare equal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_eq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 0 -$$; - ---! @brief Inequality operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if the CLLW terms compare unequal ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_neq(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 0 -$$; - ---! @brief Less-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = -1 -$$; - ---! @brief Less-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders before or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_lte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> 1 -$$; - ---! @brief Greater-than operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gt(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) = 1 -$$; - ---! @brief Greater-than-or-equal operator backing function for `eql_v2.ore_cllw` ---! @internal ---! ---! @param a eql_v2.ore_cllw Left operand ---! @param b eql_v2.ore_cllw Right operand ---! @return boolean True if `a` orders after or equal to `b` ---! ---! @see eql_v2.compare_ore_cllw_term -CREATE FUNCTION eql_v2.ore_cllw_gte(a eql_v2.ore_cllw, b eql_v2.ore_cllw) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.compare_ore_cllw_term(a, b) <> -1 -$$; - - -CREATE OPERATOR = ( - FUNCTION = eql_v2.ore_cllw_eq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel -); - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.ore_cllw_neq, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - -CREATE OPERATOR < ( - FUNCTION = eql_v2.ore_cllw_lt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.ore_cllw_lte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - -CREATE OPERATOR > ( - FUNCTION = eql_v2.ore_cllw_gt, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.ore_cllw_gte, - LEFTARG = eql_v2.ore_cllw, - RIGHTARG = eql_v2.ore_cllw, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - - ---! @brief Extract Bloom filter index term from JSONB payload ---! ---! Extracts the Bloom filter array from the 'bf' field of an encrypted ---! data payload. Used internally for pattern-match queries (LIKE operator). ---! ---! @param jsonb containing encrypted EQL payload ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! @throws Exception if 'bf' field is missing when bloom_filter index is expected ---! ---! @see eql_v2.has_bloom_filter ---! @see eql_v2."~~" -CREATE FUNCTION eql_v2.bloom_filter(val jsonb) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.has_bloom_filter(val) THEN - RETURN ARRAY(SELECT jsonb_array_elements(val->'bf'))::eql_v2.bloom_filter; - END IF; - - RAISE 'Expected a match index (bf) value in json: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract Bloom filter index term from encrypted column value ---! ---! Extracts the Bloom filter from an encrypted column value by accessing ---! its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return eql_v2.bloom_filter Bloom filter as smallint array ---! ---! @see eql_v2.bloom_filter(jsonb) -CREATE FUNCTION eql_v2.bloom_filter(val eql_v2_encrypted) - RETURNS eql_v2.bloom_filter - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN (SELECT eql_v2.bloom_filter(val.data)); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if JSONB payload contains Bloom filter index term ---! ---! Tests whether the encrypted data payload includes a 'bf' field, ---! indicating a Bloom filter is available for pattern-match queries. ---! ---! @param jsonb containing encrypted EQL payload ---! @return Boolean True if 'bf' field is present and non-null ---! ---! @see eql_v2.bloom_filter -CREATE FUNCTION eql_v2.has_bloom_filter(val jsonb) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN val ->> 'bf' IS NOT NULL; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Check if encrypted column value contains Bloom filter index term ---! ---! Tests whether an encrypted column value includes a Bloom filter ---! by checking its underlying JSONB data field. ---! ---! @param eql_v2_encrypted Encrypted column value ---! @return Boolean True if Bloom filter is present ---! ---! @see eql_v2.has_bloom_filter(jsonb) -CREATE FUNCTION eql_v2.has_bloom_filter(val eql_v2_encrypted) - RETURNS boolean - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.has_bloom_filter(val.data); - END; -$$ LANGUAGE plpgsql; - ---! @file src/ste_vec/eq_term.sql ---! @brief XOR-aware equality term extractor for `eql_v2.ste_vec_entry` ---! ---! Returns the bytea representation of whichever deterministic term ---! the sv entry carries — `hm` (HMAC-256) for bool leaves / array ---! roots / object roots, or `oc` (CLLW ORE) for string / number ---! leaves. The two byte distributions are disjoint by construction ---! (different keys, different protocols), so byte equality on the ---! coalesce is unambiguous: equal terms imply equal plaintexts under ---! the same selector, and unequal terms imply different plaintexts ---! (or different protocols, which can't happen for a single ---! selector). ---! ---! This is the canonical equality extractor used by `=` and `<>` on ---! `eql_v2.ste_vec_entry` — see `src/operators/ste_vec_entry.sql`. ---! The recipe for field-level equality on encrypted JSON is: ---! ---! @example ---! -- Functional hash index covers both hm-bearing and oc-bearing selectors ---! CREATE INDEX ON users USING hash (eql_v2.eq_term(data -> '<selector>')); ---! -- Bare-form predicate matches via the inlined `=` on ste_vec_entry ---! SELECT * FROM users WHERE data -> '<selector>' = $1::eql_v2.ste_vec_entry; ---! ---! @param entry eql_v2.ste_vec_entry STE-vec entry (extracted via `->`) ---! @return bytea Decoded `hm` or `oc` bytes (NULL if entry is NULL). ---! ---! @note The XOR contract (each sv entry carries exactly one of `hm` ---! or `oc` — enforced by the `ste_vec_entry` DOMAIN CHECK) means ---! the coalesce always picks the one present term. ---! ---! @see eql_v2.hmac_256(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/ste_vec_entry.sql -CREATE FUNCTION eql_v2.eq_term(entry eql_v2.ste_vec_entry) - RETURNS bytea - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT decode(coalesce(entry ->> 'hm', entry ->> 'oc'), 'hex') -$$; - ---! @brief Extract ORE index term for ordering encrypted values ---! ---! Helper function that extracts the ore_block_u64_8_256 index term from an encrypted value ---! for use in ORDER BY clauses when comparison operators are not appropriate or available. ---! ---! @param eql_v2_encrypted Encrypted value to extract order term from ---! @return eql_v2.ore_block_u64_8_256 ORE index term for ordering ---! ---! @example ---! -- Order encrypted values without using comparison operators ---! SELECT * FROM users ORDER BY eql_v2.order_by(encrypted_age); ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.order_by(a eql_v2_encrypted) - RETURNS eql_v2.ore_block_u64_8_256 - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.ore_block_u64_8_256(a); - END; -$$ LANGUAGE plpgsql; - ---! @brief Fallback literal comparison for encrypted values ---! @internal ---! ---! Compares two encrypted values by their raw JSONB representation when no ---! suitable index terms are available. This ensures consistent ordering required ---! for btree correctness and prevents "lock BufferContent is not held" errors. ---! ---! Used as a last resort fallback in eql_v2.compare() when encrypted values ---! lack matching index terms (hmac_256, ore). ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note This compares the encrypted payloads directly, not the plaintext values ---! @note Ordering is consistent but not meaningful for range queries ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.compare_literal(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - LANGUAGE SQL -AS $$ - SELECT CASE - WHEN a.data < b.data THEN -1 - WHEN a.data > b.data THEN 1 - ELSE 0 - END; -$$; - - ---! @brief Compare two encrypted values using ORE block index terms ---! ---! Performs a three-way comparison (returns -1/0/1) of encrypted values using ---! their ORE block index terms. Used internally by range operators (<, <=, >, >=) ---! for order-revealing comparisons without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value to compare ---! @param b eql_v2_encrypted Second encrypted value to compare ---! @return Integer -1 if a < b, 0 if a = b, 1 if a > b ---! ---! @note NULL values are sorted before non-NULL values ---! @note Uses ORE cryptographic protocol for secure comparisons ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.has_ore_block_u64_8_256 ---! @see eql_v2."<" ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.compare_ore_block_u64_8_256(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - a_term eql_v2.ore_block_u64_8_256; - b_term eql_v2.ore_block_u64_8_256; - BEGIN - - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - a_term := eql_v2.ore_block_u64_8_256(a); - END IF; - - IF eql_v2.has_ore_block_u64_8_256(a) THEN - b_term := eql_v2.ore_block_u64_8_256(b); - END IF; - - IF a_term IS NULL AND b_term IS NULL THEN - RETURN 0; - END IF; - - IF a_term IS NULL THEN - RETURN -1; - END IF; - - IF b_term IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a_term.terms, b_term.terms); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Less-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for less-than ---! testing. The `<` operator wrappers no longer call this helper — they ---! inline a direct `ore_block_u64_8_256` comparison instead (see the ---! inlinable bodies below). ---! ---! @warning Behaviour now diverges from the `<` operator: this helper ---! still walks `eql_v2.compare`'s priority list (ore_block → ore_cllw ---! → hm), whereas `<` goes straight to `ore_block_u64_8_256` and raises ---! on missing `ob`. Callers relying on the dispatcher fallback should ---! migrate to the extractor form: `eql_v2.ore_cllw(col) < ---! eql_v2.ore_cllw($1::jsonb)`. See U-005. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a < b (compare result = -1) ---! ---! @see eql_v2.compare ---! @see eql_v2."<" -CREATE FUNCTION eql_v2.lt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = -1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than operator for encrypted values ---! ---! Implements the < operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term (configured ---! via the `ore` index in the EQL schema). ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is less than b ---! ---! @example ---! -- Range query on encrypted timestamps ---! SELECT * FROM events ---! WHERE encrypted_timestamp < '2024-01-01'::timestamp::text::eql_v2_encrypted; ---! ---! -- Compare encrypted numeric columns ---! SELECT * FROM products WHERE encrypted_price < encrypted_discount_price; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col < val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) < eql_v2.ore_block_u64_8_256(val)` --- and matches a functional btree index built on --- `eql_v2.ore_block_u64_8_256(col)` (using the DEFAULT --- `eql_v2.ore_block_u64_8_256_operator_class`). Bare range queries --- (`WHERE col < $1`) engage the functional ORE index on Supabase and any --- install that doesn't ship `eql_v2.encrypted_operator_class`. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2."<"` walked `eql_v2.compare`, which dispatched through --- ore_block / ore_cllw_u64 / ore_cllw_var / ope. Now `<` requires the --- column to have `ore_block_u64_8_256` configured (i.e. carry an `ob` --- field). Calling `<` on a column with only `ore_cllw_*` or OPE terms --- now raises from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where it --- previously returned a Boolean. Loud failure surfaces config errors --- rather than silently producing zero rows — see U-005. -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for encrypted value and JSONB ---! ---! Overload of < operator accepting JSONB on the right side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides; the jsonb ---! extractor `eql_v2.ore_block_u64_8_256(jsonb)` reads `b->'ob'` directly. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE encrypted_age < '{"ob":[...]}'::jsonb; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than operator for JSONB and encrypted value ---! ---! Overload of < operator accepting JSONB on the left side. Reduces to a ---! direct comparison of the `ob` ORE term on both sides. ---! ---! @param a JSONB Left operand ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a < b ---! ---! @example ---! SELECT * FROM events WHERE '{"ob":[...]}'::jsonb < encrypted_date; ---! ---! @see eql_v2."<"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) < eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <( - FUNCTION=eql_v2."<", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - ---! @brief Less-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `<=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `<=` testing. ---! The `<=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `<=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `<=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a <= b (compare result <= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2."<=" -CREATE FUNCTION eql_v2.lte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) <= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Less-than-or-equal operator for encrypted values ---! ---! Implements the <= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a <= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age <= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for encrypted value and JSONB ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief <= operator for JSONB and encrypted value ---! @see eql_v2."<="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) <= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR <=( - FUNCTION = eql_v2."<=", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - ---! @brief Equality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `=` operator's body: reduces to ---! `hmac_256(a) = hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.eq`); equivalent to using the `=` operator ---! directly. ---! ---! Equality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `=` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms match ---! ---! @see eql_v2."=" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.eq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - ---! @brief Equality operator for encrypted values ---! ---! Implements the = operator for comparing two encrypted values using their ---! encrypted index terms (hmac_256). Enables WHERE clause comparisons ---! without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are equal ---! ---! @example ---! -- Compare encrypted columns ---! SELECT * FROM users WHERE encrypted_email = other_encrypted_email; ---! ---! -- Search using encrypted literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2.add_search_config --- Inlinable: `LANGUAGE sql IMMUTABLE` with a single SELECT body and no --- `SET` clause. The Postgres planner inlines the body into the calling --- query during planning, so `WHERE col = val` reduces to --- `WHERE eql_v2.hmac_256(col) = eql_v2.hmac_256(val)` and matches a --- functional hash index built on `eql_v2.hmac_256(col)`. Bare equality --- queries (including those issued by PostgREST and ORMs that don't --- wrap columns themselves) become fast on Supabase and any --- --exclude-operator-family install. --- --- Behaviour change vs the previous dispatcher-based impl: the old --- `eql_v2.eq` walked `eql_v2.compare`, which fell back to ORE / Blake3 / --- literal comparison when HMAC wasn't present. Now `=` requires the --- column to have `equality` configured (i.e. carry an `hm` field). --- Calling `=` on an ORE-only column will return NULL where it --- previously returned a Boolean. This is intentional — it surfaces --- config errors loudly. See the predicate/extractor RFC for context. -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - ---! @brief Equality operator for encrypted value and JSONB ---! ---! Overload of = operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Useful for comparing ---! against JSONB literals or columns. ---! ---! @param eql_v2_encrypted Left operand (encrypted value) ---! @param b JSONB Right operand (will be cast to eql_v2_encrypted) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare encrypted column to JSONB literal ---! SELECT * FROM users ---! WHERE encrypted_email = '{"c":"...","i":{"unique":"..."}}'::jsonb; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) = eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Equality operator for JSONB and encrypted value ---! ---! Overload of = operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for comparison. Enables commutative ---! equality comparisons. ---! ---! @param a JSONB Left operand (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if values are equal ---! ---! @example ---! -- Compare JSONB literal to encrypted column ---! SELECT * FROM users ---! WHERE '{"c":"...","i":{"unique":"..."}}'::jsonb = encrypted_email; ---! ---! @see eql_v2."="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) = eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR = ( - FUNCTION=eql_v2."=", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - ---! @brief Greater-than-or-equal comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>=` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for `>=` testing. ---! The `>=` operator wrappers no longer go through this helper — see the ---! inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>=` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>=` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a >= b (compare result >= 0) ---! ---! @see eql_v2.compare ---! @see eql_v2.">=" -CREATE FUNCTION eql_v2.gte(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) >= 0; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than-or-equal operator for encrypted values ---! ---! Implements the >= operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Requires the column to carry an ---! `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a >= b ---! ---! @example ---! SELECT * FROM users WHERE encrypted_age >= '18'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = eql_v2_encrypted, - RIGHTARG=jsonb, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief >= operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a >= b ---! @see eql_v2.">="(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">="(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) >= eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >=( - FUNCTION = eql_v2.">=", - LEFTARG = jsonb, - RIGHTARG =eql_v2_encrypted, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @brief Greater-than comparison helper for encrypted values ---! @internal ---! @deprecated Slated for removal in EQL 3.0. Use the `>` operator instead. ---! ---! Internal helper that delegates to `eql_v2.compare` for greater-than ---! testing. The `>` operator wrappers no longer go through this helper — ---! see the inlinable bodies below. ---! ---! @warning Behaviour now diverges from the `>` operator: this helper ---! still walks `eql_v2.compare`'s priority list, whereas `>` goes ---! straight to `ore_block_u64_8_256` and raises on missing `ob`. See ---! the matching note on `eql_v2.lt` and U-005 for migration guidance. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if a > b (compare result = 1) ---! ---! @see eql_v2.compare ---! @see eql_v2.">" -CREATE FUNCTION eql_v2.gt(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2.compare(a, b) = 1; - END; -$$ LANGUAGE plpgsql; - ---! @brief Greater-than operator for encrypted values ---! ---! Implements the > operator for comparing two encrypted values via their ---! `ob` (ore_block_u64_8_256) ORE term. Enables range queries and sorting ---! without decryption. Requires the column to carry an `ob` term. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if a is greater than b ---! ---! @example ---! SELECT * FROM events ---! WHERE encrypted_value > '100'::int::text::eql_v2_encrypted; ---! ---! @see eql_v2.ore_block_u64_8_256 ---! @see eql_v2.add_search_config --- Inlinable: see `src/operators/<.sql` for the rationale. Predicate --- `WHERE col > val` reduces to --- `WHERE eql_v2.ore_block_u64_8_256(col) > eql_v2.ore_block_u64_8_256(val)` --- and matches a functional ORE index built on the same expression. --- Breaking impact: columns with only `ore_cllw_*` or OPE terms now --- raise from the `ore_block_u64_8_256(jsonb)` extractor --- (`Expected an ore index (ob) value in json: ...`) where they --- previously fell through `eql_v2.compare`. See U-005. -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION=eql_v2.">", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for encrypted value and JSONB ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b jsonb Right operand ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = eql_v2_encrypted, - RIGHTARG = jsonb, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief > operator for JSONB and encrypted value ---! @param a jsonb Left operand ---! @param b eql_v2_encrypted Right operand (encrypted value) ---! @return Boolean True if a > b ---! @see eql_v2.">"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2.">"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_block_u64_8_256(a) > eql_v2.ore_block_u64_8_256(b) -$$; - - -CREATE OPERATOR >( - FUNCTION = eql_v2.">", - LEFTARG = jsonb, - RIGHTARG = eql_v2_encrypted, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - ---! @brief Compute hash integer for encrypted value ---! ---! Produces a 32-bit integer hash suitable for PostgreSQL hash joins, GROUP BY, ---! DISTINCT, and hash aggregate operations. Used by the `eql_v2_encrypted` hash ---! operator class (`FUNCTION 1`). Inlinable single-statement SQL — the SQL ---! function machinery is much cheaper per row than plpgsql, which matters ---! because HashAggregate / hash-join call this once per input row. ---! ---! Returns `hashtext` of the root payload's `hm` term. This is the canonical ---! bucket for equality groups, since `=` on `eql_v2_encrypted` reduces to ---! `hmac_256(a) = hmac_256(b)` post-#193. ---! ---! @par Contract ---! Callers using `GROUP BY` / `DISTINCT` / hash joins on `eql_v2_encrypted` ---! MUST configure the column with a `unique` index so the crypto layer ---! emits `hm` — `hm` is assumed present. A missing `hm` is a misconfiguration ---! that surfaces upstream via [U-002](docs/upgrading/v2.3.md#u-002-equality-and-hashing-require-hmac). ---! ---! @param val eql_v2_encrypted Encrypted value to hash ---! @return integer 32-bit hash value derived from `hm` ---! ---! @note For grouping a value extracted from an encrypted JSON document, use ---! the field-level recipe directly: `GROUP BY eql_v2.eq_term(col -> '<selector>')` ---! (covers both hm-bearing and oc-bearing selectors via the XOR-aware ---! extractor — see `src/ste_vec/eq_term.sql`). That bypasses ---! `hash_encrypted` entirely. ---! ---! @see eql_v2.hmac_256 ---! @see eql_v2.has_hmac_256 ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.hash_encrypted(val eql_v2_encrypted) - RETURNS integer - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT pg_catalog.hashtext(eql_v2.hmac_256(val)::text) -$$; - ---! @brief Contains operator for encrypted values (@>) ---! ---! Implements the @> (contains) operator for testing if left encrypted value ---! contains the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2_encrypted Right operand (contained value) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Check if encrypted array contains value ---! SELECT * FROM documents ---! WHERE encrypted_tags @> '["security"]'::jsonb::eql_v2_encrypted; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.add_search_config --- Marked IMMUTABLE STRICT PARALLEL SAFE so the planner inlines the body --- and a functional GIN index on `eql_v2.ste_vec(col)` can match --- `WHERE col @> val`. The previous default-VOLATILE declaration prevented --- inlining and forced seq scan even on Supabase installs that have the --- ste_vec functional index in place. -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ste_vec_contains(a, b) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contains operator (@>) with an `eql_v2.stevec_query` needle ---! ---! Type-safe containment for the recommended recipe: the right-hand ---! side is an `stevec_query` (sv-shaped payload, no `c` fields). The ---! body inlines to a native `jsonb @>` over `eql_v2.to_stevec_query(a)::jsonb`, ---! so the planner can match a functional GIN index built on the same ---! expression — engaging Bitmap Index Scan for bare-form containment ---! across both `hm`-bearing and `oc`-bearing selectors with a single ---! index. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.stevec_query Right operand (query payload) ---! @return Boolean True if a contains b ---! ---! @example ---! -- Functional GIN index (covers all selectors, hm and oc): ---! CREATE INDEX ON users USING gin ( ---! eql_v2.to_stevec_query(encrypted_doc)::jsonb jsonb_path_ops ---! ); ---! -- Bare-form predicate engages the index: ---! SELECT * FROM users ---! WHERE encrypted_doc @> '{"sv":[{"s":"<sel>","hm":"<hm>"}]}'::eql_v2.stevec_query; ---! ---! @see eql_v2.stevec_query ---! @see eql_v2.to_stevec_query -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.stevec_query) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Single-expression body so the planner can inline. The haystack - -- normalisation happens in `to_stevec_query`; the needle is trusted - -- to be clean (sv elements of shape `{s, hm-or-oc}` — the documented - -- stevec_query contract). For untrusted needles, callers should - -- normalise via the json-shape `{"sv":[{"s":"<sel>","hm":"<term>"}]}`. - SELECT eql_v2.to_stevec_query(a)::jsonb @> b::jsonb -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.stevec_query -); - - ---! @brief Contains operator (@>) with an `eql_v2.ste_vec_entry` needle ---! ---! Convenience overload for the common pattern "does this encrypted ---! payload include this specific sv entry?". Wraps the entry into a ---! single-element sv array (stripping `c`) and reduces to the same ---! `to_stevec_query(a)::jsonb @> needle::jsonb` form as the ---! `stevec_query` overload — so it engages the same functional GIN ---! index. Inlinable. ---! ---! @param a eql_v2_encrypted Left operand (container) ---! @param b eql_v2.ste_vec_entry Right operand (single entry) ---! @return Boolean True if a contains an sv entry matching `b` ---! ---! @example ---! -- Does this row's encrypted doc contain the same name as this other doc? ---! SELECT a.* FROM docs a, docs b ---! WHERE a.doc @> (b.doc -> '<name-sel>'); ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."@>"(a eql_v2_encrypted, b eql_v2.ste_vec_entry) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.to_stevec_query(a)::jsonb - @> jsonb_build_object( - 'sv', - jsonb_build_array( - jsonb_strip_nulls( - jsonb_build_object( - 's', b -> 's', - 'hm', b -> 'hm', - 'oc', b -> 'oc' - ) - ) - ) - ) -$$; - -CREATE OPERATOR @>( - FUNCTION=eql_v2."@>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2.ste_vec_entry -); - ---! @file config/tables.sql ---! @brief Encryption configuration storage table ---! ---! Defines the main table for storing EQL v2 encryption configurations. ---! Each row represents a configuration specifying which tables/columns to encrypt ---! and what index types to use. Configurations progress through lifecycle states. ---! ---! @see config/types.sql for state ENUM definition ---! @see config/indexes.sql for state uniqueness constraints ---! @see config/constraints.sql for data validation - - ---! @brief Encryption configuration table ---! ---! Stores encryption configurations with their state and metadata. ---! The 'data' JSONB column contains the full configuration structure including ---! table/column mappings, index types, and casting rules. ---! ---! @note Only one configuration can be 'active', 'pending', or 'encrypting' at once ---! @note 'id' is auto-generated identity column ---! @note 'state' defaults to 'pending' for new configurations ---! @note 'data' validated by CHECK constraint (see config/constraints.sql) -CREATE TABLE IF NOT EXISTS public.eql_v2_configuration -( - id bigint GENERATED ALWAYS AS IDENTITY, - state eql_v2_configuration_state NOT NULL DEFAULT 'pending', - data jsonb, - created_at timestamptz not null default current_timestamp, - PRIMARY KEY(id) -); - - ---! @brief Initialize default configuration structure ---! @internal ---! ---! Creates a default configuration object if input is NULL. Used internally ---! by public configuration functions to ensure consistent structure. ---! ---! @param config JSONB Existing configuration or NULL ---! @return JSONB Configuration with default structure (version 1, empty tables) -CREATE FUNCTION eql_v2.config_default(config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF config IS NULL THEN - SELECT jsonb_build_object('v', 1, 'tables', jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add table to configuration if not present ---! @internal ---! ---! Ensures the specified table exists in the configuration structure. ---! Creates empty table entry if needed. Idempotent operation. ---! ---! @param table_name Text Name of table to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with table entry -CREATE FUNCTION eql_v2.config_add_table(table_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - tbl jsonb; - BEGIN - IF NOT config #> array['tables'] ? table_name THEN - SELECT jsonb_insert(config, array['tables', table_name], jsonb_build_object()) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add column to table configuration if not present ---! @internal ---! ---! Ensures the specified column exists in the table's configuration structure. ---! Creates empty column entry with indexes object if needed. Idempotent operation. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column to add ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with column entry -CREATE FUNCTION eql_v2.config_add_column(table_name text, column_name text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - col jsonb; - BEGIN - IF NOT config #> array['tables', table_name] ? column_name THEN - SELECT jsonb_build_object('indexes', jsonb_build_object()) into col; - SELECT jsonb_set(config, array['tables', table_name, column_name], col) INTO config; - END IF; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Set cast type for column in configuration ---! @internal ---! ---! Updates the cast_as field for a column, specifying the PostgreSQL type ---! that decrypted values should be cast to. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param cast_as Text PostgreSQL type for casting (e.g., 'text', 'int', 'jsonb') ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with cast_as set -CREATE FUNCTION eql_v2.config_add_cast(table_name text, column_name text, cast_as text, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_set(config, array['tables', table_name, column_name, 'cast_as'], to_jsonb(cast_as)) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Add search index to column configuration ---! @internal ---! ---! Inserts a search index entry (unique, match, ore, ste_vec) with its options ---! into the column's indexes object. ---! ---! @param table_name Text Name of parent table ---! @param column_name Text Name of column ---! @param index_name Text Type of index to add ---! @param opts JSONB Index-specific options ---! @param config JSONB Configuration object ---! @return JSONB Updated configuration with index added -CREATE FUNCTION eql_v2.config_add_index(table_name text, column_name text, index_name text, opts jsonb, config jsonb) - RETURNS jsonb - IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - SELECT jsonb_insert(config, array['tables', table_name, column_name, 'indexes', index_name], opts) INTO config; - RETURN config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Generate default options for match index ---! @internal ---! ---! Returns default configuration for match (LIKE) indexes: k=6, bf=2048, ---! ngram tokenizer with token_length=3, downcase filter, include_original=true. ---! ---! @return JSONB Default match index options -CREATE FUNCTION eql_v2.config_match_default() - RETURNS jsonb -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_build_object( - 'k', 6, - 'bf', 2048, - 'include_original', true, - 'tokenizer', json_build_object('kind', 'ngram', 'token_length', 3), - 'token_filters', json_build_array(json_build_object('kind', 'downcase'))); -END; --- AUTOMATICALLY GENERATED FILE --- Source is version-template.sql - -DROP FUNCTION IF EXISTS eql_v2.version(); - ---! @file version.sql ---! @brief EQL version reporting ---! ---! This file is auto-generated from version.template during build. ---! The version string placeholder is replaced with the actual release version. - ---! @brief Get EQL library version string ---! ---! Returns the version string for the installed EQL library. ---! This value is set at build time from the project version. ---! ---! @return text Version string (e.g., "2.1.0" or "DEV" for development builds) ---! ---! @note Auto-generated during build from version.template ---! ---! @example ---! -- Check installed EQL version ---! SELECT eql_v2.version(); ---! -- Returns: '2.1.0' -CREATE FUNCTION eql_v2.version() - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT 'eql-2.3.1'; -$$ LANGUAGE SQL; - - ---! @file src/ore_cllw/operator_class.sql ---! @brief Btree operator class on the `eql_v2.ore_cllw` composite type ---! ---! Registers the CLLW per-byte comparison operators as a btree opclass for ---! the `eql_v2.ore_cllw` composite type. With `DEFAULT FOR TYPE`, a functional ---! btree index on `eql_v2.ore_cllw(col)` (or any expression returning the ---! composite) automatically picks up this opclass — no annotation needed at ---! index creation time. ---! ---! Why this matters. After the consolidation in #219, ordered comparison on ---! sv-element values (via `eql_v2.ore_cllw(value -> '<selector>'::text)`) ---! has correct semantics through the operator backing functions (each ---! reduces to `compare_ore_cllw_term <op> 0`), but PostgreSQL won't engage ---! a functional index for `ORDER BY ...` or `WHERE ... < $1` unless the ---! type has a registered btree opclass that the planner can structurally ---! match. Without this opclass, `field_order/*` queries on sv-element CLLW ---! columns fall back to seq scan + Top-N sort (measured 20s+ on 1M rows). ---! With it, the same queries become Index Scan + LIMIT — milliseconds. ---! ---! FUNCTION 1 is the three-way comparator that btree's internal sort uses ---! (returns -1 / 0 / +1). We point it at `compare_ore_cllw_term` directly: ---! that's plpgsql by design (the per-byte CLLW protocol needs iteration), ---! and btree calls it once per index entry pair during build / search — ---! not per-row in the outer query. ---! ---! @note Deliberately no operator family registration beyond the opclass ---! itself: no cross-type operators on `eql_v2.ore_cllw` × `jsonb`, no ---! hash support — see operators.sql for the rationale. ---! @note Excluded from the Supabase build variant (the build glob ---! `**/*operator_class.sql` strips operator classes for Supabase ---! compatibility). ---! ---! @see src/ore_cllw/operators.sql ---! @see src/ore_cllw/functions.sql - -CREATE OPERATOR FAMILY eql_v2.ore_cllw_ops USING btree; - -CREATE OPERATOR CLASS eql_v2.ore_cllw_ops - DEFAULT FOR TYPE eql_v2.ore_cllw - USING btree FAMILY eql_v2.ore_cllw_ops AS - OPERATOR 1 < (eql_v2.ore_cllw, eql_v2.ore_cllw), - OPERATOR 2 <= (eql_v2.ore_cllw, eql_v2.ore_cllw), - OPERATOR 3 = (eql_v2.ore_cllw, eql_v2.ore_cllw), - OPERATOR 4 >= (eql_v2.ore_cllw, eql_v2.ore_cllw), - OPERATOR 5 > (eql_v2.ore_cllw, eql_v2.ore_cllw), - FUNCTION 1 eql_v2.compare_ore_cllw_term(eql_v2.ore_cllw, eql_v2.ore_cllw); - - ---! @brief B-tree operator family for ORE block types ---! ---! Defines the operator family for creating B-tree indexes on ORE block types. ---! ---! @see eql_v2.ore_block_u64_8_256_operator_class -CREATE OPERATOR FAMILY eql_v2.ore_block_u64_8_256_operator_family USING btree; - ---! @brief B-tree operator class for ORE block encrypted values ---! ---! Defines the operator class required for creating B-tree indexes on columns ---! using the ore_block_u64_8_256 type. Enables range queries and ORDER BY on ---! ORE-encrypted data without decryption. ---! ---! Supports operators: <, <=, =, >=, > ---! Uses comparison function: compare_ore_block_u64_8_256_terms ---! ---! ---! @example ---! -- Would be used like (if enabled): ---! CREATE INDEX ON events USING btree ( ---! (encrypted_timestamp::jsonb->'ob')::eql_v2.ore_block_u64_8_256 ---! ); ---! ---! @see CREATE OPERATOR CLASS in PostgreSQL documentation ---! @see eql_v2.compare_ore_block_u64_8_256_terms -CREATE OPERATOR CLASS eql_v2.ore_block_u64_8_256_operator_class DEFAULT FOR TYPE eql_v2.ore_block_u64_8_256 USING btree FAMILY eql_v2.ore_block_u64_8_256_operator_family AS - OPERATOR 1 <, - OPERATOR 2 <=, - OPERATOR 3 =, - OPERATOR 4 >=, - OPERATOR 5 >, - FUNCTION 1 eql_v2.compare_ore_block_u64_8_256_terms(a eql_v2.ore_block_u64_8_256, b eql_v2.ore_block_u64_8_256); - ---! @brief Cast text to ORE block term ---! @internal ---! ---! Converts text to bytea and wraps in ore_block_u64_8_256_term type. ---! Used internally for ORE block extraction and manipulation. ---! ---! @param t Text Text value to convert ---! @return eql_v2.ore_block_u64_8_256_term ORE term containing bytea representation ---! ---! @see eql_v2.ore_block_u64_8_256_term -CREATE FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(t text) - RETURNS eql_v2.ore_block_u64_8_256_term - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN t::bytea; -END; - ---! @brief Implicit cast from text to ORE block term ---! ---! Defines an implicit cast allowing automatic conversion of text values ---! to ore_block_u64_8_256_term type for ORE operations. ---! ---! @see eql_v2.text_to_ore_block_u64_8_256_term -CREATE CAST (text AS eql_v2.ore_block_u64_8_256_term) - WITH FUNCTION eql_v2.text_to_ore_block_u64_8_256_term(text) AS IMPLICIT; - ---! @brief Pattern matching helper using bloom filters ---! @internal ---! ---! Internal helper for LIKE-style pattern matching on encrypted values. ---! Uses bloom filter index terms to test substring containment without decryption. ---! Requires 'match' index configuration on the column. ---! ---! Marked IMMUTABLE so the planner inlines the body and a functional index on ---! `eql_v2.bloom_filter(col)` can match `WHERE eql_v2.like(col, val)`. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @see eql_v2."~~" ---! @see eql_v2.bloom_filter ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.like(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief Case-insensitive pattern matching helper ---! @internal ---! ---! Internal helper for ILIKE-style case-insensitive pattern matching. ---! Case sensitivity is controlled by index configuration (token_filters with downcase). ---! This function has same implementation as like() - actual case handling is in index terms. ---! ---! @param a eql_v2_encrypted Haystack (value to search in) ---! @param b eql_v2_encrypted Needle (pattern to search for) ---! @return Boolean True if bloom filter of a contains bloom filter of b ---! ---! @note Case sensitivity depends on match index token_filters configuration ---! @see eql_v2."~~" ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.ilike(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL -IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b); -$$; - ---! @brief LIKE operator for encrypted values (pattern matching) ---! ---! Implements the ~~ (LIKE) operator for substring/pattern matching on encrypted ---! text using bloom filter index terms. Enables WHERE col LIKE '%pattern%' queries ---! without decryption. Requires 'match' index configuration on the column. ---! ---! Pattern matching uses n-gram tokenization configured in match index. Token length ---! and filters affect matching behavior. ---! ---! @param a eql_v2_encrypted Haystack (encrypted text to search in) ---! @param b eql_v2_encrypted Needle (encrypted pattern to search for) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! -- Search for substring in encrypted email ---! SELECT * FROM users ---! WHERE encrypted_email ~~ '%@example.com%'::text::eql_v2_encrypted; ---! ---! -- Pattern matching on encrypted names ---! SELECT * FROM customers ---! WHERE encrypted_name ~~ 'John%'::text::eql_v2_encrypted; ---! ---! @brief SQL LIKE operator (~~ operator) for encrypted text pattern matching ---! ---! @param a eql_v2_encrypted Left operand (encrypted value) ---! @param b eql_v2_encrypted Right operand (encrypted pattern) ---! @return boolean True if pattern matches ---! ---! @note Requires match index: eql_v2.add_search_config(table, column, 'match') ---! @see eql_v2.like ---! @see eql_v2.add_search_config --- Inlinable: delegates to `eql_v2.like` which is itself an inlinable --- single-statement SQL function. Two levels of inlining produce --- `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)`, which matches a --- functional GIN index built on `eql_v2.bloom_filter(col)`. PostgREST --- and ORM `~~`/`~~*` queries engage the bloom-filter index without --- the caller wrapping the column themselves. -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b) -$$; - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief Case-insensitive LIKE operator (~~*) ---! ---! Implements ~~* (ILIKE) operator for case-insensitive pattern matching. ---! Case handling depends on match index token_filters configuration (use downcase filter). ---! Same implementation as ~~, with case sensitivity controlled by index configuration. ---! ---! @param a eql_v2_encrypted Haystack ---! @param b eql_v2_encrypted Needle ---! @return Boolean True if a contains b (case-insensitive) ---! ---! @note Configure match index with downcase token filter for case-insensitivity ---! @see eql_v2."~~" -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for encrypted value and JSONB ---! ---! Overload of ~~ operator accepting JSONB on the right side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param eql_v2_encrypted Haystack (encrypted value) ---! @param b JSONB Needle (will be cast to eql_v2_encrypted) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE encrypted_email ~~ '%gmail%'::jsonb; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a, b::eql_v2_encrypted) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief LIKE operator for JSONB and encrypted value ---! ---! Overload of ~~ operator accepting JSONB on the left side. Automatically ---! casts JSONB to eql_v2_encrypted for bloom filter pattern matching. ---! ---! @param a JSONB Haystack (will be cast to eql_v2_encrypted) ---! @param eql_v2_encrypted Needle (encrypted pattern) ---! @return Boolean True if a contains b as substring ---! ---! @example ---! SELECT * FROM users WHERE 'test@example.com'::jsonb ~~ encrypted_pattern; ---! ---! @see eql_v2."~~"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."~~"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.like(a::eql_v2_encrypted, b) -$$; - - -CREATE OPERATOR ~~( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - -CREATE OPERATOR ~~*( - FUNCTION=eql_v2."~~", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - --- ----------------------------------------------------------------------------- - ---! @file src/operators/ste_vec_entry.sql ---! @brief Comparison operators on `eql_v2.ste_vec_entry` ---! ---! Equality (`=`, `<>`) reduces to `eq_term(a) = eq_term(b)` — a bytea ---! comparison of `coalesce(hm, oc)`. Ordering (`<`, `<=`, `>`, `>=`) ---! reduces to `ore_cllw(a) <op> ore_cllw(b)`. Each backing function is ---! inlinable single-statement SQL, so the planner can fold the ---! operator body into the calling query — `WHERE col -> 'sel' = $1` ---! and `WHERE col -> 'sel' < $1` therefore match functional indexes ---! built on `eql_v2.eq_term(col -> 'sel')` / ---! `eql_v2.ore_cllw(col -> 'sel')` without per-query rewriting. ---! ---! XOR contract. Each sv entry carries exactly one of `hm` (bool ---! leaves, array / object roots) or `oc` (string / number leaves) — ---! enforced by the `ste_vec_entry` DOMAIN CHECK. Equality coalesces ---! across both protocols because both are deterministic and the byte ---! distributions are disjoint; ordering strictly uses `ore_cllw` ---! (range on hm-only entries is meaningless and produces silent NULL, ---! which the lint subsystem `src/lint/lints.sql` flags as a ---! configuration error). ---! ---! Same convention as the `eql_v2_encrypted` operators (#193 / #211): the ---! operator-class function-matching layer is what makes index match work ---! structurally, the backing functions just need to inline cleanly through ---! to the extractor calls. ---! ---! @see eql_v2.eq_term(eql_v2.ste_vec_entry) ---! @see eql_v2.ore_cllw(eql_v2.ste_vec_entry) ---! @see src/operators/=.sql ---! @see src/operators/<.sql - ---! @brief Equality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if both entries share the same deterministic ---! equality term (hm-or-oc, via `eq_term`). -CREATE FUNCTION eql_v2.eq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) = eql_v2.eq_term(b) -$$; - -CREATE OPERATOR = ( - FUNCTION = eql_v2.eq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = =, - NEGATOR = <>, - RESTRICT = eqsel, - JOIN = eqjoinsel, - HASHES, - MERGES -); - - ---! @brief Inequality backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if the entries' equality terms (hm-or-oc, via ---! `eq_term`) differ. -CREATE FUNCTION eql_v2.neq(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.eq_term(a) <> eql_v2.eq_term(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION = eql_v2.neq, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <>, - NEGATOR = =, - RESTRICT = neqsel, - JOIN = neqjoinsel -); - - ---! @brief Less-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before `b`'s -CREATE FUNCTION eql_v2.lt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) < eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR < ( - FUNCTION = eql_v2.lt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >, - NEGATOR = >=, - RESTRICT = scalarltsel, - JOIN = scalarltjoinsel -); - - ---! @brief Less-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts before or equal to `b`'s -CREATE FUNCTION eql_v2.lte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) <= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR <= ( - FUNCTION = eql_v2.lte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = >=, - NEGATOR = >, - RESTRICT = scalarlesel, - JOIN = scalarlejoinsel -); - - ---! @brief Greater-than backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after `b`'s -CREATE FUNCTION eql_v2.gt(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) > eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR > ( - FUNCTION = eql_v2.gt, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <, - NEGATOR = <=, - RESTRICT = scalargtsel, - JOIN = scalargtjoinsel -); - - ---! @brief Greater-than-or-equal backing function for `eql_v2.ste_vec_entry` ---! @internal ---! @param a eql_v2.ste_vec_entry Left operand ---! @param b eql_v2.ste_vec_entry Right operand ---! @return boolean True if `a`'s CLLW ORE term sorts after or equal to `b`'s -CREATE FUNCTION eql_v2.gte(a eql_v2.ste_vec_entry, b eql_v2.ste_vec_entry) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.ore_cllw(a) >= eql_v2.ore_cllw(b) -$$; - -CREATE OPERATOR >= ( - FUNCTION = eql_v2.gte, - LEFTARG = eql_v2.ste_vec_entry, - RIGHTARG = eql_v2.ste_vec_entry, - COMMUTATOR = <=, - NEGATOR = <, - RESTRICT = scalargesel, - JOIN = scalargejoinsel -); - ---! @file operators/sort.sql ---! @brief Comparison-based sorting functions for encrypted values without operator classes ---! ---! Provides O(n log n) quicksort-based sorting using eql_v2.compare() for environments ---! where btree operator classes are unavailable (e.g., Supabase). This is significantly ---! faster than the O(n^2) correlated subquery workaround. ---! ---! When all input rows share an ORE term (`ob`) the sort path pre-extracts the ---! ORE order key once per row and compares those keys directly. Rows lacking ---! an ORE term entirely fall back to `eql_v2.compare()` per pair. - - ---! @internal ---! @brief Compare pre-extracted ORE order keys with encrypted NULL semantics ---! ---! Mirrors eql_v2.compare() for NULL handling, then delegates to the ---! ore_block_u64_8_256 comparator when both keys are present. ---! ---! @param a eql_v2.ore_block_u64_8_256 First order key ---! @param b eql_v2.ore_block_u64_8_256 Second order key ---! @return integer -1 if a < b, 0 if a = b, 1 if a > b -CREATE FUNCTION eql_v2._compare_order_key( - a eql_v2.ore_block_u64_8_256, - b eql_v2.ore_block_u64_8_256 -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF a IS NULL AND b IS NULL THEN - RETURN 0; - END IF; - - IF a IS NULL THEN - RETURN -1; - END IF; - - IF b IS NULL THEN - RETURN 1; - END IF; - - RETURN eql_v2.compare_ore_block_u64_8_256_terms(a, b); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare two elements from aligned arrays using the selected sort strategy ---! ---! @param vals eql_v2_encrypted[] Encrypted values (used when strategy = 'compare') ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (strategy = 'ore') ---! @param left_idx integer Index of the left element ---! @param right_idx integer Index of the right element ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if left < right, 0 if equal, 1 if left > right -CREATE FUNCTION eql_v2._compare_sort_elements( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - left_idx integer, - right_idx integer, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[left_idx], ore_keys[right_idx]); - END IF; - - RETURN eql_v2.compare(vals[left_idx], vals[right_idx]); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Compare an array element against a captured pivot using the selected strategy ---! ---! @param vals eql_v2_encrypted[] Array of encrypted values ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys ---! @param idx integer Index of the element to compare ---! @param pivot_val eql_v2_encrypted Pivot encrypted value (strategy = 'compare') ---! @param pivot_ore_key eql_v2.ore_block_u64_8_256 Pivot ORE key (strategy = 'ore') ---! @param strategy text One of 'ore' or 'compare' ---! @return integer -1 if element < pivot, 0 if equal, 1 if element > pivot -CREATE FUNCTION eql_v2._compare_sort_pivot( - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - idx integer, - pivot_val eql_v2_encrypted, - pivot_ore_key eql_v2.ore_block_u64_8_256, - strategy text -) -RETURNS integer -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - IF strategy = 'ore' THEN - RETURN eql_v2._compare_order_key(ore_keys[idx], pivot_ore_key); - END IF; - - RETURN eql_v2.compare(vals[idx], pivot_val); -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place insertion sort on parallel id/value/key arrays ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Array of pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._insertion_sort( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - i integer; - j integer; - key_id bigint; - key_val eql_v2_encrypted; - sort_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - IF lo >= hi THEN - RETURN; - END IF; - - FOR i IN lo + 1..hi LOOP - key_id := ids[i]; - key_val := vals[i]; - sort_ore_key := ore_keys[i]; - j := i - 1; - - WHILE j >= lo LOOP - EXIT WHEN strategy = 'compare' - AND eql_v2.compare(vals[j], key_val) <= 0; - EXIT WHEN strategy = 'ore' - AND eql_v2._compare_order_key(ore_keys[j], sort_ore_key) <= 0; - - ids[j + 1] := ids[j]; - vals[j + 1] := vals[j]; - ore_keys[j + 1] := ore_keys[j]; - j := j - 1; - END LOOP; - - ids[j + 1] := key_id; - vals[j + 1] := key_val; - ore_keys[j + 1] := sort_ore_key; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief In-place quicksort on parallel id/value/key arrays ---! ---! Sorts aligned arrays simultaneously using Hoare partition with median-of-three pivot ---! selection. The median-of-three strategy avoids O(n^2) degradation on already-sorted ---! input, which is common with sequential test data. ---! ---! @param ids bigint[] Array of row identifiers (reordered in place) ---! @param vals eql_v2_encrypted[] Array of encrypted values to compare (reordered in place) ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (reordered in place) ---! @param lo integer Lower bound index (1-based, inclusive) ---! @param hi integer Upper bound index (1-based, inclusive) ---! @param strategy text One of 'ore' or 'compare' ---! ---! @return ids bigint[] Sorted array of row identifiers ---! @return vals eql_v2_encrypted[] Sorted array of encrypted values ---! @return ore_keys eql_v2.ore_block_u64_8_256[] Sorted array of pre-extracted ORE keys -CREATE FUNCTION eql_v2._quicksort_sorter( - INOUT ids bigint[], - INOUT vals eql_v2_encrypted[], - INOUT ore_keys eql_v2.ore_block_u64_8_256[], - lo integer, - hi integer, - strategy text -) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - insertion_threshold CONSTANT integer := 16; - pivot_val eql_v2_encrypted; - pivot_ore_key eql_v2.ore_block_u64_8_256; - mid integer; - i integer; - j integer; - left_hi integer; - right_lo integer; - tmp_id bigint; - tmp_val eql_v2_encrypted; - tmp_ore_key eql_v2.ore_block_u64_8_256; -BEGIN - WHILE lo < hi LOOP - IF hi - lo <= insertion_threshold THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._insertion_sort(ids, vals, ore_keys, lo, hi, strategy) q; - RETURN; - END IF; - - -- Median-of-three pivot selection: sort lo, mid, hi then use mid as pivot - mid := lo + (hi - lo) / 2; - - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, mid, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[mid]; ids[mid] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[mid]; vals[mid] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[mid]; ore_keys[mid] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, lo, hi, strategy) > 0 THEN - tmp_id := ids[lo]; ids[lo] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[lo]; vals[lo] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[lo]; ore_keys[lo] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - IF eql_v2._compare_sort_elements(vals, ore_keys, mid, hi, strategy) > 0 THEN - tmp_id := ids[mid]; ids[mid] := ids[hi]; ids[hi] := tmp_id; - tmp_val := vals[mid]; vals[mid] := vals[hi]; vals[hi] := tmp_val; - tmp_ore_key := ore_keys[mid]; ore_keys[mid] := ore_keys[hi]; ore_keys[hi] := tmp_ore_key; - END IF; - - pivot_val := vals[mid]; - pivot_ore_key := ore_keys[mid]; - i := lo; - j := hi; - - LOOP - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, i, - pivot_val, pivot_ore_key, strategy - ) < 0 LOOP - i := i + 1; - END LOOP; - WHILE eql_v2._compare_sort_pivot( - vals, ore_keys, j, - pivot_val, pivot_ore_key, strategy - ) > 0 LOOP - j := j - 1; - END LOOP; - - EXIT WHEN i >= j; - - tmp_id := ids[i]; ids[i] := ids[j]; ids[j] := tmp_id; - tmp_val := vals[i]; vals[i] := vals[j]; vals[j] := tmp_val; - tmp_ore_key := ore_keys[i]; ore_keys[i] := ore_keys[j]; ore_keys[j] := tmp_ore_key; - - i := i + 1; - j := j - 1; - END LOOP; - - left_hi := j; - right_lo := j + 1; - - IF left_hi - lo < hi - right_lo THEN - IF lo < left_hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, lo, left_hi, strategy) q; - END IF; - lo := right_lo; - ELSE - IF right_lo < hi THEN - SELECT q.ids, q.vals, q.ore_keys - INTO ids, vals, ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, right_lo, hi, strategy) q; - END IF; - hi := left_hi; - END IF; - END LOOP; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Emit aligned arrays as rows in ASC or DESC order ---! ---! @param ids bigint[] Array of sorted row identifiers ---! @param vals eql_v2_encrypted[] Array of sorted encrypted values ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Rows emitted in the requested order -CREATE FUNCTION eql_v2._emit_sorted_rows( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - i integer; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - IF upper(direction) = 'DESC' THEN - FOR i IN REVERSE n..1 LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - ELSE - FOR i IN 1..n LOOP - id := ids[i]; - val := vals[i]; - RETURN NEXT; - END LOOP; - END IF; -END; -$$ LANGUAGE plpgsql; - - ---! @internal ---! @brief Sort encrypted values using precomputed ORE keys when available ---! ---! Shared implementation for public sorting entrypoints. The `strategy` ---! parameter selects the comparison path: `'ore'` uses the aligned `ore_keys` ---! array; `'compare'` falls back to `eql_v2.compare()` on the encrypted values ---! directly. ---! ---! @param ids bigint[] Row identifiers aligned with `vals` ---! @param vals eql_v2_encrypted[] Encrypted values to sort ---! @param ore_keys eql_v2.ore_block_u64_8_256[] Pre-extracted ORE keys (used when strategy = 'ore') ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param strategy text One of 'ore' or 'compare' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows -CREATE FUNCTION eql_v2._sort_compare_precomputed( - ids bigint[], - vals eql_v2_encrypted[], - ore_keys eql_v2.ore_block_u64_8_256[], - direction text DEFAULT 'ASC', - strategy text DEFAULT 'ore' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - m integer; - k integer; - sorted_ids bigint[]; - sorted_vals eql_v2_encrypted[]; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; -BEGIN - n := coalesce(array_length(ids, 1), 0); - m := coalesce(array_length(vals, 1), 0); - - IF n <> m THEN - RAISE EXCEPTION 'ids and vals must have the same length'; - END IF; - - IF strategy = 'ore' THEN - k := coalesce(array_length(ore_keys, 1), 0); - IF n <> k THEN - RAISE EXCEPTION 'ids and ore_keys must have the same length when strategy = ''ore'''; - END IF; - END IF; - - IF n = 0 THEN - RETURN; - END IF; - - IF n = 1 THEN - id := ids[1]; - val := vals[1]; - RETURN NEXT; - RETURN; - END IF; - - SELECT q.ids, q.vals, q.ore_keys - INTO sorted_ids, sorted_vals, sorted_ore_keys - FROM eql_v2._quicksort_sorter(ids, vals, ore_keys, 1, n, strategy) q; - - RETURN QUERY - SELECT emitted.id, emitted.val - FROM eql_v2._emit_sorted_rows(sorted_ids, sorted_vals, direction) emitted; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values using comparison-based quicksort ---! ---! Sorts parallel arrays of identifiers and encrypted values using O(n log n) ---! quicksort with eql_v2.compare(). Returns sorted rows as a table, avoiding ---! the need for unnest() or other array manipulation by callers. ---! ---! When all input rows share an `ore` term the sort uses pre-extracted ORE ---! keys; otherwise it falls back to `eql_v2.compare()` per pair. ---! ---! This function is designed for environments without operator classes (e.g., Supabase) ---! where direct ORDER BY on encrypted columns is not available. ---! ---! @param ids bigint[] Array of row identifiers ---! @param vals eql_v2_encrypted[] Array of encrypted values (must be same length as ids) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @example ---! -- Sort all rows from an encrypted table ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore), ---! 'ASC' ---! ); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore WHERE id > 42), ---! (SELECT array_agg(e ORDER BY id) FROM ore WHERE id > 42), ---! 'DESC' ---! ); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare( ---! (SELECT array_agg(id ORDER BY id) FROM ore), ---! (SELECT array_agg(e ORDER BY id) FROM ore) ---! ) LIMIT 5; ---! ---! @see eql_v2.compare ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - ids bigint[], - vals eql_v2_encrypted[], - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - n integer; - sorted_ore_keys eql_v2.ore_block_u64_8_256[]; - i integer; - use_ore boolean := true; - strategy text; -BEGIN - n := coalesce(array_length(ids, 1), 0); - - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - FOR i IN 1..n LOOP - IF vals[i] IS NULL THEN - sorted_ore_keys[i] := NULL; - ELSE - IF use_ore THEN - IF eql_v2.has_ore_block_u64_8_256(vals[i]) THEN - sorted_ore_keys[i] := eql_v2.order_by(vals[i]); - ELSE - use_ore := false; - END IF; - END IF; - - EXIT WHEN NOT use_ore; - END IF; - END LOOP; - - IF use_ore THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - ids, vals, sorted_ore_keys, direction, strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a table using column and table references ---! ---! Convenience overload that accepts column names, a table name, and an optional ---! filter clause instead of pre-aggregated arrays. Internally constructs the ---! query and delegates to eql_v2.order_by_compare(). ---! ---! @param id_column text Name of the bigint identifier column ---! @param val_column text Name of the eql_v2_encrypted value column ---! @param tbl text Table name (may be schema-qualified) ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @param filter text Optional WHERE clause (without the WHERE keyword) ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note The id column must be castable to bigint. Uses dynamic SQL internally. ---! @warning The filter parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ascending (default) ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore'); ---! ---! -- Sort descending ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'DESC'); ---! ---! -- Sort with a filter ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore', 'ASC', 'id > 42'); ---! ---! -- Compose with LIMIT ---! SELECT * FROM eql_v2.sort_compare('id', 'e', 'ore') LIMIT 10; ---! ---! @see eql_v2.sort_compare(bigint[], eql_v2_encrypted[], text) ---! @see eql_v2.order_by_compare -CREATE FUNCTION eql_v2.sort_compare( - id_column text, - val_column text, - tbl text, - direction text DEFAULT 'ASC', - filter text DEFAULT NULL -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - query text; - resolved_tbl regclass; -BEGIN - resolved_tbl := to_regclass(tbl); - - IF resolved_tbl IS NULL THEN - RAISE EXCEPTION 'table "%" does not exist', tbl; - END IF; - - query := format('SELECT %I, %I FROM %s', id_column, val_column, resolved_tbl); - - IF filter IS NOT NULL THEN - query := query || ' WHERE ' || filter; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2.order_by_compare(query, direction) sc; -END; -$$ LANGUAGE plpgsql; - - ---! @brief Sort encrypted values from a query using comparison-based quicksort ---! ---! Convenience wrapper that accepts a SQL query string, executes it, collects the ---! results, and returns them sorted. For ORE-backed values this pre-extracts the ---! order key once per row and sorts on that key; other inputs fall back to ---! eql_v2.compare(). The query must return exactly two columns: a bigint ---! identifier and an eql_v2_encrypted value. ---! ---! @param query text SQL query returning (bigint, eql_v2_encrypted) columns ---! @param direction text Sort direction: 'ASC' (default) or 'DESC' ---! @return TABLE(id bigint, val eql_v2_encrypted) Sorted rows ---! ---! @note Uses dynamic SQL (EXECUTE) so cannot be IMMUTABLE or PARALLEL SAFE ---! @warning The query parameter is executed as dynamic SQL. Use only with trusted input. ---! ---! @example ---! -- Sort all rows ---! SELECT * FROM eql_v2.order_by_compare('SELECT id, e FROM ore'); ---! ---! -- Sort with WHERE clause ---! SELECT * FROM eql_v2.order_by_compare( ---! 'SELECT id, e FROM ore WHERE id > 42', ---! 'DESC' ---! ); ---! ---! @see eql_v2.sort_compare ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.order_by_compare( - query text, - direction text DEFAULT 'ASC' -) -RETURNS TABLE(id bigint, val eql_v2_encrypted) - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - all_ids bigint[]; - all_vals eql_v2_encrypted[]; - all_ore_keys eql_v2.ore_block_u64_8_256[]; - all_have_ore_keys boolean; - strategy text; -BEGIN - -- Pre-extract sort keys. ORE wins if every non-NULL row carries `ob`, - -- otherwise fall back to eql_v2.compare() per pair. - EXECUTE format( - 'WITH input_rows AS ( - SELECT row_number() OVER () AS ord, - sub.id, - sub.val, - CASE - WHEN sub.val IS NULL THEN NULL - WHEN eql_v2.has_ore_block_u64_8_256(sub.val) THEN eql_v2.order_by(sub.val) - ELSE NULL - END AS ore_key, - CASE - WHEN sub.val IS NULL THEN TRUE - ELSE eql_v2.has_ore_block_u64_8_256(sub.val) - END AS has_ore_key - FROM (%s) sub(id, val) - ) - SELECT array_agg(id ORDER BY ord), - array_agg(val ORDER BY ord), - array_agg(ore_key ORDER BY ord), - coalesce(bool_and(has_ore_key), TRUE) - FROM input_rows', - query - ) INTO all_ids, all_vals, all_ore_keys, all_have_ore_keys; - - IF all_ids IS NULL THEN - RETURN; - END IF; - - IF all_have_ore_keys THEN - strategy := 'ore'; - ELSE - strategy := 'compare'; - END IF; - - RETURN QUERY - SELECT sc.id, sc.val - FROM eql_v2._sort_compare_precomputed( - all_ids, - all_vals, - all_ore_keys, - direction, - strategy - ) sc; -END; -$$ LANGUAGE plpgsql; - ---! @file src/operators/operator_class.sql ---! @brief Btree operator class for the `eql_v2_encrypted` composite type ---! ---! `eql_v2_encrypted` is a composite type. PostgreSQL gives every composite ---! type an implicit row-wise btree comparison (`record_ops`) — but that ---! compares the raw ciphertext byte-for-byte, so two encryptions of the same ---! plaintext (same `hm`, different `c`) would sort and group as *distinct*. ---! `eql_v2.encrypted_operator_class` is registered `DEFAULT ... USING btree` ---! specifically to override `record_ops` with a comparison that is correct ---! for encrypted data: `GROUP BY`, `DISTINCT`, `ORDER BY`, sort-merge joins ---! and `ANALYZE` on a bare `eql_v2_encrypted` column all route through ---! FUNCTION 1 below. ---! ---! @note FUNCTION 1 is `eql_v2.encrypted_btree_compare`, NOT the strict ---! `eql_v2.compare`. A btree support function must be total and must ---! never raise — `ANALYZE` calls it to build column statistics on ---! every encrypted column. `eql_v2.compare` is deliberately strict ---! (it raises without a Block-ORE `ob` term — see U-005); it backs ---! the `<` / `>` range operators, not this opclass. ---! ---! @note Functional indexes are the canonical recipe for *building* indexes ---! on encrypted columns (see U-001 and docs/reference/database-indexes.md). ---! This opclass exists to keep the composite type's built-in ---! comparison correct — not as an index-building recommendation. ---! ---! @see eql_v2.encrypted_hash_operator_class (hash — GROUP BY / hash joins) ---! @see eql_v2.compare - --------------------- - ---! @brief Total, non-raising btree comparator for `eql_v2_encrypted` ---! ---! Three-way comparison (`-1` / `0` / `1`) used as FUNCTION 1 of ---! `eql_v2.encrypted_operator_class`. Unlike `eql_v2.compare`, it never ---! raises: a btree support function is invoked by `ANALYZE`, sort, and ---! `GROUP BY` on every value, so raising is not an option. ---! ---! Comparison priority: ---! 1. Both operands carry `ob` (Block ORE) — order-preserving comparison ---! via `eql_v2.compare_ore_block_u64_8_256`. ---! 2. Both operands carry `hm` (HMAC-256) — a total order on the hmac ---! bytes. Not order-preserving on plaintext (hmac is not), but ---! deterministic, total, and `= 0` exactly when the hmac terms match ---! — consistent with the `=` operator, so `GROUP BY` / `DISTINCT` ---! deduplicate correctly. ---! 3. Otherwise — a deterministic order on the raw payload. Reached only ---! for term-less / mixed payloads; present so the function stays total. ---! ---! @param a eql_v2_encrypted First value ---! @param b eql_v2_encrypted Second value ---! @return integer -1, 0, or 1 ---! ---! @internal ---! @see eql_v2.encrypted_operator_class ---! @see eql_v2.compare -CREATE FUNCTION eql_v2.encrypted_btree_compare(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - hm_a text; - hm_b text; - BEGIN - -- Block ORE on both sides: order-preserving comparison. - IF eql_v2.has_ore_block_u64_8_256(a) AND eql_v2.has_ore_block_u64_8_256(b) THEN - RETURN eql_v2.compare_ore_block_u64_8_256(a, b); - END IF; - - -- HMAC on both sides: total order on the hmac bytes. `= 0` iff the hmac - -- terms match, consistent with the `=` operator and the hash opclass. - hm_a := eql_v2.hmac_256(a)::text; - hm_b := eql_v2.hmac_256(b)::text; - IF hm_a IS NOT NULL AND hm_b IS NOT NULL THEN - RETURN CASE - WHEN hm_a < hm_b THEN -1 - WHEN hm_a > hm_b THEN 1 - ELSE 0 - END; - END IF; - - -- Fallback for term-less / mixed payloads: a deterministic, non-raising - -- total order on the raw payload. Not a normal column shape — this - -- branch only keeps the btree FUNCTION 1 contract (total, never raises). - RETURN CASE - WHEN (a).data::text < (b).data::text THEN -1 - WHEN (a).data::text > (b).data::text THEN 1 - ELSE 0 - END; - END; -$$ LANGUAGE plpgsql; - --------------------- - -CREATE OPERATOR FAMILY eql_v2.encrypted_operator_family USING btree; - -CREATE OPERATOR CLASS eql_v2.encrypted_operator_class DEFAULT FOR TYPE eql_v2_encrypted USING btree FAMILY eql_v2.encrypted_operator_family AS - OPERATOR 1 <, - OPERATOR 2 <=, - OPERATOR 3 =, - OPERATOR 4 >=, - OPERATOR 5 >, - FUNCTION 1 eql_v2.encrypted_btree_compare(a eql_v2_encrypted, b eql_v2_encrypted); - ---! @brief PostgreSQL hash operator class for encrypted value hashing ---! ---! Defines the hash operator family and operator class required for hash-based ---! operations on encrypted values. This enables PostgreSQL to use hash strategies for: ---! - Hash joins (cross-row equality via hash) ---! - GROUP BY (hash aggregation) ---! - DISTINCT (hash-based deduplication) ---! - UNION (hash-based set operations) ---! ---! Only the same-type equality operator (eql_v2_encrypted = eql_v2_encrypted) is ---! registered. Cross-type operators (encrypted/jsonb) are excluded because hash ---! joins require independent hashing of each side before comparison. ---! ---! @note Requires hmac_256 index terms for correct hashing ---! @see eql_v2.hash_encrypted ---! @see eql_v2.encrypted_operator_class (btree) - -CREATE OPERATOR FAMILY eql_v2.encrypted_hash_operator_family USING hash; - -CREATE OPERATOR CLASS eql_v2.encrypted_hash_operator_class - DEFAULT FOR TYPE eql_v2_encrypted USING hash - FAMILY eql_v2.encrypted_hash_operator_family AS - OPERATOR 1 = (eql_v2_encrypted, eql_v2_encrypted), - FUNCTION 1 eql_v2.hash_encrypted(eql_v2_encrypted); - ---! @brief Contained-by operator for encrypted values (<@) ---! ---! Implements the <@ (contained-by) operator for testing if left encrypted value ---! is contained by the right encrypted value. Uses ste_vec (secure tree encoding vector) ---! index terms for containment testing without decryption. Reverse of @> operator. ---! ---! Primarily used for encrypted array or set containment queries. ---! ---! @param a eql_v2_encrypted Left operand (contained value) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if a is contained by b ---! ---! @example ---! -- Check if value is contained in encrypted array ---! SELECT * FROM documents ---! WHERE '["security"]'::jsonb::eql_v2_encrypted <@ encrypted_tags; ---! ---! @note Requires ste_vec index configuration ---! @see eql_v2.ste_vec_contains ---! @see eql_v2.\"@>\" ---! @see eql_v2.add_search_config - --- Marked IMMUTABLE STRICT PARALLEL SAFE — see operators/@>.sql for rationale. -CREATE FUNCTION eql_v2."<@"(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - -- Contains with reversed arguments - SELECT eql_v2.ste_vec_contains(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.stevec_query` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.stevec_query)`. Mirrors the ---! typed needle convention: "is this query payload contained in that ---! encrypted document?". ---! ---! @param a eql_v2.stevec_query Left operand (query payload) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.stevec_query) -CREATE FUNCTION eql_v2."<@"(a eql_v2.stevec_query, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.stevec_query, - RIGHTARG=eql_v2_encrypted -); - - ---! @brief Contained-by operator (<@) with an `eql_v2.ste_vec_entry` LHS ---! ---! Reverse of `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)`. Convenience ---! shape for "is this entry contained in that encrypted document?". ---! ---! @param a eql_v2.ste_vec_entry Left operand (single entry) ---! @param b eql_v2_encrypted Right operand (container) ---! @return Boolean True if `b` contains `a` ---! @see eql_v2."@>"(eql_v2_encrypted, eql_v2.ste_vec_entry) -CREATE FUNCTION eql_v2."<@"(a eql_v2.ste_vec_entry, b eql_v2_encrypted) -RETURNS boolean -LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."@>"(b, a) -$$; - -CREATE OPERATOR <@( - FUNCTION=eql_v2."<@", - LEFTARG=eql_v2.ste_vec_entry, - RIGHTARG=eql_v2_encrypted -); - ---! @brief Inequality helper for encrypted values ---! @internal ---! ---! Inlinable SQL helper mirroring the `<>` operator's body: reduces to ---! `hmac_256(a) <> hmac_256(b)`. Kept for callers that invoked the ---! pre-#193 form (`eql_v2.neq`); equivalent to using the `<>` operator ---! directly. ---! ---! Inequality on `eql_v2_encrypted` is strictly hmac-based (see U-002). ---! Returns NULL when either side lacks an `hm` term — matching the ---! `<>` operator's behaviour. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return Boolean True if hmac terms differ ---! ---! @see eql_v2."<>" ---! @see eql_v2.hmac_256 -CREATE FUNCTION eql_v2.neq(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - ---! @brief Not-equal operator for encrypted values ---! ---! Implements the <> (not equal) operator for comparing encrypted values using their ---! encrypted index terms. Enables WHERE clause inequality comparisons without decryption. ---! ---! @param a eql_v2_encrypted Left operand ---! @param b eql_v2_encrypted Right operand ---! @return Boolean True if encrypted values are not equal ---! ---! @example ---! -- Find records with non-matching values ---! SELECT * FROM users ---! WHERE encrypted_email <> 'admin@example.com'::text::eql_v2_encrypted; ---! ---! @see eql_v2.compare ---! @see eql_v2."=" --- Inlinable; mirrors `=` (see operators/=.sql for rationale). --- Returns NULL on ORE-only encrypted columns (no `hm` field) instead --- of falling back to a slower comparison path; surface the config --- error rather than hide it. -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b) -$$; - - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for encrypted value and JSONB ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a eql_v2_encrypted, b jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a) <> eql_v2.hmac_256(b::eql_v2_encrypted) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=jsonb, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - ---! @brief <> operator for JSONB and encrypted value ---! ---! @param jsonb Plain JSONB value ---! @param eql_v2_encrypted Encrypted value ---! @return boolean True if values are not equal ---! ---! @see eql_v2."<>"(eql_v2_encrypted, eql_v2_encrypted) -CREATE FUNCTION eql_v2."<>"(a jsonb, b eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.hmac_256(a::eql_v2_encrypted) <> eql_v2.hmac_256(b) -$$; - -CREATE OPERATOR <> ( - FUNCTION=eql_v2."<>", - LEFTARG=jsonb, - RIGHTARG=eql_v2_encrypted, - NEGATOR = =, - RESTRICT = eqsel, - JOIN = eqjoinsel, - MERGES -); - - - - - ---! @brief JSONB field accessor operator alias (->>) ---! ---! Implements the ->> operator as an alias of -> for encrypted JSONB data. This mirrors ---! PostgreSQL semantics where ->> returns text via implicit casts. The underlying ---! implementation delegates to eql_v2."->" and allows PostgreSQL to coerce the result. ---! ---! Provides two overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! ---! @see eql_v2."->" ---! @see eql_v2.selector - ---! @brief ->> operator with text selector ---! @param eql_v2_encrypted Encrypted JSONB data ---! @param text Field name to extract ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @example ---! SELECT encrypted_json ->> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector text) - RETURNS text -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - found eql_v2_encrypted; - BEGIN - -- found = eql_v2."->"(e, selector); - -- RETURN eql_v2.ciphertext(found); - RETURN eql_v2."->"(e, selector); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - - - ---------------------------------------------------- - ---! @brief ->> operator with encrypted selector ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted field selector ---! @return text Encrypted value at selector, implicitly cast from eql_v2_encrypted ---! @see eql_v2."->>"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->>"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN eql_v2."->>"(e, eql_v2._selector(selector)); - END; -$$ LANGUAGE plpgsql; - - -CREATE OPERATOR ->> ( - FUNCTION=eql_v2."->>", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - ---! @brief JSONB field accessor operator for encrypted values (->) ---! ---! Implements the -> operator to access fields/elements from encrypted JSONB data. ---! Returns the matching sv entry as `eql_v2.ste_vec_entry` (or NULL on miss). ---! ---! Encrypted JSON is represented as an array of sv elements in the ---! StEVec format. Each element has a selector, ciphertext, and index ---! terms: `{"sv": [{"c": "...", "s": "...", "hm": "..."}, ...]}`. ---! ---! Provides three overloads: ---! - (eql_v2_encrypted, text) - Field name selector ---! - (eql_v2_encrypted, eql_v2_encrypted) - Encrypted selector ---! - (eql_v2_encrypted, integer) - Array index selector (0-based) ---! ---! All three return `eql_v2.ste_vec_entry` and preserve the source ---! payload's root `i` / `v` envelope metadata in the returned entry ---! (the DOMAIN CHECK on `ste_vec_entry` doesn't forbid extra fields). ---! ---! @note Operator resolution: Assignment casts are considered (PostgreSQL standard behavior). ---! To use text selector, parameter may need explicit cast to text. ---! ---! @see eql_v2.ste_vec_entry ---! @see eql_v2.selector ---! @see eql_v2."->>" - ---! @brief -> operator with text selector ---! ---! Returns the sv entry whose `s` selector equals @p selector, with ---! the source payload's `i` / `v` metadata merged in. Selectors are ---! deterministic per (path, key) within a document, so at most one ---! entry matches; `jsonb_path_query_first` returns the first match ---! and stops scanning. ---! ---! Inlinable single-statement SQL: the planner folds this body into ---! the calling query, so `WHERE col -> 'sel' = $1` reduces structurally ---! to `eql_v2.eq_term(col -> 'sel') = eql_v2.eq_term($1)` and matches ---! a functional index built on `eql_v2.eq_term(col -> 'sel')`. ---! ---! @param e eql_v2_encrypted Encrypted JSONB payload (root) ---! @param selector text Selector hash (the `s` field value) ---! @return eql_v2.ste_vec_entry Matching entry merged with root meta, ---! NULL if no element matches. ---! ---! @note The returned entry carries `i` / `v` from the root in addition ---! to the sv-element fields. This is intentional: per-entry ---! extractors (`eql_v2.eq_term`, `eql_v2.ore_cllw`, ...) read ---! only their own fields and ignore `i` / `v`; callers that need ---! the root envelope (e.g. for decryption) still see it. ---! ---! @example ---! SELECT encrypted_json -> 'field_name' FROM table; -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector text) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT ( - eql_v2.meta_data(e) || - jsonb_path_query_first( - (e).data, - '$.sv[*] ? (@.s == $sel)'::jsonpath, - jsonb_build_object('sel', selector) - ) - )::eql_v2.ste_vec_entry -$$; - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=text -); - ---------------------------------------------------- - ---! @brief -> operator with encrypted selector ---! ---! Convenience overload: extracts the selector text from an encrypted ---! selector payload and delegates to the (text) form. Inlinable. ---! ---! @param e eql_v2_encrypted Encrypted JSONB data ---! @param selector eql_v2_encrypted Encrypted selector payload ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @see eql_v2."->"(eql_v2_encrypted, text) -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2."->"(e, eql_v2._selector(selector)) -$$; - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=eql_v2_encrypted -); - - ---------------------------------------------------- - ---! @brief -> operator with integer array index ---! ---! Returns the sv entry at the given (0-based, JSONB-style) array ---! index, merged with the root payload's `i` / `v` metadata. Returns ---! NULL when the underlying value isn't an sv-array payload or when ---! the index is out of bounds. ---! ---! @param e eql_v2_encrypted Encrypted sv-array payload ---! @param selector integer Array index (0-based, JSONB convention) ---! @return eql_v2.ste_vec_entry Matching entry, NULL on miss ---! @note Array index is 0-based (JSONB standard) despite PostgreSQL arrays being 1-based ---! @example ---! SELECT encrypted_array -> 0 FROM table; ---! @see eql_v2.is_ste_vec_array -CREATE FUNCTION eql_v2."->"(e eql_v2_encrypted, selector integer) - RETURNS eql_v2.ste_vec_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT CASE - WHEN eql_v2.is_ste_vec_array(e) THEN - (eql_v2.meta_data(e) || ((e).data -> 'sv' -> selector))::eql_v2.ste_vec_entry - ELSE NULL - END -$$; - - - - - -CREATE OPERATOR ->( - FUNCTION=eql_v2."->", - LEFTARG=eql_v2_encrypted, - RIGHTARG=integer -); - - ---! @brief EQL lint: detect non-inlinable operator implementation functions ---! ---! Returns one row per violation found in the installed EQL surface. The ---! Postgres planner can only inline a function during index matching when: ---! ---! * `LANGUAGE sql` (plpgsql / C / etc. cannot be inlined) ---! * `IMMUTABLE` or `STABLE` volatility (VOLATILE cannot be inlined into ---! index expressions) ---! * No `SET` clauses (e.g. `SET search_path = ...`) ---! * Not `SECURITY DEFINER` ---! * Single-statement SELECT body ---! ---! @note The single-statement SELECT body condition is **not yet checked** by ---! this lint. A `LANGUAGE sql` function with a multi-statement body, a CTE, ---! or any pre-SELECT statement will pass all four implemented checks while ---! remaining non-inlinable. Implementing the check requires walking `prosrc` ---! (or `pg_get_functiondef`); tracked as a follow-up to #194. ---! ---! Operators on encrypted types (`eql_v2_encrypted`, `eql_v2.bloom_filter`, ---! `eql_v2.ore_*`, etc.) whose implementation functions fail any of these ---! rules silently fall back to seq scan when the documented functional ---! indexes (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, ---! `eql_v2.ste_vec(col)`) are in place. This lint surfaces every such case. ---! ---! Severity: ---! `error` — fixable, blocks index matching, ship-blocking. ---! `warning` — likely-fixable, may not block matching but signals intent. ---! `info` — observational; useful for review, not a defect on its own. ---! ---! Categories: ---! `inlinability_language` — implementation function isn't `LANGUAGE sql`. ---! `inlinability_volatility` — implementation function is VOLATILE. ---! `inlinability_set_clause` — implementation function has a `SET` clause. ---! `inlinability_secdef` — implementation function is `SECURITY DEFINER`. ---! `inlinability_transitive` — implementation function is itself inlinable ---! but its body invokes a non-inlinable function ---! (depth 1; the planner can't peek through ---! that boundary). ---! ---! @example ---! ``` ---! SELECT severity, category, object_name, message ---! FROM eql_v2.lints() ---! WHERE severity = 'error' ---! ORDER BY category, object_name; ---! ``` ---! ---! @return SETOF record (severity text, category text, object_name text, message text) -CREATE OR REPLACE FUNCTION eql_v2.lints() -RETURNS TABLE ( - severity text, - category text, - object_name text, - message text -) -LANGUAGE sql STABLE -AS $$ - WITH - -- All operators where at least one operand involves an EQL type. Limits - -- the scope of the lint to the operator surface customers actually hit - -- via SQL (`col = val`, `col LIKE '...'`, `col @> '...'` and friends). - eql_operators AS ( - SELECT - op.oid AS oprid, - op.oprname AS opname, - op.oprcode AS implfunc, - op.oprleft::regtype AS lhs, - op.oprright::regtype AS rhs, - op.oprcode::regprocedure AS impl_signature - FROM pg_operator op - WHERE EXISTS ( - SELECT 1 FROM pg_type t - WHERE t.oid IN (op.oprleft, op.oprright) - AND (t.typname LIKE 'eql_v2%' - OR t.typnamespace = 'eql_v2'::regnamespace) - ) - ), - - -- Cross-join with each operator's implementation function metadata. - -- One row per operator; columns describe the inlinability of the impl. - op_impl AS ( - SELECT - eo.opname, - eo.lhs, - eo.rhs, - eo.impl_signature::text AS impl_signature, - lang_l.lanname AS lang, - p.provolatile AS volatility, - p.proconfig AS config, - p.prosecdef AS secdef, - p.prosrc AS body - FROM eql_operators eo - JOIN pg_proc p ON p.oid = eo.implfunc - JOIN pg_language lang_l ON lang_l.oid = p.prolang - ) - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Direct inlinability checks: each row examines one operator's │ - -- │ implementation function and emits a violation if any rule is │ - -- │ broken. Multiple violations on the same function become │ - -- │ multiple rows (developers see every reason it doesn't inline). │ - -- └─────────────────────────────────────────────────────────────────┘ - - SELECT - 'error' AS severity, - 'inlinability_language' AS category, - format('operator %s(%s, %s) -> %s', - opname, lhs, rhs, impl_signature) AS object_name, - format( - 'Operator implementation function is `LANGUAGE %s`; only `LANGUAGE sql` functions can be inlined by the planner. Bare `col %s val` queries fall back to seq scan even when a matching functional index exists.', - lang, opname) AS message - FROM op_impl - WHERE lang <> 'sql' - - UNION ALL - - SELECT - 'error', - 'inlinability_volatility', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function is `VOLATILE`. The Postgres planner refuses to inline volatile functions into index expressions, so functional indexes never engage. Mark the function `IMMUTABLE` (or `STABLE` if it depends on session state).', - opname) - FROM op_impl - WHERE volatility = 'v' - - UNION ALL - - SELECT - 'error', - 'inlinability_set_clause', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - format( - 'Operator implementation function has a `SET` clause (e.g. `SET search_path = ...`). Per Postgres function-inlining rules, any `SET` clause blocks inlining. Use schema-qualified identifiers in the body and remove the `SET` clause to allow the planner to inline.') - FROM op_impl - WHERE config IS NOT NULL - - UNION ALL - - SELECT - 'error', - 'inlinability_secdef', - format('operator %s(%s, %s) -> %s', opname, lhs, rhs, impl_signature), - 'Operator implementation function is `SECURITY DEFINER`. Such functions cannot be inlined; remove `SECURITY DEFINER` or use a non-inlinable wrapper layer.' - FROM op_impl - WHERE secdef - - -- ┌─────────────────────────────────────────────────────────────────┐ - -- │ Transitive inlinability: an operator implementation function │ - -- │ that's itself inlinable can still fail to inline if its body │ - -- │ calls a non-inlinable function. Walk one level via pg_depend. │ - -- │ │ - -- │ Postgres records function-to-function dependencies in │ - -- │ pg_depend with deptype 'n' (normal) when one function references│ - -- │ another in its body — but only at CREATE time and only for │ - -- │ direct calls. This is good enough for v1; deeper transitive │ - -- │ analysis is a follow-up. │ - -- └─────────────────────────────────────────────────────────────────┘ - - UNION ALL - - SELECT - 'error', - 'inlinability_transitive', - format('operator %s(%s, %s) -> %s', oi.opname, oi.lhs, oi.rhs, - oi.impl_signature), - format( - 'Operator implementation function is inlinable but invokes non-inlinable function `%s` (lang=%s, volatility=%s%s). The chain blocks at depth 1: the planner inlines the outer call but cannot reduce the inner call into an index expression.', - called.proname, - called_lang.lanname, - CASE called.provolatile - WHEN 'i' THEN 'IMMUTABLE' - WHEN 's' THEN 'STABLE' - WHEN 'v' THEN 'VOLATILE' - END, - CASE WHEN called.proconfig IS NOT NULL - THEN ', has SET clause' - ELSE '' END) - FROM op_impl oi - -- Only worth the transitive check if the outer function is otherwise - -- inlinable — otherwise the direct lints above already report it. - JOIN pg_proc outer_p ON outer_p.oid = oi.impl_signature::regprocedure - JOIN pg_depend d - ON d.classid = 'pg_proc'::regclass - AND d.objid = outer_p.oid - AND d.refclassid = 'pg_proc'::regclass - AND d.deptype = 'n' - JOIN pg_proc called ON called.oid = d.refobjid - JOIN pg_language called_lang ON called_lang.oid = called.prolang - WHERE oi.lang = 'sql' - AND oi.volatility IN ('i', 's') - AND oi.config IS NULL - AND NOT oi.secdef - AND called.oid <> outer_p.oid - AND ( - called_lang.lanname <> 'sql' - OR called.provolatile = 'v' - OR called.proconfig IS NOT NULL - OR called.prosecdef - ) - - ORDER BY 1, 2, 3; -$$; - -COMMENT ON FUNCTION eql_v2.lints() IS - 'EQL lint: returns one row per non-inlinable operator implementation. ' - 'Run `SELECT * FROM eql_v2.lints() WHERE severity = ''error''` for a ' - 'CI-gateable check that all operator implementations on EQL types are ' - 'eligible for planner inlining.'; - ---! @file jsonb/functions.sql ---! @brief JSONB path query and array manipulation functions for encrypted data ---! ---! These functions provide PostgreSQL-compatible operations on encrypted JSONB values ---! using Structured Transparent Encryption (STE). They support: ---! - Path-based queries to extract nested encrypted values ---! - Existence checks for encrypted fields ---! - Array operations (length, elements extraction) ---! - Field-level HMAC term extraction for equality / GROUP BY / DISTINCT ---! ---! @note STE stores encrypted JSONB as a vector of encrypted elements ('sv') with selectors ---! @note Functions suppress errors for missing fields, type mismatches (similar to PostgreSQL jsonpath) ---! @note `selector` parameters in this module are *encrypted-side* selector ---! hashes — the deterministic hash that the crypto layer (e.g. ---! `@cipherstash/protect`) emits in the `s` field of each `sv` element ---! (e.g. `'a7cea93975ed8c01f861ccb6bd082784'`). Plaintext JSONPaths ---! like `'$.address.city'` are never accepted at runtime; the proxy / ---! client rewrites them to selector hashes before the query reaches EQL. - - ---! @brief Query encrypted JSONB for elements matching selector ---! ---! Searches the Structured Transparent Encryption (STE) vector for elements matching ---! the given selector path. Returns all matching encrypted elements. If multiple ---! matches form an array, they are wrapped with array metadata. ---! ---! @param jsonb Encrypted JSONB payload containing STE vector ('sv') ---! @param text Path selector to match against encrypted elements ---! @return SETOF eql_v2_encrypted Matching encrypted elements (may return multiple rows) ---! ---! @note Returns empty set if selector is not found (does not throw exception) ---! @note Array elements use same selector; multiple matches wrapped with 'a' flag ---! @note Returns a set containing NULL if val is NULL; returns empty set if no matches found ---! @see eql_v2.jsonb_path_query_first ---! @see eql_v2.jsonb_path_exists -CREATE FUNCTION eql_v2.jsonb_path_query(val jsonb, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT - CASE - WHEN bool_or(eql_v2.is_ste_vec_array(elem)) THEN - (eql_v2.meta_data(val) || jsonb_build_object('sv', jsonb_agg(elem), 'a', 1))::eql_v2_encrypted - ELSE - (eql_v2.meta_data(val) || (array_agg(elem))[1])::eql_v2_encrypted - END - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - HAVING count(*) > 0 -$$; - - ---! @brief Query encrypted JSONB with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its plaintext value ---! before delegating to main jsonb_path_query implementation. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Query encrypted JSONB with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector, ---! extracting the JSONB payload before querying. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match against ---! @return SETOF eql_v2_encrypted Matching encrypted elements ---! ---! @example ---! -- Query encrypted JSONB for the sv element at a given selector hash ---! SELECT * FROM eql_v2.jsonb_path_query(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query(val eql_v2_encrypted, selector text) - RETURNS SETOF eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT * FROM eql_v2.jsonb_path_query((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Check if selector path exists in encrypted JSONB ---! ---! Tests whether any encrypted elements match the given selector path. ---! More efficient than jsonb_path_query when only existence check is needed. ---! ---! @param jsonb Encrypted JSONB payload to check ---! @param text Path selector to test ---! @return boolean True if matching element exists, false otherwise ---! ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val jsonb, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT 1 FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - ); -$$; - - ---! @brief Check existence with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before checking existence. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to check ---! @param selector eql_v2_encrypted Encrypted selector to test ---! @return boolean True if path exists ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Check existence with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to check ---! @param text Path selector to test ---! @return boolean True if path exists ---! ---! @example ---! -- Check if the encrypted document has an sv element at a given selector hash ---! SELECT eql_v2.jsonb_path_exists(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_exists(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_exists(val eql_v2_encrypted, selector text) - RETURNS boolean - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_exists((val).data, selector); -$$; - - ------------------------------------------------------------------------------------- - - ---! @brief Get first element matching selector ---! ---! Returns only the first encrypted element matching the selector path, ---! or NULL if no match found. More efficient than jsonb_path_query when ---! only one result is needed. ---! ---! @param jsonb Encrypted JSONB payload to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @note Uses LIMIT 1 internally for efficiency ---! @see eql_v2.jsonb_path_query(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val jsonb, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT (eql_v2.meta_data(val) || elem)::eql_v2_encrypted - FROM jsonb_array_elements(val -> 'sv') elem - WHERE elem ->> 's' = selector - LIMIT 1 -$$; - - ---! @brief Get first element with encrypted selector ---! ---! Overload that accepts encrypted selector and extracts its value ---! before querying for first match. ---! ---! @param val eql_v2_encrypted Encrypted JSONB value to query ---! @param selector eql_v2_encrypted Encrypted selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector eql_v2_encrypted) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, eql_v2._selector(selector)); -$$; - - ---! @brief Get first element with text selector ---! ---! Overload that accepts encrypted JSONB value and text selector. ---! ---! @param eql_v2_encrypted Encrypted JSONB value to query ---! @param text Path selector to match ---! @return eql_v2_encrypted First matching element or NULL ---! ---! @example ---! -- Get the first matching sv element from an encrypted document ---! SELECT eql_v2.jsonb_path_query_first(encrypted_document, 'a7cea93975ed8c01f861ccb6bd082784'); ---! ---! @see eql_v2.jsonb_path_query_first(jsonb, text) -CREATE FUNCTION eql_v2.jsonb_path_query_first(val eql_v2_encrypted, selector text) - RETURNS eql_v2_encrypted - LANGUAGE sql - IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT eql_v2.jsonb_path_query_first((val).data, selector); -$$; - - - ------------------------------------------------------------------------------------- - - ---! @brief Get length of encrypted JSONB array ---! ---! Returns the number of elements in an encrypted JSONB array by counting ---! elements in the STE vector ('sv'). The encrypted value must have the ---! array flag ('a') set to true. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return integer Number of elements in the array ---! @throws Exception 'cannot get array length of a non-array' if 'a' flag is missing or not true ---! ---! @note Array flag 'a' must be present and set to true value ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_length(val jsonb) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - - IF val IS NULL THEN - RETURN NULL; - END IF; - - IF eql_v2.is_ste_vec_array(val) THEN - sv := eql_v2.ste_vec(val); - RETURN array_length(sv, 1); - END IF; - - RAISE 'cannot get array length of a non-array'; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get array length from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts the ---! JSONB payload before computing array length. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return integer Number of elements in the array ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get length of encrypted array ---! SELECT eql_v2.jsonb_array_length(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_length(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_length(val eql_v2_encrypted) - RETURNS integer - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN ( - SELECT eql_v2.jsonb_array_length(val.data) - ); - END; -$$ LANGUAGE plpgsql; - - - - ---! @brief Extract elements from encrypted JSONB array ---! ---! Returns each element of an encrypted JSONB array as a separate row. ---! Each element is returned as an eql_v2_encrypted value with metadata ---! preserved from the parent array. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Each element inherits metadata (version, ident) from parent ---! @see eql_v2.jsonb_array_length ---! @see eql_v2.jsonb_array_elements_text -CREATE FUNCTION eql_v2.jsonb_array_elements(val jsonb) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - meta jsonb; - item jsonb; - BEGIN - - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - -- Column identifier and version - meta := eql_v2.meta_data(val); - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - item = sv[idx]; - RETURN NEXT (meta || item)::eql_v2_encrypted; - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract elements from encrypted array type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element as a separate row. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF eql_v2_encrypted One row per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Expand encrypted array into rows ---! SELECT * FROM eql_v2.jsonb_array_elements(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements(val eql_v2_encrypted) - RETURNS SETOF eql_v2_encrypted - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements(val.data); - END; -$$ LANGUAGE plpgsql; - - - ---! @brief Extract encrypted array elements as ciphertext ---! ---! Returns each element of an encrypted JSONB array as its raw ciphertext ---! value (text representation). Unlike jsonb_array_elements, this returns ---! only the ciphertext 'c' field without metadata. ---! ---! @param jsonb Encrypted JSONB payload representing an array ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array (missing 'a' flag) ---! ---! @note Returns ciphertext only, not full encrypted structure ---! @see eql_v2.jsonb_array_elements -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val jsonb) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - sv eql_v2_encrypted[]; - found eql_v2_encrypted[]; - BEGIN - IF NOT eql_v2.is_ste_vec_array(val) THEN - RAISE 'cannot extract elements from non-array'; - END IF; - - sv := eql_v2.ste_vec(val); - - FOR idx IN 1..array_length(sv, 1) LOOP - RETURN NEXT eql_v2.ciphertext(sv[idx]); - END LOOP; - - RETURN; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Extract array elements as ciphertext from encrypted type ---! ---! Overload that accepts encrypted composite type and extracts each ---! array element's ciphertext as text. ---! ---! @param eql_v2_encrypted Encrypted array value ---! @return SETOF text One ciphertext string per array element ---! @throws Exception if value is not an array ---! ---! @example ---! -- Get ciphertext of each array element ---! SELECT * FROM eql_v2.jsonb_array_elements_text(encrypted_tags); ---! ---! @see eql_v2.jsonb_array_elements_text(jsonb) -CREATE FUNCTION eql_v2.jsonb_array_elements_text(val eql_v2_encrypted) - RETURNS SETOF text - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - SELECT * FROM eql_v2.jsonb_array_elements_text(val.data); - END; -$$ LANGUAGE plpgsql; - - ------------------------------------------------------------------------------------- - --- `eql_v2.hmac_256_terms(eql_v2_encrypted)` was added under #205 as a --- GIN-indexable {s, hm} aggregate. It's been removed: under the XOR --- contract each sv element carries exactly one of `hm` (bool leaves, --- array / object roots) or `oc` (string / number leaves), and --- `hmac_256_terms` filters out everything without `hm` — so containment --- queries via this index could never match on string / number selectors. --- The canonical XOR-aware replacement is the typed --- `@>(eql_v2_encrypted, eql_v2.stevec_query)` overload, which inlines --- to `eql_v2.to_stevec_query(col)::jsonb @> needle::jsonb` and engages --- a functional GIN on `(eql_v2.to_stevec_query(col)::jsonb) jsonb_path_ops`. --- See U-007 / U-008 in `docs/upgrading/v2.3.md`. ---! @file encryptindex/functions.sql ---! @brief Configuration lifecycle and column encryption management ---! ---! Provides functions for managing encryption configuration transitions: ---! - Comparing configurations to identify changes ---! - Identifying columns needing encryption ---! - Creating and renaming encrypted columns during initial setup ---! - Tracking encryption progress ---! ---! These functions support the workflow of activating a pending configuration ---! and performing the initial encryption of plaintext columns. - - ---! @brief Compare two configurations and find differences ---! @internal ---! ---! Returns table/column pairs where configuration differs between two configs. ---! Used to identify which columns need encryption when activating a pending config. ---! ---! @param a jsonb First configuration to compare ---! @param b jsonb Second configuration to compare ---! @return TABLE(table_name text, column_name text) Columns with differing configuration ---! ---! @note Compares configuration structure, not just presence/absence ---! @see eql_v2.select_pending_columns -CREATE FUNCTION eql_v2.diff_config(a JSONB, b JSONB) - RETURNS TABLE(table_name TEXT, column_name TEXT) -IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - RETURN QUERY - WITH table_keys AS ( - SELECT jsonb_object_keys(a->'tables') AS key - UNION - SELECT jsonb_object_keys(b->'tables') AS key - ), - column_keys AS ( - SELECT tk.key AS table_key, jsonb_object_keys(a->'tables'->tk.key) AS column_key - FROM table_keys tk - UNION - SELECT tk.key AS table_key, jsonb_object_keys(b->'tables'->tk.key) AS column_key - FROM table_keys tk - ) - SELECT - ck.table_key AS table_name, - ck.column_key AS column_name - FROM - column_keys ck - WHERE - (a->'tables'->ck.table_key->ck.column_key IS DISTINCT FROM b->'tables'->ck.table_key->ck.column_key); - END; -$$ LANGUAGE plpgsql; - - ---! @brief Get columns with pending configuration changes ---! ---! Compares 'pending' and 'active' configurations to identify columns that need ---! encryption or re-encryption. Returns columns where configuration differs. ---! ---! @return TABLE(table_name text, column_name text) Columns needing encryption ---! @throws Exception if no pending configuration exists ---! ---! @note Treats missing active config as empty config ---! @see eql_v2.diff_config ---! @see eql_v2.select_target_columns -CREATE FUNCTION eql_v2.select_pending_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - active JSONB; - pending JSONB; - config_id BIGINT; - BEGIN - SELECT data INTO active FROM eql_v2_configuration WHERE state = 'active'; - - -- set default config - IF active IS NULL THEN - active := '{}'; - END IF; - - SELECT id, data INTO config_id, pending FROM eql_v2_configuration WHERE state = 'pending'; - - -- set default config - IF config_id IS NULL THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - RETURN QUERY - SELECT d.table_name, d.column_name FROM eql_v2.diff_config(active, pending) as d; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Map pending columns to their encrypted target columns ---! ---! For each column with pending configuration, identifies the corresponding ---! encrypted column. During initial encryption, target is '{column_name}_encrypted'. ---! Returns NULL for target_column if encrypted column doesn't exist yet. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Column mappings ---! ---! @note Target column is NULL if no column exists matching either 'column_name' or 'column_name_encrypted' with type eql_v2_encrypted ---! @note The LEFT JOIN checks both original and '_encrypted' suffix variations with type verification ---! @see eql_v2.select_pending_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.select_target_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT - c.table_name, - c.column_name, - s.column_name as target_column - FROM - eql_v2.select_pending_columns() c - LEFT JOIN information_schema.columns s ON - s.table_name = c.table_name AND - (s.column_name = c.column_name OR s.column_name = c.column_name || '_encrypted') AND - s.udt_name = 'eql_v2_encrypted'; -$$ LANGUAGE sql; - - ---! @brief Check if database is ready for encryption ---! ---! Verifies that all columns with pending configuration have corresponding ---! encrypted target columns created. Returns true if encryption can proceed. ---! ---! @return boolean True if all pending columns have target encrypted columns ---! ---! @note Returns false if any pending column lacks encrypted column ---! @see eql_v2.select_target_columns ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.ready_for_encryption() - RETURNS BOOLEAN - STABLE STRICT PARALLEL SAFE -AS $$ - SELECT EXISTS ( - SELECT * - FROM eql_v2.select_target_columns() AS c - WHERE c.target_column IS NOT NULL); -$$ LANGUAGE sql; - - ---! @brief Create encrypted columns for initial encryption ---! ---! For each plaintext column with pending configuration that lacks an encrypted ---! target column, creates a new column '{column_name}_encrypted' of type ---! eql_v2_encrypted. This prepares the database schema for initial encryption. ---! ---! @return TABLE(table_name text, column_name text) Created encrypted columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE ADD COLUMN) - modifies database schema ---! @note Only creates columns that don't already exist ---! @see eql_v2.select_target_columns ---! @see eql_v2.rename_encrypted_columns -CREATE FUNCTION eql_v2.create_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name IN - SELECT c.table_name, (c.column_name || '_encrypted') FROM eql_v2.select_target_columns() AS c WHERE c.target_column IS NULL - LOOP - EXECUTE format('ALTER TABLE %I ADD column %I eql_v2_encrypted;', table_name, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Finalize initial encryption by renaming columns ---! ---! After initial encryption completes, renames columns to complete the transition: ---! - Plaintext column '{column_name}' → '{column_name}_plaintext' ---! - Encrypted column '{column_name}_encrypted' → '{column_name}' ---! ---! This makes the encrypted column the primary column with the original name. ---! ---! @return TABLE(table_name text, column_name text, target_column text) Renamed columns ---! ---! @warning Executes dynamic DDL (ALTER TABLE RENAME COLUMN) - modifies database schema ---! @note Only renames columns where target is '{column_name}_encrypted' ---! @see eql_v2.create_encrypted_columns -CREATE FUNCTION eql_v2.rename_encrypted_columns() - RETURNS TABLE(table_name TEXT, column_name TEXT, target_column TEXT) - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - FOR table_name, column_name, target_column IN - SELECT * FROM eql_v2.select_target_columns() as c WHERE c.target_column = c.column_name || '_encrypted' - LOOP - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, column_name, column_name || '_plaintext'); - EXECUTE format('ALTER TABLE %I RENAME %I TO %I;', table_name, target_column, column_name); - RETURN NEXT; - END LOOP; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Count rows encrypted with active configuration ---! @internal ---! ---! Counts rows in a table where the encrypted column was encrypted using ---! the currently active configuration. Used to track encryption progress. ---! ---! @param table_name text Name of table to check ---! @param column_name text Name of encrypted column to check ---! @return bigint Count of rows encrypted with active configuration ---! ---! @note The 'v' field in encrypted payloads stores the payload version ("2"), not the configuration ID ---! @note Configuration tracking mechanism is implementation-specific -CREATE FUNCTION eql_v2.count_encrypted_with_active_config(table_name TEXT, column_name TEXT) - RETURNS BIGINT - SET search_path = pg_catalog, extensions, public -AS $$ -DECLARE - result BIGINT; -BEGIN - EXECUTE format( - 'SELECT COUNT(%I) FROM %s t WHERE %I->>%L = (SELECT id::TEXT FROM eql_v2_configuration WHERE state = %L)', - column_name, table_name, column_name, 'v', 'active' - ) - INTO result; - RETURN result; -END; -$$ LANGUAGE plpgsql; - - - ---! @brief Validate presence of ident field in encrypted payload ---! @internal ---! ---! Checks that the encrypted JSONB payload contains the required 'i' (ident) field. ---! The ident field tracks which table and column the encrypted value belongs to. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'i' field is present ---! @throws Exception if 'i' field is missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF val ? 'i' THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ident (i) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate table and column fields in ident ---! @internal ---! ---! Checks that the 'i' (ident) field contains both 't' (table) and 'c' (column) ---! subfields, which identify the origin of the encrypted value. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if both 't' and 'c' subfields are present ---! @throws Exception if 't' or 'c' subfields are missing ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_i_ct(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val->'i' ?& array['t', 'c']) THEN - RETURN true; - END IF; - RAISE 'Encrypted column ident (i) missing table (t) or column (c) fields: %', val; - END; -$$ LANGUAGE plpgsql; - ---! @brief Validate version field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload has version field 'v' set to '2', ---! the current EQL v2 payload version. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if 'v' field is present and equals '2' ---! @throws Exception if 'v' field is missing or not '2' ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_v(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - - IF val->>'v' <> '2' THEN - RAISE 'Expected encrypted column version (v) 2'; - RETURN false; - END IF; - - RETURN true; - END IF; - RAISE 'Encrypted column missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ciphertext field in encrypted payload ---! @internal ---! ---! Checks that the encrypted payload carries the required root-level ciphertext ---! envelope. The v2.3 payload schema admits two mutually exclusive top-level ---! shapes (`docs/reference/schema/eql-payload-v2.3.schema.json`): ---! ---! - `EncryptedPayload` (scalar) — carries `c` at the root. ---! - `SteVecPayload` (jsonb / structured) — carries `sv` at the root; the ---! root document ciphertext lives inside `sv[0].c`, so `c` is absent at ---! the root. ---! ---! Either shape satisfies this check. Per-element ciphertext validity on ---! `sv` entries is enforced separately by the `eql_v2.ste_vec_entry` DOMAIN. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if either 'c' or 'sv' is present at the root ---! @throws Exception if neither 'c' nor 'sv' is present ---! ---! @note Used in CHECK constraints to ensure payload structure ---! @see eql_v2.check_encrypted -CREATE FUNCTION eql_v2._encrypted_check_c(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'c') OR (val ? 'sv') THEN - RETURN true; - END IF; - RAISE 'Encrypted column missing ciphertext (c) or ste_vec (sv) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate complete encrypted payload structure ---! ---! Comprehensive validation function that checks all required fields in an ---! encrypted JSONB payload: version ('v'), ciphertext ('c'), ident ('i'), ---! and ident subfields ('t', 'c'). ---! ---! This function is used in CHECK constraints to ensure encrypted column ---! data integrity at the database level. ---! ---! @param jsonb Encrypted payload to validate ---! @return Boolean True if all structure checks pass ---! @throws Exception if any required field is missing or invalid ---! ---! @example ---! -- Add validation constraint to encrypted column ---! ALTER TABLE users ADD CONSTRAINT check_email_encrypted ---! CHECK (eql_v2.check_encrypted(encrypted_email::jsonb)); ---! ---! @see eql_v2._encrypted_check_v ---! @see eql_v2._encrypted_check_c ---! @see eql_v2._encrypted_check_i ---! @see eql_v2._encrypted_check_i_ct -CREATE FUNCTION eql_v2.check_encrypted(val jsonb) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN ( - eql_v2._encrypted_check_v(val) AND - eql_v2._encrypted_check_c(val) AND - eql_v2._encrypted_check_i(val) AND - eql_v2._encrypted_check_i_ct(val) - ); -END; - - ---! @brief Validate encrypted composite type structure ---! ---! Validates an eql_v2_encrypted composite type by checking its underlying ---! JSONB payload. Delegates to eql_v2.check_encrypted(jsonb). ---! ---! @param eql_v2_encrypted Encrypted value to validate ---! @return Boolean True if structure is valid ---! @throws Exception if any required field is missing or invalid ---! ---! @see eql_v2.check_encrypted(jsonb) -CREATE FUNCTION eql_v2.check_encrypted(val eql_v2_encrypted) - RETURNS BOOLEAN -LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN eql_v2.check_encrypted(val.data); -END; - - --- Aggregate functions for ORE - ---! @brief State transition function for min aggregate ---! @internal ---! ---! Returns the smaller of two encrypted values for use in MIN aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The smaller of the two values ---! ---! @see eql_v2.min(eql_v2_encrypted) -CREATE FUNCTION eql_v2.min(a eql_v2_encrypted, b eql_v2_encrypted) - RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a < b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find minimum encrypted value in a group ---! ---! Aggregate function that returns the minimum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Minimum value in the group ---! ---! @example ---! -- Find minimum age per department ---! SELECT department, eql_v2.min(encrypted_age) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.min(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.min(eql_v2_encrypted) -( - sfunc = eql_v2.min, - stype = eql_v2_encrypted -); - - ---! @brief State transition function for max aggregate ---! @internal ---! ---! Returns the larger of two encrypted values for use in MAX aggregate. ---! Comparison uses ORE index terms without decryption. ---! ---! @param a eql_v2_encrypted First encrypted value ---! @param b eql_v2_encrypted Second encrypted value ---! @return eql_v2_encrypted The larger of the two values ---! ---! @see eql_v2.max(eql_v2_encrypted) -CREATE FUNCTION eql_v2.max(a eql_v2_encrypted, b eql_v2_encrypted) -RETURNS eql_v2_encrypted -STRICT - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF a > b THEN - RETURN a; - ELSE - RETURN b; - END IF; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Find maximum encrypted value in a group ---! ---! Aggregate function that returns the maximum encrypted value in a group ---! using ORE index term comparisons without decryption. ---! ---! @param input eql_v2_encrypted Encrypted values to aggregate ---! @return eql_v2_encrypted Maximum value in the group ---! ---! @example ---! -- Find maximum salary per department ---! SELECT department, eql_v2.max(encrypted_salary) ---! FROM employees ---! GROUP BY department; ---! ---! @note Requires 'ore' index configuration on the column ---! @see eql_v2.max(eql_v2_encrypted, eql_v2_encrypted) -CREATE AGGREGATE eql_v2.max(eql_v2_encrypted) -( - sfunc = eql_v2.max, - stype = eql_v2_encrypted -); - - ---! @file config/indexes.sql ---! @brief Configuration state uniqueness indexes ---! ---! Creates partial unique indexes to enforce that only one configuration ---! can be in 'active', 'pending', or 'encrypting' state at any time. ---! Multiple 'inactive' configurations are allowed. ---! ---! @note Uses partial indexes (WHERE clauses) for efficiency ---! @note Prevents conflicting configurations from being active simultaneously ---! @see config/types.sql for state definitions - - ---! @brief Unique active configuration constraint ---! @note Only one configuration can be 'active' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'active'; - ---! @brief Unique pending configuration constraint ---! @note Only one configuration can be 'pending' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'pending'; - ---! @brief Unique encrypting configuration constraint ---! @note Only one configuration can be 'encrypting' at once -CREATE UNIQUE INDEX ON public.eql_v2_configuration (state) WHERE state = 'encrypting'; - - ---! @brief Add a search index configuration for an encrypted column ---! ---! Configures a searchable encryption index (unique, match, ore, ope, or ste_vec) ---! on an encrypted column. Creates or updates the pending configuration, then ---! migrates and activates it unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to configure ---! @param index_name Text Type of index ('unique', 'match', 'ore', 'ope', 'ste_vec') ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB Index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if index already exists for this column ---! @throws Exception if cast_as is not a valid type ---! ---! @example ---! -- Add unique index for exact-match searches ---! SELECT eql_v2.add_search_config('users', 'email', 'unique'); ---! ---! -- Add match index for LIKE searches with custom token length ---! SELECT eql_v2.add_search_config('posts', 'content', 'match', 'text', ---! '{"token_filters": [{"kind": "downcase"}], "tokenizer": {"kind": "ngram", "token_length": 3}}' ---! ); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.add_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - o jsonb; - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if index exists - IF _config #> array['tables', table_name, column_name, 'indexes'] ? index_name THEN - RAISE EXCEPTION '% index exists for column: % %', index_name, table_name, column_name; - END IF; - - IF NOT cast_as = ANY('{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}') THEN - RAISE EXCEPTION '% is not a valid cast type', cast_as; - END IF; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- set default options for index if opts empty - IF index_name = 'match' AND opts = '{}' THEN - SELECT eql_v2.config_match_default() INTO opts; - END IF; - - SELECT eql_v2.config_add_index(table_name, column_name, index_name, opts, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a search index configuration from an encrypted column ---! ---! Removes a previously configured search index from an encrypted column. ---! Updates the pending configuration, then migrates and activates it ---! unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove match index from column ---! SELECT eql_v2.remove_search_config('posts', 'content', 'match'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.modify_search_config -CREATE FUNCTION eql_v2.remove_search_config(table_name text, column_name text, index_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _config jsonb; - BEGIN - - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the index does not exist - -- IF NOT _config->key ? index_name THEN - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No % index exists for column: % %', index_name, table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the index - SELECT _config #- array['tables', table_name, column_name, 'indexes', index_name] INTO _config; - - -- update the config and migrate (even if empty) - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Modify a search index configuration for an encrypted column ---! ---! Updates an existing search index configuration by removing and re-adding it ---! with new options. Convenience function that combines remove and add operations. ---! If index does not exist, it is added. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column ---! @param index_name Text Type of index to modify ---! @param cast_as Text PostgreSQL type for decrypted values (default: 'text') ---! @param opts JSONB New index-specific options (default: '{}') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! ---! @example ---! -- Change match index tokenizer settings ---! SELECT eql_v2.modify_search_config('posts', 'content', 'match', 'text', ---! '{"tokenizer": {"kind": "ngram", "token_length": 4}}' ---! ); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.modify_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - PERFORM eql_v2.remove_search_config(table_name, column_name, index_name, migrating); - RETURN eql_v2.add_search_config(table_name, column_name, index_name, cast_as, opts, migrating); - END; -$$ LANGUAGE plpgsql; - ---! @brief Migrate pending configuration to encrypting state ---! ---! Transitions the pending configuration to encrypting state, validating that ---! all configured columns have encrypted target columns ready. This is part of ---! the configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if migration succeeds ---! @throws Exception if encryption already in progress ---! @throws Exception if no pending configuration exists ---! @throws Exception if configured columns lack encrypted targets ---! ---! @example ---! -- Manually migrate configuration (normally done automatically) ---! SELECT eql_v2.migrate_config(); ---! ---! @see eql_v2.activate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.migrate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - RAISE EXCEPTION 'An encryption is already in progress'; - END IF; - - IF NOT EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - RAISE EXCEPTION 'No pending configuration exists to encrypt'; - END IF; - - IF NOT eql_v2.ready_for_encryption() THEN - RAISE EXCEPTION 'Some pending columns do not have an encrypted target'; - END IF; - - UPDATE public.eql_v2_configuration SET state = 'encrypting' WHERE state = 'pending'; - RETURN true; - END; -$$ LANGUAGE plpgsql; - ---! @brief Activate encrypting configuration ---! ---! Transitions the encrypting configuration to active state, making it the ---! current operational configuration. Marks previous active configuration as ---! inactive. Final step in configuration lifecycle: pending → encrypting → active. ---! ---! @return Boolean True if activation succeeds ---! @throws Exception if no encrypting configuration exists to activate ---! ---! @example ---! -- Manually activate configuration (normally done automatically) ---! SELECT eql_v2.activate_config(); ---! ---! @see eql_v2.migrate_config ---! @see eql_v2.add_column -CREATE FUNCTION eql_v2.activate_config() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'encrypting') THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'encrypting'; - RETURN true; - ELSE - RAISE EXCEPTION 'No encrypting configuration exists to activate'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Discard pending configuration ---! ---! Deletes the pending configuration without applying changes. Use this to ---! abandon configuration changes before they are migrated and activated. ---! ---! @return Boolean True if discard succeeds ---! @throws Exception if no pending configuration exists to discard ---! ---! @example ---! -- Discard uncommitted configuration changes ---! SELECT eql_v2.discard(); ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.discard() - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF EXISTS (SELECT FROM public.eql_v2_configuration c WHERE c.state = 'pending') THEN - DELETE FROM public.eql_v2_configuration WHERE state = 'pending'; - RETURN true; - ELSE - RAISE EXCEPTION 'No pending configuration exists to discard'; - END IF; - END; -$$ LANGUAGE plpgsql; - ---! @brief Configure a column for encryption ---! ---! Adds a column to the encryption configuration, making it eligible for ---! encrypted storage and search indexes. Creates or updates pending configuration, ---! adds encrypted constraint, then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to encrypt ---! @param cast_as Text PostgreSQL type to cast decrypted values (default: 'text') ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if column already configured for encryption ---! ---! @example ---! -- Configure email column for encryption ---! SELECT eql_v2.add_column('users', 'email', 'text'); ---! ---! -- Configure age column with integer casting ---! SELECT eql_v2.add_column('users', 'age', 'int'); ---! ---! @see eql_v2.add_search_config ---! @see eql_v2.remove_column -CREATE FUNCTION eql_v2.add_column(table_name text, column_name text, cast_as text DEFAULT 'text', migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- set default config - SELECT eql_v2.config_default(_config) INTO _config; - - -- if index exists - IF _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'Config exists for column: % %', table_name, column_name; - END IF; - - SELECT eql_v2.config_add_table(table_name, _config) INTO _config; - - SELECT eql_v2.config_add_column(table_name, column_name, _config) INTO _config; - - SELECT eql_v2.config_add_cast(table_name, column_name, cast_as, _config) INTO _config; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO UPDATE - SET data = _config; - - IF NOT migrating THEN - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - - PERFORM eql_v2.add_encrypted_constraint(table_name, column_name); - - -- exeunt - RETURN _config; - END; -$$ LANGUAGE plpgsql; - ---! @brief Remove a column from encryption configuration ---! ---! Removes a column from the encryption configuration, including all associated ---! search indexes. Removes encrypted constraint, updates pending configuration, ---! then migrates and activates unless migrating flag is set. ---! ---! @param table_name Text Name of the table containing the column ---! @param column_name Text Name of the column to remove ---! @param migrating Boolean Skip auto-migration if true (default: false) ---! @return JSONB Updated configuration object ---! @throws Exception if no active or pending configuration exists ---! @throws Exception if table is not configured ---! @throws Exception if column is not configured ---! ---! @example ---! -- Remove email column from encryption ---! SELECT eql_v2.remove_column('users', 'email'); ---! ---! @see eql_v2.add_column ---! @see eql_v2.remove_search_config -CREATE FUNCTION eql_v2.remove_column(table_name text, column_name text, migrating boolean DEFAULT false) - RETURNS jsonb - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - key text; - _config jsonb; - BEGIN - -- set the active config - SELECT data INTO _config FROM public.eql_v2_configuration WHERE state = 'active' OR state = 'pending' ORDER BY state DESC; - - -- if no config - IF _config IS NULL THEN - RAISE EXCEPTION 'No active or pending configuration exists'; - END IF; - - -- if the table doesn't exist - IF NOT _config #> array['tables'] ? table_name THEN - RAISE EXCEPTION 'No configuration exists for table: %', table_name; - END IF; - - -- if the column does not exist - IF NOT _config #> array['tables', table_name] ? column_name THEN - RAISE EXCEPTION 'No configuration exists for column: % %', table_name, column_name; - END IF; - - -- create a new pending record if we don't have one - INSERT INTO public.eql_v2_configuration (state, data) VALUES ('pending', _config) - ON CONFLICT (state) - WHERE state = 'pending' - DO NOTHING; - - -- remove the column - SELECT _config #- array['tables', table_name, column_name] INTO _config; - - -- if table is now empty, remove the table - IF _config #> array['tables', table_name] = '{}' THEN - SELECT _config #- array['tables', table_name] INTO _config; - END IF; - - PERFORM eql_v2.remove_encrypted_constraint(table_name, column_name); - - -- update the config (even if empty) and activate - UPDATE public.eql_v2_configuration SET data = _config WHERE state = 'pending'; - - IF NOT migrating THEN - -- For empty configs, skip migration validation and directly activate - IF _config #> array['tables'] = '{}' THEN - UPDATE public.eql_v2_configuration SET state = 'inactive' WHERE state = 'active'; - UPDATE public.eql_v2_configuration SET state = 'active' WHERE state = 'pending'; - ELSE - PERFORM eql_v2.migrate_config(); - PERFORM eql_v2.activate_config(); - END IF; - END IF; - - -- exeunt - RETURN _config; - - END; -$$ LANGUAGE plpgsql; - ---! @brief Reload configuration from CipherStash Proxy ---! ---! Placeholder function for reloading configuration from the CipherStash Proxy. ---! Currently returns NULL without side effects. ---! ---! @return Void ---! ---! @note This function may be used for configuration synchronization in future versions -CREATE FUNCTION eql_v2.reload_config() - RETURNS void -LANGUAGE sql STRICT PARALLEL SAFE -BEGIN ATOMIC - RETURN NULL; -END; - ---! @brief Query encryption configuration in tabular format ---! ---! Returns the active encryption configuration as a table for easier querying ---! and filtering. Shows all configured tables, columns, cast types, and indexes. ---! ---! @return TABLE Contains configuration state, relation name, column name, cast type, and indexes ---! ---! @example ---! -- View all encrypted columns ---! SELECT * FROM eql_v2.config(); ---! ---! -- Find all columns with match indexes ---! SELECT relation, col_name FROM eql_v2.config() ---! WHERE indexes ? 'match'; ---! ---! @see eql_v2.add_column ---! @see eql_v2.add_search_config -CREATE FUNCTION eql_v2.config() RETURNS TABLE ( - state eql_v2_configuration_state, - relation text, - col_name text, - decrypts_as text, - indexes jsonb -) - SET search_path = pg_catalog, extensions, public -AS $$ -BEGIN - RETURN QUERY - WITH tables AS ( - SELECT cfg.state, tables.key AS table, tables.value AS tbl_config - FROM public.eql_v2_configuration cfg, jsonb_each(data->'tables') tables - WHERE cfg.data->>'v' = '1' - ) - SELECT - tables.state, - tables.table, - column_config.key, - COALESCE(column_config.value->>'plaintext_type', column_config.value->>'cast_as'), - column_config.value->'indexes' - FROM tables, jsonb_each(tables.tbl_config) column_config; -END; -$$ LANGUAGE plpgsql; - ---! @file config/constraints.sql ---! @brief Configuration validation functions and constraints ---! ---! Provides CHECK constraint functions to validate encryption configuration structure. ---! Ensures configurations have required fields (version, tables) and valid values ---! for index types and cast types before being stored. ---! ---! @see config/tables.sql where constraints are applied - - ---! @brief Extract index type names from configuration ---! @internal ---! ---! Helper function that extracts all index type names from the configuration's ---! 'indexes' sections across all tables and columns. ---! ---! @param jsonb Configuration data to extract from ---! @return SETOF text Index type names (e.g., 'match', 'ore', 'unique', 'ste_vec') ---! ---! @note Used by config_check_indexes for validation ---! @see eql_v2.config_check_indexes -CREATE FUNCTION eql_v2.config_get_indexes(val jsonb) - RETURNS SETOF text - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -BEGIN ATOMIC - SELECT jsonb_object_keys(jsonb_path_query(val,'$.tables.*.*.indexes')); -END; - - ---! @brief Validate index types in configuration ---! @internal ---! ---! Checks that all index types specified in the configuration are valid. ---! Valid index types are: match, ore, ope, unique, ste_vec. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all index types are valid ---! @throws Exception if any invalid index type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @see eql_v2.config_get_indexes -CREATE FUNCTION eql_v2.config_check_indexes(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - - IF (SELECT EXISTS (SELECT eql_v2.config_get_indexes(val))) THEN - IF (SELECT bool_and(index = ANY('{match, ore, ope, unique, ste_vec}')) FROM eql_v2.config_get_indexes(val) AS index) THEN - RETURN true; - END IF; - RAISE 'Configuration has an invalid index (%). Index should be one of {match, ore, ope, unique, ste_vec}', val; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate cast types in configuration ---! @internal ---! ---! Checks that all 'cast_as' and 'plaintext_type' types specified in the configuration are valid. ---! Valid cast types are: text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if all cast types are valid or no cast types specified ---! @throws Exception if any invalid cast type found ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Empty configurations (no cast_as/plaintext_type fields) are valid ---! @note Cast type names are EQL's internal representations, not PostgreSQL native types ---! @note 'plaintext_type' is accepted as a canonical alias for 'cast_as' -CREATE FUNCTION eql_v2.config_check_cast(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_types text[] := '{text, int, small_int, big_int, real, double, boolean, date, jsonb, json, float, decimal, timestamp}'; - BEGIN - -- Validate cast_as fields - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as'))) THEN - IF NOT (SELECT bool_and(cast_as = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.cast_as')) AS cast_as) casts) THEN - RAISE 'Configuration has an invalid cast_as (%). Cast should be one of %', val, _valid_types; - END IF; - END IF; - - -- Validate plaintext_type fields (canonical alias for cast_as) - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type'))) THEN - IF NOT (SELECT bool_and(pt = ANY(_valid_types)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.plaintext_type')) AS pt) types) THEN - RAISE 'Configuration has an invalid plaintext_type (%). Type should be one of %', val, _valid_types; - END IF; - END IF; - - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate tables field presence ---! @internal ---! ---! Ensures the configuration has a 'tables' field, which is required ---! to specify which database tables contain encrypted columns. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'tables' field exists ---! @throws Exception if 'tables' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_tables(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'tables') THEN - RETURN true; - END IF; - RAISE 'Configuration missing tables (tables) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate version field presence ---! @internal ---! ---! Ensures the configuration has a 'v' (version) field, which tracks ---! the configuration format version. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if 'v' field exists ---! @throws Exception if 'v' field is missing ---! ---! @note Used in CHECK constraint on eql_v2_configuration table -CREATE FUNCTION eql_v2.config_check_version(val jsonb) - RETURNS boolean - SET search_path = pg_catalog, extensions, public -AS $$ - BEGIN - IF (val ? 'v') THEN - RETURN true; - END IF; - RAISE 'Configuration missing version (v) field: %', val; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Validate ste_vec index mode option ---! @internal ---! ---! Checks that the optional `mode` field on `ste_vec` index configurations is ---! one of the recognised values. Valid modes are: standard, compat. ---! Configurations without a `mode` field (the default) pass unconditionally. ---! ---! @param jsonb Configuration data to validate ---! @return boolean True if every ste_vec mode is valid, or none are set ---! @throws Exception if any ste_vec.mode value is not in the allowed set ---! ---! @note Used in CHECK constraint on eql_v2_configuration table ---! @note Mode is optional — only configurations that set it are validated -CREATE FUNCTION eql_v2.config_check_ste_vec_mode(val jsonb) - RETURNS BOOLEAN - IMMUTABLE STRICT PARALLEL SAFE - SET search_path = pg_catalog, extensions, public -AS $$ - DECLARE - _valid_modes text[] := '{standard, compat}'; - BEGIN - IF EXISTS (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode'))) THEN - IF NOT (SELECT bool_and(mode = ANY(_valid_modes)) - FROM (SELECT jsonb_array_elements_text(jsonb_path_query_array(val, '$.tables.*.*.indexes.ste_vec.mode')) AS mode) modes) THEN - RAISE 'Configuration has an invalid ste_vec mode (%). Mode should be one of %', val, _valid_modes; - END IF; - END IF; - RETURN true; - END; -$$ LANGUAGE plpgsql; - - ---! @brief Drop existing data validation constraint if present ---! @note Allows constraint to be recreated during upgrades -ALTER TABLE public.eql_v2_configuration DROP CONSTRAINT IF EXISTS eql_v2_configuration_data_check; - - ---! @brief Comprehensive configuration data validation ---! ---! CHECK constraint that validates all aspects of configuration data: ---! - Version field presence ---! - Tables field presence ---! - Valid cast_as types ---! - Valid index types ---! - Valid ste_vec mode (when set) ---! ---! @note Combines all config_check_* validation functions ---! @see eql_v2.config_check_version ---! @see eql_v2.config_check_tables ---! @see eql_v2.config_check_cast ---! @see eql_v2.config_check_indexes ---! @see eql_v2.config_check_ste_vec_mode -ALTER TABLE public.eql_v2_configuration - ADD CONSTRAINT eql_v2_configuration_data_check CHECK ( - eql_v2.config_check_version(data) AND - eql_v2.config_check_tables(data) AND - eql_v2.config_check_cast(data) AND - eql_v2.config_check_indexes(data) AND - eql_v2.config_check_ste_vec_mode(data) -); - - ---! @file pin_search_path.sql ---! @brief Post-install: pin search_path on every eql_v2.* function ---! ---! This file is appended verbatim by `tasks/build.sh` to the end of every ---! release variant (main, supabase, protect/stack), AFTER all `src/**/*.sql` ---! files have been concatenated. It lives outside `src/` so it stays out of ---! the dependency graph entirely — each variant has a different leaf set ---! (supabase excludes `**/*operator_class.sql`; protect excludes `src/config/*` ---! and `src/encryptindex/*`), and threading REQUIREs to be ordered last in ---! every variant simultaneously is fragile. ---! ---! Iterates over functions in the `eql_v2` schema and applies a fixed ---! `search_path` via `ALTER FUNCTION ... SET search_path = ...`. This is the ---! only way to satisfy Supabase splinter's `function_search_path_mutable` ---! lint, which checks `pg_proc.proconfig` directly. ---! ---! @note A SET clause disables PostgreSQL's SQL-function inlining (see ---! inline_function() in src/backend/optimizer/util/clauses.c). For most ---! eql_v2 helpers this is irrelevant. The exceptions are wrappers that ---! must inline to expose `eql_v2.jsonb_array(col) @> ...` to the planner ---! so the GIN index on `jsonb_array(e)` can be matched. Those are ---! deliberately skipped here and allowlisted in `tasks/test/splinter.sh`. ---! ---! @see tasks/test/splinter.sh ---! @see tasks/build.sh - -DO $$ -DECLARE - fn_oid oid; - inline_critical_oids oid[]; - enc_oid oid; - jsonb_oid oid; - text_oid oid; - entry_oid oid; -BEGIN - -- Resolve type oids without depending on caller search_path. The encrypted - -- composite type is created in `public`; jsonb / text are in `pg_catalog`; - -- the ste_vec_entry DOMAIN lives in `eql_v2`. - SELECT t.oid INTO enc_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'public' AND t.typname = 'eql_v2_encrypted'; - - IF enc_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type public.eql_v2_encrypted not found — ' - 'this script must run after all EQL src/**/*.sql files have been loaded'; - END IF; - - SELECT t.oid INTO jsonb_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'jsonb'; - - IF jsonb_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.jsonb not found'; - END IF; - - SELECT t.oid INTO text_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'text'; - - IF text_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type pg_catalog.text not found'; - END IF; - - SELECT t.oid INTO entry_oid - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'ste_vec_entry'; - - IF entry_oid IS NULL THEN - RAISE EXCEPTION 'pin_search_path: type eql_v2.ste_vec_entry not found'; - END IF; - - -- Wrappers that must remain inlinable for functional-index matching. - -- Verified empirically: with SET, EXPLAIN drops to Seq Scan; without, - -- it uses Bitmap Index Scan / Index Scan. - -- - -- Phase 1 operator inlining (#193): `=`, `<>`, `~~`, `~~*`, `@>`, `<@` - -- on `eql_v2_encrypted` and the cross-type (encrypted, jsonb) / - -- (jsonb, encrypted) overloads emitted by ORMs that bind parameters - -- as jsonb (Drizzle, PostgREST, encryptedSupabase). The implementation - -- functions reduce to `extractor(a) op extractor(b)` and must inline - -- to match the documented functional indexes - -- (`eql_v2.hmac_256(col)`, `eql_v2.bloom_filter(col)`, - -- `eql_v2.ste_vec(col)`). - -- - -- For `~~` / `~~*` the planner must inline two layers — the operator - -- function `eql_v2."~~"` and the helper `eql_v2.like` / `eql_v2.ilike` - -- — to reach the canonical `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)` - -- form that the documented functional index matches. The helpers are - -- allowlisted alongside the operator wrappers below; pinning either - -- layer breaks the chain and reverts to Seq Scan. - -- - -- Note: pg_proc.proargtypes is an oidvector with 0-based bounds, so we - -- compare elements individually rather than using array equality (which - -- requires matching bounds, not just contents). - SELECT pg_catalog.array_agg(p.oid) INTO inline_critical_oids - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - AND ( - -- Same-type (encrypted, encrypted) operators that must inline. - -- `like`/`ilike` are the SQL helpers that `~~`/`~~*` delegate to; - -- both layers must inline to reach `bloom_filter(a) @> bloom_filter(b)`. - -- `<`, `<=`, `>`, `>=` inline to `ore_block_u64_8_256(a) op - -- ore_block_u64_8_256(b)`; they must reach the functional ORE index - -- expression `eql_v2.ore_block_u64_8_256(col)` for bare range - -- queries to engage Index Scan. - (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', '@>', '<@', - 'jsonb_contains', 'jsonb_contained_by', - 'like', 'ilike') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = enc_oid) - -- Cross-type (encrypted, jsonb). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = enc_oid AND p.proargtypes[1] = jsonb_oid) - -- Cross-type (jsonb, encrypted). - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - '~~', '~~*', - 'jsonb_contains', 'jsonb_contained_by') - AND p.proargtypes[0] = jsonb_oid AND p.proargtypes[1] = enc_oid) - -- Root-level HMAC extractor (#205): all 1-arg overloads are now - -- inlinable SQL. Must stay unpinned so the planner can fold extractor - -- calls inside the inlined equality operator bodies into the calling - -- query, preserving the functional-index match. - OR (p.pronargs = 1 - AND p.proname = 'hmac_256' - AND (p.proargtypes[0] = enc_oid OR p.proargtypes[0] = jsonb_oid)) - -- Field-level JSONB extractors (#205): inlinable SQL replacements for - -- the previous plpgsql bodies. Inlining lets the planner fold the - -- `jsonb_array_elements(...) WHERE elem->>'s' = selector` body into - -- the calling query, eliminating per-row function call overhead on - -- large ste_vec scans. - OR (p.pronargs = 2 - AND p.proname IN ('jsonb_path_query', - 'jsonb_path_query_first', - 'jsonb_path_exists')) - -- Inner ORE-block comparison helpers backing the `<`, `<=`, `>`, `>=` - -- operators on `eql_v2.ore_block_u64_8_256`. The outer operators on - -- `eql_v2_encrypted` inline to `ore_block(a) <op> ore_block(b)`, and - -- PG only carries the inlined form through to index matching if the - -- inner operator function is also inlinable (no SET, IMMUTABLE). - -- Pinning these would prevent the planner from structurally matching - -- predicates against a functional `eql_v2.ore_block_u64_8_256(col)` - -- index. The inner functions are deterministic comparisons of - -- composite type bytes, declared IMMUTABLE STRICT PARALLEL SAFE. - OR (p.pronargs = 2 - AND p.proname IN ('ore_block_u64_8_256_eq', 'ore_block_u64_8_256_neq', - 'ore_block_u64_8_256_lt', 'ore_block_u64_8_256_lte', - 'ore_block_u64_8_256_gt', 'ore_block_u64_8_256_gte')) - -- Hash operator class FUNCTION 1: called once per row by HashAggregate, - -- hash joins, DISTINCT. Inlinable SQL avoids the per-row plpgsql - -- interpreter overhead — without this, `GROUP BY value` on - -- `eql_v2_encrypted` at 1M rows degrades super-linearly because the - -- plpgsql cost compounds with HashAggregate work_mem spillage. - OR (p.pronargs = 1 - AND p.proname = 'hash_encrypted' - AND p.proargtypes[0] = enc_oid) - -- Consolidated ORE-CLLW extractor (U-006). Inlinable SQL — pinning - -- would silently undo it and prevent the planner from folding - -- `eql_v2.ore_cllw(col)` calls into the calling query. The - -- `compare_ore_cllw_term` comparator stays plpgsql by design (per-byte - -- protocol can't be expressed as a single inlinable SELECT), so it is - -- NOT on this list. The (jsonb) form is a RHS-parameter helper for - -- comparisons against literal jsonb; the (eql_v2.ste_vec_entry) form - -- is the typed extractor for the result of `col -> '<selector>'`. - OR (p.pronargs = 1 - AND p.proname IN ('ore_cllw', 'has_ore_cllw') - AND (p.proargtypes[0] = jsonb_oid OR p.proargtypes[0] = entry_oid)) - -- Typed HMAC extractor on a ste_vec entry (#219 strict separation). - -- Same rationale as `ore_cllw(ste_vec_entry)` — must inline so - -- `eql_v2.hmac_256(col -> 'sel')` folds into the calling query and - -- matches a functional hash index built on the same expression. - OR (p.pronargs = 1 - AND p.proname IN ('hmac_256', 'has_hmac_256', 'selector') - AND p.proargtypes[0] = entry_oid) - -- `eql_v2.ste_vec_entry × eql_v2.ste_vec_entry` operators (#219). - -- Inline to `hmac_256(a) = hmac_256(b)` (equality) or - -- `ore_cllw(a) <op> ore_cllw(b)` (ordering); both chains must remain - -- unpinned for functional-index match through extractor form. - OR (p.pronargs = 2 - AND p.proname IN ('=', '<>', '<', '<=', '>', '>=', - 'eq', 'neq', 'lt', 'lte', 'gt', 'gte') - AND p.proargtypes[0] = entry_oid AND p.proargtypes[1] = entry_oid) - -- Inner ORE-CLLW comparison helpers backing the `<`, `<=`, `=`, - -- `>=`, `>`, `<>` operators on `eql_v2.ore_cllw` (the composite - -- type, registered via `eql_v2.ore_cllw_ops` opclass — #221). Same - -- precedent as the `ore_block_u64_8_256_*` helpers above: PG only - -- carries the inlined operator wrapper through to functional-index - -- match if the inner backing function is also inlinable. Pinning - -- these would break the index match for `ORDER BY eql_v2.ore_cllw - -- (value -> '<selector>'::text)` and the matching `WHERE` form. - OR (p.pronargs = 2 - AND p.proname IN ('ore_cllw_eq', 'ore_cllw_neq', - 'ore_cllw_lt', 'ore_cllw_lte', - 'ore_cllw_gt', 'ore_cllw_gte')) - -- `->` selector lookup: inlinable SQL post the type flip - -- (returns `eql_v2.ste_vec_entry`). Must stay unpinned so the - -- planner can fold `col -> '<selector>'` into the calling query - -- — without this, the chained recipe - -- `WHERE col -> 'sel' = $1::ste_vec_entry` would not match a - -- functional hash index on `eql_v2.eq_term(col -> 'sel')`. - OR (p.proname = '->' - AND p.pronargs = 2 - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = text_oid - OR p.proargtypes[1] = enc_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'pg_catalog' AND t.typname = 'int4'))) - -- XOR-aware equality term extractor on a ste_vec entry. Must - -- inline so `eql_v2.eq_term(col -> 'sel')` folds into the - -- calling query and matches a functional hash index built on - -- the same expression. - OR (p.pronargs = 1 - AND p.proname = 'eq_term' - AND p.proargtypes[0] = entry_oid) - -- Type-safe `@>` / `<@` overloads with typed needles - -- (`stevec_query`, `ste_vec_entry`). Inline to the existing - -- `ste_vec_contains` machinery — must stay unpinned to engage - -- the GIN index on `eql_v2.ste_vec(col)` structurally for - -- bare-form containment. - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[0] = enc_oid - AND (p.proargtypes[1] = entry_oid - OR p.proargtypes[1] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - OR (p.pronargs = 2 - AND p.proname IN ('@>', '<@') - AND p.proargtypes[1] = enc_oid - AND (p.proargtypes[0] = entry_oid - OR p.proargtypes[0] = (SELECT t.oid FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace - WHERE n.nspname = 'eql_v2' AND t.typname = 'stevec_query'))) - ); - - FOR fn_oid IN - SELECT p.oid - FROM pg_catalog.pg_proc p - JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname = 'eql_v2' - -- Only normal functions ('f') and window functions ('w') accept - -- ALTER FUNCTION ... SET. Aggregates ('a') would be rejected by - -- ALTER ROUTINE/FUNCTION, and procedures ('p') would need ALTER - -- PROCEDURE. The 3 affected aggregates (min, max, grouped_value) - -- are allowlisted in splinter. - AND p.prokind IN ('f', 'w') - AND NOT EXISTS ( - SELECT 1 FROM pg_catalog.unnest(coalesce(p.proconfig, '{}'::text[])) c - WHERE c LIKE 'search_path=%' - ) - AND NOT (p.oid = ANY (coalesce(inline_critical_oids, '{}'::oid[]))) - LOOP - -- oid::regprocedure renders as `schema.name(argtype, argtype)` and is a - -- valid target for ALTER FUNCTION regardless of caller search_path. - EXECUTE pg_catalog.format( - 'ALTER FUNCTION %s SET search_path = pg_catalog, extensions, public', - fn_oid::regprocedure - ); - END LOOP; -END $$; diff --git a/packages/cli/tests/e2e/command-help.e2e.test.ts b/packages/cli/tests/e2e/command-help.e2e.test.ts index 0134ebec3..411dcf3e9 100644 --- a/packages/cli/tests/e2e/command-help.e2e.test.ts +++ b/packages/cli/tests/e2e/command-help.e2e.test.ts @@ -40,7 +40,8 @@ describe('per-command --help', () => { }) expect(r.exitCode).toBe(0) expect(r.output).toContain('Usage: npx stash eql install [options]') - expect(r.output).toContain('--eql-version <2|3>') + expect(r.output).not.toContain('--eql-version') + expect(r.output).not.toContain('--latest') expect(r.output).toContain('Also settable via DATABASE_URL.') }) diff --git a/packages/cli/tests/e2e/smoke.e2e.test.ts b/packages/cli/tests/e2e/smoke.e2e.test.ts index d2eecbcc1..febcf3b56 100644 --- a/packages/cli/tests/e2e/smoke.e2e.test.ts +++ b/packages/cli/tests/e2e/smoke.e2e.test.ts @@ -117,9 +117,8 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('bogus-sub') }) - // `--migration` without `--supabase` fails flag validation before any I/O - // or prompt, so these two cases can observe the install entry path - // deterministically without a database. + // The retired `--migration` flag fails before any I/O or prompt, so these + // cases can observe the install entry path deterministically without a DB. it('db install still works as a deprecated alias and warns', async () => { const r = render(['db', 'install', '--migration']) const { exitCode } = await r.exit @@ -128,7 +127,8 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('stash db install" is deprecated') expect(r.output).toContain('eql install" instead') // The alias reaches the real install command (its flag validation ran). - expect(r.output).toContain('requires `--supabase`') + expect(r.output).toContain('eql install --migration` has been removed') + expect(r.output).toContain('eql migration --drizzle') }) it('eql install routes to the install command without a deprecation warning', async () => { @@ -136,7 +136,8 @@ describe('stash CLI — non-interactive smoke', () => { const { exitCode } = await r.exit expect(exitCode).toBe(1) expect(r.output).not.toContain('is deprecated') - expect(r.output).toContain('requires `--supabase`') + expect(r.output).toContain('eql install --migration` has been removed') + expect(r.output).toContain('eql migration --drizzle') }) it('db migrate is a stub that exits 0 with a "not yet implemented" warning', async () => { diff --git a/packages/cli/tests/e2e/v2-retirement.e2e.test.ts b/packages/cli/tests/e2e/v2-retirement.e2e.test.ts new file mode 100644 index 000000000..1b5aad7da --- /dev/null +++ b/packages/cli/tests/e2e/v2-retirement.e2e.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, it } from 'vitest' +import { runPiped } from '../helpers/spawn-piped.js' + +const output = (result: { stdout: string; stderr: string }) => + `${result.stdout}\n${result.stderr}` + +describe('retired EQL v2 CLI surface', () => { + it('rejects v2 installation before database access and links recovery SQL', async () => { + const result = await runPiped(['eql', 'install', '--eql-version', '2']) + + expect(result.exitCode).toBe(1) + expect(output(result)).toContain( + 'https://github.com/cipherstash/encrypt-query-language/releases/tag/eql-2.3.1', + ) + }) + + it('rejects the obsolete operator-family install flag before database access', async () => { + const result = await runPiped([ + 'eql', + 'install', + '--exclude-operator-family', + ]) + + expect(result.exitCode).toBe(1) + expect(output(result)).toMatch(/self-adapts/i) + }) + + it.each([ + [['db', 'push'], 'db push'], + [['db', 'activate'], 'db activate'], + [['encrypt', 'cutover'], 'encrypt cutover'], + ] as const)('rejects removed `%s` routing', async (args, command) => { + const result = await runPiped([...args]) + + expect(result.exitCode).toBe(1) + expect(output(result)).toContain(command) + expect(output(result)).toMatch(/removed/i) + }) +}) diff --git a/packages/cli/tsup.config.ts b/packages/cli/tsup.config.ts index 12aa690b8..feb085885 100644 --- a/packages/cli/tsup.config.ts +++ b/packages/cli/tsup.config.ts @@ -79,8 +79,6 @@ export default defineConfig([ options.define = { ...options.define, ...buildDefines } }, onSuccess: async () => { - // Copy bundled SQL files into dist so they ship with the package - cpSync('src/sql', 'dist/sql', { recursive: true }) // Skills live at the monorepo root and ship inside the CLI tarball so // `stash init` can copy them into the user's `.claude/skills/` or // `.codex/skills/` directory at handoff time. Mirror of diff --git a/packages/migrate/README.md b/packages/migrate/README.md index bd41337ff..7c9355580 100644 --- a/packages/migrate/README.md +++ b/packages/migrate/README.md @@ -1,35 +1,29 @@ # @cipherstash/migrate -Primitives for migrating existing plaintext columns to CipherStash encrypted columns (EQL v2 `eql_v2_encrypted` or the concrete EQL v3 `eql_v3_*` domains) in production Postgres databases, safely and resumably. +Primitives for safely and resumably migrating existing plaintext columns to concrete EQL v3 `eql_v3_*` domains in production PostgreSQL databases. -Backs the `stash encrypt` CLI command group, but also exported for direct use — embed `runBackfill()` in your own worker or cron job when you'd rather not pipe gigabytes through a CLI process. +The package backs `stash encrypt backfill` and `stash encrypt drop`. You can also embed `runBackfill()` in a worker or cron job when a large migration should not run inside a CLI process. ## Lifecycle -Each column walks through these phases — the ladder depends on the column's EQL version, and detection is one-sided: `detectColumnEqlVersion` recognises an `eql_v3_*` Postgres domain as **v3**, and everything else as *unknown* (`null`). The v2 ladder is the fallback for an unknown column, not a detection result: - ```text -EQL v2: schema-added → dual-writing → backfilling → backfilled → cut-over → dropped -EQL v3: schema-added → dual-writing → backfilling → backfilled ————————————→ dropped +schema-added → dual-writing → backfilling → backfilled → dropped ``` -State is tracked in an append-only `cipherstash.cs_migrations` table installed by `stash eql install`. +EQL v3 has no configuration table and no rename cut-over. The application switches to the encrypted column by name after backfill, then drops the original plaintext column after verifying coverage. State is tracked in the append-only `cipherstash.cs_migrations` table installed by `stash eql install`. -- **EQL v2** additionally keeps its intent (indexes, cast_as) in `eql_v2_configuration` so CipherStash Proxy works against the same database, and finishes with a **cut-over**: `eql_v2.rename_encrypted_columns()` swaps `<col>_encrypted` into place (`<col>` becomes `<col>_plaintext`) alongside a config promotion. -- **EQL v3** has **no configuration table and no cut-over** — each column's domain type encodes its own configuration. The v3 types are *self-describing*, so tooling resolves encrypted columns from the domain types themselves; the `<col>_encrypted` naming is a convention only, never enforced or relied upon (`resolveEncryptedColumn`). The application switches to the encrypted column *by name*, and the original plaintext `<col>` is dropped once verified: `stash encrypt drop` refuses to generate the migration while any row still has the plaintext set and the encrypted column NULL (`countUnencrypted`); the concrete `eql_v3_*` domain's CHECK constraint guarantees every non-null value is a valid v3 envelope. +State readers still accept the legacy `cut_over` event and its `cut-over` phase. Manifest readers retain the `cut-over` target phase and `eqlVersion: 2`. Those fields are kept only so status tools can display existing migration history; this package no longer exports the EQL v2 Proxy configuration or rename primitives. ## API ```ts import { - installMigrationsSchema, appendEvent, + installMigrationsSchema, latestByColumn, progress, - runBackfill, - renameEncryptedColumns, - reloadConfig, readManifest, + runBackfill, writeManifest, } from '@cipherstash/migrate' ``` @@ -38,38 +32,23 @@ import { Creates `cipherstash.cs_migrations` idempotently. Normally called by `stash eql install`. -### `runBackfill({ db, encryptionClient, tableSchema, tableName, plaintextColumn, encryptedColumn, pkColumn, schemaColumnKey, chunkSize?, signal?, onProgress? })` - -Chunked, resumable, idempotent backfill of plaintext → encrypted. Per chunk, in a single transaction: select next page → encrypt via `client.bulkEncryptModels` → `UPDATE … FROM (VALUES …)` → `INSERT` a `backfill_checkpoint` event. Guards with `encrypted IS NULL` so re-runs never double-write. - -- `db`: a `pg.PoolClient` (the runner drives transactions on it). -- `encryptionClient`: your initialised `@cipherstash/stack` client (or anything that exposes `bulkEncryptModels(models, table)` returning `{ data } | { failure }`). For an EQL v3 column pass an `Encryption` client (from `@cipherstash/stack/v3`) — it pins the v3 wire format; the engine itself is version-agnostic and writes whatever envelope the client produces. -- `tableSchema`: the `EncryptedTable` for the target table from your encryption client file. -- `signal`: optional `AbortSignal`. If aborted between chunks, the backfill exits cleanly and leaves a resumable checkpoint. - -Returns `{ resumed, rowsProcessed, rowsTotal, completed }`. - -### `appendEvent(client, { tableName, columnName, event, phase, … })` / `progress(client, table, column)` / `latestByColumn(client)` - -Direct access to the `cs_migrations` event log. Use these if you're building your own migration UI or orchestration on top. - -### `renameEncryptedColumns(client)` / `reloadConfig(client)` +### `runBackfill(options)` -Thin wrappers around `eql_v2.rename_encrypted_columns()` (the **v2** cut-over primitive) and `eql_v2.reload_config()` (Proxy refresh hint — no-op when connected directly to Postgres). Not used in the v3 lifecycle — v3 has no rename step. +Runs a chunked, resumable, idempotent plaintext-to-encrypted backfill. Each chunk selects the next keyset page, encrypts through `bulkEncryptModels`, updates the encrypted column, and records a checkpoint in one transaction. The `encrypted IS NULL` guard makes retries converge. -### `detectColumnEqlVersion` / `resolveEncryptedColumn` / `listEncryptedColumns` / `classifyEqlDomain` +Pass an initialized EQL v3 `Encryption` client and an EQL v3 `encryptedTable`. The lower-level runner writes the envelope produced by the supplied client; the `stash encrypt backfill` command additionally verifies that the destination is an `eql_v3_*` domain before invoking it. -The EQL types are self-describing, and these are the domain-type primitives everything version-specific above branches on. `detectColumnEqlVersion(client, table, column)` inspects one column's Postgres domain type and returns `3` (an `eql_v3_*` domain) or `null` — it never returns `2`. `null` means *unknown*: a plaintext column, or a legacy `eql_v2_encrypted` one, is indistinguishable here, and callers fall through to the v2 lifecycle (a v2 column's version is carried by the manifest's recorded `eqlVersion`, if it has one). Resolution is case-exact (quoted-identifier semantics, matching the rest of the pipeline) and honours `search_path`. `resolveEncryptedColumn(client, table, plaintextColumn, hint?)` finds a plaintext column's encrypted counterpart from the domain types — an explicit hint (e.g. the manifest's recorded `encryptedColumn`) wins, then the `<col>_encrypted` convention, then the table's sole EQL column; the name is never assumed. `listEncryptedColumns` returns every EQL-domain column on a table, classified. +### State and manifest helpers -### `countEncrypted` / `countUnencrypted` +`appendEvent`, `progress`, and `latestByColumn` access the migration event log. `readManifest` and `writeManifest` manage the Zod-validated `.cipherstash/migrations.json` intent file. -Coverage counts over the live table. `countUnencrypted(client, table, plaintextColumn, encryptedColumn)` counts rows with plaintext set and ciphertext NULL — the check `stash encrypt drop` runs before generating the v3 plaintext-drop migration (a non-zero count means rows were written without dual-writes since the backfill). `countEncrypted` counts populated target-column rows (v2 verifies through `eql_v2.count_encrypted_with_active_config`, which needs the config table v3 doesn't have). Both are full-table scans — fine as one-shot verification, not per-row primitives. +### Domain and coverage helpers -### `readManifest(cwd)` / `writeManifest(manifest, cwd)` +`detectColumnEqlVersion`, `classifyEqlDomain`, `listEncryptedColumns`, and `resolveEncryptedColumn` recognize concrete EQL v3 domains and resolve the encrypted counterpart without relying on a naming convention. A legacy `eql_v2_encrypted` column returns `null`; callers must not treat that as an authorable generation. -Read/write `.cipherstash/migrations.json` — the repo-side intent declaration. Zod-validated. The manifest is optional; commands work without it but you lose the `plan` diff. +`countUnencrypted(client, table, plaintextColumn, encryptedColumn)` counts rows with plaintext set and ciphertext NULL. `stash encrypt drop` uses it before generating the v3 plaintext-drop migration. `countEncrypted` counts populated target rows. Both are full-table scans intended for one-shot verification. -## Drop-in usage in a BullMQ/Inngest worker +## Worker example ```ts import pg from 'pg' @@ -86,13 +65,12 @@ export async function handler({ signal }: { signal: AbortSignal }) { encryptionClient, tableSchema: usersTable, tableName: 'users', - schemaColumnKey: 'email', + schemaColumnKey: 'email_encrypted', plaintextColumn: 'email', encryptedColumn: 'email_encrypted', pkColumn: 'id', chunkSize: 2000, signal, - onProgress: (p) => console.log(`${p.rowsProcessed}/${p.rowsTotal}`), }) } finally { db.release() diff --git a/packages/migrate/src/__tests__/backfill-v3.integration.test.ts b/packages/migrate/src/__tests__/backfill-v3.integration.test.ts index bbd73f31e..4c426f282 100644 --- a/packages/migrate/src/__tests__/backfill-v3.integration.test.ts +++ b/packages/migrate/src/__tests__/backfill-v3.integration.test.ts @@ -10,7 +10,7 @@ * (implicit jsonb→domain cast + CHECK enforcement, the same assignment * path a real `eql_v3_*` column takes); * - `countEncrypted` (the v3 verification primitive — v3 has no - * `eql_v2.count_encrypted_with_active_config`) counts them. + * the encrypted-column coverage check counts them. * * Skipped unless `PG_TEST_URL` is set (same harness as the v2 file): * diff --git a/packages/migrate/src/__tests__/manifest.test.ts b/packages/migrate/src/__tests__/manifest.test.ts index c98bc8cd8..41cbd1837 100644 --- a/packages/migrate/src/__tests__/manifest.test.ts +++ b/packages/migrate/src/__tests__/manifest.test.ts @@ -1,4 +1,4 @@ -import { mkdtempSync, rmSync } from 'node:fs' +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import { describe, expect, it } from 'vitest' @@ -76,6 +76,44 @@ describe('manifest', () => { } }) + it('retains legacy v2 status fields for existing migration history', async () => { + const tmp = mkdtempSync(join(tmpdir(), 'cs-manifest-')) + try { + const legacy = { + version: 1, + tables: { + users: [ + { + column: 'email', + castAs: 'text', + indexes: ['unique'], + targetPhase: 'cut-over', + encryptedColumn: 'email_encrypted', + eqlVersion: 2, + }, + ], + }, + } + const file = manifestPath(tmp) + mkdirSync(join(tmp, '.cipherstash'), { recursive: true }) + writeFileSync(file, JSON.stringify(legacy), 'utf-8') + + expect(await readManifest(tmp)).toMatchObject({ + tables: { + users: [ + { + targetPhase: 'cut-over', + encryptedColumn: 'email_encrypted', + eqlVersion: 2, + }, + ], + }, + }) + } finally { + rmSync(tmp, { recursive: true, force: true }) + } + }) + it('rejects invalid index kinds', async () => { const tmp = mkdtempSync(join(tmpdir(), 'cs-manifest-')) try { diff --git a/packages/migrate/src/__tests__/v2-retirement.test.ts b/packages/migrate/src/__tests__/v2-retirement.test.ts new file mode 100644 index 000000000..8f5eb110a --- /dev/null +++ b/packages/migrate/src/__tests__/v2-retirement.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from 'vitest' +import * as migrate from '../index.js' + +describe('EQL v2 lifecycle retirement', () => { + it('does not export Proxy configuration or cutover primitives', () => { + for (const name of [ + 'activateConfig', + 'countEncryptedWithActiveConfig', + 'discardPendingConfig', + 'migrateConfig', + 'readyForEncryption', + 'reloadConfig', + 'renameEncryptedColumns', + 'selectPendingColumns', + ]) { + expect(migrate).not.toHaveProperty(name) + } + }) +}) diff --git a/packages/migrate/src/backfill.ts b/packages/migrate/src/backfill.ts index 0b1ce5c5c..d20153b5a 100644 --- a/packages/migrate/src/backfill.ts +++ b/packages/migrate/src/backfill.ts @@ -158,8 +158,8 @@ export interface BackfillOptions { */ plaintextColumn: string /** - * Physical column that receives the `eql_v2_encrypted` ciphertext JSON, - * e.g. `email_encrypted`. Must already exist (typically created by + * Physical EQL v3 domain column that receives ciphertext, e.g. + * `email_encrypted`. Must already exist (typically created by * `drizzle-kit` / a prior migration) before backfill starts. */ encryptedColumn: string diff --git a/packages/migrate/src/cursor.ts b/packages/migrate/src/cursor.ts index 4546daf4f..15b2a7eb3 100644 --- a/packages/migrate/src/cursor.ts +++ b/packages/migrate/src/cursor.ts @@ -123,10 +123,7 @@ export async function countUnencrypted( /** * Count rows whose encrypted column is populated: `encrypted IS NOT NULL`. * - * The v3 backfill-verification primitive. EQL v2 verified via - * `eql_v2.count_encrypted_with_active_config(...)`, which reads the - * `eql_v2_configuration` table — v3 has no configuration table, so the - * equivalent check is a plain count of the target column (the concrete + * Backfill verification for EQL v3 is a plain count of the target column (the concrete * `eql_v3_*` domain's CHECK constraint already guarantees every non-null * value is a valid v3 envelope). */ diff --git a/packages/migrate/src/eql.ts b/packages/migrate/src/eql.ts deleted file mode 100644 index f169442a5..000000000 --- a/packages/migrate/src/eql.ts +++ /dev/null @@ -1,166 +0,0 @@ -import type { ClientBase } from 'pg' - -/** - * Thin, typed wrappers around the EQL (Encrypt Query Language) functions - * installed by `stash eql install`. These mirror the canonical SQL API that - * CipherStash Proxy also drives, so every action we take here stays - * visible to Proxy using the same column-level config. - * - * Defined by the EQL project at - * https://github.com/cipherstash/encrypt-query-language — see - * `src/config/functions.sql` and `src/encryptindex/functions.sql` for the - * source of truth. - */ - -/** - * A column that has been registered in the `pending` EQL configuration but - * is not yet part of the `active` config. Returned by - * {@link selectPendingColumns}. - */ -export interface PendingColumn { - tableName: string - columnName: string -} - -/** - * Return columns present in the `pending` EQL config but absent (or - * different) in the `active` one. Wraps `eql_v2.select_pending_columns()`. - * Useful for showing "what's about to change" before calling - * {@link readyForEncryption} + activating the pending config. - */ -export async function selectPendingColumns( - client: ClientBase, -): Promise<PendingColumn[]> { - const result = await client.query<{ - table_name: string - column_name: string - }>('SELECT table_name, column_name FROM eql_v2.select_pending_columns()') - return result.rows.map((row) => ({ - tableName: row.table_name, - columnName: row.column_name, - })) -} - -/** - * Check EQL's precondition for activating a pending configuration: every - * pending column must have a matching `eql_v2_encrypted`-typed target - * column in the schema. Returns `true` if activation is safe. - * Wraps `eql_v2.ready_for_encryption()`. - */ -export async function readyForEncryption(client: ClientBase): Promise<boolean> { - const result = await client.query<{ ready: boolean }>( - 'SELECT eql_v2.ready_for_encryption() AS ready', - ) - return result.rows[0]?.ready === true -} - -/** - * Atomically rename every `<col>` → `<col>_plaintext` and - * `<col>_encrypted` → `<col>` across tables in the **pending** EQL config. - * Wraps `eql_v2.rename_encrypted_columns()`. - * - * This is the **cut-over primitive**: after this returns, any SQL that - * reads `<col>` transparently receives the encrypted column (decrypted on - * read by Proxy or Protect). Call inside a transaction. - * - * **Requires a pending configuration.** The underlying EQL function calls - * `select_pending_columns()` and raises `'No pending configuration exists - * to encrypt'` if none is registered. Call `db push` to register a pending - * config first; if no rename targets are present in the diff, the loop - * inside `rename_encrypted_columns()` does nothing — but the function - * still requires pending to exist. - * - * Renames physical columns only — does not advance the EQL state machine. - * Pair with {@link migrateConfig} + {@link activateConfig} to finish the - * pending → encrypting → active transition. - */ -export async function renameEncryptedColumns( - client: ClientBase, -): Promise<void> { - await client.query('SELECT eql_v2.rename_encrypted_columns()') -} - -/** - * Advance the EQL state machine: `pending → encrypting`. Wraps - * `eql_v2.migrate_config()`. - * - * Throws when: - * - There is no pending configuration to migrate. - * - There is already an encrypting configuration in flight. - * - Some pending column lacks its encrypted target column (`<col>_encrypted` - * doesn't exist in the schema with `eql_v2_encrypted` UDT). - */ -export async function migrateConfig(client: ClientBase): Promise<void> { - await client.query('SELECT eql_v2.migrate_config()') -} - -/** - * Advance the EQL state machine: `encrypting → active`. Wraps - * `eql_v2.activate_config()`. Marks any prior `active` row as `inactive` - * in the same call. - * - * Throws when there is no encrypting configuration to activate. Always - * call after {@link migrateConfig} has flipped the pending row to - * encrypting (typically inside the same transaction as - * {@link renameEncryptedColumns} for cutover; or alone for non-rename - * activations like adding a new column to an existing config). - */ -export async function activateConfig(client: ClientBase): Promise<void> { - await client.query('SELECT eql_v2.activate_config()') -} - -/** - * Discard the pending configuration without applying it. Wraps - * `eql_v2.discard()`. Used by `db push` to clean up any stale pending - * before writing a new one — the state machine only allows one pending - * row at a time, and a stale pending blocks new pushes. - * - * No-op when no pending exists. - */ -export async function discardPendingConfig(client: ClientBase): Promise<void> { - // The EQL `discard()` function raises when there's no pending row — - // we want a no-op in that case, so DELETE directly. Safer than - // wrapping eql_v2.discard() in a try/catch that swallows shape errors. - await client.query( - "DELETE FROM public.eql_v2_configuration WHERE state = 'pending'", - ) -} - -/** - * Nudge Proxy to re-read its config immediately instead of waiting for its - * next 60-second refresh tick. Wraps `eql_v2.reload_config()`. - * - * **Must be executed through a CipherStash Proxy connection** — when - * connected directly to Postgres, `reload_config()` is a no-op (by design, - * per the EQL documentation). The CLI's `cutover` command accepts a - * `--proxy-url` flag and will connect to that separately to issue this. - */ -export async function reloadConfig(client: ClientBase): Promise<void> { - await client.query('SELECT eql_v2.reload_config()') -} - -/** - * Return EQL's count of rows in `<tableName>.<columnName>` whose encrypted - * payload's config version matches the currently active config. Useful as - * a cheap sanity check — 0 after a backfill generally means something's - * wrong (wrong config active, or the backfill wrote with a stale version). - * - * Wraps `eql_v2.count_encrypted_with_active_config(table, column)`. - * - * Returns `bigint` because the underlying Postgres function returns - * `BIGINT` and naively coercing to JS `number` loses precision past - * `Number.MAX_SAFE_INTEGER` — exactly the row counts large-table users - * are running this against. Callers that need a JS number can do their - * own range check. - */ -export async function countEncryptedWithActiveConfig( - client: ClientBase, - tableName: string, - columnName: string, -): Promise<bigint> { - const result = await client.query<{ count: string }>( - 'SELECT eql_v2.count_encrypted_with_active_config($1, $2) AS count', - [tableName, columnName], - ) - return BigInt(result.rows[0]?.count ?? '0') -} diff --git a/packages/migrate/src/index.ts b/packages/migrate/src/index.ts index 026b5f6e5..cb34f42da 100644 --- a/packages/migrate/src/index.ts +++ b/packages/migrate/src/index.ts @@ -1,23 +1,20 @@ /** * `@cipherstash/migrate` — primitives for migrating existing plaintext - * columns to EQL-encrypted columns (`eql_v2_encrypted` or the - * self-describing `eql_v3_*` domains) in production Postgres databases. + * columns to self-describing EQL v3 domains in production Postgres databases. * * Powers the `stash encrypt` CLI command group, and is usable directly * from a user's own worker/cron when they'd rather not pipe gigabytes * through a CLI process. * - * Per-column lifecycle (version-dependent — EQL v3 has no cut-over rename; - * the application switches to the encrypted column by name): + * Per-column lifecycle (the application switches to the encrypted column by + * name; there is no automated cut-over rename): * * ``` - * v2: schema-added → dual-writing → backfilling → backfilled → cut-over → dropped * v3: schema-added → dual-writing → backfilling → backfilled → dropped * ``` * * State is split across three stores on purpose: * - `.cipherstash/migrations.json` — repo-side intent ({@link Manifest}) - * - `eql_v2_configuration` — EQL intent (unchanged; Proxy's source of truth) * - `cipherstash.cs_migrations` — append-only runtime state written here * * The primary entry point is {@link runBackfill}. The state DAO @@ -41,16 +38,6 @@ export { type KeysetPageOptions, qualifyTable, } from './cursor.js' -export { - activateConfig, - countEncryptedWithActiveConfig, - discardPendingConfig, - migrateConfig, - readyForEncryption, - reloadConfig, - renameEncryptedColumns, - selectPendingColumns, -} from './eql.js' export { installMigrationsSchema, MIGRATIONS_SCHEMA_SQL } from './install.js' export { type Manifest, diff --git a/packages/migrate/src/install.ts b/packages/migrate/src/install.ts index 2a3981a4e..b29d19a6b 100644 --- a/packages/migrate/src/install.ts +++ b/packages/migrate/src/install.ts @@ -8,14 +8,8 @@ import type { ClientBase } from 'pg' * All statements are `CREATE … IF NOT EXISTS` so running the installer * multiple times or alongside an existing deployment is safe. * - * This table is intentionally kept separate from `eql_v2_configuration`: - * - That table's `data` JSONB has a strict CHECK constraint that forbids - * custom metadata, so we cannot stuff backfill progress into it. - * - Its `state` enum is global (`pending`/`encrypting`/`active`/`inactive` - * — only one of the first three at a time), which cannot represent - * multiple columns in different phases simultaneously. - * - Checkpoint writes during backfill would collide with Proxy's 60s - * config refresh cycle. + * This table is intentionally independent of the installed EQL schemas, so + * checkpoint writes never depend on query-configuration state. */ export const MIGRATIONS_SCHEMA_SQL = ` CREATE SCHEMA IF NOT EXISTS cipherstash; diff --git a/packages/migrate/src/manifest.ts b/packages/migrate/src/manifest.ts index 872c83f54..e67755597 100644 --- a/packages/migrate/src/manifest.ts +++ b/packages/migrate/src/manifest.ts @@ -3,8 +3,7 @@ import * as path from 'node:path' import { z } from 'zod' /** - * The four EQL index kinds recognised by Proxy. Keep in sync with the - * `indexes` CHECK constraint in `eql_v2_configuration`. + * Legacy EQL v2 index kinds retained so existing manifests remain readable. */ const IndexKind = z.enum(['unique', 'match', 'ore', 'ste_vec']) @@ -47,10 +46,8 @@ const ManifestColumnSchema = z.object({ /** Desired EQL index set. Driver of the `indexes: {…}` block in EQL config. */ indexes: z.array(IndexKind).default([]), /** - * The phase the user wants this column to reach. `cut-over` is the - * typical end state (reads transparently decrypted); advance to - * `dropped` only once you're confident the plaintext column is no - * longer needed. + * The phase the user wants this column to reach. `cut-over` is retained for + * legacy EQL v2 manifests; new EQL v3 entries target `dropped`. */ targetPhase: z .enum(['schema-added', 'dual-writing', 'backfilled', 'cut-over', 'dropped']) diff --git a/packages/migrate/src/state.ts b/packages/migrate/src/state.ts index 6910ccb52..a3f4ff2ca 100644 --- a/packages/migrate/src/state.ts +++ b/packages/migrate/src/state.ts @@ -23,14 +23,14 @@ export type MigrationEvent = /** * The per-column lifecycle phase as surfaced in status/plan output and - * driven by the `stash encrypt {backfill,cutover,drop}` commands. + * driven by the `stash encrypt {backfill,drop}` commands. * * ``` * schema-added → the <col>_encrypted column exists and is registered with EQL * dual-writing → app writes both plaintext and encrypted on inserts/updates * backfilling → runBackfill is (or has been) encrypting historical rows * backfilled → all historical rows encrypted; safe to cut over reads - * cut-over → columns renamed (via eql_v2.rename_encrypted_columns) + * cut-over → legacy EQL v2 history token (status compatibility only) * dropped → old plaintext column removed * ``` */ diff --git a/packages/migrate/src/version.ts b/packages/migrate/src/version.ts index 88c496170..36b268b79 100644 --- a/packages/migrate/src/version.ts +++ b/packages/migrate/src/version.ts @@ -1,15 +1,12 @@ import type { ClientBase } from 'pg' /** - * Which EQL generation an encrypted column belongs to. The migration lifecycle - * differs between them: v2 is driven by the `eql_v2_configuration` state machine - * (see {@link import('./eql.js')}), while v3 is domain-native — configuration - * lives in the column's own type and there is no configuration table, so its - * lifecycle is backfill-then-drop with no cut-over rename. + * Which EQL generation an encrypted column belongs to. EQL v3 is domain-native: + * configuration lives in the column's own type and its lifecycle is + * backfill-then-drop with no cut-over rename. The `2` member remains solely for + * reading legacy manifest and status records. * - * Numeric (`2 | 3`) to match the manifest's `eqlVersion` field and the CLI - * installer's `--eql-version` — one representation everywhere, no - * string↔number translation at boundaries. + * Numeric (`2 | 3`) to match the manifest's `eqlVersion` field. */ export type EqlVersion = 2 | 3 diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 4a279964b..13054ee8e 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: stash-cli -description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql install/upgrade/status`, `db validate`, `encrypt backfill/cutover/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, the credential rules for `~/.cipherstash`, and the rollout-then-cutover lifecycle. Use when setting up CipherStash EQL in a database, running any `stash` command, creating `stash.config.ts`, or rolling encryption out to production. +description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql install/upgrade/status`, `db validate`, `encrypt backfill/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, credential rules, and the staged EQL v3 rollout lifecycle. --- # CipherStash CLI (`stash`) @@ -130,7 +130,6 @@ There is **no global `--non-interactive` or `--json` flag** (and no global `--ye | Region (`auth login`, `init`) | `--region <slug>` or `STASH_REGION` | | Database URL (all `db` / `eql` / `schema` commands) | `--database-url <url>` or `DATABASE_URL` | | Agent target (`plan`, `impl`) | `--target <claude-code\|codex\|agents-md\|wizard>` | -| Proxy choice (`init`) | `--proxy` / `--no-proxy` (default) | | Dual-write confirmation (`encrypt backfill`) | `--confirm-dual-writes-deployed` | | Machine-readable output | `--json` on `status`, `manifest`, `auth login`, `auth regions` | @@ -199,7 +198,7 @@ export default defineConfig({ | Option | Required | Default | Purpose | |---|---|---|---| | `databaseUrl` | yes | — | PostgreSQL connection string | -| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db push`, `db validate`, `encrypt backfill` (`schema build` only *writes* here; `encrypt cutover`/`drop` resolve against the database) | +| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db validate` and `encrypt backfill` (`schema build` only writes here; `encrypt drop` resolves against the database) | Resolved by walking up from `process.cwd()`, like `tsconfig.json`. `stash init` scaffolds it; `stash eql install` offers to. @@ -209,29 +208,28 @@ Four explicit save-points. Each runs standalone; chain prompts make first-time s | Command | Owns | Ends with | |---|---|---| -| `stash init` | Auth, database, proxy choice, encryption client, deps, EQL install, `.cipherstash/context.json` | Default-yes prompt → chains to `stash plan` | +| `stash init` | Auth, database, encryption client, deps, EQL install, `.cipherstash/context.json` | Default-yes prompt → chains to `stash plan` | | `stash plan` | Drafts `.cipherstash/plan.md` via agent handoff. State-driven: auto-detects rollout vs. cutover. | Default-yes prompt → chains to `stash impl` | | `stash impl` | Executes the plan via agent handoff. Enforces the deploy gate. | Deploy-gate banner (rollout) or "verify state" | | `stash status` | The rollout quest log — per-column "where am I", runs in ms | — | ### `init` — scaffold -Seven mechanical steps, no agent handoff. It prompts only when it can't pick a sensible default. +Six mechanical steps, no agent handoff. It prompts only when it can't pick a sensible default. 1. **Authenticate** — silent when a valid token exists. 2. **Resolve database** — per the resolution order above; verifies the connection. -3. **Resolve proxy choice** — CipherStash Proxy or direct SDK access (the default). Stored as `usesProxy` in `context.json`. Set by `--proxy` / `--no-proxy`; non-TTY without a flag defaults to SDK. -4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. The placeholder declares one sentinel table, `__stash_placeholder__`, because `Encryption` requires at least one — replace it with the tables you actually encrypt. Until you do, the commands that read the client file — `db push`, `db validate`, and `encrypt backfill` — exit 1 and point back at it. -5. **Install dependencies** — one combined prompt for `@cipherstash/stack` and `stash`. Skipped when both are present. -6. **Install EQL** — always EQL v3. **Drizzle** projects generate a v3 install migration (the same output as `eql migration --drizzle`, including the `cs_migrations` tracking schema) so the install lands in your migration history — apply it with `drizzle-kit migrate`; requires `drizzle-kit` to be installed and configured. **Prisma Next** is skipped (it installs EQL via `prisma-next migrate`). Everything else runs `eql install` directly against the resolved database, and is skipped when EQL is already installed. -7. **Gather context** — detects available coding agents and writes `.cipherstash/context.json`. +3. **Build schema** — auto-detects Drizzle, Supabase, and Prisma Next and writes the placeholder encryption client. +4. **Install dependencies** — one combined prompt for `@cipherstash/stack` and `stash`. +5. **Install EQL** — always EQL v3. Drizzle generates `eql migration --drizzle`; Prisma Next installs through `prisma-next migrate`; other integrations install directly. +6. **Gather context** — detects available coding agents and writes `.cipherstash/context.json`. -Flags: `--supabase`, `--drizzle`, `--prisma-next`, `--proxy` / `--no-proxy`, `--region <slug>`. +Flags: `--supabase`, `--drizzle`, `--prisma-next`, `--region <slug>`. | Generated file | Purpose | |---|---| | `./src/encryption/index.ts` | Placeholder encryption client — declare encrypted columns here, or let `plan`/`impl` do it | -| `.cipherstash/context.json` | Detected facts: integration, package manager, schemas, env key names, agents, `usesProxy`. CLI-owned; never hand-edit | +| `.cipherstash/context.json` | Detected facts: integration, package manager, schemas, env key names, and agents. CLI-owned; never hand-edit | | `stash.config.ts` | Scaffolded if missing | ### `plan` — draft for review @@ -250,7 +248,7 @@ Pre-flights `.cipherstash/context.json` (errors with "Run `stash init` first" if | Detected state | Plan written | |---|---| | No `dual_writing` event recorded | **Encryption rollout** — schema-add + dual-write code. Ends at the deploy gate. | -| A column has `dual_writing` or later | **Encryption cutover** — backfill, schema rename, read-path switch, drop. Requires the rollout to be deployed. | +| A column has `dual_writing` or later | **Encryption cutover** — backfill, switch schema/query references to the EQL v3 encrypted column by name, wire the read path, then drop plaintext. Requires the rollout to be deployed. | | `--complete-rollout` passed | **Complete rollout** — schema-add through drop, no deploy gate. Needs consent: an interactive default-no confirm, or `--yes` non-interactively (without it, a non-interactive run **exits non-zero without drafting** rather than silently doing nothing). | The agent writes a machine-readable header into the plan: @@ -307,7 +305,7 @@ The split is invisible — keep running `plan` and `impl`; the CLI reads `cs_mig ### Why the split exists -There is no atomic way to replace a populated plaintext column with an encrypted one without corrupting data. The rollout phase deploys the *capability* to write encrypted values (the twin column and the dual-write code). The cutover phase deploys the *transition*: backfill historical rows, rename-swap the columns, switch the application read path through the encryption client, then drop the plaintext column. +There is no atomic way to replace a populated plaintext column with an encrypted one without corrupting data. The rollout phase deploys the *capability* to write encrypted values (the twin column and the dual-write code). The cutover phase deploys the *transition*: backfill historical rows, switch schema/query references to the EQL v3 encrypted column by name, wire the application read path through the encryption client, then drop the plaintext column. There is no rename swap. Backfill is only safe once dual-writes are running in production. Any row written *during* the backfill window must land in both columns — otherwise it stays plaintext-only and creates silent migration drift. The gate makes that precondition explicit. @@ -345,31 +343,19 @@ stash eql status #### `eql install` -Gets a project from zero to installed EQL. It loads an existing `stash.config.ts` (or offers to scaffold one), scaffolds the encryption client if missing, then auto-detects the install path: **Drizzle** generates a migration via `drizzle-kit generate --custom` (apply it with `npx drizzle-kit migrate`); **Supabase without Drizzle** prompts between a migration file and a direct install, pre-selecting migration when `supabase/migrations/` exists; otherwise it installs directly. +Gets a project from zero to a direct EQL v3 install. It loads an existing `stash.config.ts` (or offers to scaffold one), scaffolds the encryption client if missing, and applies the pinned `@cipherstash/eql` bundle. To put installation in Drizzle migration history, use `eql migration --drizzle` instead. | Flag | Description | |---|---| | `--force` | Reinstall even if EQL is present | | `--dry-run` | Show what would happen | -| `--supabase` | Supabase-compatible install (no operator families; grants `anon`, `authenticated`, `service_role`) | -| `--drizzle` | Generate a Drizzle migration (**v2 only — requires `--eql-version 2`**; for v3 use `eql migration --drizzle`). `--name`, `--out` tune it — `--name` accepts letters, numbers, `-`, `_` only; `--out` is passed to `drizzle-kit --out`, so set it to match your `drizzle.config.ts` | -| `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly. `--migration` is **v2 only — requires `--eql-version 2`**; for a v3 install as a migration use `eql migration` | -| `--migrations-dir <path>` | Supabase migrations directory (default `supabase/migrations`). **v2 only — requires `--eql-version 2`** | -| `--exclude-operator-family` | Skip operator families (non-superuser roles) | -| `--eql-version <2\|3>` | EQL generation. **Default `3`** (the native `public.eql_v3_*` domain schema — the documented approach). `2` is the legacy composite schema. | -| `--latest` | Fetch latest EQL from GitHub instead of the bundled copy (**v2 only**) | +| `--supabase` | Supabase-compatible install; grants `anon`, `authenticated`, and `service_role` | | `--database-url <url>` | One-shot install (see below) | -`--migration`, `--direct`, and `--migrations-dir` require an explicit `--supabase`; they never auto-enable it. - -**`eql install` for EQL v3 runs the direct path only.** Passing `--eql-version 3` with an explicit `--drizzle`, `--migration`, `--migrations-dir`, or `--latest` is an error. When Drizzle or a Supabase migrations directory is merely *auto-detected*, v3 falls back to a direct install and prints a notice. To get a **v3 install as a migration** (preferred for real projects), use `eql migration` (below) instead of `eql install --drizzle`. +The removed `--eql-version`, `--latest`, `--drizzle`, `--migration`, `--direct`, `--migrations-dir`, and `--exclude-operator-family` options fail clearly instead of being ignored. A request for EQL v2 points dump-recovery users to the upstream EQL 2.3.1 SQL release. New installs are EQL v3 only; its pinned bundle self-adapts when a database role cannot create the optional operator family. **`--database-url` is a one-shot.** It installs against that database and leaves the project untouched — no config is loaded, and none is scaffolded, nor is an encryption client. This lets `npx stash eql install --database-url postgres://...` run in a bare project with no CipherStash dependencies. It also means the flag always wins: loading a config could pick up a parent-directory `databaseUrl` literal and install against the wrong database. -**`eql install --supabase --migration`** writes `supabase/migrations/00000000000000_cipherstash_eql.sql`. The all-zero timestamp guarantees it runs before any user migration referencing `eql_v2_encrypted`. Apply with `supabase db reset` (local) or `supabase migration up` (remote). - -Direct installs (`--supabase --direct`) do **not** survive `supabase db reset` — the reset drops the database and replays only files in `supabase/migrations/`. Use `--migration` if you reset. - #### `eql migration` Generates an **EQL v3 install migration** for your ORM, instead of running SQL directly against the database (`eql install`). Migration-first is the preferred path: the install lands in your migration history and ships to every environment through the ORM's own migrate step. v3 only — there is no `--eql-version` here. @@ -390,17 +376,15 @@ stash eql migration --drizzle --supabase # also grant eql_v3 to anon/authentic Pass exactly one of `--drizzle` / `--prisma`. The generated migration also installs the `cs_migrations` tracking schema, so one `drizzle-kit migrate` covers everything `stash encrypt …` needs. -After writing the migration, `--drizzle` sweeps the output directory for sibling migrations containing an in-place `ALTER COLUMN … SET DATA TYPE <eql_v2_encrypted | eql_v3_*>` — drizzle-kit emits these when you change a plaintext column to an encrypted one, and Postgres rejects them (there is no cast from `text`/`numeric` to an EQL type). Each is rewritten into an `ADD COLUMN` + `DROP` + `RENAME` sequence and the rewritten files are listed. This is equivalent to DROP+ADD — it fixes the type but does **not** preserve data — so it is safe **only on an empty table**. On a populated table the new column starts NULL and the old one is dropped in the same migration, destroying the plaintext; do not run it there. Use the staged `stash encrypt` path (add → backfill → cutover → drop) instead. - -The sweep is fail-closed: it rewrites a statement only when the same directory also contains the `CREATE TABLE` or `ADD COLUMN` that declared the column, and the column is not already encrypted **in the corpus**. That is a guarantee about what the migration files say, not about the live database — the sweep never queries the database, so a column that has drifted from its migration history (for example, renamed directly by `stash encrypt cutover`, or altered by hand via psql or the Supabase dashboard) can already hold ciphertext while the corpus still describes it as plaintext. A statement it cannot place — the declaration lives in another migration directory, or the history was squashed — is listed as needing review rather than rewritten, because a column of unknown type may already hold ciphertext that the rewrite's `DROP COLUMN` would destroy. Either way, check the column's actual type in the database before applying a flagged or corpus-cleared rewrite by hand. +After writing the migration, `--drizzle` sweeps sibling migrations containing an in-place `ALTER COLUMN … SET DATA TYPE <encrypted domain>`. When the source declaration is provably plaintext, it replaces the ALTER with an `ADD COLUMN` + `DROP` + `RENAME` sequence and lists the rewritten files. This is equivalent to DROP+ADD: it is safe only on an empty table and does not preserve data, constraints, defaults, or indexes. On a populated table, do not run that rewrite; use the staged EQL v3 rollout (add an encrypted twin, dual-write, backfill, switch the application to the encrypted column by name, then drop plaintext). If the source type cannot be proven, the statement remains unchanged and the command warns that the migration directory needs review. #### `eql upgrade` -The install SQL is safe to re-run — columns and data survive — but it is not fully idempotent: it begins with `DROP SCHEMA IF EXISTS eql_v3 CASCADE`, which cascade-drops any **functional indexes** built on the `eql_v3` extractors (see `stash-indexing`). After an upgrade, recreate them: migration runners skip migrations already recorded as applied, so add a *new* migration that re-issues the `CREATE INDEX` statements (or run the DDL directly), then `ANALYZE` the affected tables. `upgrade` checks the current version, re-runs the install SQL, and reports the new one. If EQL isn't installed it points you at `eql install`. Same `--supabase`, `--exclude-operator-family`, `--eql-version`, `--latest`, `--dry-run`, `--database-url` flags. +The install SQL is safe to re-run — columns and data survive — but it cascade-drops functional indexes that depend on `eql_v3`; recreate them afterward. `upgrade` is v3-only and accepts `--supabase`, `--dry-run`, and `--database-url`. #### `eql status` -Whether EQL is installed and at which version; database permission status; whether an active encrypt config exists (relevant only to Proxy users). +Whether EQL is installed and at which version, plus database permission status. It retains read-only EQL v2/config-table diagnostics for existing deployments. ### Database @@ -435,10 +419,7 @@ For AI-guided integration that edits your existing schema files in place, prefer The database-side toolset that takes an existing plaintext column the rest of the way, **after** the rollout PR is deployed and dual-writes are live. It drives `@cipherstash/migrate`, recording every transition in `cipherstash.cs_migrations` (installed by `eql install`) and reading intent from `.cipherstash/migrations.json`. -The phase ladder depends on the column's EQL version, which the commands read off the column's **domain type** — never off the `<col>_encrypted` naming, which is a convention only. Detection is one-sided: a `public.eql_v3_*` domain is recognised as v3, and everything else (including a legacy `eql_v2_encrypted` column) falls through to the v2 ladder: - -- **EQL v3 (the default):** `schema-added → dual-writing → backfilling → backfilled → dropped`. There is no cut-over — the application switches to the encrypted column by name, then the plaintext column is dropped. -- **EQL v2:** `schema-added → dual-writing → backfilling → backfilled → cut-over → dropped`, where cut-over renames the encrypted twin into the original column name. +The authored lifecycle is EQL v3 only: `schema-added → dual-writing → backfilling → backfilled → dropped`. There is no rename cut-over — the application switches to the encrypted column by name, then the plaintext column is dropped. Status readers still display legacy v2 manifest/history fields, but mutation commands refuse an unclassified `eql_v2_encrypted` target. #### `encrypt status` / `encrypt plan` @@ -453,7 +434,7 @@ stash encrypt backfill --table users --column email --chunk-size 5000 Chunked, resumable, idempotent. Walks the table in keyset-pagination order, encrypts each chunk via `bulkEncryptModels`, and writes one `UPDATE ... FROM (VALUES ...)` per chunk in a transaction that also checkpoints to `cs_migrations`. SIGINT/SIGTERM finishes the current chunk and exits cleanly; re-running resumes. The `<col> IS NOT NULL AND <col>_encrypted IS NULL` guard makes concurrent runners and re-runs converge. -Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgres domain type and records it (plus the target phase) in `.cipherstash/migrations.json`. A column that is not a v3 domain — including a legacy `eql_v2_encrypted` one — does not classify and takes the v2 lifecycle, recording no version. On an EQL v3 column it finishes by printing the v3 next steps: switch the application to `<col>_encrypted` by name, then `stash encrypt drop` — there is no cut-over. +Backfill requires a `public.eql_v3_*` target column, records version 3 and the `dropped` target phase in `.cipherstash/migrations.json`, then prints the next steps: switch the application to the encrypted column by name and run `stash encrypt drop`. A missing, plaintext, or legacy v2 target is rejected before encryption begins. **Dual-write precondition.** The application must already write both `<col>` and `<col>_encrypted` on every insert and update. Otherwise rows written *during* the backfill land in plaintext only, silently. The first run prompts (interactive) or requires `--confirm-dual-writes-deployed` (non-interactive), then records `dual_writing`. Resumes don't re-prompt. @@ -469,29 +450,13 @@ Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgr | `--confirm-dual-writes-deployed` | Non-interactive equivalent of the prompt | | `--force` | Re-encrypt every plaintext row, including ones with existing ciphertext. Recovery path for drift. Expensive, not destructive. Flagged in the audit trail. | -#### `encrypt cutover` - -```bash -stash encrypt cutover --table users --column email -``` - -**EQL v2 only** — v3 has no cut-over: the application switches to the encrypted column by name. Running this command on a **backfilled** v3 column reports "not applicable" (exit 0) with the next step — *provided the encrypted column can be identified*. If it was resolved only by elimination (the table's one remaining EQL column, with no `encryptedColumn` recorded in `.cipherstash/migrations.json` and no `<col>_encrypted` name match), it refuses and exits 1 rather than report an outcome for a column it guessed. `.cipherstash/` is gitignored, so a clone or CI runner without that manifest can hit this on a table whose encrypted column is named unconventionally; re-run `stash encrypt backfill … --encrypted-column <name>` to record the pairing; before backfill completes it exits 1 and says to finish the backfill (never "switch now" onto a half-populated column). For v2, the preconditions are: the column is in the `backfilled` phase, **and** a pending EQL configuration exists (on a v3-only database — no `eql_v2_configuration` table — it explains that and exits 1). - -In one transaction it renames `<col>` → `<col>_plaintext` and `<col>_encrypted` → `<col>`, advances the pending config to `encrypting`, activates it, and appends a `cut_over` event. With a Proxy URL configured (`--proxy-url` or `CIPHERSTASH_PROXY_URL`) it then calls `eql_v2.reload_config()` so Proxy picks up the new shape. - -> **After cutover, `<col>` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. -> -> **Known gap (v2).** The pending-configuration precondition is satisfied by `stash db push`. SDK-only users (who otherwise never need `db push`) must therefore run it once before `encrypt cutover`. This gap is specific to the legacy v2 path and is not being decoupled — EQL v3 columns sidestep it entirely (no configuration table, no cut-over; see above), which is how [issue #585](https://github.com/cipherstash/stack/issues/585) was resolved when v3 became the default. - -Flags: `--table`, `--column`, `--proxy-url <url>`, `--migrations-dir <path>`. - #### `encrypt drop` ```bash stash encrypt drop --table users --column email ``` -Version-aware. For **EQL v2** columns in the `cut-over` phase it emits `ALTER TABLE <table> DROP COLUMN <col>_plaintext;` (the post-rename name). For **EQL v3** columns it runs from the `backfilled` phase and drops the ORIGINAL `<col>` — v3 has no rename, so make sure the application reads/writes the encrypted column first. Before generating, the v3 path **verifies live coverage** (refuses if any row still has `<col>` set with the encrypted column NULL — e.g. rows written without dual-writes since the backfill; re-run `encrypt backfill` to fix). Either way it does **not** apply the migration — review and run your own migrate command. +Runs from the `backfilled` phase and drops the original plaintext `<col>`. It verifies live coverage first and refuses if any row still has plaintext set with the encrypted column NULL. Legacy EQL v2 targets are rejected; the old rename/drop lifecycle is no longer automated. The command generates a migration but does not apply it. Flags: `--table`, `--column`, `--migrations-dir <path>`. @@ -562,7 +527,7 @@ Flags: `--name <name>`, `--write [path]`, `--json`. ```typescript import { defineConfig, loadStashConfig, resolveDatabaseUrl, - EQLInstaller, loadBundledEqlSql, downloadEqlSql, + EQLInstaller, loadBundledEqlSql, } from 'stash' ``` @@ -571,8 +536,7 @@ import { | `defineConfig` | `(config: StashConfig) => StashConfig` — identity function for type-checking | | `loadStashConfig` | `(resolverOptions?: ResolveDatabaseUrlOptions, knownConfigPath?: string) => Promise<ResolvedStashConfig>` — walks up for the config, validates with Zod, applies defaults, exits 1 if missing or invalid | | `resolveDatabaseUrl` | `(opts?: ResolveDatabaseUrlOptions) => Promise<string>` — the resolution chain documented above | -| `loadBundledEqlSql` | `(options?: { supabase?, excludeOperatorFamily?, eqlVersion?: 2 \| 3 }) => string` | -| `downloadEqlSql` | `(options?: { excludeOperatorFamily?, supabase? } \| boolean) => Promise<string>` — latest EQL from GitHub releases | +| `loadBundledEqlSql` | `() => string` — pinned EQL v3 SQL from `@cipherstash/eql` | ### `EQLInstaller` @@ -580,18 +544,18 @@ import { const installer = new EQLInstaller({ databaseUrl: 'postgresql://...' }) await installer.checkPermissions() // PermissionCheckResult -await installer.isInstalled({ eqlVersion: 3 }) // boolean +await installer.isInstalled() // boolean (v3) await installer.getInstalledVersion() // string | 'unknown' | null await installer.install({ supabase: true }) // executes in a transaction ``` -`isInstalled`, `getInstalledVersion`, and `install` all accept `eqlVersion?: 2 | 3` (default `3`, matching the CLI's `--eql-version` default), selecting the `eql_v3` or `eql_v2` schema. `install` also takes `excludeOperatorFamily`, `supabase`, and `latest` (v2 only). +`install` installs EQL v3 only and accepts `supabase`. `isInstalled` and `getInstalledVersion` retain an optional `{ eqlVersion: 2 | 3 }` solely for read-only diagnostics of existing v2 databases. ```typescript type PermissionCheckResult = { ok: boolean // all required permissions present missing: string[] // what's absent - isSuperuser: boolean // drives the automatic no-operator-family fallback + isSuperuser: boolean // permission diagnostic; the v3 bundle self-adapts } ``` diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index d9d03b22e..662459ec2 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -423,9 +423,9 @@ Run `ANALYZE <table>` after the migration applies — an expression index gather The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (On a legacy EQL v2 + CipherStash Proxy database, the rollout also includes `stash db push` to register the encryption config in `eql_v2_configuration`; EQL v3 ships no configuration table, so on the v3-only database the schema below assumes, `db push` reports "Nothing to do." and a v3 rollout never needs it.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads by name + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. -> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) now mutates **EQL v3 only**. A `public.eql_v3_*` target is required for backfill and drop. Legacy `eql_v2_encrypted` columns and migration history remain visible in status, but mutation commands reject them. The v3 lifecycle is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with no rename. > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which Drizzle tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. @@ -477,18 +477,6 @@ export const encryptionClient = await Encryption({ schemas: [usersEncryptionSche Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN "email_encrypted" "eql_v3_text_search";` — drizzle-kit emits the **bare** domain name, which resolves to the `public.eql_v3_text_search` domain via `search_path` (a schema-qualified custom type would be quoted as one identifier and fail, so the bare name is deliberate). Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) -> **Using CipherStash Proxy?** -> -> `stash db push` registers the encryption config in `eql_v2_configuration`, which only exists on a database that has EQL v2 installed. The Database Setup above installs EQL v3 only, and `types.TextSearch('email_encrypted')` is a `public.eql_v3_text_search` column — v3 keeps a column's config in its domain type and ships no configuration table, so on that database `stash db push` prints "Nothing to do." and exits 0. There is nothing to push for this schema. -> -> ```bash -> stash db push # EQL v2 + Proxy databases only -> ``` -> -> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected, and `stash db activate` promotes the pending row to active. -> -> SDK-only users can skip this step on EQL v3 (this section's case) — there is nothing to push. On a legacy EQL v2 column they cannot: `stash encrypt cutover` requires a pending EQL config, so an SDK-only v2 rollout must still run `stash db push` once before cutover (see the SDK-only note under Backfill below). - #### Dual-writing: write to both columns from app code Find **every** code path that writes to `users.email` and update it to encrypt and also write to `email_encrypted`: @@ -542,14 +530,10 @@ Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination ord If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. -> **SDK-only note (EQL v2 only):** `stash encrypt cutover` requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. EQL v3 columns never hit this — cut-over doesn't apply to them. - #### Switch reads to the encrypted column -**EQL v3: there is no cut-over.** The encrypted column keeps its own name — you -switch the application to it by name, verify reads, then drop the plaintext -column. Running `stash encrypt cutover` on a **backfilled** v3 column reports -"not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +The EQL v3 encrypted column keeps its own name. Switch the application to it by +name, verify reads, then drop the plaintext column. There is no rename command. Point your read paths at `email_encrypted` and decrypt the selected envelopes with the encryption client. **This is the moment that breaks read paths if they diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index e8016b013..b412aa652 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -271,13 +271,13 @@ type UserEncrypted = InferEncrypted<typeof users> Install the EQL v3 SQL with the stash CLI (v3 is the default): ```bash -stash eql install --eql-version 3 +stash eql install # Supabase targets: add --supabase to apply role grants -stash eql install --eql-version 3 --supabase +stash eql install --supabase ``` -EQL v3 ships **one SQL bundle for every target, including Supabase** — no separate Supabase or no-operator-family variants. For Supabase, though, still pass the `--supabase` flag: it applies the `anon`/`authenticated`/`service_role` grants on the `eql_v3` and `eql_v3_internal` schemas. Without it, encrypted queries fail with `permission denied for schema eql_v3_internal`. v3 installs via the direct path only (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are v2-only flags and are not supported for v3). +EQL v3 ships one SQL bundle for every target. For Supabase, pass `--supabase` to apply grants on `eql_v3` and `eql_v3_internal`. For a Drizzle migration, use `stash eql migration --drizzle`; the old v2 install flags were removed. In migrations, declare each encrypted column as its domain type: @@ -825,7 +825,7 @@ try { ## Rolling Encryption Out to Production -> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column — *unless* the table also holds EQL v3 columns and `.cipherstash/migrations.json` records an `encryptedColumn` that is one of the unclassified ones. That combination is ambiguous, so `cutover`/`drop` fail closed and name the recorded column instead of guessing. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `<col>_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `<col>_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext `<col>`. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). +> **EQL version note.** Rollout mutation tooling is EQL v3 only. It requires a `public.eql_v3_*` destination domain; legacy v2 columns remain readable and visible in status/history diagnostics but cannot be installed, backfilled, cut over, or dropped through `stash`. Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. @@ -856,8 +856,6 @@ Everything that lands in the repo and ships in **one** PR: | Schema-add | Migration adds `<col>_encrypted` (nullable `jsonb`) alongside the existing plaintext column. Plaintext column unchanged; application still writes only plaintext. | | Dual-write code | Application now writes both `<col>` and `<col>_encrypted` on every persistence path that mutates the row, in the same transaction, on every code branch. Reads still come from the plaintext column. | -> **If you use CipherStash Proxy (the EQL v2 path):** `stash db push` and `stash db activate` manage `eql_v2_configuration`, so they only apply to a database that has EQL v2 installed — on a v3-only database (the default) `db push` reports "Nothing to do." and exits 0, and `db activate` errors out. EQL v3 ships no configuration table, so a v3 rollout has nothing to push. On a v2 + Proxy database, run `stash db push` after the schema-add to register the new column. With no active config yet it writes directly to `active`; with an existing active config it writes `pending`, which `stash db activate` promotes to active (the v2 `stash encrypt cutover` also promotes it as part of its rename). Required for Proxy-based queries. - **The dual-write definition matters.** "Writes both columns" is not enough. The rule is: every persistence path that mutates this row writes both columns, in the same transaction, on every code branch. A single missed branch — a CSV import, an admin action, a background job, a third-party webhook handler — means rows inserted in production after deploy land in plaintext only, and backfill won't catch them. Grep for every site that writes the plaintext column before declaring rollout complete. ### ⛔ Deploy gate @@ -875,25 +873,23 @@ Once dual-writes are recorded as live in `cs_migrations`: | Action | What changes | |---|---| | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | -| Schema rename (**v2 only**) | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. **v3:** there is no rename — leave `<col>_encrypted` under its own name and point the schema/queries at that name instead. | -| `stash encrypt cutover` (**v2 only**) | One transaction: renames `<col>` → `<col>_plaintext`, `<col>_encrypted` → `<col>`, and promotes the `eql_v2_configuration` row `pending` → `active`. Reads of `<col>` through **CipherStash Proxy** are decrypted on the wire; SDK/ORM reads still return ciphertext and need the decrypt path in the next row. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. Exception: if the encrypted column was identified only by elimination (no recorded `encryptedColumn`, no `<col>_encrypted` name match), it exits 1 rather than report an outcome for a guess — record the pairing with `stash encrypt backfill … --encrypted-column <name>`. | +| Switch schema/query references | Leave `<col>_encrypted` under its own name and point the schema and queries at it. EQL v3 has no rename cut-over. | | Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | -| Remove dual-write code | The plaintext column is no longer authoritative — **v2:** it is now `<col>_plaintext` (the cutover renamed it); **v3:** it is still the original `<col>`, since nothing was renamed. Either way, delete the dual-write logic once reads are served from the encrypted column. | -| `stash encrypt drop` | Emits a migration that drops the plaintext column — and *which* column that is depends on the generation. **v2** (precondition: phase `cut-over`): a plain `ALTER TABLE … DROP COLUMN "<col>_plaintext"`. **v3** (precondition: phase `backfilled`): it drops the **original `<col>`** — there is no `<col>_plaintext` — and the generated SQL is a `DO` block that takes `ACCESS EXCLUSIVE` on the table, re-counts rows with `<col>` set and `<col>_encrypted` NULL *at apply time*, and raises instead of dropping if any remain. Apply with the project's normal migration tooling. | +| Remove dual-write code | Delete dual-write logic once reads are served from the encrypted column. | +| `stash encrypt drop` | Emits a migration that drops the original plaintext `<col>`. The generated SQL locks the table, re-checks coverage at apply time, and raises instead of dropping if plaintext-only rows remain. | -**Create the functional indexes between backfill and the read switch** (EQL v3 columns). After `stash encrypt backfill` completes and before reads move to the encrypted column, create the `eql_v3.*` extractor indexes for every queried capability (and `ANALYZE`) — one bulk build instead of per-row maintenance during backfill, and the switched reads engage an index from the first query. Recipes in the `stash-indexing` skill. (Legacy v2 rollouts have no extractor indexes to create — skip this step.) +**Create functional indexes between backfill and the read switch.** Build the `eql_v3.*` extractor indexes for every queried capability and run `ANALYZE`. ### State storage -Three sources of truth, kept separate on purpose: +Two current sources of truth, kept separate on purpose: - **`.cipherstash/migrations.json`** (repo) — *intent*. Which columns the developer wants to encrypt and at which phase, code-reviewable. -- **`eql_v2_configuration`** (DB, EQL-managed) — *EQL intent*. **EQL v2 + Proxy only.** Which columns are encrypted and with which indexes; drives the CipherStash Proxy. EQL v3 encodes a column's config in its Postgres domain type and ships no configuration table, so a v3-only database has just the other two. - **`cipherstash.cs_migrations`** (DB, CipherStash-managed) — *runtime state*. Append-only event log: phase transitions, backfill cursors, error rows. Latest row per `(table, column)` is the current state. `stash encrypt status` shows all three side-by-side and flags drift (e.g. EQL says registered, the physical `<col>_encrypted` column is missing). `stash status` (the quest log) rolls them up into the per-column "what's the next move" view used during a rollout. -> **Note on internal phase names.** The runtime event log uses machine-readable phase names that depend on the column's EQL version: v3 (the default) runs `schema-added → dual-writing → backfilling → backfilled → dropped` (no cut-over — the app switches to the encrypted column by name), while v2 runs `schema-added → dual-writing → backfilling → backfilled → cut-over → dropped`. They appear in `cs_migrations` rows and `stash encrypt status` output. Treat them as internal mechanism detail — the user-facing story is "encryption rollout, then switch reads to encrypted, with a deploy gate in between." +> **Note on internal phase names.** Current runs use `schema-added → dual-writing → backfilling → backfilled → dropped`. Readers still accept legacy `cut-over` rows so old history remains displayable. ### CLI sequence for a single column @@ -930,8 +926,8 @@ stash encrypt backfill --table users --column email --force # after backfill and before reads move over. Recipes: `stash-indexing`. # Point the application at the encrypted column BY NAME — -# `email_encrypted`. There is no rename and no `stash encrypt cutover` -# on v3. Wire the read paths through the encryption client so they +# `email_encrypted`. There is no rename command. Wire the read paths through +# the encryption client so they # decrypt, deploy, and verify reads return plaintext. # Then remove the dual-write code and drop the plaintext column. @@ -940,97 +936,6 @@ stash encrypt backfill --table users --column email --force stash encrypt drop --table users --column email ``` -#### EQL v2 (legacy) - -> **Known limitation (v2):** `stash encrypt cutover` requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked separately. (EQL v3 columns never hit this — cutover doesn't apply to them.) - -```bash -# Run this often — it's the canonical "where am I?" command. -stash status - -# ---- ENCRYPTION ROLLOUT (one PR, one deploy) ---- -# 1. Add the encrypted twin column via your normal migration tooling -# (drizzle-kit / supabase migrations / etc.). -# 2. Edit application code so every persistence path writes both -# `<col>` and `<col>_encrypted` in the same transaction, on every -# code branch. -# 3. Ship the PR to production. - -# ---- ⛔ DEPLOY GATE ---- -# Verify dual-writes are live, then redraft the plan for cutover work: -stash status -stash plan - -# ---- ENCRYPTION CUTOVER ---- -stash encrypt backfill --table users --column email -# Prompts to confirm dual-writes are live (or pass -# --confirm-dual-writes-deployed in CI). Resumable; SIGINT-safe. - -# Recovery — if dual-writes weren't actually live when backfill ran, -# re-run with --force to encrypt every plaintext row regardless. -stash encrypt backfill --table users --column email --force - -# Edit the schema to drop the `_encrypted` suffix, then register the -# pending EQL config — cutover requires it (see Known limitation above), -# so SDK-only deployments must run `stash db push` once here too: -stash db push -stash encrypt cutover --table users --column email -# In one transaction: rename physical columns, promote pending → active. - -# Wire the read paths through the encryption client. Remove dual-write -# code. Then drop the plaintext column: -stash encrypt drop --table users --column email -``` - -#### EQL v2 + CipherStash Proxy - -Register and promote encryption config at each phase. This applies only to a database with EQL v2 installed — `eql_v2_configuration` is where `stash db push` writes, and a v3-only database has no such table (`db push` reports "Nothing to do."). A v3 rollout uses the v3 sequence above whether or not Proxy is in front of it. - -```bash -# Run this often — it's the canonical "where am I?" command. -stash status - -# ---- ENCRYPTION ROLLOUT (one PR, one deploy) ---- -# 1. Add the encrypted twin column via your normal migration tooling -# (drizzle-kit / supabase migrations / etc.). -# 2. Register the new encryption config with EQL: -stash db push -# First push (no active config yet) → writes directly to active. -# Subsequent push (active already exists) → writes pending, which -# `stash db activate` promotes — as does the v2 `stash encrypt -# cutover` below, as part of its rename. -# 3. Edit application code so every persistence path writes both -# `<col>` and `<col>_encrypted` in the same transaction, on every -# code branch. -# 4. Ship the PR to production. - -# ---- ⛔ DEPLOY GATE ---- -# Verify dual-writes are live, then redraft the plan for cutover work: -stash status -stash plan - -# ---- ENCRYPTION CUTOVER ---- -stash encrypt backfill --table users --column email -# Prompts to confirm dual-writes are live (or pass -# --confirm-dual-writes-deployed in CI). Resumable; SIGINT-safe. - -# Recovery — if dual-writes weren't actually live when backfill ran, -# re-run with --force to encrypt every plaintext row regardless. -stash encrypt backfill --table users --column email --force - -# Edit the schema to drop the `_encrypted` suffix, then re-push: -stash db push -# → writes the renamed-shape config as `pending`. The active config -# keeps serving until cutover finishes. - -stash encrypt cutover --table users --column email -# In one transaction: rename physical columns, promote pending → active. - -# Wire the read paths through the encryption client. Remove dual-write -# code. Then drop the plaintext column: -stash encrypt drop --table users --column email -``` - ### Library use Long-running backfills can also embed the engine directly without the CLI: @@ -1059,11 +964,11 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a ### Invariants the rollout preserves -- **Reads never return the wrong value.** Until cutover, reads come from the plaintext column. After cutover, the same `SELECT email` returns the decrypted ciphertext via Proxy or the encryption client. There is no in-between. -- **Writes never drop.** Dual-writing keeps both columns in sync until the cutover moment. After cutover, writes go to the encrypted column. +- **Reads never return the wrong value.** Before the application switch, reads come from the plaintext column. After the switch, queries target the encrypted column by name and decrypt through the integration/client. +- **Writes never drop.** Dual-writing keeps both columns in sync until the application switches to the encrypted column. - **The deploy gate is a one-way door for production.** Backfill against rows the dual-write code never saw produces silent drift. The CLI refuses to run cutover-step plans without a `dual_writing` event recorded; do not paper over that refusal. - **Re-runs are safe.** Backfill is idempotent (`<col> IS NOT NULL AND <col>_encrypted IS NULL` guards every chunk). `cs_migrations` is append-only. -- **Rollback is possible up to cutover.** Until the rename happens, the plaintext column is authoritative; aborting just leaves the encrypted twin partially populated. After cutover, rollback is a manual restore — treat cutover as the one-way door for data. +- **Rollback is possible until plaintext is dropped.** Before the final drop, aborting leaves the original plaintext column intact; after the drop, recovery requires a restore. ## Integrations @@ -1118,6 +1023,6 @@ const users = encryptedTable("users", { const client = await Encryption({ schemas: [users] }) ``` -**v2 is a read path now, not an authoring surface.** `decrypt` / `decryptModel` still read stored v2 payloads, and `stash eql install --eql-version 2` still installs the v2 SQL, so existing deployments keep working. What is gone: the **Supabase** and **Drizzle** adapters are EQL v3 only — `encryptedSupabase` is the v3 factory (there is no v2 form), and `@cipherstash/stack-drizzle` dropped `encryptedType` and the `like`/`ilike` operators. A v2 column reached through either adapter is read-only; decrypt it through `@cipherstash/stack` instead. The `@cipherstash/stack/schema` builders (`encryptedColumn(...).equality()`, …) remain exported but are `@deprecated` — do not author new schemas with them. **Legacy v2 `searchableJson()`** cannot be emitted by protect-ffi 0.30 (the selector envelope was removed), so migrate those columns to v3 `types.Json`. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) +**v2 is a read path now, not an authoring or rollout surface.** `decrypt` / `decryptModel` still read stored v2 payloads, but `stash` no longer installs EQL v2 or drives its Proxy configuration, backfill, rename, or drop lifecycle. For dump recovery, obtain the EQL 2.3.1 SQL from the upstream encrypt-query-language release. Migrate maintained deployments to v3 `types.*` domains. > **DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) now **encrypts EQL v3 only** — author tables with `types.*` from `@cipherstash/stack/eql/v3`. Its decrypt methods still accept a v2 table so previously stored v2 items remain readable. See the `stash-dynamodb` skill. diff --git a/skills/stash-indexing/SKILL.md b/skills/stash-indexing/SKILL.md index d7f7d981e..d62efad76 100644 --- a/skills/stash-indexing/SKILL.md +++ b/skills/stash-indexing/SKILL.md @@ -268,8 +268,8 @@ Index not being used: ## Reference -- `stash-encryption` — the `types.*` domain catalog, wire-format operators and ordering, and the rollout/cutover lifecycle. -- `stash-cli` — `stash eql install`, `stash db validate` (its "No indexes on an encrypted column" Info finding is resolved by this skill), `stash encrypt backfill` / `cutover`. +- `stash-encryption` — the `types.*` domain catalog, wire-format operators and ordering, and the staged rollout lifecycle. +- `stash-cli` — `stash eql install`, `stash db validate` (its "No indexes on an encrypted column" Info finding is resolved by this skill), and `stash encrypt backfill` / `drop`. - `stash-drizzle`, `stash-supabase`, `stash-prisma-next` — per-integration query patterns; index DDL placement per the section above. - `stash-postgres` — the hand-written predicate forms these indexes serve (`pg` / `postgres-js`, no ORM). - `stash-edge` — the WASM entry, for apps whose queries run on Deno / Workers / Supabase Edge Functions. diff --git a/skills/stash-postgres/SKILL.md b/skills/stash-postgres/SKILL.md index 8e3260b32..5cfd92e01 100644 --- a/skills/stash-postgres/SKILL.md +++ b/skills/stash-postgres/SKILL.md @@ -30,12 +30,11 @@ Supabase client, those integrations emit correct operands for you: see > decrypts on read, so there is no `encryptQuery`, no `eql_v3.query_*` cast, > and no payload to bind. > -> Proxy's schema lifecycle — `stash db push` into `eql_v2_configuration`, -> promoted at cutover — belongs to EQL **v2**. This skill is EQL v3, where a -> column's encryption config lives in its own domain and there is no -> configuration table to push. The `stash` CLI does not track a -> Proxy-versus-SDK choice; it targets the direct-connection path this skill -> describes. +> Proxy's former schema lifecycle belongs to EQL **v2** and is no longer +> installed or mutated by `stash`; legacy state remains visible through status +> diagnostics only. This skill is EQL v3, where a column's encryption config +> lives in its own domain and there is no configuration table to push. The +> `stash` CLI targets the direct-connection path this skill describes. ## When to Use This Skill diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index deacdf19c..8c5f3f1c8 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -606,11 +606,9 @@ alias of its unsuffixed counterpart above. The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + switch reads to the encrypted column + drop — under EQL v3, which this section's schema uses, the switch is an application-side change with no rename; under legacy EQL v2 it is a rename). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + switch reads to the encrypted column by name + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. -> **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). - -> **Using CipherStash Proxy?** `stash db push` and `stash db activate` manage the `eql_v2_configuration` table, so they only apply to a database that has EQL v2 installed. EQL v3 ships no configuration table — on the v3-only database this section's schema assumes, `stash db push` reports "Nothing to do." and exits 0, and there is no cutover to run it before. If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) against a legacy EQL v2 database instead of the SDK, run `stash db push` after schema-add to register the encrypted column shape with EQL. +> **EQL version note.** The `stash encrypt *` tooling now mutates **EQL v3 only**. A `public.eql_v3_*` target is required for backfill and drop. Legacy `eql_v2_encrypted` columns and migration history remain visible in status, but mutation commands reject them. The v3 lifecycle is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with no rename. > **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash <command>` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. @@ -666,16 +664,6 @@ export const users = encryptedTable('users', { }) ``` -> **Using CipherStash Proxy?** `stash db push` registers the encryption config in `eql_v2_configuration` — an EQL v2 + Proxy artifact. The `public.eql_v3_text_search` column added above needs nothing registered: EQL v3 keeps a column's config in its domain type and ships no configuration table, so on a v3-only database `db push` prints "Nothing to do." and exits 0. -> -> ```bash -> stash db push # EQL v2 + Proxy databases only -> ``` -> -> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config. If an active config already exists, it writes the new config as `pending` — that's expected, and `stash db activate` promotes it to active. -> -> **SDK users:** Skip this step. Your encryption config lives in app code. - #### Dual-writing: write to both columns from app code Find **every** code path that writes to `users.email` and update it to also write the encrypted twin. With the v3 wrapper this is a single insert: `email` is a plaintext column and passes through unchanged, while `email_encrypted` is a v3 domain column the wrapper encrypts automatically. Wrap it in one function so callers can't forget one half: @@ -723,30 +711,15 @@ stash encrypt backfill --table users --column email # (CI: pass --confirm-dual-writes-deployed instead.) ``` -Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It detects a `public.eql_v3_*` column as EQL v3 and records that in `cs_migrations`; a legacy `eql_v2_encrypted` column does not classify and takes the v2 lifecycle with no version recorded. +Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It requires a `public.eql_v3_*` target and records EQL version 3; a legacy `eql_v2_encrypted` target is rejected before encryption begins. If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. #### Switch reads to the encrypted column -**EQL v3 (the schema above): there is no cut-over.** The encrypted column keeps -its own name — point your application at `email_encrypted` through the -`encryptedSupabase` wrapper, deploy, verify reads decrypt correctly, then skip -ahead to the drop step. Running `stash encrypt cutover` on a **backfilled** v3 -column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't -finished). - -> **EQL v2 cutover (`stash encrypt cutover`) via this SDK is no longer -> supported.** `@cipherstash/stack-supabase` authors and queries EQL v3 only — -> the v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper that -> read the post-cutover `eql_v2_encrypted` column has been removed. The CLI -> lifecycle commands still handle a v2 column — detection is one-sided, so a v2 -> column classifies as *unknown* and falls through to the v2 lifecycle — but the -> SDK read path for one is gone. If you have an in-flight v2 cutover, either pin -> the last `@cipherstash/stack-supabase` release that shipped the v2 wrapper to -> finish it, or (recommended) create an `eql_v3_*` twin and run the v3 rollout -> above. New -> encryption should always target an `eql_v3_*` domain. +The EQL v3 encrypted column keeps its own name. Point the application at +`email_encrypted` through the `encryptedSupabase` wrapper, deploy, verify reads +decrypt correctly, then continue to the drop step. There is no rename command. #### Drop: remove the plaintext column @@ -756,10 +729,13 @@ Once read paths are routing through the wrapper and you're confident reads are d stash encrypt drop --table users --column email ``` -The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version.** Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else classifies as *unknown* and falls through to the **v2** path: - -- **v3** — drops the original plaintext column, `email`. There was no rename, so no `email_plaintext` exists. The SQL is not a bare `ALTER TABLE`: it's a `DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply time*, `RAISE EXCEPTION`s if any remain, and only then executes the `ALTER TABLE ... DROP COLUMN` — so a row written after generation can't be silently destroyed. Requires the `backfilled` phase plus a live coverage check at generation time. -- **v2** — drops the post-rename leftover, `ALTER TABLE users DROP COLUMN email_plaintext;`. Requires the `cut-over` phase. +The CLI emits an EQL v3 drop migration for the original plaintext column, +`email`. There was no rename, so no `email_plaintext` exists. The SQL is not a +bare `ALTER TABLE`: it is a `DO $stash_drop$` block that takes `LOCK TABLE users +IN ACCESS EXCLUSIVE MODE`, re-counts rows where `email IS NOT NULL AND +email_encrypted IS NULL` at apply time, raises if any remain, and only then +drops the column. It requires the `backfilled` phase plus a live coverage check +at generation time. Legacy v2 state is rejected. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — the plaintext column is gone; only the encrypted column is written now, through the wrapper. @@ -797,17 +773,12 @@ A v2 `encryptedTable` is structurally identical to a v3 one apart from that marker, so TypeScript alone will not always catch the swap — re-author the table with `encryptedTable`/`types` from `@cipherstash/stack/eql/v3`. -Existing v2 deployments have two options: - -- **Migrate to EQL v3** (recommended): add an `eql_v3_*` twin column and run the - rollout in "Migrating an Existing Column to Encrypted" above. -- **Stay on v2 for now:** pin the last `@cipherstash/stack-supabase` release that - shipped the v2 wrapper. The CLI rollout tooling (`stash encrypt backfill` / - `cutover` / `drop`) still drives a v2 column, but only because detection is - one-sided: a v2 column is never detected as v2, it classifies as *unknown* and - falls through to the v2 lifecycle. No EQL version is recorded for it in - `.cipherstash/migrations.json`, so `stash encrypt status` reports no version - for that column. +Existing v2 deployments should add an `eql_v3_*` twin column and run the rollout +in "Migrating an Existing Column to Encrypted" above. Current `stash` releases +do not install EQL v2 or mutate its Proxy configuration, backfill, rename, or +drop lifecycle. They retain read-only status and manifest diagnostics so the +legacy state remains visible. For dump recovery, use the upstream EQL 2.3.1 SQL +release; do not treat it as a supported new-install path. For the removed v2 wrapper's historical API and semantics, see the docs at https://cipherstash.com/docs. From e26868dfff9288eee90c4685e795a5cb26f6d484 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 09:47:08 +1000 Subject: [PATCH 101/123] fix(cli): remove stale v2 rollout guidance --- .../init/lib/__tests__/setup-prompt.test.ts | 8 ++++ .../cli/src/commands/init/lib/setup-prompt.ts | 4 +- packages/migrate/README.md | 2 +- .../no-removed-eql-version-flag.test.mjs | 42 +++++++++++++++++++ skills/stash-drizzle/SKILL.md | 2 +- skills/stash-supabase/SKILL.md | 2 +- 6 files changed, 56 insertions(+), 4 deletions(-) create mode 100644 scripts/__tests__/no-removed-eql-version-flag.test.mjs diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 512ad68a4..9c3eb6e60 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -591,6 +591,14 @@ describe('renderSetupPrompt — plan templates are EQL v3-only', () => { it('does not emit the removed cutover invocation', () => { expect(plan('cutover')).not.toMatch(/encrypt cutover/) }) + + it('gives the complete plan the supported backfill, read-switch, and drop sequence', () => { + const out = plan('complete') + expect(out).toContain('`pnpm dlx stash encrypt backfill`') + expect(out).toMatch(/application read switch.*encrypted column by name/i) + expect(out).toContain('`pnpm dlx stash encrypt drop`') + expect(out).not.toContain('`cutover`') + }) }) describe('renderSetupPrompt — honours what the handoff actually wrote', () => { diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 5e190612e..6f66d06c7 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -760,7 +760,9 @@ function renderCompletePlanPrompt(ctx: SetupPromptContext): string { bullet( 'For migrate columns: the full step list with the exact CLI invocations (`' + cli + - ' encrypt backfill`, `cutover`, `drop`) and concrete `--table` / `--column` values.', + ' encrypt backfill`, an application read switch to the EQL v3 encrypted column by name and deploy, `' + + cli + + ' encrypt drop`) and concrete `--table` / `--column` values.', ), bullet('For new columns: the additive single-deploy walkthrough.'), bullet( diff --git a/packages/migrate/README.md b/packages/migrate/README.md index 7c9355580..310a8d150 100644 --- a/packages/migrate/README.md +++ b/packages/migrate/README.md @@ -34,7 +34,7 @@ Creates `cipherstash.cs_migrations` idempotently. Normally called by `stash eql ### `runBackfill(options)` -Runs a chunked, resumable, idempotent plaintext-to-encrypted backfill. Each chunk selects the next keyset page, encrypts through `bulkEncryptModels`, updates the encrypted column, and records a checkpoint in one transaction. The `encrypted IS NULL` guard makes retries converge. +Runs a chunked, resumable, idempotent plaintext-to-encrypted backfill. For each chunk, it selects the next keyset page and encrypts it through `bulkEncryptModels` before `BEGIN`. The database transaction commits only the encrypted-column writes and the corresponding `cs_migrations` checkpoint. The `encrypted IS NULL` guard makes retries converge. Pass an initialized EQL v3 `Encryption` client and an EQL v3 `encryptedTable`. The lower-level runner writes the envelope produced by the supplied client; the `stash encrypt backfill` command additionally verifies that the destination is an `eql_v3_*` domain before invoking it. diff --git a/scripts/__tests__/no-removed-eql-version-flag.test.mjs b/scripts/__tests__/no-removed-eql-version-flag.test.mjs new file mode 100644 index 000000000..04866236c --- /dev/null +++ b/scripts/__tests__/no-removed-eql-version-flag.test.mjs @@ -0,0 +1,42 @@ +import { execFileSync } from 'node:child_process' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') + +/** Tracked skills ship with the CLI and are copied into customer projects. */ +function shippedSkills() { + const out = execFileSync( + 'git', + ['ls-files', '-z', ':(glob)skills/*/SKILL.md'], + { + cwd: REPO_ROOT, + encoding: 'utf8', + }, + ) + return out.split('\0').filter(Boolean) +} + +describe('shipped skill eql install examples use the current CLI', () => { + const files = shippedSkills() + + it('finds tracked shipped skills (guards against a silently-empty glob)', () => { + expect(files.length).toBeGreaterThan(5) + expect(files).toContain('skills/stash-drizzle/SKILL.md') + expect(files).toContain('skills/stash-supabase/SKILL.md') + }) + + it.each(files)('%s', (file) => { + const body = readFileSync(resolve(REPO_ROOT, file), 'utf8') + const obsoleteExamples = body.match( + /\bstash\s+eql\s+install\b[^\n]*--eql-version\b/g, + ) + + expect( + obsoleteExamples ?? [], + `${file} contains an executable stash eql install example with the removed --eql-version flag.`, + ).toEqual([]) + }) +}) diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 662459ec2..90927a599 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -49,7 +49,7 @@ EQL (Encrypt Query Language) provides the PostgreSQL functions and domains that **Direct install** — run the SQL straight against the database (quick, good for dev): ```bash -stash eql install --eql-version 3 +stash eql install ``` **Migration (preferred for real projects)** — generate a Drizzle custom migration that carries the EQL v3 install SQL, so it lands in your migration history and ships to every environment through `drizzle-kit migrate`: diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 8c5f3f1c8..2bfcb1d09 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -67,7 +67,7 @@ this is also how **Supabase Edge Functions** get credentials in local dev — ### 1. Install EQL v3 on the database ```bash -stash eql install --eql-version 3 --supabase +stash eql install --supabase ``` Since eql-3.0.0 there is **one** v3 SQL artifact for every target — there is From 73e1483b7c3f38eaf12ec88995c2526e91cc974c Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 09:53:40 +1000 Subject: [PATCH 102/123] test(cli): harden removed command regressions --- .../init/lib/__tests__/setup-prompt.test.ts | 17 ++++++-- .../no-removed-eql-version-flag.test.mjs | 41 ++++++++++++++++--- 2 files changed, 50 insertions(+), 8 deletions(-) diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 9c3eb6e60..0afe7b30a 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -24,6 +24,12 @@ const baseCtx: SetupPromptContext = { }, } +function hasCompletePlanSequence(prompt: string): boolean { + return /`pnpm dlx stash encrypt backfill`[\s\S]*application read switch.*encrypted column by name and deploy[\s\S]*`pnpm dlx stash encrypt drop`/i.test( + prompt, + ) +} + describe('renderSetupPrompt — orient + route (implement mode)', () => { it('emits integration + package manager in the header', () => { const out = renderSetupPrompt(baseCtx) @@ -594,11 +600,16 @@ describe('renderSetupPrompt — plan templates are EQL v3-only', () => { it('gives the complete plan the supported backfill, read-switch, and drop sequence', () => { const out = plan('complete') - expect(out).toContain('`pnpm dlx stash encrypt backfill`') - expect(out).toMatch(/application read switch.*encrypted column by name/i) - expect(out).toContain('`pnpm dlx stash encrypt drop`') + expect(hasCompletePlanSequence(out)).toBe(true) expect(out).not.toContain('`cutover`') }) + + it('requires the application read switch to be deployed before drop', () => { + const missingDeploy = + '`pnpm dlx stash encrypt backfill`, an application read switch to the EQL v3 encrypted column by name, `pnpm dlx stash encrypt drop`' + + expect(hasCompletePlanSequence(missingDeploy)).toBe(false) + }) }) describe('renderSetupPrompt — honours what the handoff actually wrote', () => { diff --git a/scripts/__tests__/no-removed-eql-version-flag.test.mjs b/scripts/__tests__/no-removed-eql-version-flag.test.mjs index 04866236c..ba8bb08c8 100644 --- a/scripts/__tests__/no-removed-eql-version-flag.test.mjs +++ b/scripts/__tests__/no-removed-eql-version-flag.test.mjs @@ -19,23 +19,54 @@ function shippedSkills() { return out.split('\0').filter(Boolean) } +function obsoleteEqlInstallExamples(body) { + return [ + ...body.matchAll(/```(?:bash|sh|shell|zsh)[^\n]*\r?\n([\s\S]*?)```/gi), + ].flatMap( + (match) => + match[1] + .replace(/\\\r?\n[ \t]*/g, ' ') + .match( + /^[ \t]*(?:\$[ \t]+)?(?:pnpm[ \t]+dlx[ \t]+|npx[ \t]+)?stash[ \t]+eql[ \t]+install\b[^\n]*--eql-version\b[^\n]*/gm, + ) ?? [], + ) +} + +describe('obsolete eql install example detection', () => { + it('does not treat explanatory prose as an executable example', () => { + const prose = + 'The removed stash eql install --eql-version flag is no longer accepted.' + + expect(obsoleteEqlInstallExamples(prose)).toEqual([]) + }) + + it('detects a backslash-wrapped executable shell command', () => { + const shellExample = [ + '```bash', + 'stash eql install \\', + ' --eql-version 3', + '```', + ].join('\n') + + expect(obsoleteEqlInstallExamples(shellExample)).toHaveLength(1) + }) +}) + describe('shipped skill eql install examples use the current CLI', () => { const files = shippedSkills() it('finds tracked shipped skills (guards against a silently-empty glob)', () => { - expect(files.length).toBeGreaterThan(5) + expect(files).not.toHaveLength(0) expect(files).toContain('skills/stash-drizzle/SKILL.md') expect(files).toContain('skills/stash-supabase/SKILL.md') }) it.each(files)('%s', (file) => { const body = readFileSync(resolve(REPO_ROOT, file), 'utf8') - const obsoleteExamples = body.match( - /\bstash\s+eql\s+install\b[^\n]*--eql-version\b/g, - ) + const obsoleteExamples = obsoleteEqlInstallExamples(body) expect( - obsoleteExamples ?? [], + obsoleteExamples, `${file} contains an executable stash eql install example with the removed --eql-version flag.`, ).toEqual([]) }) From 3a182bfad09e16eb2d28f220a9a704284c86c741 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 09:56:33 +1000 Subject: [PATCH 103/123] test(cli): cover all skill command runners --- .../__tests__/no-removed-eql-version-flag.test.mjs | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/scripts/__tests__/no-removed-eql-version-flag.test.mjs b/scripts/__tests__/no-removed-eql-version-flag.test.mjs index ba8bb08c8..778bf95a5 100644 --- a/scripts/__tests__/no-removed-eql-version-flag.test.mjs +++ b/scripts/__tests__/no-removed-eql-version-flag.test.mjs @@ -27,12 +27,24 @@ function obsoleteEqlInstallExamples(body) { match[1] .replace(/\\\r?\n[ \t]*/g, ' ') .match( - /^[ \t]*(?:\$[ \t]+)?(?:pnpm[ \t]+dlx[ \t]+|npx[ \t]+)?stash[ \t]+eql[ \t]+install\b[^\n]*--eql-version\b[^\n]*/gm, + /^[ \t]*(?:\$[ \t]+)?(?:(?:bunx|npx)[ \t]+|(?:pnpm|yarn)[ \t]+dlx[ \t]+)?stash[ \t]+eql[ \t]+install\b[^\n]*--eql-version\b[^\n]*/gm, ) ?? [], ) } describe('obsolete eql install example detection', () => { + it.each([ + 'stash', + 'npx stash', + 'bunx stash', + 'pnpm dlx stash', + 'yarn dlx stash', + ])('detects the documented %s runner form', (runner) => { + const shellExample = `\`\`\`bash\n${runner} eql install --eql-version 3\n\`\`\`` + + expect(obsoleteEqlInstallExamples(shellExample)).toHaveLength(1) + }) + it('does not treat explanatory prose as an executable example', () => { const prose = 'The removed stash eql install --eql-version flag is no longer accepted.' From ac9289ec4cb080d51d1f9d2752b5de460fc30ec6 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 10:14:31 +1000 Subject: [PATCH 104/123] fix(cli): finish EQL v2 retirement cleanup --- docs/reference/supabase-sdk.md | 19 +++---- packages/cli/src/bin/main.ts | 20 +++++--- .../src/commands/encrypt/lib/resolve-eql.ts | 34 +++++-------- packages/cli/src/commands/init/index.ts | 10 ++++ .../init/lib/__tests__/setup-prompt.test.ts | 6 +++ .../cli/src/commands/init/lib/setup-prompt.ts | 2 +- .../cli/tests/e2e/v2-retirement.e2e.test.ts | 32 ++++++++++++ .../__tests__/supabase-v3-factory.test.ts | 9 ++-- .../integration/wire.integration.test.ts | 2 +- packages/stack-supabase/src/index.ts | 2 +- packages/stack/README.md | 4 +- .../shared/harness.integration.test.ts | 2 +- .../test-kit/src/__tests__/install.test.ts | 51 +++++++++++++++++++ packages/test-kit/src/catalog.ts | 2 +- packages/test-kit/src/install.ts | 10 ++-- .../wizard/src/__tests__/post-agent.test.ts | 35 ++++++++++--- packages/wizard/src/lib/gather.ts | 2 +- packages/wizard/src/lib/post-agent.ts | 20 ++------ .../no-removed-eql-version-flag.test.mjs | 20 +++++--- 19 files changed, 199 insertions(+), 83 deletions(-) create mode 100644 packages/test-kit/src/__tests__/install.test.ts diff --git a/docs/reference/supabase-sdk.md b/docs/reference/supabase-sdk.md index 220ea7734..cc768d20a 100644 --- a/docs/reference/supabase-sdk.md +++ b/docs/reference/supabase-sdk.md @@ -137,17 +137,14 @@ The domains use SQL-standard type names (`integer`, `smallint`, `real`, ### Install EQL ```bash -# v3 (the default) stash eql install --supabase - -# v2 (legacy installs only) -stash eql install --eql-version 2 --supabase ``` -For **v2**, `--supabase` selects the opclass-stripped bundle (operator -classes / families require superuser, which Supabase does not grant) and -applies the schema grants for `anon`, `authenticated`, and `service_role`. -Without the grants, encrypted queries fail with `42501`. +`stash` no longer installs EQL v2 or mutates its Proxy configuration. To +recover or restore an existing v2 database dump, use the upstream +[EQL 2.3.1 release SQL](https://github.com/cipherstash/encrypt-query-language/releases/tag/eql-2.3.1), +then migrate maintained deployments to v3 domains. Do not use v2 for new +authoring. For **v3**, since eql-3.0.0 there is **one** SQL artifact for every target — no separate Supabase variant. The bundle's only superuser-requiring @@ -168,9 +165,9 @@ denied for schema eql_v3_internal`). ### Exposed schemas -**v2 (manual, required):** for a bare `col <op> term` filter to reach the -custom operator, `eql_v2` must be on PostgREST's request-time search_path — -add it to **Dashboard → Settings → API → Exposed schemas** +**Legacy v2 recovery deployments only:** for a bare `col <op> term` filter to +reach the custom operator, `eql_v2` must be on PostgREST's request-time +search_path — add it to **Dashboard → Settings → API → Exposed schemas** ([Supabase custom-schemas guide](https://supabase.com/docs/guides/api/using-custom-schemas)). > **Warning — silent fallback (v2).** If the schema is not exposed, the diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index 94576492d..773e41c3a 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -161,7 +161,13 @@ function parseArgs(argv: string[]): ParsedArgs { for (let i = 0; i < rest.length; i++) { const arg = rest[i] if (arg.startsWith('--')) { - const key = arg.slice(2) + const raw = arg.slice(2) + const equals = raw.indexOf('=') + if (equals >= 0) { + values[raw.slice(0, equals)] = raw.slice(equals + 1) + continue + } + const key = raw const nextArg = rest[i + 1] if (nextArg !== undefined && !nextArg.startsWith('-')) { values[key] = nextArg @@ -216,16 +222,18 @@ function rejectRetiredEqlFlags( flags: Record<string, boolean>, values: Record<string, string>, ): void { + const present = (key: string) => + flags[key] === true || Object.hasOwn(values, key) const error = validateInstallFlags({ eqlVersion: values['eql-version'], - latest: flags.latest, - drizzle: flags.drizzle, + latest: present('latest'), + drizzle: present('drizzle'), name: values.name, out: values.out, - migration: flags.migration, - direct: flags.direct, + migration: present('migration'), + direct: present('direct'), migrationsDir: values['migrations-dir'], - excludeOperatorFamily: flags['exclude-operator-family'], + excludeOperatorFamily: present('exclude-operator-family'), }) if (error) { p.log.error(error) diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index 6f7f513bd..eac1f21f1 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -37,11 +37,8 @@ export interface ResolvedLifecycle { * * Deliberately NOT set when `candidates` is empty. That is the pure-v2 table * — a `<col>` / `<col>_encrypted` pair and nothing else — where there is no - * v3 column to mis-claim and so nothing to protect against. The v2 lifecycle - * is still implemented in `cutover.ts` / `drop.ts`, and those commands must - * keep reaching it; failing closed here would refuse a lifecycle this same - * build still performs, and tell the user to downgrade to reach it (#787 - * review). + * v3 column to mis-claim. The caller then emits its explicit fail-closed + * legacy-v2 diagnostic; no v2 mutation lifecycle remains. */ unresolvedHint?: string } @@ -111,18 +108,13 @@ export async function resolveColumnLifecycle( /** * Explain a failed resolution (`info === null`) to the user, or return - * `null` when the failure is fine to fall through to the v2 lifecycle. + * `null` when no v3 candidate exists and the caller should emit its explicit + * fail-closed legacy-v2 diagnostic. * - * The one fall-through case is "no EQL v3 columns at all", which the v2 - * phase/config preconditions turn into an accurate error ("not backfilled", - * "no pending config", …). Since `classifyEqlDomain` recognises `eql_v3_*` - * only, that case covers every pure-v2 table — both the pre-cutover pair - * (`<col>` / `<col>_encrypted`) and the post-cutover state where `<col>` was - * renamed onto the ciphertext. Neither column is ever a candidate, so a pure-v2 - * table reaches the v2 ladder here regardless of what the manifest recorded; - * a recorded `encryptedColumn` must NOT turn that into a refusal, because - * `cutover.ts` / `drop.ts` in this same build still implement that ladder - * (#787 review). + * The one fall-through case is "no EQL v3 columns at all". Since + * `classifyEqlDomain` recognises `eql_v3_*` only, that covers pure-v2 tables. + * Neither v2 column is a candidate, so the caller reaches its v2-retirement + * error regardless of what the manifest recorded; no mutation is attempted. * * A non-empty candidate list therefore means EQL v3 columns exist but none is * identifiable — the caller must fail closed with this message rather than @@ -137,11 +129,11 @@ export function explainUnresolved( unresolvedHint?: string, ): string | null { // "No EQL v3 columns at all" always falls through, even with a recorded hint. - // That is the pure-v2 table, and the v2 ladder in `cutover.ts` / `drop.ts` - // still handles it — the caller's own preconditions produce the accurate - // error. Ordered ahead of the hint branch deliberately: `resolveColumnLifecycle` - // already declines to set `unresolvedHint` on an empty candidate list, and this - // keeps the two agreeing for direct callers of this function (#787 review). + // That is the pure-v2 table; the caller's own fail-closed retirement branch + // produces the actionable error. Ordered ahead of the hint branch + // deliberately: `resolveColumnLifecycle` already declines to set + // `unresolvedHint` on an empty candidate list, and this keeps the two agreeing + // for direct callers of this function (#787 review). if (candidates.length === 0) return null // The recorded pairing points at a real column that is not an EQL v3 column, diff --git a/packages/cli/src/commands/init/index.ts b/packages/cli/src/commands/init/index.ts index 11c52960b..7a8288458 100644 --- a/packages/cli/src/commands/init/index.ts +++ b/packages/cli/src/commands/init/index.ts @@ -71,6 +71,16 @@ export async function initCommand( flags: Record<string, boolean>, values: Record<string, string> = {}, ) { + const retiredProxyFlag = ['proxy', 'no-proxy'].find( + (name) => flags[name] === true || Object.hasOwn(values, name), + ) + if (retiredProxyFlag) { + p.log.error( + `\`--${retiredProxyFlag}\` has been removed. EQL v3 stores query configuration in column domains and does not use CipherStash Proxy; remove this flag and select only the project integration (for example, \`--supabase\` or \`--drizzle\`).`, + ) + throw new CliExit(1) + } + const provider = resolveProvider(flags) p.intro('CipherStash Stack Setup') diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 0afe7b30a..08bb04a37 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -598,6 +598,12 @@ describe('renderSetupPrompt — plan templates are EQL v3-only', () => { expect(plan('cutover')).not.toMatch(/encrypt cutover/) }) + it('does not list cutover inside a brace-form or comma-separated encrypt command index', () => { + const out = renderSetupPrompt(baseCtx) + expect(out).not.toMatch(/encrypt\s*\{[^}]*\bcutover\b[^}]*\}/i) + expect(out).not.toMatch(/encrypt[^\n]*,\s*cutover\b/i) + }) + it('gives the complete plan the supported backfill, read-switch, and drop sequence', () => { const out = plan('complete') expect(hasCompletePlanSequence(out)).toBe(true) diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 6f66d06c7..86ccda87a 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -205,7 +205,7 @@ const SKILL_PURPOSES: Record<string, string> = { 'stash-dynamodb': 'DynamoDB encryption: per-item encrypt/decrypt, HMAC attribute keys, audit logging', 'stash-cli': - '`stash` command reference — `status`, `plan`, `impl`, `eql install`, `encrypt {backfill,cutover,drop}`, etc.', + '`stash` command reference — `status`, `plan`, `impl`, `eql install`, the staged EQL v3 read-switch workflow, and `encrypt {backfill,drop}`.', 'stash-supply-chain-security': 'supply-chain controls (post-install policy, lockfile integrity, etc.)', } diff --git a/packages/cli/tests/e2e/v2-retirement.e2e.test.ts b/packages/cli/tests/e2e/v2-retirement.e2e.test.ts index 1b5aad7da..eeaa73dca 100644 --- a/packages/cli/tests/e2e/v2-retirement.e2e.test.ts +++ b/packages/cli/tests/e2e/v2-retirement.e2e.test.ts @@ -25,6 +25,38 @@ describe('retired EQL v2 CLI surface', () => { expect(output(result)).toMatch(/self-adapts/i) }) + it.each([ + ['--proxy', '--proxy'], + ['--no-proxy', '--no-proxy'], + ['--proxy=true', '--proxy'], + ['--no-proxy=true', '--no-proxy'], + ] as const)('rejects retired init flag `%s` before init work', async (arg, flag) => { + const result = await runPiped(['init', arg], { timeoutMs: 2_000 }) + + expect(result.timedOut).toBe(false) + expect(result.exitCode).toBe(1) + expect(output(result)).toContain(flag) + expect(output(result)).toMatch(/removed|retired/i) + expect(output(result)).toMatch(/EQL v3/i) + }) + + it.each([ + '--eql-version=2', + '--latest=true', + '--drizzle=true', + '--name=install-eql', + '--out=drizzle', + '--migration=true', + '--migrations-dir=drizzle', + '--direct=true', + '--exclude-operator-family=true', + ])('rejects retired eql install equals form `%s`', async (arg) => { + const result = await runPiped(['eql', 'install', arg]) + + expect(result.exitCode).toBe(1) + expect(output(result)).toMatch(/removed/i) + }) + it.each([ [['db', 'push'], 'db push'], [['db', 'activate'], 'db activate'], diff --git a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts index 93bab05f9..5fdea7230 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts @@ -161,9 +161,12 @@ describe('encryptedSupabaseV3 factory', () => { ], }), ) - await expect( - encryptedSupabaseV3(fakeClient, { databaseUrl: 'postgres://x' }), - ).rejects.toThrow(/no EQL v3 encrypted columns found/) + const result = encryptedSupabaseV3(fakeClient, { + databaseUrl: 'postgres://x', + }) + await expect(result).rejects.toThrow(/no EQL v3 encrypted columns found/) + await expect(result).rejects.toThrow(/stash eql install --supabase/) + await expect(result).rejects.not.toThrow(/--eql-version/) expect(encryptionMock).not.toHaveBeenCalled() }) diff --git a/packages/stack-supabase/integration/wire.integration.test.ts b/packages/stack-supabase/integration/wire.integration.test.ts index cd10218a0..a890dc61d 100644 --- a/packages/stack-supabase/integration/wire.integration.test.ts +++ b/packages/stack-supabase/integration/wire.integration.test.ts @@ -223,7 +223,7 @@ const ADA_CREATED = new Date('2026-01-02T03:04:05.000Z') beforeAll(async () => { // EQL v3 and the Supabase grants are installed once per run by `globalSetup`, - // which shells out to the real `stash eql install --eql-version 3 --supabase`. + // which shells out to the real `stash eql install --supabase`. // Re-applying them here would only test a hand-rolled approximation. await sql.unsafe(`DROP TABLE IF EXISTS ${TABLE}`) await sql.unsafe(` diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index a05c7b81f..a2f7e6f75 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -208,7 +208,7 @@ export async function encryptedSupabase( if (encryptionSchemas.length === 0) { throw new Error( '[supabase v3]: no EQL v3 encrypted columns found in schema "public". ' + - 'Check that EQL v3 is installed (`stash eql install --eql-version 3`) ' + + 'Check that EQL v3 is installed (`stash eql install --supabase`) ' + 'and that at least one column uses an eql_v3 domain type.', ) } diff --git a/packages/stack/README.md b/packages/stack/README.md index 9e5710eda..b74a8caf6 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -153,11 +153,11 @@ Prefer the plain `Ord` domains unless you know your database supports the ORE op Install the EQL v3 SQL into your database with the stash CLI: ```bash -npx stash eql install --eql-version 3 +npx stash eql install # On Supabase, add --supabase to grant the anon/authenticated/service_role # roles access to the eql_v3 schemas — without it, encrypted queries fail with # "permission denied for schema eql_v3_internal": -npx stash eql install --eql-version 3 --supabase +npx stash eql install --supabase ``` In migrations, declare each encrypted column as its domain type: diff --git a/packages/stack/integration/shared/harness.integration.test.ts b/packages/stack/integration/shared/harness.integration.test.ts index ba92ee617..afe95bd82 100644 --- a/packages/stack/integration/shared/harness.integration.test.ts +++ b/packages/stack/integration/shared/harness.integration.test.ts @@ -7,7 +7,7 @@ import { afterAll, expect, it } from 'vitest' * Proves the harness itself, so a failure in the adapter suites is never * ambiguous between "the adapter is broken" and "the database was never set up". * - * `globalSetup` has already run `stash eql install --eql-version 3` against the + * `globalSetup` has already run `stash eql install` against the * configured database by the time this file executes. Nothing here is skippable: * an unconfigured run throws in `globalSetup`, before any test is collected. */ diff --git a/packages/test-kit/src/__tests__/install.test.ts b/packages/test-kit/src/__tests__/install.test.ts new file mode 100644 index 000000000..341734109 --- /dev/null +++ b/packages/test-kit/src/__tests__/install.test.ts @@ -0,0 +1,51 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const { execFileMock } = vi.hoisted(() => ({ execFileMock: vi.fn() })) + +vi.mock('node:child_process', () => ({ execFile: execFileMock })) +vi.mock('node:fs', () => ({ existsSync: vi.fn(() => true) })) + +const { installEqlV3 } = await import('../install.js') + +beforeEach(() => { + execFileMock.mockReset().mockImplementation((...args: unknown[]) => { + const callback = args.at(-1) as ( + error: Error | null, + stdout: string, + stderr: string, + ) => void + callback(null, '', '') + }) +}) + +describe('installEqlV3 CLI argv', () => { + it.each([ + ['postgres', []], + ['supabase', ['--supabase']], + ] as const)('uses the current %s install surface', async (variant, flags) => { + await installEqlV3('postgres://integration', variant) + + expect(execFileMock).toHaveBeenCalledTimes(1) + expect(execFileMock.mock.calls[0]?.[1]).toEqual([ + expect.stringMatching(/packages\/cli\/dist\/bin\/stash\.js$/), + 'eql', + 'install', + ...flags, + '--database-url', + 'postgres://integration', + ]) + }) + + it('reports the supported Supabase invocation when the installer fails', async () => { + execFileMock.mockImplementationOnce((...args: unknown[]) => { + const callback = args.at(-1) as (error: Error) => void + callback(new Error('installer exploded')) + }) + + const result = installEqlV3('postgres://integration', 'supabase') + await expect(result).rejects.toThrow( + /^stash eql install --supabase failed\./, + ) + await expect(result).rejects.not.toThrow(/--eql-version|--direct/) + }) +}) diff --git a/packages/test-kit/src/catalog.ts b/packages/test-kit/src/catalog.ts index 21935994f..93bd1e18a 100644 --- a/packages/test-kit/src/catalog.ts +++ b/packages/test-kit/src/catalog.ts @@ -287,7 +287,7 @@ const BIGINT_ERR = [9223372036854775808n, -9223372036854775809n] as const * the eql-3.0.0 bundle self-skips that statement on `insufficient_privilege`. * * Measured against `supabase/postgres:17.4.1.048` after `stash eql install - * --eql-version 3 --supabase --direct`, as the non-superuser `postgres` role: + * --supabase`, as the non-superuser `postgres` role: * the domains ARE created, but they cannot hold data. Their CHECK calls * `eql_v3_internal.ore_domain_unavailable()`, so the first INSERT raises * diff --git a/packages/test-kit/src/install.ts b/packages/test-kit/src/install.ts index cbb528fc0..04b73324e 100644 --- a/packages/test-kit/src/install.ts +++ b/packages/test-kit/src/install.ts @@ -44,8 +44,8 @@ export function dbVariant(): DbVariant { * Notes on the flags: * - `--supabase` applies the `eql_v3` AND `eql_v3_internal` grants to * anon/authenticated/service_role. The SECURITY INVOKER extractors need both. - * - `--direct` is required alongside it: `--supabase` alone prompts for an - * install mode (migration vs direct) and would hang a CI job. + * - EQL v3 is the only installable generation, and `eql install` is always a + * direct install, so no generation or mode flags are required. * - Verified against `supabase/postgres:17.4.1.048` as the NON-superuser * `postgres` role: no superuser connection is needed. */ @@ -61,8 +61,8 @@ export async function installEqlV3( ) } - const args = ['eql', 'install', '--eql-version', '3'] - if (variant === 'supabase') args.push('--supabase', '--direct') + const args = ['eql', 'install'] + if (variant === 'supabase') args.push('--supabase') args.push('--database-url', databaseUrl) try { @@ -76,7 +76,7 @@ export async function installEqlV3( } catch (cause) { const detail = cause instanceof Error ? cause.message : String(cause) throw new Error( - `stash eql install --eql-version 3${variant === 'supabase' ? ' --supabase --direct' : ''} failed.\n` + + `stash eql install${variant === 'supabase' ? ' --supabase' : ''} failed.\n` + 'This is the real installer, so a failure here is a real bug — do not work around it.\n' + detail, ) diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index e104be5a3..0816129a6 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -41,7 +41,28 @@ describe('runPostAgentSteps execution commands', () => { vi.mocked(childProcess.execSync).mockImplementation(() => Buffer.from('')) }) - it('executes eql install/db push using the detected runner (bun → bunx) when usesProxy=true', async () => { + it('reads legacy usesProxy context without running or recommending retired db push', async () => { + const info = vi.spyOn(p.log, 'info') + + await runPostAgentSteps({ + cwd: '/tmp/fake', + integration: 'supabase', + packageManager: bun, + gathered: { + installCommand: 'bun add @cipherstash/stack', + hasStashConfig: true, + usesProxy: true, + } as never, + }) + + const commands = vi + .mocked(childProcess.execSync) + .mock.calls.map((c) => c[0] as string) + expect(commands).not.toContain('bunx stash db push') + expect(info).not.toHaveBeenCalledWith(expect.stringMatching(/db push/i)) + }) + + it('executes eql install using the detected runner (bun → bunx) and ignores legacy usesProxy=true', async () => { await runPostAgentSteps({ cwd: '/tmp/fake', integration: 'supabase', @@ -58,14 +79,14 @@ describe('runPostAgentSteps execution commands', () => { .mocked(childProcess.execSync) .mock.calls.map((c) => c[0] as string) expect(commands).toContain('bunx stash eql install') - expect(commands).toContain('bunx stash db push') + expect(commands).not.toContain('bunx stash db push') // Sanity: no leftover npx forms for the cipherstash binaries. for (const cmd of commands) { expect(cmd).not.toMatch(/^npx @cipherstash/) } }) - it('skips eql install when hasStashConfig=true and still uses bunx for db push when usesProxy=true', async () => { + it('skips eql install when hasStashConfig=true and ignores legacy usesProxy=true', async () => { await runPostAgentSteps({ cwd: '/tmp/fake', integration: 'supabase', @@ -79,11 +100,11 @@ describe('runPostAgentSteps execution commands', () => { const commands = vi .mocked(childProcess.execSync) .mock.calls.map((c) => c[0] as string) - expect(commands).toContain('bunx stash db push') + expect(commands).not.toContain('bunx stash db push') expect(commands).not.toContain('bunx stash eql install') }) - it('falls back to npx when packageManager is undefined and usesProxy=true', async () => { + it('falls back to npx when packageManager is undefined and ignores legacy usesProxy=true', async () => { await runPostAgentSteps({ cwd: '/tmp/fake', integration: 'supabase', @@ -98,10 +119,10 @@ describe('runPostAgentSteps execution commands', () => { .mocked(childProcess.execSync) .mock.calls.map((c) => c[0] as string) expect(commands).toContain('npx stash eql install') - expect(commands).toContain('npx stash db push') + expect(commands).not.toContain('npx stash db push') }) - it('skips db push when usesProxy=false', async () => { + it('never runs retired db push when legacy usesProxy=false', async () => { await runPostAgentSteps({ cwd: '/tmp/fake', integration: 'supabase', diff --git a/packages/wizard/src/lib/gather.ts b/packages/wizard/src/lib/gather.ts index c190d16fb..66f4242b9 100644 --- a/packages/wizard/src/lib/gather.ts +++ b/packages/wizard/src/lib/gather.ts @@ -46,7 +46,7 @@ export interface GatheredContext { installCommand: string /** Whether stash.config.ts already exists. */ hasStashConfig: boolean - /** Whether the user runs CipherStash Proxy. False = SDK-only (post-agent skips `stash db push`). Sourced from .cipherstash/context.json; defaults to false when the file is missing or the field is absent. */ + /** Legacy context field retained for compatibility with existing context files. */ usesProxy: boolean } diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index df8f7ec0c..f34ff1c38 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -27,7 +27,7 @@ interface PostAgentOptions { const DRIZZLE_OUT_DIRS = ['drizzle', 'migrations', 'src/db/migrations'] /** - * Run all post-agent steps: install packages, push config, run migrations. + * Run all post-agent steps: install packages, install EQL, run migrations. */ export async function runPostAgentSteps(opts: PostAgentOptions): Promise<void> { const { cwd, integration, gathered, packageManager } = opts @@ -53,21 +53,9 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise<void> { ) } - // Step 3: Push encryption config (only when using Proxy) - if (gathered.usesProxy) { - await runStep( - 'Pushing encryption config to database...', - 'Encryption config pushed', - `${runner} stash db push`, - cwd, - ) - } else { - p.log.info( - 'Skipping `stash db push` — not using CipherStash Proxy. Run it manually if you ever switch to Proxy.', - ) - } - - // Step 4: Integration-specific migrations + // Step 3: Integration-specific migrations. Older gathered context may still + // carry `usesProxy`; it is compatibility data only. EQL v3 has no Proxy + // configuration to push, and the retired `stash db push` must never run. if (integration === 'drizzle') { await runStep( 'Generating Drizzle migration...', diff --git a/scripts/__tests__/no-removed-eql-version-flag.test.mjs b/scripts/__tests__/no-removed-eql-version-flag.test.mjs index 778bf95a5..cbc8323af 100644 --- a/scripts/__tests__/no-removed-eql-version-flag.test.mjs +++ b/scripts/__tests__/no-removed-eql-version-flag.test.mjs @@ -6,11 +6,17 @@ import { describe, expect, it } from 'vitest' const REPO_ROOT = resolve(fileURLToPath(import.meta.url), '../../..') -/** Tracked skills ship with the CLI and are copied into customer projects. */ -function shippedSkills() { +/** Tracked executable examples that describe the current public surface. */ +function publicCommandDocs() { const out = execFileSync( 'git', - ['ls-files', '-z', ':(glob)skills/*/SKILL.md'], + [ + 'ls-files', + '-z', + ':(glob)skills/*/SKILL.md', + ':(glob)packages/*/README.md', + 'docs/reference/supabase-sdk.md', + ], { cwd: REPO_ROOT, encoding: 'utf8', @@ -64,13 +70,15 @@ describe('obsolete eql install example detection', () => { }) }) -describe('shipped skill eql install examples use the current CLI', () => { - const files = shippedSkills() +describe('public eql install examples use the current CLI', () => { + const files = publicCommandDocs() - it('finds tracked shipped skills (guards against a silently-empty glob)', () => { + it('finds tracked public command docs (guards against a silently-empty glob)', () => { expect(files).not.toHaveLength(0) expect(files).toContain('skills/stash-drizzle/SKILL.md') expect(files).toContain('skills/stash-supabase/SKILL.md') + expect(files).toContain('packages/stack/README.md') + expect(files).toContain('docs/reference/supabase-sdk.md') }) it.each(files)('%s', (file) => { From 5b06d3ee2f55fea17ef629cebc9fda8f4c0084e8 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 10:33:13 +1000 Subject: [PATCH 105/123] fix(ci): sync EQL v3 migration guidance --- e2e/tests/package-managers.e2e.test.ts | 2 +- .../src/__tests__/rewrite-migrations.test.ts | 4 +++- packages/wizard/src/lib/rewrite-migrations.ts | 20 ++++++++++--------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/e2e/tests/package-managers.e2e.test.ts b/e2e/tests/package-managers.e2e.test.ts index bfe42de02..498f929cf 100644 --- a/e2e/tests/package-managers.e2e.test.ts +++ b/e2e/tests/package-managers.e2e.test.ts @@ -55,7 +55,7 @@ describe('CLI init providers — package-manager-aware Next Steps', () => { label: 'drizzle', create: createDrizzleProvider, firstStep: (r) => - `Set up your database: ${r} stash eql install --drizzle`, + `Set up your database: ${r} stash eql migration --drizzle`, }, { label: 'supabase', diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 1914febcf..5cefe86e2 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -328,7 +328,9 @@ describe('rewriteEncryptedAlterColumns', () => { const updated = fs.readFileSync(filePath, 'utf-8') expect(updated).toContain('safe ONLY if') expect(updated).toContain('constraints, defaults, and indexes') - expect(updated).toContain('stash encrypt') + expect(updated).toContain('EQL v3 path') + expect(updated).toContain('dual-write') + expect(updated).toContain('switch the app to the encrypted column') }) it('separates ADD/DROP/RENAME with --> statement-breakpoint', async () => { diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index 74393eac6..a050b4d05 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -683,10 +683,11 @@ export interface RewriteResult { * plaintext is destroyed. The commented UPDATE is a placeholder that can never * become real SQL (the encrypted value is the EQL envelope produced by ZeroKMS * via the client — there is no expression Postgres can evaluate to fill it), so - * a populated table must instead use the staged `stash encrypt` lifecycle - * (add → backfill via `@cipherstash/stack`'s `encryptModel` → cutover → drop), - * which keeps both columns alive across deploys. Each rewritten file carries a - * header comment saying exactly this. + * a populated table must instead use the staged EQL v3 lifecycle (add an + * encrypted twin → dual-write → backfill via `@cipherstash/stack`'s + * `encryptModel` → switch the application to the encrypted column by name → + * drop plaintext), which keeps both columns alive across deploys. Each + * rewritten file carries a header comment saying exactly this. * * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements * left for a human — ones outside the strict matcher (a hand-authored @@ -956,9 +957,10 @@ export async function sweepMigrationDirs( * composite, but this is equally true on both surfaces.) * * So the guidance does NOT tell the user to backfill and run this migration — - * that would still lose data on cutover. It points a populated table at the - * staged `stash encrypt` lifecycle (add → backfill → cutover → drop), which - * keeps both columns alive across deploys. + * that would still lose data. It points a populated table at the staged EQL v3 + * lifecycle (add an encrypted twin → dual-write → backfill → switch the + * application to the encrypted column by name → drop plaintext), which keeps + * both columns alive across deploys. */ function renderSafeAlter( table: string, @@ -975,8 +977,8 @@ function renderSafeAlter( `-- ${domain}. This ADD+DROP+RENAME equals DROP+ADD and is safe ONLY if`, `-- ${qualifiedTable} is empty. On a populated table it DESTROYS existing "${column}"`, '-- data (the new column starts NULL) — do NOT run it there. Use the staged', - "-- `stash encrypt` path instead: add -> backfill via @cipherstash/stack's", - '-- encryptModel in application code -> cutover -> drop.', + '-- EQL v3 path instead: add an encrypted twin -> dual-write -> backfill via', + '-- encryptModel -> switch the app to the encrypted column -> drop plaintext.', '-- NOTE: constraints, defaults, and indexes on the original column are NOT', '-- carried over by this ADD/DROP/RENAME — re-add any NOT NULL, DEFAULT,', '-- UNIQUE, or index definitions manually.', From 7d988c3f821b348a2380209bd8e44b4ec13c7b7c Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 11:02:25 +1000 Subject: [PATCH 106/123] fix(cli): address review feedback --- packages/cli/README.md | 2 +- packages/cli/src/__tests__/release-train.test.ts | 14 ++++++++++++++ .../db/__tests__/find-generated-migration.test.ts | 1 + packages/cli/src/commands/eql/migration.ts | 3 ++- .../init/lib/__tests__/setup-prompt.test.ts | 9 ++++----- packages/cli/src/commands/init/lib/setup-prompt.ts | 2 +- skills/stash-cli/SKILL.md | 2 +- 7 files changed, 24 insertions(+), 9 deletions(-) diff --git a/packages/cli/README.md b/packages/cli/README.md index 64d70e5e7..7bb470e2b 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -66,7 +66,7 @@ export default defineConfig({ The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory. -Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `db validate`, `eql status`, `db test-connection`, and `schema build`. +Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `db validate`, `eql status`, `db test-connection`, `schema build`, and `encrypt *`. --- diff --git a/packages/cli/src/__tests__/release-train.test.ts b/packages/cli/src/__tests__/release-train.test.ts index 7776d1e41..64dc1da48 100644 --- a/packages/cli/src/__tests__/release-train.test.ts +++ b/packages/cli/src/__tests__/release-train.test.ts @@ -33,6 +33,20 @@ describe('release train coverage', () => { } }) + it('pins the bare-project stash invocation in the published CLI skill', () => { + const cliManifest = JSON.parse( + readFileSync(resolve(CLI_ROOT, 'package.json'), 'utf8'), + ) as { version: string } + const skill = readFileSync( + resolve(REPO_ROOT, 'skills/stash-cli/SKILL.md'), + 'utf8', + ) + + expect(skill).toContain( + `npx --package=stash@${cliManifest.version} stash eql install --database-url 'postgres://...'`, + ) + }) + it('every train manifest exists and carries a version (what tsup will embed)', () => { // Exercises the exact inputs tsup.config.ts reads at build time, so a // renamed/moved workspace package fails HERE in source-mode tests, not diff --git a/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts b/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts index a5cfe479f..f75dfc948 100644 --- a/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts +++ b/packages/cli/src/commands/db/__tests__/find-generated-migration.test.ts @@ -36,6 +36,7 @@ describe('findGeneratedMigration', () => { '0010_install-eql.sql', '0011_install-eql.txt', // not .sql '0001_users.sql', // doesn't match the name + '9999_install-eql-backup.sql', // contains the name, but is not an exact match ]) { writeFileSync(join(dir, f), '') } diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index e211a6367..e68cb0b44 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -31,8 +31,9 @@ export async function findGeneratedMigration( `Drizzle output directory not found: ${outDir}\nMake sure drizzle-kit is configured correctly.`, ) } + const migrationSuffix = `_${migrationName}.sql` const matchingFiles = (await readdir(outDir)) - .filter((entry) => entry.endsWith('.sql') && entry.includes(migrationName)) + .filter((entry) => entry.endsWith(migrationSuffix)) .sort() if (matchingFiles.length === 0) { throw new Error( diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index 08bb04a37..5876b9f55 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -25,7 +25,7 @@ const baseCtx: SetupPromptContext = { } function hasCompletePlanSequence(prompt: string): boolean { - return /`pnpm dlx stash encrypt backfill`[\s\S]*application read switch.*encrypted column by name and deploy[\s\S]*`pnpm dlx stash encrypt drop`/i.test( + return /`pnpm dlx stash encrypt backfill`[\s\S]*application read switch.*encrypted column by name[\s\S]*`pnpm dlx stash encrypt drop`/i.test( prompt, ) } @@ -610,11 +610,10 @@ describe('renderSetupPrompt — plan templates are EQL v3-only', () => { expect(out).not.toContain('`cutover`') }) - it('requires the application read switch to be deployed before drop', () => { - const missingDeploy = - '`pnpm dlx stash encrypt backfill`, an application read switch to the EQL v3 encrypted column by name, `pnpm dlx stash encrypt drop`' + it('does not require a deployment in the no-deployed-application flow', () => { + const out = plan('complete') - expect(hasCompletePlanSequence(missingDeploy)).toBe(false) + expect(out).not.toMatch(/encrypted column by name and deploy/i) }) }) diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 86ccda87a..e35b5a449 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -760,7 +760,7 @@ function renderCompletePlanPrompt(ctx: SetupPromptContext): string { bullet( 'For migrate columns: the full step list with the exact CLI invocations (`' + cli + - ' encrypt backfill`, an application read switch to the EQL v3 encrypted column by name and deploy, `' + + ' encrypt backfill`, an application read switch to the EQL v3 encrypted column by name, `' + cli + ' encrypt drop`) and concrete `--table` / `--column` values.', ), diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 13054ee8e..44784cc2f 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -354,7 +354,7 @@ Gets a project from zero to a direct EQL v3 install. It loads an existing `stash The removed `--eql-version`, `--latest`, `--drizzle`, `--migration`, `--direct`, `--migrations-dir`, and `--exclude-operator-family` options fail clearly instead of being ignored. A request for EQL v2 points dump-recovery users to the upstream EQL 2.3.1 SQL release. New installs are EQL v3 only; its pinned bundle self-adapts when a database role cannot create the optional operator family. -**`--database-url` is a one-shot.** It installs against that database and leaves the project untouched — no config is loaded, and none is scaffolded, nor is an encryption client. This lets `npx stash eql install --database-url postgres://...` run in a bare project with no CipherStash dependencies. It also means the flag always wins: loading a config could pick up a parent-directory `databaseUrl` literal and install against the wrong database. +**`--database-url` is a one-shot.** It installs against that database and leaves the project untouched — no config is loaded, and none is scaffolded, nor is an encryption client. This lets `npx --package=stash@1.0.0-rc.4 stash eql install --database-url 'postgres://...'` run in a bare project with no CipherStash dependencies while pinning the CLI to this skill's release. It also means the flag always wins: loading a config could pick up a parent-directory `databaseUrl` literal and install against the wrong database. #### `eql migration` From 78b9ab36e75ed80eefb956306121f917cbd78faa Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 11:57:45 +1000 Subject: [PATCH 107/123] test(scripts): cover repository-linter race guards --- .../lint-no-hardcoded-runners.test.mjs | 37 ++++++++++++++++++- scripts/__tests__/vitest-config.test.mjs | 8 ++++ 2 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 scripts/__tests__/vitest-config.test.mjs diff --git a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs index 709ac63ed..a2c1a5d8e 100644 --- a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs +++ b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs @@ -12,8 +12,10 @@ const SCRIPT = resolve( function runScript(script, ...targets) { try { - execFileSync(process.execPath, [script, ...targets], { encoding: 'utf8' }) - return { exitCode: 0, output: '' } + const output = execFileSync(process.execPath, [script, ...targets], { + encoding: 'utf8', + }) + return { exitCode: 0, output } } catch (err) { return { exitCode: err.status, @@ -26,6 +28,25 @@ function run(target) { return runScript(SCRIPT, target) } +function runWithReaddirError(code) { + const probe = resolve( + fileURLToPath(import.meta.url), + '../../walk-error-probe.mjs', + ) + const dir = mkdtempSync(join(tmpdir(), 'lint-hardcoded-runners-walk-')) + const src = readFileSync(SCRIPT, 'utf8').replace( + "import { readdir } from 'node:fs/promises'", + `async function readdir() { const err = new Error('readdir failed'); err.code = '${code}'; throw err }`, + ) + try { + writeFileSync(probe, src) + return runScript(probe, dir) + } finally { + rmSync(probe, { force: true }) + rmSync(dir, { recursive: true, force: true }) + } +} + describe('lint-no-hardcoded-runners', () => { const fx = (name) => resolve(fileURLToPath(import.meta.url), `../fixtures/${name}`) @@ -51,6 +72,18 @@ describe('lint-no-hardcoded-runners', () => { expect(r.output).not.toMatch(/at ModuleJob/) }) + it('skips a directory that vanished while being walked', () => { + const r = runWithReaddirError('ENOENT') + expect(r.exitCode).toBe(0) + expect(r.output).toContain('OK') + }) + + it('rethrows unexpected errors encountered while walking', () => { + const r = runWithReaddirError('EACCES') + expect(r.exitCode).not.toBe(0) + expect(r.output).toContain('EACCES') + }) + // Matches the sibling linter: a target outside the repo rendered as a // `../../../../..` chain out of the root instead of naming the file. it('renders an absolute path for an offender outside the repo root', () => { diff --git a/scripts/__tests__/vitest-config.test.mjs b/scripts/__tests__/vitest-config.test.mjs new file mode 100644 index 000000000..1ad8a56da --- /dev/null +++ b/scripts/__tests__/vitest-config.test.mjs @@ -0,0 +1,8 @@ +import { describe, expect, it } from 'vitest' +import scriptsVitestConfig from '../vitest.config.mjs' + +describe('scripts vitest config', () => { + it('keeps script test files serialized', () => { + expect(scriptsVitestConfig.test.fileParallelism).toBe(false) + }) +}) From 2d21debf4fd9da0d76436e967a25a97f27edcac7 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:05:39 +1000 Subject: [PATCH 108/123] test(scripts): cover nested filesystem races --- .../lint-no-hardcoded-runners.test.mjs | 64 +++++++++++++++++-- scripts/lint-no-hardcoded-runners.mjs | 12 +++- 2 files changed, 71 insertions(+), 5 deletions(-) diff --git a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs index a2c1a5d8e..9c93b8e1d 100644 --- a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs +++ b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs @@ -28,16 +28,60 @@ function run(target) { return runScript(SCRIPT, target) } -function runWithReaddirError(code) { +function runWithReaddirError(code, { afterParent = false } = {}) { const probe = resolve( fileURLToPath(import.meta.url), '../../walk-error-probe.mjs', ) const dir = mkdtempSync(join(tmpdir(), 'lint-hardcoded-runners-walk-')) + const replacement = afterParent + ? `let readdirCalls = 0 +async function readdir() { + readdirCalls += 1 + if (readdirCalls === 1) { + return [{ name: 'vanished', isDirectory: () => true }] + } + const err = new Error('readdir failed') + err.code = '${code}' + throw err +}` + : `async function readdir() { const err = new Error('readdir failed'); err.code = '${code}'; throw err }` const src = readFileSync(SCRIPT, 'utf8').replace( "import { readdir } from 'node:fs/promises'", - `async function readdir() { const err = new Error('readdir failed'); err.code = '${code}'; throw err }`, + replacement, + ) + try { + writeFileSync(probe, src) + return runScript(probe, dir) + } finally { + rmSync(probe, { force: true }) + rmSync(dir, { recursive: true, force: true }) + } +} + +function runWithReadFileError(code) { + const probe = resolve( + fileURLToPath(import.meta.url), + '../../read-error-probe.mjs', ) + const dir = mkdtempSync(join(tmpdir(), 'lint-hardcoded-runners-read-')) + const src = readFileSync(SCRIPT, 'utf8') + .replace( + "import { readFileSync, statSync } from 'node:fs'", + `import { readFileSync as realReadFileSync, statSync } from 'node:fs' +function readFileSync(path, ...args) { + if (String(path).endsWith('vanished.ts')) { + const err = new Error('read failed') + err.code = '${code}' + throw err + } + return realReadFileSync(path, ...args) +}`, + ) + .replace( + "import { readdir } from 'node:fs/promises'", + "async function readdir() { return [{ name: 'vanished.ts', isDirectory: () => false }] }", + ) try { writeFileSync(probe, src) return runScript(probe, dir) @@ -72,8 +116,8 @@ describe('lint-no-hardcoded-runners', () => { expect(r.output).not.toMatch(/at ModuleJob/) }) - it('skips a directory that vanished while being walked', () => { - const r = runWithReaddirError('ENOENT') + it('skips a directory that vanished after its parent was enumerated', () => { + const r = runWithReaddirError('ENOENT', { afterParent: true }) expect(r.exitCode).toBe(0) expect(r.output).toContain('OK') }) @@ -84,6 +128,18 @@ describe('lint-no-hardcoded-runners', () => { expect(r.output).toContain('EACCES') }) + it('skips a file that vanished after its directory was enumerated', () => { + const r = runWithReadFileError('ENOENT') + expect(r.exitCode).toBe(0) + expect(r.output).toContain('OK') + }) + + it('rethrows unexpected errors reading an enumerated file', () => { + const r = runWithReadFileError('EACCES') + expect(r.exitCode).not.toBe(0) + expect(r.output).toContain('EACCES') + }) + // Matches the sibling linter: a target outside the repo rendered as a // `../../../../..` chain out of the root instead of naming the file. it('renders an absolute path for an offender outside the repo root', () => { diff --git a/scripts/lint-no-hardcoded-runners.mjs b/scripts/lint-no-hardcoded-runners.mjs index 26c6a2548..bdf058650 100644 --- a/scripts/lint-no-hardcoded-runners.mjs +++ b/scripts/lint-no-hardcoded-runners.mjs @@ -159,7 +159,17 @@ for (const target of TARGETS) { const shown = rel.startsWith('..') ? file : rel if (ALLOWLISTED_PATHS.has(rel)) continue if (/\.(test|spec)\.(ts|tsx|mts|cts)$/.test(file)) continue - const lines = readFileSync(file, 'utf8').split('\n') + let source + try { + source = readFileSync(file, 'utf8') + } catch (err) { + // A file can vanish after its directory was enumerated, just as a + // directory can vanish before the recursive readdir above. There is + // nothing left to lint; unexpected read failures must still surface. + if (err?.code === 'ENOENT') continue + throw err + } + const lines = source.split('\n') lines.forEach((line, idx) => { const matches = NPX_TOKEN.test(line) if (!matches) return From 487dc9b7feaae92b62cdaa56476dbfeb962ee59c Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:07:42 +1000 Subject: [PATCH 109/123] fix(cli): distinguish missing encrypted column from legacy EQL v2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `detectColumnEqlVersion` returns null both for a column whose domain isn't `eql_v3_*` and for a column that isn't there at all, so backfill's single error accused a user who had merely not added `<col>_encrypted` yet of running legacy EQL v2 — and handed them a remedy (migrate the domain) for a column that does not exist. Extract the guard into `assertEqlV3Target` and probe `columnExists` on the failure path only; a v3 answer already proves existence, so the happy path still makes one catalog query. Also mark the retired surface in two internal design plans (`stash db push`, `stash encrypt cutover`, the `eql_v2.*` wrappers, `commands/db/push.ts`) so a future reader doesn't chase files that were deleted with EQL v2. The historical narrative is left intact. --- .changeset/backfill-missing-column-message.md | 12 ++ docs/plans/cli-help-and-manifest.md | 11 +- docs/plans/encryption-migrations.md | 65 ++++++++--- .../encrypt/__tests__/backfill-target.test.ts | 103 ++++++++++++++++++ packages/cli/src/commands/encrypt/backfill.ts | 43 ++++++-- 5 files changed, 208 insertions(+), 26 deletions(-) create mode 100644 .changeset/backfill-missing-column-message.md create mode 100644 packages/cli/src/commands/encrypt/__tests__/backfill-target.test.ts diff --git a/.changeset/backfill-missing-column-message.md b/.changeset/backfill-missing-column-message.md new file mode 100644 index 000000000..4586f422e --- /dev/null +++ b/.changeset/backfill-missing-column-message.md @@ -0,0 +1,12 @@ +--- +'stash': patch +--- + +`stash encrypt backfill` now distinguishes a missing encrypted column from a +legacy EQL v2 one. The domain probe returns the same "not v3" answer for both, +so a user who had simply not added the `<col>_encrypted` column yet was told +they were on a legacy EQL v2 column and advised to migrate a domain that did not +exist. The command now reports that the column is absent, points at adding an +`eql_v3_*`-domain column and applying the migration, and mentions +`--encrypted-column` for non-standard names. The EQL v2 message is unchanged for +columns that really are present. diff --git a/docs/plans/cli-help-and-manifest.md b/docs/plans/cli-help-and-manifest.md index f79cd992b..60e620595 100644 --- a/docs/plans/cli-help-and-manifest.md +++ b/docs/plans/cli-help-and-manifest.md @@ -1,6 +1,11 @@ # Proposal: a command-descriptor registry for `stash` help + a `manifest --json` -**Status:** proposal / for discussion +**Status:** implemented — the registry and `stash manifest --json` shipped. +Kept as the design record. **Note (2026-07-29):** the command names used as +examples below are a snapshot of the surface at proposal time. `db push`, `db +install`, `db upgrade`, and `db status` no longer exist (the installer is `stash +eql install`), and `stash encrypt cutover` was removed with EQL v2. Run `stash +manifest --json` for the current surface — that is the point of the registry. **Area:** `packages/cli` **Motivated by:** the docs V2 CLI reference, which is generated from the CLI (cipherstash/docs#45). Today it parses `stash --help`; this proposal gives it — @@ -20,7 +25,9 @@ and agents — a real structured source, and makes per-command help consistent. 2. **Per-command help is inconsistent.** `auth` implements its own `--help` ([`commands/auth/index.ts`](../../packages/cli/src/commands/auth/index.ts)), but `stash eql install --help`, `stash db push --help`, etc. fall through to - the top-level help. There is no per-command help for most commands. + the top-level help. There is no per-command help for most commands. (`db + push` has since been removed — the drift it illustrates was real, and is + what the registry now prevents.) 3. **No machine-readable output.** Docs have to scrape `--help`, and agents running `npx stash` have no authoritative, versioned command surface to read. diff --git a/docs/plans/encryption-migrations.md b/docs/plans/encryption-migrations.md index b672867c9..eb47ee008 100644 --- a/docs/plans/encryption-migrations.md +++ b/docs/plans/encryption-migrations.md @@ -1,5 +1,27 @@ # Encryption Migrations — Implementation Plan +> **Status (2026-07-29): historical. Parts of the surface below no longer exist.** +> +> This document is kept as the design record for `cs_migrations`, the manifest, +> and the backfill engine — all of which shipped and are current. What was +> retired with the EQL v2 removal: +> +> - **`stash encrypt cutover` and the whole `cut-over` phase.** EQL v3 has no +> rename swap. The lifecycle is `schema-added → dual-writing → backfilling → +> backfilled → dropped`: backfill, switch the application to the encrypted +> column *by name*, then drop the plaintext column (which is the original +> `<col>`, not a `<col>_plaintext` left behind by a rename). +> - **`stash db push`, `db install`, `db upgrade`, `db status`.** `db push` and +> its `eql_v2_configuration` DAO are gone entirely; the installer is now +> `stash eql install`. +> - **Every `eql_v2.*` function call.** `stash` no longer installs or drives EQL +> v2. Existing v2 ciphertext stays readable; it is not an authoring or rollout +> target. +> +> Read the sections below as "what we planned in Phase 1", not as a description +> of the current CLI. `skills/stash-cli/SKILL.md` and +> `skills/stash-encryption/SKILL.md` are the current lifecycle reference. + ## Context CipherStash today can encrypt a column at rest via EQL + either Stack/Protect.js (client-side) or the CipherStash Proxy (transparent). What it *doesn't* have is a first-class way to migrate an **existing plaintext column** into an encrypted one safely in production. EQL ships the schema/config primitives (`add_column`, `migrate_config`, `rename_encrypted_columns`) but no backfill orchestrator, no per-column phase tracking, and no resumable data mover. Today users have to wire this up themselves, which is both the biggest onboarding friction and the biggest correctness risk (partial backfills, reads on the wrong column, silent plaintext leaks). @@ -10,6 +32,10 @@ This plan adds a shared migration substrate — CLI + library — that walks eac schema-added → dual-writing → backfilling → backfilled → cut-over → dropped ``` +*The shipped lifecycle drops `cut-over`: `schema-added → dual-writing → +backfilling → backfilled → dropped`. Status readers still display legacy +`cut_over` rows so old history remains printable.* + The same mechanism serves Stack and Proxy users. Phase 1 ships the status inspector and the backfill engine (the two pieces with no good existing workaround). The other phases get lightweight commands that mostly orchestrate existing EQL functions and delegate the code changes (e.g. wiring dual-writes into the persistence layer) to the agent handoff that `stash init` set up — Claude / Codex / AGENTS.md, with the relevant skills already installed. ## Scope (Phase 1) @@ -19,8 +45,8 @@ The same mechanism serves Stack and Proxy users. Phase 1 ships the status inspec 3. A new `cs_migrations` table + small library (`@cipherstash/migrate` or co-located in `@cipherstash/stack`) that the CLI commands drive. Library is exported so users can embed backfill in their own workers/cron later without new infra. 4. `.cipherstash/migrations.json` repo manifest = intent (desired columns + index set + target phase). `stash encrypt plan` diffs intent vs. observed state. 5. Thin wrappers for the post-backfill phases so users can drive end-to-end from the CLI today, even if those phases are mostly pass-throughs: - - `stash encrypt cutover` — wraps `eql_v2.rename_encrypted_columns()` + `eql_v2.reload_config()` (via Proxy if present). - - `stash encrypt drop` — emits a migration file that drops `<col>_plaintext`. + - `stash encrypt cutover` — wraps `eql_v2.rename_encrypted_columns()` + `eql_v2.reload_config()` (via Proxy if present). *(Removed — see Status. There is no cut-over rename.)* + - `stash encrypt drop` — emits a migration file that drops `<col>_plaintext`. *(Shipped, but it drops the original `<col>`: with no rename, there is no `<col>_plaintext`.)* **Out of Phase 1:** Proxy-mode backfill (Phase 2), CS-hosted backfill runner (Phase 3), upstreaming `cs_migrations` into EQL as `eql_v2_migrations` (Phase 3), `stash encrypt update` for re-encrypting an already-cut-over column with new EQL config (next change). @@ -131,7 +157,12 @@ Backfill now owns the bookmark. The first time backfill runs against a column, i If the user lies (says yes but dual-writes aren't actually live), rows inserted during the backfill land in plaintext only and the recovery is `stash encrypt backfill --force`, which drops the `<col>_encrypted IS NULL` guard and re-encrypts every row regardless of current state. Audit trail: the `force` run is recorded with `details.force = true` in `cs_migrations` so it shows up in `encrypt status` history as a distinct event. -### 5. `stash encrypt cutover` +### 5. `stash encrypt cutover` *(never shipped in this form; removed)* + +> Nothing in this section exists. The command, the `cut-over` phase transition, +> and every `eql_v2.*` call below were removed with EQL v2. In EQL v3 the +> application moves to the encrypted column by name after backfill — no rename, +> no config promotion, no Proxy reload. For each column in `backfilled` phase, in a single transaction: @@ -155,6 +186,10 @@ Record `cut_over` event. App's existing `SELECT email FROM users` returns the en ### 6. `stash encrypt drop` +> Shipped, but gated on `backfilled` (not `cut_over`, which no longer exists) +> and it drops the original `<col>`. It also re-verifies coverage under an +> `ACCESS EXCLUSIVE` lock inside the generated migration. + For columns in `cut_over` phase: 1. Read Drizzle / Prisma / other migration tooling from repo (we already detect this in init). @@ -168,17 +203,17 @@ For columns in `cut_over` phase: - `status.ts` — new - `plan.ts` — new (diffs intent vs. observed) - `backfill.ts` — new (also handles the dual-write confirmation + `--force` recovery path) - - `cutover.ts` — new + - `cutover.ts` — new *(never shipped / removed)* - `drop.ts` — new - `stack/packages/cli/src/bin/stash.ts` — register `encrypt` subcommand (analogous to existing `db` registration at ~line 237) - `stack/packages/migrate/` — **new package** (library the CLI drives) - `src/state.ts` — `cs_migrations` DAO (append event, get latest, get progress) - `src/backfill.ts` — the chunked loop, exported as `runBackfill({ table, column, client, db, chunkSize, signal })` - `src/cursor.ts` — keyset pagination primitive - - `src/eql.ts` — thin wrappers over `eql_v2.*` functions (rename, reload, config read) + - `src/eql.ts` — thin wrappers over `eql_v2.*` functions (rename, reload, config read) *(never shipped; `@cipherstash/migrate` has no `eql_v2` wrappers — see `src/version.ts` for the v3 domain classifier that replaced this idea)* - `src/manifest.ts` — read/write `.cipherstash/migrations.json` - - `src/schema.sql` — `cs_migrations` DDL, installed by `db install` or an explicit `encrypt install` step -- `stack/packages/cli/src/commands/db/install.ts` — extend to install `cs_migrations` schema alongside EQL + - `src/schema.sql` — `cs_migrations` DDL, installed by `db install` or an explicit `encrypt install` step *(`db install` is now `stash eql install`)* +- `stack/packages/cli/src/commands/db/install.ts` — extend to install `cs_migrations` schema alongside EQL *(moved: the installer is `commands/eql/install.ts`)* - `stack/packages/cli/src/commands/init/lib/introspect.ts` — `introspectDatabase` lives here post-#395; `status.ts` reuses it via direct import - `stack/packages/cli/src/config/` — extend `stash.config.ts` loader so backfill subprocess can dynamically import user's encryption client - `stack/packages/cli/package.json` — add `@cipherstash/migrate` dep @@ -190,8 +225,8 @@ For columns in `cut_over` phase: - `introspectDatabase` in `packages/cli/src/commands/init/lib/introspect.ts` (moved here from the old wizard package as part of the #395 init handoff work). - `loadStashConfig` + dynamic encryption-client import lives at `packages/cli/src/config/`. Re-export from `@cipherstash/migrate` so library consumers don't need a hidden cross-package import. - `rewriteEncryptedAlterColumns` in `packages/cli/src/commands/db/rewrite-migrations.ts` — the phase-1 schema-add is already solved by drizzle-kit + this rewriter. The new commands **will not** re-solve it. -- EQL functions (Postgres): `eql_v2.add_column`, `eql_v2.add_search_config`, `eql_v2.migrate_config`, `eql_v2.activate_config`, `eql_v2.rename_encrypted_columns`, `eql_v2.reload_config`, `eql_v2.count_encrypted_with_active_config`, `eql_v2.select_pending_columns`, `eql_v2.ready_for_encryption`. -- `db push` in `packages/cli/src/commands/db/push.ts` — already handles writing to `eql_v2_configuration`; reuse the DAO. +- EQL functions (Postgres): `eql_v2.add_column`, `eql_v2.add_search_config`, `eql_v2.migrate_config`, `eql_v2.activate_config`, `eql_v2.rename_encrypted_columns`, `eql_v2.reload_config`, `eql_v2.count_encrypted_with_active_config`, `eql_v2.select_pending_columns`, `eql_v2.ready_for_encryption`. *(None of these are called any more — `stash` installs and drives EQL v3 only. EQL v3 needs no configuration table: the domain types carry the config.)* +- `db push` in `packages/cli/src/commands/db/push.ts` — already handles writing to `eql_v2_configuration`; reuse the DAO. *(Both the command and that file were deleted; there is no `packages/cli/src/commands/db/push.ts` to read.)* ## Verification @@ -201,15 +236,15 @@ For columns in `cut_over` phase: - Manifest reader: schema validation, drift detection. 2. **Integration (Drizzle, local Postgres)** - Seed 100k-row `users` table with plaintext `email`. - - `stash db install` → EQL + `cs_migrations` installed. + - `stash db install` → EQL + `cs_migrations` installed. *(now `stash eql install`)* - Manually wire dual-write in the test app's insert code (simulates user + agent handoff). - `stash encrypt backfill --table users --column email` → interactive prompt confirms dual-writes are deployed, appends `dual_writing` event, runs to completion; progress output sane; `COUNT(*) WHERE email_encrypted IS NULL` = 0. - Same flow non-interactively: `--confirm-dual-writes-deployed` accepted, loud warning printed. - Kill mid-backfill (SIGINT) → re-run with `--resume` → completes without duplicate encryption; `cs_migrations` shows continuous cursor progression; no second `dual_writing` event. - `stash encrypt backfill --force` after manually corrupting an encrypted row → re-encrypts every row; `details.force = true` recorded in `cs_migrations`. - `stash encrypt status` → shows `backfilled`. - - `stash encrypt cutover` → rename executes; app (still running, reads `email`) now gets decrypted ciphertext transparently. - - `stash encrypt drop` → migration file emitted; apply; `email_plaintext` gone. + - `stash encrypt cutover` → rename executes; app (still running, reads `email`) now gets decrypted ciphertext transparently. *(Removed. The v3 equivalent is a deploy: point the app at `email_encrypted` and decrypt through the encryption client.)* + - `stash encrypt drop` → migration file emitted; apply; `email_plaintext` gone. *(Drops `email`, the original plaintext column.)* 3. **Idempotency** - Run `backfill` twice with no kill — second run does 0 writes. - Concurrent runners on two shells — both converge, no duplicate writes, no missed rows. @@ -237,8 +272,10 @@ For columns in `cut_over` phase: ## Open items flagged (decisions already made) +> Two of these were later reversed by the move to EQL v3 — marked inline. + - Phase 1 runtime mode = Protect/Stack client-side only. -- Phase 4 default cutover mechanism = `eql_v2.rename_encrypted_columns()` (transparent to app code). -- State store = repo manifest + `eql_v2_configuration` (EQL intent) + new `cs_migrations` table (runtime state). +- Phase 4 default cutover mechanism = `eql_v2.rename_encrypted_columns()` (transparent to app code). *(Reversed: no cut-over mechanism at all.)* +- State store = repo manifest + `eql_v2_configuration` (EQL intent) + new `cs_migrations` table (runtime state). *(Reversed: EQL v3 carries intent in the column's domain type, so there is no configuration table.)* - Phase 1 shipping scope = status + backfill first-class; other phases as thin wrappers. - `cs_migrations` is CLI-owned for now, explicitly designed to be upstreamed into EQL as `eql_v2_migrations` in a later release so both Stack and Proxy own it jointly. diff --git a/packages/cli/src/commands/encrypt/__tests__/backfill-target.test.ts b/packages/cli/src/commands/encrypt/__tests__/backfill-target.test.ts new file mode 100644 index 000000000..04ae3adad --- /dev/null +++ b/packages/cli/src/commands/encrypt/__tests__/backfill-target.test.ts @@ -0,0 +1,103 @@ +/** + * Pins `assertEqlV3Target`'s two distinct failure modes. + * + * `detectColumnEqlVersion` returns `null` for BOTH "the column exists but its + * type is not an `eql_v3_*` domain" AND "there is no such column". Collapsing + * them into one message told a user who simply had not added the encrypted + * column yet that they were on a legacy EQL v2 column — a diagnosis with no + * relation to their actual problem, and a remedy (migrate the domain) they + * cannot act on. + */ + +import { beforeEach, describe, expect, it, vi } from 'vitest' + +// Only the two catalog probes are replaced; everything else in +// `@cipherstash/migrate` stays real so a rename on either side fails loudly +// rather than silently mocking a function that no longer exists. +const detectColumnEqlVersion = vi.hoisted(() => + vi.fn(async (): Promise<2 | 3 | null> => null), +) +const columnExists = vi.hoisted(() => vi.fn(async (): Promise<boolean> => true)) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal<typeof import('@cipherstash/migrate')>()), + detectColumnEqlVersion, + columnExists, +})) + +// `backfill.ts` pulls in the encryption-client loader and clack at module +// scope; neither is reachable from the guard, but both must import cleanly. +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + confirm: vi.fn(async () => true), + isCancel: vi.fn(() => false), + log: { + info: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + step: vi.fn(), + }, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('../context.js', () => ({ + loadEncryptionContext: vi.fn(async () => ({ client: {}, tables: new Map() })), + requireTable: vi.fn(), +})) + +import type pg from 'pg' +import { assertEqlV3Target, BackfillConfigError } from '../backfill.js' + +// The guard only ever hands this to the mocked probes. +const db = {} as pg.ClientBase + +describe('assertEqlV3Target', () => { + beforeEach(() => { + vi.clearAllMocks() + }) + + it('accepts an EQL v3 domain without probing for existence', async () => { + detectColumnEqlVersion.mockResolvedValue(3) + + await expect( + assertEqlV3Target(db, 'users', 'email_encrypted'), + ).resolves.toBe(3) + // The happy path must not pay for a second catalog round-trip. + expect(columnExists).not.toHaveBeenCalled() + }) + + it('reports a MISSING column as missing, not as legacy EQL v2', async () => { + detectColumnEqlVersion.mockResolvedValue(null) + columnExists.mockResolvedValue(false) + + const error = await assertEqlV3Target(db, 'users', 'email_encrypted').catch( + (e: unknown) => e, + ) + + expect(error).toBeInstanceOf(BackfillConfigError) + const message = (error as Error).message + expect(message).toContain('does not exist on users') + expect(message).toContain('eql_v3_*') + // The remedy is to add the column, not to migrate a domain that isn't there. + expect(message).toContain('--encrypted-column') + expect(message).not.toContain('EQL v2') + }) + + it('keeps the legacy-v2 diagnosis for a column that DOES exist', async () => { + detectColumnEqlVersion.mockResolvedValue(null) + columnExists.mockResolvedValue(true) + + const error = await assertEqlV3Target(db, 'users', 'email_encrypted').catch( + (e: unknown) => e, + ) + + expect(error).toBeInstanceOf(BackfillConfigError) + const message = (error as Error).message + expect(message).toContain('is not an EQL v3 domain') + expect(message).toContain('no longer backfills legacy EQL v2 columns') + expect(message).not.toContain('does not exist') + }) +}) diff --git a/packages/cli/src/commands/encrypt/backfill.ts b/packages/cli/src/commands/encrypt/backfill.ts index 4da4b63eb..b4e3fdcf4 100644 --- a/packages/cli/src/commands/encrypt/backfill.ts +++ b/packages/cli/src/commands/encrypt/backfill.ts @@ -1,5 +1,6 @@ import { appendEvent, + columnExists, detectColumnEqlVersion, type ManifestColumn, progress, @@ -137,19 +138,11 @@ export async function backfillCommand(options: BackfillCommandOptions) { const encryptedColumn = options.encryptedColumn ?? `${options.column}_encrypted` - // Backfill authors ciphertext, so it accepts only the current EQL v3 - // domains. Legacy v2 remains visible to status/read diagnostics but is no - // longer a writable rollout target. - const eqlVersion = await detectColumnEqlVersion( + const eqlVersion = await assertEqlV3Target( db, options.table, encryptedColumn, ) - if (eqlVersion !== 3) { - throw new BackfillConfigError( - `${options.table}.${encryptedColumn} is not an EQL v3 domain. stash no longer backfills legacy EQL v2 columns; migrate the schema to an eql_v3_* domain first.`, - ) - } p.log.info( `${options.table}.${encryptedColumn} is EQL v3 — lifecycle is backfill → switch the app to the encrypted column by name → drop (no cut-over rename).`, ) @@ -314,13 +307,43 @@ export async function backfillCommand(options: BackfillCommandOptions) { * upstream encryption errors, which can embed plaintext samples and are * suppressed by the catch block in {@link backfillCommand}. */ -class BackfillConfigError extends Error { +export class BackfillConfigError extends Error { constructor(message: string) { super(message) this.name = 'BackfillConfigError' } } +/** + * Backfill authors ciphertext, so it accepts only the current EQL v3 domains. + * Legacy v2 remains visible to status/read diagnostics but is no longer a + * writable rollout target. + * + * `detectColumnEqlVersion` answers `null` for a non-v3 domain AND for a column + * that isn't there, so the failure path pays for one extra probe to tell them + * apart. Conflating them accused a user who had merely not added the encrypted + * twin yet of running legacy v2, and handed them a remedy — migrate the domain + * — for a column that does not exist. The probe stays off the happy path: a v3 + * answer already proves existence. + */ +export async function assertEqlV3Target( + db: pg.ClientBase, + table: string, + encryptedColumn: string, +): Promise<3> { + const eqlVersion = await detectColumnEqlVersion(db, table, encryptedColumn) + if (eqlVersion === 3) return 3 + + if (!(await columnExists(db, table, encryptedColumn))) { + throw new BackfillConfigError( + `Column ${encryptedColumn} does not exist on ${table}. Add the encrypted destination column with an eql_v3_* domain type to your schema and apply the migration before backfilling. If your schema names it something other than ${encryptedColumn}, pass --encrypted-column <name>.`, + ) + } + throw new BackfillConfigError( + `${table}.${encryptedColumn} is not an EQL v3 domain. stash no longer backfills legacy EQL v2 columns; migrate the schema to an eql_v3_* domain first.`, + ) +} + /** * Pick the schema-column key for this physical column. The drizzle * helper (`extractEncryptionSchema`) keys by the physical encrypted name; From 9c7ff43b329ffd72a9459b72f1663a3c3c58c669 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:08:48 +1000 Subject: [PATCH 110/123] test(scripts): capture successful stderr output --- .../lint-no-hardcoded-runners.test.mjs | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs index 9c93b8e1d..d13c2ad9f 100644 --- a/scripts/__tests__/lint-no-hardcoded-runners.test.mjs +++ b/scripts/__tests__/lint-no-hardcoded-runners.test.mjs @@ -1,4 +1,4 @@ -import { execFileSync } from 'node:child_process' +import { spawnSync } from 'node:child_process' import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { join, resolve } from 'node:path' @@ -11,16 +11,12 @@ const SCRIPT = resolve( ) function runScript(script, ...targets) { - try { - const output = execFileSync(process.execPath, [script, ...targets], { - encoding: 'utf8', - }) - return { exitCode: 0, output } - } catch (err) { - return { - exitCode: err.status, - output: String(err.stdout) + String(err.stderr), - } + const result = spawnSync(process.execPath, [script, ...targets], { + encoding: 'utf8', + }) + return { + exitCode: result.status, + output: String(result.stdout) + String(result.stderr), } } @@ -99,6 +95,19 @@ describe('lint-no-hardcoded-runners', () => { expect(run(fx('clean.ts')).exitCode).toBe(0) }) + it('captures stderr from a successful script', () => { + const dir = mkdtempSync(join(tmpdir(), 'lint-hardcoded-runners-stderr-')) + const probe = join(dir, 'stderr-probe.mjs') + try { + writeFileSync(probe, "process.stderr.write('successful warning\\n')\n") + const r = runScript(probe) + expect(r.exitCode).toBe(0) + expect(r.output).toContain('successful warning') + } finally { + rmSync(dir, { recursive: true, force: true }) + } + }) + it('fails on a hardcoded `npx ...` string literal', () => { const r = run(fx('offender.ts')) expect(r.exitCode).toBe(1) From e155956d71a04a8b6aee6db343deeb3b279a7c6a Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:26:43 +1000 Subject: [PATCH 111/123] fix: finish v2 removal release gates --- .changeset/adapter-release-readiness.md | 28 ++++ .changeset/decrypt-chaining-docs.md | 5 +- .changeset/dynamodb-eql-v3.md | 11 +- .changeset/remove-eql-v2-drizzle-root.md | 15 +-- .../remove-eql-v2-migrate-classifier.md | 2 +- .changeset/remove-eql-v2-packages.md | 1 - .../remove-eql-v2-supabase-authoring.md | 26 ++-- .changeset/remove-eql-v2-supabase-skill.md | 3 +- e2e/package.json | 1 + .../init/lib/__tests__/install-skills.test.ts | 8 ++ .../init/lib/__tests__/setup-prompt.test.ts | 3 +- .../cli/src/commands/init/lib/setup-prompt.ts | 2 +- .../lock-context.integration.test.ts | 24 ++-- .../null-persistence.integration.test.ts | 10 +- .../relational.integration.test.ts | 107 +++++++-------- .../stack-drizzle/tsconfig.typecheck.json | 6 +- .../__tests__/supabase-v3-builder.test.ts | 125 +++++++++++++++++- .../__tests__/supabase-v3-factory.test.ts | 27 ++++ packages/stack-supabase/src/helpers.ts | 72 +++++++++- packages/stack-supabase/src/index.ts | 15 +++ packages/stack-supabase/src/query-builder.ts | 35 ++++- packages/stack-supabase/src/query-dbspace.ts | 10 +- packages/stack-supabase/src/query-encrypt.ts | 69 ++++++---- packages/stack-supabase/src/query-filters.ts | 16 ++- packages/stack-supabase/src/query-results.ts | 17 ++- packages/stack-supabase/src/types.ts | 16 ++- .../stack/__tests__/typed-client-v3.test.ts | 36 +++++ .../__tests__/wasm-inline-models.test.ts | 12 ++ packages/stack/src/adapter-kit.ts | 4 + packages/stack/src/encryption/v3.ts | 9 +- .../stack/src/eql/v3/date-reconstruction.ts | 57 ++++++++ packages/stack/src/wasm-inline.ts | 11 +- packages/wizard/src/lib/install-skills.ts | 13 +- pnpm-lock.yaml | 3 + 34 files changed, 609 insertions(+), 190 deletions(-) create mode 100644 .changeset/adapter-release-readiness.md create mode 100644 packages/stack/src/eql/v3/date-reconstruction.ts diff --git a/.changeset/adapter-release-readiness.md b/.changeset/adapter-release-readiness.md new file mode 100644 index 000000000..f0ced1557 --- /dev/null +++ b/.changeset/adapter-release-readiness.md @@ -0,0 +1,28 @@ +--- +'@cipherstash/stack': patch +'@cipherstash/stack-supabase': patch +'stash': patch +'@cipherstash/wizard': patch +--- + +Finish the EQL v2-removal release gates and adapter correctness pass. + +- Supabase preserves nested PostgREST boolean expressions and + `referencedTable`, never sends nullish encrypted search operands as + plaintext, honours escaped LIKE metacharacters, rejects CSV result mode + before decryption, and diagnoses the removed object-form factory call. +- Native, WASM, and Supabase model decryption reconstruct valid date and + timestamp values consistently, including nested paths, aliases, and bulk + results, while leaving invalid values unchanged. +- `stash init` names the concrete `public.eql_v3_*` domain family and gives + `public.eql_v3_text_search` as a valid Supabase example. +- CLI and wizard skill selection stay in parity for every integration, + including the Prisma Next skill, and verify that each selected skill has a + `SKILL.md`. + +The final 1.0 integration surface is `Encryption` from +`@cipherstash/stack/v3`, the `@cipherstash/stack-drizzle` package root, and +`encryptedSupabase` from `@cipherstash/stack-supabase`. DynamoDB decrypt +operations retain `.audit()` on the typed `Encryption` client. Existing EQL v2 +ciphertext remains readable through the core client; authoring and adapter +writes use EQL v3. diff --git a/.changeset/decrypt-chaining-docs.md b/.changeset/decrypt-chaining-docs.md index 19bcbb8ea..36afc506b 100644 --- a/.changeset/decrypt-chaining-docs.md +++ b/.changeset/decrypt-chaining-docs.md @@ -31,8 +31,9 @@ other two to guess: for Drizzle, the `encryptedSupabase` wrapper's own filters for Supabase, the `eql*` column operators for Prisma Next, and `client.encryptQuery(...)` for a plain Postgres project. -- **Schema authoring.** The `types.*` column factories for Drizzle, the - `eql_v3_encrypted` domain in migration SQL for Supabase, the `cipherstash.*` +- **Schema authoring.** The `types.*` column factories for Drizzle, a concrete + `public.eql_v3_*` domain such as `public.eql_v3_text_search` in migration SQL + for Supabase, the `cipherstash.*` field constructors in `schema.prisma` for Prisma Next, and `encryptedTable` for plain Postgres. Prisma Next was previously sent at `types.*` / `encryptedTable` — the client `stash schema build` explicitly refuses to diff --git a/.changeset/dynamodb-eql-v3.md b/.changeset/dynamodb-eql-v3.md index 54b960c92..8ba065f9d 100644 --- a/.changeset/dynamodb-eql-v3.md +++ b/.changeset/dynamodb-eql-v3.md @@ -6,10 +6,9 @@ `encryptedDynamoDB` now accepts EQL v3 tables. Pass a table built with `encryptedTable` + the `types.*` domains from -`@cipherstash/stack/v3` (or `@cipherstash/stack/eql/v3`) to any of -`encryptModel`, `bulkEncryptModels`, `decryptModel`, `bulkDecryptModels`. Both -the typed client from `EncryptionV3` and the nominal client from -`Encryption({ config: { eqlVersion: 3 } })` are accepted. +`@cipherstash/stack/v3` to any of `encryptModel`, `bulkEncryptModels`, +`decryptModel`, or `bulkDecryptModels`. Build the typed client with +`Encryption({ schemas: [table] })`. EQL v2 tables continue to be **readable** — `decryptModel` / `bulkDecryptModels` still accept one, so existing items stay accessible. Writing @@ -36,8 +35,8 @@ Notes on capability: path — `{ 'profile.ssn': types.TextEq('profile.ssn') }`. The model is matched by dotted path, so `{ profile: { ssn } }` resolves, and the nested attribute keeps its `__hmac` for key conditions. -- Audit metadata on `decryptModel` / `bulkDecryptModels` requires the nominal - client; the `EncryptionV3` client has no audit surface on decrypt. +- The typed `Encryption` client supports `.audit()` on `decryptModel` and + `bulkDecryptModels`, including when used through the DynamoDB adapter. The DynamoDB adapter also gains its first test coverage — across the v2 and v3 paths, where it previously had none. diff --git a/.changeset/remove-eql-v2-drizzle-root.md b/.changeset/remove-eql-v2-drizzle-root.md index 64d918746..eb06bd693 100644 --- a/.changeset/remove-eql-v2-drizzle-root.md +++ b/.changeset/remove-eql-v2-drizzle-root.md @@ -11,17 +11,16 @@ Remove the EQL v2 authoring surface from `@cipherstash/stack-drizzle` and collap - The EQL v2 root exports are gone: `encryptedType`, the v2 `extractEncryptionSchema`, the v2 `createEncryptionOperators` (including the `like` / `ilike` operators), and `EncryptionConfigError`. Authoring or querying `eql_v2_encrypted` columns through Drizzle is no longer supported. - The `./v3` subpath is **removed** from the package `exports` map and `typesVersions`. The EQL v3 implementation is now the package root, and the `*V3` names are de-suffixed (`createEncryptionOperatorsV3` → `createEncryptionOperators`, `extractEncryptionSchemaV3` → `extractEncryptionSchema`). This is a **hard break with no alias**: post-collapse the root names would collide with the removed v2 names, and keeping an alias would silently type-check v2 call sites against v3 semantics. -**Migration** — import the v3 surface from the package root instead of `./v3`, and drop the `V3` suffix: - -```diff -- import { types, extractEncryptionSchemaV3, createEncryptionOperatorsV3 } from '@cipherstash/stack-drizzle/v3' -+ import { types, extractEncryptionSchema, createEncryptionOperators } from '@cipherstash/stack-drizzle' -``` +**Migration** — import `types`, `extractEncryptionSchema`, and +`createEncryptionOperators` from the `@cipherstash/stack-drizzle` package root. The `types.*` column factories, `makeEqlV3Column` / `getEqlV3Column` / `isEqlV3Column`, the codec helpers (`v3ToDriver` / `v3FromDriver` / `EqlV3CodecError`), and `EncryptionOperatorError` are unchanged apart from moving to the root. Existing EQL v2 ciphertext remains decryptable via `@cipherstash/stack` — only the Drizzle-side v2 authoring and query-building is removed. -**`stash` (patch):** `stash init --drizzle` scaffolded the removed surface — the generated encryption-client file and the Drizzle placeholder both imported `extractEncryptionSchemaV3` from `@cipherstash/stack-drizzle/v3`, so a freshly-initialised project would not resolve against this release. Both now emit the collapsed root import. The bundled `stash-drizzle` and `stash-encryption` skills are updated to match. +**`stash` (patch):** `stash init --drizzle` now emits the package-root +`extractEncryptionSchema` import. The bundled `stash-drizzle` and +`stash-encryption` skills are updated to match. -**`@cipherstash/stack` (patch):** README only — its Drizzle section documented the removed `./v3` subpath and the `*V3` export names. +**`@cipherstash/stack` (patch):** README only — its Drizzle section now +documents the final package-root exports. diff --git a/.changeset/remove-eql-v2-migrate-classifier.md b/.changeset/remove-eql-v2-migrate-classifier.md index e6ae23fb6..04d74dda1 100644 --- a/.changeset/remove-eql-v2-migrate-classifier.md +++ b/.changeset/remove-eql-v2-migrate-classifier.md @@ -1,5 +1,5 @@ --- -'@cipherstash/migrate': patch +'@cipherstash/migrate': minor --- Drop EQL v2 from the domain-type classifier. `classifyEqlDomain` (and the diff --git a/.changeset/remove-eql-v2-packages.md b/.changeset/remove-eql-v2-packages.md index 3cc56bc0a..772a72e62 100644 --- a/.changeset/remove-eql-v2-packages.md +++ b/.changeset/remove-eql-v2-packages.md @@ -1,6 +1,5 @@ --- '@cipherstash/stack': patch -'@cipherstash/nextjs': patch --- Remove the EQL v2-only published packages `@cipherstash/protect`, diff --git a/.changeset/remove-eql-v2-supabase-authoring.md b/.changeset/remove-eql-v2-supabase-authoring.md index ad783dc52..59c6e036d 100644 --- a/.changeset/remove-eql-v2-supabase-authoring.md +++ b/.changeset/remove-eql-v2-supabase-authoring.md @@ -5,23 +5,16 @@ Remove the EQL v2 authoring surface and de-suffix the v3 API to the canonical unsuffixed names (part of the EQL v2 removal, #707). -- **`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory** - (formerly `encryptedSupabaseV3`). `encryptedSupabaseV3` remains as a - type-identical `@deprecated` alias, so existing imports keep working. +- **`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory.** - **The legacy v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper is removed** — with it the two-argument `from(tableName, schema)` form and the hand-written client-side v2 schema. Its `EncryptedSupabaseConfig` and the v2 `EncryptedSupabaseInstance`/`EncryptedQueryBuilder` type shapes are gone; the unsuffixed type names now denote the v3 surface. -- **The `*V3` type exports are de-suffixed** to their canonical names — - `EncryptedSupabaseV3Options` → `EncryptedSupabaseOptions`, - `EncryptedSupabaseV3Instance` → `EncryptedSupabaseInstance`, - `TypedEncryptedSupabaseV3Instance` → `TypedEncryptedSupabaseInstance`, - `EncryptedQueryBuilderV3` → `EncryptedQueryBuilder`, - `EncryptedQueryBuilderV3Untyped` → `EncryptedQueryBuilderUntyped`, - `V3FilterableKeys` → `FilterableKeys`, `V3OrderableKeys` → `OrderableKeys`, and - the rest of the `*V3` key-helper types. Each keeps a type-identical - `@deprecated` `*V3` alias. +- **The public types use canonical unsuffixed names:** + `EncryptedSupabaseOptions`, `EncryptedSupabaseInstance`, + `TypedEncryptedSupabaseInstance`, `EncryptedQueryBuilder`, + `EncryptedQueryBuilderUntyped`, `FilterableKeys`, and `OrderableKeys`. **Reading existing v2 data.** Only the v2 *authoring/emission* surface is removed — no v2 ciphertext is stranded. Decryption in `@cipherstash/stack` is @@ -37,9 +30,6 @@ Internally the v3 query builder (`query-builder-v3.ts`) was folded into the base `EncryptedQueryBuilderImpl`, which is now natively EQL v3; no runtime behaviour or wire encoding changed. -**Migration:** rename `encryptedSupabaseV3` → `encryptedSupabase` (or keep using -the alias). If you still use the v2 `encryptedSupabase({ encryptionClient, -supabaseClient }).from(table, schema)` wrapper, migrate the table to an -`eql_v3_*` column domain and switch to the introspecting factory — -`await encryptedSupabase(supabaseUrl, supabaseKey)` — see the `stash-supabase` -skill and https://cipherstash.com/docs. +**Migration:** use `await encryptedSupabase(supabaseUrl, supabaseKey)` with +`eql_v3_*` column domains. See the `stash-supabase` skill and +https://cipherstash.com/docs. diff --git a/.changeset/remove-eql-v2-supabase-skill.md b/.changeset/remove-eql-v2-supabase-skill.md index c4c33dedd..29d4b5b89 100644 --- a/.changeset/remove-eql-v2-supabase-skill.md +++ b/.changeset/remove-eql-v2-supabase-skill.md @@ -3,8 +3,7 @@ --- Update the bundled `stash-supabase` agent skill for the EQL v2 removal (#707): -`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory (with -`encryptedSupabaseV3` kept as a `@deprecated` alias), and the legacy v2 +`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory, and the legacy v2 `encryptedSupabase({ encryptionClient, supabaseClient })` authoring wrapper has been removed. The skill's examples, exported-type list, and migration/cutover guidance are corrected accordingly. Skills ship inside the `stash` tarball, so diff --git a/e2e/package.json b/e2e/package.json index dd10ac340..0d328cec2 100644 --- a/e2e/package.json +++ b/e2e/package.json @@ -16,6 +16,7 @@ "devDependencies": { "@types/semver": "7.7.1", "semver": "^7.8.0", + "typescript": "catalog:repo", "vitest": "catalog:repo", "yaml": "^2.9.0" } diff --git a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts index 052496cc0..5e315fd48 100644 --- a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts @@ -17,6 +17,7 @@ import type { Integration } from '../../types.js' vi.mock('@clack/prompts', () => ({ log: { warn: vi.fn() } })) import * as p from '@clack/prompts' +import { SKILL_MAP as WIZARD_SKILL_MAP } from '../../../../../../wizard/src/lib/install-skills.js' import { availableSkills, installSkills, @@ -40,6 +41,13 @@ const ALL_INTEGRATIONS: Integration[] = [ ] describe('SKILL_MAP', () => { + it('stays in parity with every wizard integration', () => { + expect(WIZARD_SKILL_MAP.drizzle).toEqual(SKILL_MAP.drizzle) + expect(WIZARD_SKILL_MAP.supabase).toEqual(SKILL_MAP.supabase) + expect(WIZARD_SKILL_MAP.prisma).toEqual(SKILL_MAP['prisma-next']) + expect(WIZARD_SKILL_MAP.generic).toEqual(SKILL_MAP.postgresql) + }) + it('has a non-empty entry for every integration (no undefined → crash)', () => { for (const integration of ALL_INTEGRATIONS) { const skills = SKILL_MAP[integration] diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index ee1c302b7..b80a6854e 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -188,7 +188,8 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { }) it('names the eql_v3 domain for supabase', () => { - expect(render('supabase')).toContain('eql_v3_encrypted') + expect(render('supabase')).toContain('email public.eql_v3_text_search') + expect(render('supabase')).not.toContain('eql_v3_encrypted') }) it('names the cipherstash.* field constructors for prisma-next', () => { diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 3deefd3f3..404ede44f 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -100,7 +100,7 @@ function schemaAuthoringGuidance(integration: Integration): string { case 'drizzle': return 'Declare it with the `types.*` column factories from `@cipherstash/stack-drizzle` on the existing `pgTable`' case 'supabase': - return 'Declare the column with the `eql_v3_encrypted` domain in the migration SQL (`encryptedSupabase` derives its encryption config by introspecting those domains)' + return 'Declare the column with a concrete `public.eql_v3_*` domain in the migration SQL — for example, `email public.eql_v3_text_search` (`encryptedSupabase` derives its encryption config by introspecting those domains)' case 'prisma-next': return 'Declare the field with the `cipherstash.*` constructors in `prisma/schema.prisma` (`cipherstash.TextSearch()`, `cipherstash.DoubleOrd()`, …), with the `cipherstash` extension pack wired up per `@cipherstash/prisma-next/control`' default: diff --git a/packages/stack-drizzle/integration/lock-context.integration.test.ts b/packages/stack-drizzle/integration/lock-context.integration.test.ts index 489a78100..fd5dbd568 100644 --- a/packages/stack-drizzle/integration/lock-context.integration.test.ts +++ b/packages/stack-drizzle/integration/lock-context.integration.test.ts @@ -32,12 +32,8 @@ * a distinct `sub`) and asserts B cannot read A's row, with A reading it as the * control. */ -import { OidcFederationStrategy } from '@cipherstash/stack' -import { - type AnyV3Table, - type EncryptionClientFor, - EncryptionV3, -} from '@cipherstash/stack/v3' +import { type AuthFailure, OidcFederationStrategy } from '@cipherstash/stack' +import { type EncryptionClientFor, EncryptionV3 } from '@cipherstash/stack/v3' import { databaseUrl, unwrapResult, V3_MATRIX } from '@cipherstash/test-kit' import { clerkJwtProvider } from '@cipherstash/test-kit/integration-clerk' import { and, asc as drizzleAsc, eq as drizzleEq, type SQL } from 'drizzle-orm' @@ -69,13 +65,13 @@ const secretTable = pgTable(TABLE_NAME, { rowKey: text('row_key').notNull(), testRunId: text('test_run_id').notNull(), secret: makeEqlV3Column(V3_MATRIX['public.eql_v3_text_eq'].builder('secret')), -} as never) +}) const schema = extractEncryptionSchema(secretTable) type SelectRow = { rowKey: string } -let client: EncryptionClientFor<readonly AnyV3Table[]> +let client: EncryptionClientFor<readonly [typeof schema]> let ops: ReturnType<typeof createEncryptionOperators> let db: ReturnType<typeof drizzle> @@ -133,6 +129,12 @@ const IDENTITY_DENIAL = const INFRA_FAULT = /ECONNREFUSED|ECONNRESET|ETIMEDOUT|ENOTFOUND|EAI_AGAIN|socket hang up|timed? ?out|network error/i +function authFailureMessage(failure: AuthFailure): string { + return 'error' in failure && failure.error instanceof Error + ? failure.error.message + : failure.type +} + /** Run-scoped SELECT of row keys under an already-encrypted SQL condition. */ async function selectRowKeys(condition: SQL): Promise<string[]> { const rows = (await db @@ -157,7 +159,7 @@ beforeAll(async () => { // `config.authStrategy` expects (it calls `.getToken()` on it). const federation = OidcFederationStrategy.create(crn, clerkJwtProvider()) if (federation.failure) { - throw new Error(`[federation]: ${federation.failure.message}`) + throw new Error(`[federation]: ${authFailureMessage(federation.failure)}`) } client = await EncryptionV3({ schemas: [schema], @@ -314,7 +316,9 @@ describe('v3 drizzle operators with lock context (live pg)', () => { clerkJwtProvider('CLERK_MACHINE_TOKEN_B'), ) if (federationB.failure) { - throw new Error(`[federation B]: ${federationB.failure.message}`) + throw new Error( + `[federation B]: ${authFailureMessage(federationB.failure)}`, + ) } const clientB = await EncryptionV3({ schemas: [schema], diff --git a/packages/stack-drizzle/integration/null-persistence.integration.test.ts b/packages/stack-drizzle/integration/null-persistence.integration.test.ts index 6a0e0f418..a9cc7d5d7 100644 --- a/packages/stack-drizzle/integration/null-persistence.integration.test.ts +++ b/packages/stack-drizzle/integration/null-persistence.integration.test.ts @@ -13,11 +13,7 @@ * as SQL NULL, and the present cell still decrypts to its plaintext. */ -import { - type AnyV3Table, - type EncryptionClientFor, - EncryptionV3, -} from '@cipherstash/stack/v3' +import { type EncryptionClientFor, EncryptionV3 } from '@cipherstash/stack/v3' import { databaseUrl, V3_MATRIX } from '@cipherstash/test-kit' import { and, asc as drizzleAsc, eq as drizzleEq, type SQL } from 'drizzle-orm' import { integer, pgTable, text } from 'drizzle-orm/pg-core' @@ -54,7 +50,7 @@ const nullableTable = pgTable(TABLE_NAME, { matchText: makeEqlV3Column( V3_MATRIX['public.eql_v3_text_match'].builder('match_text'), ), -} as never) +}) // Tier metadata: property (drizzle) + DB column + a present-row plaintext. const TIERS = [ @@ -88,7 +84,7 @@ const schema = extractEncryptionSchema(nullableTable) type SelectRow = { rowKey: string } -let client: EncryptionClientFor<readonly AnyV3Table[]> +let client: EncryptionClientFor<readonly [typeof schema]> let ops: ReturnType<typeof createEncryptionOperators> let db: ReturnType<typeof drizzle> diff --git a/packages/stack-drizzle/integration/relational.integration.test.ts b/packages/stack-drizzle/integration/relational.integration.test.ts index 5fac50151..7173a7fb5 100644 --- a/packages/stack-drizzle/integration/relational.integration.test.ts +++ b/packages/stack-drizzle/integration/relational.integration.test.ts @@ -19,11 +19,7 @@ * `stash eql install`. This suite throws rather than skips when unconfigured. */ -import { - type AnyV3Table, - type EncryptionClientFor, - EncryptionV3, -} from '@cipherstash/stack/v3' +import { type EncryptionClientFor, EncryptionV3 } from '@cipherstash/stack/v3' import { type DomainSpec, databaseUrl, @@ -82,9 +78,9 @@ const ROWS = [ROW_A, ROW_B, ROW_C] as const // and the seed INSERT raises — so a table built from every row can only ever run // against a superuser database. Filtering here is what lets this suite run on // both the plain-Postgres and Supabase variants. -const matrixEntries = typedEntries(V3_MATRIX).filter(([, spec]) => - isCovered(spec), -) +const matrixEntries: Array<readonly [EqlV3TypeName, DomainSpec]> = typedEntries( + V3_MATRIX, +).filter(([, spec]) => isCovered(spec)) const matrixColumns = Object.fromEntries( matrixEntries.map(([eqlType, spec]) => [ slug(eqlType), @@ -140,41 +136,13 @@ type RowKey = (typeof ROWS)[number] type MatrixPlainRow = Record<string, PlainValue | null | string> type SelectRow = { rowKey: string } type Db = ReturnType<typeof drizzle> -type Client = EncryptionClientFor<readonly AnyV3Table[]> +type Client = EncryptionClientFor<readonly [typeof schema, typeof bigintSchema]> type Ops = ReturnType<typeof createEncryptionOperators> -type ComparisonOperator = 'gt' | 'gte' | 'lt' | 'lte' - let client: Client let ops: Ops let db: Db -const equalityDomains = matrixEntries.filter( - ([, spec]) => spec.indexes.unique || spec.indexes.ore || spec.indexes.ope, -) -const orderDomains = matrixEntries.filter( - ([, spec]) => spec.indexes.ore || spec.indexes.ope, -) const matchDomains = matrixEntries.filter(([, spec]) => spec.indexes.match) -const noEqualityDomains = matrixEntries.filter( - ([, spec]) => !spec.indexes.unique && !spec.indexes.ore && !spec.indexes.ope, -) -const noOrderDomains = matrixEntries.filter( - ([, spec]) => !spec.indexes.ore && !spec.indexes.ope, -) -const noMatchDomains = matrixEntries.filter(([, spec]) => !spec.indexes.match) -const comparisonOperators: Array< - [ComparisonOperator, (cmp: number) => boolean] -> = [ - ['gt', (cmp) => cmp > 0], - ['gte', (cmp) => cmp >= 0], - ['lt', (cmp) => cmp < 0], - ['lte', (cmp) => cmp <= 0], -] -const comparisonDomains = orderDomains.flatMap(([eqlType, spec]) => - comparisonOperators.map( - ([operator, predicate]) => [eqlType, spec, operator, predicate] as const, - ), -) const matrixColumn = (eqlType: EqlV3TypeName): SQLWrapper => (matrixTable as unknown as Record<string, SQLWrapper>)[slug(eqlType)] @@ -215,6 +183,24 @@ function encryptedInsertRows(): MatrixPlainRow[] { }) } +function checkedInsertRows<T extends { rowKey: string; testRunId: string }>( + rows: unknown[], +): T[] { + for (const row of rows) { + if ( + row === null || + typeof row !== 'object' || + !('rowKey' in row) || + typeof row.rowKey !== 'string' || + !('testRunId' in row) || + typeof row.testRunId !== 'string' + ) { + throw new Error('Encrypted integration row lost plaintext scope keys') + } + } + return rows as T[] +} + beforeAll(async () => { client = await EncryptionV3({ schemas: [schema, bigintSchema] }) ops = createEncryptionOperators(client) @@ -267,8 +253,8 @@ beforeAll(async () => { ) `) - const encryptedRows = unwrapResult( - await client.bulkEncryptModels(encryptedInsertRows(), schema), + const encryptedRows = checkedInsertRows<typeof matrixTable.$inferInsert>( + unwrapResult(await client.bulkEncryptModels(encryptedInsertRows(), schema)), ) await db.insert(matrixTable).values(encryptedRows) await db.insert(accountsTable).values([ @@ -276,30 +262,33 @@ beforeAll(async () => { { rowKey: ROW_B, label: 'secondary', testRunId: RUN }, ]) - // A3 end-to-end, cast-free: encrypt a native bigint model, insert the - // resulting envelope rows (typed against the column's `Encrypted` data slot), - // no `as never` anywhere. + // A3 end-to-end: encrypt a native bigint model and insert the resulting + // envelope rows against the column's `Encrypted` data slot. The extracted + // schema is intentionally runtime-shaped (`AnyV3Table`), so restore the + // concrete Drizzle insert shape at this boundary. // // ROW_B exists so the filter proofs below have a row they must EXCLUDE. On a // one-row table `gt(balance, 0n)` returning every row is indistinguishable // from it returning the right row. - const bigintRows = unwrapResult( - await client.bulkEncryptModels( - [ - { - rowKey: ROW_A, - testRunId: RUN, - balance: BIGINT_BALANCE, - ledger: BIGINT_LEDGER, - }, - { - rowKey: ROW_B, - testRunId: RUN, - balance: BIGINT_B_BALANCE, - ledger: BIGINT_B_LEDGER, - }, - ], - bigintSchema, + const bigintRows = checkedInsertRows<typeof bigintTable.$inferInsert>( + unwrapResult( + await client.bulkEncryptModels( + [ + { + rowKey: ROW_A, + testRunId: RUN, + balance: BIGINT_BALANCE, + ledger: BIGINT_LEDGER, + }, + { + rowKey: ROW_B, + testRunId: RUN, + balance: BIGINT_B_BALANCE, + ledger: BIGINT_B_LEDGER, + }, + ], + bigintSchema, + ), ), ) await db.insert(bigintTable).values(bigintRows) diff --git a/packages/stack-drizzle/tsconfig.typecheck.json b/packages/stack-drizzle/tsconfig.typecheck.json index 16d312f2f..051ef8107 100644 --- a/packages/stack-drizzle/tsconfig.typecheck.json +++ b/packages/stack-drizzle/tsconfig.typecheck.json @@ -9,9 +9,5 @@ // (#778). Widen this `include` as that count comes down; do not widen it to // the whole project until it is zero, or the gate is useless. "extends": "./tsconfig.json", - "include": [ - "__tests__/**/*.test-d.ts", - "integration/adapter.ts", - "integration/json-adapter.ts" - ] + "include": ["__tests__/**/*.test-d.ts", "integration/**/*.ts"] } diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index ce9835a2f..5163918b5 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -223,6 +223,18 @@ describe('encryptedSupabaseV3 wire encoding', () => { expect(supabase.callsFor('contains')).toHaveLength(0) }) + it('rejects nullish matches operands before they can reach the wire', () => { + const { es, supabase } = v3Instance() + + expect(() => + es.from('users', users).select('id').matches('email', undefined), + ).toThrow(/requires a non-null search term/) + expect(() => + es.from('users', users).select('id').matches('email', null), + ).toThrow(/requires a non-null search term/) + expect(supabase.callsFor('filter')).toHaveLength(0) + }) + it('refuses contains() on an encrypted column, naming matches', async () => { const { es } = v3Instance() @@ -263,6 +275,22 @@ describe('encryptedSupabaseV3 wire encoding', () => { ).toThrow(/cannot honor/) }) + it('treats escaped LIKE wildcards as literal search characters', async () => { + for (const [pattern, needle] of [ + ['%100\\%%', '100%'], + ['%under\\_score%', 'under_score'], + ] as const) { + const { es, supabase } = v3Instance() + + await es.from('users', users).select('id').like('email', pattern) + + const envelope = JSON.parse( + supabase.callsFor('filter')[0].args[2] as string, + ) + expect(envelope.pt).toBe(needle) + } + }) + it('passes contains through to native cs on a plaintext column', async () => { const { es, supabase } = v3Instance() @@ -512,11 +540,41 @@ describe('encryptedSupabaseV3 wire encoding', () => { expect(emitted.endsWith(',note.eq.x')).toBe(true) }) - // An all-plaintext or() string is forwarded verbatim, so its containment - // literal is never parsed. Naming an encrypted column forces the rebuild - // path — and there the literal's own commas must not be mistaken for - // condition separators, or `note` is filtered on the truncated `{vip`. - it('preserves a plaintext containment literal when rebuilding a mixed or() string', async () => { + it('preserves nested PostgREST logic while replacing encrypted leaves', async () => { + const { es, supabase } = v3Instance() + + await es + .from('users', users) + .select('id') + .or('and(createdAt.gte.2026-01-01,note.eq.x),id.eq.1', { + referencedTable: 'profiles', + }) + + const [or] = supabase.callsFor('or') + const emitted = or.args[0] as string + expect(emitted).toMatch(/^and\(created_at\.gte\."/) + expect(emitted).toContain(',note.eq.x),id.eq.1') + expect(or.args[1]).toEqual({ referencedTable: 'profiles' }) + }) + + it('preserves referencedTable for structured or() filters', async () => { + const { es, supabase } = v3Instance() + + await es + .from('users', users) + .select('id') + .or([{ column: 'nickname', op: 'eq', value: 'ada' }], { + foreignTable: 'profiles', + }) + + expect(supabase.callsFor('or')[0].args[1]).toEqual({ + referencedTable: 'profiles', + }) + }) + + // Encrypted leaves are substituted in place, so a plaintext containment + // sibling remains byte-for-byte identical even though its comma is nested. + it('preserves a plaintext containment literal in a mixed or() string', async () => { const { es, supabase } = v3Instance() await es @@ -526,7 +584,7 @@ describe('encryptedSupabaseV3 wire encoding', () => { const emitted = supabase.callsFor('or')[0].args[0] as string expect(emitted).toMatch(/^email\.eq\./) - expect(emitted.endsWith(',note.cs."{vip,admin}"')).toBe(true) + expect(emitted.endsWith(',note.cs.{vip,admin}')).toBe(true) }) it('keeps every filter array correlated in a combined query', async () => { @@ -852,6 +910,19 @@ describe('encryptedSupabaseV3 wire encoding', () => { ) }) + it('rejects a nullish encrypted search operand in structured or()', async () => { + const { es, supabase } = v3Instance() + + const { error, status } = await es + .from('users', users) + .select('id') + .or([{ column: 'email', op: 'matches', value: undefined }]) + + expect(status).toBe(500) + expect(error?.message).toMatch(/requires a non-null operand/) + expect(supabase.callsFor('or')).toHaveLength(0) + }) + // The operator token depends on the OPERATOR, never on whether the operand // was encrypted. `note` is a plaintext passthrough, so nothing encrypts and // the rewrite used to be skipped, emitting `note.contains.plain` — which @@ -897,6 +968,15 @@ describe('encryptedSupabaseV3 wire encoding', () => { }) describe('update / delete / single / maybeSingle', () => { + it('rejects csv() before serialized ciphertext can enter model decryption', () => { + const { es, supabase } = v3Instance() + + expect(() => es.from('users', users).select('id, email').csv()).toThrow( + /serializes rows before the adapter can decrypt/, + ) + expect(supabase.callsFor('csv')).toHaveLength(0) + }) + it('updates with raw envelopes keyed by DB column name', async () => { const { es, supabase } = v3Instance() @@ -1000,6 +1080,39 @@ describe('encryptedSupabaseV3 wire encoding', () => { ) }) + it('leaves an invalid date value untouched', async () => { + const { es } = v3Instance([{ id: 1, createdAt: 'not-a-date' }]) + + const { data } = await es.from('users', users).select('id, createdAt') + + expect(data?.[0].createdAt).toBe('not-a-date') + }) + + it('reconstructs a nested dotted date path', async () => { + const nestedTable = encryptedTable('profiles', { + 'profile.createdAt': types.Timestamp('profile_created_at'), + }) + const supabase = createMockSupabase([ + { profile: { createdAt: '2026-01-02T03:04:05.000Z' } }, + ]) + const builder = new EncryptedQueryBuilderV3Impl( + 'profiles', + nestedTable, + createMockEncryptionClient(), + supabase.client, + ['profile_created_at'], + ) + + const { data } = await builder.select('profile.createdAt') + + if (!data) throw new Error('Expected a decrypted row') + const profile = (data[0] as Record<string, unknown>).profile as Record< + string, + unknown + > + expect(profile.createdAt).toBeInstanceOf(Date) + }) + // Selecting by raw DB name means the row comes back keyed `created_at`, // the only way to reach the `dbName` half of the two-key branch. It also // exercises the `value == null` skip on the absent `createdAt` key. diff --git a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts index 93bab05f9..966d5366e 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-factory.test.ts @@ -79,6 +79,33 @@ describe('encryptedSupabaseV3 factory', () => { expect(createClientMock).not.toHaveBeenCalled() }) + it('diagnoses the removed v2 object call shape before introspection', async () => { + await expect( + encryptedSupabaseV3( + { + encryptionClient: {}, + supabaseClient: fakeClient, + } as unknown as SupabaseClientLike, + { databaseUrl: 'postgres://x' }, + ), + ).rejects.toThrow(/removed EQL v2 API.*Pass the Supabase client directly/) + expect(introspectMock).not.toHaveBeenCalled() + }) + + it('rejects an invalid supplied client before introspection', async () => { + await expect( + encryptedSupabaseV3({} as SupabaseClientLike, { + databaseUrl: 'postgres://x', + }), + ).rejects.toThrow(/Supabase client with a from\(\) method/) + await expect( + encryptedSupabaseV3(null as unknown as SupabaseClientLike, { + databaseUrl: 'postgres://x', + }), + ).rejects.toThrow(/Supabase client with a from\(\) method/) + expect(introspectMock).not.toHaveBeenCalled() + }) + it('falls back to process.env.DATABASE_URL', async () => { process.env.DATABASE_URL = 'postgres://env' await encryptedSupabaseV3(fakeClient) diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index 23168ffe0..3eb6e27ef 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -209,13 +209,46 @@ export function mapFilterOpToQueryType(op: FilterOp): QueryTypeName { * of that name is untouched. */ export function parseOrString(orString: string): PendingOrCondition[] { + return parseOrStringAt(orString, 0, false).map( + ({ sourceSpan: _, ...condition }) => condition, + ) +} + +/** Adapter-only parser that flattens nested groups and records every leaf's + * source span, allowing encrypted substitution without rebuilding the group. */ +export function parseOrStringWithSpans(orString: string): PendingOrCondition[] { + return parseOrStringAt(orString, 0, true) +} + +/** Recursively flatten PostgREST logic groups while retaining each leaf's + * exact location in the caller's original expression. The adapter must + * encrypt leaves inside `and(...)` / `or(...)`, but rebuilding the whole + * expression from a flat condition array destroys those groups. */ +function parseOrStringAt( + orString: string, + baseOffset: number, + recurseGroups: boolean, +): PendingOrCondition[] { const conditions: PendingOrCondition[] = [] - // Split on commas that are not inside parentheses (nested or/and) const parts = splitTopLevel(orString) + let cursor = 0 for (const part of parts) { + const partStart = orString.indexOf(part, cursor) + cursor = partStart + part.length const trimmed = part.trim() if (!trimmed) continue + const leading = part.length - part.trimStart().length + const sourceStart = baseOffset + partStart + leading + + const group = /^(?:not\.)?(?:and|or)\(([\s\S]*)\)$/.exec(trimmed) + if (recurseGroups && group) { + const open = trimmed.indexOf('(') + conditions.push( + ...parseOrStringAt(group[1], sourceStart + open + 1, true), + ) + continue + } // Format: column.op.value — or column.not.op.value const firstDot = trimmed.indexOf('.') @@ -245,7 +278,13 @@ export function parseOrString(orString: string): PendingOrCondition[] { // Handle special value formats const parsedValue = parseOrValue(value, op) - conditions.push({ column, op, negate, value: parsedValue }) + conditions.push({ + column, + op, + negate, + value: parsedValue, + sourceSpan: { start: sourceStart, end: sourceStart + trimmed.length }, + }) } return conditions @@ -297,6 +336,35 @@ export function rebuildOrString( .join(',') as DbFilterString } +/** + * Rebuild only encrypted leaves of a string-form `.or()` expression. + * + * The original delimiters, whitespace, nesting, and plaintext conditions are + * retained byte-for-byte. Replacements run from right to left so source spans + * remain stable when ciphertext is longer than the plaintext operand. + */ +export function substituteOrStringLeaves( + original: string, + conditions: DbPendingOrCondition[], + shouldReplace: (condition: DbPendingOrCondition) => boolean, +): DbFilterString { + let result = original + const replacements = conditions + .filter((condition) => condition.sourceSpan && shouldReplace(condition)) + .sort( + (left, right) => + (right.sourceSpan?.start ?? 0) - (left.sourceSpan?.start ?? 0), + ) + + for (const condition of replacements) { + const span = condition.sourceSpan + if (!span) continue + const replacement = rebuildOrString([condition]) + result = result.slice(0, span.start) + replacement + result.slice(span.end) + } + return result as DbFilterString +} + // --------------------------------------------------------------------------- // Internal helpers // --------------------------------------------------------------------------- diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index a05c7b81f..aae094425 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -141,6 +141,21 @@ export async function encryptedSupabase( } supabaseClient = createClient(url, key) as unknown as SupabaseClientLike } else { + if (clientOrUrl === null || typeof clientOrUrl !== 'object') { + throw new Error( + '[supabase v3]: encryptedSupabase expected a Supabase client with a from() method. Pass encryptedSupabase(supabaseClient, options), or use encryptedSupabase(url, key, options).', + ) + } + if ('encryptionClient' in clientOrUrl && 'supabaseClient' in clientOrUrl) { + throw new Error( + '[supabase v3]: encryptedSupabase({ encryptionClient, supabaseClient }) was the removed EQL v2 API. Pass the Supabase client directly instead: encryptedSupabase(supabaseClient, { databaseUrl, schemas? }).', + ) + } + if (typeof clientOrUrl.from !== 'function') { + throw new Error( + '[supabase v3]: encryptedSupabase expected a Supabase client with a from() method. Pass encryptedSupabase(supabaseClient, options), or use encryptedSupabase(url, key, options).', + ) + } supabaseClient = clientOrUrl options = (keyOrOptions as EncryptedSupabaseOptions<V3Schemas | undefined>) ?? {} diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index d7356a1a5..6516af1cf 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -313,6 +313,11 @@ export class EncryptedQueryBuilderImpl< * silently accept as containment of the raw string. */ matches(column: string, value: unknown): this { + if (value == null) { + throw new Error( + `[supabase v3]: matches("${column}", …) requires a non-null search term; null and undefined cannot be encrypted and would otherwise reach PostgREST as plaintext.`, + ) + } if (this.columns.isSearchableJsonColumn(column)) { throw new Error( `[supabase v3]: matches() is encrypted free-text search and does not apply to encrypted JSON column "${column}". Use contains("${column}", subDocument) or selectorEq("${column}", path, value).`, @@ -391,6 +396,7 @@ export class EncryptedQueryBuilderImpl< this.orFilters.push({ kind: 'structured', conditions: filtersOrConditions, + referencedTable: options?.referencedTable ?? options?.foreignTable, }) } return this @@ -486,8 +492,9 @@ export class EncryptedQueryBuilderImpl< } csv(): this { - this.transforms.push({ kind: 'csv' }) - return this + throw new Error( + '[supabase v3]: csv() is unavailable on encrypted queries because PostgREST serializes rows before the adapter can decrypt them. Select rows normally, then serialize the decrypted data to CSV.', + ) } abortSignal(signal: AbortSignal): this { @@ -861,12 +868,32 @@ export class EncryptedQueryBuilderImpl< * per (op, column) that the delegation is approximate. */ private likeNeedle(column: string, op: string, pattern: string): string { - const needle = pattern.replace(/^%+/, '').replace(/%+$/, '') - if (needle.includes('%') || pattern.includes('_')) { + const tokens: Array<{ value: string; wildcard: boolean }> = [] + for (let i = 0; i < pattern.length; i++) { + const char = pattern[i] + if (char === '\\' && i + 1 < pattern.length) { + tokens.push({ value: pattern[++i], wildcard: false }) + } else { + tokens.push({ + value: char, + wildcard: char === '%' || char === '_', + }) + } + } + + while (tokens[0]?.wildcard && tokens[0].value === '%') tokens.shift() + while ( + tokens[tokens.length - 1]?.wildcard && + tokens[tokens.length - 1]?.value === '%' + ) + tokens.pop() + + if (tokens.some((token) => token.wildcard)) { throw new Error( `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an internal "%" or any "_"). Use matches("${column}", term) with a literal search term.`, ) } + const needle = tokens.map((token) => token.value).join('') const key = `${op}:${column}` if (!warnedLikeDelegation.has(key)) { warnedLikeDelegation.add(key) diff --git a/packages/stack-supabase/src/query-dbspace.ts b/packages/stack-supabase/src/query-dbspace.ts index ae68df562..fed373a64 100644 --- a/packages/stack-supabase/src/query-dbspace.ts +++ b/packages/stack-supabase/src/query-dbspace.ts @@ -1,5 +1,5 @@ import type { ColumnMap } from './column-map' -import { parseOrString } from './helpers' +import { parseOrStringWithSpans } from './helpers' import type { DbConflictList, DbMutationOp, @@ -51,11 +51,15 @@ function orFilterToDbSpace( return { kind: 'string', original: of_.value, - conditions: parseOrString(of_.value).map(toDbCondition), + conditions: parseOrStringWithSpans(of_.value).map(toDbCondition), referencedTable: of_.referencedTable, } } - return { kind: 'structured', conditions: of_.conditions.map(toDbCondition) } + return { + kind: 'structured', + conditions: of_.conditions.map(toDbCondition), + referencedTable: of_.referencedTable, + } } function transformToDbSpace(t: TransformOp, columns: ColumnMap): DbTransformOp { diff --git a/packages/stack-supabase/src/query-encrypt.ts b/packages/stack-supabase/src/query-encrypt.ts index ae67e4bfe..6d617ae10 100644 --- a/packages/stack-supabase/src/query-encrypt.ts +++ b/packages/stack-supabase/src/query-encrypt.ts @@ -206,6 +206,24 @@ export function queryTypeForOrOp(op: FilterOp): QueryTypeName { return queryTypeForRawOp(op) } +/** A nullish encrypted search operand is never a SQL-NULL predicate. Skipping + * encryption would put the raw operand on the wire under `cs`, so fail closed + * for every spelling (`matches`, `contains`, raw `cs`, `not`, and `.or()`). */ +function assertEncryptedSearchOperand( + queryType: QueryTypeName, + value: unknown, + column: string, +): void { + if ( + value == null && + (queryType === 'freeTextSearch' || queryType === 'searchableJson') + ) { + throw new Error( + `[supabase v3]: encrypted search on column "${column}" requires a non-null operand; null and undefined cannot be sent through the plaintext PostgREST filter path.`, + ) + } +} + function encryptionFailure( tableName: string, message: string, @@ -287,19 +305,19 @@ export async function encryptFilterValues( const column = tableColumns[f.column] if (!column) continue + const queryType = mapFilterOpToQueryType(f.op) + assertEncryptedSearchOperand(queryType, f.value, f.column) if (f.op === 'in' && Array.isArray(f.value)) { - collectInListTerms( - f.op, - f.value, - column, - mapFilterOpToQueryType(f.op), - (inIndex) => ({ source: 'filter', filterIndex: i, inIndex }), - ) + collectInListTerms(f.op, f.value, column, queryType, (inIndex) => ({ + source: 'filter', + filterIndex: i, + inIndex, + })) } else if (!isEncryptableTerm(f.op, f.value)) { // `is` predicate or null operand — forwarded unencrypted. } else { - pushTerm(f.value as JsPlaintext, column, mapFilterOpToQueryType(f.op), { + pushTerm(f.value as JsPlaintext, column, queryType, { source: 'filter', filterIndex: i, }) @@ -328,9 +346,11 @@ export async function encryptFilterValues( for (let i = 0; i < dbSpace.notFilters.length; i++) { const nf = dbSpace.notFilters[i] if (!isEncryptedColumn(nf.column, encryptedColumnNames)) continue - if (!isEncryptableTerm(nf.op, nf.value)) continue const column = tableColumns[nf.column] if (!column) continue + const queryType = mapFilterOpToQueryType(nf.op) + assertEncryptedSearchOperand(queryType, nf.value, nf.column) + if (!isEncryptableTerm(nf.op, nf.value)) continue if (nf.op === 'in') { // A PostgREST list literal (`'(a,b)'`) cannot be encrypted element-wise, @@ -342,17 +362,15 @@ export async function encryptFilterValues( `not a PostgREST list literal — each element must be encrypted separately`, ) } - collectInListTerms( - nf.op, - nf.value, - column, - mapFilterOpToQueryType(nf.op), - (inIndex) => ({ source: 'not', notIndex: i, inIndex }), - ) + collectInListTerms(nf.op, nf.value, column, queryType, (inIndex) => ({ + source: 'not', + notIndex: i, + inIndex, + })) continue } - pushTerm(nf.value as JsPlaintext, column, mapFilterOpToQueryType(nf.op), { + pushTerm(nf.value as JsPlaintext, column, queryType, { source: 'not', notIndex: i, }) @@ -374,6 +392,7 @@ export async function encryptFilterValues( // `queryTypeForOrOp`, not `mapFilterOpToQueryType`: an or-condition may // carry a raw PostgREST operator (`cs`), which is not a `FilterOp`. const queryType = queryTypeForOrOp(cond.op) + assertEncryptedSearchOperand(queryType, cond.value, cond.column) const mappingFor = (inIndex?: number): TermMapping => ({ source, orIndex: i, @@ -397,6 +416,8 @@ export async function encryptFilterValues( if (!isEncryptedColumn(rf.column, encryptedColumnNames)) continue const column = tableColumns[rf.column] if (!column) continue + const queryType = queryTypeForRawOp(rf.operator) + assertEncryptedSearchOperand(queryType, rf.value, rf.column) if (rf.operator === 'in') { // Same contract as the `not(…, 'in', …)` path: a PostgREST list literal @@ -409,19 +430,17 @@ export async function encryptFilterValues( `not a PostgREST list literal — each element must be encrypted separately`, ) } - collectInListTerms( - 'in', - rf.value, - column, - queryTypeForRawOp(rf.operator), - (inIndex) => ({ source: 'raw', rawIndex: i, inIndex }), - ) + collectInListTerms('in', rf.value, column, queryType, (inIndex) => ({ + source: 'raw', + rawIndex: i, + inIndex, + })) continue } if (!isEncryptableTerm(rf.operator, rf.value)) continue - pushTerm(rf.value as JsPlaintext, column, queryTypeForRawOp(rf.operator), { + pushTerm(rf.value as JsPlaintext, column, queryType, { source: 'raw', rawIndex: i, }) diff --git a/packages/stack-supabase/src/query-filters.ts b/packages/stack-supabase/src/query-filters.ts index fdad1d10a..12d8ba14a 100644 --- a/packages/stack-supabase/src/query-filters.ts +++ b/packages/stack-supabase/src/query-filters.ts @@ -4,6 +4,7 @@ import { formatInListOperand, isEncryptedColumn, rebuildOrString, + substituteOrStringLeaves, } from './helpers' import { assertPostgrestCanQueryEncryptedOperator, @@ -338,9 +339,14 @@ export function applyFilters( ) if (referencesEncrypted) { - q = q.or(rebuildOrString(parsed), { - referencedTable: of_.referencedTable, - }) + q = q.or( + substituteOrStringLeaves(of_.original, parsed, (condition) => + isEncryptedColumn(condition.column, encryptedColumnNames), + ), + { + referencedTable: of_.referencedTable, + }, + ) } else { // Every condition names a plaintext column, whose property name IS // its DB name — nothing to map. Forward the caller's ORIGINAL string @@ -357,7 +363,9 @@ export function applyFilters( return sub ? { ...cond, value: sub.value } : cond }) - q = q.or(rebuildOrString(conditions)) + q = q.or(rebuildOrString(conditions), { + referencedTable: of_.referencedTable, + }) } } diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts index 128c06fca..425ca8c75 100644 --- a/packages/stack-supabase/src/query-results.ts +++ b/packages/stack-supabase/src/query-results.ts @@ -1,4 +1,9 @@ -import { DATE_LIKE_CASTS, logger } from '@cipherstash/stack/adapter-kit' +import { + DATE_LIKE_CASTS, + logger, + reconstructDatePaths, + reconstructDateValue, +} from '@cipherstash/stack/adapter-kit' import { selectKeyToDbV3 } from './helpers' import { type EncryptionContext, @@ -58,14 +63,14 @@ function postprocessDecryptedRow( keyToDb[dbName] ??= dbName } - const out: Record<string, unknown> = { ...row } + let out: Record<string, unknown> = { ...row } for (const [key, dbName] of Object.entries(keyToDb)) { const castAs = ctx.columns.schemaFor(dbName)?.cast_as if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue - const value = out[key] - if (value == null || value instanceof Date) continue - if (typeof value === 'string' || typeof value === 'number') { - out[key] = new Date(value) + if (key.includes('.')) { + out = reconstructDatePaths(out, [key]) + } else if (Object.hasOwn(out, key)) { + out[key] = reconstructDateValue(out[key]) } } return out diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index e19f685e9..f6a53ba61 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -539,7 +539,11 @@ export type PendingFilter = { } export type PendingOrFilter = - | { kind: 'structured'; conditions: PendingOrCondition[] } + | { + kind: 'structured' + conditions: PendingOrCondition[] + referencedTable?: string + } | { kind: 'string'; value: string; referencedTable?: string } export type PendingOrCondition = { @@ -549,6 +553,10 @@ export type PendingOrCondition = { * `in`-list split and the query-type mapping both key on the real operator. */ negate?: boolean value: unknown + /** Character range in the caller's original string-form `.or()` expression. + * Internal only: lets the encrypted adapter replace this leaf without + * rebuilding (and corrupting) surrounding `and(...)` / `or(...)` groups. */ + sourceSpan?: { start: number; end: number } } export type PendingMatchFilter = { @@ -694,7 +702,11 @@ export type DbPendingMatchFilter = { * nested `and()` or quoted values) alongside the parsed DB-space conditions * used by the encrypt-and-rebuild path. Parsing happens once, here. */ export type DbPendingOrFilter = - | { kind: 'structured'; conditions: DbPendingOrCondition[] } + | { + kind: 'structured' + conditions: DbPendingOrCondition[] + referencedTable?: string + } | { kind: 'string' original: string diff --git a/packages/stack/__tests__/typed-client-v3.test.ts b/packages/stack/__tests__/typed-client-v3.test.ts index c506d787f..3497021ec 100644 --- a/packages/stack/__tests__/typed-client-v3.test.ts +++ b/packages/stack/__tests__/typed-client-v3.test.ts @@ -78,6 +78,42 @@ describe('typedClient — decrypt reconstruction', () => { expect(data.note).toBeNull() }) + it('leaves invalid date values untouched', async () => { + const client = typedClient(fakeClient({ when: 'not-a-date' }), table) + + const result = await client.decryptModel({}, table) + if (result.failure) return + + expect((result.data as Record<string, unknown>).when).toBe('not-a-date') + }) + + it('reconstructs a dotted date path without mutating the decrypted row', async () => { + const nested = encryptedTable('nested', { + 'profile.when': types.Timestamp('profile_when'), + 'profile.birthday': types.Date('profile_birthday'), + }) + const decrypted = { + profile: { + when: '2020-01-02T03:04:05.000Z', + birthday: '1990-04-05', + untouched: true, + }, + } + const client = typedClient(fakeClient(decrypted), nested) + + const result = await client.decryptModel({}, nested) + if (result.failure) return + + const data = result.data as Record<string, unknown> + const profile = data.profile as Record<string, unknown> + expect(profile.when).toBeInstanceOf(Date) + expect(profile.birthday).toBeInstanceOf(Date) + expect(profile.untouched).toBe(true) + expect(profile).not.toBe(decrypted.profile) + expect(decrypted.profile.when).toBe('2020-01-02T03:04:05.000Z') + expect(decrypted.profile.birthday).toBe('1990-04-05') + }) + it('reconstructs each row for bulkDecryptModels', async () => { const client = typedClient( fakeClient({ when: '2021-06-01T00:00:00.000Z', note: 'x' }), diff --git a/packages/stack/__tests__/wasm-inline-models.test.ts b/packages/stack/__tests__/wasm-inline-models.test.ts index acb2b9fea..eef1e0a33 100644 --- a/packages/stack/__tests__/wasm-inline-models.test.ts +++ b/packages/stack/__tests__/wasm-inline-models.test.ts @@ -54,6 +54,7 @@ const users = encryptedTable('users', { // model carries `{ profile: { ssn } }`; the walk matches it via the path. const patients = encryptedTable('patients', { 'profile.ssn': types.TextEq('profile.ssn'), + 'profile.seenAt': types.Timestamp('profile.seen_at'), }) async function client() { @@ -217,6 +218,17 @@ describe('WasmEncryptionClient.decryptModel', () => { ) }) + it('rebuilds a date-like value at a dotted model path', async () => { + ffi.decryptBulkFallible.mockResolvedValueOnce([ + { data: '2026-07-22T01:02:03.000Z' }, + ] as never) + + const c = await client() + const out = await c.decryptModel({ profile: { seenAt: ct('d') } }, patients) + + expect(expectData(out).profile.seenAt).toBeInstanceOf(Date) + }) + it('names every failed field by its model path', async () => { ffi.decryptBulkFallible.mockResolvedValueOnce([ { error: 'boom', code: 'CT_ERROR' }, diff --git a/packages/stack/src/adapter-kit.ts b/packages/stack/src/adapter-kit.ts index a46706d8d..7376e9af7 100644 --- a/packages/stack/src/adapter-kit.ts +++ b/packages/stack/src/adapter-kit.ts @@ -33,6 +33,10 @@ export { DATE_LIKE_CASTS, EncryptedV3Column, } from './eql/v3/columns.js' +export { + reconstructDatePaths, + reconstructDateValue, +} from './eql/v3/date-reconstruction.js' // Domain registry: the Supabase adapter classifies introspected columns by their // Postgres domain and rebuilds each column's encryption config from it. export { diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index cc79cb306..6e25b29f0 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -10,6 +10,7 @@ import type { V3ModelInput, } from '@/eql/v3' import { DATE_LIKE_CASTS } from '@/eql/v3/columns' +import { reconstructDatePaths } from '@/eql/v3/date-reconstruction' import { type EncryptionError, EncryptionErrorTypes } from '@/errors' import type { LockContextInput } from '@/identity' import type { @@ -217,13 +218,7 @@ function rowReconstructor( .map(([property]) => property) return (row) => { - const out: Record<string, unknown> = { ...row } - for (const property of dateProperties) { - const value = out[property] - if (value == null) continue - out[property] = new Date(value as string | number | Date) - } - return out + return reconstructDatePaths(row, dateProperties) } } diff --git a/packages/stack/src/eql/v3/date-reconstruction.ts b/packages/stack/src/eql/v3/date-reconstruction.ts new file mode 100644 index 000000000..fcefb081e --- /dev/null +++ b/packages/stack/src/eql/v3/date-reconstruction.ts @@ -0,0 +1,57 @@ +/** Reconstruct a decrypted date-like plaintext without manufacturing an + * `Invalid Date`. Non-date values and already-constructed Dates pass through. */ +export function reconstructDateValue(value: unknown): unknown { + if ( + value == null || + value instanceof Date || + (typeof value !== 'string' && typeof value !== 'number') + ) { + return value + } + const date = new Date(value) + return Number.isNaN(date.getTime()) ? value : date +} + +/** Reconstruct date-like values at dotted model paths without mutating the + * decrypted input row. Missing paths and non-object intermediates are ignored. */ +export function reconstructDatePaths( + row: Record<string, unknown>, + paths: readonly string[], +): Record<string, unknown> { + const out = { ...row } + for (const path of paths) { + const segments = path.split('.') + let source: Record<string, unknown> = row + let target: Record<string, unknown> = out + let reachable = true + + for (let i = 0; i < segments.length - 1; i++) { + const segment = segments[i] + const sourceChild = source[segment] + if ( + sourceChild === null || + typeof sourceChild !== 'object' || + Array.isArray(sourceChild) + ) { + reachable = false + break + } + const targetChild = target[segment] + const cloned = { + ...((targetChild !== null && + typeof targetChild === 'object' && + !Array.isArray(targetChild) + ? targetChild + : sourceChild) as Record<string, unknown>), + } + target[segment] = cloned + source = sourceChild as Record<string, unknown> + target = cloned + } + + const leaf = segments.at(-1) + if (!reachable || !leaf || !Object.hasOwn(source, leaf)) continue + target[leaf] = reconstructDateValue(source[leaf]) + } + return out +} diff --git a/packages/stack/src/wasm-inline.ts b/packages/stack/src/wasm-inline.ts index 2b2d8dd6d..af9d467b8 100644 --- a/packages/stack/src/wasm-inline.ts +++ b/packages/stack/src/wasm-inline.ts @@ -117,6 +117,7 @@ import { type V3ModelInput, } from '@/eql/v3' import { DATE_LIKE_CASTS } from '@/eql/v3/columns' +import { reconstructDateValue } from '@/eql/v3/date-reconstruction' import { type EncryptionError, EncryptionErrorTypes } from '@/errors' import { type CastAs, @@ -614,12 +615,6 @@ function datePropertyPaths(table: AnyV3Table): Set<string> { * return the raw value in that case rather than silently handing back an * Invalid Date whose later `.toISOString()` throws far from here (#742 review). */ -function reconstructDate(value: WasmPlaintext, isDateColumn: boolean): unknown { - if (!isDateColumn || value == null) return value - const date = new Date(value as string | number) - return Number.isNaN(date.getTime()) ? value : date -} - /** * Internal token used to gate the {@link WasmEncryptionClient} * constructor. Symbols are unique by reference, so external code can't @@ -1368,7 +1363,9 @@ export class WasmEncryptionClient { setNestedValue( rebuilt, field.fieldKey.split('.'), - reconstructDate(item.data, dateFields.has(field.fieldKey)), + dateFields.has(field.fieldKey) + ? reconstructDateValue(item.data) + : item.data, ) }) return rebuilt diff --git a/packages/wizard/src/lib/install-skills.ts b/packages/wizard/src/lib/install-skills.ts index b3f5d2f96..53a4e06ae 100644 --- a/packages/wizard/src/lib/install-skills.ts +++ b/packages/wizard/src/lib/install-skills.ts @@ -11,7 +11,7 @@ import type { Integration } from './types.js' * skills into the user's project so Claude Code picks them up during * follow-up work. */ -const SKILL_MAP: Record<Integration, readonly string[]> = { +export const SKILL_MAP: Record<Integration, readonly string[]> = { drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'], // `stash-postgres` / `stash-edge` mirror the CLI's SKILL_MAP — see the comments // there for why Supabase and the generic (no-ORM) path get them (#754). @@ -23,7 +23,12 @@ const SKILL_MAP: Record<Integration, readonly string[]> = { 'stash-edge', 'stash-cli', ], - prisma: ['stash-encryption', 'stash-indexing', 'stash-cli'], + prisma: [ + 'stash-encryption', + 'stash-prisma-next', + 'stash-indexing', + 'stash-cli', + ], generic: [ 'stash-encryption', 'stash-indexing', @@ -62,7 +67,9 @@ export async function maybeInstallSkills( return { copied: [], failed: [] } } - const available = skills.filter((name) => existsSync(join(bundledRoot, name))) + const available = skills.filter((name) => + existsSync(join(bundledRoot, name, 'SKILL.md')), + ) if (available.length === 0) return { copied: [], failed: [] } const confirmed = await p.confirm({ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 8a666f68e..cb4888fe8 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -102,6 +102,9 @@ importers: semver: specifier: ^7.8.0 version: 7.8.5 + typescript: + specifier: catalog:repo + version: 5.9.3 vitest: specifier: catalog:repo version: 3.2.7(@types/node@26.1.1)(jiti@2.7.0)(lightningcss@1.30.2)(terser@5.44.1)(tsx@4.23.0)(yaml@2.9.0) From 738741461cc7d313d7387d68d289785ea7f505e8 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:50:08 +1000 Subject: [PATCH 112/123] test: close issue 816 release gaps --- .changeset/nextjs-stack-metadata.md | 8 ++++ .github/workflows/tests.yml | 12 +++--- .../init/lib/__tests__/install-skills.test.ts | 12 ++++++ .../stack-drizzle/tsconfig.typecheck.json | 12 +++--- packages/stack-supabase/src/like-pattern.ts | 39 +++++++++++++++++++ packages/stack-supabase/src/query-builder.ts | 25 ++---------- .../__tests__/wasm-inline-models.test.ts | 20 ++++++++++ 7 files changed, 92 insertions(+), 36 deletions(-) create mode 100644 .changeset/nextjs-stack-metadata.md create mode 100644 packages/stack-supabase/src/like-pattern.ts diff --git a/.changeset/nextjs-stack-metadata.md b/.changeset/nextjs-stack-metadata.md new file mode 100644 index 000000000..2df589862 --- /dev/null +++ b/.changeset/nextjs-stack-metadata.md @@ -0,0 +1,8 @@ +--- +'@cipherstash/nextjs': patch +--- + +Correct the published package metadata to reference `@cipherstash/stack` +instead of the removed `@cipherstash/protect` package. The package now also +ships with its own source typecheck command and keeps its Vitest mock typing +compatible with the repository-pinned test runner. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 4ece7ea51..a6de8d262 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -166,13 +166,11 @@ jobs: # workspace dependencies through `dist/*.d.ts`, hence the turbo filters # (`^build` builds the dependencies first). # - # NOT gated yet, and deliberately: `@cipherstash/stack` (147 errors under - # its own tsconfig), `@cipherstash/stack-drizzle` (63), `stash` (21) and - # `@cipherstash/stack-supabase` (11). Their `test:types` scripts only - # cover `__tests__/**/*.test-d.ts`, so `src`, the runtime test suites and - # `integration/**` compile nowhere. Most of the drizzle and supabase count - # is one root cause — `V3_MATRIX`'s `indexes` union in - # `@cipherstash/test-kit` — see #778. + # NOT fully gated yet, and deliberately: `@cipherstash/stack`, `stash` + # and `@cipherstash/stack-supabase` still have source/runtime-test files + # outside their type-test configs. stack-drizzle's type-test config now + # includes every `integration/**` source; its remaining runtime source + # sets are tracked separately. - name: Typecheck (migrate) run: pnpm exec turbo run typecheck --filter @cipherstash/migrate diff --git a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts index 5e315fd48..07fb875f8 100644 --- a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts @@ -48,6 +48,18 @@ describe('SKILL_MAP', () => { expect(WIZARD_SKILL_MAP.generic).toEqual(SKILL_MAP.postgresql) }) + it('selects only skills that contain a bundled SKILL.md', () => { + for (const integration of ALL_INTEGRATIONS) { + const available = new Set(availableSkills(integration)) + const skills = SKILL_MAP[integration] + for (const skill of skills) { + expect(available.has(skill), `${integration}: ${skill}/SKILL.md`).toBe( + true, + ) + } + } + }) + it('has a non-empty entry for every integration (no undefined → crash)', () => { for (const integration of ALL_INTEGRATIONS) { const skills = SKILL_MAP[integration] diff --git a/packages/stack-drizzle/tsconfig.typecheck.json b/packages/stack-drizzle/tsconfig.typecheck.json index 051ef8107..1dad75198 100644 --- a/packages/stack-drizzle/tsconfig.typecheck.json +++ b/packages/stack-drizzle/tsconfig.typecheck.json @@ -1,13 +1,11 @@ { // The typecheck gate CI runs (`test:types` -> vitest typecheck) covers the - // `.test-d.ts` files plus the two `integration/**` adapters that #772's - // review found were erroring in a file nothing compiled. + // `.test-d.ts` files plus every `integration/**` source that #772's review + // found was otherwise compiled by no CI job. // - // `src/`, the runtime `__tests__/*.test.ts` and the rest of `integration/**` - // are still ungated: 63 errors under the full `tsconfig.json`, most of them - // one root cause — `V3_MATRIX`'s `indexes` union in `@cipherstash/test-kit` - // (#778). Widen this `include` as that count comes down; do not widen it to - // the whole project until it is zero, or the gate is useless. + // `src/` and runtime `__tests__/*.test.ts` are still outside this explicit + // gate. Keep this include focused until those remaining source sets are + // clean under the full project config. "extends": "./tsconfig.json", "include": ["__tests__/**/*.test-d.ts", "integration/**/*.ts"] } diff --git a/packages/stack-supabase/src/like-pattern.ts b/packages/stack-supabase/src/like-pattern.ts new file mode 100644 index 000000000..51874fc2a --- /dev/null +++ b/packages/stack-supabase/src/like-pattern.ts @@ -0,0 +1,39 @@ +type LikeToken = { value: string; wildcard: boolean } + +export type LikeNeedle = { + needle: string + hasUnsupportedWildcard: boolean +} + +/** + * Reduce a SQL LIKE pattern to the literal needle used by encrypted fuzzy + * matching. Only unescaped leading/trailing `%` tokens are approximable; + * escaped metacharacters remain literal and every other wildcard is reported. + */ +export function parseLikeNeedle(pattern: string): LikeNeedle { + const tokens: LikeToken[] = [] + for (let i = 0; i < pattern.length; i++) { + const char = pattern[i] + if (char === '\\' && i + 1 < pattern.length) { + tokens.push({ value: pattern[++i], wildcard: false }) + } else { + tokens.push({ + value: char, + wildcard: char === '%' || char === '_', + }) + } + } + + while (tokens[0]?.wildcard && tokens[0].value === '%') tokens.shift() + while ( + tokens[tokens.length - 1]?.wildcard && + tokens[tokens.length - 1]?.value === '%' + ) { + tokens.pop() + } + + return { + needle: tokens.map((token) => token.value).join(''), + hasUnsupportedWildcard: tokens.some((token) => token.wildcard), + } +} diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index 6516af1cf..eb9c2ee80 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -10,6 +10,7 @@ import type { AnyV3Table } from '@cipherstash/stack/eql/v3' import type { LockContextInput } from '@cipherstash/stack/identity' import { ColumnMap } from './column-map' import { addJsonbCastsV3 } from './helpers' +import { parseLikeNeedle } from './like-pattern' import { toDbSpace } from './query-dbspace' import { assertJsonContainmentOperand, @@ -868,32 +869,12 @@ export class EncryptedQueryBuilderImpl< * per (op, column) that the delegation is approximate. */ private likeNeedle(column: string, op: string, pattern: string): string { - const tokens: Array<{ value: string; wildcard: boolean }> = [] - for (let i = 0; i < pattern.length; i++) { - const char = pattern[i] - if (char === '\\' && i + 1 < pattern.length) { - tokens.push({ value: pattern[++i], wildcard: false }) - } else { - tokens.push({ - value: char, - wildcard: char === '%' || char === '_', - }) - } - } - - while (tokens[0]?.wildcard && tokens[0].value === '%') tokens.shift() - while ( - tokens[tokens.length - 1]?.wildcard && - tokens[tokens.length - 1]?.value === '%' - ) - tokens.pop() - - if (tokens.some((token) => token.wildcard)) { + const { needle, hasUnsupportedWildcard } = parseLikeNeedle(pattern) + if (hasUnsupportedWildcard) { throw new Error( `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an internal "%" or any "_"). Use matches("${column}", term) with a literal search term.`, ) } - const needle = tokens.map((token) => token.value).join('') const key = `${op}:${column}` if (!warnedLikeDelegation.has(key)) { warnedLikeDelegation.add(key) diff --git a/packages/stack/__tests__/wasm-inline-models.test.ts b/packages/stack/__tests__/wasm-inline-models.test.ts index eef1e0a33..d0d5a56bf 100644 --- a/packages/stack/__tests__/wasm-inline-models.test.ts +++ b/packages/stack/__tests__/wasm-inline-models.test.ts @@ -298,6 +298,26 @@ describe('WasmEncryptionClient.bulkDecryptModels', () => { ]) }) + it('reconstructs valid Dates in bulk and preserves invalid date values', async () => { + ffi.decryptBulkFallible.mockResolvedValueOnce([ + { data: '2026-07-22T01:02:03.000Z' }, + { data: 'not-a-date' }, + ] as never) + + const c = await client() + const out = await c.bulkDecryptModels( + [{ createdOn: ct('a') }, { createdOn: ct('b') }], + users, + ) + + const data = expectData(out) + expect(data[0].createdOn).toBeInstanceOf(Date) + expect((data[0].createdOn as Date).toISOString()).toBe( + '2026-07-22T01:02:03.000Z', + ) + expect(data[1].createdOn).toBe('not-a-date') + }) + it('labels failures with the model index and field path', async () => { ffi.decryptBulkFallible.mockResolvedValueOnce([ { data: 'ok' }, From 4dab29abe1e087e2e21ffce483ad04331d5f415d Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:58:09 +1000 Subject: [PATCH 113/123] test(stack-supabase): cover multi-leaf or() substitution and not.and() recursion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two guards in the string-form `.or()` path had no test that could see them fail. `substituteOrStringLeaves` sorts its replacements right-to-left so an earlier source span stays valid after a later one grew — and every v3 envelope is far longer than the plaintext operand it replaces. Sorting ascending instead splices the second envelope INSIDE the first and leaks the trailing condition verbatim: `email.eq."{\"created_at.gte."{…}"…}",createdAt.gte.2026-01-01`. Only one existing test replaced two leaves at all (the `is`-on-an-encrypted- column case, where the second replacement is `created_at.is.null` — shorter than the span it fills), so the growth case that motivates the sort was untested. `parseOrStringAt`'s group regex admits a `not.` prefix. Drop it and `not.and(email.eq.ada,…)` stops being a group: the leaf parser splits it at the first dot into the pseudo-column `not`, which matches no encrypted column, so the whole or-string takes the verbatim branch and `ada` goes to PostgREST in the clear against an encrypted column. Both were verified red against those exact mutations and green on restore. --- .../__tests__/supabase-v3-builder.test.ts | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index 5163918b5..ebd8eb202 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -557,6 +557,71 @@ describe('encryptedSupabaseV3 wire encoding', () => { expect(or.args[1]).toEqual({ referencedTable: 'profiles' }) }) + // `substituteOrStringLeaves` splices each encrypted leaf back into the + // caller's original string by source span, and sorts the replacements + // RIGHT-TO-LEFT so an earlier span stays valid after a later one grew — every + // v3 envelope is far longer than the plaintext operand it replaces. Every + // other string-form or() test carries exactly ONE encrypted leaf, where the + // sort direction cannot matter; with two, a left-to-right pass splices the + // second replacement at an offset that now lands INSIDE the first envelope. + it('substitutes every encrypted leaf of a multi-leaf or() string', async () => { + const { es, supabase } = v3Instance() + + await es + .from('users', users) + .select('id') + .or('email.eq.ada,createdAt.gte.2026-01-01') + + const emitted = supabase.callsFor('or')[0].args[0] as string + + // Neither operand may reach PostgREST as plaintext. + expect(emitted).not.toContain('email.eq.ada') + expect(emitted).not.toContain('.gte.2026-01-01') + + // The two conditions are still separated by exactly one top-level comma, + // and the second still carries its DB column name. Split on the delimiter + // rather than on `,`: every quote inside an envelope is backslash-escaped, + // so this boundary occurs once. + const boundary = emitted.indexOf('",created_at.gte."') + expect(boundary).toBeGreaterThan(0) + const first = emitted.slice(0, boundary + 1) + const second = emitted.slice(boundary + 2) + + expect(first).toMatch(/^email\.eq\."/) + expect(second).toMatch(/^created_at\.gte\."/) + expect(JSON.parse(orOperand(first, 'email.eq.')).pt).toBe('ada') + expect(JSON.parse(orOperand(second, 'created_at.gte.')).pt).toBe( + '2026-01-01', + ) + }) + + // `parseOrStringAt`'s group regex admits a `not.` prefix + // (`/^(?:not\.)?(?:and|or)\(…\)$/`). Without it, `not.and(…)` is not a group: + // the leaf parser splits it at the first dot into the pseudo-column `not`, + // which matches no encrypted column — so the whole or-string takes the + // verbatim branch and the encrypted operand goes to the database in the clear. + it('encrypts a leaf nested inside a not.and() group', async () => { + const { es, supabase } = v3Instance() + + await es + .from('users', users) + .select('id') + .or('not.and(email.eq.ada,note.eq.x),id.eq.1') + + const emitted = supabase.callsFor('or')[0].args[0] as string + + // The `not.and(` wrapper and both plaintext siblings survive byte-for-byte. + expect(emitted).toMatch(/^not\.and\(email\.eq\."/) + expect(emitted.endsWith(',note.eq.x),id.eq.1')).toBe(true) + expect(emitted).not.toContain('email.eq.ada') + + const operand = emitted.slice( + 'not.and('.length, + emitted.length - ',note.eq.x),id.eq.1'.length, + ) + expect(JSON.parse(orOperand(operand, 'email.eq.')).pt).toBe('ada') + }) + it('preserves referencedTable for structured or() filters', async () => { const { es, supabase } = v3Instance() From 34c48eb84568ff1514c1c471ac77fdd9df6cd663 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 12:59:21 +1000 Subject: [PATCH 114/123] docs(stack): move the date-reconstruction rationale onto the function it explains MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `reconstructDate` was deleted from `wasm-inline.ts` when date reconstruction moved to `eql/v3/date-reconstruction.ts`, but its doc comment survived and came to rest above `INTERNAL_CONSTRUCT` — two JSDoc blocks stacked over one constant, the first documenting nothing. The explanation it carried is still the reason `reconstructDateValue` guards on `Number.isNaN` at all, so it lands there: an unparseable stored value must come back as the raw value, not as an Invalid Date whose later `.toISOString()` throws far from the column that produced it (#742 review). --- packages/stack/src/eql/v3/date-reconstruction.ts | 13 +++++++++++-- packages/stack/src/wasm-inline.ts | 7 ------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/packages/stack/src/eql/v3/date-reconstruction.ts b/packages/stack/src/eql/v3/date-reconstruction.ts index fcefb081e..9679fd1bf 100644 --- a/packages/stack/src/eql/v3/date-reconstruction.ts +++ b/packages/stack/src/eql/v3/date-reconstruction.ts @@ -1,5 +1,14 @@ -/** Reconstruct a decrypted date-like plaintext without manufacturing an - * `Invalid Date`. Non-date values and already-constructed Dates pass through. */ +/** + * Rebuild a decrypted plaintext for a date-like model field into a `Date`. + * Non-date values and already-constructed Dates pass through untouched. + * + * The `Number.isNaN` guard is the point of the function, not a formality: an + * unparseable stored value — a date column written in a non-ISO format, or + * corrupted — makes `new Date(...)` an Invalid Date. Returning the raw value + * instead means the caller sees what is actually stored, rather than an + * Invalid Date whose later `.toISOString()` throws far from the column that + * produced it (#742 review). + */ export function reconstructDateValue(value: unknown): unknown { if ( value == null || diff --git a/packages/stack/src/wasm-inline.ts b/packages/stack/src/wasm-inline.ts index af9d467b8..f3d31e3fe 100644 --- a/packages/stack/src/wasm-inline.ts +++ b/packages/stack/src/wasm-inline.ts @@ -608,13 +608,6 @@ function datePropertyPaths(table: AnyV3Table): Set<string> { return paths } -/** - * Rebuild a decrypted plaintext for a model field into a `Date` when the - * column is date-like. An unparseable stored value (a date column written in a - * non-ISO format, or corrupted) would make `new Date(...)` an Invalid Date; - * return the raw value in that case rather than silently handing back an - * Invalid Date whose later `.toISOString()` throws far from here (#742 review). - */ /** * Internal token used to gate the {@link WasmEncryptionClient} * constructor. Symbols are unique by reference, so external code can't From d6ded7ff58f14952a946d01da9886d2d466f17ca Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:01:07 +1000 Subject: [PATCH 115/123] refactor(stack-supabase,skills): drop the unreachable csv transform, and correct the skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `csv()` throws unconditionally — PostgREST serializes rows server-side, so a CSV response carries ciphertext the adapter never gets to decrypt. Nothing can therefore push `{ kind: 'csv' }`, yet the transform survived in three places: the `TransformOp` union, the `toDbSpace` pass-through switch, and the `buildAndExecuteQuery` apply switch. All three go; the fail-closed throw stays exactly as it is. `skills/stash-supabase/SKILL.md` still listed `.csv()` among the transforms "passed through to Supabase directly". That ships inside the `stash` tarball and gets copied into customer repos, so it was about to become wrong guidance in someone else's codebase. It now documents the throw and shows serializing the decrypted rows instead. --- packages/stack-supabase/src/query-builder.ts | 3 --- packages/stack-supabase/src/query-dbspace.ts | 1 - packages/stack-supabase/src/types.ts | 10 +++++++++- skills/stash-supabase/SKILL.md | 10 +++++++++- 4 files changed, 18 insertions(+), 6 deletions(-) diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index eb9c2ee80..e583aa055 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -727,9 +727,6 @@ export class EncryptedQueryBuilderImpl< case 'maybeSingle': query = query.maybeSingle() break - case 'csv': - query = query.csv() - break case 'abortSignal': query = query.abortSignal(t.signal) break diff --git a/packages/stack-supabase/src/query-dbspace.ts b/packages/stack-supabase/src/query-dbspace.ts index fed373a64..42ae9cb9e 100644 --- a/packages/stack-supabase/src/query-dbspace.ts +++ b/packages/stack-supabase/src/query-dbspace.ts @@ -71,7 +71,6 @@ function transformToDbSpace(t: TransformOp, columns: ColumnMap): DbTransformOp { case 'range': case 'single': case 'maybeSingle': - case 'csv': case 'abortSignal': case 'throwOnError': case 'returns': diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index f6a53ba61..a2500a2e6 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -599,7 +599,8 @@ export type TransformOp = } | { kind: 'single' } | { kind: 'maybeSingle' } - | { kind: 'csv' } + // No `csv` member: `csv()` throws rather than recording a transform (see + // {@link EncryptedQueryBuilderCore.csv}), so nothing can ever push one. | { kind: 'abortSignal'; signal: AbortSignal } | { kind: 'throwOnError' } | { kind: 'returns' } @@ -969,6 +970,13 @@ export interface EncryptedQueryBuilderCore< * error. Same `T | null` awaited shape — `single()` reports the missing row * through `error` instead. */ maybeSingle(): EncryptedSingleQueryBuilder<T> + /** + * Always THROWS. PostgREST serializes rows server-side, so a CSV response + * carries ciphertext the adapter never gets to decrypt. Declared so the call + * is a loud runtime failure rather than a missing method, and typed `Self` + * only to keep the chain shape — it never returns. Select rows normally and + * serialize the decrypted data yourself. + */ csv(): Self abortSignal(signal: AbortSignal): Self throwOnError(): Self diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index deacdf19c..66eb932d6 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -372,12 +372,20 @@ These are passed through to Supabase directly: .order("email", { ascending: true }) // encrypted columns: see behaviour below .limit(10) .range(0, 9) -.csv() .abortSignal(signal) .throwOnError() .returns<U>() ``` +`csv()` is the exception — it **throws**. PostgREST serializes rows +server-side, so a CSV response would carry ciphertext the wrapper never gets +to decrypt. Select rows normally and serialize the decrypted data yourself: + +```typescript +const { data } = await es.from("users").select("id, email") +const csv = data!.map((r) => `${r.id},${r.email}`).join("\n") +``` + `order()` works on plaintext columns and on OPE-backed encrypted ordering columns — see the `order()` bullet in the next section for exactly which domains qualify. From fcb627b16c2b4d1ffa6d4f3c6ce2eb4b5eaabe4d Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:02:32 +1000 Subject: [PATCH 116/123] perf(stack-supabase): reconstruct nested date paths in one pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `postprocessDecryptedRow` called `reconstructDatePaths(out, [key])` once per dotted date key, and that helper allocates a fresh shallow row copy on every call — so a row with N nested date columns paid N clones to produce one result. Collect the dotted keys and make a single call. The batching is also the correctness-preserving form for two paths under a shared prefix: inside one call the second path clones the intermediate the first already rebuilt (`targetChild` wins over `sourceChild`), so both Dates and their plaintext siblings survive. Covered by a new `profile.createdAt` / `profile.updatedAt` test, which goes red if that preference is dropped. The flat-key branch is unchanged. --- .../__tests__/supabase-v3-builder.test.ts | 48 +++++++++++++++++++ packages/stack-supabase/src/query-results.ts | 13 +++-- 2 files changed, 58 insertions(+), 3 deletions(-) diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index ebd8eb202..0ee07e9d5 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -1178,6 +1178,54 @@ describe('encryptedSupabaseV3 wire encoding', () => { expect(profile.createdAt).toBeInstanceOf(Date) }) + // Two dotted paths under the SAME prefix. `postprocessDecryptedRow` now + // batches every dotted key into one `reconstructDatePaths` call instead of + // one call per key; within a single call the second path must clone the + // `profile` object the first already rebuilt, or one of the two Dates is + // silently thrown away along with any plaintext sibling. + it('reconstructs two dotted date paths sharing a prefix', async () => { + const nestedTable = encryptedTable('profiles', { + 'profile.createdAt': types.Timestamp('profile_created_at'), + 'profile.updatedAt': types.Timestamp('profile_updated_at'), + }) + const supabase = createMockSupabase([ + { + profile: { + createdAt: '2026-01-02T03:04:05.000Z', + updatedAt: '2026-03-04T05:06:07.000Z', + nickname: 'ada', + }, + }, + ]) + const builder = new EncryptedQueryBuilderV3Impl( + 'profiles', + nestedTable, + createMockEncryptionClient(), + supabase.client, + ['profile_created_at', 'profile_updated_at'], + ) + + const { data } = await builder.select( + 'profile.createdAt, profile.updatedAt', + ) + + if (!data) throw new Error('Expected a decrypted row') + const profile = (data[0] as Record<string, unknown>).profile as Record< + string, + unknown + > + expect(profile.createdAt).toBeInstanceOf(Date) + expect((profile.createdAt as Date).toISOString()).toBe( + '2026-01-02T03:04:05.000Z', + ) + expect(profile.updatedAt).toBeInstanceOf(Date) + expect((profile.updatedAt as Date).toISOString()).toBe( + '2026-03-04T05:06:07.000Z', + ) + // The plaintext sibling rides along in the same nested object. + expect(profile.nickname).toBe('ada') + }) + // Selecting by raw DB name means the row comes back keyed `created_at`, // the only way to reach the `dbName` half of the two-key branch. It also // exercises the `value == null` skip on the absent `createdAt` key. diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts index 425ca8c75..b8daa6ccd 100644 --- a/packages/stack-supabase/src/query-results.ts +++ b/packages/stack-supabase/src/query-results.ts @@ -63,17 +63,24 @@ function postprocessDecryptedRow( keyToDb[dbName] ??= dbName } - let out: Record<string, unknown> = { ...row } + const out: Record<string, unknown> = { ...row } + // Dotted paths are collected and reconstructed in ONE pass: + // `reconstructDatePaths` allocates a fresh shallow row copy per call, so + // calling it per key spent one clone on every nested date column. Batching + // is also what keeps two paths sharing a prefix (`profile.createdAt` / + // `profile.updatedAt`) correct — within a single call the second path clones + // the intermediate the first already rebuilt, rather than the raw source. + const dottedKeys: string[] = [] for (const [key, dbName] of Object.entries(keyToDb)) { const castAs = ctx.columns.schemaFor(dbName)?.cast_as if (!DATE_LIKE_CAST_SET.has(castAs as string)) continue if (key.includes('.')) { - out = reconstructDatePaths(out, [key]) + dottedKeys.push(key) } else if (Object.hasOwn(out, key)) { out[key] = reconstructDateValue(out[key]) } } - return out + return dottedKeys.length > 0 ? reconstructDatePaths(out, dottedKeys) : out } /** From 673cc071f18dc0b160e58dc8f62cd2d349fd2de1 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:03:20 +1000 Subject: [PATCH 117/123] test(stack-drizzle): name the insert-row helper for what it actually checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `checkedInsertRows` reads as though it validates the rows it returns. It validates `rowKey` and `testRunId` are strings and then does `return rows as T[]` — the encrypted columns, the ones that could plausibly be malformed, are never looked at. Renamed to `assertScopeKeys`, with a doc comment saying plainly that it re-establishes the Drizzle insert type after asserting only that the two plaintext scope keys survived encryption. Verified through `pnpm --filter @cipherstash/stack-drizzle run test:types`, which is the script driving `tsconfig.typecheck.json` — now widened to `integration/**/*.ts`. Confirmed the gate really compiles this file by planting a deliberate type error in it and watching the run exit 1. --- .../integration/relational.integration.test.ts | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/packages/stack-drizzle/integration/relational.integration.test.ts b/packages/stack-drizzle/integration/relational.integration.test.ts index 7173a7fb5..892e73077 100644 --- a/packages/stack-drizzle/integration/relational.integration.test.ts +++ b/packages/stack-drizzle/integration/relational.integration.test.ts @@ -183,7 +183,19 @@ function encryptedInsertRows(): MatrixPlainRow[] { }) } -function checkedInsertRows<T extends { rowKey: string; testRunId: string }>( +/** + * Re-establish the Drizzle insert type on rows that came back from + * `bulkEncryptModels`, which is typed against the runtime-shaped `AnyV3Table` + * and so returns `Record<string, unknown>[]`. + * + * The assertion is the narrow part of the contract: it checks ONLY that the + * two plaintext scope keys — `rowKey` and `testRunId`, the ones every query + * and the run-scoped cleanup filter on — survived encryption as strings. The + * encrypted columns are NOT validated; their shape is what the assertions in + * the tests themselves are for. Named for that scope so the cast at the end + * does not read as a checked conversion of the whole row. + */ +function assertScopeKeys<T extends { rowKey: string; testRunId: string }>( rows: unknown[], ): T[] { for (const row of rows) { @@ -253,7 +265,7 @@ beforeAll(async () => { ) `) - const encryptedRows = checkedInsertRows<typeof matrixTable.$inferInsert>( + const encryptedRows = assertScopeKeys<typeof matrixTable.$inferInsert>( unwrapResult(await client.bulkEncryptModels(encryptedInsertRows(), schema)), ) await db.insert(matrixTable).values(encryptedRows) @@ -270,7 +282,7 @@ beforeAll(async () => { // ROW_B exists so the filter proofs below have a row they must EXCLUDE. On a // one-row table `gt(balance, 0n)` returning every row is indistinguishable // from it returning the right row. - const bigintRows = checkedInsertRows<typeof bigintTable.$inferInsert>( + const bigintRows = assertScopeKeys<typeof bigintTable.$inferInsert>( unwrapResult( await client.bulkEncryptModels( [ From 5b03135dbacefa36d462ed460950f2a651604bdb Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:05:03 +1000 Subject: [PATCH 118/123] test(e2e): assert CLI/wizard SKILL_MAP parity where both packages are declared MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The parity assertion lived in the CLI's unit suite and reached the wizard through `../../../../../../wizard/src/lib/install-skills.js` — a deep relative path into a package `packages/cli` declares no dependency on, targeting a non-published internal module. If the wizard relocated that file the only signal was a module-not-found in a different package's test run. Chose option (a), moving it to `e2e/`, over adding a devDependency and a comment (option (b)), for three reasons: - `e2e/` already declares BOTH `stash` and `@cipherstash/wizard` as workspace dependencies, so the layering is honest without widening the CLI's dep graph for a single assertion. - `e2e` has a `typecheck` script over `tests/**` and CI runs it ("Typecheck (e2e)"). `packages/cli` has no typecheck script at all, so this is the only placement where a relocated module is a COMPILE error. Verified: pointing the import at a non-existent path fails `pnpm --filter @cipherstash/e2e run typecheck` with TS2307. - Cross-package behaviour is what that workspace is for, per its own description — and reading source rather than a built binary is an existing idiom there (`package-managers.e2e.test.ts` Suite A), so no build is needed. `SKILL_MAP` stays unexported from the wizard's package entry — the published surface should not grow for a test. The assertion also got stronger in the move: the wizard→CLI name mapping is now a `Record<WizardIntegration, CliIntegration>`, so a new integration on either side fails to compile or fails the key-coverage test, instead of being silently omitted from a hand-listed set of four expectations. --- e2e/README.md | 1 + e2e/tests/skill-map-parity.e2e.test.ts | 60 +++++++++++++++++++ .../init/lib/__tests__/install-skills.test.ts | 12 ++-- 3 files changed, 65 insertions(+), 8 deletions(-) create mode 100644 e2e/tests/skill-map-parity.e2e.test.ts diff --git a/e2e/README.md b/e2e/README.md index 3e2e625ac..c3ee77197 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -25,6 +25,7 @@ pnpm --filter @cipherstash/e2e exec vitest run tests/package-managers.e2e.test.t | `tests/package-managers.e2e.test.ts` | The `init` providers and the wizard binary render `bunx`/`pnpm dlx`/`yarn dlx`/`npx` based on detected package manager. | | `tests/supply-chain.e2e.test.ts` | Lockfile/registry/CODEOWNERS controls from `skills/stash-supply-chain-security` are still enforced. | | `tests/prisma-example-readme.e2e.test.ts` | Parses `examples/prisma/README.md`'s "Run it" section and asserts every command (skipping `stash auth login`) exits 0. | +| `tests/skill-map-parity.e2e.test.ts` | The CLI's and the wizard's `SKILL_MAP` install the same skills for each equivalent integration. Lives here because this is the only workspace declaring both packages. | ## Auth-dependent suites diff --git a/e2e/tests/skill-map-parity.e2e.test.ts b/e2e/tests/skill-map-parity.e2e.test.ts new file mode 100644 index 000000000..3766a48d1 --- /dev/null +++ b/e2e/tests/skill-map-parity.e2e.test.ts @@ -0,0 +1,60 @@ +/** + * The CLI and the wizard each own a `SKILL_MAP`, and the two must agree: both + * install skills into the same `.claude/skills` / `.codex/skills` directory of + * the same user project, so a user who ran `npx stash init` and a user who ran + * the wizard would otherwise end up with different guidance for the same + * integration. The maps drift silently — nothing imports one from the other. + * + * This assertion lives HERE, not in either package's own suite, because it is + * the only workspace that declares both `stash` and `@cipherstash/wizard` as + * dependencies. It reads source rather than a built binary (the same idiom as + * `package-managers.e2e.test.ts` Suite A) so it needs no build step, and + * `pnpm --filter @cipherstash/e2e run typecheck` compiles `tests/**` — which + * makes a relocated module a compile error in CI rather than a module-not-found + * inside an unrelated package's unit run. + * + * The integration names differ by design: the CLI names the packages + * (`prisma-next`, `postgresql`), the wizard names the user's situation + * (`prisma`, `generic`). The mapping below is the whole contract. + */ + +import { describe, expect, it } from 'vitest' + +import { SKILL_MAP as CLI_SKILL_MAP } from '../../packages/cli/src/commands/init/lib/install-skills.js' +import type { Integration as CliIntegration } from '../../packages/cli/src/commands/init/types.js' +import { SKILL_MAP as WIZARD_SKILL_MAP } from '../../packages/wizard/src/lib/install-skills.js' +import type { Integration as WizardIntegration } from '../../packages/wizard/src/lib/types.js' + +// Exhaustive over the wizard union: a new wizard integration added without a +// CLI counterpart fails to compile here, rather than shipping a divergent set. +const EQUIVALENT: Record<WizardIntegration, CliIntegration> = { + drizzle: 'drizzle', + supabase: 'supabase', + prisma: 'prisma-next', + generic: 'postgresql', +} + +describe('CLI and wizard SKILL_MAP parity', () => { + it('installs the same skills for every equivalent integration', () => { + for (const [wizardName, cliName] of Object.entries(EQUIVALENT) as Array< + [WizardIntegration, CliIntegration] + >) { + expect( + WIZARD_SKILL_MAP[wizardName], + `${wizardName} (wizard) vs ${cliName} (cli)`, + ).toEqual(CLI_SKILL_MAP[cliName]) + } + }) + + // Both unions are closed, so parity over `EQUIVALENT` is only complete while + // it covers every CLI integration too — a CLI-only integration would slip + // through the loop above unnoticed. + it('covers every integration on both sides', () => { + expect(Object.keys(WIZARD_SKILL_MAP).sort()).toEqual( + Object.keys(EQUIVALENT).sort(), + ) + expect(Object.keys(CLI_SKILL_MAP).sort()).toEqual( + Object.values(EQUIVALENT).sort(), + ) + }) +}) diff --git a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts index 07fb875f8..5da026907 100644 --- a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts @@ -17,7 +17,6 @@ import type { Integration } from '../../types.js' vi.mock('@clack/prompts', () => ({ log: { warn: vi.fn() } })) import * as p from '@clack/prompts' -import { SKILL_MAP as WIZARD_SKILL_MAP } from '../../../../../../wizard/src/lib/install-skills.js' import { availableSkills, installSkills, @@ -40,14 +39,11 @@ const ALL_INTEGRATIONS: Integration[] = [ 'postgresql', ] +// Parity with the wizard's own SKILL_MAP is asserted in +// `e2e/tests/skill-map-parity.e2e.test.ts` — the only workspace that declares +// both `stash` and `@cipherstash/wizard`, and whose `typecheck` script compiles +// the cross-package import. describe('SKILL_MAP', () => { - it('stays in parity with every wizard integration', () => { - expect(WIZARD_SKILL_MAP.drizzle).toEqual(SKILL_MAP.drizzle) - expect(WIZARD_SKILL_MAP.supabase).toEqual(SKILL_MAP.supabase) - expect(WIZARD_SKILL_MAP.prisma).toEqual(SKILL_MAP['prisma-next']) - expect(WIZARD_SKILL_MAP.generic).toEqual(SKILL_MAP.postgresql) - }) - it('selects only skills that contain a bundled SKILL.md', () => { for (const integration of ALL_INTEGRATIONS) { const available = new Set(availableSkills(integration)) From 7f708791e7b1b44b5972d2d537fcf0cd5d275027 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:06:00 +1000 Subject: [PATCH 119/123] docs(changeset): restore the encryptedSupabaseV3 alias, and say what the nested-or() fix was MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changesets had been edited to delete the statement that `encryptedSupabaseV3` survives as a type-identical `@deprecated` alias. It does — `packages/stack-supabase/src/index.ts:275`, alongside the `*V3` type aliases at 282-297 — the repo's own tests import it, and `AGENTS.md` documents it. A changeset that contradicts the shipped code is worse than a terse one, so both get a tight sentence back. `adapter-release-readiness` compressed the nested-group fix into "preserves nested PostgREST boolean expressions", which reads as a formatting nicety. It was a disclosure fix: with no group recursion, `and(createdAt.gte.…,…)` was cut at the first dot into the pseudo-column `and(createdAt`, matched no encrypted column, and took the verbatim branch — so the operand reached PostgREST as plaintext against an encrypted column, under the JS property name rather than the DB column name. Users need that stated outright. Also records the `stash-supabase` skill correction (csv() throws, it is not a pass-through transform) on the changeset that already bumps `stash`. --- .changeset/adapter-release-readiness.md | 22 +++++++++++++++---- .../remove-eql-v2-supabase-authoring.md | 7 ++++-- .changeset/remove-eql-v2-supabase-skill.md | 7 +++--- 3 files changed, 27 insertions(+), 9 deletions(-) diff --git a/.changeset/adapter-release-readiness.md b/.changeset/adapter-release-readiness.md index f0ced1557..dc3540e00 100644 --- a/.changeset/adapter-release-readiness.md +++ b/.changeset/adapter-release-readiness.md @@ -7,10 +7,24 @@ Finish the EQL v2-removal release gates and adapter correctness pass. -- Supabase preserves nested PostgREST boolean expressions and - `referencedTable`, never sends nullish encrypted search operands as - plaintext, honours escaped LIKE metacharacters, rejects CSV result mode - before decryption, and diagnoses the removed object-form factory call. +- **Supabase encrypts leaves nested inside a PostgREST boolean group.** This + is a disclosure fix, not a formatting one. `parseOrString` had no group + recursion, so `.or('and(createdAt.gte.2026-01-01,note.eq.x)')` came back from + the top-level split as one part and the leaf parser cut it at the first dot + into the pseudo-column `and(createdAt`. That name matched no encrypted + column, so the whole expression took the verbatim branch: the operand + `2026-01-01` reached PostgREST **as plaintext, against an encrypted column**, + under the JS property name `createdAt` rather than the DB column name + `created_at`. Every encrypted leaf nested inside `and(...)` / `or(...)` / + `not.and(...)` leaked its operand to the database and returned wrong results. + Nested groups and `referencedTable` are now preserved while each encrypted + leaf is substituted in place. +- Supabase never sends nullish encrypted search operands as plaintext, honours + escaped LIKE metacharacters, rejects CSV result mode before decryption, and + diagnoses the removed object-form factory call. The bundled `stash-supabase` + skill no longer lists `csv()` among the transforms passed through to + Supabase — it throws, and the skill now says so and shows serializing the + decrypted rows instead. - Native, WASM, and Supabase model decryption reconstruct valid date and timestamp values consistently, including nested paths, aliases, and bulk results, while leaving invalid values unchanged. diff --git a/.changeset/remove-eql-v2-supabase-authoring.md b/.changeset/remove-eql-v2-supabase-authoring.md index 59c6e036d..0bf7fb5f5 100644 --- a/.changeset/remove-eql-v2-supabase-authoring.md +++ b/.changeset/remove-eql-v2-supabase-authoring.md @@ -5,7 +5,9 @@ Remove the EQL v2 authoring surface and de-suffix the v3 API to the canonical unsuffixed names (part of the EQL v2 removal, #707). -- **`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory.** +- **`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory** + (formerly `encryptedSupabaseV3`). `encryptedSupabaseV3` remains a + type-identical `@deprecated` alias, so existing imports keep working. - **The legacy v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper is removed** — with it the two-argument `from(tableName, schema)` form and the hand-written client-side v2 schema. Its `EncryptedSupabaseConfig` and @@ -14,7 +16,8 @@ unsuffixed names (part of the EQL v2 removal, #707). - **The public types use canonical unsuffixed names:** `EncryptedSupabaseOptions`, `EncryptedSupabaseInstance`, `TypedEncryptedSupabaseInstance`, `EncryptedQueryBuilder`, - `EncryptedQueryBuilderUntyped`, `FilterableKeys`, and `OrderableKeys`. + `EncryptedQueryBuilderUntyped`, `FilterableKeys`, and `OrderableKeys`. Each + keeps a type-identical `@deprecated` `*V3` alias. **Reading existing v2 data.** Only the v2 *authoring/emission* surface is removed — no v2 ciphertext is stranded. Decryption in `@cipherstash/stack` is diff --git a/.changeset/remove-eql-v2-supabase-skill.md b/.changeset/remove-eql-v2-supabase-skill.md index 29d4b5b89..c632acc7f 100644 --- a/.changeset/remove-eql-v2-supabase-skill.md +++ b/.changeset/remove-eql-v2-supabase-skill.md @@ -3,8 +3,9 @@ --- Update the bundled `stash-supabase` agent skill for the EQL v2 removal (#707): -`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory, and the legacy v2 -`encryptedSupabase({ encryptionClient, supabaseClient })` authoring wrapper has -been removed. The skill's examples, exported-type list, and migration/cutover +`encryptedSupabase` is now the connect-time-introspecting EQL v3 factory (with +`encryptedSupabaseV3` kept as a type-identical `@deprecated` alias), and the +legacy v2 `encryptedSupabase({ encryptionClient, supabaseClient })` authoring +wrapper has been removed. The skill's examples, exported-type list, and migration/cutover guidance are corrected accordingly. Skills ship inside the `stash` tarball, so the stale v2 guidance would otherwise land in a user's project. From 2db092d57ee5a31d4f41aa4ee836097dc7446c54 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 13:43:56 +1000 Subject: [PATCH 120/123] docs(ci,stack-drizzle): keep the error counts and #778 in the gate comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Widening `stack-drizzle`'s typecheck `include` to `integration/**/*.ts` made the surrounding comments stale, and rewriting them dropped the two things that made them actionable: the per-package error counts, and the `#778` root cause (`V3_MATRIX`'s `indexes` union in `@cipherstash/test-kit`). Those counts are the mechanism the original comment describes — "widen this include as that count comes down" is not a usable instruction once the count is gone, and "tracked separately" does not say where. Restore both, keeping the corrected scope: drizzle now type-tests every `integration/**` source, so only its `src` and runtime tests remain ungated. Comment-only; no build or gate behaviour changes. --- .github/workflows/tests.yml | 13 ++++++++----- packages/stack-drizzle/tsconfig.typecheck.json | 8 +++++--- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a6de8d262..4a656554d 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -166,11 +166,14 @@ jobs: # workspace dependencies through `dist/*.d.ts`, hence the turbo filters # (`^build` builds the dependencies first). # - # NOT fully gated yet, and deliberately: `@cipherstash/stack`, `stash` - # and `@cipherstash/stack-supabase` still have source/runtime-test files - # outside their type-test configs. stack-drizzle's type-test config now - # includes every `integration/**` source; its remaining runtime source - # sets are tracked separately. + # NOT fully gated yet, and deliberately: `@cipherstash/stack` (147 errors + # under its own tsconfig), `stash` (21) and `@cipherstash/stack-supabase` + # (11). Their `test:types` scripts only cover `__tests__/**/*.test-d.ts`, + # so `src` and the runtime test suites compile nowhere. + # `@cipherstash/stack-drizzle` now type-tests every `integration/**` + # source as well; its remaining `src` / runtime-test errors share one + # root cause — `V3_MATRIX`'s `indexes` union in `@cipherstash/test-kit` + # — see #778. - name: Typecheck (migrate) run: pnpm exec turbo run typecheck --filter @cipherstash/migrate diff --git a/packages/stack-drizzle/tsconfig.typecheck.json b/packages/stack-drizzle/tsconfig.typecheck.json index 1dad75198..18f06548b 100644 --- a/packages/stack-drizzle/tsconfig.typecheck.json +++ b/packages/stack-drizzle/tsconfig.typecheck.json @@ -3,9 +3,11 @@ // `.test-d.ts` files plus every `integration/**` source that #772's review // found was otherwise compiled by no CI job. // - // `src/` and runtime `__tests__/*.test.ts` are still outside this explicit - // gate. Keep this include focused until those remaining source sets are - // clean under the full project config. + // `src/` and the runtime `__tests__/*.test.ts` are still ungated under the + // full `tsconfig.json`, most of it one root cause — `V3_MATRIX`'s `indexes` + // union in `@cipherstash/test-kit` (#778). Widen this `include` as that + // count comes down; do not widen it to the whole project until it is zero, + // or the gate is useless. "extends": "./tsconfig.json", "include": ["__tests__/**/*.test-d.ts", "integration/**/*.ts"] } From 0f54c0e181572588096ee18c49a3de23b9a2cd47 Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 14:11:16 +1000 Subject: [PATCH 121/123] refactor(stack-supabase): collapse the or-string parser to the one production runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `parseOrStringWithSpans` became the only parser any production path calls when the nested-group fix landed, but `parseOrString` survived alongside it and kept the entire helpers suite — including assertions that pinned the NON-recursing behaviour, e.g. `and(a.eq.1,b.eq.2),c.eq.3` yielding two conditions, one of them a junk leaf on a column literally named `and(a`. The parser with the coverage was the dead one; the live one's group recursion and source spans were exercised only indirectly, through the builder. Delete the wrapper, repoint the suite at `parseOrStringWithSpans`, and drop the now-constant `recurseGroups` parameter. A local `parseConditions` strips `sourceSpan` so every existing `toEqual` stays exactly as strict about the condition shape as it was. Adds the span-level coverage that existed nowhere: a leaf one and two groups deep, inside `not.and(`, after array and jsonb containment literals whose commas are not delimiters, with surrounding whitespace excluded, and two identical leaves getting distinct spans — plus `substituteOrStringLeaves` replacing leaves at two nesting depths. Both are real regressions tests: flipping the splice to left-to-right fails two of them, and shifting the group base offset by one fails seven. Nothing in the suite caught either mutation before. --- .../__tests__/supabase-helpers.test.ts | 350 +++++++++++++++--- packages/stack-supabase/src/helpers.ts | 37 +- packages/stack-supabase/src/query-dbspace.ts | 3 +- packages/stack-supabase/src/query-filters.ts | 2 +- 4 files changed, 319 insertions(+), 73 deletions(-) diff --git a/packages/stack-supabase/__tests__/supabase-helpers.test.ts b/packages/stack-supabase/__tests__/supabase-helpers.test.ts index cb3ef2f88..52aa1a73e 100644 --- a/packages/stack-supabase/__tests__/supabase-helpers.test.ts +++ b/packages/stack-supabase/__tests__/supabase-helpers.test.ts @@ -1,5 +1,10 @@ import { describe, expect, it } from 'vitest' -import { addJsonbCastsV3, parseOrString, rebuildOrString } from '../src/helpers' +import { + addJsonbCastsV3, + parseOrStringWithSpans, + rebuildOrString, + substituteOrStringLeaves, +} from '../src/helpers' import type { DbPendingOrCondition } from '../src/types' // `createdAt` is a renamed property (DB column `created_at`); `email` is a @@ -83,6 +88,21 @@ function cond(column: string, op: string, value: unknown, negate?: boolean) { return { column, op, value, negate } as unknown as DbPendingOrCondition } +/** + * The parsed leaves with `sourceSpan` dropped, for the assertions that are about + * column / op / negate / value. + * + * Spans are pinned exactly — and against the source text they must slice back + * to — in their own describe below, so removing them here costs no coverage and + * keeps these `toEqual`s as strict about the condition shape as they have + * always been. + */ +function parseConditions(orString: string) { + return parseOrStringWithSpans(orString).map( + ({ sourceSpan: _sourceSpan, ...condition }) => condition, + ) +} + describe('rebuildOrString quoting', () => { it('escapes the double quotes inside a quoted operand', () => { const out = rebuildOrString([cond('email', 'eq', ENVELOPE)]) @@ -188,13 +208,13 @@ describe('rebuildOrString containment', () => { }) }) -describe('parseOrString / rebuildOrString round-trip', () => { +describe('parseOrStringWithSpans / rebuildOrString round-trip', () => { it('round-trips an encrypted JSON envelope operand', () => { const conditions = [ { column: 'email', op: 'eq', negate: false, value: ENVELOPE }, ] expect( - parseOrString( + parseConditions( rebuildOrString(conditions.map((c) => cond(c.column, c.op, c.value))), ), ).toEqual(conditions) @@ -205,7 +225,7 @@ describe('parseOrString / rebuildOrString round-trip', () => { { column: 'a', op: 'eq', negate: false, value: 'x\\"y,z' }, ] expect( - parseOrString( + parseConditions( rebuildOrString(conditions.map((c) => cond(c.column, c.op, c.value))), ), ).toEqual(conditions) @@ -216,7 +236,7 @@ describe('parseOrString / rebuildOrString round-trip', () => { cond('email', 'eq', ENVELOPE), cond('id', 'eq', '7'), ]) - const parsed = parseOrString(s) + const parsed = parseConditions(s) expect(parsed).toHaveLength(2) expect(parsed[0].value).toBe(ENVELOPE) expect(parsed[1].value).toBe('7') @@ -238,7 +258,7 @@ describe('parseOrString / rebuildOrString round-trip', () => { const s = rebuildOrString( conditions.map((c) => cond(c.column, c.op, c.value)), ) - expect(parseOrString(s)).toEqual(conditions) + expect(parseConditions(s)).toEqual(conditions) }) }) @@ -259,23 +279,23 @@ describe('parseOrString / rebuildOrString round-trip', () => { // entirely — a filter that silently matches the wrong rows. Only or-strings // that also reference an encrypted column are rebuilt from the parse, so this // corrupts precisely the mixed encrypted/plaintext case. -describe('parseOrString containment literals', () => { +describe('parseOrStringWithSpans containment literals', () => { it('does not split on a comma inside an array literal', () => { - expect(parseOrString('note.eq.hello,tags.cs.{vip,admin}')).toEqual([ + expect(parseConditions('note.eq.hello,tags.cs.{vip,admin}')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'hello' }, { column: 'tags', op: 'cs', negate: false, value: '{vip,admin}' }, ]) }) it('does not split on a comma inside a jsonb literal', () => { - expect(parseOrString('meta.cs.{"a":1,"b":2},note.eq.x')).toEqual([ + expect(parseConditions('meta.cs.{"a":1,"b":2},note.eq.x')).toEqual([ { column: 'meta', op: 'cs', negate: false, value: '{"a":1,"b":2}' }, { column: 'note', op: 'eq', negate: false, value: 'x' }, ]) }) it('round-trips a plaintext containment literal through rebuild', () => { - const parsed = parseOrString('tags.cs.{vip,admin}') + const parsed = parseOrStringWithSpans('tags.cs.{vip,admin}') expect(rebuildOrString(parsed as DbPendingOrCondition[])).toBe( 'tags.cs."{vip,admin}"', ) @@ -290,16 +310,16 @@ describe('parseOrString containment literals', () => { // into the preceding operand — and only or-strings that also reference an // encrypted column are rebuilt from the parse, so it corrupts precisely the // mixed encrypted/plaintext case. -describe('parseOrString structural characters inside values', () => { +describe('parseOrStringWithSpans structural characters inside values', () => { it('does not close an array literal on a brace inside a quoted element', () => { - expect(parseOrString('tags.cs.{"a}b"},email.eq.secret')).toEqual([ + expect(parseConditions('tags.cs.{"a}b"},email.eq.secret')).toEqual([ { column: 'tags', op: 'cs', negate: false, value: '{"a}b"}' }, { column: 'email', op: 'eq', negate: false, value: 'secret' }, ]) }) it('does not close a jsonb literal on a brace inside a quoted value', () => { - expect(parseOrString('meta.cs.{"a":"v}"},id.eq.1')).toEqual([ + expect(parseConditions('meta.cs.{"a":"v}"},id.eq.1')).toEqual([ { column: 'meta', op: 'cs', negate: false, value: '{"a":"v}"}' }, { column: 'id', op: 'eq', negate: false, value: '1' }, ]) @@ -316,7 +336,7 @@ describe('parseOrString structural characters inside values', () => { '(', ])('keeps a quoted %s inside a jsonb literal out of the depth count', (char) => { expect( - parseOrString(`email.eq.x,meta.cs.{"a":"${char}"},note.eq.y`), + parseConditions(`email.eq.x,meta.cs.{"a":"${char}"},note.eq.y`), ).toEqual([ { column: 'email', op: 'eq', negate: false, value: 'x' }, { column: 'meta', op: 'cs', negate: false, value: `{"a":"${char}"}` }, @@ -326,20 +346,22 @@ describe('parseOrString structural characters inside values', () => { it('keeps an escaped quote inside a jsonb value opaque', () => { // `\"` must not close the element, or the `}` behind it decrements depth. - expect(parseOrString('a.eq.1,meta.cs.{"a":"\\"}"},b.eq.2')).toHaveLength(3) + expect(parseConditions('a.eq.1,meta.cs.{"a":"\\"}"},b.eq.2')).toHaveLength( + 3, + ) }) it('splits after an unmatched brace in an unquoted value', () => { // `}` is not a PostgREST reserved character, so `a}b` is a valid unquoted // scalar operand. - expect(parseOrString('nickname.eq.a}b,id.eq.1')).toEqual([ + expect(parseConditions('nickname.eq.a}b,id.eq.1')).toEqual([ { column: 'nickname', op: 'eq', negate: false, value: 'a}b' }, { column: 'id', op: 'eq', negate: false, value: '1' }, ]) }) it('splits after an unmatched paren in an unquoted value', () => { - expect(parseOrString('a.eq.x),b.eq.y')).toEqual([ + expect(parseConditions('a.eq.x),b.eq.y')).toEqual([ { column: 'a', op: 'eq', negate: false, value: 'x)' }, { column: 'b', op: 'eq', negate: false, value: 'y' }, ]) @@ -352,14 +374,14 @@ describe('parseOrString structural characters inside values', () => { // then forwarded VERBATIM (nothing looks encrypted), so PostgREST runs the // swallowed `email.eq.ada` with a plaintext operand against a ciphertext column. it('splits after an unmatched opening brace in an unquoted value', () => { - expect(parseOrString('note.eq.a{b,email.eq.ada')).toEqual([ + expect(parseConditions('note.eq.a{b,email.eq.ada')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'a{b' }, { column: 'email', op: 'eq', negate: false, value: 'ada' }, ]) }) it('splits after an unmatched opening paren in an unquoted value', () => { - expect(parseOrString('note.eq.a(b,email.eq.ada')).toEqual([ + expect(parseConditions('note.eq.a(b,email.eq.ada')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'a(b' }, { column: 'email', op: 'eq', negate: false, value: 'ada' }, ]) @@ -368,31 +390,32 @@ describe('parseOrString structural characters inside values', () => { // A stray opener must not cost the or-string its REAL containment literals. // Discarding the depth pass wholesale on an unbalanced count re-splits inside // `{vip,admin}`, and the dotless `admin}` fragment is then dropped by - // `parseOrString` — the same silent condition loss, moved one operand along. + // `parseOrStringWithSpans` — the same silent condition loss, moved one + // operand along. // A structural brace opens a group or an operand; anywhere else it is data. it('keeps a sibling array literal intact past a stray opening brace', () => { - expect(parseOrString('note.eq.a{b,tags.cs.{vip,admin}')).toEqual([ + expect(parseConditions('note.eq.a{b,tags.cs.{vip,admin}')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'a{b' }, { column: 'tags', op: 'cs', negate: false, value: '{vip,admin}' }, ]) }) it('keeps an array literal intact when the stray opener follows it', () => { - expect(parseOrString('tags.cs.{vip,admin},note.eq.a{b')).toEqual([ + expect(parseConditions('tags.cs.{vip,admin},note.eq.a{b')).toEqual([ { column: 'tags', op: 'cs', negate: false, value: '{vip,admin}' }, { column: 'note', op: 'eq', negate: false, value: 'a{b' }, ]) }) it('keeps a sibling jsonb literal intact past a stray opening brace', () => { - expect(parseOrString('note.eq.a{b,meta.cs.{"a":1,"b":2}')).toEqual([ + expect(parseConditions('note.eq.a{b,meta.cs.{"a":1,"b":2}')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'a{b' }, { column: 'meta', op: 'cs', negate: false, value: '{"a":1,"b":2}' }, ]) }) it('keeps a sibling array literal intact past a stray opening paren', () => { - expect(parseOrString('note.eq.a(b,tags.cs.{vip,admin}')).toEqual([ + expect(parseConditions('note.eq.a(b,tags.cs.{vip,admin}')).toEqual([ { column: 'note', op: 'eq', negate: false, value: 'a(b' }, { column: 'tags', op: 'cs', negate: false, value: '{vip,admin}' }, ]) @@ -402,19 +425,244 @@ describe('parseOrString structural characters inside values', () => { // scalar carrying an in-value dot still fools it. The unbalanced-depth // re-split is what recovers this one; both mechanisms are load-bearing. it('recovers a scalar whose brace follows an in-value dot', () => { - expect(parseOrString('x.eq.a.{b,y.eq.1')).toEqual([ + expect(parseConditions('x.eq.a.{b,y.eq.1')).toEqual([ { column: 'x', op: 'eq', negate: false, value: 'a.{b' }, { column: 'y', op: 'eq', negate: false, value: '1' }, ]) }) - it('treats and/or group parens as structure', () => { - expect(parseOrString('and(a.eq.1,b.eq.2),c.eq.3')).toHaveLength(2) - expect(parseOrString('not.and(a.eq.1,b.eq.2),c.eq.3')).toHaveLength(2) - expect(parseOrString('and(a.eq.1,or(b.eq.2,c.eq.3)),d.eq.4')).toHaveLength( - 2, + // A logic group is STRUCTURE, and its body is recursed into: every leaf comes + // back flat, in source order, each carrying the span it occupies in the + // caller's original string. + // + // The alternative — treating `and(a.eq.1,b.eq.2)` as one pseudo-condition on a + // column literally named `and(a` — is the disclosure this PR fixes. Such a + // condition matches no encrypted column, so nothing in the group looked + // encrypted, and the whole `.or()` was forwarded VERBATIM: an encrypted + // column inside the group was compared against a PLAINTEXT operand on the + // wire. Assert the leaves, not just the count, so a regression that flattens + // to the right number of wrong conditions cannot pass. + it('flattens an and() group to its leaf conditions', () => { + expect(parseConditions('and(a.eq.1,b.eq.2),c.eq.3')).toEqual([ + { column: 'a', op: 'eq', negate: false, value: '1' }, + { column: 'b', op: 'eq', negate: false, value: '2' }, + { column: 'c', op: 'eq', negate: false, value: '3' }, + ]) + }) + + it('flattens a not.and() group to its leaf conditions', () => { + // The `not.` belongs to the GROUP, not to any leaf: negation of the group is + // preserved by leaving the original text in place (see + // `substituteOrStringLeaves`), so no leaf comes back with `negate: true`. + expect(parseConditions('not.and(a.eq.1,b.eq.2),c.eq.3')).toEqual([ + { column: 'a', op: 'eq', negate: false, value: '1' }, + { column: 'b', op: 'eq', negate: false, value: '2' }, + { column: 'c', op: 'eq', negate: false, value: '3' }, + ]) + }) + + it('flattens an or() group nested inside an and() group', () => { + expect(parseConditions('and(a.eq.1,or(b.eq.2,c.eq.3)),d.eq.4')).toEqual([ + { column: 'a', op: 'eq', negate: false, value: '1' }, + { column: 'b', op: 'eq', negate: false, value: '2' }, + { column: 'c', op: 'eq', negate: false, value: '3' }, + { column: 'd', op: 'eq', negate: false, value: '4' }, + ]) + }) +}) + +// --------------------------------------------------------------------------- +// Source spans +// +// Flattening a group is only half the fix: the leaves come back detached from +// their nesting, so the ONLY thing that can put an encrypted operand back where +// it belongs is `sourceSpan`. Every span is an offset into the caller's +// ORIGINAL string, across group recursion, containment literals whose commas +// and braces are not delimiters, and whitespace the parser trims. +// +// A span that is off by even one character does not throw — it splices +// ciphertext into the middle of a neighbouring condition, producing an or-string +// PostgREST either rejects or, worse, silently reads as a different filter. +// --------------------------------------------------------------------------- + +/** + * Every leaf's span resolved back through `input.slice(start, end)` — the + * property that actually matters, since that is exactly the slice + * {@link substituteOrStringLeaves} overwrites. + */ +function spanTexts(orString: string): string[] { + return parseOrStringWithSpans(orString).map((condition) => { + const span = condition.sourceSpan + if (!span) { + throw new Error(`leaf ${condition.column}.${condition.op} has no span`) + } + return orString.slice(span.start, span.end) + }) +} + +/** The raw spans, for the assertions that pin exact offsets. */ +function spans(orString: string) { + return parseOrStringWithSpans(orString).map((c) => c.sourceSpan) +} + +describe('parseOrStringWithSpans source spans', () => { + it('spans each top-level leaf, and nothing of the delimiter', () => { + const input = 'email.eq.ada,note.eq.x' + expect(spans(input)).toEqual([ + { start: 0, end: 12 }, + { start: 13, end: 22 }, + ]) + expect(spanTexts(input)).toEqual(['email.eq.ada', 'note.eq.x']) + }) + + it('spans a leaf one group deep against the ORIGINAL string', () => { + // The recursion re-parses the group BODY, whose own offsets start at zero. + // The `open + 1` base offset is what translates them back into the original + // string; without it every inner span is short by the group's opener. + const input = 'and(a.eq.1,b.eq.2),c.eq.3' + expect(spans(input)).toEqual([ + { start: 4, end: 10 }, + { start: 11, end: 17 }, + { start: 19, end: 25 }, + ]) + expect(spanTexts(input)).toEqual(['a.eq.1', 'b.eq.2', 'c.eq.3']) + }) + + it('spans a leaf two groups deep, accumulating both base offsets', () => { + const input = 'and(a.eq.1,and(b.eq.2,or(c.eq.3,d.eq.4)))' + expect(spans(input)).toEqual([ + { start: 4, end: 10 }, + { start: 15, end: 21 }, + { start: 25, end: 31 }, + { start: 32, end: 38 }, + ]) + expect(spanTexts(input)).toEqual(['a.eq.1', 'b.eq.2', 'c.eq.3', 'd.eq.4']) + }) + + it('spans a leaf inside not.and(), counting the prefix', () => { + // The group regex matches `not.and(` as well as `and(`, so the opener is at + // index 7, not 3. Measuring from a hard-coded `and(` length would shift + // every leaf in a negated group by four characters. + const input = 'not.and(email.eq.ada,note.eq.x)' + expect(spans(input)).toEqual([ + { start: 8, end: 20 }, + { start: 21, end: 30 }, + ]) + expect(spanTexts(input)).toEqual(['email.eq.ada', 'note.eq.x']) + }) + + it('spans a leaf that follows an array containment literal', () => { + // `{vip,admin}` holds a comma that is NOT a delimiter. Splitting on it puts + // the following leaf's span inside the literal, so substitution would + // overwrite part of `{vip,admin}` rather than replace `email.eq.ada`. + const input = 'and(note.cs.{vip,admin},email.eq.ada)' + expect(spans(input)).toEqual([ + { start: 4, end: 23 }, + { start: 24, end: 36 }, + ]) + expect(spanTexts(input)).toEqual(['note.cs.{vip,admin}', 'email.eq.ada']) + }) + + it('spans a leaf that follows a jsonb containment literal', () => { + const input = 'and(meta.cs.{"a":1,"b":2},email.eq.ada)' + expect(spans(input)).toEqual([ + { start: 4, end: 25 }, + { start: 26, end: 38 }, + ]) + expect(spanTexts(input)).toEqual(['meta.cs.{"a":1,"b":2}', 'email.eq.ada']) + }) + + it('excludes the whitespace surrounding a leaf from its span', () => { + // The condition is parsed from the TRIMMED token, so the span must start + // after the leading spaces and stop before the trailing ones — otherwise + // substitution eats the caller's formatting, and a trailing-space span on + // the last leaf runs past a shorter replacement. + const input = ' email.eq.ada , note.eq.x ' + expect(spans(input)).toEqual([ + { start: 1, end: 13 }, + { start: 16, end: 25 }, + ]) + expect(spanTexts(input)).toEqual(['email.eq.ada', 'note.eq.x']) + }) + + it('gives two identical leaves distinct spans', () => { + // The token search runs from a moving cursor, not from index 0. Restarting + // it would give the second occurrence the first one's span, so both + // substitutions would rewrite the first leaf and the second would keep its + // plaintext operand. + const input = 'email.eq.ada,email.eq.ada' + expect(spans(input)).toEqual([ + { start: 0, end: 12 }, + { start: 13, end: 25 }, + ]) + expect(spanTexts(input)).toEqual(['email.eq.ada', 'email.eq.ada']) + }) +}) + +// --------------------------------------------------------------------------- +// substituteOrStringLeaves +// +// Replacements run RIGHT TO LEFT. Every span is an offset into the original +// string, so the moment one splice changes the string's length — and an +// encrypted operand is always far longer than the plaintext it replaces — +// every span to its right is stale. A left-to-right pass therefore splices the +// second replacement INSIDE the first one's ciphertext. +// --------------------------------------------------------------------------- + +describe('substituteOrStringLeaves', () => { + /** Re-value the parsed leaves, then splice. `map` keeps each leaf's span. */ + function substitute( + input: string, + revalue: (column: string, value: unknown) => unknown, + shouldReplace: (column: string) => boolean, + ) { + const conditions = parseOrStringWithSpans(input).map((c) => ({ + ...c, + value: revalue(c.column, c.value), + })) as DbPendingOrCondition[] + return substituteOrStringLeaves(input, conditions, (c) => + shouldReplace(c.column), + ) + } + + it('replaces leaves at two nesting depths, longest-first, in place', () => { + // Both replacements are LONGER than the operands they displace, and the + // top-level leaf sits at a higher offset than the grouped one — so a + // left-to-right pass, working from spans the first splice already + // invalidated, would write `CT-FOR-BOB…` into the middle of `CT-FOR-ADA…`. + // Distinct replacement values, so a splice landing on the wrong leaf shows + // up rather than cancelling out. + const input = 'and(email.eq.ada,note.eq.x),email.eq.bob' + expect( + substitute( + input, + (_column, value) => + value === 'ada' + ? 'CT-FOR-ADA-XXXXXXXXXXXXXXXXXXXX' + : value === 'bob' + ? 'CT-FOR-BOB-YYYYYYYYYYYYYYYYYYYY' + : value, + (column) => column === 'email', + ), + ).toBe( + 'and(email.eq.CT-FOR-ADA-XXXXXXXXXXXXXXXXXXXX,note.eq.x),email.eq.CT-FOR-BOB-YYYYYYYYYYYYYYYYYYYY', ) }) + + it('replaces leaves across three depths and leaves the rest byte-for-byte', () => { + // Depths 1, 2 and 0 in one expression, with the depth-2 `b` leaf skipped: + // the group syntax, the untouched leaf, and the delimiters must all survive + // verbatim — that byte-for-byte survival is the whole reason the adapter + // splices rather than rebuilding the expression from the flat leaves. + const input = 'and(a.eq.1,or(b.eq.2,c.eq.3)),d.eq.4' + expect( + substitute( + input, + (column, value) => (column === 'b' ? value : `${column}`.repeat(10)), + (column) => column !== 'b', + ), + ).toBe('and(a.eq.aaaaaaaaaa,or(b.eq.2,c.eq.cccccccccc)),d.eq.dddddddddd') + }) }) // An `in`-list element is quoted exactly like any other operand, so the list must @@ -422,26 +670,28 @@ describe('parseOrString structural characters inside values', () => { // string on every comma tore `("a,b",c)` into three fragments and left the quotes // embedded in them — on an encrypted column each fragment is encrypted as its own // term, so the intended element never matches. -describe('parseOrString in-list elements', () => { +describe('parseOrStringWithSpans in-list elements', () => { it('does not split on a comma inside a quoted element', () => { - expect(parseOrString('email.in.("a,b",c)')).toEqual([ + expect(parseConditions('email.in.("a,b",c)')).toEqual([ { column: 'email', op: 'in', negate: false, value: ['a,b', 'c'] }, ]) }) it('unescapes a quoted element', () => { - expect(parseOrString('a.in.("x\\"y",z)')).toEqual([ + expect(parseConditions('a.in.("x\\"y",z)')).toEqual([ { column: 'a', op: 'in', negate: false, value: ['x"y', 'z'] }, ]) }) it('round-trips a comma-bearing element through rebuild', () => { const s = 'name.in.("Doe, John",Smith)' - expect(rebuildOrString(parseOrString(s) as DbPendingOrCondition[])).toBe(s) + expect( + rebuildOrString(parseOrStringWithSpans(s) as DbPendingOrCondition[]), + ).toBe(s) }) it('round-trips an encrypted envelope element', () => { - const parsed = parseOrString( + const parsed = parseConditions( rebuildOrString([cond('email', 'in', [ENVELOPE, 'x'])]), ) expect(parsed).toEqual([ @@ -450,7 +700,7 @@ describe('parseOrString in-list elements', () => { }) it('splits a negated list on top-level commas only', () => { - expect(parseOrString('email.not.in.("a,b",c)')).toEqual([ + expect(parseConditions('email.not.in.("a,b",c)')).toEqual([ { column: 'email', op: 'in', negate: true, value: ['a,b', 'c'] }, ]) }) @@ -460,7 +710,7 @@ describe('parseOrString in-list elements', () => { // `(`: parsed as an array, an encrypted `eq` operand is encrypted as a JS array // rather than the intended string, and the filter matches nothing. it('does not read a parenthesized scalar as a list for a scalar operator', () => { - expect(parseOrString('email.eq.(foo)')).toEqual([ + expect(parseConditions('email.eq.(foo)')).toEqual([ { column: 'email', op: 'eq', negate: false, value: '(foo)' }, ]) }) @@ -477,10 +727,12 @@ describe('parseOrString in-list elements', () => { 'adj', ])('round-trips a paren-delimited %s operand', (op) => { const s = `period.${op}.(1,10)` - expect(parseOrString(s)).toEqual([ + expect(parseConditions(s)).toEqual([ { column: 'period', op, negate: false, value: ['1', '10'] }, ]) - expect(rebuildOrString(parseOrString(s) as DbPendingOrCondition[])).toBe(s) + expect( + rebuildOrString(parseOrStringWithSpans(s) as DbPendingOrCondition[]), + ).toBe(s) }) }) @@ -507,36 +759,36 @@ describe('rebuildOrString reserved words', () => { }) }) -describe('parseOrString negation', () => { +describe('parseOrStringWithSpans negation', () => { it('lifts a not. prefix off the operator', () => { - expect(parseOrString('nickname.not.eq.ada')).toEqual([ + expect(parseConditions('nickname.not.eq.ada')).toEqual([ { column: 'nickname', op: 'eq', negate: true, value: 'ada' }, ]) }) it('parses a negated in-list as a real list, not a literal string', () => { - expect(parseOrString('nickname.not.in.(ada,grace)')).toEqual([ + expect(parseConditions('nickname.not.in.(ada,grace)')).toEqual([ { column: 'nickname', op: 'in', negate: true, value: ['ada', 'grace'] }, ]) }) it('parses not.is.null', () => { - expect(parseOrString('email.not.is.null')).toEqual([ + expect(parseConditions('email.not.is.null')).toEqual([ { column: 'email', op: 'is', negate: true, value: null }, ]) }) it('leaves a non-negated condition unnegated', () => { - expect(parseOrString('nickname.in.(ada,grace)')).toEqual([ + expect(parseConditions('nickname.in.(ada,grace)')).toEqual([ { column: 'nickname', op: 'in', negate: false, value: ['ada', 'grace'] }, ]) }) it('does not mistake a column or value named "not" for the prefix', () => { - expect(parseOrString('not.eq.ada')).toEqual([ + expect(parseConditions('not.eq.ada')).toEqual([ { column: 'not', op: 'eq', negate: false, value: 'ada' }, ]) - expect(parseOrString('nickname.eq.not')).toEqual([ + expect(parseConditions('nickname.eq.not')).toEqual([ { column: 'nickname', op: 'eq', negate: false, value: 'not' }, ]) }) @@ -545,12 +797,12 @@ describe('parseOrString negation', () => { // `col.not.<value>` is malformed PostgREST. Consuming the prefix would leave // no operator, and the condition would be silently DROPPED from the or-string // — quietly widening the result set. Pass it through so PostgREST rejects it. - expect(parseOrString('nickname.not.ada')).toEqual([ + expect(parseConditions('nickname.not.ada')).toEqual([ { column: 'nickname', op: 'not', negate: false, value: 'ada' }, ]) expect( rebuildOrString( - parseOrString('nickname.not.ada') as DbPendingOrCondition[], + parseOrStringWithSpans('nickname.not.ada') as DbPendingOrCondition[], ), ).toBe('nickname.not.ada') }) @@ -564,7 +816,7 @@ describe('rebuildOrString negation', () => { }) it('round-trips a negated in-list through parse → rebuild', () => { - const parsed = parseOrString('nickname.not.in.(ada,grace)') + const parsed = parseOrStringWithSpans('nickname.not.in.(ada,grace)') expect(rebuildOrString(parsed as DbPendingOrCondition[])).toBe( 'nickname.not.in.(ada,grace)', ) diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index 3eb6e27ef..8f3a55240 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -195,10 +195,11 @@ export function mapFilterOpToQueryType(op: FilterOp): QueryTypeName { } /** - * Parse a Supabase `.or()` filter string into structured conditions. + * Parse a Supabase `.or()` filter string into structured conditions, flattening + * nested `and(...)` / `or(...)` groups and recording every leaf's source span. * * Input: `'email.eq.john@example.com,name.ilike.%john%'` - * Output: `[{ column: 'email', op: 'eq', negate: false, value: 'john@example.com' }, …]` + * Output: `[{ column: 'email', op: 'eq', negate: false, value: 'john@example.com', sourceSpan: … }, …]` * * PostgREST spells negation `column.not.<op>.<value>`. It is lifted onto its own * `negate` flag rather than left as the operator: the term collector keys the @@ -207,27 +208,21 @@ export function mapFilterOpToQueryType(op: FilterOp): QueryTypeName { * string `in.(a,b)` as a single plaintext — a filter that silently matched * nothing. Only a `not` in the OPERATOR position is a prefix; a column or value * of that name is untouched. + * + * The spans are what let {@link substituteOrStringLeaves} rewrite a leaf in + * place, so groups survive byte-for-byte instead of being rebuilt (and + * destroyed) from the flat condition array. */ -export function parseOrString(orString: string): PendingOrCondition[] { - return parseOrStringAt(orString, 0, false).map( - ({ sourceSpan: _, ...condition }) => condition, - ) -} - -/** Adapter-only parser that flattens nested groups and records every leaf's - * source span, allowing encrypted substitution without rebuilding the group. */ export function parseOrStringWithSpans(orString: string): PendingOrCondition[] { - return parseOrStringAt(orString, 0, true) + return parseOrStringAt(orString, 0) } /** Recursively flatten PostgREST logic groups while retaining each leaf's - * exact location in the caller's original expression. The adapter must - * encrypt leaves inside `and(...)` / `or(...)`, but rebuilding the whole - * expression from a flat condition array destroys those groups. */ + * exact location in the caller's original expression. `baseOffset` translates + * the group body's own coordinates back into that original string. */ function parseOrStringAt( orString: string, baseOffset: number, - recurseGroups: boolean, ): PendingOrCondition[] { const conditions: PendingOrCondition[] = [] const parts = splitTopLevel(orString) @@ -242,11 +237,9 @@ function parseOrStringAt( const sourceStart = baseOffset + partStart + leading const group = /^(?:not\.)?(?:and|or)\(([\s\S]*)\)$/.exec(trimmed) - if (recurseGroups && group) { + if (group) { const open = trimmed.indexOf('(') - conditions.push( - ...parseOrStringAt(group[1], sourceStart + open + 1, true), - ) + conditions.push(...parseOrStringAt(group[1], sourceStart + open + 1)) continue } @@ -399,12 +392,12 @@ const OR_GROUP_TOKEN = /^(?:not\.)?(?:and|or)$/ * opener. `depth !== 0` at the end proves the counting was fooled, so the pass * is discarded and the input re-split honouring quotes alone — a backstop, not * the primary mechanism. It must stay narrow: applied to an input whose braces - * WERE structure, it re-splits inside `{vip,admin}` and `parseOrString` then + * WERE structure, it re-splits inside `{vip,admin}` and `parseOrStringAt` then * drops the dotless `admin}` fragment. * * `trackDepth` is the recursion's own flag, never passed by callers. NEVER - * throws — `query-builder.ts` relies on `parseOrString` being total so that - * capability errors surface in filter order. + * throws — `query-builder.ts` relies on `parseOrStringWithSpans` being total so + * that capability errors surface in filter order. */ function splitTopLevel(input: string, trackDepth = true): string[] { const parts: string[] = [] diff --git a/packages/stack-supabase/src/query-dbspace.ts b/packages/stack-supabase/src/query-dbspace.ts index 42ae9cb9e..194ba6055 100644 --- a/packages/stack-supabase/src/query-dbspace.ts +++ b/packages/stack-supabase/src/query-dbspace.ts @@ -103,7 +103,8 @@ function mutationToDbSpace(m: MutationOp, columns: ColumnMap): DbMutationOp { * `buildAndExecuteQuery`) consumes only the branded result, so a column can * no longer reach PostgREST untranslated — that is a compile error. * - * Total: `filterColumnName`, `parseOrString`, and `resolveMutationOptions` + * Total: `filterColumnName`, `parseOrStringWithSpans`, and + * `resolveMutationOptions` * never throw, so this introduces no new early-throw point and cannot perturb * the order in which capability errors surface. * diff --git a/packages/stack-supabase/src/query-filters.ts b/packages/stack-supabase/src/query-filters.ts index 12d8ba14a..8eb545e0c 100644 --- a/packages/stack-supabase/src/query-filters.ts +++ b/packages/stack-supabase/src/query-filters.ts @@ -351,7 +351,7 @@ export function applyFilters( // Every condition names a plaintext column, whose property name IS // its DB name — nothing to map. Forward the caller's ORIGINAL string // byte-for-byte: relied on for nested `and()` and quoted values that - // `parseOrString`/`rebuildOrString` cannot round-trip. + // `parseOrStringWithSpans`/`rebuildOrString` cannot round-trip. q = q.or(of_.original as DbFilterString, { referencedTable: of_.referencedTable, }) From 2b0eeb2f7b68ad536fde5439e72d351df02627fe Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 14:11:16 +1000 Subject: [PATCH 122/123] test(stack-supabase): pin parseLikeNeedle's escape handling directly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `parseLikeNeedle` decides which `like`/`ilike` patterns can be delegated to encrypted fuzzy matching and what literal term the search receives, but its only coverage was two end-to-end cases in the builder suite, which pin the wire envelope rather than the parse. Its predecessor (`replace(/^%+/, '')` plus `pattern.includes('_')`) got the escaping wrong in both directions, so the rules are worth holding directly: 9 of the 22 new tests fail against that predecessor. Pins in particular that `hasUnsupportedWildcard` tracks UNESCAPED wildcards only — an escaped `\_` is a literal and must not reject the query — and that only unescaped leading/trailing `%` are strippable, so a literal `\%` at either end stops the strip instead of being eaten by it. A trailing lone backslash stays a literal backslash rather than throwing; Postgres rejects such a pattern outright, but this needle is an approximation and the plaintext `like` path never reaches here. Documented as deliberate in the JSDoc so it does not get "fixed" into a throw. Also corrects the throw's parenthetical, which still read `an internal "%" or any "_"` — escaped metacharacters are honoured now, so it names unescaped ones. --- .../__tests__/like-pattern.test.ts | 161 ++++++++++++++++++ packages/stack-supabase/src/like-pattern.ts | 5 + packages/stack-supabase/src/query-builder.ts | 14 +- 3 files changed, 174 insertions(+), 6 deletions(-) create mode 100644 packages/stack-supabase/__tests__/like-pattern.test.ts diff --git a/packages/stack-supabase/__tests__/like-pattern.test.ts b/packages/stack-supabase/__tests__/like-pattern.test.ts new file mode 100644 index 000000000..113a3ed21 --- /dev/null +++ b/packages/stack-supabase/__tests__/like-pattern.test.ts @@ -0,0 +1,161 @@ +import { describe, expect, it } from 'vitest' +import { parseLikeNeedle } from '../src/like-pattern' + +/** + * Direct unit coverage for `parseLikeNeedle`. + * + * The function is the whole of the like/ilike → matches() approximation: it + * decides which patterns are delegable and what literal term the encrypted + * fuzzy search actually receives. Its only other coverage is two end-to-end + * cases in `supabase-v3-builder.test.ts`, which exercise it through a builder + * and so pin the wire envelope rather than the parse. These tests pin the parse + * itself — in particular the escaping rules, which the naive predecessor + * (`replace(/^%+/, '').replace(/%+$/, '')` plus `pattern.includes('_')`) got + * wrong in both directions. + * + * Every pattern below is written as a JS string literal, so `\\` in the source + * is a single backslash in the pattern the parser sees. + */ + +type Case = { + /** What the row pins, and the test's name. */ + name: string + /** The SQL LIKE pattern, as a caller would pass it. */ + pattern: string + expected: { needle: string; hasUnsupportedWildcard: boolean } +} + +const CASES: Case[] = [ + { + name: 'strips the unescaped leading and trailing %', + pattern: '%abc%', + expected: { needle: 'abc', hasUnsupportedWildcard: false }, + }, + { + name: 'passes a wildcard-free pattern through untouched', + pattern: 'abc', + expected: { needle: 'abc', hasUnsupportedWildcard: false }, + }, + { + name: 'reports an interior % as unsupported, keeping it in the needle', + pattern: '%a%b%', + expected: { needle: 'a%b', hasUnsupportedWildcard: true }, + }, + { + name: 'reports an unescaped _ as unsupported', + pattern: '%a_b%', + expected: { needle: 'a_b', hasUnsupportedWildcard: true }, + }, + { + name: 'treats an escaped % as a literal percent in the needle', + // The 7-char pattern `%100\%%`: strippable wildcard, `100`, an escaped + // literal `%`, strippable wildcard. + pattern: '%100\\%%', + expected: { needle: '100%', hasUnsupportedWildcard: false }, + }, + { + name: 'treats an escaped _ as literal, without flagging it unsupported', + pattern: '%under\\_score%', + expected: { needle: 'under_score', hasUnsupportedWildcard: false }, + }, + { + name: 'reduces an all-wildcard pattern to an empty needle', + pattern: '%%', + expected: { needle: '', hasUnsupportedWildcard: false }, + }, + { + name: 'reduces an empty pattern to an empty needle', + pattern: '', + expected: { needle: '', hasUnsupportedWildcard: false }, + }, + { + name: 'keeps an escaped backslash as a literal backslash', + // `%a\\%` — the `\\` escapes a backslash, so the trailing `%` is still an + // unescaped, strippable wildcard. + pattern: '%a\\\\%', + expected: { needle: 'a\\', hasUnsupportedWildcard: false }, + }, +] + +describe('parseLikeNeedle', () => { + it.each(CASES)('$name', ({ pattern, expected }) => { + expect(parseLikeNeedle(pattern)).toEqual(expected) + }) + + /** + * Only LEADING and TRAILING `%` are strippable, and only UNESCAPED ones. + * + * Fuzzy matching is inherently unanchored, so a `%` at either end adds + * nothing and can be dropped. An ESCAPED `\%` in the same position is not a + * wildcard at all — it is a literal percent the user is searching for — so it + * must stop the strip rather than be eaten by it. The old + * `replace(/^%+/, '')` / `replace(/%+$/, '')` pair could not tell the two + * apart and silently deleted the literal. + */ + describe('strips only unescaped leading/trailing %', () => { + it('stops stripping at an escaped % on either end', () => { + expect(parseLikeNeedle('\\%abc\\%')).toEqual({ + needle: '%abc%', + hasUnsupportedWildcard: false, + }) + }) + + it('keeps an escaped % that sits between two strippable ones', () => { + // `%\%%` — the outer two are wildcards, the middle is the literal. + expect(parseLikeNeedle('%\\%%')).toEqual({ + needle: '%', + hasUnsupportedWildcard: false, + }) + }) + + it('does not strip an interior %, and reports it instead', () => { + expect(parseLikeNeedle('a%b')).toEqual({ + needle: 'a%b', + hasUnsupportedWildcard: true, + }) + }) + }) + + /** + * A trailing lone backslash is kept as a literal backslash, deliberately. + * + * Postgres rejects such a pattern outright ("LIKE pattern must not end with + * escape character"), so we could throw here too. We don't: this needle is + * only ever an approximation feeding encrypted fuzzy matching, and the + * plaintext `like` path never reaches this function, so refusing would turn a + * database-level error into an earlier, less informative client-side one for + * no gain. Treating the dangling escape as a literal keeps the needle + * well-defined. This is pinned, not incidental — don't "fix" it into a throw. + */ + it('keeps a trailing lone backslash as a literal backslash', () => { + expect(parseLikeNeedle('abc\\')).toEqual({ + needle: 'abc\\', + hasUnsupportedWildcard: false, + }) + }) + + // The interaction that actually decides whether a caller's query is accepted + // or thrown out: `hasUnsupportedWildcard` must track UNESCAPED wildcards + // only. The predecessor tested `pattern.includes('_')`, so every one of the + // escaped cases below was rejected even though it is perfectly matchable. + describe('hasUnsupportedWildcard tracks unescaped wildcards only', () => { + it.each([ + ['\\_', 'a bare escaped underscore'], + ['%under\\_score%', 'an escaped underscore between strippable wildcards'], + ['a\\_b\\_c', 'several escaped underscores'], + ['a\\%b', 'an escaped interior percent'], + ])('stays false for %s (%s)', (pattern) => { + expect(parseLikeNeedle(pattern).hasUnsupportedWildcard).toBe(false) + }) + + it.each([ + ['_abc', 'leading'], + ['a_bc', 'interior'], + ['abc_', 'trailing'], + ['%a_c%', 'interior, inside strippable wildcards'], + ['\\_a_b', 'interior, alongside an escaped underscore'], + ])('goes true for an unescaped _ (%s: %s)', (pattern) => { + expect(parseLikeNeedle(pattern).hasUnsupportedWildcard).toBe(true) + }) + }) +}) diff --git a/packages/stack-supabase/src/like-pattern.ts b/packages/stack-supabase/src/like-pattern.ts index 51874fc2a..9853a71cf 100644 --- a/packages/stack-supabase/src/like-pattern.ts +++ b/packages/stack-supabase/src/like-pattern.ts @@ -9,6 +9,11 @@ export type LikeNeedle = { * Reduce a SQL LIKE pattern to the literal needle used by encrypted fuzzy * matching. Only unescaped leading/trailing `%` tokens are approximable; * escaped metacharacters remain literal and every other wildcard is reported. + * + * A trailing lone backslash (which Postgres itself rejects, "LIKE pattern must + * not end with escape character") is deliberately kept as a literal backslash + * rather than throwing: the needle is only an approximation feeding encrypted + * fuzzy matching, and the plaintext `like` path never reaches here. */ export function parseLikeNeedle(pattern: string): LikeNeedle { const tokens: LikeToken[] = [] diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index e583aa055..7bfbeaa57 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -257,9 +257,10 @@ export class EncryptedQueryBuilderImpl< * delegated to `matches`. EQL v3 free-text search is fuzzy bloom token * matching, not SQL pattern matching, so the result is APPROXIMATE — matching * is case-insensitive and one-sided (may false-positive), and anchoring is - * lost. Leading/trailing `%` are stripped; an internal `%` or any `_` cannot be - * approximated by trigram matching and throws. A plaintext column keeps real - * SQL LIKE. + * lost. Unescaped leading/trailing `%` are stripped; an unescaped internal `%` + * or any unescaped `_` cannot be approximated by trigram matching and throws. + * Escaped metacharacters (`\%`, `\_`) are literal search characters. A + * plaintext column keeps real SQL LIKE. */ like(column: string, pattern: string): this { if (!this.columns.isEncryptedV3Column(column)) { @@ -862,14 +863,15 @@ export class EncryptedQueryBuilderImpl< /** * Reduce a SQL LIKE pattern to a fuzzy-match needle, or throw when it cannot be * approximated. Strips surrounding `%` (prefix/suffix wildcards, which fuzzy - * matching subsumes); an internal `%` or any `_` is unapproximable. Warns once - * per (op, column) that the delegation is approximate. + * matching subsumes); an unescaped internal `%` or any unescaped `_` is + * unapproximable. Escaped metacharacters (`\%`, `\_`) are literals and pass + * through. Warns once per (op, column) that the delegation is approximate. */ private likeNeedle(column: string, op: string, pattern: string): string { const { needle, hasUnsupportedWildcard } = parseLikeNeedle(pattern) if (hasUnsupportedWildcard) { throw new Error( - `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an internal "%" or any "_"). Use matches("${column}", term) with a literal search term.`, + `[supabase v3]: "${op}" pattern "${pattern}" on encrypted column "${column}" has wildcards fuzzy free-text matching cannot honor (an unescaped internal "%" or any unescaped "_"). Use matches("${column}", term) with a literal search term.`, ) } const key = `${op}:${column}` From c62d23c45b19d92ccea4db36a2d5411a21c116dd Mon Sep 17 00:00:00 2001 From: Toby Hede <toby@cipherstash.com> Date: Wed, 29 Jul 2026 14:11:30 +1000 Subject: [PATCH 123/123] docs(stack-supabase,changeset): mark csv() deprecated and state the date-shape break MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things this release changes that a reader could only find at runtime. `csv()` throws on every encrypted query, but the interface member is still typed `Self` with no `@deprecated` tag, so an existing call site got no editor or compile signal and would discover the removal on its first request. Tag it; the `Self` return type stays, since it is a deliberate choice to keep the chain shape. `docs/reference/supabase-sdk.md` still advertised `.csv` among the transforms passed through to Supabase — the bundled skill was corrected in this PR, the reference was not. Date reconstruction now returns the raw value when a stored value does not parse. Before this branch the native typed client pushed every date-like column through `new Date(...)` unconditionally, so an unparseable value came back as an Invalid `Date`: `instanceof Date` held and `.getTime()` yielded `NaN`. It now comes back as the raw string and that call throws a `TypeError`, while the declared column type is still `Date`. The changeset said only "leaving invalid values unchanged", which undersells it. Both `stash-encryption` and `stash-drizzle` asserted flatly that `Date` columns are reconstructed to real `Date` instances — Drizzle decrypts through the same typed client, so both now carry the caveat. Bumps `@cipherstash/stack-supabase` and `@cipherstash/stack` from patch to minor. `csv()` and `matches(column, null)` throwing where they previously did something are behavioural breaks, and the date-shape change originates in `stack`; the migrate classifier changeset was raised for the same reason. The emitted version is unchanged under rc prerelease mode with a fixed group — this is signal, not version math. Also names the parser that actually runs in the nested-group bullet, since `parseOrString` no longer exists. --- .changeset/adapter-release-readiness.md | 36 +++++++++++++++---------- docs/reference/supabase-sdk.md | 4 ++- packages/stack-supabase/src/types.ts | 9 ++++--- skills/stash-drizzle/SKILL.md | 2 +- skills/stash-encryption/SKILL.md | 2 +- 5 files changed, 33 insertions(+), 20 deletions(-) diff --git a/.changeset/adapter-release-readiness.md b/.changeset/adapter-release-readiness.md index dc3540e00..f633bd8df 100644 --- a/.changeset/adapter-release-readiness.md +++ b/.changeset/adapter-release-readiness.md @@ -1,6 +1,6 @@ --- -'@cipherstash/stack': patch -'@cipherstash/stack-supabase': patch +'@cipherstash/stack': minor +'@cipherstash/stack-supabase': minor 'stash': patch '@cipherstash/wizard': patch --- @@ -8,17 +8,17 @@ Finish the EQL v2-removal release gates and adapter correctness pass. - **Supabase encrypts leaves nested inside a PostgREST boolean group.** This - is a disclosure fix, not a formatting one. `parseOrString` had no group - recursion, so `.or('and(createdAt.gte.2026-01-01,note.eq.x)')` came back from - the top-level split as one part and the leaf parser cut it at the first dot - into the pseudo-column `and(createdAt`. That name matched no encrypted - column, so the whole expression took the verbatim branch: the operand - `2026-01-01` reached PostgREST **as plaintext, against an encrypted column**, - under the JS property name `createdAt` rather than the DB column name - `created_at`. Every encrypted leaf nested inside `and(...)` / `or(...)` / - `not.and(...)` leaked its operand to the database and returned wrong results. - Nested groups and `referencedTable` are now preserved while each encrypted - leaf is substituted in place. + is a disclosure fix, not a formatting one. The `.or()` string parser had + no group recursion, so `.or('and(createdAt.gte.2026-01-01,note.eq.x)')` + came back from the top-level split as one part and the leaf parser cut it + at the first dot into the pseudo-column `and(createdAt`. That name matched + no encrypted column, so the whole expression took the verbatim branch: the + operand `2026-01-01` reached PostgREST **as plaintext, against an + encrypted column**, under the JS property name `createdAt` rather than the + DB column name `created_at`. Every encrypted leaf nested inside `and(...)` + / `or(...)` / `not.and(...)` leaked its operand to the database and + returned wrong results. Nested groups and `referencedTable` are now + preserved while each encrypted leaf is substituted in place. - Supabase never sends nullish encrypted search operands as plaintext, honours escaped LIKE metacharacters, rejects CSV result mode before decryption, and diagnoses the removed object-form factory call. The bundled `stash-supabase` @@ -27,7 +27,15 @@ Finish the EQL v2-removal release gates and adapter correctness pass. decrypted rows instead. - Native, WASM, and Supabase model decryption reconstruct valid date and timestamp values consistently, including nested paths, aliases, and bulk - results, while leaving invalid values unchanged. + results, while leaving invalid values unchanged. That last clause is a + behavioural change on the native typed client and the Supabase adapter, + which previously pushed every date-like column through `new Date(...)` + unconditionally: a stored value that does not parse used to come back as an + Invalid `Date` and now comes back as the raw string, matching what the WASM + entry already did. The declared column type is still `Date`, so code that + assumed `instanceof Date` held for every date column — or called a `Date` + method on it unguarded, so that `.getTime()` used to yield `NaN` and now + throws a `TypeError` — has to handle the raw value. - `stash init` names the concrete `public.eql_v3_*` domain family and gives `public.eql_v3_text_search` as a valid Supabase example. - CLI and wizard skill selection stay in parity for every integration, diff --git a/docs/reference/supabase-sdk.md b/docs/reference/supabase-sdk.md index 220ea7734..a33b27e17 100644 --- a/docs/reference/supabase-sdk.md +++ b/docs/reference/supabase-sdk.md @@ -58,7 +58,9 @@ capabilities come from the introspected domains. The builder surface is `.select/.insert/.update/.upsert/.delete`, `.eq/.neq/.in/.is/.gt/.gte/.lt/.lte/.match/.or/.not/.filter`, -transforms (`.order/.limit/.range/.single/.maybeSingle/.csv/.abortSignal/.throwOnError`), +transforms (`.order/.limit/.range/.single/.maybeSingle/.abortSignal/.throwOnError` +— `.csv()` is declared but always throws, since PostgREST serializes rows +before the wrapper can decrypt them), plus `.withLockContext(lockContext)` and `.audit(config)`. For free-text search it exposes `.matches()` (fuzzy bloom token search) on encrypted columns, keeps `.contains()` for native (exact) containment on plaintext diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index a2500a2e6..e58cf74c9 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -699,9 +699,9 @@ export type DbPendingMatchFilter = { } /** Retains the caller's ORIGINAL text for the verbatim fallback (which must be - * forwarded byte-for-byte — `parseOrString`/`rebuildOrString` do not round-trip - * nested `and()` or quoted values) alongside the parsed DB-space conditions - * used by the encrypt-and-rebuild path. Parsing happens once, here. */ + * forwarded byte-for-byte — `parseOrStringWithSpans`/`rebuildOrString` do not + * round-trip nested `and()` or quoted values) alongside the parsed DB-space + * conditions used by the encrypt-and-rebuild path. Parsing happens once, here. */ export type DbPendingOrFilter = | { kind: 'structured' @@ -976,6 +976,9 @@ export interface EncryptedQueryBuilderCore< * is a loud runtime failure rather than a missing method, and typed `Self` * only to keep the chain shape — it never returns. Select rows normally and * serialize the decrypted data yourself. + * + * @deprecated Always throws on an encrypted query — select the rows normally, + * then serialize the decrypted data to CSV yourself. */ csv(): Self abortSignal(signal: AbortSignal): Self diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index d9d03b22e..2d540b68c 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -370,7 +370,7 @@ if (!decrypted.failure) { } ``` -`Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. Non-schema fields pass through unchanged. +`Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. Non-schema fields pass through unchanged. Drizzle decrypts through the same typed client as every other integration, so the one caveat applies here too: a stored value that does **not** parse as a date is handed back unchanged rather than as an `Invalid Date`, even though the declared type is `Date` — guard with `instanceof Date` before calling `Date` methods on a column whose stored values you don't control. ## Indexing Encrypted Columns diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index e8016b013..3fbb0bd3e 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -391,7 +391,7 @@ if (!enc.failure) { Typed-client model notes: - `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a chainable `AuditableDecryptModelOperation` — await it for the `Result`, or chain `.audit({ metadata })` / `.withLockContext(lockContext)` first. A lock context may instead be passed as the optional third argument; use one form or the other, not both (chaining `.withLockContext()` onto a decrypt that already took a positional lock context throws). -- `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. +- `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. A stored value that does **not** parse as a date (a legacy non-ISO string, say) is handed back unchanged rather than as an `Invalid Date`, even though the declared type is `Date` — guard with `instanceof Date` before calling `Date` methods on a column whose stored values you don't control. - Nullable schema fields stay nullable through the round trip. ## Bulk Operations