feat: add tokenization API with format-preserving token workflows#53
Merged
feat: add tokenization API with format-preserving token workflows#53
Conversation
Implements format-preserving tokenization service with UUID, numeric, luhn-preserving, and alphanumeric token formats. Supports deterministic and non-deterministic token generation, token lifecycle management (revoke, validate, expire), and dual database persistence (PostgreSQL/MySQL). Major changes: - New tokenization domain layer with FormatType, Token, and TokenizationKey models - Tokenization use cases with metrics decorators for observability - Token generator factory with format-specific implementations (UUID, numeric, Luhn, alphanumeric) - MySQL and PostgreSQL repository implementations for tokenization persistence - HTTP handlers with DTO validation for key management and token operations - CLI commands: create-tokenization-key, rotate-tokenization-key, clean-expired-tokens - Database migrations: 000002_add_tokenization (up/down for both PostgreSQL and MySQL) API endpoints: - POST /v1/tokenization/keys (create key with format + algorithm) - POST /v1/tokenization/keys/:name/rotate (version increment) - DELETE /v1/tokenization/keys/:id (soft delete) - POST /v1/tokenization/keys/:name/tokenize (generate token with optional TTL) - POST /v1/tokenization/detokenize (retrieve plaintext) - POST /v1/tokenization/validate (check token validity) - POST /v1/tokenization/revoke (mark token as revoked) Capability mapping: encrypt (tokenize), decrypt (detokenize), read (validate), delete (revoke/delete key), write (create key), rotate (rotate key version). Documentation updates: - Added docs/api/tokenization.md with endpoint contracts and security notes - Added docs/api/capability-matrix.md as canonical capability reference - Added docs/releases/v0.4.0.md with upgrade checklist and migration rollback guidance - Added docs/operations/policy-smoke-tests.md for capability verification runbook - Added docs/metadata.json for centralized release/API version tracking - Added docs/tools/check_docs_metadata.py consistency checker - Updated smoke test script/docs with tokenization round-trip validation - Expanded monitoring operations guide with tokenization metrics - Updated README with v0.4.0 feature highlights and pinned Docker image version - Cross-linked capability matrix from API endpoint docs and policy cookbook CI/CD enhancements: - Added API/docs consistency guard for PRs (enforces doc updates for API changes) - Added docs metadata check validation in CI workflow - Integrated check_docs_metadata.py into make docs-lint Testing: - Comprehensive test coverage for domain, service, use case, repository, and HTTP layers - Integration tests using real PostgreSQL and MySQL test containers - Handler tests with Gin test context and mock use cases - DTO validation tests for request/response contracts Breaking changes: None (additive migration, new endpoints under /v1/tokenization). Rollback note: Down migration available; safe to roll back before production tokenization usage.
Addresses 12 gosec lint warnings by adding appropriate nolint directives for false positives and refactoring one type conversion for clarity. Changes by category: Test fixture false positives (9 files, G101): - Add `//nolint:gosec // test fixture data` to test database credentials and token hash strings in repository and use case tests - Files: internal/app/di_test.go, internal/testutil/database.go, internal/tokenization/service/*_test.go, internal/auth/repository/*_token_repository_test.go, internal/auth/usecase/token_usecase_test.go Domain model fields (4 files, G117): - Add `//nolint:gosec` to legitimate Secret/ClientSecret fields used in authentication flows (hashed secrets, not plaintext credentials) - Files: internal/auth/domain/client.go, internal/auth/domain/token.go, internal/auth/http/dto/request.go, internal/auth/http/dto/response.go Integer conversion safety (2 files, G115): - numeric_generator.go: Add nolint comment (conversion bounded 0-9 by design) - rules.go: Replace `string(rune(p.MinLength+48))` with `strconv.Itoa(p.MinLength)` for clearer intent and import strconv package HTTP test SSRF false positive (1 file, G704): - Add `//nolint:gosec // controlled test environment with localhost URLs` to integration test HTTP client - File: test/integration/api_test.go All lint warnings are legitimate false positives or have been refactored for improved clarity. No actual security vulnerabilities were present. Verification: `make lint` now passes with 0 issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements format-preserving tokenization service with UUID, numeric, luhn-preserving, and alphanumeric token formats. Supports deterministic and non-deterministic token generation, token lifecycle management (revoke, validate, expire), and dual database persistence (PostgreSQL/MySQL).
Major changes:
API endpoints:
Capability mapping: encrypt (tokenize), decrypt (detokenize), read (validate), delete (revoke/delete key), write (create key), rotate (rotate key version).
Documentation updates:
CI/CD enhancements:
Testing:
Breaking changes: None (additive migration, new endpoints under /v1/tokenization).
Rollback note: Down migration available; safe to roll back before production tokenization usage.