From 368a9fd55bcfa327ddb9556e599c6b602968e04b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 7 Jun 2026 13:15:39 +0000 Subject: [PATCH 1/2] docs: add CLAUDE.md updates for 5 active repos (June 2026) Review and update CLAUDE.md files for the 5 most active resonate repositories based on merged PRs from contributors: SayaliPat, shrivastavakapil2000, JoeVsVolcano, mike-brant, and nathan-resonate. Repos covered: - step-function-workflow-orchestrator (updated): EMR7 upgrades, cookiejar deprecation - batch-expression-modeling (updated): BlockGraph vendor support (CDP-118913) - resonate-terraform (new): monitoring/alerting stack overhaul (DE-13849) - batch-audience-delivery-syndication (new): BlockGraph/FreeWheel file syndication - identity-graph (new): prism_dbt v1.0, NAME_ADDRESS UDF, CI workflows https://claude.ai/code/session_011tgujgCjwfxgGcDx6XNnH4 --- claude-md-updates/README.md | 56 ++ .../CLAUDE.md | 128 +++++ .../batch-expression-modeling/CLAUDE.md | 486 ++++++++++++++++++ claude-md-updates/identity-graph/CLAUDE.md | 221 ++++++++ .../resonate-terraform/CLAUDE.md | 114 ++++ .../CLAUDE.md | 185 +++++++ 6 files changed, 1190 insertions(+) create mode 100644 claude-md-updates/README.md create mode 100644 claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md create mode 100644 claude-md-updates/batch-expression-modeling/CLAUDE.md create mode 100644 claude-md-updates/identity-graph/CLAUDE.md create mode 100644 claude-md-updates/resonate-terraform/CLAUDE.md create mode 100644 claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md diff --git a/claude-md-updates/README.md b/claude-md-updates/README.md new file mode 100644 index 0000000..870c0b6 --- /dev/null +++ b/claude-md-updates/README.md @@ -0,0 +1,56 @@ +# CLAUDE.md Updates + +This directory contains updated `CLAUDE.md` files for the 5 most active Resonate repositories (based on merged PRs from June 2025–June 2026 by contributors: SayaliPat, shrivastavakapil2000, JoeVsVolcano, PallaviJagarlamudi, mike-brant, nathan-resonate). + +## Repositories + +| Repository | Status | PR Count | Key Recent Changes | +|---|---|---|---| +| `resonate/step-function-workflow-orchestrator` | **Updated** | 42 PRs | EMR7 upgrades for 5 pipelines; cookiejar-sample-export decommission | +| `resonate/batch-expression-modeling` | **Updated** | 8 PRs | BlockGraph vendor support (CDP-118913); batch stitch throttle fix | +| `resonate/resonate-terraform` | **New** | 6 PRs | Monitoring/alerting overhaul (DE-13849); Nagios replacement with Slack+PD | +| `resonate/batch-audience-delivery-syndication` | **New** | 5 PRs | BlockGraph/FreeWheel file syndication Lambdas (T06–T08) | +| `resonate/identity-graph` | **New** | 4 PRs | prism_dbt v1.0 release; NAME_ADDRESS UDF + SP; CI workflows | + +## How to Apply + +These CLAUDE.md files need to be applied to their respective repositories. For each repo: + +1. **Repos with "Updated" status**: Replace the existing `CLAUDE.md` at the repo root. +2. **Repos with "New" status**: Create a new `CLAUDE.md` at the repo root. + +### Quick Apply Commands + +```bash +# step-function-workflow-orchestrator +gh api -X PUT repos/resonate/step-function-workflow-orchestrator/contents/CLAUDE.md \ + --field message="docs: update CLAUDE.md with EMR7 upgrades and cookiejar deprecation" \ + --field content="$(base64 -i step-function-workflow-orchestrator/CLAUDE.md)" + +# batch-expression-modeling +gh api -X PUT repos/resonate/batch-expression-modeling/contents/CLAUDE.md \ + --field message="docs: update CLAUDE.md with BlockGraph vendor support" \ + --field content="$(base64 -i batch-expression-modeling/CLAUDE.md)" + +# resonate-terraform (new file) +gh api -X PUT repos/resonate/resonate-terraform/contents/CLAUDE.md \ + --field message="docs: add CLAUDE.md" \ + --field content="$(base64 -i resonate-terraform/CLAUDE.md)" + +# batch-audience-delivery-syndication (new file) +gh api -X PUT repos/resonate/batch-audience-delivery-syndication/contents/CLAUDE.md \ + --field message="docs: add CLAUDE.md" \ + --field content="$(base64 -i batch-audience-delivery-syndication/CLAUDE.md)" + +# identity-graph (new file) +gh api -X PUT repos/resonate/identity-graph/contents/CLAUDE.md \ + --field message="docs: add CLAUDE.md with prism_dbt v1.0 documentation" \ + --field content="$(base64 -i identity-graph/CLAUDE.md)" +``` + +> **Note:** For repos with existing CLAUDE.md, you'll need to include the current file SHA in the PUT request. +> Run `gh api repos/resonate//contents/CLAUDE.md | jq .sha` to get the current SHA. + +## Generated By + +This update was generated by a Claude Code agent (claude.ai/code) on 2026-06-07, triggered by the presence of merged PRs from the specified contributor list. The agent reviewed recent commits, PR titles, and existing CLAUDE.md content to produce contextually accurate documentation. diff --git a/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md b/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md new file mode 100644 index 0000000..17d1fd9 --- /dev/null +++ b/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md @@ -0,0 +1,128 @@ +# batch-audience-delivery-syndication + +## Project Purpose and Architecture Overview + +This repository manages the **batch audience delivery syndication** workflows — Lambda functions and Terraform infrastructure for distributing processed audience data to third-party partners. The initial use case is **BlockGraph/FreeWheel** file syndication. + +The repo sits downstream of `batch-expression-modeling`: once BEM has evaluated audience expressions and produced output files, this repo's Lambda functions handle the final-mile delivery steps (rename files to partner conventions, generate taxonomy manifests, upload to partner SFTP/S3). + +**Top-level layout:** + +``` +workflows/ + lambdas/ + blockgraph-rename-files/ # Renames BEM output part-files to BlockGraph naming convention + blockgraph-create-taxonomy-file/ # Generates BlockGraph taxonomy CSV manifest + blockgraph-publish-files/ # Uploads files to BlockGraph (FreeWheel SFTP or S3) + step-functions/ # Step Function ASL definitions (if any) +terraform/ + lambdas/ + blockgraph-rename-files// terragrunt.hcl + blockgraph-create-taxonomy-file// terragrunt.hcl + blockgraph-publish-files// terragrunt.hcl +.github/ + workflows/ # GitHub Actions for CI/CD +``` + +--- + +## Key Lambda Functions + +### blockgraph-rename-files (CDP-118916) + +Renames BEM output `part-N.csv.gz` files to the naming convention expected by BlockGraph/FreeWheel. + +- Input: S3 prefix containing BEM output part files +- Output: Same S3 prefix with files renamed to partner convention +- Part-file regex is broad enough to match bare `part-N.csv.gz` patterns (not just `part-00000-*.csv.gz`) +- IAM: Least-privilege S3 permissions scoped to the delivery bucket + +### blockgraph-create-taxonomy-file (CDP-118915) + +Generates a BlockGraph taxonomy CSV manifest describing the audience segments delivered. + +- Output S3 location for taxonomy file uses `SPI=N` (confirmed by product — not sensitive personal information) +- Does **not** use `ExpectedBucketOwner` verification (dropped after product confirmation) +- Runs SonarQube-clean code (all findings resolved) + +### blockgraph-publish-files (CDP-118917) + +Uploads renamed files and the taxonomy manifest to BlockGraph (FreeWheel's receiving endpoint). + +- SSM Parameter Store is used for all secrets (SFTP credentials, API keys) — boto3 config follows standard SSM patterns +- Upload ordering: taxonomy file uploaded before data files (ensures manifest is present when partner processes data) +- IAM: Least-privilege — only the specific S3 paths and SSM parameters needed + +--- + +## Development + +### Running Lambda Tests + +Each Lambda has its own test suite: + +```bash +cd workflows/lambdas/ +pip install -r requirements.txt +python -m pytest tests/ +``` + +### AWS Authentication + +```bash +aws sso login +``` + +--- + +## Deployment + +All deployments go through **GitHub Actions** — do not run `terragrunt apply` locally. + +### Environments + +| Environment | Purpose | +|---|---| +| `dev` / `dev2`-`devN` | Development testing | +| `qa` | QA/staging environment | +| `prod` | Production | + +### Workflow Dispatch + +```bash +# Deploy a specific Lambda to an environment +gh workflow run lambdas.yml -f lambda=blockgraph-publish-files -f environment=qa --ref +``` + +### Terraform Module Source + +Lambda infrastructure uses the shared module from `resonate-terraform`: +``` +git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/lambda +``` + +--- + +## Project-Specific Rules and Gotchas + +- **SPI classification:** BlockGraph delivery files are classified `SPI=N` — confirmed with product. Do not add `ExpectedBucketOwner` checks unless this changes. +- **Upload ordering matters:** Taxonomy file must be uploaded before data files. The `blockgraph-publish-files` Lambda enforces this order — do not parallelize these uploads. +- **Part-file naming:** The rename regex matches bare `part-N.csv.gz` (not just Spark's `part-00000-uuid-*.csv.gz`). Test with actual BEM output when changing the regex. +- **SSM secrets:** All credentials are in Parameter Store under a `/blockgraph/` prefix. Keys are environment-specific — dev uses dev credentials, prod uses prod credentials. +- **Downstream of BEM:** This repo processes output from `batch-expression-modeling`. If BEM changes its output format or S3 path structure, update the input configuration here accordingly. + +--- + +## End-to-End BlockGraph Flow + +``` +batch-expression-modeling + └─> BEM evaluates expressions → Stitch joins with person_identity_graph_beta + └─> Output files in S3 (part-N.csv.gz) + └─> batch-audience-delivery-syndication + ├─> blockgraph-rename-files (rename to partner convention) + ├─> blockgraph-create-taxonomy-file (generate manifest) + └─> blockgraph-publish-files (upload to FreeWheel) +``` + +The step function orchestrating these three Lambdas lives in `step-function-workflow-orchestrator` (or this repo — check `workflows/step-functions/`). diff --git a/claude-md-updates/batch-expression-modeling/CLAUDE.md b/claude-md-updates/batch-expression-modeling/CLAUDE.md new file mode 100644 index 0000000..1e1bc61 --- /dev/null +++ b/claude-md-updates/batch-expression-modeling/CLAUDE.md @@ -0,0 +1,486 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +This repository contains the **Batch Expression Modeling (BEM)** system for Resonate's audience delivery platform. It evaluates audience expressions against user behavioral data at scale, processes the results through identity stitching, and formats the output for various downstream vendors (Meta, TikTok, TheTradeDesk, BlockGraph, etc.). + +### High-Level Architecture + +The system consists of three main components orchestrated by AWS Step Functions: + +1. **BEM (Batch Expression Modeling)** - Scala/Spark job that evaluates ~350 audience expressions against 2TB+ of SuperTagAv data +2. **Stitch** - Scala/Spark job that joins evaluated expressions with vendor identity tables (cookies, HEMs, device IDs) +3. **Formatter** - Scala/Spark job that formats stitched data for specific vendor requirements + +Each component runs on AWS EMR clusters and is configured/orchestrated by Lambda functions and Step Functions state machines. + +## Repository Structure + +``` +├── src/main/scala/com/resonate/ +│ ├── bem/ # Batch Expression Modeling (evaluates expressions against SuperTagAv) +│ ├── stitch/ # Identity stitching (joins RCID with vendor IDs) +│ ├── delivery/ # Delivery formatting and vendor-specific logic +│ └── utils/ # Shared utilities +├── workflows/ +│ ├── lambdas/ # Lambda functions for pipeline configuration +│ │ ├── batch-audience-delivery-config/ # Main config lambda (generates pipeline configuration) +│ │ ├── batch-audience-delivery-processing-config/ # Processing config lambda +│ │ ├── batch-vendor-stitch/ # Stitch config lambda +│ │ ├── bem-gen-job-parameters/ # BEM job parameters +│ │ └── ... +│ └── step-functions/ # Step function state machines (ASL JSON) +│ ├── batch-audience-delivery-processing/ # Main pipeline +│ ├── batch-audience-delivery-daily-pipeline/ # Daily orchestrator +│ ├── bem-pipeline/ # BEM execution +│ ├── batch-stitch-pipeline/ # Stitch execution +│ └── batch-delivery-formatter-pipeline/ # Formatter execution +├── terraform/ # Infrastructure as code (one directory per Lambda/Step Function) +├── integration-tests/ # End-to-end integration tests +└── build.sbt # SBT build configuration +``` + +## Common Development Tasks + +### Scala/Spark Development + +**Build and test:** +```bash +# Compile Scala code +./compile.sh # or: sbt compile + +# Run tests +sbt test + +# Run specific test +sbt "testOnly com.resonate.bem.BatchExpressionModelJobTest" + +# Package JAR for EMR deployment +./package.sh # or: sbt assembly +# Output: target/batch-expression-modeling.jar +``` + +**AWS authentication:** +```bash +# Required before uploading JARs to S3 or accessing AWS resources +aws sso login +``` + +### Python Lambda Development + +**Lambda functions are in `workflows/lambdas/`**. Each has its own `requirements.txt`. + +**Run tests for a Lambda:** +```bash +cd workflows/lambdas/ +python -m pytest tests/ +``` + +**Common Lambda functions:** +- `batch-audience-delivery-config` - Generates delivery configuration per vendor (including BlockGraph); queries RTP DB for schedule +- `batch-audience-delivery-processing-config` - Generates pipeline configuration from delivery schedule +- `bem-gen-job-parameters` - Generates EMR step parameters for BEM job +- `batch-vendor-stitch` - Generates stitch configuration per vendor + +### Integration Tests + +**Run full integration test:** +```bash +cd integration-tests + +# First time: install dependencies +pip install -r requirements.txt + +# Run setup + execution + validation +bash run_full_integration_test.sh +``` + +The integration test: +1. Sets up test data in S3 with datetime-based paths (e.g., `integration/data/20250914_143022/`) +2. Triggers the `aud-delivery-proc-integration` step function +3. Polls for completion and validates SUCCESS status + +**Cost:** ~$0.25-0.30 per run, ~15-25 minutes with minimal test data. + +### GitHub Actions Workflows + +**Workflow files (in `.github/workflows/`):** + +| Workflow File | Description | +|--------------|-------------| +| `bem-jar.yml` | Builds and publishes the Scala JAR to S3 | +| `bem-lambdas.yml` | Deploys Lambda functions | +| `bem-step-functions.yml` | Deploys Step Functions | +| `bem-unit-tests.yml` | Runs Scala unit tests | + +**Deploy to an environment:** + +⚠️ **CRITICAL: Always specify `--ref` when deploying from a feature branch!** + +Without `--ref`, `gh workflow run` deploys from the **default branch (main)**, NOT your current branch. This is a common mistake that results in deploying old code. + +```bash +# Deploy JAR (required input: environment) +# JAR goes to qa (integration environment uses qa JAR) +gh workflow run bem-jar.yml -f environment=qa --ref + +# Deploy Lambdas (required input: environment) +gh workflow run bem-lambdas.yml -f environment=integration --ref + +# Deploy Step Functions (required inputs: environment, deployment_type) +gh workflow run bem-step-functions.yml -f environment=integration -f deployment_type=All --ref +``` + +**Example deploying all components from a feature branch:** +```bash +BRANCH="feature/CDP-123456-my-feature" +gh workflow run bem-jar.yml -f environment=qa --ref $BRANCH +gh workflow run bem-lambdas.yml -f environment=integration --ref $BRANCH +gh workflow run bem-step-functions.yml -f environment=integration -f deployment_type=All --ref $BRANCH +``` + +**Step Functions deployment_type options:** +- `All` - Deploy all step functions +- `BEM Pipeline Step Function` +- `Batch Stitch Pipeline` +- `Batch Delivery Formatter Pipeline` +- `Audience Delivery Pipeline` +- `Audience Delivery Custom Delivery Pipeline` +- `Audience Delivery Processing Pipeline` +- `Audience Delivery Daily Pipeline` + +**Environment values:** +- `integration` - Integration testing environment (uses QA JAR) +- `dev`, `dev2`-`dev7` - Development environments +- `nonprod` - QA/staging environment +- `prod` - Production environment + +**Check deployment status:** +```bash +# List recent runs for a workflow +gh run list --workflow=bem-jar.yml --limit=3 +gh run list --workflow=bem-lambdas.yml --limit=3 +gh run list --workflow=bem-step-functions.yml --limit=3 + +# Watch a specific run +gh run watch +``` + +**Note:** The integration environment uses the QA JAR. When testing changes in integration: +1. Deploy JAR to `qa` +2. Deploy lambdas to `integration` +3. Deploy step functions to `integration` + +### Working with Step Functions + +Step functions are defined in `workflows/step-functions/*/statemachine/*.asl.json` and use **JSONata query language** (not JSONPath). + +**Key step functions:** +- `batch-audience-delivery-daily-pipeline` - Entry point triggered daily by EventBridge +- `batch-audience-delivery-processing` - Main pipeline orchestrating BEM → Stitch → Formatter +- `bem-pipeline` - EMR cluster creation → BEM job execution → cluster termination +- `batch-stitch-pipeline` - Parallel execution of stitch jobs per vendor +- `batch-delivery-formatter-pipeline` - Formatting for vendor-specific requirements + +## Key Concepts + +### Expression Evaluation (BEM) + +The BEM job (`BatchExpressionModelJob.scala`) evaluates audience expressions against SuperTagAv data: + +1. **Input:** + - `surveyexpressions.csv` - List of expressions to evaluate (e.g., `NOT (E999999998)`) + - SuperTagAv parquet data - User behavioral data with attribute bitmaps (evmap, avmap, cvmap) + - Models CSV - Expression evaluation models + +2. **Processing:** + - Reads SuperTagAv data (~2TB) with optimized schema (only needed bitmap columns) + - Evaluates each expression against each RCID's attribute bitmaps + - Uses `AudienceEvaluator` from `das-expression` library + - Outputs matches partitioned by `exprHash` + +3. **Output:** Parquet files with schema `(rcid, exprHash, score)` partitioned by `exprHash=` + +### Identity Stitching + +The Stitch job (`TableStitch.scala`) joins BEM output with vendor identity tables: + +1. **Input:** + - BEM cache (evaluated expressions with RCIDs) + - Vendor stitch tables (e.g., `tmid`, `hemsha2`, `viant_id` columns) + - Audience metadata (expression hash to audience ID mappings) + +2. **Processing:** + - Joins BEM cache with stitch table on `rcid` + - Maps `exprHash` to `audienceIds` and explodes + - Filters empty vendor keys + +3. **Output:** `(akey, vkey, score)` where `akey` = audience key, `vkey` = vendor key (e.g., cookie, email hash) + +### Delivery Configuration + +The `batch-audience-delivery-config` Lambda generates pipeline configuration: + +- Queries RTP database for delivery schedules (via `RTP_PSQL_CONNECTION_STRING`) +- Determines which vendors need delivery based on schedule windows +- Generates S3 paths for tagav data, cookiejar, expressions, and output locations +- Sets EMR cluster configurations (instance types, counts, etc.) + +**Important fields:** +- `delivery_type` - "daily", "training", "custom", "blockgraph_syndicated", "blockgraph_custom" +- `tagav_date_folder` - Date folder for SuperTagAv input (format: YYYYMMDD) +- `cookie_jar_date` - Date folder for cookiejar data +- `start_time_of_delivery` / `end_time_of_delivery` - Time window for vendor deliveries (epoch milliseconds) + +### BlockGraph Vendor (CDP-118913) + +BlockGraph is a regular vendor added in June 2026. Key specifics: +- **Stitch table:** `person_identity_graph_beta` (not the standard cookie/HEM stitch tables) +- **Stitch columns:** Uses normalized RID-address columns (`stitch_column` singular, not `stitch_columns` plural for single-column vendors) +- **Delivery types:** `blockgraph_syndicated` and `blockgraph_custom` — both configured via `batch-audience-delivery-config` Lambda +- **Downstream syndication:** After BEM/Stitch, BlockGraph output is picked up by the `batch-audience-delivery-syndication` repo for file renaming, taxonomy generation, and SFTP upload to FreeWheel + +## AWS Step Functions: Converting JSONPath to JSONata + +### Top-Level Changes + +Add `"QueryLanguage": "JSONata"` at the state machine or state level. + +### Field Replacements + +JSONPath's five fields are replaced with two in JSONata: +- **Arguments** - Replaces `Parameters` for sending data to integrated actions +- **Output** - Replaces `ResultPath`, `ResultSelector`, and `OutputPath` for transforming state output + +### JSONPath Expression Syntax + +- **JSONPath**: `"field.$": "$.path.to.value"` +- **JSONata**: `"field": "{% $states.input.path.to.value %}"` + +Remove the `.$` suffix and wrap expressions in `{% %}` delimiters. + +### ResultPath Conversions + +**Merge result into input at a path:** +```json +// JSONPath +"ResultPath": "$.ClusterCreationResult" + +// JSONata +"Output": "{% $merge([$states.input, {'ClusterCreationResult': $states.result}]) %}" +``` + +**Discard result, pass through input only:** +```json +// JSONPath +"ResultPath": null + +// JSONata +// Simply omit the Output field - input passes through by default +``` + +**Replace entire input with result (default):** +```json +// JSONPath +"ResultPath": "$" // or omit ResultPath + +// JSONata +// Omit Output field, or use: +"Output": "{% $states.result %}" +``` + +### OutputPath Conversions + +**Extract specific field from result:** +```json +// JSONPath +"OutputPath": "$.Payload" + +// JSONata +"Output": "{% $states.result.Payload %}" +``` + +### Catch Block Error Handling + +**Merge error into input:** +```json +// JSONPath +"Catch": [ + { + "ErrorEquals": ["States.ALL"], + "ResultPath": "$.Error", + "Next": "HandleError" + } +] + +// JSONata +"Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Output": "{% $merge([$states.input, {'Error': $states.errorOutput}]) %}", + "Next": "HandleError" + } +] +``` + +Note: `$states.errorOutput` is only available in Catch blocks. + +### Map State Conversions + +**ItemsPath:** +```json +// JSONPath +"ItemsPath": "$.step_configs" + +// JSONata +"Items": "{% $states.input.step_configs %}" +``` + +**ItemSelector:** +```json +// JSONPath +"ItemSelector": { + "step_config.$": "$$.Map.Item.Value", + "clusterId.$": "$.ClusterCreationResult.ClusterId" +} + +// JSONata +"ItemSelector": { + "step_config": "{% $states.context.Map.Item.Value %}", + "clusterId": "{% $states.input.ClusterCreationResult.ClusterId %}" +} +``` + +**IMPORTANT:** Use `$states.context.Map.Item.Value` to access the current item in a Map state. There is **NO** `$states.item` variable - this is a common mistake that will cause "Field '$states.item' does not exist" errors. + +**⚠️ CRITICAL: Map States Must Preserve Input** + +By default, a Map state outputs an **array of results** from processing each item, which **loses the original input fields**. If subsequent states need access to fields from the original input (like a ClusterId from cluster creation), you MUST add an Output field to preserve the input. + +```json +// WRONG - This loses ClusterCreationResult from input +"Step Map": { + "Type": "Map", + "ItemSelector": { + "ClusterId": "{% $states.input.ClusterCreationResult.ClusterId %}" + }, + "Items": "{% $states.input.step_configs %}", + "Next": "TerminateCluster" // ❌ Will fail - ClusterCreationResult is lost! +} + +// CORRECT - Preserve input for next state +"Step Map": { + "Type": "Map", + "ItemSelector": { + "ClusterId": "{% $states.input.ClusterCreationResult.ClusterId %}" + }, + "Items": "{% $states.input.step_configs %}", + "Output": "{% $states.input %}", // ✅ Preserves all input fields + "Next": "TerminateCluster" +} +``` + +### Array Construction + +**Intrinsic function to array literal:** +```json +// JSONPath +"Args.$": "States.Array('cmd', '--flag', $.input.value, '--opt', $.input.other)" + +// JSONata - Use regular JSON array with individual JSONata expressions +"Args": [ + "cmd", + "--flag", + "{% $states.input.value %}", + "--opt", + "{% $states.input.other %}" +] +``` + +### Reserved Variables in JSONata + +- `$states.input` - Original input to the current state +- `$states.result` - Result from API/sub-workflow (Task, Parallel, Map states) +- `$states.errorOutput` - Error output (only in Catch blocks) +- `$states.context` - Execution metadata (StartTime, task token, Map.Item.Value, etc.) + - `$states.context.Map.Item.Value` - Current item in Map state iteration + - `$states.context.Map.Item.Index` - Current iteration index in Map state + +### ⚠️ Common Pitfall: $states.item Does NOT Exist + +**Error:** `Field '$states.item' does not exist` + +**WRONG:** +```json +"ItemSelector": { + "item": "{% $states.item %}" // ❌ This will fail! +} +``` + +**CORRECT:** +```json +"ItemSelector": { + "item": "{% $states.context.Map.Item.Value %}" // ✅ Use this instead +} +``` + +Many online examples and AI suggestions incorrectly reference `$states.item`, but this variable does not exist in AWS Step Functions JSONata. Always use `$states.context.Map.Item.Value` to access the current Map iteration item. + +### Key Differences from JSONPath + +1. **No `.$` suffix** - Regular field names with JSONata expressions in `{% %}` +2. **No Path fields** - InputPath, ResultPath, OutputPath are replaced by Arguments and Output +3. **Assign vs Output** - `Assign` creates variables accessible in all future states; `Output` only affects the immediate next state +4. **Parallel processing** - Assign and Output are processed in parallel; variable assignments don't affect Output + +### Common Patterns + +**Pass through input unchanged:** +- Omit the Output field entirely, OR explicitly use `"Output": "{% $states.input %}"` + +**IMPORTANT:** Any Task state that needs to pass data to subsequent states should preserve the input explicitly. Without an Output field, the Task's result replaces the entire input, losing all previous data. + +**Add result as new field while preserving input:** +```jsonata +"Output": "{% $merge([$states.input, {'resultField': $states.result}]) %}" +``` + +**Transform and extract specific data:** +```jsonata +"Output": { + "field1": "{% $states.input.someValue %}", + "field2": "{% $states.result.someOtherValue %}" +} +``` + +**Create variables for use in later states:** +```jsonata +"Assign": { + "myVariable": "{% $states.input.someValue %}" +} +// Later states can reference {% $myVariable %} +``` + +### Reference Documentation + +- [AWS Step Functions - Transforming data with JSONata](https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html) +- [AWS Step Functions - Passing data between states with variables](https://docs.aws.amazon.com/step-functions/latest/dg/workflow-variables.html) + +--- + +## Recent Changes (as of June 2026) + +### BlockGraph Vendor Support (CDP-118913, June 2026) + +The `batch-audience-delivery-config` Lambda was extended to support BlockGraph as a fully generic vendor. Key implementation details: +- BlockGraph uses `person_identity_graph_beta` as its stitch table (not a standard cookie/HEM table) +- Single-column vendor: uses `stitch_column` (singular key) not `stitch_columns` (plural) +- Delivery types `blockgraph_syndicated` and `blockgraph_custom` are valid `delivery_type` values +- The `audience_bitmap_path` field is read from the Lambda event for BlockGraph runs +- The `person_jar` logical name + path is provided in the event (not hardcoded in the config Lambda) + +### Batch Stitch Throttle Fix (CDP-118972) + +A throttling issue in the batch stitch pipeline was fixed. If you observe stitch jobs failing with throttling-related errors in CloudWatch, check the retry/backoff configuration in the stitch step function ASL. diff --git a/claude-md-updates/identity-graph/CLAUDE.md b/claude-md-updates/identity-graph/CLAUDE.md new file mode 100644 index 0000000..746aeb3 --- /dev/null +++ b/claude-md-updates/identity-graph/CLAUDE.md @@ -0,0 +1,221 @@ +# identity-graph + +## Project Purpose and Architecture Overview + +This repository hosts the **PRISM Identity Graph** — Resonate's person identity resolution system. It maps customer-supplied identifiers (HEM, MAID, ZIP11, name+address, etc.) to a canonical `person_id`, enabling cross-channel identity stitching at scale. + +The core deliverable is **`prism_dbt`** — a Snowflake-native dbt package deployed via `CREATE DBT PROJECT` / `EXECUTE DBT PROJECT`. Consumers invoke it from Step Functions (via Lambda) or directly from Snowflake SQL. + +**Top-level layout:** + +``` +dbt/ + prism_dbt/ # The Snowflake dbt package (v1.0+) + macros/ # Core primitives (waterfall_match, identifier_expand, etc.) + models/ + service/ # Lambda-invokable service models + waterfall.sql # Main integration point for consumers + tests/ + unit/ # Offline dbt unit tests (no Snowflake needed) + *.sql # Singular tests (require live Snowflake mock graph) + dbt_project.yml + profiles.yml.example # Template for local dev +snowflake/ + function/ + name_address_hash/ # NAME_ADDRESS_HASH UDF (SHA-256 PII normalization) + stored_procedures/ + prism_lookup/ # PRISM_LOOKUP stored procedure + name_address_lookup/ # NAME_ADDRESS_LOOKUP stored procedure +.github/ + workflows/ + snowflake_dbt.yml # Deploys prism_dbt package to Snowflake + snowflake_stored_procedure.yml # Deploys PRISM_LOOKUP / NAME_ADDRESS_LOOKUP SPs + snowflake_function.yml # Deploys NAME_ADDRESS_HASH UDF +``` + +**Production JAR path (Scala identity graph):** +`s3://resonate-core-applications/identity-graph/jars/identity-graph-latest.jar` + +--- + +## Core Primitives + +| Primitive | Direction | Purpose | +|---|---|---| +| `prism_dbt.waterfall_match` | identifiers → ONE person_id per row | Resolve mixed-identifier input to a canonical `person_id` via priority-ordered waterfall (first-match-wins) | +| `prism_dbt.identifier_expand` | person_id → identifiers of ONE type | Fan out `person_id`s to deliverable identifiers (HEM, MAID, TTD, etc.) | +| `prism_dbt.persons_project` | person_id → persons attributes | Project `persons` columns (name, address, lalvoterid, …) onto `person_id`s | +| `prism_dbt.lookup` | one identifier → one person + linked IDs | Single-row debug lookup; also deployed as `PRISM_LOOKUP` stored procedure | +| `NAME_ADDRESS_LOOKUP` (SP) | PII tuple → one person | Resolves raw `(first_name, last_name, address_line, zip)` to a person record | + +**Picking the right primitive:** +- *"I have customer rows with mixed identifiers, give me one person_id each"* → `waterfall_match` +- *"I have person_ids, give me their RAMP_IDs (or TTDs, or CTV_IDs)"* → `identifier_expand` (one call per type) +- *"I have person_ids, give me name + address"* → `persons_project` +- *"I have one identifier and want to debug who it resolves to"* → `lookup` macro or `PRISM_LOOKUP` SP +- *"I have raw name+address and want the person_id"* → `NAME_ADDRESS_LOOKUP` SP + +--- + +## Common Development Tasks + +### Local Development (prism_dbt) + +```bash +cd dbt/prism_dbt + +# Install dependencies +dbt deps + +# Run against mock graph (requires ~/.dbt/profiles.yml — copy from profiles.yml.example) +dbt seed --profiles-dir +dbt run --select waterfall --profiles-dir +dbt test --profiles-dir +``` + +**Two profile files — don't confuse them:** +- `profiles.yml.example` → copy to `~/.dbt/profiles.yml` for local `dbt run` / `dbt test` (uses real SSO auth) +- `snowflake_profiles/profiles.yml` → stub for `snow dbt deploy` only (account/user = `'_'`). **Do not edit.** + +**Mock tables** live in `RESONATE.PRISM.*` on `resonate-dev`, provisioned from `.context/cdp-119018-mock-tables.sql`. + +### Running Tests + +```bash +dbt test # All tests (25 total: 16 unit + 9 singular) +dbt test --select test_type:unit # Unit tests only (offline — no Snowflake needed) +dbt test --select test_type:data # Singular tests only (require live Snowflake) +``` + +**Unit tests** (`tests/unit/*.yml`) use dbt's `unit_tests:` framework — mocked inline, no Snowflake required. Fast and deterministic. + +**Singular tests** (`tests/test_*.sql`) run against the live `RESONATE.PRISM.*` mock graph on `resonate-dev`. Used for scenarios that can't be mocked (UDF calls, stored procedures, `input_column_aliases` with custom column names). + +### Deploying to Snowflake + +All deployments go through GitHub Actions: + +| Workflow | What it does | +|---|---| +| `snowflake_dbt.yml` | Uploads `prism_dbt` via `snow dbt deploy`. Runs `dbt parse` first as fast-fail gate. | +| `snowflake_stored_procedure.yml` | Deploys `PRISM_LOOKUP` or `NAME_ADDRESS_LOOKUP` SPs with placeholder substitution. | +| `snowflake_function.yml` | Deploys the `NAME_ADDRESS_HASH` UDF with placeholder substitution. | + +--- + +## Key Concepts + +### `waterfall_match` Step Config + +Caller-supplied list of step dicts; first match wins per input row: + +| Key | Required | Description | +|---|---|---| +| `identifier_type` | yes | PRISM identifier type (`HEM_SHA256`, `MAID`, `ZIP11`, …) | +| `stitch_label` | yes | String emitted in `prism_matched_step` on match | +| `input_column` | no | Caller input column. Defaults: `HEM_TYPE`-gated for HEM variants; else `identifier_type` as column name | +| `input_type_column`, `input_type_value` | no | Gate the step on a sibling column value | +| `link_basis` | no | Crosswalk filter (`DIRECT`, `VIA_PII_MATCH`, `VIA_HEM`, etc.) | +| `output_value` | no | `'rid'` (default) or `'identifier_value'` | + +**ZIP11 is special:** It lives as a column on `persons`, NOT in `identifier_index`. `waterfall_match` routes ZIP11 lookups directly to `persons.zip11` automatically — no special caller config needed. + +### Default Waterfall Priority + +`prism_default_waterfall` (PRISM's recommended order, high-precision first): +HEM_MD5 → HEM_SHA1 → HEM_SHA256 → MAID → RAMP_ID → TTD → CTV_ID → RCID → L2_VOTER_ID → L2_CONSUMER_ID → ZIP11 + +**Intentionally excluded from default** (opt-in only): `IP`, `PHONE_CELL`, `PHONE_LANDLINE`, `NONID`, `HOUSEHOLD_ID`, `L2_FAMILY_ID`, `EXPERIAN_*`, `NAME_ADDRESS`. + +**`RID` is not published by PRISM** — it's a Resonate-internal bridge owned by Append's downstream pipeline. + +### NAME_ADDRESS Normalization (v1.0) + +UDF normalization recipe: `LOWER + TRIM + strip non-alphanumeric + LPAD zip + pipe-concat + SHA-256`. + +To enable NAME_ADDRESS in a waterfall: +```sql +SELECT + *, + {{ prism_dbt.name_address_hash('first_name', 'last_name', 'address_line', 'zip') }} AS NAME_ADDRESS +FROM normalized_input_raw +``` +Then add `{identifier_type: NAME_ADDRESS, stitch_label: name_address}` to `waterfall_order`. + +v1.0 does **not** do canonical-first-name lookup (Bill ≠ William) — v1.1 adds it. + +### `waterfall_match` Output Schema + +One row per input row (input pass-through + PRISM columns): +``` +prism_matched_step -- stitch_label of winning step (NULL if no match) +prism_matched_priority -- 1-based priority of winning step +prism_matched_identifier_type +prism_matched_identifier_value +prism_matched_person_id -- THE key output +prism_matched_output_mode +prism_consumer +prism_called_at +``` + +Unmatched rows are preserved with NULL `prism_matched_*` columns. + +### Invoking from a Step Function / Lambda + +```python +import json, snowflake.connector, os + +def handler(event, context): + args = { + "input_relation": event["input_relation"], + "consumer": event["consumer"], + "job_run_id": event["job_run_id"], + "waterfall_order": event["waterfall_order"], + "output_database": event["output_database"], + "output_schema": event["output_schema"], + "waterfall_output_alias": event["output_alias"], + } + sql = f""" + EXECUTE DBT PROJECT RESONATE.PRISM.prism_dbt + ARGS = $$run --select waterfall --vars '{json.dumps(args)}'$$ + """ + with snowflake.connector.connect( + account=os.environ["SF_ACCOUNT"], + user=os.environ["SF_USER"], + password=os.environ["SF_PASSWORD"], + warehouse="RESONATE_X3LARGE_WH", + role="ACCOUNTADMIN", + ) as conn: + with conn.cursor() as cur: + cur.execute(sql) +``` + +--- + +## Project-Specific Rules and Gotchas + +- **`CROSSWALK_LATEST` is NOT used by v1.0 macros.** The prod crosswalk has a 2-column schema missing metadata needed for confidence filtering. v1.0 uses `PERSONS_LATEST` and `IDENTIFIER_INDEX_LATEST` only. +- **`identifier_rank = 1` filter is enforced.** `waterfall_match` only considers canonical-person rows from `identifier_index`. Rank-2+ rows (fan-out) are excluded from match. Use `identifier_expand` if you want all linked identifiers. +- **Deterministic tiebreak always applies.** Even for fan-out identifier types (RCID, TTD, RAMP_ID), `waterfall_match` picks one person deterministically. Order: priority → confidence DESC → source_count DESC → max_effective_weight DESC → person_id ASC. +- **`min_confidence` default is 0.7.** Override to `0.0` for Append parity baseline. This filters on `identifier_index.confidence`. +- **Service role is `ACCOUNTADMIN`.** Cross-DB CTAS into any consumer DB works without per-client grants. Do not downgrade this role without verifying all consumer DB writes still succeed. +- **`prism_database` / `prism_schema` vars:** In prod, set `prism_database=PRISM`, `prism_schema=PUBLIC`. In dev/non-prod, defaults are `RESONATE` / `PRISM`. + +--- + +## Recent Changes (as of June 2026) + +### prism_dbt v1.0 Release (CDP-119018, June 2026) + +The `prism_dbt` package reached v1.0. Key additions: +- **`waterfall_match` macro** with full ZIP11 routing (ZIP11 now resolved via `persons.zip11` column, not `identifier_index`) +- **`NAME_ADDRESS_HASH` UDF** deployed to Snowflake; callable via `prism_dbt.name_address_hash()` macro +- **`NAME_ADDRESS_LOOKUP` stored procedure** for single-row PII-based lookups +- **CI workflows** added for automated dbt package deployment, stored procedure deployment, and UDF deployment +- **Schema-level data tests** for all three service models +- **`prism_default_waterfall`**: `IP` and `ZIP11` routing clarified — `MAID` and `IP` are single-type (not split by platform/version); ZIP11 is excluded from default (household-precision) +- **`PRISM` database**: Snowflake DB choices in workflows no longer include a separate `PRISM` DB option — everything lives in `RESONATE.PRISM.*` + +### ZIP11 Routing Fix + +ZIP11 was previously included in `identifier_index` lookups. It's now correctly routed to `persons.zip11` column directly (see `_internal/lookup_strategy.sql`). If you see unexpected ZIP11 behavior, check that you're on v1.0 of the package. diff --git a/claude-md-updates/resonate-terraform/CLAUDE.md b/claude-md-updates/resonate-terraform/CLAUDE.md new file mode 100644 index 0000000..92e46dc --- /dev/null +++ b/claude-md-updates/resonate-terraform/CLAUDE.md @@ -0,0 +1,114 @@ +# resonate-terraform + +## Project Purpose and Architecture Overview + +This repository contains all Terraform/Terragrunt infrastructure-as-code for Resonate's AWS infrastructure, as well as Terraform for GitHub org management, Lacework security, and supporting operational scripts. + +**All infrastructure changes are deployed through GitHub Actions — never run `terragrunt apply` locally.** + +**Top-level layout:** + +``` +aws/ + global/ # Cross-account AWS resources (SQS, Batch, etc.) + dev/ # Non-prod environment configs + prod_us_east_2/ # Prod environment configs +modules/ + resources/ # Reusable Terraform modules + lambda/ # Lambda function module (used by all Lambda repos) + step_function/ # Step Function module (used by all pipeline repos) + ... +github/ # Terraform for GitHub org (teams, repos, branch protection) +lacework/ # Terraform for Lacework security platform +scripts/ # Operational helper scripts (drift detection, etc.) +templates/ # Base templates updated by scripts +tests/ # Tests for Terraform templates +``` + +**Key module sources (referenced by other repos):** +- Lambda deployments: `git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/lambda` +- Step Function deployments: `git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/step_function` + +These modules are pinned by tag in consuming repos' `terragrunt.hcl` files. + +--- + +## Key Commands + +### Drift Detection + +```bash +# Run drift detection across all prod Terraform templates +python3 scripts/drift_detection.py + +# Check known drift causes at: +# https://resonate-jira.atlassian.net/wiki/spaces/ASD/pages/3951329285/Terraform+Drift+Causes +``` + +### Terraform / Terragrunt (local plan only — applies must go through CI) + +```bash +# Plan changes for a specific resource (non-prod) +cd aws/global/dev/ +terragrunt plan + +# Plan changes for prod (requires role assumption) +cd aws/global/prod_us_east_2/ +terragrunt plan --terragrunt-iam-role arn:aws:iam::694585954309:role/ProdTerraform +``` + +--- + +## GitHub Actions Workflows + +All deploys are triggered via GitHub Actions. The primary workflows are: + +| Workflow | Trigger | What it deploys | +|---|---|---| +| `terraform-plan.yml` | PR to main | Runs `terragrunt plan` across changed directories | +| `terraform-apply.yml` | Push to main | Runs `terragrunt apply` for changed directories | +| `github-terraform.yml` | Push to main (github/ changes) | GitHub org Terraform | +| `lacework-terraform.yml` | Push to main (lacework/ changes) | Lacework Terraform | + +**Infrastructure accounts:** +- Non-prod: default AWS account +- Prod: `arn:aws:iam::694585954309:role/ProdTerraform` (assumed via `role_arn` in root `terragrunt.hcl`) + +**State buckets:** +- Non-prod: `resonate-terraforming-state` +- Prod: `resonate-terraforming-state-prod` + +--- + +## Creating New Lambda Templates + +Full process documented at: +https://resonate-jira.atlassian.net/wiki/spaces/ASD/pages/683540663/Lambda+Terraform+Creation+Deployment+Process + +**Quick summary:** +1. Add a new directory under `modules/resources/` or use existing `lambda` module +2. Create a `terragrunt.hcl` in the consuming repo pointing to the module +3. Push to main — GitHub Actions applies the change + +--- + +## Project-Specific Rules and Gotchas + +- **Never apply locally.** All infra changes go through GitHub Actions. Local `terragrunt plan` is acceptable for verification, but never `apply`. +- **Terraform version:** Pinned to **1.5.7** in all workflows. Do not upgrade without coordinating with the team — version bumps require testing across all consuming repos. +- **Module versioning:** Other repos reference modules via `git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/?ref=`. When making breaking changes to a module, bump the version tag and update all consumers. +- **GitHub Terraform:** Changes to `github/` manage org-level settings (teams, repo permissions, branch protection). Test carefully — incorrect configs can lock people out of repos. +- **Drift:** Known drift exists in prod due to manual emergency changes. Always check the drift causes wiki before assuming a plan diff indicates a bug. + +--- + +## Monitoring & Alerting Infrastructure (Recent — DE-13849, June 2026) + +The monitoring stack was overhauled in June 2026, replacing Nagios-based alerting with a Slack + PagerDuty stack via AWS CloudWatch + SNS + Chatbot: + +- **Slack channel:** `#monitoring-alerts-slack` receives CloudWatch alarms for EC2 status, Grafana ALB, synthetics canaries, and `pg_rtp` database +- **Chatbot scope:** Narrowed to only `monitoring-alerts-slack` channel (was previously broader) +- **PagerDuty:** Techops PagerDuty SNS topic referenced directly for Splunk dev EC2 status alarms +- **Decommissioned:** Nagios monitoring infrastructure removed; all active alarms now go through CloudWatch → SNS → Slack/PagerDuty + +When adding new alarms, follow the pattern in `aws/global/dev/monitoring/` — publish to the relevant SNS topic (`monitoring-alerts-slack` for non-paging alerts, `techops-pagerduty` for paging alerts). diff --git a/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md b/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md new file mode 100644 index 0000000..dd3efb4 --- /dev/null +++ b/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md @@ -0,0 +1,185 @@ +# step-function-workflow-orchestrator + +## Project Purpose and Architecture Overview + +This repo contains the X-Men team's AWS Step Function pipelines and the supporting Lambda functions that power the Connected Profiles data platform. It is **not** a single application — it is a collection of ~50 independent pipelines (state machines) and ~30 Lambda functions grouped by runtime. + +**Top-level layout:** + +``` +pipelines/ # Step Function definitions — one subdir per pipeline + / + statemachine/ # ASL JSON state machine definition + config/ # EMR cluster configs + per-environment params.json + events/ # EventBridge event payloads per environment + tests/ # Unit + integration tests +terraform/ + pipelines/ # Terragrunt config for deploying step functions + //terragrunt.hcl + python_lambdas/ # Terragrunt config for Python lambdas + java_lambdas/ # Terragrunt config for Java lambdas + javascript-lambdas/ + lambda.hcl # Shared lambda Terragrunt root config + step_function.hcl # Shared step function Terragrunt root config +python_lambdas/ # Python Lambda source (one subdir per function) +java_lambdas/ # Java Lambda source (dos-lambdas, metrics-formatter) +javascript-lambdas/ # Node.js Lambda source (human-approval-*) +scripts/ # Operational helper scripts (Python, not deployed) +``` + +**Infrastructure accounts:** +- Non-prod: default AWS account (no `--profile` flag needed) +- Prod: `arn:aws:iam::694585954309:role/ProdTerraform` (assumed via `role_arn` in `step_function.hcl`) + +**State bucket:** `resonate-terraforming-state` (non-prod) / `resonate-terraforming-state-prod` (prod) +**State key prefix:** `environments/mgmt/lambdas/x_men/` (lambdas) or `environments/mgmt/lambdas/x_men|agents_of_shield/` (step functions) + +--- + +## Key Commands + +### Python Lambdas + +All Python lambdas live under `python_lambdas//`. Dependencies are per-lambda `requirements.txt` files. The workspace-level dev tooling uses `uv` (see `python_lambdas/pyproject.toml`). + +```bash +# Install dev deps (from python_lambdas/) +uv sync + +# Run unit tests for a specific lambda +cd python_lambdas/ +pytest tests/ + +# Run unit tests across all Python lambdas (from python_lambdas/) +pytest + +# Lint +flake8 python_lambdas/ + +# Build/run a single lambda locally with SAM +cd python_lambdas/ +sam build --use-container +sam local invoke --event events/event.json +``` + +### Java Lambdas + +```bash +# Build dos-lambdas fat JAR (shadow JAR) +cd java_lambdas/dos-lambdas +./gradlew shadowJar + +# Build metrics-formatter fat JAR +cd java_lambdas/metrics-formatter +./gradlew shadowJar + +# Run tests +./gradlew test +``` + +### JavaScript Lambdas + +```bash +# No build step — plain Node.js +cd javascript-lambdas/human-approval-emailer +npm install + +cd javascript-lambdas/human-approval-callback +npm install +``` + +### Integration Tests (Step Functions) + +Integration tests start the real Step Function in AWS and poll to completion. They require AWS credentials for the dev/integration environment. + +```bash +cd pipelines//tests/integration +pytest --state-machine-name -integration +# e.g. +pytest --state-machine-name behavior-stitch-integration +``` + +### Operational Scripts + +```bash +# Generate environment-specific configs from integration config (run from scripts/) +python3 generate_environment_configs.py [path-to-pipelines] +# e.g. +python3 generate_environment_configs.py dev6 + +# Mark DOS unprocessed files as processed (recovery helper, run from scripts/) +python3 mark_dos_unprocessed_as_processed.py \ + --onboard_path s3://... \ + --completion_status_path s3://... +``` + +--- + +## Terraform / Infrastructure Specifics + +All deployments go through **GitHub Actions** — there are no local `terragrunt apply` commands in normal workflow. Use the Actions UI (or `workflow_dispatch`) for all deploys. + +### GitHub Actions Workflows + +| Workflow | Trigger | What it deploys | +|---|---|---| +| `step_function.yml` | push to `pipelines/**` or manual | Step Functions via `terraform/pipelines///` | +| `python_lambda.yml` | push to `python_lambdas/**` or manual | Python Lambdas via `terraform/python_lambdas///` | +| `java_lambda.yml` | push to `java_lambdas/**` or manual | Java Lambdas via `terraform/java_lambdas///` | +| `nodejs_lambda.yml` | push to `javascript-lambdas/**` or manual | JS Lambdas via `terraform/javascript-lambdas///` | +| `unzip-copy.yml` | push to `common/unzip-copy/**` or manual | Docker image → ECR → ECS task definition | + +**Supported environments by workflow:** +- `step_function.yml`: `dev`, `dev2`–`dev9`, `integration`, `qa`, `prod` +- `python_lambda.yml`: `dev`, `dev2`–`dev9`, `nonprod`, `qa`, `prod` +- `java_lambda.yml` / `nodejs_lambda.yml`: varies per pipeline's deployed env folders +- `unzip-copy.yml`: `dev`, `prod` only + +### Terragrunt Module Sources + +- **Step functions:** `git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/step_function` +- **Lambdas:** `git::ssh://github.com/resonate/resonate-terraform.git//modules/resources/lambda` +- Terraform version pinned to **1.5.7** in all workflows. + +### Pipeline Config Layout + +Each pipeline has per-environment configs at `pipelines//config//params.json` and shared configs at `pipelines//config/emr.json`, `emr-on-demand.json`, `steps.json`. The `generate_environment_configs.py` script derives `dev6` (or other dev variants) configs from the `integration` config by rewriting S3 bucket suffixes. + +### Lambda ARN Injection + +State machine ASL files use template variables like `${DynamicDatesFunctionArn}`. These are resolved at deploy time via `lambda_replacement_vars` in each pipeline's `terragrunt.hcl`. + +### unzip-copy is Different + +`python_lambdas/unzip-copy` is deployed as a **Docker container to ECS** (not as a Lambda ZIP). It has its own `Dockerfile`, `taskdef-dev.json` / `taskdef-prod.json`, and a dedicated `unzip-copy.yml` workflow. + +--- + +## Project-Specific Rules and Gotchas + +- **Never run `terragrunt apply` locally** — all infra changes must go through GitHub Actions CI/CD. +- **Config drift in dev variants:** When creating a new dev environment (e.g. `dev7`), run `scripts/generate_environment_configs.py dev7` to generate correct S3 bucket paths — do not hand-edit `params.json` copies. +- **S3 bucket naming convention:** Non-prod buckets append `-` (e.g. `resonate-core-datasets-dev`). The `generate_environment_configs.py` script handles rewriting these automatically. +- **Lambda ARN template variables** (`${XxxFunctionArn}`) in ASL JSON are **not** real ARNs — they are substituted by Terraform at deploy time using `lambda_replacement_vars` in terragrunt.hcl. +- **Prod-only step function scheduling:** Step functions are only enabled (`is_enabled = true`) in prod environments; all other environments deploy the state machine but leave the EventBridge schedule disabled. +- **Python runtime:** Local dev tooling requires Python 3.13+ (enforced by `python_lambdas/pyproject.toml`). The AWS runtime deployed per-lambda varies: currently `python3.7`, `python3.9`, `python3.11`, `python3.12`, or `python3.13` depending on the function — check the relevant `terraform/python_lambdas//terragrunt.hcl`. Dev tooling uses `uv`; individual lambdas declare their own `requirements.txt`. +- **Java lambdas use shadow JARs:** The Gradle `shadowJar` task produces the fat JAR uploaded to Lambda. Do not use the plain `jar` task. +- **Integration tests are long-running:** The behavior-stitch integration test has a 2-hour timeout (`EXECUTION_TIMEOUT = 7200`). Do not run locally unless you intend to wait. +- **SNS topics for failure/pass alerts:** Pipelines use `sns_topic_replacement_vars` in each environment's `terragrunt.hcl` to inject `TopicFailArn`, `TopicPassArn`, and (where applicable) `TopicWarnArn`. Topic names are environment-specific: dev environments typically use `datapipeline-fail-qa-topic` / `datapipeline-pass-qa-topic`, dev6 uses `datapipeline-fail-dev6-topic` / `datapipeline-pass-dev6-topic`, and prod pipelines use pipeline-specific topics (e.g. `dos-datapipeline-fail-prod-topic`, `modeling-team-pagerduty`). These SNS topics are managed outside this repo. + +--- + +## Recent Changes (as of June 2026) + +### EMR7 Upgrades (CDP-118269) +Multiple pipelines were upgraded from EMR6 to EMR7. If you're working on a pipeline and its `emr.json` / `emr-on-demand.json` references an old release label, check the corresponding PR for the correct EMR7 release label. Affected pipelines include: +- `maid-onboarder` +- `topic-aggregation` +- `idsync-overlap` +- `marketops-overlap` +- `behavior-stitch` + +Each upgrade also updated the jar S3 bucket path for dev environments (`integration` → `dev` bucket suffix). + +### Decommissioned: cookiejar-sample-export (June 2026) +The `cookiejar-sample-export` Lambda **infrastructure has been removed** (Terraform config deleted), but the source code under `python_lambdas/cookiejar-sample-export/` is intentionally kept for reference (marked deprecated). Do not redeploy this Lambda — the business use case has been retired. If you need to clean up, only the code directory remains. From 46cc106293003f7360ac150e8ce08b3a81d0c009 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 7 Jun 2026 13:21:00 +0000 Subject: [PATCH 2/2] fix: address Copilot review comments on CLAUDE.md updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README: note PallaviJagarlamudi had 0 merged PRs (consistent with PR body) - README: add SHA fetch + placeholder to gh api update commands for existing CLAUDE.md repos (step-function-workflow-orchestrator, batch-expression-modeling) - identity-graph: correct ZIP11 in Recent Changes — it IS in the default waterfall (resolves via persons.zip11, not excluded) - batch-audience-delivery-syndication: replace ambiguous step function location with actionable search instructions - step-function-workflow-orchestrator: clarify the literal `|` in the step function state key prefix https://claude.ai/code/session_011tgujgCjwfxgGcDx6XNnH4 --- claude-md-updates/README.md | 15 +++++++++------ .../batch-audience-delivery-syndication/CLAUDE.md | 2 +- claude-md-updates/identity-graph/CLAUDE.md | 2 +- .../step-function-workflow-orchestrator/CLAUDE.md | 4 +++- 4 files changed, 14 insertions(+), 9 deletions(-) diff --git a/claude-md-updates/README.md b/claude-md-updates/README.md index 870c0b6..58fb91d 100644 --- a/claude-md-updates/README.md +++ b/claude-md-updates/README.md @@ -1,6 +1,6 @@ # CLAUDE.md Updates -This directory contains updated `CLAUDE.md` files for the 5 most active Resonate repositories (based on merged PRs from June 2025–June 2026 by contributors: SayaliPat, shrivastavakapil2000, JoeVsVolcano, PallaviJagarlamudi, mike-brant, nathan-resonate). +This directory contains updated `CLAUDE.md` files for the 5 most active Resonate repositories (based on merged PRs from June 2025–June 2026 by contributors: SayaliPat, shrivastavakapil2000, JoeVsVolcano, PallaviJagarlamudi, mike-brant, nathan-resonate). Note: PallaviJagarlamudi had no merged PRs in this period, so no repo contributions are attributed to that user. ## Repositories @@ -22,14 +22,20 @@ These CLAUDE.md files need to be applied to their respective repositories. For e ### Quick Apply Commands ```bash -# step-function-workflow-orchestrator +# step-function-workflow-orchestrator (existing CLAUDE.md — SHA required) +# Get current SHA first: +SFWO_SHA=$(gh api repos/resonate/step-function-workflow-orchestrator/contents/CLAUDE.md | jq -r .sha) gh api -X PUT repos/resonate/step-function-workflow-orchestrator/contents/CLAUDE.md \ --field message="docs: update CLAUDE.md with EMR7 upgrades and cookiejar deprecation" \ + --field sha="$SFWO_SHA" \ --field content="$(base64 -i step-function-workflow-orchestrator/CLAUDE.md)" -# batch-expression-modeling +# batch-expression-modeling (existing CLAUDE.md — SHA required) +# Get current SHA first: +BEM_SHA=$(gh api repos/resonate/batch-expression-modeling/contents/CLAUDE.md | jq -r .sha) gh api -X PUT repos/resonate/batch-expression-modeling/contents/CLAUDE.md \ --field message="docs: update CLAUDE.md with BlockGraph vendor support" \ + --field sha="$BEM_SHA" \ --field content="$(base64 -i batch-expression-modeling/CLAUDE.md)" # resonate-terraform (new file) @@ -48,9 +54,6 @@ gh api -X PUT repos/resonate/identity-graph/contents/CLAUDE.md \ --field content="$(base64 -i identity-graph/CLAUDE.md)" ``` -> **Note:** For repos with existing CLAUDE.md, you'll need to include the current file SHA in the PUT request. -> Run `gh api repos/resonate//contents/CLAUDE.md | jq .sha` to get the current SHA. - ## Generated By This update was generated by a Claude Code agent (claude.ai/code) on 2026-06-07, triggered by the presence of merged PRs from the specified contributor list. The agent reviewed recent commits, PR titles, and existing CLAUDE.md content to produce contextually accurate documentation. diff --git a/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md b/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md index 17d1fd9..ed48da6 100644 --- a/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md +++ b/claude-md-updates/batch-audience-delivery-syndication/CLAUDE.md @@ -125,4 +125,4 @@ batch-expression-modeling └─> blockgraph-publish-files (upload to FreeWheel) ``` -The step function orchestrating these three Lambdas lives in `step-function-workflow-orchestrator` (or this repo — check `workflows/step-functions/`). +To locate the step function that orchestrates these three Lambdas, search for references to `blockgraph-rename-files`, `blockgraph-create-taxonomy-file`, or `blockgraph-publish-files` in the ASL JSON files under `step-function-workflow-orchestrator/pipelines/` or under `workflows/step-functions/` in this repo. diff --git a/claude-md-updates/identity-graph/CLAUDE.md b/claude-md-updates/identity-graph/CLAUDE.md index 746aeb3..96afd7f 100644 --- a/claude-md-updates/identity-graph/CLAUDE.md +++ b/claude-md-updates/identity-graph/CLAUDE.md @@ -213,7 +213,7 @@ The `prism_dbt` package reached v1.0. Key additions: - **`NAME_ADDRESS_LOOKUP` stored procedure** for single-row PII-based lookups - **CI workflows** added for automated dbt package deployment, stored procedure deployment, and UDF deployment - **Schema-level data tests** for all three service models -- **`prism_default_waterfall`**: `IP` and `ZIP11` routing clarified — `MAID` and `IP` are single-type (not split by platform/version); ZIP11 is excluded from default (household-precision) +- **`prism_default_waterfall`**: `IP` and `ZIP11` routing clarified — `MAID` and `IP` are single-type (not split by platform/version); ZIP11 IS included in the default waterfall (it resolves via `persons.zip11` column, not `identifier_index`) - **`PRISM` database**: Snowflake DB choices in workflows no longer include a separate `PRISM` DB option — everything lives in `RESONATE.PRISM.*` ### ZIP11 Routing Fix diff --git a/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md b/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md index dd3efb4..8ade6cc 100644 --- a/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md +++ b/claude-md-updates/step-function-workflow-orchestrator/CLAUDE.md @@ -32,7 +32,9 @@ scripts/ # Operational helper scripts (Python, not deployed) - Prod: `arn:aws:iam::694585954309:role/ProdTerraform` (assumed via `role_arn` in `step_function.hcl`) **State bucket:** `resonate-terraforming-state` (non-prod) / `resonate-terraforming-state-prod` (prod) -**State key prefix:** `environments/mgmt/lambdas/x_men/` (lambdas) or `environments/mgmt/lambdas/x_men|agents_of_shield/` (step functions) +**State key prefixes:** +- Lambdas: `environments/mgmt/lambdas/x_men/` +- Step Functions: `environments/mgmt/lambdas/x_men|agents_of_shield/` (the `|` is a literal character in the S3 key, not an "or") ---