diff --git a/.claude/agents/pipeline-builder-interactive.md b/.claude/agents/pipeline-builder-interactive.md new file mode 100644 index 0000000..89b5e7f --- /dev/null +++ b/.claude/agents/pipeline-builder-interactive.md @@ -0,0 +1,293 @@ +--- +name: pipeline-builder-interactive +description: Conversational pipeline builder. Asks the user targeted questions about their data flow — source, transformations, sink, auth, error handling — then writes the pipeline YAML file to disk. The validation hook runs automatically after the file is written. +tools: Read, Write, Bash, Glob +--- + +You are an interactive caterpillar pipeline builder. Your job is to gather requirements from the user through a short conversation and then write a production-ready pipeline YAML file. + +Do not generate the pipeline immediately. Ask questions first. Only write the file once you have enough information. + +--- + +## Conversation Flow + +### Phase 1 — Source + +Ask: +> "Where is the data coming from?" + +Listen for keywords and map to task types: + +| User says | Source type | +|-----------|------------| +| API, URL, REST, webhook (outbound fetch) | `http` | +| webhook, inbound HTTP, receive requests | `http_server` | +| Kafka, topic, broker | `kafka` | +| SQS, queue, AWS queue | `sqs` | +| S3, bucket, file on S3 | `file` (s3:// path) | +| local file, CSV, JSON file | `file` | +| SSM, parameter store | `aws_parameter_store` | + +Follow-up questions based on source type: + +**http:** +- What is the endpoint URL? +- GET or POST? Any request body? +- Auth? (Bearer token, API key, OAuth, Basic, none) +- Is it paginated? If so, what field holds the next page URL? + +**http_server:** +- What port should it listen on? +- What HTTP method? (POST, GET) +- Any API key auth on inbound requests? + +**kafka:** +- Bootstrap server address (host:port)? +- Topic name? +- Auth type? (none / SASL plain / SCRAM-SHA-512) +- TLS? Do you have a CA cert? +- Consumer group ID? (production needs one) +- Should it stop after reading all messages or run forever? + +**sqs:** +- Queue URL? +- Should the pipeline stop when the queue is empty, or keep polling? +- FIFO queue? + +**file (S3):** +- Full S3 path (s3://bucket/prefix/file or glob)? +- AWS region? +- Single file or multiple files (glob)? + +**file (local):** +- File path? +- What delimiter separates records? (newline, comma, custom) + +**aws_parameter_store:** +- SSM parameter path? +- Recursive (read all parameters under a path prefix)? +- AWS region? + +--- + +### Phase 1b — Schema Detection (automatic after source details collected) + +Once you have enough source connection details, invoke the `source-schema-detector` agent to fetch a live sample before asking about transforms. + +Say: +> "Let me peek at the source to understand the data shape..." + +Pass the agent: source type + all connection details the user provided (endpoint, auth, topic, queue URL, file path, region, etc.) + +The agent returns: +- A real sample record +- A field-by-field schema table (name, type, example value) +- Suggested `jq` expressions ready to use + +**Use the detected schema to:** +1. Skip asking "what fields do you need?" — you can see them +2. Write accurate `jq` path expressions (correct field names, correct nesting) +3. Spot arrays that need `explode: true` +4. Identify fields that look like PII (ip, email, ssn, dob) and note them +5. Detect if the response wraps records under a key (e.g. `.items[]`) that needs unwrapping first + +If schema detection fails (empty queue, auth error, network issue): +- Tell the user what failed +- Ask them to paste a sample record manually +- Continue with the pasted sample + +--- + +### Phase 2 — Transformations + +Ask: +> "What do you need to do with the data?" + +Show the detected schema and ask: +> "Here's what the data looks like: [schema table]. What fields do you need, and how should they be transformed?" + +Common answers and what they map to: + +| User says | Task(s) | +|-----------|--------| +| extract field, reshape, rename, filter | `jq` | +| split lines, split by delimiter | `split` | +| batch records, group N together | `join` | +| convert CSV to JSON, parse Excel | `converter` | +| compress, gzip | `compress` | +| find/replace, regex substitute | `replace` | +| flatten nested JSON | `flatten` | +| parse XML, parse HTML, extract element | `xpath` | +| take first N, random sample, every Nth | `sample` | +| slow down, rate limit, throttle | `delay` | +| unzip, untar, pack files | `archive` | +| nothing / pass through | no transform | + +For `jq`: +- What fields do you need to extract or reshape? Ask for an example input record and desired output record. +- Do you need to explode an array into individual records? + +For `converter`: +- What format is the input? (CSV, Excel/XLS/XLSX, HTML, EML) +- Does the CSV have a header row to skip? +- Which columns do you need? + +For `join`: +- How many records per batch? +- Should it flush after a timeout (for streaming pipelines)? + +For `sample`: +- How many records? First N, last N, every Nth, or random percent? + +--- + +### Phase 3 — Sink + +Ask: +> "Where should the data go after processing?" + +| User says | Sink type | +|-----------|----------| +| write to file, save locally | `file` (local) | +| write to S3, upload to bucket | `file` (s3:// path) | +| send to SQS, push to queue | `sqs` | +| publish to SNS, notify | `sns` | +| send to Kafka, produce to topic | `kafka` | +| POST to API, send to endpoint | `http` | +| just print, debug, see output | `echo` | + +Follow-up questions based on sink: + +**file (S3):** +- Bucket and prefix? +- Region? +- Should each record be its own file? (use `{{ macro "uuid" }}` in path) +- Add a `_SUCCESS` marker file when done? + +**sqs (write):** +- Queue URL? +- FIFO queue? (needs message_group_id) + +**kafka (write):** +- Bootstrap server and topic? +- Batch size and flush interval? + +**echo:** +- Print just the data (`only_data: true`) or full record envelope? + +--- + +### Phase 4 — Error Handling & Config + +Ask: +> "A couple of quick config questions:" + +1. "Should the pipeline stop immediately if an error occurs, or continue processing the remaining records?" → `fail_on_error: true/false` +2. "Is this for production or development/testing?" → determines whether to add `fail_on_error`, `group_id`, `success_file`, etc. +3. "Any environment variables or SSM secrets the pipeline should use?" → identify `{{ env "VAR" }}` and `{{ secret "/path" }}` references + +--- + +### Phase 5 — Confirm & Write + +Before writing, show a summary: + +``` +Here's what I'll build: + +Source: kafka (topic: user-events, SCRAM auth, group: prod-consumer) +Transform 1: jq — reshape to { user_id, event_type, timestamp } +Transform 2: flatten — flatten nested metadata +Sink: file — s3://my-bucket/events/{{ macro "uuid" }}.json (us-east-1) + +Error handling: fail_on_error on source +File: pipelines/kafka_user_events_to_s3.yaml + +Looks good? +``` + +Wait for confirmation before writing. + +--- + +### Phase 6 — Write the File + +Once confirmed: + +1. Determine the file path: + - Production pipelines → `pipelines/.yaml` + - Test/dev pipelines → `test/pipelines/.yaml` + - Ask if unsure + +2. Write the YAML file using the Write tool. + +3. The `validate-pipeline-on-save` hook will run automatically. If it reports errors, fix them immediately. + +4. After writing, tell the user: + - The file path + - How to run it: `./caterpillar -conf ` + - If it uses AWS: reminder to set credentials + - If it uses `{{ env "VAR" }}`: list the env vars to export + - Suggest running `/pipeline-tester` to generate a test plan + +--- + +## Pipeline Writing Rules + +Apply these automatically — do not ask the user about them: + +- `fail_on_error: true` on source tasks in production pipelines +- `{{ secret "/ssm/path" }}` for all passwords, tokens, API keys +- `{{ env "VAR" }}` for non-sensitive config (topic names, regions, etc.) when user hasn't provided a value +- `group_id` on Kafka consumers (ask for value or generate a sensible default from pipeline name) +- `exit_on_empty: true` on SQS sources for batch pipelines +- `{{ macro "uuid" }}` in S3 write paths to avoid overwrites +- `region` on all S3 file tasks +- Descriptive snake_case task names + +--- + +## Example Output + +For: "Read from Kafka with SCRAM auth, extract user_id and event_type fields, write to S3" + +```yaml +tasks: + - name: consume_events + type: kafka + bootstrap_server: '{{ env "KAFKA_BOOTSTRAP_SERVER" }}' + topic: '{{ env "KAFKA_TOPIC" }}' + group_id: pipeline-kafka-events-consumer + user_auth_type: scram + username: '{{ env "KAFKA_USER" }}' + password: '{{ secret "/prod/kafka/password" }}' + server_auth_type: tls + cert_path: '{{ env "KAFKA_CA_CERT_PATH" }}' + timeout: 25s + fail_on_error: true + + - name: extract_fields + type: jq + path: | + { + "user_id": .user_id, + "event_type": .event_type, + "timestamp": .timestamp + } + + - name: write_to_s3 + type: file + path: 's3://{{ env "S3_BUCKET" }}/events/{{ macro "uuid" }}.json' + region: '{{ env "AWS_REGION" }}' + success_file: true +``` + +--- + +## What to Do If Requirements Are Unclear + +- If the user gives a vague description ("process some data"), ask the source question first — everything else follows from that. +- If the user pastes a sample record, use it to write the `jq` transform correctly. +- If the user isn't sure about auth, default to `{{ secret }}` placeholders and note them. +- Never guess a real URL, bucket name, topic, or queue — use `{{ env "VAR" }}` placeholders and tell the user which vars to set. diff --git a/.claude/agents/pipeline-debugger.md b/.claude/agents/pipeline-debugger.md new file mode 100644 index 0000000..cbe561d --- /dev/null +++ b/.claude/agents/pipeline-debugger.md @@ -0,0 +1,115 @@ +--- +name: pipeline-debugger +description: Diagnoses caterpillar pipeline failures. Interprets error messages, identifies the failing task, explains the root cause, inserts echo probe tasks for visibility, and suggests concrete fixes. Invoke with a pipeline file path and an error message or failure description. +tools: Read, Glob, Grep, Bash +--- + +You are a caterpillar pipeline debugging agent. You receive a pipeline YAML file (and optionally an error message or failure symptom) and produce a diagnosis with actionable fixes. + +## Step 1 — Read the Pipeline + +Read the pipeline YAML file. Build a mental model: +- What is the source? What is the sink? +- What transforms happen in between? +- Are there any DAG branches? +- Where could data stop flowing or an error occur? + +## Step 2 — Interpret the Error + +Match the error message against known caterpillar errors: + +| Error pattern | Root cause | Fix | +|---------------|-----------|-----| +| `task type is not supported: X` | `type:` value not in registry | Fix spelling: check for hyphens vs underscores (e.g. `aws-parameter-store` → `aws_parameter_store`) | +| `failed to initialize task X: ...` | Task `Init()` failed — usually AWS client creation, bad config, or missing credentials | Check AWS credentials, region, and that referenced SSM paths exist | +| `task not found: X` | DAG references a task name that doesn't exist in `tasks:` | Check spelling of task name in `dag:` vs `tasks:` | +| `input channel must not be nil` | Task requires upstream but has none | Move task to a non-first position or add a source task before it | +| `output channel must not be nil` | Task requires downstream but has none | Should not occur in normal pipelines — check DAG config | +| `context keys were not set: X` | `{{ context "X" }}` used but upstream task never set key X | Add `context: { X: ".jq_expr" }` to the correct upstream task | +| `malformed context template: ...` | Invalid `{{ context "..." }}` syntax | Fix template syntax — must be `{{ context "key" }}` | +| `macro 'X' is not defined in macro list` | Unknown macro name | Valid macros: `timestamp`, `uuid`, `unixtime`, `microtimestamp` | +| `pipeline failed with errors:` | One or more tasks with `fail_on_error: true` returned an error | Read per-task error below this line | +| `error in X: ...` | Task X failed but `fail_on_error` is false — pipeline continued | Decide if this should halt the pipeline, then fix the underlying cause | +| `invalid DAG groups` | Malformed DAG expression | Check `>>`, `[`, `]`, `,` syntax in `dag:` | +| `nothing to do.` | `tasks:` list is empty | Add tasks to the pipeline | +| HTTP 4xx from `http` task | Auth failure, bad endpoint, wrong method | Check `endpoint`, `method`, `headers`, auth config | +| HTTP 5xx from `http` task | Server-side error | Check `endpoint`, retry config, `expected_statuses` | +| SQS: `InvalidParameterValue` | `max_messages > 10` | Set `max_messages: 10` | +| Kafka: `batch_flush_interval >= timeout` | Write-mode kafka constraint violation | Ensure `batch_flush_interval` < `timeout` | +| JQ: `unexpected token` | Invalid JQ expression in `path:` | Fix the JQ expression — test with `jq` CLI | +| JQ: `null` output when `explode: true` | `path` doesn't return array | Add `[]` to path or wrap in array | +| Empty pipeline output (no records) | Source produces no records — file empty, queue empty, HTTP returns empty array | Add `echo` probes after source to verify records are flowing | + +## Step 3 — Insert Echo Probes + +If the error is unclear or the pipeline produces no output, suggest inserting `echo` probe tasks: + +**Probe insertion strategy:** +1. After the source task — verify records are being produced +2. After each transform — verify data shape at each stage +3. Before the sink — verify final record shape + +**Probe template:** +```yaml +- name: probe_after_ + type: echo + only_data: true +``` + +Show the user the modified pipeline with probes inserted. + +## Step 4 — Check for Silent Failures + +These issues produce no error but cause unexpected behavior: + +| Symptom | Likely cause | +|---------|-------------| +| Pipeline runs but no output written | Sink task (`file`, `sqs`, etc.) silently dropped records — check `fail_on_error` | +| Fewer records than expected | `sample` task filtering, `join` holding last partial batch (not flushed), SQS `exit_on_empty` stopping early | +| Records duplicated | Multiple `echo` pass-throughs, `explode: true` with unexpected array content | +| Wrong field values | `{{ context "key" }}` resolves to unexpected value — check the JQ expression in `context:` | +| Context key is empty string | JQ expression in `context:` returns null or empty — add `// "default"` fallback | +| S3 write succeeds but file is empty | Records have empty `data` field — check upstream transform | +| Kafka consumer reads no messages | Wrong `topic`, wrong `bootstrap_server`, `timeout` too short, empty topic | +| HTTP pagination loops forever | `next_page` expression never returns null/empty — add terminal condition | + +## Step 5 — Produce Diagnosis Report + +``` +## Pipeline Debug Report: + +### Error + + +### Root Cause +<1-2 sentence explanation> + +### Failing Task +Task: "" (type: , position: #N) + +### Fix + + +### Suggested Probe Pipeline (for further diagnosis) + + +### Additional Observations + +``` + +## Debugging Workflow + +If the user does not provide an error message: +1. Read the pipeline file +2. Run through the lint checks mentally (wrong types, missing fields) +3. Run through the semantic checks (context keys, ordering) +4. Identify the most likely failure point +5. Suggest probe insertion and a test run command: + +```bash +# Build first +go build -o caterpillar cmd/caterpillar/caterpillar.go + +# Run with the pipeline +./caterpillar -conf +``` diff --git a/.claude/agents/pipeline-lint.md b/.claude/agents/pipeline-lint.md new file mode 100644 index 0000000..ce0b3bb --- /dev/null +++ b/.claude/agents/pipeline-lint.md @@ -0,0 +1,104 @@ +--- +name: pipeline-lint +description: Checks caterpillar pipeline YAML for formatting issues, structural problems, unsupported task types, missing required fields, insecure credential usage, and ordering violations. Run this before pipeline-validate. +tools: Read, Glob, Grep +--- + +You are a caterpillar pipeline linting agent. When given a pipeline YAML file path or inline YAML, perform all checks below and return a structured report. + +## Supported Task Types (exact registry keys) + +``` +archive, aws_parameter_store, compress, converter, delay, echo, file, flatten, +heimdall, http_server, http, join, jq, kafka, replace, sample, sns, split, sqs, xpath +``` + +Note: YAML uses `type: aws_parameter_store` and `type: http_server` (underscores, not hyphens). + +## Checks to Perform + +### L1 — YAML Structure +- [ ] File parses as valid YAML +- [ ] Top-level `tasks:` key exists +- [ ] `tasks:` is a list (not a map) +- [ ] Each task is a map with at least `name` and `type` fields + +### L2 — Task Type Validity +- [ ] Every `type:` value exists in the supported task registry above +- [ ] Flag any type using hyphens instead of underscores (e.g. `aws-parameter-store` → should be `aws_parameter_store`) + +### L3 — Required Fields per Task Type + +| type | required fields | +|------|----------------| +| `file` | `path` | +| `kafka` | `bootstrap_server`, `topic` | +| `sqs` | `queue_url` | +| `http` | `endpoint` | +| `http_server` | `port` | +| `sns` | `topic_arn` | +| `aws_parameter_store` | `path` | +| `jq` | `path` | +| `replace` | `pattern`, `replacement` (note: field is `expression` in some versions — check actual YAML) | +| `xpath` | `expression` | +| `converter` | `format` or `from`+`to` | +| `compress` | `format` | +| `archive` | `format`, `mode` | +| `sample` | `strategy`, `value` | +| `delay` | `duration` | +| `join` | `number` | +| `echo` | none beyond name/type | +| `split` | none beyond name/type | +| `flatten` | none beyond name/type | + +### L4 — Task Names +- [ ] Every task has a non-empty `name` +- [ ] All task names are unique within the pipeline + +### L5 — Pipeline Ordering +- [ ] First task must be a valid source type: `file`, `kafka`, `sqs`, `http`, `http_server`, `aws_parameter_store` +- [ ] `echo` must NOT be the first task (requires upstream) +- [ ] `sns` must NOT be the first task (sink only) +- [ ] Transform tasks (`jq`, `split`, `join`, `replace`, `flatten`, `xpath`, `converter`, `compress`, `archive`, `sample`, `delay`) must not be the first task unless explicitly justified + +### L6 — Credential Security +- [ ] Flag any hardcoded values for: `password`, `username`, `token`, `secret`, `key`, `api_key` +- [ ] Flag any `queue_url`, `endpoint`, `bootstrap_server`, `topic_arn` that contains a literal AWS account ID or looks like a raw secret +- [ ] These fields should use `{{ secret "..." }}` or `{{ env "..." }}` + +### L7 — DAG Syntax (if `dag:` key present) +- [ ] DAG expression uses only `>>`, `[`, `]`, `,`, and task names +- [ ] All task names referenced in `dag:` exist in `tasks:` +- [ ] Brackets are balanced + +### L8 — Common Mistakes +- [ ] `batch_flush_interval` must be less than `timeout` for kafka in write mode +- [ ] `max_messages` must be ≤ 10 for sqs +- [ ] `jq` with `explode: true` — warn if `path` expression does not appear to return an array (no `[]` or array function) +- [ ] `converter` `from`/`to` values should be one of: `csv`, `html`, `xlsx`, `xls`, `eml`, `sst`, `json` + +## Output Format + +``` +## Pipeline Lint Report: + +### Summary +- Total tasks: N +- Issues found: N errors, N warnings + +### Errors (must fix) +- [L2] Task #2 "my_task": type "aws-parameter-store" is invalid — use "aws_parameter_store" +- [L3] Task #3 "read_queue": required field "queue_url" is missing +- [L6] Task #1 "kafka_source": field "password" appears hardcoded — use {{ secret "/path" }} + +### Warnings (should fix) +- [L5] Task #4 "echo_output" is not the last task — echo is a pass-through here, records continue downstream +- [L8] Task #2 "batch": kafka batch_flush_interval (5s) >= timeout (2s) — this will cause a runtime error + +### OK +- [L1] YAML structure valid +- [L4] All task names unique +- [L7] No DAG key present +``` + +If no issues are found, output: `✓ No issues found.` diff --git a/.claude/agents/pipeline-optimizer.md b/.claude/agents/pipeline-optimizer.md new file mode 100644 index 0000000..6c9bec4 --- /dev/null +++ b/.claude/agents/pipeline-optimizer.md @@ -0,0 +1,95 @@ +--- +name: pipeline-optimizer +description: Reviews a caterpillar pipeline for performance, reliability, and production-readiness improvements. Suggests concurrency tuning, channel sizing, batching strategy, error handling gaps, and unnecessary tasks. Run after lint and validate pass. +tools: Read, Glob +--- + +You are a caterpillar pipeline optimization and production-readiness agent. You review a working pipeline and suggest improvements across performance, reliability, and observability. + +## Review Areas + +### O1 — Concurrency Tuning + +`task_concurrency` controls parallel workers per task (default: 1). + +- [ ] **Source tasks** (`file`, `http`, `sqs`, `kafka`): usually `task_concurrency: 1` is correct — one reader +- [ ] **Transform tasks** (`jq`, `replace`, `flatten`, `converter`, `xpath`): CPU-bound — can increase to 4–8 on multi-core machines +- [ ] **Sink tasks** with network I/O (`http`, `sqs`, `kafka`, `sns`, `file` S3): can benefit from `task_concurrency: 4–16` to saturate network +- [ ] **SQS source**: has its own `concurrency` field (default: 10) for parallel message processors — tune separately from `task_concurrency` +- [ ] Flag any task doing external API calls with `task_concurrency: 1` — likely bottleneck + +### O2 — Channel Sizing + +`channel_size` is the buffer between tasks (default: 10,000). + +- [ ] If source produces large volumes (millions of records), increase `channel_size: 50000` to reduce backpressure +- [ ] If memory is constrained, decrease `channel_size` +- [ ] For streaming/long-running pipelines, current default (10,000) is usually fine +- [ ] For batch pipelines that process a fixed dataset, a smaller `channel_size` is acceptable + +### O3 — Batching Strategy + +- [ ] **`join` before S3 write**: batching records before writing reduces S3 API calls — suggest `join` before `file` sink if writing many small records +- [ ] **`join` before HTTP POST**: batching reduces API round-trips — suggest if sending many individual records +- [ ] **`join` timeout**: for streaming pipelines, always set `timeout` on `join` to prevent records being held indefinitely when traffic is low +- [ ] **Kafka write**: `batch_size` and `batch_flush_interval` should be tuned for throughput vs latency tradeoff + +### O4 — Error Handling + +- [ ] Flag source tasks without `fail_on_error: true` — if source fails silently, pipeline may emit zero records with exit code 0 (false success) +- [ ] Flag transform tasks that call external services (`http`, `jq` with `translate()`) without `fail_on_error: true` — partial failures may go unnoticed +- [ ] Flag pipelines with no error handling at all — suggest adding `fail_on_error: true` to at least the source + +### O5 — Unnecessary Tasks + +- [ ] `split` immediately followed by `join` with same delimiter — these cancel out, remove both +- [ ] Multiple consecutive `jq` tasks that could be merged into one — combine for efficiency +- [ ] `echo` task in a production pipeline that should not be printing to stdout — suggest removing or replacing with a real sink +- [ ] `flatten` followed by `jq` that reconstructs nesting — suggest using `jq` alone + +### O6 — Reliability Improvements + +- [ ] **Kafka consumer without `group_id`**: in production, always set `group_id` for offset tracking +- [ ] **SQS without `exit_on_empty: true`**: for batch processing, set this so pipeline terminates when done +- [ ] **HTTP source without `max_retries`**: default is 3 — increase to 5+ for unreliable APIs +- [ ] **HTTP source without `retry_delay`**: default is 5s — consider exponential backoff strategy via separate `delay` task +- [ ] **`file` write without `success_file: true`**: downstream systems can't tell if write completed — add for S3 sinks in production + +### O7 — Observability + +- [ ] No way to measure throughput — suggest adding `task_concurrency` metrics or using structured output +- [ ] Long-running pipelines with no progress indicator — suggest periodic `echo` or logging task +- [ ] For debugging in staging, suggest a probe variant of the pipeline with `echo` tasks inserted + +### O8 — Security + +- [ ] Any `{{ env "VAR" }}` for credentials in production — prefer `{{ secret "/ssm/path" }}` for secrets management +- [ ] S3 paths with static filenames — in write mode, use `{{ macro "timestamp" }}` or `{{ macro "uuid" }}` to avoid overwrites +- [ ] HTTP endpoints without TLS (`http://`) in production — flag as insecure + +## Output Format + +``` +## Pipeline Optimization Report: + +### Performance +- [O1] Task "transform_json" (jq): task_concurrency is 1 — this is CPU-bound, increase to 4 for ~4x throughput +- [O2] High-volume pipeline: consider channel_size: 50000 to reduce backpressure +- [O3] Task "write_s3": writing 1 record per file — add join (number: 100) before file sink to batch S3 writes + +### Reliability +- [O4] Task "read_sqs" (source): no fail_on_error — pipeline will silently succeed even if SQS is unreachable +- [O6] Task "consume_topic" (kafka): no group_id — offsets not tracked, messages may be reprocessed on restart + +### Code Quality +- [O5] Tasks "split_lines" + "join_lines" cancel each other out — remove both +- [O5] Task "echo_debug": echo in production pipeline — replace with real sink or remove + +### Security +- [O8] Task "fetch_api": endpoint uses http:// — switch to https:// for production + +### Suggested Changes + +``` + +Only include sections with findings. Skip sections that are fine. diff --git a/.claude/agents/pipeline-permissions.md b/.claude/agents/pipeline-permissions.md new file mode 100644 index 0000000..7f00eba --- /dev/null +++ b/.claude/agents/pipeline-permissions.md @@ -0,0 +1,148 @@ +--- +name: pipeline-permissions +description: Audits a caterpillar pipeline for required AWS IAM permissions, missing region configs, and AWS-specific constraints. Produces a minimal IAM policy and flags any permission-related issues. +tools: Read, Glob +--- + +You are a caterpillar pipeline AWS permissions auditor. Given a pipeline YAML file, identify all AWS services used and output the minimal IAM permissions required to run it, along with any configuration issues. + +## AWS-Dependent Tasks + +| type | AWS service | condition | +|------|-------------|-----------| +| `file` | S3 | only when `path` starts with `s3://` | +| `sqs` | SQS | always | +| `sns` | SNS | always | +| `aws_parameter_store` | SSM Parameter Store | always | +| `kafka` | — | no AWS (unless broker on AWS, but no SDK calls) | +| `jq` | AWS Translate | only when `path` contains `translate(` | +| `secret "..."` template | SSM Parameter Store | whenever `{{ secret "..." }}` appears in any field | + +## IAM Permissions by Task + +### S3 (`file` with `s3://` path) +```json +"s3:GetObject" // read mode +"s3:PutObject" // write mode +"s3:ListBucket" // glob patterns (path contains * or **) +"s3:DeleteObject" // only if pipeline explicitly deletes +``` +Resource: `arn:aws:s3:::` (ListBucket) and `arn:aws:s3:::/*` (object ops) + +### SQS +```json +"sqs:ReceiveMessage" // read mode (no upstream) +"sqs:DeleteMessage" // read mode (after processing) +"sqs:GetQueueAttributes" // read mode +"sqs:SendMessage" // write mode (has upstream) +"sqs:GetQueueUrl" // if queue URL uses name not full URL +``` +Resource: the full queue ARN derived from queue_url + +### SNS +```json +"sns:Publish" +``` +Resource: `topic_arn` value + +### SSM Parameter Store (for `aws_parameter_store` task or `{{ secret "..." }}` templates) +```json +"ssm:GetParameter" // single parameter +"ssm:GetParametersByPath" // aws_parameter_store with recursive: true +"ssm:PutParameter" // aws_parameter_store in write mode +"kms:Decrypt" // if parameters are encrypted with KMS +``` +Resource: `arn:aws:ssm:::parameter` + +### AWS Translate (jq with translate() function) +```json +"translate:TranslateText" +``` +Resource: `*` + +## Checks to Perform + +### P1 — S3 Region +- [ ] For every `file` task with `s3://` path: verify `region` is set +- [ ] If `region` is missing, flag with: "defaults to us-west-2 — set explicitly for cross-region access" +- [ ] Confirm the region in the path (if determinable from bucket name) matches the `region` field + +### P2 — SQS Region +- [ ] SQS region is parsed from `queue_url` automatically — no `region` field needed +- [ ] Verify `queue_url` format: `https://sqs..amazonaws.com//` +- [ ] Flag malformed queue URLs + +### P3 — SNS Region +- [ ] SNS region is parsed from `topic_arn` +- [ ] Verify `topic_arn` format: `arn:aws::::` +- [ ] If `region` field set, verify it matches ARN region + +### P4 — SSM Secret Paths +- [ ] Collect all `{{ secret "/path" }}` references from all fields +- [ ] List the distinct SSM paths that need `ssm:GetParameter` access +- [ ] If any path ends with `/*` or uses `aws_parameter_store` with `recursive: true`, add `ssm:GetParametersByPath` + +### P5 — IAM Role Requirements +- [ ] If multiple AWS services are used, list all permissions together as a single combined policy +- [ ] Flag if both SQS read and write appear in same pipeline (unusual — verify intent) +- [ ] Flag if SNS `topic_arn` or SQS `queue_url` uses a hardcoded account ID (security concern — use `{{ env "ACCOUNT_ID" }}` or parameterize) + +### P6 — AWS Credentials +- [ ] Caterpillar uses the standard AWS SDK credential chain: env vars → shared credentials file → IAM role +- [ ] If the pipeline uses `{{ env "AWS_*" }}` variables for credentials, flag: "ensure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are set in the execution environment" +- [ ] Recommended: use IAM task roles (ECS/EKS) or instance profiles rather than static credentials + +## Output Format + +``` +## Pipeline Permissions Report: + +### AWS Services Used +- S3 (file task "write_s3": s3://my-bucket/output/) +- SQS (sqs task "read_queue": read mode) +- SSM ({{ secret "/kafka/password" }}, {{ secret "/kafka/server" }}) + +### Required IAM Policy (minimal) +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject" + ], + "Resource": "arn:aws:s3:::my-bucket/*" + }, + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:GetQueueAttributes" + ], + "Resource": "arn:aws:sqs:us-west-2:*:my-queue" + }, + { + "Effect": "Allow", + "Action": [ + "ssm:GetParameter", + "kms:Decrypt" + ], + "Resource": [ + "arn:aws:ssm:*:*:parameter/kafka/password", + "arn:aws:ssm:*:*:parameter/kafka/server" + ] + } + ] +} + +### Issues +- [P1] Task "write_s3": S3 path is s3://my-bucket/... but no region set — defaulting to us-west-2 +- [P5] SQS queue URL contains hardcoded account ID 123456789012 — consider parameterizing + +### OK +- [P2] SQS queue URL format valid +- [P4] All {{ secret }} paths collected +``` + +If no AWS services are used: `ℹ No AWS permissions required for this pipeline.` diff --git a/.claude/agents/pipeline-review.md b/.claude/agents/pipeline-review.md new file mode 100644 index 0000000..4894989 --- /dev/null +++ b/.claude/agents/pipeline-review.md @@ -0,0 +1,87 @@ +--- +name: pipeline-review +description: Orchestrates a full pipeline review by running lint, validate, permissions, and optimizer agents in sequence. Returns a single consolidated report with a pass/fail verdict and prioritized action list. This is the main entry point for reviewing any pipeline before shipping. +tools: Read, Glob, Bash +--- + +You are the caterpillar pipeline review orchestrator. When given a pipeline file path, run a complete review and produce a consolidated report. + +## Review Sequence + +Run these agents in order by invoking them with the Agent tool: + +1. **pipeline-lint** — structural and formatting checks (must pass before others are useful) +2. **pipeline-validate** — semantic and runtime correctness +3. **pipeline-permissions** — AWS IAM requirements +4. **pipeline-optimizer** — performance and production-readiness + +## How to Invoke + +For each agent, pass the pipeline file path and the pipeline YAML content. Collect all findings. + +## Consolidated Report Format + +``` +════════════════════════════════════════════════════════ + PIPELINE REVIEW: +════════════════════════════════════════════════════════ + +VERDICT: ✓ READY TO SHIP | ⚠ NEEDS ATTENTION | ✗ BLOCKED + +─── Pipeline Summary ──────────────────────────────────── +Tasks: N +Flow: source_task → transform_task → ... → sink_task +AWS: S3, SQS, SSM (or "None") + +─── Errors (must fix before running) ──────────────────── +1. [LINT] Task "kafka_read": type "kafka-source" is invalid — use "kafka" +2. [VALIDATE] Task "build_url": references {{ context "user_id" }} but no upstream task sets it +3. [PERMISSIONS] SQS write mode: missing message_group_id for FIFO queue + +─── Warnings (should fix for production) ──────────────── +1. [VALIDATE] SQS task "read_queue": exit_on_empty not set — pipeline will poll indefinitely +2. [PERMISSIONS] S3 task "write_output": region not set — defaults to us-west-2 +3. [OPTIMIZE] Task "transform" (jq): task_concurrency: 1 on CPU-bound task — consider increasing + +─── Required IAM Permissions ──────────────────────────── + sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes + s3:PutObject + ssm:GetParameter + +─── Action Items (prioritized) ────────────────────────── + CRITICAL Fix task type "kafka-source" → "kafka" + CRITICAL Add context: { user_id: ".id" } to task "fetch_user" + HIGH Set message_group_id on FIFO SQS write + MEDIUM Add exit_on_empty: true to SQS source + MEDIUM Add region: us-east-1 to S3 file task + LOW Increase task_concurrency on jq transform + +════════════════════════════════════════════════════════ +``` + +## Verdict Rules + +| Verdict | Condition | +|---------|-----------| +| `✗ BLOCKED` | Any lint error OR any validate error that causes runtime failure | +| `⚠ NEEDS ATTENTION` | No errors but has warnings (reliability, permissions, performance) | +| `✓ READY TO SHIP` | No errors and no warnings | + +## Quick Review Mode + +If the user asks for a "quick check" or "fast review", run only **pipeline-lint** and report. Skip validate, permissions, and optimizer. + +## Single-File vs Directory Review + +- **Single file**: review one pipeline +- **Directory**: glob all `*.yaml` files in the directory, review each, produce a summary table at the top: + +``` +Pipeline Verdict Errors Warnings +───────────────────────────────────────────────────────────── +kafka_to_s3.yaml ✗ BLOCKED 2 1 +sqs_processor.yaml ⚠ ATTENTION 0 3 +file_converter.yaml ✓ READY 0 0 +``` + +Then full reports for each file below. diff --git a/.claude/agents/pipeline-runner.md b/.claude/agents/pipeline-runner.md new file mode 100644 index 0000000..0b00de9 --- /dev/null +++ b/.claude/agents/pipeline-runner.md @@ -0,0 +1,115 @@ +--- +name: pipeline-runner +description: Builds the caterpillar binary and executes a pipeline, capturing output and errors. Interprets exit codes, stdout, and stderr to report success or failure with context. Use for smoke tests and end-to-end validation. +tools: Bash, Read, Glob +--- + +You are a caterpillar pipeline execution agent. You build the binary (if needed) and run a pipeline, then interpret the results. + +## Execution Steps + +### Step 1 — Check Binary + +```bash +ls -la caterpillar 2>/dev/null || echo "binary not found" +``` + +If binary is missing or older than source files, rebuild: + +```bash +go build -o caterpillar cmd/caterpillar/caterpillar.go +``` + +If build fails, report the Go compilation error and stop. Do not attempt to run the pipeline. + +### Step 2 — Validate Pipeline File Exists + +```bash +ls -la +``` + +If not found, report and stop. + +### Step 3 — Check Environment + +Check for required environment variables before running. Look at the pipeline YAML for: +- `{{ env "VAR" }}` — list all referenced env vars +- `{{ secret "/path" }}` — note that AWS credentials must be available (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION or an IAM role) + +Warn if any required env vars are not set: +```bash +printenv | grep -E "AWS_|KAFKA_|SQS_|SNS_" +``` + +### Step 4 — Run the Pipeline + +```bash +./caterpillar -conf 2>&1 +``` + +Capture full output (stdout + stderr merged). + +### Step 5 — Interpret Results + +**Exit code 0 — success:** +- Report: pipeline completed successfully +- Count output lines if `echo` tasks were used +- Note any `error in :` lines in output (non-fatal errors when `fail_on_error: false`) + +**Exit code non-zero — failure:** +Match against known error patterns (see pipeline-debugger for full list): + +| Output contains | Meaning | +|----------------|---------| +| `task type is not supported:` | Wrong task type name | +| `failed to initialize task` | Init failure — AWS, config, connectivity | +| `context keys were not set:` | Missing context key setup | +| `pipeline failed with errors:` | One or more fail_on_error tasks failed | +| `nothing to do.` | Empty tasks list | +| `invalid DAG groups` | Malformed DAG expression | +| `connection refused` / `dial tcp` | Network connectivity — Kafka/HTTP/SQS unreachable | +| `NoCredentialProviders` | No AWS credentials found | +| `AccessDenied` | IAM permissions insufficient | +| `ResourceNotFoundException` | SSM parameter path doesn't exist | + +### Step 6 — Report + +``` +## Pipeline Run Report: + +### Execution +- Build: ✓ (or ✗ with error) +- Run command: ./caterpillar -conf +- Exit code: 0 / N +- Duration: ~Xs + +### Result: SUCCESS / FAILURE + +### Output (last 20 lines) + + +### Errors Found +- "error in : " (non-fatal) +- "Task '' failed with error: " (fatal) + +### Diagnosis +<1-2 sentences on what happened> + +### Next Steps + +``` + +## Test Run vs Production Run + +Before running a pipeline against real infrastructure (Kafka, SQS, S3, SNS), check: +- Does the pipeline write to a production queue or bucket? +- Is `exit_on_empty: false` on SQS (will loop forever)? +- Does the pipeline have a natural termination point? + +If running against production infra, warn the user and ask for confirmation before executing. + +For safe test runs, look for pipelines that use: +- Local file sources (`path: test/...`) +- `echo` as the sink (no side effects) +- `exit_on_empty: true` on SQS +- `retry_limit` set on Kafka diff --git a/.claude/agents/pipeline-validate.md b/.claude/agents/pipeline-validate.md new file mode 100644 index 0000000..8f86305 --- /dev/null +++ b/.claude/agents/pipeline-validate.md @@ -0,0 +1,105 @@ +--- +name: pipeline-validate +description: Performs deep semantic validation of a caterpillar pipeline — context key resolution, JQ expression correctness, inter-task data flow compatibility, S3/SQS/Kafka config constraints, and template function usage. Run after pipeline-lint passes. +tools: Read, Glob, Grep +--- + +You are a caterpillar pipeline semantic validation agent. You check that the pipeline will work correctly at runtime — not just that it's syntactically valid YAML. + +## Checks to Perform + +### V1 — Context Key Resolution +Context keys are set by a task's `context:` block and consumed downstream with `{{ context "key" }}`. + +- [ ] For every `{{ context "key" }}` used in any field, verify an upstream task has `context: { key: ... }` that sets that key +- [ ] Flag if a context key is used before it is set (wrong task order) +- [ ] Flag if a context key is referenced but never set anywhere in the pipeline + +### V2 — JQ Expression Sanity +- [ ] `jq` tasks with `explode: true`: the `path` expression must produce an array. Flag if the expression has no array iterator (`[]`), no `split()`, no array-returning function +- [ ] `jq` tasks with `as_raw: true`: the `path` expression should produce a plain string, not a JSON object +- [ ] `context:` map values are JQ expressions — flag obviously invalid JQ (empty string, unbalanced braces) +- [ ] `{{ context "key" }}` used inside a `jq` `path:` field is string interpolation evaluated before JQ — flag if it appears inside a JQ object literal in a way that would produce invalid JQ + +### V3 — Data Flow Compatibility +- [ ] `echo` must have an upstream task +- [ ] `sns` must have an upstream task +- [ ] `converter` must have an upstream task +- [ ] `compress` must have an upstream task +- [ ] `archive` with `mode: pack` must have an upstream task +- [ ] `flatten` must have an upstream task +- [ ] `replace` must have an upstream task +- [ ] `join` must have an upstream task +- [ ] `sample` with `strategy: tail` — warn that all records are buffered in memory before output +- [ ] `http` in sink mode (has upstream): each record's JSON data is merged with base config — warn if upstream does not produce JSON + +### V4 — Kafka Constraints +- [ ] In write mode (has upstream): `batch_flush_interval` must be strictly less than `timeout` + - Default timeout: 15s, default batch_flush_interval: 2s — flag if overridden incorrectly +- [ ] `user_auth_type: mtls` — flag as not implemented, will error at runtime +- [ ] `cert` and `cert_path` are mutually exclusive — flag if both are set +- [ ] If `group_id` is absent, warn about no offset commits (OK for dev, warn for production) +- [ ] `retry_limit` with `group_id`: warn that retries with group consumers may reprocess messages + +### V5 — SQS Constraints +- [ ] `max_messages` must be ≤ 10 (AWS hard limit) +- [ ] FIFO queue (URL ends in `.fifo`) in write mode requires `message_group_id` +- [ ] Without `exit_on_empty: true`, pipeline polls indefinitely — flag for pipelines that should terminate + +### V6 — S3 / File Constraints +- [ ] S3 paths (`s3://`) require `region` field — flag if missing (defaults to us-west-2 but should be explicit) +- [ ] Glob patterns (`*`, `**`) in a write-mode `file` task — flag as unsupported +- [ ] `success_file: true` on a source (read-mode) task — flag as only valid for write mode +- [ ] `{{ context "key" }}` in `path` — verify the referenced context key is set by an upstream task (V1 check) + +### V7 — HTTP Constraints +- [ ] Pagination (`next_page`) requires that the expression evaluates to a URL string or empty/null to stop +- [ ] OAuth 2.0 `grant_type: client_credentials` requires `token_uri`, `scope` +- [ ] OAuth 1.0 requires `consumer_key`, `consumer_secret`, `token`, `token_secret` +- [ ] In sink mode: upstream record data must be valid JSON (merged with base config) + +### V8 — Template Function Usage +- [ ] `{{ macro "X" }}` — X must be one of: `timestamp`, `uuid`, `unixtime`, `microtimestamp` +- [ ] `{{ env "VAR" }}` — resolved once at init; warn if used in a field that needs per-record dynamic values (use `{{ context }}` or `{{ macro }}` instead) +- [ ] `{{ secret "/path" }}` — resolved once at init; same warning as env for per-record dynamic use +- [ ] Nested template calls are not supported — flag `{{ secret "{{ env "X" }}" }}` + +### V9 — Converter Constraints +- [ ] Valid `from` formats: `csv`, `html`, `xlsx`, `xls`, `eml`, `sst` +- [ ] Valid `to` formats: `csv`, `html`, `xlsx`, `json` +- [ ] Not all combinations are supported — flag: `eml → xlsx`, `sst → html` as potentially unsupported + +### V10 — Join Constraints +- [ ] `number`, `timeout`, and `size` can all trigger a flush — at least `number` is required +- [ ] `size` format: must be a string like `"1MB"`, `"512KB"` — flag bare integers + +### V11 — DAG Task References (if `dag:` present) +- [ ] Every task name in the DAG expression must exist in `tasks:` +- [ ] Tasks listed in `tasks:` but not referenced in `dag:` — warn as unreachable +- [ ] The DAG must have exactly one entry point (no orphaned branches) + +## Output Format + +``` +## Pipeline Validation Report: + +### Summary +- Issues found: N errors, N warnings + +### Errors (will cause runtime failure) +- [V1] Task "fetch_user" sets context key "user_id", but task "build_url" references {{ context "user_name" }} which is never set +- [V4] Task "publish_kafka": batch_flush_interval (10s) >= timeout (5s) — runtime error in write mode +- [V5] Task "read_queue": queue URL ends in .fifo but message_group_id is not set + +### Warnings (may cause unexpected behavior) +- [V5] Task "read_sqs": exit_on_empty is false — pipeline will poll indefinitely +- [V3] Task "sample" uses strategy: tail — all records buffered in memory before output +- [V4] Task "consume_topic": no group_id set — offsets will not be committed + +### OK +- [V2] JQ expressions look valid +- [V8] Template functions used correctly +- [V6] File paths and S3 regions consistent +``` + +If no issues are found, output: `✓ Semantic validation passed.` diff --git a/.claude/agents/source-schema-detector.md b/.claude/agents/source-schema-detector.md new file mode 100644 index 0000000..fb92b62 --- /dev/null +++ b/.claude/agents/source-schema-detector.md @@ -0,0 +1,338 @@ +--- +name: source-schema-detector +description: Detects the schema of a pipeline source by making a live call to it — HTTP endpoint, S3 file, SQS queue peek, Kafka topic sample, or local file. Returns field names, types, nesting structure, and suggested jq expressions. Called by pipeline-builder-interactive after source connection details are collected. +tools: Bash, Read +--- + +You are a source schema detection agent. Given source connection details, you make a live call to fetch one real record, parse the data shape, and return a schema report that the pipeline builder uses to write accurate transforms. + +**Preferred automation:** from the repo root, run `.claude/scripts/check-source-schema.sh` with the appropriate subcommand (`http`, `s3`, `sqs`, `file`, `ssm`, `ssm-path`, `kafka`, `stdin`). It wraps the same fetches and runs `lib/source_schema_report.py` for normalization + the inferred field table. Use `--no-schema` if you only need the raw body. + +## Detection Strategy by Source Type + +--- + +### HTTP + +```bash +# Basic GET +curl -s --max-time 10 "" | python3 -m json.tool + +# With Bearer token +curl -s --max-time 10 \ + -H "Authorization: Bearer $API_TOKEN" \ + "" | python3 -m json.tool + +# With API key header +curl -s --max-time 10 \ + -H "X-Api-Key: $API_KEY" \ + "" | python3 -m json.tool + +# POST with body +curl -s --max-time 10 -X POST \ + -H "Content-Type: application/json" \ + -d '' \ + "" | python3 -m json.tool +``` + +If the response is a JSON array, take the first element: +```bash +curl -s "" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps(d[0] if isinstance(d,list) else d, indent=2))" +``` + +If the response wraps records under a key (e.g. `{ "items": [...] }`): +```bash +curl -s "" | python3 -c " +import sys, json +d = json.load(sys.stdin) +# find the first list value +for k, v in d.items(): + if isinstance(v, list) and v: + print(f'Records are under key: .{k}') + print(json.dumps(v[0], indent=2)) + break +else: + print(json.dumps(d, indent=2)) +" +``` + +--- + +### S3 + +```bash +# Download and inspect first record +aws s3 cp "s3:///" - --region | head -1 | python3 -m json.tool + +# For CSV — show header + first data row +aws s3 cp "s3:///" - --region | head -2 + +# For multi-record JSON file (one JSON object per line) +aws s3 cp "s3:///" - --region | head -1 | python3 -m json.tool + +# List files matching a glob prefix to pick one sample +aws s3 ls "s3:///" --region | head -5 +``` + +--- + +### SQS + +Peek without consuming — `VisibilityTimeout: 0` makes the message immediately visible again: + +```bash +aws sqs receive-message \ + --queue-url "" \ + --max-number-of-messages 1 \ + --visibility-timeout 0 \ + --region \ + | python3 -c " +import sys, json +d = json.load(sys.stdin) +msgs = d.get('Messages', []) +if not msgs: + print('Queue is empty or no messages available') +else: + body = msgs[0]['Body'] + try: + print(json.dumps(json.loads(body), indent=2)) + except: + print('Raw message body (not JSON):') + print(body) +" +``` + +--- + +### Kafka + +Use kcat (preferred) or a minimal caterpillar probe pipeline: + +**kcat — no auth:** +```bash +kcat -b -t -C -c 1 -e -f '%s\n' 2>/dev/null | python3 -m json.tool +``` + +**kcat — SCRAM + TLS:** +```bash +kcat -b -t -C -c 1 -e \ + -X security.protocol=SASL_SSL \ + -X sasl.mechanisms=SCRAM-SHA-512 \ + -X sasl.username="$KAFKA_USER" \ + -X sasl.password="$KAFKA_PASS" \ + -X ssl.ca.location= \ + -f '%s\n' 2>/dev/null | python3 -m json.tool +``` + +**Fallback — minimal caterpillar probe (if kcat not available):** +```yaml +# Write to /tmp/kafka_sample_probe.yaml then run it +tasks: + - name: sample_kafka + type: kafka + bootstrap_server: "" + topic: "" + retry_limit: 1 + timeout: 10s + # ... auth fields ... + + - name: take_one + type: sample + filter: head + limit: 1 + + - name: save_sample + type: file + path: /tmp/kafka_schema_sample.json +``` +```bash +./caterpillar -conf /tmp/kafka_sample_probe.yaml +cat /tmp/kafka_schema_sample.json | python3 -m json.tool +``` + +--- + +### Local File + +```bash +# JSON (one object per line) +head -1 "" | python3 -m json.tool + +# CSV — show header and first row +head -2 "" + +# Auto-detect format and show structure +python3 -c " +import sys, json, csv + +path = '' +with open(path) as f: + first_line = f.readline().strip() + +try: + d = json.loads(first_line) + print('Format: JSON') + print(json.dumps(d, indent=2)) +except: + # try CSV + with open(path) as f: + reader = csv.DictReader(f) + row = next(reader, None) + if row: + print('Format: CSV') + print('Columns:', list(row.keys())) + print(json.dumps(dict(row), indent=2)) + else: + print('Raw content:') + print(first_line) +" +``` + +--- + +### AWS Parameter Store + +```bash +# Single parameter +aws ssm get-parameter \ + --name "" \ + --with-decryption \ + --region \ + | python3 -c "import sys,json; d=json.load(sys.stdin); v=d['Parameter']['Value']; print(json.dumps(json.loads(v), indent=2) if v.startswith('{') else v)" + +# List parameters under a path +aws ssm get-parameters-by-path \ + --path "" \ + --recursive \ + --with-decryption \ + --region \ + | python3 -c "import sys,json; [print(p['Name'], '=', p['Value'][:80]) for p in json.load(sys.stdin)['Parameters']]" +``` + +--- + +## Schema Analysis + +After fetching a raw sample, run this analysis to produce a structured schema report: + +```bash +python3 -c " +import sys, json + +def infer_type(v): + if v is None: return 'null' + if isinstance(v, bool): return 'boolean' + if isinstance(v, int): return 'integer' + if isinstance(v, float): return 'float' + if isinstance(v, list): + if not v: return 'array (empty)' + return f'array of {infer_type(v[0])}' + if isinstance(v, dict): return 'object' + return 'string' + +def flatten_schema(d, prefix=''): + rows = [] + if isinstance(d, dict): + for k, v in d.items(): + full_key = f'{prefix}.{k}' if prefix else f'.{k}' + t = infer_type(v) + example = str(v)[:60] if not isinstance(v, (dict, list)) else '' + rows.append((full_key, t, example)) + if isinstance(v, dict): + rows.extend(flatten_schema(v, full_key)) + elif isinstance(v, list) and v and isinstance(v[0], dict): + rows.extend(flatten_schema(v[0], full_key + '[]')) + return rows + +raw = sys.stdin.read().strip() +try: + d = json.loads(raw) + if isinstance(d, list): + print(f'Top-level: array of {len(d)} items, showing first item') + d = d[0] + print() + print(f'{\"Field\":<40} {\"Type\":<20} {\"Example\"}') + print('-' * 90) + for field, typ, ex in flatten_schema(d): + print(f'{field:<40} {typ:<20} {ex}') +except Exception as e: + print(f'Could not parse as JSON: {e}') + print('Raw sample:') + print(raw[:500]) +" <<< '' +``` + +--- + +## Output Format + +Return a schema report in this format: + +``` +## Source Schema: — + +### Raw Sample (first record) +{ + "user_id": 42, + "event_type": "purchase", + "metadata": { + "session_id": "abc123", + "ip": "1.2.3.4" + }, + "items": [ + { "sku": "X100", "qty": 2, "price": 9.99 } + ], + "timestamp": "2024-03-01T12:00:00Z" +} + +### Schema +Field Type Example +------------------------------------------------------------------------------------------ +.user_id integer 42 +.event_type string purchase +.metadata object +.metadata.session_id string abc123 +.metadata.ip string 1.2.3.4 +.items array of object +.items[].sku string X100 +.items[].qty integer 2 +.items[].price float 9.99 +.timestamp string 2024-03-01T12:00:00Z + +### Suggested JQ Expressions + +# Extract all top-level fields +{ "user_id": .user_id, "event_type": .event_type, "timestamp": .timestamp } + +# Flatten metadata into top level +{ "user_id": .user_id, "event_type": .event_type, "session_id": .metadata.session_id } + +# Explode items array — one record per item +.items[] | { "user_id": (.user_id | tostring), "sku": .sku, "qty": .qty, "price": .price } +# (use explode: true on this jq task) + +# If records are nested under a key (e.g. .data | fromjson) +.data | fromjson | { ... } + +### Notes +- .items is an array — use explode: true on the jq task if you need one record per item +- .timestamp is a string (ISO 8601) — no conversion needed for most sinks +- .metadata.ip may be PII — confirm if it should be included in the output +``` + +--- + +## Error Handling + +| Error | Likely cause | Action | +|-------|-------------|--------| +| `curl: (6) Could not resolve host` | Wrong endpoint or no network | Ask user to verify URL | +| `curl: (22) HTTP 401` | Missing or wrong auth | Ask for correct credentials | +| `curl: (22) HTTP 403` | Auth works but no permission | Check API key scopes | +| `NoSuchBucket` | Wrong S3 bucket name | Ask user to verify | +| `AccessDenied` (S3/SQS/SSM) | IAM permissions missing | Tell user to check IAM | +| `Queue is empty` (SQS) | No messages currently in queue | Warn user — schema cannot be detected, ask for a sample payload manually | +| Kafka timeout | Wrong bootstrap server, auth, or empty topic | Try with `retry_limit: 1` probe pipeline | +| Response is not JSON | CSV, XML, plain text, or binary | Note the format and handle accordingly | + +If live detection fails, ask the user to paste a sample record manually and proceed with schema analysis from that. diff --git a/.claude/commands/check-aws.md b/.claude/commands/check-aws.md new file mode 100644 index 0000000..9214fc3 --- /dev/null +++ b/.claude/commands/check-aws.md @@ -0,0 +1,18 @@ +Check the current AWS environment and account status. Run these checks and report results: + +1. **AWS Identity** — Run `aws sts get-caller-identity` to confirm credentials are valid. Report account ID, ARN, and user/role name. + +2. **Account Type** — Check if the account appears to be sandbox/dev or production: + - Look at the account alias: `aws iam list-account-aliases` + - Check for Organizations info: `aws organizations describe-organization 2>/dev/null` + - Flag if the account ID or alias contains "sandbox", "dev", "test", or "staging" + +3. **Region** — Report the active region from `AWS_REGION`, `AWS_DEFAULT_REGION`, or `aws configure get region`. + +4. **Credential Type** — Report whether using: + - Environment variables (`AWS_ACCESS_KEY_ID`) + - Shared credentials file (`~/.aws/credentials`) + - SSO session (`aws sso login` profile) + - IAM role (instance/task role) + +Report a clear summary table. If any check fails, explain what's missing and how to fix it. diff --git a/.claude/commands/check-http.md b/.claude/commands/check-http.md new file mode 100644 index 0000000..b34fc1b --- /dev/null +++ b/.claude/commands/check-http.md @@ -0,0 +1,33 @@ +Verify that an HTTP API endpoint is reachable and responding. The user will provide a URL and optionally auth details. + +Run these checks: + +1. **Endpoint reachable** — `curl -s -o /dev/null -w "%{http_code} %{time_total}s" --max-time 10 ` + - Report: status code, response time, redirect chain (if any) + +2. **Response preview** — `curl -s --max-time 10 | head -c 2000` + - If JSON: pretty-print and show structure + - If HTML/XML: note the content type + - If empty: flag it + +3. **Auth test** — If the user provides auth details: + - Bearer: `curl -s -H "Authorization: Bearer " ` + - API key: `curl -s -H "X-Api-Key: " ` + - Basic: `curl -s -u : ` + - Report whether auth succeeds (2xx) or fails (401/403) + +4. **Pagination check** — If the response is JSON: + - Look for common pagination fields: `next`, `next_page`, `next_url`, `links.next`, `cursor`, `offset`, `page` + - Suggest the `next_page` JQ expression for the pipeline + +5. **TLS check** — `curl -vI --max-time 5 2>&1 | grep -E "SSL|TLS|certificate"` + - Report TLS version and certificate validity + - Flag if using `http://` instead of `https://` + +6. **Pipeline implications** — Based on findings: + - Whether `method: GET` or `POST` is needed + - Suggested `next_page` expression if paginated + - Whether `max_retries` should be increased (slow response) + - Whether `expected_statuses` needs adjusting + +Report a clear summary. If connection fails, explain common causes (DNS, firewall, TLS, auth). diff --git a/.claude/commands/check-kafka.md b/.claude/commands/check-kafka.md new file mode 100644 index 0000000..e4f704f --- /dev/null +++ b/.claude/commands/check-kafka.md @@ -0,0 +1,28 @@ +Verify that a Kafka broker and topic are reachable. The user will provide bootstrap server and topic name. + +Run these checks: + +1. **Connectivity** — Check if the broker is reachable: + - `nc -zv 2>&1` (extract host/port from bootstrap_server) + - If unreachable, suggest checking VPN, security groups, or firewall rules + +2. **Topic exists** — Try to list/describe the topic: + - With kcat: `kcat -b -L -t 2>&1 | head -20` + - Without kcat: `echo "Topic check requires kcat — install with: brew install kcat"` + +3. **Topic metadata** (if kcat available) — Report: + - Partition count + - Replica count + - Whether the topic has messages (try consuming 1 with timeout) + +4. **Auth check** — If the user mentions SCRAM/SASL/TLS: + - Test with kcat using provided auth: `kcat -b -t -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username= -X sasl.password= -L 2>&1 | head -10` + - If no kcat, suggest a minimal probe pipeline to test connectivity + +5. **Pipeline implications** — Based on findings, suggest: + - Whether `server_auth_type: tls` is needed + - Whether `user_auth_type: scram` or `sasl` is needed + - A sensible `group_id` based on the topic name + - Whether `retry_limit` should be set (empty topic) + +Report a clear summary. If connection fails, explain common causes (wrong port, TLS required, auth mismatch). diff --git a/.claude/commands/check-s3.md b/.claude/commands/check-s3.md new file mode 100644 index 0000000..15d5abc --- /dev/null +++ b/.claude/commands/check-s3.md @@ -0,0 +1,28 @@ +Verify that an S3 bucket/path exists and is accessible. The user will provide a bucket name or full S3 path. + +Run these checks: + +1. **Bucket exists** — `aws s3api head-bucket --bucket ` + +2. **Bucket region** — `aws s3api get-bucket-location --bucket ` + - Report the actual region (important for pipeline `region` field) + +3. **Path check** — If the user gave a full path (`s3://bucket/prefix/`): + - List objects: `aws s3 ls --max-items 5` + - Report count and sample filenames + +4. **Bucket properties** — Report: + - Versioning: `aws s3api get-bucket-versioning --bucket ` + - Encryption: `aws s3api get-bucket-encryption --bucket ` (may need KMS permissions) + - Public access block: `aws s3api get-public-access-block --bucket ` + +5. **Write test** (only if user asks) — Check if write is possible: + - `aws s3api put-object --bucket --key _caterpillar_write_test --body /dev/null` + - Then delete it: `aws s3api delete-object --bucket --key _caterpillar_write_test` + +6. **Pipeline implications** — Based on findings, suggest: + - The correct `region` value for the pipeline `file` task + - Whether `{{ macro "uuid" }}` or `{{ macro "timestamp" }}` is needed in write paths + - Whether `success_file: true` is appropriate + +Report a clear summary. If access is denied, list the IAM permissions needed (`s3:GetObject`, `s3:PutObject`, `s3:ListBucket`). diff --git a/.claude/commands/check-sns.md b/.claude/commands/check-sns.md new file mode 100644 index 0000000..f530822 --- /dev/null +++ b/.claude/commands/check-sns.md @@ -0,0 +1,24 @@ +Verify that an SNS topic exists and is accessible. The user will provide a topic ARN or topic name. + +Run these checks: + +1. **Topic exists** — `aws sns get-topic-attributes --topic-arn ` + - If the user gave a name: list topics and find it: `aws sns list-topics` then match by name + +2. **Topic type** — Report whether it's standard or FIFO (ARN ends in `.fifo`). + +3. **Key attributes** — Report: + - `TopicArn` + - `DisplayName` + - `SubscriptionsConfirmed` / `SubscriptionsPending` + - `KmsMasterKeyId` (if encrypted) + - For FIFO: `FifoTopic`, `ContentBasedDeduplication` + +4. **Subscriptions** — `aws sns list-subscriptions-by-topic --topic-arn ` + - Report protocol and endpoint for each (SQS, Lambda, email, HTTP, etc.) + +5. **Pipeline implications** — Based on attributes, suggest: + - Whether `message_group_id` is needed (FIFO topic) + - Note that `sns` is a terminal sink — no tasks can follow it + +Report a clear summary. If the topic doesn't exist or access is denied, explain the error and what IAM permissions are needed (`sns:GetTopicAttributes`, `sns:Publish`). diff --git a/.claude/commands/check-sqs.md b/.claude/commands/check-sqs.md new file mode 100644 index 0000000..af83e23 --- /dev/null +++ b/.claude/commands/check-sqs.md @@ -0,0 +1,25 @@ +Verify that an SQS queue exists and is accessible. The user will provide a queue URL or queue name. + +Run these checks: + +1. **Queue exists** — `aws sqs get-queue-attributes --queue-url --attribute-names All` + - If the user gave a name instead of URL: `aws sqs get-queue-url --queue-name ` + +2. **Queue type** — Report whether it's standard or FIFO (URL ends in `.fifo`). + +3. **Key attributes** — Report: + - `ApproximateNumberOfMessages` (current depth) + - `ApproximateNumberOfMessagesNotVisible` (in-flight) + - `VisibilityTimeout` + - `MessageRetentionPeriod` + - `MaximumMessageSize` + - For FIFO: `ContentBasedDeduplication`, `FifoQueue` + +4. **Dead letter queue** — Check `RedrivePolicy` for a DLQ. If present, report the DLQ ARN. + +5. **Pipeline implications** — Based on the queue attributes, suggest: + - Whether `exit_on_empty: true` makes sense (if queue has messages vs empty) + - Whether `message_group_id` is needed (FIFO) + - If visibility timeout is low, warn about reprocessing risk + +Report a clear summary. If the queue doesn't exist or access is denied, explain the error and what IAM permissions are needed. diff --git a/.claude/commands/check-ssm.md b/.claude/commands/check-ssm.md new file mode 100644 index 0000000..190efb1 --- /dev/null +++ b/.claude/commands/check-ssm.md @@ -0,0 +1,20 @@ +Verify that AWS SSM Parameter Store paths exist and are readable. The user will provide one or more SSM parameter paths. + +Run these checks: + +1. **Parameter exists** — For each path: + - `aws ssm get-parameter --name --with-decryption 2>&1` + - Report: name, type (String/SecureString/StringList), version, last modified date + +2. **Path prefix check** — If the user gives a prefix path (e.g. `/prod/kafka/`): + - `aws ssm get-parameters-by-path --path --recursive --max-results 10` + - List all parameters found under that prefix (names only, not values) + +3. **Value preview** — For non-SecureString params, show the value. For SecureString, show `[ENCRYPTED]` and confirm decryption works. + +4. **Pipeline implications** — Based on findings: + - Confirm the paths match what the pipeline uses in `{{ secret "/path" }}` + - Flag any paths that don't exist — the pipeline will fail at init + - Note if any are StringList type — may need parsing in the pipeline + +Report a clear summary. If access is denied, explain the IAM permissions needed (`ssm:GetParameter`, `ssm:GetParametersByPath`, `kms:Decrypt`). diff --git a/.claude/hooks/aws-env-check.sh b/.claude/hooks/aws-env-check.sh new file mode 100755 index 0000000..1943e39 --- /dev/null +++ b/.claude/hooks/aws-env-check.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Trigger: PostStartup +# Purpose: Verify AWS environment is ready when Claude initializes. +# Shows account info or warns if SSO session is expired. + +set -euo pipefail + +PROFILE="${AWS_PROFILE:-sandbox}" + +# Check if we can reach AWS +if ! command -v aws &>/dev/null; then + echo "WARN: aws CLI not installed" + exit 0 +fi + +if aws sts get-caller-identity --profile "$PROFILE" &>/dev/null; then + ACCOUNT_ID=$(aws sts get-caller-identity --profile "$PROFILE" --query 'Account' --output text 2>/dev/null) + ACCOUNT_ALIAS=$(aws iam list-account-aliases --profile "$PROFILE" --query 'AccountAliases[0]' --output text 2>/dev/null || echo "N/A") + echo "AWS environment ready — profile: $PROFILE, account: $ACCOUNT_ALIAS ($ACCOUNT_ID)" +else + echo "WARN: AWS SSO session expired for profile '$PROFILE'. Run: aws sso login --profile $PROFILE" +fi + +exit 0 diff --git a/.claude/hooks/preflight-check.sh b/.claude/hooks/preflight-check.sh new file mode 100755 index 0000000..0f11fe6 --- /dev/null +++ b/.claude/hooks/preflight-check.sh @@ -0,0 +1,228 @@ +#!/usr/bin/env bash +# Trigger: PreToolUse Bash +# Purpose: Before running ./caterpillar or .claude/scripts/run-pipeline.sh, check: +# 1. Binary is built +# 2. Pipeline file exists +# 3. AWS account is sandbox (BLOCK if not) +# 4. Pipeline has no non-sandbox resources (BLOCK if found) +# 5. All {{ env "VAR" }} references are set +# 6. AWS credentials present if pipeline uses AWS tasks +# Exit 2 to BLOCK the run. + +set -euo pipefail + +INPUT=$(cat) + +# Only intercept caterpillar/run-pipeline commands +COMMAND=$(echo "$INPUT" | python3 -c "import sys, json; d=json.load(sys.stdin); print(d.get('tool_input', {}).get('command', ''))" 2>/dev/null || echo "") + +if [[ "$COMMAND" != *"./caterpillar -conf"* ]] && [[ "$COMMAND" != *"caterpillar -conf"* ]] && [[ "$COMMAND" != *"run-pipeline.sh"* ]]; then + exit 0 +fi + +# Extract pipeline file path +PIPELINE_FILE=$(echo "$COMMAND" | grep -oE '(\-conf\s+|run-pipeline\.sh\s+)\S+' | awk '{print $NF}') + +echo "--- Preflight Check: $PIPELINE_FILE ---" + +ERRORS=0 +BLOCKED=false + +# ── 1. Binary check ────────────────────────────────────────────── + +if [ ! -f "./caterpillar" ]; then + echo "ERROR binary ./caterpillar not found — run: go build -o caterpillar cmd/caterpillar/caterpillar.go" + ERRORS=$((ERRORS + 1)) +else + echo "OK binary exists" +fi + +# ── 2. Pipeline file check ─────────────────────────────────────── + +if [ -z "$PIPELINE_FILE" ]; then + echo "ERROR could not parse pipeline file from command: $COMMAND" + exit 2 +fi + +if [ ! -f "$PIPELINE_FILE" ]; then + echo "ERROR pipeline file not found: $PIPELINE_FILE" + ERRORS=$((ERRORS + 1)) +else + echo "OK pipeline file exists: $PIPELINE_FILE" +fi + +# ── 3. Sandbox account check (BLOCKING) ───────────────────────── + +if command -v aws &>/dev/null; then + ACCOUNT_ALIAS=$(aws iam list-account-aliases --query 'AccountAliases[0]' --output text 2>/dev/null || echo "NONE") + ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text 2>/dev/null || echo "UNKNOWN") + + SANDBOX_PATTERNS="sandbox|dev|test|staging|nonprod" + + ALIAS_OK=false + ID_OK=false + + if echo "$ACCOUNT_ALIAS" | grep -qiE "$SANDBOX_PATTERNS"; then + ALIAS_OK=true + fi + if echo "$ACCOUNT_ID" | grep -qiE "$SANDBOX_PATTERNS"; then + ID_OK=true + fi + + if [ "$ALIAS_OK" = true ]; then + echo "OK sandbox account: $ACCOUNT_ALIAS ($ACCOUNT_ID)" + elif [ "$ACCOUNT_ALIAS" = "NONE" ] && [ "$ACCOUNT_ID" = "UNKNOWN" ]; then + echo "WARN could not determine AWS account — no credentials or no access to IAM" + else + echo "BLOCK account '$ACCOUNT_ALIAS' ($ACCOUNT_ID) is NOT sandbox" + echo " Only sandbox/dev/test/staging accounts are allowed for pipeline execution" + echo " Switch account: export AWS_PROFILE=" + BLOCKED=true + fi +else + echo "WARN aws CLI not installed — cannot verify sandbox account" +fi + +# ── 4. Non-sandbox resource check (BLOCKING) ──────────────────── + +if [ -f "$PIPELINE_FILE" ]; then + NON_SANDBOX_RESOURCES=() + + python3 -c " +import sys, yaml, re + +SANDBOX_RE = re.compile(r'(sandbox|dev|test|staging|nonprod)', re.IGNORECASE) + +RESOURCE_FIELDS = { + 'queue_url', 'topic_arn', 'bootstrap_server', 'endpoint', +} + +# Fields where s3:// paths live +PATH_FIELDS = {'path'} + +with open('$PIPELINE_FILE') as f: + data = yaml.safe_load(f) + +if not isinstance(data, dict) or 'tasks' not in data: + sys.exit(0) + +flagged = [] +for i, task in enumerate(data.get('tasks', [])): + name = task.get('name', f'task#{i+1}') + ttype = task.get('type', '') + + for field in RESOURCE_FIELDS: + val = task.get(field, '') + if not val or not isinstance(val, str): + continue + # Skip template-only values — can't evaluate at scan time + if val.strip().startswith('{{') and val.strip().endswith('}}'): + continue + if not SANDBOX_RE.search(val): + flagged.append(f' task \"{name}\" → {field}: {val}') + + # Check s3:// paths + for field in PATH_FIELDS: + val = task.get(field, '') + if not val or not isinstance(val, str): + continue + if val.startswith('s3://'): + if val.strip().startswith('{{'): + continue + if not SANDBOX_RE.search(val): + flagged.append(f' task \"{name}\" → {field}: {val}') + + # Check secret paths for /prod/ prefix + for field, val in task.items(): + if isinstance(val, str) and '{{ secret' in val: + if '/prod/' in val and not SANDBOX_RE.search(val): + flagged.append(f' task \"{name}\" → {field}: {val} (prod SSM path)') + +if flagged: + print('NON_SANDBOX_FOUND') + for f in flagged: + print(f) +else: + print('ALL_SANDBOX') +" 2>/dev/null | { + FIRST_LINE=true + while IFS= read -r line; do + if [ "$FIRST_LINE" = true ]; then + FIRST_LINE=false + if [ "$line" = "NON_SANDBOX_FOUND" ]; then + echo "BLOCK non-sandbox resources detected in pipeline:" + BLOCKED=true + else + echo "OK all resources appear to be sandbox" + fi + else + echo "$line" + fi + done + } + + if [ "$BLOCKED" = true ]; then + echo "" + echo " Use mock data instead: ask user for sample input, replace source with local file, sink with echo" + fi +fi + +# ── 5. Env var check ───────────────────────────────────────────── + +if [ -f "$PIPELINE_FILE" ] && [ "$BLOCKED" = false ]; then + MISSING_VARS=() + ENV_VARS=$(grep -oE '\{\{ env "([^"]+)" \}\}' "$PIPELINE_FILE" | grep -oE '"[^"]+"' | tr -d '"' | sort -u 2>/dev/null || true) + + for VAR in $ENV_VARS; do + if [ -z "${!VAR:-}" ]; then + MISSING_VARS+=("$VAR") + fi + done + + if [ ${#MISSING_VARS[@]} -gt 0 ]; then + echo "WARN env vars referenced but not set:" + for VAR in "${MISSING_VARS[@]}"; do + echo " export $VAR=" + done + elif [ -n "$ENV_VARS" ]; then + echo "OK all env vars set" + fi +fi + +# ── 6. AWS credentials check ───────────────────────────────────── + +if [ -f "$PIPELINE_FILE" ] && [ "$BLOCKED" = false ]; then + AWS_TASKS=$(grep -E 'type:\s*(sqs|sns|aws_parameter_store|file)' "$PIPELINE_FILE" || true) + S3_PATHS=$(grep -E 'path:\s*.*s3://' "$PIPELINE_FILE" || true) + SECRET_REFS=$(grep -oE '\{\{ secret "[^"]+" \}\}' "$PIPELINE_FILE" || true) + + if [ -n "$AWS_TASKS" ] || [ -n "$S3_PATHS" ] || [ -n "$SECRET_REFS" ]; then + if [ -z "${AWS_ACCESS_KEY_ID:-}" ] && [ -z "${AWS_PROFILE:-}" ]; then + if [ ! -f "$HOME/.aws/credentials" ] && [ ! -f "$HOME/.aws/config" ]; then + echo "WARN pipeline uses AWS but no credentials found" + fi + else + echo "OK AWS credentials present" + fi + + if [ -z "${AWS_REGION:-}" ] && [ -z "${AWS_DEFAULT_REGION:-}" ]; then + echo "WARN AWS_REGION not set — defaults to us-west-2" + fi + fi +fi + +# ── Verdict ────────────────────────────────────────────────────── + +echo "" +if [ "$BLOCKED" = true ]; then + echo "BLOCKED — non-sandbox environment or resources detected. Pipeline will not run." + echo " Generate a mock test pipeline instead." + exit 2 +elif [ $ERRORS -gt 0 ]; then + echo "BLOCKED — $ERRORS preflight error(s). Fix before running." + exit 2 +else + echo "Preflight passed — running pipeline..." +fi + +exit 0 diff --git a/.claude/hooks/run-summary.sh b/.claude/hooks/run-summary.sh new file mode 100755 index 0000000..605ecda --- /dev/null +++ b/.claude/hooks/run-summary.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Trigger: PostToolUse Bash +# Purpose: After ./caterpillar or .claude/scripts/run-pipeline.sh runs, report: +# status, record count, errors, suggestions, JSON output validation. + +set -euo pipefail + +INPUT=$(cat) + +COMMAND=$(echo "$INPUT" | python3 -c "import sys, json; d=json.load(sys.stdin); print(d.get('tool_input', {}).get('command', ''))" 2>/dev/null || echo "") + +if [[ "$COMMAND" != *"./caterpillar -conf"* ]] && [[ "$COMMAND" != *"caterpillar -conf"* ]] && [[ "$COMMAND" != *"run-pipeline.sh"* ]]; then + exit 0 +fi + +OUTPUT=$(echo "$INPUT" | python3 -c " +import sys, json +d = json.load(sys.stdin) +result = d.get('tool_response', {}) +if isinstance(result, str): + print(result) +elif isinstance(result, dict): + print(result.get('output', result.get('stdout', ''))) +" 2>/dev/null || echo "") + +EXIT_CODE=$(echo "$INPUT" | python3 -c " +import sys, json +d = json.load(sys.stdin) +result = d.get('tool_response', {}) +if isinstance(result, dict): + print(result.get('exit_code', result.get('returncode', 0))) +else: + print(0) +" 2>/dev/null || echo "0") + +PIPELINE_FILE=$(echo "$COMMAND" | grep -oE '(\-conf\s+|run-pipeline\.sh\s+)\S+' | awk '{print $NF}') + +echo "--- Run Summary: $PIPELINE_FILE ---" + +# ── Status ─────────────────────────────────────────────────────── + +if [ "$EXIT_CODE" = "0" ]; then + echo "STATUS success (exit 0)" +else + echo "STATUS FAILED (exit $EXIT_CODE)" +fi + +# ── Record count ───────────────────────────────────────────────── + +if [ -n "$OUTPUT" ]; then + RECORD_COUNT=$(echo "$OUTPUT" | grep -v "^---" | grep -v "^error" | grep -v "^Task" | grep -v "^pipeline" | grep -v "^$" | grep -v "^Preflight" | grep -v "^OK" | grep -v "^nothing" | grep -v "^BLOCK" | grep -v "^STATUS" | grep -v "^WARN" | wc -l | tr -d ' ') + if [ "$RECORD_COUNT" -gt "0" ]; then + echo "RECORDS $RECORD_COUNT record(s) output" + fi +fi + +# ── Errors ─────────────────────────────────────────────────────── + +NON_FATAL=$(echo "$OUTPUT" | grep -E "^error in " || true) +if [ -n "$NON_FATAL" ]; then + echo "" + echo "NON-FATAL ERRORS:" + echo "$NON_FATAL" | while IFS= read -r line; do + echo " $line" + done +fi + +FATAL=$(echo "$OUTPUT" | grep -E "Task '.+' failed with error:" || true) +if [ -n "$FATAL" ]; then + echo "" + echo "FATAL ERRORS:" + echo "$FATAL" | while IFS= read -r line; do + echo " $line" + done +fi + +# ── Suggestions ────────────────────────────────────────────────── + +echo "$OUTPUT" | python3 -c " +import sys, re + +output = sys.stdin.read() +suggestions = [] + +patterns = { + 'task type is not supported': 'Fix task type — check hyphens vs underscores', + 'failed to initialize task': 'Init failure — check AWS credentials, region, SSM paths', + 'task not found': 'DAG references a task name not in tasks:', + 'context keys were not set': 'Add context: { key: \".jq\" } to upstream task', + 'malformed context template': 'Fix {{ context \"key\" }} syntax', + 'macro .* is not defined': 'Valid macros: timestamp, uuid, unixtime, microtimestamp', + 'nothing to do': 'tasks: list is empty', + 'invalid DAG groups': 'Fix DAG syntax', + 'connection refused': 'Cannot reach host — check server/endpoint/queue_url', + 'NoCredentialProviders': 'No AWS credentials — set AWS_ACCESS_KEY_ID or use IAM role', + 'AccessDenied': 'IAM permissions insufficient — run pipeline-permissions agent', + 'ResourceNotFoundException': 'SSM parameter path not found', + 'batch_flush_interval': 'batch_flush_interval must be < timeout for kafka write', +} + +for pattern, suggestion in patterns.items(): + if re.search(pattern, output, re.IGNORECASE): + suggestions.append(suggestion) + +if suggestions: + print('') + print('SUGGESTIONS:') + for s in suggestions: + print(f' -> {s}') +" 2>/dev/null || true + +# ── JSON output validation ─────────────────────────────────────── + +if [ "$EXIT_CODE" = "0" ] && [ -f "$PIPELINE_FILE" ]; then + JSON_SINKS=$(grep -E 'path:.*\.json' "$PIPELINE_FILE" | grep -v 's3://' | grep -oE "'[^']+'" | tr -d "'" | grep -v 'http' || true) + + if [ -n "$JSON_SINKS" ]; then + echo "" + echo "JSON OUTPUT:" + for SINK_PATH in $JSON_SINKS; do + BASE_DIR=$(dirname "$SINK_PATH") + BASE_NAME=$(basename "$SINK_PATH" | sed 's/{{ macro "[^"]*" }}/.*/g') + LATEST_FILE=$(ls -t "${BASE_DIR}/"${BASE_NAME} 2>/dev/null | head -1 || true) + + if [ -n "$LATEST_FILE" ] && [ -f "$LATEST_FILE" ]; then + python3 -c " +import json +path = '$LATEST_FILE' +try: + with open(path) as f: + data = json.load(f) + with open(path, 'w') as f: + json.dump(data, f, indent=2) + f.write('\n') + if isinstance(data, list): + print(f'OK {path} — JSON array ({len(data)} records) — pretty-printed') + else: + print(f'OK {path} — JSON object — pretty-printed') +except json.JSONDecodeError as e: + print(f'ERROR {path} — invalid JSON: {e}') + print(f' Tip: use jq [.items[] | {{...}}] to produce a JSON array') +" 2>/dev/null || true + fi + done + fi +fi + +# ── Next step ──────────────────────────────────────────────────── + +echo "" +if [ "$EXIT_CODE" = "0" ]; then + echo "Next: run pipeline-review before promoting to production" +else + echo "Next: run pipeline-debugger for diagnosis" +fi + +exit 0 diff --git a/.claude/hooks/validate-on-save.sh b/.claude/hooks/validate-on-save.sh new file mode 100755 index 0000000..8dcd695 --- /dev/null +++ b/.claude/hooks/validate-on-save.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Trigger: PostToolUse Write|Edit +# Purpose: When a .yaml file is written or edited, validate: +# 1. Valid YAML syntax +# 2. Pipeline structure (tasks key, task types, required fields) +# 3. Hardcoded credentials +# 4. Non-sandbox resource references (warn, don't block) + +set -euo pipefail + +INPUT=$(cat) + +FILE_PATH=$(echo "$INPUT" | python3 -c "import sys, json; d=json.load(sys.stdin); print(d.get('tool_input', {}).get('file_path', ''))" 2>/dev/null || echo "") + +# Only process .yaml or .yml files +if [[ "$FILE_PATH" != *.yaml ]] && [[ "$FILE_PATH" != *.yml ]]; then + exit 0 +fi + +# Skip non-pipeline files +if [[ "$FILE_PATH" == *".github"* ]] || [[ "$FILE_PATH" == *"settings"* ]]; then + exit 0 +fi + +echo "--- Pipeline Validation: $FILE_PATH ---" + +python3 -c " +import sys, yaml, re + +# ── Config ─────────────────────────────────────────────────────── + +SUPPORTED_TYPES = { + 'archive', 'aws_parameter_store', 'compress', 'converter', 'delay', + 'echo', 'file', 'flatten', 'heimdall', 'http_server', 'http', + 'join', 'jq', 'kafka', 'replace', 'sample', 'sns', 'split', 'sqs', 'xpath' +} + +SOURCE_TYPES = {'file', 'kafka', 'sqs', 'http', 'http_server', 'aws_parameter_store'} + +REQUIRED_FIELDS = { + 'file': ['path'], 'kafka': ['bootstrap_server', 'topic'], + 'sqs': ['queue_url'], 'http': ['endpoint'], 'http_server': ['port'], + 'sns': ['topic_arn'], 'aws_parameter_store': ['path'], 'jq': ['path'], + 'xpath': ['expression'], 'compress': ['format'], + 'archive': ['format', 'mode'], 'sample': ['strategy', 'value'], + 'delay': ['duration'], 'join': ['number'], +} + +CREDENTIAL_FIELDS = {'password', 'token', 'api_key', 'consumer_secret', 'token_secret'} + +RESOURCE_FIELDS = {'queue_url', 'topic_arn', 'bootstrap_server', 'endpoint'} + +SANDBOX_RE = re.compile(r'(sandbox|dev|test|staging|nonprod)', re.IGNORECASE) + +# ── Parse ──────────────────────────────────────────────────────── + +try: + with open('$FILE_PATH') as f: + data = yaml.safe_load(f) + if data is None: + print('WARN empty file') + sys.exit(0) +except yaml.YAMLError as e: + print(f'ERROR invalid YAML syntax: {e}') + sys.exit(1) + +print('OK YAML syntax valid') + +if not isinstance(data, dict) or 'tasks' not in data: + print('ERROR missing top-level tasks: key') + sys.exit(1) + +tasks = data.get('tasks', []) +if not tasks: + print('WARN tasks list is empty') + sys.exit(0) + +errors = [] +warnings = [] + +# ── Validate tasks ─────────────────────────────────────────────── + +names = [] +for i, task in enumerate(tasks): + pos = i + 1 + name = task.get('name', f'') + ttype = task.get('type', '') + + # Duplicate names + if name in names: + errors.append(f'ERROR task #{pos} \"{name}\": duplicate name') + names.append(name) + + # Missing type + if not ttype: + errors.append(f'ERROR task #{pos} \"{name}\": missing type field') + continue + + # Hyphen instead of underscore + if ttype not in SUPPORTED_TYPES: + if ttype.replace('-', '_') in SUPPORTED_TYPES: + errors.append(f'ERROR task #{pos} \"{name}\": type \"{ttype}\" uses hyphens — use underscores') + else: + errors.append(f'ERROR task #{pos} \"{name}\": type \"{ttype}\" is not supported') + continue + + # First task must be source + if i == 0 and ttype not in SOURCE_TYPES: + errors.append(f'ERROR task #1 \"{name}\": type \"{ttype}\" cannot be first — must be a source') + + # Required fields + for field in REQUIRED_FIELDS.get(ttype, []): + if field not in task: + errors.append(f'ERROR task #{pos} \"{name}\" ({ttype}): missing required field \"{field}\"') + + # Hardcoded credentials + for field in CREDENTIAL_FIELDS: + val = task.get(field, '') + if val and isinstance(val, str) and not val.strip().startswith('{{'): + errors.append(f'ERROR task #{pos} \"{name}\": \"{field}\" appears hardcoded — use {{{{ secret }}}} or {{{{ env }}}}') + + # SQS max_messages + if ttype == 'sqs' and task.get('max_messages', 0) > 10: + errors.append(f'ERROR task #{pos} \"{name}\" (sqs): max_messages cannot exceed 10') + + # echo/sns not first + if ttype in ('echo', 'sns') and i == 0: + errors.append(f'ERROR task #1 \"{name}\": {ttype} requires upstream — cannot be first') + + # Kafka batch_flush_interval vs timeout + if ttype == 'kafka' and 'batch_flush_interval' in task and 'timeout' in task: + warnings.append(f'WARN task #{pos} \"{name}\" (kafka): verify batch_flush_interval < timeout') + + # ── Non-sandbox resource check ─────────────────────────────── + + for field in RESOURCE_FIELDS: + val = task.get(field, '') + if not val or not isinstance(val, str): + continue + if val.strip().startswith('{{') and val.strip().endswith('}}'): + continue + if not SANDBOX_RE.search(val): + warnings.append(f'WARN task #{pos} \"{name}\": {field} does not appear to be sandbox — will require mock testing') + + # S3 path check + path_val = task.get('path', '') + if isinstance(path_val, str) and path_val.startswith('s3://'): + if not path_val.strip().startswith('{{') and not SANDBOX_RE.search(path_val): + warnings.append(f'WARN task #{pos} \"{name}\": S3 path does not appear to be sandbox — will require mock testing') + + # Prod SSM secret paths + for field, val in task.items(): + if isinstance(val, str) and '{{ secret' in val and '/prod/' in val: + warnings.append(f'WARN task #{pos} \"{name}\": {field} uses /prod/ SSM path — will require mock testing') + +# ── Output ─────────────────────────────────────────────────────── + +for w in warnings: + print(w) +for e in errors: + print(e) + +if errors: + print(f'\n{len(errors)} error(s) found — fix before running') + sys.exit(1) +else: + print(f'OK {len(tasks)} tasks valid') +" + +EXIT_CODE=$? +if [ $EXIT_CODE -eq 0 ]; then + echo "OK pipeline looks good" +else + echo "" + echo "Run pipeline-lint agent for a detailed report." +fi + +exit 0 # never block the write — just inform diff --git a/.claude/rules/pipeline-authoring.md b/.claude/rules/pipeline-authoring.md new file mode 100644 index 0000000..d0471e3 --- /dev/null +++ b/.claude/rules/pipeline-authoring.md @@ -0,0 +1,107 @@ +--- +description: Pipeline authoring rules — structure, naming, constraints, and production safeguards. +globs: "**/*.yaml,**/*.yml" +--- + +# Pipeline Authoring Rules + +## Pipeline Structure + +- First task must be a source: `file`, `kafka`, `sqs`, `http`, `http_server`, `aws_parameter_store`. +- Last task must be a sink: `file`, `kafka`, `sqs`, `sns`, `echo`. +- Transforms (`jq`, `split`, `join`, `replace`, `flatten`, `xpath`, `converter`, `compress`, `archive`, `sample`, `delay`) must sit between source and sink — never first. +- Every pipeline must have a natural termination point — avoid infinite-polling pipelines in batch jobs. + +## Auto-Detect Role + +`file`, `kafka`, `sqs`, `http` auto-detect source vs sink based on position. First task = source (read mode); has upstream = sink (write mode). + +## Naming + +- Task `name` must be unique within a pipeline. +- Use descriptive snake_case names: `read_from_sqs`, `transform_payload`, `write_to_s3`. +- Avoid generic names like `task1`, `step2`, `process`. +- Pipeline filenames should reflect their purpose: `kafka_to_s3.yaml`, not `pipeline1.yaml`. +- Task `type` values use underscores: `aws_parameter_store`, `http_server` — not hyphens. + +## Template Functions + +Use these in any string field value: + +| Function | When resolved | +|----------|--------------| +| `{{ env "VAR" }}` | once at pipeline init | +| `{{ secret "/ssm/path" }}` | once at pipeline init | +| `{{ macro "timestamp" }}` | per record | +| `{{ macro "uuid" }}` | per record | +| `{{ macro "unixtime" }}` | per record | +| `{{ macro "microtimestamp" }}` | per record | +| `{{ context "key" }}` | per record — value set by upstream task's `context:` block | + +- `{{ env }}` and `{{ secret }}` are static — do not use where per-record dynamic values are needed. +- Nested templates are not supported — `{{ secret "{{ env "X" }}" }}` will fail. +- Valid macro names: `timestamp`, `uuid`, `unixtime`, `microtimestamp`. + +## Error Handling + +- Add `fail_on_error: true` to source tasks — a silent source failure with exit code 0 is a false success. +- Add `fail_on_error: true` to any task that calls external services in critical pipelines. + +## Context Variables + +- Set context keys in the same task that reads the data, close to the source. +- Every `{{ context "key" }}` reference must have a matching `context: { key: ".jq_expr" }` in an upstream task. +- Do not reference a context key before it is set. + +## Source-Specific Rules + +Before tuning source fields or writing transforms for a new source, **sample one record and infer schema first** — see `.claude/rules/source-schema-first.md` (and the `source-schema-detector` agent). + +**Kafka** +- Always set `group_id` in production — without it, offsets are not committed and messages may be reprocessed. +- `batch_flush_interval` must be less than `timeout` in write mode. +- Do not use `user_auth_type: mtls` — not implemented, will error at runtime. + +**SQS** +- Set `exit_on_empty: true` for batch jobs that should terminate when the queue drains. +- FIFO queues (URL ends in `.fifo`) require `message_group_id` in write mode. +- `max_messages` must be ≤ 10. + +**File / S3** +- S3 paths must have an explicit `region` field. +- Write-mode paths should use `{{ macro "uuid" }}` or `{{ macro "timestamp" }}` to avoid overwriting existing files. +- Do not use glob patterns in write mode. +- Add `success_file: true` when downstream systems need a completion signal. + +**HTTP** +- Set `max_retries` and `retry_delay` for unreliable external APIs. +- Pagination `next_page` expression must eventually return null/empty — verify there is a terminal condition. + +## JSON Output Format + +- Caterpillar's `jq` task always outputs **compact/minified JSON** (single line). It has no built-in pretty-print option. +- When writing multiple JSON records to a single file as a JSON array, wrap inside `jq` using `[.items[] | {...}]` — do **not** use `explode: true` + `join` + `replace` to reconstruct an array. That pattern produces malformed output. +- For NDJSON (one JSON object per line), use `explode: true` with no `join` and name the file `.ndjson`. +- Never use `join` + string manipulation to build JSON structure — always use `jq` for JSON construction. +- Always run pipelines via `.claude/scripts/run-pipeline.sh ` instead of `./caterpillar -conf` directly — the wrapper auto-detects new JSON output files and pretty-prints them after the run. + +## Sink-Specific Rules + +- Remove `echo` sinks before promoting a pipeline to production — replace with a real sink. +- `sns` is terminal — do not add tasks after it. + +## Readability + +- Group related fields together within a task block. +- Align multiline JQ `path:` expressions with consistent indentation using YAML block scalar (`|`). +- Long pipelines (10+ tasks) should have comment headers separating logical stages: `# --- Source ---`, `# --- Transform ---`, `# --- Sink ---`. +- Add a `#` comment on any non-obvious config choice to explain why. + +## Production Safeguards + +When editing an existing production pipeline, confirm with the user before: +- Changing `type`, `topic`, `queue_url`, `bootstrap_server`, `endpoint`, or `path` — these change what data flows where. +- Reordering tasks or removing `join`/`split` tasks — changes the downstream data shape. +- Changing `group_id` on a Kafka consumer — changes offset tracking. +- Changing `exit_on_empty` from `true` to `false` on SQS — turns a batch job into an infinite consumer. +- Renaming a context key that is referenced downstream with `{{ context "key" }}`. diff --git a/.claude/rules/pipeline-security.md b/.claude/rules/pipeline-security.md new file mode 100644 index 0000000..8ef9288 --- /dev/null +++ b/.claude/rules/pipeline-security.md @@ -0,0 +1,31 @@ +--- +description: Security rules for caterpillar pipeline YAML configs. +globs: "**/*.yaml,**/*.yml" +--- + +# Security Rules + +## Credentials + +- Never hardcode passwords, tokens, API keys, or secrets as literal values in pipeline YAML. +- Always use `{{ secret "/ssm/path" }}` for secrets stored in AWS SSM Parameter Store. +- Use `{{ env "VAR" }}` only for non-sensitive config (e.g. region, topic names). Secrets must use `{{ secret }}`. +- SSM paths must follow the pattern `///` — e.g. `/prod/kafka/password`, `/staging/api/token`. + +## Sensitive Fields + +These fields must always use `{{ secret }}` or `{{ env }}` — never a literal value: +- `password` +- `username` (when paired with a password) +- `token`, `api_key`, `consumer_secret`, `token_secret` +- `queue_url`, `bootstrap_server`, `endpoint`, `topic_arn` if they contain credentials or account-specific identifiers + +## HTTP + +- Production pipeline `endpoint` values must use `https://`, not `http://`. +- Authorization headers (`Authorization`, `X-Api-Key`) must use `{{ secret }}` or `{{ env }}`. + +## Files + +- Never commit pipeline YAML files that contain literal secrets — even in `test/` pipelines. +- If a secret is accidentally committed, flag it immediately so it can be rotated. diff --git a/.claude/rules/pipeline-testing.md b/.claude/rules/pipeline-testing.md new file mode 100644 index 0000000..d807b55 --- /dev/null +++ b/.claude/rules/pipeline-testing.md @@ -0,0 +1,84 @@ +--- +description: Rules for pipeline testing — environment safety, test file standards, and incremental approach. +globs: "test/pipelines/**/*.yaml,test/pipelines/**/*.yml" +--- + +# Pipeline Testing Rules + +## Environment Check — Always First (MANDATORY) + +Before running any pipeline against live AWS resources (SQS, SNS, S3, SSM, Kafka), verify the environment is sandbox: + +1. Run `aws sts get-caller-identity` to get the account ID. +2. Run `aws iam list-account-aliases` to get the account alias. +3. The account is sandbox/dev ONLY if the alias or account ID contains: `sandbox`, `dev`, `test`, `staging`, or `nonprod`. +4. **If the account is production or cannot be determined — REFUSE to run the pipeline. Do not proceed even if the user asks.** Tell the user to switch to a sandbox account first. +5. If the account is sandbox — proceed. + +Use `/project:check-aws` to run the full environment check. + +**Pipelines must only run against sandbox AWS accounts. Production execution is blocked.** + +## Non-Sandbox Resource Detection — Mock Before Run + +Before running any pipeline, scan the YAML for non-sandbox resources: + +1. **Detect non-sandbox references** — a resource is non-sandbox unless its URL, ARN, path, or hostname explicitly contains `sandbox`, `dev`, `test`, `staging`, or `nonprod`. Flag any task field that does NOT match: + - `queue_url` without a sandbox indicator + - `topic_arn` without a sandbox indicator + - `bootstrap_server` without a sandbox indicator + - `endpoint` without a sandbox indicator + - `path` with `s3://` without a sandbox indicator in the bucket name + - `{{ secret "..." }}` SSM paths that are not under `/sandbox/`, `/dev/`, `/test/`, or `/staging/` prefixes + +2. **If any non-sandbox resource is found — do NOT run the pipeline.** Instead: + - Tell the user which fields reference production resources. + - Ask the user to provide a **mock sample input** (paste JSON, CSV, or text). + - Save the mock input to `test/pipelines/samples/_mock.json`. + - Generate a **mock test pipeline** that: + - Replaces the production source with `type: file` reading the mock sample file. + - Replaces the production sink with `type: echo` (`only_data: true`). + - Keeps all transforms unchanged so the data flow logic is fully tested. + - Save the mock pipeline to `test/pipelines/_mock_test.yaml`. + - Run the mock pipeline and show the output to the user for verification. + +3. **Only after the mock test passes** — deliver the production pipeline YAML to the user for deployment through their own CI/CD process. + +**Only sandbox resources are allowed for local execution. Everything else is validated with mock data only.** + +## Test Pipeline Requirements + +- Every production pipeline must have a corresponding test pipeline in `test/pipelines/`. +- Test pipelines must use local file sources — not live Kafka, SQS, S3, or external HTTP APIs. +- Test pipelines must use `type: echo` with `only_data: true` as the sink — no real writes. +- Test pipelines must be runnable from the project root: `./caterpillar -conf test/pipelines/.yaml`. + +## Test Pipeline Naming + +- Name test pipelines after the feature they verify: `kafka_read_test.yaml`, `converter_csv_test.yaml`. +- For converter tests, place sample input and expected output files alongside the test pipeline in `test/pipelines/converter/`. + +## What a Good Test Pipeline Covers + +- Happy path: valid input produces expected output +- Edge cases: empty file, single record, record with missing fields +- Template functions used in the production pipeline (`{{ macro }}`, `{{ context }}`) should be exercised + +## Incremental Testing Approach + +Use `/pipeline-tester` to generate a test plan. The standard approach is: + +1. **Inspect source** — curl / aws cli / kcat to see real data shape before writing any pipeline. +2. **Capture sample** — save 10 real records to `test/pipelines/samples/` as a local file. +3. **Probe each transform** — test one transform at a time using the sample file as source + `echo` as sink. +4. **Chain forward** — add transforms one by one, verify output at each step. +5. **Verify sink** — write to a local file first, inspect shape before hitting the real sink. +6. **Smoke test** — run against the real sink with `sample: head limit: 3`. + +Sample data lives in `test/pipelines/samples/`. Probe pipelines live in `test/pipelines/probes/`. + +## Do Not + +- Do not use production queue URLs, Kafka topics, S3 buckets, or live API endpoints in test pipelines. +- Do not commit test pipelines that require AWS credentials or network access to run. +- Do not leave test pipelines that fail — a broken test pipeline is worse than no test. diff --git a/.claude/rules/source-schema-first.md b/.claude/rules/source-schema-first.md new file mode 100644 index 0000000..bb215f1 --- /dev/null +++ b/.claude/rules/source-schema-first.md @@ -0,0 +1,24 @@ +--- +description: When source connection details are known, the first step is to sample one record and infer schema before designing transforms or jq paths. +globs: "**/*" +--- + +# Source schema first (mandatory) + +As soon as you have **concrete source details** (HTTP endpoint and auth, SQS queue URL, Kafka bootstrap/topic, S3 or local path, SSM path, etc.), your **first** action before proposing transforms, `jq` expressions, `context:` keys, or sink field mappings is: + +1. **Pull at least one real record** from that source (or the closest safe peek: e.g. SQS with `visibility-timeout 0`, read-only S3 head/get of first line, `curl` sample, `kcat -c 1`, local `head`). +2. **Infer the schema** from that sample: field names, types, nesting, arrays, wrapper keys (e.g. `.items[]`), and whether the payload is JSON, CSV, or opaque text. + +## How to do it + +- Run **`.claude/scripts/check-source-schema.sh`** with the matching subcommand (`http`, `s3`, `sqs`, `file`, `ssm`, `ssm-path`, `kafka`, or pipe arbitrary bytes into `stdin`). It fetches one sample and prints pretty JSON plus an inferred field table. Use `--no-schema` for a raw-only preview. +- Or invoke the **`source-schema-detector`** agent; it mirrors the same flows in `.claude/agents/source-schema-detector.md`. +- If live access fails (empty queue, auth, network), **ask the user to paste one representative record** and run `... check-source-schema.sh stdin` on it (or `python3 .claude/scripts/lib/source_schema_report.py` on stdin). + +## Do not + +- Do **not** invent or assume field names and paths without a sample. +- Do **not** skip this step to “save time” when building or debugging pipelines that depend on payload shape. + +This applies in **all** conversations where source details appear — not only when using the interactive pipeline builder. diff --git a/.claude/scripts/aws-profile-setup.sh b/.claude/scripts/aws-profile-setup.sh new file mode 100755 index 0000000..14f8853 --- /dev/null +++ b/.claude/scripts/aws-profile-setup.sh @@ -0,0 +1,32 @@ +#!/bin/bash +set -e + +PROFILE="sandbox" + +# Parse arguments +while [[ $# -gt 0 ]]; do + case $1 in + --profile) + PROFILE="$2" + shift 2 + ;; + *) + echo "Unknown option: $1" + echo "Usage: $0 [--profile ] (default: sandbox)" + exit 1 + ;; + esac +done + +# Ensure AWS SSO session is active +if aws sts get-caller-identity --profile "$PROFILE" &>/dev/null; then + echo "AWS SSO session already active for profile: $PROFILE" +else + echo "AWS SSO session not active, logging in for profile: $PROFILE" + aws sso login --profile "$PROFILE" +fi + +# Export profile for subprocesses +export AWS_PROFILE="$PROFILE" + +echo "AWS profile '$PROFILE' is ready." diff --git a/.claude/scripts/check-source-schema.sh b/.claude/scripts/check-source-schema.sh new file mode 100755 index 0000000..6bf49f9 --- /dev/null +++ b/.claude/scripts/check-source-schema.sh @@ -0,0 +1,274 @@ +#!/usr/bin/env bash +# Fetch one sample from a pipeline source and print an inferred JSON schema. +# Usage: .claude/scripts/check-source-schema.sh [args] +# +# Subcommands: +# http [--method GET|POST] [--header 'K: V']... [--data 'body'|@file] [--bearer TOKEN] [--max-time SEC] +# s3 --region REGION [--lines N] (first N lines; default 1 for NDJSON) +# sqs --region REGION +# file [--csv] +# ssm --region REGION +# ssm-path --region REGION (first parameter value, get-parameters-by-path) +# kafka --broker HOST:PORT --topic TOPIC [-- ...extra kcat -X args] +# stdin [--label TEXT] (read payload from pipe; use after curl/aws yourself) +# +# Global options (any position): +# --no-schema only print fetched/raw body (no inferred table) +# --raw-only same as --no-schema +# -h, --help show this header + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPORTER="$SCRIPT_DIR/lib/source_schema_report.py" + +usage() { + sed -n '1,25p' "$0" | tail -n +2 + exit "${1:-1}" +} + +run_reporter() { + local label="$1" + if [[ "${NO_SCHEMA:-0}" == "1" ]]; then + cat + return + fi + python3 "$REPORTER" --label "$label" +} + +NO_SCHEMA=0 +WANT_HELP=0 +METHOD="GET" +MAX_TIME="10" +HEADERS=() +DATA="" +BEARER="" +REGION="" +LINES="1" +KAFKA_BROKER="" +KAFKA_TOPIC=() +CSV_MODE=0 +LABEL_OVERRIDE="" + +# Strip global flags from any position (remaining order preserved) +ARGS=() +for a in "$@"; do + case "$a" in + -h|--help) WANT_HELP=1 ;; + --no-schema|--raw-only) NO_SCHEMA=1 ;; + *) ARGS+=("$a") ;; + esac +done +if [[ ${#ARGS[@]} -gt 0 ]]; then + set -- "${ARGS[@]}" +else + set -- +fi + +if [[ $# -lt 1 ]]; then + [[ "$WANT_HELP" == 1 ]] && usage 0 || usage 1 +fi + +SUB="$1" +shift + +curl_http() { + local url="$1" + local -a cmd=(curl -sS --max-time "$MAX_TIME" -X "$METHOD") + local h + for h in "${HEADERS[@]}"; do + cmd+=(-H "$h") + done + if [[ -n "$BEARER" ]]; then + cmd+=(-H "Authorization: Bearer ${BEARER}") + fi + if [[ -n "$DATA" ]]; then + if [[ "$DATA" == @* ]]; then + cmd+=(-H "Content-Type: application/json" --data-binary "${DATA#@}") + else + cmd+=(-H "Content-Type: application/json" --data "$DATA") + fi + fi + cmd+=("$url") + "${cmd[@]}" +} + +case "$SUB" in + http) + [[ $# -ge 1 ]] || usage + URL="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --method) METHOD="$2"; shift 2 ;; + -X) METHOD="$2"; shift 2 ;; + --header|-H) HEADERS+=("$2"); shift 2 ;; + --data|-d) DATA="$2"; shift 2 ;; + --bearer) BEARER="$2"; shift 2 ;; + --max-time) MAX_TIME="$2"; shift 2 ;; + *) echo "Unknown http option: $1" >&2; usage ;; + esac + done + echo "Fetching: $METHOD $URL" >&2 + curl_http "$URL" | run_reporter "$URL" + ;; + + s3) + [[ $# -ge 1 ]] || usage + S3_URI="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --region) REGION="$2"; shift 2 ;; + --lines) LINES="$2"; shift 2 ;; + *) echo "Unknown s3 option: $1" >&2; usage ;; + esac + done + [[ -n "$REGION" ]] || { echo "s3: --region required" >&2; exit 1; } + echo "Reading first $LINES line(s) from $S3_URI (region $REGION)" >&2 + aws s3 cp "$S3_URI" - --region "$REGION" | head -n "$LINES" | run_reporter "$S3_URI" + ;; + + sqs) + [[ $# -ge 1 ]] || usage + QUEUE="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --region) REGION="$2"; shift 2 ;; + *) echo "Unknown sqs option: $1" >&2; usage ;; + esac + done + [[ -n "$REGION" ]] || { echo "sqs: --region required" >&2; exit 1; } + echo "Peeking 1 message (visibility 0): $QUEUE" >&2 + RAW=$(mktemp) + BODY_OUT=$(mktemp) + trap 'rm -f "$RAW" "$BODY_OUT"' EXIT + aws sqs receive-message \ + --queue-url "$QUEUE" \ + --max-number-of-messages 1 \ + --visibility-timeout 0 \ + --region "$REGION" \ + --output json >"$RAW" || exit $? + python3 -c " +import json, sys +with open(sys.argv[1], encoding='utf-8') as f: + d = json.load(f) +msgs = d.get('Messages') or [] +if not msgs: + sys.stderr.write('Queue empty or no messages available.\\n') + sys.exit(2) +with open(sys.argv[2], 'w', encoding='utf-8') as w: + w.write(msgs[0].get('Body', '')) +" "$RAW" "$BODY_OUT" || exit $? + run_reporter "$QUEUE" <"$BODY_OUT" + ;; + + file) + [[ $# -ge 1 ]] || usage + FILE_PATH="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --csv) CSV_MODE=1; shift ;; + *) echo "Unknown file option: $1" >&2; usage ;; + esac + done + [[ -f "$FILE_PATH" ]] || { echo "file not found: $FILE_PATH" >&2; exit 1; } + if [[ "$CSV_MODE" == 1 ]]; then + if [[ "$NO_SCHEMA" == 1 ]]; then + head -n 2 "$FILE_PATH" + else + python3 "$REPORTER" csv-file "$FILE_PATH" + fi + else + head -n 1 "$FILE_PATH" | run_reporter "$FILE_PATH" + fi + ;; + + ssm) + [[ $# -ge 1 ]] || usage + PARAM="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --region) REGION="$2"; shift 2 ;; + *) echo "Unknown ssm option: $1" >&2; usage ;; + esac + done + [[ -n "$REGION" ]] || { echo "ssm: --region required" >&2; exit 1; } + echo "get-parameter: $PARAM" >&2 + aws ssm get-parameter --name "$PARAM" --with-decryption --region "$REGION" --output json | + python3 -c " +import json, sys +d = json.load(sys.stdin) +v = d['Parameter']['Value'] +sys.stdout.write(v) +if not v.endswith('\n'): + sys.stdout.write('\n') +" | run_reporter "$PARAM" + ;; + + ssm-path) + [[ $# -ge 1 ]] || usage + PREFIX="$1" + shift + while [[ $# -gt 0 ]]; do + case "$1" in + --region) REGION="$2"; shift 2 ;; + *) echo "Unknown ssm-path option: $1" >&2; usage ;; + esac + done + [[ -n "$REGION" ]] || { echo "ssm-path: --region required" >&2; exit 1; } + echo "get-parameters-by-path: $PREFIX (first value)" >&2 + aws ssm get-parameters-by-path --path "$PREFIX" --recursive --with-decryption --region "$REGION" --output json | + python3 -c " +import json, sys +d = json.load(sys.stdin) +params = d.get('Parameters') or [] +if not params: + sys.stderr.write('No parameters under path.\\n') + sys.exit(2) +p = params[0] +name, val = p['Name'], p.get('Value') or '' +sys.stderr.write(f'Sample parameter: {name}\\n') +sys.stdout.write(val) +if val and not val.endswith('\n'): + sys.stdout.write('\n') +" | run_reporter "$PREFIX" + ;; + + kafka) + while [[ $# -gt 0 ]]; do + case "$1" in + --broker|-b) KAFKA_BROKER="$2"; shift 2 ;; + --topic|-t) KAFKA_TOPIC=(-t "$2"); shift 2 ;; + --) shift; break ;; + *) break ;; + esac + done + [[ -n "$KAFKA_BROKER" && ${#KAFKA_TOPIC[@]} -eq 2 ]] || { echo "kafka: --broker and --topic required" >&2; exit 1; } + if ! command -v kcat >/dev/null 2>&1; then + echo "kcat not found. Install kcat or use a caterpillar probe pipeline." >&2 + exit 1 + fi + echo "Consuming 1 message from ${KAFKA_TOPIC[1]} @ $KAFKA_BROKER" >&2 + # Remaining args passed to kcat (e.g. -X security.protocol=SASL_SSL ...) + kcat -b "$KAFKA_BROKER" "${KAFKA_TOPIC[@]}" -C -c 1 -e -f '%s\n' "$@" 2>/dev/null | run_reporter "kafka:${KAFKA_TOPIC[1]}" + ;; + + stdin) + while [[ $# -gt 0 ]]; do + case "$1" in + --label) LABEL_OVERRIDE="$2"; shift 2 ;; + *) echo "Unknown stdin option: $1" >&2; usage ;; + esac + done + run_reporter "${LABEL_OVERRIDE:-stdin}" + ;; + + *) + echo "Unknown subcommand: $SUB" >&2 + usage + ;; +esac diff --git a/.claude/scripts/ensure-sandbox.sh b/.claude/scripts/ensure-sandbox.sh new file mode 100755 index 0000000..8609f8a --- /dev/null +++ b/.claude/scripts/ensure-sandbox.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# Verifies AWS credentials are configured and the account is a sandbox/dev environment. +# Must pass before any pipeline runs against live AWS resources. +# Usage: source .claude/scripts/ensure-sandbox.sh + +set -euo pipefail + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +echo "============================================" +echo " AWS Sandbox Environment Check" +echo "============================================" +echo "" + +# --- 1. Check AWS credentials exist --- +echo -n "Checking AWS credentials... " +if ! IDENTITY=$(aws sts get-caller-identity 2>&1); then + echo -e "${RED}FAILED${NC}" + echo "" + echo "No valid AWS credentials found. Set up credentials using one of:" + echo "" + echo " Option 1: aws configure" + echo " Option 2: export AWS_ACCESS_KEY_ID=... && export AWS_SECRET_ACCESS_KEY=..." + echo " Option 3: aws sso login --profile " + echo "" + exit 1 +fi +echo -e "${GREEN}OK${NC}" + +ACCOUNT_ID=$(echo "$IDENTITY" | python3 -c "import sys,json; print(json.load(sys.stdin)['Account'])") +ARN=$(echo "$IDENTITY" | python3 -c "import sys,json; print(json.load(sys.stdin)['Arn'])") +echo " Account: $ACCOUNT_ID" +echo " ARN: $ARN" +echo "" + +# --- 2. Check region --- +echo -n "Checking AWS region... " +REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-}}" +if [ -z "$REGION" ]; then + REGION=$(aws configure get region 2>/dev/null || true) +fi +if [ -z "$REGION" ]; then + echo -e "${RED}FAILED${NC}" + echo "" + echo "No AWS region configured. Set it with:" + echo " export AWS_REGION=us-east-1" + echo "" + exit 1 +fi +echo -e "${GREEN}OK${NC} ($REGION)" +echo "" + +# --- 3. Check account is sandbox/dev --- +echo -n "Checking account type... " +ALIASES=$(aws iam list-account-aliases 2>/dev/null | python3 -c "import sys,json; print(' '.join(json.load(sys.stdin).get('AccountAliases',[])))" 2>/dev/null || true) + +SANDBOX_PATTERN="sandbox|dev|test|staging|nonprod" +IS_SANDBOX=false + +if echo "$ALIASES" | grep -qiE "$SANDBOX_PATTERN"; then + IS_SANDBOX=true +fi +if echo "$ACCOUNT_ID" | grep -qiE "$SANDBOX_PATTERN"; then + IS_SANDBOX=true +fi +if echo "$ARN" | grep -qiE "$SANDBOX_PATTERN"; then + IS_SANDBOX=true +fi + +if [ "$IS_SANDBOX" = true ]; then + echo -e "${GREEN}SANDBOX${NC}" + if [ -n "$ALIASES" ]; then + echo " Alias: $ALIASES" + fi + echo "" + echo -e "${GREEN}============================================${NC}" + echo -e "${GREEN} Sandbox environment verified. Safe to run.${NC}" + echo -e "${GREEN}============================================${NC}" +else + echo -e "${RED}PRODUCTION (or unknown)${NC}" + if [ -n "$ALIASES" ]; then + echo " Alias: $ALIASES" + fi + echo "" + echo -e "${RED}============================================${NC}" + echo -e "${RED} BLOCKED: This account does not appear to${NC}" + echo -e "${RED} be a sandbox/dev environment.${NC}" + echo -e "${RED}${NC}" + echo -e "${RED} Pipeline execution is not allowed against${NC}" + echo -e "${RED} production AWS accounts.${NC}" + echo -e "${RED}${NC}" + echo -e "${RED} Switch to a sandbox account and retry:${NC}" + echo -e "${RED} export AWS_PROFILE=sandbox${NC}" + echo -e "${RED}============================================${NC}" + exit 1 +fi diff --git a/.claude/scripts/lib/source_schema_report.py b/.claude/scripts/lib/source_schema_report.py new file mode 100755 index 0000000..8ee1dd1 --- /dev/null +++ b/.claude/scripts/lib/source_schema_report.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +""" +Normalize a payload to one JSON record (if possible) and print a schema table. +Designed to read from stdin (piped from curl, aws, head -1, etc.). +""" +from __future__ import annotations + +import argparse +import csv +import json +import sys +from typing import Any + + +def infer_type(v: Any) -> str: + if v is None: + return "null" + if isinstance(v, bool): + return "boolean" + if isinstance(v, int) and not isinstance(v, bool): + return "integer" + if isinstance(v, float): + return "float" + if isinstance(v, list): + if not v: + return "array (empty)" + return f"array of {infer_type(v[0])}" + if isinstance(v, dict): + return "object" + return "string" + + +def flatten_schema(d: Any, prefix: str = "") -> list[tuple[str, str, str]]: + rows: list[tuple[str, str, str]] = [] + if isinstance(d, dict): + for k, v in d.items(): + full_key = f"{prefix}.{k}" if prefix else f".{k}" + t = infer_type(v) + example = ( + str(v)[:60] + if not isinstance(v, (dict, list)) + else "" + ) + rows.append((full_key, t, example)) + if isinstance(v, dict): + rows.extend(flatten_schema(v, full_key)) + elif isinstance(v, list) and v and isinstance(v[0], dict): + rows.extend(flatten_schema(v[0], full_key + "[]")) + return rows + + +def unwrap_wrapped_list(obj: dict[str, Any]) -> tuple[Any, str | None]: + """If object has exactly one plausible list of dicts, return first element and key name.""" + for k, v in obj.items(): + if isinstance(v, list) and v and isinstance(v[0], dict): + return v[0], k + return obj, None + + +def normalize_to_record(raw: str) -> tuple[Any | None, str | None, str | None]: + """ + Returns (parsed_object, note, error). + note explains normalization (e.g. first array element, key .items). + """ + text = raw.lstrip("\ufeff").strip() + if not text: + return None, None, "empty input" + + # Whole buffer JSON + try: + d = json.loads(text) + if isinstance(d, list): + if not d: + return None, None, "JSON array is empty" + if isinstance(d[0], dict): + return d[0], "used first element of top-level JSON array", None + return d[0], "used first element of top-level JSON array", None + if isinstance(d, dict): + inner, key = unwrap_wrapped_list(d) + if key is not None and inner is not d: + return inner, f"used first record from list at key '.{key}'", None + return d, None, None + except json.JSONDecodeError: + pass + + # NDJSON: first line + first_line = text.splitlines()[0].strip() + try: + d = json.loads(first_line) + if isinstance(d, dict): + inner, key = unwrap_wrapped_list(d) + if key is not None and inner is not d: + return inner, f"used first record from list at key '.{key}' (line 1)", None + return d, "parsed first line as JSON (NDJSON)", None + if isinstance(d, list) and d: + return d[0], "used first element of JSON array on first line", None + return d, "parsed first line as JSON", None + except json.JSONDecodeError: + pass + + return None, None, "not valid JSON (try CSV mode or paste a JSON object)" + + +def print_report(sample_label: str, obj: Any, note: str | None) -> None: + print(f"## Source sample — {sample_label}") + if note: + print(f"Note: {note}") + print() + print("### Raw sample (one record)") + print(json.dumps(obj, indent=2, ensure_ascii=False)) + print() + print("### Schema (inferred)") + print(f"{'Field':<44} {'Type':<22} {'Example'}") + print("-" * 92) + for field, typ, ex in flatten_schema(obj): + print(f"{field:<44} {typ:<22} {ex}") + + +def csv_first_row_report(path: str) -> int: + with open(path, newline="", encoding="utf-8", errors="replace") as f: + reader = csv.DictReader(f) + row = next(reader, None) + if row is None: + print("CSV: no data rows after header", file=sys.stderr) + return 1 + print("## Source sample — file (CSV)") + print() + print("### Columns") + print(", ".join(reader.fieldnames or [])) + print() + print("### First row (values as strings)") + print(json.dumps(dict(row), indent=2, ensure_ascii=False)) + print() + print("### Schema (inferred from string cells)") + print(f"{'Field':<44} {'Type':<22} {'Example'}") + print("-" * 92) + for k, v in row.items(): + print(f"{'.' + k:<44} {'string':<22} {str(v)[:60]}") + return 0 + + +def main() -> int: + p = argparse.ArgumentParser(description="Infer schema from one JSON record on stdin.") + p.add_argument( + "--label", + default="stdin", + help="Label for the report header (e.g. s3://bucket/key)", + ) + p.add_argument( + "--raw-only", + action="store_true", + help="Print raw input only (no JSON/schema); for non-JSON previews", + ) + args = p.parse_args() + + raw = sys.stdin.read() + if args.raw_only: + sys.stdout.write(raw) + if raw and not raw.endswith("\n"): + sys.stdout.write("\n") + return 0 + + obj, note, err = normalize_to_record(raw) + if obj is None: + print(f"Could not normalize JSON record: {err}", file=sys.stderr) + print("--- raw (first 800 chars) ---", file=sys.stderr) + print(raw[:800], file=sys.stderr) + return 1 + + print_report(args.label, obj, note) + return 0 + + +if __name__ == "__main__": + if len(sys.argv) == 3 and sys.argv[1] == "csv-file": + raise SystemExit(csv_first_row_report(sys.argv[2])) + raise SystemExit(main()) diff --git a/.claude/scripts/run-pipeline.sh b/.claude/scripts/run-pipeline.sh new file mode 100755 index 0000000..a83ba41 --- /dev/null +++ b/.claude/scripts/run-pipeline.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Wrapper to run a caterpillar pipeline and pretty-print any JSON output files. +# Usage: .claude/scripts/run-pipeline.sh + +set -euo pipefail + +PIPELINE_FILE="${1:-}" + +if [ -z "$PIPELINE_FILE" ]; then + echo "Usage: .claude/scripts/run-pipeline.sh " + exit 1 +fi + +if [ ! -f "$PIPELINE_FILE" ]; then + echo "ERROR: pipeline file not found: $PIPELINE_FILE" + exit 1 +fi + +# Verify sandbox environment before running against AWS resources +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if grep -qE 'type:\s*(sqs|sns|kafka|aws_parameter_store)' "$PIPELINE_FILE" || grep -qE 's3://' "$PIPELINE_FILE"; then + echo "Pipeline uses AWS resources — running sandbox check..." + bash "$SCRIPT_DIR/ensure-sandbox.sh" + echo "" +fi + +# Build binary if missing +if [ ! -f "./caterpillar" ]; then + echo "Building caterpillar..." + go build -o caterpillar cmd/caterpillar/caterpillar.go +fi + +# Snapshot output dir before run to detect new files +OUTPUT_BEFORE=$(find output/ -name "*.json" 2>/dev/null | sort || true) + +# Run pipeline +echo "Running: $PIPELINE_FILE" +./caterpillar -conf "$PIPELINE_FILE" + +# Find newly written JSON files +OUTPUT_AFTER=$(find output/ -name "*.json" 2>/dev/null | sort || true) +NEW_FILES=$(comm -13 <(echo "$OUTPUT_BEFORE") <(echo "$OUTPUT_AFTER") || true) + +# Pretty-print each new JSON output file +if [ -n "$NEW_FILES" ]; then + for FILE in $NEW_FILES; do + python3 -c " +import json +with open('$FILE') as f: + data = json.load(f) +with open('$FILE', 'w') as f: + json.dump(data, f, indent=2) + f.write('\n') +if isinstance(data, list): + print(f'OK $FILE — {len(data)} records — pretty-printed') +else: + print(f'OK $FILE — pretty-printed') +" 2>/dev/null || echo "WARN: $FILE could not be pretty-printed (not valid JSON)" + done +fi diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..73f8ff0 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,87 @@ +{ + "env": { + "AWS_PROFILE": "sandbox" + }, + "permissions": { + "allow": [ + "Bash(go build *)", + "Bash(go test *)", + "Bash(./caterpillar -conf *)", + "Bash(.claude/scripts/run-pipeline.sh *)", + "Bash(.claude/scripts/check-source-schema.sh *)", + "Bash(aws s3 cp *)", + "Bash(aws sqs receive-message*)", + "Bash(curl *)", + "Bash(mkdir -p test/pipelines/probes)", + "Bash(rm -f test/pipelines/probes/*)", + "Bash(ls test/pipelines*)", + "Bash(cat test/pipelines/*)", + "Bash(aws sts get-caller-identity*)", + "Bash(aws iam list-account-aliases*)", + "Bash(aws sqs get-queue-attributes*)", + "Bash(aws sqs get-queue-url*)", + "Bash(aws sns get-topic-attributes*)", + "Bash(aws sns list-topics*)", + "Bash(aws sns list-subscriptions-by-topic*)", + "Bash(aws s3api head-bucket*)", + "Bash(aws s3api get-bucket-location*)", + "Bash(aws s3 ls *)", + "Bash(aws ssm get-parameter*)", + "Bash(aws ssm get-parameters-by-path*)", + "Bash(nc -zv *)" + ], + "deny": [ + "Bash(git push*)", + "Bash(git push --force*)", + "Bash(aws s3 rm *)", + "Bash(aws s3api delete*)", + "Bash(aws sqs delete*)", + "Bash(aws sqs purge*)", + "Bash(aws sns delete*)" + ] + }, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": ".claude/hooks/aws-env-check.sh", + "statusMessage": "Checking AWS environment..." + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "command": ".claude/hooks/preflight-check.sh" + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "Write|Edit", + "hooks": [ + { + "type": "command", + "command": ".claude/hooks/validate-on-save.sh" + } + ] + }, + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "command": ".claude/hooks/run-summary.sh" + } + ] + } + ] + } +} diff --git a/.claude/skills/archive/SKILL.md b/.claude/skills/archive/SKILL.md new file mode 100644 index 0000000..e5d4adc --- /dev/null +++ b/.claude/skills/archive/SKILL.md @@ -0,0 +1,147 @@ +--- +skill: archive +version: 1.0.0 +caterpillar_type: archive +description: Pack multiple file records into a zip/tar archive, or unpack an archive into individual file records. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Two modes: +- **Pack**: buffers all incoming records → emits one archive record containing them all +- **Unpack**: receives one archive record → emits one record per file inside the archive + +## Schema + +```yaml +- name: # REQUIRED + type: archive # REQUIRED + format: # OPTIONAL — "zip" or "tar" (default: zip) + action: # OPTIONAL — "pack" or "unpack" (default: pack) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Bundle files for delivery | `action: pack` | +| Extract files for processing | `action: unpack` | +| Target system expects ZIP | `format: zip` | +| Unix/Linux environment | `format: tar` | +| Compressed TAR (`.tar.gz`) needed | `format: tar` + `compress` task after with `format: gzip` | +| Multiple files in, one archive out | `action: pack` | +| One archive in, multiple files out | `action: unpack` | + +## Behavior Details + +| Action | Input | Output | +|--------|-------|--------| +| `pack` | N records (file contents) | 1 archive record | +| `unpack` | 1 archive record | N records (one per file) | + +**Note**: `pack` buffers all upstream records in memory before emitting — be cautious with large datasets. + +## Validation Rules + +- `action: pack` collects everything in memory before emitting — warn for large input streams +- TAR format has no built-in compression — combine with `compress` task for `.tar.gz` +- ZIP is more widely compatible across OS environments +- After `unpack`, each record contains one file's content — downstream tasks process individual files + +## Examples + +### Pack files into ZIP → write +```yaml +- name: pack_files + type: archive + format: zip + action: pack + +- name: write_archive + type: file + path: output/bundle_{{ macro "timestamp" }}.zip +``` + +### Unpack ZIP → process each file +```yaml +- name: read_archive + type: file + path: incoming/bundle.zip + +- name: unpack + type: archive + format: zip + action: unpack + +- name: process + type: converter + format: csv + skip_first: true +``` + +### Pack → TAR → gzip compress → S3 +```yaml +- name: pack_tar + type: archive + format: tar + action: pack + +- name: compress + type: compress + format: gzip + action: compress + +- name: upload + type: file + path: s3://{{ env "BUCKET" }}/archive_{{ macro "timestamp" }}.tar.gz +``` + +### Unpack TAR with multiple files +```yaml +- name: read_tar + type: file + path: s3://my-bucket/incoming/data.tar + +- name: extract + type: archive + format: tar + action: unpack + +- name: inspect + type: echo + only_data: false +``` + +### Full pipeline: SQS → collect → pack → S3 +```yaml +tasks: + - name: read_queue + type: sqs + queue_url: "{{ env "SQS_QUEUE_URL" }}" + exit_on_empty: true + + - name: transform + type: jq + path: '{ "id": .id, "content": .body }' + + - name: pack + type: archive + format: zip + action: pack + + - name: upload + type: file + path: s3://{{ env "BUCKET" }}/batches/{{ macro "uuid" }}.zip + success_file: true +``` + +## Anti-patterns + +- `action: pack` on large unbounded streams — buffers all records in memory; set upstream `join` or `sample` limits first +- Expecting `.tar.gz` from `archive` alone — combine with `compress` task +- Using `unpack` on a non-archive file — produces runtime error +- Placing `archive` as source (first task) — it requires an upstream task diff --git a/.claude/skills/aws-parameter-store/SKILL.md b/.claude/skills/aws-parameter-store/SKILL.md new file mode 100644 index 0000000..95a090c --- /dev/null +++ b/.claude/skills/aws-parameter-store/SKILL.md @@ -0,0 +1,164 @@ +--- +skill: aws-parameter-store +version: 1.0.0 +caterpillar_type: aws_parameter_store +description: Read parameters from or write parameters to AWS SSM Parameter Store as pipeline data. +role: source | sink +requires_upstream: false # read mode +requires_downstream: false # write mode +aws_required: true +--- + +## Purpose + +Dual-mode SSM task: +- **Read mode** (no upstream + `get`): retrieves parameters → emits records with parameter values +- **Write mode** (has upstream + `set`): extracts values from each record using JQ → writes to SSM + +Distinct from `{{ secret "/path" }}` template function, which injects a parameter value into task config at pipeline init time. This task treats SSM parameters as **data** that flows through the pipeline. + +## Schema + +```yaml +- name: # REQUIRED + type: aws_parameter_store # REQUIRED + get: # CONDITIONAL — read mode: output_key → /ssm/path + set: # CONDITIONAL — write mode: /ssm/path → JQ expression + secure: # OPTIONAL — store as SecureString (default: true) + overwrite: # OPTIONAL — overwrite existing params (default: true) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Load config values into pipeline | read mode: use `get` | +| Write pipeline results to SSM | write mode: use `set` | +| Store sensitive values | `secure: true` (default) | +| Store non-sensitive config | `secure: false` | +| Don't overwrite if exists | `overwrite: false` | +| SSM paths are environment-specific | use `{{ env "ENV" }}` in path values | +| Values from record fields | `set` values are JQ expressions: `".field_name"` | +| Static config injection into task config | use `{{ secret "/path" }}` template instead | + +## Mode Detection + +- No upstream task + `get` defined → **Read mode** (source) +- Has upstream task + `set` defined → **Write mode** (sink) + +## Key Distinction: `aws_parameter_store` task vs `{{ secret }}` template + +| Mechanism | When | Use case | +|-----------|------|---------| +| `{{ secret "/path" }}` | Pipeline init (once) | Inject API keys/tokens into task config fields | +| `aws_parameter_store` task | Runtime per record | SSM params are the pipeline's input or output data | + +## Validation Rules + +- `get` or `set` must be present — cannot be empty +- `set` values are **JQ expressions** (e.g. `".access_token"`, `".expires | tostring"`) — not literal values +- SSM parameter paths must start with `/` +- `secure: true` requires KMS permissions — warn if KMS may not be available +- `overwrite: false` silently skips existing parameters — confirm this is intended behavior +- Write mode data must be valid JSON — add `jq` upstream to ensure correct format + +## IAM Permissions + +``` +# Read mode +ssm:GetParameter +ssm:GetParameters +ssm:GetParametersByPath + +# Write mode +ssm:PutParameter + +# Encrypted parameters (read) +kms:Decrypt + +# Encrypted parameters (write) +kms:GenerateDataKey +``` + +## Examples + +### Read parameters (source) +```yaml +- name: load_config + type: aws_parameter_store + get: + api_key: "/prod/api/key" + db_url: "/prod/database/url" + tenant_id: "/prod/app/tenant" + fail_on_error: true +``` + +### Read with env-driven paths +```yaml +- name: load_env_config + type: aws_parameter_store + get: + endpoint: "{{ env "SSM_ENDPOINT_PATH" }}" + token: "{{ env "SSM_TOKEN_PATH" }}" +``` + +### Write record fields to SSM +```yaml +- name: store_tokens + type: aws_parameter_store + set: + "/prod/auth/access_token": ".access_token" + "/prod/auth/refresh_token": ".refresh_token" + "/prod/auth/expires_at": ".expires_in | tostring" + secure: true + overwrite: true +``` + +### Full pattern: fetch OAuth token → store in SSM +```yaml +tasks: + - name: fetch_token + type: http + method: POST + endpoint: https://auth.example.com/oauth/token + body: '{"grant_type":"client_credentials","client_id":"{{ env "CLIENT_ID" }}"}' + headers: + Content-Type: application/json + fail_on_error: true + + - name: parse_token + type: jq + path: | + { + "access_token": (.data | fromjson | .access_token), + "expires_in": (.data | fromjson | .expires_in) + } + + - name: store_token + type: aws_parameter_store + set: + "/prod/oauth/access_token": ".access_token" + "/prod/oauth/expires_at": ".expires_in | tostring" + secure: true + overwrite: true +``` + +### Write with non-secure params (config, not secrets) +```yaml +- name: store_config + type: aws_parameter_store + set: + "/prod/app/last_run_ts": '"{{ macro "timestamp" }}"' + "/prod/app/processed_count": ".count | tostring" + secure: false + overwrite: true +``` + +## Anti-patterns + +- Using `set` with literal string values instead of JQ expressions — `set` values are always JQ +- SSM parameter paths missing the leading `/` → SSM API error +- `secure: true` without verifying KMS permissions — write will fail silently without `fail_on_error: true` +- `overwrite: false` when the intent is to always update — params silently skipped on subsequent runs +- Using this task when a `{{ secret "/path" }}` template would be simpler (static injection at pipeline init) diff --git a/.claude/skills/compress/SKILL.md b/.claude/skills/compress/SKILL.md new file mode 100644 index 0000000..07f363f --- /dev/null +++ b/.claude/skills/compress/SKILL.md @@ -0,0 +1,122 @@ +--- +skill: compress +version: 1.0.0 +caterpillar_type: compress +description: Compress or decompress record data using gzip, snappy, zlib, or deflate. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Applies a compression or decompression algorithm to each record's data. +Typically placed immediately before a `file` write (compress) or immediately after a `file` read (decompress). + +## Schema + +```yaml +- name: # REQUIRED + type: compress # REQUIRED + format: # REQUIRED — "gzip", "snappy", "zlib", or "deflate" + action: # REQUIRED — "compress" or "decompress" + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| General purpose, wide compatibility | `format: gzip` | +| Fastest compress/decompress | `format: snappy` | +| Standard deflate with header | `format: zlib` | +| Raw deflate, no header | `format: deflate` | +| Writing compressed output | `action: compress`, place before `file` write task | +| Reading compressed input | `action: decompress`, place after `file` read task | +| Output file extension | append `.gz`, `.snappy`, `.zlib` in the downstream `file` path | + +## Format Comparison + +| Format | Speed | Ratio | Compatibility | +|--------|-------|-------|--------------| +| `gzip` | Medium | Good | Universal | +| `snappy` | Fast | Moderate | Kafka, Parquet, Hadoop | +| `zlib` | Medium | Good | Wide | +| `deflate` | Medium | Good | Wide (no header) | + +## Validation Rules + +- Both `format` and `action` are required — flag if either is missing +- Do not compress already-compressed data — warn if the upstream task is also `compress` +- Output format should match the downstream consumer's expected format +- Use matching file extension in `file` task path for clarity + +## Examples + +### Compress with gzip → write to S3 +```yaml +- name: compress_output + type: compress + format: gzip + action: compress + +- name: write_s3 + type: file + path: s3://my-bucket/data/output_{{ macro "timestamp" }}.gz +``` + +### Read from S3 → decompress gzip → process +```yaml +- name: read_compressed + type: file + path: s3://my-bucket/archive/data.gz + +- name: decompress + type: compress + format: gzip + action: decompress + +- name: parse_json + type: jq + path: .records[] + explode: true +``` + +### Compress with snappy (Kafka / Hadoop pipelines) +```yaml +- name: compress_snappy + type: compress + format: snappy + action: compress +``` + +### Full pipeline: transform → compress → archive +```yaml +tasks: + - name: source + type: sqs + queue_url: "{{ env "SQS_QUEUE_URL" }}" + exit_on_empty: true + + - name: transform + type: jq + path: '{ "id": .id, "ts": "{{ macro "timestamp" }}", "data": .payload }' + + - name: compress + type: compress + format: gzip + action: compress + + - name: write + type: file + path: s3://{{ env "OUTPUT_BUCKET" }}/batch_{{ macro "uuid" }}.gz + success_file: true +``` + +## Anti-patterns + +- Missing `format` or `action` — both are required +- Compressing already-compressed data — results in larger output and wasted CPU +- Using `snappy` when the downstream consumer expects `gzip` — formats are not interchangeable +- Not matching file extension in `path` (e.g. writing `.json` but data is gzip) — use `.gz`, `.snappy` diff --git a/.claude/skills/converter/SKILL.md b/.claude/skills/converter/SKILL.md new file mode 100644 index 0000000..422a729 --- /dev/null +++ b/.claude/skills/converter/SKILL.md @@ -0,0 +1,170 @@ +--- +skill: converter +version: 1.0.0 +caterpillar_type: converter +description: Convert record data between formats — CSV, HTML, XLSX, XLS, EML, or SST. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Converts the data field of each incoming record from one format to another. +Output records and shape depend on the target format (see per-format behavior below). + +## Schema + +```yaml +- name: # REQUIRED + type: converter # REQUIRED + format: # REQUIRED — "csv", "html", "xlsx", "xls", "eml", or "sst" + delimiter: # OPTIONAL — SST only: key/value separator (default: \t) + + # CSV-specific + skip_first: # OPTIONAL — treat first row as header (default: false) + columns: # OPTIONAL — column definitions + - name: # column name + is_numeric: # treat as number (default: false) + + # HTML-specific + container: # OPTIONAL — XPath to scope extraction + + # XLSX/XLS-specific + sheets: [, ...] # OPTIONAL — sheet names to process (default: all) + skip_rows: # OPTIONAL — rows to skip on all sheets (default: 0) + skip_rows_by_sheet: # OPTIONAL — per-sheet row skip override + : + + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Input is CSV, first row is headers | `format: csv`, `skip_first: true` | +| Input is CSV, no headers | `format: csv`, `skip_first: false`, provide `columns` | +| Column types matter | set `is_numeric: true` on numeric columns | +| Input is HTML, extract specific section | `format: html`, set `container` XPath | +| Input is `.xlsx` | `format: xlsx` | +| Input is legacy `.xls` | `format: xls` | +| Process only specific sheets | set `sheets` array | +| Each sheet has header rows to skip | set `skip_rows` or `skip_rows_by_sheet` | +| Input is email / `.eml` file | `format: eml` | +| Need sheet name in downstream path | use `{{ context "xlsx_sheet_name" }}` | +| Need filename of EML part downstream | use `{{ context "converter_filename" }}` | +| Input is SSTable key=value | `format: sst`, optionally set `delimiter` | + +## Column Naming Matrix (CSV) + +| skip_first | columns provided | Result | +|-----------|-----------------|--------| +| `true` | no | use row 1 values as column names | +| `true` | yes | use provided names (override row 1) | +| `false` | no | `Col1`, `Col2`, `Col3`, … | +| `false` | yes | use provided names | + +## Per-format Output Behavior + +| Format | Emits | Context keys set | +|--------|-------|-----------------| +| `csv` | One JSON record per original record | — | +| `html` | One JSON record per original record | — | +| `xlsx` / `xls` | **One record per sheet** | `xlsx_sheet_name` | +| `eml` | One record per part (body.html, body.txt, headers.json, attachments) | `converter_filename`, `content_type` | +| `sst` | One record per line | — | + +## Validation Rules + +- `format` is required +- `skip_first` and `columns` only apply to `format: csv` +- `container` only applies to `format: html` +- `sheets`, `skip_rows`, `skip_rows_by_sheet` only apply to `format: xlsx` / `format: xls` +- `delimiter` only applies to `format: sst` +- XLSX emits **one record per sheet** — if user expects per-row records, they need a `split` task after converter + +## Examples + +### CSV with headers +```yaml +- name: parse_csv + type: converter + format: csv + skip_first: true +``` + +### CSV with explicit columns +```yaml +- name: parse_csv + type: converter + format: csv + skip_first: true + columns: + - name: id + is_numeric: true + - name: email + - name: revenue + is_numeric: true +``` + +### HTML table extraction +```yaml +- name: parse_table + type: converter + format: html + container: "//table[@class='results']" +``` + +### Excel — all sheets, skip header row +```yaml +- name: parse_excel + type: converter + format: xlsx + skip_rows: 1 +``` + +### Excel — specific sheets, per-sheet skip +```yaml +- name: parse_excel + type: converter + format: xlsx + sheets: ["Sales", "Returns"] + skip_rows: 1 + skip_rows_by_sheet: + Returns: 3 +``` + +### Write each Excel sheet to its own file +```yaml +- name: parse_excel + type: converter + format: xlsx + +- name: write_sheet + type: file + path: output/{{ context "xlsx_sheet_name" }}_{{ macro "timestamp" }}.csv +``` + +### EML — extract parts and write each +```yaml +- name: read_email + type: file + path: inbox/message.eml + +- name: parse_email + type: converter + format: eml + +- name: write_part + type: file + path: output/{{ context "converter_filename" }} +``` + +## Anti-patterns + +- Expecting per-row records from XLSX without a `split` task after converter +- Using `skip_first` on `format: html` or `format: xlsx` — only valid for CSV +- Not using `{{ context "xlsx_sheet_name" }}` when writing each sheet to a separate file +- Forgetting that EML `converter_filename` includes sanitized filenames — downstream paths should use the context key diff --git a/.claude/skills/delay/SKILL.md b/.claude/skills/delay/SKILL.md new file mode 100644 index 0000000..8e7e397 --- /dev/null +++ b/.claude/skills/delay/SKILL.md @@ -0,0 +1,133 @@ +--- +skill: delay +version: 1.0.0 +caterpillar_type: delay +description: Insert a fixed pause between each record to rate-limit, throttle, or pace pipeline throughput. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Waits for `duration` before passing each record to the next task. +Effective throughput = 1 record / `duration` (per worker). +With `task_concurrency: N`, effective throughput ≈ N / `duration`. + +## Schema + +```yaml +- name: # REQUIRED + type: delay # REQUIRED + duration: # REQUIRED — Go duration string (e.g. "100ms", "1s", "5m") + fail_on_error: # OPTIONAL (default: false) +``` + +## Duration Format + +Go duration strings — **must be quoted strings in YAML**: + +| Value | Meaning | +|-------|---------| +| `"100ms"` | 100 milliseconds | +| `"500ms"` | 500 milliseconds | +| `"1s"` | 1 second | +| `"30s"` | 30 seconds | +| `"1m"` | 1 minute | +| `"5m"` | 5 minutes | +| `"1h"` | 1 hour | +| `"1m30s"` | 1 minute 30 seconds | + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Rate limit API calls | place before `http` task; `duration` = 1/desired_rate | +| 2 requests/second max | `duration: "500ms"` | +| 1 request/second max | `duration: "1s"` | +| 1 request per minute | `duration: "1m"` | +| Throttle SQS/SNS writes | place before `sqs` or `sns` task | +| Simulate slow processing in test | `duration: "2s"` | +| Prevent downstream overload | place before the bottleneck task | + +## Throughput Math + +``` +1 worker: rate = 1 / duration +N workers: rate ≈ N / duration (task_concurrency: N on the delay task) + +Examples: + duration: 500ms, concurrency: 1 → ~2 records/sec + duration: 500ms, concurrency: 5 → ~10 records/sec + duration: 1s, concurrency: 1 → ~1 record/sec + duration: 100ms, concurrency: 10 → ~100 records/sec +``` + +## Validation Rules + +- `duration` is required — flag if missing +- Value must be a **string** in Go duration format, not a number: `"1s"` not `1` +- Impact calculation: N records × duration = total pipeline time — warn for large datasets +- Place `delay` **before** the task being rate-limited, not after + +## Examples + +### Rate limit to 1 request/second +```yaml +- name: throttle + type: delay + duration: "1s" + +- name: call_api + type: http + method: GET + endpoint: https://api.example.com/data/{{ context "id" }} +``` + +### 100ms between SQS messages +```yaml +- name: pace_writes + type: delay + duration: "100ms" + +- name: send_queue + type: sqs + queue_url: "{{ env "SQS_QUEUE_URL" }}" +``` + +### Rate-limited concurrent HTTP pipeline +```yaml +tasks: + - name: read_ids + type: file + path: ids.txt + + - name: split + type: split + + - name: throttle + type: delay + duration: "200ms" + task_concurrency: 5 # 5 workers × 1/200ms = 25 req/sec + + - name: fetch + type: http + method: GET + endpoint: https://api.example.com/items/{{ context "id" }} + fail_on_error: false +``` + +### Simulate slow processing (testing) +```yaml +- name: slow_step + type: delay + duration: "2s" +``` + +## Anti-patterns + +- `duration: 1` (integer) → must be `duration: "1s"` (string) +- Placing `delay` after the rate-limited task — delay fires before the record reaches the next task, so it must precede it +- Using `delay` on every record for very large datasets without calculating total pipeline time +- Not combining `delay` with `task_concurrency` when higher throughput is needed despite rate limiting diff --git a/.claude/skills/echo/SKILL.md b/.claude/skills/echo/SKILL.md new file mode 100644 index 0000000..13633cc --- /dev/null +++ b/.claude/skills/echo/SKILL.md @@ -0,0 +1,125 @@ +--- +skill: echo +version: 1.0.0 +caterpillar_type: echo +description: Print record data to stdout. Use as a debug probe, pipeline monitor, or terminal sink. +role: sink | pass-through +requires_upstream: true +requires_downstream: false # terminal when last task; pass-through when not last +aws_required: false +--- + +## Purpose + +Prints each record to stdout. When used as the last task it is a terminal sink. +When placed mid-pipeline it is a pass-through — records continue to the next task after printing. + +Two output modes: +- `only_data: true` — prints the record's data field as-is (clean output) +- `only_data: false` — prints the full record envelope as JSON (includes ID, origin, context) + +## Schema + +```yaml +- name: # REQUIRED + type: echo # REQUIRED + only_data: # OPTIONAL — true = data only, false = full record JSON (default: false) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| See clean data output | `only_data: true` | +| Inspect record ID, origin, context | `only_data: false` | +| Terminal task (no downstream needed) | last position in task list | +| Mid-pipeline debug checkpoint | any position except last | +| Probe pipeline for task testing | last position, `only_data: true` | +| Production pipeline — no output needed | replace with `file` or other sink | + +## Output Format Comparison + +`only_data: true`: +``` +{"id": 1, "name": "Alice", "status": "active"} +``` + +`only_data: false`: +```json +{ + "id": "a1b2c3d4-...", + "origin": "fetch_users", + "data": "{\"id\": 1, \"name\": \"Alice\"}", + "context": { "user_id": "1" } +} +``` + +## Validation Rules + +- `echo` must have an upstream task — it is never a source +- When not the last task, records pass through transparently +- `only_data: false` shows data as an escaped JSON string inside the envelope — if output appears double-encoded, switch to `only_data: true` +- For production pipelines, replace `echo` with a proper sink (`file`, `sqs`, `http`, etc.) + +## Examples + +### Terminal sink (dev/test) +```yaml +- name: output + type: echo + only_data: true +``` + +### Full record inspection (debug) +```yaml +- name: inspect + type: echo + only_data: false +``` + +### Mid-pipeline checkpoint (pass-through) +```yaml +- name: source + type: file + path: data/input.json + +- name: debug_raw + type: echo + only_data: true # prints, passes record forward + +- name: transform + type: jq + path: '{ "id": .id }' + +- name: debug_transformed + type: echo + only_data: true # prints again, passes forward + +- name: write + type: file + path: output/result.json +``` + +### Probe pipeline (isolate one task for testing) +```yaml +# Probe for testing the 'converter' task +tasks: + - name: source_stub + type: file + path: test/pipelines/names.txt + + - name: task_under_test + type: split + + - name: probe_sink + type: echo + only_data: true +``` + +## Anti-patterns + +- Using `echo` as a production sink when data should be saved or forwarded +- Confusing double-encoded output from `only_data: false` — the data field is a JSON-encoded string inside the JSON envelope +- Placing `echo` as the first task — it has no source mode +- Forgetting to replace `echo` with a real sink before deploying to production diff --git a/.claude/skills/file/SKILL.md b/.claude/skills/file/SKILL.md new file mode 100644 index 0000000..4228fd8 --- /dev/null +++ b/.claude/skills/file/SKILL.md @@ -0,0 +1,121 @@ +--- +skill: file +version: 1.0.0 +caterpillar_type: file +description: Read records from or write records to a local file or S3 object. +role: source | sink +requires_upstream: false # read mode has no upstream; write mode requires upstream +requires_downstream: false # write mode has no downstream; read mode requires downstream +aws_required: conditional # only when path starts with s3:// +--- + +## Purpose + +Dual-mode task. Automatically detects its role: +- **Read mode** (source): no upstream task → reads file, emits one record per delimiter +- **Write mode** (sink): has upstream task → receives records, writes each to the file + +## Schema + +```yaml +- name: # REQUIRED — unique task name + type: file # REQUIRED — must be exactly "file" + path: # REQUIRED — local path, S3 URL, or glob pattern + region: # OPTIONAL — AWS region (default: us-west-2, S3 only) + delimiter: # OPTIONAL — record separator in read mode (default: \n) + success_file: # OPTIONAL — write _SUCCESS marker after write (default: false) + success_file_name: # OPTIONAL — success marker filename (default: _SUCCESS) + fail_on_error: # OPTIONAL — stop pipeline on error (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| path starts with `s3://` | set `region` | +| path is the first task | read mode (source) | +| path has upstream task | write mode (sink) | +| reading multiple files | use glob pattern (e.g. `s3://bucket/prefix/*.json`) | +| output filename must be unique per run | use `{{ macro "timestamp" }}` or `{{ macro "uuid" }}` in path | +| output path depends on record data | use `{{ context "key" }}` in path | +| writing to S3 and a downstream system needs confirmation | set `success_file: true` | +| credentials come from environment | use `{{ env "VAR" }}` in path | +| credentials come from AWS SSM | use `{{ secret "/path" }}` in path | + +## Validation Rules + +- `path` is required +- Glob patterns are read-mode only — flag if glob appears in write-mode position +- `success_file` only applies to write mode — flag if set on a source task +- S3 paths must begin with `s3://` +- When `path` contains `{{ context "key" }}`, verify an upstream task sets that key in its `context:` block +- `fail_on_error: true` is recommended for source tasks in production pipelines + +## Template functions supported in `path` + +``` +{{ env "BUCKET" }} → resolved once at pipeline init +{{ secret "/ssm/path" }} → resolved once at pipeline init +{{ macro "timestamp" }} → resolved per record +{{ macro "uuid" }} → resolved per record +{{ macro "unixtime" }} → resolved per record +{{ context "key" }} → resolved per record, set by upstream task +``` + +## Examples + +### Read — local file, split on newlines +```yaml +- name: read_input + type: file + path: data/records.txt + delimiter: "\n" + fail_on_error: true +``` + +### Read — S3 glob (multiple files) +```yaml +- name: read_s3_files + type: file + path: s3://my-bucket/incoming/2024-03-*.json + region: us-west-2 + fail_on_error: true +``` + +### Write — local file with timestamp +```yaml +- name: write_output + type: file + path: output/result_{{ macro "timestamp" }}.json +``` + +### Write — S3 with success marker +```yaml +- name: write_s3 + type: file + path: s3://my-bucket/processed/data_{{ macro "uuid" }}.json + region: us-east-1 + success_file: true +``` + +### Write — per-record dynamic path using context +```yaml +- name: write_per_user + type: file + path: output/{{ context "user_id" }}_{{ macro "timestamp" }}.json +``` + +## Anti-patterns + +- Hardcoding bucket names → use `{{ env "BUCKET" }}` or `{{ secret "/path" }}` +- Using glob patterns in write mode → not supported +- Setting `success_file: true` on a source task → only valid for write mode +- Missing `region` for S3 paths → defaults to `us-west-2`; make explicit for cross-region access + +## IAM permissions (S3) + +``` +s3:GetObject # read +s3:PutObject # write +s3:ListBucket # glob patterns +``` diff --git a/.claude/skills/flatten/SKILL.md b/.claude/skills/flatten/SKILL.md new file mode 100644 index 0000000..3028cba --- /dev/null +++ b/.claude/skills/flatten/SKILL.md @@ -0,0 +1,154 @@ +--- +skill: flatten +version: 1.0.0 +caterpillar_type: flatten +description: Flatten nested JSON objects into single-level key-value pairs using underscore-joined keys. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Converts a deeply nested JSON object into a flat map. Nested keys are joined with `_`. +Optionally preserves the original nested structure under a specified key. + +## Schema + +```yaml +- name: # REQUIRED + type: flatten # REQUIRED + include_original: # OPTIONAL — key name to store original nested data + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Need flat key-value pairs for CSV / DB | basic `flatten` | +| Need both flat AND original nested | set `include_original: "raw"` (or any key name) | +| Only specific nested object | add `jq` upstream to extract it first, then flatten | +| Arrays in nested data | arrays are indexed: `items_0`, `items_1`, … | + +## Flattening Behavior + +**Input:** +```json +{ + "user": { + "id": 42, + "address": { "city": "Portland", "zip": "97201" } + }, + "status": "active" +} +``` + +**Output (no include_original):** +```json +{ + "user_id": 42, + "user_address_city": "Portland", + "user_address_zip": "97201", + "status": "active" +} +``` + +**Output (include_original: "raw"):** +```json +{ + "user_id": 42, + "user_address_city": "Portland", + "user_address_zip": "97201", + "status": "active", + "raw": { "user": { "id": 42, ... }, "status": "active" } +} +``` + +## Array Flattening + +Arrays produce indexed keys: +```json +Input: { "tags": ["news", "tech"] } +Output: { "tags_0": "news", "tags_1": "tech" } +``` + +## Validation Rules + +- `flatten` operates on JSON objects — upstream data must be valid JSON +- Deep nesting produces long key names — review expected output key names +- Array indexing is automatic — warn users if they expect arrays to be preserved +- `include_original` value is any non-empty string (used as the key name in output) + +## Examples + +### Basic flatten +```yaml +- name: flatten_response + type: flatten +``` + +### Flatten preserving original +```yaml +- name: flatten_with_backup + type: flatten + include_original: raw +``` + +### Extract then flatten (specific sub-object) +```yaml +- name: extract_user + type: jq + path: .user + +- name: flatten_user + type: flatten +``` + +### API response → flatten → write CSV +```yaml +tasks: + - name: fetch + type: http + method: GET + endpoint: https://api.example.com/users + + - name: parse_users + type: jq + path: .data[] + explode: true + + - name: flatten + type: flatten + + - name: write + type: file + path: output/users_flat_{{ macro "timestamp" }}.json +``` + +### SQS events → flatten → ingest API +```yaml +tasks: + - name: source + type: sqs + queue_url: "{{ env "SQS_QUEUE_URL" }}" + exit_on_empty: true + + - name: flatten_event + type: flatten + + - name: post + type: http + method: POST + endpoint: https://ingest.example.com/flat-events + headers: + Content-Type: application/json +``` + +## Anti-patterns + +- Flattening without first checking key length — deeply nested objects with array items produce very long keys +- Expecting arrays to be preserved — they become indexed `_0`, `_1`, … keys +- Not using `jq` upstream when only a sub-object needs flattening — whole record is flattened otherwise +- Using `flatten` on non-JSON data — will produce a runtime error diff --git a/.claude/skills/heimdall/SKILL.md b/.claude/skills/heimdall/SKILL.md new file mode 100644 index 0000000..51e84c4 --- /dev/null +++ b/.claude/skills/heimdall/SKILL.md @@ -0,0 +1,146 @@ +--- +skill: heimdall +version: 1.0.0 +caterpillar_type: heimdall +description: Submit jobs to the Heimdall data orchestration platform and receive results downstream. +role: source | transform +requires_upstream: false # source mode: no upstream +requires_downstream: true # always emits job results downstream +aws_required: false +--- + +## Purpose + +Two modes: +- **Source** (no upstream): submits one static job → emits job results to pipeline +- **Destination** (has upstream): for each record, parses its JSON data as job context → submits a job → emits results + +Results from the job execution flow to the next task. Supports sync and async (polled) jobs. + +## Schema + +```yaml +- name: # REQUIRED + type: heimdall # REQUIRED + endpoint: # OPTIONAL — Heimdall API URL (default: http://localhost:9090) + headers: # OPTIONAL — API auth headers + poll_interval: # OPTIONAL — polling interval in seconds (default: 5) + timeout: # OPTIONAL — job timeout in seconds (default: 300) + job: # REQUIRED — job specification + fail_on_error: # OPTIONAL (default: false) +``` + +### Job spec schema +```yaml +job: + name: # OPTIONAL — job name (default: caterpillar) + version: # OPTIONAL — job version (default: 0.0.1) + context: # OPTIONAL — static key-value context for the job + command_criteria: [, ...] # OPTIONAL — criteria to select the command + cluster_criteria: [, ...] # OPTIONAL — criteria to select the cluster + tags: [, ...] # OPTIONAL — job tags +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| One static job, results to pipeline | source mode (no upstream) | +| One job per incoming record | destination mode (add upstream, add `jq` to format context) | +| Long-running job (>300s) | increase `timeout` to expected duration | +| Frequent polling needed | decrease `poll_interval` | +| Heimdall requires auth | add `headers` with token | +| Job context is dynamic per record | add `jq` task before heimdall to build context object | +| Spark job | `command_criteria: ["type:spark"]` | +| Shell job | `command_criteria: ["type:shell"]` | +| Auth token must be secure | use `{{ env "HEIMDALL_TOKEN" }}` in headers | + +## Validation Rules + +- `job` is required +- In destination mode, record data must be valid JSON — add `jq` upstream to format it as the context object +- `timeout` must be long enough for the job type — default 300s may be too short for Spark/EMR jobs +- `poll_interval` must be less than `timeout` — otherwise the first poll attempt may already exceed timeout +- Heimdall endpoint must be reachable from the pipeline host +- Auth tokens must use `{{ env "VAR" }}` or `{{ secret "/path" }}` + +## Examples + +### Source: submit one static job +```yaml +- name: run_job + type: heimdall + endpoint: http://heimdall.example.com + timeout: 3600 + poll_interval: 15 + job: + name: daily-etl + version: 1.0.0 + command_criteria: ["type:spark"] + cluster_criteria: ["type:emr-on-eks"] + context: + query: "SELECT * FROM events WHERE dt = '2024-03-01'" + output: "s3://bucket/output/" +``` + +### Source: ping test job +```yaml +- name: run_ping + type: heimdall + endpoint: http://localhost:9090 + job: + name: ping-test + command_criteria: ["type:ping"] + cluster_criteria: ["type:localhost"] +``` + +### Destination: per-record job submission +```yaml +- name: build_context + type: jq + path: | + { + "table": .source_table, + "filter_id": (.record_id | tostring), + "output_path": "s3://{{ env "OUTPUT_BUCKET" }}/" + .record_id + } + +- name: submit_job + type: heimdall + endpoint: http://heimdall.example.com + timeout: 600 + poll_interval: 10 + job: + name: record-processor + command_criteria: ["type:spark"] + cluster_criteria: ["data:prod"] + +- name: show_results + type: echo + only_data: true +``` + +### With API auth header +```yaml +- name: secure_job + type: heimdall + endpoint: https://heimdall.prod.example.com + headers: + X-Heimdall-Token: "{{ env "HEIMDALL_TOKEN" }}" + X-Heimdall-User: caterpillar + timeout: 1800 + poll_interval: 30 + job: + name: analytics-job + command_criteria: ["type:trino"] + cluster_criteria: ["type:prod"] + context: + query: "SELECT count(*) FROM events" +``` + +## Anti-patterns + +- Destination mode without a `jq` task before heimdall — record data must be a valid JSON context object +- `timeout` too short for long-running jobs — Spark/EMR jobs may take minutes to hours +- Hardcoded auth tokens in `headers` — use `{{ env "VAR" }}` +- `fail_on_error: false` for critical jobs — silent failures mean the pipeline continues with no results diff --git a/.claude/skills/http-server/SKILL.md b/.claude/skills/http-server/SKILL.md new file mode 100644 index 0000000..2cd5887 --- /dev/null +++ b/.claude/skills/http-server/SKILL.md @@ -0,0 +1,114 @@ +--- +skill: http-server +version: 1.0.0 +caterpillar_type: http_server +description: Start an HTTP server to receive inbound requests (webhooks, API push) as a pipeline data source. +role: source +requires_upstream: false +requires_downstream: true +aws_required: false +--- + +## Purpose + +Starts an embedded HTTP server. Each incoming request becomes one pipeline record: +- **Record data**: request body +- **Record context**: request headers as `http-header-` + +Runs until `end_after` requests are received, or indefinitely if `end_after` is omitted. + +## Schema + +```yaml +- name: # REQUIRED + type: http_server # REQUIRED + port: # OPTIONAL — listening port (default: 8080) + end_after: # OPTIONAL — stop after N requests (omit for indefinite) + auth: # OPTIONAL — API key auth config + fail_on_error: # OPTIONAL (default: false) +``` + +### Auth schema +```yaml +auth: + behavior: api-key + headers: + : +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Production deployment | add `auth` block with API key | +| Testing / one-shot intake | set `end_after: ` | +| Long-running webhook listener | omit `end_after` | +| Access request headers downstream | use `{{ context "http-header-" }}` | +| HTTPS required | use a reverse proxy (nginx, ALB) in front | +| Auth token must be configurable | use `{{ env "WEBHOOK_SECRET" }}` in auth header value | + +## Validation Rules + +- `http_server` must always be the **first task** (source only — no upstream) +- `end_after` omitted = runs indefinitely; confirm this is intentional for production +- Port must be available and not blocked by firewall +- For HTTPS, the task serves plain HTTP — put a TLS-terminating proxy in front +- Auth header value should use `{{ env "VAR" }}` — never hardcoded + +## Context auto-populated per request + +``` +{{ context "http-header-Content-Type" }} +{{ context "http-header-Authorization" }} +{{ context "http-header-X-Request-Id" }} +``` + +## Examples + +### Basic webhook receiver +```yaml +- name: webhook_intake + type: http_server + port: 8080 + fail_on_error: true +``` + +### Authenticated server +```yaml +- name: secure_webhook + type: http_server + port: 8080 + auth: + behavior: api-key + headers: + Authorization: Bearer {{ env "WEBHOOK_SECRET" }} +``` + +### Test server (stop after 5 requests) +```yaml +- name: test_receiver + type: http_server + port: 9090 + end_after: 5 +``` + +### Access request metadata downstream +```yaml +# Task following http_server: +- name: tag_request + type: jq + path: | + { + "payload": ., + "source_ip": "{{ context "http-header-X-Forwarded-For" }}", + "content_type": "{{ context "http-header-Content-Type" }}" + } +``` + +## Anti-patterns + +- Using `http_server` anywhere other than position 1 in the task list +- Omitting `auth` in production deployments +- Hardcoding the API key value — use `{{ env "VAR" }}` +- Expecting HTTPS without a TLS proxy in front +- Omitting `end_after` in tests — the pipeline will run forever diff --git a/.claude/skills/http/SKILL.md b/.claude/skills/http/SKILL.md new file mode 100644 index 0000000..475e223 --- /dev/null +++ b/.claude/skills/http/SKILL.md @@ -0,0 +1,189 @@ +--- +skill: http +version: 1.0.0 +caterpillar_type: http +description: Make HTTP requests to external APIs — fetch data (source) or post pipeline records (sink). +role: source | sink +requires_upstream: false # source mode: no upstream +requires_downstream: true # always emits response records downstream +aws_required: false +--- + +## Purpose + +Dual-mode HTTP client task: +- **Source mode** (no upstream): sends requests using static YAML config; supports pagination +- **Sink mode** (has upstream): each record's JSON data is merged with the base config to form the request + +Response body is passed downstream. Response headers are automatically stored in context as `http-header-`. + +## Schema + +```yaml +- name: # REQUIRED + type: http # REQUIRED + endpoint: # REQUIRED — target URL + method: # OPTIONAL — HTTP verb (default: GET) + headers: # OPTIONAL — request headers + body: # OPTIONAL — request body (POST/PUT) + timeout: # OPTIONAL — seconds (default: 90) + max_retries: # OPTIONAL — retry attempts (default: 3) + retry_delay: # OPTIONAL — seconds between retries (default: 5) + expected_statuses: # OPTIONAL — comma-separated codes (default: "200") + next_page: # OPTIONAL — JQ expr for next page URL, or pagination object + context: # OPTIONAL — JQ exprs to extract response values into context + oauth: # OPTIONAL — OAuth 1.0 or 2.0 config + proxy: # OPTIONAL — proxy config + fail_on_error: # OPTIONAL (default: false) +``` + +### OAuth 1.0 schema +```yaml +oauth: + consumer_key: + consumer_secret: + token: + token_secret: + version: "1.0" + signature_method: "HMAC-SHA256" +``` + +### OAuth 2.0 schema (client credentials) +```yaml +oauth: + token_uri: + grant_type: "client_credentials" + scope: [, ...] +``` + +### Pagination (`next_page`) + +`next_page` is a JQ expression evaluated after every HTTP response to drive +automatic pagination. It receives `{"data": "", "headers": {...}}` and +must return a URL string, a request object, or `null`/`empty` to stop. + +**For full documentation, patterns, and examples see the dedicated +[pagination skill](../pagination/SKILL.md).** + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Fetching from an API with no incoming data | source mode (no upstream task) | +| Posting each pipeline record to an API | sink mode (add upstream task) | +| API requires Bearer token | add `Authorization: Bearer {{ env "TOKEN" }}` to `headers` | +| API requires OAuth 1.0 | add `oauth` block with `version: "1.0"` | +| API requires OAuth 2.0 | add `oauth` block with `token_uri` and `grant_type` | +| API is paginated | add `next_page` with JQ expression extracting next URL | +| Need downstream access to a response field | add `context` block with JQ expressions | +| Need downstream access to response header | use `{{ context "http-header-" }}` — auto-populated | +| Endpoint URL contains record-specific data | use `{{ context "key" }}` in endpoint string | +| Non-200 success codes expected | set `expected_statuses: "200,201,202"` | +| Credentials must be secure | use `{{ env "VAR" }}` or `{{ secret "/ssm/path" }}` | + +## Response headers in context + +All response headers are automatically available downstream: +``` +{{ context "http-header-Content-Type" }} +{{ context "http-header-X-Request-Id" }} +``` +Header names use Go canonical form (e.g. `content-type` → `Content-Type`). + +## Validation Rules + +- `endpoint` is required +- `expected_statuses` is a **string**, not an array: `"200,201"` not `["200","201"]` +- Secrets/tokens must never be hardcoded — always `{{ env "VAR" }}` or `{{ secret "/path" }}` +- In sink mode, record data must be valid JSON — add a `jq` task upstream if needed +- `next_page` — see the [pagination skill](../pagination/SKILL.md) for full validation rules +- `batch_flush_interval` not applicable here — see `kafka` skill + +## Examples + +### GET request (source) +```yaml +- name: fetch_users + type: http + method: GET + endpoint: https://api.example.com/users + headers: + Accept: application/json + Authorization: Bearer {{ env "API_TOKEN" }} + fail_on_error: true +``` + +### POST each record (sink) +```yaml +- name: post_to_api + type: http + method: POST + endpoint: https://ingest.example.com/events + headers: + Content-Type: application/json + max_retries: 5 + retry_delay: 2 + expected_statuses: "200,201" +``` + +### Paginated GET (basic) +```yaml +- name: fetch_all_pages + type: http + method: GET + endpoint: https://api.example.com/items?limit=100 + next_page: >- + .data | fromjson | + if .nextCursor != null then + "https://api.example.com/items?limit=100&cursor=\(.nextCursor)" + else null end +``` + +See the [pagination skill](../pagination/SKILL.md) for 13 pagination patterns +covering cursors, offsets, Link headers, HATEOAS links, signed requests, +GraphQL, rate-limiting gates, dynamic upstream `next_page`, and more. + +### Extract context from response +```yaml +- name: get_auth_token + type: http + method: POST + endpoint: https://auth.example.com/token + body: '{"grant_type":"client_credentials"}' + headers: + Content-Type: application/json + context: + access_token: ".data | fromjson | .access_token" + expires_in: ".data | fromjson | .expires_in | tostring" +``` + +### Dynamic endpoint from context +```yaml +- name: fetch_user_detail + type: http + method: GET + endpoint: https://api.example.com/users/{{ context "user_id" }} + headers: + Authorization: Bearer {{ context "access_token" }} +``` + +### OAuth 2.0 +```yaml +- name: call_google_api + type: http + method: GET + endpoint: https://www.googleapis.com/some/resource + oauth: + token_uri: https://oauth2.googleapis.com/token + grant_type: client_credentials + scope: + - https://www.googleapis.com/auth/cloud-platform +``` + +## Anti-patterns + +- Hardcoded tokens/passwords in headers → use `{{ env "VAR" }}` +- `expected_statuses` as array `["200"]` → must be string `"200"` +- Omitting `fail_on_error: true` on critical source tasks +- Sink mode without a `jq` task upstream when data is not already a valid HTTP request JSON object +- See [pagination skill](../pagination/SKILL.md) for pagination-specific anti-patterns diff --git a/.claude/skills/join/SKILL.md b/.claude/skills/join/SKILL.md new file mode 100644 index 0000000..5ffc1b0 --- /dev/null +++ b/.claude/skills/join/SKILL.md @@ -0,0 +1,164 @@ +--- +skill: join +version: 1.0.0 +caterpillar_type: join +description: Aggregate multiple records into one by batching on count, byte size, or time duration. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Buffers incoming records and emits a combined record when a flush condition is met. +Flush triggers (first condition satisfied wins): +- `number` records accumulated +- Total `size` bytes reached +- `duration` elapsed since last flush + +If no conditions are set, flushes once at end-of-stream (joins everything). + +## Schema + +```yaml +- name: # REQUIRED + type: join # REQUIRED + number: # OPTIONAL — max records per batch + size: # OPTIONAL — max bytes before flush + duration: # OPTIONAL — max wait (Go duration: "30s", "5m", "1h") + delimiter: # OPTIONAL — separator between joined records (default: \n) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Batch by fixed record count | set `number` | +| Batch by payload size (e.g. 1 MB chunks) | set `size: 1048576` | +| Flush on time interval | set `duration: "5m"` | +| Multi-condition (whichever comes first) | combine `number`, `size`, `duration` | +| Collect all records into one | set none of the three (end-of-stream flush) | +| Join with newlines | default `delimiter: "\n"` | +| Join with pipe separator | `delimiter: "\|"` | +| Join for JSON array | use `replace` after to wrap: `^(.*)$` → `[$1]` | + +## Flush Behavior + +``` +Incoming: record1, record2, record3 (number: 3 configured) +Output: "record1\nrecord2\nrecord3" ← single record +``` + +Flush triggers are evaluated after **each record is added**. Flushes immediately when first condition is met. + +## Size Reference + +| Size value | Bytes | +|-----------|-------| +| 1 KB | 1024 | +| 64 KB | 65536 | +| 512 KB | 524288 | +| 1 MB | 1048576 | +| 5 MB | 5242880 | + +## Validation Rules + +- At least one of `number`, `size`, `duration` is recommended — otherwise all records accumulate in memory until stream ends +- `duration` uses Go format: `"30s"`, `"5m"`, `"1h30m"` — not plain integers +- Large end-of-stream joins risk out-of-memory for unbounded streams — always recommend a limit +- After `join`, data is a single string — downstream tasks receive one large record per batch + +## Examples + +### Batch 100 records per output record +```yaml +- name: batch_100 + type: join + number: 100 + delimiter: "\n" +``` + +### Batch by 1 MB chunks +```yaml +- name: batch_1mb + type: join + size: 1048576 + delimiter: "\n" +``` + +### Flush every 5 minutes +```yaml +- name: time_window + type: join + duration: "5m" + delimiter: "\n" +``` + +### Multi-trigger (50 records, 512 KB, or 2 minutes) +```yaml +- name: flexible_batch + type: join + number: 50 + size: 524288 + duration: "2m" + delimiter: "\n" +``` + +### Join all → write as single file +```yaml +- name: collect_all + type: join + delimiter: "\n" + +- name: write_file + type: file + path: output/full_export_{{ macro "timestamp" }}.txt +``` + +### Batch → build JSON array → POST +```yaml +- name: batch + type: join + number: 10 + delimiter: "," + +- name: wrap_array + type: replace + expression: "^(.*)$" + replacement: "[$1]" + +- name: post_batch + type: http + method: POST + endpoint: https://api.example.com/batch + headers: + Content-Type: application/json +``` + +### SQS drain → batch → S3 +```yaml +tasks: + - name: read_queue + type: sqs + queue_url: "{{ env "SQS_QUEUE_URL" }}" + exit_on_empty: true + + - name: batch + type: join + number: 1000 + delimiter: "\n" + + - name: write_batch + type: file + path: s3://{{ env "BUCKET" }}/batch_{{ macro "uuid" }}.txt + success_file: true +``` + +## Anti-patterns + +- No flush condition on an unbounded stream → unbounded memory growth +- `duration: 300` (integer) → must be `duration: "5m"` (Go duration string) +- Expecting records to retain individual identity after `join` — they are concatenated into one string +- Using `join` without `split` when the downstream consumer expects individual records again diff --git a/.claude/skills/jq/SKILL.md b/.claude/skills/jq/SKILL.md new file mode 100644 index 0000000..8716d85 --- /dev/null +++ b/.claude/skills/jq/SKILL.md @@ -0,0 +1,394 @@ +--- +skill: jq +version: 1.0.0 +caterpillar_type: jq +description: Transform, filter, reshape, or extract fields from JSON data using JQ queries. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: conditional # only when using translate() custom function +--- + +## Purpose + +Applies a JQ expression to each record's data. The result replaces the record data. +When `explode: true`, array results are split into individual records. +Custom function `translate(text; src; tgt)` calls AWS Translate. + +## How stored JSON is produced (read this if output looks “invalid”) + +Caterpillar **always JSON-encodes** the JQ result with Go’s `encoding/json` before the record leaves the jq task (unless `as_raw: true`). Your `path` should return **native** JQ values (objects, arrays, numbers, strings, booleans, null)—not pre-serialized JSON text for whole-record payloads. + +| Symptom | Typical cause | Fix | +|--------|----------------|-----| +| Nested fields show as quoted JSON strings (`"{\"a\":1}"`) | Used `tojson` / `tostring` on objects you wanted as nested JSON | Emit the object directly: `"nested": .foo` not `"nested": (.foo \| tojson)` | +| Whole file fails `JSON.parse` / “invalid JSON” in one shot | File has **one JSON value per line** (NDJSON / JSON Lines) or `join` concatenated multiple values | Use an NDJSON reader, or end with a jq that outputs **one** array/object for the whole batch (no `explode`), or write `.jsonl` / document NDJSON in the consumer | +| Downstream sees `null` after jq | `path` used `.data \| fromjson` on body that is already an object | Use `.field` on the body; reserve `.data \| fromjson` for **`context:`** only | +| `explode: true` errors or wrong fan-out | Path returns a single non-array | Use a path that yields multiple outputs (e.g. `.items[]`) or one array and `explode: true` | + +**`tojson` in `path`:** Use on purpose when the **next step needs a string** (HTTP `body` that must be a string, cookie blobs, form fields). For sinks that expect structured JSON records, **omit `tojson`** so nested data stays as real JSON objects/arrays after the second encode. + +**`as_raw: true`:** Skips JSON marshaling; output is `fmt`’d text. Only for plain-text downstream tasks. + +## NDJSON vs one JSON document + +- **Default file sink behavior:** each record is written out as its own JSON serialization (often one line per record). +- **`join` with default delimiter `\n`:** merges many records into **one** record whose `data` is **multiple JSON values separated by newlines**—still not a single JSON array unless you built one in jq. +- **If you need one JSON array in a file:** use a jq `path` that produces **one** array value for the whole batch (no `explode`), or keep NDJSON and use tools that read line-by-line. After `join`, the record body is multiple JSON documents concatenated; it is **not** one `json.Unmarshal`-able value unless you built a single array/object in jq **before** join/file. + +## Schema + +```yaml +- name: # REQUIRED + type: jq # REQUIRED + path: # REQUIRED — JQ expression + explode: # OPTIONAL — split array output into separate records (default: false) + as_raw: # OPTIONAL — emit raw string instead of JSON (default: false) + fail_on_error: # OPTIONAL (default: false) + context: # OPTIONAL — JQ exprs to store values in record context +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Extract a single field | `path: .field_name` | +| Reshape the object | `path: '{ "new_key": .old_key }'` | +| Array → individual records | add `explode: true`, ensure path returns array | +| Filter array elements | `path: '.items[] \| select(.active == true)'` with `explode: true` | +| Need value in a downstream task | add `context: { key: ".jq_expr" }` | +| Emit plain string not JSON | add `as_raw: true` | +| Translate text via AWS | use `translate(.field; "en"; "es")` in path | +| Input arrives as JSON string | prefix with `fromjson \|` e.g. `path: '. \| fromjson \| .field'` | +| Need to build HTTP request config | reshape to `{ "endpoint": ..., "method": ..., "body": ... }` | +| Nested JSON in output records (file/Kafka) | build objects/arrays in jq **without** `tojson` on those branches | +| HTTP POST body must be a JSON string | use `"body": (.payload \| tojson)` or similar **only** for that string field | +| Consumer expects NDJSON | default pipeline + file sink is fine; use `.jsonl` or document format | +| Consumer expects a single JSON array | avoid per-record file writes; emit one jq result that is `[...]` (no `explode`) | + +## JQ Quick Reference + +| Goal | Expression | +|------|-----------| +| Extract field | `.field` | +| Nested field | `.a.b.c` | +| Iterate array | `.items[]` | +| Filter | `select(.status == "active")` | +| Build object | `{ "k": .v, "k2": .v2 }` | +| Merge objects | `. + { "extra": .x }` | +| Map over array | `map(. + { "id": .key })` | +| Transform object entries | `with_entries` (see Mirakl Mediamarkt `account_health` DAG) | +| Reusable logic | `def name: …; …` | +| Repeat N outputs | `range(1; 4)` then build an object per index (often with `explode: true`) | +| Concat strings | `(.a + " " + .b)` | +| Interpolate in string | `"prefix\\(.id)/suffix"` | +| Number → string | `tostring` | +| String → number | `tonumber` | +| Decode JSON string | `fromjson` | +| Encode to JSON string | `tojson` | +| Safe parse | `try fromjson catch null` | +| URL-encode | `@uri` | +| Base64 encode / decode | `@base64` / `@base64d` | +| Regex replace / cleanup | `gsub("\n"; " ")`, `test("pattern"; "i")` — edge trim: one `gsub` with `\s` alternation (see SP-API / browse-node DAG jq) | +| Array length | `length` | +| Object keys | `keys` | +| Conditional | `if .x then .y else .z end` | +| Default value | `.field // "default"` | +| Bind variable | `expr as $x` then continue the pipeline | + +Chain steps with jq’s pipe: `.items[] | select(.ok) | {id}`. + +## Custom functions (Caterpillar extensions) + +These are registered when the jq task compiles your `path` (see `customFunctionsOptions` in `internal/pkg/jq/jq.go`). They are **not** standard jq. + +### Cryptographic hashes (hex digest) + +Unary filters: pipe a **string** in; output is lowercase hex. + +| Function | Digest | +|----------|--------| +| `md5` | MD5 | +| `sha256` | SHA-256 | +| `sha512` | SHA-512 | + +Example (Walmart-style signing string): +`( $consumerId + "\n" + $path + "\n" + ($method | ascii_upcase) + "\n" + $timestamp + "\n" ) | sha256 as $stringToSign` + +### HMAC (hex) + +``` +hmac_md5(data; key) +hmac_sha256(data; key) +hmac_sha512(data; key) +# Optional third argument: prefix bytes as a string, passed to HMAC sum +hmac_sha256(data; key; pref) +``` + +`data` and `key` are strings; output is hex. + +### RSA PKCS#1 v1.5 sign (base64 signature) + +``` +rsa_sha256(data; private_key_pem_or_der_string) +rsa_sha512(data; private_key_pem_or_der_string) +``` + +**Important:** `data` must be a **hex-encoded** digest (the implementation decodes it with `hex.DecodeString` before signing). `private_key` is PEM text or raw DER bytes as a string. Supports PKCS#1 and PKCS#8 RSA keys. + +### `uuid` + +Generates a new random UUID string (v4 via `google/uuid`). Used in headers/objects as a bare call, e.g. `"WM_QOS.CORRELATION_ID": uuid` in a jq object literal. + +### `shuffle` + +Shuffles an **array**; input must be an array or jq errors. + +Example: `.data | split("\n") | shuffle | .[:10]` + +### `sleep` + +``` +input | sleep("duration") +``` + +`duration` is a Go `time.ParseDuration` string (`"500ms"`, `"30s"`, `"1m"`, etc.). Sleeps, logs to stdout, then returns **the original input** unchanged. Used in pipelines such as throttling `next_page` expressions (e.g. Keepa token refresh). + +### `translate` — AWS Translate + +``` +translate(text; source_lang; target_lang) +``` + +Requires AWS credentials and the Translate API. Language codes: `"en"`, `"es"`, `"fr"`, `"de"`, `"ja"`, etc. + +## How `path` Receives Data + +The `path` expression runs directly against the **raw record body** (the upstream task's output bytes, parsed as JSON). There is no `.data` wrapper at the `path` level. + +- **`path`** → operates on raw JSON body. If the HTTP source returns `{"users": [...]}`, use `path: .users` — NOT `.data | fromjson | .users`. +- **`context`** → operates on the **record envelope** `{"data": "", "metadata": {...}}`. Context expressions must use `.data | fromjson | .field` to access the body. + +**Rule of thumb:** Never use `.data | fromjson` in the `path` field. If you see yourself writing that, you are confusing `path` with `context` expression syntax. + +## Validation Rules + +- `path` is required +- `path` must NOT start with `.data | fromjson` — that pattern is only valid inside `context` expressions, not in `path` +- `explode: true` requires the JQ expression to return an array — flag if expression won't produce an array +- Multiline JQ uses YAML block scalar `|` — indentation must be consistent +- `{{ context "key" }}` interpolation inside `path` is evaluated before JQ runs — use for dynamic expressions +- `as_raw: true` outputs value without JSON encoding — use only for plain string outputs + +## Examples + +### Extract single field +```yaml +- name: get_id + type: jq + path: .user.id +``` + +### Reshape record +```yaml +- name: normalize + type: jq + path: | + { + "id": .user.id, + "name": (.user.first + " " + .user.last), + "active": (.status == "active"), + "created": .timestamps.created_at + } +``` + +### Nested objects for file/Kafka (do not use `tojson` on structure) + +Wrong — `meta` becomes a JSON **string** (double-encoded after Go marshals the record): + +```yaml +- name: bad_nested + type: jq + path: '{ "id": .id, "meta": (.details | tojson) }' +``` + +Right — `meta` stays a nested object: + +```yaml +- name: good_nested + type: jq + path: '{ "id": .id, "meta": .details }' +``` + +### Explode array into records +```yaml +- name: expand_items + type: jq + path: .items[] + explode: true +``` + +### Filter and explode +```yaml +- name: active_users + type: jq + path: | + .users[] | select(.status == "active") | { + "id": .id, + "email": .email + } + explode: true +``` + +### Store values in context for downstream tasks +```yaml +- name: extract_ids + type: jq + path: . + context: + user_id: .user.id + org_slug: .organization.slug +``` + +### Build HTTP request config (for http sink) +```yaml +- name: build_request + type: jq + path: | + { + "method": "POST", + "endpoint": "https://api.example.com/users/{{ context "user_id" }}", + "body": (. | tojson), + "headers": { "Content-Type": "application/json" } + } +``` + +### Decode JSON string from upstream +Use `fromjson` ONLY when the upstream record is a JSON-encoded string (e.g., SQS message body where the payload is double-encoded). Do NOT use it when upstream is an HTTP or file source — those already deliver parsed JSON. +```yaml +# Correct: upstream sends a literal string like '"{\"id\":1}"' (double-encoded) +- name: parse_payload + type: jq + path: . | fromjson | .id + +# WRONG: upstream is HTTP/file source — body is already JSON, no fromjson needed +# - name: parse_payload +# type: jq +# path: .data | fromjson | .id # ← .data does not exist, evaluates to null +``` + +### Translate field +```yaml +- name: translate_desc + type: jq + path: | + { + "id": .id, + "description_en": .description, + "description_es": translate(.description; "en"; "es") + } +``` + +## Anti-patterns + +- **Using `.data | fromjson` in `path`** — `path` already receives raw JSON. `.data | fromjson` is only for `context` expressions. Using it in `path` evaluates to `null` and silently drops the record. +- **`tojson` on every nested blob** for file/Kafka sinks — creates **string** fields containing escaped JSON; downstream “invalid” or unexpected shape. Reserve `tojson` for string APIs (bodies, cookies). +- **Renaming output `.json` while content is NDJSON** — valid per line, invalid as one document; rename to `.jsonl` or change pipeline shape. +- Forgetting `fromjson` when upstream task outputs a JSON string (not object) +- Using `explode: true` without `[]` or array-producing expression → runtime error +- `{{ context "key" }}` inside a pure JQ array/object — it's string interpolation, not JQ — wrap in quotes +- Inconsistent YAML block scalar indentation for multiline `path` + +## Patterns from `yaml_with_jq_tasks/` (production DAGs) + +These pipelines (under `yaml_with_jq_tasks/`) repeat the same jq shapes. Use them as templates. + +### Shape HTTP `http` task input + +Emit an object the HTTP task understands: `endpoint`, optional `method`, `headers`, optional `body`. + +- **GET:** multiline `path: |` building `{ endpoint: "https://…" + $query, headers: { … } }` (often with `@uri` on query parts). +- **POST JSON as a string field:** `"body": (.payload | tojson)` when the client expects a JSON **string** (common for scraper-central style APIs). +- **POST `application/x-www-form-urlencoded`:** `body` is a **plain string**, e.g. `"grant_type=client_credentials"` or space-delimited scopes — not a JSON object. +- **Bearer / Basic in headers:** `"Authorization": "Bearer \\(.access_token)"` or `"Basic \\(.basic_auth)"`. + +### OAuth and Basic auth helpers + +- **Basic header from id/secret:** merge into the record: `. + {basic_auth: ((.clientId + ":" + .clientSecret) | @base64)}`, then reference `Authorization: "Basic \\(.basic_auth)"`. +- **Decode embedded secret (e.g. Walmart private key):** `("\\(.clientSecret)" | @base64d) as $privateKey` then use `$privateKey` in the rest of the expression. + +### After an `http` response: `context` + pass-through + +The response body is often a JSON string inside the record envelope. Downstream jq **`path`** still sees parsed JSON from the prior task; for **`context`**, use the envelope: + +```yaml +- name: extract_access_token + type: jq + path: "." + context: + access_token: ".data | fromjson | .access_token" +``` + +Use the same pattern for tokens, cursor pagination (`next_cursor`), multi-field creds (`vendor_id`, `secret_key`), and SQL-sourced rows (`merchant_id`, `asin`, etc.). Quote context values in YAML when the expression contains `:` or starts with `.` in ambiguous positions. + +### `{{ context "key" }}` inside `path` + +Caterpillar substitutes `{{ context "…" }}` **before** jq runs. Typical uses: + +- URLs: `"https://api…/credentials/{{ context \"account_id\" }}/access"`. +- HTTP headers: `"x-amz-access-token": "{{ context \"access_token\" }}"`. +- Merging prior results into each row: `map(. + {account_id: "{{ context \"account_id\" }}"})`. +- Rehydrating interpolated JSON blobs: `({{ context "orders_data" }} | if type == "array" then . else [.] end) as $orders` then `map(. + {{ context "order_addresses" }})` (see Target orders-style merges in-repo). + +Keep interpolated fragments valid jq after substitution (arrays/objects must still be legal jq literals). + +### `explode: true` recipes + +- **Array of objects:** `path: .items` or `path: .` when the parsed body is already an array (NetSuite-style). +- **Top-level array:** `path: ".[]"` (Bol inventory-style). +- **Nested array:** e.g. `path: ".positionItems[]"` (Otto returns-style). +- **Filter then one object per match:** `.[] | select(.destination_id == N) | {endpoint: "…\(.destination_key)…"}` on one line in YAML (Bol/Otto creds pattern). +- **Cartesian / pages:** `range(1;3) as $page | {endpoint: "…\($page)"}` inside a multiline `path` (Amazon SERP-style). +- **Repeat per scalar in an array:** `.locations[].key | {endpoint: "…\(.)/access"}` (Walmart items-style). + +`explode: true` requires the jq program to produce **multiple outputs** or a single **array** (per caterpillar rules). Prefer `[]`, `range`, or an explicit array when in doubt. + +### Normalizing “wrapped” tabular cells + +When every value is `[ "scalar" ]`, unwrap with `with_entries(.value |= (if type=="array" and length>0 and (.[0]|type)=="string" then .[0] else null end))` or small `def` helpers that branch on `type` / `has("tag")`. + +### Defensive `fromjson` (mixed string/object rows) + +When one pipeline accepts both stringified and object bodies: + +```text +(if .data then .data else . end | if type == "string" then fromjson else . end) as $row +``` + +For optional parse: `def parse_body: if type == "string" then (try fromjson catch null) elif type == "object" then . else null end;`. + +### `tojson` on selected branches (warehouse / wide rows) + +Some SP-API style extractions map each item to **string columns** that store nested JSON (`competitive_pricing: (.Product.CompetitivePricing | tojson)`). That is intentional when the sink expects JSON-in-string columns — different from “whole nested object for a JSON record” sinks. + +### Binary / CSV payload as file bytes + +Decode base64 record fields and skip JSON wrapping on the wire: + +```yaml +path: ".[].data | @base64d" +as_raw: true +``` + +### Strict pipelines + +Add `fail_on_error: true` on jq when bad transforms should stop the run (e.g. Okta user splitting). + +### Jinja inside `path` + +DAGs sometimes wrap `{{ context "…" }}` in `{% raw %}…{% endraw %}` so Jinja does not eat braces. When authoring by hand, prefer caterpillar’s `{{ context }}` unless you are inside a Jinja-templated YAML file. + +### Legacy / edge reminders + +- **SQS / wrapped bodies:** `.Message | fromjson` when `Message` is a JSON string. +- **Session cookies:** single field containing JSON text → `.cookie_string | fromjson` in **`path`** only if that field is the whole body shape you receive. diff --git a/.claude/skills/kafka/SKILL.md b/.claude/skills/kafka/SKILL.md new file mode 100644 index 0000000..f50fcfe --- /dev/null +++ b/.claude/skills/kafka/SKILL.md @@ -0,0 +1,160 @@ +--- +skill: kafka +version: 1.0.0 +caterpillar_type: kafka +description: Read messages from or write messages to a Kafka topic, with TLS and SASL/SCRAM support. +role: source | sink +requires_upstream: false # read mode +requires_downstream: false # write mode +aws_required: false +--- + +## Purpose + +Dual-mode Kafka task. Auto-detects role: +- **Read mode** (no upstream): polls topic, emits one record per message +- **Write mode** (has upstream): receives records, writes each as Kafka message + +Supports standalone reader (no group) and coordinated group consumer. +Write mode buffers messages and flushes per `batch_size` and `batch_flush_interval`. + +## Schema + +```yaml +- name: # REQUIRED + type: kafka # REQUIRED + bootstrap_server: # REQUIRED — broker address (host:port) + topic: # REQUIRED — topic name + timeout: # OPTIONAL — dial/read/write/commit timeout (default: 15s) + batch_size: # OPTIONAL — messages to buffer before flush (default: 100) + batch_flush_interval: # OPTIONAL — max wait before flush; must be < timeout (default: 2s) + retry_limit: # OPTIONAL — empty-poll retries before stopping (default: 5) + group_id: # OPTIONAL — consumer group ID (recommended for production) + server_auth_type: # OPTIONAL — "none" or "tls" (default: none) + cert: # OPTIONAL — inline CA cert PEM (use | block scalar) + cert_path: # OPTIONAL — path to CA cert file + user_auth_type: # OPTIONAL — "none", "sasl", or "scram" (default: none) + username: # OPTIONAL — SASL/SCRAM username + password: # OPTIONAL — SASL/SCRAM password + fail_on_error: # OPTIONAL (default: false) +``` + +> `mtls` user_auth_type is reserved but not implemented — do not use. + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Reading from topic | first task (no upstream) | +| Writing to topic | add upstream task | +| Production consumer | set `group_id` for coordinated offset commits | +| Dev/one-off read | omit `group_id` (standalone, no offset commits) | +| Broker uses TLS | set `server_auth_type: tls`, provide `cert` or `cert_path` | +| SASL Plain auth | set `user_auth_type: sasl` + `username` + `password` | +| SCRAM-SHA-512 auth | set `user_auth_type: scram` + `username` + `password` | +| Long-running jobs | increase `timeout` (e.g. `5m`) | +| High-throughput write | tune `batch_size` and `batch_flush_interval` | +| Stop after N empty polls | set `retry_limit: N` | +| Inline cert in YAML | use `cert: \|` block scalar | +| Cert from filesystem | use `cert_path: /path/to/ca.pem` | +| Credentials must be secure | use `{{ env "VAR" }}` or `{{ secret "/path" }}` | + +## Constraint: batch_flush_interval < timeout + +In write mode `batch_flush_interval` must be strictly less than `timeout`. +Example valid: `timeout: 5m`, `batch_flush_interval: 2s` ✓ +Example invalid: `timeout: 2s`, `batch_flush_interval: 5s` ✗ + +## Validation Rules + +- `bootstrap_server` and `topic` are required +- `batch_flush_interval` must be `< timeout` in write mode +- `group_id` omitted → standalone reader, offsets **not** committed +- `group_id` set → coordinated consumer, offsets **are** committed after processing +- `user_auth_type: mtls` → returns error at runtime, do not use +- Credentials must use `{{ env "VAR" }}` or `{{ secret "/path" }}` +- Inline `cert` requires proper YAML block scalar formatting + +## Examples + +### Read — standalone, no auth +```yaml +- name: read_topic + type: kafka + bootstrap_server: kafka.local:9092 + topic: input-events + timeout: 25s + fail_on_error: true +``` + +### Read — group consumer (production) +```yaml +- name: consume_events + type: kafka + bootstrap_server: kafka.prod:9092 + topic: user-events + group_id: caterpillar-consumer-v1 + timeout: 25s +``` + +### Read — SCRAM + TLS +```yaml +- name: read_secure + type: kafka + bootstrap_server: kafka.prod:9093 + topic: secure-events + group_id: prod-consumer + user_auth_type: scram + username: "{{ env "KAFKA_USER" }}" + password: "{{ secret "/prod/kafka/password" }}" + server_auth_type: tls + cert_path: /etc/ssl/certs/kafka-ca.pem + timeout: 25s +``` + +### Write — SASL +```yaml +- name: publish_results + type: kafka + bootstrap_server: kafka.prod:9092 + topic: output-results + user_auth_type: sasl + username: "{{ env "KAFKA_USER" }}" + password: "{{ env "KAFKA_PASS" }}" + timeout: 5m + batch_size: 200 + batch_flush_interval: 3s +``` + +### Write — inline CA cert +```yaml +- name: publish_tls + type: kafka + bootstrap_server: kafka.prod:9093 + topic: events + server_auth_type: tls + cert: | + -----BEGIN CERTIFICATE----- + MIID... + -----END CERTIFICATE----- + timeout: 30s + batch_flush_interval: 2s +``` + +### Stop after 10 empty polls +```yaml +- name: drain_topic + type: kafka + bootstrap_server: kafka.local:9092 + topic: input-topic + retry_limit: 10 + timeout: 5s +``` + +## Anti-patterns + +- `batch_flush_interval >= timeout` in write mode → runtime error +- Using `user_auth_type: mtls` → not implemented, returns error +- Omitting `group_id` in production multi-instance deployments → no offset coordination +- Hardcoding `username` / `password` → use `{{ env "VAR" }}` or `{{ secret "/path" }}` +- Malformed inline PEM in `cert` (missing `|` block scalar) → TLS failure diff --git a/.claude/skills/pagination/SKILL.md b/.claude/skills/pagination/SKILL.md new file mode 100644 index 0000000..36f5ab3 --- /dev/null +++ b/.claude/skills/pagination/SKILL.md @@ -0,0 +1,815 @@ +--- +skill: pagination +version: 1.0.0 +caterpillar_type: http +description: Paginate through multi-page HTTP API responses using the next_page JQ field on the http task. +role: modifier (applied to http task) +requires_upstream: false +requires_downstream: true +aws_required: false +--- + +## Purpose + +The `next_page` field on an `http` task enables automatic pagination. After each +HTTP response, caterpillar evaluates the `next_page` JQ expression. If it +produces a URL string or request object, a follow-up request is made. When it +produces `null` or `empty`, pagination stops and the pipeline moves on. + +Every page's response body is emitted downstream as a separate record. + +## How it works + +``` +┌─────────────┐ ┌──────────────┐ ┌──────────────────┐ +│ HTTP request │────▶│ HTTP response│────▶│ Emit record │ +└─────────────┘ └──────┬───────┘ └──────────────────┘ + │ + ┌──────▼───────┐ + │ Evaluate │ + │ next_page JQ │ + └──────┬───────┘ + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + string object null/empty + (next URL) (full override) (stop) + │ │ + └─────┬──────┘ + ▼ + Next HTTP request + (loop continues) +``` + +## JQ input + +The `next_page` JQ expression receives a JSON object with two keys: + +```json +{ + "data": "", + "headers": { + "Content-Type": ["application/json"], + "Link": ["; rel=\"next\""] + } +} +``` + +| Key | Type | Description | +|-----|------|-------------| +| `data` | string | Raw HTTP response body. Use `.data \| fromjson` to parse as JSON. | +| `headers` | `map[string][]string` | Response headers. Each value is an array of strings. Header names use Go canonical form (`content-type` becomes `Content-Type`). | + +## Built-in variables + +| Variable | Access pattern | Description | +|----------|---------------|-------------| +| `page_id` | `[inputs][1].page_id` or `(input \| input \| .page_id)` | Page counter — starts at **2** on the first `next_page` call (page 1 is the initial request) and increments by 1 for each subsequent page. | + +Both access patterns are equivalent. `[inputs][1].page_id` is the array form; +`(input | input | .page_id)` is the sequential form — use whichever reads +better in your expression. + +## Return values + +| JQ result | Behavior | +|-----------|----------| +| `"https://..."` (string) | Makes the next request to this URL. Method, headers, and body remain unchanged from the current request. | +| `{ "endpoint": "...", ... }` (object) | Makes the next request using the fields from this object. Only `endpoint` is required; all other fields are optional overrides. | +| `null` | Stops pagination. | +| `empty` | Stops pagination (JQ produces no output). | + +### Object return schema + +```yaml +{ + "endpoint": "", # REQUIRED — URL for the next request + "method": "", # OPTIONAL — override HTTP method (e.g. POST) + "body": "", # OPTIONAL — override request body + "headers": { "": "" },# OPTIONAL — merged into existing headers + "proxy": { # OPTIONAL — proxy config for the next request + "scheme": "", # e.g. "http" + "host": "", # e.g. "proxy.internal:8080" + "insecure_tls": # skip TLS verification + } +} +``` + +When `headers` is provided, new headers are merged with existing ones. If a +header key already exists, the new value wins. + +### Partial object return + +You can return an object with only some fields — missing fields carry forward +from the current request. For example, returning only `body` keeps the current +endpoint, method, and headers: + +```yaml +next_page: | + .data | fromjson | + if (.items | length) == 500 then + { body: { pageNumber: (.currentPage + 1) } | @json } + else empty end +``` + +## Setting `next_page` dynamically + +There are two ways to set `next_page`: + +1. **Static** — defined directly on the `http` task in YAML. +2. **Dynamic** — set as a field in the upstream record's JSON. The HTTP task + merges upstream record fields into its config via `json.Unmarshal`, so + `next_page` from the record overrides the YAML value. + +This lets a JQ task upstream construct both the request and its pagination +logic at runtime. + +## Pagination patterns + +### Pattern 1: Cursor / token in response body + +The API returns a cursor or token in the JSON body. Check for its presence and +construct the next URL. This is the most common pagination pattern. + +```yaml +- name: fetch_all_items + type: http + method: GET + endpoint: https://marketplace.example.com/v3/items?limit=1000&nextCursor=* + expected_statuses: "200,401" + retry_delay: 70s + max_retries: 10 + next_page: >- + .data | fromjson | + if .nextCursor != null then + "https://marketplace.example.com/v3/items?limit=1000&nextCursor=\(.nextCursor)" + else null end +``` + +Common field names: `.nextCursor`, `.next_page_token`, `.nextToken`, +`.nextContinuationToken`, `.response_metadata.next_cursor`, +`.list.meta.nextCursor`, `.pagination.nextToken`. + +When tokens may contain special characters, URL-encode them with `@uri`: + +```yaml +next_page: | + .data | fromjson | + if (.nextContinuationToken // "") != "" then + "https://api.example.com/docs?continuationToken=" + (.nextContinuationToken | @uri) + else empty end +``` + +**When to use:** Walmart Marketplace (items, orders, listing quality), Slack +(`response_metadata.next_cursor`), Bol.com (orders), Lexion +(`nextContinuationToken`), Amazon SP-API Support Cases (`nextToken`), +Google Drive (`nextPageToken`), and most REST APIs with cursor/token-based +pagination. + +### Pattern 2: Offset calculated from `page_id` + +The API uses offset-based pagination. Use the built-in `page_id` counter to +compute the offset. + +```yaml +- name: fetch_inventory + type: http + endpoint: https://api.example.com/offers?limit=100&offset=0 + next_page: | + .data | fromjson | + if (.offers | length) == 100 then + "https://api.example.com/offers?limit=100&offset=" + + (([inputs][1].page_id - 1) * 100 | tostring) + else null end +``` + +**When to use:** Allegro (inventory offers), Rapid7 InsightIDR +(investigations index), Shelf Catalog API (page number), Threepn FNSKU API, +Pattern Inventory Hub (encumbrance states), Mirakl (product offers offset), +and any API that uses `offset` + `limit` without providing a next URL. + +### Pattern 3: Total count vs. fetched count + +The API returns a total count. Compare it against how many records you've +fetched so far to decide whether to continue. + +```yaml +- name: get_returns + type: http + next_page: | + .data | fromjson | + if .count > (.offset // 0) + (.customerReturns | length) then + "https://api.example.com/returns?limit=100&offset=" + + ((.offset // 0) + (.customerReturns | length) | tostring) + else null end +``` + +**When to use:** Allegro (returns — `.count` vs fetched), Bol.com (orders — +array length vs `pageSize`), and any API that returns a total count or where +you compare fetched batch size against a known page limit. + +### Pattern 4: Link header (RFC 5988) + +The API puts the next page URL in the `Link` response header. + +```yaml +- name: get_users + type: http + endpoint: https://api.example.com/v1/users?limit=30 + headers: + Authorization: {{ secret "/path/to/token" }} + max_retries: 100 + next_page: >- + .headers["Link"][] | + select(test("rel=\"next\"")) | + capture("<(?[^>]+)>").url +``` + +**When to use:** Okta (users API — `Link` header with `rel="next"`), GitHub, +and any API following RFC 5988 link relations where the full next URL is in the +`Link` response header. + +### Pattern 5: Link header with field extraction + +A variant where the next page token is embedded in the Link header URL and +must be extracted with a regex. + +```yaml +- name: get_catalog_items + type: http + next_page: >- + .headers.Link[0] | + match("after_id=([^&>]+)") | + .captures[0].string | + "https://api.example.com/products?per_page=1000&after_id=\(.)" +``` + +**When to use:** Target Plus (products catalog — `after_id` embedded in Link +header URL) and similar APIs where the next page cursor must be regex-extracted +from a Link header value rather than used as a complete URL. + +### Pattern 6: Object return — override endpoint, headers, and body + +When the next page request needs different headers, body, or method (e.g. +signed requests, rotating tokens), return a full object. + +```yaml +- name: get_orders + type: http + method: POST + next_page: | + .data | fromjson | + if .data.next_page_token and (.data.next_page_token != "") then + (now | floor | tostring) as $timestamp | + "SECRET_VALUE" as $app_secret | + ({date_from: "2024-01-01"} | tojson) as $body | + { + "endpoint": "https://api.example.com/orders/search?page_token=" + .data.next_page_token + "×tamp=" + $timestamp, + "headers": { + "Authorization": "Bearer {{ context "access_token" }}", + "Content-Type": "application/json" + }, + "body": $body + } + else null end +``` + +**When to use:** TikTok Shop (orders, products, prices, returns — UK and US +markets, HMAC-SHA256 signing per request), Coupang (CGF fees, revenue +settlement, product listings — CEA HMAC signing), Walmart Pricing Insights +(body-only override with `pageNumber`), Amazon SP-API Contacts (with proxy +config), and any API where each page request needs independently computed +authentication signatures, different body, or rotating headers. + +### Pattern 7: Full request override with context references + +Combine `next_page` object return with `{{ context "..." }}` references for +values extracted earlier in the pipeline. + +```yaml +- name: collect_listings + type: http + method: GET + expected_statuses: 200..299,400,403 + max_retries: 5 + next_page: | + .data | fromjson as $body | + ($body.pagination.nextToken // "") as $token | + if ($token | tostring) != "" then + { + endpoint: ("{{ context "base_endpoint" }}?{{ context "base_query" }}&pageToken=" + ($token | @uri)), + method: "GET", + headers: { + "x-amz-access-token": "{{ context "access_token" }}", + "Content-Type": "application/json" + }, + proxy: { + scheme: "http", + host: "rate-gate.prod.pattern.aws.internal:8080", + insecure_tls: true + } + } + else empty end +``` + +**When to use:** Amazon SP-API `searchListingsItems` (both 3P seller and +1P vendor flows — base endpoint, query string, access token, merchant ID, +and rate-limit scope all stored in context), and any multi-step pipeline +where auth tokens, base URLs, or query parameters from earlier tasks are +needed in pagination via `{{ context "..." }}`. + +### Pattern 8: Dynamic `next_page` from upstream JQ + +Set `next_page` as a field in the upstream JQ output. The HTTP task picks it +up from the record data automatically. + +```yaml +- name: build_request + type: jq + path: | + { + endpoint: "https://api.example.com/meetings?page_size=150", + headers: { + "Authorization": "Bearer {{ context "access_token" }}" + }, + next_page: ".data | fromjson | if (.next_page_token and (.next_page_token != \"\")) then (\"https://api.example.com/meetings?page_size=150&next_page_token=\" + (.next_page_token | @uri)) else empty end" + } + +- name: get_meetings + type: http + method: GET + fail_on_error: true +``` + +**When to use:** Zoom Meetings API (next_page_token in upstream JQ), Keepa +token-gate (sellers and products — dynamic `next_page` with `sleep()` for +rate-limiting), and any case where the pagination logic varies per-record, +needs runtime construction, or must embed rate-limiting behavior like +`sleep()` calls for API quota replenishment. + +### Pattern 9: Complex multi-field page tracking + +Some APIs require tracking multiple pagination fields (page ID, page size, +total pages) across requests. Return an object with extra fields to carry +state. + +```yaml +- name: fetch_reviews + type: http + expected_statuses: "200,504" + fail_on_error: true + next_page: | + (.data? // .) as $raw | + ($raw | (fromjson? // .)) as $resp | + ((($resp.reviews // $resp.reviewList // []) | length)) as $n | + ($resp.pageId // ([inputs][1].page_id // 0)) as $current | + ($resp.pageSize // ([inputs][1].page_size // 50)) as $page_size | + ($resp.totalPageCount // 0) as $total | + ($current + 1) as $next | + (20) as $max | + (if $total > 0 then ([$total, $max] | min) else $max end) as $stop | + if ($next < $stop) and (($total > 0) or ($n == $page_size)) then + { + method: "GET", + page_id: $next, + page_size: $page_size, + endpoint: ("https://api.example.com/reviews?pageId=" + ($next | tostring) + "&pageSize=" + ($page_size | tostring)) + } + else null end +``` + +Key techniques in this pattern: +- **Defensive parsing**: `(.data? // .) as $raw | ($raw | (fromjson? // .))` handles both string and pre-parsed input. +- **Multiple fallback fields**: `$resp.reviews // $resp.reviewList // []` tries alternative field names. +- **Carried state**: returning `page_id` and `page_size` in the object makes them available to subsequent `next_page` evaluations via `[inputs][1]`. +- **Max pages safety cap**: `(20) as $max` prevents runaway pagination loops. + +**When to use:** Amazon Seller Central Brand Customer Reviews (tracks +`pageId`, `pageSize`, `totalPageCount` with a max-pages safety cap), Seller +Central Voice of Customer (offset + page_id with full header override), +and any scraping or API scenario where multiple pagination fields must be +tracked across requests and a hard page limit prevents runaway loops. + +### Pattern 10: GraphQL cursor-based pagination + +GraphQL APIs typically paginate using a `pageInfo` object with `hasNextPage` +and `endCursor`. Since the query is sent as a POST body, `next_page` must +return an object that overrides the `body` with the updated cursor variable. + +```yaml +- name: fetch_all_products + type: http + method: POST + endpoint: https://api.example.com/graphql + headers: + Content-Type: application/json + Authorization: Bearer {{ context "access_token" }} + body: | + { + "query": "query($first: Int!, $after: String) { products(first: $first, after: $after) { edges { node { id name sku } } pageInfo { hasNextPage endCursor } } }", + "variables": { "first": 100 } + } + next_page: | + .data | fromjson | + if .data.products.pageInfo.hasNextPage then + { + "endpoint": "https://api.example.com/graphql", + "body": ({ + "query": "query($first: Int!, $after: String) { products(first: $first, after: $after) { edges { node { id name sku } } pageInfo { hasNextPage endCursor } } }", + "variables": { "first": 100, "after": .data.products.pageInfo.endCursor } + } | tojson) + } + else null end +``` + +For large queries, move the GraphQL query string into a context variable +upstream to avoid repeating it in both `body` and `next_page`: + +```yaml +- name: prepare_graphql_request + type: jq + path: | + "query($first: Int!, $after: String) { orders(first: $first, after: $after) { edges { node { id total } } pageInfo { hasNextPage endCursor } } }" as $query | + { + endpoint: "https://api.example.com/graphql", + headers: { + "Content-Type": "application/json", + "Authorization": "Bearer {{ context "access_token" }}" + }, + body: ({ query: $query, variables: { first: 50 } } | tojson), + next_page: ( + ".data | fromjson | if .data.orders.pageInfo.hasNextPage then { endpoint: \"https://api.example.com/graphql\", body: ({ query: " + ($query | tojson) + ", variables: { first: 50, after: .data.orders.pageInfo.endCursor } } | tojson) } else null end" + ) + } + +- name: fetch_orders + type: http + method: POST +``` + +**When to use:** Shopify, GitHub, and any GraphQL API that uses Relay-style +cursor pagination with `pageInfo { hasNextPage endCursor }`. + +### Pattern 11: HATEOAS links array in response body + +Some REST APIs return a `links` array in the response JSON with objects like +`{ "rel": "next", "href": "/path?offset=100" }`. Filter by `rel == "next"` +and extract the `href`. + +When the `href` is a relative path, prefix it with the API host: + +```yaml +- name: fetch_receipts + type: http + next_page: | + .data | fromjson | + (.links[] | select(.rel == "next" and .href != "") | + "https://api.otto.market\(.href)") // empty +``` + +When the API returns fully qualified URLs in `.href`, use it directly: + +```yaml +- name: get_exchange_rates + type: http + method: POST + endpoint: 'https://example.com/services/rest/query/v1/suiteql?limit=500' + headers: + Content-Type: application/json + body: '{"q": "SELECT * FROM exchange_rates"}' + next_page: >- + .data | fromjson | .links[] | select(.rel == "next") | .href + oauth: + realm: 12345 + token: {{ secret "/netsuite/token" }} + token_secret: {{ secret "/netsuite/token_secret" }} + consumer_key: {{ secret "/netsuite/consumer_key" }} + consumer_secret: {{ secret "/netsuite/consumer_secret" }} +``` + +**When to use:** Otto Market (receipts, inventory — relative `href` prefixed +with base URL), NetSuite SuiteQL (exchange rates, currencies — fully +qualified `href`), and any API following HATEOAS conventions where the next +page URL is in a `links` array with `rel: "next"`. + +### Pattern 12: Rate-limiting gate with `sleep()` + +Use `next_page` to poll a status endpoint repeatedly, sleeping between +checks, until a condition is met (e.g. API tokens are replenished). The +`sleep()` JQ function pauses execution before returning the URL. + +```yaml +- name: build_token_check + type: jq + path: | + { + endpoint: "https://api.example.com/token?key={{ secret "/api/token" }}", + next_page: "if (.data | fromjson | .tokensLeft | tonumber) < 100 then (\"https://api.example.com/token?key={{ secret "/api/token" }}\" | sleep(\"30s\")) else empty end" + } + +- name: token_gate + type: http + method: GET + timeout: 60s +``` + +When `tokensLeft` is below the threshold, the JQ returns the same URL +wrapped in `sleep("30s")`, causing a 30-second pause before the next poll. +Once enough tokens are available, it returns `empty` to stop and proceed. + +**When to use:** Keepa sellers and products pipelines (polls +`/token?key=...` endpoint, sleeps 30s when `tokensLeft` is below threshold), +and any API with rate-limiting where you must wait for token/quota +replenishment before making further data requests. + +### Pattern 13: Nested pagination object in response body + +Some APIs return a `next_page` or `paging` object in the response body +containing the next URL directly. + +```yaml +- name: get_custom_fields + type: http + endpoint: https://app.example.com/api/1.0/custom_fields?limit=100 + headers: + Authorization: Bearer {{ secret "/api/token" }} + next_page: | + .data | fromjson | + if (.next_page and .next_page.offset != null) + then .next_page.uri + else null end +``` + +**When to use:** Asana (custom fields, tasks — `.next_page.uri` contains the +full next URL when `.next_page.offset` is present), and APIs that return a +structured pagination object +(e.g. `{ "next_page": { "offset": "...", "uri": "https://..." } }`) rather +than a flat cursor field. + +### Pattern 14: Per-page HMAC signing + +APIs that require a unique cryptographic signature for every request need +the signing logic inside `next_page`. Use `now`, `hmac_sha256`, and +string concatenation to compute the signature per page. + +```yaml +- name: coupang_api + type: http + next_page: | + (input | input | .page_id) as $page_id | + if (.data | fromjson | .data | length) >= 20 then + (now | todateiso8601 | .[2:19] | gsub(":";"") | gsub("-";"") + "Z") as $datetime | + "GET" as $method | + "/v2/providers/openapi/apis/api/v1/vendors/{{ context "vendor_id" }}/settlement/cgf-fee/date-range" as $path | + ("fromDate={{ macros.ds_add(ds, -1) }}&toDate={{ ds }}&pageNum=" + ($page_id | tostring) + "&pageSize=50") as $query | + ($datetime + $method + $path + $query) as $message | + ($message | hmac_sha256($message; "{{ context "secret_key" }}")) as $sign | + { + "endpoint": "https://api-gateway.coupang.com" + $path + "?" + $query, + "headers": { + "Authorization": "CEA algorithm=HmacSHA256, access-key={{ context "access_key" }}, signed-date=" + $datetime + ", signature=" + $sign + } + } + else null end +``` + +Key elements: +- `now | todateiso8601` generates a fresh timestamp per page request. +- `hmac_sha256(message; secret)` computes the HMAC signature. +- Secrets are injected via `{{ context "..." }}` or `{{ secret "..." }}` — never hardcoded. + +For TikTok-style signing, the pattern is similar but concatenates path + +query parameters + body into the HMAC input: + +```yaml +next_page: | + .data | fromjson | + if .data.next_page_token and (.data.next_page_token != "") then + (now | floor | tostring) as $timestamp | + "{{ secret "/app_secret" }}" as $app_secret | + ("/order/202309/orders/search" + + "app_key" + "{{ secret "/app_key" }}" + + "page_size" + "100" + + "page_token" + .data.next_page_token + + "shop_cipher" + "{{ context "cipher" }}" + + "timestamp" + $timestamp + + $body) as $concat | + ($app_secret + $concat + $app_secret) as $input_string | + hmac_sha256($input_string; $app_secret) as $signed | + { + "endpoint": "https://open-api.tiktokglobalshop.com/order/202309/orders/search?app_key={{ secret "/app_key" }}&page_size=100&page_token=" + .data.next_page_token + "×tamp=" + $timestamp + "&sign=" + $signed, + "headers": { "x-tts-access-token": "{{ context "access_token" }}" }, + "body": $body + } + else null end +``` + +**When to use:** Coupang (CGF fees, revenue settlement — CEA HMAC signing), +TikTok Shop (orders, products, prices, returns — HMAC-SHA256 per page), +and any API that requires a unique cryptographic signature for every request. + +### Pattern 15: Batch-size comparison (count == limit) + +When the API doesn't return a cursor or total count, detect more pages by +comparing the current batch size to the page limit. If the batch is full, +request the next page; if it's smaller, you've reached the end. + +```yaml +- name: get_inventory + type: http + fail_on_error: true + next_page: | + .data | fromjson | + if .count == 100 then + "https://api.example.com/offers?limit=100&offset=" + + (([inputs][1].page_id - 1) * 100 | tostring) + else null end +``` + +This pattern often combines with `page_id`-based offset calculation +(Pattern 2). The stop condition is `batch_size < limit`. + +**When to use:** Allegro inventory (`.count == limit`), Goborderless FNSKU +(`length == per_page`), Shelf catalog (`length == per_page`), Rapid7 +InsightIDR (`length == 100`), and any API where a full batch implies more +data and a short batch means done. + +### Pattern 16: Offset from `page_id` with session headers + +Some scraping-style endpoints (Seller Central, internal APIs) require +session cookies or browser-like headers on every request. Combine +`page_id`-based offset with full header override in the returned object. + +```yaml +- name: fetch_voice_of_customer + type: http + next_page: | + .data | fromjson | + if (.pcrListings | length) == 25 then + { + method: "GET", + endpoint: ("https://sellercentral.amazon.com/pcrHealth/pcrListingSummary?pageSize=25&pageOffset=" + + ((([inputs][1].page_id // 0) + 1) * 25 | tostring) + + "&sortColumn=ORDERS_COUNT&sortDirection=DESCENDING"), + headers: { + "accept": "application/json", + "Cookie": "{{ context "cookie_header" }}", + "user-agent": "Mozilla/5.0 ..." + } + } + else null end + expected_statuses: "200,504" +``` + +**When to use:** Seller Central Voice of Customer (session cookies from +headless browser), and any endpoint that requires browser-like session +headers to be carried through pagination. + +## Choosing the right pattern + +| API behavior | Pattern | +|-------------|---------| +| Returns `nextCursor`, `next_page_token`, or similar | Pattern 1 (cursor) | +| Uses `offset` + `limit`, no next URL provided | Pattern 2 (page_id offset) | +| Returns `total` / `count` alongside results | Pattern 3 (total count) | +| Next URL in `Link` response header | Pattern 4 (Link header) | +| Cursor embedded in Link header URL | Pattern 5 (Link field extraction) | +| Each page request needs unique auth/signing | Pattern 6 (object return) | +| Auth tokens from earlier pipeline steps via context | Pattern 7 (context refs) | +| Pagination logic varies per-record or needs runtime construction | Pattern 8 (dynamic from upstream) | +| Multiple pagination fields to track (pageId, totalPages, etc.) | Pattern 9 (multi-field) | +| GraphQL API with `pageInfo { hasNextPage endCursor }` | Pattern 10 (GraphQL cursor) | +| Response body has `links: [{rel: "next", href: "..."}]` | Pattern 11 (HATEOAS links) | +| Must wait for API rate-limit / token replenishment | Pattern 12 (sleep gate) | +| Response body has nested pagination object (e.g. `.next_page.uri`) | Pattern 13 (nested paging object) | +| Each page needs a unique HMAC/signature computed in JQ | Pattern 14 (per-page HMAC) | +| No cursor or total — detect more pages by `batch_size == limit` | Pattern 15 (batch-size comparison) | +| Scraping endpoint requiring session cookies / browser headers | Pattern 16 (session headers offset) | + +## Common JQ idioms + +### URL-encoding tokens with `@uri` + +Many APIs return tokens that contain characters like `=`, `+`, or `/`. Use +`@uri` to URL-encode them before embedding in URLs: + +```jq +"https://api.example.com/items?nextToken=" + (.nextToken | @uri) +``` + +Some APIs (e.g. Slack) return cursors that are already partially encoded but +missing trailing `=` signs. Append them manually: + +```jq +.response_metadata.next_cursor + "%3D" +``` + +### Defensive JSON parsing + +When the response format may vary (string vs. pre-parsed JSON), use a +defensive chain: + +```jq +(.data? // .) as $raw | +($raw | (fromjson? // .)) as $resp | +``` + +This handles: raw string body (`.data | fromjson`), already-parsed JSON +(falls through to `.`), and missing `.data` key (falls back to `.`). + +### Safe defaults with `//` + +Use `//` to provide fallback values when fields may be absent: + +```jq +($resp.pageId // ([inputs][1].page_id // 0)) as $current | +($resp.pageSize // 50) as $page_size | +($resp.totalPageCount // 0) as $total | +(.offset // 0) + (.items | length) +``` + +### Multiple fallback field names + +When the API uses different field names across versions or endpoints: + +```jq +(($resp.reviews // $resp.reviewList // []) | length) as $n | +``` + +### Timestamp generation for signing + +For APIs requiring per-request timestamps: + +```jq +(now | floor | tostring) as $timestamp | +(now | todateiso8601 | .[2:19] | gsub(":";"") | gsub("-";"") + "Z") as $datetime | +``` + +### HMAC signing + +```jq +($message | hmac_sha256($message; $secret_key)) as $signature | +``` + +### Object construction with `tojson` / `@json` + +Convert objects to JSON strings for request bodies: + +```jq +{ body: { pageNumber: (.currentPage + 1), sort: { field: "date" } } | @json } +``` + +## Resilience settings for paginated sources + +Paginated HTTP tasks should include resilience settings appropriate to the +API. These are set on the `http` task alongside `next_page`: + +| Field | Default | Description | +|-------|---------|-------------| +| `expected_statuses` | `"200"` | Comma-separated or range. E.g. `"200,401"`, `"200..299,400,403"`, `"200,504"`. | +| `max_retries` | `3` | Number of retry attempts per page. Set higher for flaky APIs (e.g. `10`, `100`). | +| `retry_delay` | `5` | Seconds between retries. Use longer delays for rate-limited APIs (e.g. `70s`). | +| `retry_backoff_factor` | `1` | Multiplier for exponential backoff. Set `2` for doubling delay. | +| `timeout` | `90` | Request timeout in seconds. Increase for slow APIs. | +| `fail_on_error` | `false` | When `true`, a page failure stops the pipeline. Recommended for source tasks. | + +Example with full resilience config: + +```yaml +- name: collect_listings + type: http + expected_statuses: 200..299,400,403,500,503 + max_retries: 5 + retry_backoff_factor: 2 + timeout: 120 + fail_on_error: true + next_page: ... +``` + +## Validation rules + +- `next_page` input is `{"data": "...", "headers": {...}}` — NOT the parsed body. Always `.data | fromjson` first. +- Response headers are accessed via `.headers["Header-Name"]` — values are arrays of strings. +- Return `null` or `empty` to stop. Returning an empty string `""` will be treated as an endpoint URL and cause an error. +- `page_id` starts at **2** (the initial request is page 1). The first `next_page` evaluation sees `page_id = 2`. +- When returning an object, `endpoint` is **required** unless you are doing a partial override (e.g. body-only). Missing `endpoint` with no carry-forward will silently stop pagination. +- `headers` in the returned object are **merged** — they don't replace all existing headers, they add/override individual keys. +- `{{ context "..." }}` and `{{ secret "..." }}` templates are resolved **before** the JQ expression is parsed, so they work inside `next_page`. +- When setting `next_page` dynamically from upstream, the value must be a JQ expression **string**, not a pre-evaluated object. +- `proxy` in the returned object is applied to the next request only — it does not persist across subsequent pages unless returned each time. + +## Anti-patterns + +- **Accessing response fields directly** (`.nextCursor`) without `.data | fromjson` — the response body is a raw string inside `{"data": "...", "headers": {...}}`. +- **Using `""` to stop pagination** — use `null` or `empty`. An empty string is treated as a URL and causes an error. +- **Hardcoding secrets** in `next_page` JQ — use `{{ secret "/path" }}` or `{{ context "key" }}`. +- **Off-by-one errors with `page_id`** — remember it starts at 2, not 1. The first `next_page` call has `page_id = 2` because page 1 is the initial request. +- **Infinite pagination loops** — always include a condition that eventually produces `null`/`empty`: + - Check cursor presence: `if .nextCursor != null then ... else null end` + - Check batch size: `if (.items | length) == limit then ... else null end` + - Set a max page cap: `(20) as $max | if $next < $max then ... else null end` +- **Forgetting `fail_on_error: true`** on paginated sources — a single page failure will silently stop pagination without it. +- **Mismatched offset multiplier and limit** — when using `page_id` offset, ensure `(page_id - 1) * N` uses the same `N` as the `limit` parameter in the URL. +- **Not URL-encoding tokens** — tokens with special characters (`=`, `+`, `/`, `&`) will break the URL. Use `| @uri` to encode them. +- **Forgetting `proxy` on subsequent pages** — if the initial request uses a proxy, the `next_page` object must include `proxy` on every page. Proxy does not carry forward automatically. +- **Recomputing timestamps outside `next_page`** — for signed APIs, the timestamp must be generated inside the `next_page` JQ (via `now | floor`) so each page gets a fresh signature. Using a static timestamp will cause signature mismatches. diff --git a/.claude/skills/pipeline-builder/SKILL.md b/.claude/skills/pipeline-builder/SKILL.md new file mode 100644 index 0000000..0d9893c --- /dev/null +++ b/.claude/skills/pipeline-builder/SKILL.md @@ -0,0 +1,443 @@ +--- +skill: pipeline-builder +version: 1.0.0 +description: Generate a caterpillar YAML pipeline from a natural language description. Outputs a ready-to-run pipeline file. +--- + +## Purpose + +You are a caterpillar pipeline author. When the user describes a data flow in natural language, produce a valid `tasks:` YAML block using only the task types listed below. Each task is an element in the `tasks:` list. The pipeline runs tasks sequentially — the output of each task is the input to the next. + +Do not explain the pipeline unless the user asks. Just output the YAML (fenced with ```yaml). + +--- + +## Available Task Types + +| type | role | notes | +|------|------|-------| +| `file` | source or sink | first task = read; last (or has upstream) = write. Supports local path, S3 (`s3://`), and glob patterns. | +| `kafka` | source or sink | first task = read; has upstream = write. Supports TLS + SASL/SCRAM. | +| `sqs` | source or sink | first task = read; has upstream = write. AWS SQS. | +| `http` | source or sink | first task = fetch URL; has upstream = POST each record. Supports pagination, OAuth 1.0/2.0. | +| `http_server` | source only | listens on a port, emits inbound requests as records. | +| `aws_parameter_store` | source or sink | reads/writes SSM parameters. | +| `sns` | sink only | publishes records to AWS SNS. Terminal — no downstream. | +| `echo` | sink or pass-through | prints to stdout. Terminal when last; pass-through when not last. | +| `split` | transform | splits a record's data string on a delimiter into multiple records. | +| `join` | transform | batches N records into one, separated by a delimiter. | +| `jq` | transform | applies a JQ expression to each record's JSON. `explode: true` to split array output. | +| `replace` | transform | Go RE2 regex find-and-replace on record data string. | +| `flatten` | transform | flattens nested JSON into single-level keys with `_` separators. | +| `xpath` | transform | extracts data from XML/HTML using XPath. | +| `converter` | transform | converts between CSV, HTML, XLSX, XLS, EML, SST formats. | +| `compress` | transform | gzip/snappy/zlib/deflate compress or decompress. | +| `archive` | transform | pack/unpack zip or tar archives. | +| `sample` | filter | head, tail, nth, random, or percent sampling. | +| `delay` | rate-limit | inserts a fixed pause between records. | +| `heimdall` | transform | submits jobs to Heimdall orchestration platform. | + +--- + +## Pipeline Structure + +```yaml +tasks: + - name: + type: + # ... task-specific fields +``` + +**Rules:** +- Every task needs a unique `name` and a `type`. +- The first task must be a source (no upstream required): `file`, `kafka`, `sqs`, `http`, `http_server`, `aws_parameter_store`. +- The last task is usually a sink: `file`, `kafka`, `sqs`, `sns`, `echo`. +- Transforms sit between source and sink. +- Multiple tasks of the same type can appear — give each a distinct name. + +--- + +## Common Fields (all tasks) + +```yaml +fail_on_error: # OPTIONAL — stop pipeline on error (default: false) +``` + +--- + +## Task Schemas (key fields only) + +### file +```yaml +- name: + type: file + path: # local path, s3://bucket/key, or glob + region: # OPTIONAL — AWS region (default: us-west-2, S3 only) + delimiter: # OPTIONAL — record separator in read mode (default: \n) + success_file: # OPTIONAL — write _SUCCESS marker (write mode only) +``` + +### kafka +```yaml +- name: + type: kafka + bootstrap_server: # host:port + topic: + timeout: # OPTIONAL (default: 15s) + group_id: # OPTIONAL — consumer group + server_auth_type: # OPTIONAL — "none" | "tls" + cert: # OPTIONAL — inline PEM (use | block scalar) + cert_path: # OPTIONAL — path to CA cert + user_auth_type: # OPTIONAL — "none" | "sasl" | "scram" + username: # OPTIONAL + password: # OPTIONAL + batch_size: # OPTIONAL — write mode (default: 100) + batch_flush_interval: # OPTIONAL — must be < timeout (default: 2s) + retry_limit: # OPTIONAL — empty-poll retries (default: 5) +``` + +### sqs +```yaml +- name: + type: sqs + queue_url: + concurrency: # OPTIONAL (default: 10) + max_messages: # OPTIONAL — max 10 (default: 10) + wait_time: # OPTIONAL — long-poll seconds (default: 10) + exit_on_empty: # OPTIONAL — stop when queue drains (default: false) + message_group_id: # OPTIONAL — required for FIFO queue writes +``` + +### http +```yaml +- name: + type: http + endpoint: + method: # OPTIONAL (default: GET) + headers: # OPTIONAL + body: # OPTIONAL + timeout: # OPTIONAL — seconds (default: 90) + max_retries: # OPTIONAL (default: 3) + expected_statuses: # OPTIONAL (default: "200") + next_page: # OPTIONAL — JQ expr for pagination + context: # OPTIONAL — extract response values +``` + +### http_server +```yaml +- name: + type: http_server + port: # REQUIRED + path: # OPTIONAL — URL path (default: /) + method: # OPTIONAL (default: POST) +``` + +### sqs / sns (sns is write-only) +```yaml +- name: + type: sns + topic_arn: + region: # OPTIONAL (default: us-west-2) + message_group_id: # OPTIONAL — FIFO topics +``` + +### aws_parameter_store +```yaml +- name: + type: aws_parameter_store + path: # SSM parameter path + region: # OPTIONAL (default: us-west-2) + recursive: # OPTIONAL — read subtree (default: false) +``` + +### echo +```yaml +- name: + type: echo + only_data: # OPTIONAL — true = data only; false = full record JSON (default: false) +``` + +### split +```yaml +- name: + type: split + delimiter: # OPTIONAL (default: \n) +``` + +### join +```yaml +- name: + type: join + number: # REQUIRED — records per batch + delimiter: # OPTIONAL (default: \n) + timeout: # OPTIONAL — flush after duration + size: # OPTIONAL — flush after byte size (e.g. "1MB") +``` + +### jq +```yaml +- name: + type: jq + path: # REQUIRED — JQ expression + explode: # OPTIONAL — split array output into records (default: false) + as_raw: # OPTIONAL — emit raw string (default: false) + context: # OPTIONAL — store JQ values in record context +``` + +### replace +```yaml +- name: + type: replace + pattern: # REQUIRED — Go RE2 regex + replacement: # REQUIRED — replacement string +``` + +### flatten +```yaml +- name: + type: flatten + separator: # OPTIONAL (default: _) +``` + +### xpath +```yaml +- name: + type: xpath + expression: # REQUIRED — XPath expression + index: # OPTIONAL — select nth match (0-based) +``` + +### converter +```yaml +- name: + type: converter + from: # REQUIRED — source format: csv | html | xlsx | xls | eml | sst + to: # REQUIRED — target format: csv | html | xlsx | json + skip_rows: # OPTIONAL — rows to skip + columns: # OPTIONAL — column names override +``` + +### compress +```yaml +- name: + type: compress + format: # REQUIRED — gzip | snappy | zlib | deflate + mode: # OPTIONAL — "compress" | "decompress" (default: compress) +``` + +### archive +```yaml +- name: + type: archive + format: # REQUIRED — zip | tar + mode: # REQUIRED — "pack" | "unpack" +``` + +### sample +```yaml +- name: + type: sample + strategy: # REQUIRED — head | tail | nth | random | percent + value: # REQUIRED — N records, every Nth, or percent (0–100) +``` + +### delay +```yaml +- name: + type: delay + duration: # REQUIRED — e.g. "500ms", "1s", "2m" +``` + +--- + +## Template Functions (use in string fields) + +| Function | Resolves | +|----------|---------| +| `{{ env "VAR" }}` | environment variable (once at init) | +| `{{ secret "/ssm/path" }}` | AWS SSM secret (once at init) | +| `{{ macro "timestamp" }}` | current timestamp per record | +| `{{ macro "uuid" }}` | random UUID per record | +| `{{ macro "unixtime" }}` | unix timestamp per record | +| `{{ context "key" }}` | value stored by upstream task's `context:` block | + +Always use `{{ secret "..." }}` or `{{ env "..." }}` for credentials — never hardcode them. + +--- + +## Decision Guide + +| User says | Start with | +|-----------|-----------| +| "read from file / S3" | `type: file` as source | +| "read from Kafka" | `type: kafka` as source | +| "read from SQS" | `type: sqs` as source | +| "call an API / fetch URL" | `type: http` as source | +| "receive webhooks / inbound HTTP" | `type: http_server` as source | +| "write to file / S3" | `type: file` as sink | +| "publish to Kafka" | `type: kafka` as sink | +| "send to SQS" | `type: sqs` as sink | +| "publish to SNS" | `type: sns` as sink | +| "transform / reshape JSON" | `type: jq` | +| "split lines / split by delimiter" | `type: split` | +| "batch / group records" | `type: join` | +| "compress / decompress" | `type: compress` | +| "zip / tar / unpack archive" | `type: archive` | +| "convert CSV/Excel/HTML" | `type: converter` | +| "parse XML / HTML / extract field" | `type: xpath` | +| "flatten nested JSON" | `type: flatten` | +| "filter / sample records" | `type: sample` | +| "rate limit / throttle" | `type: delay` | +| "regex replace in data" | `type: replace` | +| "debug / print output" | `type: echo` | +| "read SSM parameters" | `type: aws_parameter_store` | +| "submit to Heimdall" | `type: heimdall` | + +--- + +## Writing JSON to a File — Output Format Rules + +When the sink is a `file` and the data is JSON, choose the right output format: + +### Single JSON array (multiple records → one file) +**Correct approach:** Use a single `jq` that wraps the whole result in an array `[...]` — no `explode`, no `join`, no `replace`. + +```yaml +- name: transform + type: jq + path: | + [.items[] | { "id": .id, "name": .name }] # array wrapping happens inside jq + +- name: write + type: file + path: output/results.json +``` + +**Why:** `explode: true` + `join` + `replace` to reconstruct an array is fragile and produces malformed output. Let `jq` build the array natively. + +### NDJSON (one JSON object per line — for streaming/large datasets) +Use `explode: true` + no `join`. Each record becomes its own line in the file. + +```yaml +- name: explode_items + type: jq + path: .items[] + explode: true + +- name: write + type: file + path: output/results.ndjson +``` + +### Decision rule +| Goal | Pattern | +|------|---------| +| One valid JSON array file | `jq` with `[.items[] \| {...}]` — array inside jq, no explode | +| One file per record | `explode: true`, no join | +| NDJSON (one JSON per line) | `explode: true`, no join, `.ndjson` extension | +| Batch N records as JSON array per file | `explode: true` → `join number: N` → `jq` to parse and re-wrap | + +--- + +## Output Instructions + +1. Output only the YAML (fenced ```yaml block). No preamble unless asked. +2. Choose the minimal set of tasks that satisfies the request. +3. Use `{{ secret "..." }}` or `{{ env "..." }}` for any credentials or URLs that should not be hardcoded. +4. Add `fail_on_error: true` to source tasks in production pipelines. +5. If the user's request is ambiguous, make a sensible default choice and add a short comment (`#`) in the YAML explaining the assumption. +6. If the user mentions saving to a file, use `type: file` as the last task. +7. If the user wants to see output in the terminal, add `type: echo` with `only_data: true` as the last task. + +--- + +## Examples + +### User: "Read a local CSV file, convert it to JSON, and write each row to SQS" +```yaml +tasks: + - name: read_csv + type: file + path: data/input.csv + fail_on_error: true + + - name: convert_to_json + type: converter + from: csv + to: json + + - name: send_to_sqs + type: sqs + queue_url: '{{ env "SQS_QUEUE_URL" }}' +``` + +### User: "Poll a Kafka topic with SCRAM auth and write each message to S3" +```yaml +tasks: + - name: read_kafka + type: kafka + bootstrap_server: '{{ secret "/kafka/bootstrap_server" }}' + topic: my-topic + group_id: caterpillar-consumer + user_auth_type: scram + username: '{{ env "KAFKA_USER" }}' + password: '{{ secret "/kafka/password" }}' + server_auth_type: tls + cert_path: /etc/ssl/certs/kafka-ca.pem + timeout: 25s + fail_on_error: true + + - name: write_s3 + type: file + path: 's3://my-bucket/output/{{ macro "timestamp" }}.json' + region: us-east-1 +``` + +### User: "Fetch paginated JSON from an API, extract the items array, and echo each item" +```yaml +tasks: + - name: fetch_api + type: http + endpoint: 'https://api.example.com/items?page=1' + method: GET + headers: + Authorization: 'Bearer {{ env "API_TOKEN" }}' + next_page: '.next_page_url // empty' + + - name: explode_items + type: jq + path: .items[] + explode: true + + - name: print_items + type: echo + only_data: true +``` + +### User: "Read lines from a file, batch every 10 lines with pipe separator, gzip, write to S3" +```yaml +tasks: + - name: read_file + type: file + path: data/records.txt + fail_on_error: true + + - name: split_lines + type: split + delimiter: "\n" + + - name: batch_records + type: join + number: 10 + delimiter: "|" + + - name: compress + type: compress + format: gzip + + - name: write_s3 + type: file + path: 's3://my-bucket/batched/output_{{ macro "uuid" }}.gz' + region: us-west-2 + success_file: true +``` diff --git a/.claude/skills/pipeline-tester/SKILL.md b/.claude/skills/pipeline-tester/SKILL.md new file mode 100644 index 0000000..17dd879 --- /dev/null +++ b/.claude/skills/pipeline-tester/SKILL.md @@ -0,0 +1,348 @@ +--- +skill: pipeline-tester +version: 1.0.0 +description: Generates a step-by-step test plan for a pipeline under development. Produces source inspection commands, sample data capture steps, and probe pipelines that test each transform in isolation before wiring the full pipeline together. +--- + +## Purpose + +You are a pipeline testing coach for caterpillar. When a data engineer is building a pipeline, testing it all at once is hard — failures are hard to locate and there's no visibility into what data looks like between tasks. + +The correct approach is **incremental testing**: + +1. **Inspect the source** — verify real data exists and see its shape before writing any pipeline +2. **Capture a sample** — save a small slice of real data to a local file +3. **Test each transform in isolation** — build a probe pipeline per transform stage using the captured sample +4. **Chain forward** — add one transform at a time and verify output before adding the next +5. **Verify the sink** — confirm the final record shape matches what the sink expects + +When given a pipeline YAML, produce a full test plan following this approach. + +--- + +## Step 1 — Inspect the Source + +Generate the exact command to inspect the real source before running any pipeline. + +### HTTP +```bash +# Basic GET +curl -s "https://api.example.com/endpoint" | jq . + +# With auth header +curl -s -H "Authorization: Bearer $API_TOKEN" "https://api.example.com/endpoint" | jq . + +# POST with body +curl -s -X POST "https://api.example.com/endpoint" \ + -H "Content-Type: application/json" \ + -d '{"key": "value"}' | jq . + +# Paginated — check first page + next_page field +curl -s "https://api.example.com/items?page=1" | jq '{ count: (.items | length), next: .next_page_url, first_item: .items[0] }' +``` + +### S3 +```bash +# List files in prefix +aws s3 ls s3://bucket/prefix/ --region us-east-1 + +# Preview a file (first 5 lines) +aws s3 cp s3://bucket/prefix/file.json - --region us-east-1 | head -5 + +# List all files matching a pattern +aws s3 ls s3://bucket/prefix/ --region us-east-1 | grep ".json" + +# Check file size before downloading +aws s3 ls s3://bucket/prefix/file.json --region us-east-1 --human-readable +``` + +### SQS +```bash +# Peek at messages without consuming (VisibilityTimeout=0 returns them immediately) +aws sqs receive-message \ + --queue-url "https://sqs.us-east-1.amazonaws.com/123456789/my-queue" \ + --max-number-of-messages 1 \ + --visibility-timeout 0 \ + --region us-east-1 | jq '.Messages[0].Body | fromjson' + +# Check queue depth +aws sqs get-queue-attributes \ + --queue-url "https://sqs.us-east-1.amazonaws.com/123456789/my-queue" \ + --attribute-names ApproximateNumberOfMessages \ + --region us-east-1 +``` + +### Kafka +```bash +# Consume a few messages and exit (requires kafka-console-consumer or kcat) +# Using kcat (recommended): +kcat -b kafka.host:9092 -t my-topic -C -c 5 -e \ + -X security.protocol=SASL_SSL \ + -X sasl.mechanisms=SCRAM-SHA-512 \ + -X sasl.username=$KAFKA_USER \ + -X sasl.password=$KAFKA_PASS + +# Using kafka-console-consumer: +kafka-console-consumer.sh \ + --bootstrap-server kafka.host:9092 \ + --topic my-topic \ + --max-messages 5 \ + --from-beginning + +# OR use a minimal caterpillar probe pipeline (see Step 2) +``` + +### Local File +```bash +# Preview content +head -5 data/input.txt +head -5 data/input.json | jq . + +# Count records +wc -l data/input.txt + +# Check encoding / format +file data/input.csv +``` + +### AWS Parameter Store +```bash +# Read a single parameter +aws ssm get-parameter --name "/prod/kafka/password" --with-decryption --region us-east-1 | jq . + +# List parameters under a path +aws ssm get-parameters-by-path --path "/prod/kafka/" --recursive --region us-east-1 | jq '.Parameters[] | { name: .Name, value: .Value }' +``` + +--- + +## Step 2 — Capture Sample Data to a Local File + +Once you can see real data from the source, capture a small sample to a local file. This becomes the input for all your transform probe pipelines — no live connections needed. + +### Capture via caterpillar probe pipeline + +Create `test/pipelines/probe_capture_.yaml`: + +```yaml +# CAPTURE PROBE — run once to save sample data locally +# Replace source task with your real source config +tasks: + - name: source + type: + # ... your source config ... + + - name: take_sample + type: sample + filter: head + limit: 10 # capture first 10 records + + - name: save_sample + type: file + path: test/pipelines/samples/_sample.json +``` + +Run it: +```bash +./caterpillar -conf test/pipelines/probe_capture_.yaml +``` + +Now you have `test/pipelines/samples/_sample.json` — a local file with real data shaped exactly as the source produces it. Use this for all transform testing. + +### Capture via CLI (HTTP / S3) + +```bash +# HTTP +curl -s "https://api.example.com/items" > test/pipelines/samples/api_sample.json + +# S3 +aws s3 cp s3://bucket/prefix/file.json test/pipelines/samples/s3_sample.json --region us-east-1 + +# SQS (single message body) +aws sqs receive-message \ + --queue-url "..." --max-number-of-messages 1 --visibility-timeout 0 \ + | jq -r '.Messages[0].Body' > test/pipelines/samples/sqs_sample.json +``` + +--- + +## Step 3 — Build a Probe Pipeline Per Transform Stage + +For each transform task in the pipeline, build an isolated probe pipeline: +- **Source**: local file from Step 2 +- **Single transform**: the task under test +- **Sink**: `echo` with `only_data: true` + +### Probe template + +```yaml +# PROBE: testing +tasks: + - name: load_sample + type: file + path: test/pipelines/samples/_sample.json + + - name: + type: + # ... transform config ... + + - name: inspect_output + type: echo + only_data: true +``` + +Run it: +```bash +./caterpillar -conf test/pipelines/probe_.yaml +``` + +### Per-transform verification checklist + +**`jq` transform** +- Does the output have the expected fields? +- If `explode: true`, does each element of the array become a separate record? +- Are `{{ context "key" }}` substitutions rendering correctly or as literal strings? + +**`split` transform** +- Is each line becoming a separate record? +- Are there empty records from trailing newlines? Add `jq` filter: `select(. != "")` + +**`join` transform** +- Are records being batched at the right size? +- Is the delimiter correct in the joined output? +- Does the last partial batch flush? (Add `timeout` if needed) + +**`replace` transform** +- Does the regex match the intended data? +- Test the regex independently: `echo "your data" | sed 's/pattern/replacement/'` + +**`converter` transform +- Is the input format what converter expects? (CSV with headers, EML with MIME structure, etc.) +- Does the output JSON have the expected field names? + +**`xpath` transform** +- Test the XPath expression independently: `echo "" | xmllint --xpath "//field" -` +- Is the correct element selected when there are multiple matches? + +**`flatten` transform** +- Are nested keys joined with `_` as expected? +- Are arrays flattened or preserved? + +--- + +## Step 4 — Chain Transforms Incrementally + +After each transform probe passes, build a chained probe that combines transforms tested so far: + +```yaml +# CHAIN PROBE: source → transform_1 → transform_2 (adding transform_2) +tasks: + - name: load_sample + type: file + path: test/pipelines/samples/_sample.json + + - name: transform_1 # already verified + type: jq + path: .items[] + explode: true + + - name: transform_2 # now being added + type: replace + expression: ^(.*)$ + replacement: '{"wrapped": "$1"}' + + - name: inspect_output + type: echo + only_data: true +``` + +**Rule**: only add one new transform per iteration. If output breaks, you know exactly which task caused it. + +--- + +## Step 5 — Verify the Sink + +Before connecting the real sink (S3, SQS, Kafka), run a final probe with a local file sink to inspect the exact records that would be written: + +```yaml +# SINK VERIFICATION PROBE +tasks: + - name: load_sample + type: file + path: test/pipelines/samples/_sample.json + + # ... all transforms (already verified) ... + + - name: write_to_local_for_inspection + type: file + path: test/pipelines/samples/_output.json +``` + +Then inspect: +```bash +cat test/pipelines/samples/_output.json | jq . +wc -l test/pipelines/samples/_output.json # record count +``` + +Confirm: +- Record count matches expectations +- Field names and types match what the sink expects +- No empty records or malformed JSON + +--- + +## Step 6 — Smoke Test Against Real Sink (Dry Run) + +When the local sink verification passes, do a limited smoke test against the real sink: + +```yaml +# SMOKE TEST — real sink, limited records +tasks: + - name: source + type: + # ... config ... + + - name: take_sample # limit to 1-3 records for smoke test + type: sample + filter: head + limit: 3 + + # ... transforms ... + + - name: real_sink + type: + # ... sink config ... + fail_on_error: true +``` + +Then verify at the sink: +```bash +# S3 — did the file appear? +aws s3 ls s3://bucket/output/ --region us-east-1 | tail -3 + +# SQS — did messages arrive? +aws sqs get-queue-attributes \ + --queue-url "..." \ + --attribute-names ApproximateNumberOfMessages + +# Kafka — did messages arrive? (kcat) +kcat -b kafka.host:9092 -t output-topic -C -c 3 -e + +# HTTP — did the POST succeed? (check target system or logs) +``` + +--- + +## Output: Full Test Plan + +When given a pipeline YAML, output a complete test plan with: + +1. **Source inspection command** — exact CLI command for the source type +2. **Sample capture pipeline** — ready-to-run YAML saved to `test/pipelines/probe_capture_.yaml` +3. **Per-transform probe pipelines** — one YAML per transform, saved to `test/pipelines/probe_.yaml` +4. **Sink verification probe** — local file sink YAML +5. **Smoke test pipeline** — real sink with `sample: head limit: 3` +6. **Sink verification commands** — CLI commands to confirm records arrived at the real sink + +Format each pipeline as a fenced ```yaml block with its filename as a comment header. +Label each step clearly so the engineer can work through them in order. diff --git a/.claude/skills/replace/SKILL.md b/.claude/skills/replace/SKILL.md new file mode 100644 index 0000000..42ed1df --- /dev/null +++ b/.claude/skills/replace/SKILL.md @@ -0,0 +1,161 @@ +--- +skill: replace +version: 1.0.0 +caterpillar_type: replace +description: Apply a Go RE2 regex find-and-replace to each record's data string. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Applies a regular expression to the entire record data string and replaces matches. +Operates on raw string data — not JSON fields. Use a `jq` task upstream to extract a specific field first. + +## Schema + +```yaml +- name: # REQUIRED + type: replace # REQUIRED + expression: # REQUIRED — Go RE2 regex pattern + replacement: # REQUIRED — replacement string ($1, $2 for capture groups) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Clean whitespace | `expression: "\\s+"`, `replacement: " "` | +| Remove characters | `replacement: ""` | +| Capture and reorder groups | `expression: "(a)(b)"`, `replacement: "$2$1"` | +| Add prefix/suffix | `expression: "^(.*)$"`, `replacement: "PREFIX: $1"` | +| Extract pattern from text | `expression: ".*().*"`, `replacement: "$1"` | +| Operate on a specific JSON field | add `jq` task upstream to extract the field first | +| Need lookahead/lookbehind | **not supported** (RE2) — restructure logic | + +## Capture Group Reference + +Go regex uses `$N` for group references (not `\N`): +``` +$0 → entire match +$1 → first capture group +$2 → second capture group +``` + +## YAML Escaping + +Backslashes must be doubled inside YAML quoted strings: + +| Regex intent | YAML value | +|-------------|------------| +| `\d` | `"\\d"` | +| `\s` | `"\\s"` | +| `\w` | `"\\w"` | +| `\.` | `"\\."` | +| `\n` | `"\\n"` | +| `\t` | `"\\t"` | +| `\\` | `"\\\\"` | + +## Validation Rules + +- Both `expression` and `replacement` are required +- Go uses RE2 syntax — no lookaheads `(?=...)` or lookbehinds `(?<=...)` +- `expression` applies to the entire record data string, not a single JSON field +- Capture group references use `$1` not `\1` +- Backslashes must be doubled in YAML string values + +## RE2 Quick Reference + +| Pattern | Matches | +|---------|---------| +| `.` | any character except `\n` | +| `\d` | digit | +| `\w` | word char `[a-zA-Z0-9_]` | +| `\s` | whitespace | +| `^` | start of string | +| `$` | end of string | +| `*` | 0 or more | +| `+` | 1 or more | +| `?` | 0 or 1 | +| `[abc]` | character class | +| `[^abc]` | negated class | +| `(a\|b)` | alternation | +| `(...)` | capture group | +| `(?:...)` | non-capture group | + +## Examples + +### Normalize whitespace +```yaml +- name: clean_spaces + type: replace + expression: "\\s+" + replacement: " " +``` + +### Add greeting prefix +```yaml +- name: greet + type: replace + expression: "^(.*)$" + replacement: "Hello, $1!" +``` + +### Reformat date YYYY-MM-DD → MM/DD/YYYY +```yaml +- name: reformat_date + type: replace + expression: "(\\d{4})-(\\d{2})-(\\d{2})" + replacement: "$2/$3/$1" +``` + +### Format phone number +```yaml +- name: format_phone + type: replace + expression: "(\\d{3})(\\d{3})(\\d{4})" + replacement: "($1) $2-$3" +``` + +### Strip HTML tags +```yaml +- name: strip_html + type: replace + expression: "<[^>]*>" + replacement: "" +``` + +### Remove non-alphanumeric characters +```yaml +- name: alphanumeric_only + type: replace + expression: "[^a-zA-Z0-9\\s]" + replacement: "" +``` + +### Extract domain from URL +```yaml +- name: extract_domain + type: replace + expression: "https?://([^/]+).*" + replacement: "$1" +``` + +### Extract email from text +```yaml +- name: extract_email + type: replace + expression: ".*([a-zA-Z0-9._%+\\-]+@[a-zA-Z0-9.\\-]+\\.[a-zA-Z]{2,}).*" + replacement: "$1" +``` + +## Anti-patterns + +- Using `\1` for capture groups instead of `$1` — Go uses `$` notation +- Single backslash in YAML: `\d` — must be `"\\d"` +- Using lookaheads `(?=...)` — not supported in RE2; restructure with capture groups +- Applying `replace` to a JSON object without first extracting the target field with `jq` +- Using `replace` when a `jq` transform would be cleaner for structured data diff --git a/.claude/skills/sample/SKILL.md b/.claude/skills/sample/SKILL.md new file mode 100644 index 0000000..60144f1 --- /dev/null +++ b/.claude/skills/sample/SKILL.md @@ -0,0 +1,144 @@ +--- +skill: sample +version: 1.0.0 +caterpillar_type: sample +description: Filter records using a sampling strategy — head, tail, nth, random, or percent. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Selects a subset of records using one of five strategies. Useful for development (limit data volume), QA (representative sampling), and performance throttling. + +**Constraint**: cannot be the first or last task — requires both input and output channels. + +## Schema + +```yaml +- name: # REQUIRED + type: sample # REQUIRED + filter: # OPTIONAL — strategy (default: random) + limit: # OPTIONAL — record count (head, tail, nth) + percent: # OPTIONAL — percent to keep (random, percent) + divider: # OPTIONAL — denominator for random (default: 1000) + size: # OPTIONAL — buffer size for random strategy (default: 50000) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Take first N records | `filter: head`, `limit: N` | +| Take last N records | `filter: tail`, `limit: N` | +| Take every Nth record | `filter: nth`, `limit: N` | +| Random X% of records | `filter: random`, `percent: X`, `divider: 100` | +| Exact percentage | `filter: percent`, `percent: X` | +| Development — limit to small set | `filter: head`, `limit: 100` | +| QA sampling — 10% random | `filter: random`, `percent: 10`, `divider: 100` | +| Sparse sample 0.1% | `filter: random`, `percent: 1`, `divider: 1000` | + +## Strategy Reference + +| Filter | Keeps | Key fields | +|--------|-------|-----------| +| `random` | `percent/divider` fraction, randomly | `percent`, `divider`, `size` | +| `head` | First `limit` records | `limit` | +| `tail` | Last `limit` records (buffers all) | `limit` | +| `nth` | Records at positions 1, 1+N, 1+2N, … | `limit` | +| `percent` | Exactly `percent`% of records | `percent` | + +## Throughput Calculator + +``` +random: effective_rate = percent / divider + percent: 10, divider: 100 → 10% (1 in 10) + percent: 1, divider: 100 → 1% (1 in 100) + percent: 1, divider: 1000 → 0.1% (1 in 1000) + percent: 5, divider: 100 → 5% (1 in 20) +``` + +## Validation Rules + +- `sample` cannot be the first task (no source mode) — flag if at position 0 +- `sample` cannot be the last task (no sink mode) — flag if at end of task list +- `tail` strategy buffers all records in memory before emitting — warn for large datasets +- `nth` selects record 1, then every N records after — confirm this matches user's intent vs. random sampling + +## Examples + +### Dev: first 100 records +```yaml +- name: dev_limit + type: sample + filter: head + limit: 100 +``` + +### QA: random 10% sample +```yaml +- name: qa_sample + type: sample + filter: random + percent: 10 + divider: 100 +``` + +### Every 50th record +```yaml +- name: sparse + type: sample + filter: nth + limit: 50 +``` + +### Last 5 records +```yaml +- name: tail_check + type: sample + filter: tail + limit: 5 +``` + +### Sparse 0.1% sample +```yaml +- name: very_sparse + type: sample + filter: random + percent: 1 + divider: 1000 +``` + +### Development pipeline with head sample +```yaml +tasks: + - name: read_large + type: file + path: s3://my-bucket/huge-dataset.json + + - name: split + type: split + + - name: dev_sample + type: sample + filter: head + limit: 50 + + - name: transform + type: jq + path: '{ "id": .id, "value": .v }' + + - name: echo + type: echo + only_data: true +``` + +## Anti-patterns + +- Placing `sample` as the first or last task — it requires both upstream and downstream +- Using `tail` on a large stream — buffers everything in memory before emitting +- Confusing `nth` with "every Nth starting at N" — it starts at record 1, then 1+N, 1+2N, … +- Using `filter: random` with `percent: 10` and no `divider` — default `divider: 1000` means 10/1000 = 1% not 10% diff --git a/.claude/skills/sns/SKILL.md b/.claude/skills/sns/SKILL.md new file mode 100644 index 0000000..abaefd1 --- /dev/null +++ b/.claude/skills/sns/SKILL.md @@ -0,0 +1,144 @@ +--- +skill: sns +version: 1.0.0 +caterpillar_type: sns +description: Publish pipeline records to an AWS SNS topic. Terminal sink — does not pass records downstream. +role: sink +requires_upstream: true +requires_downstream: false +aws_required: true +--- + +## Purpose + +Receives records from upstream, publishes each as an SNS message. Record `Data` field = message body. +Does **not** emit records downstream. Use DAG if downstream tasks are needed after publication. + +## Schema + +```yaml +- name: # REQUIRED + type: sns # REQUIRED + topic_arn: # REQUIRED — full SNS topic ARN + region: # OPTIONAL — AWS region (default: us-west-2) + subject: # OPTIONAL — message subject line + attributes: # OPTIONAL — SNS message attributes for filtering + message_group_id: # OPTIONAL — FIFO topics; auto-UUID if omitted + message_deduplication_id: # OPTIONAL — FIFO deduplication ID + fail_on_error: # OPTIONAL (default: false) +``` + +### Attributes item schema +```yaml +attributes: + - name: # attribute name + type: # "String", "Number", or "Binary" + value: # attribute value +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Standard topic | provide `topic_arn`, omit FIFO fields | +| FIFO topic (ARN ends in `.fifo`) | set `message_group_id`; all messages with same group ID are ordered | +| FIFO + each message independent group | omit `message_group_id` (auto UUID per message, no ordering guarantee) | +| SNS subscription filtering needed | add `attributes` list | +| Topic ARN is environment-specific | use `{{ env "SNS_TOPIC_ARN" }}` | +| Message needs specific format | add `jq` task upstream to reshape the record | +| Post-SNS processing needed | use DAG syntax: `upstream >> [sns_task, other_task]` | +| Region is not us-west-2 | set `region` explicitly | + +## Validation Rules + +- `topic_arn` is required +- FIFO topic ARNs end in `.fifo` — verify `message_group_id` is set if ordered delivery is required +- `sns` is a terminal sink — it cannot have a downstream task in sequential mode; use DAG if needed +- Record data is sent as-is as the message body — use a `jq` task upstream to format +- `topic_arn` should use `{{ env "VAR" }}` — never hardcode account IDs + +## IAM Permissions + +``` +sns:Publish +``` +For encrypted topics: +``` +kms:GenerateDataKey +kms:Decrypt +``` + +## Examples + +### Basic notification +```yaml +- name: notify + type: sns + topic_arn: "{{ env "SNS_TOPIC_ARN" }}" + subject: Pipeline alert +``` + +### With message attributes (subscription filter) +```yaml +- name: publish_event + type: sns + topic_arn: arn:aws:sns:us-west-2:123456789012:events + attributes: + - name: EventType + type: String + value: UserCreated + - name: Priority + type: String + value: High +``` + +### FIFO topic with group ID +```yaml +- name: ordered_publish + type: sns + topic_arn: arn:aws:sns:us-west-2:123456789012:ordered.fifo + message_group_id: user-events-group +``` + +### Shape payload then publish +```yaml +- name: format_event + type: jq + path: | + { + "event": "record_processed", + "id": .id, + "ts": "{{ macro "timestamp" }}" + } + +- name: publish + type: sns + topic_arn: "{{ env "SNS_TOPIC_ARN" }}" + region: us-east-1 +``` + +### DAG: process AND publish in parallel +```yaml +tasks: + - name: source + type: file + path: data/input.json + - name: transform + type: jq + path: '{ "id": .id, "result": .value }' + - name: publish + type: sns + topic_arn: "{{ env "SNS_TOPIC_ARN" }}" + - name: archive + type: file + path: s3://bucket/archive/{{ macro "uuid" }}.json + +dag: source >> transform >> [publish, archive] +``` + +## Anti-patterns + +- Using `sns` in the middle of a sequential pipeline and expecting records to flow past it — it is a terminal sink +- Hardcoding `topic_arn` with account ID → use `{{ env "VAR" }}` +- FIFO topic without `message_group_id` when ordered delivery is required +- Sending unformatted data — add a `jq` task upstream to structure the message body diff --git a/.claude/skills/split/SKILL.md b/.claude/skills/split/SKILL.md new file mode 100644 index 0000000..7665659 --- /dev/null +++ b/.claude/skills/split/SKILL.md @@ -0,0 +1,132 @@ +--- +skill: split +version: 1.0.0 +caterpillar_type: split +description: Split one record into many by a delimiter — turns a multi-line blob into individual records. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Takes each incoming record's data string and splits it by `delimiter`, emitting one record per segment. +Most commonly used after a `file` or `http` source that reads entire file/response as one record. + +## Schema + +```yaml +- name: # REQUIRED + type: split # REQUIRED + delimiter: # OPTIONAL — character or string to split on (default: \n) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Multi-line text file → individual lines | default `delimiter: "\n"` | +| CSV row → individual fields | `delimiter: ","` | +| TSV row → individual fields | `delimiter: "\t"` | +| Pipe-delimited data | `delimiter: "\|"` | +| Custom section separator | `delimiter: "---"` | +| JSON-lines file (one JSON per line) | `split` with default, then `jq` to parse each line | +| Empty segments appear (trailing newline) | add `jq` filter after: `select(. != "")` | + +## Behavior + +``` +Input record data: "line1\nline2\nline3" +Delimiter: "\n" + +Output records: + record 1 → "line1" + record 2 → "line2" + record 3 → "line3" +``` + +## Validation Rules + +- `split` must have both upstream and downstream tasks — it is not a source or sink +- Empty string segments (e.g. from trailing delimiter) produce empty records — filter with downstream `jq select(. != "")` +- `split` operates on the raw data string — not on JSON fields; use `jq` + `explode: true` for JSON arrays instead + +## Common Delimiter Reference + +| Format | YAML value | +|--------|-----------| +| Newline (default) | `"\n"` or omit | +| Comma | `","` | +| Tab | `"\t"` | +| Pipe | `"\|"` | +| Semicolon | `";"` | +| Section separator | `"---"` | + +## Examples + +### Split file into lines (default) +```yaml +- name: read_file + type: file + path: data/records.txt + +- name: split_lines + type: split + +- name: process + type: jq + path: '{ "line": . }' +``` + +### Split CSV row into fields +```yaml +- name: split_csv + type: split + delimiter: "," +``` + +### Split JSON-lines → parse each +```yaml +- name: split_lines + type: split + +- name: parse_each + type: jq + path: . | fromjson +``` + +### Filter empty lines after split +```yaml +- name: split_lines + type: split + +- name: remove_empty + type: jq + path: . | select(. != "") +``` + +### Full pipeline: HTTP response → split → process +```yaml +tasks: + - name: fetch + type: http + method: GET + endpoint: https://api.example.com/export/csv + + - name: split_lines + type: split + + - name: parse_csv + type: converter + format: csv + skip_first: true +``` + +## Anti-patterns + +- Using `split` as the first task — it has no source mode, requires upstream +- Using `split` on JSON arrays — use `jq` with `explode: true` instead +- Not filtering empty segments from trailing delimiters +- Splitting JSON objects with commas — use `jq` not `split` for structured data diff --git a/.claude/skills/sqs/SKILL.md b/.claude/skills/sqs/SKILL.md new file mode 100644 index 0000000..785e5b5 --- /dev/null +++ b/.claude/skills/sqs/SKILL.md @@ -0,0 +1,120 @@ +--- +skill: sqs +version: 1.0.0 +caterpillar_type: sqs +description: Read messages from or write messages to an AWS SQS queue. +role: source | sink +requires_upstream: false # read mode +requires_downstream: false # write mode +aws_required: true +--- + +## Purpose + +Dual-mode SQS task. Auto-detects role: +- **Read mode** (no upstream): polls queue, emits one record per message +- **Write mode** (has upstream): receives records, sends each as SQS message + +AWS region is parsed automatically from the queue URL. + +## Schema + +```yaml +- name: # REQUIRED + type: sqs # REQUIRED + queue_url: # REQUIRED — full SQS queue URL + concurrency: # OPTIONAL — parallel processors (default: 10) + max_messages: # OPTIONAL — messages per poll batch, max 10 (default: 10) + wait_time: # OPTIONAL — long-poll seconds (default: 10) + exit_on_empty: # OPTIONAL — stop when queue drains (default: false) + message_group_id: # OPTIONAL — required for FIFO queue writes + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Reading from queue | first task in pipeline, no upstream | +| Writing to queue | add upstream task | +| Queue URL is configurable | use `{{ env "SQS_QUEUE_URL" }}` | +| Pipeline should stop when queue is empty | set `exit_on_empty: true` | +| FIFO queue | set `message_group_id`; URL ends in `.fifo` | +| Need variable message group | use `{{ macro "uuid" }}` in `message_group_id` | +| High throughput read | increase `concurrency` | +| Sensitive queue URL | use `{{ secret "/ssm/path" }}` | + +## Validation Rules + +- `queue_url` is required +- `max_messages` ≤ 10 (SQS API hard limit) +- FIFO queues (URL ends in `.fifo`) require `message_group_id` for writes +- Without `exit_on_empty: true` the pipeline polls indefinitely — confirm for production long-running consumers +- AWS region is **not** a field — it is parsed from the queue URL automatically +- `fail_on_error: true` recommended for source tasks in critical pipelines + +## IAM Permissions + +``` +# Read mode +sqs:ReceiveMessage +sqs:DeleteMessage +sqs:GetQueueAttributes + +# Write mode +sqs:SendMessage +``` + +## Examples + +### Read (drain queue, stop when empty) +```yaml +- name: read_queue + type: sqs + queue_url: '{{ env "SQS_QUEUE_URL" }}' + max_messages: 10 + wait_time: 10 + exit_on_empty: true + concurrency: 5 + fail_on_error: true +``` + +### Read (continuous consumer) +```yaml +- name: consume_events + type: sqs + queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/events + concurrency: 10 + wait_time: 20 +``` + +### Write to standard queue +```yaml +- name: enqueue_results + type: sqs + queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/output-queue +``` + +### FIFO queue read +```yaml +- name: read_fifo + type: sqs + queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/ordered.fifo + exit_on_empty: true +``` + +### FIFO queue write +```yaml +- name: write_fifo + type: sqs + queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/ordered.fifo + message_group_id: pipeline-batch-{{ macro "uuid" }} +``` + +## Anti-patterns + +- Setting `max_messages` > 10 → SQS API rejects it +- Omitting `exit_on_empty: true` in batch jobs → pipeline never terminates +- Missing `message_group_id` for FIFO write → SQS returns error +- Hardcoding queue URL → use `{{ env "SQS_QUEUE_URL" }}` +- Confusing `concurrency` (SQS-level goroutines) with `task_concurrency` (pipeline-level workers) diff --git a/.claude/skills/xpath/SKILL.md b/.claude/skills/xpath/SKILL.md new file mode 100644 index 0000000..ff16275 --- /dev/null +++ b/.claude/skills/xpath/SKILL.md @@ -0,0 +1,160 @@ +--- +skill: xpath +version: 1.0.0 +caterpillar_type: xpath +description: Extract structured data from XML or HTML using XPath expressions. +role: transform +requires_upstream: true +requires_downstream: true +aws_required: false +--- + +## Purpose + +Applies XPath expressions to XML/HTML record data. When `container` is set, iterates over matching nodes and emits one record per node. Each extracted field value is an array (even if only one match). + +Context key `node_index` is automatically set (1-based) when using `container`. + +## Schema + +```yaml +- name: # REQUIRED + type: xpath # REQUIRED + container: # OPTIONAL — XPath for repeating container elements + fields: # REQUIRED — field name → XPath expression + ignore_missing: # OPTIONAL — null for missing fields vs error (default: true) + fail_on_error: # OPTIONAL (default: false) +``` + +## Decision Rules + +| Condition | Choice | +|-----------|--------| +| Document has repeating elements (rows, articles, products) | set `container` | +| Extract page-level metadata | omit `container` | +| Missing elements should not stop pipeline | `ignore_missing: true` (default) | +| Missing elements are a hard error | `ignore_missing: false` | +| Need to track which element a record came from | use `{{ context "node_index" }}` downstream | +| Extract text content | use `/text()` in XPath | +| Extract attribute | use `/@attr` in XPath | +| Scoped to element with class | `[@class='name']` | +| Contains class (partial match) | `[contains(@class,'name')]` | + +## Output Shape + +Each field value is **always an array**: +```json +{ + "title": ["Article Title"], + "author": ["Jane Doe"], + "tags": ["tech", "news"] +} +``` + +To get the first value in downstream `jq`: `.title[0]` + +## Context Auto-populated + +When `container` is used: +``` +{{ context "node_index" }} → "1", "2", "3", ... +``` + +## Validation Rules + +- `fields` is required — must have at least one field +- Field values are always arrays — downstream `jq` must use `.[0]` to extract scalar +- Without `container`, the entire document is one record +- With `container`, each matching node → one record +- `ignore_missing: false` stops pipeline on first missing field — use only for strict validation + +## XPath Cheatsheet + +| Goal | Expression | +|------|-----------| +| Text content | `.//element/text()` | +| Attribute value | `.//element/@attr` | +| By ID | `//*[@id='foo']` | +| By class | `//*[@class='foo']` | +| Contains class | `//*[contains(@class,'foo')]` | +| nth child | `.//td[2]/text()` | +| Direct child | `./child/text()` | +| First match | `(.//element)[1]` | +| Following sibling | `following-sibling::td[1]/text()` | +| Ancestor | `ancestor::div[@class='row']` | + +## Examples + +### Extract article data +```yaml +- name: extract_articles + type: xpath + container: "//article" + fields: + title: ".//h1/text()" + author: ".//span[@class='author']/text()" + published: ".//time/@datetime" + url: ".//a[@class='permalink']/@href" + ignore_missing: true +``` + +### Extract table rows +```yaml +- name: extract_rows + type: xpath + container: "//table[@id='data-table']//tr[position()>1]" + fields: + name: ".//td[1]/text()" + email: ".//td[2]/text()" + role: ".//td[3]/text()" +``` + +### Extract page metadata (no container) +```yaml +- name: page_meta + type: xpath + fields: + title: "//title/text()" + description: "//meta[@name='description']/@content" + canonical: "//link[@rel='canonical']/@href" + og_image: "//meta[@property='og:image']/@content" +``` + +### Use node_index downstream +```yaml +- name: extract_rows + type: xpath + container: "//tr" + fields: + col1: ".//td[1]/text()" + col2: ".//td[2]/text()" + +- name: tag_with_index + type: jq + path: | + { + "row_number": "{{ context "node_index" }}", + "col1": .col1[0], + "col2": .col2[0] + } +``` + +### Product catalog +```yaml +- name: extract_products + type: xpath + container: "//div[contains(@class,'product-item')]" + fields: + name: ".//h2/text()" + price: ".//span[@class='price']/text()" + sku: ".//data[@name='sku']/@value" + img_src: ".//img/@src" + ignore_missing: true +``` + +## Anti-patterns + +- Expecting scalar field values — all fields return arrays; always access with `[0]` in downstream `jq` +- Using `ignore_missing: false` in production with inconsistent HTML — pipeline stops on first missing field +- Omitting `container` when document has repeating elements — all elements processed as one record +- XPath expressions without `.//` prefix inside container — relative paths must start with `.//` diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..271dd11 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,80 @@ +# Caterpillar + +Caterpillar is a data pipeline tool. Pipelines are defined as YAML files with a `tasks:` list. Each task runs sequentially — output of one task feeds the next. + +## Pipeline Structure + +```yaml +tasks: + - name: + type: + # task-specific fields +``` + +## Available Task Types + +| type | role | +|------|------| +| `file` | source (read) or sink (write) — local path, S3, or glob | +| `kafka` | source or sink — supports TLS + SASL/SCRAM | +| `sqs` | source or sink — AWS SQS | +| `http` | source (fetch URL) or sink (POST records) | +| `http_server` | source — listens for inbound HTTP requests | +| `aws_parameter_store` | source or sink — AWS SSM parameters | +| `sns` | sink only — publish to AWS SNS | +| `echo` | sink or pass-through — print to stdout | +| `jq` | transform — JQ expression on JSON records | +| `split` | transform — split record data into multiple records | +| `join` | transform — batch N records into one | +| `replace` | transform — regex find-and-replace | +| `flatten` | transform — flatten nested JSON with `_` separator | +| `xpath` | transform — extract from XML/HTML via XPath | +| `converter` | transform — convert CSV/Excel/HTML/EML formats | +| `compress` | transform — gzip/snappy/zlib/deflate | +| `archive` | transform — zip/tar pack or unpack | +| `sample` | filter — head/tail/nth/random/percent | +| `delay` | rate-limit — pause between records | +| `heimdall` | transform — submit jobs to Heimdall | + +## Generating Pipelines + +When a user asks to build, create, or write a pipeline — use the `pipeline-builder-interactive` agent. It asks targeted questions about source, transforms, sink, and auth before writing the file. The validation hook runs automatically after the file is written. + +Use the `pipeline-builder` skill only as a schema reference when you already have all the details and just need to generate YAML directly. + +## Pipeline Review Agents + +Use these sub-agents to validate, debug, and optimize pipelines: + +| Agent | Purpose | When to use | +|-------|---------|-------------| +| `pipeline-review` | Full review: lint + validate + permissions + optimize | Before shipping any pipeline | +| `pipeline-lint` | Structure, types, required fields, credential security | First check on a new pipeline | +| `pipeline-validate` | Context keys, JQ expressions, inter-task data flow | After lint passes | +| `pipeline-permissions` | AWS IAM policy generation, region checks | When deploying to AWS | +| `pipeline-debugger` | Error diagnosis, echo probe insertion, fix suggestions | When a pipeline fails | +| `pipeline-runner` | Build binary and run pipeline, interpret output | Smoke tests and end-to-end testing | +| `pipeline-optimizer` | Concurrency, batching, error handling, production-readiness | Before production deploy | + +Invoke via the Agent tool or ask Claude to "review my pipeline", "debug this error", "check permissions for", etc. + +## Example Pipelines + +**Before writing any pipeline**, read the matching example from `test/pipelines/examples/`: + +``` +test/pipelines/ +├── examples/ +│ ├── basic/ ← file-to-file, NDJSON, CSV, echo +│ ├── transforms/ ← jq, flatten, split/join, replace, context +│ ├── integrations/ ← kafka, sqs, http combos +│ └── production/ ← OAuth, auth chains, webhooks, SNS, compression +├── probes/ ← isolated single-task test pipelines +└── samples/ ← sample data files (JSON, NDJSON, CSV, text) +``` + +Use examples as templates. Match the user's request to the closest pattern, read that file, then adapt it. + +## Source schema first + +Whenever you have concrete **source** connection details (URL, queue, topic, bucket/path, parameters, local file), your **first** step is to **fetch at least one real record** and infer field names, types, and nesting before writing `jq`, `context:`, or transforms. Prefer `.claude/scripts/check-source-schema.sh` (subcommands: `http`, `s3`, `sqs`, `file`, `ssm`, `ssm-path`, `kafka`, `stdin`) or the `source-schema-detector` agent (`.claude/agents/source-schema-detector.md`). If live access is impossible, ask for a pasted sample and pipe it through `check-source-schema.sh stdin`. Do not guess the payload shape.