feat: webhook notifications for media processing events

[RndmCodeGuy20]

Summary

Adds a complete webhook notification system that notifies users when their media processing jobs transition through lifecycle states — and makes the whole pipeline demo-ready by letting a host-run client drive presign → upload → complete → worker → variants → webhooks exactly like a real client, for both image and video.

Changes

Webhooks

Events: job.starting, job.started, job.done, job.failed

Architecture:

• Worker (Python) inserts webhook_deliveries rows atomically with job status transitions
• Go API polls pending deliveries and dispatches HTTP POST with HMAC-SHA256 signature
• Exponential backoff with jitter (5 attempts, 5min ceiling) to avoid thundering herd
• Per-tenant scoping via assets.owner_id → webhook_registrations.user_id

New endpoints:

• POST /api/v1/webhooks — register a webhook
• GET /api/v1/webhooks — list user's webhooks
• DELETE /api/v1/webhooks/{id} — delete a webhook

Migrations:

• 000003: event outbox (transactional enqueue)
• 000004: add owner_id to assets
• 000005: add user_id + events to webhook_registrations

Observability: OTel metrics for delivery count, duration, failures, and pending gauge.

Split storage endpoint (internal vs public)

Presigned upload URLs were signed against the compose-internal host minio:9000, which a host client cannot resolve — and SigV4 signs the Host header, so the URL can't be rewritten afterwards. Storage now supports two endpoints:

• S3_ENDPOINT_URL — internal/server-side endpoint for object I/O (http://minio:9000)

• S3_PUBLIC_ENDPOINT_URL — client-facing endpoint baked into presigned + persisted variant URLs (http://localhost:9000)

• Go storagex: separate presign client bound to the public endpoint; PublicURL prefers it. Back-compat — falls back to the internal endpoint when public is unset.

• Worker S3Storage: public_url() uses the public endpoint; object I/O stays internal.

• docker-compose.yml: both endpoints wired for api and worker.

Health check fix

cmd/server --health-check was unimplemented — it booted a second server that failed to bind port 5010, leaving the api container permanently unhealthy and blocking the worker (which depends_on api: service_healthy). It is now a lightweight /healthz probe.

Demo + docs

• scripts/demo-e2e.sh — host-run end-to-end driver for image and video, including the job.starting → job.started → job.done webhook lifecycle, with a PASS/FAIL summary and non-zero exit on failure. Adds tests/test_assets/sample.mp4.
• README / CLAUDE.md — corrected port (5010) and endpoints (/api/v1/storage/presign, GET /api/v1/assets/{id}/complete), documented auth + the split endpoint, added webhooks and run-the-demo sections, refreshed roadmap.

Testing

• Go unit tests: service validation, backoff math, HMAC, storagex split-endpoint presign/PublicURL (+ back-compat)
• Go integration tests: full dispatcher flow with httptest / testcontainers
• Python unit tests: webhook delivery insertion, S3 public-endpoint behavior (fixed a stale image-pipeline test)
• End-to-end: scripts/demo-e2e.sh → 23/23 checks pass (image 3 webp variants; video poster + 720p + preview; all variants fetchable from host over localhost:9000; all webhooks delivered)

How to test

docker compose -f docker-compose.yml -f docker-compose.webhooks.yml up -d --build

# full pipeline (image + video + webhooks), run from the host:
./scripts/demo-e2e.sh

# or the webhook-only dev script:
./scripts/test-webhooks.sh
docker logs mpiper-webhook-receiver -f