PulseOps

Event-driven analytics backend with PostgreSQL time-series partitioning, Kafka event queue, Redis caching, and GraphQL API.

Designed to evaluate heterogeneous event ingestion, asynchronous aggregation, tenant-skew behavior, and dashboard query performance with benchmark evidence rather than hard-coded performance claims.

Article Evidence

PulseOps includes architecture and benchmark artifacts for evidence-based writeups. The repository supports honest claims about the shape of the system: Fastify ingest, Kafka decoupling, worker aggregation, PostgreSQL raw/aggregate storage, Redis-backed GraphQL caching, synthetic skew generation, and k6 benchmark scripts.

Do not publish specific throughput, latency, cache-hit, Kafka-lag, or availability numbers unless they are tied to a dated benchmark report under docs/benchmarks/ with raw k6 output and environment notes.

• Evidence guide: docs/article-evidence.md
• Observability map: docs/observability.md
• Benchmark report template: docs/benchmarks/YYYY-MM-DD-pulseops-benchmark.md
• Clean publish local benchmark: docs/benchmarks/2026-06-16-clean-publish-benchmark.md
• Full local benchmark evidence: docs/benchmarks/2026-06-16-clean-full-benchmark.md (records Dirty tree | yes; rerun after committing before citing final article numbers)
• Canonical local smoke report: docs/benchmarks/2026-06-16-final-benchmark-smoke-pulseops-benchmark.md
• Heavier ingest-scale report: docs/benchmarks/2026-06-16-ingest-scale-pulseops-benchmark.md
• Synthetic skew generator: scripts/generate-skewed-events.ts
• k6 load tests: tests/load/

Architecture Evidence

Event Ingestion — HTTP API accepts validated JSON events and publishes them to Kafka topic events-raw. Batches reduce per-event request overhead and keep PostgreSQL writes out of the request path.

(services/ingest-api/src/index.ts)

Event Processing — Worker consumes from Kafka, writes raw events to PostgreSQL, and updates daily aggregates. Late-arrival behavior should be benchmarked before making correctness or freshness claims.

(services/worker/src/index.ts, services/worker/src/processing.ts, services/worker/src/aggregators/daily.ts)

Storage — PostgreSQL 16 with time-series optimization:

• events partitioned by timestamp in the initial schema
• daily_aggregates keyed by org_id, project_id, metric, date, and dimensions
• Tenant/time indexes for raw event lookup
• Aggregate lookup indexes for dashboard queries

(scripts/init-db.sql, migrations/006_performance_optimizations.sql)

Query Layer — Apollo GraphQL API:

• Redis cache with tenant/project cache-version keys and 300-second TTL
• Queries require X-API-Key and reject cross-tenant orgId/projectId
• Aggregate-backed paths when no property filters are supplied
• Raw event fallback for supported property filters

(services/graphql-api/src/schema.ts)

Benchmark Tooling:

• scripts/generate-skewed-events.ts produces hot/medium/quiet tenant workloads with late arrivals, duplicates, bursts, multiple metrics, and 7/30/90-day windows.
• tests/load/ingest-throughput.js measures ingest acceptance behavior.
• tests/load/hot-tenant.js stresses skewed tenant distribution.
• tests/load/dashboard-query.js measures GraphQL dashboard query behavior.
• tests/load/backpressure.js captures behavior under burst pressure.

Dashboard — React 18 + Recharts, renders:

• Event count trends from GraphQL queries
• Custom metric dashboards (user-defined dimensions)
• Anomaly detection (threshold alerts)
• Timezone-aware charting

(web/src/components/Dashboard.tsx)

Infrastructure — Docker Compose local dev stack:

• PostgreSQL 16 (time-series partitioned)
• Redis 7 (query cache)
• Apache Kafka (message queue, 3 partitions)
• Ingest API (Node.js, port 3001)
• GraphQL API (Node.js, port 3002)
• Worker (Node.js, Kafka consumer)
• React frontend (Vite, port 5173, run through pnpm dev:web)

Docker Compose health checks cover PostgreSQL, Redis, Kafka, and the app services; the Node app services also use restart policies.

(docker-compose.yml)

Stack — Node.js 20, Fastify, Apollo GraphQL, React 18, PostgreSQL 16, Redis 7, Kafka, Playwright (E2E), k6 (load testing), pnpm workspaces.

How It Works

1. Event source sends JSON → POST to ingest API
2. API validates + authenticates → binds org/project from X-API-Key
3. API publishes → Kafka topic events-raw
4. Worker consumes → writes idempotent raw events and updates daily aggregates
5. GraphQL query → Checks cache (Redis) → if miss, queries DB → returns to dashboard
6. Dashboard renders → React queries GraphQL and can be refreshed by the client

Getting Started

One-Command Setup

pnpm bootstrap  # Installs deps, starts Docker services, runs migrations
pnpm dev        # Starts ingest API, GraphQL API, worker, and frontend concurrently

Visit http://localhost:5173.

Manual Setup

pnpm install
docker-compose up -d

# Create schema
pnpm db:migrate
pnpm db:verify:fresh

# Seed sample data
pnpm db:seed

# Start services
pnpm dev

Services

• Ingest API (port 3001): curl -X POST http://localhost:3001/api/v1/events -H 'X-API-Key: demo_key_change_this' -H 'Content-Type: application/json' -d '{"event_name":"signup","user_id":"123"}'; metrics at http://localhost:3001/metrics
• GraphQL API (port 3002): http://localhost:3002/graphql; metrics at http://localhost:3002/metrics
• Worker (metrics port 3003): http://localhost:3003/metrics
• Frontend (port 5173): http://localhost:5173

API Examples

Ingest Event

curl -X POST http://localhost:3001/api/v1/events \
  -H "X-API-Key: demo_key_change_this" \
  -H "Content-Type: application/json" \
  -d '{
    "event_name": "page_view",
    "user_id": "u123",
    "properties": {
      "page": "/pricing"
    },
    "timestamp": "2026-06-16T12:00:00.000Z"
  }'

Query Metrics (GraphQL)

query {
  metrics(
    orgId: "00000000-0000-0000-0000-000000000001"
    projectId: "00000000-0000-0000-0000-000000000002"
    startDate: "2026-06-01"
    endDate: "2026-06-16"
  ) {
    totalEvents
    topEvents {
      eventName
      count
    }
  }
}

Testing

Unit Tests

pnpm test:unit  # Jest

Integration Tests

pnpm test:integration  # Postgres + Redis required

E2E Tests

pnpm test:e2e  # Playwright

Load Testing

# Publishable local evidence must start from a clean tree.
git status --short

pnpm --silent benchmark:generate -- --tenants 100 --events 100000 --days 30 --hot-tenant-ratio 0.6 --late-arrival-ratio 0.05 --duplicate-ratio 0.01 --output jsonl > docs/benchmarks/evidence/events.jsonl
RUN_ID=local-smoke API_URL=http://localhost:3001 GRAPHQL_URL=http://localhost:3002/graphql API_KEY=demo_key_change_this pnpm benchmark
RUN_ID=local-smoke pnpm query-plans:capture
pnpm benchmark:report -- --run-id local-smoke --output docs/benchmarks/local-smoke-pulseops-benchmark.md --force
RUN_ID=local-smoke pnpm validate:evidence  # writes docs/benchmarks/latest-pulseops-benchmark.md
pnpm db:verify:fresh
API_URL=http://localhost:3001 API_KEY=demo_key_change_this pnpm benchmark:ingest
pnpm benchmark:seed-tenants -- --tenants 100 --hot-tenants 1 --medium-tenants 10 --manifest tmp/benchmark-tenants.json
TENANT_KEYS_FILE=tmp/benchmark-tenants.json API_URL=http://localhost:3001 pnpm benchmark:hot-tenant
RUN_ID=local-smoke TENANT_KEYS_FILE=tmp/benchmark-tenants.json pnpm benchmark:hot-db -- --require-complete
GRAPHQL_URL=http://localhost:3002/graphql API_KEY=demo_key_change_this ORG_ID=00000000-0000-0000-0000-000000000001 PROJECT_ID=00000000-0000-0000-0000-000000000002 pnpm benchmark:dashboard
RUN_ID=cache-smoke WARM_ITERATIONS=12 pnpm benchmark:cache -- --run-id cache-smoke --warm-iterations 12
pnpm benchmark:worker -- --run-id worker-catchup-smoke --events 1000 --batch-size 100 --poll-ms 500 --timeout-ms 60000
docker compose stop worker
pnpm prove:worker-retry-offsets -- --timeout-ms 120000 --poll-ms 500
docker compose start worker
API_URL=http://localhost:3001 API_KEY=demo_key_change_this pnpm benchmark:backpressure

Performance Characteristics

These are benchmark targets and measurement areas, not measured claims.

Metric	Status	Notes
Ingest throughput	Measured locally	Use docs/benchmarks/2026-06-16-clean-publish-benchmark.md for clean-tree article numbers. docs/benchmarks/2026-06-16-ingest-scale-pulseops-benchmark.md remains dirty-tree stress evidence. The 1000 RPS target was not sustained locally.
Ingest p95 latency	Measured locally	See dated benchmark reports; request acceptance latency is not aggregate visibility latency.
Dashboard query p95	Measured locally	See dated reports; cache smoke is cold-vs-warm local evidence, not a production cache-hit-ratio benchmark.
Worker catch-up	Measured locally	200-event bounded local smoke run; cite the worker catch-up evidence file for the exact run ID.
Kafka lag	Measured locally	Smoke run returned lag to 0; heavier ingest-scale snapshot captured 10,254,305 queued messages. Do not claim a lag limit or freshness guarantee.
Tenant skew impact	Smoke measured locally	Canonical local smoke reconciled 249 persisted hot-test events with Kafka lag 0: hot 201, quiet 40, medium 8. Evidence: docs/benchmarks/evidence/hot-tenant-db-2026-06-16-final-benchmark-smoke.json; full long-duration skew benchmark still needed
Hot-tenant DB pressure	Measured locally when benchmark:hot-db -- --require-complete is run	Aggregate-key pressure, request/persistence/lag reconciliation, and after-run DB snapshot; not continuous lock sampling
Backpressure behavior	k6 load-script evidence only	Correlate with Kafka lag, worker catch-up, and DB metrics before making stronger backpressure claims.

Deployment

Frontend

npm run build
# Deploy to Vercel (free tier)

Backend Services

• Ingest API + GraphQL API + Worker: Deploy to Railway, Render, or AWS Lambda
• PostgreSQL: Managed service (Neon free tier, Railway, AWS RDS)
• Redis: Upstash (free tier for dev)
• Kafka: Managed Kafka (Confluent Cloud, AWS MSK) or self-hosted

Estimated dev cost: $0 (free tiers); small prod workloads typically $50-200/month.

Architecture

Event Source
      ↓
  Ingest API (HTTP, port 3001)
      ↓
   Kafka (3 partitions)
      ↓
   Worker (aggregation)
      ↓
PostgreSQL 16 (time-series partitioned)
      ↓
GraphQL API (port 3002, Redis cache layer)
      ↓
React Dashboard (port 5173, polling refresh)

Security

• No secrets in git — .env files gitignored, secrets in environment
• API key hashing — SHA-256 fingerprint lookup plus bcrypt verification
• Multi-tenant isolation — API keys bind org/project context; GraphQL rejects cross-tenant args
• Rate limiting — Token bucket per API-key fingerprint
• SQL injection prevention — Parameterized PostgreSQL queries
• Automated scans — ESLint security plugin, npm audit in CI

License

MIT