Skip to content

IMPLEMENTATION_PLAN.md

Latest commit

 

History

History
320 lines (255 loc) · 8.64 KB

IMPLEMENTATION_PLAN.md

File metadata and controls

320 lines (255 loc) · 8.64 KB
Raw

Sentinel — AI Pull Request Reviewer Implementation Plan

A comprehensive plan for building a production-style AI system that automatically reviews GitHub pull requests using a multi-agent LangGraph workflow.

System Architecture Overview

flowchart TB
    subgraph External["External Systems"]
        GH["🐙 GitHub"]
    end

    subgraph Sentinel["Sentinel System"]
        subgraph API["API Service"]
            WH["Webhook Handler<br/>(FastAPI)"]
            SIG["Signature Verifier"]
            DUP["Deduplication Logic"]
        end
        
        subgraph Queue["Queue Layer"]
            REDIS["Redis<br/>Job Queue + Locks"]
        end
        
        subgraph Workers["Worker Service (Scalable)"]
            W1["Worker 1"]
            W2["Worker 2"]
            W3["Worker N..."]
        end
        
        subgraph AI["LangGraph Workflow"]
            CLASS["PR Classifier"]
            RAG["Context Retriever<br/>(Optional RAG)"]
            AGENTS["Reviewer Agents"]
            AGG["Aggregator"]
            GUARD["Guardrails"]
        end
        
        subgraph DB["Persistence Layer"]
            PG["PostgreSQL<br/>Jobs + Reviews + Audit"]
        end
    end

    GH -->|"Webhook Event"| WH
    WH --> SIG --> DUP
    DUP -->|"Create Job"| PG
    DUP -->|"Enqueue"| REDIS
    
    REDIS --> W1
    REDIS --> W2
    REDIS --> W3
    
    W1 & W2 & W3 -->|"Run Workflow"| CLASS
    CLASS --> RAG --> AGENTS --> AGG --> GUARD
    
    GUARD -->|"Post Review"| GH
    GUARD -->|"Save Results"| PG
Loading

Request Flow: Webhook to Review

sequenceDiagram
    participant GH as GitHub
    participant API as API Service
    participant PG as PostgreSQL
    participant RD as Redis Queue
    participant WK as Worker
    participant LG as LangGraph
    
    GH->>API: PR Webhook (opened/updated)
    API->>API: Verify Signature
    API->>API: Extract: repo, PR#, SHA, install_id
    
    API->>PG: Check existing job for PR
    alt Same SHA already reviewed
        API-->>GH: Skip (idempotent)
    else New SHA or first review
        API->>PG: Upsert job record
        API->>RD: Enqueue review job
        API-->>GH: 202 Accepted
    end
    
    WK->>RD: Poll for jobs
    RD-->>WK: Job payload
    WK->>PG: Mark job "running"
    
    WK->>GH: Fetch PR diff & metadata
    GH-->>WK: PR data
    
    WK->>LG: Run multi-agent review
    LG-->>WK: Structured review output
    
    WK->>GH: Post PR review + comments
    WK->>PG: Save review artifacts
    WK->>PG: Mark job "completed"
Loading

LangGraph Multi-Agent Workflow

stateDiagram-v2
    [*] --> Classify
    
    Classify --> FetchContext: PR classified
    
    FetchContext --> ParallelReview: Context ready
    
    state ParallelReview {
        [*] --> Correctness
        [*] --> Security
        [*] --> Maintainability
        [*] --> TestRisk
        
        Correctness --> [*]
        Security --> [*]
        Maintainability --> [*]
        TestRisk --> [*]
    }
    
    ParallelReview --> Aggregate: All agents complete
    
    Aggregate --> Guardrails: Deduplicated & ranked
    
    Guardrails --> [*]: Final review output
Loading

Agent Details

Agent Focus Example Findings
Correctness Logic bugs, edge cases, incorrect assumptions "Off-by-one error in loop boundary"
Security Auth, injection, secrets, unsafe defaults "SQL injection via unsanitized input"
Maintainability Clarity, duplication, naming, structure "Extract repeated logic to helper function"
Test & Risk Missing tests, rollout risks, migrations "No tests for new error handling path"

Database Schema

erDiagram
    JOBS ||--o{ REVIEWS : "produces"
    
    JOBS {
        uuid id PK
        string repo
        int pr_number
        string head_sha
        string installation_id
        enum status "queued|running|completed|failed"
        timestamp created_at
        timestamp updated_at
        timestamp started_at
        timestamp completed_at
        string idempotency_key UK "repo:pr:sha"
    }
    
    REVIEWS {
        uuid id PK
        uuid job_id FK
        jsonb summary "PR-level review"
        jsonb findings "Array of issues"
        jsonb inline_comments "Line-anchored comments"
        int token_usage
        int latency_ms
        float confidence_score
        timestamp created_at
    }
Loading

Implementation Phases

Phase 1: Foundation (Week 1)

Core infrastructure and basic webhook handling

  • Project scaffolding (monorepo structure)
  • Docker Compose for local dev (Redis, Postgres)
  • FastAPI webhook service skeleton
  • GitHub App registration & auth
  • Signature verification
  • Basic Postgres schema (jobs table)

Phase 2: Queue & Workers (Week 2)

Job processing pipeline

  • Redis queue implementation
  • Job producer (API → Redis)
  • Worker consumer loop
  • Idempotency + deduplication logic
  • Job locking (one worker per PR)
  • Retry logic with exponential backoff

Phase 3: AI Review Engine (Week 3-4)

LangGraph workflow implementation

  • PR data fetcher (GitHub REST API)
  • Diff parser (unified diff → structured)
  • LangGraph state definition
  • PR classifier agent
  • Correctness reviewer agent
  • Security reviewer agent
  • Maintainability reviewer agent
  • Test/Risk reviewer agent (optional)
  • Aggregator node
  • Guardrails layer

Phase 4: Output & Integration (Week 4)

Publishing reviews back to GitHub

  • Review formatter (structured → GitHub format)
  • PR review posting (GitHub API)
  • Inline comment anchoring
  • Review artifact persistence
  • Error handling & dead letter queue

Phase 5: Polish & Observability (Week 5)

Production readiness

  • Structured logging
  • LangSmith tracing integration
  • Metrics (Prometheus)
  • Rate limiting
  • Health checks
  • Documentation

Directory Structure

sentinel/
├── docker-compose.yml          # Local dev environment
├── README.md
├── pyproject.toml              # Project dependencies
│
├── api/                        # Webhook ingestion service
│   ├── Dockerfile
│   ├── main.py                 # FastAPI app
│   ├── routes/
│   │   └── webhooks.py
│   ├── services/
│   │   ├── signature.py        # Webhook verification
│   │   └── job_producer.py     # Queue jobs
│   └── models/
│       └── events.py           # Webhook payloads
│
├── worker/                     # Review engine service
│   ├── Dockerfile
│   ├── main.py                 # Worker loop
│   ├── consumer.py             # Redis consumer
│   ├── github/
│   │   ├── client.py           # GitHub API wrapper
│   │   └── diff_parser.py      # Parse unified diffs
│   ├── workflow/
│   │   ├── graph.py            # LangGraph definition
│   │   ├── state.py            # Workflow state
│   │   └── agents/
│   │       ├── classifier.py
│   │       ├── correctness.py
│   │       ├── security.py
│   │       ├── maintainability.py
│   │       └── aggregator.py
│   └── output/
│       └── github_publisher.py # Post reviews
│
├── shared/                     # Shared utilities
│   ├── db/
│   │   ├── models.py           # SQLAlchemy models
│   │   └── session.py          # DB connection
│   ├── queue/
│   │   └── redis_client.py
│   └── config.py               # Settings
│
└── tests/
    ├── api/
    ├── worker/
    └── fixtures/
        └── sample_diffs/

Verification Plan

Since this is a greenfield project with no existing code yet, verification will be established as we build:

Unit Tests

  • Each agent will have unit tests with mocked LLM responses
  • Queue operations will be tested with Redis testcontainers
  • Signature verification tested with known GitHub payloads

Integration Tests

  • End-to-end webhook → job → review flow with Docker Compose
  • Test against mock GitHub API responses

Manual Testing

  1. Set up a test repository with GitHub App installed
  2. Open a PR with intentional issues (security, bugs, style)
  3. Verify Sentinel posts a coherent review
  4. Test rapid PR updates (force push) to verify debouncing

Next Steps

Ready to start building? I recommend beginning with:

  1. Phase 1 — Set up the project scaffolding and Docker Compose
  2. Create the GitHub App configuration
  3. Implement basic webhook handling

Let me know when you're ready to proceed!