🌉 Incident Bridge

Incident Bridge is an AI-agent-orchestrated, Model Context Protocol (MCP)-driven incident response and automated communication engine. By bridging the gap between SRE observability alerts (Dynatrace), customer support queues (Zendesk), and collaboration hubs (Slack), Incident Bridge automates the complex stages of identifying, triaging, drafting, and dispatching communications during critical system outages.

Implemented as a serverless, programmatic code-first Google Cloud Agent Builder runtime powered by the Google Gemini API (@google/genai) and deployed directly to Google Cloud Run, it enables SREs and Support Leads to coordinate real-time incident resolution, proactive risk assessment, and customer reassurance from a single, unified workspace.

The entire cognitive and triage lifecycle is fully code-driven, running via an active fast-polling and adaptive decaying loop inside src/index.ts. This loop leverages @google/genai structured outputs, coordinating tools and state transitions across our SRE Triage Agent (src/agents/triage.ts) and Drafting Agent (src/agents/drafting.ts) under a unified deterministic coordinator (src/agents/coordinator.ts).

🗺️ Architectural Workflow

During a Sev-1 outage, information must flow swiftly and securely. Below is a representation of how Incident Bridge orchestrates data across platforms, driven by Gemini AI, SRE agents, and MCP servers:

sequenceDiagram
    autonumber
    participant DT as Dynatrace (Observability)
    participant IB as Incident Bridge (Backend)
    participant FB as Firestore (State Store)
    participant Slack as Slack Channel (#incidents)
    participant ZD_MCP as Zendesk MCP Server (In-Process SSE)
    participant Gemini as Gemini AI (2.5 Flash / Pro)
    participant DB as SRE Command Center Dashboard (Glass UI)

    %% Stage 1: Triage
    DT->>IB: Post Problem Webhook (/dynatrace/problem)
    activate IB
    IB->>FB: Create Incident Document (EVALUATING)
    IB->>Slack: Post Initial Alert w/ "Declare" Button
    deactivate IB

    Note over IB, ZD_MCP: Fast-Polling Triage Loop (Every 10 Seconds)
    rect rgb(30, 41, 59)
        IB->>DT: Query Grail logs via execute-dql tool (Dynatrace MCP)
        DT-->>IB: Return unique failing merchant IDs
        IB->>ZD_MCP: Query Recent Customer Support Tickets
        ZD_MCP-->>IB: Return Support Ticket Bodies & Tags (e.g. merchant:shopstream)
        IB->>Gemini: Classify Support Tickets (2.5 Flash) & Compute Business Risk (2.5 Pro)
        Gemini-->>IB: Return Probable Cause, related tickets count, ARR risk, Complainants & Silent Sufferers
        IB->>Slack: Update Slack Alert Block with Live Triage Stats & ARR exposure
    end

    %% Stage 2: Declaration & Drafting
    Slack->>IB: SRE clicks "Declare Incident"
    activate IB
    IB->>FB: Transition Incident State to PENDING_APPROVAL
    IB->>Slack: Update Slack Message (Remove Actions, Show Declaration Info)
    
    IB->>Gemini: Draft multi-channel communications (Gemini 2.5 Pro)
    Gemini-->>IB: Returns Status Page updates, Slack Briefing, Zendesk Templates, AM DMs
    IB->>FB: Persist AI Drafts in Firestore
    deactivate IB

    %% Stage 3: Approval & Dispatching
    DB->>IB: Support Lead / IC edits & approves draft comms
    activate IB
    IB->>FB: Approve & Transition State to EXECUTING
    IB->>Slack: Post Approved Comms / Update Incident Thread
    IB->>IB: Create Zendesk Master Ticket & Link Correlated Tickets & Sync AI Internal Notes
    deactivate IB
    
    %% Resolution
    DT->>IB: Post Resolution Webhook (RESOLVED)
    IB->>FB: Update State to RESOLVED
    IB->>Slack: Thread "All Clear: Incident Resolved" message

For a deep dive into the multi-agent cognitive design, see our AGENTS.md.

⚡ The 3 Stages of the Incident Life Cycle

Incident Bridge separates incident management into three distinct, structured stages, moving from alert to resolution:

🔍 Stage 1: Triaging (`EVALUATING` State)

Webhook Ingestion: When Dynatrace detects a problem (e.g., failure rate increase on payment-svc-v2), it fires a webhook to the /dynatrace/problem endpoint.
State Store Initialization: The backend instantiates a record in Cloud Firestore under the EVALUATING state and broadcasts an initial alert to Slack.
Continuous Background Triaging: A 10-second fast-polling loop initiates the SRE Triage Agent:
- Targeted DQL Lakehouse Extraction: Uses the custom Dynatrace MCP Server (execute-dql tool) to execute targeted DQL queries directly against Dynatrace's Grail data lakehouse to extract precise, high-fidelity error contexts (such as specific transaction merchant IDs and customer session traces).
- Employs Gemini 2.5 Flash for high-speed ticket classification and Gemini 2.5 Pro for the SRE Solutions Architect, along with a semantic mapping file src/config/service_map.json to translate infrastructure errors into customer-facing terms.
- Uses the Zendesk MCP Server to fetch and filter customer tickets filed within the last 36 hours.
- Proactive Risk Matrix: Correlates the telemetry failing list with open Zendesk support tickets. It computes the total portfolio Business Exposure (impacted ARR) and splits customers into 🔴 Active Complainants (failing + opened ticket) and 🟢 Silent Sufferers (failing in telemetry + has not opened a support ticket yet).
- Automatically posts these insights to the active Slack incident card.

📝 Stage 2: Declaring (`PENDING_APPROVAL` State)

SRE Confirmation: Once an SRE confirms the outage in Slack (by clicking Declare Incident), the incident shifts to PENDING_APPROVAL.
Gemini 2.5 Pro Comms Engine: The backend invokes the Drafting Agent to generate:
- Public Status Page Update: Markdown-formatted updates for customer transparency.
- Internal Slack Briefing: A formatted Slack briefing (utilizing strictly Slack-friendly mrkdwn syntax).
- Zendesk Ticket Response: Tailored email replies pre-staged for support agents.
- Proactive AM DMs: Direct warnings mapped to Account Managers (amEmail) whose high-value customer accounts are suffering silent failures, pre-packed with draft outreach emails.
- Zendesk Master Ticket Schema: Ready-to-deploy problem schemas to link tickets.

🚀 Stage 3: Dispatching (`EXECUTING` & `RESOLVED` States)

Command Center Review: Users log into the glassmorphic Admin Dashboard (/). They can review SRE logs, telemetry metrics, and toggle between AI drafts.
Execution: Approving both communication tracks advances the incident to EXECUTING, executing automated dispatches:
- Establishing a Master Problem Ticket on Zendesk.
- Automatically linking all child customer tickets to the newly created master Problem Ticket.
- Syncing the AI-generated responses directly into each child ticket as a private internal note in Zendesk for immediate agent review.
Resolution Loop: Once services return to normal, Dynatrace posts a RESOLVED webhook. Incident Bridge automatically transitions the state to RESOLVED, updates the dashboard, and threads an "All Clear" notice onto the original Slack incident alert.

🛠️ Technology Stack

Runtime & Web Framework: Node.js & Express configured with TypeScript.
Frontend SPA: Vite & React compiling a reactive Tailwind CSS v4 interface with glassmorphic layouts, modern typography (Inter), and custom scrolling consoles.
Database & State Store: Cloud Firestore utilizing the official firebase-admin SDK.
AI Core: Google Gen AI SDK (@google/genai) harnessing structured output schemas for deterministic JSON responses.
Model Context Protocol (MCP): @modelcontextprotocol/sdk utilized to integrate in-process Zendesk and a custom Dynatrace server. The custom Dynatrace MCP server executes targeted DQL queries directly against Dynatrace's Grail data lakehouse to extract precise, high-fidelity error contexts (such as specific transaction merchant IDs and customer session traces).

📁 Repository Directory Map

incident-bridge/
├── dist/                   # Compiled backend production build
├── public/                 # Statically served compiled Frontend SPA assets
├── src/                    # Backend & Frontend Source Code (TypeScript)
│   ├── index.ts            # Entrypoint: Express startup, SSE handlers, fast-poll & adaptive loop
│   ├── agents/             # SRE & Drafting Cognitive Agents
│   │   ├── coordinator.ts  # Orchestrates high-level triage & drafting processes
│   │   ├── drafting.ts     # Drafting Agent: Generates multi-channel communications
│   │   └── triage.ts       # Triage Agent: queries DQL, calls Flash (classifier) & Pro (architect)
│   ├── middlewares/        # Express Middlewares
│   │   └── auth.ts         # Authentication gate for secure endpoints
│   ├── mcp/                # Model Context Protocol Clients & Servers
│   │   ├── zendeskServer.ts# In-process Zendesk MCP SSE Server & tools definitions
│   │   ├── zendesk.ts      # Zendesk MCP Client connector (local SSE)
│   │   └── dynatrace.ts    # Dynatrace MCP Gateway client (exposes execute-dql)
│   ├── routes/             # Express API Routers
│   │   ├── index.ts        # Routes hub
│   │   ├── dynatrace.ts    # Mounted at /dynatrace - Webhook ingestion for live problems
│   │   ├── slack.ts        # Mounted at /slack - Interactive button action handlers
│   │   ├── cron.ts         # Mounted at /cron - Adaptive decaying sweeps scheduler
│   │   ├── status.ts       # Mounted at /api/incidents - Dashboard states, approval routes
│   │   └── demo.ts         # Mounted at /api/demo - Simulates events, mocks Dynatrace, DQL executor
│   ├── services/           # Backend Data Integrations
│   │   ├── firebase.ts     # Cloud Firestore queries, updates, and schema models
│   │   └── zendesk.ts      # Fetch-based Zendesk Search & Org lookups, seeding utility
│   ├── utils/              # Helper utilities
│   │   ├── slack.ts        # Slack WebClient wrapper (posts/updates block kit structures)
│   │   └── telemetry.ts    # Trace providers initializations
│   ├── config/
│   │   └── service_map.json# Infrastructure component-to-customer symptom map
│   └── frontend/           # React Single Page Application (Vite project)
│       ├── index.html      # SPA HTML structure
│       └── src/
│           ├── App.tsx     # Unified Admin Command Center & SRE tab layout
│           ├── components/ # Modular React UI components (SreCenter.tsx, SupportHub.tsx)
│           ├── index.css   # Tailwind CSS setup and custom utilities
│           └── main.tsx    # App React mounter
├── scripts/                # Verification, Seeding & Utility Scripts
│   ├── create_tickets.js   # Seeds mock customer support tickets in Zendesk
│   ├── delete_evaluating.js# Cleans up evaluating incidents
│   ├── get_incidents.js    # Retrieves list of current incidents
│   ├── simulate_webhook.js # Mimics a live Dynatrace problem payload firing to the app
│   └── clear_incidents.js  # Cleans up existing incidents from the Firestore DB
├── vite.config.ts          # Vite compile configuration (Root level)
└── tsconfig.json           # TS Compiler configurations

⚙️ Environment Setup & Configuration

Incident Bridge integrates multiple enterprise platforms to synchronize SRE and support response channels. For a streamlined, step-by-step setup guide for each integration, please refer to the dedicated documentation files:

👑 Dynatrace Setup Guide: Configuring classic endpoint ingestion, platform API tokens, and MCP server permissions.
💬 Slack App Setup Guide: Creating custom Slack apps, configuring Bot User scopes, and inviting your bot to channels.
📞 Zendesk Setup Guide: Authenticating Admin API tokens, adding organization domains, and automated ticket seeding.

Core Environment Template (`.env`)

Create a .env file in the root directory. You can use the provided .env.example as a baseline template:

# --- AUTH GATE CONFIGURATION ---
ALLOWED_INDIVIDUAL_EMAILS=developer@yourdomain.com
DEMO_BYPASS_PASSCODE=DEMO-BYPASS_CODE
GOOGLE_CLIENT_ID=your-google-oauth-client-id.apps.googleusercontent.com

# --- DYNATRACE CONFIGURATION ---
DYNATRACE_ENDPOINT=https://your-env.live.dynatrace.com # Your Dynatrace Tenant URL
DYNATRACE_API_TOKEN=dt0c01.XXXX.XXXX                  # API token with scopes: entities.write, events.ingest, logs.ingest
DYNATRACE_MCP_URL=http://localhost:XXXX/rpc            # Or the URL of your Dynatrace MCP Gateway
DYNATRACE_MCP_TOKEN=your-dynatrace-mcp-bearer-token    # Bearer token for authenticating with Dynatrace MCP

# --- SLACK CONFIGURATION ---
SLACK_BOT_TOKEN=xoxb-XXXX-XXXX-XXXX                    # Slack Bot API Token
SLACK_ALERTS_CHANNEL=#incidents                        # Channel where SRE alerts are posted

# --- ZENDESK CONFIGURATION ---
ZENDESK_SUBDOMAIN=your-subdomain                       # Zendesk subdomain (e.g. 'company-help')
ZENDESK_EMAIL=sre-admin@company.com                    # Login email for Zendesk account
ZENDESK_API_TOKEN=XXXX                                 # API Token generated in Zendesk Admin Center

# --- FIREBASE & GEMINI CONFIGURATION ---
GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/sa.json # Path to Service Account JSON key
GCLOUD_PROJECT_ID=project-xxxxx                        # GCP Project Id
GEMINI_API_KEY=AIzaSyXXXX                              # Google Gemini API Key
PORT=8080                                              # Running Port (default: 8080)

🎪 Hackathon Demo Walkthrough

Incident Bridge comes equipped with a dedicated Demo Control Panel at http://localhost:8080/ to fully showcase SRE & Support integrations.

1. Seed and Run

Launch the Express server using npm run start (this triggers the fast-polling loop).
Open http://localhost:8080/ in your browser.

2. Trigger the Simulated Failure

On the Demo Control Panel, select a scenario:
- Outage A: Payment Gateway Timeout: Simulates database pool exhaustion on payment-svc-v2. Ingests structured logs into Grail carrying merchant.id attributes.
- Outage B: API Gateway OAuth Failure: Simulates validation signature failures on gateway-svc.
Click Trigger Incident Simulation.
Behind the scenes: The app seeds matching customer tickets, auto-provisions a custom device, injects custom alert events into your Dynatrace Problem feed, and ingests structured transaction logs directly into Dynatrace Grail.

3. SRE Command Center & Retro Terminal

Under the SRE Command Center tab in your active incident dashboard:
- Retro Diagnostic Console: Watch the typewriter-animated terminal process raw log files, invoke MCP tools, and classify tickets.
- Business Exposure Widget: Instantly review total portfolio risk (e.g. $425,000 ARR at risk) along with a detailed list of Active Complainants vs Silent Sufferers.

4. 5-Minute Auto-Declaration Failsafe

SREs have 5 minutes from incident alert creation to declare.
If unattended, the background poll scheduler automatically declares the incident, generating the communications drafts and updating state to PENDING_APPROVAL, ensuring judges can still view the drafts even if they step away!

5. Adaptive Decaying Ticket Sweeps

Support tickets arrive progressively rather than all at once. Constant polling degrades API resources.
The Solution: Scheduled automatic, silent sweeps run at the 2-minute, 5-minute, and 10-minute marks.
If a sweep finds a late ticket matching the failing telemetry tags, the engine automatically recalculates ARR exposure, appends the event to the retro terminal logs, and updates the Slack alert block.

6. Reset Simulation

Click the red "Reset Simulation" button in the header.
This wipes the active Firestore entries and uses a bulk-deletion API to purge all Zendesk tickets tagged with incident-test, leaving your testing sandbox spotless.

☁️ Google Cloud Run Deployment

Build the container image using Cloud Build and deploy the service, attaching all Secret Manager variables to the container environment:

gcloud run deploy incident-bridge \
  --source . \
  --region us-central1 \
  --min-instances=0 \
  --max-instances=2 \
  --update-secrets=\
SLACK_BOT_TOKEN=SLACK_BOT_TOKEN:latest,\
SLACK_ALERTS_CHANNEL=SLACK_ALERTS_CHANNEL:latest,\
ZENDESK_API_TOKEN=ZENDESK_API_TOKEN:latest,\
ZENDESK_SUBDOMAIN=ZENDESK_SUBDOMAIN:latest,\
ZENDESK_EMAIL=ZENDESK_EMAIL:latest,\
GEMINI_API_KEY=GEMINI_API_KEY:latest,\
DYNATRACE_ENDPOINT=DYNATRACE_ENDPOINT:latest,\
DYNATRACE_API_TOKEN=DYNATRACE_API_TOKEN:latest,\
DYNATRACE_MCP_URL=DYNATRACE_MCP_URL:latest,\
DYNATRACE_MCP_TOKEN=DYNATRACE_MCP_TOKEN:latest,\
DYNATRACE_WEBHOOK_SECRET=DYNATRACE_WEBHOOK_SECRET:latest,\
ALLOWED_INDIVIDUAL_EMAILS=ALLOWED_INDIVIDUAL_EMAILS:latest,\
DEMO_BYPASS_PASSCODE=DEMO_BYPASS_PASSCODE:latest,\
GOOGLE_CLIENT_ID=GOOGLE_CLIENT_ID:latest

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
data		data
docs		docs
public		public
scripts		scripts
src		src
tests		tests
.clinerules		.clinerules
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
Dockerfile		Dockerfile
LICENSE.txt		LICENSE.txt
PROJECT_STORY.md		PROJECT_STORY.md
README.md		README.md
cleanup-policy.json		cleanup-policy.json
lifecycle.json		lifecycle.json
package-lock.json		package-lock.json
package.json		package.json
tsconfig.frontend.json		tsconfig.frontend.json
tsconfig.json		tsconfig.json
vite.config.ts		vite.config.ts

Navigation Menu

Repository files navigation

🌉 Incident Bridge

🗺️ Architectural Workflow

⚡ The 3 Stages of the Incident Life Cycle

🔍 Stage 1: Triaging (`EVALUATING` State)

📝 Stage 2: Declaring (`PENDING_APPROVAL` State)

🚀 Stage 3: Dispatching (`EXECUTING` & `RESOLVED` States)

🛠️ Technology Stack

📁 Repository Directory Map

⚙️ Environment Setup & Configuration

Core Environment Template (`.env`)

🎪 Hackathon Demo Walkthrough

1. Seed and Run

2. Trigger the Simulated Failure

3. SRE Command Center & Retro Terminal

4. 5-Minute Auto-Declaration Failsafe

5. Adaptive Decaying Ticket Sweeps

6. Reset Simulation

☁️ Google Cloud Run Deployment

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

🌉 Incident Bridge

🗺️ Architectural Workflow

⚡ The 3 Stages of the Incident Life Cycle

🔍 Stage 1: Triaging (EVALUATING State)

📝 Stage 2: Declaring (PENDING_APPROVAL State)

🚀 Stage 3: Dispatching (EXECUTING & RESOLVED States)

🛠️ Technology Stack

📁 Repository Directory Map

⚙️ Environment Setup & Configuration

Core Environment Template (.env)

🎪 Hackathon Demo Walkthrough

1. Seed and Run

2. Trigger the Simulated Failure

3. SRE Command Center & Retro Terminal

4. 5-Minute Auto-Declaration Failsafe

5. Adaptive Decaying Ticket Sweeps

6. Reset Simulation

☁️ Google Cloud Run Deployment

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

🔍 Stage 1: Triaging (`EVALUATING` State)

📝 Stage 2: Declaring (`PENDING_APPROVAL` State)

🚀 Stage 3: Dispatching (`EXECUTING` & `RESOLVED` States)

Core Environment Template (`.env`)