Skip to content

fix: feat: add a Grafana dashboard config (deploy/grafana/) for the existing Prometheus metrics — the gateway already exposes telos_exec_blocks_total, telos_network_blocks_total, and latency metrics at /metrics, but there is no pre-built dash - #408

Open
webbrain-one wants to merge 1 commit into
AnkanMisra:mainfrom
webbrain-one:webbrain/issue-303

Conversation

@webbrain-one

@webbrain-one webbrain-one commented Aug 9, 2026

Copy link
Copy Markdown

Closes #303

Summary by CodeRabbit

  • New Features
    • Added an operations dashboard for monitoring request volume, verification results, cache performance, rate limiting, latency, active requests, and AI provider response times.
    • Added automatic dashboard and Prometheus datasource provisioning.
    • Added monitoring configuration for gateway and verifier service metrics with regular collection and evaluation intervals.

Adds pre-built dashboard and Prometheus configuration for existing gateway
metrics. Updates docker-compose.yml with a `monitoring` profile and
documents the setup in DEPLOY.md.
@vercel

vercel Bot commented Aug 9, 2026

Copy link
Copy Markdown

@webbrain-one is attempting to deploy a commit to the ankanmisra's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the type:devops Deployment, observability, operations, CI, or infrastructure work. label Aug 9, 2026
@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Added Prometheus scraping for gateway and verifier metrics. Added Grafana provisioning for the Prometheus datasource and a MicroAI Paygate operations dashboard with request, verification, cache, rate-limit, latency, concurrency, and AI provider panels.

Changes

Observability stack

Layer / File(s) Summary
Prometheus metrics collection
deploy/prometheus/prometheus.yml
Prometheus uses 15-second scrape and evaluation intervals. It scrapes gateway and verifier /metrics endpoints.
Paygate operations dashboard
deploy/grafana/dashboards/microai-paygate.json
The Grafana 10 dashboard refreshes every 10 seconds and displays request, verification, cache, rate-limit, API latency, verifier latency, active request, and AI provider metrics.
Grafana provisioning
deploy/grafana/provisioning/dashboards/microai.yaml, deploy/grafana/provisioning/datasources/prometheus.yaml
Grafana loads the dashboard from the configured path and provisions a default, non-editable Prometheus datasource through the proxy.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Gateway
  participant Verifier
  participant Prometheus
  participant Grafana
  Prometheus->>Gateway: Scrape /metrics
  Prometheus->>Verifier: Scrape /metrics
  Grafana->>Prometheus: Query dashboard metrics
  Prometheus-->>Grafana: Return metric data
Loading

Possibly related PRs

Suggested labels: SWoC26, level:advanced

Suggested reviewers: ankanmisra

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description only states that issue #303 closes and omits the required summary, change type, affected areas, checklist, verification, and reviewer notes. Complete the pull request template with the change summary, selected change and affected-area types, checklist status, exact verification commands, and relevant notes.
Linked Issues check ⚠️ Warning The dashboard and Prometheus provisioning objectives are addressed, but the required Docker Compose monitoring services and DEPLOY.md documentation are missing. Add Prometheus and Grafana services under the monitoring profile in docker-compose.yml, and document startup and access steps in DEPLOY.md.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the Grafana dashboard change and its use of existing Prometheus metrics, but it is unnecessarily long and contains duplicate prefixes.
Out of Scope Changes check ✅ Passed The added Grafana and Prometheus configuration files support the linked monitoring objectives, and no unrelated changes are shown.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Warning

⚠️ This pull request shows signs of AI-generated slop (redundant_comments, description_diff_mismatch). It has been flagged by CodeRabbit slop detection and should be reviewed carefully.

- job_name: microai-gateway
static_configs:
- targets: ["gateway:3000"]
metrics_path: /metrics

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium prometheus/prometheus.yml:10

The gateway scrape job hard-codes metrics_path: /metrics, but the gateway's METRICS_PATH setting can move the metrics endpoint to a different path. Any deployment that sets a custom METRICS_PATH (and docker-compose.yml already passes this through) causes Prometheus to hit a 404 and silently lose all gateway metrics. Consider aligning this path with the configured METRICS_PATH value, or document why the Prometheus config intentionally ignores that setting.

🤖 Copy this AI Prompt to have your agent fix this:
In file @deploy/prometheus/prometheus.yml around line 10:

The gateway scrape job hard-codes `metrics_path: /metrics`, but the gateway's `METRICS_PATH` setting can move the metrics endpoint to a different path. Any deployment that sets a custom `METRICS_PATH` (and `docker-compose.yml` already passes this through) causes Prometheus to hit a 404 and silently lose all gateway metrics. Consider aligning this path with the configured `METRICS_PATH` value, or document why the Prometheus config intentionally ignores that setting.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/grafana/dashboards/microai-paygate.json`:
- Around line 92-104: Add a p95 target to the “Verifier Latency (p50/p95/p99)”
timeseries panel using the existing verifier_request_duration_seconds_bucket
histogram and 5-minute rate pattern, with legendFormat set to p95; preserve the
existing p50 and p99 targets.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e9a0f998-dfa9-44ad-a63d-7c31486b029b

📥 Commits

Reviewing files that changed from the base of the PR and between e8481a3 and c7d933e.

📒 Files selected for processing (4)
  • deploy/grafana/dashboards/microai-paygate.json
  • deploy/grafana/provisioning/dashboards/microai.yaml
  • deploy/grafana/provisioning/datasources/prometheus.yaml
  • deploy/prometheus/prometheus.yml

Comment on lines +92 to +104
"title": "Verifier Latency (p50/p95/p99)",
"type": "timeseries",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"targets": [
{
"expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p99"
}
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Add the verifier p95 query.

The panel title specifies p50, p95, and p99. The targets only return p50 and p99. Add the p95 target so the dashboard meets the latency visualization objective.

Proposed fix
         {
+          "expr": "histogram_quantile(0.95, rate(verifier_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p95"
+        },
+        {
           "expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))",
           "legendFormat": "p99"
         }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"title": "Verifier Latency (p50/p95/p99)",
"type": "timeseries",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"targets": [
{
"expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p99"
}
]
"title": "Verifier Latency (p50/p95/p99)",
"type": "timeseries",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"targets": [
{
"expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.95, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p95"
},
{
"expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p99"
}
]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/grafana/dashboards/microai-paygate.json` around lines 92 - 104, Add a
p95 target to the “Verifier Latency (p50/p95/p99)” timeseries panel using the
existing verifier_request_duration_seconds_bucket histogram and 5-minute rate
pattern, with legendFormat set to p95; preserve the existing p50 and p99
targets.

@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
microai-paygate Ready Ready Preview Aug 10, 2026 5:18am

@AnkanMisra AnkanMisra left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow cr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:devops Deployment, observability, operations, CI, or infrastructure work.

Projects

None yet

2 participants