fix: feat: add a Grafana dashboard config (deploy/grafana/) for the existing Prometheus metrics — the gateway already exposes telos_exec_blocks_total, telos_network_blocks_total, and latency metrics at /metrics, but there is no pre-built dash - #408
Conversation
Adds pre-built dashboard and Prometheus configuration for existing gateway metrics. Updates docker-compose.yml with a `monitoring` profile and documents the setup in DEPLOY.md.
|
@webbrain-one is attempting to deploy a commit to the ankanmisra's projects Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughAdded Prometheus scraping for gateway and verifier metrics. Added Grafana provisioning for the Prometheus datasource and a MicroAI Paygate operations dashboard with request, verification, cache, rate-limit, latency, concurrency, and AI provider panels. ChangesObservability stack
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Gateway
participant Verifier
participant Prometheus
participant Grafana
Prometheus->>Gateway: Scrape /metrics
Prometheus->>Verifier: Scrape /metrics
Grafana->>Prometheus: Query dashboard metrics
Prometheus-->>Grafana: Return metric data
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Warning |
| - job_name: microai-gateway | ||
| static_configs: | ||
| - targets: ["gateway:3000"] | ||
| metrics_path: /metrics |
There was a problem hiding this comment.
🟡 Medium prometheus/prometheus.yml:10
The gateway scrape job hard-codes metrics_path: /metrics, but the gateway's METRICS_PATH setting can move the metrics endpoint to a different path. Any deployment that sets a custom METRICS_PATH (and docker-compose.yml already passes this through) causes Prometheus to hit a 404 and silently lose all gateway metrics. Consider aligning this path with the configured METRICS_PATH value, or document why the Prometheus config intentionally ignores that setting.
🤖 Copy this AI Prompt to have your agent fix this:
In file @deploy/prometheus/prometheus.yml around line 10:
The gateway scrape job hard-codes `metrics_path: /metrics`, but the gateway's `METRICS_PATH` setting can move the metrics endpoint to a different path. Any deployment that sets a custom `METRICS_PATH` (and `docker-compose.yml` already passes this through) causes Prometheus to hit a 404 and silently lose all gateway metrics. Consider aligning this path with the configured `METRICS_PATH` value, or document why the Prometheus config intentionally ignores that setting.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/grafana/dashboards/microai-paygate.json`:
- Around line 92-104: Add a p95 target to the “Verifier Latency (p50/p95/p99)”
timeseries panel using the existing verifier_request_duration_seconds_bucket
histogram and 5-minute rate pattern, with legendFormat set to p95; preserve the
existing p50 and p99 targets.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e9a0f998-dfa9-44ad-a63d-7c31486b029b
📒 Files selected for processing (4)
deploy/grafana/dashboards/microai-paygate.jsondeploy/grafana/provisioning/dashboards/microai.yamldeploy/grafana/provisioning/datasources/prometheus.yamldeploy/prometheus/prometheus.yml
| "title": "Verifier Latency (p50/p95/p99)", | ||
| "type": "timeseries", | ||
| "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 }, | ||
| "targets": [ | ||
| { | ||
| "expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))", | ||
| "legendFormat": "p50" | ||
| }, | ||
| { | ||
| "expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))", | ||
| "legendFormat": "p99" | ||
| } | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add the verifier p95 query.
The panel title specifies p50, p95, and p99. The targets only return p50 and p99. Add the p95 target so the dashboard meets the latency visualization objective.
Proposed fix
{
+ "expr": "histogram_quantile(0.95, rate(verifier_request_duration_seconds_bucket[5m]))",
+ "legendFormat": "p95"
+ },
+ {
"expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))",
"legendFormat": "p99"
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "title": "Verifier Latency (p50/p95/p99)", | |
| "type": "timeseries", | |
| "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 }, | |
| "targets": [ | |
| { | |
| "expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))", | |
| "legendFormat": "p50" | |
| }, | |
| { | |
| "expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))", | |
| "legendFormat": "p99" | |
| } | |
| ] | |
| "title": "Verifier Latency (p50/p95/p99)", | |
| "type": "timeseries", | |
| "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 }, | |
| "targets": [ | |
| { | |
| "expr": "histogram_quantile(0.50, rate(verifier_request_duration_seconds_bucket[5m]))", | |
| "legendFormat": "p50" | |
| }, | |
| { | |
| "expr": "histogram_quantile(0.95, rate(verifier_request_duration_seconds_bucket[5m]))", | |
| "legendFormat": "p95" | |
| }, | |
| { | |
| "expr": "histogram_quantile(0.99, rate(verifier_request_duration_seconds_bucket[5m]))", | |
| "legendFormat": "p99" | |
| } | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/grafana/dashboards/microai-paygate.json` around lines 92 - 104, Add a
p95 target to the “Verifier Latency (p50/p95/p99)” timeseries panel using the
existing verifier_request_duration_seconds_bucket histogram and 5-minute rate
pattern, with legendFormat set to p95; preserve the existing p50 and p99
targets.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Closes #303
Summary by CodeRabbit