feat(infra): add alerting pipeline (AlertManager + Grafana Alerting → Discord)#3501
feat(infra): add alerting pipeline (AlertManager + Grafana Alerting → Discord)#3501manamana32321 wants to merge 2 commits intomainfrom
Conversation
- AlertManager: add alertmanagerConfigSecret reference + HighNodeMemoryUsage PrometheusRule - Grafana: add Discord contact point, notification policy, and RDS FreeStorageSpace alert rule - Apply same structure to both production and stage environments - SealedSecrets to be added separately (requires correct cluster context) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…dSecrets
- Create overlays/{production,stage} for environment-specific SealedSecrets
- Add Kustomize source to prometheus ApplicationSet
- SealedSecret files are placeholder (need re-seal with correct cluster context)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef1f0ec382
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - admin-user.yaml | ||
| - aws-credentials.yaml | ||
| - github-credentials.yaml | ||
| - discord-webhook.yaml |
There was a problem hiding this comment.
This overlay now references discord-webhook.yaml, but that file is not present in this commit for either grafana/overlays/production or grafana/overlays/stage, so kustomize build for the overlay fails at render time. Since infra/k8s/argocd/applications/monitoring/grafana.yaml uses these overlays as an ArgoCD source, Grafana sync will fail before any of the alerting changes are applied.
Useful? React with 👍 / 👎.
|
|
||
| alertmanager: | ||
| alertmanagerSpec: | ||
| alertmanagerConfigSecret: alertmanager-config |
There was a problem hiding this comment.
The new key alertmanager.alertmanagerSpec.alertmanagerConfigSecret is not a supported Alertmanager spec field for kube-prometheus-stack; the chart/operator expects configSecret (typically paired with useExistingSecret). As written, this setting will be ignored or rejected, so the alertmanager-config sealed secret will not become the active Alertmanager config and Discord routing will not be enabled.
Useful? React with 👍 / 👎.
1 participant
Summary
alertmanagerConfigSecret(SealedSecret) +HighNodeMemoryUsagePrometheusRuleRDS FreeStorageSpace Lowalert rule (production only)TODO (머지 전 필요)
prometheus/overlays/production/alertmanager-config.yamlprometheus/overlays/stage/alertmanager-config.yamlgrafana/overlays/production/discord-webhook.yamlgrafana/overlays/stage/discord-webhook.yaml알림 경로
Test plan
🤖 Generated with Claude Code