feat(infra): add alerting pipeline (alertmanager + grafana alerting → discord)#3502
feat(infra): add alerting pipeline (alertmanager + grafana alerting → discord)#3502manamana32321 wants to merge 9 commits intomainfrom
Conversation
- AlertManager: add alertmanagerConfigSecret reference + HighNodeMemoryUsage PrometheusRule - Grafana: add Discord contact point, notification policy, and RDS FreeStorageSpace alert rule - Apply same structure to both production and stage environments - SealedSecrets to be added separately (requires correct cluster context) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…dSecrets
- Create overlays/{production,stage} for environment-specific SealedSecrets
- Add Kustomize source to prometheus ApplicationSet
- SealedSecret files are placeholder (need re-seal with correct cluster context)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
manamana32321
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef1f0ec382
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AlertManager(prometheus)와 Grafana Alerting 양쪽에 Discord webhook URL을 SealedSecret으로 추가한다. prod/stage 각각 별도 클러스터 공개키로 암호화. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ed00a8936
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| alertmanager: | ||
| alertmanagerSpec: | ||
| alertmanagerConfigSecret: alertmanager-config |
There was a problem hiding this comment.
configSecret instead of alertmanagerConfigSecret
The value key under alertmanager.alertmanagerSpec is misspelled: kube-prometheus-stack reads configSecret, so alertmanagerConfigSecret is ignored and the Alertmanager CR keeps using the chart-generated default config instead of your sealed alertmanager-config secret. That means the Discord receiver/routing in the custom config will not be applied (the same typo is also present in infra/k8s/monitoring/prometheus/values-stage.yaml).
Useful? React with 👍 / 👎.
infra/k8s/monitoring/grafana/overlays/production/discord-webhook.yaml
Outdated
Show resolved
Hide resolved
alertmanagerConfigSecret(config 전체 암호화) 대신 alertmanager.config를 values에 직접 작성하고, webhook URL만 Secret 볼륨 마운트(url_file)로 주입한다. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
envFromSecrets는 Secret key를 환경변수명으로 사용하므로 webhook-url(dash 포함) 대신 DISCORD_WEBHOOK_URL로 변경한다. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- FreeableMemory < 300MB 5분 지속 시 경고 - DatabaseConnections > 150 5분 지속 시 경고 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- RDS alert rules를 values-aws-alerts.yaml로 분리 - stage는 RDS 미사용이므로 빈 파일로 처리 - FreeableMemory 룰 제거 - ApplicationSet valueFiles에 values-aws-alerts.yaml 추가 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Description
AlertManager와 Grafana Alerting 두 경로의 알림 파이프라인 뼈대를 구축합니다.
alertmanagerConfigSecret(SealedSecret)으로 Discord webhook receiver 설정 +HighNodeMemoryUsagePrometheusRule 추가RDS FreeStorageSpace Lowalert rule (production only)Additional context
Discord webhook은 SealedSecret으로 관리. prod/stage 각각 올바른 클러스터 컨텍스트에서 kubeseal로 생성 완료.
Before submitting the PR, please make sure you do the following
fixes #123).Closes TAS-2644