diff --git a/docs/assets/images/alerts/advanced-config-status.png b/docs/assets/images/alerts/advanced-config-status.png new file mode 100644 index 0000000000..d4154697be Binary files /dev/null and b/docs/assets/images/alerts/advanced-config-status.png differ diff --git a/docs/assets/images/alerts/project-alert-receiver-pending-disabled.png b/docs/assets/images/alerts/project-alert-receiver-pending-disabled.png new file mode 100644 index 0000000000..3d5ccd1b22 Binary files /dev/null and b/docs/assets/images/alerts/project-alert-receiver-pending-disabled.png differ diff --git a/docs/assets/images/alerts/project-alert-receiver-pending.png b/docs/assets/images/alerts/project-alert-receiver-pending.png new file mode 100644 index 0000000000..971496766c Binary files /dev/null and b/docs/assets/images/alerts/project-alert-receiver-pending.png differ diff --git a/docs/assets/images/alerts/project-alert-receivers.png b/docs/assets/images/alerts/project-alert-receivers.png new file mode 100644 index 0000000000..18dbac45fe Binary files /dev/null and b/docs/assets/images/alerts/project-alert-receivers.png differ diff --git a/docs/assets/images/alerts/project-triggered-alerts.png b/docs/assets/images/alerts/project-triggered-alerts.png new file mode 100644 index 0000000000..8621a5dd0e Binary files /dev/null and b/docs/assets/images/alerts/project-triggered-alerts.png differ diff --git a/docs/setup_installation/admin/alert.md b/docs/setup_installation/admin/alert.md index a5e351da33..c3d9330660 100644 --- a/docs/setup_installation/admin/alert.md +++ b/docs/setup_installation/admin/alert.md @@ -100,7 +100,7 @@ A global receiver is created when a webhook is configured and can be used by any ### Step 6: Advanced configuration If you are familiar with Prometheus' [Alert manager](https://prometheus.io/docs/alerting/latest/alertmanager/) -you can also configure alerts by editing the _yaml/json_ file directly by going to the advaced page and clicking the edit button. +you can also configure alerts by editing the _yaml/json_ file directly by going to the advanced page and clicking the edit button. The advanced page shows the configuration currently loaded on the alert manager. After editing the configuration it takes some time to propagate changes to the alertmanager. @@ -113,6 +113,17 @@ It will try to load the new configuration to the alertmanager and show any error
Advanced configuration
+A status indicator next to the _Advanced configuration_ heading shows whether the configuration Hopsworks has saved matches what the alert manager currently has loaded. + +- _Synced_: the loaded configuration matches the saved configuration. +- _Pending reload_: a change has been saved but the alert manager has not loaded it yet. This is expected right after an edit because changes take some time to propagate. +- _Warning_: the change has stayed unloaded past the timeout, which usually means the alert manager rejected the configuration. The reason reported by the alert manager is shown so you can find and fix the offending section. + +
+ Advanced configuration load status +
Configuration load status next to the Advanced configuration heading
+
+ !!!warning If you make any changes to the configuration ensure that the changes are valid by reloading the configuration until the changes are loaded and visible in the advanced page. @@ -130,7 +141,7 @@ global: ... ``` -To test the alerts by creating triggers from Jobs and Feature group validations see [Alerts](../../user_guides/fs/feature_group/data_validation_best_practices.md#setup-alerts). +To manage a project's receivers and create alerts on Jobs and Feature group validations see [Alerts](../../user_guides/projects/alerts/index.md). The yaml syntax in the UI is slightly different in that it does not allow double quotes (it will ignore the values but give no error). Below is an example configuration, that can be used in the UI, with both email and slack receivers configured for system alerts. diff --git a/docs/user_guides/projects/alerts/index.md b/docs/user_guides/projects/alerts/index.md new file mode 100644 index 0000000000..91ae48fd60 --- /dev/null +++ b/docs/user_guides/projects/alerts/index.md @@ -0,0 +1,80 @@ +# Alerts + +## Introduction + +Hopsworks can notify you when something happens in your project, such as a job failing or a feature ingestion succeeding. +Alerts are delivered through Prometheus' [Alert manager](https://prometheus.io/docs/alerting/latest/alertmanager/) to receivers that you define per project. +Email, Slack and PagerDuty alerts require an administrator to first configure the corresponding channel for the cluster, so that Hopsworks knows how to reach the provider. +Webhook receivers can be created directly in a project without any cluster-level channel configuration. +See [Configure Alerts](../../../setup_installation/admin/alert.md) for the administrator setup. + +You manage a project's alerts under _Project Settings_ → _Alerts_. + +## Alert receivers + +A receiver is a destination that an alert is sent to, for example a Slack channel or a webhook URL. +The _Alert receivers_ table lists the receivers available to the project, both the receivers you created in this project and the global receivers shared across the cluster. + +
+ Alert receivers with load status +
Alert receivers with their load status
+
+ +To add a receiver click _Add receiver_, choose a channel, give it a name and fill in the channel details. + +### Receiver load status + +When you create or edit a receiver, Hopsworks writes the change to the Alert manager configuration and then the Alert manager loads it asynchronously. +The receiver is not usable until it has been loaded, so each receiver shows a _Status_ that tells you whether it is ready. + +| Status | Meaning | +| --- | --- | +| Loaded | The receiver is active in the Alert manager and can be used in an alert. | +| Pending | The receiver has been saved but the Alert manager has not loaded it yet. It usually becomes _Loaded_ within a minute. | +| Warning | The receiver has stayed unloaded past the timeout. The Alert manager most likely rejected the configuration, for example because it is malformed. | + +A newly created receiver appears immediately in the list as _Pending_ and switches to _Loaded_ once the Alert manager has picked it up. + +
+ A pending receiver +
A newly created receiver shown as Pending until the Alert manager loads it
+
+ +If a receiver stays in _Warning_, edit it to correct the configuration or delete it. +The status refreshes on its own after a create or edit, and you can trigger a refresh manually with the _Refresh_ button. + +## Creating alerts + +Alerts are triggered by events such as a job succeeding or failing, a feature ingestion completing, or feature monitoring detecting a shift. +You create project-wide alerts under _Global alerts_, and you can also attach alerts to an individual job or feature group from its own page. +For each alert you choose a trigger, a severity and the receiver that the notification is sent to. + +Only receivers that are _Loaded_ can be selected. +A receiver that is not yet loaded is shown as disabled in the receiver dropdown, because routing an alert to a receiver the Alert manager has not loaded would silently drop the notification. + +
+ A pending receiver disabled in the receiver dropdown +
A pending receiver cannot be selected until it is loaded
+
+ +Each configured global alert shows a _Route_ status that mirrors the receiver status, so you can confirm that the route is loaded in the Alert manager. + +## Triggered alerts (debugging) + +The _Triggered alerts (debugging)_ panel shows the Alert manager's live view of the alerts currently firing for the project. +Use it to understand why a triggered alert may not have been delivered. + +The panel shows the routing timing in effect (`group_wait`, `group_interval`, `repeat_interval` and `group_by`) and a table of the alerts the Alert manager currently holds. + +
+ Triggered alerts debugging panel +
Live view of alerts firing for the project
+
+ +If an alert you expected is missing or was not delivered, the panel explains the common reasons: + +- Not listed below: the alert is not firing, or its labels do not match any route. Compare against the `group_by` shown above. +- Suppressed: the alert is muted by a silence or inhibited by another alert. See the _Suppressed by_ column. +- Held back by timing: the first alert in a new group waits `group_wait`, and later alerts added to an existing group are sent on the next `group_interval`. +- Grouped or already firing: alerts that share the same `group_by` labels are bundled into a single notification, and an alert whose labels match one already firing is not re-sent until `repeat_interval`. Either way several firings can arrive as one message. +- Wrong receiver: the alert resolved to `default-receiver` or to a receiver outside this project, so its route or labels are misconfigured. Global receivers are valid for every project. diff --git a/mkdocs.yml b/mkdocs.yml index b53bc4becc..7317bf2e92 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -184,6 +184,7 @@ nav: - Overview: user_guides/projects/airflow/airflow.md - Airflow 3 upgrade: user_guides/projects/airflow/airflow3_upgrade.md - Security model: user_guides/projects/airflow/security_model.md + - Alerts: user_guides/projects/alerts/index.md - OpenSearch: - Connect: user_guides/projects/opensearch/connect.md - KNN: user_guides/projects/opensearch/knn.md