Who decides that something is an alert: Prometheus or the Alertmanager? What does the Alertmanager do?
Prometheus decides. Alert rules are defined and evaluated in Prometheus, which sends firing alerts to the Alertmanager. The Alertmanager only handles delivery: grouping, silencing, routing and paging.
* Prometheus decides, the Alertmanager delivers. *
The split of duties:
| Prometheus | Alertmanager |
|---|---|
Holds the alert rules (e.g. in first_rules.yml) |
Receives alerts from one or many Prometheus servers |
| Evaluates them against the metrics | Groups related alerts into one notification |
Sends firing alerts to the Alertmanagers listed in prometheus.yml |
Routes by alert type and escalates; allows silencing |
| Hands the actual paging to e-mail or services like PagerDuty or OpsGenie |
The Alertmanager is fairly minimal. It keeps no long alert history and cannot do complex escalations such as moving up a management chain when an alert stays unhandled. That is why it is usually connected to an external paging service; using a cloud service like OpsGenie brings its own data-protection questions. For high availability, several Alertmanagers can form a cluster (mesh), and Prometheus can notify all of them.
Go deeper:
Prometheus — Alerting overview — the split between rules in Prometheus and delivery in the Alertmanager.- Prometheus — Alertmanager — grouping, inhibition, silences and high availability.