Many companies use Microsoft Teams. Receiving alerts in Teams allows you to react faster. It makes working on an incident visible, as you can answer the message. Let's add it to our monitoring stack!
Head over to GitHub to get the full code.
Teams configuration
Alertmanager does not support Teams out of the box. You must use webhooks to achieve this. The webhook is used by prom2teams to write messages.
Add a channel to the team which should receive the notifications. Click on the three dots beside the team name and choose Add channel
.
To add the webhook, click on the dots beside the channel and click on Connectors
.
Configure an Incoming Webhook
. Teams shows the webhook URL once you click Create
. Copy this URL in the next step.
Add prom2teams to the stack
prom2teams will use the webhook to send messages to Teams.
Add prom2teams to your docker-compose.yml
:
prom2teams:
image: idealista/prom2teams:2.7.0
restart: unless-stopped
environment:
PROM2TEAMS_CONNECTOR: "url from the webhook"
ports:
- 8089:8089
Update your alerts
The Alertmanager uses labels to decide which alert goes to which notification channel. Change the prometheus/alerts.yml
to contain the following:
groups:
- name: DemoAlerts
rules:
- alert: InstanceDown
expr: up{job="services"} < 1
for: 1m
labels: # labels and annotations are new
severity: low
annotations:
summary: 'Alert with low severity.'
- alert: InstanceDownCritical
expr: up{job="services"} < 1
for: 1m
labels:
severity: high
annotations:
summary: 'Alert with high severity.'
Now you have two alerts with different labels.
Configure the Alertmanager
Change the alertmanager/alertmanager.yml
so that the Alertmanager is aware of prom2teams.
route:
group_by: [ alertname ]
receiver: 'mail' # default receiver
repeat_interval: 24h
routes:
- receiver: 'teams'
repeat_interval: 12h
matchers:
- severity="medium"
- receiver: 'teams'
repeat_interval: 4h
matchers:
- severity="high"
receivers:
- name: 'mail'
email_configs:
- smarthost: 'yourmailhost.com:465'
auth_username: 'yourmail@yourmailhost.com'
auth_password: "your mail password"
from: 'yourmail@yourmailhost.com'
to: 'someonesmail@yourmailhost.com'
require_tls: false
- name: 'teams'
webhook_configs:
- url: "http://prom2teams:8089"
send_resolved: true
Now, the Alertmanager can publish to two channels.
matchers
tell the Alertmanager which channel to use.
The Alertmanager sends alerts with medium
severity to Teams. This is repeated every four hours - until you fix it!
The Alertmanager uses email if no route matcher matches.
You can use the routing tree editor to check if the configuration is what you need. Add labels and values of your alert to see which route they will take.
Check the result
Start the stack with docker-compose up
and wait until the alerts fire.
You will get a notification from Teams for the InstanceDownCritical
alert:
And an email for the InstanceDown
alert:
Which notification channels do you want to see next? Leave me a message!
Head over to GitHub to get the full code.
If this article was helpful for your, please consider to buy me a coffee :-)
Top comments (0)