Day 77 (DevOps)Alerting setup on Grafana Cloud

Grafana Alerting allows you to learn about problems in your systems moments after they occur. Create, manage, and take action on your alerts in a single, consolidated view, and improve your team’s ability to identify and resolve issues quickly.

Alerting

You can use Grafana Cloud to avoid installing, maintaining, and scaling your own instance of Grafana. Create a free account to get started, which includes free forever access to 10k metrics, 50GB logs, 50GB traces, 500VUh k6 testing & more.

Grafana Alerting allows you to learn about problems in your systems moments after they occur.

Monitor your incoming metrics data or log entries and set up your Alerting system to watch for specific events or circumstances and then send notifications when those things are found.

In this way, you eliminate the need for manual monitoring and provide a first line of defense against system outages or changes that could turn into major incidents.

Using Grafana Alerting, you create queries and expressions from multiple data sources — no matter where your data is stored — giving you the flexibility to combine your data and alert on your metrics and logs in new and unique ways. You can then create, manage, and take action on your alerts from a single, consolidated view, and improve your team’s ability to identify and resolve issues quickly.

Grafana Alerting is available for Grafana OSS, Grafana Enterprise, or Grafana Cloud. With Mimir and Loki alert rules you can run alert expressions closer to your data and at massive scale, all managed by the Grafana UI you are already familiar with.

Which are the key metrics for your business that you want to monitor and alert on?

Find events that are important to know about and not so trivial or frequent that recipients ignore them.
Alerts should only be created for big events that require immediate attention or intervention.
Consider quality over quantity.

Which type of Alerting do you want to use?

Choose between Grafana-managed Alerting or Grafana Mimir or Loki-managed Alerting; or both.

How do you want to organize your alerts and notifications?

Be selective about who you set to receive alerts. Consider sending them to whoever is on call or a specific Slack channel.
Automate as far as possible using the Alerting API or alerts as code (Terraform).

How can you reduce alert fatigue?

Avoid noisy, unnecessary alerts by using silences, mute timings, or pausing alert rule evaluation.
Continually tune your alert rules to review effectiveness. Remove alert rules to avoid duplication or ineffective alerts.
Think carefully about priority and severity levels.
Continually review your thresholds and evaluation rules.