Grafana Alerts

BlameTrail can receive Grafana alert webhooks and automatically create or resolve incidents based on alert state changes. When a Grafana alert fires, BlameTrail creates an incident. When the alert clears, the incident is auto-resolved. Fingerprint-based deduplication prevents duplicate incidents from repeated alert notifications.

Prerequisites

• A running Grafana instance with alerting configured
• A BlameTrail service to receive the incidents

Configuring the integration in BlameTrail

1. Navigate to Organization > Integrations.
2. In the Grafana section, select the BlameTrail service where Grafana alert incidents should be created.
3. Copy the Webhook Secret — you will use this as the Bearer token when configuring Grafana.

Setting up Grafana

1. Create a contact point

1. In Grafana, go to Alerting > Contact points.
2. Click Add contact point.
3. Configure:

The Authorization header value must be Bearer followed by the webhook secret you copied from BlameTrail.

4. Click Test to verify connectivity.
5. Click Save.

2. Assign the contact point to a notification policy

1. Go to Alerting > Notification policies.
2. Edit an existing policy or create a new one.
3. Set the Contact point to BlameTrail.
4. Configure matching labels if you only want certain alerts to forward to BlameTrail.
5. Click Save.

How it works

Incident creation

When Grafana sends an alert with a firing status:

1. BlameTrail verifies the Bearer token in the Authorization header.
2. It extracts the alert fingerprint, name, and labels.
3. It checks if an open incident with the same fingerprint already exists (deduplication).
4. If no existing incident is found, it creates a new incident on the configured service.

Auto-resolve

When Grafana sends an alert with a resolved status:

1. BlameTrail looks up the open incident matching the alert fingerprint.
2. If found, the incident is automatically resolved.
3. If no matching incident exists (e.g., it was already manually resolved), the webhook is acknowledged without action.

Fingerprint-based deduplication

Grafana assigns a fingerprint to each unique alert (based on the alert rule and labels). BlameTrail uses this fingerprint to:

• Prevent duplicate incidents when Grafana sends repeated firing notifications for the same alert.
• Match resolved notifications to the correct open incident.

This means you do not need to worry about noisy alerts creating multiple incidents.

Incident details

Incidents created from Grafana alerts include:

Testing

1. In Grafana, go to Alerting > Contact points.
2. Find the BlameTrail contact point and click Test.
3. Check your BlameTrail service's incidents page. A test incident should appear.
4. Send a resolve test to verify auto-resolve works.

Troubleshooting