Slack
Send incident notifications to Slack channels with service name, monitor details, incident type, duration, and suspect deploy information.
BlameTrail sends real-time Slack notifications when incidents open and resolve, and when fix proposals are ready or their PRs are created or merged. Each notification includes context relevant to the event, giving your team immediate visibility without leaving Slack.
Setting up Slack notifications
1. Create an incoming webhook in Slack
- Go to Slack API: Incoming Webhooks or your Slack workspace's app management page.
- Create a new app (or use an existing one) and enable Incoming Webhooks.
- Click Add New Webhook to Workspace.
- Select the channel where you want incident notifications (e.g.,
#incidents,#on-call,#production-alerts). - Copy the webhook URL. It looks like:
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
2. Add the webhook to BlameTrail
- In BlameTrail, navigate to Organization > Integrations.
- In the Slack section, click Add Webhook.
- Enter:
- Name — A label for this webhook (e.g.,
#incidents,Production Alerts). - Webhook URL — The Slack incoming webhook URL you copied.
- Name — A label for this webhook (e.g.,
- Click Save.
3. Test the notification
Click Send Test next to the webhook to verify it is working. A test message should appear in your Slack channel.
Multiple channels
You can add multiple Slack webhooks to send notifications to different channels. All configured webhooks receive the same notifications. This is useful for:
- Sending to a primary
#incidentschannel and a team-specific channel - Routing to both engineering and management channels
- Maintaining a dedicated channel for critical alerts alongside a general channel
To add another webhook, repeat the setup steps with a different Slack channel.
What notifications include
Incident opened
When an incident is created, the Slack notification includes:
| Field | Description |
|---|---|
| Service name | The affected service |
| Monitor | The monitor that detected the problem (if applicable) |
| Incident type | Availability or latency |
| Severity | Critical, error, warning, or info |
| Started at | When the incident was first detected |
| Suspect deploy | The most likely suspect deploy (commit SHA, message, deployer) |
Incident resolved
When an incident resolves, the notification includes:
| Field | Description |
|---|---|
| Service name | The affected service |
| Monitor | The monitor that recovered |
| Incident type | Availability or latency |
| Duration | How long the incident lasted |
| Resolved at | When the incident was resolved |
Fix proposal ready
When an AI fix proposal is generated and ready for review:
| Field | Description |
|---|---|
| Fix type | Revert or AI Fix |
| Confidence | Confidence score percentage (AI fixes only) |
| Incident | Link to the incident in BlameTrail |
| Summary | Brief description of the proposed fix |
Fix PR created
When a fix proposal PR is opened on GitHub:
| Field | Description |
|---|---|
| Fix type | Revert or AI Fix |
| PR link | Link to the pull request on GitHub |
| Incident | Link to the incident in BlameTrail |
Fix PR merged
When a fix PR is merged on GitHub:
| Field | Description |
|---|---|
| PR link | Link to the merged pull request |
| Incident | Link to the incident in BlameTrail |
Managing webhooks
Editing a webhook
- Navigate to Organization > Integrations.
- Find the webhook in the Slack section.
- Click Edit to update the name or URL.
- Click Save.
Removing a webhook
- Navigate to Organization > Integrations.
- Find the webhook in the Slack section.
- Click Delete and confirm.
The webhook is removed immediately. No further notifications are sent to that channel.
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| No notifications received | Webhook URL incorrect or app removed from Slack | Verify the URL is correct and the Slack app is still installed |
| Test works but incident notifications do not | No active monitors or no incidents triggered | Confirm that monitors are running and check the incidents page |
| Notifications go to the wrong channel | Webhook URL points to a different channel | Create a new webhook for the correct channel in Slack and update BlameTrail |
| Slack shows "channel_not_found" | The channel was deleted or the app was removed | Recreate the incoming webhook in Slack for the correct channel |