Grafana Alerting integration

IncidentRelay can receive Grafana Alerting notifications through a dedicated webhook endpoint. Each Grafana alert instance is normalized, routed, grouped, deduplicated, and processed through the standard IncidentRelay alert lifecycle.

Endpoint

POST /api/integrations/grafana

The endpoint requires the intake token of an active IncidentRelay route whose source is grafana.

Authorization: Bearer <route-intake-token>
Content-Type: application/json

Create an IncidentRelay route

Open Routes in IncidentRelay.
Create a route or edit an existing one.
Select Grafana as the source.
Select the team that owns the alerts.
Configure matchers, grouping, and the assignment target.
Make sure the route is active.
Copy the generated intake URL and route token.

A typical grouping configuration is:

[
  "alertname",
  "grafana_folder",
  "instance"
]

Choose only labels that identify the logical incident. Avoid grouping by labels whose values change frequently.

Example route matchers:

{
  "environment": "production",
  "team": "sre"
}

Grafana labels are available to route matchers after common labels and per-alert labels are merged.

Configure Grafana

In Grafana:

Open Alerts & IRM → Alerting → Notification configuration.
Open the Contact points tab.
Add a contact point.
Select Webhook as the integration.

Set the URL to the IncidentRelay Grafana endpoint:

https://incidentrelay.example.com/api/integrations/grafana

Set the HTTP method to POST.
In the authorization settings, set:
- Authentication Header Scheme: Bearer
- Authentication Header Credentials: the IncidentRelay route token
Keep Disable resolved message disabled so IncidentRelay receives recovery notifications.
Save the contact point.
Use Grafana's test action to verify delivery.

Attach the contact point to the required Grafana notification policy.

Recommended Grafana labels

Add stable labels to Grafana alert rules so IncidentRelay can route and group them predictably.

labels:
  team: sre
  environment: production
  severity: critical

Useful labels include:

Label	Purpose
`team`	Team routing
`environment`	Production, staging, development
`severity`	Alert priority
`service`	Affected service
`instance`	Affected host or instance
`grafana_folder`	Grafana folder
`alertname`	Alert rule name

The normalizer also recognizes oncall_team as a fallback team label and priority or level as fallback severity labels.

Alert normalization

IncidentRelay processes every object in Grafana's alerts array independently.

Status

The alert instance status takes precedence over the top-level notification status.

Resolved-like values are normalized to:

resolved

Other values are treated as:

firing

This allows one Grafana notification group to contain alert instances with different states.

Title

IncidentRelay selects the first available value from:

annotations.summary
labels.alertname
top-level title
Grafana alert

Message

IncidentRelay selects the first available value from:

annotations.description
annotations.message
top-level message
valueString

Severity

IncidentRelay selects the first available label from:

severity
priority
level

Team

IncidentRelay selects the first available value from:

labels.team
labels.oncall_team
top-level team

Route matching remains authoritative. The label does not bypass normal route access or matcher checks.

Labels added by IncidentRelay

When present in the Grafana payload, the integration exposes the following values as labels:

IncidentRelay label	Grafana field
`dashboard_url`	`dashboardURL`
`panel_url`	`panelURL`
`generator_url`	`generatorURL`
`silence_url`	`silenceURL`
`grafana_url`	`externalURL`
`grafana_org_id`	`orgId`
`grafana_receiver`	`receiver`
`grafana_group_key`	`groupKey`
`grafana_state`	`state`

The first available Grafana link is also stored as event_link. The selection order is:

dashboard URL
panel URL
alert rule URL
silence URL
Grafana base URL

Deduplication

When Grafana provides fingerprint, IncidentRelay uses it as the deduplication key.

When fingerprint is missing, IncidentRelay generates a stable key from:

Grafana source;
alert rule UID when available;
alert title;
stable labels;
Grafana organization ID.

A firing notification followed by a resolved notification with the same deduplication key updates the existing alert instead of creating another alert.

Stored payload

IncidentRelay stores the original Grafana group context, but each normalized IncidentRelay alert keeps only its own Grafana alert instance in the stored alerts array.

This preserves fields such as:

orgId;
receiver;
groupKey;
common labels and annotations;
dashboard and panel links;
expression values;
timestamps.

Example payload

{
  "receiver": "incidentrelay",
  "status": "firing",
  "orgId": 1,
  "groupKey": "{}:{alertname=\"DiskFull\"}",
  "commonLabels": {
    "team": "sre",
    "environment": "production"
  },
  "commonAnnotations": {
    "runbook_url": "https://example.com/runbooks/disk"
  },
  "externalURL": "https://grafana.example.com/",
  "title": "[FIRING:1] DiskFull",
  "state": "alerting",
  "message": "Grafana notification",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "DiskFull",
        "severity": "critical",
        "instance": "host1",
        "grafana_folder": "Infrastructure",
        "__alert_rule_uid__": "disk-full-rule"
      },
      "annotations": {
        "summary": "Disk is full",
        "description": "/var is 95% full"
      },
      "startsAt": "2026-06-21T10:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://grafana.example.com/alerting/grafana/disk-full-rule/view",
      "fingerprint": "grafana-disk-full-host1",
      "silenceURL": "https://grafana.example.com/alerting/silence/new",
      "dashboardURL": "https://grafana.example.com/d/system-overview",
      "panelURL": "https://grafana.example.com/d/system-overview?viewPanel=12",
      "values": {
        "A": 95
      },
      "valueString": "[ var='A' value=95 ]"
    }
  ]
}

Manual test

curl -X POST \
  "https://incidentrelay.example.com/api/integrations/grafana" \
  -H "Authorization: Bearer ROUTE_INTAKE_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @grafana-payload.json

A successful response contains one result for every object in the Grafana alerts array.

[
  {
    "created": true,
    "alert_id": 123,
    "group_id": 45,
    "status": "firing",
    "team_id": 2,
    "team_slug": "sre",
    "route_id": 7,
    "routing_error": null,
    "trace_id": "..."
  }
]

HTTP responses

Status	Meaning
`200`	All alert instances were processed successfully
`202`	Processing was accepted but not fully completed synchronously
`207`	The notification contains mixed processing outcomes
`400`	Invalid payload or routing failure
`401`	Route intake token is missing or invalid

When routing fails, the response includes a trace_id. An administrator can use the alert explain trace to inspect matcher evaluation and the exact routing failure.

Troubleshooting

Route intake token is required

Grafana did not send an authorization header.

Verify:

Authentication Header Scheme: Bearer
Authentication Header Credentials: <route-intake-token>

Do not add the word Bearer to the credentials field when the scheme is configured separately.

Alert did not match any active route

Check that:

the route is active;
the route source is grafana;
the token belongs to that route;
the incoming labels satisfy every configured matcher;
the route's team is active and accessible.

Use the returned trace_id to inspect route evaluation.

Resolved alerts are not received

Make sure Disable resolved message is not enabled in the Grafana webhook contact point.

IncidentRelay uses the status of each item in the alerts array, not only the top-level status.

Alerts are duplicated

Check that Grafana sends a stable fingerprint. When the fingerprint is unavailable, keep rule UID and routing labels stable between firing and resolved notifications.

Do not include volatile values in labels used to identify an alert instance.

Dashboard or panel links are empty

Grafana includes dashboard and panel links only when the alert rule is associated with the relevant dashboard and panel metadata. The alert rule URL remains available through generatorURL.

Test notification does not match production routing

Grafana test payload labels can differ from real alert-rule labels. Compare the received labels in the explain trace with the route matchers.

Security

Use HTTPS for the webhook endpoint.
Store route intake tokens as secrets.
Use a separate route token for each integration or trust boundary.
Rotate a token immediately if it is exposed.
Do not place the token in the URL.
Restrict route matchers so a leaked token cannot route unrelated alerts.

Navigation Menu

Uh oh!

Grafana Alerting integration

Endpoint

Create an IncidentRelay route

Configure Grafana

Recommended Grafana labels

Alert normalization

Status

Title

Message

Severity

Team

Labels added by IncidentRelay

Deduplication

Stored payload

Example payload

Manual test

HTTP responses

Troubleshooting

Route intake token is required

Alert did not match any active route

Resolved alerts are not received

Alerts are duplicated

Dashboard or panel links are empty

Test notification does not match production routing

Security

Uh oh!

FilesExpand file tree

grafana.md

Latest commit

History

grafana.md

File metadata and controls

Grafana Alerting integration

Endpoint

Create an IncidentRelay route

Configure Grafana

Recommended Grafana labels

Alert normalization

Status

Title

Message

Severity

Team

Labels added by IncidentRelay

Deduplication

Stored payload

Example payload

Manual test

HTTP responses

Troubleshooting

Route intake token is required

Alert did not match any active route

Resolved alerts are not received

Alerts are duplicated

Dashboard or panel links are empty

Test notification does not match production routing

Security