Observability Integrations

Connect to Grafana, PagerDuty, Datadog, Netdata, Dynatrace, Splunk, Jenkins, and CloudBees for automated incident detection

Aurora integrates with leading observability and CI/CD platforms to automatically detect and investigate incidents in real-time.

Supported Platforms

Grafana

Alert webhooks from Grafana Cloud or self-hosted instances

PagerDuty

Incident webhooks with runbook integration

Datadog

Event stream webhooks for monitors and alerts

Netdata

Real-time health monitoring alerts

Dynatrace

Problem webhooks with Davis AI insights

Splunk

Search-based alerts and log anomalies

Jenkins

Build failure detection with RCA

CloudBees

CloudBees CI build/deployment failures

Integration Architecture

Webhook Ingestion

Each platform sends alerts to Aurora via webhook:

Platform → POST /api/webhooks/{platform} → Aurora Backend
                                            ↓
                                    Create Incident
                                            ↓
                                    Launch Background RCA
                                            ↓
                                    Stream Thoughts to UI

Common Webhook Flow

1. Alert triggered in platform

   Grafana/PagerDuty/Datadog monitor detects an issue

2. Platform sends webhook

   POST request to Aurora’s webhook endpoint with alert payload

3. Aurora validates & stores

   Webhook handler validates payload and stores in platform-specific table (e.g., grafana_alerts)

4. Incident creation

   Alert is converted to an incident in the incidents table

5. Correlation check

   Aurora checks if this alert correlates with existing incidents

6. RCA initiated

   If new incident, Aurora launches background investigation via Celery task

Platform-Specific Features

Grafana

Setup

1. Create a Contact Point in Grafana:
   • Type: Webhook
   • URL: https://your-aurora.com/api/webhooks/grafana
   • Method: POST
2. Attach contact point to alert rules

Webhook Payload

{
  "receiver": "aurora",
  "status": "firing",
  "alerts": [{
    "status": "firing",
    "labels": {
      "alertname": "HighErrorRate",
      "severity": "critical",
      "service": "api"
    },
    "annotations": {
      "summary": "Error rate above 5%",
      "description": "API service is experiencing high error rates"
    },
    "startsAt": "2026-03-03T10:30:00Z",
    "fingerprint": "a1b2c3d4"
  }]
}

Features

• Alert fingerprint used as unique ID for correlation
• Supports alert grouping by labels
• Auto-resolves incidents when alert status changes to resolved

See Grafana Integration for API token configuration.

PagerDuty

Setup

1. Go to Integrations → Generic Webhooks in PagerDuty
2. Add webhook URL: https://your-aurora.com/api/webhooks/pagerduty
3. Subscribe to events: incident.triggered, incident.acknowledged, incident.resolved

Webhook Payload

{
  "event": {
    "id": "incident_123",
    "event_type": "incident.triggered",
    "resource_type": "incident",
    "occurred_at": "2026-03-03T10:30:00Z",
    "data": {
      "id": "PD12345",
      "title": "Database connection failures",
      "status": "triggered",
      "urgency": "high",
      "service": {
        "id": "SERVICE_ID",
        "summary": "Production Database"
      }
    }
  }
}

Features

• Runbook Integration: Extract runbook URLs from incident metadata
• Event Consolidation: Aggregate all events for an incident (triggered, escalated, acknowledged, resolved)
• Bidirectional Sync: Update PagerDuty incident status from Aurora

Runbook Extraction

# server/routes/pagerduty/runbook_utils.py
def extract_runbook_from_pagerduty_event(incident_id: str, user_id: str):
    """Extract runbook URL from PagerDuty incident metadata."""
    events = fetch_and_consolidate_pagerduty_events(user_id, incident_id)
    
    for event in events:
        if 'body' in event and 'details' in event['body']:
            details = event['body']['details']
            if 'runbook_url' in details:
                return details['runbook_url']
    return None

Datadog

Setup

1. Go to Integrations → Webhooks in Datadog
2. Create new webhook:
   • Name: Aurora
   • URL: https://your-aurora.com/api/webhooks/datadog
3. Add webhook to monitor notification list: @webhook-aurora

Webhook Payload

{
  "id": "1234567890",
  "title": "[Triggered] CPU usage above 90%",
  "text": "CPU usage has been above 90% for 5 minutes",
  "priority": "normal",
  "tags": ["env:production", "service:api"],
  "alert_type": "error",
  "alert_transition": "Triggered",
  "date": 1709467800,
  "org": {"id": 123, "name": "My Org"},
  "link": "https://app.datadoghq.com/monitors/12345"
}

Features

• Tag-based correlation (match alerts with same service tag)
• Priority mapping: normal → medium, high → high, low → low
• Auto-linking to Datadog monitor dashboard

Netdata

Setup

1. In Netdata Cloud, go to Spaces → Notification Settings
2. Add Aurora webhook: https://your-aurora.com/api/webhooks/netdata
3. Select alert types to forward

Webhook Payload

{
  "alarm": "cpu_usage",
  "status": "CRITICAL",
  "hostname": "prod-server-01",
  "chart": "system.cpu",
  "family": "cpu",
  "value": 95.2,
  "units": "%",
  "when": 1709467800,
  "info": "CPU usage is critically high"
}

Features

• Real-time metric alerts
• Host-based correlation (group alerts by hostname)
• Chart data embedded in incident context

Dynatrace

Setup

1. Go to Settings → Integration → Problem notifications
2. Add custom integration:
   • Type: Custom webhook
   • URL: https://your-aurora.com/api/webhooks/dynatrace

Webhook Payload

{
  "ProblemID": "12345",
  "State": "OPEN",
  "ProblemTitle": "High response time detected",
  "ImpactedEntity": "SERVICE-A1B2C3D4",
  "Tags": "Environment:Production,Service:API",
  "ProblemDetailsJSON": {
    "displayName": "API Service",
    "impactLevel": "SERVICE",
    "severityLevel": "ERROR"
  }
}

Features

• Davis AI root cause insights included in incident context
• Entity-based correlation (group by impacted service)
• Automatic severity mapping from Dynatrace severity levels

Splunk

Setup

1. Create a Webhook Alert Action in Splunk:
   • URL: https://your-aurora.com/api/webhooks/splunk
   • Method: POST
2. Add webhook to alert actions in Search

Webhook Payload

{
  "sid": "scheduler__admin__search__RMD123",
  "search_name": "Error Rate Alert",
  "result": {
    "error_count": "150",
    "service": "api",
    "severity": "high"
  },
  "results_link": "https://splunk.example.com/app/search/..."
}

Features

• Search results embedded in incident context
• Link back to Splunk search results
• Custom field extraction from search results

Jenkins / CloudBees

Setup

1. Install Generic Webhook Trigger plugin in Jenkins
2. Add post-build action:
   • URL: https://your-aurora.com/api/webhooks/jenkins (or /cloudbees)
   • Trigger on: Build failure

Webhook Payload

{
  "build": {
    "number": 42,
    "status": "FAILURE",
    "url": "https://jenkins.example.com/job/api-build/42/",
    "full_url": "https://jenkins.example.com/job/api-build/42/",
    "scm": {
      "commit": "a1b2c3d4",
      "branch": "main"
    }
  },
  "name": "api-build",
  "url": "job/api-build/"
}

Features

• RCA for Build Failures: Optional setting to trigger RCA only for failures (not all builds)
• Build log extraction for investigation context
• Commit SHA and branch tracking
• Link back to Jenkins build console

Enable RCA for Jenkins/CloudBees via the Connectors page. When disabled, build events are logged but don’t create incidents.

Alert Correlation

Aurora uses multiple strategies to correlate related alerts:

1. Service-Based Correlation

Alerts affecting the same service are grouped:

# Check if alert matches existing incident by service
if alert_service == incident.alert_service:
    correlation_score = 0.9
    correlation_strategy = "service_match"

2. Time-Based Clustering

Alerts within a 5-minute window may be related:

time_diff = abs(alert_timestamp - incident.started_at)
if time_diff < timedelta(minutes=5):
    correlation_score = 0.7
    correlation_strategy = "time_cluster"

3. Semantic Similarity

Using Weaviate vector search for alert description similarity:

# server/chat/backend/agent/weaviate_client.py
results = weaviate_client.query(
    collection="Incidents",
    query_vector=embed(alert_description),
    limit=5,
    distance_threshold=0.3  # High similarity
)

if results:
    correlation_score = 1.0 - results[0].distance
    correlation_strategy = "semantic_similarity"

Correlated alerts are added to the incident_alerts table and displayed in the Correlated Alerts section of the incident detail page.

User Workflows

Configuring Webhooks

1. Navigate to Connectors page in Aurora
2. Click “Configure” next to the observability platform
3. Copy the webhook URL for your platform
4. Follow platform-specific setup instructions
5. Send test webhook to verify connectivity

Webhook configuration with unique URL per platform

Viewing Incidents from Observability Platforms

1. Alert fires in Grafana/PagerDuty/etc.
2. Webhook received by Aurora (within 1-2 seconds)
3. Incident appears on Incidents page with status investigating
4. Click incident to view:
   • Raw Alert: Original payload from platform
   • Thoughts: AI investigation progress
   • Suggestions: Diagnostic commands and fixes
   • Source Link: Deep link back to alerting platform

Manual Incident Creation

For platforms without native webhooks, manually create incidents via API:

curl -X POST https://your-aurora.com/api/incidents \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "custom",
    "alert_title": "Manual incident",
    "alert_service": "api",
    "severity": "high",
    "alert_metadata": {
      "description": "User-reported issue"
    }
  }'

Incident Lifecycle

From alert to resolution:

Status Transitions

From Status	To Status	Trigger
-	investigating	Alert webhook received, RCA started
investigating	analyzed	RCA completed, waiting for user action
analyzed	resolved	User marks incident as resolved
investigating	merged	Alert merged into another incident
resolved	resolved	Postmortem generated

Real-Time Updates

The Incidents page uses Server-Sent Events (SSE) for real-time updates:

// client/src/app/incidents/page.tsx
useEffect(() => {
  const eventSource = new EventSource('/api/incidents/stream');
  
  eventSource.onmessage = (event) => {
    const data = JSON.parse(event.data);
    if (data.type === 'incident_update') {
      refreshIncidents(true); // Silent refresh
    }
  };
  
  return () => eventSource.close();
}, []);

Backend sends SSE events when:

• New incident created
• Incident status changes
• RCA completes
• Suggestions added

API Reference

Webhook Endpoints

Grafana

POST /api/webhooks/grafana

All webhook endpoints:

• Accept JSON payloads
• Return 202 Accepted on success
• Validate payload structure
• Rate-limited per platform

Get Platform Status

GET /api/{platform}/status

Returns connection status and configuration:

{
  "connected": true,
  "base_url": "https://app.datadoghq.com",
  "webhook_url": "https://aurora.example.com/api/webhooks/datadog",
  "last_webhook_received": "2026-03-03T10:30:00Z"
}

Related Features

Incident Investigation

Automatic root cause analysis for incoming alerts

Cloud Integrations

Execute diagnostic commands during RCA investigations