> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Arvo-AI/aurora/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability Integrations

> Connect to Grafana, PagerDuty, Datadog, Netdata, Dynatrace, Splunk, Jenkins, and CloudBees for automated incident detection

Aurora integrates with leading observability and CI/CD platforms to automatically detect and investigate incidents in real-time.

## Supported Platforms

<CardGroup cols={2}>
  <Card title="Grafana" icon="chart-line">
    Alert webhooks from Grafana Cloud or self-hosted instances
  </Card>

  <Card title="PagerDuty" icon="bell">
    Incident webhooks with runbook integration
  </Card>

  <Card title="Datadog" icon="dog">
    Event stream webhooks for monitors and alerts
  </Card>

  <Card title="Netdata" icon="server">
    Real-time health monitoring alerts
  </Card>

  <Card title="Dynatrace" icon="chart-area">
    Problem webhooks with Davis AI insights
  </Card>

  <Card title="Splunk" icon="magnifying-glass">
    Search-based alerts and log anomalies
  </Card>

  <Card title="Jenkins" icon="wrench">
    Build failure detection with RCA
  </Card>

  <Card title="CloudBees" icon="cloud">
    CloudBees CI build/deployment failures
  </Card>
</CardGroup>

## Integration Architecture

### Webhook Ingestion

Each platform sends alerts to Aurora via webhook:

```
Platform → POST /api/webhooks/{platform} → Aurora Backend
                                            ↓
                                    Create Incident
                                            ↓
                                    Launch Background RCA
                                            ↓
                                    Stream Thoughts to UI
```

### Common Webhook Flow

<Steps>
  <Step title="Alert triggered in platform">
    Grafana/PagerDuty/Datadog monitor detects an issue
  </Step>

  <Step title="Platform sends webhook">
    POST request to Aurora's webhook endpoint with alert payload
  </Step>

  <Step title="Aurora validates & stores">
    Webhook handler validates payload and stores in platform-specific table (e.g., `grafana_alerts`)
  </Step>

  <Step title="Incident creation">
    Alert is converted to an incident in the `incidents` table
  </Step>

  <Step title="Correlation check">
    Aurora checks if this alert correlates with existing incidents
  </Step>

  <Step title="RCA initiated">
    If new incident, Aurora launches background investigation via Celery task
  </Step>
</Steps>

## Platform-Specific Features

<Tabs>
  <Tab title="Grafana">
    ### Setup

    1. Create a **Contact Point** in Grafana:
       * Type: Webhook
       * URL: `https://your-aurora.com/api/webhooks/grafana`
       * Method: POST
    2. Attach contact point to alert rules

    ### Webhook Payload

    ```json theme={null}
    {
      "receiver": "aurora",
      "status": "firing",
      "alerts": [{
        "status": "firing",
        "labels": {
          "alertname": "HighErrorRate",
          "severity": "critical",
          "service": "api"
        },
        "annotations": {
          "summary": "Error rate above 5%",
          "description": "API service is experiencing high error rates"
        },
        "startsAt": "2026-03-03T10:30:00Z",
        "fingerprint": "a1b2c3d4"
      }]
    }
    ```

    ### Features

    * Alert fingerprint used as unique ID for correlation
    * Supports alert grouping by labels
    * Auto-resolves incidents when alert status changes to `resolved`

    <Info>
      See [Grafana Integration](/integrations/observability/grafana) for API token configuration.
    </Info>
  </Tab>

  <Tab title="PagerDuty">
    ### Setup

    1. Go to **Integrations** → **Generic Webhooks** in PagerDuty
    2. Add webhook URL: `https://your-aurora.com/api/webhooks/pagerduty`
    3. Subscribe to events: `incident.triggered`, `incident.acknowledged`, `incident.resolved`

    ### Webhook Payload

    ```json theme={null}
    {
      "event": {
        "id": "incident_123",
        "event_type": "incident.triggered",
        "resource_type": "incident",
        "occurred_at": "2026-03-03T10:30:00Z",
        "data": {
          "id": "PD12345",
          "title": "Database connection failures",
          "status": "triggered",
          "urgency": "high",
          "service": {
            "id": "SERVICE_ID",
            "summary": "Production Database"
          }
        }
      }
    }
    ```

    ### Features

    * **Runbook Integration**: Extract runbook URLs from incident metadata
    * **Event Consolidation**: Aggregate all events for an incident (triggered, escalated, acknowledged, resolved)
    * **Bidirectional Sync**: Update PagerDuty incident status from Aurora

    <CodeGroup>
      ```python Runbook Extraction theme={null}
      # server/routes/pagerduty/runbook_utils.py
      def extract_runbook_from_pagerduty_event(incident_id: str, user_id: str):
          """Extract runbook URL from PagerDuty incident metadata."""
          events = fetch_and_consolidate_pagerduty_events(user_id, incident_id)
          
          for event in events:
              if 'body' in event and 'details' in event['body']:
                  details = event['body']['details']
                  if 'runbook_url' in details:
                      return details['runbook_url']
          return None
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Datadog">
    ### Setup

    1. Go to **Integrations** → **Webhooks** in Datadog
    2. Create new webhook:
       * Name: Aurora
       * URL: `https://your-aurora.com/api/webhooks/datadog`
    3. Add webhook to monitor notification list: `@webhook-aurora`

    ### Webhook Payload

    ```json theme={null}
    {
      "id": "1234567890",
      "title": "[Triggered] CPU usage above 90%",
      "text": "CPU usage has been above 90% for 5 minutes",
      "priority": "normal",
      "tags": ["env:production", "service:api"],
      "alert_type": "error",
      "alert_transition": "Triggered",
      "date": 1709467800,
      "org": {"id": 123, "name": "My Org"},
      "link": "https://app.datadoghq.com/monitors/12345"
    }
    ```

    ### Features

    * Tag-based correlation (match alerts with same `service` tag)
    * Priority mapping: `normal` → `medium`, `high` → `high`, `low` → `low`
    * Auto-linking to Datadog monitor dashboard
  </Tab>

  <Tab title="Netdata">
    ### Setup

    1. In Netdata Cloud, go to **Spaces** → **Notification Settings**
    2. Add Aurora webhook: `https://your-aurora.com/api/webhooks/netdata`
    3. Select alert types to forward

    ### Webhook Payload

    ```json theme={null}
    {
      "alarm": "cpu_usage",
      "status": "CRITICAL",
      "hostname": "prod-server-01",
      "chart": "system.cpu",
      "family": "cpu",
      "value": 95.2,
      "units": "%",
      "when": 1709467800,
      "info": "CPU usage is critically high"
    }
    ```

    ### Features

    * Real-time metric alerts
    * Host-based correlation (group alerts by hostname)
    * Chart data embedded in incident context
  </Tab>

  <Tab title="Dynatrace">
    ### Setup

    1. Go to **Settings** → **Integration** → **Problem notifications**
    2. Add custom integration:
       * Type: Custom webhook
       * URL: `https://your-aurora.com/api/webhooks/dynatrace`

    ### Webhook Payload

    ```json theme={null}
    {
      "ProblemID": "12345",
      "State": "OPEN",
      "ProblemTitle": "High response time detected",
      "ImpactedEntity": "SERVICE-A1B2C3D4",
      "Tags": "Environment:Production,Service:API",
      "ProblemDetailsJSON": {
        "displayName": "API Service",
        "impactLevel": "SERVICE",
        "severityLevel": "ERROR"
      }
    }
    ```

    ### Features

    * Davis AI root cause insights included in incident context
    * Entity-based correlation (group by impacted service)
    * Automatic severity mapping from Dynatrace severity levels
  </Tab>

  <Tab title="Splunk">
    ### Setup

    1. Create a **Webhook Alert Action** in Splunk:
       * URL: `https://your-aurora.com/api/webhooks/splunk`
       * Method: POST
    2. Add webhook to alert actions in Search

    ### Webhook Payload

    ```json theme={null}
    {
      "sid": "scheduler__admin__search__RMD123",
      "search_name": "Error Rate Alert",
      "result": {
        "error_count": "150",
        "service": "api",
        "severity": "high"
      },
      "results_link": "https://splunk.example.com/app/search/..."
    }
    ```

    ### Features

    * Search results embedded in incident context
    * Link back to Splunk search results
    * Custom field extraction from search results
  </Tab>

  <Tab title="Jenkins / CloudBees">
    ### Setup

    1. Install **Generic Webhook Trigger** plugin in Jenkins
    2. Add post-build action:
       * URL: `https://your-aurora.com/api/webhooks/jenkins` (or `/cloudbees`)
       * Trigger on: Build failure

    ### Webhook Payload

    ```json theme={null}
    {
      "build": {
        "number": 42,
        "status": "FAILURE",
        "url": "https://jenkins.example.com/job/api-build/42/",
        "full_url": "https://jenkins.example.com/job/api-build/42/",
        "scm": {
          "commit": "a1b2c3d4",
          "branch": "main"
        }
      },
      "name": "api-build",
      "url": "job/api-build/"
    }
    ```

    ### Features

    * **RCA for Build Failures**: Optional setting to trigger RCA only for failures (not all builds)
    * Build log extraction for investigation context
    * Commit SHA and branch tracking
    * Link back to Jenkins build console

    <Note>
      Enable RCA for Jenkins/CloudBees via the **Connectors** page. When disabled, build events are logged but don't create incidents.
    </Note>
  </Tab>
</Tabs>

## Alert Correlation

Aurora uses multiple strategies to correlate related alerts:

### 1. Service-Based Correlation

Alerts affecting the same service are grouped:

```python theme={null}
# Check if alert matches existing incident by service
if alert_service == incident.alert_service:
    correlation_score = 0.9
    correlation_strategy = "service_match"
```

### 2. Time-Based Clustering

Alerts within a 5-minute window may be related:

```python theme={null}
time_diff = abs(alert_timestamp - incident.started_at)
if time_diff < timedelta(minutes=5):
    correlation_score = 0.7
    correlation_strategy = "time_cluster"
```

### 3. Semantic Similarity

Using Weaviate vector search for alert description similarity:

```python theme={null}
# server/chat/backend/agent/weaviate_client.py
results = weaviate_client.query(
    collection="Incidents",
    query_vector=embed(alert_description),
    limit=5,
    distance_threshold=0.3  # High similarity
)

if results:
    correlation_score = 1.0 - results[0].distance
    correlation_strategy = "semantic_similarity"
```

<Info>
  Correlated alerts are added to the `incident_alerts` table and displayed in the **Correlated Alerts** section of the incident detail page.
</Info>

## User Workflows

### Configuring Webhooks

1. Navigate to **Connectors** page in Aurora
2. Click **"Configure"** next to the observability platform
3. Copy the webhook URL for your platform
4. Follow platform-specific setup instructions
5. Send test webhook to verify connectivity

<Frame>
  <img src="https://mintlify.s3.us-west-1.amazonaws.com/arvo-ai-aurora/images/placeholder-webhook-setup.png" alt="Webhook configuration page showing unique URL and test button" />

  *Webhook configuration with unique URL per platform*
</Frame>

### Viewing Incidents from Observability Platforms

1. Alert fires in Grafana/PagerDuty/etc.
2. Webhook received by Aurora (within 1-2 seconds)
3. Incident appears on **Incidents** page with status `investigating`
4. Click incident to view:
   * **Raw Alert**: Original payload from platform
   * **Thoughts**: AI investigation progress
   * **Suggestions**: Diagnostic commands and fixes
   * **Source Link**: Deep link back to alerting platform

### Manual Incident Creation

For platforms without native webhooks, manually create incidents via API:

```bash theme={null}
curl -X POST https://your-aurora.com/api/incidents \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "custom",
    "alert_title": "Manual incident",
    "alert_service": "api",
    "severity": "high",
    "alert_metadata": {
      "description": "User-reported issue"
    }
  }'
```

## Incident Lifecycle

From alert to resolution:

```mermaid theme={null}
flowchart LR
    A[Alert Webhook] --> B[Incident Created]
    B --> C{Correlate?}
    C -->|Yes| D[Add to Existing Incident]
    C -->|No| E[Launch New RCA]
    E --> F[Investigation Running]
    F --> G[Thoughts + Suggestions]
    G --> H[User Reviews]
    H --> I{Resolved?}
    I -->|Yes| J[Mark Resolved]
    I -->|No| K[Continue Investigation]
    J --> L[Generate Postmortem]
```

### Status Transitions

| From Status     | To Status       | Trigger                                |
| --------------- | --------------- | -------------------------------------- |
| -               | `investigating` | Alert webhook received, RCA started    |
| `investigating` | `analyzed`      | RCA completed, waiting for user action |
| `analyzed`      | `resolved`      | User marks incident as resolved        |
| `investigating` | `merged`        | Alert merged into another incident     |
| `resolved`      | `resolved`      | Postmortem generated                   |

## Real-Time Updates

The Incidents page uses Server-Sent Events (SSE) for real-time updates:

```typescript theme={null}
// client/src/app/incidents/page.tsx
useEffect(() => {
  const eventSource = new EventSource('/api/incidents/stream');
  
  eventSource.onmessage = (event) => {
    const data = JSON.parse(event.data);
    if (data.type === 'incident_update') {
      refreshIncidents(true); // Silent refresh
    }
  };
  
  return () => eventSource.close();
}, []);
```

Backend sends SSE events when:

* New incident created
* Incident status changes
* RCA completes
* Suggestions added

## API Reference

### Webhook Endpoints

<CodeGroup>
  ```http Grafana theme={null}
  POST /api/webhooks/grafana
  ```

  ```http PagerDuty theme={null}
  POST /api/webhooks/pagerduty
  ```

  ```http Datadog theme={null}
  POST /api/webhooks/datadog
  ```

  ```http Netdata theme={null}
  POST /api/webhooks/netdata
  ```

  ```http Dynatrace theme={null}
  POST /api/webhooks/dynatrace
  ```

  ```http Splunk theme={null}
  POST /api/webhooks/splunk
  ```

  ```http Jenkins theme={null}
  POST /api/webhooks/jenkins
  ```

  ```http CloudBees theme={null}
  POST /api/webhooks/cloudbees
  ```
</CodeGroup>

All webhook endpoints:

* Accept JSON payloads
* Return `202 Accepted` on success
* Validate payload structure
* Rate-limited per platform

### Get Platform Status

```http theme={null}
GET /api/{platform}/status
```

Returns connection status and configuration:

```json theme={null}
{
  "connected": true,
  "base_url": "https://app.datadoghq.com",
  "webhook_url": "https://aurora.example.com/api/webhooks/datadog",
  "last_webhook_received": "2026-03-03T10:30:00Z"
}
```

## Related Features

<CardGroup cols={2}>
  <Card title="Incident Investigation" icon="bug" href="/features/incident-investigation">
    Automatic root cause analysis for incoming alerts
  </Card>

  <Card title="Cloud Integrations" icon="cloud" href="/features/cloud-integrations">
    Execute diagnostic commands during RCA investigations
  </Card>
</CardGroup>