> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Arvo-AI/aurora/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and Restore

> Back up and restore Aurora incident data, configurations, and storage

Aurora stores critical incident data across multiple systems. This guide covers backing up and restoring PostgreSQL, Weaviate, Vault, and object storage.

## What to Back Up

Aurora's data is distributed across:

1. **PostgreSQL** - Incidents, alerts, suggestions, thoughts, users, credentials
2. **Weaviate** - Semantic search vectors, knowledge base embeddings
3. **Vault** - User credentials, API tokens, secrets
4. **Object Storage** (SeaweedFS/S3) - File uploads, Terraform state, exported data
5. **Redis** - Celery tasks, cache (ephemeral - can be rebuilt)

## Backup Strategy

<Tip>
  **Recommended**: Run automated backups daily to S3-compatible storage with 30-day retention.
</Tip>

### Backup Schedule

* **PostgreSQL**: Daily full backup + continuous WAL archiving
* **Weaviate**: Weekly snapshot
* **Vault**: Daily backup
* **Object Storage**: Continuous replication to secondary region
* **Redis**: No backup needed (cache only)

## Backing Up PostgreSQL

### Manual Backup

<Steps>
  <Step title="Create a backup directory">
    ```bash theme={null}
    mkdir -p ~/aurora-backups/postgres
    ```
  </Step>

  <Step title="Run pg_dump">
    ```bash theme={null}
    docker exec aurora-postgres pg_dump \
      -U $POSTGRES_USER \
      -d $POSTGRES_DB \
      --format=custom \
      --compress=9 \
      --file=/tmp/aurora_backup.dump

    docker cp aurora-postgres:/tmp/aurora_backup.dump \
      ~/aurora-backups/postgres/aurora_$(date +%Y%m%d_%H%M%S).dump
    ```

    The `--format=custom` option enables parallel restore and selective table restore.
  </Step>

  <Step title="Verify the backup">
    ```bash theme={null}
    docker exec aurora-postgres pg_restore \
      --list /tmp/aurora_backup.dump | head -20
    ```
  </Step>
</Steps>

### Automated Backup Script

Create `scripts/backup-postgres.sh`:

```bash theme={null}
#!/bin/bash
set -e

# Configuration
BACKUP_DIR="/var/backups/aurora/postgres"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/aurora_$DATE.dump"

# Ensure backup directory exists
mkdir -p $BACKUP_DIR

# Run backup
docker exec aurora-postgres pg_dump \
  -U $POSTGRES_USER \
  -d $POSTGRES_DB \
  --format=custom \
  --compress=9 \
  --file=/tmp/backup.dump

docker cp aurora-postgres:/tmp/backup.dump $BACKUP_FILE

# Upload to S3 (optional)
if [ -n "$AWS_S3_BACKUP_BUCKET" ]; then
  aws s3 cp $BACKUP_FILE s3://$AWS_S3_BACKUP_BUCKET/postgres/
fi

# Clean up old backups
find $BACKUP_DIR -name "aurora_*.dump" -mtime +$RETENTION_DAYS -delete

echo "Backup completed: $BACKUP_FILE"
```

Run via cron:

```bash theme={null}
0 2 * * * /path/to/scripts/backup-postgres.sh
```

### Continuous WAL Archiving

For point-in-time recovery, enable WAL archiving in PostgreSQL.

Add to `config/postgres/postgresql.conf`:

```
wal_level = replica
archive_mode = on
archive_command = 'aws s3 cp %p s3://your-backup-bucket/wal/%f'
```

Update `docker-compose.yaml` to mount the config:

```yaml theme={null}
postgres:
  volumes:
    - ./config/postgres/postgresql.conf:/etc/postgresql/postgresql.conf:ro
  command: postgres -c config_file=/etc/postgresql/postgresql.conf
```

## Backing Up Weaviate

<Steps>
  <Step title="Create a Weaviate backup">
    Weaviate supports backups via API:

    ```bash theme={null}
    curl -X POST "http://localhost:8080/v1/backups/filesystem" \
      -H "Content-Type: application/json" \
      -d '{
        "id": "aurora-backup-'$(date +%Y%m%d)'",
        "include": ["AuroraKnowledge", "AuroraIncidents"]
      }'
    ```
  </Step>

  <Step title="Check backup status">
    ```bash theme={null}
    curl "http://localhost:8080/v1/backups/filesystem/aurora-backup-20260303"
    ```
  </Step>

  <Step title="Export backup files">
    Weaviate stores backups in `/var/lib/weaviate/backups` inside the container:

    ```bash theme={null}
    docker cp weaviate:/var/lib/weaviate/backups/aurora-backup-20260303 \
      ~/aurora-backups/weaviate/
    ```
  </Step>
</Steps>

### Automated Weaviate Backup

```bash theme={null}
#!/bin/bash
set -e

BACKUP_ID="aurora-backup-$(date +%Y%m%d)"
BACKUP_DIR="/var/backups/aurora/weaviate"

# Create backup
curl -X POST "http://localhost:8080/v1/backups/filesystem" \
  -H "Content-Type: application/json" \
  -d '{"id": "'$BACKUP_ID'"}'

# Wait for completion
while true; do
  STATUS=$(curl -s "http://localhost:8080/v1/backups/filesystem/$BACKUP_ID" | jq -r '.status')
  if [ "$STATUS" = "SUCCESS" ]; then
    break
  fi
  sleep 5
done

# Export backup
mkdir -p $BACKUP_DIR
docker cp weaviate:/var/lib/weaviate/backups/$BACKUP_ID $BACKUP_DIR/

echo "Weaviate backup completed: $BACKUP_ID"
```

## Backing Up Vault

<Warning>
  Vault backups contain sensitive credentials. Encrypt and restrict access.
</Warning>

<Steps>
  <Step title="Take a Vault snapshot">
    ```bash theme={null}
    docker exec aurora-vault vault operator raft snapshot save /tmp/vault-snapshot.snap

    docker cp aurora-vault:/tmp/vault-snapshot.snap \
      ~/aurora-backups/vault/vault_$(date +%Y%m%d_%H%M%S).snap
    ```
  </Step>

  <Step title="Encrypt the snapshot">
    ```bash theme={null}
    # Encrypt with GPG
    gpg --symmetric --cipher-algo AES256 \
      ~/aurora-backups/vault/vault_20260303_020000.snap

    # Or use age
    age -p ~/aurora-backups/vault/vault_20260303_020000.snap > \
      ~/aurora-backups/vault/vault_20260303_020000.snap.age
    ```
  </Step>

  <Step title="Upload to secure storage">
    ```bash theme={null}
    aws s3 cp ~/aurora-backups/vault/vault_20260303_020000.snap.gpg \
      s3://your-backup-bucket/vault/ \
      --sse AES256
    ```
  </Step>
</Steps>

### Vault Backup Script

```bash theme={null}
#!/bin/bash
set -e

BACKUP_DIR="/var/backups/aurora/vault"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/vault_$DATE.snap"
ENCRYPTION_KEY_FILE="/etc/aurora/vault-backup-key.txt"

mkdir -p $BACKUP_DIR

# Take snapshot
docker exec aurora-vault vault operator raft snapshot save /tmp/snapshot.snap
docker cp aurora-vault:/tmp/snapshot.snap $BACKUP_FILE

# Encrypt
if [ -f "$ENCRYPTION_KEY_FILE" ]; then
  gpg --batch --yes --passphrase-file $ENCRYPTION_KEY_FILE \
    --symmetric --cipher-algo AES256 $BACKUP_FILE
  rm $BACKUP_FILE  # Remove unencrypted file
  BACKUP_FILE="$BACKUP_FILE.gpg"
fi

# Upload to S3
if [ -n "$AWS_S3_BACKUP_BUCKET" ]; then
  aws s3 cp $BACKUP_FILE s3://$AWS_S3_BACKUP_BUCKET/vault/ --sse AES256
fi

echo "Vault backup completed: $BACKUP_FILE"
```

## Backing Up Object Storage (SeaweedFS)

SeaweedFS backups depend on your deployment:

### Option 1: S3 Sync

Sync to another S3-compatible service:

```bash theme={null}
#!/bin/bash
aws s3 sync \
  --endpoint-url http://localhost:8333 \
  s3://aurora-storage \
  s3://aurora-storage-backup \
  --region us-east-1
```

### Option 2: Volume Backup

Back up Docker volumes:

```bash theme={null}
docker run --rm \
  -v aurora_seaweedfs_filer_data:/data \
  -v ~/aurora-backups/seaweedfs:/backup \
  alpine tar czf /backup/seaweedfs_$(date +%Y%m%d).tar.gz /data
```

### Option 3: SeaweedFS Replication

Configure cross-datacenter replication in `docker-compose.yaml`:

```yaml theme={null}
seaweedfs-master:
  command: >
    master
    -defaultReplication=010  # 0 same rack, 1 different datacenter, 0 different racks
```

## Restoring from Backup

### Restore PostgreSQL

<Steps>
  <Step title="Stop Aurora services">
    ```bash theme={null}
    make down
    ```
  </Step>

  <Step title="Drop and recreate database">
    ```bash theme={null}
    docker-compose up -d postgres
    docker exec -it aurora-postgres psql -U $POSTGRES_USER -c "DROP DATABASE IF EXISTS $POSTGRES_DB;"
    docker exec -it aurora-postgres psql -U $POSTGRES_USER -c "CREATE DATABASE $POSTGRES_DB;"
    ```
  </Step>

  <Step title="Restore from dump">
    ```bash theme={null}
    docker cp ~/aurora-backups/postgres/aurora_20260303_020000.dump aurora-postgres:/tmp/restore.dump

    docker exec aurora-postgres pg_restore \
      -U $POSTGRES_USER \
      -d $POSTGRES_DB \
      --verbose \
      /tmp/restore.dump
    ```
  </Step>

  <Step title="Restart Aurora">
    ```bash theme={null}
    make dev
    ```
  </Step>
</Steps>

### Restore Weaviate

<Steps>
  <Step title="Stop Weaviate">
    ```bash theme={null}
    docker-compose stop weaviate
    ```
  </Step>

  <Step title="Copy backup into container">
    ```bash theme={null}
    docker cp ~/aurora-backups/weaviate/aurora-backup-20260303 \
      weaviate:/var/lib/weaviate/backups/
    ```
  </Step>

  <Step title="Restore via API">
    ```bash theme={null}
    docker-compose up -d weaviate

    curl -X POST "http://localhost:8080/v1/backups/filesystem/aurora-backup-20260303/restore" \
      -H "Content-Type: application/json"
    ```
  </Step>

  <Step title="Verify restore">
    ```bash theme={null}
    curl "http://localhost:8080/v1/schema"
    ```
  </Step>
</Steps>

### Restore Vault

<Steps>
  <Step title="Stop Vault">
    ```bash theme={null}
    docker-compose stop vault vault-init
    ```
  </Step>

  <Step title="Decrypt backup">
    ```bash theme={null}
    gpg --decrypt ~/aurora-backups/vault/vault_20260303_020000.snap.gpg > \
      ~/aurora-backups/vault/vault_20260303_020000.snap
    ```
  </Step>

  <Step title="Restore snapshot">
    ```bash theme={null}
    docker cp ~/aurora-backups/vault/vault_20260303_020000.snap aurora-vault:/tmp/restore.snap

    docker-compose up -d vault
    docker exec aurora-vault vault operator raft snapshot restore /tmp/restore.snap
    ```
  </Step>

  <Step title="Unseal Vault">
    ```bash theme={null}
    docker exec aurora-vault vault operator unseal
    ```
  </Step>
</Steps>

### Restore Object Storage

Restore SeaweedFS volumes:

```bash theme={null}
docker run --rm \
  -v aurora_seaweedfs_filer_data:/data \
  -v ~/aurora-backups/seaweedfs:/backup \
  alpine tar xzf /backup/seaweedfs_20260303.tar.gz -C /
```

Or sync from backup S3:

```bash theme={null}
aws s3 sync s3://aurora-storage-backup s3://aurora-storage \
  --endpoint-url http://localhost:8333
```

## Disaster Recovery Testing

<Tip>
  Test your restore process monthly. Backups are useless if you can't restore them.
</Tip>

### DR Test Checklist

1. **Create a test environment**
   ```bash theme={null}
   cp .env .env.dr-test
   # Update ports to avoid conflicts
   ```

2. **Restore all components** (PostgreSQL, Weaviate, Vault, object storage)

3. **Verify data integrity**
   * Check incident count: `SELECT COUNT(*) FROM incidents;`
   * Test authentication
   * Verify file uploads work

4. **Run a test investigation** to ensure all systems function

5. **Document any issues** and update procedures

## Backup Monitoring

Monitor backup health:

```bash theme={null}
#!/bin/bash
# scripts/check-backup-freshness.sh

MAX_AGE_HOURS=48
BACKUP_DIR="/var/backups/aurora/postgres"

LATEST_BACKUP=$(find $BACKUP_DIR -name "aurora_*.dump" -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" ")

if [ -z "$LATEST_BACKUP" ]; then
  echo "ERROR: No backups found in $BACKUP_DIR"
  exit 1
fi

BACKUP_AGE=$(( ( $(date +%s) - $(stat -c %Y "$LATEST_BACKUP") ) / 3600 ))

if [ $BACKUP_AGE -gt $MAX_AGE_HOURS ]; then
  echo "WARNING: Latest backup is $BACKUP_AGE hours old (max: $MAX_AGE_HOURS)"
  exit 1
fi

echo "OK: Latest backup is $BACKUP_AGE hours old"
```

Integrate with monitoring:

```bash theme={null}
# Add to crontab
0 * * * * /path/to/scripts/check-backup-freshness.sh || curl https://healthchecks.io/your-check-id
```

## Backup Retention Policy

<Tabs>
  <Tab title="Development">
    * Daily backups: 7 days retention
    * Weekly backups: 4 weeks retention
    * No long-term archival
  </Tab>

  <Tab title="Production">
    * Daily backups: 30 days retention
    * Weekly backups: 12 weeks retention
    * Monthly backups: 12 months retention
    * Yearly backups: 7 years (compliance)
  </Tab>
</Tabs>

Implement with S3 lifecycle policies:

```json theme={null}
{
  "Rules": [
    {
      "Id": "aurora-backup-lifecycle",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 2555
      }
    }
  ]
}
```

## Security Considerations

<Warning>
  Backups contain sensitive data. Always encrypt and restrict access.
</Warning>

1. **Encrypt backups at rest** - Use GPG, age, or S3 SSE
2. **Encrypt backups in transit** - Use HTTPS/TLS for uploads
3. **Restrict access** - Use IAM policies, bucket policies, and MFA
4. **Audit access logs** - Enable CloudTrail or equivalent
5. **Test encryption** - Verify you can decrypt backups
6. **Rotate encryption keys** - Update keys annually
7. **Separate backup credentials** - Use dedicated service accounts

## Troubleshooting

<AccordionGroup>
  <Accordion title="Backup script fails with permission denied">
    Ensure the script can access Docker:

    ```bash theme={null}
    sudo usermod -aG docker $USER
    # Log out and back in
    ```
  </Accordion>

  <Accordion title="Restore fails with constraint violations">
    Restore with `--no-owner --no-privileges`:

    ```bash theme={null}
    pg_restore -U $POSTGRES_USER -d $POSTGRES_DB \
      --no-owner --no-privileges /tmp/restore.dump
    ```
  </Accordion>

  <Accordion title="Weaviate backup times out">
    Increase timeout for large datasets:

    ```bash theme={null}
    curl -X POST -m 3600 "http://localhost:8080/v1/backups/filesystem" ...
    ```
  </Accordion>

  <Accordion title="Vault unseal keys lost">
    If you lose unseal keys, you **cannot** recover Vault data. This is by design.

    Always store unseal keys in multiple secure locations:

    * Encrypted USB drives
    * Hardware security modules (HSMs)
    * Split across key personnel (Shamir's Secret Sharing)
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="First Investigation" icon="play" href="/guides/first-investigation">
    Run your first incident investigation
  </Card>

  <Card title="Custom Connectors" icon="code" href="/guides/custom-connectors">
    Build integrations for proprietary systems
  </Card>
</CardGroup>