Scaling Aurora

Scale Aurora horizontally and vertically for high-traffic production deployments

Scaling Aurora

Guidelines for scaling Aurora to handle increased load, traffic, and concurrent users.

Scaling Strategy

Aurora components fall into three categories:

Stateless Services (Horizontally Scalable)

These services can be scaled by increasing replica count:

• aurora-server - REST API (handles HTTP requests)
• celery-worker - Background tasks (RCA analysis, integrations)
• chatbot - WebSocket server (chat interface)
• frontend - Next.js UI (serves web pages)
• searxng - Web search engine
• t2v-transformers - ML embeddings

Stateful Services (Requires Special Configuration)

These services require additional setup for horizontal scaling:

• postgres - Database (replication, read replicas)
• redis - Cache and queue (Redis Cluster)
• weaviate - Vector database (multi-node cluster)
• vault - Secrets management (Raft HA)

Single-Instance Services

These MUST remain at 1 replica:

• celery-beat - Task scheduler (multiple instances cause duplicate tasks)

Horizontal Scaling

Kubernetes (Helm)

Increase replica counts in values.generated.yaml:

replicaCounts:
  # Scale based on traffic
  server: 5          # API requests
  celeryWorker: 10   # Background tasks
  chatbot: 3         # WebSocket connections
  frontend: 3        # Web traffic
  
  # Scale for performance
  searxng: 2         # Web search
  transformers: 2    # ML embeddings
  
  # Keep at 1
  celeryBeat: 1      # DO NOT SCALE

Apply changes:

helm upgrade aurora-oss ./deploy/helm/aurora \
  --namespace aurora \
  -f values.generated.yaml

Docker Compose

Scale services manually:

# Scale specific service
docker compose up -d --scale celery_worker=5

# Scale multiple services
docker compose up -d \
  --scale celery_worker=5 \
  --scale aurora-server=3

Or edit docker-compose.yaml:

services:
  celery_worker:
    # ... existing config ...
    deploy:
      replicas: 5

Auto-Scaling

Horizontal Pod Autoscaler (HPA)

Automatically scale based on CPU/memory:

1. Enable metrics server

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   
2. Create HPA for API server

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: aurora-server-hpa
     namespace: aurora
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: aurora-oss-server
     minReplicas: 3
     maxReplicas: 10
     metrics:
       - type: Resource
         resource:
           name: cpu
           target:
             type: Utilization
             averageUtilization: 70
       - type: Resource
         resource:
           name: memory
           target:
             type: Utilization
             averageUtilization: 80
     behavior:
       scaleUp:
         stabilizationWindowSeconds: 60
         policies:
           - type: Percent
             value: 50
             periodSeconds: 60
       scaleDown:
         stabilizationWindowSeconds: 300
         policies:
           - type: Pods
             value: 1
             periodSeconds: 120
   

   Apply:

   kubectl apply -f aurora-server-hpa.yaml
   
3. Create HPA for Celery workers

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: aurora-celery-worker-hpa
     namespace: aurora
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: aurora-oss-celery-worker
     minReplicas: 2
     maxReplicas: 20
     metrics:
       - type: Resource
         resource:
           name: cpu
           target:
             type: Utilization
             averageUtilization: 80
   
4. Monitor autoscaling

   # Check HPA status
   kubectl get hpa -n aurora
   
   # Watch scaling events
   kubectl get hpa -n aurora -w
   
   # View scaling events
   kubectl describe hpa aurora-server-hpa -n aurora
   

Custom Metrics Autoscaling

Scale based on application metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: aurora-celery-queue-hpa
  namespace: aurora
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aurora-oss-celery-worker
  minReplicas: 2
  maxReplicas: 50
  metrics:
    # Scale based on Celery queue length
    - type: External
      external:
        metric:
          name: redis_celery_queue_length
          selector:
            matchLabels:
              queue: celery
        target:
          type: AverageValue
          averageValue: "10"  # 10 tasks per worker

Vertical Scaling

Increase Resource Limits

For Kubernetes, update values.generated.yaml:

resources:
  server:
    requests:
      cpu: "1000m"      # Increased from 500m
      memory: "2Gi"     # Increased from 1Gi
    limits:
      cpu: "4000m"      # Increased from 2000m
      memory: "8Gi"     # Increased from 4Gi
  
  celeryWorker:
    requests:
      cpu: "500m"       # Increased from 200m
      memory: "4Gi"     # Increased from 2Gi
    limits:
      cpu: "2000m"      # Increased from 1000m
      memory: "16Gi"    # Increased from 8Gi

Apply changes:

helm upgrade aurora-oss ./deploy/helm/aurora \
  --namespace aurora \
  -f values.generated.yaml

Vertical Pod Autoscaler (VPA)

Automatically adjust resource requests:

1. Install VPA

   git clone https://github.com/kubernetes/autoscaler.git
   cd autoscaler/vertical-pod-autoscaler
   ./hack/vpa-up.sh
   
2. Create VPA for API server

   apiVersion: autoscaling.k8s.io/v1
   kind: VerticalPodAutoscaler
   metadata:
     name: aurora-server-vpa
     namespace: aurora
   spec:
     targetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: aurora-oss-server
     updatePolicy:
       updateMode: "Auto"  # or "Recreate" or "Initial"
     resourcePolicy:
       containerPolicies:
         - containerName: '*'
           minAllowed:
             cpu: 100m
             memory: 256Mi
           maxAllowed:
             cpu: 4000m
             memory: 8Gi
   
3. Monitor recommendations

   kubectl get vpa aurora-server-vpa -n aurora
   kubectl describe vpa aurora-server-vpa -n aurora
   

Database Scaling

PostgreSQL

Read Replicas

For read-heavy workloads:

Managed Database (Recommended):

• AWS RDS: Create read replicas via console
• GCP Cloud SQL: Enable read replicas
• Azure Database: Add read replicas

Configure application:

config:
  POSTGRES_HOST: "aurora-primary.xyz.rds.amazonaws.com"
  POSTGRES_READ_REPLICA_HOST: "aurora-replica.xyz.rds.amazonaws.com"

Self-Managed:

# Deploy read replica
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-replica
spec:
  serviceName: postgres-replica
  replicas: 2
  template:
    spec:
      containers:
        - name: postgres
          image: postgres:15-alpine
          env:
            - name: POSTGRES_PRIMARY_HOST
              value: "aurora-oss-postgres-0.aurora-oss-postgres"
            - name: POSTGRES_REPLICATION_MODE
              value: "slave"

Connection Pooling

Use PgBouncer to reduce database connections:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
  namespace: aurora
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: pgbouncer
          image: edoburu/pgbouncer:latest
          env:
            - name: DATABASE_URL
              value: "postgresql://aurora:password@aurora-oss-postgres:5432/aurora_db"
            - name: POOL_MODE
              value: "transaction"
            - name: MAX_CLIENT_CONN
              value: "1000"
            - name: DEFAULT_POOL_SIZE
              value: "25"

Update application:

config:
  POSTGRES_HOST: "pgbouncer"

Redis Scaling

Redis Cluster

For high availability and sharding:

services:
  redis:
    enabled: false  # Disable built-in Redis

config:
  REDIS_URL: "redis://redis-cluster:6379/0"

Deploy Redis Cluster:

helm install redis bitnami/redis-cluster \
  --namespace aurora \
  --set cluster.nodes=6 \
  --set cluster.replicas=1

Redis Sentinel

For failover without sharding:

helm install redis bitnami/redis \
  --namespace aurora \
  --set sentinel.enabled=true \
  --set master.persistence.size=20Gi \
  --set replica.replicaCount=2

Vector Database Scaling

Weaviate Clustering

For production, use Weaviate Cloud or multi-node cluster:

Weaviate Cloud (Recommended):

services:
  weaviate:
    enabled: false

config:
  WEAVIATE_HOST: "aurora-cluster.weaviate.network"
  WEAVIATE_PORT: "443"
  WEAVIATE_SCHEME: "https"

Self-Managed Cluster:

replicaCounts:
  weaviate: 3

weaviate:
  cluster:
    enabled: true
    replicas: 3

Load Balancing

Ingress Session Affinity

For WebSocket connections, enable session affinity:

ingress:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "aurora-ws-affinity"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"

External Load Balancer

For cloud deployments:

AWS ALB:

ingress:
  className: "alb"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health

GCP Load Balancer:

ingress:
  className: "gce"
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: "aurora-ip"

Monitoring Scaling

Key Metrics to Track

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: 'aurora-metrics'
        metrics_path: '/metrics'
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names:
                - aurora
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: aurora-.*

Metrics to monitor:

• Request rate (requests/sec)
• Response time (p50, p95, p99)
• Error rate (%)
• CPU usage (%)
• Memory usage (%)
• Celery queue length
• Database connections
• Redis memory usage

Grafana Dashboard

Import Aurora dashboard:

kubectl create configmap grafana-dashboard-aurora \
  --from-file=aurora-dashboard.json \
  -n monitoring

Performance Optimization

Caching

Enable aggressive caching:

config:
  # Cloud provider API caching
  AURORA_SETUP_CACHE_ENABLED: "true"
  AURORA_SETUP_CACHE_TTL: "7200"  # 2 hours
  
  # Storage caching
  STORAGE_CACHE_ENABLED: "true"
  STORAGE_CACHE_TTL: "300"  # 5 minutes

Cost Optimization

Reduce LLM costs during scale:

config:
  RCA_OPTIMIZE_COSTS: "true"  # Use cheaper models when possible
  AGENT_RECURSION_LIMIT: "120"  # Reduce from 240 for faster completion

Testing Scaling

Load Testing

Use k6 for load testing:

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 10 },   // Ramp up to 10 users
    { duration: '5m', target: 50 },   // Ramp up to 50 users
    { duration: '10m', target: 100 }, // Stay at 100 users
    { duration: '2m', target: 0 },    // Ramp down
  ],
};

export default function () {
  const res = http.get('https://api.aurora.example.com/health');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

Run test:

k6 run load-test.js

Stress Testing

export let options = {
  stages: [
    { duration: '5m', target: 1000 },  // Ramp to 1000 users
    { duration: '10m', target: 1000 }, // Stay at peak
  ],
};

Scaling Checklist

Before scaling to production:

• Resource requests and limits configured
• HPA configured for stateless services
• Database connection pooling enabled
• Redis clustering or Sentinel configured
• Weaviate clustering or cloud instance
• Session affinity enabled for WebSocket
• Monitoring and alerting set up
• Load testing performed
• Cost optimization settings reviewed
• Backup strategy scales with data growth
• Log aggregation handles increased volume
• Network policies allow pod-to-pod communication

Common Scaling Issues

Pod OOMKilled

Increase memory limits:

resources:
  celeryWorker:
    limits:
      memory: "16Gi"  # Increased from 8Gi

Database Connection Exhaustion

Add PgBouncer or increase connection limits:

postgres:
  config:
    max_connections: "500"  # Increased from 100

Redis Memory Issues

Increase Redis memory or add eviction policy:

redis:
  config:
    maxmemory: "2gb"
    maxmemory-policy: "allkeys-lru"

Slow Response Times

Profile application:

# Enable profiling
kubectl exec -it deployment/aurora-oss-server -n aurora -- \
  python -m cProfile -s cumtime main_compute.py

Next Steps

Production Best Practices

Security and reliability for production

Monitoring

Set up comprehensive monitoring

Performance Tuning

Optimize Aurora performance

Troubleshooting

Common scaling issues