# Disaster Recovery — Süper Panel

## What can fail

| Failure | Blast radius | Detection |
|---------|--------------|-----------|
| `panel-postgres` data loss | Audit log, project registry, events history, user/session. Spoke data is **untouched** (lives in spoke DBs). | Coolify health red; `/api/health` 503; login impossible. |
| `panel-redis` data loss | Lose in-flight BullMQ jobs + Redis Streams consumer offset. Worker re-reads from offset 0 on next start; duplicate event persistence is blocked by `Event.streamId` unique constraint. | Worker logs `BUSYGROUP` / reconnect spam. |
| `panel-web` container down | Panel unreachable (login & dashboards). Spokes keep running. | Tailscale curl `https://sp.semih.ai/api/health` fails. |
| `panel-worker` down | Events stop accumulating; cron jobs miss (BullMQ catches up on restart, but window jobs may skip). | `/events` SSE shows no new rows for hours. |
| MinIO down | Backups + audit archive uploads fail. Existing dumps unaffected. | Worker job logs `[panel-backup]` errors. |
| Maestro host down | Everything down. Spokes on other Tailscale hosts unaffected. | Tailscale ping fails. |

## Backups

- **Daily 04:00 UTC** — worker runs `pg_dump -Fc -Z9 DATABASE_URL_PANEL` and uploads to MinIO bucket `panel-backups/panel-postgres/`. Job name `panel-backup` in BullMQ queue `nightly`.
- **Daily 03:30 UTC** — audit entries older than `AUDIT_RETENTION_DAYS` (default 90) shipped to MinIO `panel-audit-archive/audit/` as JSONL, then deleted from DB. Job `audit-archive`.
- **Retention** — MinIO bucket lifecycle is not set yet; backups grow unbounded. TODO: add lifecycle rule `panel-postgres/` keep 30 days, `audit/` keep forever.

## Restore drill

Run on the maestro host. Restores latest dump into a throwaway `panel_drill` database and runs a few SELECTs.

```bash
./scripts/restore-drill.sh
# or pin to a specific dump:
./scripts/restore-drill.sh 2026-05-13T03-00-00
```

Cleanup after the drill:

```bash
sudo docker exec coolify-db psql -U coolify -d postgres -c 'DROP DATABASE panel_drill;'
```

**Cadence:** run monthly. Log the run in `docs/dr-log.md` (date, dump key, sanity-check counts).

## Real restore (panel-postgres lost)

1. Stop the panel-web + panel-worker containers (Coolify UI → stop).
2. Drop and recreate the `panel` database in `coolify-db`:
   ```bash
   sudo docker exec coolify-db psql -U coolify -d postgres \
     -c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
   ```
   *(adjust the actual DB user if it differs — see `DATABASE_URL_PANEL`)*
3. Find the dump key in MinIO and pg_restore:
   ```bash
   sudo docker exec coolify-db pg_restore --clean --if-exists -U panel -d panel /path/to/dump
   ```
4. Start panel-web — entrypoint runs `prisma db push` (idempotent) and `seed` (upsert).
5. Start panel-worker — Redis Streams resume from `>` (only new entries; old events are in the restored Event table).

## What's NOT covered here (yet)

- Spoke DB restores — each spoke owns its DB and its own DR runbook.
- Coolify itself going down — would need to rebuild Coolify from scratch on a new host and re-import the project. Panel can be redeployed from Gitea repo.
- MinIO going down — its backups live on the same host. **Off-host backup is a Phase 5+ concern.**