909cacf7b3
- src/lib/rate-limit.ts (Redis sliding window, fail-open) - before-hook on /sign-in/email: 5 attempts/min per IP+email - worker job audit-archive (daily 03:30, JSONL → MinIO, 90d retention) - worker job panel-backup (daily 04:00, pg_dump -Fc -Z9 → MinIO) - Dockerfile adds postgresql16-client - scripts/restore-drill.sh restores latest dump into panel_drill - docs/disaster-recovery.md + docs/phase4-deferred.md (mTLS + Infisical rationale)
59 lines
3.2 KiB
Markdown
59 lines
3.2 KiB
Markdown
# Disaster Recovery — Süper Panel
|
|
|
|
## What can fail
|
|
|
|
| Failure | Blast radius | Detection |
|
|
|---------|--------------|-----------|
|
|
| `panel-postgres` data loss | Audit log, project registry, events history, user/session. Spoke data is **untouched** (lives in spoke DBs). | Coolify health red; `/api/health` 503; login impossible. |
|
|
| `panel-redis` data loss | Lose in-flight BullMQ jobs + Redis Streams consumer offset. Worker re-reads from offset 0 on next start; duplicate event persistence is blocked by `Event.streamId` unique constraint. | Worker logs `BUSYGROUP` / reconnect spam. |
|
|
| `panel-web` container down | Panel unreachable (login & dashboards). Spokes keep running. | Tailscale curl `https://sp.semih.ai/api/health` fails. |
|
|
| `panel-worker` down | Events stop accumulating; cron jobs miss (BullMQ catches up on restart, but window jobs may skip). | `/events` SSE shows no new rows for hours. |
|
|
| MinIO down | Backups + audit archive uploads fail. Existing dumps unaffected. | Worker job logs `[panel-backup]` errors. |
|
|
| Maestro host down | Everything down. Spokes on other Tailscale hosts unaffected. | Tailscale ping fails. |
|
|
|
|
## Backups
|
|
|
|
- **Daily 04:00 UTC** — worker runs `pg_dump -Fc -Z9 DATABASE_URL_PANEL` and uploads to MinIO bucket `panel-backups/panel-postgres/`. Job name `panel-backup` in BullMQ queue `nightly`.
|
|
- **Daily 03:30 UTC** — audit entries older than `AUDIT_RETENTION_DAYS` (default 90) shipped to MinIO `panel-audit-archive/audit/` as JSONL, then deleted from DB. Job `audit-archive`.
|
|
- **Retention** — MinIO bucket lifecycle is not set yet; backups grow unbounded. TODO: add lifecycle rule `panel-postgres/` keep 30 days, `audit/` keep forever.
|
|
|
|
## Restore drill
|
|
|
|
Run on the maestro host. Restores latest dump into a throwaway `panel_drill` database and runs a few SELECTs.
|
|
|
|
```bash
|
|
./scripts/restore-drill.sh
|
|
# or pin to a specific dump:
|
|
./scripts/restore-drill.sh 2026-05-13T03-00-00
|
|
```
|
|
|
|
Cleanup after the drill:
|
|
|
|
```bash
|
|
sudo docker exec coolify-db psql -U coolify -d postgres -c 'DROP DATABASE panel_drill;'
|
|
```
|
|
|
|
**Cadence:** run monthly. Log the run in `docs/dr-log.md` (date, dump key, sanity-check counts).
|
|
|
|
## Real restore (panel-postgres lost)
|
|
|
|
1. Stop the panel-web + panel-worker containers (Coolify UI → stop).
|
|
2. Drop and recreate the `panel` database in `coolify-db`:
|
|
```bash
|
|
sudo docker exec coolify-db psql -U coolify -d postgres \
|
|
-c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
|
|
```
|
|
*(adjust the actual DB user if it differs — see `DATABASE_URL_PANEL`)*
|
|
3. Find the dump key in MinIO and pg_restore:
|
|
```bash
|
|
sudo docker exec coolify-db pg_restore --clean --if-exists -U panel -d panel /path/to/dump
|
|
```
|
|
4. Start panel-web — entrypoint runs `prisma db push` (idempotent) and `seed` (upsert).
|
|
5. Start panel-worker — Redis Streams resume from `>` (only new entries; old events are in the restored Event table).
|
|
|
|
## What's NOT covered here (yet)
|
|
|
|
- Spoke DB restores — each spoke owns its DB and its own DR runbook.
|
|
- Coolify itself going down — would need to rebuild Coolify from scratch on a new host and re-import the project. Panel can be redeployed from Gitea repo.
|
|
- MinIO going down — its backups live on the same host. **Off-host backup is a Phase 5+ concern.**
|