root/sp

Files

Semih 909cacf7b3 feat(phase4): rate limit + audit archive + panel backup + DR docs

- src/lib/rate-limit.ts (Redis sliding window, fail-open)
- before-hook on /sign-in/email: 5 attempts/min per IP+email
- worker job audit-archive (daily 03:30, JSONL → MinIO, 90d retention)
- worker job panel-backup (daily 04:00, pg_dump -Fc -Z9 → MinIO)
- Dockerfile adds postgresql16-client
- scripts/restore-drill.sh restores latest dump into panel_drill
- docs/disaster-recovery.md + docs/phase4-deferred.md (mTLS + Infisical rationale)

2026-05-13 11:14:36 +00:00

3.2 KiB

Raw Blame History

Disaster Recovery — Süper Panel

What can fail

Failure	Blast radius	Detection
`panel-postgres` data loss	Audit log, project registry, events history, user/session. Spoke data is untouched (lives in spoke DBs).	Coolify health red; `/api/health` 503; login impossible.
`panel-redis` data loss	Lose in-flight BullMQ jobs + Redis Streams consumer offset. Worker re-reads from offset 0 on next start; duplicate event persistence is blocked by `Event.streamId` unique constraint.	Worker logs `BUSYGROUP` / reconnect spam.
`panel-web` container down	Panel unreachable (login & dashboards). Spokes keep running.	Tailscale curl `https://sp.semih.ai/api/health` fails.
`panel-worker` down	Events stop accumulating; cron jobs miss (BullMQ catches up on restart, but window jobs may skip).	`/events` SSE shows no new rows for hours.
MinIO down	Backups + audit archive uploads fail. Existing dumps unaffected.	Worker job logs `[panel-backup]` errors.
Maestro host down	Everything down. Spokes on other Tailscale hosts unaffected.	Tailscale ping fails.

Backups

Daily 04:00 UTC — worker runs pg_dump -Fc -Z9 DATABASE_URL_PANEL and uploads to MinIO bucket panel-backups/panel-postgres/. Job name panel-backup in BullMQ queue nightly.
Daily 03:30 UTC — audit entries older than AUDIT_RETENTION_DAYS (default 90) shipped to MinIO panel-audit-archive/audit/ as JSONL, then deleted from DB. Job audit-archive.
Retention — MinIO bucket lifecycle is not set yet; backups grow unbounded. TODO: add lifecycle rule panel-postgres/ keep 30 days, audit/ keep forever.

Restore drill

Run on the maestro host. Restores latest dump into a throwaway panel_drill database and runs a few SELECTs.

./scripts/restore-drill.sh
# or pin to a specific dump:
./scripts/restore-drill.sh 2026-05-13T03-00-00

Cleanup after the drill:

sudo docker exec coolify-db psql -U coolify -d postgres -c 'DROP DATABASE panel_drill;'

Cadence: run monthly. Log the run in docs/dr-log.md (date, dump key, sanity-check counts).

Real restore (panel-postgres lost)

Stop the panel-web + panel-worker containers (Coolify UI → stop).

Drop and recreate the panel database in coolify-db:

sudo docker exec coolify-db psql -U coolify -d postgres \
  -c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'

(adjust the actual DB user if it differs — see DATABASE_URL_PANEL)

Find the dump key in MinIO and pg_restore:

sudo docker exec coolify-db pg_restore --clean --if-exists -U panel -d panel /path/to/dump

Start panel-web — entrypoint runs prisma db push (idempotent) and seed (upsert).
Start panel-worker — Redis Streams resume from > (only new entries; old events are in the restored Event table).

What's NOT covered here (yet)

Spoke DB restores — each spoke owns its DB and its own DR runbook.
Coolify itself going down — would need to rebuild Coolify from scratch on a new host and re-import the project. Panel can be redeployed from Gitea repo.
MinIO going down — its backups live on the same host. Off-host backup is a Phase 5+ concern.

3.2 KiB Raw Blame History