feat(phase4): rate limit + audit archive + panel backup + DR docs
- src/lib/rate-limit.ts (Redis sliding window, fail-open) - before-hook on /sign-in/email: 5 attempts/min per IP+email - worker job audit-archive (daily 03:30, JSONL → MinIO, 90d retention) - worker job panel-backup (daily 04:00, pg_dump -Fc -Z9 → MinIO) - Dockerfile adds postgresql16-client - scripts/restore-drill.sh restores latest dump into panel_drill - docs/disaster-recovery.md + docs/phase4-deferred.md (mTLS + Infisical rationale)
This commit is contained in:
Semih
58
docs/disaster-recovery.md
Normal file
58
docs/disaster-recovery.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Disaster Recovery — Süper Panel
|
||||
|
||||
## What can fail
|
||||
|
||||
| Failure | Blast radius | Detection |
|
||||
|---------|--------------|-----------|
|
||||
| `panel-postgres` data loss | Audit log, project registry, events history, user/session. Spoke data is **untouched** (lives in spoke DBs). | Coolify health red; `/api/health` 503; login impossible. |
|
||||
| `panel-redis` data loss | Lose in-flight BullMQ jobs + Redis Streams consumer offset. Worker re-reads from offset 0 on next start; duplicate event persistence is blocked by `Event.streamId` unique constraint. | Worker logs `BUSYGROUP` / reconnect spam. |
|
||||
| `panel-web` container down | Panel unreachable (login & dashboards). Spokes keep running. | Tailscale curl `https://sp.semih.ai/api/health` fails. |
|
||||
| `panel-worker` down | Events stop accumulating; cron jobs miss (BullMQ catches up on restart, but window jobs may skip). | `/events` SSE shows no new rows for hours. |
|
||||
| MinIO down | Backups + audit archive uploads fail. Existing dumps unaffected. | Worker job logs `[panel-backup]` errors. |
|
||||
| Maestro host down | Everything down. Spokes on other Tailscale hosts unaffected. | Tailscale ping fails. |
|
||||
|
||||
## Backups
|
||||
|
||||
- **Daily 04:00 UTC** — worker runs `pg_dump -Fc -Z9 DATABASE_URL_PANEL` and uploads to MinIO bucket `panel-backups/panel-postgres/`. Job name `panel-backup` in BullMQ queue `nightly`.
|
||||
- **Daily 03:30 UTC** — audit entries older than `AUDIT_RETENTION_DAYS` (default 90) shipped to MinIO `panel-audit-archive/audit/` as JSONL, then deleted from DB. Job `audit-archive`.
|
||||
- **Retention** — MinIO bucket lifecycle is not set yet; backups grow unbounded. TODO: add lifecycle rule `panel-postgres/` keep 30 days, `audit/` keep forever.
|
||||
|
||||
## Restore drill
|
||||
|
||||
Run on the maestro host. Restores latest dump into a throwaway `panel_drill` database and runs a few SELECTs.
|
||||
|
||||
```bash
|
||||
./scripts/restore-drill.sh
|
||||
# or pin to a specific dump:
|
||||
./scripts/restore-drill.sh 2026-05-13T03-00-00
|
||||
```
|
||||
|
||||
Cleanup after the drill:
|
||||
|
||||
```bash
|
||||
sudo docker exec coolify-db psql -U coolify -d postgres -c 'DROP DATABASE panel_drill;'
|
||||
```
|
||||
|
||||
**Cadence:** run monthly. Log the run in `docs/dr-log.md` (date, dump key, sanity-check counts).
|
||||
|
||||
## Real restore (panel-postgres lost)
|
||||
|
||||
1. Stop the panel-web + panel-worker containers (Coolify UI → stop).
|
||||
2. Drop and recreate the `panel` database in `coolify-db`:
|
||||
```bash
|
||||
sudo docker exec coolify-db psql -U coolify -d postgres \
|
||||
-c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
|
||||
```
|
||||
*(adjust the actual DB user if it differs — see `DATABASE_URL_PANEL`)*
|
||||
3. Find the dump key in MinIO and pg_restore:
|
||||
```bash
|
||||
sudo docker exec coolify-db pg_restore --clean --if-exists -U panel -d panel /path/to/dump
|
||||
```
|
||||
4. Start panel-web — entrypoint runs `prisma db push` (idempotent) and `seed` (upsert).
|
||||
5. Start panel-worker — Redis Streams resume from `>` (only new entries; old events are in the restored Event table).
|
||||
|
||||
## What's NOT covered here (yet)
|
||||
|
||||
- Spoke DB restores — each spoke owns its DB and its own DR runbook.
|
||||
- Coolify itself going down — would need to rebuild Coolify from scratch on a new host and re-import the project. Panel can be redeployed from Gitea repo.
|
||||
- MinIO going down — its backups live on the same host. **Off-host backup is a Phase 5+ concern.**
|
||||
45
docs/phase4-deferred.md
Normal file
45
docs/phase4-deferred.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# Phase 4 deferred: mTLS + Infisical
|
||||
|
||||
The PRD lists mTLS (Layer 2 admin API hardening) and Infisical (secret manager) as Phase 4 work. Both are deliberately deferred. Reasons + trigger criteria below.
|
||||
|
||||
## mTLS for `/internal/admin/*`
|
||||
|
||||
**Why deferred:** no spoke currently exposes `/internal/admin/*`. Layer 2 is wire-ready (`AdminClient` with `X-Internal-Token`) but unused. Standing up a CA, distributing client certs, and adding mTLS verification on every spoke before any spoke actually has admin endpoints is premature engineering — we'd ship infrastructure for traffic that doesn't exist.
|
||||
|
||||
**Threat model today:** spokes have no admin endpoints. There is no Layer 2 traffic to intercept. Tailscale already gates network access to panel-web; the panel cannot be reached from the public internet at all.
|
||||
|
||||
**Trigger to revisit:**
|
||||
- Second spoke ships `/internal/admin/*` (i.e. we have 2+ spokes calling admin endpoints).
|
||||
- OR token rotation becomes a chore (manual env updates start drifting).
|
||||
- OR a regulatory requirement appears (KVKK audit, customer SOC2 ask, etc.).
|
||||
|
||||
**When triggered, do:**
|
||||
1. Generate a self-signed CA (long-lived) and per-spoke client certs (1-year, rotated by cron).
|
||||
2. Add `caCert` / `clientCert` / `clientKey` options to `AdminClient` (already structured for it).
|
||||
3. Spoke-side: configure their gateway (Traefik/Caddy/Nginx) to require client cert on `/internal/admin/*`.
|
||||
4. Keep `X-Internal-Token` as defense-in-depth (mTLS authenticates client *machine*, token authenticates *application*).
|
||||
|
||||
## Infisical self-hosted
|
||||
|
||||
**Why deferred:** no Infisical instance running in this Coolify. Adding it requires (a) deploying Infisical, (b) creating projects + service tokens, (c) installing the agent or SDK in panel-web + panel-worker, (d) migrating ~12 env vars. That's a weekend of work for marginal benefit at this scale — Coolify already encrypts env at rest and audits writes.
|
||||
|
||||
**Trigger to revisit:**
|
||||
- Need to share secrets across multiple machines (right now everything's on `maestro`).
|
||||
- Need a secret-rotation cadence (e.g. quarterly rotation of `BETTER_AUTH_SECRET`).
|
||||
- Need versioned, time-windowed secrets (preview environments with their own credentials).
|
||||
- Need an audit trail of *who looked at* a secret (Coolify only audits writes).
|
||||
|
||||
**When triggered, do:**
|
||||
1. Deploy Infisical via Coolify (1 web + 1 db).
|
||||
2. Create project `super-panel` with envs `production` and `staging`.
|
||||
3. Install `@infisical/sdk` in both `apps/web` and `apps/worker`, swap `process.env.X` reads through the SDK with cache.
|
||||
4. Remove the corresponding Coolify env vars (one at a time; verify on each).
|
||||
5. Add CI gate: fail PRs that introduce raw `process.env.<SECRET>` access for known-sensitive keys.
|
||||
|
||||
## What we *do* have today (Phase 4 actually delivered)
|
||||
|
||||
- Rate limit on `/sign-in/email` (5/min per IP+email, Redis-backed, fail-open).
|
||||
- Daily audit archive to MinIO (90-day retention).
|
||||
- Daily `pg_dump` of panel-postgres to MinIO (`panel-backups/`).
|
||||
- Restore drill script (`scripts/restore-drill.sh`) — monthly cadence target.
|
||||
- DR runbook (`docs/disaster-recovery.md`).
|
||||
Reference in New Issue
Block a user