Adds the active half of Phase 8: dashboard was passive, this pushes
when something breaks. Targets MTTD < 10 minutes from the PRD.
Detection (panel-web)
- detectVinAnomalies() compares a 15-minute current window against a
baseline built from the same 15-minute slot across the previous 7
days (same-hour-of-day, average across days that had ≥ 1 sample).
- Five hits, two severities:
* success_rate_drop — high at ≥ 5pp drop, critical at ≥ 15pp
* p95_latency_spike — high at 2× baseline (and > 1s), critical at 3×
* volume_drop — high at ≥ 80% below baseline, critical at ≥ 95%
* volume_spike — high at ≥ 5× baseline
* timeout_dominance — high when ≥ 50% of failures are timeouts
- Minimum-volume guards on both current and baseline so quiet hours
don't generate noise (MIN_CURRENT_VOLUME=5, MIN_BASELINE_VOLUME=10).
- Each anomaly carries a 15-min-bucket dedupe key — same anomaly type
fires at most once per bucket regardless of cron cadence.
Endpoint
- GET /api/internal/vin-anomaly-check, gated by x-internal-worker-token
header (constant-time compare against INTERNAL_WORKER_TOKEN env).
Returns { current, baseline, anomalies[] }.
Worker
- New job vin-anomaly-detect, BullMQ scheduler */5 * * * *.
Fetches the panel endpoint, then for each anomaly calls
alertVinAnomaly() — sendTelegram with the bucket dedupe key, so
Redis SETEX NX dedupes across the 1h cooldown window.
- alertVinAnomaly() in worker/lib/telegram.ts formats the message with
severity icon + baseline/observed/volume context + dashboard link.
- runVinAnomalyDetect returns { ok, checked, anomalies, alertsFired,
alertsDeduped }; pipeline logs only when something happened or the
check failed.
Env
- INTERNAL_WORKER_TOKEN set on both panel-web and panel-worker
(32-byte hex, generated in Coolify).
- PANEL_INTERNAL_URL on panel-worker → coolify-network UUID hostname
for panel-web, no Cloudflare/Tailscale hop on internal calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4694c1ffdb