feat(sase): VIN decode anomaly detection + Telegram alerts
Adds the active half of Phase 8: dashboard was passive, this pushes
when something breaks. Targets MTTD < 10 minutes from the PRD.
Detection (panel-web)
- detectVinAnomalies() compares a 15-minute current window against a
baseline built from the same 15-minute slot across the previous 7
days (same-hour-of-day, average across days that had ≥ 1 sample).
- Five hits, two severities:
* success_rate_drop — high at ≥ 5pp drop, critical at ≥ 15pp
* p95_latency_spike — high at 2× baseline (and > 1s), critical at 3×
* volume_drop — high at ≥ 80% below baseline, critical at ≥ 95%
* volume_spike — high at ≥ 5× baseline
* timeout_dominance — high when ≥ 50% of failures are timeouts
- Minimum-volume guards on both current and baseline so quiet hours
don't generate noise (MIN_CURRENT_VOLUME=5, MIN_BASELINE_VOLUME=10).
- Each anomaly carries a 15-min-bucket dedupe key — same anomaly type
fires at most once per bucket regardless of cron cadence.
Endpoint
- GET /api/internal/vin-anomaly-check, gated by x-internal-worker-token
header (constant-time compare against INTERNAL_WORKER_TOKEN env).
Returns { current, baseline, anomalies[] }.
Worker
- New job vin-anomaly-detect, BullMQ scheduler */5 * * * *.
Fetches the panel endpoint, then for each anomaly calls
alertVinAnomaly() — sendTelegram with the bucket dedupe key, so
Redis SETEX NX dedupes across the 1h cooldown window.
- alertVinAnomaly() in worker/lib/telegram.ts formats the message with
severity icon + baseline/observed/volume context + dashboard link.
- runVinAnomalyDetect returns { ok, checked, anomalies, alertsFired,
alertsDeduped }; pipeline logs only when something happened or the
check failed.
Env
- INTERNAL_WORKER_TOKEN set on both panel-web and panel-worker
(32-byte hex, generated in Coolify).
- PANEL_INTERNAL_URL on panel-worker → coolify-network UUID hostname
for panel-web, no Cloudflare/Tailscale hop on internal calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
25
apps/web/src/app/api/internal/vin-anomaly-check/route.ts
Normal file
25
apps/web/src/app/api/internal/vin-anomaly-check/route.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
import { NextResponse } from "next/server";
|
||||
import { timingSafeEqual } from "node:crypto";
|
||||
import { detectVinAnomalies } from "@/lib/sase/vin-anomaly";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
const INTERNAL_TOKEN = process.env.INTERNAL_WORKER_TOKEN ?? "";
|
||||
|
||||
function validToken(req: Request): boolean {
|
||||
if (!INTERNAL_TOKEN) return false;
|
||||
const provided = req.headers.get("x-internal-worker-token") ?? "";
|
||||
if (!provided) return false;
|
||||
const a = Buffer.from(provided);
|
||||
const b = Buffer.from(INTERNAL_TOKEN);
|
||||
if (a.length !== b.length) return false;
|
||||
return timingSafeEqual(a, b);
|
||||
}
|
||||
|
||||
export async function GET(req: Request) {
|
||||
if (!validToken(req)) {
|
||||
return NextResponse.json({ ok: false, error: "unauthorized" }, { status: 401 });
|
||||
}
|
||||
const result = await detectVinAnomalies();
|
||||
return NextResponse.json({ ok: true, ...result });
|
||||
}
|
||||
Reference in New Issue
Block a user