3947528fec
New workspace package fusion-plugin-telemetry-watcher that turns a
Grafana Alerting webhook payload into a Fusion incident task in the
triage column. Hooks the dedup/severity/rate-limit primitives that
PostHog/Sentry/Slack sources will reuse in Phase 2.
Pipeline:
POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
→ bearer-secret check
→ parseGrafanaPayload (one signal per firing alert; resolved alerts
are dropped — recovery is verified post-deploy by the QA agent)
→ classifySeverity P0/P1/P2/P3 with critical-path keyword
escalation (payment/auth/billing/subscription)
→ DedupCache 4h fingerprint window — repeat fires log against the
existing task instead of opening duplicates
→ IncidentRateLimiter 5/h, 20/d — overflow becomes a "telemetry
storm" mega-task in a future phase
→ taskStore.createTask({ column: "triage", priority })
→ optional auto-assign to Triage Agent
14 unit tests cover severity buckets, critical-path escalation, dedup
windowing/eviction, hourly+daily rate caps, and the Grafana payload
parser (firing vs resolved, label-based domain inference).
Settings expose all thresholds + secret + dedup window + rate limits
through the dashboard plugin settings UI. README documents the deploy
+ register + Grafana contact-point wiring.
119 lines
4.4 KiB
Markdown
119 lines
4.4 KiB
Markdown
# fusion-plugin-telemetry-watcher
|
||
|
||
Phase-1 watcher: ingests **Grafana Alerting webhooks** and opens incident tasks
|
||
scoped for the sase autonomous engineering pipeline.
|
||
|
||
PostHog polling, Sentry webhook, and Slack feedback channels land in Phase 2 —
|
||
the dedup/severity/rate-limit primitives are already in place to receive them.
|
||
|
||
## Pipeline
|
||
|
||
```
|
||
Grafana Alerting webhook
|
||
→ POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
|
||
→ parseGrafanaPayload() (signal + fingerprint)
|
||
→ classifySeverity() (P0..P3 with critical-path escalation)
|
||
→ DedupCache.hit() (4h fingerprint window, in-memory)
|
||
→ IncidentRateLimiter (5/hr, 20/day defaults)
|
||
→ taskStore.createTask({ column: "triage", priority })
|
||
→ optional auto-assign to Triage Agent
|
||
```
|
||
|
||
Triage Agent reads the task on its next heartbeat, applies the routing rules
|
||
in `.fusion/memory/team-charter.md`, and delegates per severity.
|
||
|
||
## Settings
|
||
|
||
| Key | Type | Default | Notes |
|
||
| --- | --- | --- | --- |
|
||
| `grafanaWebhookSecret` | password | — | Bearer token expected in `Authorization: Bearer …`. Empty disables verification. |
|
||
| `triageAgentId` | string | — | Agent to auto-assign incident tasks to. Falls back to unassigned (heartbeat scan picks them up). |
|
||
| `latencyMs` | number | 500 | P95 floor (ms). |
|
||
| `errorRate` | number | 0.05 | Error fraction floor. |
|
||
| `funnelDropPp` | number | 10 | Funnel drop pp (PostHog phase). |
|
||
| `trafficFloorRpm` | number | 60 | Below this, ratio-based alerts are not trusted. |
|
||
| `dedupWindowMinutes` | number | 240 | Rolling fingerprint window. |
|
||
| `rateLimitPerHour` | number | 5 | Hard cap. |
|
||
| `rateLimitPerDay` | number | 20 | Hard cap. |
|
||
|
||
## Severity classification
|
||
|
||
```
|
||
grafana:
|
||
errorRate ≥ 0.5 AND traffic ≥ floor → P0
|
||
errorRate ≥ 4× threshold → P1
|
||
latency ≥ 4× threshold → P1
|
||
latency ≥ 2× threshold OR errorRate≥thr → P2
|
||
else → P3
|
||
|
||
posthog:
|
||
drop ≥ 4× pp threshold → P0
|
||
drop ≥ 2× pp threshold → P1
|
||
drop ≥ pp threshold → P2
|
||
else → P3
|
||
|
||
sentry: by event count (10/50/200 buckets)
|
||
feedback: default P2
|
||
```
|
||
|
||
Critical-path keywords (`payment`, `iyzico`, `eft`, `auth`, `sign-in`, `sign-up`,
|
||
`billing`, `subscription`) escalate the bucket by one step.
|
||
|
||
## Deploy + register (Phase 1 smoke)
|
||
|
||
```bash
|
||
# 1. Local install (updates pnpm-lock.yaml)
|
||
cd /home/s/fusion && pnpm install
|
||
|
||
# 2. Commit + push (Coolify auto-deploys)
|
||
git add plugins/fusion-plugin-telemetry-watcher pnpm-workspace.yaml pnpm-lock.yaml
|
||
git commit -m "plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor"
|
||
git push origin main
|
||
|
||
# 3. After deploy lands, register the plugin in the sase project
|
||
curl -k -X POST \
|
||
-H "Authorization: Bearer fn_..." -H "Content-Type: application/json" \
|
||
-d '{
|
||
"mode": "register",
|
||
"id": "fusion-plugin-telemetry-watcher",
|
||
"name": "Telemetry Watcher",
|
||
"version": "0.1.0",
|
||
"path": "/app/plugins/fusion-plugin-telemetry-watcher/src/index.ts",
|
||
"enabled": true,
|
||
"settings": {
|
||
"grafanaWebhookSecret": "<GENERATE A SECRET>",
|
||
"latencyMs": 500,
|
||
"rateLimitPerHour": 5
|
||
}
|
||
}' \
|
||
"https://fusion.semih.ai/api/plugins?projectId=proj_155fecc31ef14928"
|
||
|
||
# 4. Configure the same secret in Grafana contact point and point its webhook URL to:
|
||
# https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
|
||
|
||
# 5. Synthetic smoke test:
|
||
curl -k -X POST -H 'Authorization: Bearer <secret>' -H 'Content-Type: application/json' \
|
||
https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook \
|
||
-d '{
|
||
"status": "firing",
|
||
"alerts": [{
|
||
"status": "firing",
|
||
"labels": { "alertname": "ApiP95High", "endpoint": "/api/vehicles/decode", "severity": "warning" },
|
||
"annotations": { "summary": "P95 latency rose to 2200ms over 5min" },
|
||
"values": { "B": 2200 },
|
||
"fingerprint": "smoke-test-1"
|
||
}]
|
||
}'
|
||
```
|
||
|
||
Expected response: `{ "ok": true, "accepted": 1, "opened": 1, "deduped": 0, "throttled": 0, "taskIds": ["FN-XXX"] }`. The task lands in the `triage` column with priority=`high` (P1 due to 4× latency threshold).
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
cd /home/s/fusion/plugins/fusion-plugin-telemetry-watcher && pnpm test
|
||
```
|
||
|
||
Covers severity classification (incl. critical-path escalation), dedup window
|
||
eviction, rate-limit hourly+daily caps, and Grafana payload parsing.
|