root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor

Browse Source 
New workspace package fusion-plugin-telemetry-watcher that turns a
Grafana Alerting webhook payload into a Fusion incident task in the
triage column. Hooks the dedup/severity/rate-limit primitives that
PostHog/Sentry/Slack sources will reuse in Phase 2.

Pipeline:
  POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
    → bearer-secret check
    → parseGrafanaPayload (one signal per firing alert; resolved alerts
      are dropped — recovery is verified post-deploy by the QA agent)
    → classifySeverity P0/P1/P2/P3 with critical-path keyword
      escalation (payment/auth/billing/subscription)
    → DedupCache 4h fingerprint window — repeat fires log against the
      existing task instead of opening duplicates
    → IncidentRateLimiter 5/h, 20/d — overflow becomes a "telemetry
      storm" mega-task in a future phase
    → taskStore.createTask({ column: "triage", priority })
    → optional auto-assign to Triage Agent

14 unit tests cover severity buckets, critical-path escalation, dedup
windowing/eviction, hourly+daily rate caps, and the Grafana payload
parser (firing vs resolved, label-based domain inference).

Settings expose all thresholds + secret + dedup window + rate limits
through the dashboard plugin settings UI. README documents the deploy
+ register + Grafana contact-point wiring.
This commit is contained in:
Semih
2026-05-09 16:55:38 +00:00
parent accd19b18d
commit 3947528fec
12 changed files with 1017 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

118
plugins/fusion-plugin-telemetry-watcher/README.md Normal file
View File

@@ -0,0 +1,118 @@
# fusion-plugin-telemetry-watcher
Phase-1 watcher: ingests **Grafana Alerting webhooks** and opens incident tasks
scoped for the sase autonomous engineering pipeline.
PostHog polling, Sentry webhook, and Slack feedback channels land in Phase 2 —
the dedup/severity/rate-limit primitives are already in place to receive them.
## Pipeline
```
Grafana Alerting webhook
→ POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
→ parseGrafanaPayload() (signal + fingerprint)
→ classifySeverity() (P0..P3 with critical-path escalation)
→ DedupCache.hit() (4h fingerprint window, in-memory)
→ IncidentRateLimiter (5/hr, 20/day defaults)
→ taskStore.createTask({ column: "triage", priority })
→ optional auto-assign to Triage Agent
```
Triage Agent reads the task on its next heartbeat, applies the routing rules
in `.fusion/memory/team-charter.md`, and delegates per severity.
## Settings
| Key | Type | Default | Notes |
| --- | --- | --- | --- |
| `grafanaWebhookSecret` | password | — | Bearer token expected in `Authorization: Bearer …`. Empty disables verification. |
| `triageAgentId` | string | — | Agent to auto-assign incident tasks to. Falls back to unassigned (heartbeat scan picks them up). |
| `latencyMs` | number | 500 | P95 floor (ms). |
| `errorRate` | number | 0.05 | Error fraction floor. |
| `funnelDropPp` | number | 10 | Funnel drop pp (PostHog phase). |
| `trafficFloorRpm` | number | 60 | Below this, ratio-based alerts are not trusted. |
| `dedupWindowMinutes` | number | 240 | Rolling fingerprint window. |
| `rateLimitPerHour` | number | 5 | Hard cap. |
| `rateLimitPerDay` | number | 20 | Hard cap. |
## Severity classification
```
grafana:
errorRate ≥ 0.5 AND traffic ≥ floor → P0
errorRate ≥ 4× threshold → P1
latency ≥ 4× threshold → P1
latency ≥ 2× threshold OR errorRate≥thr → P2
else → P3
posthog:
drop ≥ 4× pp threshold → P0
drop ≥ 2× pp threshold → P1
drop ≥ pp threshold → P2
else → P3
sentry: by event count (10/50/200 buckets)
feedback: default P2
```
Critical-path keywords (`payment`, `iyzico`, `eft`, `auth`, `sign-in`, `sign-up`,
`billing`, `subscription`) escalate the bucket by one step.
## Deploy + register (Phase 1 smoke)
```bash
# 1. Local install (updates pnpm-lock.yaml)
cd /home/s/fusion && pnpm install
# 2. Commit + push (Coolify auto-deploys)
git add plugins/fusion-plugin-telemetry-watcher pnpm-workspace.yaml pnpm-lock.yaml
git commit -m "plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor"
git push origin main
# 3. After deploy lands, register the plugin in the sase project
curl -k -X POST \
-H "Authorization: Bearer fn_..." -H "Content-Type: application/json" \
-d '{
"mode": "register",
"id": "fusion-plugin-telemetry-watcher",
"name": "Telemetry Watcher",
"version": "0.1.0",
"path": "/app/plugins/fusion-plugin-telemetry-watcher/src/index.ts",
"enabled": true,
"settings": {
"grafanaWebhookSecret": "<GENERATE A SECRET>",
"latencyMs": 500,
"rateLimitPerHour": 5
}
}' \
"https://fusion.semih.ai/api/plugins?projectId=proj_155fecc31ef14928"
# 4. Configure the same secret in Grafana contact point and point its webhook URL to:
# https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
# 5. Synthetic smoke test:
curl -k -X POST -H 'Authorization: Bearer <secret>' -H 'Content-Type: application/json' \
https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook \
-d '{
"status": "firing",
"alerts": [{
"status": "firing",
"labels": { "alertname": "ApiP95High", "endpoint": "/api/vehicles/decode", "severity": "warning" },
"annotations": { "summary": "P95 latency rose to 2200ms over 5min" },
"values": { "B": 2200 },
"fingerprint": "smoke-test-1"
}]
}'
```
Expected response: `{ "ok": true, "accepted": 1, "opened": 1, "deduped": 0, "throttled": 0, "taskIds": ["FN-XXX"] }`. The task lands in the `triage` column with priority=`high` (P1 due to 4× latency threshold).
## Tests
```bash
cd /home/s/fusion/plugins/fusion-plugin-telemetry-watcher && pnpm test
```
Covers severity classification (incl. critical-path escalation), dedup window
eviction, rate-limit hourly+daily caps, and Grafana payload parsing.