Files

Semih 3947528fec plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor

New workspace package fusion-plugin-telemetry-watcher that turns a
Grafana Alerting webhook payload into a Fusion incident task in the
triage column. Hooks the dedup/severity/rate-limit primitives that
PostHog/Sentry/Slack sources will reuse in Phase 2.

Pipeline:
  POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
    → bearer-secret check
    → parseGrafanaPayload (one signal per firing alert; resolved alerts
      are dropped — recovery is verified post-deploy by the QA agent)
    → classifySeverity P0/P1/P2/P3 with critical-path keyword
      escalation (payment/auth/billing/subscription)
    → DedupCache 4h fingerprint window — repeat fires log against the
      existing task instead of opening duplicates
    → IncidentRateLimiter 5/h, 20/d — overflow becomes a "telemetry
      storm" mega-task in a future phase
    → taskStore.createTask({ column: "triage", priority })
    → optional auto-assign to Triage Agent

14 unit tests cover severity buckets, critical-path escalation, dedup
windowing/eviction, hourly+daily rate caps, and the Grafana payload
parser (firing vs resolved, label-based domain inference).

Settings expose all thresholds + secret + dedup window + rate limits
through the dashboard plugin settings UI. README documents the deploy
+ register + Grafana contact-point wiring.

2026-05-09 16:55:38 +00:00

4.4 KiB

Raw Blame History

fusion-plugin-telemetry-watcher

Phase-1 watcher: ingests Grafana Alerting webhooks and opens incident tasks scoped for the sase autonomous engineering pipeline.

PostHog polling, Sentry webhook, and Slack feedback channels land in Phase 2 — the dedup/severity/rate-limit primitives are already in place to receive them.

Pipeline

Grafana Alerting webhook
   → POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
   → parseGrafanaPayload()       (signal + fingerprint)
   → classifySeverity()          (P0..P3 with critical-path escalation)
   → DedupCache.hit()            (4h fingerprint window, in-memory)
   → IncidentRateLimiter         (5/hr, 20/day defaults)
   → taskStore.createTask({ column: "triage", priority })
   → optional auto-assign to Triage Agent

Triage Agent reads the task on its next heartbeat, applies the routing rules in .fusion/memory/team-charter.md, and delegates per severity.

Settings

Key	Type	Default	Notes
`grafanaWebhookSecret`	password	—	Bearer token expected in `Authorization: Bearer …`. Empty disables verification.
`triageAgentId`	string	—	Agent to auto-assign incident tasks to. Falls back to unassigned (heartbeat scan picks them up).
`latencyMs`	number	500	P95 floor (ms).
`errorRate`	number	0.05	Error fraction floor.
`funnelDropPp`	number	10	Funnel drop pp (PostHog phase).
`trafficFloorRpm`	number	60	Below this, ratio-based alerts are not trusted.
`dedupWindowMinutes`	number	240	Rolling fingerprint window.
`rateLimitPerHour`	number	5	Hard cap.
`rateLimitPerDay`	number	20	Hard cap.

Severity classification

grafana:
  errorRate ≥ 0.5  AND traffic ≥ floor      → P0
  errorRate ≥ 4× threshold                  → P1
  latency   ≥ 4× threshold                  → P1
  latency   ≥ 2× threshold OR errorRate≥thr → P2
  else                                      → P3

posthog:
  drop ≥ 4× pp threshold → P0
  drop ≥ 2× pp threshold → P1
  drop ≥     pp threshold → P2
  else                    → P3

sentry:    by event count (10/50/200 buckets)
feedback:  default P2

Critical-path keywords (payment, iyzico, eft, auth, sign-in, sign-up, billing, subscription) escalate the bucket by one step.

Deploy + register (Phase 1 smoke)

# 1. Local install (updates pnpm-lock.yaml)
cd /home/s/fusion && pnpm install

# 2. Commit + push (Coolify auto-deploys)
git add plugins/fusion-plugin-telemetry-watcher pnpm-workspace.yaml pnpm-lock.yaml
git commit -m "plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor"
git push origin main

# 3. After deploy lands, register the plugin in the sase project
curl -k -X POST \
  -H "Authorization: Bearer fn_..." -H "Content-Type: application/json" \
  -d '{
    "mode": "register",
    "id": "fusion-plugin-telemetry-watcher",
    "name": "Telemetry Watcher",
    "version": "0.1.0",
    "path": "/app/plugins/fusion-plugin-telemetry-watcher/src/index.ts",
    "enabled": true,
    "settings": {
      "grafanaWebhookSecret": "<GENERATE A SECRET>",
      "latencyMs": 500,
      "rateLimitPerHour": 5
    }
  }' \
  "https://fusion.semih.ai/api/plugins?projectId=proj_155fecc31ef14928"

# 4. Configure the same secret in Grafana contact point and point its webhook URL to:
#    https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook

# 5. Synthetic smoke test:
curl -k -X POST -H 'Authorization: Bearer <secret>' -H 'Content-Type: application/json' \
  https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook \
  -d '{
    "status": "firing",
    "alerts": [{
      "status": "firing",
      "labels": { "alertname": "ApiP95High", "endpoint": "/api/vehicles/decode", "severity": "warning" },
      "annotations": { "summary": "P95 latency rose to 2200ms over 5min" },
      "values": { "B": 2200 },
      "fingerprint": "smoke-test-1"
    }]
  }'

Expected response: { "ok": true, "accepted": 1, "opened": 1, "deduped": 0, "throttled": 0, "taskIds": ["FN-XXX"] }. The task lands in the triage column with priority=high (P1 due to 4× latency threshold).

Tests

cd /home/s/fusion/plugins/fusion-plugin-telemetry-watcher && pnpm test

Covers severity classification (incl. critical-path escalation), dedup window eviction, rate-limit hourly+daily caps, and Grafana payload parsing.

4.4 KiB Raw Blame History Unescape Escape