fusion/plugins/fusion-plugin-telemetry-watcher

fusion-plugin-telemetry-watcher

Phase-1 watcher: ingests Grafana Alerting webhooks and opens incident tasks scoped for the sase autonomous engineering pipeline.

PostHog polling, Sentry webhook, and Slack feedback channels land in Phase 2 — the dedup/severity/rate-limit primitives are already in place to receive them.

Pipeline

Grafana Alerting webhook
   → POST /api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook
   → parseGrafanaPayload()       (signal + fingerprint)
   → classifySeverity()          (P0..P3 with critical-path escalation)
   → DedupCache.hit()            (4h fingerprint window, in-memory)
   → IncidentRateLimiter         (5/hr, 20/day defaults)
   → taskStore.createTask({ column: "triage", priority })
   → optional auto-assign to Triage Agent

Triage Agent reads the task on its next heartbeat, applies the routing rules in .fusion/memory/team-charter.md, and delegates per severity.

Settings

Key	Type	Default	Notes
grafanaWebhookSecret	password	—	Bearer token expected in Authorization: Bearer …. Empty disables verification.
triageAgentId	string	—	Agent to auto-assign incident tasks to. Falls back to unassigned (heartbeat scan picks them up).
latencyMs	number	500	P95 floor (ms).
errorRate	number	0.05	Error fraction floor.
funnelDropPp	number	10	Funnel drop pp (PostHog phase).
trafficFloorRpm	number	60	Below this, ratio-based alerts are not trusted.
dedupWindowMinutes	number	240	Rolling fingerprint window.
rateLimitPerHour	number	5	Hard cap.
rateLimitPerDay	number	20	Hard cap.

Severity classification

grafana:
  errorRate ≥ 0.5  AND traffic ≥ floor      → P0
  errorRate ≥ 4× threshold                  → P1
  latency   ≥ 4× threshold                  → P1
  latency   ≥ 2× threshold OR errorRate≥thr → P2
  else                                      → P3

posthog:
  drop ≥ 4× pp threshold → P0
  drop ≥ 2× pp threshold → P1
  drop ≥     pp threshold → P2
  else                    → P3

sentry:    by event count (10/50/200 buckets)
feedback:  default P2

Critical-path keywords (payment, iyzico, eft, auth, sign-in, sign-up, billing, subscription) escalate the bucket by one step.

Deploy + register (Phase 1 smoke)

# 1. Local install (updates pnpm-lock.yaml)
cd /home/s/fusion && pnpm install

# 2. Commit + push (Coolify auto-deploys)
git add plugins/fusion-plugin-telemetry-watcher pnpm-workspace.yaml pnpm-lock.yaml
git commit -m "plugin(telemetry-watcher): Phase 1 — Grafana webhook ingestor"
git push origin main

# 3. After deploy lands, register the plugin in the sase project
curl -k -X POST \
  -H "Authorization: Bearer fn_..." -H "Content-Type: application/json" \
  -d '{
    "mode": "register",
    "id": "fusion-plugin-telemetry-watcher",
    "name": "Telemetry Watcher",
    "version": "0.1.0",
    "path": "/app/plugins/fusion-plugin-telemetry-watcher/src/index.ts",
    "enabled": true,
    "settings": {
      "grafanaWebhookSecret": "<GENERATE A SECRET>",
      "latencyMs": 500,
      "rateLimitPerHour": 5
    }
  }' \
  "https://fusion.semih.ai/api/plugins?projectId=proj_155fecc31ef14928"

# 4. Configure the same secret in Grafana contact point and point its webhook URL to:
#    https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook

# 5. Synthetic smoke test:
curl -k -X POST -H 'Authorization: Bearer <secret>' -H 'Content-Type: application/json' \
  https://fusion.semih.ai/api/plugins/fusion-plugin-telemetry-watcher/grafana-webhook \
  -d '{
    "status": "firing",
    "alerts": [{
      "status": "firing",
      "labels": { "alertname": "ApiP95High", "endpoint": "/api/vehicles/decode", "severity": "warning" },
      "annotations": { "summary": "P95 latency rose to 2200ms over 5min" },
      "values": { "B": 2200 },
      "fingerprint": "smoke-test-1"
    }]
  }'

Expected response: { "ok": true, "accepted": 1, "opened": 1, "deduped": 0, "throttled": 0, "taskIds": ["FN-XXX"] }. The task lands in the triage column with priority=high (P1 due to 4× latency threshold).

Tests

cd /home/s/fusion/plugins/fusion-plugin-telemetry-watcher && pnpm test

Covers severity classification (incl. critical-path escalation), dedup window eviction, rate-limit hourly+daily caps, and Grafana payload parsing.