fusion/docs/diagnostics.md

# Diagnostics

## Insight run sweeper (`[insight-sweeper]`)

The dashboard insight router runs stale-run recovery sweeps for `project_insight_runs` rows stuck in `pending`/`running` without a live controller owner.

- Recovery writes `terminalCause: "orphaned_active_run_recovered"` and lifecycle failure metadata (`failureClass: "non_retryable"`, `retryable: false`).
- Recovery appends both `warning` and `status_changed` events on `project_insight_run_events` with `metadata.recovery = "orphaned_active_run"`.
- `metadata.recoverySource` indicates where recovery occurred: `startup`, `periodic`, `drive_by`, or `manual`.

## Dependency-blocked Todo backlog health (`[dependency-blocked-todo]`)

Self-healing now runs `surface-dependency-blocked-todos` during both startup recovery and periodic maintenance.

- Normal path emits a workflow insight titled `Backlog health: dependency-blocked todos YYYY-MM-DD`.
- Fallback path (insight store unavailable) writes a per-task log entry prefixed with `[dependency-blocked-todo]` against the top blocker task.
- Reporter summary warnings include group count, total blocked Todo count, and top blocker IDs.

Operator interpretation:
- `ageBucket: "fresh"` → expected dependency queueing.
- `ageBucket: "aging"` → review blocker progress.
- `ageBucket: "stale"` → emerging stall; escalate/unblock blocker.

## Process supervisor (`[process-supervisor]`)

The process supervisor logs when it registers a supervised child, starts teardown, expires the grace window, escalates to `SIGKILL`, or observes a natural child exit.

- `spawned pid=<pid> pgid=<pgid|n/a> command=<cmd>` — child registered for parent-death supervision.
- `terminating pid=<pid> pgid=<pgid|n/a> reason=<reason>` — teardown cascade started.
- `grace expired for pid=<pid>; escalating to SIGKILL` — child ignored the grace window.
- `sent SIGKILL to pid=<pid> pgid=<pgid|n/a>` — hard-kill escalation sent.
- `maxLifetime exceeded for pid=<pid> after <ms>ms` — lifetime watchdog fired.
- `child pid=<pid> exited naturally code=<n|null> signal=<n|null>` — child deregistered after exit.

## Self-healing surfacing passes (`[self-healing]`)

- `surface-in-review-stalls`
  - Log prefix: `In-review stall surfaced [`
  - Purpose: reason-driven in-review stall detector (`merge-blocker`, retry exhaustion, no-worktree, transient merge-status orphaning).
- `surface-in-review-stalled`
  - Log prefix: `In-review stalled surfaced [in-review-stalled]: quiet ...`
  - Purpose: time-quiet detector for unpaused in-review tasks beyond `inReviewStalledThresholdMs`.
  - Non-overlap: skipped when reason-driven `In-review stall surfaced [` is fresh, and skipped for paused tasks (owned by stale-paused-review).
- `surface-stale-paused-reviews`
  - Log prefix: `Stale paused review surfaced [stale-paused-review]: paused ...`
  - Purpose: paused in-review backlog-health detector gated by `stalePausedReviewThresholdMs`.

## No-progress churn stuck-task escalation (`[executor]`, `[stuck-detector]`, `[self-healing]`)

Time-based stuck/stalled/stale surfaces now floor activity timestamps using `settings.engineActiveSinceMs` plus `settings.engineActivationGraceMs` (default `300000`). The runtime stamps `engineActiveSinceMs` on startup and each unpause transition so engine pause/downtime does not count as quiet time.

- Trigger shape: one loop classification/compact-and-resume has already fired for the current `execute()` lifecycle, then ignored `fn_task_update` rebuffs accumulate to `ignoredStepUpdateCount >= 25` without intervening progress.
- Executor diagnostic: `[executor] <taskId>: no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — escalating to STUCK_NO_PROGRESS_CHURN`.
- Self-healing diagnostic: `<taskId> no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — marking failed`.
- Audit event: `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason: "no-progress-churn" }`.
- Outcome: task is marked `status: "failed"`, moved to `in-review`, and not requeued; operators should decompose/rescope the task instead of waiting for more automatic stuck-kill retries.

## Stale self-owned active-session cleanup diagnostics (`[executor]`)

FN-5346 adds a same-task stale-binding reconcile marker before worktree removal:

- `[FN-5346] <taskId>: dropped stale self-owned activeSessionRegistry entry before removeWorktree at <worktreePath>`
- Follow-up task log entry: `Cleared stale self-owned active-session entry before remove`

## Reports health stale-classifier diagnostics (`[reports-health]`)

Direct-report stale decisions in `HeartbeatMonitor.buildReportsHealthSection()` now emit a structured log when an agent is marked `**stale**`.

- Log shape: `[reports-health] stale report <agentId> intervalSource=<source> staleThresholdMs=<n> heartbeatAgeMs=<n>`
- `intervalSource` values:
  - `runtimeConfig` — interval came from cached per-agent runtime config
  - `persisted-agent` — cache was missing/sparse; interval came from persisted `getAgent()` row
  - `monitor-default` — no per-agent interval available; monitor default interval used
- `staleThresholdMs` is the computed stale threshold (`max(1.5 × interval, 5m floor)`)
- `heartbeatAgeMs` is the report's current heartbeat age at classification time
- Healthy reports do not emit this diagnostic; only stale decisions do

## Resume instrumentation (FN-5389, Phase 1)

Dashboard Phase 1 resume instrumentation adds observation-only client/server traces for refetch/reconnect attribution. It does not change visibility/pageshow/SSE behavior; FN-5392 consumes this data for fixes.

- Client event shape (`ResumeEvent`): `{ ts, view, trigger, projectId?, gapMs?, replayAttempted, replayFromEventId?, lastEventId?, sseChannel?, reason?, detail? }`.
- Trigger taxonomy: `visibility`, `pageshow`, `sse-error`, `sse-reconnect`, `sse-open`, `remount`, `route-active`, `route-inactive`, `project-context-change`.
- Sources:
  - `sse-bus` (`pageshow`, visible `visibilitychange`, `openChannel`, `forceReconnect`, EventSource `error`)
  - Hooks: `useTasks` (`visibility`, `sse-reconnect`), `useChatRooms` (`sse-reconnect`), `useChat` (`sse-open`, `project-context-change`)
  - Components: `Board` and `ChatView` mount/unmount route markers (`remount` / `route-active` / `route-inactive`)
- Access paths:
  - Client ring (500): `window.__fusionDebug.resumeInstrumentation.get()` / `.clear()`
  - Server ring (5000, in-memory): `GET /api/diagnostics/resume-events?limit=&since=&view=` returns `{ events, droppedSinceLastRead }`
- Client batching: POST `/api/diagnostics/resume-events` in idle batches (`<=25` per POST).
- Disable knob: `window.__fusionDebug.resumeInstrumentation.setEnabled(false)`.

FN-5415 extends this coverage across remaining board/data visibility hooks: `useNodes`, `useMeshState`, `useProjects`, and `useManagedDockerNodes`. Each now emits `trigger: "visibility"` with `reason: "debounced-refresh"` when refresh is taken and `reason: "debounce-skipped"` (including `detail.timeSinceLastRefreshMs`) when suppressed by debounce. This completes board/data-hook resume-correlation coverage needed for FN-5392 Phase 2 remediation analysis.