# Diagnostics

## Goal injection diagnostics (`[goal-injection]`)

Executor, heartbeat, and planning runs emit one goal-injection diagnostic with outcome `applied`, `no-goals`, or `disabled-or-failed`.

- Run-audit event: `prompt:goal-injection` (`database` domain, target lane) with metadata `{ lane, outcome, goalCount, goalIds, truncated, reason?, errorClass?, runId?, agentId?, taskId? }`.
- Goal anchoring events also persist `metadata.goalIds` (alongside existing count/tool fields):
  - `goal:injection-applied` / `goal:injection-skipped` → `{ lane, count, goalIds, truncated?, reason? }`
  - `goal:retrieval-invoked` → `{ toolName, count, goalIds, notFound }`
- Run cited-goals read path: `GET /api/agents/:id/runs/:runId/cited-goals` returns `{ runId, taskId?, injectedGoalIds, retrievedGoalIds, citedGoalIds }` aggregated from `goal:*` + `prompt:goal-injection` run-audit events.
- Task log (executor lane with `taskId`): `[goal-injection] <outcome> count=<n> ids=<json-array> truncated=<bool> ...`.
- Guardrail: diagnostics persist goal IDs/counts only; never prompt text, goal titles, or goal descriptions.

## Insight run sweeper (`[insight-sweeper]`)

The dashboard insight router runs stale-run recovery sweeps for `project_insight_runs` rows stuck in `pending`/`running` without a live controller owner.

- Recovery writes `terminalCause: "orphaned_active_run_recovered"` and lifecycle failure metadata (`failureClass: "non_retryable"`, `retryable: false`).
- Recovery appends both `warning` and `status_changed` events on `project_insight_run_events` with `metadata.recovery = "orphaned_active_run"`.
- `metadata.recoverySource` indicates where recovery occurred: `startup`, `periodic`, `drive_by`, or `manual`.

## Dependency-blocked Todo backlog health (`[dependency-blocked-todo]`)

Self-healing now runs `surface-dependency-blocked-todos` during both startup recovery and periodic maintenance.

- Normal path emits a workflow insight titled `Backlog health: dependency-blocked todos YYYY-MM-DD`.
- Fallback path (insight store unavailable) writes a per-task log entry prefixed with `[dependency-blocked-todo]` against the top blocker task.
- Reporter summary warnings include group count, total blocked Todo count, and top blocker IDs.

Operator interpretation:
- `ageBucket: "fresh"` → expected dependency queueing.
- `ageBucket: "aging"` → review blocker progress.
- `ageBucket: "stale"` → emerging stall; escalate/unblock blocker.

## Process supervisor (`[process-supervisor]`)

The process supervisor logs when it registers a supervised child, starts teardown, expires the grace window, escalates to `SIGKILL`, or observes a natural child exit.

- `spawned pid=<pid> pgid=<pgid|n/a> command=<cmd>` — child registered for parent-death supervision.
- `terminating pid=<pid> pgid=<pgid|n/a> reason=<reason>` — teardown cascade started.
- `grace expired for pid=<pid>; escalating to SIGKILL` — child ignored the grace window.
- `sent SIGKILL to pid=<pid> pgid=<pgid|n/a>` — hard-kill escalation sent.
- `maxLifetime exceeded for pid=<pid> after <ms>ms` — lifetime watchdog fired.
- `child pid=<pid> exited naturally code=<n|null> signal=<n|null>` — child deregistered after exit.

## Self-healing surfacing passes (`[self-healing]`)

- `surface-in-review-stalls`
  - Log prefix: `In-review stall surfaced [`
  - Purpose: reason-driven in-review stall detector (`merge-blocker`, retry exhaustion, no-worktree, transient merge-status orphaning).
- `surface-in-review-stalled`
  - Log prefix: `In-review stalled surfaced [in-review-stalled]: quiet ...`
  - Purpose: time-quiet detector for unpaused in-review tasks beyond `inReviewStalledThresholdMs`.
  - Non-overlap: skipped when reason-driven `In-review stall surfaced [` is fresh, and skipped for paused tasks (owned by stale-paused-review).
- `surface-stale-paused-reviews`
  - Log prefix: `Stale paused review surfaced [stale-paused-review]: paused ...`
  - Purpose: paused in-review backlog-health detector gated by `stalePausedReviewThresholdMs`.
- FN-5335 backward-move annotations
  - Log prefix shape: `[<stage-name>] <taskId>: triple-proof not satisfied — no action (operator-decides)`
  - Representative stage names: `no-progress-no-task-done`, `partial-progress-no-task-done`, `stale-incomplete-review`, `ghost-review`, `missing-worktree-review`, `stuck-merge-deadlock`, `finalize-no-op-review`, `reclaim-pr-conflict`, `reclaim-self-owned-branch-conflict`, `auto-rebound-paused-scope-decay`.

## No-progress churn stuck-task escalation (`[executor]`, `[stuck-detector]`, `[self-healing]`)

Time-based stuck/stalled/stale surfaces now floor activity timestamps using `settings.engineActiveSinceMs` plus `settings.engineActivationGraceMs` (default `300000`). The runtime stamps `engineActiveSinceMs` on startup and each unpause transition so engine pause/downtime does not count as quiet time.

- Trigger shape: one loop classification/compact-and-resume has already fired for the current `execute()` lifecycle, then ignored `fn_task_update` rebuffs accumulate to `ignoredStepUpdateCount >= 25` without intervening progress.
- Executor diagnostic: `[executor] <taskId>: no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — escalating to STUCK_NO_PROGRESS_CHURN`.
- Self-healing diagnostic: `<taskId> no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — marking failed`.
- Audit event: `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason: "no-progress-churn" }`.
- Outcome: task is marked `status: "failed"`, moved to `in-review`, and not requeued; operators should decompose/rescope the task instead of waiting for more automatic stuck-kill retries.

## Stale self-owned active-session cleanup diagnostics (`[executor]`)

FN-5346 adds a same-task stale-binding reconcile marker before worktree removal:

- `[FN-5346] <taskId>: dropped stale self-owned activeSessionRegistry entry before removeWorktree at <worktreePath>`
- Follow-up task log entry: `Cleared stale self-owned active-session entry before remove`

## Runtime stop diagnostics (`[runtime-stop]`, `[executor]`)

Engine stop now aborts in-flight executor AI sessions before the runtime drain wait.

- Executor summary log: `[executor] abortAllInFlight: aborted N task surface(s) — engine stop`
- Runtime warning when in-flight work still exists after configured post-abort drain: `[runtime-stop] post-abort drain timeout reached with N tasks still in-flight`

Use these together to distinguish expected immediate session teardown from genuinely stuck cleanup surfaces that outlive the configured `runtimeStopDrainMs` window.

## Reports health stale-classifier diagnostics (`[reports-health]`)

Direct-report stale decisions in `HeartbeatMonitor.buildReportsHealthSection()` now emit a structured log when an agent is marked `**stale**`.

- Log shape: `[reports-health] stale report <agentId> intervalSource=<source> staleThresholdMs=<n> heartbeatAgeMs=<n>`
- `intervalSource` values:
  - `runtimeConfig` — interval came from cached per-agent runtime config
  - `persisted-agent` — cache was missing/sparse; interval came from persisted `getAgent()` row
  - `monitor-default` — no per-agent interval available; monitor default interval used
- `staleThresholdMs` is the computed stale threshold (`max(1.5 × interval, 5m floor)`)
- `heartbeatAgeMs` is the report's current heartbeat age at classification time
- Healthy reports do not emit this diagnostic; only stale decisions do

## Resume instrumentation (FN-5389, Phase 1)

Dashboard Phase 1 resume instrumentation adds observation-only client/server traces for refetch/reconnect attribution. It does not change visibility/pageshow/SSE behavior; FN-5392 consumes this data for fixes.

- Client event shape (`ResumeEvent`): `{ ts, view, trigger, projectId?, gapMs?, replayAttempted, replayFromEventId?, lastEventId?, sseChannel?, reason?, detail? }`.
- Trigger taxonomy: `visibility`, `pageshow`, `sse-error`, `sse-reconnect`, `sse-open`, `remount`, `route-active`, `route-inactive`, `project-context-change`.
- Sources:
  - `sse-bus` (`pageshow`, visible `visibilitychange`, `openChannel`, `forceReconnect`, EventSource `error`)
  - Hooks: `useTasks` (`visibility`, `sse-reconnect`), `useChatRooms` (`sse-reconnect`), `useChat` (`sse-open`, `project-context-change`)
  - Components: `Board` and `ChatView` mount/unmount route markers (`remount` / `route-active` / `route-inactive`)
- Access paths:
  - Client ring (500): `window.__fusionDebug.resumeInstrumentation.get()` / `.clear()`
  - Server ring (5000, in-memory): `GET /api/diagnostics/resume-events?limit=&since=&view=` returns `{ events, droppedSinceLastRead }`
- Client batching: POST `/api/diagnostics/resume-events` in idle batches (`<=25` per POST).
- Disable knob: `window.__fusionDebug.resumeInstrumentation.setEnabled(false)`.

FN-5415 extends this coverage across remaining board/data visibility hooks: `useNodes`, `useMeshState`, `useProjects`, and `useManagedDockerNodes`. Each now emits `trigger: "visibility"` with `reason: "debounced-refresh"` when refresh is taken and `reason: "debounce-skipped"` (including `detail.timeSinceLastRefreshMs`) when suppressed by debounce. This completes board/data-hook resume-correlation coverage needed for FN-5392 Phase 2 remediation analysis.

### Phase 3 coverage (FN-5416)

FN-5416 extends resume-correlation coverage to stream-focused hooks and their primary route shells:

- Hooks
  - `usePrChecksStream`: `remount`, `visibility`
  - `useDevServerLogs`: `project-context-change`, `sse-open`, `sse-reconnect`
  - `useResearch`: `sse-open`, `sse-reconnect`
  - `useBackgroundSessions`: `sse-open`, `sse-reconnect`
  - `useAgentLogs`: `project-context-change`, `sse-open`, `sse-reconnect` on `/api/tasks/:id/logs/stream`
- Route shells
  - `DevServerView`: `remount` / `route-active` / `route-inactive`
  - `ResearchView`: `remount` / `route-active` / `route-inactive`