root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fab2377fd160c9647851daf4b6311986686822e0
fusion/docs/diagnostics.md
gsxdsm 194dfa9387 FN-5758: add cited-goal audit aggregation and runtime route 
Add end-to-end goal-citation audit trail plumbing across core, engine, CLI, and dashboard.

- extend goal citation extraction with aggregation logic and new core coverage
- expose cited-goal runtime data through dashboard routes with dedicated API tests
- wire goal-citation audit details into engine diagnostics and CLI extension audit expectations
- document the new audit behavior and publish a patch changeset for @runfusion/fusion

Files changed:
 .changeset/fn-5758-goal-citation-audit.md          |  9 +++
 docs/dashboard-guide.md                            |  2 +
 docs/diagnostics.md                                |  4 ++
 .../__tests__/extension-goal-tools-audit.test.ts   |  6 +-
 packages/cli/src/extension.ts                      |  3 +
 .../goal-citation-audit-aggregation.test.ts        | 63 ++++++++++++++++++
 packages/core/src/goal-citation-extractor.ts       | 54 ++++++++++++++-
 packages/core/src/index.ts                         |  1 +
 .../src/__tests__/routes-run-cited-goals.test.ts   | 77 ++++++++++++++++++++++
 packages/dashboard/src/routes.ts                   | 11 ++++
 .../src/routes/register-agent-runtime-routes.ts    | 49 ++++++++++++++
 .../src/__tests__/goal-anchoring-audit.test.ts     | 40 +++++++----
 packages/engine/src/goal-anchoring-audit.ts        |  4 ++
 packages/engine/src/goal-injection-diagnostics.ts  |  1 +
 14 files changed, 308 insertions(+), 16 deletions(-)

Fusion-Task-Id: FN-5758

Fusion-Task-Lineage: b077c0b7-6ca4-489d-8c36-73d80a2afdb2
2026-05-30 22:53:24 -07:00

133 lines
9.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Diagnostics
## Goal injection diagnostics (`[goal-injection]`)
Executor, heartbeat, and planning runs emit one goal-injection diagnostic with outcome `applied`, `no-goals`, or `disabled-or-failed`.
- Run-audit event: `prompt:goal-injection` (`database` domain, target lane) with metadata `{ lane, outcome, goalCount, goalIds, truncated, reason?, errorClass?, runId?, agentId?, taskId? }`.
- Goal anchoring events also persist `metadata.goalIds` (alongside existing count/tool fields):
- `goal:injection-applied` / `goal:injection-skipped` → `{ lane, count, goalIds, truncated?, reason? }`
- `goal:retrieval-invoked` → `{ toolName, count, goalIds, notFound }`
- Run cited-goals read path: `GET /api/agents/:id/runs/:runId/cited-goals` returns `{ runId, taskId?, injectedGoalIds, retrievedGoalIds, citedGoalIds }` aggregated from `goal:*` + `prompt:goal-injection` run-audit events.
- Task log (executor lane with `taskId`): `[goal-injection] <outcome> count=<n> ids=<json-array> truncated=<bool> ...`.
- Guardrail: diagnostics persist goal IDs/counts only; never prompt text, goal titles, or goal descriptions.
## Insight run sweeper (`[insight-sweeper]`)
The dashboard insight router runs stale-run recovery sweeps for `project_insight_runs` rows stuck in `pending`/`running` without a live controller owner.
- Recovery writes `terminalCause: "orphaned_active_run_recovered"` and lifecycle failure metadata (`failureClass: "non_retryable"`, `retryable: false`).
- Recovery appends both `warning` and `status_changed` events on `project_insight_run_events` with `metadata.recovery = "orphaned_active_run"`.
- `metadata.recoverySource` indicates where recovery occurred: `startup`, `periodic`, `drive_by`, or `manual`.
## Dependency-blocked Todo backlog health (`[dependency-blocked-todo]`)
Self-healing now runs `surface-dependency-blocked-todos` during both startup recovery and periodic maintenance.
- Normal path emits a workflow insight titled `Backlog health: dependency-blocked todos YYYY-MM-DD`.
- Fallback path (insight store unavailable) writes a per-task log entry prefixed with `[dependency-blocked-todo]` against the top blocker task.
- Reporter summary warnings include group count, total blocked Todo count, and top blocker IDs.
Operator interpretation:
- `ageBucket: "fresh"` → expected dependency queueing.
- `ageBucket: "aging"` → review blocker progress.
- `ageBucket: "stale"` → emerging stall; escalate/unblock blocker.
## Process supervisor (`[process-supervisor]`)
The process supervisor logs when it registers a supervised child, starts teardown, expires the grace window, escalates to `SIGKILL`, or observes a natural child exit.
- `spawned pid=<pid> pgid=<pgid|n/a> command=<cmd>` — child registered for parent-death supervision.
- `terminating pid=<pid> pgid=<pgid|n/a> reason=<reason>` — teardown cascade started.
- `grace expired for pid=<pid>; escalating to SIGKILL` — child ignored the grace window.
- `sent SIGKILL to pid=<pid> pgid=<pgid|n/a>` — hard-kill escalation sent.
- `maxLifetime exceeded for pid=<pid> after <ms>ms` — lifetime watchdog fired.
- `child pid=<pid> exited naturally code=<n|null> signal=<n|null>` — child deregistered after exit.
## Self-healing surfacing passes (`[self-healing]`)
- `surface-in-review-stalls`
- Log prefix: `In-review stall surfaced [`
- Purpose: reason-driven in-review stall detector (`merge-blocker`, retry exhaustion, no-worktree, transient merge-status orphaning).
- `surface-in-review-stalled`
- Log prefix: `In-review stalled surfaced [in-review-stalled]: quiet ...`
- Purpose: time-quiet detector for unpaused in-review tasks beyond `inReviewStalledThresholdMs`.
- Non-overlap: skipped when reason-driven `In-review stall surfaced [` is fresh, and skipped for paused tasks (owned by stale-paused-review).
- `surface-stale-paused-reviews`
- Log prefix: `Stale paused review surfaced [stale-paused-review]: paused ...`
- Purpose: paused in-review backlog-health detector gated by `stalePausedReviewThresholdMs`.
- FN-5335 backward-move annotations
- Log prefix shape: `[<stage-name>] <taskId>: triple-proof not satisfied — no action (operator-decides)`
- Representative stage names: `no-progress-no-task-done`, `partial-progress-no-task-done`, `stale-incomplete-review`, `ghost-review`, `missing-worktree-review`, `stuck-merge-deadlock`, `finalize-no-op-review`, `reclaim-pr-conflict`, `reclaim-self-owned-branch-conflict`, `auto-rebound-paused-scope-decay`.
## No-progress churn stuck-task escalation (`[executor]`, `[stuck-detector]`, `[self-healing]`)
Time-based stuck/stalled/stale surfaces now floor activity timestamps using `settings.engineActiveSinceMs` plus `settings.engineActivationGraceMs` (default `300000`). The runtime stamps `engineActiveSinceMs` on startup and each unpause transition so engine pause/downtime does not count as quiet time.
- Trigger shape: one loop classification/compact-and-resume has already fired for the current `execute()` lifecycle, then ignored `fn_task_update` rebuffs accumulate to `ignoredStepUpdateCount >= 25` without intervening progress.
- Executor diagnostic: `[executor] <taskId>: no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — escalating to STUCK_NO_PROGRESS_CHURN`.
- Self-healing diagnostic: `<taskId> no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — marking failed`.
- Audit event: `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason: "no-progress-churn" }`.
- Outcome: task is marked `status: "failed"`, moved to `in-review`, and not requeued; operators should decompose/rescope the task instead of waiting for more automatic stuck-kill retries.
## Stale self-owned active-session cleanup diagnostics (`[executor]`)
FN-5346 adds a same-task stale-binding reconcile marker before worktree removal:
- `[FN-5346] <taskId>: dropped stale self-owned activeSessionRegistry entry before removeWorktree at <worktreePath>`
- Follow-up task log entry: `Cleared stale self-owned active-session entry before remove`
## Runtime stop diagnostics (`[runtime-stop]`, `[executor]`)
Engine stop now aborts in-flight executor AI sessions before the runtime drain wait.
- Executor summary log: `[executor] abortAllInFlight: aborted N task surface(s) — engine stop`
- Runtime warning when in-flight work still exists after configured post-abort drain: `[runtime-stop] post-abort drain timeout reached with N tasks still in-flight`
Use these together to distinguish expected immediate session teardown from genuinely stuck cleanup surfaces that outlive the configured `runtimeStopDrainMs` window.
## Reports health stale-classifier diagnostics (`[reports-health]`)
Direct-report stale decisions in `HeartbeatMonitor.buildReportsHealthSection()` now emit a structured log when an agent is marked `**stale**`.
- Log shape: `[reports-health] stale report <agentId> intervalSource=<source> staleThresholdMs=<n> heartbeatAgeMs=<n>`
- `intervalSource` values:
- `runtimeConfig` — interval came from cached per-agent runtime config
- `persisted-agent` — cache was missing/sparse; interval came from persisted `getAgent()` row
- `monitor-default` — no per-agent interval available; monitor default interval used
- `staleThresholdMs` is the computed stale threshold (`max(1.5 × interval, 5m floor)`)
- `heartbeatAgeMs` is the report's current heartbeat age at classification time
- Healthy reports do not emit this diagnostic; only stale decisions do
## Resume instrumentation (FN-5389, Phase 1)
Dashboard Phase 1 resume instrumentation adds observation-only client/server traces for refetch/reconnect attribution. It does not change visibility/pageshow/SSE behavior; FN-5392 consumes this data for fixes.
- Client event shape (`ResumeEvent`): `{ ts, view, trigger, projectId?, gapMs?, replayAttempted, replayFromEventId?, lastEventId?, sseChannel?, reason?, detail? }`.
- Trigger taxonomy: `visibility`, `pageshow`, `sse-error`, `sse-reconnect`, `sse-open`, `remount`, `route-active`, `route-inactive`, `project-context-change`.
- Sources:
- `sse-bus` (`pageshow`, visible `visibilitychange`, `openChannel`, `forceReconnect`, EventSource `error`)
- Hooks: `useTasks` (`visibility`, `sse-reconnect`), `useChatRooms` (`sse-reconnect`), `useChat` (`sse-open`, `project-context-change`)
- Components: `Board` and `ChatView` mount/unmount route markers (`remount` / `route-active` / `route-inactive`)
- Access paths:
- Client ring (500): `window.__fusionDebug.resumeInstrumentation.get()` / `.clear()`
- Server ring (5000, in-memory): `GET /api/diagnostics/resume-events?limit=&since=&view=` returns `{ events, droppedSinceLastRead }`
- Client batching: POST `/api/diagnostics/resume-events` in idle batches (`<=25` per POST).
- Disable knob: `window.__fusionDebug.resumeInstrumentation.setEnabled(false)`.
FN-5415 extends this coverage across remaining board/data visibility hooks: `useNodes`, `useMeshState`, `useProjects`, and `useManagedDockerNodes`. Each now emits `trigger: "visibility"` with `reason: "debounced-refresh"` when refresh is taken and `reason: "debounce-skipped"` (including `detail.timeSinceLastRefreshMs`) when suppressed by debounce. This completes board/data-hook resume-correlation coverage needed for FN-5392 Phase 2 remediation analysis.
### Phase 3 coverage (FN-5416)
FN-5416 extends resume-correlation coverage to stream-focused hooks and their primary route shells:
- Hooks
- `usePrChecksStream`: `remount`, `visibility`
- `useDevServerLogs`: `project-context-change`, `sse-open`, `sse-reconnect`
- `useResearch`: `sse-open`, `sse-reconnect`
- `useBackgroundSessions`: `sse-open`, `sse-reconnect`
- `useAgentLogs`: `project-context-change`, `sse-open`, `sse-reconnect` on `/api/tasks/:id/logs/stream`
- Route shells
- `DevServerView`: `remount` / `route-active` / `route-inactive`
- `ResearchView`: `remount` / `route-active` / `route-inactive`
Reference in New Issue View Git Blame Copy Permalink