feat(FN-3958): add heartbeat timer reconciliation self-healing

Adds scheduler heartbeat timer reconciliation with automatic self-healing when timers drift, including tests for tracked-only monitor recovery and documentation in the agents reference.

Fusion-Task-Id: FN-3958
This commit is contained in:
Fusion
2026-05-10 23:24:27 -07:00
committed by gsxdsm
parent e151cf2c55
commit 44a9fb8ed8
7 changed files with 176 additions and 5 deletions

View File

@@ -970,6 +970,21 @@ Effects:
- Resume triggers one on-demand heartbeat restart only when `runtimeConfig.enabled !== false`
- `onTerminated` is a run-level callback for terminated heartbeat runs and is not used by unresponsive recovery
### Timer Reconciliation Self-Healing (FN-3958)
`HeartbeatTriggerScheduler` owns a periodic registration audit that reconciles durable-agent truth in `AgentStore` against the in-memory timer map.
- Audit cadence: once immediately on scheduler start, then every 60 seconds while running
- Repair target: durable, heartbeat-enabled agents in tickable states (`active`, `running`, `idle`) that are missing a timer entry
- Safety guards: skip ephemeral/task-worker agents, skip disabled agents, skip non-tickable states, and skip agents with an active heartbeat run
- Existing timer entries are left untouched (no interval reset/jitter churn)
This covers the untracked timer-loss failure mode where no `agent:updated` event fires after a timer entry disappears. Manual stop/start is no longer required to re-arm the timer in that case.
Separation of responsibilities:
- **HeartbeatMonitor recovery** handles **tracked stale sessions** (stuck in-memory run/session cleanup + pause/resume restart)
- **HeartbeatTriggerScheduler audit** handles **untracked missing-timer registration drift** (re-arm scheduling)
## Dashboard Health Status
The dashboard displays agent health status in AgentsView, AgentListModal, and AgentDetailView using a centralized health evaluation utility (`packages/dashboard/app/utils/agentHealth.ts`).