root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

FN-7672: recover durable agents stuck in error state despite active manager

Browse Source 
Root-causes 4 correlated CTO-report agent failures where durable non-ephemeral agents got stuck in `error` state indefinitely because the heartbeat scheduler stops ticking error-state agents entirely, and self-healing's recovery sweep previously only considered them when their manager row was missing.

- SelfHealingManager: scope the `managerMissing` gate to the "running" orphan-detection path only, so "error"-state durable agents with a present/active manager now fall through to the existing transient/operator-actionable/active-execution/cooldown/retry-budget recovery guards instead of being skipped outright
- Add FNXC:AgentHeartbeat comment documenting the FN-7672 incident and rationale for the scoping change
- Extend self-healing.test.ts with coverage for manager-present durable agents in error state
- Add changeset (patch) describing the fix for release notes
- Update docs/agents.md accordingly

Files changed:
 .changeset/fn-7672-durable-agent-recovery.md       |   7 ++
 docs/agents.md                                     |   2 +
 packages/engine/src/__tests__/self-healing.test.ts | 129 ++++++++++++++++++++-
 packages/engine/src/self-healing.ts                |  24 +++-
 4 files changed, 160 insertions(+), 2 deletions(-)

Fusion-Task-Id: FN-7672

Fusion-Task-Lineage: 6676dc9e-66e7-4f70-804a-cccf77e8d337

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-07-08 05:38:01 -07:00
parent b8fa2a6652
commit aa534c19af
4 changed files with 160 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/agents.md
View File

@@ -364,6 +364,8 @@ On restart attempts, the runtime triggers the normal heartbeat pipeline with `so
Self-healing intentionally leaves agents in `error` (no auto-restart) when blockers are operator-actionable or non-transient, when cooldown has not elapsed, when active execution is present, or when retry budget is exhausted.
**Manager presence does not gate this sweep (FN-7672):** eligibility for durable `state="error"` recovery does *not* depend on whether the agent's `reportsTo` manager is present/active. `HeartbeatTriggerScheduler` clears timers entirely once an agent enters `state="error"`, so this recovery sweep is the *only* path back to a healthy heartbeat for a durable agent stuck in `error` — a present manager does not make the agent any less stuck. (A separate, unrelated `managerMissing` check still gates recovery of orphaned `state="running"` agents — a different failure mode where a live process's manager row was deleted.) FN-7672 root-caused a correlated 4-agent error cluster reporting to one active manager (a transient upstream auth/session blip) that could never have self-healed under the old manager-missing-only gate, even once the underlying cause resolved.
- **Timer trigger:** run completes and the durable agent returns to `state="active"` (recoverable soft-fail).
- **Assignment / on-demand trigger:** run completes with `resultJson.actionRequired = true`, then the durable agent is paused with `pauseReason="heartbeat-model-unavailable"` and `lastError` set to actionable credential guidance (including the missing provider name when detectable).