root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(FN-4005): harden durable transient recovery auto-restart flow

Browse Source 
- Add bounded auto-restart handling in in-process runtime for durable agents after transient failures
- Extend self-healing recovery logic and transient error detection to classify and recover retryable runtime interruptions
- Add regression coverage across heartbeat executor, self-healing, and transient detector test suites
- Document durable agent transient recovery behavior in docs/agents.md

Fusion-Task-Id: FN-4005
This commit is contained in:
Fusion
2026-05-11 09:17:08 -07:00
committed by gsxdsm
parent bf4786e550
commit 631b384837
7 changed files with 330 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/agents.md
View File

@@ -218,6 +218,26 @@ When the runtime model is present and differs from execution-lane settings, hear
If a heartbeat cannot create/run a session due to unavailable provider credentials or missing provider registration, Fusion records `resultJson.reason = "heartbeat_model_unavailable"` with actionable diagnostics in `resultJson.detail`/`stderrExcerpt`.
### Durable-agent transient error auto-recovery
Self-healing may auto-recover **durable (non-ephemeral)** agents stuck in `state="error"` when all eligibility checks pass:
- agent is non-ephemeral (`isEphemeralAgent(...) === false`)
- heartbeat runtime is enabled (`runtimeConfig.enabled !== false`)
- no active heartbeat execution is already running for the agent
- `lastError` classifies as transient network/infrastructure failure
- `lastError` is **not** operator-actionable (credentials/model/billing-style failures)
When eligible, self-healing uses bounded retries with persisted metadata at `agent.metadata.durableErrorRecovery`:
- exponential cooldown (`30s` base, capped at `15m`)
- retry budget cap (`5` attempts)
- persisted `attempts`, `lastAttemptAt`, `nextRetryAt`, `exhausted`, and `lastReason`
On restart attempts, the runtime triggers the normal heartbeat pipeline with `source: "automation"` and a structured `contextSnapshot.selfHealing` payload so operators can audit recovery runs in heartbeat history.
Self-healing intentionally leaves agents in `error` (no auto-restart) when blockers are operator-actionable or non-transient, when cooldown has not elapsed, when active execution is present, or when retry budget is exhausted.
- **Timer trigger:** run completes and the durable agent returns to `state="active"` (recoverable soft-fail).
- **Assignment / on-demand trigger:** run completes with `resultJson.actionRequired = true`, then the durable agent is paused with `pauseReason="heartbeat-model-unavailable"` and `lastError` set to actionable credential guidance (including the missing provider name when detectable).