FN-7884: reset durable-agent error state on engine restart

Engine startup now treats itself as an implicit operator retry for durable heartbeat agents stuck in error, clearing eligible error states and re-arming heartbeats instead of waiting for the steady-state sweep's cooldown/exhaustion gates.

- Add SelfHealingManager.resetDurableAgentErrorStateOnStartup(), run first in runStartupRecovery(), which resets shared heartbeatErrorRecovery/legacy durableErrorRecovery metadata, clears lastError/pauseReason, flips eligible error and error-retry-exhausted-parked durable agents to active, and re-arms their heartbeat
- Preserve suppression for operator-actionable, stale worktree/module-resolution, user-paused, error-unrecoverable, ephemeral, disabled-runtime, and actively-executing agents
- Add agent:reset-error-state-on-startup run-audit mutation type with ids/counts/outcomes-only metadata (agentId, priorState, priorPauseReason, source)
- Add changeset FN-7884 (patch) documenting the operator-facing behavior
- Update AGENTS.md and docs/agents.md, docs/architecture.md to describe the new startup reset path alongside existing FN-7835/FN-7844/FN-7859/FN-7878 recovery docs
- Extend self-healing.test.ts with coverage for the new startup reset behavior and its exclusions

Files changed:
 .changeset/fn-7884-restart-error-reset.md          |   7 ++
 AGENTS.md                                          |   1 +
 docs/agents.md                                     |   4 +-
 docs/architecture.md                               |   2 +-
 packages/engine/src/__tests__/self-healing.test.ts | 127 ++++++++++++++++++++-
 packages/engine/src/run-audit.ts                   |   1 +
 packages/engine/src/self-healing.ts                |  88 +++++++++++++-
 7 files changed, 223 insertions(+), 7 deletions(-)

Fusion-Task-Id: FN-7884
Fusion-Task-Lineage: fe64f6af-3ff3-4876-8308-8a75591c45f1
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>

docs/agents.md

@@ -362,14 +362,14 @@ Durable-agent error recovery is coordinated between the heartbeat timer path and
 - `lastError` is **not** operator-actionable (credentials/model/billing-style failures)
 - stale worktree/module-resolution failures remain suppressed instead of auto-restarted
 
-Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details:
+Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Engine startup adds a clean-slate reset before the steady-state sweep: eligible `state="error"` agents and durable agents parked with `pauseReason="error-retry-exhausted"` have `heartbeatErrorRecovery` and legacy `durableErrorRecovery` reset, `lastError`/exhaustion pause state cleared, and their heartbeat re-armed immediately. That restart-triggered reset intentionally bypasses the sweep's staleness, cooldown, and exhausted-budget gates because restarting the engine is treated like an operator Retry click. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details:
 
 - exponential cooldown (`30s` base, capped at `15m`)
 - persisted `attempts`, `lastAttemptAt`, `nextRetryAt`, `exhausted`, `lastReason`, and stale missing-module path counters
 
 On restart attempts, the runtime triggers the normal heartbeat pipeline with `source: "automation"` and a structured `contextSnapshot.selfHealing` payload so operators can audit recovery runs in heartbeat history. The sweep flips `error → active` before calling `executeHeartbeat`, so the heartbeat run does not re-enter run-entry error recovery or double-count the same recovery.
 
-Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, cooldown windows, active execution, runtime-disabled agents, and ephemeral agents remain excluded from this parking/retry path.
+Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, active execution, runtime-disabled agents, ephemeral agents, user-paused agents, and `error-unrecoverable` parks remain excluded from both the steady-state retry path and the startup clean-slate reset; cooldown windows and exhausted-budget gates are bypassed only by the startup reset for otherwise recoverable agents.
 
 **Manager presence does not gate this sweep (FN-7672/FN-7844):** eligibility for durable `state="error"` recovery does *not* depend on whether the agent's `reportsTo` manager is present/active. The timer path is now the fast path for heartbeat-managed error agents, while this recovery sweep remains the maintenance backstop for durable agents that are still stale in `error`; a present manager does not make the agent any less stuck. (A separate, unrelated `managerMissing` check still gates recovery of orphaned `state="running"` agents — a different failure mode where a live process's manager row was deleted.) FN-7672 root-caused a correlated 4-agent error cluster reporting to one active manager (a transient upstream auth/session blip) that could never have self-healed under the old manager-missing-only gate, even once the underlying cause resolved.
 

docs/architecture.md

@@ -676,7 +676,7 @@ Runtime action-gate flow (v1):
 - `StuckTaskDetector` (`stuck-task-detector.ts`) — inactivity/loop stall detection
 - `GridlockDetector` (`gridlock-detector.ts`) — detects all-blocked todo pipelines and emits notification events (plus explicit clear signals when gridlock resolves)
 - `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
-- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors are not restarted; both timer and sweep paths park them `paused` with `pauseReason:"error-unrecoverable"` and emit `agent:error-parked-unrecoverable`, while preserving exclusions for disabled runtime agents, ephemeral agents, active executions, and stale worktree/module-resolution suppression.
+- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878/FN-7884): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. On engine startup, `SelfHealingManager.resetDurableAgentErrorStateOnStartup()` runs before the steady-state sweep and treats restart as an explicit operator retry: eligible `error` and `error-retry-exhausted` durable agents have shared/legacy retry metadata reset, `lastError` and the exhaustion pause cleared, state set to `active`, heartbeat re-armed, and `agent:reset-error-state-on-startup` emitted without applying the sweep's staleness/cooldown/exhaustion gates. Non-recoverable durable heartbeat errors are not restarted; timer, startup, and sweep paths preserve exclusions for disabled runtime agents, ephemeral agents, active executions, user pauses, `error-unrecoverable` parks, operator-actionable errors, and stale worktree/module-resolution suppression.
 - `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
   - Batch 1 maintenance now includes `reconcile-orphaned-task-dirs` (FN-6783), a paused-safe housekeeping step that calls `TaskStore.reconcileOrphanedTaskDirs()` so valid live `.fusion/tasks/{ID}/task.json` records missing from the SQLite index become visible without waiting for process restart. The store-level guard skips any ID already present in active, soft-deleted, archived, or tombstoned storage and emits `task:reconcile-orphaned-task-dir` only for recovered rows.
   - Batch 1 maintenance also includes `reconcile-phantom-committed-reservations` (FN-7069), which calls `TaskStore.reconcilePhantomCommittedReservations()` for committed task-ID reservations that have no live/soft-deleted/archived task row and no `.fusion/tasks/{ID}/task.json`. The sweep prunes orphaned `activityLog` rows and `agents`/cascaded `agentRuns`, preserves `runAuditEvents`, and keeps the reservation `committed` per FN-5105 so the ID is permanently reserved rather than resurrected or handed out again.