FN-7878: default heartbeat error recovery to recoverable for generic durable-agent failures

Durable agents were parking as error-unrecoverable on any non-transient-pattern failure, even generic/unknown blips that manual Retry immediately fixed; this changes the default to recoverable and reserves immediate unrecoverable parking for operator-actionable errors.

- isHeartbeatErrorRecoverable now returns true unless the error is operator-actionable (auth/model/billing/scope) or a stale worktree/module-resolution error, instead of requiring a transient-pattern match via classifyError
- Add OAuth scope-requirement and insufficient-scope patterns to the operator-actionable error detector so those still park immediately
- Update heartbeat-error-recovery, heartbeat-executor, self-healing, and transient-error-detector tests to cover the new default-recoverable behavior
- Update AGENTS.md and docs/architecture.md durable-agent error recovery notes to describe the new recoverable-by-default policy
- Add changeset documenting the fix

Files changed:
 .changeset/fn-7878-recoverable-default.md          |  7 ++
 AGENTS.md                                          |  2 +-
 docs/architecture.md                               |  4 +-
 .../src/__tests__/heartbeat-error-recovery.test.ts | 90 +++++++++++++++++++---
 .../src/__tests__/heartbeat-executor.test.ts       | 17 ++--
 packages/engine/src/__tests__/self-healing.test.ts | 45 ++++++-----
 .../src/__tests__/transient-error-detector.test.ts |  7 +-
 packages/engine/src/agent-heartbeat.ts             |  8 +-
 packages/engine/src/transient-error-detector.ts    |  2 +
 9 files changed, 137 insertions(+), 45 deletions(-)

Fusion-Task-Id: FN-7878

Fusion-Task-Lineage: 6f929af9-ceef-404f-95c9-98f26478f020

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>

docs/architecture.md

@@ -676,7 +676,7 @@ Runtime action-gate flow (v1):
 - `StuckTaskDetector` (`stuck-task-detector.ts`) — inactivity/loop stall detection
 - `GridlockDetector` (`gridlock-detector.ts`) — detects all-blocked todo pipelines and emits notification events (plus explicit clear signals when gridlock resolves)
 - `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
-- Durable agent error recovery (FN-7835/FN-7844/FN-7859): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is transient and non-operator-actionable. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors are not restarted; both timer and sweep paths park them `paused` with `pauseReason:"error-unrecoverable"` and emit `agent:error-parked-unrecoverable`, while preserving exclusions for disabled runtime agents, ephemeral agents, active executions, and stale worktree/module-resolution suppression.
+- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors are not restarted; both timer and sweep paths park them `paused` with `pauseReason:"error-unrecoverable"` and emit `agent:error-parked-unrecoverable`, while preserving exclusions for disabled runtime agents, ephemeral agents, active executions, and stale worktree/module-resolution suppression.
 - `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
   - Batch 1 maintenance now includes `reconcile-orphaned-task-dirs` (FN-6783), a paused-safe housekeeping step that calls `TaskStore.reconcileOrphanedTaskDirs()` so valid live `.fusion/tasks/{ID}/task.json` records missing from the SQLite index become visible without waiting for process restart. The store-level guard skips any ID already present in active, soft-deleted, archived, or tombstoned storage and emits `task:reconcile-orphaned-task-dir` only for recovered rows.
   - Batch 1 maintenance also includes `reconcile-phantom-committed-reservations` (FN-7069), which calls `TaskStore.reconcilePhantomCommittedReservations()` for committed task-ID reservations that have no live/soft-deleted/archived task row and no `.fusion/tasks/{ID}/task.json`. The sweep prunes orphaned `activityLog` rows and `agents`/cascaded `agentRuns`, preserves `runAuditEvents`, and keeps the reservation `committed` per FN-5105 so the ID is permanently reserved rather than resurrected or handed out again.
@@ -2162,7 +2162,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
 - **Completion fan-out is synchronous**: `SelfHealingManager.reconcileCompletedTask()` runs on `in-review → done`. Downstream stale `blockedBy` links and residual `fusion/<task-id>` branch/worktree artifacts are reconciled immediately, not on a periodic sweep.
 - **In-review stall deadlock**: identical stalls (same code + reason) repeated past `inReviewStallDeadlockThreshold` (default 3) auto-pause with `pausedReason: "in-review-stall-deadlock"` and `status: "failed"`. User-initiated retry paths (dashboard retry, `fn_task_retry`, and CLI `task retry`) clear that automatic deadlock pause so the retry can execute, but they never override explicit/manual pauses or unrelated automatic pause reasons.
 - **Restart recovery**: `RestartRecoveryCoordinator` classifies interrupted `in-progress` runs. Unusable-worktree session-start failures (`missing`, `incomplete`, `unregistered git worktree`) are recoverable; retries are capped at `MAX_WORKTREE_SESSION_RETRIES=3` before escalating. `recoverMissingWorktreeReviewFailures` also owns durable unusable-worktree session-start failures that reached `in-review` with a merge-active status (`merging`, `merging-pr`, or `merging-fix`): before interrupted/deadlocked merge sweeps can re-drive the same phantom path, it applies auto-merge eligibility, workspace-task exclusion, and triple-proof, clears stale `worktree`/`branch`/`sessionFile`, resets the exhausted worktree-session retry budget, increments `recoveryRetryCount` as the bounded merge-active stale-metadata clear counter, and requeues to `todo` preserving progress. Operator retry surfaces (`fn_task_retry`, CLI `task retry`, dashboard retry) have the same signature-only reset primitive so a missing-worktree failure does not require a valid `merging` transition.
-- **Durable agent heartbeat error recovery (FN-7835/FN-7859)**: `HeartbeatTriggerScheduler` keeps timers armed for durable heartbeat-managed agents in `state:"error"` only when `lastError` classifies as transient and is not operator-actionable (credential, quota, model-access, or permanent configuration failures do not restart). `HeartbeatMonitor.executeHeartbeat()` clears recoverable errors at run entry (`error → active`, clears `lastError`), increments a consecutive `metadata.heartbeatErrorRecovery` counter, and emits `agent:auto-recover-error-state`; success resets the counter. Once the bounded budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable) is exhausted, the agent is parked `paused` with `pauseReason:"error-retry-exhausted"` and `agent:error-retry-exhausted` is emitted. Non-recoverable durable heartbeat errors are parked `paused` with `pauseReason:"error-unrecoverable"` and `agent:error-parked-unrecoverable` rather than remaining indefinitely in bare `error`.
+- **Durable agent heartbeat error recovery (FN-7835/FN-7859/FN-7878)**: `HeartbeatTriggerScheduler` keeps timers armed for durable heartbeat-managed agents in `state:"error"` when `lastError` is recoverable: generic/unknown errors and transient credential-rotation failures get the bounded retry budget, while operator-actionable credential/scope, quota/billing, and model-access failures do not restart. Stale worktree/module-resolution errors also skip naive retry recovery so the dedicated stale-host/worktree suppression path can handle them. `HeartbeatMonitor.executeHeartbeat()` clears recoverable errors at run entry (`error → active`, clears `lastError`), increments a consecutive `metadata.heartbeatErrorRecovery` counter, and emits `agent:auto-recover-error-state`; success resets the counter. Once the bounded budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable) is exhausted, the agent is parked `paused` with `pauseReason:"error-retry-exhausted"` and `agent:error-retry-exhausted` is emitted. Non-recoverable durable heartbeat errors are parked `paused` with `pauseReason:"error-unrecoverable"` and `agent:error-parked-unrecoverable` rather than remaining indefinitely in bare `error`.
 - **Executor pre-session liveness gate (FN-4935/FN-6861)**: the gate now skips for fresh acquisitions (`acquisition.source === "fresh"`), emits structured `not_usable_task_worktree:<classification>` diagnostics (including canonicalized registered-path snapshots) and a `worktree:incomplete-detected` audit event with `source: "executor-liveness-gate"`, while preserving the existing `taskDoneRetryCount` / `MAX_TASK_DONE_REQUEUE_RETRIES` requeue contract. The project repo root is never a usable task worktree even though it is a legitimately registered Git worktree; `classifyTaskWorktree` returns `repo-root` for canonical root-equal paths, and resume acquisition treats that as self-healable stale metadata by clearing `task.worktree` and creating a fresh checkout under the configured worktrees directory. FN-5772 adds a bounded nested-root self-heal: when `task.worktree` points at a strict descendant of a registered worktree root inside the configured worktrees dir, executor re-anchors `task.worktree` to the git top-level, emits `worktree:reanchored` (`fromPath`, `toPath`, `source`), and proceeds; repo-root/outside-dir/unregistered top-level mismatches still fail. FN-4651 `worktreeSessionRetryCount` remains scoped to the in-review/session-start recovery path.
 - **Stale self-owned active-session reconcile on conflict cleanup (FN-4973)**: when executor worktree-conflict cleanup finds only a same-task stale `activeSessionRegistry` entry and no live in-memory `activeWorktrees` binding for that task/path, it must unregister the stale entry before `removeWorktree` (plus one-shot backstop reconcile on same-task `ActiveSessionWorktreeRemovalError` races). Foreign-task entries remain protected by FN-4811 and must never be reconciled by the requesting task.
 - **Same-task stale removal canonical helper (FN-5346)**: executor same-task cleanup paths now route pre-removal reconciliation through `reconcileSelfOwnedActiveSessionForRemoval` (via executor helper wiring), so stale self-owned `activeSessionRegistry` residues are cleared only when no live in-memory binding exists, while FN-4811 foreign-owner refusals and live-owner protections remain intact.