diff --git a/.changeset/fn-7884-restart-error-reset.md b/.changeset/fn-7884-restart-error-reset.md new file mode 100644 index 0000000000..b1860b59dc --- /dev/null +++ b/.changeset/fn-7884-restart-error-reset.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": patch +--- + +summary: Durable agents in error state are cleared and retried automatically on engine restart. +category: fix +dev: New SelfHealingManager.resetDurableAgentErrorStateOnStartup() runs first in runStartupRecovery(): it resets the shared heartbeatErrorRecovery/durableErrorRecovery budget+cooldown, clears lastError, flips eligible error and error-retry-exhausted-parked durable agents to active, re-arms the heartbeat, and emits agent:reset-error-state-on-startup — bypassing the steady-state staleness/cooldown/exhaustion gates while preserving operator-actionable / stale-module / user-paused / error-unrecoverable suppression (FN-7884). diff --git a/AGENTS.md b/AGENTS.md index ad78d662fc..f56a473da4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -225,6 +225,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme - FN-7528: a deterministic, non-LLM post-task performance capture (`AgentReflectionService.captureTaskPerformance`) runs once per completed task and emits `reflection:captured` with ids/counts/outcomes-only metadata (`retryReworkCount?`, `filesTouchedCount?`, `packagesTouchedCount?`, `verificationFileScoped?`, `durationMs?`); never persists `verificationScopeReason` free-text or summary prose in run-audit. - FN-7787: `createResolvedAgentSession` enriches `session:runtime-resolved` with `noModelResolved: true` and `runtimeBuiltInFallbackModel` when a non-mock/non-test session reaches runtime creation without a complete provider/model pair; this is a visibility signal for runtime built-in fallback usage, not a fabricated model-resolution verdict. - FN-7835/FN-7844/FN-7859/FN-7878: durable-agent error-state recovery emits `agent:auto-recover-error-state` when either the heartbeat timer or the self-healing sweep clears a recoverable, non-operator-actionable `error` and retries; metadata stays ids/counts/outcomes-only (`agentId`, attempt, limit, source), where `source` is `timer`/`automation`/`self-healing`. Generic/unknown heartbeat failures are recoverable by default because manual Retry often proves they were transient; both entry paths share the `heartbeatErrorRecovery` budget (self-healing keeps `durableErrorRecovery` only for cooldown/stale-path bookkeeping) and emit `agent:error-retry-exhausted` when the shared budget is exhausted and the agent is parked `paused` with `pauseReason:"error-retry-exhausted"`. Only operator-actionable durable heartbeat errors (credentials/OAuth scope, model access, billing/quota, excluding transient auth rotation), plus stale worktree/module-resolution errors handled by their dedicated suppression path, skip the retry budget and emit `agent:error-parked-unrecoverable` with ids/counts/outcomes-only metadata (`agentId`, `source`, optional `attempts`, `limit`) before parking `paused` with `pauseReason:"error-unrecoverable"` for human repair. +- FN-7884: self-healing startup recovery emits `agent:reset-error-state-on-startup` when an engine restart clears an eligible durable-agent `error` or `pauseReason:"error-retry-exhausted"` park, resets shared `heartbeatErrorRecovery` plus legacy `durableErrorRecovery` budget/cooldown metadata, clears `lastError`/exhaustion pause state, and re-arms the heartbeat. Metadata stays ids/counts/outcomes-only (`agentId`, `priorState`, optional `priorPauseReason`, `source`). This startup-only path bypasses steady-state staleness/cooldown/exhaustion gates while preserving operator-actionable, stale-module, user-paused, `error-unrecoverable`, ephemeral, disabled-runtime, and active-execution suppression. - FN-7802: self-healing emits `task:reconcile-missing-worktree-merge-active` when it proves an `in-review` merge-active task (`merging`/`merging-pr`/`merging-fix`) is stranded by an unusable-worktree session-start failure, clears stale `worktree`/`branch`/`sessionFile`, resets the worktree-session retry budget, increments `recoveryRetryCount` as the bounded stale-metadata clear counter, and requeues to `todo`; it emits `task:reconcile-missing-worktree-merge-active-no-action` when `autoMerge:false`, workspace-task ownership, or triple-proof blocks the backward move. - FN-7863: executor emits `task:execution-dispatch-loop-terminalized` when an execute-node self-requeue loop reaches `MAX_EXECUTE_REQUEUE_LOOP_CYCLES` with an unchanged progress signature; metadata stays ids/counts/outcomes-only (`taskId`, `cycleCount`, `maxCycles`, `progressSignature`, `failureValue`) and the task is visibly failed with `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` while preserving worktree/branch/step progress. - FN-7011: self-healing emits `task:reconcile-engine-downtime-active-timing` when startup recovery shifts active task segment anchors to exclude proven engine-process downtime, and `task:reconcile-engine-downtime-active-timing-no-action` when no active task qualifies. diff --git a/docs/agents.md b/docs/agents.md index c97aacfcdf..d93c78c52f 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -362,14 +362,14 @@ Durable-agent error recovery is coordinated between the heartbeat timer path and - `lastError` is **not** operator-actionable (credentials/model/billing-style failures) - stale worktree/module-resolution failures remain suppressed instead of auto-restarted -Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details: +Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Engine startup adds a clean-slate reset before the steady-state sweep: eligible `state="error"` agents and durable agents parked with `pauseReason="error-retry-exhausted"` have `heartbeatErrorRecovery` and legacy `durableErrorRecovery` reset, `lastError`/exhaustion pause state cleared, and their heartbeat re-armed immediately. That restart-triggered reset intentionally bypasses the sweep's staleness, cooldown, and exhausted-budget gates because restarting the engine is treated like an operator Retry click. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details: - exponential cooldown (`30s` base, capped at `15m`) - persisted `attempts`, `lastAttemptAt`, `nextRetryAt`, `exhausted`, `lastReason`, and stale missing-module path counters On restart attempts, the runtime triggers the normal heartbeat pipeline with `source: "automation"` and a structured `contextSnapshot.selfHealing` payload so operators can audit recovery runs in heartbeat history. The sweep flips `error → active` before calling `executeHeartbeat`, so the heartbeat run does not re-enter run-entry error recovery or double-count the same recovery. -Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, cooldown windows, active execution, runtime-disabled agents, and ephemeral agents remain excluded from this parking/retry path. +Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, active execution, runtime-disabled agents, ephemeral agents, user-paused agents, and `error-unrecoverable` parks remain excluded from both the steady-state retry path and the startup clean-slate reset; cooldown windows and exhausted-budget gates are bypassed only by the startup reset for otherwise recoverable agents. **Manager presence does not gate this sweep (FN-7672/FN-7844):** eligibility for durable `state="error"` recovery does *not* depend on whether the agent's `reportsTo` manager is present/active. The timer path is now the fast path for heartbeat-managed error agents, while this recovery sweep remains the maintenance backstop for durable agents that are still stale in `error`; a present manager does not make the agent any less stuck. (A separate, unrelated `managerMissing` check still gates recovery of orphaned `state="running"` agents — a different failure mode where a live process's manager row was deleted.) FN-7672 root-caused a correlated 4-agent error cluster reporting to one active manager (a transient upstream auth/session blip) that could never have self-healed under the old manager-missing-only gate, even once the underlying cause resolved. diff --git a/docs/architecture.md b/docs/architecture.md index d9441f618b..f77c96f3e2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -676,7 +676,7 @@ Runtime action-gate flow (v1): - `StuckTaskDetector` (`stuck-task-detector.ts`) — inactivity/loop stall detection - `GridlockDetector` (`gridlock-detector.ts`) — detects all-blocked todo pipelines and emits notification events (plus explicit clear signals when gridlock resolves) - `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification -- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors are not restarted; both timer and sweep paths park them `paused` with `pauseReason:"error-unrecoverable"` and emit `agent:error-parked-unrecoverable`, while preserving exclusions for disabled runtime agents, ephemeral agents, active executions, and stale worktree/module-resolution suppression. +- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878/FN-7884): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. On engine startup, `SelfHealingManager.resetDurableAgentErrorStateOnStartup()` runs before the steady-state sweep and treats restart as an explicit operator retry: eligible `error` and `error-retry-exhausted` durable agents have shared/legacy retry metadata reset, `lastError` and the exhaustion pause cleared, state set to `active`, heartbeat re-armed, and `agent:reset-error-state-on-startup` emitted without applying the sweep's staleness/cooldown/exhaustion gates. Non-recoverable durable heartbeat errors are not restarted; timer, startup, and sweep paths preserve exclusions for disabled runtime agents, ephemeral agents, active executions, user pauses, `error-unrecoverable` parks, operator-actionable errors, and stale worktree/module-resolution suppression. - `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions - Batch 1 maintenance now includes `reconcile-orphaned-task-dirs` (FN-6783), a paused-safe housekeeping step that calls `TaskStore.reconcileOrphanedTaskDirs()` so valid live `.fusion/tasks/{ID}/task.json` records missing from the SQLite index become visible without waiting for process restart. The store-level guard skips any ID already present in active, soft-deleted, archived, or tombstoned storage and emits `task:reconcile-orphaned-task-dir` only for recovered rows. - Batch 1 maintenance also includes `reconcile-phantom-committed-reservations` (FN-7069), which calls `TaskStore.reconcilePhantomCommittedReservations()` for committed task-ID reservations that have no live/soft-deleted/archived task row and no `.fusion/tasks/{ID}/task.json`. The sweep prunes orphaned `activityLog` rows and `agents`/cascaded `agentRuns`, preserves `runAuditEvents`, and keeps the reservation `committed` per FN-5105 so the ID is permanently reserved rather than resurrected or handed out again. diff --git a/packages/engine/src/__tests__/self-healing.test.ts b/packages/engine/src/__tests__/self-healing.test.ts index a4a37643fd..dc339c9cf3 100644 --- a/packages/engine/src/__tests__/self-healing.test.ts +++ b/packages/engine/src/__tests__/self-healing.test.ts @@ -112,7 +112,7 @@ vi.mock("../merger.js", () => ({ })); import { SelfHealingManager, isBranchAheadOfBase, MAX_AUTO_MERGE_RETRIES } from "../self-healing.js"; -import { HEARTBEAT_ERROR_RECOVERY_METADATA_KEY, HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON } from "../agent-heartbeat.js"; +import { HEARTBEAT_ERROR_RECOVERY_METADATA_KEY, HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON, readHeartbeatErrorRetryCount } from "../agent-heartbeat.js"; import type { TaskStore, Settings, Task, AgentStore, Agent, NotificationProvider } from "@fusion/core"; import { EventEmitter } from "node:events"; import { execSync } from "node:child_process"; @@ -815,6 +815,7 @@ describe("SelfHealingManager", () => { const recoverPartialProgressNoTaskDoneFailures = vi.spyOn(manager, "recoverPartialProgressNoTaskDoneFailures").mockResolvedValue(1); const recoverOrphanedExecutions = vi.spyOn(manager, "recoverOrphanedExecutions").mockResolvedValue(1); const recoverApprovedTriageTasks = vi.spyOn(manager, "recoverApprovedTriageTasks").mockResolvedValue(1); + const resetDurableAgentErrorStateOnStartup = vi.spyOn(manager, "resetDurableAgentErrorStateOnStartup").mockResolvedValue(1); const recoverOrphanedAgents = vi.spyOn(manager, "recoverOrphanedAgents").mockResolvedValue(1); const recoverAgentsRunningOnInactiveTasks = vi.spyOn(manager, "recoverAgentsRunningOnInactiveTasks").mockResolvedValue(1); const clearStaleBlockedBy = vi.spyOn(manager, "clearStaleBlockedBy").mockResolvedValue(1); @@ -833,6 +834,7 @@ describe("SelfHealingManager", () => { expect(recoverPartialProgressNoTaskDoneFailures).toHaveBeenCalledTimes(1); expect(recoverOrphanedExecutions).toHaveBeenCalledTimes(1); expect(recoverApprovedTriageTasks).toHaveBeenCalledTimes(1); + expect(resetDurableAgentErrorStateOnStartup).toHaveBeenCalledTimes(1); expect(recoverOrphanedAgents).toHaveBeenCalledTimes(1); expect(recoverAgentsRunningOnInactiveTasks).toHaveBeenCalledTimes(1); expect(clearStaleBlockedBy).toHaveBeenCalledTimes(1); @@ -885,10 +887,12 @@ describe("SelfHealingManager", () => { enginePaused: true, } as unknown as Settings); const recoverCompletedTasks = vi.spyOn(manager, "recoverCompletedTasks").mockResolvedValue(1); + const resetDurableAgentErrorStateOnStartup = vi.spyOn(manager, "resetDurableAgentErrorStateOnStartup").mockResolvedValue(1); await manager.runStartupRecovery(); expect(recoverCompletedTasks).not.toHaveBeenCalled(); + expect(resetDurableAgentErrorStateOnStartup).not.toHaveBeenCalled(); }); it("runStartupRecovery skips while globalPause is active", async () => { @@ -907,6 +911,127 @@ describe("SelfHealingManager", () => { }); }); + describe("resetDurableAgentErrorStateOnStartup", () => { + function createStatefulMockAgentStore(agents: Agent[]): AgentStore & { getAgent(id: string): Agent | undefined } { + const agentMap = new Map(agents.map((agent) => [agent.id, { ...agent, metadata: agent.metadata ? { ...agent.metadata } : agent.metadata }])); + return { + getAgent: (id: string) => agentMap.get(id), + listAgents: vi.fn().mockImplementation(async (filter?: { state?: string }) => { + const values = Array.from(agentMap.values()); + return filter?.state ? values.filter((agent) => agent.state === filter.state) : values; + }), + updateAgentState: vi.fn().mockImplementation(async (id: string, state: Agent["state"]) => { + const agent = agentMap.get(id); + if (agent) { + agentMap.set(id, { ...agent, state }); + } + }), + updateAgent: vi.fn().mockImplementation(async (id: string, patch: Partial) => { + const agent = agentMap.get(id); + if (agent) { + agentMap.set(id, { ...agent, ...patch }); + } + }), + } as unknown as AgentStore & { getAgent(id: string): Agent | undefined }; + } + + it("returns 0 when no agentStore", async () => { + const result = await manager.resetDurableAgentErrorStateOnStartup(); + expect(result).toBe(0); + }); + + it("resets fresh error and exhausted parked agents on runStartupRecovery while preserving suppression guards", async () => { + const now = Date.now(); + const staleModuleError = "Error: Cannot find module '/tmp/fusion-old/node_modules/@fusion/engine/dist/index.js' imported from /tmp/fusion-old/packages/engine/src/agent.js"; + const agents = [ + { + id: "fresh-error", + state: "error", + lastError: "socket hang up", + updatedAt: new Date(now).toISOString(), + metadata: { + unrelated: "keep", + [HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]: { consecutiveAttempts: 3, nextRetryAt: new Date(now + 360_000).toISOString() }, + durableErrorRecovery: { attempts: 3, nextRetryAt: new Date(now + 360_000).toISOString(), exhausted: false }, + }, + } as unknown as Agent, + { + id: "exhausted-parked", + state: "paused", + pauseReason: HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, + lastError: "Failed to start agent session: spawn ENOENT", + updatedAt: new Date(now).toISOString(), + metadata: { + unrelated: "keep-too", + [HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]: { consecutiveAttempts: 5, updatedAt: new Date(now).toISOString() }, + durableErrorRecovery: { attempts: 5, exhausted: true, nextRetryAt: new Date(now + 600_000).toISOString() }, + }, + } as unknown as Agent, + { id: "operator-actionable", state: "error", lastError: "OAuth token does not meet scope requirements", updatedAt: new Date(now).toISOString(), metadata: { untouched: true } } as unknown as Agent, + { id: "stale-module", state: "error", lastError: staleModuleError, updatedAt: new Date(now).toISOString(), metadata: { untouched: true } } as unknown as Agent, + { id: "error-unrecoverable", state: "paused", pauseReason: HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON, lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "user-paused", state: "paused", pauseReason: "manual", lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "ephemeral", state: "error", lastError: "socket hang up", metadata: { agentKind: "task-worker" }, updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "disabled", state: "error", lastError: "socket hang up", runtimeConfig: { enabled: false }, updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "live-agent", state: "error", lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "healthy-active", state: "active", updatedAt: new Date(now).toISOString() } as unknown as Agent, + { id: "healthy-idle", state: "idle", updatedAt: new Date(now).toISOString() } as unknown as Agent, + ]; + const agentStore = createStatefulMockAgentStore(agents); + const restartDurableAgentHeartbeat = vi.fn().mockResolvedValue(true); + const recordRunAuditEvent = vi.fn().mockResolvedValue(undefined); + const storeWithSettings = createMockStore({ + getSettings: vi.fn().mockResolvedValue({ globalPause: false, enginePaused: false, taskStuckTimeoutMs: 60_000 } as unknown as Settings), + recordRunAuditEvent, + }); + const managerWithAgents = new SelfHealingManager(storeWithSettings, { + rootDir: "/tmp/test-project", + agentStore, + restartDurableAgentHeartbeat, + hasActiveAgentExecution: (agentId) => agentId === "live-agent", + }); + + await managerWithAgents.runStartupRecovery(); + + for (const agentId of ["fresh-error", "exhausted-parked"]) { + const agent = agentStore.getAgent(agentId)!; + expect(agent.state).toBe("active"); + expect(agent.lastError).toBeUndefined(); + expect(agent.pauseReason).toBeUndefined(); + expect(readHeartbeatErrorRetryCount(agent)).toBe(0); + expect(agent.metadata?.durableErrorRecovery).toBeUndefined(); + expect(agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]).toEqual(expect.objectContaining({ consecutiveAttempts: 0 })); + expect((agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY] as Record).nextRetryAt).toBeUndefined(); + expect((agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY] as Record).exhausted).toBeUndefined(); + } + expect(agentStore.getAgent("fresh-error")?.metadata?.unrelated).toBe("keep"); + expect(agentStore.getAgent("exhausted-parked")?.metadata?.unrelated).toBe("keep-too"); + expect(restartDurableAgentHeartbeat).toHaveBeenCalledTimes(2); + expect(restartDurableAgentHeartbeat).toHaveBeenCalledWith("fresh-error", { reason: "startup-error-reset", attempt: 1 }); + expect(restartDurableAgentHeartbeat).toHaveBeenCalledWith("exhausted-parked", { reason: "startup-error-reset", attempt: 1 }); + + const resetAudits = recordRunAuditEvent.mock.calls + .map(([event]) => event) + .filter((event) => event.mutationType === "agent:reset-error-state-on-startup"); + expect(resetAudits).toHaveLength(2); + expect(resetAudits).toEqual(expect.arrayContaining([ + expect.objectContaining({ target: "fresh-error", metadata: expect.objectContaining({ agentId: "fresh-error", priorState: "error", source: "self-healing" }) }), + expect.objectContaining({ target: "exhausted-parked", metadata: expect.objectContaining({ agentId: "exhausted-parked", priorState: "paused", priorPauseReason: HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, source: "self-healing" }) }), + ])); + expect(recordRunAuditEvent.mock.calls.map(([event]) => event.mutationType).filter((type) => type === "agent:auto-recover-error-state")).toHaveLength(0); + expect(agentStore.updateAgentState).toHaveBeenCalledTimes(2); + + for (const untouchedId of ["operator-actionable", "stale-module", "error-unrecoverable", "user-paused", "ephemeral", "disabled", "live-agent", "healthy-active", "healthy-idle"]) { + expect(agentStore.updateAgentState).not.toHaveBeenCalledWith(untouchedId, expect.anything()); + expect(agentStore.updateAgent).not.toHaveBeenCalledWith(untouchedId, expect.anything()); + } + expect(agentStore.getAgent("operator-actionable")?.lastError).toBe("OAuth token does not meet scope requirements"); + expect(agentStore.getAgent("stale-module")?.lastError).toBe(staleModuleError); + expect(agentStore.getAgent("error-unrecoverable")?.pauseReason).toBe(HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON); + managerWithAgents.stop(); + }); + }); + describe("recoverOrphanedAgents", () => { function createMockAgentStore(agents: Agent[]): AgentStore { return { diff --git a/packages/engine/src/run-audit.ts b/packages/engine/src/run-audit.ts index 0d7439cab4..724945a239 100644 --- a/packages/engine/src/run-audit.ts +++ b/packages/engine/src/run-audit.ts @@ -435,6 +435,7 @@ export type DatabaseMutationType = | "task:assign" | "task:checkout" | "agent:auto-recover-error-state" + | "agent:reset-error-state-on-startup" | "agent:error-retry-exhausted" | "agent:error-parked-unrecoverable" | "task:release" diff --git a/packages/engine/src/self-healing.ts b/packages/engine/src/self-healing.ts index 1c7a5f015d..8273c487ca 100644 --- a/packages/engine/src/self-healing.ts +++ b/packages/engine/src/self-healing.ts @@ -51,6 +51,7 @@ import { HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON, isHeartbeatErrorRecoverable, readHeartbeatErrorRetryCount, + resetHeartbeatErrorRecoveryMetadata, resolveErrorRecoveryLimit, } from "./agent-heartbeat.js"; import { classifyForeignOnlyContamination, deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js"; @@ -1386,6 +1387,7 @@ export class SelfHealingManager { { name: "approved-triage", fn: () => this.recoverApprovedTriageTasks().then(() => undefined) }, { name: "recover-starved-refinement", fn: () => this.recoverStarvedRefinementTriageTasks().then(() => undefined) }, { name: "orphaned-planning", fn: () => this.recoverOrphanedPlanningTasks().then(() => undefined) }, + { name: "reset-durable-agent-error-state-on-startup", fn: () => this.resetDurableAgentErrorStateOnStartup().then(() => undefined) }, { name: "recover-orphaned-agents", fn: () => this.recoverOrphanedAgents().then(() => undefined) }, { name: "recover-stale-heartbeat-runs", fn: () => this.recoverStaleHeartbeatRuns().then(() => undefined) }, { name: "reattach-orphaned-assigned-executions", fn: () => this.reattachOrphanedAssignedExecutions().then(() => undefined) }, @@ -10264,10 +10266,12 @@ export class SelfHealingManager { private async emitDurableAgentErrorRecoveryAudit(options: { agentId: string; - type: "agent:auto-recover-error-state" | "agent:error-retry-exhausted" | "agent:error-parked-unrecoverable"; + type: "agent:auto-recover-error-state" | "agent:reset-error-state-on-startup" | "agent:error-retry-exhausted" | "agent:error-parked-unrecoverable"; attempt?: number; attempts?: number; - limit: number; + limit?: number; + priorState?: Agent["state"]; + priorPauseReason?: string; source: "self-healing"; }): Promise { try { @@ -10283,7 +10287,9 @@ export class SelfHealingManager { agentId: options.agentId, ...(options.attempt !== undefined ? { attempt: options.attempt } : {}), ...(options.attempts !== undefined ? { attempts: options.attempts } : {}), - limit: options.limit, + ...(options.limit !== undefined ? { limit: options.limit } : {}), + ...(options.priorState !== undefined ? { priorState: options.priorState } : {}), + ...(options.priorPauseReason !== undefined ? { priorPauseReason: options.priorPauseReason } : {}), source: options.source, }, }); @@ -10456,6 +10462,82 @@ export class SelfHealingManager { return clearedAgentIds.size; } + /* + FNXC:AgentHeartbeat 2026-07-12-17:26: + FN-7884: Engine restart is an explicit operator retry boundary for durable heartbeat agents. Startup recovery must immediately clear recoverable `error` and `error-retry-exhausted` parks, reset shared heartbeatErrorRecovery/durableErrorRecovery budget state, and re-arm heartbeats without steady-state staleness/cooldown/exhaustion gates; operator-actionable, stale-module, user-paused, error-unrecoverable, disabled, ephemeral, and actively executing agents remain suppressed. + */ + async resetDurableAgentErrorStateOnStartup(): Promise { + const agentStore = this.options.agentStore; + if (!agentStore) { + return 0; + } + + let resetCount = 0; + try { + const allAgents = await agentStore.listAgents({ includeEphemeral: true }); + for (const agent of allAgents) { + const isErrorRetryExhaustedPark = + agent.state === "paused" && agent.pauseReason === HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON; + if (agent.state !== "error" && !isErrorRetryExhaustedPark) { + continue; + } + if (isEphemeralAgent(agent)) { + continue; + } + const runtimeConfig = (agent.runtimeConfig ?? {}) as Record; + if (runtimeConfig.enabled === false) { + continue; + } + if (this.options.hasActiveAgentExecution?.(agent.id) === true) { + continue; + } + if (!isHeartbeatErrorRecoverable(agent) || isStaleWorktreeModuleResolutionError(agent.lastError ?? "")) { + log.warn(`Startup durable-agent error reset suppressed for ${agent.id}: unrecoverable or stale-module error requires existing recovery path`); + continue; + } + + const priorState = agent.state; + const priorPauseReason = agent.pauseReason; + const resetMetadata = resetHeartbeatErrorRecoveryMetadata(agent); + try { + await agentStore.updateAgentState(agent.id, "active"); + await agentStore.updateAgent(agent.id, { + lastError: undefined, + pauseReason: undefined, + metadata: resetMetadata, + }); + await this.emitDurableAgentErrorRecoveryAudit({ + agentId: agent.id, + type: "agent:reset-error-state-on-startup", + priorState, + ...(priorPauseReason ? { priorPauseReason } : {}), + source: "self-healing", + }); + if (!this.options.restartDurableAgentHeartbeat) { + log.log(`Durable-agent startup error reset heartbeat restart unavailable for ${agent.id}; state reset only`); + } else { + const restartOk = await this.options.restartDurableAgentHeartbeat(agent.id, { + reason: "startup-error-reset", + attempt: 1, + }); + if (!restartOk) { + log.warn(`Durable-agent startup error reset heartbeat restart skipped for ${agent.id}`); + } + } + resetCount++; + log.log(`Startup reset durable-agent error state for ${agent.id}; heartbeat re-armed when available`); + } catch (error) { + log.warn(`Failed to reset durable-agent error state on startup for ${agent.id}: ${error instanceof Error ? error.message : String(error)}`); + } + } + } catch (error) { + log.warn(`Startup durable-agent error reset failed: ${error instanceof Error ? error.message : String(error)}`); + return resetCount; + } + + return resetCount; + } + async recoverOrphanedAgents(): Promise { const agentStore = this.options.agentStore; if (!agentStore) {