FN-7884: reset durable-agent error state on engine restart
Engine startup now treats itself as an implicit operator retry for durable heartbeat agents stuck in error, clearing eligible error states and re-arming heartbeats instead of waiting for the steady-state sweep's cooldown/exhaustion gates. - Add SelfHealingManager.resetDurableAgentErrorStateOnStartup(), run first in runStartupRecovery(), which resets shared heartbeatErrorRecovery/legacy durableErrorRecovery metadata, clears lastError/pauseReason, flips eligible error and error-retry-exhausted-parked durable agents to active, and re-arms their heartbeat - Preserve suppression for operator-actionable, stale worktree/module-resolution, user-paused, error-unrecoverable, ephemeral, disabled-runtime, and actively-executing agents - Add agent:reset-error-state-on-startup run-audit mutation type with ids/counts/outcomes-only metadata (agentId, priorState, priorPauseReason, source) - Add changeset FN-7884 (patch) documenting the operator-facing behavior - Update AGENTS.md and docs/agents.md, docs/architecture.md to describe the new startup reset path alongside existing FN-7835/FN-7844/FN-7859/FN-7878 recovery docs - Extend self-healing.test.ts with coverage for the new startup reset behavior and its exclusions Files changed: .changeset/fn-7884-restart-error-reset.md | 7 ++ AGENTS.md | 1 + docs/agents.md | 4 +- docs/architecture.md | 2 +- packages/engine/src/__tests__/self-healing.test.ts | 127 ++++++++++++++++++++- packages/engine/src/run-audit.ts | 1 + packages/engine/src/self-healing.ts | 88 +++++++++++++- 7 files changed, 223 insertions(+), 7 deletions(-) Fusion-Task-Id: FN-7884 Fusion-Task-Lineage: fe64f6af-3ff3-4876-8308-8a75591c45f1 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-7884-restart-error-reset.md
Normal file
7
.changeset/fn-7884-restart-error-reset.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Durable agents in error state are cleared and retried automatically on engine restart.
|
||||
category: fix
|
||||
dev: New SelfHealingManager.resetDurableAgentErrorStateOnStartup() runs first in runStartupRecovery(): it resets the shared heartbeatErrorRecovery/durableErrorRecovery budget+cooldown, clears lastError, flips eligible error and error-retry-exhausted-parked durable agents to active, re-arms the heartbeat, and emits agent:reset-error-state-on-startup — bypassing the steady-state staleness/cooldown/exhaustion gates while preserving operator-actionable / stale-module / user-paused / error-unrecoverable suppression (FN-7884).
|
||||
@@ -225,6 +225,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
- FN-7528: a deterministic, non-LLM post-task performance capture (`AgentReflectionService.captureTaskPerformance`) runs once per completed task and emits `reflection:captured` with ids/counts/outcomes-only metadata (`retryReworkCount?`, `filesTouchedCount?`, `packagesTouchedCount?`, `verificationFileScoped?`, `durationMs?`); never persists `verificationScopeReason` free-text or summary prose in run-audit.
|
||||
- FN-7787: `createResolvedAgentSession` enriches `session:runtime-resolved` with `noModelResolved: true` and `runtimeBuiltInFallbackModel` when a non-mock/non-test session reaches runtime creation without a complete provider/model pair; this is a visibility signal for runtime built-in fallback usage, not a fabricated model-resolution verdict.
|
||||
- FN-7835/FN-7844/FN-7859/FN-7878: durable-agent error-state recovery emits `agent:auto-recover-error-state` when either the heartbeat timer or the self-healing sweep clears a recoverable, non-operator-actionable `error` and retries; metadata stays ids/counts/outcomes-only (`agentId`, attempt, limit, source), where `source` is `timer`/`automation`/`self-healing`. Generic/unknown heartbeat failures are recoverable by default because manual Retry often proves they were transient; both entry paths share the `heartbeatErrorRecovery` budget (self-healing keeps `durableErrorRecovery` only for cooldown/stale-path bookkeeping) and emit `agent:error-retry-exhausted` when the shared budget is exhausted and the agent is parked `paused` with `pauseReason:"error-retry-exhausted"`. Only operator-actionable durable heartbeat errors (credentials/OAuth scope, model access, billing/quota, excluding transient auth rotation), plus stale worktree/module-resolution errors handled by their dedicated suppression path, skip the retry budget and emit `agent:error-parked-unrecoverable` with ids/counts/outcomes-only metadata (`agentId`, `source`, optional `attempts`, `limit`) before parking `paused` with `pauseReason:"error-unrecoverable"` for human repair.
|
||||
- FN-7884: self-healing startup recovery emits `agent:reset-error-state-on-startup` when an engine restart clears an eligible durable-agent `error` or `pauseReason:"error-retry-exhausted"` park, resets shared `heartbeatErrorRecovery` plus legacy `durableErrorRecovery` budget/cooldown metadata, clears `lastError`/exhaustion pause state, and re-arms the heartbeat. Metadata stays ids/counts/outcomes-only (`agentId`, `priorState`, optional `priorPauseReason`, `source`). This startup-only path bypasses steady-state staleness/cooldown/exhaustion gates while preserving operator-actionable, stale-module, user-paused, `error-unrecoverable`, ephemeral, disabled-runtime, and active-execution suppression.
|
||||
- FN-7802: self-healing emits `task:reconcile-missing-worktree-merge-active` when it proves an `in-review` merge-active task (`merging`/`merging-pr`/`merging-fix`) is stranded by an unusable-worktree session-start failure, clears stale `worktree`/`branch`/`sessionFile`, resets the worktree-session retry budget, increments `recoveryRetryCount` as the bounded stale-metadata clear counter, and requeues to `todo`; it emits `task:reconcile-missing-worktree-merge-active-no-action` when `autoMerge:false`, workspace-task ownership, or triple-proof blocks the backward move.
|
||||
- FN-7863: executor emits `task:execution-dispatch-loop-terminalized` when an execute-node self-requeue loop reaches `MAX_EXECUTE_REQUEUE_LOOP_CYCLES` with an unchanged progress signature; metadata stays ids/counts/outcomes-only (`taskId`, `cycleCount`, `maxCycles`, `progressSignature`, `failureValue`) and the task is visibly failed with `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` while preserving worktree/branch/step progress.
|
||||
- FN-7011: self-healing emits `task:reconcile-engine-downtime-active-timing` when startup recovery shifts active task segment anchors to exclude proven engine-process downtime, and `task:reconcile-engine-downtime-active-timing-no-action` when no active task qualifies.
|
||||
|
||||
@@ -362,14 +362,14 @@ Durable-agent error recovery is coordinated between the heartbeat timer path and
|
||||
- `lastError` is **not** operator-actionable (credentials/model/billing-style failures)
|
||||
- stale worktree/module-resolution failures remain suppressed instead of auto-restarted
|
||||
|
||||
Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details:
|
||||
Both paths use the same persisted retry budget, `agent.metadata.heartbeatErrorRecovery.consecutiveAttempts`, with the default cap of `5` attempts (settings-overridable through `heartbeatErrorRecoveryAttempts`). The timer path provides fast recovery on the agent's own interval; the self-healing sweep is the backstop for stale `error` agents whose timer was lost, delayed, or did not re-tick. Engine startup adds a clean-slate reset before the steady-state sweep: eligible `state="error"` agents and durable agents parked with `pauseReason="error-retry-exhausted"` have `heartbeatErrorRecovery` and legacy `durableErrorRecovery` reset, `lastError`/exhaustion pause state cleared, and their heartbeat re-armed immediately. That restart-triggered reset intentionally bypasses the sweep's staleness, cooldown, and exhausted-budget gates because restarting the engine is treated like an operator Retry click. Self-healing still persists `agent.metadata.durableErrorRecovery` for sweep-specific cooldown and stale-path details:
|
||||
|
||||
- exponential cooldown (`30s` base, capped at `15m`)
|
||||
- persisted `attempts`, `lastAttemptAt`, `nextRetryAt`, `exhausted`, `lastReason`, and stale missing-module path counters
|
||||
|
||||
On restart attempts, the runtime triggers the normal heartbeat pipeline with `source: "automation"` and a structured `contextSnapshot.selfHealing` payload so operators can audit recovery runs in heartbeat history. The sweep flips `error → active` before calling `executeHeartbeat`, so the heartbeat run does not re-enter run-entry error recovery or double-count the same recovery.
|
||||
|
||||
Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, cooldown windows, active execution, runtime-disabled agents, and ephemeral agents remain excluded from this parking/retry path.
|
||||
Self-healing intentionally refuses to auto-restart agents when blockers are operator-actionable or non-transient. Runtime-enabled durable agents in that terminal bucket are parked `paused` with `pauseReason="error-unrecoverable"` so operators see that credential/model/configuration repair is required; transient retry-budget exhaustion still parks with `pauseReason="error-retry-exhausted"`. Stale worktree/module-resolution suppression, active execution, runtime-disabled agents, ephemeral agents, user-paused agents, and `error-unrecoverable` parks remain excluded from both the steady-state retry path and the startup clean-slate reset; cooldown windows and exhausted-budget gates are bypassed only by the startup reset for otherwise recoverable agents.
|
||||
|
||||
**Manager presence does not gate this sweep (FN-7672/FN-7844):** eligibility for durable `state="error"` recovery does *not* depend on whether the agent's `reportsTo` manager is present/active. The timer path is now the fast path for heartbeat-managed error agents, while this recovery sweep remains the maintenance backstop for durable agents that are still stale in `error`; a present manager does not make the agent any less stuck. (A separate, unrelated `managerMissing` check still gates recovery of orphaned `state="running"` agents — a different failure mode where a live process's manager row was deleted.) FN-7672 root-caused a correlated 4-agent error cluster reporting to one active manager (a transient upstream auth/session blip) that could never have self-healed under the old manager-missing-only gate, even once the underlying cause resolved.
|
||||
|
||||
|
||||
@@ -676,7 +676,7 @@ Runtime action-gate flow (v1):
|
||||
- `StuckTaskDetector` (`stuck-task-detector.ts`) — inactivity/loop stall detection
|
||||
- `GridlockDetector` (`gridlock-detector.ts`) — detects all-blocked todo pipelines and emits notification events (plus explicit clear signals when gridlock resolves)
|
||||
- `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
|
||||
- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors are not restarted; both timer and sweep paths park them `paused` with `pauseReason:"error-unrecoverable"` and emit `agent:error-parked-unrecoverable`, while preserving exclusions for disabled runtime agents, ephemeral agents, active executions, and stale worktree/module-resolution suppression.
|
||||
- Durable agent error recovery (FN-7835/FN-7844/FN-7859/FN-7878/FN-7884): a heartbeat-managed, runtime-enabled non-ephemeral agent that lands in `state:"error"` remains timer-eligible and clears `lastError` by transitioning `error → active` at the next heartbeat run entry when `lastError` is recoverable. Generic/unknown errors are recoverable by default; immediate `error-unrecoverable` parking is reserved for operator-actionable auth/model/billing/quota failures, while stale worktree/module-resolution errors stay on their dedicated self-healing suppression/rebuild path. Recovery is bounded by one shared `heartbeatErrorRecovery` attempt budget (`MAX_HEARTBEAT_ERROR_RECOVERY_ATTEMPTS`, settings-overridable through the engine's optional cast-based knob) across both the timer path and `SelfHealingManager.recoverOrphanedAgents()`. Self-healing is the stale-agent backstop and still stores `durableErrorRecovery` cooldown/stale-module metadata, but it writes/reads the shared heartbeat counter and emits the same `agent:auto-recover-error-state` / `agent:error-retry-exhausted` audit surface with `source:"self-healing"`. The sweep flips `error → active` before `restartDurableAgentHeartbeat()` calls `executeHeartbeat()`, preventing run-entry recovery from re-counting or double-emitting for the same recovery. Success resets the shared counter and clears legacy sweep retry state; budget exhaustion parks the agent `paused` with `pauseReason:"error-retry-exhausted"`. On engine startup, `SelfHealingManager.resetDurableAgentErrorStateOnStartup()` runs before the steady-state sweep and treats restart as an explicit operator retry: eligible `error` and `error-retry-exhausted` durable agents have shared/legacy retry metadata reset, `lastError` and the exhaustion pause cleared, state set to `active`, heartbeat re-armed, and `agent:reset-error-state-on-startup` emitted without applying the sweep's staleness/cooldown/exhaustion gates. Non-recoverable durable heartbeat errors are not restarted; timer, startup, and sweep paths preserve exclusions for disabled runtime agents, ephemeral agents, active executions, user pauses, `error-unrecoverable` parks, operator-actionable errors, and stale worktree/module-resolution suppression.
|
||||
- `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
|
||||
- Batch 1 maintenance now includes `reconcile-orphaned-task-dirs` (FN-6783), a paused-safe housekeeping step that calls `TaskStore.reconcileOrphanedTaskDirs()` so valid live `.fusion/tasks/{ID}/task.json` records missing from the SQLite index become visible without waiting for process restart. The store-level guard skips any ID already present in active, soft-deleted, archived, or tombstoned storage and emits `task:reconcile-orphaned-task-dir` only for recovered rows.
|
||||
- Batch 1 maintenance also includes `reconcile-phantom-committed-reservations` (FN-7069), which calls `TaskStore.reconcilePhantomCommittedReservations()` for committed task-ID reservations that have no live/soft-deleted/archived task row and no `.fusion/tasks/{ID}/task.json`. The sweep prunes orphaned `activityLog` rows and `agents`/cascaded `agentRuns`, preserves `runAuditEvents`, and keeps the reservation `committed` per FN-5105 so the ID is permanently reserved rather than resurrected or handed out again.
|
||||
|
||||
@@ -112,7 +112,7 @@ vi.mock("../merger.js", () => ({
|
||||
}));
|
||||
|
||||
import { SelfHealingManager, isBranchAheadOfBase, MAX_AUTO_MERGE_RETRIES } from "../self-healing.js";
|
||||
import { HEARTBEAT_ERROR_RECOVERY_METADATA_KEY, HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON } from "../agent-heartbeat.js";
|
||||
import { HEARTBEAT_ERROR_RECOVERY_METADATA_KEY, HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON, readHeartbeatErrorRetryCount } from "../agent-heartbeat.js";
|
||||
import type { TaskStore, Settings, Task, AgentStore, Agent, NotificationProvider } from "@fusion/core";
|
||||
import { EventEmitter } from "node:events";
|
||||
import { execSync } from "node:child_process";
|
||||
@@ -815,6 +815,7 @@ describe("SelfHealingManager", () => {
|
||||
const recoverPartialProgressNoTaskDoneFailures = vi.spyOn(manager, "recoverPartialProgressNoTaskDoneFailures").mockResolvedValue(1);
|
||||
const recoverOrphanedExecutions = vi.spyOn(manager, "recoverOrphanedExecutions").mockResolvedValue(1);
|
||||
const recoverApprovedTriageTasks = vi.spyOn(manager, "recoverApprovedTriageTasks").mockResolvedValue(1);
|
||||
const resetDurableAgentErrorStateOnStartup = vi.spyOn(manager, "resetDurableAgentErrorStateOnStartup").mockResolvedValue(1);
|
||||
const recoverOrphanedAgents = vi.spyOn(manager, "recoverOrphanedAgents").mockResolvedValue(1);
|
||||
const recoverAgentsRunningOnInactiveTasks = vi.spyOn(manager, "recoverAgentsRunningOnInactiveTasks").mockResolvedValue(1);
|
||||
const clearStaleBlockedBy = vi.spyOn(manager, "clearStaleBlockedBy").mockResolvedValue(1);
|
||||
@@ -833,6 +834,7 @@ describe("SelfHealingManager", () => {
|
||||
expect(recoverPartialProgressNoTaskDoneFailures).toHaveBeenCalledTimes(1);
|
||||
expect(recoverOrphanedExecutions).toHaveBeenCalledTimes(1);
|
||||
expect(recoverApprovedTriageTasks).toHaveBeenCalledTimes(1);
|
||||
expect(resetDurableAgentErrorStateOnStartup).toHaveBeenCalledTimes(1);
|
||||
expect(recoverOrphanedAgents).toHaveBeenCalledTimes(1);
|
||||
expect(recoverAgentsRunningOnInactiveTasks).toHaveBeenCalledTimes(1);
|
||||
expect(clearStaleBlockedBy).toHaveBeenCalledTimes(1);
|
||||
@@ -885,10 +887,12 @@ describe("SelfHealingManager", () => {
|
||||
enginePaused: true,
|
||||
} as unknown as Settings);
|
||||
const recoverCompletedTasks = vi.spyOn(manager, "recoverCompletedTasks").mockResolvedValue(1);
|
||||
const resetDurableAgentErrorStateOnStartup = vi.spyOn(manager, "resetDurableAgentErrorStateOnStartup").mockResolvedValue(1);
|
||||
|
||||
await manager.runStartupRecovery();
|
||||
|
||||
expect(recoverCompletedTasks).not.toHaveBeenCalled();
|
||||
expect(resetDurableAgentErrorStateOnStartup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("runStartupRecovery skips while globalPause is active", async () => {
|
||||
@@ -907,6 +911,127 @@ describe("SelfHealingManager", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("resetDurableAgentErrorStateOnStartup", () => {
|
||||
function createStatefulMockAgentStore(agents: Agent[]): AgentStore & { getAgent(id: string): Agent | undefined } {
|
||||
const agentMap = new Map<string, Agent>(agents.map((agent) => [agent.id, { ...agent, metadata: agent.metadata ? { ...agent.metadata } : agent.metadata }]));
|
||||
return {
|
||||
getAgent: (id: string) => agentMap.get(id),
|
||||
listAgents: vi.fn().mockImplementation(async (filter?: { state?: string }) => {
|
||||
const values = Array.from(agentMap.values());
|
||||
return filter?.state ? values.filter((agent) => agent.state === filter.state) : values;
|
||||
}),
|
||||
updateAgentState: vi.fn().mockImplementation(async (id: string, state: Agent["state"]) => {
|
||||
const agent = agentMap.get(id);
|
||||
if (agent) {
|
||||
agentMap.set(id, { ...agent, state });
|
||||
}
|
||||
}),
|
||||
updateAgent: vi.fn().mockImplementation(async (id: string, patch: Partial<Agent>) => {
|
||||
const agent = agentMap.get(id);
|
||||
if (agent) {
|
||||
agentMap.set(id, { ...agent, ...patch });
|
||||
}
|
||||
}),
|
||||
} as unknown as AgentStore & { getAgent(id: string): Agent | undefined };
|
||||
}
|
||||
|
||||
it("returns 0 when no agentStore", async () => {
|
||||
const result = await manager.resetDurableAgentErrorStateOnStartup();
|
||||
expect(result).toBe(0);
|
||||
});
|
||||
|
||||
it("resets fresh error and exhausted parked agents on runStartupRecovery while preserving suppression guards", async () => {
|
||||
const now = Date.now();
|
||||
const staleModuleError = "Error: Cannot find module '/tmp/fusion-old/node_modules/@fusion/engine/dist/index.js' imported from /tmp/fusion-old/packages/engine/src/agent.js";
|
||||
const agents = [
|
||||
{
|
||||
id: "fresh-error",
|
||||
state: "error",
|
||||
lastError: "socket hang up",
|
||||
updatedAt: new Date(now).toISOString(),
|
||||
metadata: {
|
||||
unrelated: "keep",
|
||||
[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]: { consecutiveAttempts: 3, nextRetryAt: new Date(now + 360_000).toISOString() },
|
||||
durableErrorRecovery: { attempts: 3, nextRetryAt: new Date(now + 360_000).toISOString(), exhausted: false },
|
||||
},
|
||||
} as unknown as Agent,
|
||||
{
|
||||
id: "exhausted-parked",
|
||||
state: "paused",
|
||||
pauseReason: HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON,
|
||||
lastError: "Failed to start agent session: spawn ENOENT",
|
||||
updatedAt: new Date(now).toISOString(),
|
||||
metadata: {
|
||||
unrelated: "keep-too",
|
||||
[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]: { consecutiveAttempts: 5, updatedAt: new Date(now).toISOString() },
|
||||
durableErrorRecovery: { attempts: 5, exhausted: true, nextRetryAt: new Date(now + 600_000).toISOString() },
|
||||
},
|
||||
} as unknown as Agent,
|
||||
{ id: "operator-actionable", state: "error", lastError: "OAuth token does not meet scope requirements", updatedAt: new Date(now).toISOString(), metadata: { untouched: true } } as unknown as Agent,
|
||||
{ id: "stale-module", state: "error", lastError: staleModuleError, updatedAt: new Date(now).toISOString(), metadata: { untouched: true } } as unknown as Agent,
|
||||
{ id: "error-unrecoverable", state: "paused", pauseReason: HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON, lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "user-paused", state: "paused", pauseReason: "manual", lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "ephemeral", state: "error", lastError: "socket hang up", metadata: { agentKind: "task-worker" }, updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "disabled", state: "error", lastError: "socket hang up", runtimeConfig: { enabled: false }, updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "live-agent", state: "error", lastError: "socket hang up", updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "healthy-active", state: "active", updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
{ id: "healthy-idle", state: "idle", updatedAt: new Date(now).toISOString() } as unknown as Agent,
|
||||
];
|
||||
const agentStore = createStatefulMockAgentStore(agents);
|
||||
const restartDurableAgentHeartbeat = vi.fn().mockResolvedValue(true);
|
||||
const recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
|
||||
const storeWithSettings = createMockStore({
|
||||
getSettings: vi.fn().mockResolvedValue({ globalPause: false, enginePaused: false, taskStuckTimeoutMs: 60_000 } as unknown as Settings),
|
||||
recordRunAuditEvent,
|
||||
});
|
||||
const managerWithAgents = new SelfHealingManager(storeWithSettings, {
|
||||
rootDir: "/tmp/test-project",
|
||||
agentStore,
|
||||
restartDurableAgentHeartbeat,
|
||||
hasActiveAgentExecution: (agentId) => agentId === "live-agent",
|
||||
});
|
||||
|
||||
await managerWithAgents.runStartupRecovery();
|
||||
|
||||
for (const agentId of ["fresh-error", "exhausted-parked"]) {
|
||||
const agent = agentStore.getAgent(agentId)!;
|
||||
expect(agent.state).toBe("active");
|
||||
expect(agent.lastError).toBeUndefined();
|
||||
expect(agent.pauseReason).toBeUndefined();
|
||||
expect(readHeartbeatErrorRetryCount(agent)).toBe(0);
|
||||
expect(agent.metadata?.durableErrorRecovery).toBeUndefined();
|
||||
expect(agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY]).toEqual(expect.objectContaining({ consecutiveAttempts: 0 }));
|
||||
expect((agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY] as Record<string, unknown>).nextRetryAt).toBeUndefined();
|
||||
expect((agent.metadata?.[HEARTBEAT_ERROR_RECOVERY_METADATA_KEY] as Record<string, unknown>).exhausted).toBeUndefined();
|
||||
}
|
||||
expect(agentStore.getAgent("fresh-error")?.metadata?.unrelated).toBe("keep");
|
||||
expect(agentStore.getAgent("exhausted-parked")?.metadata?.unrelated).toBe("keep-too");
|
||||
expect(restartDurableAgentHeartbeat).toHaveBeenCalledTimes(2);
|
||||
expect(restartDurableAgentHeartbeat).toHaveBeenCalledWith("fresh-error", { reason: "startup-error-reset", attempt: 1 });
|
||||
expect(restartDurableAgentHeartbeat).toHaveBeenCalledWith("exhausted-parked", { reason: "startup-error-reset", attempt: 1 });
|
||||
|
||||
const resetAudits = recordRunAuditEvent.mock.calls
|
||||
.map(([event]) => event)
|
||||
.filter((event) => event.mutationType === "agent:reset-error-state-on-startup");
|
||||
expect(resetAudits).toHaveLength(2);
|
||||
expect(resetAudits).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ target: "fresh-error", metadata: expect.objectContaining({ agentId: "fresh-error", priorState: "error", source: "self-healing" }) }),
|
||||
expect.objectContaining({ target: "exhausted-parked", metadata: expect.objectContaining({ agentId: "exhausted-parked", priorState: "paused", priorPauseReason: HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON, source: "self-healing" }) }),
|
||||
]));
|
||||
expect(recordRunAuditEvent.mock.calls.map(([event]) => event.mutationType).filter((type) => type === "agent:auto-recover-error-state")).toHaveLength(0);
|
||||
expect(agentStore.updateAgentState).toHaveBeenCalledTimes(2);
|
||||
|
||||
for (const untouchedId of ["operator-actionable", "stale-module", "error-unrecoverable", "user-paused", "ephemeral", "disabled", "live-agent", "healthy-active", "healthy-idle"]) {
|
||||
expect(agentStore.updateAgentState).not.toHaveBeenCalledWith(untouchedId, expect.anything());
|
||||
expect(agentStore.updateAgent).not.toHaveBeenCalledWith(untouchedId, expect.anything());
|
||||
}
|
||||
expect(agentStore.getAgent("operator-actionable")?.lastError).toBe("OAuth token does not meet scope requirements");
|
||||
expect(agentStore.getAgent("stale-module")?.lastError).toBe(staleModuleError);
|
||||
expect(agentStore.getAgent("error-unrecoverable")?.pauseReason).toBe(HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON);
|
||||
managerWithAgents.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("recoverOrphanedAgents", () => {
|
||||
function createMockAgentStore(agents: Agent[]): AgentStore {
|
||||
return {
|
||||
|
||||
@@ -435,6 +435,7 @@ export type DatabaseMutationType =
|
||||
| "task:assign"
|
||||
| "task:checkout"
|
||||
| "agent:auto-recover-error-state"
|
||||
| "agent:reset-error-state-on-startup"
|
||||
| "agent:error-retry-exhausted"
|
||||
| "agent:error-parked-unrecoverable"
|
||||
| "task:release"
|
||||
|
||||
@@ -51,6 +51,7 @@ import {
|
||||
HEARTBEAT_ERROR_UNRECOVERABLE_PAUSE_REASON,
|
||||
isHeartbeatErrorRecoverable,
|
||||
readHeartbeatErrorRetryCount,
|
||||
resetHeartbeatErrorRecoveryMetadata,
|
||||
resolveErrorRecoveryLimit,
|
||||
} from "./agent-heartbeat.js";
|
||||
import { classifyForeignOnlyContamination, deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js";
|
||||
@@ -1386,6 +1387,7 @@ export class SelfHealingManager {
|
||||
{ name: "approved-triage", fn: () => this.recoverApprovedTriageTasks().then(() => undefined) },
|
||||
{ name: "recover-starved-refinement", fn: () => this.recoverStarvedRefinementTriageTasks().then(() => undefined) },
|
||||
{ name: "orphaned-planning", fn: () => this.recoverOrphanedPlanningTasks().then(() => undefined) },
|
||||
{ name: "reset-durable-agent-error-state-on-startup", fn: () => this.resetDurableAgentErrorStateOnStartup().then(() => undefined) },
|
||||
{ name: "recover-orphaned-agents", fn: () => this.recoverOrphanedAgents().then(() => undefined) },
|
||||
{ name: "recover-stale-heartbeat-runs", fn: () => this.recoverStaleHeartbeatRuns().then(() => undefined) },
|
||||
{ name: "reattach-orphaned-assigned-executions", fn: () => this.reattachOrphanedAssignedExecutions().then(() => undefined) },
|
||||
@@ -10264,10 +10266,12 @@ export class SelfHealingManager {
|
||||
|
||||
private async emitDurableAgentErrorRecoveryAudit(options: {
|
||||
agentId: string;
|
||||
type: "agent:auto-recover-error-state" | "agent:error-retry-exhausted" | "agent:error-parked-unrecoverable";
|
||||
type: "agent:auto-recover-error-state" | "agent:reset-error-state-on-startup" | "agent:error-retry-exhausted" | "agent:error-parked-unrecoverable";
|
||||
attempt?: number;
|
||||
attempts?: number;
|
||||
limit: number;
|
||||
limit?: number;
|
||||
priorState?: Agent["state"];
|
||||
priorPauseReason?: string;
|
||||
source: "self-healing";
|
||||
}): Promise<void> {
|
||||
try {
|
||||
@@ -10283,7 +10287,9 @@ export class SelfHealingManager {
|
||||
agentId: options.agentId,
|
||||
...(options.attempt !== undefined ? { attempt: options.attempt } : {}),
|
||||
...(options.attempts !== undefined ? { attempts: options.attempts } : {}),
|
||||
limit: options.limit,
|
||||
...(options.limit !== undefined ? { limit: options.limit } : {}),
|
||||
...(options.priorState !== undefined ? { priorState: options.priorState } : {}),
|
||||
...(options.priorPauseReason !== undefined ? { priorPauseReason: options.priorPauseReason } : {}),
|
||||
source: options.source,
|
||||
},
|
||||
});
|
||||
@@ -10456,6 +10462,82 @@ export class SelfHealingManager {
|
||||
return clearedAgentIds.size;
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:AgentHeartbeat 2026-07-12-17:26:
|
||||
FN-7884: Engine restart is an explicit operator retry boundary for durable heartbeat agents. Startup recovery must immediately clear recoverable `error` and `error-retry-exhausted` parks, reset shared heartbeatErrorRecovery/durableErrorRecovery budget state, and re-arm heartbeats without steady-state staleness/cooldown/exhaustion gates; operator-actionable, stale-module, user-paused, error-unrecoverable, disabled, ephemeral, and actively executing agents remain suppressed.
|
||||
*/
|
||||
async resetDurableAgentErrorStateOnStartup(): Promise<number> {
|
||||
const agentStore = this.options.agentStore;
|
||||
if (!agentStore) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
let resetCount = 0;
|
||||
try {
|
||||
const allAgents = await agentStore.listAgents({ includeEphemeral: true });
|
||||
for (const agent of allAgents) {
|
||||
const isErrorRetryExhaustedPark =
|
||||
agent.state === "paused" && agent.pauseReason === HEARTBEAT_ERROR_RETRY_EXHAUSTED_PAUSE_REASON;
|
||||
if (agent.state !== "error" && !isErrorRetryExhaustedPark) {
|
||||
continue;
|
||||
}
|
||||
if (isEphemeralAgent(agent)) {
|
||||
continue;
|
||||
}
|
||||
const runtimeConfig = (agent.runtimeConfig ?? {}) as Record<string, unknown>;
|
||||
if (runtimeConfig.enabled === false) {
|
||||
continue;
|
||||
}
|
||||
if (this.options.hasActiveAgentExecution?.(agent.id) === true) {
|
||||
continue;
|
||||
}
|
||||
if (!isHeartbeatErrorRecoverable(agent) || isStaleWorktreeModuleResolutionError(agent.lastError ?? "")) {
|
||||
log.warn(`Startup durable-agent error reset suppressed for ${agent.id}: unrecoverable or stale-module error requires existing recovery path`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const priorState = agent.state;
|
||||
const priorPauseReason = agent.pauseReason;
|
||||
const resetMetadata = resetHeartbeatErrorRecoveryMetadata(agent);
|
||||
try {
|
||||
await agentStore.updateAgentState(agent.id, "active");
|
||||
await agentStore.updateAgent(agent.id, {
|
||||
lastError: undefined,
|
||||
pauseReason: undefined,
|
||||
metadata: resetMetadata,
|
||||
});
|
||||
await this.emitDurableAgentErrorRecoveryAudit({
|
||||
agentId: agent.id,
|
||||
type: "agent:reset-error-state-on-startup",
|
||||
priorState,
|
||||
...(priorPauseReason ? { priorPauseReason } : {}),
|
||||
source: "self-healing",
|
||||
});
|
||||
if (!this.options.restartDurableAgentHeartbeat) {
|
||||
log.log(`Durable-agent startup error reset heartbeat restart unavailable for ${agent.id}; state reset only`);
|
||||
} else {
|
||||
const restartOk = await this.options.restartDurableAgentHeartbeat(agent.id, {
|
||||
reason: "startup-error-reset",
|
||||
attempt: 1,
|
||||
});
|
||||
if (!restartOk) {
|
||||
log.warn(`Durable-agent startup error reset heartbeat restart skipped for ${agent.id}`);
|
||||
}
|
||||
}
|
||||
resetCount++;
|
||||
log.log(`Startup reset durable-agent error state for ${agent.id}; heartbeat re-armed when available`);
|
||||
} catch (error) {
|
||||
log.warn(`Failed to reset durable-agent error state on startup for ${agent.id}: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
log.warn(`Startup durable-agent error reset failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return resetCount;
|
||||
}
|
||||
|
||||
return resetCount;
|
||||
}
|
||||
|
||||
async recoverOrphanedAgents(): Promise<number> {
|
||||
const agentStore = this.options.agentStore;
|
||||
if (!agentStore) {
|
||||
|
||||
Reference in New Issue
Block a user