fix(FN-8004): treat heartbeat soft-delete races as benign instead of stranding agents
A task soft-deleted concurrently with a heartbeat-driven moveTask raised TaskDeletedError from the engine's own board path, leaving the agent in `error` with a non-empty lastError and requiring a stop/start cycle to recover. The race is benign by construction: the task is gone, so the move is a no-op. The heartbeat now classifies it via isConcurrentSoftDeleteRaceError (matching the canonical message and serialized/typed forms), keeps the agent active, clears stale error/recovery state, and emits agent:heartbeat-move-skipped-soft-delete with ids/counts-only metadata. Concurrent operator pauses are preserved. Squash-merged by hand from fusion/fn-8004. The engine's AI merge approved this content twice (squash a3a3cc6a8) but could not land it: main advances every ~8 minutes and each merge cycle took ~10, so every attempt lost to a concurrent advance and rebuilt. Each cycle also burned a corrective pass on a first-pass review rejection with no stated reason — the issue #1946 class of bug that this task's own report cites as a sibling. Reconciled against #2157, which refactored transient-error-detector.ts: the new classifier coexists with the extracted transient-error-patterns.ts leaf. Verified on the merged tree — 123 tests green across FN-8004's suites and #2157's, engine typecheck clean. Fusion-Task-Id: FN-8004 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
7
.changeset/fn-8004-soft-delete-heartbeat-race.md
Normal file
7
.changeset/fn-8004-soft-delete-heartbeat-race.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Concurrent soft-delete during a heartbeat move no longer strands an agent in error.
|
||||
category: fix
|
||||
dev: New engine classifier isConcurrentSoftDeleteRaceError short-circuits the agent-heartbeat failed-run handler for TaskDeletedError races (agent stays active, budget untouched) and emits run-audit agent:heartbeat-move-skipped-soft-delete.
|
||||
@@ -267,6 +267,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
- Workspace (Phase D U1): self-healing emits `task:reconcile-orphaned-workspace-worktree` when it removes a done/dead workspace task's recorded per-repo worktree from its stored `worktreePath` (guarded by `isPathActive`; no temp-root walk).
|
||||
- FN-7514: the planner overseer's per-task oversight loop (`PlannerRecoveryController.tick`) emits `overseer:oversight-withheld-human-control` when the pure `evaluateOverseerHumanControl` guard withholds ALL oversight action (no steering, retry, targeted-fix, or pending confirmation) for a task that is user-paused (`task.userPaused===true`, or `task.paused===true` with no `pausedReason`) or ineligible for auto-merge processing per `allowsAutoMergeProcessing` (`autoMerge:false`/PR-based human-review terminal contract). The guard runs BEFORE FN-7513's confirmation classification, so a withheld task never records a pending confirmation. Metadata: `{ taskId, reason: "user-paused" | "auto-merge-off-human-review", stage, oversightLevel }`; deduped per (taskId, withheld reason) so it is not re-emitted every poll while the reason is unchanged.
|
||||
- FN-7720: `TaskStore.bypassFailedPreMergeReviewStep` emits `task:bypass-review` when a privileged operator bypasses the latest failed pre-merge review step of an `in-review` task; metadata includes `workflowStepId`, `workflowStepName`, `bypassedFromStatus`, `bypassedFromVerdict`, and the mandatory `reason`. The bypass rewrites the step's `status` to `"skipped"` with `bypassedBy`/`bypassedAt`/`bypassReason`/`bypassedFromStatus` fields; it never fabricates a reviewer `verdict` and clears only the failed-pre-merge-step `getTaskMergeBlocker` reason. Reachable via `fn_task_bypass_review` (CLI/pi-extension operator tool surface only — not executor/reviewer/triage) and `POST /tasks/:id/bypass-review`.
|
||||
- FN-8004: `agent:heartbeat-move-skipped-soft-delete` records a heartbeat move that races a soft-deleted task without parking the durable agent. Metadata remains ids/timestamps/source only (`agentId`, optional `taskId`/`deletedAt`, `moveAttemptedAt`, optional `source`); it never stores error prose.
|
||||
|
||||
|
||||
## Reference docs (deeper detail)
|
||||
|
||||
@@ -2273,3 +2273,7 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
|
||||
- FN-5223 backstop: `packages/engine/src/__tests__/reliability-interactions/engine-active-since-floor.test.ts` covers engine-activation floor + grace composition across startup, pause/unpause, global-pause gating, and StuckTaskDetector lifecycle interactions.
|
||||
|
||||
The auto-recovery dispatcher at `packages/engine/src/auto-recovery.ts` (FN-4533) composes on top of existing layers (FN-4500 fast-path, FN-4508 deterministic branch-conflict, FN-4499 bootstrap-misbinding, FN-4428 contamination, `mergeAuditAutoRecovery` Stages 1–5, self-healing) to handle six residual classes: file-scope violation at squash, branch misbinding / ghost worktree, verification-fix scope leak, contamination, `branch-conflict-unrecoverable` residuals, and room-post/message-send failures. Invocation is additive — no existing layer's behavior changes.
|
||||
|
||||
### Concurrent soft-delete heartbeat races (FN-8004)
|
||||
|
||||
A heartbeat `moveTask` failure with the typed `TaskDeletedError` soft-delete message is a benign board miss: the durable agent stays active, clears `lastError` and heartbeat recovery metadata, and emits `agent:heartbeat-move-skipped-soft-delete`. Its audit metadata is structured only: `agentId`, `taskId`, `deletedAt`, `moveAttemptedAt`, and `source`.
|
||||
|
||||
@@ -240,6 +240,69 @@ describe("HeartbeatMonitor error-state recovery", () => {
|
||||
}));
|
||||
});
|
||||
|
||||
it("keeps an agent active when a heartbeat move races a soft-delete", async () => {
|
||||
const deletedAt = "2026-07-13T10:18:51.000Z";
|
||||
const raceMessage = `Task FN-8004 is soft-deleted (deletedAt=${deletedAt}) and cannot be read or mutated`;
|
||||
const store = createAgentStore(baseAgent({
|
||||
state: "active",
|
||||
lastError: "previous failure",
|
||||
metadata: buildHeartbeatErrorRecoveryMetadata(baseAgent(), 3),
|
||||
}));
|
||||
const taskStore = createNoTaskStore();
|
||||
const monitor = new HeartbeatMonitor({ store, taskStore, rootDir: process.cwd() });
|
||||
const run = await monitor.startRun(store.agent.id, { source: "timer" });
|
||||
|
||||
await monitor.completeRun(store.agent.id, run.id, { status: "failed", errorMessage: raceMessage });
|
||||
|
||||
expect(store.agent.state).toBe("active");
|
||||
expect(store.agent.lastError).toBeUndefined();
|
||||
expect(readHeartbeatErrorRetryCount(store.agent)).toBe(0);
|
||||
expect(taskStore.recordRunAuditEvent).toHaveBeenCalledWith(expect.objectContaining({
|
||||
mutationType: "agent:heartbeat-move-skipped-soft-delete",
|
||||
target: store.agent.id,
|
||||
metadata: expect.objectContaining({
|
||||
agentId: store.agent.id,
|
||||
taskId: "FN-8004",
|
||||
deletedAt,
|
||||
moveAttemptedAt: expect.any(String),
|
||||
source: "timer",
|
||||
}),
|
||||
}));
|
||||
expect(store.updateAgentState).not.toHaveBeenCalledWith(store.agent.id, "error");
|
||||
expect(store.updateAgentState).not.toHaveBeenCalledWith(store.agent.id, "paused");
|
||||
});
|
||||
|
||||
it("recognizes a soft-delete race from stderrExcerpt without making empty failures benign", async () => {
|
||||
const raceMessage = "Task FN-8004 is soft-deleted (deletedAt=2026-07-13T10:18:51.000Z) and cannot be read or mutated";
|
||||
const raceStore = createAgentStore(baseAgent({ state: "active" }));
|
||||
const raceMonitor = new HeartbeatMonitor({ store: raceStore, taskStore: createNoTaskStore(), rootDir: process.cwd() });
|
||||
const raceRun = await raceMonitor.startRun(raceStore.agent.id, { source: "timer" });
|
||||
|
||||
await raceMonitor.completeRun(raceStore.agent.id, raceRun.id, { status: "failed", stderrExcerpt: raceMessage });
|
||||
|
||||
expect(raceStore.agent.state).toBe("active");
|
||||
expect(raceStore.agent.lastError).toBeUndefined();
|
||||
|
||||
const emptyStore = createAgentStore(baseAgent({ state: "active" }));
|
||||
const emptyMonitor = new HeartbeatMonitor({ store: emptyStore, taskStore: createNoTaskStore(), rootDir: process.cwd() });
|
||||
const emptyRun = await emptyMonitor.startRun(emptyStore.agent.id, { source: "timer" });
|
||||
await emptyMonitor.completeRun(emptyStore.agent.id, emptyRun.id, { status: "failed" });
|
||||
|
||||
expect(emptyStore.agent.state).toBe("error");
|
||||
expect(emptyStore.agent.lastError).toBe("Run failed");
|
||||
});
|
||||
|
||||
it("keeps genuine failed heartbeat runs on the existing error path", async () => {
|
||||
const store = createAgentStore(baseAgent({ state: "active" }));
|
||||
const monitor = new HeartbeatMonitor({ store, taskStore: createNoTaskStore(), rootDir: process.cwd() });
|
||||
const run = await monitor.startRun(store.agent.id, { source: "timer" });
|
||||
|
||||
await monitor.completeRun(store.agent.id, run.id, { status: "failed", errorMessage: "socket hang up" });
|
||||
|
||||
expect(store.agent.state).toBe("error");
|
||||
expect(store.agent.lastError).toBe("socket hang up");
|
||||
});
|
||||
|
||||
it("auto-retries a false heartbeat-model-unavailable park on the next heartbeat without operator Retry", async () => {
|
||||
const session = createSession(async () => undefined);
|
||||
mockedCreateFnAgent.mockResolvedValueOnce(session as never);
|
||||
|
||||
@@ -8,7 +8,9 @@ import {
|
||||
isTransientAuthCredentialError,
|
||||
classifyError,
|
||||
isSilentTransientError,
|
||||
extractConcurrentSoftDeleteRaceDetails,
|
||||
extractMissingModulePath,
|
||||
isConcurrentSoftDeleteRaceError,
|
||||
isOperatorActionableAgentError,
|
||||
isStaleWorktreeModuleResolutionError,
|
||||
isModelAuthTierIncompatibilityError,
|
||||
@@ -278,6 +280,44 @@ describe("Transient Error Detector", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("isConcurrentSoftDeleteRaceError", () => {
|
||||
const raceMessage = "Task FN-8004 is soft-deleted (deletedAt=2026-07-13T10:18:51.000Z) and cannot be read or mutated";
|
||||
|
||||
it("matches the typed soft-delete move race and extracts audit-safe details", () => {
|
||||
expect(isConcurrentSoftDeleteRaceError(raceMessage)).toBe(true);
|
||||
expect(extractConcurrentSoftDeleteRaceDetails(`Error: ${raceMessage}`)).toEqual({
|
||||
taskId: "FN-8004",
|
||||
deletedAt: "2026-07-13T10:18:51.000Z",
|
||||
});
|
||||
});
|
||||
|
||||
it("matches canonical TaskDeletedError serialization even when its message changes", () => {
|
||||
const serializedError = "TaskDeletedError: task FN-8004 was deleted";
|
||||
expect(isConcurrentSoftDeleteRaceError(serializedError)).toBe(true);
|
||||
expect(extractConcurrentSoftDeleteRaceDetails(serializedError)).toEqual({ taskId: "FN-8004", deletedAt: undefined });
|
||||
});
|
||||
|
||||
it("extracts available fields from the JSON form of a serialized TaskDeletedError", () => {
|
||||
const serializedError = '{"name":"TaskDeletedError","taskId":"FN-8004","deletedAt":"2026-07-13T10:18:51.000Z"}';
|
||||
expect(isConcurrentSoftDeleteRaceError(serializedError)).toBe(true);
|
||||
expect(extractConcurrentSoftDeleteRaceDetails(serializedError)).toEqual({
|
||||
taskId: "FN-8004",
|
||||
deletedAt: "2026-07-13T10:18:51.000Z",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
"invalid api key",
|
||||
"socket hang up",
|
||||
"Task FN-8004 is soft-deleted (deletedAt=2026-07-13T10:18:51.000Z) and cannot be recreated",
|
||||
"",
|
||||
undefined,
|
||||
])("does not match unrelated or empty input: %s", (errorMessage) => {
|
||||
expect(isConcurrentSoftDeleteRaceError(errorMessage as string)).toBe(false);
|
||||
expect(extractConcurrentSoftDeleteRaceDetails(errorMessage as string)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("isStaleWorktreeModuleResolutionError", () => {
|
||||
it("returns true for cannot-find-module node_modules imported-from stale worktree signature", () => {
|
||||
const message =
|
||||
|
||||
@@ -49,7 +49,12 @@ import { resolveHeartbeatPromptTemplate, resolveHeartbeatScopeDisciplineMode, se
|
||||
import { buildPromptLayers, collapsePromptLayers } from "./prompt-layers.js";
|
||||
import { resolveAndEmitGoalContext } from "./goal-injection-diagnostics.js";
|
||||
import { createLogger, heartbeatLog, formatError } from "./logger.js";
|
||||
import { isOperatorActionableAgentError, isStaleWorktreeModuleResolutionError } from "./transient-error-detector.js";
|
||||
import {
|
||||
extractConcurrentSoftDeleteRaceDetails,
|
||||
isConcurrentSoftDeleteRaceError,
|
||||
isOperatorActionableAgentError,
|
||||
isStaleWorktreeModuleResolutionError,
|
||||
} from "./transient-error-detector.js";
|
||||
|
||||
/**
|
||||
* FNXC:WorktreeAcquisition 2026-07-09-00:00:
|
||||
@@ -1435,6 +1440,47 @@ export class HeartbeatMonitor {
|
||||
const failedWithRecoverableError = isHeartbeatErrorRecoverable({ lastError: failedError });
|
||||
const failedWithUnrecoverableError = !failedWithRecoverableError && !isStaleWorktreeModuleResolutionError(failedError);
|
||||
/*
|
||||
FNXC:HeartbeatRecovery 2026-07-15-00:00:
|
||||
FN-8004 requires a typed TaskDeletedError from a heartbeat move racing soft-delete to bypass every error, exhaustion, and unrecoverable branch below. The task is already intentionally gone, so keep the durable agent active, clear stale recovery state, and retain only structured audit evidence.
|
||||
*/
|
||||
if (isConcurrentSoftDeleteRaceError(failedError)) {
|
||||
const raceDetails = extractConcurrentSoftDeleteRaceDetails(failedError);
|
||||
const runWithSource = run as unknown as { source?: unknown };
|
||||
const runSource = typeof runWithSource.source === "string" ? runWithSource.source : undefined;
|
||||
const moveAttemptedAt = new Date().toISOString();
|
||||
|
||||
await this.store.updateAgentState(agentId, "active");
|
||||
await this.store.updateAgent(agentId, {
|
||||
lastError: undefined,
|
||||
...(latestAgent ? { metadata: resetHeartbeatErrorRecoveryMetadata(latestAgent) } : {}),
|
||||
});
|
||||
heartbeatLog.log(`Agent ${agentId} heartbeat move skipped because task ${raceDetails?.taskId ?? "unknown"} was soft-deleted concurrently`);
|
||||
|
||||
if (this.taskStore) {
|
||||
try {
|
||||
const audit = createRunAuditor(this.taskStore, {
|
||||
runId,
|
||||
agentId,
|
||||
phase: "heartbeat",
|
||||
source: runSource,
|
||||
});
|
||||
await audit.database({
|
||||
type: "agent:heartbeat-move-skipped-soft-delete",
|
||||
target: agentId,
|
||||
metadata: {
|
||||
agentId,
|
||||
taskId: raceDetails?.taskId,
|
||||
deletedAt: raceDetails?.deletedAt,
|
||||
moveAttemptedAt,
|
||||
source: runSource,
|
||||
},
|
||||
});
|
||||
} catch (auditErr) {
|
||||
heartbeatLog.warn(`Agent ${agentId} soft-delete race audit failed: ${auditErr instanceof Error ? auditErr.message : String(auditErr)} — continuing`);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
/*
|
||||
FNXC:HeartbeatRecovery 2026-07-11-19:57:
|
||||
FN-7835's primary timer path cannot rely on a future heartbeat to perform exhaustion bookkeeping: once retryCount reaches the limit, timer eligibility intentionally stops dispatching error-state agents. Park the agent paused on the failing boundary run so the bounded retry contract is reachable in production.
|
||||
*/
|
||||
@@ -1510,6 +1556,7 @@ export class HeartbeatMonitor {
|
||||
await this.store.updateAgentState(agentId, "error");
|
||||
await this.store.updateAgent(agentId, { lastError: failedError });
|
||||
}
|
||||
}
|
||||
} else if (completionResult.status === "terminated") {
|
||||
await this.store.updateAgentState(agentId, "paused");
|
||||
} else {
|
||||
|
||||
@@ -438,6 +438,11 @@ export type DatabaseMutationType =
|
||||
| "agent:reset-error-state-on-startup"
|
||||
| "agent:error-retry-exhausted"
|
||||
| "agent:error-parked-unrecoverable"
|
||||
/*
|
||||
FNXC:RunAudit 2026-07-15-00:00:
|
||||
FN-8004 records a heartbeat move that lost a concurrent soft-delete using identifiers and timestamps only. Never place the failed run text or agent lastError in this event because the race is benign and audit metadata must remain structured.
|
||||
*/
|
||||
| "agent:heartbeat-move-skipped-soft-delete"
|
||||
| "task:release"
|
||||
| "task:pause"
|
||||
| "task:unpause"
|
||||
|
||||
@@ -136,6 +136,41 @@ export function classifyError(errorMessage: string): "transient" | "usage-limit"
|
||||
const STALE_WORKTREE_MODULE_RESOLUTION_PATTERN = /Cannot find module\s+['"][^'"]*node_modules[^'"]*['"][\s\S]*imported from\s+/i;
|
||||
const STALE_WORKTREE_MODULE_PATH_PATTERN = /Cannot find module\s+['"]([^'"]*node_modules[^'"]*)['"]/i;
|
||||
|
||||
/*
|
||||
FNXC:Reliability-ErrorClassification 2026-07-15-00:00:
|
||||
FN-8004 treats only the typed TaskDeletedError message emitted when a heartbeat move races a soft-delete as a benign board miss. This must not classify broader deleted-task failures as harmless because genuine heartbeat failures still require normal recovery or parking.
|
||||
*/
|
||||
const CONCURRENT_SOFT_DELETE_RACE_PATTERN = /Task\s+([^\s]+)\s+is\s+soft-deleted\s+\(deletedAt=([^)]+)\)\s+and\s+cannot\s+be\s+read\s+or\s+mutated/i;
|
||||
const TASK_DELETED_ERROR_TYPE_PATTERN = /(?:\bTaskDeletedError\s*:|["']name["']\s*:\s*["']TaskDeletedError["']|\bname\s*[=:]\s*TaskDeletedError\b)/i;
|
||||
const SERIALIZED_TASK_ID_PATTERN = /["']taskId["']\s*:\s*["']([^"']+)["']/i;
|
||||
const SERIALIZED_DELETED_AT_PATTERN = /["']deletedAt["']\s*:\s*["']([^"']+)["']/i;
|
||||
const TYPED_TASK_ID_PATTERN = /\bTaskDeletedError\s*:\s*(?:task\s+)?([A-Za-z][A-Za-z0-9_-]*)\b/i;
|
||||
|
||||
export function isConcurrentSoftDeleteRaceError(errorMessage: string): boolean {
|
||||
if (!errorMessage || typeof errorMessage !== "string") {
|
||||
return false;
|
||||
}
|
||||
|
||||
// A serialized core error can preserve only its canonical name, not Error.message.
|
||||
return CONCURRENT_SOFT_DELETE_RACE_PATTERN.test(errorMessage) || TASK_DELETED_ERROR_TYPE_PATTERN.test(errorMessage);
|
||||
}
|
||||
|
||||
export function extractConcurrentSoftDeleteRaceDetails(errorMessage: string): { taskId?: string; deletedAt?: string } | null {
|
||||
if (!errorMessage || typeof errorMessage !== "string" || !isConcurrentSoftDeleteRaceError(errorMessage)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const canonicalMatch = errorMessage.match(CONCURRENT_SOFT_DELETE_RACE_PATTERN);
|
||||
if (canonicalMatch?.[1] && canonicalMatch[2]) {
|
||||
return { taskId: canonicalMatch[1], deletedAt: canonicalMatch[2] };
|
||||
}
|
||||
|
||||
const taskId = errorMessage.match(SERIALIZED_TASK_ID_PATTERN)?.[1]
|
||||
?? errorMessage.match(TYPED_TASK_ID_PATTERN)?.[1];
|
||||
const deletedAt = errorMessage.match(SERIALIZED_DELETED_AT_PATTERN)?.[1];
|
||||
return taskId || deletedAt ? { taskId, deletedAt } : null;
|
||||
}
|
||||
|
||||
export function isStaleWorktreeModuleResolutionError(errorMessage: string): boolean {
|
||||
if (!errorMessage || typeof errorMessage !== "string") {
|
||||
return false;
|
||||
|
||||
Reference in New Issue
Block a user