fix(engine): honest BLOCKED park survives pause-abort and workflow-graph teardown (#2264)

## What

Follow-up 1 to the FN-8141 guard series (#2254–#2260). Makes the honest
`fn_task_done(outcome="blocked")` park (`status:"failed"`,
`error:"BLOCKED: <reason>"`, blockedBy → dependencies, added in #2256)
**survive the graph-teardown machinery** that bounced FN-8141's failed
park back to `todo`.

## Why

In the original FN-8141 incident, the executor's parked-failed state did
not stick: the pause-abort classifier and the workflow-graph failure
handler either rehomed the task to `todo` (clearing `status`/`error`) or
overwrote the distinctive `BLOCKED:` error with a generic "Workflow
graph terminated with failure" string. #2256 added the blocked exit but
nobody proved the park survives that bounce. Any path that
clears/overwrites the marker re-opens the laundering hole, because
self-healing (#2257/#2260) and dependency-gated scheduling key off
exactly that `BLOCKED:` error plus the recorded `blockedBy`
dependencies.

`handleGraphFailure` now detects a live blocked park (`status ===
"failed" && error.startsWith("BLOCKED:")`) **before every other
classifier** and honors it, following the existing non-graph honor-park
precedent (executor `~12163`):

- no requeue to `todo`, no engine-internal auto-continue, no `BLOCKED:`
error overwrite;
- clears the in-memory pause-abort marker so
`recoverPausedAbortFailures` has nothing to chase;
- **releases the worktree / `maxWorktrees` slot** (FN-6782 leaked-holder
precedent — the graph `finally` does not delete `activeWorktrees`);
- leaves `status`/`error`/`column`/`dependencies`/steps untouched.

Unblocking still works: the operator requeue (`moveTask`
in-progress→todo, `moves.ts ~628`) and `buildManualRetryResetPatch`
clear the `BLOCKED:` error; the guard keys off the **live** error, so a
cleared row is never re-wedged, and dependency-gated scheduling leaves
the parked row untouched while `blockedBy` deps are unmet.

## Surfaces covered

Pause-abort classifier (hard-cancel), engine-internal auto-continue, and
the plain terminal graph-failure sink — all routed through
`handleGraphFailure`, so a single top-of-method guard composes across
them.

## Test evidence

Extended `executor-task-done-blocked.test.ts` (drives
`handleGraphFailure` against a live blocked park):
- honors the park under a hard-cancel pause-abort bounce (no requeue /
clear / auto-continue);
- honors it under a plain terminal graph failure (sink never overwrites
`BLOCKED:`);
- releases the worktree/concurrency slot + clears the pause-abort
marker;
- NON-blocked failed park keeps existing behavior (guard scoped to
`BLOCKED:`);
- a cleared (unblocked) row is NOT re-honor-parked.

```
pnpm --filter @fusion/engine exec vitest run src/__tests__/executor-task-done-blocked.test.ts  → 13 passed
pnpm --filter @fusion/engine exec vitest run executor-paused-abort-todo-benign + executor-graph-requeue-gate  → 53 passed
pnpm --filter @fusion/engine exec tsc --noEmit  → clean
pnpm verify:fast  → PASS (3 steps green)
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-07-16 21:39:16 -07:00
committed by GitHub
parent bc7dfe4bbf
commit f116d05c41
3 changed files with 248 additions and 1 deletions

View File

@@ -0,0 +1,7 @@
---
"@runfusion/fusion": patch
---
summary: A task honestly parked as blocked now stays parked through engine pause/abort and workflow-graph teardown.
category: fix
dev: handleGraphFailure honors a live blocked park (status "failed", error "BLOCKED:") before every pause-abort/graph-failure classifier — no requeue-to-todo, no auto-continue, no BLOCKED: error overwrite — and releases its worktree/maxWorktrees slot. FN-8141 follow-up 1.

View File

@@ -1,4 +1,4 @@
import { beforeEach, describe, expect, it, vi } from "vitest"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import "./executor-test-helpers.js"; import "./executor-test-helpers.js";
import { TaskExecutor } from "../executor.js"; import { TaskExecutor } from "../executor.js";
import * as worktreePool from "../worktree-pool.js"; import * as worktreePool from "../worktree-pool.js";
@@ -254,3 +254,213 @@ describe("FN-8141 blocked-parked task is not auto-recovered by the completed-tod
expect(typeof evaluateNoCommitsNoOpFinalize).toBe("function"); expect(typeof evaluateNoCommitsNoOpFinalize).toBe("function");
}); });
}); });
/*
FNXC:Lifecycle 2026-07-16-21:22:
FN-8141 follow-up 1 — the honest blocked park must SURVIVE the graph-teardown machinery that undid the
original incident's failed park. In FN-8141 the pause-abort classifier and the workflow-graph failure
handler bounced the parked-failed task back to `todo` (clearing status/error) or overwrote the distinctive
`BLOCKED:` error with a generic graph-failure string — either of which re-opens the laundering hole because
self-healing (#2257/#2260) and dependency-gated scheduling key off exactly that error + the recorded
blockedBy dependencies. These tests drive handleGraphFailure (the graph-teardown sink) against a live
blocked park across the enumerated surfaces: pause-abort (hard-cancel), engine-internal auto-continue, and
the plain terminal graph-failure sink. Invariant asserted on every surface: the row stays parked
(status:"failed", error starts with "BLOCKED:", dependencies intact), is NOT moved to todo, is NOT
auto-continued into a new agent session, keeps its steps, and releases its worktree/concurrency slot.
*/
describe("FN-8141 follow-up 1 — blocked park survives graph teardown", () => {
const now = "2026-07-16T00:00:00.000Z";
function blockedParkedDetail(overrides: Record<string, unknown> = {}) {
return {
id: "FN-8141",
title: "Blocked exit test",
description: "",
column: "in-progress",
status: "failed",
error: "BLOCKED: pi 0.80.10 removed AuthStorage; SDK bump cannot pass verify:fast",
dependencies: ["FN-8145"],
worktree: "/repo/.worktrees/swift-falcon",
branch: "fusion/fn-8141",
baseBranch: "main",
paused: false,
userPaused: false,
autoMerge: true,
mergeRetries: 0,
// True statuses — NOT all done/skipped — so a laundered "complete" state can never form.
steps: [
{ name: "Implement", status: "in-progress" as const },
{ name: "Testing & Verification", status: "pending" as const },
],
currentStep: 0,
log: [],
createdAt: now,
updatedAt: now,
...overrides,
};
}
function makeHarness(overrides: Record<string, unknown> = {}) {
const store = createMockStore();
const task = blockedParkedDetail(overrides);
store.getTask.mockResolvedValue(task as any);
store.getSettings.mockResolvedValue({
maxConcurrent: 2,
maxWorktrees: 4,
pollIntervalMs: 15000,
autoMerge: true,
maxAutoMergeRetries: 3,
} as any);
store.recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
const executor = new TaskExecutor(store as any, "/repo", {} as any);
return { store, task, executor };
}
async function invokeGraphFailure(
executor: TaskExecutor,
task: any,
resultOverrides: Record<string, unknown> = {},
) {
await (executor as any).handleGraphFailure(task, {
disposition: "failed",
outcome: "failure",
visitedNodeIds: ["plan", "execute"],
context: {},
...resultOverrides,
});
}
function logText(store: ReturnType<typeof createMockStore>): string {
return store.logEntry.mock.calls.map((call: unknown[]) => call[1]).join("\n");
}
function assertParkPreserved(store: ReturnType<typeof createMockStore>, executor: TaskExecutor) {
// Never moved to todo (the FN-8141 bounce).
expect(store.moveTask).not.toHaveBeenCalled();
// status/error never cleared to null...
const clearedStatusOrError = store.updateTask.mock.calls.some((call: unknown[]) => {
const patch = call[1] as { status?: unknown; error?: unknown } | undefined;
return patch?.status === null || patch?.error === null;
});
expect(clearedStatusOrError).toBe(false);
// ...and the distinctive BLOCKED: error is never overwritten by a generic graph-failure string.
const overwroteBlockedError = store.updateTask.mock.calls.some((call: unknown[]) => {
const patch = call[1] as { error?: unknown } | undefined;
return typeof patch?.error === "string" && !patch.error.startsWith("BLOCKED:");
});
expect(overwroteBlockedError).toBe(false);
// Dependencies never mutated.
const mutatedDeps = store.updateTask.mock.calls.some(
(call: unknown[]) => (call[1] as { dependencies?: unknown } | undefined)?.dependencies !== undefined,
);
expect(mutatedDeps).toBe(false);
// Steps left untouched.
expect(store.updateStep).not.toHaveBeenCalled();
// Worktree/concurrency slot released; no leaked maxWorktrees holder (FN-6782).
expect((executor as any).activeWorktrees.has("FN-8141")).toBe(false);
expect((executor as any).pausedAborted.has("FN-8141")).toBe(false);
// Honor-park breadcrumb logged.
expect(logText(store)).toContain("honoring park, not requeueing, retrying, or clearing state");
}
beforeEach(() => {
resetExecutorMocks();
vi.useFakeTimers();
});
afterEach(() => {
vi.useRealTimers();
});
it("honors the park under a hard-cancel pause-abort bounce (no requeue, no clear, no auto-continue)", async () => {
const { store, task, executor } = makeHarness();
(executor as any).addActiveWorktree(task.id, task.worktree);
// The exact FN-8141 teardown: pause-abort mark/classify/cleanup fired hard-cancel.
(executor as any).markPausedAborted(task.id, "hard-cancel");
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await invokeGraphFailure(executor, task, {
interruptedNodeId: "execute",
interruptedAbortKind: "engine-pause",
context: { "node:execute:value": "aborted", "node:execute:abortKind": "engine-pause" },
});
assertParkPreserved(store, executor);
// NOT auto-continued into a fresh agent session (the engine-internal auto-continue path).
await vi.advanceTimersByTimeAsync(10);
expect(executeSpy).not.toHaveBeenCalled();
});
it("honors the park under a plain terminal graph failure (no pause-abort marker) — the sink never overwrites BLOCKED:", async () => {
// Without the guard, the terminal graph-failure sink overwrites error with
// "Workflow graph terminated with failure at node 'execute'", erasing the BLOCKED: marker.
const { store, task, executor } = makeHarness();
(executor as any).addActiveWorktree(task.id, task.worktree);
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await invokeGraphFailure(executor, task, {
visitedNodeIds: ["plan", "execute"],
context: { "node:execute:value": "aborted" },
});
assertParkPreserved(store, executor);
await vi.advanceTimersByTimeAsync(10);
expect(executeSpy).not.toHaveBeenCalled();
});
it("does NOT emit a redundant blocked-parked run-audit event on the honor-park (the exit already emitted it)", async () => {
const { store, task, executor } = makeHarness();
(executor as any).addActiveWorktree(task.id, task.worktree);
(executor as any).markPausedAborted(task.id, "hard-cancel");
await invokeGraphFailure(executor, task);
expect(store.recordRunAuditEvent).not.toHaveBeenCalledWith(
expect.objectContaining({ mutationType: "task:execution-blocked-parked" }),
);
});
it("NON-blocked failed park keeps existing behavior — the honor-park guard does not fire", async () => {
// A generic terminal graph failure (error does NOT start with BLOCKED:) must fall through to
// the normal terminal sink and park failed with the generic message — the guard is scoped to BLOCKED:.
const { store, task, executor } = makeHarness({
status: null,
error: null,
});
await invokeGraphFailure(executor, task, {
visitedNodeIds: ["plan", "execute"],
context: { "node:execute:value": "some-non-blocked-failure" },
});
// Generic terminal park still happens.
const parkedFailed = store.updateTask.mock.calls.some(
(call: unknown[]) => (call[1] as { status?: string } | undefined)?.status === "failed",
);
expect(parkedFailed).toBe(true);
// The blocked honor-park breadcrumb is absent.
expect(logText(store)).not.toContain("honoring park, not requeueing, retrying, or clearing state");
});
it("a cleared (unblocked) row is NOT re-honor-parked — the stale BLOCKED: error must not wedge a requeue", async () => {
// Operator requeue via moveTask(in-progress→todo) clears status/error (moves.ts reopen); a
// re-dispatched row therefore carries no BLOCKED: error. The guard keys off the LIVE error, so
// once cleared it must not fire and re-wedge the row — normal graph-failure handling resumes.
const { store, task, executor } = makeHarness({
status: null,
error: null,
});
await invokeGraphFailure(executor, task, {
visitedNodeIds: ["plan", "execute"],
context: { "node:execute:value": "some-non-blocked-failure" },
});
// Guard did not intercept — the normal terminal sink parked failed instead.
expect(logText(store)).not.toContain("honoring park, not requeueing, retrying, or clearing state");
const parkedFailed = store.updateTask.mock.calls.some(
(call: unknown[]) => (call[1] as { status?: string } | undefined)?.status === "failed",
);
expect(parkedFailed).toBe(true);
});
});

View File

@@ -9281,6 +9281,36 @@ export class TaskExecutor {
} }
const live = loadedLive; const live = loadedLive;
/* /*
FNXC:Lifecycle 2026-07-16-21:22:
FN-8141 follow-up 1 — an honest `fn_task_done(outcome="blocked")` park (status="failed",
error "BLOCKED: <reason>", executor ~14657) must SURVIVE the same graph-teardown machinery
that undid the original incident's failed park. Every downstream classifier in this method
can wash the marker out: the genuine-pause-abort todo-rehome branch (~9504) clears
status/error on a task the abort bounced back to `todo`; the execution-resume router and the
terminal graph-failure sink (~9982) overwrite the distinctive `BLOCKED:` error with a generic
"Workflow graph terminated with failure" string; and the engine-internal auto-continue
(~9540) re-runs the doomed session. Self-healing (#2257/#2260) and dependency-gated scheduling
key off this exact `BLOCKED:` error + the recorded blockedBy dependencies, so any of those
would re-open the laundering hole. Detect the live blocked park BEFORE every other classifier
and honor it exactly like the non-graph post-loop honor-park (executor ~12163): clear the
in-memory pause-abort marker so `recoverPausedAbortFailures` has nothing to chase, RELEASE the
worktree/concurrency slot (FN-6782 leaked-`maxWorktrees`-holder precedent; the graph finally
does not delete `activeWorktrees`), and return WITHOUT touching status/error/column/
dependencies/steps — the park stays intact for the blocker/operator. Unblocking still works:
the operator requeue (moveTask in-progress→todo, moves.ts ~628) and `buildManualRetryResetPatch`
clear the `BLOCKED:` error, and the scheduler leaves the parked row untouched while blockedBy
dependencies are unmet.
*/
if (live.status === "failed" && live.error?.startsWith("BLOCKED:")) {
this.clearPausedAborted(task.id);
this.activeWorktrees.delete(task.id);
const blockedParkHonored = `Workflow graph run ended after an honest blocked park (${live.error}) — honoring park, not requeueing, retrying, or clearing state`;
executorLog.log(`${task.id}: ${blockedParkHonored}`);
await this.store.logEntry(task.id, blockedParkHonored, undefined, this.getRunContextFor(task.id));
await this.persistTokenUsage(task.id);
return;
}
/*
FNXC:MissingWorktreeRecovery 2026-07-16-18:25: FNXC:MissingWorktreeRecovery 2026-07-16-18:25:
An unusable-worktree session-start refusal inside a graph node must route to the bounded An unusable-worktree session-start refusal inside a graph node must route to the bounded
worktree-session recovery BEFORE any other classifier: FN-7977's provider-failure hold worktree-session recovery BEFORE any other classifier: FN-7977's provider-failure hold