FN-6568: route merge-seam graph aborts to retry
Classify merge-seam abort provenance so non-paused merge graph failures retry instead of parking as pauses. - Track paused-abort provenance separately from the legacy pause-abort bit. - Route merge and requestMerge graph failures through bounded auto-merge retry when they are not genuine pauses. - Preserve user/global pause parking behavior with regression coverage and document the lifecycle invariant. Files changed: .../fn-6568-merge-seam-abort-classification.md | 5 + docs/architecture.md | 1 + .../engine/src/__tests__/executor-recovery.test.ts | 131 ++++++++++++++++++++- packages/engine/src/executor.ts | 122 +++++++++++++------ 4 files changed, 221 insertions(+), 38 deletions(-) Fusion-Task-Id: FN-6568 Fusion-Task-Lineage: 5d9ee4f8-0336-4fb9-aa95-beb65e6c1ed9
This commit is contained in:
5
.changeset/fn-6568-merge-seam-abort-classification.md
Normal file
5
.changeset/fn-6568-merge-seam-abort-classification.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Fix workflow graph merge-node failures so merge-seam aborts are not misclassified as pause/resume aborts. Non-paused merge failures now route to the bounded auto-merge retry path instead of being parked failed with no merge retry count.
|
||||
@@ -1788,6 +1788,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
|
||||
- **Worktrunk-managed lifecycles**: when `worktrunk.enabled`, self-healing defers prune/idle/worktree-cap sweeps to the worktrunk backend; branch-level stale/ conflict reclaim stays native. Orphan `fusion/*` branches are operator-managed via standard git tooling (no auto-rescue task filing).
|
||||
- **Post-finalize verification no-op (FN-4944)**: when auto-merge receives a delayed `VerificationError` after a task is already `done` with `mergeDetails.mergeConfirmed === true` (already-on-main fast-path), it must log one `[verification] ... no action` diagnostic and must not bounce the task back to `in-progress` / `merging-fix`. Defense-in-depth now re-checks the done+mergeConfirmed condition immediately before each verification-failure status write site, and emits `task:post-finalize-verification-no-op` database audit events with failure metadata for forensics.
|
||||
- **Transient auto-merge retry classification (FN-5697)**: non-conflict auto-merge errors now run through `isTransientError(...)` before terminal parking. Transient provider/network failures (for example `This operation was aborted`, `socket hang up`, and `server_error` payloads) are retried with bounded exponential backoff (`5s/10s/20s`) and `status=null` for both direct and pull-request merge strategies; once `MAX_AUTO_MERGE_TRANSIENT_RETRIES` is exhausted, tasks are parked `in-review/failed` with explicit transient-exhaustion logs.
|
||||
- **Merge-seam abort provenance (FN-6568)**: workflow graph merge-node failures must not be classified as pause/resume aborts merely because the merge seam hard-canceled an in-flight session. `TaskExecutor` tracks paused-abort provenance separately (`global-pause`, `merge-seam`, `hard-cancel`); genuine user/global pauses still preserve FN-6478/FN-5147 parking, while non-paused `merge`/`requestMerge` graph failures route back into the bounded auto-merge retry path instead of being parked `status:"failed"` with `mergeRetries=NULL`.
|
||||
- **Worktree pool exclusivity (FN-4954)**: `WorktreePool.acquire(taskId)` / `release(path, taskId?)` track a `leased` map so every pooled path is either idle or leased, never both. Cross-task double-lease detection throws `PoolDoubleLeaseError` and emits `worktree:pool-double-lease-detected`; merger Step 8 now detaches HEAD and clears `task.worktree` / `task.branch` before releasing paths back to the pool.
|
||||
- **Stale registration recovery (FN-5056)**: `NativeWorktreeBackend.create` and `executor.tryCreateWorktree` detect `missing but already registered worktree` failures, run `git worktree prune` (plus `remove --force` / `add -f` fallbacks) before retrying, and emit `worktree:stale-registration-{detected,recovered,recovery-failed}` audit events.
|
||||
- **Raw worktree deletion must be paired with prune (FN-5058)**: any direct filesystem deletion of a worktree directory (`rm -rf` / `rmSync`) must be followed by best-effort `git worktree prune` via `pruneWorktreeAdminEntries` so `.git/worktrees/*` admin entries are not stranded in a missing-but-registered state (FN-5056 class).
|
||||
|
||||
@@ -341,7 +341,7 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
|
||||
// Simulate: task gets paused mid-execution → abort error
|
||||
mockedCreateFnAgent.mockRejectedValue(new Error("Aborted"));
|
||||
(executor as any).pausedAborted.add("FN-001");
|
||||
(executor as any).markPausedAborted("FN-001", "hard-cancel");
|
||||
|
||||
await executor.execute(task);
|
||||
|
||||
@@ -374,7 +374,7 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
mockedCreateFnAgent.mockRejectedValue(new Error("Aborted"));
|
||||
(executor as any).pausedAborted.add("FN-001");
|
||||
(executor as any).markPausedAborted("FN-001", "hard-cancel");
|
||||
|
||||
await executor.execute({
|
||||
id: "FN-001",
|
||||
@@ -1114,7 +1114,7 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
error: null,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
(executor as any).pausedAborted.add("FN-001");
|
||||
(executor as any).markPausedAborted("FN-001", "hard-cancel");
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
visitedNodeIds: ["execute"],
|
||||
@@ -1162,7 +1162,7 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
error: "Task reached in-review without calling fn_task_done",
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
(executor as any).pausedAborted.add("FN-001");
|
||||
(executor as any).markPausedAborted("FN-001", "hard-cancel");
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
visitedNodeIds: ["execute"],
|
||||
@@ -1314,6 +1314,129 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
},
|
||||
);
|
||||
|
||||
describe("merge-seam abort classification (FN-6568)", () => {
|
||||
/*
|
||||
Surface Enumeration coverage:
|
||||
- Pause-branch classifier: merge-seam provenance bypasses operator-action pause parking; user/global pause provenance still parks.
|
||||
- handleGraphFailure call surfaces: direct graph-failure handling for merge/requestMerge nodes plus existing execute-node hard-cancel tests.
|
||||
- pausedAborted provenance: hard-cancel, global-pause, merge-seam, and no-provenance/clean merge failure behavior are explicit.
|
||||
- Failed-node identity: legacy `merge` seam and graph primitive `requestMerge` are both treated as merge failures.
|
||||
- Column/progress states: in-review merge failures retry; existing in-progress genuine pause tests preserve pause state.
|
||||
- Data states: userPaused true, paused true, merge-seam provenance, global-pause provenance, and hard-cancel provenance are covered.
|
||||
- autoMerge:false review parking: a genuinely paused in-review task remains parked without backward movement.
|
||||
*/
|
||||
const makeGraphTask = (overrides: Partial<Task> = {}) => ({
|
||||
id: "FN-001",
|
||||
title: "Test",
|
||||
description: "Test",
|
||||
column: "in-progress",
|
||||
status: undefined,
|
||||
dependencies: [],
|
||||
steps: [{ name: "Preflight", status: "done" }],
|
||||
currentStep: 1,
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Resuming execution after unpause" }],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
...overrides,
|
||||
}) as Task;
|
||||
|
||||
it.each(["merge", "requestMerge"] as const)(
|
||||
"routes non-paused merge-seam abort at %s into bounded auto-merge retry instead of pause parking",
|
||||
async (nodeId) => {
|
||||
const store = createMockStore();
|
||||
const task = makeGraphTask();
|
||||
store.getTask.mockResolvedValue({
|
||||
...task,
|
||||
column: "in-review",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
status: undefined,
|
||||
error: null,
|
||||
mergeRetries: null,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
const mergeRequester = vi.fn(async () => ({ merged: false, noOp: false, reason: "merge-conflict" }));
|
||||
executor.setMergeRequester(mergeRequester as any);
|
||||
(executor as any).markPausedAborted("FN-001", "merge-seam");
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: [nodeId],
|
||||
});
|
||||
|
||||
const messages = store.logEntry.mock.calls.map((call) => call[1]).join("\n");
|
||||
expect(messages).toContain(`Workflow graph merge failure at node '${nodeId}' routed to bounded auto-merge retry after merge-seam abort`);
|
||||
expect(messages).not.toContain("engine abort during pause/resume");
|
||||
expect(messages).not.toContain("operator action required");
|
||||
expect(store.updateTask).not.toHaveBeenCalledWith(
|
||||
"FN-001",
|
||||
expect.objectContaining({ status: "failed" }),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
expect(mergeRequester).toHaveBeenCalledWith("FN-001");
|
||||
},
|
||||
);
|
||||
|
||||
it("preserves global-pause provenance as operator-action parking for in-review graph exits", async () => {
|
||||
const store = createMockStore();
|
||||
const task = makeGraphTask();
|
||||
store.getTask.mockResolvedValue({
|
||||
...task,
|
||||
column: "in-review",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
status: undefined,
|
||||
error: null,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
(executor as any).markPausedAborted("FN-001", "global-pause");
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: ["merge"],
|
||||
});
|
||||
|
||||
const expectedMessage = "Workflow graph failure surfaced after paused global pause in 'in-review' at node 'merge' — operator action required; retry or explicitly unpause/resume after inspecting the task";
|
||||
const messages = store.logEntry.mock.calls.map((call) => call[1]).join("\n");
|
||||
expect(messages).toContain("global pause");
|
||||
expect(messages).toContain("operator action required");
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-001", { error: expectedMessage, status: "failed" }, undefined);
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps autoMerge:false genuinely paused in-review tasks parked without moving backward", async () => {
|
||||
const store = createMockStore();
|
||||
const task = makeGraphTask({ autoMerge: false } as Partial<Task>);
|
||||
store.getTask.mockResolvedValue({
|
||||
...task,
|
||||
column: "in-review",
|
||||
paused: true,
|
||||
userPaused: true,
|
||||
status: undefined,
|
||||
error: null,
|
||||
autoMerge: false,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: ["merge"],
|
||||
});
|
||||
|
||||
const expectedMessage = "Workflow graph failure surfaced after paused explicit user pause in 'in-review' at node 'merge' — operator action required; retry or explicitly unpause/resume after inspecting the task";
|
||||
expect(store.logEntry).toHaveBeenCalledWith("FN-001", expectedMessage, undefined, undefined);
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-001", { error: expectedMessage, status: "failed" }, undefined);
|
||||
expect(store.moveTask).not.toHaveBeenCalledWith("FN-001", "todo", expect.anything());
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
it("auto-retries a bounded transient resume-after-restart graph failure instead of parking", async () => {
|
||||
const store = createMockStore();
|
||||
const task = {
|
||||
|
||||
@@ -1446,6 +1446,11 @@ export class TaskExecutor {
|
||||
private activeSubagentSessions = new Map<string, Set<AgentSession>>();
|
||||
/** Tasks that were paused mid-execution (to avoid marking them as "failed"). */
|
||||
private pausedAborted = new Set<string>();
|
||||
/**
|
||||
* FNXC:WorkflowLifecycle 2026-06-17-03:42:
|
||||
* FN-6568 separates pause provenance from the legacy pausedAborted hard-cancel bit. Merge-seam/internal aborts caused FN-6528/FN-6531/FN-6534/FN-6537 to look like pause/resume aborts and left mergeRetries=NULL, so handleGraphFailure must know whether the abort came from global pause, the merge seam, or a generic hard cancel before choosing operator-action parking.
|
||||
*/
|
||||
private pausedAbortProvenance = new Map<string, "global-pause" | "merge-seam" | "hard-cancel">();
|
||||
/** Tasks that had a dependency added mid-execution (abort + discard worktree). */
|
||||
private depAborted = new Set<string>();
|
||||
/** Tasks killed by stuck task detector. Value = shouldRequeue (budget not exhausted). */
|
||||
@@ -1469,6 +1474,16 @@ export class TaskExecutor {
|
||||
/** Set of ephemeral spawned agent IDs with in-flight cleanup (prevents duplicate deletion attempts). */
|
||||
private pendingEphemeralDeletions = new Set<string>();
|
||||
|
||||
private markPausedAborted(taskId: string, provenance: "global-pause" | "merge-seam" | "hard-cancel" = "hard-cancel"): void {
|
||||
this.pausedAborted.add(taskId);
|
||||
this.pausedAbortProvenance.set(taskId, provenance);
|
||||
}
|
||||
|
||||
private clearPausedAborted(taskId: string): void {
|
||||
this.pausedAborted.delete(taskId);
|
||||
this.pausedAbortProvenance.delete(taskId);
|
||||
}
|
||||
|
||||
private setActiveSession(taskId: string, sessionState: ActiveExecutorSessionState, worktreePath: string): void {
|
||||
this.activeSessions.set(taskId, sessionState);
|
||||
activeSessionRegistry.registerPath(worktreePath, { taskId, kind: "executor", ownerKey: taskId });
|
||||
@@ -2040,7 +2055,7 @@ export class TaskExecutor {
|
||||
if (options.userCanceled) {
|
||||
this.userCanceledTaskIds.add(taskId);
|
||||
}
|
||||
this.pausedAborted.add(taskId);
|
||||
this.markPausedAborted(taskId, "hard-cancel");
|
||||
this.options.stuckTaskDetector?.untrackTask(taskId);
|
||||
this.clearWorkflowRerunWatchdog(taskId);
|
||||
this.clearCompletedTaskWatchdog(taskId);
|
||||
@@ -2695,7 +2710,7 @@ export class TaskExecutor {
|
||||
if (settings.globalPause && !previous.globalPause) {
|
||||
for (const [taskId, controllers] of this.activeConfiguredCommandControllers) {
|
||||
executorLog.log(`Global pause — aborting configured command(s) for ${taskId}`);
|
||||
this.pausedAborted.add(taskId);
|
||||
this.markPausedAborted(taskId, "global-pause");
|
||||
this.options.stuckTaskDetector?.untrackTask(taskId);
|
||||
for (const controller of controllers) {
|
||||
controller.abort();
|
||||
@@ -2713,7 +2728,7 @@ export class TaskExecutor {
|
||||
}
|
||||
for (const [taskId, { session }] of this.activeSessions) {
|
||||
executorLog.log(`Global pause — terminating agent session for ${taskId}`);
|
||||
this.pausedAborted.add(taskId);
|
||||
this.markPausedAborted(taskId, "global-pause");
|
||||
this.options.stuckTaskDetector?.untrackTask(taskId);
|
||||
// abort() interrupts any in-flight LLM stream / tool call;
|
||||
// dispose() then releases session resources.
|
||||
@@ -2731,7 +2746,7 @@ export class TaskExecutor {
|
||||
}
|
||||
for (const [taskId, stepExecutor] of this.activeStepExecutors) {
|
||||
executorLog.log(`Global pause — terminating step sessions for ${taskId}`);
|
||||
this.pausedAborted.add(taskId);
|
||||
this.markPausedAborted(taskId, "global-pause");
|
||||
this.options.stuckTaskDetector?.untrackTask(taskId);
|
||||
stepExecutor.terminateAllSessions().catch(err =>
|
||||
executorLog.warn(`Failed to terminate step sessions for global pause ${taskId}: ${err}`)
|
||||
@@ -2743,7 +2758,7 @@ export class TaskExecutor {
|
||||
}
|
||||
for (const [taskId, workflowSession] of this.activeWorkflowStepSessions) {
|
||||
executorLog.log(`Global pause — terminating workflow step session for ${taskId}`);
|
||||
this.pausedAborted.add(taskId);
|
||||
this.markPausedAborted(taskId, "global-pause");
|
||||
this.options.stuckTaskDetector?.untrackTask(taskId);
|
||||
const sessionWithAbort = workflowSession as AgentSession & { abort?: () => Promise<void> };
|
||||
if (typeof sessionWithAbort.abort === "function") {
|
||||
@@ -3444,7 +3459,7 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(task, task.worktree, settings, undefined);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -5074,9 +5089,9 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(live, worktreePath, settings, undefined);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (await this.parkTaskAfterWorkflowStepPause(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
} else if (this.pausedAborted.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
}
|
||||
return { outcome: "success", value: "deferred-paused", data: { allPassed: false } };
|
||||
}
|
||||
@@ -5175,7 +5190,7 @@ export class TaskExecutor {
|
||||
},
|
||||
abortRun: async (_ctx, task, input) => {
|
||||
if (input.hardCancel) {
|
||||
this.pausedAborted.add(task.id);
|
||||
this.markPausedAborted(task.id, "merge-seam");
|
||||
}
|
||||
await this.store.updateTask(task.id, {
|
||||
paused: true,
|
||||
@@ -5241,9 +5256,9 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(live, worktreePath, settings, undefined);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (await this.parkTaskAfterWorkflowStepPause(seamTask.id)) {
|
||||
this.pausedAborted.delete(seamTask.id);
|
||||
this.clearPausedAborted(seamTask.id);
|
||||
} else if (this.pausedAborted.has(seamTask.id)) {
|
||||
this.pausedAborted.delete(seamTask.id);
|
||||
this.clearPausedAborted(seamTask.id);
|
||||
}
|
||||
return { outcome: "success", value: "deferred-paused" };
|
||||
}
|
||||
@@ -6330,6 +6345,29 @@ export class TaskExecutor {
|
||||
return value === "awaiting-user-input" || value === "awaiting-cli-approval";
|
||||
}
|
||||
|
||||
private isMergeGraphFailure(failedNode: string | undefined): boolean {
|
||||
return failedNode === "merge" || failedNode === "requestMerge";
|
||||
}
|
||||
|
||||
private async routeGraphMergeFailureToRetry(
|
||||
live: TaskDetail,
|
||||
result: WorkflowGraphTaskRunResult,
|
||||
abortProvenance: "global-pause" | "merge-seam" | "hard-cancel" | undefined,
|
||||
): Promise<boolean> {
|
||||
if (!this.mergeRequester) return false;
|
||||
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1] ?? "unknown";
|
||||
const message = `Workflow graph merge failure at node '${failedNode}' routed to bounded auto-merge retry${abortProvenance === "merge-seam" ? " after merge-seam abort" : ""}`;
|
||||
executorLog.warn(`${live.id}: ${message}`);
|
||||
await this.store.logEntry(live.id, message, undefined, this.getRunContextFor(live.id));
|
||||
try {
|
||||
await this.mergeRequester(live.id);
|
||||
} catch (error) {
|
||||
executorLog.warn(`${live.id}: bounded auto-merge retry request failed after graph merge failure: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
await this.persistTokenUsage(live.id);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Terminal failure of a graph run: record the error and park the task in
|
||||
* review so a human can act — never leave it invisible in in-progress. */
|
||||
private async handleGraphFailure(task: Task, result: WorkflowGraphTaskRunResult): Promise<void> {
|
||||
@@ -6341,16 +6379,29 @@ export class TaskExecutor {
|
||||
// is still in-progress — leave the pause machinery in charge instead of
|
||||
// parking the task in review.
|
||||
const pausedAborted = this.pausedAborted.has(task.id);
|
||||
if (live.paused || pausedAborted) {
|
||||
const abortProvenance = this.pausedAbortProvenance.get(task.id);
|
||||
const mergeSeamAborted = abortProvenance === "merge-seam";
|
||||
const genuinePauseAbort = Boolean(
|
||||
live.userPaused
|
||||
|| abortProvenance === "global-pause"
|
||||
|| (live.paused && !mergeSeamAborted)
|
||||
|| (pausedAborted && !mergeSeamAborted),
|
||||
);
|
||||
if (genuinePauseAbort) {
|
||||
/*
|
||||
FNXC:WorkflowLifecycle 2026-06-15-01:45:
|
||||
FN-6478: a graph exit during an in-progress pause is recoverable by explicit unpause, but the same exit after the task has already left in-progress strands the workflow graph. Preserve userPaused and autoMerge:false review parking; surface non-in-progress paused exits as operator-actionable failures without moving the task backward or re-enqueueing execution.
|
||||
|
||||
FNXC:WorkflowLifecycle 2026-06-17-03:48:
|
||||
FN-6568: merge-seam aborts are not pause provenance. A non-paused merge-node failure must bypass this operator-action pause branch so FN-6528/FN-6531/FN-6534/FN-6537-style failures route to bounded auto-merge retry instead of being parked failed with mergeRetries=NULL.
|
||||
*/
|
||||
const pauseProvenance = live.userPaused
|
||||
? "explicit user pause"
|
||||
: pausedAborted
|
||||
? "engine abort during pause/resume"
|
||||
: "task pause";
|
||||
: abortProvenance === "global-pause"
|
||||
? "global pause"
|
||||
: pausedAborted
|
||||
? "engine abort during pause/resume"
|
||||
: "task pause";
|
||||
if (live.column !== "in-progress") {
|
||||
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1] ?? "unknown";
|
||||
const message = `Workflow graph failure surfaced after paused ${pauseProvenance} in '${live.column}' at node '${failedNode}' — operator action required; retry or explicitly unpause/resume after inspecting the task`;
|
||||
@@ -6367,13 +6418,16 @@ export class TaskExecutor {
|
||||
await this.store.logEntry(task.id, benignMessage, undefined, this.getRunContextFor(task.id));
|
||||
return;
|
||||
}
|
||||
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1];
|
||||
if (this.isMergeGraphFailure(failedNode) && await this.routeGraphMergeFailureToRetry(live, result, abortProvenance)) {
|
||||
return;
|
||||
}
|
||||
if (live.column !== "in-progress") {
|
||||
const benignMessage = `Workflow graph run ended after task already advanced to '${live.column}' — no further action needed`;
|
||||
executorLog.log(`${task.id}: ${benignMessage}`);
|
||||
await this.store.logEntry(task.id, benignMessage, undefined, this.getRunContextFor(task.id));
|
||||
return;
|
||||
}
|
||||
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1];
|
||||
const failureValue = this.graphFailureValue(result);
|
||||
if (this.isAwaitingGraphFailureValue(failureValue)) {
|
||||
/*
|
||||
@@ -7150,13 +7204,13 @@ export class TaskExecutor {
|
||||
}
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
return;
|
||||
}
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
await this.store.logEntry(task.id, "Execution paused — step sessions terminated, moved to todo", undefined, this.getRunContextFor(task.id));
|
||||
await this.store.moveTask(task.id, "todo", { preserveResumeState: true });
|
||||
return;
|
||||
@@ -7322,11 +7376,11 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(task, worktreePath, settings, taskEnv);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (await this.parkTaskAfterWorkflowStepPause(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
return;
|
||||
}
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -7397,13 +7451,13 @@ export class TaskExecutor {
|
||||
await this.handleDepAbortCleanup(task.id, worktreePath);
|
||||
} else if (this.pausedAborted.has(task.id)) {
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
return;
|
||||
}
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
await this.store.logEntry(task.id, "Execution paused during step-session", undefined, this.getRunContextFor(task.id));
|
||||
await this.store.moveTask(task.id, "todo", { preserveResumeState: true });
|
||||
} else if (this.stuckAborted.has(task.id)) {
|
||||
@@ -8006,13 +8060,13 @@ export class TaskExecutor {
|
||||
// prompt to resolve gracefully instead of throwing.
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
return;
|
||||
}
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
wasPaused = true;
|
||||
if (await this.shouldFinalizeCompletedTask(task.id, taskDone)) {
|
||||
if (await this.shouldDeferCompletionForGlobalPause(task.id, "paused after completion")) {
|
||||
@@ -8038,7 +8092,7 @@ export class TaskExecutor {
|
||||
// scheduler re-dispatches while the old execution guard is still set.
|
||||
if (this.stuckAborted.has(task.id)) {
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
@@ -8100,12 +8154,12 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(task, worktreePath, settings, taskEnv);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (await this.parkTaskAfterWorkflowStepPause(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
wasPaused = true;
|
||||
return;
|
||||
}
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
wasPaused = true;
|
||||
}
|
||||
return;
|
||||
@@ -8371,12 +8425,12 @@ export class TaskExecutor {
|
||||
const workflowResult = await this.runWorkflowSteps(task, worktreePath, settings, taskEnv);
|
||||
if (workflowResult === "deferred-paused") {
|
||||
if (await this.parkTaskAfterWorkflowStepPause(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
wasPaused = true;
|
||||
return;
|
||||
}
|
||||
if (this.pausedAborted.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
wasPaused = true;
|
||||
}
|
||||
return;
|
||||
@@ -8538,13 +8592,13 @@ export class TaskExecutor {
|
||||
} else if (this.pausedAborted.has(task.id)) {
|
||||
// Task was paused mid-execution — clean up worktree and move to todo
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
return;
|
||||
}
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
const latestTask = await this.store.getTask(task.id);
|
||||
if (
|
||||
latestTask?.column === "todo" &&
|
||||
@@ -8598,7 +8652,7 @@ export class TaskExecutor {
|
||||
// Task was killed by stuck task detector — defer requeue to finally block
|
||||
// (after this.executing is cleared) to prevent re-dispatch race.
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
@@ -9155,7 +9209,7 @@ export class TaskExecutor {
|
||||
// task in "in-progress" with no active session or worktree.
|
||||
if (stuckRequeue === true) {
|
||||
if (this.userCanceledTaskIds.has(task.id)) {
|
||||
this.pausedAborted.delete(task.id);
|
||||
this.clearPausedAborted(task.id);
|
||||
this.stuckAborted.delete(task.id);
|
||||
this.userCanceledTaskIds.delete(task.id);
|
||||
await this.store.logEntry(task.id, "Execution canceled by user — leaving task in todo");
|
||||
@@ -14335,7 +14389,7 @@ Backward compat fallback: if JSON is unavailable, you may still begin output wit
|
||||
// awaitAbortInFlightTaskWork marks pausedAborted as a generic hard-cancel
|
||||
// signal. The force-requeue path has already handled the task move, so
|
||||
// clear it to prevent a later subprocess unwind from logging/moving as a pause.
|
||||
this.pausedAborted.delete(taskId);
|
||||
this.clearPausedAborted(taskId);
|
||||
|
||||
if (!preserveProgress) {
|
||||
await this.resetStepsIfWorkLost(latestTask);
|
||||
|
||||
Reference in New Issue
Block a user