FN-6796: preserve benign in-review pause aborts
Preserve completed in-review tasks when benign pause/resume aborts surface after executor handoff. - Detect non-user hard-cancel pause aborts on clean completed in-review rows and clear the transient abort marker without moving the task. - Let self-healing recover persisted safe in-review pause-abort parks in place while keeping hard-cancel, pause, autoMerge:false, terminal merge, and live-execution guards intact. - Cover executor and self-healing recovery paths with regression tests, document the audit metadata, and add a patch changeset. Files changed: .../fn-6796-pause-resume-in-review-recovery.md | 5 ++ AGENTS.md | 1 + docs/architecture.md | 2 + .../engine/src/__tests__/executor-recovery.test.ts | 97 +++++++++++++++++++++- .../active-worktree-removal-liveness.test.ts | 2 +- .../self-healing-paused-abort-recovery.test.ts | 90 +++++++++++++++++++- packages/engine/src/executor.ts | 35 ++++++++ packages/engine/src/self-healing.ts | 38 +++++++-- 8 files changed, 257 insertions(+), 13 deletions(-) Fusion-Task-Id: FN-6796 Fusion-Task-Lineage: d2fe6c6d-c118-4f2f-b19e-58e6f1b29384
This commit is contained in:
gsxdsm
5
.changeset/fn-6796-pause-resume-in-review-recovery.md
Normal file
5
.changeset/fn-6796-pause-resume-in-review-recovery.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Recover benign in-review pause/resume abort parks without requiring operator intervention while preserving hard-cancel, pause, and terminal merge safeguards.
|
||||
@@ -193,6 +193,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
|
||||
- FN-6736: self-healing emits `task:reclaim-phantom-executor-binding` when it proves an in-memory executor-active binding is stale, clears the binding, and requeues the in-progress task with worktree/progress preserved.
|
||||
- FN-6783: task-store open and self-healing housekeeping emit `task:reconcile-orphaned-task-dir` when they non-destructively re-import a valid live `.fusion/tasks/{ID}/task.json` directory that has no task row anywhere, preserving soft-deleted/archived/tombstoned IDs.
|
||||
- FN-6782/FN-6796: self-healing emits `task:auto-recover-paused-abort-park` when it clears a benign pause-abort operator park, requeueing safe `todo`/`in-progress` rows or preserving a clean auto-merge-eligible `in-review` row for review progression.
|
||||
- FN-6793/FN-6797: self-healing emits `task:reconcile-in-review-unmet-dependencies` when it rebounds an `in-review` task whose declared dependencies are still unmet, and `task:reconcile-in-review-unmet-dependencies-no-action` when pause/user-pause, `autoMerge:false`, live execution/checkout proof, or a failed rebound mutation blocks that backward move.
|
||||
|
||||
|
||||
|
||||
@@ -679,6 +679,7 @@ Runtime action-gate flow (v1):
|
||||
- Worktrees-dir sweeps that list direct children of `<worktreesDir>` (pool idle scan, orphan cleanup/reap, self-healing unregistered-orphan reap, and cap enforcement) must exclude the `.ai-merge` container by name; those one-level sweeps never inspect or recycle clean rooms beneath it. Batch 1 sweeps stale AI merge clean-room worktrees under the new `<worktreesDir>/.ai-merge/` root and still scans legacy `.fusion/ai-merge/` plus legacy `tmpdir()` locations for pre-relocation leftovers; candidates are bounded to names starting with `fusion-ai-merge-`. `runAiMerge` registers each live clean-room worktree in `activeSessionRegistry` with kind `ai-merge` as soon as the directory exists and keeps both raw and canonical paths registered for the duration of the merge, so the dedicated periodic sweep and pre-merge prune defer when either path is active (including concurrent same-task merge attempts). The default age gate is 2 hours; task-aware cleanup uses a 10-minute grace period for `done`/`archived` tasks and for genuinely missing/deleted task rows, and every removal path is clamped by the same 10-minute minimum-age floor so a freshly created worktree is never reaped. Transient `getTask` lookup failures (for example SQLite busy/parse errors) are not treated as deletion evidence; they log a warning, emit `lookup-error` only if eventually removed, and retain the conservative 2-hour gate. The sweep canonicalizes paths before checking `activeSessionRegistry`, attempts `git worktree remove --force <path>` before filesystem removal, runs `git worktree prune` after cleanup attempts, and emits `worktree:tempdir-sweep` run-audit telemetry for removal attempts and failures. Fresh directories, active-session paths, and individual removal failures are skipped/logged without aborting the maintenance cycle.
|
||||
|
||||
- `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`.
|
||||
- `recoverPausedAbortFailures()` clears executor pause/resume abort parks only when the durable row is safe to recover. `todo`/`in-progress` rows are requeued for normal scheduling, while clean `in-review` rows (completed steps, not paused/user-paused/executing, auto-merge eligible, no confirmed or terminal merge evidence) have `status`/`error` cleared in place so review progression can continue. User hard-cancel, global/user pause, `autoMerge:false`, terminal merge, and live-execution guards remain operator-actionable. Successful recovery emits `task:auto-recover-paused-abort-park` with `preservedInReview` metadata.
|
||||
- `reattach-orphaned-assigned-executions` is a forward-resume safety net for durable-agent assignments. During startup recovery and periodic maintenance, after orphaned-agent and stale-heartbeat-run repairs, self-healing finds `in-progress` tasks with an `assignedAgentId` whose agent has no active heartbeat run and no active executor session after the orphan grace window. It re-dispatches in place via `executor.resumeTaskForAgent(agentId)` (the same seam used by clean `HeartbeatMonitor.onRunCompleted` and guarded by executor double-execution checks), emits `task:reattach-orphaned-execution`, and never moves the task backward. This complements engine-start `executor.resumeOrphaned()` and leaves unassigned/role-based execution recovery to the existing startup/limbo/stuck-task paths.
|
||||
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.
|
||||
|
||||
@@ -730,6 +731,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
||||
### Observability and reflection
|
||||
- `AgentLogger` (`agent-logger.ts`) — structured per-agent run logging
|
||||
- `RunAudit` (`run-audit.ts`) — mutation audit tracking (DB/git/filesystem)
|
||||
- FN-6782/FN-6796: `task:auto-recover-paused-abort-park` records self-healing recovery of pause-abort operator parks. Metadata includes the source column and whether recovery preserved a clean `in-review` row instead of requeueing to `todo`.
|
||||
- FN-4956: Layer 3 merge-conflict arbitration now scope-partitions conflicted files before AI resolution. Out-of-scope conflicts are deterministically resolved to the integration branch (`git checkout --ours`) and unstaged, while only in-scope conflicts flow to AI. Integration branch defaults are resolved via `resolveIntegrationBranch(rootDir, settings)`. Audit events: `merge:layer3:foreign-file-skipped` and `merge:layer3:scope-override-bypass`.
|
||||
- FN-5655 goal anchoring observability adds `database`-domain mutation types `goal:injection-applied`, `goal:injection-skipped`, and `goal:retrieval-invoked` so Slice 2 cite-rate tracking has a prompt-independent signal. Metadata uses counts/IDs only (`count`, `lane`, `toolName`, optional `truncated`/`reason`/`notFound`) and never stores prompt bodies or goal titles/descriptions. These events surface through `GET /api/agents/:id/runs/:runId/audit` and support the existing `startTime`/`endTime` filters.
|
||||
|
||||
|
||||
@@ -1200,7 +1200,102 @@ describe("TaskExecutor bounded recovery retries", () => {
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("surfaces genuine hard-cancel pausedAborted in-review graph exits as workflow failures", async () => {
|
||||
it("preserves clean completed in-review rows after benign engine-restart hard-cancel provenance without finalize log", async () => {
|
||||
const store = createMockStore();
|
||||
const steps = [
|
||||
{ name: "Preflight", status: "done" },
|
||||
{ name: "Implement", status: "done" },
|
||||
];
|
||||
const task = {
|
||||
id: "FN-001",
|
||||
title: "Test",
|
||||
description: "Test",
|
||||
column: "in-progress",
|
||||
status: undefined,
|
||||
dependencies: [],
|
||||
steps,
|
||||
currentStep: 1,
|
||||
// FNXC:WorkflowLifecycle 2026-06-20-00:00:
|
||||
// FN-6796 symptom coverage must omit the paused-after-completion finalize log so this exercises the benign in-review pause-abort classifier, not the older alreadyFinalizedToReview suppression path.
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Normal review handoff without paused-completion marker" }],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
} as Task;
|
||||
store.getTask.mockResolvedValue({
|
||||
...task,
|
||||
column: "in-review",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
status: null,
|
||||
error: null,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
(executor as any).markPausedAborted("FN-001", "hard-cancel");
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: ["execute"],
|
||||
});
|
||||
|
||||
const messages = store.logEntry.mock.calls.map((call) => call[1]).join("\n");
|
||||
expect(messages).toContain("Workflow graph run ended during engine pause/resume while already in-review — benign, in-review state preserved");
|
||||
expect(messages).not.toContain("Workflow graph failure surfaced after paused engine abort during pause/resume");
|
||||
expect(messages).not.toContain("operator action required");
|
||||
expect((executor as any).pausedAborted.has("FN-001")).toBe(false);
|
||||
expect(store.updateTask).not.toHaveBeenCalledWith(
|
||||
"FN-001",
|
||||
expect.objectContaining({ status: "failed" }),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("surfaces user hard-cancel in-review graph exits with completed steps as workflow failures", async () => {
|
||||
const store = createMockStore();
|
||||
const steps = [
|
||||
{ name: "Preflight", status: "done" },
|
||||
{ name: "Implement", status: "done" },
|
||||
];
|
||||
const task = {
|
||||
id: "FN-001",
|
||||
title: "Test",
|
||||
description: "Test",
|
||||
column: "in-progress",
|
||||
status: undefined,
|
||||
dependencies: [],
|
||||
steps,
|
||||
currentStep: 1,
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Normal review handoff without paused-completion marker" }],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
} as Task;
|
||||
store.getTask.mockResolvedValue({
|
||||
...task,
|
||||
column: "in-review",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
status: null,
|
||||
error: null,
|
||||
});
|
||||
const executor = new TaskExecutor(store, "/tmp/test", {});
|
||||
await (executor as any).awaitAbortInFlightTaskWork("FN-001", "user move in-progress to todo", { userCanceled: true });
|
||||
|
||||
await (executor as any).handleGraphFailure(task, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: ["execute"],
|
||||
});
|
||||
|
||||
const expectedMessage = "Workflow graph failure surfaced after paused engine abort during pause/resume in 'in-review' at node 'execute' — operator action required; retry or explicitly unpause/resume after inspecting the task";
|
||||
expect((executor as any).userCanceledTaskIds.has("FN-001")).toBe(true);
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-001", { error: expectedMessage, status: "failed" }, undefined);
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
expect(store.handoffToReview).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("surfaces incomplete hard-cancel pausedAborted in-review graph exits as workflow failures", async () => {
|
||||
const store = createMockStore();
|
||||
const steps = [
|
||||
{ name: "Preflight", status: "pending" },
|
||||
|
||||
@@ -217,7 +217,7 @@ describe("FN-4811: active worktree removal liveness gate", () => {
|
||||
expect(execCalls.some((c) => c.includes('git branch -D "fusion/fn-9999"'))).toBe(true);
|
||||
// Must have logged the stale-path cleanup outcome — NOT the generic failure log.
|
||||
const logCalls = store.logEntry.mock.calls.map((c: any[]) => String(c[1] ?? ""));
|
||||
expect(logCalls.some((m: string) => m.includes("Cleaned up stale conflicting worktree admin entry"))).toBe(true);
|
||||
expect(logCalls.some((m: string) => m.includes("Cleaned up stale conflicting worktree"))).toBe(true);
|
||||
expect(logCalls.some((m: string) => m === "Failed to clean up conflicting worktree")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -29,6 +29,9 @@ import type { Settings, Task, TaskStore } from "@fusion/core";
|
||||
|
||||
const PARK_ERROR =
|
||||
"Workflow graph failure surfaced after paused engine abort during pause/resume in 'todo' at node 'execute' — operator action required; retry or explicitly unpause/resume after inspecting the task";
|
||||
const IN_REVIEW_PARK_ERROR =
|
||||
"Workflow graph failure surfaced after paused engine abort during pause/resume in 'in-review' at node 'execute' — operator action required; retry or explicitly unpause/resume after inspecting the task";
|
||||
const DONE_STEPS = [{ status: "done" }, { status: "done" }];
|
||||
|
||||
function createMockStore(tasks: Task[]): TaskStore & EventEmitter {
|
||||
const emitter = new EventEmitter();
|
||||
@@ -114,11 +117,45 @@ describe("recoverPausedAbortFailures", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("skips paused, executing, in-review, and non-pause-abort failures", async () => {
|
||||
it("clears a completed in-review pause-abort park without moving it backward", async () => {
|
||||
const store = createMockStore([parkTask({
|
||||
id: "FN-7002",
|
||||
column: "in-review",
|
||||
error: IN_REVIEW_PARK_ERROR,
|
||||
steps: DONE_STEPS,
|
||||
autoMerge: true,
|
||||
})]);
|
||||
const clearBinding = vi.fn().mockReturnValue(true);
|
||||
const manager = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
clearPhantomExecutorBinding: clearBinding as (taskId: string) => boolean | void,
|
||||
});
|
||||
|
||||
const recovered = await manager.recoverPausedAbortFailures();
|
||||
|
||||
expect(recovered).toBe(1);
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-7002", { status: null, error: null });
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
expect(clearBinding).toHaveBeenCalledWith("FN-7002");
|
||||
expect(store.logEntry).toHaveBeenCalledWith(
|
||||
"FN-7002",
|
||||
"Auto-recovered: in-review pause-abort park cleared — preserved for normal review progression",
|
||||
);
|
||||
expect(store.recordRunAuditEvent).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
mutationType: "task:auto-recover-paused-abort-park",
|
||||
target: "FN-7002",
|
||||
metadata: { fromColumn: "in-review", preservedInReview: true },
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("skips paused, executing, incomplete in-review, and non-pause-abort failures", async () => {
|
||||
const store = createMockStore([
|
||||
parkTask({ id: "FN-A", paused: true }),
|
||||
parkTask({ id: "FN-B", column: "in-progress" }), // executing (below)
|
||||
parkTask({ id: "FN-C", column: "in-review" }), // in-review park left for operator
|
||||
parkTask({ id: "FN-C", column: "in-review", error: IN_REVIEW_PARK_ERROR }), // incomplete in-review park left for operator
|
||||
parkTask({ id: "FN-D", error: "some other failure", status: "failed" }),
|
||||
]);
|
||||
const manager = new SelfHealingManager(store, {
|
||||
@@ -132,6 +169,55 @@ describe("recoverPausedAbortFailures", () => {
|
||||
expect(store.updateTask).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("leaves guarded in-review pause-abort parks untouched", async () => {
|
||||
const candidates = [
|
||||
parkTask({ id: "FN-U", column: "in-review", error: IN_REVIEW_PARK_ERROR, steps: DONE_STEPS, userPaused: true, autoMerge: true }),
|
||||
parkTask({ id: "FN-X", column: "in-review", error: IN_REVIEW_PARK_ERROR, steps: DONE_STEPS, autoMerge: true }),
|
||||
parkTask({ id: "FN-M", column: "in-review", error: IN_REVIEW_PARK_ERROR, steps: DONE_STEPS, autoMerge: true, mergeDetails: { mergeConfirmed: true } as any }),
|
||||
parkTask({ id: "FN-T", column: "in-review", error: `${IN_REVIEW_PARK_ERROR} merge-conflict`, steps: DONE_STEPS, autoMerge: true }),
|
||||
parkTask({ id: "FN-A", column: "in-review", error: IN_REVIEW_PARK_ERROR, steps: DONE_STEPS, autoMerge: undefined }),
|
||||
];
|
||||
const store = createMockStore(candidates);
|
||||
(store.getSettings as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
autoMerge: false,
|
||||
globalPause: false,
|
||||
enginePaused: false,
|
||||
maintenanceIntervalMs: 0,
|
||||
} as unknown as Settings);
|
||||
const manager = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
getExecutingTaskIds: () => new Set<string>(["FN-X"]),
|
||||
});
|
||||
|
||||
const recovered = await manager.recoverPausedAbortFailures();
|
||||
|
||||
expect(recovered).toBe(0);
|
||||
expect(store.updateTask).not.toHaveBeenCalled();
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("revalidates in-review recovery against fresh state before clearing the park", async () => {
|
||||
const initial = parkTask({
|
||||
id: "FN-STALE",
|
||||
column: "in-review",
|
||||
error: IN_REVIEW_PARK_ERROR,
|
||||
steps: DONE_STEPS,
|
||||
autoMerge: true,
|
||||
});
|
||||
const store = createMockStore([initial]);
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({ ...initial, paused: true });
|
||||
const manager = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
const recovered = await manager.recoverPausedAbortFailures();
|
||||
|
||||
expect(recovered).toBe(0);
|
||||
expect(store.updateTask).not.toHaveBeenCalled();
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// FNXC:WorkflowLifecycle greptile P1 (PR #1687): the method self-guards on
|
||||
// global/engine pause at its own entry, so calling it directly (test/API path)
|
||||
// while the operator has frozen the board must be a no-op.
|
||||
|
||||
@@ -6529,6 +6529,32 @@ export class TaskExecutor {
|
||||
return true;
|
||||
}
|
||||
|
||||
private isBenignInReviewPauseAbort(
|
||||
live: TaskDetail,
|
||||
result: WorkflowGraphTaskRunResult,
|
||||
abortProvenance: "global-pause" | "merge-seam" | "hard-cancel" | "completion-finalize" | undefined,
|
||||
pausedAborted: boolean,
|
||||
userCanceled: boolean,
|
||||
): boolean {
|
||||
/*
|
||||
FNXC:WorkflowLifecycle 2026-06-20-00:00:
|
||||
FN-6796: an engine restart/pause-resume abort reaches graph-failure handling as `hard-cancel` provenance even when no user canceled the task. A clean completed `in-review` row in that shape is already handed off for review and must not be stranded with the operator-action pause-abort marker; the discriminator is the in-memory `userCanceledTaskIds` set plus the resting column and clean row state, while global/user pause, merge-seam, terminal merge values, merge-confirmed partial landings, and pre-existing status/error still park exactly as before.
|
||||
*/
|
||||
if (!pausedAborted) return false;
|
||||
if (abortProvenance !== "hard-cancel") return false;
|
||||
if (userCanceled) return false;
|
||||
if (live.column !== "in-review") return false;
|
||||
if (live.userPaused === true) return false;
|
||||
if (live.status != null || live.error != null) return false;
|
||||
if (live.mergeDetails?.mergeConfirmed === true) return false;
|
||||
if (this.isTerminalMergeGraphFailureValue(this.graphFailureValue(result))) return false;
|
||||
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1];
|
||||
if (this.isMergeGraphFailure(failedNode)) return false;
|
||||
if (live.steps.length === 0) return false;
|
||||
if (!live.steps.every((step) => step.status === "done" || step.status === "skipped")) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
private async routeGraphMergeFailureToRetry(
|
||||
live: TaskDetail,
|
||||
result: WorkflowGraphTaskRunResult,
|
||||
@@ -6622,6 +6648,15 @@ export class TaskExecutor {
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (genuinePauseAbort && this.isBenignInReviewPauseAbort(live, result, abortProvenance, pausedAborted, this.userCanceledTaskIds.has(task.id))) {
|
||||
this.clearPausedAborted(task.id);
|
||||
this.activeWorktrees.delete(task.id);
|
||||
const inReviewBenign = "Workflow graph run ended during engine pause/resume while already in-review — benign, in-review state preserved";
|
||||
executorLog.log(`${task.id}: ${inReviewBenign}`);
|
||||
await this.store.logEntry(task.id, inReviewBenign, undefined, this.getRunContextFor(task.id));
|
||||
await this.persistTokenUsage(task.id);
|
||||
return;
|
||||
}
|
||||
if (genuinePauseAbort) {
|
||||
/*
|
||||
FNXC:WorkflowLifecycle 2026-06-15-01:45:
|
||||
|
||||
@@ -8132,16 +8132,34 @@ export class SelfHealingManager {
|
||||
typeof t.error === "string" &&
|
||||
t.error.includes(PAUSE_ABORT_PARK_OPERATOR_MARKER) &&
|
||||
t.error.includes(PAUSE_ABORT_PARK_ERROR_MARKER);
|
||||
const isTerminalMergePark = (t: Task): boolean => {
|
||||
const text = typeof t.error === "string" ? t.error.toLowerCase() : "";
|
||||
return text.includes("conflict")
|
||||
|| text.includes("contamination")
|
||||
|| text.includes("foreign")
|
||||
|| text.includes("retry-exhausted")
|
||||
|| text.includes("retries exhausted")
|
||||
|| text.includes("max retries");
|
||||
};
|
||||
const isRecoverableInReviewPauseAbortPark = (t: Task): boolean => {
|
||||
/*
|
||||
FNXC:WorkflowLifecycle 2026-06-20-00:00:
|
||||
FN-6796 defense-in-depth: executor memory that distinguishes benign engine aborts from user hard-cancel is gone after restart, so self-healing may recover only persisted clean `in-review` pause-abort parks: non-paused, not executing, auto-merge eligible, completed steps, no terminal/confirmed merge evidence. User hard-cancel rows rest in `todo`; global/user pauses and autoMerge:false review rows remain operator-controlled.
|
||||
*/
|
||||
return t.column === "in-review"
|
||||
&& allowsAutoMergeProcessing(t, settings)
|
||||
&& t.mergeDetails?.mergeConfirmed !== true
|
||||
&& !isTerminalMergePark(t)
|
||||
&& t.steps.length > 0
|
||||
&& t.steps.every((step) => step.status === "done" || step.status === "skipped");
|
||||
};
|
||||
|
||||
const parked = tasks.filter((t) =>
|
||||
isPausedAbortPark(t) &&
|
||||
!t.paused &&
|
||||
!t.userPaused &&
|
||||
!executingIds.has(t.id) &&
|
||||
// Only recover columns that are safe to requeue. done/archived parks are
|
||||
// terminal and in-review parks may carry merge state — leave those for
|
||||
// the existing review recoverers / operator inspection.
|
||||
(t.column === "todo" || t.column === "in-progress"),
|
||||
(t.column === "todo" || t.column === "in-progress" || isRecoverableInReviewPauseAbortPark(t)),
|
||||
);
|
||||
|
||||
if (parked.length === 0) return 0;
|
||||
@@ -8165,13 +8183,13 @@ export class SelfHealingManager {
|
||||
fresh.paused ||
|
||||
fresh.userPaused ||
|
||||
latestExecutingIds.has(fresh.id) ||
|
||||
!(fresh.column === "todo" || fresh.column === "in-progress")
|
||||
!(fresh.column === "todo" || fresh.column === "in-progress" || isRecoverableInReviewPauseAbortPark(fresh))
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
await this.store.updateTask(task.id, { status: null, error: null });
|
||||
if (fresh.column !== "todo") {
|
||||
if (fresh.column !== "todo" && fresh.column !== "in-review") {
|
||||
await this.store.moveTask(task.id, "todo", {
|
||||
preserveProgress: true,
|
||||
moveSource: "engine",
|
||||
@@ -8187,7 +8205,9 @@ export class SelfHealingManager {
|
||||
|
||||
await this.store.logEntry(
|
||||
task.id,
|
||||
"Auto-recovered: pause-abort park cleared — requeued for normal scheduling",
|
||||
fresh.column === "in-review"
|
||||
? "Auto-recovered: in-review pause-abort park cleared — preserved for normal review progression"
|
||||
: "Auto-recovered: pause-abort park cleared — requeued for normal scheduling",
|
||||
);
|
||||
// FNXC:WorkflowLifecycle 2026-06-20-00:00: audit emission is strictly
|
||||
// best-effort — an audit throw AFTER the successful state mutation must
|
||||
@@ -8201,7 +8221,7 @@ export class SelfHealingManager {
|
||||
domain: "database",
|
||||
mutationType: "task:auto-recover-paused-abort-park",
|
||||
target: task.id,
|
||||
metadata: { fromColumn: fresh.column },
|
||||
metadata: { fromColumn: fresh.column, preservedInReview: fresh.column === "in-review" },
|
||||
});
|
||||
} catch (auditErr: unknown) {
|
||||
log.warn(`Pause-abort park audit emission failed for ${task.id}: ${auditErr instanceof Error ? auditErr.message : String(auditErr)}`);
|
||||
@@ -8214,7 +8234,7 @@ export class SelfHealingManager {
|
||||
}
|
||||
|
||||
if (recovered > 0) {
|
||||
log.log(`Recovered ${recovered} pause-abort park(s) → requeued to todo`);
|
||||
log.log(`Recovered ${recovered} pause-abort park(s) → requeued to todo or preserved in review`);
|
||||
}
|
||||
return recovered;
|
||||
} catch (err: unknown) { const errorMessage = err instanceof Error ? err.message : String(err);
|
||||
|
||||
Reference in New Issue
Block a user