fix(engine): stranded-completed promoters withhold tasks whose last execution ended in a failure park (#2257)
## What & why
FN-8141 laundered a failed task into `done`. The executor correctly
parked the task `failed` ("task parked failed during no-fn_task_done
retry" / "fn_task_done refusal retry budget exhausted"), the pause-abort
machinery bounced it to `todo`, and ~12 minutes later
`recoverStrandedCompletedTodoTasks` promoted it to `in-review` because
every step was done/skipped — overriding the honest failure park. From
there the AI merger found an empty diff and finalized it as a no-op
`done`, with no reviewer ever seeing it.
Existing exclusions (`task.error`,
`evaluateNoCommitsNoOpFinalize().blocked`, active statuses, refreshing
review state) all missed it because the failure provenance lived **only
in the durable task log** by the time the promoter ran — status/error
had been cleared by the pause-abort bounce.
This PR restores the invariant: **a stranded-completed promoter must not
promote a task whose most recent execution lifecycle ended in a
failure/refusal park.**
## Change
- New pure, unit-testable evaluator
`evaluateCompletedPromotionFailureProvenance(task)` in `@fusion/core`
(next to `no-commits-finalize-guard.ts`). It scans the task-log **tail**
(bounded to 250 entries) and lets the **most-recent execution-outcome
marker** decide: a failure/refusal park → `{ blocked: true, reason:
"failure-provenance" }`; a fresh clean completion (`Task marked done by
agent` / `All steps complete — implicit fn_task_done`) that appears more
recently supersedes an earlier park; zero failure markers → not blocked.
Recency is by construction, so a failure that predates a newer clean
execution is never reached.
- Both self-healing sweeps (`recoverCompletedTasks` stuck-in-progress
**and** `recoverStrandedCompletedTodoTasks` stranded-todo) fetch the
full task for candidates that already cleared the cheap slim filters
(slim listings strip `log`) and skip when blocked, emitting a
**deduped** `task:reconcile-stranded-completed-no-action` run-audit
event (ids/outcomes-only: `taskId`, `reason`, `sweep`, `marker?`).
- Defense-in-depth: the shared executor `recoverCompletedTask`
chokepoint — which the sweeps AND the executor's own
unpause/`resumeOrphaned` fast-paths all funnel through — also refuses a
provenance-blocked promotion, so no route can launder a failed park.
**Escape hatch (documented in FNXC comments):** an operator
retrying/moving the task starts a fresh execution whose clean-completion
marker supersedes the failure park, clearing the block with no code
change.
## Surface enumeration
- `recoverCompletedTasks` (stuck-in-progress sweep, self-healing.ts) —
guarded + audited.
- `recoverStrandedCompletedTodoTasks` (stranded-todo sweep,
self-healing.ts) — guarded + audited. FN-8141 shows both columns can
launder.
- `recoverCompletedTask` executor callback (the route both sweeps +
unpause + `resumeOrphaned` share) — verified it did **not** check
log-based provenance; added the guard there as the final chokepoint.
## Test evidence
Pure-evaluator unit tests (`@fusion/core`) — marker detection,
most-recent-outcome recency, supersede-by-clean-completion,
empty/missing log, tail-scan bound:
```
pnpm --filter @fusion/core exec vitest run src/__tests__/completed-promotion-failure-provenance.test.ts
Test Files 1 passed (1) Tests 9 passed (9)
```
Self-healing integration tests (`@fusion/engine`) — FN-8141-shaped todo
(3 done + 2 skipped + refusal-exhaust/park marker) is NOT promoted and
emits the no-action event exactly once (deduped across a second cycle);
same task after a fresh clean execution IS promoted; stuck-in-progress
variant covered:
```
pnpm --filter @fusion/engine exec vitest run src/__tests__/self-healing.test.ts -t "recoverCompletedTasks|recoverStrandedCompletedTodoTasks|FN-8141"
Test Files 1 passed (1) Tests 14 passed | 382 skipped (396)
```
`@fusion/core` builds clean. My engine changes add **zero** new type
errors (verified: all 13 engine build errors are the pre-existing pi-SDK
cluster in `auth-storage.ts`/`pi.ts`/`provider-registration.ts`, none in
`self-healing.ts`/`run-audit.ts`/`executor.ts`/the new file).
## Known environmental blocker
`pnpm verify:fast` cannot go green on this branch: the `@fusion/engine`
build is **already broken at baseline** (confirmed by stashing all my
changes) by the pi 0.80.x SDK migration errors
(`ModelRegistry`/`AuthStorage`/`ModelRuntime`) — the exact FN-8145
upstream breakage described in the FN-8141 incident. That is out of
scope for this task and independent of this diff. Likewise, the 22
pre-existing
`restart.integration.test.ts`/`executor-fast-mode-workflows.test.ts`
failures are identical with and without my changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus <noreply@anthropic.com>
This commit is contained in:
gsxdsm
7
.changeset/stranded-completed-failure-provenance.md
Normal file
7
.changeset/stranded-completed-failure-provenance.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Self-healing no longer promotes a failed/refused task into review after its work was reverted.
|
||||
category: fix
|
||||
dev: FN-8141 — new pure evaluator `evaluateCompletedPromotionFailureProvenance` (@fusion/core) reads the durable task log tail; both stranded-completed promoters (`recoverCompletedTasks` stuck-in-progress and `recoverStrandedCompletedTodoTasks` stranded-todo) and the shared `recoverCompletedTask` chokepoint now withhold promotion when the most recent execution-outcome was a failure/refusal park, emitting a deduped `task:reconcile-stranded-completed-no-action` run-audit event (reason `failure-provenance`). A fresh clean execution (operator retry) supersedes the park.
|
||||
@@ -0,0 +1,121 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { evaluateCompletedPromotionFailureProvenance, type TaskLogEntry } from "../index.js";
|
||||
|
||||
/**
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 invariant: a stranded-completed promotion candidate whose MOST RECENT execution-outcome
|
||||
* in the durable task log was a failure/refusal park must be blocked; a failure superseded by a
|
||||
* newer clean completion, or a task with zero failure markers, must not be blocked.
|
||||
*/
|
||||
|
||||
let seq = 0;
|
||||
function entry(action: string): TaskLogEntry {
|
||||
// Monotonic timestamps keep log order deterministic without relying on Date.now().
|
||||
seq += 1;
|
||||
return { timestamp: `2026-07-16T10:00:${String(seq).padStart(2, "0")}.000Z`, action };
|
||||
}
|
||||
|
||||
function log(actions: string[]): TaskLogEntry[] {
|
||||
return actions.map(entry);
|
||||
}
|
||||
|
||||
const FAILURE_PARK = "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying";
|
||||
const REFUSAL_EXHAUST = "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted";
|
||||
const CLEAN_DONE = "Task marked done by agent";
|
||||
const IMPLICIT_DONE = "All steps complete — implicit fn_task_done (agent did not call tool explicitly)";
|
||||
|
||||
describe("evaluateCompletedPromotionFailureProvenance", () => {
|
||||
it("blocks when the tail failure marker is the FN-8141 park", () => {
|
||||
const result = evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([
|
||||
"Starting execution",
|
||||
IMPLICIT_DONE,
|
||||
REFUSAL_EXHAUST,
|
||||
FAILURE_PARK,
|
||||
// pause-abort bounce to todo (not an execution-outcome marker)
|
||||
"Execution paused — session preserved for resume, moved to todo",
|
||||
]),
|
||||
});
|
||||
expect(result.blocked).toBe(true);
|
||||
expect(result.reason).toBe("failure-provenance");
|
||||
expect(result.markerAction).toContain("task parked failed during no-fn_task_done retry");
|
||||
});
|
||||
|
||||
it("blocks on the refusal-budget-exhaust marker even without the terminal park line", () => {
|
||||
const result = evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([IMPLICIT_DONE, REFUSAL_EXHAUST]),
|
||||
});
|
||||
expect(result).toMatchObject({ blocked: true, reason: "failure-provenance" });
|
||||
});
|
||||
|
||||
it("blocks on the retry-budget and implicit-refusal failure variants", () => {
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({
|
||||
log: log(["boom — execution failed after task-done retry budget was exhausted"]),
|
||||
}).blocked,
|
||||
).toBe(true);
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({
|
||||
log: log(["nope — execution failed because implicit fn_task_done was refused"]),
|
||||
}).blocked,
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("does NOT block when a fresh clean completion supersedes an earlier failure park", () => {
|
||||
const result = evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([
|
||||
REFUSAL_EXHAUST,
|
||||
FAILURE_PARK,
|
||||
"Execution paused — session preserved for resume, moved to todo",
|
||||
// operator retried → fresh execution completed all steps cleanly
|
||||
"Resuming execution after unpause",
|
||||
CLEAN_DONE,
|
||||
]),
|
||||
});
|
||||
expect(result).toEqual({ blocked: false });
|
||||
});
|
||||
|
||||
it("does NOT block when the clean completion is the implicit-done variant", () => {
|
||||
const result = evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([FAILURE_PARK, IMPLICIT_DONE]),
|
||||
});
|
||||
expect(result).toEqual({ blocked: false });
|
||||
});
|
||||
|
||||
it("does NOT block a task with zero failure markers", () => {
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({
|
||||
log: log(["Starting execution", CLEAN_DONE]),
|
||||
}),
|
||||
).toEqual({ blocked: false });
|
||||
});
|
||||
|
||||
it("does NOT block an empty or missing log", () => {
|
||||
expect(evaluateCompletedPromotionFailureProvenance({ log: [] })).toEqual({ blocked: false });
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({ log: undefined as unknown as TaskLogEntry[] }),
|
||||
).toEqual({ blocked: false });
|
||||
});
|
||||
|
||||
it("treats the MOST RECENT outcome as authoritative regardless of earlier markers", () => {
|
||||
// failure → clean → failure again: the tail failure wins.
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([FAILURE_PARK, CLEAN_DONE, REFUSAL_EXHAUST]),
|
||||
}).blocked,
|
||||
).toBe(true);
|
||||
// clean → failure → clean: the tail clean wins.
|
||||
expect(
|
||||
evaluateCompletedPromotionFailureProvenance({
|
||||
log: log([CLEAN_DONE, REFUSAL_EXHAUST, CLEAN_DONE]),
|
||||
}).blocked,
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("bounds the scan to the tail: an ancient failure beyond the window is not reached", () => {
|
||||
// One failure marker followed by >250 benign, non-outcome entries: the scan window never
|
||||
// reaches the failure, so no failure provenance is detected (not blocked).
|
||||
const ancientFailure = [FAILURE_PARK, ...Array.from({ length: 300 }, (_v, i) => `heartbeat ${i}`)];
|
||||
expect(evaluateCompletedPromotionFailureProvenance({ log: log(ancientFailure) }).blocked).toBe(false);
|
||||
});
|
||||
});
|
||||
90
packages/core/src/completed-promotion-failure-provenance.ts
Normal file
90
packages/core/src/completed-promotion-failure-provenance.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
import type { Task } from "./types.js";
|
||||
|
||||
export interface CompletedPromotionFailureProvenanceEvaluation {
|
||||
/** True when the task's current execution lifecycle ended in a failure/refusal park. */
|
||||
blocked: boolean;
|
||||
/** Stable reason code for the no-action run-audit event. */
|
||||
reason?: "failure-provenance";
|
||||
/** The blocking failure-marker log action text (diagnostics only). */
|
||||
markerAction?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 laundered a failed task into `done`: the executor parked the task `failed`
|
||||
* ("task parked failed during no-fn_task_done retry" / "fn_task_done refusal retry budget exhausted"),
|
||||
* the pause-abort machinery bounced it to `todo`, and 12 minutes later the stranded-completed
|
||||
* promoters (`recoverStrandedCompletedTodoTasks` / `recoverCompletedTasks` in self-healing.ts)
|
||||
* moved it to `in-review` because every step was done/skipped — overriding the honest failure park.
|
||||
*
|
||||
* Invariant restored: a stranded-completed promoter must NOT promote a task whose MOST RECENT
|
||||
* execution-outcome in the durable task log was a failure/refusal park. The escape hatch stays
|
||||
* intact because an operator retrying/moving the task produces a fresh execution that logs a clean
|
||||
* completion marker ("Task marked done by agent" / "All steps complete — implicit fn_task_done"),
|
||||
* which is more recent than the failure marker and therefore supersedes it.
|
||||
*
|
||||
* Recency by construction: we scan the log tail and stop at the FIRST (most recent) entry that is
|
||||
* either a failure park or a clean completion. A failure that predates a newer clean execution is
|
||||
* never reached — the completion marker decides first. A task with zero failure markers is never
|
||||
* blocked. The scan is bounded to the tail so these per-housekeeping-cycle sweeps stay cheap.
|
||||
*/
|
||||
|
||||
/** Bound the tail scan; sweeps run every housekeeping cycle over many tasks. */
|
||||
const MAX_LOG_SCAN = 250;
|
||||
|
||||
/**
|
||||
* Log-action substrings that mark the current execution lifecycle ending in a failure/refusal park.
|
||||
* Sources (packages/engine/src/executor.ts):
|
||||
* - "task parked failed during no-fn_task_done retry" (FN-7965 terminal park honoring)
|
||||
* - "fn_task_done refusal retry budget exhausted" (explicit fn_task_done refusal exhaustion)
|
||||
* - "execution failed after task-done retry budget was exhausted" (retry-budget failure)
|
||||
* - "execution failed because implicit fn_task_done was refused" (implicit-completion refusal)
|
||||
*/
|
||||
const FAILURE_PARK_MARKERS = [
|
||||
"task parked failed during no-fn_task_done retry",
|
||||
"fn_task_done refusal retry budget exhausted",
|
||||
"execution failed after task-done retry budget was exhausted",
|
||||
"execution failed because implicit fn_task_done was refused",
|
||||
];
|
||||
|
||||
/**
|
||||
* Log-action substrings that mark a fresh clean execution outcome. A clean completion appearing
|
||||
* MORE RECENTLY than a failure park proves the failing lifecycle was superseded by a good one.
|
||||
* Sources (packages/engine/src/executor.ts):
|
||||
* - "Task marked done by agent" (explicit fn_task_done success)
|
||||
* - "All steps complete — implicit fn_task_done" (implicit-completion success)
|
||||
* - "Auto-recovered: task work was complete but stranded" (stranded-completion recovery)
|
||||
*/
|
||||
const CLEAN_COMPLETION_MARKERS = [
|
||||
"Task marked done by agent",
|
||||
"All steps complete — implicit fn_task_done",
|
||||
"Auto-recovered: task work was complete but stranded",
|
||||
];
|
||||
|
||||
function matchesAny(text: string, markers: string[]): boolean {
|
||||
return markers.some((marker) => text.includes(marker));
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluate whether a stranded-completed promotion candidate should be withheld because its current
|
||||
* execution lifecycle ended in a failure/refusal park. Pure and unit-testable; both self-healing
|
||||
* sweeps share it. Requires the full task `log` (slim listings strip it, so promoters must fetch the
|
||||
* full task for candidates before calling this).
|
||||
*/
|
||||
export function evaluateCompletedPromotionFailureProvenance(
|
||||
task: Pick<Task, "log">,
|
||||
): CompletedPromotionFailureProvenanceEvaluation {
|
||||
const log = task.log ?? [];
|
||||
// Walk from the tail so the most recent execution-outcome marker decides. Cap the scan window.
|
||||
const scanFloor = Math.max(0, log.length - MAX_LOG_SCAN);
|
||||
for (let i = log.length - 1; i >= scanFloor; i--) {
|
||||
const action = log[i]?.action ?? "";
|
||||
if (matchesAny(action, FAILURE_PARK_MARKERS)) {
|
||||
return { blocked: true, reason: "failure-provenance", markerAction: action };
|
||||
}
|
||||
if (matchesAny(action, CLEAN_COMPLETION_MARKERS)) {
|
||||
return { blocked: false };
|
||||
}
|
||||
}
|
||||
return { blocked: false };
|
||||
}
|
||||
@@ -723,6 +723,8 @@ export {
|
||||
} from "./no-op-completion-marker.js";
|
||||
export { evaluateNoCommitsNoOpFinalize } from "./no-commits-finalize-guard.js";
|
||||
export type { NoCommitsNoOpFinalizeEvaluation } from "./no-commits-finalize-guard.js";
|
||||
export { evaluateCompletedPromotionFailureProvenance } from "./completed-promotion-failure-provenance.js";
|
||||
export type { CompletedPromotionFailureProvenanceEvaluation } from "./completed-promotion-failure-provenance.js";
|
||||
export {
|
||||
__getDeterministicGuardMutexSize,
|
||||
deterministicGuardLocks,
|
||||
|
||||
@@ -2408,6 +2408,51 @@ describe("SelfHealingManager", () => {
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 — the same laundering can start from the in-progress column. The stuck-in-progress
|
||||
promoter must also withhold an all-steps-done/skipped task whose most recent execution ended in a
|
||||
failure/refusal park, and emit `task:reconcile-stranded-completed-no-action` (sweep stuck-in-progress).
|
||||
*/
|
||||
it("FN-8141: does NOT promote a stuck-in-progress task whose last execution ended in a failure park", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141-IP",
|
||||
column: "in-progress",
|
||||
paused: false,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "skipped" }],
|
||||
},
|
||||
]);
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141-IP",
|
||||
lineageId: "lin-8141-ip",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141-IP: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverCompletedTasks();
|
||||
|
||||
expect(result).toBe(0);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
const emitted = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls.some(
|
||||
([ev]) =>
|
||||
(ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" &&
|
||||
(ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stuck-in-progress",
|
||||
);
|
||||
expect(emitted).toBe(true);
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
it("skips tasks that are actively executing", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const getExecuting = vi.fn().mockReturnValue(new Set(["FN-001"]));
|
||||
@@ -3017,6 +3062,107 @@ describe("SelfHealingManager", () => {
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 — the stranded-todo promoter must not launder a failed task into in-review. A candidate
|
||||
with all steps done/skipped whose MOST RECENT durable-log execution-outcome is a failure/refusal
|
||||
park is withheld and emits `task:reconcile-stranded-completed-no-action` (reason failure-provenance)
|
||||
once; the same task after a fresh clean execution IS promoted; the escape hatch is operator retry.
|
||||
*/
|
||||
it("FN-8141: does NOT promote a stranded-todo task whose last execution ended in a failure park, and emits the no-action event once", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
// Slim board row: 3 done + 2 skipped, no error/active status (exactly the FN-8141 shape that
|
||||
// passed all existing exclusions).
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141",
|
||||
column: "todo",
|
||||
paused: false,
|
||||
error: null,
|
||||
reviewLevel: 2,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "done" }, { status: "skipped" }, { status: "skipped" }],
|
||||
},
|
||||
]);
|
||||
// Full task carries the durable failure-park provenance the slim row cannot.
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141",
|
||||
lineageId: "lin-8141",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:01.000Z", action: "All steps complete — implicit fn_task_done (agent did not call tool explicitly)" },
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
{ timestamp: "2026-07-16T10:00:04.000Z", action: "Execution paused — session preserved for resume, moved to todo" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
|
||||
expect(result).toBe(0);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
const auditCalls = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls;
|
||||
const noActionEvents = auditCalls.filter(
|
||||
([ev]) =>
|
||||
(ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" &&
|
||||
(ev as { metadata?: { reason?: string; sweep?: string } }).metadata?.reason === "failure-provenance" &&
|
||||
(ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stranded-todo",
|
||||
);
|
||||
expect(noActionEvents).toHaveLength(1);
|
||||
|
||||
// Deduped: a second sweep with the same unchanged provenance does not re-emit.
|
||||
await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
const noActionEventsAfter = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls.filter(
|
||||
([ev]) => (ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action",
|
||||
);
|
||||
expect(noActionEventsAfter).toHaveLength(1);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
it("FN-8141: DOES promote the same task once a fresh clean execution completes all steps after the failure park", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141",
|
||||
column: "todo",
|
||||
paused: false,
|
||||
error: null,
|
||||
reviewLevel: 2,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "done" }],
|
||||
},
|
||||
]);
|
||||
// Log: the old failure park is followed by a fresh clean completion (operator retry escape hatch).
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141",
|
||||
lineageId: "lin-8141",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
{ timestamp: "2026-07-16T10:00:04.000Z", action: "Resuming execution after unpause" },
|
||||
{ timestamp: "2026-07-16T10:00:05.000Z", action: "Task marked done by agent" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(recoverFn).toHaveBeenCalledWith(expect.objectContaining({ id: "FN-8141" }));
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("recoverMissingWorktreeReviewFailures", () => {
|
||||
|
||||
@@ -14,7 +14,7 @@ import { existsSync, lstatSync, realpathSync } from "node:fs";
|
||||
import { readFile, rm, writeFile } from "node:fs/promises";
|
||||
import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, AsyncMissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings, MergeResult, WorkflowIrNode, WorkflowIrNodeKind, WorkflowStepResult as CoreWorkflowStepResult, ThinkingLevel } from "@fusion/core";
|
||||
import { getUnmetSchedulingDependencies } from "./scheduler.js";
|
||||
import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core";
|
||||
import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, evaluateCompletedPromotionFailureProvenance, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core";
|
||||
import { finalizeProvenAutoMergeTask } from "./auto-merge-finalization.js";
|
||||
import { mergeEffectiveSettings } from "./effective-settings.js";
|
||||
import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "./replan-target.js";
|
||||
@@ -4422,6 +4422,23 @@ export class TaskExecutor {
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 defense-in-depth: recoverCompletedTask is the shared promotion chokepoint for BOTH
|
||||
self-healing sweeps AND the executor's own unpause / resumeOrphaned fast-paths. A task whose
|
||||
most recent execution-outcome in the durable log was a failure/refusal park must not be
|
||||
promoted to in-review by ANY route, even one that re-derived completion from all-steps-done/
|
||||
skipped (skipped counts as complete, which is exactly how FN-8141 laundered a failed task).
|
||||
The self-healing sweeps additionally emit the deduped no-action audit event; here we simply
|
||||
refuse. Escape hatch: an operator retrying the task starts a fresh execution whose clean
|
||||
completion marker supersedes the failure park, clearing this block with no code change.
|
||||
*/
|
||||
const failureProvenance = evaluateCompletedPromotionFailureProvenance(liveForCompletenessCheck ?? task);
|
||||
if (failureProvenance.blocked) {
|
||||
executorLog.log(`${task.id}: skipping recoverCompletedTask — most recent execution ended in a failure/refusal park (operator-decides)`);
|
||||
return false;
|
||||
}
|
||||
|
||||
const settings = await this.store.getSettings();
|
||||
if (settings.globalPause || settings.enginePaused) {
|
||||
executorLog.log(
|
||||
|
||||
@@ -807,7 +807,17 @@ export type DatabaseMutationType =
|
||||
* withheld state persists unchanged.
|
||||
* Metadata: { taskId: string; reason: "user-paused" | "auto-merge-off-human-review"; stage?: string; oversightLevel?: string }
|
||||
*/
|
||||
| "overseer:oversight-withheld-human-control";
|
||||
| "overseer:oversight-withheld-human-control"
|
||||
/**
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 no-action event: a stranded-completed promoter (`recoverCompletedTasks` stuck-in-progress
|
||||
* sweep OR `recoverStrandedCompletedTodoTasks` stranded-todo sweep in self-healing.ts) withheld
|
||||
* promotion of an all-steps-done/skipped task because its most recent execution-outcome in the
|
||||
* durable task log was a failure/refusal park (`evaluateCompletedPromotionFailureProvenance`).
|
||||
* Emitted at most once per taskId while the blocking provenance persists (deduped in-memory).
|
||||
* Metadata: { taskId, reason: "failure-provenance", sweep: "stuck-in-progress" | "stranded-todo", marker?: string }
|
||||
*/
|
||||
| "task:reconcile-stranded-completed-no-action";
|
||||
|
||||
// ── Filesystem mutation types ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ import { setImmediate as setImmediateCb } from "node:timers";
|
||||
import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { isAbsolute, join, relative, resolve } from "node:path";
|
||||
import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core";
|
||||
import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, evaluateCompletedPromotionFailureProvenance, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core";
|
||||
import type { MeshLeaseManager } from "./mesh-lease-manager.js";
|
||||
import { createLogger, schedulerLog } from "./logger.js";
|
||||
import { mergeEffectiveSettings } from "./effective-settings.js";
|
||||
@@ -743,6 +743,14 @@ export class SelfHealingManager {
|
||||
private workspacePartialLandDrops: Map<string, number> = new Map();
|
||||
private orphanWorktreeRemovalFailures: Map<string, number> = new Map();
|
||||
private finalizeUnprovenWarned = new Set<string>();
|
||||
/*
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 dedup: `task:reconcile-stranded-completed-no-action` is emitted at most once per taskId
|
||||
* while a failure-provenance block persists, so the sweeps don't spam run-audit every housekeeping
|
||||
* cycle. Cleared on stop(); a fresh clean execution that clears the block re-arms emission naturally
|
||||
* because the task leaves the promotable state before returning to it.
|
||||
*/
|
||||
private strandedCompletedFailureProvenanceWarned = new Set<string>();
|
||||
private metaResolvedSkipAuditMemo = new Map<string, string>();
|
||||
private metaStalledSkipAuditMemo = new Map<string, string>();
|
||||
private preservedQueuedOverlapLogged = new Map<string, string>();
|
||||
@@ -1045,6 +1053,68 @@ export class SelfHealingManager {
|
||||
log.log(`[${stage}] ${task.id}: triple-proof not satisfied — no action (operator-decides)`);
|
||||
}
|
||||
|
||||
/*
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 — a stranded-completed promoter must respect failure provenance. Before promoting an
|
||||
* all-steps-done/skipped candidate to in-review, verify its MOST RECENT execution-outcome in the
|
||||
* durable task log was not a failure/refusal park; a park bounced to `todo`/`in-progress` by the
|
||||
* pause-abort machinery must not be laundered into `in-review`. Slim listings strip `log`, so the
|
||||
* full task is fetched only for candidates that already cleared the cheap step/status filters.
|
||||
* On block, emits `task:reconcile-stranded-completed-no-action` once per taskId (deduped) and
|
||||
* withholds promotion. Escape hatch: an operator retrying/moving the task starts a fresh execution
|
||||
* whose clean completion marker supersedes the failure park, clearing the block with no code change.
|
||||
* Fail-closed: if the full task cannot be read this cycle, withhold promotion rather than risk
|
||||
* laundering a failed park.
|
||||
*/
|
||||
private async isStrandedCompletedPromotionBlockedByFailureProvenance(
|
||||
taskId: string,
|
||||
sweep: "stuck-in-progress" | "stranded-todo",
|
||||
): Promise<boolean> {
|
||||
let fullTask: Task;
|
||||
try {
|
||||
fullTask = await this.store.getTask(taskId);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log.warn(`[stranded-completed-provenance] ${taskId}: full-task fetch failed (${message}) — withholding ${sweep} promotion this cycle`);
|
||||
return true;
|
||||
}
|
||||
|
||||
const evaluation = evaluateCompletedPromotionFailureProvenance(fullTask);
|
||||
if (!evaluation.blocked) {
|
||||
// Provenance cleared (fresh clean execution or never-failed): re-arm the dedup so a later
|
||||
// re-failure is reported again.
|
||||
this.strandedCompletedFailureProvenanceWarned.delete(taskId);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!this.strandedCompletedFailureProvenanceWarned.has(taskId)) {
|
||||
this.strandedCompletedFailureProvenanceWarned.add(taskId);
|
||||
try {
|
||||
await createRunAuditor(this.store, {
|
||||
runId: generateSyntheticRunId("self-healing-stranded-completed-provenance", taskId),
|
||||
agentId: "self-healing",
|
||||
taskId,
|
||||
taskLineageId: fullTask.lineageId,
|
||||
phase: "stranded-completed-provenance",
|
||||
}).database({
|
||||
type: "task:reconcile-stranded-completed-no-action" as DatabaseMutationType,
|
||||
target: taskId,
|
||||
metadata: {
|
||||
taskId,
|
||||
reason: evaluation.reason ?? "failure-provenance",
|
||||
sweep,
|
||||
...(evaluation.markerAction ? { marker: evaluation.markerAction } : {}),
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log.warn(`[stranded-completed-provenance] ${taskId}: no-action audit emission failed: ${message}`);
|
||||
}
|
||||
}
|
||||
log.log(`[stranded-completed-provenance] ${taskId}: withholding ${sweep} promotion — most recent execution ended in a failure/refusal park (operator-decides)`);
|
||||
return true;
|
||||
}
|
||||
|
||||
private async listActiveHeartbeatTaskIds(): Promise<Set<string>> {
|
||||
const activeTaskIds = new Set<string>();
|
||||
if (!this.options.agentStore) {
|
||||
@@ -1409,6 +1479,7 @@ export class SelfHealingManager {
|
||||
}
|
||||
|
||||
this.finalizeUnprovenWarned.clear();
|
||||
this.strandedCompletedFailureProvenanceWarned.clear();
|
||||
this.metaResolvedSkipAuditMemo.clear();
|
||||
this.metaStalledSkipAuditMemo.clear();
|
||||
this.preservedQueuedOverlapLogged.clear();
|
||||
@@ -2810,6 +2881,11 @@ export class SelfHealingManager {
|
||||
log.log(`${task.id} started executing concurrently — skipping recovery this cycle`);
|
||||
continue;
|
||||
}
|
||||
// FN-8141: never promote a stuck-in-progress task whose most recent execution ended in a
|
||||
// failure/refusal park — honor the honest failure instead of laundering it into in-review.
|
||||
if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stuck-in-progress")) {
|
||||
continue;
|
||||
}
|
||||
log.log(`Recovering completed task ${task.id}: ${task.title || task.description?.slice(0, 60) || "(untitled)"}`);
|
||||
const success = await recoverFn(task);
|
||||
if (success) recovered++;
|
||||
@@ -2866,6 +2942,11 @@ export class SelfHealingManager {
|
||||
log.log(`${task.id} started executing concurrently — skipping stranded todo recovery this cycle`);
|
||||
continue;
|
||||
}
|
||||
// FN-8141: never promote a stranded-todo task whose most recent execution ended in a
|
||||
// failure/refusal park — the pause-abort bounce to todo must not launder the failure.
|
||||
if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stranded-todo")) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const success = await recoverFn(task);
|
||||
if (success) recovered++;
|
||||
|
||||
Reference in New Issue
Block a user