From 136958fc6097aca8aba6c19afc5b4aa23b274eeb Mon Sep 17 00:00:00 2001 From: gsxdsm Date: Thu, 16 Jul 2026 19:36:57 -0700 Subject: [PATCH] fix(engine): stranded-completed promoters withhold tasks whose last execution ended in a failure park (#2257) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What & why FN-8141 laundered a failed task into `done`. The executor correctly parked the task `failed` ("task parked failed during no-fn_task_done retry" / "fn_task_done refusal retry budget exhausted"), the pause-abort machinery bounced it to `todo`, and ~12 minutes later `recoverStrandedCompletedTodoTasks` promoted it to `in-review` because every step was done/skipped — overriding the honest failure park. From there the AI merger found an empty diff and finalized it as a no-op `done`, with no reviewer ever seeing it. Existing exclusions (`task.error`, `evaluateNoCommitsNoOpFinalize().blocked`, active statuses, refreshing review state) all missed it because the failure provenance lived **only in the durable task log** by the time the promoter ran — status/error had been cleared by the pause-abort bounce. This PR restores the invariant: **a stranded-completed promoter must not promote a task whose most recent execution lifecycle ended in a failure/refusal park.** ## Change - New pure, unit-testable evaluator `evaluateCompletedPromotionFailureProvenance(task)` in `@fusion/core` (next to `no-commits-finalize-guard.ts`). It scans the task-log **tail** (bounded to 250 entries) and lets the **most-recent execution-outcome marker** decide: a failure/refusal park → `{ blocked: true, reason: "failure-provenance" }`; a fresh clean completion (`Task marked done by agent` / `All steps complete — implicit fn_task_done`) that appears more recently supersedes an earlier park; zero failure markers → not blocked. Recency is by construction, so a failure that predates a newer clean execution is never reached. - Both self-healing sweeps (`recoverCompletedTasks` stuck-in-progress **and** `recoverStrandedCompletedTodoTasks` stranded-todo) fetch the full task for candidates that already cleared the cheap slim filters (slim listings strip `log`) and skip when blocked, emitting a **deduped** `task:reconcile-stranded-completed-no-action` run-audit event (ids/outcomes-only: `taskId`, `reason`, `sweep`, `marker?`). - Defense-in-depth: the shared executor `recoverCompletedTask` chokepoint — which the sweeps AND the executor's own unpause/`resumeOrphaned` fast-paths all funnel through — also refuses a provenance-blocked promotion, so no route can launder a failed park. **Escape hatch (documented in FNXC comments):** an operator retrying/moving the task starts a fresh execution whose clean-completion marker supersedes the failure park, clearing the block with no code change. ## Surface enumeration - `recoverCompletedTasks` (stuck-in-progress sweep, self-healing.ts) — guarded + audited. - `recoverStrandedCompletedTodoTasks` (stranded-todo sweep, self-healing.ts) — guarded + audited. FN-8141 shows both columns can launder. - `recoverCompletedTask` executor callback (the route both sweeps + unpause + `resumeOrphaned` share) — verified it did **not** check log-based provenance; added the guard there as the final chokepoint. ## Test evidence Pure-evaluator unit tests (`@fusion/core`) — marker detection, most-recent-outcome recency, supersede-by-clean-completion, empty/missing log, tail-scan bound: ``` pnpm --filter @fusion/core exec vitest run src/__tests__/completed-promotion-failure-provenance.test.ts Test Files 1 passed (1) Tests 9 passed (9) ``` Self-healing integration tests (`@fusion/engine`) — FN-8141-shaped todo (3 done + 2 skipped + refusal-exhaust/park marker) is NOT promoted and emits the no-action event exactly once (deduped across a second cycle); same task after a fresh clean execution IS promoted; stuck-in-progress variant covered: ``` pnpm --filter @fusion/engine exec vitest run src/__tests__/self-healing.test.ts -t "recoverCompletedTasks|recoverStrandedCompletedTodoTasks|FN-8141" Test Files 1 passed (1) Tests 14 passed | 382 skipped (396) ``` `@fusion/core` builds clean. My engine changes add **zero** new type errors (verified: all 13 engine build errors are the pre-existing pi-SDK cluster in `auth-storage.ts`/`pi.ts`/`provider-registration.ts`, none in `self-healing.ts`/`run-audit.ts`/`executor.ts`/the new file). ## Known environmental blocker `pnpm verify:fast` cannot go green on this branch: the `@fusion/engine` build is **already broken at baseline** (confirmed by stashing all my changes) by the pi 0.80.x SDK migration errors (`ModelRegistry`/`AuthStorage`/`ModelRuntime`) — the exact FN-8145 upstream breakage described in the FN-8141 incident. That is out of scope for this task and independent of this diff. Likewise, the 22 pre-existing `restart.integration.test.ts`/`executor-fast-mode-workflows.test.ts` failures are identical with and without my changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus --- .../stranded-completed-failure-provenance.md | 7 + ...leted-promotion-failure-provenance.test.ts | 121 +++++++++++++++ .../completed-promotion-failure-provenance.ts | 90 +++++++++++ packages/core/src/index.ts | 2 + .../engine/src/__tests__/self-healing.test.ts | 146 ++++++++++++++++++ packages/engine/src/executor.ts | 19 ++- packages/engine/src/run-audit.ts | 12 +- packages/engine/src/self-healing.ts | 83 +++++++++- 8 files changed, 477 insertions(+), 3 deletions(-) create mode 100644 .changeset/stranded-completed-failure-provenance.md create mode 100644 packages/core/src/__tests__/completed-promotion-failure-provenance.test.ts create mode 100644 packages/core/src/completed-promotion-failure-provenance.ts diff --git a/.changeset/stranded-completed-failure-provenance.md b/.changeset/stranded-completed-failure-provenance.md new file mode 100644 index 0000000000..62ffba108b --- /dev/null +++ b/.changeset/stranded-completed-failure-provenance.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": patch +--- + +summary: Self-healing no longer promotes a failed/refused task into review after its work was reverted. +category: fix +dev: FN-8141 — new pure evaluator `evaluateCompletedPromotionFailureProvenance` (@fusion/core) reads the durable task log tail; both stranded-completed promoters (`recoverCompletedTasks` stuck-in-progress and `recoverStrandedCompletedTodoTasks` stranded-todo) and the shared `recoverCompletedTask` chokepoint now withhold promotion when the most recent execution-outcome was a failure/refusal park, emitting a deduped `task:reconcile-stranded-completed-no-action` run-audit event (reason `failure-provenance`). A fresh clean execution (operator retry) supersedes the park. diff --git a/packages/core/src/__tests__/completed-promotion-failure-provenance.test.ts b/packages/core/src/__tests__/completed-promotion-failure-provenance.test.ts new file mode 100644 index 0000000000..8e12c593ea --- /dev/null +++ b/packages/core/src/__tests__/completed-promotion-failure-provenance.test.ts @@ -0,0 +1,121 @@ +import { describe, expect, it } from "vitest"; +import { evaluateCompletedPromotionFailureProvenance, type TaskLogEntry } from "../index.js"; + +/** + * FNXC:Lifecycle 2026-07-16-10:30: + * FN-8141 invariant: a stranded-completed promotion candidate whose MOST RECENT execution-outcome + * in the durable task log was a failure/refusal park must be blocked; a failure superseded by a + * newer clean completion, or a task with zero failure markers, must not be blocked. + */ + +let seq = 0; +function entry(action: string): TaskLogEntry { + // Monotonic timestamps keep log order deterministic without relying on Date.now(). + seq += 1; + return { timestamp: `2026-07-16T10:00:${String(seq).padStart(2, "0")}.000Z`, action }; +} + +function log(actions: string[]): TaskLogEntry[] { + return actions.map(entry); +} + +const FAILURE_PARK = "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying"; +const REFUSAL_EXHAUST = "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted"; +const CLEAN_DONE = "Task marked done by agent"; +const IMPLICIT_DONE = "All steps complete — implicit fn_task_done (agent did not call tool explicitly)"; + +describe("evaluateCompletedPromotionFailureProvenance", () => { + it("blocks when the tail failure marker is the FN-8141 park", () => { + const result = evaluateCompletedPromotionFailureProvenance({ + log: log([ + "Starting execution", + IMPLICIT_DONE, + REFUSAL_EXHAUST, + FAILURE_PARK, + // pause-abort bounce to todo (not an execution-outcome marker) + "Execution paused — session preserved for resume, moved to todo", + ]), + }); + expect(result.blocked).toBe(true); + expect(result.reason).toBe("failure-provenance"); + expect(result.markerAction).toContain("task parked failed during no-fn_task_done retry"); + }); + + it("blocks on the refusal-budget-exhaust marker even without the terminal park line", () => { + const result = evaluateCompletedPromotionFailureProvenance({ + log: log([IMPLICIT_DONE, REFUSAL_EXHAUST]), + }); + expect(result).toMatchObject({ blocked: true, reason: "failure-provenance" }); + }); + + it("blocks on the retry-budget and implicit-refusal failure variants", () => { + expect( + evaluateCompletedPromotionFailureProvenance({ + log: log(["boom — execution failed after task-done retry budget was exhausted"]), + }).blocked, + ).toBe(true); + expect( + evaluateCompletedPromotionFailureProvenance({ + log: log(["nope — execution failed because implicit fn_task_done was refused"]), + }).blocked, + ).toBe(true); + }); + + it("does NOT block when a fresh clean completion supersedes an earlier failure park", () => { + const result = evaluateCompletedPromotionFailureProvenance({ + log: log([ + REFUSAL_EXHAUST, + FAILURE_PARK, + "Execution paused — session preserved for resume, moved to todo", + // operator retried → fresh execution completed all steps cleanly + "Resuming execution after unpause", + CLEAN_DONE, + ]), + }); + expect(result).toEqual({ blocked: false }); + }); + + it("does NOT block when the clean completion is the implicit-done variant", () => { + const result = evaluateCompletedPromotionFailureProvenance({ + log: log([FAILURE_PARK, IMPLICIT_DONE]), + }); + expect(result).toEqual({ blocked: false }); + }); + + it("does NOT block a task with zero failure markers", () => { + expect( + evaluateCompletedPromotionFailureProvenance({ + log: log(["Starting execution", CLEAN_DONE]), + }), + ).toEqual({ blocked: false }); + }); + + it("does NOT block an empty or missing log", () => { + expect(evaluateCompletedPromotionFailureProvenance({ log: [] })).toEqual({ blocked: false }); + expect( + evaluateCompletedPromotionFailureProvenance({ log: undefined as unknown as TaskLogEntry[] }), + ).toEqual({ blocked: false }); + }); + + it("treats the MOST RECENT outcome as authoritative regardless of earlier markers", () => { + // failure → clean → failure again: the tail failure wins. + expect( + evaluateCompletedPromotionFailureProvenance({ + log: log([FAILURE_PARK, CLEAN_DONE, REFUSAL_EXHAUST]), + }).blocked, + ).toBe(true); + // clean → failure → clean: the tail clean wins. + expect( + evaluateCompletedPromotionFailureProvenance({ + log: log([CLEAN_DONE, REFUSAL_EXHAUST, CLEAN_DONE]), + }).blocked, + ).toBe(false); + }); + + it("bounds the scan to the tail: an ancient failure beyond the window is not reached", () => { + // One failure marker followed by >250 benign, non-outcome entries: the scan window never + // reaches the failure, so no failure provenance is detected (not blocked). + const ancientFailure = [FAILURE_PARK, ...Array.from({ length: 300 }, (_v, i) => `heartbeat ${i}`)]; + expect(evaluateCompletedPromotionFailureProvenance({ log: log(ancientFailure) }).blocked).toBe(false); + }); +}); diff --git a/packages/core/src/completed-promotion-failure-provenance.ts b/packages/core/src/completed-promotion-failure-provenance.ts new file mode 100644 index 0000000000..fd78c33587 --- /dev/null +++ b/packages/core/src/completed-promotion-failure-provenance.ts @@ -0,0 +1,90 @@ +import type { Task } from "./types.js"; + +export interface CompletedPromotionFailureProvenanceEvaluation { + /** True when the task's current execution lifecycle ended in a failure/refusal park. */ + blocked: boolean; + /** Stable reason code for the no-action run-audit event. */ + reason?: "failure-provenance"; + /** The blocking failure-marker log action text (diagnostics only). */ + markerAction?: string; +} + +/** + * FNXC:Lifecycle 2026-07-16-10:30: + * FN-8141 laundered a failed task into `done`: the executor parked the task `failed` + * ("task parked failed during no-fn_task_done retry" / "fn_task_done refusal retry budget exhausted"), + * the pause-abort machinery bounced it to `todo`, and 12 minutes later the stranded-completed + * promoters (`recoverStrandedCompletedTodoTasks` / `recoverCompletedTasks` in self-healing.ts) + * moved it to `in-review` because every step was done/skipped — overriding the honest failure park. + * + * Invariant restored: a stranded-completed promoter must NOT promote a task whose MOST RECENT + * execution-outcome in the durable task log was a failure/refusal park. The escape hatch stays + * intact because an operator retrying/moving the task produces a fresh execution that logs a clean + * completion marker ("Task marked done by agent" / "All steps complete — implicit fn_task_done"), + * which is more recent than the failure marker and therefore supersedes it. + * + * Recency by construction: we scan the log tail and stop at the FIRST (most recent) entry that is + * either a failure park or a clean completion. A failure that predates a newer clean execution is + * never reached — the completion marker decides first. A task with zero failure markers is never + * blocked. The scan is bounded to the tail so these per-housekeeping-cycle sweeps stay cheap. + */ + +/** Bound the tail scan; sweeps run every housekeeping cycle over many tasks. */ +const MAX_LOG_SCAN = 250; + +/** + * Log-action substrings that mark the current execution lifecycle ending in a failure/refusal park. + * Sources (packages/engine/src/executor.ts): + * - "task parked failed during no-fn_task_done retry" (FN-7965 terminal park honoring) + * - "fn_task_done refusal retry budget exhausted" (explicit fn_task_done refusal exhaustion) + * - "execution failed after task-done retry budget was exhausted" (retry-budget failure) + * - "execution failed because implicit fn_task_done was refused" (implicit-completion refusal) + */ +const FAILURE_PARK_MARKERS = [ + "task parked failed during no-fn_task_done retry", + "fn_task_done refusal retry budget exhausted", + "execution failed after task-done retry budget was exhausted", + "execution failed because implicit fn_task_done was refused", +]; + +/** + * Log-action substrings that mark a fresh clean execution outcome. A clean completion appearing + * MORE RECENTLY than a failure park proves the failing lifecycle was superseded by a good one. + * Sources (packages/engine/src/executor.ts): + * - "Task marked done by agent" (explicit fn_task_done success) + * - "All steps complete — implicit fn_task_done" (implicit-completion success) + * - "Auto-recovered: task work was complete but stranded" (stranded-completion recovery) + */ +const CLEAN_COMPLETION_MARKERS = [ + "Task marked done by agent", + "All steps complete — implicit fn_task_done", + "Auto-recovered: task work was complete but stranded", +]; + +function matchesAny(text: string, markers: string[]): boolean { + return markers.some((marker) => text.includes(marker)); +} + +/** + * Evaluate whether a stranded-completed promotion candidate should be withheld because its current + * execution lifecycle ended in a failure/refusal park. Pure and unit-testable; both self-healing + * sweeps share it. Requires the full task `log` (slim listings strip it, so promoters must fetch the + * full task for candidates before calling this). + */ +export function evaluateCompletedPromotionFailureProvenance( + task: Pick, +): CompletedPromotionFailureProvenanceEvaluation { + const log = task.log ?? []; + // Walk from the tail so the most recent execution-outcome marker decides. Cap the scan window. + const scanFloor = Math.max(0, log.length - MAX_LOG_SCAN); + for (let i = log.length - 1; i >= scanFloor; i--) { + const action = log[i]?.action ?? ""; + if (matchesAny(action, FAILURE_PARK_MARKERS)) { + return { blocked: true, reason: "failure-provenance", markerAction: action }; + } + if (matchesAny(action, CLEAN_COMPLETION_MARKERS)) { + return { blocked: false }; + } + } + return { blocked: false }; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 3a5d8b9a1b..3de83bb894 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -723,6 +723,8 @@ export { } from "./no-op-completion-marker.js"; export { evaluateNoCommitsNoOpFinalize } from "./no-commits-finalize-guard.js"; export type { NoCommitsNoOpFinalizeEvaluation } from "./no-commits-finalize-guard.js"; +export { evaluateCompletedPromotionFailureProvenance } from "./completed-promotion-failure-provenance.js"; +export type { CompletedPromotionFailureProvenanceEvaluation } from "./completed-promotion-failure-provenance.js"; export { __getDeterministicGuardMutexSize, deterministicGuardLocks, diff --git a/packages/engine/src/__tests__/self-healing.test.ts b/packages/engine/src/__tests__/self-healing.test.ts index f67a32640a..6c446466ab 100644 --- a/packages/engine/src/__tests__/self-healing.test.ts +++ b/packages/engine/src/__tests__/self-healing.test.ts @@ -2408,6 +2408,51 @@ describe("SelfHealingManager", () => { managerWithRecovery.stop(); }); + /* + FNXC:Lifecycle 2026-07-16-10:30: + FN-8141 — the same laundering can start from the in-progress column. The stuck-in-progress + promoter must also withhold an all-steps-done/skipped task whose most recent execution ended in a + failure/refusal park, and emit `task:reconcile-stranded-completed-no-action` (sweep stuck-in-progress). + */ + it("FN-8141: does NOT promote a stuck-in-progress task whose last execution ended in a failure park", async () => { + const recoverFn = vi.fn().mockResolvedValue(true); + const managerWithRecovery = new SelfHealingManager(store, { + rootDir: "/tmp/test-project", + recoverCompletedTask: recoverFn, + getExecutingTaskIds: () => new Set(), + }); + + (store.listTasks as ReturnType).mockResolvedValue([ + { + id: "FN-8141-IP", + column: "in-progress", + paused: false, + steps: [{ status: "done" }, { status: "done" }, { status: "skipped" }], + }, + ]); + (store.getTask as ReturnType).mockResolvedValue({ + id: "FN-8141-IP", + lineageId: "lin-8141-ip", + log: [ + { timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" }, + { timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141-IP: task parked failed during no-fn_task_done retry — honoring park, not retrying" }, + ], + }); + + const result = await managerWithRecovery.recoverCompletedTasks(); + + expect(result).toBe(0); + expect(recoverFn).not.toHaveBeenCalled(); + const emitted = (store.recordRunAuditEvent as ReturnType).mock.calls.some( + ([ev]) => + (ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" && + (ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stuck-in-progress", + ); + expect(emitted).toBe(true); + + managerWithRecovery.stop(); + }); + it("skips tasks that are actively executing", async () => { const recoverFn = vi.fn().mockResolvedValue(true); const getExecuting = vi.fn().mockReturnValue(new Set(["FN-001"])); @@ -3017,6 +3062,107 @@ describe("SelfHealingManager", () => { managerWithRecovery.stop(); }); + + /* + FNXC:Lifecycle 2026-07-16-10:30: + FN-8141 — the stranded-todo promoter must not launder a failed task into in-review. A candidate + with all steps done/skipped whose MOST RECENT durable-log execution-outcome is a failure/refusal + park is withheld and emits `task:reconcile-stranded-completed-no-action` (reason failure-provenance) + once; the same task after a fresh clean execution IS promoted; the escape hatch is operator retry. + */ + it("FN-8141: does NOT promote a stranded-todo task whose last execution ended in a failure park, and emits the no-action event once", async () => { + const recoverFn = vi.fn().mockResolvedValue(true); + const managerWithRecovery = new SelfHealingManager(store, { + rootDir: "/tmp/test-project", + recoverCompletedTask: recoverFn, + getExecutingTaskIds: () => new Set(), + }); + + // Slim board row: 3 done + 2 skipped, no error/active status (exactly the FN-8141 shape that + // passed all existing exclusions). + (store.listTasks as ReturnType).mockResolvedValue([ + { + id: "FN-8141", + column: "todo", + paused: false, + error: null, + reviewLevel: 2, + steps: [{ status: "done" }, { status: "done" }, { status: "done" }, { status: "skipped" }, { status: "skipped" }], + }, + ]); + // Full task carries the durable failure-park provenance the slim row cannot. + (store.getTask as ReturnType).mockResolvedValue({ + id: "FN-8141", + lineageId: "lin-8141", + log: [ + { timestamp: "2026-07-16T10:00:01.000Z", action: "All steps complete — implicit fn_task_done (agent did not call tool explicitly)" }, + { timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" }, + { timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" }, + { timestamp: "2026-07-16T10:00:04.000Z", action: "Execution paused — session preserved for resume, moved to todo" }, + ], + }); + + const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks(); + + expect(result).toBe(0); + expect(recoverFn).not.toHaveBeenCalled(); + const auditCalls = (store.recordRunAuditEvent as ReturnType).mock.calls; + const noActionEvents = auditCalls.filter( + ([ev]) => + (ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" && + (ev as { metadata?: { reason?: string; sweep?: string } }).metadata?.reason === "failure-provenance" && + (ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stranded-todo", + ); + expect(noActionEvents).toHaveLength(1); + + // Deduped: a second sweep with the same unchanged provenance does not re-emit. + await managerWithRecovery.recoverStrandedCompletedTodoTasks(); + const noActionEventsAfter = (store.recordRunAuditEvent as ReturnType).mock.calls.filter( + ([ev]) => (ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action", + ); + expect(noActionEventsAfter).toHaveLength(1); + expect(recoverFn).not.toHaveBeenCalled(); + + managerWithRecovery.stop(); + }); + + it("FN-8141: DOES promote the same task once a fresh clean execution completes all steps after the failure park", async () => { + const recoverFn = vi.fn().mockResolvedValue(true); + const managerWithRecovery = new SelfHealingManager(store, { + rootDir: "/tmp/test-project", + recoverCompletedTask: recoverFn, + getExecutingTaskIds: () => new Set(), + }); + + (store.listTasks as ReturnType).mockResolvedValue([ + { + id: "FN-8141", + column: "todo", + paused: false, + error: null, + reviewLevel: 2, + steps: [{ status: "done" }, { status: "done" }, { status: "done" }], + }, + ]); + // Log: the old failure park is followed by a fresh clean completion (operator retry escape hatch). + (store.getTask as ReturnType).mockResolvedValue({ + id: "FN-8141", + lineageId: "lin-8141", + log: [ + { timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" }, + { timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" }, + { timestamp: "2026-07-16T10:00:04.000Z", action: "Resuming execution after unpause" }, + { timestamp: "2026-07-16T10:00:05.000Z", action: "Task marked done by agent" }, + ], + }); + + const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks(); + + expect(result).toBe(1); + expect(recoverFn).toHaveBeenCalledWith(expect.objectContaining({ id: "FN-8141" })); + + managerWithRecovery.stop(); + }); }); describe("recoverMissingWorktreeReviewFailures", () => { diff --git a/packages/engine/src/executor.ts b/packages/engine/src/executor.ts index c72a9ab6b7..89e8425d31 100644 --- a/packages/engine/src/executor.ts +++ b/packages/engine/src/executor.ts @@ -14,7 +14,7 @@ import { existsSync, lstatSync, realpathSync } from "node:fs"; import { readFile, rm, writeFile } from "node:fs/promises"; import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, AsyncMissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings, MergeResult, WorkflowIrNode, WorkflowIrNodeKind, WorkflowStepResult as CoreWorkflowStepResult, ThinkingLevel } from "@fusion/core"; import { getUnmetSchedulingDependencies } from "./scheduler.js"; -import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core"; +import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, evaluateCompletedPromotionFailureProvenance, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core"; import { finalizeProvenAutoMergeTask } from "./auto-merge-finalization.js"; import { mergeEffectiveSettings } from "./effective-settings.js"; import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "./replan-target.js"; @@ -4422,6 +4422,23 @@ export class TaskExecutor { return false; } + /* + FNXC:Lifecycle 2026-07-16-10:30: + FN-8141 defense-in-depth: recoverCompletedTask is the shared promotion chokepoint for BOTH + self-healing sweeps AND the executor's own unpause / resumeOrphaned fast-paths. A task whose + most recent execution-outcome in the durable log was a failure/refusal park must not be + promoted to in-review by ANY route, even one that re-derived completion from all-steps-done/ + skipped (skipped counts as complete, which is exactly how FN-8141 laundered a failed task). + The self-healing sweeps additionally emit the deduped no-action audit event; here we simply + refuse. Escape hatch: an operator retrying the task starts a fresh execution whose clean + completion marker supersedes the failure park, clearing this block with no code change. + */ + const failureProvenance = evaluateCompletedPromotionFailureProvenance(liveForCompletenessCheck ?? task); + if (failureProvenance.blocked) { + executorLog.log(`${task.id}: skipping recoverCompletedTask — most recent execution ended in a failure/refusal park (operator-decides)`); + return false; + } + const settings = await this.store.getSettings(); if (settings.globalPause || settings.enginePaused) { executorLog.log( diff --git a/packages/engine/src/run-audit.ts b/packages/engine/src/run-audit.ts index 9b58998343..b026b5f8a4 100644 --- a/packages/engine/src/run-audit.ts +++ b/packages/engine/src/run-audit.ts @@ -807,7 +807,17 @@ export type DatabaseMutationType = * withheld state persists unchanged. * Metadata: { taskId: string; reason: "user-paused" | "auto-merge-off-human-review"; stage?: string; oversightLevel?: string } */ - | "overseer:oversight-withheld-human-control"; + | "overseer:oversight-withheld-human-control" + /** + * FNXC:Lifecycle 2026-07-16-10:30: + * FN-8141 no-action event: a stranded-completed promoter (`recoverCompletedTasks` stuck-in-progress + * sweep OR `recoverStrandedCompletedTodoTasks` stranded-todo sweep in self-healing.ts) withheld + * promotion of an all-steps-done/skipped task because its most recent execution-outcome in the + * durable task log was a failure/refusal park (`evaluateCompletedPromotionFailureProvenance`). + * Emitted at most once per taskId while the blocking provenance persists (deduped in-memory). + * Metadata: { taskId, reason: "failure-provenance", sweep: "stuck-in-progress" | "stranded-todo", marker?: string } + */ + | "task:reconcile-stranded-completed-no-action"; // ── Filesystem mutation types ───────────────────────────────────────────────── diff --git a/packages/engine/src/self-healing.ts b/packages/engine/src/self-healing.ts index 05f237f756..e74f0a4d22 100644 --- a/packages/engine/src/self-healing.ts +++ b/packages/engine/src/self-healing.ts @@ -30,7 +30,7 @@ import { setImmediate as setImmediateCb } from "node:timers"; import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { isAbsolute, join, relative, resolve } from "node:path"; -import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core"; +import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, evaluateCompletedPromotionFailureProvenance, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core"; import type { MeshLeaseManager } from "./mesh-lease-manager.js"; import { createLogger, schedulerLog } from "./logger.js"; import { mergeEffectiveSettings } from "./effective-settings.js"; @@ -743,6 +743,14 @@ export class SelfHealingManager { private workspacePartialLandDrops: Map = new Map(); private orphanWorktreeRemovalFailures: Map = new Map(); private finalizeUnprovenWarned = new Set(); + /* + * FNXC:Lifecycle 2026-07-16-10:30: + * FN-8141 dedup: `task:reconcile-stranded-completed-no-action` is emitted at most once per taskId + * while a failure-provenance block persists, so the sweeps don't spam run-audit every housekeeping + * cycle. Cleared on stop(); a fresh clean execution that clears the block re-arms emission naturally + * because the task leaves the promotable state before returning to it. + */ + private strandedCompletedFailureProvenanceWarned = new Set(); private metaResolvedSkipAuditMemo = new Map(); private metaStalledSkipAuditMemo = new Map(); private preservedQueuedOverlapLogged = new Map(); @@ -1045,6 +1053,68 @@ export class SelfHealingManager { log.log(`[${stage}] ${task.id}: triple-proof not satisfied — no action (operator-decides)`); } + /* + * FNXC:Lifecycle 2026-07-16-10:30: + * FN-8141 — a stranded-completed promoter must respect failure provenance. Before promoting an + * all-steps-done/skipped candidate to in-review, verify its MOST RECENT execution-outcome in the + * durable task log was not a failure/refusal park; a park bounced to `todo`/`in-progress` by the + * pause-abort machinery must not be laundered into `in-review`. Slim listings strip `log`, so the + * full task is fetched only for candidates that already cleared the cheap step/status filters. + * On block, emits `task:reconcile-stranded-completed-no-action` once per taskId (deduped) and + * withholds promotion. Escape hatch: an operator retrying/moving the task starts a fresh execution + * whose clean completion marker supersedes the failure park, clearing the block with no code change. + * Fail-closed: if the full task cannot be read this cycle, withhold promotion rather than risk + * laundering a failed park. + */ + private async isStrandedCompletedPromotionBlockedByFailureProvenance( + taskId: string, + sweep: "stuck-in-progress" | "stranded-todo", + ): Promise { + let fullTask: Task; + try { + fullTask = await this.store.getTask(taskId); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + log.warn(`[stranded-completed-provenance] ${taskId}: full-task fetch failed (${message}) — withholding ${sweep} promotion this cycle`); + return true; + } + + const evaluation = evaluateCompletedPromotionFailureProvenance(fullTask); + if (!evaluation.blocked) { + // Provenance cleared (fresh clean execution or never-failed): re-arm the dedup so a later + // re-failure is reported again. + this.strandedCompletedFailureProvenanceWarned.delete(taskId); + return false; + } + + if (!this.strandedCompletedFailureProvenanceWarned.has(taskId)) { + this.strandedCompletedFailureProvenanceWarned.add(taskId); + try { + await createRunAuditor(this.store, { + runId: generateSyntheticRunId("self-healing-stranded-completed-provenance", taskId), + agentId: "self-healing", + taskId, + taskLineageId: fullTask.lineageId, + phase: "stranded-completed-provenance", + }).database({ + type: "task:reconcile-stranded-completed-no-action" as DatabaseMutationType, + target: taskId, + metadata: { + taskId, + reason: evaluation.reason ?? "failure-provenance", + sweep, + ...(evaluation.markerAction ? { marker: evaluation.markerAction } : {}), + }, + }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + log.warn(`[stranded-completed-provenance] ${taskId}: no-action audit emission failed: ${message}`); + } + } + log.log(`[stranded-completed-provenance] ${taskId}: withholding ${sweep} promotion — most recent execution ended in a failure/refusal park (operator-decides)`); + return true; + } + private async listActiveHeartbeatTaskIds(): Promise> { const activeTaskIds = new Set(); if (!this.options.agentStore) { @@ -1409,6 +1479,7 @@ export class SelfHealingManager { } this.finalizeUnprovenWarned.clear(); + this.strandedCompletedFailureProvenanceWarned.clear(); this.metaResolvedSkipAuditMemo.clear(); this.metaStalledSkipAuditMemo.clear(); this.preservedQueuedOverlapLogged.clear(); @@ -2810,6 +2881,11 @@ export class SelfHealingManager { log.log(`${task.id} started executing concurrently — skipping recovery this cycle`); continue; } + // FN-8141: never promote a stuck-in-progress task whose most recent execution ended in a + // failure/refusal park — honor the honest failure instead of laundering it into in-review. + if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stuck-in-progress")) { + continue; + } log.log(`Recovering completed task ${task.id}: ${task.title || task.description?.slice(0, 60) || "(untitled)"}`); const success = await recoverFn(task); if (success) recovered++; @@ -2866,6 +2942,11 @@ export class SelfHealingManager { log.log(`${task.id} started executing concurrently — skipping stranded todo recovery this cycle`); continue; } + // FN-8141: never promote a stranded-todo task whose most recent execution ended in a + // failure/refusal park — the pause-abort bounce to todo must not launder the failure. + if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stranded-todo")) { + continue; + } const success = await recoverFn(task); if (success) recovered++;