fix(engine): stranded-completed promoters withhold tasks whose last execution ended in a failure park (#2257)
## What & why
FN-8141 laundered a failed task into `done`. The executor correctly
parked the task `failed` ("task parked failed during no-fn_task_done
retry" / "fn_task_done refusal retry budget exhausted"), the pause-abort
machinery bounced it to `todo`, and ~12 minutes later
`recoverStrandedCompletedTodoTasks` promoted it to `in-review` because
every step was done/skipped — overriding the honest failure park. From
there the AI merger found an empty diff and finalized it as a no-op
`done`, with no reviewer ever seeing it.
Existing exclusions (`task.error`,
`evaluateNoCommitsNoOpFinalize().blocked`, active statuses, refreshing
review state) all missed it because the failure provenance lived **only
in the durable task log** by the time the promoter ran — status/error
had been cleared by the pause-abort bounce.
This PR restores the invariant: **a stranded-completed promoter must not
promote a task whose most recent execution lifecycle ended in a
failure/refusal park.**
## Change
- New pure, unit-testable evaluator
`evaluateCompletedPromotionFailureProvenance(task)` in `@fusion/core`
(next to `no-commits-finalize-guard.ts`). It scans the task-log **tail**
(bounded to 250 entries) and lets the **most-recent execution-outcome
marker** decide: a failure/refusal park → `{ blocked: true, reason:
"failure-provenance" }`; a fresh clean completion (`Task marked done by
agent` / `All steps complete — implicit fn_task_done`) that appears more
recently supersedes an earlier park; zero failure markers → not blocked.
Recency is by construction, so a failure that predates a newer clean
execution is never reached.
- Both self-healing sweeps (`recoverCompletedTasks` stuck-in-progress
**and** `recoverStrandedCompletedTodoTasks` stranded-todo) fetch the
full task for candidates that already cleared the cheap slim filters
(slim listings strip `log`) and skip when blocked, emitting a
**deduped** `task:reconcile-stranded-completed-no-action` run-audit
event (ids/outcomes-only: `taskId`, `reason`, `sweep`, `marker?`).
- Defense-in-depth: the shared executor `recoverCompletedTask`
chokepoint — which the sweeps AND the executor's own
unpause/`resumeOrphaned` fast-paths all funnel through — also refuses a
provenance-blocked promotion, so no route can launder a failed park.
**Escape hatch (documented in FNXC comments):** an operator
retrying/moving the task starts a fresh execution whose clean-completion
marker supersedes the failure park, clearing the block with no code
change.
## Surface enumeration
- `recoverCompletedTasks` (stuck-in-progress sweep, self-healing.ts) —
guarded + audited.
- `recoverStrandedCompletedTodoTasks` (stranded-todo sweep,
self-healing.ts) — guarded + audited. FN-8141 shows both columns can
launder.
- `recoverCompletedTask` executor callback (the route both sweeps +
unpause + `resumeOrphaned` share) — verified it did **not** check
log-based provenance; added the guard there as the final chokepoint.
## Test evidence
Pure-evaluator unit tests (`@fusion/core`) — marker detection,
most-recent-outcome recency, supersede-by-clean-completion,
empty/missing log, tail-scan bound:
```
pnpm --filter @fusion/core exec vitest run src/__tests__/completed-promotion-failure-provenance.test.ts
Test Files 1 passed (1) Tests 9 passed (9)
```
Self-healing integration tests (`@fusion/engine`) — FN-8141-shaped todo
(3 done + 2 skipped + refusal-exhaust/park marker) is NOT promoted and
emits the no-action event exactly once (deduped across a second cycle);
same task after a fresh clean execution IS promoted; stuck-in-progress
variant covered:
```
pnpm --filter @fusion/engine exec vitest run src/__tests__/self-healing.test.ts -t "recoverCompletedTasks|recoverStrandedCompletedTodoTasks|FN-8141"
Test Files 1 passed (1) Tests 14 passed | 382 skipped (396)
```
`@fusion/core` builds clean. My engine changes add **zero** new type
errors (verified: all 13 engine build errors are the pre-existing pi-SDK
cluster in `auth-storage.ts`/`pi.ts`/`provider-registration.ts`, none in
`self-healing.ts`/`run-audit.ts`/`executor.ts`/the new file).
## Known environmental blocker
`pnpm verify:fast` cannot go green on this branch: the `@fusion/engine`
build is **already broken at baseline** (confirmed by stashing all my
changes) by the pi 0.80.x SDK migration errors
(`ModelRegistry`/`AuthStorage`/`ModelRuntime`) — the exact FN-8145
upstream breakage described in the FN-8141 incident. That is out of
scope for this task and independent of this diff. Likewise, the 22
pre-existing
`restart.integration.test.ts`/`executor-fast-mode-workflows.test.ts`
failures are identical with and without my changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus <noreply@anthropic.com>
This commit is contained in:
gsxdsm
@@ -2408,6 +2408,51 @@ describe("SelfHealingManager", () => {
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 — the same laundering can start from the in-progress column. The stuck-in-progress
|
||||
promoter must also withhold an all-steps-done/skipped task whose most recent execution ended in a
|
||||
failure/refusal park, and emit `task:reconcile-stranded-completed-no-action` (sweep stuck-in-progress).
|
||||
*/
|
||||
it("FN-8141: does NOT promote a stuck-in-progress task whose last execution ended in a failure park", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141-IP",
|
||||
column: "in-progress",
|
||||
paused: false,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "skipped" }],
|
||||
},
|
||||
]);
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141-IP",
|
||||
lineageId: "lin-8141-ip",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141-IP: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverCompletedTasks();
|
||||
|
||||
expect(result).toBe(0);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
const emitted = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls.some(
|
||||
([ev]) =>
|
||||
(ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" &&
|
||||
(ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stuck-in-progress",
|
||||
);
|
||||
expect(emitted).toBe(true);
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
it("skips tasks that are actively executing", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const getExecuting = vi.fn().mockReturnValue(new Set(["FN-001"]));
|
||||
@@ -3017,6 +3062,107 @@ describe("SelfHealingManager", () => {
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 — the stranded-todo promoter must not launder a failed task into in-review. A candidate
|
||||
with all steps done/skipped whose MOST RECENT durable-log execution-outcome is a failure/refusal
|
||||
park is withheld and emits `task:reconcile-stranded-completed-no-action` (reason failure-provenance)
|
||||
once; the same task after a fresh clean execution IS promoted; the escape hatch is operator retry.
|
||||
*/
|
||||
it("FN-8141: does NOT promote a stranded-todo task whose last execution ended in a failure park, and emits the no-action event once", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
// Slim board row: 3 done + 2 skipped, no error/active status (exactly the FN-8141 shape that
|
||||
// passed all existing exclusions).
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141",
|
||||
column: "todo",
|
||||
paused: false,
|
||||
error: null,
|
||||
reviewLevel: 2,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "done" }, { status: "skipped" }, { status: "skipped" }],
|
||||
},
|
||||
]);
|
||||
// Full task carries the durable failure-park provenance the slim row cannot.
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141",
|
||||
lineageId: "lin-8141",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:01.000Z", action: "All steps complete — implicit fn_task_done (agent did not call tool explicitly)" },
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
{ timestamp: "2026-07-16T10:00:04.000Z", action: "Execution paused — session preserved for resume, moved to todo" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
|
||||
expect(result).toBe(0);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
const auditCalls = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls;
|
||||
const noActionEvents = auditCalls.filter(
|
||||
([ev]) =>
|
||||
(ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action" &&
|
||||
(ev as { metadata?: { reason?: string; sweep?: string } }).metadata?.reason === "failure-provenance" &&
|
||||
(ev as { metadata?: { sweep?: string } }).metadata?.sweep === "stranded-todo",
|
||||
);
|
||||
expect(noActionEvents).toHaveLength(1);
|
||||
|
||||
// Deduped: a second sweep with the same unchanged provenance does not re-emit.
|
||||
await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
const noActionEventsAfter = (store.recordRunAuditEvent as ReturnType<typeof vi.fn>).mock.calls.filter(
|
||||
([ev]) => (ev as { mutationType?: string }).mutationType === "task:reconcile-stranded-completed-no-action",
|
||||
);
|
||||
expect(noActionEventsAfter).toHaveLength(1);
|
||||
expect(recoverFn).not.toHaveBeenCalled();
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
|
||||
it("FN-8141: DOES promote the same task once a fresh clean execution completes all steps after the failure park", async () => {
|
||||
const recoverFn = vi.fn().mockResolvedValue(true);
|
||||
const managerWithRecovery = new SelfHealingManager(store, {
|
||||
rootDir: "/tmp/test-project",
|
||||
recoverCompletedTask: recoverFn,
|
||||
getExecutingTaskIds: () => new Set<string>(),
|
||||
});
|
||||
|
||||
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
{
|
||||
id: "FN-8141",
|
||||
column: "todo",
|
||||
paused: false,
|
||||
error: null,
|
||||
reviewLevel: 2,
|
||||
steps: [{ status: "done" }, { status: "done" }, { status: "done" }],
|
||||
},
|
||||
]);
|
||||
// Log: the old failure park is followed by a fresh clean completion (operator retry escape hatch).
|
||||
(store.getTask as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
id: "FN-8141",
|
||||
lineageId: "lin-8141",
|
||||
log: [
|
||||
{ timestamp: "2026-07-16T10:00:02.000Z", action: "bulk-step-completion-without-review — fn_task_done refusal retry budget exhausted" },
|
||||
{ timestamp: "2026-07-16T10:00:03.000Z", action: "FN-8141: task parked failed during no-fn_task_done retry — honoring park, not retrying" },
|
||||
{ timestamp: "2026-07-16T10:00:04.000Z", action: "Resuming execution after unpause" },
|
||||
{ timestamp: "2026-07-16T10:00:05.000Z", action: "Task marked done by agent" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await managerWithRecovery.recoverStrandedCompletedTodoTasks();
|
||||
|
||||
expect(result).toBe(1);
|
||||
expect(recoverFn).toHaveBeenCalledWith(expect.objectContaining({ id: "FN-8141" }));
|
||||
|
||||
managerWithRecovery.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("recoverMissingWorktreeReviewFailures", () => {
|
||||
|
||||
@@ -14,7 +14,7 @@ import { existsSync, lstatSync, realpathSync } from "node:fs";
|
||||
import { readFile, rm, writeFile } from "node:fs/promises";
|
||||
import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, AsyncMissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings, MergeResult, WorkflowIrNode, WorkflowIrNodeKind, WorkflowStepResult as CoreWorkflowStepResult, ThinkingLevel } from "@fusion/core";
|
||||
import { getUnmetSchedulingDependencies } from "./scheduler.js";
|
||||
import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core";
|
||||
import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, evaluateCompletedPromotionFailureProvenance, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveExecutorEscalationTarget, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core";
|
||||
import { finalizeProvenAutoMergeTask } from "./auto-merge-finalization.js";
|
||||
import { mergeEffectiveSettings } from "./effective-settings.js";
|
||||
import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "./replan-target.js";
|
||||
@@ -4422,6 +4422,23 @@ export class TaskExecutor {
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:Lifecycle 2026-07-16-10:30:
|
||||
FN-8141 defense-in-depth: recoverCompletedTask is the shared promotion chokepoint for BOTH
|
||||
self-healing sweeps AND the executor's own unpause / resumeOrphaned fast-paths. A task whose
|
||||
most recent execution-outcome in the durable log was a failure/refusal park must not be
|
||||
promoted to in-review by ANY route, even one that re-derived completion from all-steps-done/
|
||||
skipped (skipped counts as complete, which is exactly how FN-8141 laundered a failed task).
|
||||
The self-healing sweeps additionally emit the deduped no-action audit event; here we simply
|
||||
refuse. Escape hatch: an operator retrying the task starts a fresh execution whose clean
|
||||
completion marker supersedes the failure park, clearing this block with no code change.
|
||||
*/
|
||||
const failureProvenance = evaluateCompletedPromotionFailureProvenance(liveForCompletenessCheck ?? task);
|
||||
if (failureProvenance.blocked) {
|
||||
executorLog.log(`${task.id}: skipping recoverCompletedTask — most recent execution ended in a failure/refusal park (operator-decides)`);
|
||||
return false;
|
||||
}
|
||||
|
||||
const settings = await this.store.getSettings();
|
||||
if (settings.globalPause || settings.enginePaused) {
|
||||
executorLog.log(
|
||||
|
||||
@@ -807,7 +807,17 @@ export type DatabaseMutationType =
|
||||
* withheld state persists unchanged.
|
||||
* Metadata: { taskId: string; reason: "user-paused" | "auto-merge-off-human-review"; stage?: string; oversightLevel?: string }
|
||||
*/
|
||||
| "overseer:oversight-withheld-human-control";
|
||||
| "overseer:oversight-withheld-human-control"
|
||||
/**
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 no-action event: a stranded-completed promoter (`recoverCompletedTasks` stuck-in-progress
|
||||
* sweep OR `recoverStrandedCompletedTodoTasks` stranded-todo sweep in self-healing.ts) withheld
|
||||
* promotion of an all-steps-done/skipped task because its most recent execution-outcome in the
|
||||
* durable task log was a failure/refusal park (`evaluateCompletedPromotionFailureProvenance`).
|
||||
* Emitted at most once per taskId while the blocking provenance persists (deduped in-memory).
|
||||
* Metadata: { taskId, reason: "failure-provenance", sweep: "stuck-in-progress" | "stranded-todo", marker?: string }
|
||||
*/
|
||||
| "task:reconcile-stranded-completed-no-action";
|
||||
|
||||
// ── Filesystem mutation types ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ import { setImmediate as setImmediateCb } from "node:timers";
|
||||
import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { isAbsolute, join, relative, resolve } from "node:path";
|
||||
import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core";
|
||||
import { IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX, IN_REVIEW_STALL_LOG_PREFIX, IN_REVIEW_STALL_TERMINAL_LOG_PREFIX, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, countRecentIdenticalStallEntries, detectDependencyCycle, detectSelfDefeatingDependency, evaluateNoCommitsNoOpFinalize, evaluateCompletedPromotionFailureProvenance, getInReviewStalledSignal, getInReviewStallReason, getPrimaryPrInfo, getStalePausedReviewSignal, getStalePausedTodoSignal, getTaskHardMergeBlocker, getTaskMergeBlocker, isEphemeralAgent, isMergeRequestContractShadowEnabled, isWorkflowColumnsEnabled, isWorkspaceTask, isSharedBranchGroupMemberIntegration, parseExplicitDuplicateMarker, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, resolveWorkflowIrForTask, AWAITING_APPROVAL_PAUSE_REASON, type Agent, type AgentStore, type ChatStore, type MessageStore, type TaskStore, type Settings, type Task, type MergeDetails, type TaskPriority, type MergeResult, type WorkflowStepResult } from "@fusion/core";
|
||||
import type { MeshLeaseManager } from "./mesh-lease-manager.js";
|
||||
import { createLogger, schedulerLog } from "./logger.js";
|
||||
import { mergeEffectiveSettings } from "./effective-settings.js";
|
||||
@@ -743,6 +743,14 @@ export class SelfHealingManager {
|
||||
private workspacePartialLandDrops: Map<string, number> = new Map();
|
||||
private orphanWorktreeRemovalFailures: Map<string, number> = new Map();
|
||||
private finalizeUnprovenWarned = new Set<string>();
|
||||
/*
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 dedup: `task:reconcile-stranded-completed-no-action` is emitted at most once per taskId
|
||||
* while a failure-provenance block persists, so the sweeps don't spam run-audit every housekeeping
|
||||
* cycle. Cleared on stop(); a fresh clean execution that clears the block re-arms emission naturally
|
||||
* because the task leaves the promotable state before returning to it.
|
||||
*/
|
||||
private strandedCompletedFailureProvenanceWarned = new Set<string>();
|
||||
private metaResolvedSkipAuditMemo = new Map<string, string>();
|
||||
private metaStalledSkipAuditMemo = new Map<string, string>();
|
||||
private preservedQueuedOverlapLogged = new Map<string, string>();
|
||||
@@ -1045,6 +1053,68 @@ export class SelfHealingManager {
|
||||
log.log(`[${stage}] ${task.id}: triple-proof not satisfied — no action (operator-decides)`);
|
||||
}
|
||||
|
||||
/*
|
||||
* FNXC:Lifecycle 2026-07-16-10:30:
|
||||
* FN-8141 — a stranded-completed promoter must respect failure provenance. Before promoting an
|
||||
* all-steps-done/skipped candidate to in-review, verify its MOST RECENT execution-outcome in the
|
||||
* durable task log was not a failure/refusal park; a park bounced to `todo`/`in-progress` by the
|
||||
* pause-abort machinery must not be laundered into `in-review`. Slim listings strip `log`, so the
|
||||
* full task is fetched only for candidates that already cleared the cheap step/status filters.
|
||||
* On block, emits `task:reconcile-stranded-completed-no-action` once per taskId (deduped) and
|
||||
* withholds promotion. Escape hatch: an operator retrying/moving the task starts a fresh execution
|
||||
* whose clean completion marker supersedes the failure park, clearing the block with no code change.
|
||||
* Fail-closed: if the full task cannot be read this cycle, withhold promotion rather than risk
|
||||
* laundering a failed park.
|
||||
*/
|
||||
private async isStrandedCompletedPromotionBlockedByFailureProvenance(
|
||||
taskId: string,
|
||||
sweep: "stuck-in-progress" | "stranded-todo",
|
||||
): Promise<boolean> {
|
||||
let fullTask: Task;
|
||||
try {
|
||||
fullTask = await this.store.getTask(taskId);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log.warn(`[stranded-completed-provenance] ${taskId}: full-task fetch failed (${message}) — withholding ${sweep} promotion this cycle`);
|
||||
return true;
|
||||
}
|
||||
|
||||
const evaluation = evaluateCompletedPromotionFailureProvenance(fullTask);
|
||||
if (!evaluation.blocked) {
|
||||
// Provenance cleared (fresh clean execution or never-failed): re-arm the dedup so a later
|
||||
// re-failure is reported again.
|
||||
this.strandedCompletedFailureProvenanceWarned.delete(taskId);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!this.strandedCompletedFailureProvenanceWarned.has(taskId)) {
|
||||
this.strandedCompletedFailureProvenanceWarned.add(taskId);
|
||||
try {
|
||||
await createRunAuditor(this.store, {
|
||||
runId: generateSyntheticRunId("self-healing-stranded-completed-provenance", taskId),
|
||||
agentId: "self-healing",
|
||||
taskId,
|
||||
taskLineageId: fullTask.lineageId,
|
||||
phase: "stranded-completed-provenance",
|
||||
}).database({
|
||||
type: "task:reconcile-stranded-completed-no-action" as DatabaseMutationType,
|
||||
target: taskId,
|
||||
metadata: {
|
||||
taskId,
|
||||
reason: evaluation.reason ?? "failure-provenance",
|
||||
sweep,
|
||||
...(evaluation.markerAction ? { marker: evaluation.markerAction } : {}),
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log.warn(`[stranded-completed-provenance] ${taskId}: no-action audit emission failed: ${message}`);
|
||||
}
|
||||
}
|
||||
log.log(`[stranded-completed-provenance] ${taskId}: withholding ${sweep} promotion — most recent execution ended in a failure/refusal park (operator-decides)`);
|
||||
return true;
|
||||
}
|
||||
|
||||
private async listActiveHeartbeatTaskIds(): Promise<Set<string>> {
|
||||
const activeTaskIds = new Set<string>();
|
||||
if (!this.options.agentStore) {
|
||||
@@ -1409,6 +1479,7 @@ export class SelfHealingManager {
|
||||
}
|
||||
|
||||
this.finalizeUnprovenWarned.clear();
|
||||
this.strandedCompletedFailureProvenanceWarned.clear();
|
||||
this.metaResolvedSkipAuditMemo.clear();
|
||||
this.metaStalledSkipAuditMemo.clear();
|
||||
this.preservedQueuedOverlapLogged.clear();
|
||||
@@ -2810,6 +2881,11 @@ export class SelfHealingManager {
|
||||
log.log(`${task.id} started executing concurrently — skipping recovery this cycle`);
|
||||
continue;
|
||||
}
|
||||
// FN-8141: never promote a stuck-in-progress task whose most recent execution ended in a
|
||||
// failure/refusal park — honor the honest failure instead of laundering it into in-review.
|
||||
if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stuck-in-progress")) {
|
||||
continue;
|
||||
}
|
||||
log.log(`Recovering completed task ${task.id}: ${task.title || task.description?.slice(0, 60) || "(untitled)"}`);
|
||||
const success = await recoverFn(task);
|
||||
if (success) recovered++;
|
||||
@@ -2866,6 +2942,11 @@ export class SelfHealingManager {
|
||||
log.log(`${task.id} started executing concurrently — skipping stranded todo recovery this cycle`);
|
||||
continue;
|
||||
}
|
||||
// FN-8141: never promote a stranded-todo task whose most recent execution ended in a
|
||||
// failure/refusal park — the pause-abort bounce to todo must not launder the failure.
|
||||
if (await this.isStrandedCompletedPromotionBlockedByFailureProvenance(task.id, "stranded-todo")) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const success = await recoverFn(task);
|
||||
if (success) recovered++;
|
||||
|
||||
Reference in New Issue
Block a user