FN-6736: reclaim phantom executor bindings

Recover wedged in-progress tasks by clearing stale executor bindings only after liveness proves the owner is gone.

- Add a guarded executor escape hatch that clears only stale in-memory task bookkeeping while refusing live session surfaces.
- Teach self-healing to identify phantom executor-active bindings using age, checkout, heartbeat, run-audit, and worktree liveness signals before requeueing preserved work.
- Record reclaim events in run audit and cover preserved-worktree recovery with reliability interaction tests.
- Document the recovery path and add a patch changeset for the published CLI package.

Files changed:
 .changeset/fn-6736-phantom-executor-binding.md     |   5 +
 AGENTS.md                                          |   1 +
 docs/architecture.md                               |   1 +
 .../reclaim-phantom-executor-binding.test.ts       | 244 +++++++++++++++++++++
 packages/engine/src/executor.ts                    |  35 +++
 packages/engine/src/run-audit.ts                   |   2 +
 packages/engine/src/runtimes/in-process-runtime.ts |   3 +-
 packages/engine/src/self-healing.ts                | 112 ++++++++++
 8 files changed, 402 insertions(+), 1 deletion(-)

Fusion-Task-Id: FN-6736

Fusion-Task-Lineage: c76191ba-f4c3-4832-a790-67676e258ba2
This commit is contained in:
gsxdsm
2026-06-19 12:37:40 -07:00
parent 25ba732598
commit df139ec84c
8 changed files with 402 additions and 1 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
Recover in-progress tasks wedged behind stale in-memory executor bindings by clearing the phantom binding and requeueing with progress and worktree preserved.

View File

@@ -191,6 +191,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
- FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes. - FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes.
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move. - FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
- FN-6736: self-healing emits `task:reclaim-phantom-executor-binding` when it proves an in-memory executor-active binding is stale, clears the binding, and requeues the in-progress task with worktree/progress preserved.
## Reference docs (deeper detail) ## Reference docs (deeper detail)

View File

@@ -703,6 +703,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule. - FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops. - `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`. - `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
- FN-6736 adds a phantom executor-binding liveness gate to the same reclaim path. When the only remaining veto is an in-memory `executor-active`/live-worktree signal, the task is `in-progress`, the execution age is far beyond grace, `checkedOutBy` is empty, no active heartbeat/agent row exists, and run-audit activity is stale, self-healing force-clears the phantom executor binding and requeues the task to `todo` with worktree and progress preserved. Live evidence still wins (FN-4811), missing-worktree limbo remains owned by `recoverInProgressLimbo()` (FN-5219), and the path does not increment FN-5704 resume-limbo counters.
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch. - Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
- Merge commit attribution is ownership-aware: a `mergeDetails.commitSha` is trusted only when reachable from `HEAD` **and** attributable to the task via `Fusion-Task-Id` trailer or task-ID-bearing subject. Reachable-but-unowned SHAs are rejected to prevent sibling done tasks from sharing misleading merge metadata. - Merge commit attribution is ownership-aware: a `mergeDetails.commitSha` is trusted only when reachable from `HEAD` **and** attributable to the task via `Fusion-Task-Id` trailer or task-ID-bearing subject. Reachable-but-unowned SHAs are rejected to prevent sibling done tasks from sharing misleading merge metadata.
- FN-4948 adds a task-worktree pre-commit branch-identity guard: provisioning paths (`NativeWorktreeBackend.create`, executor branch creation, and `StepSessionExecutor.createStepWorktree`) install a `pre-commit` hook plus `fusion-task-id` metadata under the worktree's git-path. Commits are refused unless HEAD matches `fusion/<task-id>` or the allowlist (`fusion/step-<n>-<slug>` by default). - FN-4948 adds a task-worktree pre-commit branch-identity guard: provisioning paths (`NativeWorktreeBackend.create`, executor branch creation, and `StepSessionExecutor.createStepWorktree`) install a `pre-commit` hook plus `fusion-task-id` metadata under the worktree's git-path. Commits are refused unless HEAD matches `fusion/<task-id>` or the allowlist (`fusion/step-<n>-<slug>` by default).

View File

@@ -0,0 +1,244 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { EventEmitter } from "node:events";
import { existsSync, mkdirSync, mkdtempSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import type { Settings, Task, TaskStore } from "@fusion/core";
import { SelfHealingManager, STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS } from "../../self-healing.js";
import { activeSessionRegistry } from "../../active-session-registry.js";
import * as branchConflictModule from "../../branch-conflicts.js";
import * as worktreePoolModule from "../../worktree-pool.js";
type AuditRow = { timestamp: string };
type Harness = {
rootDir: string;
worktree: string;
task: Task;
store: TaskStore & EventEmitter;
clearPhantomExecutorBinding: ReturnType<typeof vi.fn>;
manager: SelfHealingManager;
cleanup: () => void;
};
const NOW = new Date("2026-06-19T12:00:00.000Z");
const OLD_EXECUTION_STARTED_AT = new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 9.5).toISOString();
function makeTask(overrides: Partial<Task> = {}): Task {
return {
id: "FN-6736",
title: "phantom binding",
description: "test",
column: "in-progress",
branch: "fusion/fn-6736",
worktree: "/tmp/fn-6736/.worktrees/crisp-lotus",
paused: false,
userPaused: false,
checkedOutBy: undefined,
dependencies: [],
steps: [{ id: "s1", title: "step", status: "in-progress" } as any],
currentStep: 5,
log: [],
createdAt: new Date(NOW.getTime() - 2 * 60 * 60_000).toISOString(),
updatedAt: new Date(NOW.getTime() - 90 * 60_000).toISOString(),
executionStartedAt: OLD_EXECUTION_STARTED_AT,
...overrides,
} as Task;
}
function makeStore(task: Task, options: { recentAuditRows?: AuditRow[] } = {}): TaskStore & EventEmitter {
const emitter = new EventEmitter();
const settings = {
autoMerge: true,
globalPause: false,
enginePaused: false,
baseBranch: "main",
mergeStrategy: "direct",
autoRecovery: { mode: "deterministic-only", maxRetries: 3 },
} as unknown as Settings;
return Object.assign(emitter, {
getSettings: vi.fn(async () => settings),
getTask: vi.fn(async () => task),
listTasks: vi.fn(async ({ column }: { column?: string } = {}) => (column === task.column ? [task] : [])),
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => Object.assign(task, updates)),
moveTask: vi.fn(async (_id: string, column: Task["column"], opts?: Record<string, unknown>) => {
task.column = column;
(task as any).__lastMoveOpts = opts;
return task;
}),
logEntry: vi.fn(async () => undefined),
appendAgentLog: vi.fn(async () => undefined),
updateSettings: vi.fn(async () => settings),
clearStaleExecutionStartBranchReferences: vi.fn(() => []),
recordRunAuditEvent: vi.fn(async () => undefined),
getRunAuditEvents: vi.fn(() => options.recentAuditRows ?? []),
walCheckpoint: vi.fn(() => ({ busy: 0, log: 0, checkpointed: 0 })),
archiveTaskAndCleanup: vi.fn(async () => ({})),
mergeTask: vi.fn(async () => undefined),
getRootDir: vi.fn(() => "/tmp/test"),
}) as unknown as TaskStore & EventEmitter;
}
function makeHarness(overrides: Partial<Task> = {}, options: {
recentAuditRows?: AuditRow[];
activeHeartbeat?: boolean;
missingWorktree?: boolean;
} = {}): Harness {
const rootDir = mkdtempSync(join(tmpdir(), "fn-6736-"));
const worktree = join(rootDir, ".worktrees", "crisp-lotus");
if (!options.missingWorktree) {
mkdirSync(worktree, { recursive: true });
}
const task = makeTask({ worktree, ...overrides });
const store = makeStore(task, { recentAuditRows: options.recentAuditRows });
const clearPhantomExecutorBinding = vi.fn();
const agentStore = options.activeHeartbeat
? { listActiveHeartbeatRuns: vi.fn(async () => [{ startedAt: new Date(NOW.getTime() - 60_000).toISOString(), contextSnapshot: { taskId: task.id } }]) }
: { listActiveHeartbeatRuns: vi.fn(async () => []) };
const manager = new SelfHealingManager(store as any, {
rootDir,
getExecutingTaskIds: () => new Set([task.id]),
clearPhantomExecutorBinding,
agentStore,
} as any);
return {
rootDir,
worktree,
task,
store,
clearPhantomExecutorBinding,
manager,
cleanup: () => {
manager.stop();
rmSync(rootDir, { recursive: true, force: true });
},
};
}
function findAudit(store: TaskStore & EventEmitter, mutationType: string): any | undefined {
return (store.recordRunAuditEvent as any).mock.calls.find((call: any[]) => call[0].mutationType === mutationType)?.[0];
}
describe("FN-6736: phantom executor binding reclaim", () => {
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(NOW);
vi.restoreAllMocks();
activeSessionRegistry.clear();
vi.spyOn(worktreePoolModule, "isUsableTaskWorktree").mockResolvedValue(true);
vi.spyOn(branchConflictModule, "inspectBranchConflict").mockResolvedValue({ kind: "stale" } as any);
});
afterEach(() => {
activeSessionRegistry.clear();
vi.useRealTimers();
});
it("requeues an old in-progress task when executor-active is only a phantom binding", async () => {
const h = makeHarness();
expect(existsSync(h.worktree)).toBe(true);
const recovered = await h.manager.reclaimSelfOwnedBranchConflicts();
expect(recovered).toBe(1);
expect(h.clearPhantomExecutorBinding).toHaveBeenCalledWith(h.task.id);
expect(h.store.moveTask).toHaveBeenCalledWith(h.task.id, "todo", expect.objectContaining({
moveSource: "engine",
recoveryRehome: true,
preserveProgress: true,
preserveWorktree: true,
}));
expect(h.task.column).toBe("todo");
expect(h.task.userPaused).toBe(false);
expect(h.task.paused).toBe(false);
expect((h.task as any).status).not.toBe("failed");
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")).toBeUndefined();
const event = findAudit(h.store, "task:reclaim-phantom-executor-binding");
expect(event).toBeTruthy();
expect(event.metadata).toEqual(expect.objectContaining({
taskId: h.task.id,
signalReason: "executor-active",
checkedOutBy: null,
agentPresent: false,
lastActivityMs: null,
worktree: h.worktree,
branch: h.task.branch,
worktreeExists: true,
}));
expect(event.metadata.executionAgeMs).toBeGreaterThan(STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 3);
h.cleanup();
});
it("keeps FN-4811 protection when recent run-audit activity proves a live owner", async () => {
const h = makeHarness({}, { recentAuditRows: [{ timestamp: new Date(NOW.getTime() - 60_000).toISOString() }] });
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
expect(h.store.moveTask).not.toHaveBeenCalled();
expect(findAudit(h.store, "task:reclaim-phantom-executor-binding")).toBeUndefined();
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
expect(h.task.column).toBe("in-progress");
h.cleanup();
});
it("keeps FN-4811 protection when checkedOutBy is set", async () => {
const h = makeHarness({ checkedOutBy: "agent-1" } as Partial<Task>);
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
expect(h.store.moveTask).not.toHaveBeenCalled();
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
h.cleanup();
});
it("keeps FN-4811 protection when an active heartbeat row exists", async () => {
const h = makeHarness({}, { activeHeartbeat: true });
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
expect(h.store.moveTask).not.toHaveBeenCalled();
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
h.cleanup();
});
it("protects tasks just past grace but below the phantom age multiplier", async () => {
const h = makeHarness({
executionStartedAt: new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS - 1_000).toISOString(),
});
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
expect(h.store.moveTask).not.toHaveBeenCalled();
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
h.cleanup();
});
it("does not double-handle missing worktrees owned by FN-5219 in-progress limbo recovery", async () => {
const h = makeHarness({}, { missingWorktree: true });
expect(existsSync(h.worktree)).toBe(false);
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
expect(h.store.moveTask).not.toHaveBeenCalled();
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
h.cleanup();
});
it("does not increment FN-5704 resume-limbo counters on the phantom-binding requeue", async () => {
const h = makeHarness({ resumeLimboCount: 1 } as Partial<Task>);
await h.manager.reclaimSelfOwnedBranchConflicts();
await h.manager.reclaimSelfOwnedBranchConflicts();
expect(h.store.moveTask).toHaveBeenCalledTimes(1);
expect(h.task.resumeLimboCount).toBe(1);
expect(findAudit(h.store, "task:resume-limbo-escalated")).toBeUndefined();
h.cleanup();
});
});

View File

@@ -1997,6 +1997,41 @@ export class TaskExecutor {
); );
} }
/**
* FNXC:ExecutorBinding 2026-06-19-00:00:
* FN-6736 gives self-healing a narrow escape hatch for phantom in-memory executor bindings after the liveness gate proves the owner is dead. Never use this as a general task stopper: it refuses to detach observable live session surfaces, then clears only stale bookkeeping (`executing`, resume/recovery sets, process-wide graph routing, activeWorktrees, activeSessionRegistry paths, and executingTaskLock) so the scheduler can re-dispatch the preserved worktree.
*/
clearPhantomExecutorBinding(taskId: string): boolean {
const hasLiveSessionSurface = this.activeSessions.has(taskId)
|| this.activeStepExecutors.has(taskId)
|| this.activeWorkflowStepSessions.has(taskId)
|| this.activeCliTaskSessions.has(taskId);
if (hasLiveSessionSurface) {
executorLog.warn(`${taskId}: refusing to clear phantom executor binding because a live session surface is still registered`);
return false;
}
const worktreePath = this.activeWorktrees.get(taskId);
this.activeWorktrees.delete(taskId);
this.executing.delete(taskId);
this.recoveringCompleted.delete(taskId);
this.resumingUnpaused.delete(taskId);
TaskExecutor.processWideGraphRouting.delete(taskId);
executingTaskLock.release(taskId);
this.effectiveColumnAgentByTask.delete(taskId);
const registeredPaths = new Set(activeSessionRegistry.pathsForTask(taskId));
if (worktreePath) {
registeredPaths.add(worktreePath);
}
for (const path of registeredPaths) {
activeSessionRegistry.unregisterPath(path);
}
executorLog.warn(`${taskId}: cleared phantom executor binding for self-healing re-dispatch`);
return true;
}
isEphemeralDeletionPending(agentId: string): boolean { isEphemeralDeletionPending(agentId: string): boolean {
return this.pendingEphemeralDeletions.has(agentId); return this.pendingEphemeralDeletions.has(agentId);
} }

View File

@@ -505,6 +505,8 @@ export type DatabaseMutationType =
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */ /** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
| "task:auto-recover-in-progress-limbo-no-action" | "task:auto-recover-in-progress-limbo-no-action"
| "task:resume-limbo-escalated" | "task:resume-limbo-escalated"
/** Metadata: { taskId, executionAgeMs, graceMs, staleBindingAgeFloorMs, checkedOutBy, agentPresent, lastActivityMs, hasRecentRunAudit, worktree, branch, worktreeExists, signalReason } */
| "task:reclaim-phantom-executor-binding"
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */ /** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
| "task:reclaim-self-owned-branch-conflict-no-action" | "task:reclaim-self-owned-branch-conflict-no-action"
| "task:orphan-detected-no-action" | "task:orphan-detected-no-action"

View File

@@ -786,7 +786,8 @@ export class InProcessRuntime
isWorktreeResumeReserved: this.cliAgentRuntime?.isWorktreeResumeReserved, isWorktreeResumeReserved: this.cliAgentRuntime?.isWorktreeResumeReserved,
recoverCompletedTask: (task) => this.executor.recoverCompletedTask(task), recoverCompletedTask: (task) => this.executor.recoverCompletedTask(task),
recoverFailedPreMergeStep: (task) => this.executor.recoverFailedPreMergeWorkflowStep(task), recoverFailedPreMergeStep: (task) => this.executor.recoverFailedPreMergeWorkflowStep(task),
getExecutingTaskIds: () => this.executor.getExecutingTaskIds(), getExecutingTaskIds: () => this.executor?.getExecutingTaskIds() ?? new Set<string>(),
clearPhantomExecutorBinding: (taskId: string) => this.executor?.clearPhantomExecutorBinding(taskId),
recoverApprovedTriageTask: (task) => this.triageProcessor?.recoverApprovedTask(task) ?? Promise.resolve(false), recoverApprovedTriageTask: (task) => this.triageProcessor?.recoverApprovedTask(task) ?? Promise.resolve(false),
getPlanningTaskIds: () => this.triageProcessor?.getProcessingTaskIds() ?? new Set<string>(), getPlanningTaskIds: () => this.triageProcessor?.getProcessingTaskIds() ?? new Set<string>(),
evictStaleTriageProcessing: () => this.triageProcessor?.evictStaleProcessing() ?? new Set<string>(), evictStaleTriageProcessing: () => this.triageProcessor?.evictStaleProcessing() ?? new Set<string>(),

View File

@@ -92,6 +92,7 @@ const ARCHIVE_FTS_MAINTENANCE_OPTIMIZE_CADENCE_TICKS = 24;
const ARCHIVE_FTS_REBUILD_THRESHOLD_BYTES = 64 * 1024 * 1024; const ARCHIVE_FTS_REBUILD_THRESHOLD_BYTES = 64 * 1024 * 1024;
const ARCHIVE_FTS_REBUILD_BYTES_PER_TASK = 512 * 1024; const ARCHIVE_FTS_REBUILD_BYTES_PER_TASK = 512 * 1024;
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000; export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
const PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER = 3;
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000; export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3; export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
export const MAX_POST_DONE_NONCONTINUABLE_WEDGE_RECOVERIES = 3; export const MAX_POST_DONE_NONCONTINUABLE_WEDGE_RECOVERIES = 3;
@@ -241,6 +242,8 @@ export interface SelfHealingOptions {
rootDir: string; rootDir: string;
/** Optional callback to release TaskExecutor in-memory worktree ownership for a task. */ /** Optional callback to release TaskExecutor in-memory worktree ownership for a task. */
releaseExecutorWorktreeOwnership?: (taskId: string) => void; releaseExecutorWorktreeOwnership?: (taskId: string) => void;
/** Optional callback to clear a demonstrably-stale executor binding without touching live sessions. */
clearPhantomExecutorBinding?: (taskId: string) => void;
/** Optional AgentStore for agent-level self-healing checks. */ /** Optional AgentStore for agent-level self-healing checks. */
agentStore?: AgentStore; agentStore?: AgentStore;
/** Canonical stale-lease recovery manager. */ /** Canonical stale-lease recovery manager. */
@@ -896,6 +899,70 @@ export class SelfHealingManager {
return activeTaskIds; return activeTaskIds;
} }
private async getRecentRunAuditActivityAgeMs(task: Task, nowMs: number): Promise<number | null> {
const getRunAuditEvents = (this.store as unknown as {
getRunAuditEvents?: (filter: { taskId?: string; startTime?: string; limit?: number }) => Array<{ timestamp?: string }>;
}).getRunAuditEvents;
if (typeof getRunAuditEvents !== "function") {
return null;
}
try {
const since = new Date(nowMs - RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS).toISOString();
const events = getRunAuditEvents.call(this.store, { taskId: task.id, startTime: since, limit: 1 });
const newest = events.find((event) => typeof event.timestamp === "string");
if (!newest?.timestamp) return null;
const timestampMs = Date.parse(newest.timestamp);
return Number.isFinite(timestampMs) ? Math.max(0, nowMs - timestampMs) : null;
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
log.warn(`[self-healing] unable to inspect recent run-audit activity for ${task.id}: ${message}`);
return null;
}
}
/**
* FNXC:SelfHealingReclaim 2026-06-19-00:00:
* FN-6736 requires self-healing to stop treating an in-memory `executor-active` binding as live when the owner is demonstrably dead. Preserve FN-4811 by requiring every live-owner signal to be absent, leave the FN-5219 missing-worktree path untouched, and avoid FN-5704 resume-limbo counters because this path only clears a stale binding and requeues once with progress/worktree preserved.
*/
private isPhantomExecutorBinding(task: Task, options: {
executionAgeMs: number | null;
graceMs: number;
activeHeartbeatTaskIds: Set<string>;
lastActivityMs: number | null;
}): { phantom: boolean; metadata: Record<string, unknown> } {
const normalizedId = task.id.toUpperCase();
const agentPresent = options.activeHeartbeatTaskIds.has(normalizedId);
const checkedOutBy = typeof task.checkedOutBy === "string" && task.checkedOutBy.trim().length > 0 ? task.checkedOutBy : null;
const worktreeExists = Boolean(task.worktree && existsSync(task.worktree));
const hasRecentRunAudit = options.lastActivityMs !== null && options.lastActivityMs <= RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS;
const safeAgeMs = options.graceMs * PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER;
const metadata = {
taskId: task.id,
executionAgeMs: options.executionAgeMs,
graceMs: options.graceMs,
staleBindingAgeFloorMs: safeAgeMs,
checkedOutBy,
agentPresent,
lastActivityMs: options.lastActivityMs,
hasRecentRunAudit,
worktree: task.worktree ?? null,
branch: task.branch ?? null,
worktreeExists,
};
return {
phantom: task.column === "in-progress"
&& worktreeExists
&& options.executionAgeMs !== null
&& options.executionAgeMs > safeAgeMs
&& !checkedOutBy
&& !agentPresent
&& !hasRecentRunAudit,
metadata,
};
}
private getFalsePositiveRequeueSignal(task: Task, options: { private getFalsePositiveRequeueSignal(task: Task, options: {
executingIds?: Set<string>; executingIds?: Set<string>;
activeHeartbeatTaskIds?: Set<string>; activeHeartbeatTaskIds?: Set<string>;
@@ -2659,6 +2726,51 @@ export class SelfHealingManager {
includeCheckedOutLease: true, includeCheckedOutLease: true,
}); });
if (liveExecutionSignal) { if (liveExecutionSignal) {
const canEvaluatePhantomBinding = task.column === "in-progress"
&& (liveExecutionSignal.reason === "executor-active" || liveExecutionSignal.reason === "live-worktree-and-branch");
if (canEvaluatePhantomBinding) {
const nowMs = Date.now();
const executionStartedAtMs = task.executionStartedAt ? Date.parse(task.executionStartedAt) : Number.NaN;
const executionAgeMs = Number.isFinite(executionStartedAtMs) ? Math.max(0, nowMs - executionStartedAtMs) : null;
const lastActivityMs = await this.getRecentRunAuditActivityAgeMs(task, nowMs);
const phantomBinding = this.isPhantomExecutorBinding(task, {
executionAgeMs,
graceMs: STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS,
activeHeartbeatTaskIds: activeTaskIds,
lastActivityMs,
});
/*
FNXC:SelfHealingReclaim 2026-06-19-00:00:
FN-6736 makes the executor-active veto conditional for in-progress tasks whose worktree still exists: if age is far beyond grace and checkout, heartbeat, and run-audit liveness are all absent, clear only the stale in-memory binding and requeue with worktree/progress intact instead of emitting the permanent no-action wedge. Live FN-4811 owners still reach the normal no-action veto, missing worktrees remain FN-5219, and resume-limbo escalation remains FN-5704-owned.
*/
if (phantomBinding.phantom) {
this.options.clearPhantomExecutorBinding?.(task.id);
await createRunAuditor(this.store, {
runId: generateSyntheticRunId("self-healing-phantom-executor-binding", task.id),
agentId: "self-healing",
taskId: task.id,
taskLineageId: task.lineageId,
phase: "reclaim-self-owned-branch-conflict",
}).database({
type: "task:reclaim-phantom-executor-binding",
target: task.id,
metadata: {
...phantomBinding.metadata,
signalReason: liveExecutionSignal.reason,
},
});
await this.store.moveTask(task.id, "todo", {
moveSource: "engine",
recoveryRehome: true,
preserveProgress: true,
preserveWorktree: true,
});
recovered++;
continue;
}
}
await this.emitFalsePositiveRequeueNoAction( await this.emitFalsePositiveRequeueNoAction(
task, task,
"reclaim-self-owned-branch-conflict", "reclaim-self-owned-branch-conflict",