FN-6736: reclaim phantom executor bindings
Recover wedged in-progress tasks by clearing stale executor bindings only after liveness proves the owner is gone. - Add a guarded executor escape hatch that clears only stale in-memory task bookkeeping while refusing live session surfaces. - Teach self-healing to identify phantom executor-active bindings using age, checkout, heartbeat, run-audit, and worktree liveness signals before requeueing preserved work. - Record reclaim events in run audit and cover preserved-worktree recovery with reliability interaction tests. - Document the recovery path and add a patch changeset for the published CLI package. Files changed: .changeset/fn-6736-phantom-executor-binding.md | 5 + AGENTS.md | 1 + docs/architecture.md | 1 + .../reclaim-phantom-executor-binding.test.ts | 244 +++++++++++++++++++++ packages/engine/src/executor.ts | 35 +++ packages/engine/src/run-audit.ts | 2 + packages/engine/src/runtimes/in-process-runtime.ts | 3 +- packages/engine/src/self-healing.ts | 112 ++++++++++ 8 files changed, 402 insertions(+), 1 deletion(-) Fusion-Task-Id: FN-6736 Fusion-Task-Lineage: c76191ba-f4c3-4832-a790-67676e258ba2
This commit is contained in:
5
.changeset/fn-6736-phantom-executor-binding.md
Normal file
5
.changeset/fn-6736-phantom-executor-binding.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
Recover in-progress tasks wedged behind stale in-memory executor bindings by clearing the phantom binding and requeueing with progress and worktree preserved.
|
||||||
@@ -191,6 +191,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
|||||||
|
|
||||||
- FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes.
|
- FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes.
|
||||||
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
|
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
|
||||||
|
- FN-6736: self-healing emits `task:reclaim-phantom-executor-binding` when it proves an in-memory executor-active binding is stale, clears the binding, and requeues the in-progress task with worktree/progress preserved.
|
||||||
|
|
||||||
|
|
||||||
## Reference docs (deeper detail)
|
## Reference docs (deeper detail)
|
||||||
|
|||||||
@@ -703,6 +703,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
|||||||
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
|
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
|
||||||
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
||||||
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
||||||
|
- FN-6736 adds a phantom executor-binding liveness gate to the same reclaim path. When the only remaining veto is an in-memory `executor-active`/live-worktree signal, the task is `in-progress`, the execution age is far beyond grace, `checkedOutBy` is empty, no active heartbeat/agent row exists, and run-audit activity is stale, self-healing force-clears the phantom executor binding and requeues the task to `todo` with worktree and progress preserved. Live evidence still wins (FN-4811), missing-worktree limbo remains owned by `recoverInProgressLimbo()` (FN-5219), and the path does not increment FN-5704 resume-limbo counters.
|
||||||
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
||||||
- Merge commit attribution is ownership-aware: a `mergeDetails.commitSha` is trusted only when reachable from `HEAD` **and** attributable to the task via `Fusion-Task-Id` trailer or task-ID-bearing subject. Reachable-but-unowned SHAs are rejected to prevent sibling done tasks from sharing misleading merge metadata.
|
- Merge commit attribution is ownership-aware: a `mergeDetails.commitSha` is trusted only when reachable from `HEAD` **and** attributable to the task via `Fusion-Task-Id` trailer or task-ID-bearing subject. Reachable-but-unowned SHAs are rejected to prevent sibling done tasks from sharing misleading merge metadata.
|
||||||
- FN-4948 adds a task-worktree pre-commit branch-identity guard: provisioning paths (`NativeWorktreeBackend.create`, executor branch creation, and `StepSessionExecutor.createStepWorktree`) install a `pre-commit` hook plus `fusion-task-id` metadata under the worktree's git-path. Commits are refused unless HEAD matches `fusion/<task-id>` or the allowlist (`fusion/step-<n>-<slug>` by default).
|
- FN-4948 adds a task-worktree pre-commit branch-identity guard: provisioning paths (`NativeWorktreeBackend.create`, executor branch creation, and `StepSessionExecutor.createStepWorktree`) install a `pre-commit` hook plus `fusion-task-id` metadata under the worktree's git-path. Commits are refused unless HEAD matches `fusion/<task-id>` or the allowlist (`fusion/step-<n>-<slug>` by default).
|
||||||
|
|||||||
@@ -0,0 +1,244 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import { EventEmitter } from "node:events";
|
||||||
|
import { existsSync, mkdirSync, mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import type { Settings, Task, TaskStore } from "@fusion/core";
|
||||||
|
import { SelfHealingManager, STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS } from "../../self-healing.js";
|
||||||
|
import { activeSessionRegistry } from "../../active-session-registry.js";
|
||||||
|
import * as branchConflictModule from "../../branch-conflicts.js";
|
||||||
|
import * as worktreePoolModule from "../../worktree-pool.js";
|
||||||
|
|
||||||
|
type AuditRow = { timestamp: string };
|
||||||
|
|
||||||
|
type Harness = {
|
||||||
|
rootDir: string;
|
||||||
|
worktree: string;
|
||||||
|
task: Task;
|
||||||
|
store: TaskStore & EventEmitter;
|
||||||
|
clearPhantomExecutorBinding: ReturnType<typeof vi.fn>;
|
||||||
|
manager: SelfHealingManager;
|
||||||
|
cleanup: () => void;
|
||||||
|
};
|
||||||
|
|
||||||
|
const NOW = new Date("2026-06-19T12:00:00.000Z");
|
||||||
|
const OLD_EXECUTION_STARTED_AT = new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 9.5).toISOString();
|
||||||
|
|
||||||
|
function makeTask(overrides: Partial<Task> = {}): Task {
|
||||||
|
return {
|
||||||
|
id: "FN-6736",
|
||||||
|
title: "phantom binding",
|
||||||
|
description: "test",
|
||||||
|
column: "in-progress",
|
||||||
|
branch: "fusion/fn-6736",
|
||||||
|
worktree: "/tmp/fn-6736/.worktrees/crisp-lotus",
|
||||||
|
paused: false,
|
||||||
|
userPaused: false,
|
||||||
|
checkedOutBy: undefined,
|
||||||
|
dependencies: [],
|
||||||
|
steps: [{ id: "s1", title: "step", status: "in-progress" } as any],
|
||||||
|
currentStep: 5,
|
||||||
|
log: [],
|
||||||
|
createdAt: new Date(NOW.getTime() - 2 * 60 * 60_000).toISOString(),
|
||||||
|
updatedAt: new Date(NOW.getTime() - 90 * 60_000).toISOString(),
|
||||||
|
executionStartedAt: OLD_EXECUTION_STARTED_AT,
|
||||||
|
...overrides,
|
||||||
|
} as Task;
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeStore(task: Task, options: { recentAuditRows?: AuditRow[] } = {}): TaskStore & EventEmitter {
|
||||||
|
const emitter = new EventEmitter();
|
||||||
|
const settings = {
|
||||||
|
autoMerge: true,
|
||||||
|
globalPause: false,
|
||||||
|
enginePaused: false,
|
||||||
|
baseBranch: "main",
|
||||||
|
mergeStrategy: "direct",
|
||||||
|
autoRecovery: { mode: "deterministic-only", maxRetries: 3 },
|
||||||
|
} as unknown as Settings;
|
||||||
|
|
||||||
|
return Object.assign(emitter, {
|
||||||
|
getSettings: vi.fn(async () => settings),
|
||||||
|
getTask: vi.fn(async () => task),
|
||||||
|
listTasks: vi.fn(async ({ column }: { column?: string } = {}) => (column === task.column ? [task] : [])),
|
||||||
|
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => Object.assign(task, updates)),
|
||||||
|
moveTask: vi.fn(async (_id: string, column: Task["column"], opts?: Record<string, unknown>) => {
|
||||||
|
task.column = column;
|
||||||
|
(task as any).__lastMoveOpts = opts;
|
||||||
|
return task;
|
||||||
|
}),
|
||||||
|
logEntry: vi.fn(async () => undefined),
|
||||||
|
appendAgentLog: vi.fn(async () => undefined),
|
||||||
|
updateSettings: vi.fn(async () => settings),
|
||||||
|
clearStaleExecutionStartBranchReferences: vi.fn(() => []),
|
||||||
|
recordRunAuditEvent: vi.fn(async () => undefined),
|
||||||
|
getRunAuditEvents: vi.fn(() => options.recentAuditRows ?? []),
|
||||||
|
walCheckpoint: vi.fn(() => ({ busy: 0, log: 0, checkpointed: 0 })),
|
||||||
|
archiveTaskAndCleanup: vi.fn(async () => ({})),
|
||||||
|
mergeTask: vi.fn(async () => undefined),
|
||||||
|
getRootDir: vi.fn(() => "/tmp/test"),
|
||||||
|
}) as unknown as TaskStore & EventEmitter;
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeHarness(overrides: Partial<Task> = {}, options: {
|
||||||
|
recentAuditRows?: AuditRow[];
|
||||||
|
activeHeartbeat?: boolean;
|
||||||
|
missingWorktree?: boolean;
|
||||||
|
} = {}): Harness {
|
||||||
|
const rootDir = mkdtempSync(join(tmpdir(), "fn-6736-"));
|
||||||
|
const worktree = join(rootDir, ".worktrees", "crisp-lotus");
|
||||||
|
if (!options.missingWorktree) {
|
||||||
|
mkdirSync(worktree, { recursive: true });
|
||||||
|
}
|
||||||
|
const task = makeTask({ worktree, ...overrides });
|
||||||
|
const store = makeStore(task, { recentAuditRows: options.recentAuditRows });
|
||||||
|
const clearPhantomExecutorBinding = vi.fn();
|
||||||
|
const agentStore = options.activeHeartbeat
|
||||||
|
? { listActiveHeartbeatRuns: vi.fn(async () => [{ startedAt: new Date(NOW.getTime() - 60_000).toISOString(), contextSnapshot: { taskId: task.id } }]) }
|
||||||
|
: { listActiveHeartbeatRuns: vi.fn(async () => []) };
|
||||||
|
const manager = new SelfHealingManager(store as any, {
|
||||||
|
rootDir,
|
||||||
|
getExecutingTaskIds: () => new Set([task.id]),
|
||||||
|
clearPhantomExecutorBinding,
|
||||||
|
agentStore,
|
||||||
|
} as any);
|
||||||
|
return {
|
||||||
|
rootDir,
|
||||||
|
worktree,
|
||||||
|
task,
|
||||||
|
store,
|
||||||
|
clearPhantomExecutorBinding,
|
||||||
|
manager,
|
||||||
|
cleanup: () => {
|
||||||
|
manager.stop();
|
||||||
|
rmSync(rootDir, { recursive: true, force: true });
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function findAudit(store: TaskStore & EventEmitter, mutationType: string): any | undefined {
|
||||||
|
return (store.recordRunAuditEvent as any).mock.calls.find((call: any[]) => call[0].mutationType === mutationType)?.[0];
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("FN-6736: phantom executor binding reclaim", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.useFakeTimers();
|
||||||
|
vi.setSystemTime(NOW);
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
activeSessionRegistry.clear();
|
||||||
|
vi.spyOn(worktreePoolModule, "isUsableTaskWorktree").mockResolvedValue(true);
|
||||||
|
vi.spyOn(branchConflictModule, "inspectBranchConflict").mockResolvedValue({ kind: "stale" } as any);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
activeSessionRegistry.clear();
|
||||||
|
vi.useRealTimers();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("requeues an old in-progress task when executor-active is only a phantom binding", async () => {
|
||||||
|
const h = makeHarness();
|
||||||
|
expect(existsSync(h.worktree)).toBe(true);
|
||||||
|
|
||||||
|
const recovered = await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(recovered).toBe(1);
|
||||||
|
expect(h.clearPhantomExecutorBinding).toHaveBeenCalledWith(h.task.id);
|
||||||
|
expect(h.store.moveTask).toHaveBeenCalledWith(h.task.id, "todo", expect.objectContaining({
|
||||||
|
moveSource: "engine",
|
||||||
|
recoveryRehome: true,
|
||||||
|
preserveProgress: true,
|
||||||
|
preserveWorktree: true,
|
||||||
|
}));
|
||||||
|
expect(h.task.column).toBe("todo");
|
||||||
|
expect(h.task.userPaused).toBe(false);
|
||||||
|
expect(h.task.paused).toBe(false);
|
||||||
|
expect((h.task as any).status).not.toBe("failed");
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")).toBeUndefined();
|
||||||
|
const event = findAudit(h.store, "task:reclaim-phantom-executor-binding");
|
||||||
|
expect(event).toBeTruthy();
|
||||||
|
expect(event.metadata).toEqual(expect.objectContaining({
|
||||||
|
taskId: h.task.id,
|
||||||
|
signalReason: "executor-active",
|
||||||
|
checkedOutBy: null,
|
||||||
|
agentPresent: false,
|
||||||
|
lastActivityMs: null,
|
||||||
|
worktree: h.worktree,
|
||||||
|
branch: h.task.branch,
|
||||||
|
worktreeExists: true,
|
||||||
|
}));
|
||||||
|
expect(event.metadata.executionAgeMs).toBeGreaterThan(STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 3);
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps FN-4811 protection when recent run-audit activity proves a live owner", async () => {
|
||||||
|
const h = makeHarness({}, { recentAuditRows: [{ timestamp: new Date(NOW.getTime() - 60_000).toISOString() }] });
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||||
|
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-phantom-executor-binding")).toBeUndefined();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||||
|
expect(h.task.column).toBe("in-progress");
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps FN-4811 protection when checkedOutBy is set", async () => {
|
||||||
|
const h = makeHarness({ checkedOutBy: "agent-1" } as Partial<Task>);
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||||
|
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps FN-4811 protection when an active heartbeat row exists", async () => {
|
||||||
|
const h = makeHarness({}, { activeHeartbeat: true });
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||||
|
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("protects tasks just past grace but below the phantom age multiplier", async () => {
|
||||||
|
const h = makeHarness({
|
||||||
|
executionStartedAt: new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS - 1_000).toISOString(),
|
||||||
|
});
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||||
|
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not double-handle missing worktrees owned by FN-5219 in-progress limbo recovery", async () => {
|
||||||
|
const h = makeHarness({}, { missingWorktree: true });
|
||||||
|
expect(existsSync(h.worktree)).toBe(false);
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||||
|
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||||
|
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not increment FN-5704 resume-limbo counters on the phantom-binding requeue", async () => {
|
||||||
|
const h = makeHarness({ resumeLimboCount: 1 } as Partial<Task>);
|
||||||
|
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||||
|
|
||||||
|
expect(h.store.moveTask).toHaveBeenCalledTimes(1);
|
||||||
|
expect(h.task.resumeLimboCount).toBe(1);
|
||||||
|
expect(findAudit(h.store, "task:resume-limbo-escalated")).toBeUndefined();
|
||||||
|
h.cleanup();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1997,6 +1997,41 @@ export class TaskExecutor {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* FNXC:ExecutorBinding 2026-06-19-00:00:
|
||||||
|
* FN-6736 gives self-healing a narrow escape hatch for phantom in-memory executor bindings after the liveness gate proves the owner is dead. Never use this as a general task stopper: it refuses to detach observable live session surfaces, then clears only stale bookkeeping (`executing`, resume/recovery sets, process-wide graph routing, activeWorktrees, activeSessionRegistry paths, and executingTaskLock) so the scheduler can re-dispatch the preserved worktree.
|
||||||
|
*/
|
||||||
|
clearPhantomExecutorBinding(taskId: string): boolean {
|
||||||
|
const hasLiveSessionSurface = this.activeSessions.has(taskId)
|
||||||
|
|| this.activeStepExecutors.has(taskId)
|
||||||
|
|| this.activeWorkflowStepSessions.has(taskId)
|
||||||
|
|| this.activeCliTaskSessions.has(taskId);
|
||||||
|
if (hasLiveSessionSurface) {
|
||||||
|
executorLog.warn(`${taskId}: refusing to clear phantom executor binding because a live session surface is still registered`);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const worktreePath = this.activeWorktrees.get(taskId);
|
||||||
|
this.activeWorktrees.delete(taskId);
|
||||||
|
this.executing.delete(taskId);
|
||||||
|
this.recoveringCompleted.delete(taskId);
|
||||||
|
this.resumingUnpaused.delete(taskId);
|
||||||
|
TaskExecutor.processWideGraphRouting.delete(taskId);
|
||||||
|
executingTaskLock.release(taskId);
|
||||||
|
this.effectiveColumnAgentByTask.delete(taskId);
|
||||||
|
|
||||||
|
const registeredPaths = new Set(activeSessionRegistry.pathsForTask(taskId));
|
||||||
|
if (worktreePath) {
|
||||||
|
registeredPaths.add(worktreePath);
|
||||||
|
}
|
||||||
|
for (const path of registeredPaths) {
|
||||||
|
activeSessionRegistry.unregisterPath(path);
|
||||||
|
}
|
||||||
|
|
||||||
|
executorLog.warn(`${taskId}: cleared phantom executor binding for self-healing re-dispatch`);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
isEphemeralDeletionPending(agentId: string): boolean {
|
isEphemeralDeletionPending(agentId: string): boolean {
|
||||||
return this.pendingEphemeralDeletions.has(agentId);
|
return this.pendingEphemeralDeletions.has(agentId);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -505,6 +505,8 @@ export type DatabaseMutationType =
|
|||||||
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
||||||
| "task:auto-recover-in-progress-limbo-no-action"
|
| "task:auto-recover-in-progress-limbo-no-action"
|
||||||
| "task:resume-limbo-escalated"
|
| "task:resume-limbo-escalated"
|
||||||
|
/** Metadata: { taskId, executionAgeMs, graceMs, staleBindingAgeFloorMs, checkedOutBy, agentPresent, lastActivityMs, hasRecentRunAudit, worktree, branch, worktreeExists, signalReason } */
|
||||||
|
| "task:reclaim-phantom-executor-binding"
|
||||||
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
||||||
| "task:reclaim-self-owned-branch-conflict-no-action"
|
| "task:reclaim-self-owned-branch-conflict-no-action"
|
||||||
| "task:orphan-detected-no-action"
|
| "task:orphan-detected-no-action"
|
||||||
|
|||||||
@@ -786,7 +786,8 @@ export class InProcessRuntime
|
|||||||
isWorktreeResumeReserved: this.cliAgentRuntime?.isWorktreeResumeReserved,
|
isWorktreeResumeReserved: this.cliAgentRuntime?.isWorktreeResumeReserved,
|
||||||
recoverCompletedTask: (task) => this.executor.recoverCompletedTask(task),
|
recoverCompletedTask: (task) => this.executor.recoverCompletedTask(task),
|
||||||
recoverFailedPreMergeStep: (task) => this.executor.recoverFailedPreMergeWorkflowStep(task),
|
recoverFailedPreMergeStep: (task) => this.executor.recoverFailedPreMergeWorkflowStep(task),
|
||||||
getExecutingTaskIds: () => this.executor.getExecutingTaskIds(),
|
getExecutingTaskIds: () => this.executor?.getExecutingTaskIds() ?? new Set<string>(),
|
||||||
|
clearPhantomExecutorBinding: (taskId: string) => this.executor?.clearPhantomExecutorBinding(taskId),
|
||||||
recoverApprovedTriageTask: (task) => this.triageProcessor?.recoverApprovedTask(task) ?? Promise.resolve(false),
|
recoverApprovedTriageTask: (task) => this.triageProcessor?.recoverApprovedTask(task) ?? Promise.resolve(false),
|
||||||
getPlanningTaskIds: () => this.triageProcessor?.getProcessingTaskIds() ?? new Set<string>(),
|
getPlanningTaskIds: () => this.triageProcessor?.getProcessingTaskIds() ?? new Set<string>(),
|
||||||
evictStaleTriageProcessing: () => this.triageProcessor?.evictStaleProcessing() ?? new Set<string>(),
|
evictStaleTriageProcessing: () => this.triageProcessor?.evictStaleProcessing() ?? new Set<string>(),
|
||||||
|
|||||||
@@ -92,6 +92,7 @@ const ARCHIVE_FTS_MAINTENANCE_OPTIMIZE_CADENCE_TICKS = 24;
|
|||||||
const ARCHIVE_FTS_REBUILD_THRESHOLD_BYTES = 64 * 1024 * 1024;
|
const ARCHIVE_FTS_REBUILD_THRESHOLD_BYTES = 64 * 1024 * 1024;
|
||||||
const ARCHIVE_FTS_REBUILD_BYTES_PER_TASK = 512 * 1024;
|
const ARCHIVE_FTS_REBUILD_BYTES_PER_TASK = 512 * 1024;
|
||||||
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
|
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
|
||||||
|
const PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER = 3;
|
||||||
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
|
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
|
||||||
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
|
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
|
||||||
export const MAX_POST_DONE_NONCONTINUABLE_WEDGE_RECOVERIES = 3;
|
export const MAX_POST_DONE_NONCONTINUABLE_WEDGE_RECOVERIES = 3;
|
||||||
@@ -241,6 +242,8 @@ export interface SelfHealingOptions {
|
|||||||
rootDir: string;
|
rootDir: string;
|
||||||
/** Optional callback to release TaskExecutor in-memory worktree ownership for a task. */
|
/** Optional callback to release TaskExecutor in-memory worktree ownership for a task. */
|
||||||
releaseExecutorWorktreeOwnership?: (taskId: string) => void;
|
releaseExecutorWorktreeOwnership?: (taskId: string) => void;
|
||||||
|
/** Optional callback to clear a demonstrably-stale executor binding without touching live sessions. */
|
||||||
|
clearPhantomExecutorBinding?: (taskId: string) => void;
|
||||||
/** Optional AgentStore for agent-level self-healing checks. */
|
/** Optional AgentStore for agent-level self-healing checks. */
|
||||||
agentStore?: AgentStore;
|
agentStore?: AgentStore;
|
||||||
/** Canonical stale-lease recovery manager. */
|
/** Canonical stale-lease recovery manager. */
|
||||||
@@ -896,6 +899,70 @@ export class SelfHealingManager {
|
|||||||
return activeTaskIds;
|
return activeTaskIds;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async getRecentRunAuditActivityAgeMs(task: Task, nowMs: number): Promise<number | null> {
|
||||||
|
const getRunAuditEvents = (this.store as unknown as {
|
||||||
|
getRunAuditEvents?: (filter: { taskId?: string; startTime?: string; limit?: number }) => Array<{ timestamp?: string }>;
|
||||||
|
}).getRunAuditEvents;
|
||||||
|
if (typeof getRunAuditEvents !== "function") {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const since = new Date(nowMs - RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS).toISOString();
|
||||||
|
const events = getRunAuditEvents.call(this.store, { taskId: task.id, startTime: since, limit: 1 });
|
||||||
|
const newest = events.find((event) => typeof event.timestamp === "string");
|
||||||
|
if (!newest?.timestamp) return null;
|
||||||
|
const timestampMs = Date.parse(newest.timestamp);
|
||||||
|
return Number.isFinite(timestampMs) ? Math.max(0, nowMs - timestampMs) : null;
|
||||||
|
} catch (err: unknown) {
|
||||||
|
const message = err instanceof Error ? err.message : String(err);
|
||||||
|
log.warn(`[self-healing] unable to inspect recent run-audit activity for ${task.id}: ${message}`);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* FNXC:SelfHealingReclaim 2026-06-19-00:00:
|
||||||
|
* FN-6736 requires self-healing to stop treating an in-memory `executor-active` binding as live when the owner is demonstrably dead. Preserve FN-4811 by requiring every live-owner signal to be absent, leave the FN-5219 missing-worktree path untouched, and avoid FN-5704 resume-limbo counters because this path only clears a stale binding and requeues once with progress/worktree preserved.
|
||||||
|
*/
|
||||||
|
private isPhantomExecutorBinding(task: Task, options: {
|
||||||
|
executionAgeMs: number | null;
|
||||||
|
graceMs: number;
|
||||||
|
activeHeartbeatTaskIds: Set<string>;
|
||||||
|
lastActivityMs: number | null;
|
||||||
|
}): { phantom: boolean; metadata: Record<string, unknown> } {
|
||||||
|
const normalizedId = task.id.toUpperCase();
|
||||||
|
const agentPresent = options.activeHeartbeatTaskIds.has(normalizedId);
|
||||||
|
const checkedOutBy = typeof task.checkedOutBy === "string" && task.checkedOutBy.trim().length > 0 ? task.checkedOutBy : null;
|
||||||
|
const worktreeExists = Boolean(task.worktree && existsSync(task.worktree));
|
||||||
|
const hasRecentRunAudit = options.lastActivityMs !== null && options.lastActivityMs <= RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS;
|
||||||
|
const safeAgeMs = options.graceMs * PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER;
|
||||||
|
const metadata = {
|
||||||
|
taskId: task.id,
|
||||||
|
executionAgeMs: options.executionAgeMs,
|
||||||
|
graceMs: options.graceMs,
|
||||||
|
staleBindingAgeFloorMs: safeAgeMs,
|
||||||
|
checkedOutBy,
|
||||||
|
agentPresent,
|
||||||
|
lastActivityMs: options.lastActivityMs,
|
||||||
|
hasRecentRunAudit,
|
||||||
|
worktree: task.worktree ?? null,
|
||||||
|
branch: task.branch ?? null,
|
||||||
|
worktreeExists,
|
||||||
|
};
|
||||||
|
|
||||||
|
return {
|
||||||
|
phantom: task.column === "in-progress"
|
||||||
|
&& worktreeExists
|
||||||
|
&& options.executionAgeMs !== null
|
||||||
|
&& options.executionAgeMs > safeAgeMs
|
||||||
|
&& !checkedOutBy
|
||||||
|
&& !agentPresent
|
||||||
|
&& !hasRecentRunAudit,
|
||||||
|
metadata,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
private getFalsePositiveRequeueSignal(task: Task, options: {
|
private getFalsePositiveRequeueSignal(task: Task, options: {
|
||||||
executingIds?: Set<string>;
|
executingIds?: Set<string>;
|
||||||
activeHeartbeatTaskIds?: Set<string>;
|
activeHeartbeatTaskIds?: Set<string>;
|
||||||
@@ -2659,6 +2726,51 @@ export class SelfHealingManager {
|
|||||||
includeCheckedOutLease: true,
|
includeCheckedOutLease: true,
|
||||||
});
|
});
|
||||||
if (liveExecutionSignal) {
|
if (liveExecutionSignal) {
|
||||||
|
const canEvaluatePhantomBinding = task.column === "in-progress"
|
||||||
|
&& (liveExecutionSignal.reason === "executor-active" || liveExecutionSignal.reason === "live-worktree-and-branch");
|
||||||
|
if (canEvaluatePhantomBinding) {
|
||||||
|
const nowMs = Date.now();
|
||||||
|
const executionStartedAtMs = task.executionStartedAt ? Date.parse(task.executionStartedAt) : Number.NaN;
|
||||||
|
const executionAgeMs = Number.isFinite(executionStartedAtMs) ? Math.max(0, nowMs - executionStartedAtMs) : null;
|
||||||
|
const lastActivityMs = await this.getRecentRunAuditActivityAgeMs(task, nowMs);
|
||||||
|
const phantomBinding = this.isPhantomExecutorBinding(task, {
|
||||||
|
executionAgeMs,
|
||||||
|
graceMs: STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS,
|
||||||
|
activeHeartbeatTaskIds: activeTaskIds,
|
||||||
|
lastActivityMs,
|
||||||
|
});
|
||||||
|
|
||||||
|
/*
|
||||||
|
FNXC:SelfHealingReclaim 2026-06-19-00:00:
|
||||||
|
FN-6736 makes the executor-active veto conditional for in-progress tasks whose worktree still exists: if age is far beyond grace and checkout, heartbeat, and run-audit liveness are all absent, clear only the stale in-memory binding and requeue with worktree/progress intact instead of emitting the permanent no-action wedge. Live FN-4811 owners still reach the normal no-action veto, missing worktrees remain FN-5219, and resume-limbo escalation remains FN-5704-owned.
|
||||||
|
*/
|
||||||
|
if (phantomBinding.phantom) {
|
||||||
|
this.options.clearPhantomExecutorBinding?.(task.id);
|
||||||
|
await createRunAuditor(this.store, {
|
||||||
|
runId: generateSyntheticRunId("self-healing-phantom-executor-binding", task.id),
|
||||||
|
agentId: "self-healing",
|
||||||
|
taskId: task.id,
|
||||||
|
taskLineageId: task.lineageId,
|
||||||
|
phase: "reclaim-self-owned-branch-conflict",
|
||||||
|
}).database({
|
||||||
|
type: "task:reclaim-phantom-executor-binding",
|
||||||
|
target: task.id,
|
||||||
|
metadata: {
|
||||||
|
...phantomBinding.metadata,
|
||||||
|
signalReason: liveExecutionSignal.reason,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
await this.store.moveTask(task.id, "todo", {
|
||||||
|
moveSource: "engine",
|
||||||
|
recoveryRehome: true,
|
||||||
|
preserveProgress: true,
|
||||||
|
preserveWorktree: true,
|
||||||
|
});
|
||||||
|
recovered++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
await this.emitFalsePositiveRequeueNoAction(
|
await this.emitFalsePositiveRequeueNoAction(
|
||||||
task,
|
task,
|
||||||
"reclaim-self-owned-branch-conflict",
|
"reclaim-self-owned-branch-conflict",
|
||||||
|
|||||||
Reference in New Issue
Block a user