FN-6736: reclaim phantom executor bindings
Recover wedged in-progress tasks by clearing stale executor bindings only after liveness proves the owner is gone. - Add a guarded executor escape hatch that clears only stale in-memory task bookkeeping while refusing live session surfaces. - Teach self-healing to identify phantom executor-active bindings using age, checkout, heartbeat, run-audit, and worktree liveness signals before requeueing preserved work. - Record reclaim events in run audit and cover preserved-worktree recovery with reliability interaction tests. - Document the recovery path and add a patch changeset for the published CLI package. Files changed: .changeset/fn-6736-phantom-executor-binding.md | 5 + AGENTS.md | 1 + docs/architecture.md | 1 + .../reclaim-phantom-executor-binding.test.ts | 244 +++++++++++++++++++++ packages/engine/src/executor.ts | 35 +++ packages/engine/src/run-audit.ts | 2 + packages/engine/src/runtimes/in-process-runtime.ts | 3 +- packages/engine/src/self-healing.ts | 112 ++++++++++ 8 files changed, 402 insertions(+), 1 deletion(-) Fusion-Task-Id: FN-6736 Fusion-Task-Lineage: c76191ba-f4c3-4832-a790-67676e258ba2
This commit is contained in:
5
.changeset/fn-6736-phantom-executor-binding.md
Normal file
5
.changeset/fn-6736-phantom-executor-binding.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Recover in-progress tasks wedged behind stale in-memory executor bindings by clearing the phantom binding and requeueing with progress and worktree preserved.
|
||||
@@ -191,6 +191,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
|
||||
- FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes.
|
||||
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
|
||||
- FN-6736: self-healing emits `task:reclaim-phantom-executor-binding` when it proves an in-memory executor-active binding is stale, clears the binding, and requeues the in-progress task with worktree/progress preserved.
|
||||
|
||||
|
||||
## Reference docs (deeper detail)
|
||||
|
||||
@@ -703,6 +703,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
||||
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
|
||||
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
||||
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
||||
- FN-6736 adds a phantom executor-binding liveness gate to the same reclaim path. When the only remaining veto is an in-memory `executor-active`/live-worktree signal, the task is `in-progress`, the execution age is far beyond grace, `checkedOutBy` is empty, no active heartbeat/agent row exists, and run-audit activity is stale, self-healing force-clears the phantom executor binding and requeues the task to `todo` with worktree and progress preserved. Live evidence still wins (FN-4811), missing-worktree limbo remains owned by `recoverInProgressLimbo()` (FN-5219), and the path does not increment FN-5704 resume-limbo counters.
|
||||
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
||||
- Merge commit attribution is ownership-aware: a `mergeDetails.commitSha` is trusted only when reachable from `HEAD` **and** attributable to the task via `Fusion-Task-Id` trailer or task-ID-bearing subject. Reachable-but-unowned SHAs are rejected to prevent sibling done tasks from sharing misleading merge metadata.
|
||||
- FN-4948 adds a task-worktree pre-commit branch-identity guard: provisioning paths (`NativeWorktreeBackend.create`, executor branch creation, and `StepSessionExecutor.createStepWorktree`) install a `pre-commit` hook plus `fusion-task-id` metadata under the worktree's git-path. Commits are refused unless HEAD matches `fusion/<task-id>` or the allowlist (`fusion/step-<n>-<slug>` by default).
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { EventEmitter } from "node:events";
|
||||
import { existsSync, mkdirSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Settings, Task, TaskStore } from "@fusion/core";
|
||||
import { SelfHealingManager, STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS } from "../../self-healing.js";
|
||||
import { activeSessionRegistry } from "../../active-session-registry.js";
|
||||
import * as branchConflictModule from "../../branch-conflicts.js";
|
||||
import * as worktreePoolModule from "../../worktree-pool.js";
|
||||
|
||||
type AuditRow = { timestamp: string };
|
||||
|
||||
type Harness = {
|
||||
rootDir: string;
|
||||
worktree: string;
|
||||
task: Task;
|
||||
store: TaskStore & EventEmitter;
|
||||
clearPhantomExecutorBinding: ReturnType<typeof vi.fn>;
|
||||
manager: SelfHealingManager;
|
||||
cleanup: () => void;
|
||||
};
|
||||
|
||||
const NOW = new Date("2026-06-19T12:00:00.000Z");
|
||||
const OLD_EXECUTION_STARTED_AT = new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 9.5).toISOString();
|
||||
|
||||
function makeTask(overrides: Partial<Task> = {}): Task {
|
||||
return {
|
||||
id: "FN-6736",
|
||||
title: "phantom binding",
|
||||
description: "test",
|
||||
column: "in-progress",
|
||||
branch: "fusion/fn-6736",
|
||||
worktree: "/tmp/fn-6736/.worktrees/crisp-lotus",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
checkedOutBy: undefined,
|
||||
dependencies: [],
|
||||
steps: [{ id: "s1", title: "step", status: "in-progress" } as any],
|
||||
currentStep: 5,
|
||||
log: [],
|
||||
createdAt: new Date(NOW.getTime() - 2 * 60 * 60_000).toISOString(),
|
||||
updatedAt: new Date(NOW.getTime() - 90 * 60_000).toISOString(),
|
||||
executionStartedAt: OLD_EXECUTION_STARTED_AT,
|
||||
...overrides,
|
||||
} as Task;
|
||||
}
|
||||
|
||||
function makeStore(task: Task, options: { recentAuditRows?: AuditRow[] } = {}): TaskStore & EventEmitter {
|
||||
const emitter = new EventEmitter();
|
||||
const settings = {
|
||||
autoMerge: true,
|
||||
globalPause: false,
|
||||
enginePaused: false,
|
||||
baseBranch: "main",
|
||||
mergeStrategy: "direct",
|
||||
autoRecovery: { mode: "deterministic-only", maxRetries: 3 },
|
||||
} as unknown as Settings;
|
||||
|
||||
return Object.assign(emitter, {
|
||||
getSettings: vi.fn(async () => settings),
|
||||
getTask: vi.fn(async () => task),
|
||||
listTasks: vi.fn(async ({ column }: { column?: string } = {}) => (column === task.column ? [task] : [])),
|
||||
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => Object.assign(task, updates)),
|
||||
moveTask: vi.fn(async (_id: string, column: Task["column"], opts?: Record<string, unknown>) => {
|
||||
task.column = column;
|
||||
(task as any).__lastMoveOpts = opts;
|
||||
return task;
|
||||
}),
|
||||
logEntry: vi.fn(async () => undefined),
|
||||
appendAgentLog: vi.fn(async () => undefined),
|
||||
updateSettings: vi.fn(async () => settings),
|
||||
clearStaleExecutionStartBranchReferences: vi.fn(() => []),
|
||||
recordRunAuditEvent: vi.fn(async () => undefined),
|
||||
getRunAuditEvents: vi.fn(() => options.recentAuditRows ?? []),
|
||||
walCheckpoint: vi.fn(() => ({ busy: 0, log: 0, checkpointed: 0 })),
|
||||
archiveTaskAndCleanup: vi.fn(async () => ({})),
|
||||
mergeTask: vi.fn(async () => undefined),
|
||||
getRootDir: vi.fn(() => "/tmp/test"),
|
||||
}) as unknown as TaskStore & EventEmitter;
|
||||
}
|
||||
|
||||
function makeHarness(overrides: Partial<Task> = {}, options: {
|
||||
recentAuditRows?: AuditRow[];
|
||||
activeHeartbeat?: boolean;
|
||||
missingWorktree?: boolean;
|
||||
} = {}): Harness {
|
||||
const rootDir = mkdtempSync(join(tmpdir(), "fn-6736-"));
|
||||
const worktree = join(rootDir, ".worktrees", "crisp-lotus");
|
||||
if (!options.missingWorktree) {
|
||||
mkdirSync(worktree, { recursive: true });
|
||||
}
|
||||
const task = makeTask({ worktree, ...overrides });
|
||||
const store = makeStore(task, { recentAuditRows: options.recentAuditRows });
|
||||
const clearPhantomExecutorBinding = vi.fn();
|
||||
const agentStore = options.activeHeartbeat
|
||||
? { listActiveHeartbeatRuns: vi.fn(async () => [{ startedAt: new Date(NOW.getTime() - 60_000).toISOString(), contextSnapshot: { taskId: task.id } }]) }
|
||||
: { listActiveHeartbeatRuns: vi.fn(async () => []) };
|
||||
const manager = new SelfHealingManager(store as any, {
|
||||
rootDir,
|
||||
getExecutingTaskIds: () => new Set([task.id]),
|
||||
clearPhantomExecutorBinding,
|
||||
agentStore,
|
||||
} as any);
|
||||
return {
|
||||
rootDir,
|
||||
worktree,
|
||||
task,
|
||||
store,
|
||||
clearPhantomExecutorBinding,
|
||||
manager,
|
||||
cleanup: () => {
|
||||
manager.stop();
|
||||
rmSync(rootDir, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function findAudit(store: TaskStore & EventEmitter, mutationType: string): any | undefined {
|
||||
return (store.recordRunAuditEvent as any).mock.calls.find((call: any[]) => call[0].mutationType === mutationType)?.[0];
|
||||
}
|
||||
|
||||
describe("FN-6736: phantom executor binding reclaim", () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(NOW);
|
||||
vi.restoreAllMocks();
|
||||
activeSessionRegistry.clear();
|
||||
vi.spyOn(worktreePoolModule, "isUsableTaskWorktree").mockResolvedValue(true);
|
||||
vi.spyOn(branchConflictModule, "inspectBranchConflict").mockResolvedValue({ kind: "stale" } as any);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
activeSessionRegistry.clear();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("requeues an old in-progress task when executor-active is only a phantom binding", async () => {
|
||||
const h = makeHarness();
|
||||
expect(existsSync(h.worktree)).toBe(true);
|
||||
|
||||
const recovered = await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(recovered).toBe(1);
|
||||
expect(h.clearPhantomExecutorBinding).toHaveBeenCalledWith(h.task.id);
|
||||
expect(h.store.moveTask).toHaveBeenCalledWith(h.task.id, "todo", expect.objectContaining({
|
||||
moveSource: "engine",
|
||||
recoveryRehome: true,
|
||||
preserveProgress: true,
|
||||
preserveWorktree: true,
|
||||
}));
|
||||
expect(h.task.column).toBe("todo");
|
||||
expect(h.task.userPaused).toBe(false);
|
||||
expect(h.task.paused).toBe(false);
|
||||
expect((h.task as any).status).not.toBe("failed");
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")).toBeUndefined();
|
||||
const event = findAudit(h.store, "task:reclaim-phantom-executor-binding");
|
||||
expect(event).toBeTruthy();
|
||||
expect(event.metadata).toEqual(expect.objectContaining({
|
||||
taskId: h.task.id,
|
||||
signalReason: "executor-active",
|
||||
checkedOutBy: null,
|
||||
agentPresent: false,
|
||||
lastActivityMs: null,
|
||||
worktree: h.worktree,
|
||||
branch: h.task.branch,
|
||||
worktreeExists: true,
|
||||
}));
|
||||
expect(event.metadata.executionAgeMs).toBeGreaterThan(STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS * 3);
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("keeps FN-4811 protection when recent run-audit activity proves a live owner", async () => {
|
||||
const h = makeHarness({}, { recentAuditRows: [{ timestamp: new Date(NOW.getTime() - 60_000).toISOString() }] });
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
expect(findAudit(h.store, "task:reclaim-phantom-executor-binding")).toBeUndefined();
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||
expect(h.task.column).toBe("in-progress");
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("keeps FN-4811 protection when checkedOutBy is set", async () => {
|
||||
const h = makeHarness({ checkedOutBy: "agent-1" } as Partial<Task>);
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("keeps FN-4811 protection when an active heartbeat row exists", async () => {
|
||||
const h = makeHarness({}, { activeHeartbeat: true });
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("protects tasks just past grace but below the phantom age multiplier", async () => {
|
||||
const h = makeHarness({
|
||||
executionStartedAt: new Date(NOW.getTime() - STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS - 1_000).toISOString(),
|
||||
});
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("does not double-handle missing worktrees owned by FN-5219 in-progress limbo recovery", async () => {
|
||||
const h = makeHarness({}, { missingWorktree: true });
|
||||
expect(existsSync(h.worktree)).toBe(false);
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.clearPhantomExecutorBinding).not.toHaveBeenCalled();
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
expect(findAudit(h.store, "task:reclaim-self-owned-branch-conflict-no-action")?.metadata).toEqual(expect.objectContaining({ reason: "executor-active" }));
|
||||
h.cleanup();
|
||||
});
|
||||
|
||||
it("does not increment FN-5704 resume-limbo counters on the phantom-binding requeue", async () => {
|
||||
const h = makeHarness({ resumeLimboCount: 1 } as Partial<Task>);
|
||||
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
await h.manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(h.store.moveTask).toHaveBeenCalledTimes(1);
|
||||
expect(h.task.resumeLimboCount).toBe(1);
|
||||
expect(findAudit(h.store, "task:resume-limbo-escalated")).toBeUndefined();
|
||||
h.cleanup();
|
||||
});
|
||||
});
|
||||
@@ -1997,6 +1997,41 @@ export class TaskExecutor {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ExecutorBinding 2026-06-19-00:00:
|
||||
* FN-6736 gives self-healing a narrow escape hatch for phantom in-memory executor bindings after the liveness gate proves the owner is dead. Never use this as a general task stopper: it refuses to detach observable live session surfaces, then clears only stale bookkeeping (`executing`, resume/recovery sets, process-wide graph routing, activeWorktrees, activeSessionRegistry paths, and executingTaskLock) so the scheduler can re-dispatch the preserved worktree.
|
||||
*/
|
||||
clearPhantomExecutorBinding(taskId: string): boolean {
|
||||
const hasLiveSessionSurface = this.activeSessions.has(taskId)
|
||||
|| this.activeStepExecutors.has(taskId)
|
||||
|| this.activeWorkflowStepSessions.has(taskId)
|
||||
|| this.activeCliTaskSessions.has(taskId);
|
||||
if (hasLiveSessionSurface) {
|
||||
executorLog.warn(`${taskId}: refusing to clear phantom executor binding because a live session surface is still registered`);
|
||||
return false;
|
||||
}
|
||||
|
||||
const worktreePath = this.activeWorktrees.get(taskId);
|
||||
this.activeWorktrees.delete(taskId);
|
||||
this.executing.delete(taskId);
|
||||
this.recoveringCompleted.delete(taskId);
|
||||
this.resumingUnpaused.delete(taskId);
|
||||
TaskExecutor.processWideGraphRouting.delete(taskId);
|
||||
executingTaskLock.release(taskId);
|
||||
this.effectiveColumnAgentByTask.delete(taskId);
|
||||
|
||||
const registeredPaths = new Set(activeSessionRegistry.pathsForTask(taskId));
|
||||
if (worktreePath) {
|
||||
registeredPaths.add(worktreePath);
|
||||
}
|
||||
for (const path of registeredPaths) {
|
||||
activeSessionRegistry.unregisterPath(path);
|
||||
}
|
||||
|
||||
executorLog.warn(`${taskId}: cleared phantom executor binding for self-healing re-dispatch`);
|
||||
return true;
|
||||
}
|
||||
|
||||
isEphemeralDeletionPending(agentId: string): boolean {
|
||||
return this.pendingEphemeralDeletions.has(agentId);
|
||||
}
|
||||
|
||||
@@ -505,6 +505,8 @@ export type DatabaseMutationType =
|
||||
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
||||
| "task:auto-recover-in-progress-limbo-no-action"
|
||||
| "task:resume-limbo-escalated"
|
||||
/** Metadata: { taskId, executionAgeMs, graceMs, staleBindingAgeFloorMs, checkedOutBy, agentPresent, lastActivityMs, hasRecentRunAudit, worktree, branch, worktreeExists, signalReason } */
|
||||
| "task:reclaim-phantom-executor-binding"
|
||||
/** Metadata: { taskId, branch, worktree, checkedOutBy, executionStartedAt, executionAgeMs, graceMs, liveWorktreeBoundBranch, reason } */
|
||||
| "task:reclaim-self-owned-branch-conflict-no-action"
|
||||
| "task:orphan-detected-no-action"
|
||||
|
||||
@@ -786,7 +786,8 @@ export class InProcessRuntime
|
||||
isWorktreeResumeReserved: this.cliAgentRuntime?.isWorktreeResumeReserved,
|
||||
recoverCompletedTask: (task) => this.executor.recoverCompletedTask(task),
|
||||
recoverFailedPreMergeStep: (task) => this.executor.recoverFailedPreMergeWorkflowStep(task),
|
||||
getExecutingTaskIds: () => this.executor.getExecutingTaskIds(),
|
||||
getExecutingTaskIds: () => this.executor?.getExecutingTaskIds() ?? new Set<string>(),
|
||||
clearPhantomExecutorBinding: (taskId: string) => this.executor?.clearPhantomExecutorBinding(taskId),
|
||||
recoverApprovedTriageTask: (task) => this.triageProcessor?.recoverApprovedTask(task) ?? Promise.resolve(false),
|
||||
getPlanningTaskIds: () => this.triageProcessor?.getProcessingTaskIds() ?? new Set<string>(),
|
||||
evictStaleTriageProcessing: () => this.triageProcessor?.evictStaleProcessing() ?? new Set<string>(),
|
||||
|
||||
@@ -92,6 +92,7 @@ const ARCHIVE_FTS_MAINTENANCE_OPTIMIZE_CADENCE_TICKS = 24;
|
||||
const ARCHIVE_FTS_REBUILD_THRESHOLD_BYTES = 64 * 1024 * 1024;
|
||||
const ARCHIVE_FTS_REBUILD_BYTES_PER_TASK = 512 * 1024;
|
||||
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
|
||||
const PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER = 3;
|
||||
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
|
||||
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
|
||||
export const MAX_POST_DONE_NONCONTINUABLE_WEDGE_RECOVERIES = 3;
|
||||
@@ -241,6 +242,8 @@ export interface SelfHealingOptions {
|
||||
rootDir: string;
|
||||
/** Optional callback to release TaskExecutor in-memory worktree ownership for a task. */
|
||||
releaseExecutorWorktreeOwnership?: (taskId: string) => void;
|
||||
/** Optional callback to clear a demonstrably-stale executor binding without touching live sessions. */
|
||||
clearPhantomExecutorBinding?: (taskId: string) => void;
|
||||
/** Optional AgentStore for agent-level self-healing checks. */
|
||||
agentStore?: AgentStore;
|
||||
/** Canonical stale-lease recovery manager. */
|
||||
@@ -896,6 +899,70 @@ export class SelfHealingManager {
|
||||
return activeTaskIds;
|
||||
}
|
||||
|
||||
private async getRecentRunAuditActivityAgeMs(task: Task, nowMs: number): Promise<number | null> {
|
||||
const getRunAuditEvents = (this.store as unknown as {
|
||||
getRunAuditEvents?: (filter: { taskId?: string; startTime?: string; limit?: number }) => Array<{ timestamp?: string }>;
|
||||
}).getRunAuditEvents;
|
||||
if (typeof getRunAuditEvents !== "function") {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
const since = new Date(nowMs - RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS).toISOString();
|
||||
const events = getRunAuditEvents.call(this.store, { taskId: task.id, startTime: since, limit: 1 });
|
||||
const newest = events.find((event) => typeof event.timestamp === "string");
|
||||
if (!newest?.timestamp) return null;
|
||||
const timestampMs = Date.parse(newest.timestamp);
|
||||
return Number.isFinite(timestampMs) ? Math.max(0, nowMs - timestampMs) : null;
|
||||
} catch (err: unknown) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
log.warn(`[self-healing] unable to inspect recent run-audit activity for ${task.id}: ${message}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:SelfHealingReclaim 2026-06-19-00:00:
|
||||
* FN-6736 requires self-healing to stop treating an in-memory `executor-active` binding as live when the owner is demonstrably dead. Preserve FN-4811 by requiring every live-owner signal to be absent, leave the FN-5219 missing-worktree path untouched, and avoid FN-5704 resume-limbo counters because this path only clears a stale binding and requeues once with progress/worktree preserved.
|
||||
*/
|
||||
private isPhantomExecutorBinding(task: Task, options: {
|
||||
executionAgeMs: number | null;
|
||||
graceMs: number;
|
||||
activeHeartbeatTaskIds: Set<string>;
|
||||
lastActivityMs: number | null;
|
||||
}): { phantom: boolean; metadata: Record<string, unknown> } {
|
||||
const normalizedId = task.id.toUpperCase();
|
||||
const agentPresent = options.activeHeartbeatTaskIds.has(normalizedId);
|
||||
const checkedOutBy = typeof task.checkedOutBy === "string" && task.checkedOutBy.trim().length > 0 ? task.checkedOutBy : null;
|
||||
const worktreeExists = Boolean(task.worktree && existsSync(task.worktree));
|
||||
const hasRecentRunAudit = options.lastActivityMs !== null && options.lastActivityMs <= RUNNING_ON_INACTIVE_TASK_STALE_RUN_MS;
|
||||
const safeAgeMs = options.graceMs * PHANTOM_EXECUTOR_BINDING_AGE_MULTIPLIER;
|
||||
const metadata = {
|
||||
taskId: task.id,
|
||||
executionAgeMs: options.executionAgeMs,
|
||||
graceMs: options.graceMs,
|
||||
staleBindingAgeFloorMs: safeAgeMs,
|
||||
checkedOutBy,
|
||||
agentPresent,
|
||||
lastActivityMs: options.lastActivityMs,
|
||||
hasRecentRunAudit,
|
||||
worktree: task.worktree ?? null,
|
||||
branch: task.branch ?? null,
|
||||
worktreeExists,
|
||||
};
|
||||
|
||||
return {
|
||||
phantom: task.column === "in-progress"
|
||||
&& worktreeExists
|
||||
&& options.executionAgeMs !== null
|
||||
&& options.executionAgeMs > safeAgeMs
|
||||
&& !checkedOutBy
|
||||
&& !agentPresent
|
||||
&& !hasRecentRunAudit,
|
||||
metadata,
|
||||
};
|
||||
}
|
||||
|
||||
private getFalsePositiveRequeueSignal(task: Task, options: {
|
||||
executingIds?: Set<string>;
|
||||
activeHeartbeatTaskIds?: Set<string>;
|
||||
@@ -2659,6 +2726,51 @@ export class SelfHealingManager {
|
||||
includeCheckedOutLease: true,
|
||||
});
|
||||
if (liveExecutionSignal) {
|
||||
const canEvaluatePhantomBinding = task.column === "in-progress"
|
||||
&& (liveExecutionSignal.reason === "executor-active" || liveExecutionSignal.reason === "live-worktree-and-branch");
|
||||
if (canEvaluatePhantomBinding) {
|
||||
const nowMs = Date.now();
|
||||
const executionStartedAtMs = task.executionStartedAt ? Date.parse(task.executionStartedAt) : Number.NaN;
|
||||
const executionAgeMs = Number.isFinite(executionStartedAtMs) ? Math.max(0, nowMs - executionStartedAtMs) : null;
|
||||
const lastActivityMs = await this.getRecentRunAuditActivityAgeMs(task, nowMs);
|
||||
const phantomBinding = this.isPhantomExecutorBinding(task, {
|
||||
executionAgeMs,
|
||||
graceMs: STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS,
|
||||
activeHeartbeatTaskIds: activeTaskIds,
|
||||
lastActivityMs,
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:SelfHealingReclaim 2026-06-19-00:00:
|
||||
FN-6736 makes the executor-active veto conditional for in-progress tasks whose worktree still exists: if age is far beyond grace and checkout, heartbeat, and run-audit liveness are all absent, clear only the stale in-memory binding and requeue with worktree/progress intact instead of emitting the permanent no-action wedge. Live FN-4811 owners still reach the normal no-action veto, missing worktrees remain FN-5219, and resume-limbo escalation remains FN-5704-owned.
|
||||
*/
|
||||
if (phantomBinding.phantom) {
|
||||
this.options.clearPhantomExecutorBinding?.(task.id);
|
||||
await createRunAuditor(this.store, {
|
||||
runId: generateSyntheticRunId("self-healing-phantom-executor-binding", task.id),
|
||||
agentId: "self-healing",
|
||||
taskId: task.id,
|
||||
taskLineageId: task.lineageId,
|
||||
phase: "reclaim-self-owned-branch-conflict",
|
||||
}).database({
|
||||
type: "task:reclaim-phantom-executor-binding",
|
||||
target: task.id,
|
||||
metadata: {
|
||||
...phantomBinding.metadata,
|
||||
signalReason: liveExecutionSignal.reason,
|
||||
},
|
||||
});
|
||||
await this.store.moveTask(task.id, "todo", {
|
||||
moveSource: "engine",
|
||||
recoveryRehome: true,
|
||||
preserveProgress: true,
|
||||
preserveWorktree: true,
|
||||
});
|
||||
recovered++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
await this.emitFalsePositiveRequeueNoAction(
|
||||
task,
|
||||
"reclaim-self-owned-branch-conflict",
|
||||
|
||||
Reference in New Issue
Block a user