FN-6257: make AI merge cleanup idempotent for absent worktrees

AI merge cleanup now tolerates already-removed worktrees without failing a landed merge.

- Treat ENOENT and de-registered worktree removal errors as successful idempotent cleanup.
- Register AI merge temp paths before worktree creation and keep raw/canonical paths active.
- Add cleanup regression coverage and include the cleanup suite in reliability tests.
- Document the idempotent cleanup telemetry contract.

Files changed:
 docs/architecture.md                               |   2 +-
 .../engine/src/__tests__/merger-ai-cleanup.test.ts |  48 +++++++-
 .../ai-merge-cleanup-enoent-idempotent.test.ts     | 128 +++++++++++++++++++++
 packages/engine/src/merger-ai.ts                   |  73 +++++++++---
 packages/engine/vitest.config.ts                   |   5 +-
 5 files changed, 239 insertions(+), 17 deletions(-)

Fusion-Task-Id: FN-6257

Fusion-Task-Lineage: c669fa1e-2c92-4e2f-8726-4939a3146955
This commit is contained in:
gsxdsm
2026-06-11 18:14:45 -07:00
parent 9a78814418
commit 94c3cce7bc
5 changed files with 239 additions and 17 deletions

View File

@@ -670,7 +670,7 @@ Runtime action-gate flow (v1):
- `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
- `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
- Batch 1 maintenance now includes one `fts-maintenance` step for both search indexes. The live `tasks_fts` branch still runs `merge` every tick, `optimize` every 4th tick, and `rebuild` above `32 MiB` or `1 MiB × live task count`. The archive `archived_tasks_fts` branch is lighter because archive writes are mostly append-only: `merge` every 8th tick, `optimize` every 24th tick, and `rebuild` above `64 MiB` or `512 KiB × archived row count`. Each branch is independently guarded by `fts5Available` and emits `task:fts-maintenance` run-audit telemetry with distinct `target` values (`tasks_fts` vs `archived_tasks_fts`).
- Batch 1 also sweeps stale AI merge clean-room worktrees under `tmpdir()` whose names start with `fusion-ai-merge-`. `runAiMerge` registers each live clean-room worktree in `activeSessionRegistry` with kind `ai-merge` for the duration of the merge, and both the periodic sweep and pre-merge prune defer when the canonical or raw path is active. The default age gate is 2 hours; task-aware cleanup uses a 10-minute grace period for `done`/`archived` tasks and for genuinely missing/deleted task rows, and every removal path is clamped by the same 10-minute minimum-age floor so a freshly created worktree is never reaped. Transient `getTask` lookup failures (for example SQLite busy/parse errors) are not treated as deletion evidence; they log a warning, emit `lookup-error` only if eventually removed, and retain the conservative 2-hour gate. The sweep canonicalizes paths before checking `activeSessionRegistry`, attempts `git worktree remove --force <path>` before filesystem removal, and emits `worktree:tempdir-sweep` run-audit telemetry for removal attempts and failures. It is intentionally native even when `worktrunk.enabled` because these temp-dir worktrees are outside the worktrunk-managed project layout. Fresh directories, active-session paths, and individual removal failures are skipped/logged without aborting the maintenance cycle.
- Batch 1 also sweeps stale AI merge clean-room worktrees under `tmpdir()` whose names start with `fusion-ai-merge-`. `runAiMerge` registers each live clean-room worktree in `activeSessionRegistry` with kind `ai-merge` as soon as the temp directory exists and keeps both raw and canonical paths registered for the duration of the merge, so both the periodic sweep and pre-merge prune defer when either path is active. The default age gate is 2 hours; task-aware cleanup uses a 10-minute grace period for `done`/`archived` tasks and for genuinely missing/deleted task rows, and every removal path is clamped by the same 10-minute minimum-age floor so a freshly created worktree is never reaped. Transient `getTask` lookup failures (for example SQLite busy/parse errors) are not treated as deletion evidence; they log a warning, emit `lookup-error` only if eventually removed, and retain the conservative 2-hour gate. The sweep canonicalizes paths before checking `activeSessionRegistry`, attempts `git worktree remove --force <path>` before filesystem removal, and emits `worktree:tempdir-sweep` run-audit telemetry for removal attempts and failures. Standalone AI-merge cleanup is idempotent for an already-removed/de-registered temp worktree (`ENOENT`, `spawn git ENOENT`, `No such file or directory`, or `is not a working tree`): it emits successful `merge:ai-worktree-cleanup` telemetry with `alreadyAbsent`/`idempotent` markers and must not turn a confirmed land into a failed merge, while genuine removal failures still emit `success:false`. It is intentionally native even when `worktrunk.enabled` because these temp-dir worktrees are outside the worktrunk-managed project layout. Fresh directories, active-session paths, and individual removal failures are skipped/logged without aborting the maintenance cycle.
- `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`.
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.

View File

@@ -169,6 +169,52 @@ describe("AI merge temp worktree cleanup", () => {
]));
});
it("treats spawn git ENOENT during cleanup as idempotent already-absent success", async () => {
const mergeRoot = mkdtempSync(join(tmpdir(), "fusion-ai-merge-fn-1-enoent-cleanup-test-"));
tracked.add(mergeRoot);
const err = Object.assign(new Error("spawn git ENOENT"), { code: "ENOENT" });
const gitRunner = vi.fn(async () => { throw err; });
const { events, logs } = await cleanup({
mergeRoot,
gitRunner,
});
expect(gitRunner).toHaveBeenCalledWith(["worktree", "remove", "--force", mergeRoot], process.cwd());
expect(events).toEqual(expect.arrayContaining([
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "git-remove", success: true, alreadyAbsent: true, idempotent: true, code: "ENOENT" }) }),
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "fs-rm", success: true, alreadyAbsent: true, idempotent: true }) }),
]));
expect(logs.join("\n")).toContain("treating cleanup as idempotent");
});
it("removes the directory after git reports the temp path is not a working tree", async () => {
const err = new Error("Command failed: git worktree remove --force /tmp/fusion-ai-merge-fn-1\nfatal: '/tmp/fusion-ai-merge-fn-1' is not a working tree");
const { mergeRoot, events } = await cleanup({
gitRunner: vi.fn(async () => { throw err; }),
});
expect(existsSync(mergeRoot)).toBe(false);
expect(events).toEqual(expect.arrayContaining([
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "git-remove", success: true, alreadyAbsent: true, idempotent: true }) }),
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "fs-rm", success: true, alreadyAbsent: true, idempotent: true }) }),
]));
});
it("still surfaces genuine filesystem cleanup failures", async () => {
const err = new Error("Directory not empty") as NodeJS.ErrnoException;
err.code = "ENOTEMPTY";
const { events, logs } = await cleanup({ rmRunner: vi.fn(async () => { throw err; }) as typeof rm });
expect(events).toEqual(expect.arrayContaining([
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "fs-rm", success: false, code: "ENOTEMPTY", error: "Directory not empty" }) }),
]));
expect(logs.join("\n")).toContain("filesystem rm failed");
});
it("skips git removal but still audits filesystem cleanup when worktree was not added", async () => {
const gitRunner = vi.fn(async () => "");
@@ -191,7 +237,7 @@ describe("AI merge temp worktree cleanup", () => {
expect(existsSync(stale)).toBe(false);
expect(events).toEqual(expect.arrayContaining([
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ taskId: "FN-777", mergeRoot: realpathSync(tmpdir()) + "/fusion-ai-merge-fn-777-stale", phase: "pre-merge-prune", success: true }) }),
expect.objectContaining({ type: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ taskId: "FN-777", mergeRoot: realpathSync(tmpdir()) + "/fusion-ai-merge-fn-777-stale", phase: "pre-merge-prune", success: true, alreadyAbsent: true, idempotent: true }) }),
]));
});

View File

@@ -0,0 +1,128 @@
import { afterAll, describe, expect, it, vi } from "vitest";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { execSync } from "node:child_process";
import { runAiMerge } from "../../merger-ai.js";
import { hasGit } from "./_helpers.js";
const tracked = new Set<string>();
const RM = { recursive: true, force: true, maxRetries: 5, retryDelay: 50 } as const;
afterAll(() => {
for (const dir of tracked) {
try { rmSync(dir, RM); } catch { /* best effort cleanup */ }
}
});
function git(cwd: string, args: string): string {
return execSync(`git ${args}`, { cwd, encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim();
}
function createRepo(taskId: string): { rootDir: string; branch: string } {
const branch = `fusion/${taskId.toLowerCase()}`;
const rootDir = mkdtempSync(join(tmpdir(), "fusion-ai-merge-enoent-"));
tracked.add(rootDir);
git(rootDir, "init -q -b main");
git(rootDir, 'config user.email "test@example.com"');
git(rootDir, 'config user.name "Test User"');
writeFileSync(join(rootDir, "README.md"), "# fixture\n");
git(rootDir, "add README.md");
git(rootDir, 'commit -q -m "chore: init"');
git(rootDir, `checkout -q -b ${branch}`);
writeFileSync(join(rootDir, "feature.txt"), "feature work\n");
git(rootDir, "add feature.txt");
git(rootDir, 'commit -q -m "feat: task work"');
git(rootDir, "checkout -q main");
return { rootDir, branch };
}
function makeStore(taskId: string, branch: string) {
const task: any = {
id: taskId,
column: "in-review",
status: null,
branch,
baseBranch: "main",
worktree: null,
title: "AI merge cleanup ENOENT fixture",
steps: [{ title: "ready", status: "done" }],
};
const audits: any[] = [];
const logs: string[] = [];
const store: any = {
getTask: vi.fn(async () => task),
getSettings: vi.fn(async () => ({
autoMerge: true,
includeTaskIdInCommit: true,
commitAuthorEnabled: false,
merger: { mode: "ai", maxReviewPasses: 1 },
})),
updateTask: vi.fn(async (_id: string, patch: Record<string, unknown>) => { Object.assign(task, patch); return task; }),
moveTask: vi.fn(async (_id: string, column: string) => { task.column = column; return task; }),
emit: vi.fn(),
logEntry: vi.fn(async (_id: string, message: string) => { logs.push(message); }),
appendAgentLog: vi.fn(async (_id: string, message: string) => { logs.push(message); }),
recordRunAuditEvent: vi.fn(async (event: any) => { audits.push(event); }),
};
return { store, task, audits, logs };
}
function realMergeAgent(branch: string, onCwd?: (cwd: string) => void) {
return vi.fn(async (cwd: string) => {
onCwd?.(cwd);
execSync(`git merge --squash ${branch}`, { cwd, stdio: "pipe" });
execSync("git add -A", { cwd, stdio: "pipe" });
execSync('git commit -q -m "squash: feature"', { cwd, stdio: "pipe" });
});
}
describe("FN-6257 AI-merge cleanup ENOENT idempotency (real git)", () => {
it.skipIf(!hasGit)("finalizes done when the temp worktree vanishes after the squash lands", async () => {
const taskId = "FN-6257-RI";
const { rootDir, branch } = createRepo(taskId);
const { store, task, audits } = makeStore(taskId, branch);
const originalRecordRunAuditEvent = store.recordRunAuditEvent;
let observedMergeRoot = "";
let removedAfterConfirmedLand = false;
store.recordRunAuditEvent = vi.fn(async (event: any) => {
const confirmedLandEvent = (
event.mutationType === "merge:integration-ref-advance" && event.metadata?.succeeded === true
) || (
event.mutationType === "merge:ai-local-sync" && ["ff", "skipped-other-branch", "stash-ff-restore", "stash-ff-airesolved", "stash-ff-conflict"].includes(String(event.metadata?.outcome ?? ""))
);
if (confirmedLandEvent && observedMergeRoot && !removedAfterConfirmedLand) {
removedAfterConfirmedLand = true;
rmSync(observedMergeRoot, RM);
}
await originalRecordRunAuditEvent(event);
});
const mainBefore = git(rootDir, "rev-parse main");
const result = await runAiMerge(store, rootDir, taskId, { manual: true, allowDirtyLocalCheckoutSync: true }, {
mergeAgent: realMergeAgent(branch, (cwd) => { observedMergeRoot = cwd; }),
reviewAgent: vi.fn(async () => "REVIEW_VERDICT: approve"),
});
expect(removedAfterConfirmedLand).toBe(true);
expect(result).toMatchObject({ ok: true, merged: true, mergeConfirmed: true });
expect(git(rootDir, "rev-parse main")).not.toBe(mainBefore);
expect(task.column).toBe("done");
expect(task.status ?? null).toBeNull();
expect(task.error).toBeUndefined();
expect(task.mergeRetries ?? 0).not.toBeGreaterThanOrEqual(3);
expect(task.mergeDetails).toEqual(expect.objectContaining({
commitSha: result.commitSha,
mergeConfirmed: true,
}));
expect(audits).toEqual(expect.arrayContaining([
expect.objectContaining({ mutationType: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "git-remove", success: true, alreadyAbsent: true, idempotent: true }) }),
expect.objectContaining({ mutationType: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ phase: "fs-rm", success: true, alreadyAbsent: true, idempotent: true }) }),
]));
expect(audits).not.toEqual(expect.arrayContaining([
expect.objectContaining({ mutationType: "merge:ai-worktree-cleanup", metadata: expect.objectContaining({ success: false }) }),
]));
}, 20_000);
});

View File

@@ -32,7 +32,7 @@
*/
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { readdirSync, realpathSync, rmSync, statSync } from "node:fs";
import { existsSync, readdirSync, realpathSync, rmSync, statSync } from "node:fs";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -106,6 +106,13 @@ function describeCleanupError(err: unknown): string {
return stderr ? `${message}: ${stderr.trim()}` : message;
}
export function isBenignAbsentWorktreeError(err: unknown): boolean {
const code = getErrorStringProperty(err, "code");
if (code === "ENOENT") return true;
const description = describeCleanupError(err);
return /is not a working tree|No such file or directory|spawn\s+.*\bENOENT\b/i.test(description);
}
export async function pruneExistingAiMergeWorktrees(
taskId: string,
projectRootDir: string,
@@ -149,20 +156,32 @@ export async function pruneExistingAiMergeWorktrees(
continue;
}
let alreadyAbsent = false;
try {
await execFileAsync("git", ["worktree", "remove", "--force", canonicalPath], {
cwd: projectRootDir,
timeout: 30_000,
});
} catch (err: unknown) {
await log(`AI merge pre-merge prune: git worktree remove failed for ${canonicalPath}: ${describeCleanupError(err)} — falling back to filesystem removal`);
if (isBenignAbsentWorktreeError(err)) {
alreadyAbsent = true;
await log(`AI merge pre-merge prune: worktree ${canonicalPath} was already absent/de-registered; treating cleanup as idempotent`);
} else {
await log(`AI merge pre-merge prune: git worktree remove failed for ${canonicalPath}: ${describeCleanupError(err)} — falling back to filesystem removal`);
}
}
try {
rmSync(canonicalPath, { recursive: true, force: true });
await audit.git({ type: "merge:ai-worktree-cleanup", target: canonicalPath, metadata: { taskId, mergeRoot: canonicalPath, phase: "pre-merge-prune", success: true } });
await audit.git({ type: "merge:ai-worktree-cleanup", target: canonicalPath, metadata: { taskId, mergeRoot: canonicalPath, phase: "pre-merge-prune", success: true, ...(alreadyAbsent ? { alreadyAbsent: true, idempotent: true } : {}) } });
pruned++;
} catch (err: unknown) {
if (isBenignAbsentWorktreeError(err)) {
await log(`AI merge pre-merge prune: worktree ${canonicalPath} was already absent during filesystem cleanup; treating cleanup as idempotent`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: canonicalPath, metadata: { taskId, mergeRoot: canonicalPath, phase: "pre-merge-prune", success: true, alreadyAbsent: true, idempotent: true } });
pruned++;
continue;
}
const error = getErrorMessage(err);
const code = getErrorStringProperty(err, "code");
await log(`AI merge pre-merge prune: filesystem rm failed for ${canonicalPath}${code ? ` (${code})` : ""}: ${error}`);
@@ -184,23 +203,41 @@ export async function cleanupAiMergeWorktree(input: {
rmRunner?: typeof rm;
}): Promise<void> {
const { taskId, mergeRoot, projectRootDir, worktreeAdded, audit, log, gitRunner = git, rmRunner = rm } = input;
let alreadyAbsent = false;
if (worktreeAdded) {
try {
await gitRunner(["worktree", "remove", "--force", mergeRoot], projectRootDir);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: true } });
} catch (err: unknown) {
const error = describeCleanupError(err);
const code = getErrorStringProperty(err, "code");
await log(`AI merge cleanup: git worktree remove failed for ${mergeRoot}${code ? ` (${code})` : ""}: ${error}`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: false, error, ...(code ? { code } : {}) } });
if (!existsSync(mergeRoot)) {
alreadyAbsent = true;
await log(`AI merge cleanup: worktree ${mergeRoot} was already absent before git removal; treating cleanup as idempotent`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: true, alreadyAbsent: true, idempotent: true, code: "ENOENT" } });
} else {
try {
await gitRunner(["worktree", "remove", "--force", mergeRoot], projectRootDir);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: true } });
} catch (err: unknown) {
const error = describeCleanupError(err);
const code = getErrorStringProperty(err, "code");
if (isBenignAbsentWorktreeError(err)) {
alreadyAbsent = true;
await log(`AI merge cleanup: worktree ${mergeRoot} was already absent/de-registered during git removal; treating cleanup as idempotent`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: true, alreadyAbsent: true, idempotent: true, error, ...(code ? { code } : {}) } });
} else {
await log(`AI merge cleanup: git worktree remove failed for ${mergeRoot}${code ? ` (${code})` : ""}: ${error}`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "git-remove", success: false, error, ...(code ? { code } : {}) } });
}
}
}
}
try {
await rmRunner(mergeRoot, { recursive: true, force: true });
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "fs-rm", success: true } });
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "fs-rm", success: true, ...(alreadyAbsent ? { alreadyAbsent: true, idempotent: true } : {}) } });
} catch (err: unknown) {
const error = getErrorMessage(err);
const code = getErrorStringProperty(err, "code");
if (isBenignAbsentWorktreeError(err)) {
await log(`AI merge cleanup: worktree ${mergeRoot} was already absent during filesystem cleanup; treating cleanup as idempotent`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "fs-rm", success: true, alreadyAbsent: true, idempotent: true, error, ...(code ? { code } : {}) } });
return;
}
await log(`AI merge cleanup: filesystem rm failed for ${mergeRoot}${code ? ` (${code})` : ""}: ${error}`);
await audit.git({ type: "merge:ai-worktree-cleanup", target: mergeRoot, metadata: { taskId, mergeRoot, phase: "fs-rm", success: false, error, ...(code ? { code } : {}) } });
}
@@ -924,6 +961,15 @@ export async function runAiMerge(
const mergeRoot = await mkdtemp(join(tmpdir(), `fusion-ai-merge-${taskId.toLowerCase()}-`));
let worktreeAdded = false;
const registeredMergePaths = new Set<string>();
const registerMergeRoot = (pathToRegister: string): void => {
if (registeredMergePaths.has(pathToRegister)) return;
activeSessionRegistry.registerPath(pathToRegister, { taskId, kind: "ai-merge", ownerKey: `ai-merge:${taskId}` });
registeredMergePaths.add(pathToRegister);
};
// Register the tmpdir path as soon as it exists, before `git worktree add`,
// so the self-healing tmpdir sweep cannot reap a just-created clean room in
// the small window before canonical registration is available.
registerMergeRoot(mergeRoot);
try {
await git(["worktree", "add", "--detach", mergeRoot, tipSha], projectRootDir);
worktreeAdded = true;
@@ -934,8 +980,7 @@ export async function runAiMerge(
canonicalMergeRoot = mergeRoot;
}
for (const pathToRegister of new Set([canonicalMergeRoot, mergeRoot])) {
activeSessionRegistry.registerPath(pathToRegister, { taskId, kind: "ai-merge", ownerKey: `ai-merge:${taskId}` });
registeredMergePaths.add(pathToRegister);
registerMergeRoot(pathToRegister);
}
await audit.git({ type: "merge:ai-clean-room", target: integrationBranch, metadata: { taskId, tipSha, mergeRoot } });
await log(`AI merge: merging ${branch} into ${integrationBranch} (clean room at ${short(tipSha)})${advanceRetries ? ` — retry ${advanceRetries} after concurrent advance` : ""}`);

View File

@@ -116,7 +116,10 @@ export default defineConfig({
extends: true,
test: {
name: "engine-reliability",
include: ["src/__tests__/reliability-interactions/**/*.test.ts"],
include: [
"src/__tests__/reliability-interactions/**/*.test.ts",
"src/__tests__/merger-ai-cleanup.test.ts",
],
// Mirror the engine-default exclusion so reliability slow tests
// also tier into engine-slow.
exclude: ["src/**/*.slow.test.ts"],