FN-6227: handle non-empty worktree removals

Recover native worktree cleanup when git refuses to remove a non-empty worktree.

- Detect recoverable git worktree remove errors and fall back to filesystem removal plus admin pruning.
- Add a worktree remove fallback audit event for recovery visibility.
- Cover non-empty removal recovery with unit and real-git regression tests.
- Keep flaky-test quarantine exclusions merged with current main.

Files changed:
 docs/architecture.md                               |   4 +-
 ...tree-remove-non-empty-recovery.real-git.test.ts | 139 +++++++++++++++++++++
 .../engine/src/__tests__/worktree-backend.test.ts  | 113 ++++++++++++++++-
 packages/engine/src/run-audit.ts                   |   1 +
 packages/engine/src/worktree-backend.ts            |  57 ++++++++-
 packages/engine/vitest.config.ts                   |   1 +
 scripts/lib/test-quarantine.json                   |   5 +
 7 files changed, 311 insertions(+), 9 deletions(-)

Fusion-Task-Id: FN-6227

Fusion-Task-Lineage: 98673303-8109-4fe8-8d9f-eb867fb28bdd
This commit is contained in:
gsxdsm
2026-06-11 16:15:07 -07:00
parent 3e69d3990d
commit 39b2478f01
7 changed files with 311 additions and 9 deletions

View File

@@ -1062,7 +1062,7 @@ Mesh configuration and post-provision managed-node operations are registered sep
### Run Audit API
The run-audit system records every mutation performed by the engine across four domains:
- **Database** — task:create, task:update, task:move, etc. Node handoff/recovery emits structured events: `node:handoff:parked` (handoff denied/parked), `node:handoff:reassign-local` (local takeover approved), `node:handoff:reassign-any` (any-healthy takeover approved), and `node:lease:recovered` (abandoned lease cleared and task requeued).
- **Git** — worktree:create, commit:create, merge:resolve, merge:audit-failure, `worktree:reanchored`, and worktrunk lifecycle events (`worktree:worktrunk-install|create|sync|prune|remove`, plus `worktree:worktrunk-fallback`, `worktree:worktrunk-failure`, and `worktree:worktrunk-fallback-native`). Worktrunk events share metadata `{ op, binaryPath?, worktreePath?, durationMs?, exitCode?, stderrPreview?, installSource?, prunedCount? }` with `installSource` (`"release-binary" | "cargo"`) limited to successful `worktree:worktrunk-install` events and `prunedCount` limited to successful prune events when known. `worktree:worktrunk-install` is emitted only for true install actions; cache hits, configured `worktrunk.binaryPath` overrides, and `$PATH` resolutions intentionally remain silent. Dirty post-merge audit outcomes emit `merge:audit-failure` with metadata `{ mode, strategy, action, reason, issueCount, duplicateSubjectCount, touchedFileOverlapCount, verificationPassed, auditTargetLabel }`. FN-5279 adds `merge:reuse-handoff-acquired`, `merge:reuse-handoff-refused`, `merge:reuse-handoff-released`, and `merge:reuse-handoff-deferred-to-worktrunk` for task-worktree auto-merge handoff visibility. FN-5351 adds `merge:integration-worktree-state` (pre-handoff checkout/dirty snapshot for resolved integration branch), `merge:cwd-integration-fallback-refused` (terminal refusal park event), and `merge:integration-ref-advance` (integration ref advance outcome telemetry).
- **Git** — worktree:create, worktree:remove, `worktree:remove-fallback` (metadata `{ fallback: "filesystem-non-empty", error }` when native git removal falls back to filesystem removal + admin prune), commit:create, merge:resolve, merge:audit-failure, `worktree:reanchored`, and worktrunk lifecycle events (`worktree:worktrunk-install|create|sync|prune|remove`, plus `worktree:worktrunk-fallback`, `worktree:worktrunk-failure`, and `worktree:worktrunk-fallback-native`). Worktrunk events share metadata `{ op, binaryPath?, worktreePath?, durationMs?, exitCode?, stderrPreview?, installSource?, prunedCount? }` with `installSource` (`"release-binary" | "cargo"`) limited to successful `worktree:worktrunk-install` events and `prunedCount` limited to successful prune events when known. `worktree:worktrunk-install` is emitted only for true install actions; cache hits, configured `worktrunk.binaryPath` overrides, and `$PATH` resolutions intentionally remain silent. Dirty post-merge audit outcomes emit `merge:audit-failure` with metadata `{ mode, strategy, action, reason, issueCount, duplicateSubjectCount, touchedFileOverlapCount, verificationPassed, auditTargetLabel }`. FN-5279 adds `merge:reuse-handoff-acquired`, `merge:reuse-handoff-refused`, `merge:reuse-handoff-released`, and `merge:reuse-handoff-deferred-to-worktrunk` for task-worktree auto-merge handoff visibility. FN-5351 adds `merge:integration-worktree-state` (pre-handoff checkout/dirty snapshot for resolved integration branch), `merge:cwd-integration-fallback-refused` (terminal refusal park event), and `merge:integration-ref-advance` (integration ref advance outcome telemetry).
- **Git / `merge:file-scope-violation`** — emitted by the merger when `FileScopeViolationError` aborts a squash. `target` is the task ID; metadata includes `stagedFiles`, `declaredScope`, `resetLabel`, `stagedFileCount`, and `declaredScopeCount`. Consumed by `fileScopeInvariantFailuresPerDay` in `GET /api/health/reliability` (FN-4360).
- **Git / `merge:no-op-attribution-mismatch`** — emitted by the rebase landed-files attribution guard (FN-5304) when `<rebaseBaseSha>..HEAD` has zero attributable own commits but the source `fusion/<id>` tip still carries attributable own commits. `target` is the task ID; metadata includes `recordedSha`, `rebaseMergeBaseSha`, `sourceBranchRef`, `sourceBranchOwnCommitCount`, and `sourceBranchOwnCommitShas`.
- **Git / `merge:no-op-attribution-mismatch-skipped`** — emitted when the FN-5304 source-tip guard cannot run because the source branch ref is unavailable (for example already pruned). `target` is the task ID; metadata includes `reason` (`"source-ref-unavailable"`).
@@ -1593,7 +1593,7 @@ The GitHub tracking state listener now attaches to every registered project stor
- Worktrunk layout is authoritative on create: after `wt switch --create`, Fusion resolves the actual registered worktree path via `git worktree list --porcelain` and uses that path (instead of assuming `resolveTaskWorktreePath` alignment).
- Delegated operation surface in the interface: `create`, `sync`, `prune`, `remove` (plus backend path resolution via `resolveWorktreePath`).
- Executor acquisition paths (`worktree-acquisition.ts`) resolve backend selection centrally, so create flow stays backend-agnostic above the pool/acquisition layer.
- Worktree removal is backend-mediated across merger, self-healing, worktree-pool, executor, and step-session cleanup paths via `removeWorktree(...)` (`WorktreeBackend.remove()`).
- Worktree removal is backend-mediated across merger, self-healing, worktree-pool, executor, and step-session cleanup paths via `removeWorktree(...)` (`WorktreeBackend.remove()`). Native removal first runs `git worktree remove --force`; when git reports recoverable on-disk cleanup failures such as `Directory not empty`, `failed to delete`, or modified/untracked content, it falls back to async filesystem removal and `git worktree prune` (`pruneWorktreeAdminEntries`) so both the directory and dangling admin entry are cleared.
- Self-healing is worktrunk-aware for failure recovery: tasks paused with `pausedReason: "worktrunk_operation_failed"` are explicitly skipped in reclaim sweeps (`self-healing.ts`) until operator intervention.
- Failure contract: delegated worktrunk errors preserve stderr context (`WorktrunkOperationError`) and are handled by `worktrunk.onFailure` — `"fail"` pauses the task, while `"fallback-native"` retries on the native backend and emits one-shot fallback telemetry.
- Install contract: Fusion only auto-installs from a source-of-truth manifest. The shipped placeholder manifest intentionally stays in `upstream-pending-verification` until a human verifies upstream asset URLs and checksums, so install attempts fail closed rather than guessing release metadata.

View File

@@ -0,0 +1,139 @@
import { access, chmod, mkdtemp, mkdir, realpath, rm, writeFile } from "node:fs/promises";
import { constants } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { NativeWorktreeBackend, RemovalReason, removeWorktree } from "../../worktree-backend.js";
import { git, hasGit } from "./_helpers.js";
async function pathExists(path: string): Promise<boolean> {
try {
await access(path, constants.F_OK);
return true;
} catch {
return false;
}
}
describe.skipIf(!hasGit)("reliability interactions: worktree remove non-empty recovery", () => {
const roots: string[] = [];
let originalPath: string | undefined;
let originalFailPath: string | undefined;
afterEach(async () => {
if (originalPath === undefined) {
delete process.env.PATH;
} else {
process.env.PATH = originalPath;
}
if (originalFailPath === undefined) {
delete process.env.FUSION_FAIL_GIT_WORKTREE_REMOVE_PATH;
} else {
process.env.FUSION_FAIL_GIT_WORKTREE_REMOVE_PATH = originalFailPath;
}
await Promise.all(roots.map((root) => rm(root, { recursive: true, force: true })));
roots.length = 0;
});
async function setupRepo(prefix = "fusion-remove-non-empty-") {
const root = await mkdtemp(join(tmpdir(), prefix));
roots.push(root);
git(root, "git init -b main");
git(root, 'git config user.email "test@example.com"');
git(root, 'git config user.name "Test User"');
await writeFile(join(root, "README.md"), "# repo\n", "utf-8");
git(root, "git add README.md");
git(root, 'git commit -m "init"');
return root;
}
async function createWorktree(root: string, name: string, branch: string): Promise<string> {
const worktreePath = join(root, ".worktrees", name);
git(root, `git worktree add -b ${JSON.stringify(branch)} ${JSON.stringify(worktreePath)}`);
return worktreePath;
}
async function installGitRemoveFailureShim(
targetPath: string,
stderr = "error: failed to delete '$4': Directory not empty",
): Promise<void> {
const realGit = git(process.cwd(), "command -v git");
const shimDir = await mkdtemp(join(tmpdir(), "fusion-fake-git-"));
roots.push(shimDir);
const shimPath = join(shimDir, "git");
await writeFile(
shimPath,
`#!/bin/sh\nif [ "$1" = "worktree" ] && [ "$2" = "remove" ] && [ "$3" = "--force" ] && [ "$4" = "$FUSION_FAIL_GIT_WORKTREE_REMOVE_PATH" ]; then\n echo ${JSON.stringify(stderr)} >&2\n exit 1\nfi\nexec ${JSON.stringify(realGit)} "$@"\n`,
"utf-8",
);
await chmod(shimPath, 0o755);
originalPath = process.env.PATH;
originalFailPath = process.env.FUSION_FAIL_GIT_WORKTREE_REMOVE_PATH;
process.env.PATH = `${shimDir}${process.env.PATH ? `:${process.env.PATH}` : ""}`;
process.env.FUSION_FAIL_GIT_WORKTREE_REMOVE_PATH = targetPath;
}
async function expectWorktreeRemoved(root: string, worktreePath: string): Promise<void> {
expect(await pathExists(worktreePath)).toBe(false);
const porcelain = git(root, "git worktree list --porcelain");
expect(porcelain).not.toContain(`worktree ${worktreePath}`);
expect(porcelain).not.toContain(`worktree ${await realpath(dirname(worktreePath)).catch(() => dirname(worktreePath))}/${worktreePath.split("/").pop()}`);
}
it("removes and prunes a worktree with untracked-only content when git remove reports Directory not empty", async () => {
const root = await setupRepo();
const worktreePath = await createWorktree(root, "fn-untracked", "fusion/fn-untracked");
const resolvedWorktreePath = await realpath(worktreePath);
await mkdir(join(worktreePath, "dist"), { recursive: true });
await writeFile(join(worktreePath, "dist", "artifact.txt"), "artifact\n", "utf-8");
await installGitRemoveFailureShim(worktreePath);
const events: string[] = [];
await removeWorktree({
rootDir: root,
worktreePath,
settings: {},
reason: RemovalReason.ExecutorDispose,
force: true,
audit: { git: async (event) => void events.push(event.type) },
});
await expectWorktreeRemoved(root, resolvedWorktreePath);
expect(events).toContain("worktree:remove-fallback");
expect(events).toContain("worktree:admin-entry-pruned");
expect(events).toContain("worktree:remove");
});
it("removes and prunes a worktree with nested-git content when native removal falls back", async () => {
const root = await setupRepo();
const worktreePath = await createWorktree(root, "fn-nested", "fusion/fn-nested");
const resolvedWorktreePath = await realpath(worktreePath);
const nestedRepo = join(worktreePath, "node_modules", "inner-repo");
await mkdir(nestedRepo, { recursive: true });
git(nestedRepo, "git init -b main");
await writeFile(join(nestedRepo, "package.json"), "{}\n", "utf-8");
await installGitRemoveFailureShim(worktreePath);
await new NativeWorktreeBackend().remove({ rootDir: root, worktreePath });
await expectWorktreeRemoved(root, resolvedWorktreePath);
});
it("preserves native already-missing validation-failed behavior", async () => {
const root = await setupRepo();
const worktreePath = await createWorktree(root, "fn-missing", "fusion/fn-missing");
await rm(worktreePath, { recursive: true, force: true });
await installGitRemoveFailureShim(worktreePath, "fatal: validation failed, cannot remove working tree");
await expect(new NativeWorktreeBackend().remove({ rootDir: root, worktreePath })).rejects.toThrow(/validation failed/i);
});
it("still rethrows non-recoverable native removal failures", async () => {
const root = await setupRepo();
const notAWorktreePath = join(root, "not-a-worktree");
await mkdir(notAWorktreePath);
await expect(new NativeWorktreeBackend().remove({ rootDir: root, worktreePath: notAWorktreePath })).rejects.toThrow();
expect(await pathExists(notAWorktreePath)).toBe(true);
});
});

View File

@@ -10,7 +10,19 @@ import {
} from "../worktree-backend.js";
import { activeSessionRegistry } from "../active-session-registry.js";
const { execMock, accessMock, rmMock, existsSyncMock, parseIndexLockPathMock, classifyStaleLockMock, tryRemoveStaleLockMock, parseStaleRegistrationPathMock, recoverStaleRegistrationMock, installGuardMock } = vi.hoisted(() => {
const {
execMock,
accessMock,
rmMock,
existsSyncMock,
parseIndexLockPathMock,
classifyStaleLockMock,
tryRemoveStaleLockMock,
parseStaleRegistrationPathMock,
recoverStaleRegistrationMock,
installGuardMock,
pruneWorktreeAdminEntriesMock,
} = vi.hoisted(() => {
const mock = vi.fn();
(mock as any)[Symbol.for("nodejs.util.promisify.custom")] = mock;
return {
@@ -24,6 +36,7 @@ const { execMock, accessMock, rmMock, existsSyncMock, parseIndexLockPathMock, cl
parseStaleRegistrationPathMock: vi.fn(),
recoverStaleRegistrationMock: vi.fn(),
installGuardMock: vi.fn(),
pruneWorktreeAdminEntriesMock: vi.fn(),
};
});
@@ -58,6 +71,9 @@ vi.mock("../worktree-stale-registration.js", () => ({
parseStaleRegistrationPath: parseStaleRegistrationPathMock,
recoverStaleRegistration: recoverStaleRegistrationMock,
}));
vi.mock("../worktree-prune.js", () => ({
pruneWorktreeAdminEntries: pruneWorktreeAdminEntriesMock,
}));
beforeEach(() => {
execMock.mockReset();
@@ -72,6 +88,8 @@ beforeEach(() => {
tryRemoveStaleLockMock.mockReset();
installGuardMock.mockReset();
installGuardMock.mockResolvedValue(undefined);
pruneWorktreeAdminEntriesMock.mockReset();
pruneWorktreeAdminEntriesMock.mockResolvedValue(undefined);
parseIndexLockPathMock.mockReturnValue(null);
parseStaleRegistrationPathMock.mockReset();
parseStaleRegistrationPathMock.mockReturnValue(null);
@@ -150,6 +168,99 @@ describe("NativeWorktreeBackend", () => {
'git worktree remove --force "/repo/.worktrees/fn-1"',
expect.objectContaining({ cwd: "/repo", timeout: 60000, maxBuffer: 10485760 }),
);
expect(rmMock).not.toHaveBeenCalled();
expect(pruneWorktreeAdminEntriesMock).not.toHaveBeenCalled();
});
it("falls back to filesystem removal and prunes admin entries when native remove leaves a non-empty directory", async () => {
const audit = { git: vi.fn().mockResolvedValue(undefined) };
execMock.mockRejectedValueOnce({
message: "Command failed: git worktree remove --force /repo/.worktrees/fn-1",
stderr: "error: failed to delete '/repo/.worktrees/fn-1': Directory not empty",
});
await new NativeWorktreeBackend({ audit }).remove({
rootDir: "/repo",
worktreePath: "/repo/.worktrees/fn-1",
});
expect(rmMock).toHaveBeenCalledWith("/repo/.worktrees/fn-1", { recursive: true, force: true });
expect(pruneWorktreeAdminEntriesMock).toHaveBeenCalledWith({
rootDir: "/repo",
auditor: audit,
reason: "remove-non-empty-fallback",
target: "/repo/.worktrees/fn-1",
logger: undefined,
});
expect(audit.git).toHaveBeenCalledWith({
type: "worktree:remove-fallback",
target: "/repo/.worktrees/fn-1",
metadata: expect.objectContaining({ fallback: "filesystem-non-empty", error: expect.stringContaining("Directory not empty") }),
});
});
it("falls back for modified or untracked file native remove failures", async () => {
execMock.mockRejectedValueOnce({
message: "fatal: '/repo/.worktrees/fn-1' contains modified or untracked files, use --force to delete it",
stderr: "",
});
await new NativeWorktreeBackend().remove({
rootDir: "/repo",
worktreePath: "/repo/.worktrees/fn-1",
});
expect(rmMock).toHaveBeenCalledWith("/repo/.worktrees/fn-1", { recursive: true, force: true });
expect(pruneWorktreeAdminEntriesMock).toHaveBeenCalledWith(
expect.objectContaining({ rootDir: "/repo", reason: "remove-non-empty-fallback", target: "/repo/.worktrees/fn-1" }),
);
});
it("falls back for failed-to-delete native remove failures without a directory-not-empty suffix", async () => {
execMock.mockRejectedValueOnce({
message: "Command failed: git worktree remove --force /repo/.worktrees/fn-1",
stderr: "error: failed to delete '/repo/.worktrees/fn-1'",
});
await new NativeWorktreeBackend().remove({
rootDir: "/repo",
worktreePath: "/repo/.worktrees/fn-1",
});
expect(rmMock).toHaveBeenCalledWith("/repo/.worktrees/fn-1", { recursive: true, force: true });
expect(pruneWorktreeAdminEntriesMock).toHaveBeenCalledWith(
expect.objectContaining({ rootDir: "/repo", reason: "remove-non-empty-fallback", target: "/repo/.worktrees/fn-1" }),
);
});
it("rethrows non-recoverable native remove failures without filesystem fallback", async () => {
const error = { message: "fatal: not a git repository", stderr: "fatal: not a git repository" };
execMock.mockRejectedValueOnce(error);
await expect(
new NativeWorktreeBackend().remove({
rootDir: "/repo",
worktreePath: "/repo/.worktrees/fn-1",
}),
).rejects.toBe(error);
expect(rmMock).not.toHaveBeenCalled();
expect(pruneWorktreeAdminEntriesMock).not.toHaveBeenCalled();
});
it("rethrows filesystem removal failure after recoverable native remove failure", async () => {
const rmError = new Error("EACCES: permission denied");
execMock.mockRejectedValueOnce({ stderr: "error: failed to delete '/repo/.worktrees/fn-1': Directory not empty" });
rmMock.mockRejectedValueOnce(rmError as never);
await expect(
new NativeWorktreeBackend().remove({
rootDir: "/repo",
worktreePath: "/repo/.worktrees/fn-1",
}),
).rejects.toBe(rmError);
expect(pruneWorktreeAdminEntriesMock).not.toHaveBeenCalled();
});
it("syncs by fetching then rebasing", async () => {

View File

@@ -91,6 +91,7 @@ export interface EngineRunContext {
export type GitMutationType =
| "worktree:create"
| "worktree:remove"
| "worktree:remove-fallback"
| "worktree:reuse"
| "worktree:incomplete-detected"
| "worktree:reanchored"

View File

@@ -137,6 +137,22 @@ function getErrorExitCode(error: unknown): number | null {
return null;
}
function getErrorMessageWithStderr(error: unknown): string {
const message =
error instanceof Error
? error.message
: error && typeof error === "object" && "message" in error
? String((error as { message?: unknown }).message)
: String(error);
const stderr = getErrorStderr(error);
return stderr ? `${message}\n${stderr}` : message;
}
function isRecoverableNativeWorktreeRemoveError(error: unknown): boolean {
const message = getErrorMessageWithStderr(error);
return /Directory not empty/i.test(message) || /failed to delete/i.test(message) || /contains modified or untracked files/i.test(message);
}
function findStringByKey(value: unknown, key: string): string | null {
if (!value || typeof value !== "object") return null;
if (Array.isArray(value)) {
@@ -396,12 +412,41 @@ export class NativeWorktreeBackend implements WorktreeBackend {
}
async remove(input: WorktreeRemoveInput): Promise<void> {
await execAsync(`git worktree remove --force ${quoteShellArg(input.worktreePath)}`, {
cwd: input.rootDir,
encoding: "utf-8",
timeout: REMOVE_TIMEOUT_MS,
maxBuffer: MAX_BUFFER,
});
try {
await execAsync(`git worktree remove --force ${quoteShellArg(input.worktreePath)}`, {
cwd: input.rootDir,
encoding: "utf-8",
timeout: REMOVE_TIMEOUT_MS,
maxBuffer: MAX_BUFFER,
});
return;
} catch (error) {
if (!isRecoverableNativeWorktreeRemoveError(error)) {
throw error;
}
const errorMessage = getErrorMessageWithStderr(error);
this.deps.logger?.warn?.(
`[worktree-backend] git worktree remove failed for ${input.worktreePath}: ${errorMessage} — falling back to filesystem removal`,
);
await this.deps.audit?.git({
type: "worktree:remove-fallback",
target: input.worktreePath,
metadata: {
fallback: "filesystem-non-empty",
error: errorMessage,
},
});
await rm(input.worktreePath, { recursive: true, force: true });
await pruneWorktreeAdminEntries({
rootDir: input.rootDir,
auditor: this.deps.audit,
reason: "remove-non-empty-fallback",
target: input.worktreePath,
logger: this.deps.logger,
});
}
}
async sync(input: WorktreeSyncInput): Promise<{ skipped: boolean }> {

View File

@@ -108,6 +108,7 @@ export default defineConfig({
"src/__tests__/merger-file-scope-invariant.test.ts",
"src/__tests__/project-engine-manager.test.ts",
"src/__tests__/self-healing-already-merged.real-git.test.ts",
"src/__tests__/merger-ai-cleanup.test.ts",
],
},
},

View File

@@ -15,6 +15,11 @@
"file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts",
"reason": "Flake observed during FN-6226 verification: full `pnpm --filter @fusion/engine test` expected two run-audit events but saw four after unrelated real-git/self-healing cleanup activity. The failure is outside fast-mode workflow changes and indicates suite-order/temp-state sensitivity.",
"quarantinedAt": "2026-06-10"
},
{
"file": "packages/engine/src/__tests__/merger-ai-cleanup.test.ts",
"reason": "Flake observed during FN-6206 verification: `pruneExistingAiMergeWorktrees skips active-session paths` failed in full `pnpm --filter @fusion/engine test` runs while the file passed standalone, indicating suite-order/concurrency sensitivity. Follow-up FN-6207.",
"quarantinedAt": "2026-06-10"
}
]
}