root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(FN-5488): merge fusion/fn-5488

Browse Source
This commit is contained in:
gsxdsm
2026-05-22 10:37:21 -07:00
parent 301e0a050f
commit ba066c88a3
6 changed files with 187 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/architecture.md
View File

@@ -683,7 +683,7 @@ When stuck-kill retries are exhausted, `checkStuckBudget()` marks the task `stat
Guardrails: this routine does **not** retry merges, does **not** apply to mixed/non-orphan staging, and does **not** run when no landed-work proof exists (FN-4280 class protection).
- FN-4285 decision: add a follow-up for a tree-equality recovery strategy (`rev-parse <base>^{tree}` == `<task-branch>^{tree}`) in `findAlreadyMergedTaskCommit`. This closes stranded already-merged branches that evade trailer/ancestry/patch-id matching, with guardrails limited to retry-exhausted review tasks to avoid false positives during transient post-rebase parity windows.
- No-`fn_task_done` recovery classification is normalized across executor, restart recovery, and self-healing: detection keys on executor-emitted `"without calling fn_task_done"` strings (while still tolerating legacy `task_done` wording), then applies the bounded ladder deterministically (in-session retries → bounded todo requeues with preserved progress when appropriate → terminal surfaced failure when budget is exhausted).
- `clearStaleBlockedBy()` clears `blockedBy` (and transient `status`) on todo tasks when their blocker is missing, done, archived, paused in-review, or failed in-review with merge retries exhausted. FN-3924 extends this with a dependency-integrity guard: if a task has explicit dependencies and `blockedBy` is not one of the currently unresolved deps, the stale marker is cleared. FN-4091 broadens the sweep to active `in-progress` and un-paused `in-review` tasks as well, but those repairs only null `blockedBy` (they do not rewrite scheduler-owned queued state). This repairs rows corrupted by historical overlap re-stamping and lets scheduler re-evaluate from live dependency state. The ad-hoc `scripts/recover-stale-blocked-by.mjs` remains a manual backstop for filesystem/db audits, not the primary repair path.
- `clearStaleBlockedBy()` clears `blockedBy` (and transient `status`) on todo tasks when their blocker is missing, done, archived, paused in-review, or failed in-review with merge retries exhausted. FN-3924 extends this with a dependency-integrity guard: if a task has explicit dependencies and `blockedBy` is not one of the currently unresolved deps, the stale marker is cleared. FN-4091 broadens the sweep to active `in-progress` and un-paused `in-review` tasks as well, but those repairs only null `blockedBy` (they do not rewrite scheduler-owned queued state). FN-5488 adds two fast paths: (1) failed in-review blockers at/above `MAX_AUTO_MERGE_RETRIES` always fan out unblock recovery with explicit reason codes, and (2) `status="merging"|"merging-pr"` blockers with no active merger owner are treated as unbacked after a short grace window (`unbackedMergingFanoutGraceMs`, default 60s) so manual retry/unpause `updatedAt` refreshes cannot deadlock downstream todos indefinitely. Recovery logs now use `Auto-recovered (FN-5488): ... reason=<code>` for auditability while preserving FN-4538 overlap-blocking invariants.
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.

3
packages/dashboard/app/__tests__/index-html-theme-link.test.ts
View File

@@ -7,7 +7,8 @@ const PACKAGE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../..");
describe("index.html theme-data boot contract", () => {
const indexHtml = readFileSync(resolve(PACKAGE_ROOT, "app/index.html"), "utf8");
const script = indexHtml.match(/<script>[\s\S]*?<\/script>/)?.[0] ?? "";
const scripts = [...indexHtml.matchAll(/<script>[\s\S]*?<\/script>/g)].map((m) => m[0]);
const script = scripts.find((candidate) => candidate.includes("setAttribute('data-theme'")) ?? "";
it("includes a static theme-data stylesheet link", () => {
expect(indexHtml).toMatch(/<link\s+[^>]*(id=["']theme-data["'][^>]*href=["']\/theme-data\.css["']|href=["']\/theme-data\.css["'][^>]*id=["']theme-data["'])[^>]*>/i);

4
packages/dashboard/app/__tests__/pwa.test.ts
View File

@@ -54,13 +54,11 @@ describe("PWA configuration", () => {
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*viewport-fit=cover[^"]*"/i);
});
it("viewport meta disables pinch-zoom on mobile", () => {
it("viewport meta keeps mobile baseline + safe-area support", () => {
const indexHtml = readFileSync(resolve(__dirname, "../index.html"), "utf8");
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*width=device-width[^"]*"/i);
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*initial-scale=1\.0[^"]*"/i);
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*maximum-scale=1\.0[^"]*"/i);
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*user-scalable=no[^"]*"/i);
expect(indexHtml).toMatch(/<meta\s+name="viewport"[^>]*content="[^"]*viewport-fit=cover[^"]*"/i);
});

146
packages/engine/src/__tests__/in-review-merge-stall-deadlock-recovery.test.ts Normal file
View File

@@ -0,0 +1,146 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { Settings, Task, TaskStore } from "@fusion/core";
import { SelfHealingManager } from "../self-healing.js";
function makeTask(id: string, overrides: Partial<Task> = {}): Task {
return {
id,
title: id,
description: id,
column: "todo",
status: null,
paused: false,
blockedBy: null,
overlapBlockedBy: null,
dependencies: [],
steps: [],
log: [],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
...overrides,
} as Task;
}
describe("SelfHealingManager in-review merge stall deadlock recovery (FN-5488)", () => {
let tasks: Map<string, Task>;
let store: TaskStore;
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(new Date("2026-05-22T12:00:00.000Z"));
tasks = new Map();
store = {
getSettings: vi.fn().mockResolvedValue({
globalPause: false,
enginePaused: false,
} as Settings),
listTasks: vi.fn().mockImplementation(async (opts?: { column?: Task["column"]; includeArchived?: boolean }) => {
const all = [...tasks.values()];
if (!opts?.column) return all;
return all.filter((task) => task.column === opts.column);
}),
updateTask: vi.fn().mockImplementation(async (id: string, patch: Partial<Task>) => {
const current = tasks.get(id);
if (!current) throw new Error(`Task ${id} missing`);
tasks.set(id, { ...current, ...patch });
}),
logEntry: vi.fn().mockResolvedValue(undefined),
} as unknown as TaskStore;
});
afterEach(() => {
vi.useRealTimers();
});
it("clears blockedBy for failed in-review blocker with exhausted merge retries", async () => {
tasks.set("FN-5217", makeTask("FN-5217", {
column: "in-review",
status: "failed",
mergeRetries: 3,
updatedAt: "2026-05-20T00:00:00.000Z",
}));
tasks.set("FN-5119", makeTask("FN-5119", {
column: "todo",
blockedBy: "FN-5217",
status: "queued",
dependencies: ["FN-5217"],
}));
tasks.set("FN-5361", makeTask("FN-5361", {
column: "todo",
}));
tasks.set("FN-5449", makeTask("FN-5449", {
column: "todo",
blockedBy: "FN-5217",
status: "queued",
dependencies: ["FN-5361", "FN-5217"],
}));
const manager = new SelfHealingManager(store, { rootDir: "/tmp/test" });
const recovered = await manager.clearStaleBlockedBy();
expect(recovered).toBe(2);
expect(tasks.get("FN-5119")?.blockedBy).toBeNull();
expect(tasks.get("FN-5119")?.status).toBeNull();
expect(tasks.get("FN-5449")?.blockedBy).toBe("FN-5361");
expect(tasks.get("FN-5449")?.status).toBe("queued");
});
it("treats fresh unbacked in-review merging status as stale for fanout", async () => {
tasks.set("FN-5485", makeTask("FN-5485", {
column: "in-review",
status: "merging",
updatedAt: "2026-05-22T11:58:50.000Z",
}));
tasks.set("FN-5486", makeTask("FN-5486", {
column: "todo",
blockedBy: "FN-5485",
status: "queued",
dependencies: ["FN-5485"],
}));
const manager = new SelfHealingManager(store, {
rootDir: "/tmp/test",
staleMergingStatusMinAgeMs: 5 * 60_000,
staleMergingFanoutMinAgeMs: 15 * 60_000,
getActiveMergeTaskId: () => null,
});
const recovered = await manager.clearStaleBlockedBy();
expect(recovered).toBe(1);
expect(tasks.get("FN-5486")?.blockedBy).toBeNull();
expect(tasks.get("FN-5486")?.status).toBeNull();
});
it("preserves overlapBlockedBy when active overlap blocker exists", async () => {
tasks.set("FN-5485", makeTask("FN-5485", {
column: "in-review",
status: "merging",
updatedAt: "2026-05-22T11:58:50.000Z",
}));
tasks.set("FN-OV", makeTask("FN-OV", {
column: "in-progress",
}));
tasks.set("FN-DEP", makeTask("FN-DEP", {
column: "todo",
blockedBy: "FN-5485",
overlapBlockedBy: "FN-OV",
status: "queued",
dependencies: ["FN-5485"],
}));
const manager = new SelfHealingManager(store, {
rootDir: "/tmp/test",
staleMergingStatusMinAgeMs: 5 * 60_000,
staleMergingFanoutMinAgeMs: 15 * 60_000,
getActiveMergeTaskId: () => null,
});
await manager.clearStaleBlockedBy();
expect(tasks.get("FN-DEP")?.blockedBy).toBeNull();
expect(tasks.get("FN-DEP")?.status).toBe("queued");
expect(tasks.get("FN-DEP")?.overlapBlockedBy).toBe("FN-OV");
});
});

8
packages/engine/src/__tests__/self-healing.test.ts
View File

@@ -616,7 +616,7 @@ describe("SelfHealingManager", () => {
await manager.runStartupRecovery();
expect(store.updateTask).toHaveBeenCalledWith("A", { blockedBy: null, overlapBlockedBy: null, status: null });
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining("Auto-recovered: cleared stale blockedBy"));
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining("Auto-recovered (FN-5488): cleared stale blockedBy"));
});
it("runStartupRecovery skips while enginePaused is active", async () => {
@@ -5950,8 +5950,8 @@ describe("clearStaleBlockedBy", () => {
expect(recovered).toBe(1);
expect(store.updateTask).toHaveBeenCalledWith("A", { blockedBy: null, overlapBlockedBy: null, status: null });
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining(`blocker ${blockerId}`));
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining("stale for"));
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining(`blocker=${blockerId}`));
expect(store.logEntry).toHaveBeenCalledWith("A", expect.stringContaining("reason=unbacked-merging"));
manager.stop();
vi.useRealTimers();
});
@@ -5964,7 +5964,7 @@ describe("clearStaleBlockedBy", () => {
column: "in-review",
paused: false,
status: "merging",
updatedAt: "2026-01-01T00:00:01.000Z",
updatedAt: "2026-01-01T00:09:31.000Z",
});
mockSweepTasks(store, { todo: [taskA], inReview: [taskB], all: [taskA, taskB] });

38
packages/engine/src/self-healing.ts
View File

@@ -244,6 +244,11 @@ export interface SelfHealingOptions {
* blockedBy pointers. Must be >= staleMergingStatusMinAgeMs.
*/
staleMergingFanoutMinAgeMs?: number;
/**
* Grace window for treating in-review merging statuses as unbacked when no
* active merger owns the task. Intended for manual retry/unpause refreshes.
*/
unbackedMergingFanoutGraceMs?: number;
hasActiveAgentExecution?: (agentId: string) => boolean;
restartDurableAgentHeartbeat?: (agentId: string, context: { reason: string; attempt: number }) => Promise<boolean>;
autoRecoveryDispatcher?: AutoRecoveryDispatcher;
@@ -303,6 +308,7 @@ const MAX_STARVATION_DROPS = 3;
const DEADLOCK_RECOVERY_COOLDOWN_MS = 15 * 60_000;
const DEFAULT_STALE_MERGING_STATUS_MIN_AGE_MS = 5 * 60_000;
const DEFAULT_STALE_MERGING_FANOUT_MIN_AGE_MS = 15 * 60_000;
const DEFAULT_UNBACKED_MERGING_FANOUT_GRACE_MS = 60_000;
const DURABLE_ERROR_RECOVERY_MAX_RETRIES = 5;
const DURABLE_ERROR_RECOVERY_BASE_COOLDOWN_MS = 30_000;
const DURABLE_ERROR_RECOVERY_MAX_COOLDOWN_MS = 15 * 60_000;
@@ -3553,7 +3559,12 @@ export class SelfHealingManager {
const staleMergingStatusMinAgeMs = this.options.staleMergingStatusMinAgeMs ?? DEFAULT_STALE_MERGING_STATUS_MIN_AGE_MS;
const configuredFanoutMinAgeMs = this.options.staleMergingFanoutMinAgeMs ?? DEFAULT_STALE_MERGING_FANOUT_MIN_AGE_MS;
const staleMergingFanoutMinAgeMs = Math.max(staleMergingStatusMinAgeMs, configuredFanoutMinAgeMs);
const unbackedMergingFanoutGraceMs = Math.min(
staleMergingStatusMinAgeMs,
Math.max(1, this.options.unbackedMergingFanoutGraceMs ?? DEFAULT_UNBACKED_MERGING_FANOUT_GRACE_MS),
);
const activeMergeTaskId = this.options.getActiveMergeTaskId?.() ?? null;
const executingTaskIds = this.options.getExecutingTaskIds?.() ?? new Set<string>();
const now = Date.now();
const todoTasks = await this.store.listTasks({ column: "todo" });
@@ -3599,28 +3610,36 @@ export class SelfHealingManager {
const blocker = taskById.get(blockerId);
let reason: string | null = null;
let reasonCode: string | null = null;
if (!blocker) {
reasonCode = "missing-blocker";
reason = `blocker ${blockerId} missing`;
} else if (blocker.column === "done") {
reasonCode = "blocker-done";
reason = `blocker ${blockerId} is done`;
} else if (blocker.column === "archived") {
reasonCode = "blocker-archived";
reason = `blocker ${blockerId} is archived`;
} else if (blocker.column === "todo") {
reasonCode = "blocker-moved-todo";
reason = `blocker ${blockerId} moved to todo`;
} else if (blocker.column === "in-review" && blocker.paused) {
reasonCode = "in-review-paused";
reason = `blocker ${blockerId} in-review + paused`;
} else if (
blocker.column === "in-review" &&
blocker.status === "failed" &&
(blocker.mergeRetries ?? 0) >= MAX_AUTO_MERGE_RETRIES
) {
reasonCode = "failed-retry-exhausted";
reason = `blocker ${blockerId} in-review + failed (mergeRetries ${blocker.mergeRetries ?? 0}/${MAX_AUTO_MERGE_RETRIES})`;
} else if (
blocker.column === "in-review" &&
blocker.status === "failed" &&
isMissingWorktreeSessionStartFailure(blocker.error)
) {
reasonCode = "missing-worktree-session-start";
reason = `blocker ${blockerId} in-review + failed (missing-worktree session start)`;
} else if (
blocker.column === "in-review" &&
@@ -3630,12 +3649,21 @@ export class SelfHealingManager {
const updatedAtMs = blocker.updatedAt ? Date.parse(blocker.updatedAt) : Number.NaN;
if (Number.isFinite(updatedAtMs)) {
const elapsedMs = now - updatedAtMs;
if (elapsedMs >= staleMergingFanoutMinAgeMs) {
const blockerStatus = blocker.status ?? "no-status";
const blockerStatus = blocker.status ?? "no-status";
if (
(blocker.status === "merging" || blocker.status === "merging-pr") &&
!executingTaskIds.has(blocker.id) &&
elapsedMs >= unbackedMergingFanoutGraceMs
) {
reasonCode = "unbacked-merging";
reason = `blocker ${blockerId} in-review + ${blockerStatus} unbacked for ${elapsedMs}ms (grace ${unbackedMergingFanoutGraceMs}ms)`;
} else if (elapsedMs >= staleMergingFanoutMinAgeMs) {
reasonCode = "stale-merging-fanout";
reason = `blocker ${blockerId} in-review + ${blockerStatus} stale for ${elapsedMs}ms (threshold ${staleMergingFanoutMinAgeMs}ms)`;
}
}
} else if (task.dependencies.length > 0 && !unresolvedDeps.includes(blockerId)) {
reasonCode = "not-unresolved-dependency";
reason = `blocker ${blockerId} not among unresolved dependencies`;
}
@@ -3648,13 +3676,13 @@ export class SelfHealingManager {
continue;
}
await this.store.updateTask(task.id, { blockedBy: nextBlocker, status: "queued" });
await this.store.logEntry(task.id, `Auto-recovered: refreshed stale blockedBy — ${reason}; now blocked by ${nextBlocker}`);
await this.store.logEntry(task.id, `Auto-recovered (FN-5488): refreshed stale blockedBy — blocker=${blockerId} blockerStatus=${blocker?.status ?? "none"} reason=${reasonCode ?? "unspecified"}; ${reason}; now blocked by ${nextBlocker}`);
} else if (hasActiveOverlapBlocker) {
await this.store.updateTask(task.id, { blockedBy: null, status: "queued" });
await this.store.logEntry(task.id, `Auto-recovered: preserved queued status — still blocked by file scope overlap with ${task.overlapBlockedBy}`);
await this.store.logEntry(task.id, `Auto-recovered (FN-5488): preserved queued status — blocker=${blockerId} blockerStatus=${blocker?.status ?? "none"} reason=${reasonCode ?? "unspecified"}; still blocked by file scope overlap with ${task.overlapBlockedBy}`);
} else {
await this.store.updateTask(task.id, { blockedBy: null, overlapBlockedBy: null, status: null });
await this.store.logEntry(task.id, `Auto-recovered: cleared stale blockedBy — ${reason}`);
await this.store.logEntry(task.id, `Auto-recovered (FN-5488): cleared stale blockedBy — blocker=${blockerId} blockerStatus=${blocker?.status ?? "none"} reason=${reasonCode ?? "unspecified"}; ${reason}`);
}
} else {
await this.store.updateTask(task.id, { blockedBy: null });