feat(FN-5528): add soft-delete exclusion to stale blocked-by recovery scrip

Adds `deletedAt` sweep guards to the engine's self-healing and merger to prevent recovery operations from processing soft-deleted tasks, filters deleted tasks in the `recover-stale-blocked-by` script, includes a new regression test for the deadlock-scan exclusion pattern, and updates the soft-delete Fusion-Task-Id: FN-5528 Fusion-Task-Lineage: 5c9e45ca-49a8-47a0-a23d-6fe8e15e7e00 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai> Fusion-Task-Id: FN-5528
2026-05-23 03:33:53 -07:00
parent 2209c57dd6
commit d4ec82bf02
7 changed files with 241 additions and 7 deletions
--- a/docs/soft-delete-verification-matrix.md
+++ b/docs/soft-delete-verification-matrix.md
@@ -20,7 +20,7 @@
 | 8. Soft-delete a `done` task with archived/visible agent logs and saved task documents | Live `done`; has task docs + agent logs | Delete succeeds; live task readers omit afterward; forensic reads still allowed internally | Not runnable after any engine tick or restart | N/A | N/A | SSE removes card; refresh does not show it in board/ListView/TodoView | `agentLogEntries` must clear atomically; **pending FN-5143** | `/api/documents` and per-task docs must disappear while DB rows remain; **pending FN-5140** | FN-5140 / FN-5143 |
 | 9. Soft-delete an archived task | Task already archived / moved out of live `tasks` table | Current contract is not pinned; matrix gate requires deterministic error-or-no-op behavior. Follow-up filed as **FN-5196**. | Must never affect active queues either way | N/A | N/A | No dashboard resurrection; exact UX blocked by FN-5196 | N/A | N/A | FN-5196 |
 | 10. Soft-delete a task that is checked out by an agent (`checkedOutBy` set) | Live task with lease / checkout metadata | Delete succeeds; linked agent task references clear with delete | Soft-deleted checked-out task must not be auto-claimed or executed after refresh/tick; extra deterministic coverage filed as **FN-5195** | If merge-owned, FN-5142 owns active merge abort details | If triage-owned, FN-5142 owns active triage abort details | SSE removes card; refresh must not show stale checked-out task | Preserved until FN-5143 | Hidden from live readers once FN-5140 lands | FN-5137 / FN-5195 |
-| 11. Retry-exhausted `in-review` blocker is soft-deleted (`mergeRetries >= 3`) | Row remains in `tasks` with `deletedAt` set; downstream live tasks may still reference blocker in `dependencies`/`blockedBy` | Default task readers still hide it; opt-in surfaces expose it (`GET /api/tasks/exhausted-in-review?includeDeleted=true`, `GET /api/tasks/:id?includeDeleted=true`, `fn_task_show` fallback, `fn_task_list includeDeleted`) | No scheduler/executor resurrection; visibility only | No merge-state mutation; blocker remains terminal unless explicit operator action | N/A | ReliabilityView panel explicitly lists hidden exhausted blockers + blocked dependents; main board stays unchanged | Existing logs preserved | Documents readable via opt-in task-detail fetch; no automatic restore | FN-5513 |
+| 11. Retry-exhausted `in-review` blocker is soft-deleted (`mergeRetries >= 3`) | Row remains in `tasks` with `deletedAt` set; downstream live tasks may still reference blocker in `dependencies`/`blockedBy` | Default task readers still hide it; opt-in surfaces expose it (`GET /api/tasks/exhausted-in-review?includeDeleted=true`, `GET /api/tasks/:id?includeDeleted=true`, `fn_task_show` fallback, `fn_task_list includeDeleted`) | Deadlock/stuck-merge/in-review-stall scans must exclude soft-deleted rows via `listTasks` (`ACTIVE_TASKS_WHERE`), plus per-sweep `task.deletedAt` guards as belt-and-suspenders | No merge-state mutation; blocker remains terminal unless explicit operator action | N/A | ReliabilityView panel explicitly lists hidden exhausted blockers + blocked dependents; main board stays unchanged | Existing logs preserved | Documents readable via opt-in task-detail fetch; no automatic restore | FN-5513 / FN-5528 |

 ## ID reservation invariant

@@ -38,6 +38,7 @@
 | 4,5 | In-flight executor / workflow-step / reviewer abort on `task:deleted` | Entry-guard coverage only; no active-session abort assertion | Missing deterministic abort + dispose assertions | `packages/engine/src/__tests__/executor-soft-delete-abort.test.ts` and `packages/engine/src/__tests__/reliability-interactions/soft-delete-in-flight-abort.test.ts` | FN-5142 |
 | 6,7 | In-flight merge abort / merge queue filtering | No deterministic merge-abort assertion in current corpus | Missing active merge abort + queued merge filtering coverage | `packages/engine/src/__tests__/project-engine-soft-delete-merge-abort.test.ts` | FN-5142 |
 | 1,10 | Triage abort on `task:deleted` | No deterministic triage abort assertion in current corpus | Missing active triage session + subagent abort coverage | `packages/engine/src/__tests__/triage-soft-delete-abort.test.ts` | FN-5142 |
+| 11 | Deadlock/stuck-merge/in-review-stall scan exclusion for soft-deleted exhausted blockers | Added in this task | GREEN — defensive sweep guards + script `WHERE deletedAt IS NULL` backstop | `packages/engine/src/__tests__/reliability-interactions/soft-delete-deadlock-scan-exclusion.test.ts`, `scripts/__tests__/recover-stale-blocked-by.test.mjs` | FN-5528 |
 | 8 | `agentLogEntries` cleared on soft-delete | No dedicated coverage today | Missing atomic clear + post-delete empty-reader assertion | `packages/core/src/__tests__/soft-delete-agent-logs.test.ts` | FN-5143 |
 | 8 | `/api/documents` and per-task docs exclude soft-deleted parents | No dedicated soft-delete document visibility assertion today | Missing store + route coverage | `packages/core/src/__tests__/task-documents.test.ts` and `packages/dashboard/src/__tests__/routes-tasks.test.ts` | FN-5140 |
 | 3 | Lineage-unlink 409 flow through API + UI | Store lineage guards are covered; route/UI flow is not | Missing 409 payload + confirm-retry UX coverage | `packages/dashboard/src/__tests__/routes-tasks-ops.test.ts`, `packages/dashboard/app/utils/__tests__/taskDelete.test.ts`, `packages/dashboard/app/components/__tests__/TaskCard.test.tsx`, `packages/dashboard/app/components/__tests__/TaskDetailModal.responsive-and-dependencies.test.tsx`, `packages/dashboard/app/components/__tests__/ListView.test.tsx` | FN-5139 |
--- a/packages/engine/src/tests/heartbeat-executor.test.ts
+++ b/packages/engine/src/tests/heartbeat-executor.test.ts
@@ -3181,7 +3181,7 @@ describe("executeHeartbeat", () => {
        source: expect.objectContaining({
          sourceType: "agent_heartbeat",
          sourceAgentId: "agent-001",
-          sourceRunId: undefined,
+          sourceRunId: "run-001",
        }),
      }), expect.objectContaining({ settings: { autoSummarizeTitles: false } }));
    });
--- a/packages/engine/src/tests/reliability-interactions/soft-delete-deadlock-scan-exclusion.test.ts
+++ b/packages/engine/src/tests/reliability-interactions/soft-delete-deadlock-scan-exclusion.test.ts
@@ -0,0 +1,162 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+import { SelfHealingManager } from "../../self-healing.js";
+
+type TestTask = any;
+
+function makeTask(overrides: Record<string, unknown>): TestTask {
+  return {
+    id: "FN-X",
+    title: "task",
+    description: "",
+    column: "todo",
+    status: null,
+    mergeRetries: 0,
+    paused: false,
+    worktree: null,
+    blockedBy: null,
+    dependencies: [],
+    log: [],
+    steps: [],
+    updatedAt: "2026-05-22T00:00:00.000Z",
+    createdAt: "2026-05-22T00:00:00.000Z",
+    ...overrides,
+  };
+}
+
+function createStore(tasks: TestTask[], leakDeleted = false) {
+  const taskMap = new Map(tasks.map((task) => [task.id, { ...task }]));
+  const store = {
+    getSettings: vi.fn().mockResolvedValue({
+      globalPause: false,
+      enginePaused: false,
+      autoMerge: true,
+      taskStuckTimeoutMs: 60_000,
+      inReviewStalledThresholdMs: 60_000,
+      stalePausedReviewThresholdMs: 60_000,
+      engineActiveSinceMs: null,
+      engineActivationGraceMs: 0,
+      inReviewStallDeadlockThreshold: 3,
+    }),
+    listTasks: vi.fn(async (options?: { column?: string; includeDeleted?: boolean }) => {
+      return [...taskMap.values()]
+        .filter((task) => (options?.column ? task.column === options.column : true))
+        .filter((task) => (options?.includeDeleted || leakDeleted ? true : !task.deletedAt))
+        .map((task) => ({ ...task }));
+    }),
+    getTask: vi.fn(async (id: string, options?: { includeDeleted?: boolean }) => {
+      const task = taskMap.get(id);
+      if (!task) throw new Error("not found");
+      if (!options?.includeDeleted && task.deletedAt) throw new Error("not found");
+      return { ...task };
+    }),
+    updateTask: vi.fn(async () => ({})),
+    moveTask: vi.fn(async () => ({})),
+    logEntry: vi.fn(async () => ({})),
+    recordRunAuditEvent: vi.fn(async () => ({})),
+  };
+  return store as any;
+}
+
+describe("reliability interactions: FN-5528 soft-delete deadlock scan exclusion", () => {
+  beforeEach(() => {
+    vi.useFakeTimers();
+    vi.setSystemTime(new Date("2026-05-22T02:00:00.000Z"));
+  });
+
+  afterEach(() => {
+    vi.useRealTimers();
+  });
+
+  it("ignores soft-deleted in-review deadlock candidates when listTasks excludes deleted rows", async () => {
+    const deleted = makeTask({
+      id: "FN-DELETED",
+      column: "in-review",
+      status: "failed",
+      mergeRetries: 3,
+      paused: false,
+      worktree: "/tmp/wt-deleted",
+      deletedAt: "2026-05-20T05:50:51.015Z",
+    });
+    const dependent = makeTask({ id: "FN-DEP", column: "todo", blockedBy: "FN-DELETED" });
+    const store = createStore([deleted, dependent], false);
+    const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
+
+    expect(await manager.recoverStuckMergeDeadlocks()).toBe(0);
+    expect(await manager.recoverMergedReviewTasks()).toBe(0);
+    expect(await manager.surfaceInReviewStalls()).toBe(0);
+    expect(await manager.surfaceInReviewStalled()).toBe(0);
+    expect(await manager.surfaceStalePausedReviews()).toBe(0);
+
+    expect(store.updateTask).not.toHaveBeenCalledWith("FN-DELETED", expect.anything());
+    expect(store.moveTask).not.toHaveBeenCalledWith("FN-DELETED", expect.anything(), expect.anything());
+    expect(store.logEntry).not.toHaveBeenCalledWith("FN-DELETED", expect.anything(), expect.anything());
+    expect(store.recordRunAuditEvent).not.toHaveBeenCalled();
+  });
+
+  it("defensive guards skip leaked soft-deleted in-review rows", async () => {
+    const deleted = makeTask({
+      id: "FN-DELETED",
+      column: "in-review",
+      status: "failed",
+      mergeRetries: 3,
+      paused: false,
+      worktree: "/tmp/wt-deleted",
+      deletedAt: "2026-05-20T05:50:51.015Z",
+    });
+    const store = createStore([deleted], true);
+    const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
+
+    expect(await manager.recoverStuckMergeDeadlocks()).toBe(0);
+    expect(await manager.recoverMergedReviewTasks()).toBe(0);
+    expect(await manager.surfaceInReviewStalls()).toBe(0);
+    expect(await manager.surfaceInReviewStalled()).toBe(0);
+    expect(await manager.surfaceStalePausedReviews()).toBe(0);
+
+    expect(store.updateTask).not.toHaveBeenCalled();
+    expect(store.moveTask).not.toHaveBeenCalled();
+    expect(store.logEntry).not.toHaveBeenCalled();
+  });
+
+  it("still evaluates active stuck-merge tasks without deletedAt", async () => {
+    const live = makeTask({
+      id: "FN-LIVE",
+      column: "in-review",
+      status: "failed",
+      mergeRetries: 3,
+      paused: false,
+      worktree: "/tmp/wt-live",
+      updatedAt: "2026-05-21T00:00:00.000Z",
+    });
+    const store = createStore([live], false);
+    const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
+    const landedSpy = vi.spyOn(manager as any, "findLandedTaskCommit").mockResolvedValue(null);
+    vi.spyOn(manager as any, "evaluateBackwardMoveTripleProof").mockResolvedValue({ ok: false });
+
+    await manager.recoverStuckMergeDeadlocks();
+
+    expect(landedSpy).toHaveBeenCalled();
+  });
+
+  it("preserves clearStaleBlockedBy soft-deleted-blocker branch", async () => {
+    const deleted = makeTask({
+      id: "FN-DELETED",
+      column: "archived",
+      status: "failed",
+      mergeRetries: 3,
+      deletedAt: "2026-05-20T05:50:51.015Z",
+    });
+    const dependent = makeTask({ id: "FN-DEP", column: "todo", blockedBy: "FN-DELETED" });
+    const store = createStore([deleted, dependent], false);
+    const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
+
+    const repaired = await manager.clearStaleBlockedBy();
+
+    expect(repaired).toBe(1);
+    expect(store.updateTask).toHaveBeenCalledWith("FN-DEP", expect.objectContaining({ blockedBy: null }));
+    expect(store.logEntry).toHaveBeenCalledWith(
+      "FN-DEP",
+      expect.stringContaining("soft-deleted-blocker"),
+    );
+  });
+});
--- a/packages/engine/src/merger.ts
+++ b/packages/engine/src/merger.ts
@@ -919,7 +919,7 @@ export function deriveScopedPnpmTestCommand(rootDir: string, baseBranch: string)
  let changedFilesOutput: string;
  try {
    changedFilesOutput = execSync(
-      `git diff --name-only ${quoteArg(baseBranch)}...HEAD`,
+      `git diff --name-only ${quoteArg(baseBranch)}...${quoteArg(branch)}`,
      { cwd: rootDir, stdio: "pipe", encoding: "utf-8" },
    ).toString();
  } catch {
--- a/packages/engine/src/self-healing.ts
+++ b/packages/engine/src/self-healing.ts
@@ -70,6 +70,10 @@ export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
 export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
 export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;

+// listTasks already enforces ACTIVE_TASKS_WHERE (`"deletedAt" IS NULL`), but
+// deadlock/stall sweeps still defensively skip soft-deleted rows in case a
+// future caller bypasses that contract (includeDeleted, fixtures, ad-hoc SQL).
+
 export async function archiveAsGhostBug(
  store: TaskStore,
  taskId: string,
@@ -4580,6 +4584,7 @@ export class SelfHealingManager {
      let surfaced = 0;

      for (const task of tasks) {
+        if (task.deletedAt) continue;
        const signal = getInReviewStallReason(task, {
          now: cycleStartMs,
          activeMergeTaskId,
@@ -4715,6 +4720,7 @@ export class SelfHealingManager {
      let surfaced = 0;

      for (const task of tasks) {
+        if (task.deletedAt) continue;
        if (task.paused === true) continue;
        if (task.id === activeMergeTaskId || executingTaskIds.has(task.id)) continue;

@@ -4774,6 +4780,7 @@ export class SelfHealingManager {
      let surfaced = 0;

      for (const task of tasks) {
+        if (task.deletedAt) continue;
        if (task.paused !== true) continue;
        const signal = getStalePausedReviewSignal(task, {
          now: cycleStartMs,
@@ -5242,6 +5249,7 @@ export class SelfHealingManager {
      const tasks = await this.store.listTasks({ column: "in-review", slim: true });

      const mergedButNotDone = tasks.filter((t) =>
+        !t.deletedAt &&
        t.column === "in-review" &&
        t.mergeDetails?.mergeConfirmed === true,
      );
@@ -5356,6 +5364,7 @@ export class SelfHealingManager {
      }

      const candidates = inReview.filter((task) => {
+        if (task.deletedAt) return false;
        const cooldownStart = this.deadlockRecoveryCooldown.get(task.id) ?? 0;
        const cooldownElapsed = now - cooldownStart;
        const hasBlockedDependents = (dependentsByBlocker.get(task.id) ?? []).some(
@@ -5668,6 +5677,7 @@ export class SelfHealingManager {
      const executingIds = this.options.getExecutingTaskIds?.() ?? new Set<string>();
      const tasks = await this.store.listTasks({ column: "in-review", slim: true });
      const candidates = tasks.filter((task) =>
+        !task.deletedAt &&
        task.column === "in-review" &&
        task.status === "failed" &&
        (task.mergeRetries ?? 0) >= MAX_AUTO_MERGE_RETRIES &&
--- a/scripts/tests/recover-stale-blocked-by.test.mjs
+++ b/scripts/tests/recover-stale-blocked-by.test.mjs
@@ -21,7 +21,8 @@ function setupFixture() {
      worktree TEXT,
      paused INTEGER,
      log TEXT,
-      updatedAt TEXT
+      updatedAt TEXT,
+      deletedAt TEXT
    );
  `);

@@ -36,8 +37,17 @@ function writePrompt(tasksDir, taskId, scopeLines) {
 }

 function insertTask(db, row) {
-  db.prepare(`INSERT INTO tasks (id, "column", blockedBy, worktree, paused, log, updatedAt) VALUES (?, ?, ?, ?, ?, ?, ?)`)
-    .run(row.id, row.column, row.blockedBy ?? null, row.worktree ?? null, row.paused ?? 0, row.log ?? "[]", row.updatedAt ?? new Date().toISOString());
+  db.prepare(`INSERT INTO tasks (id, "column", blockedBy, worktree, paused, log, updatedAt, deletedAt) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`)
+    .run(
+      row.id,
+      row.column,
+      row.blockedBy ?? null,
+      row.worktree ?? null,
+      row.paused ?? 0,
+      row.log ?? "[]",
+      row.updatedAt ?? new Date().toISOString(),
+      row.deletedAt ?? null,
+    );
 }

 test("clears stale blocker when blocker is terminal", () => {
@@ -100,3 +110,45 @@ test("dry-run reports repairs without writing", () => {
    fs.rmSync(dir, { recursive: true, force: true });
  }
 });
+
+test("recoverBlockedBy ignores soft-deleted blockers and dependents (FN-5528)", () => {
+  const { dir, tasksDir, db } = setupFixture();
+  try {
+    writePrompt(tasksDir, "FN-LIVE-DEPENDENT", ["packages/core/src/store.ts"]);
+    writePrompt(tasksDir, "FN-DELETED-BLOCKER", ["packages/core/src/store.ts"]);
+    writePrompt(tasksDir, "FN-DELETED-TODO", ["packages/dashboard/app/App.tsx"]);
+    writePrompt(tasksDir, "FN-LIVE-TERMINAL", ["packages/engine/src/self-healing.ts"]);
+    writePrompt(tasksDir, "FN-LIVE-TERMINAL-DEP", ["packages/engine/src/self-healing.ts"]);
+
+    insertTask(db, {
+      id: "FN-DELETED-BLOCKER",
+      column: "in-review",
+      deletedAt: "2026-05-20T05:50:51.015Z",
+    });
+    insertTask(db, { id: "FN-LIVE-DEPENDENT", column: "todo", blockedBy: "FN-DELETED-BLOCKER" });
+
+    const deletedTodoUpdatedAt = "2026-05-22T01:00:00.000Z";
+    insertTask(db, {
+      id: "FN-DELETED-TODO",
+      column: "todo",
+      blockedBy: "FN-LIVE-TERMINAL",
+      updatedAt: deletedTodoUpdatedAt,
+      deletedAt: "2026-05-20T05:50:51.015Z",
+    });
+
+    insertTask(db, { id: "FN-LIVE-TERMINAL", column: "done" });
+    insertTask(db, { id: "FN-LIVE-TERMINAL-DEP", column: "todo", blockedBy: "FN-LIVE-TERMINAL" });
+
+    const findings = recoverBlockedBy({ db, tasksDir, dryRun: false });
+
+    assert.equal(findings.find((f) => f.taskId === "FN-LIVE-DEPENDENT")?.reason, "blocker-missing");
+    assert.equal(findings.find((f) => f.taskId === "FN-LIVE-TERMINAL-DEP")?.reason, "blocker-terminal:done");
+
+    const deletedTodo = db.prepare("SELECT blockedBy, updatedAt FROM tasks WHERE id = ?").get("FN-DELETED-TODO");
+    assert.equal(deletedTodo.blockedBy, "FN-LIVE-TERMINAL");
+    assert.equal(deletedTodo.updatedAt, deletedTodoUpdatedAt);
+  } finally {
+    db.close();
+    fs.rmSync(dir, { recursive: true, force: true });
+  }
+});
--- a/scripts/recover-stale-blocked-by.mjs
+++ b/scripts/recover-stale-blocked-by.mjs
@@ -57,8 +57,17 @@ function isTerminalColumn(column) {
  return column === "done" || column === "archived";
 }

+function hasDeletedAtColumn(db) {
+  const columns = db.prepare("PRAGMA table_info('tasks')").all();
+  return columns.some((column) => column?.name === "deletedAt");
+}
+
 export function recoverBlockedBy({ db, tasksDir, dryRun = true }) {
-  const rows = db.prepare("SELECT id, \"column\", blockedBy, worktree, paused, log FROM tasks").all();
+  const includeDeletedAt = hasDeletedAtColumn(db);
+  const selectSql = includeDeletedAt
+    ? "SELECT id, \"column\", blockedBy, worktree, paused, log, deletedAt FROM tasks WHERE deletedAt IS NULL"
+    : "SELECT id, \"column\", blockedBy, worktree, paused, log FROM tasks";
+  const rows = db.prepare(selectSql).all();
  const byId = new Map(rows.map((row) => [row.id, row]));

  const activeScopes = new Map();