FN-7968: defer slow cleanup off task deletion critical path

Make soft-delete return after the DB mutation while branch and agent cleanup run in the background.

- Schedule cleanupBranchForTask after the soft-delete transaction instead of awaiting it under withTaskLock
- Persist cleaned-branch log entries on the deleted row asynchronously; warn on deferred failures
- Respond from DELETE /tasks/:id after deleteTask and schedule execution-agent binding release off the HTTP path
- Add core and dashboard regression tests for non-blocking delete cleanup
- Document the fast-path contract in architecture.md and add a patch changeset

Files changed:
 .changeset/fn-7968-task-delete-latency.md          |   7 +
 docs/architecture.md                               |   1 +
 .../task-delete-nonblocking-cleanup.test.ts        | 160 +++++++++++++++++++++
 packages/core/src/task-store/archive-lifecycle.ts  |  57 +++++++-
 .../routes-task-delete-nonblocking.test.ts         | 139 ++++++++++++++++++
 .../src/routes/register-task-workflow-routes.ts    |  19 ++-
 6 files changed, 370 insertions(+), 13 deletions(-)

Fusion-Task-Id: FN-7968

Fusion-Task-Lineage: f218a91e-aee3-46c9-a80f-182751b3ccc4

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-07-15 12:44:30 -07:00
parent 836e53c6c0
commit 6e3a338cac
6 changed files with 370 additions and 13 deletions

View File

@@ -0,0 +1,7 @@
---
"@runfusion/fusion": patch
---
summary: Make task deletion return faster while cleanup continues in the background.
category: performance
dev: Defers branch cleanup and dashboard agent-binding release off the user-visible delete path.

View File

@@ -2227,6 +2227,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
- **Soft-delete in-flight abort (FN-5142)**: `task:deleted` must immediately abort/dispose active executor work (`activeSessions`, `activeStepExecutors`, `activeWorkflowStepSessions`, reviewer subagents), interrupt active merge state (`mergeAbortController`, `activeMergeSession`, `activeMergeTaskId`, `mergeActive`, `mergeQueue`, `pausedReviewTaskIds`), and abort triage specify/subagent sessions for that id. Handlers are per-task and idempotent.
- **Archive releases active-session locks (FN-7717)**: `task:moved` with `to === "archived"` (from ANY column, including in-progress via a direct single-hop `fn_task_archive`, and triage/planning where Plan Review and other workflow-step sessions run) now disposes in-flight session surfaces via `awaitAbortInFlightTaskWork` and sweeps any remaining `activeSessionRegistry` paths for the task, so a leaked lock can never survive archive and block a successor task's `registerPath` on the same session path. The `to === "archived"` check is ordered BEFORE the `from === "in-progress"` branch so a direct in-progress→archived move gets the same full cleanup instead of falling into the narrower in-progress-only branch. `to === "done"`/`"in-review"` are deliberately excluded — those columns legitimately hold `ai-merge`/`workspace-repo-land` merge leases.
- **Soft-delete audit + column reconcile (FN-5175)**: `TaskStore.deleteTask` records a `runAuditEvents` row (`mutationType: "task:deleted"`, `domain: "database"`) inside the same transaction that sets `deletedAt`, and sets `"column" = 'archived'` on the row. Callers without a heartbeat run context (`fn task delete`, pi extension, dashboard delete route) pass an `auditContext` with `agentId: "system"` and a synthetic `runId`. The watcher cross-instance emit path does NOT re-record the audit event. The row stays in `tasks` (not `archivedTasks`); `archiveTask` is unchanged.
- **Soft-delete fast path cleanup (FN-7968)**: user-visible `TaskStore.deleteTask` completion is bounded by the soft-delete transaction, audit row, cache/event emission, dependency/lineage gates, and near-duplicate cleanup. Potentially slow branch cleanup (`cleanupBranchForTask` git subprocesses) is scheduled after the row is already soft-deleted; it must still clear stale execution-start branch references and persist the cleaned-branch log entry on the deleted row. Dashboard `DELETE /tasks/:id` likewise responds after `deleteTask` and schedules execution-agent binding release off the HTTP critical path; the release remains observable through warning logs on failure.
- **Soft-delete resurrection guard (FN-5208)**: `TaskStore.readTaskJson()` must never fall back to `.fusion/tasks/<id>/task.json` when the DB row exists with `deletedAt` set — it throws `TaskDeletedError`. `atomicCreateTaskJson` / `atomicWriteTaskJson` / `atomicWriteTaskJsonWithAudit` refuse to upsert a task whose row is currently soft-deleted (unless the in-memory task carries `deletedAt` itself, for soft-delete maintenance paths), emit a `[soft-delete-resurrection-blocked]` log line, and record a `task:resurrection-blocked` run-audit event. Stale in-flight planner/triage writes for a soft-deleted ID surface `TaskDeletedError` and abort cleanly without emitting `task:created`.
- **Exhausted in-review visibility surfaces (FN-5513/FN-6569)**: retry-exhausted merge failures (`column='in-review'`, `status='failed'`, `mergeRetries >= maxAutoMergeRetries`, default `3`) can remain soft-deleted for lifecycle safety, but are now intentionally discoverable through opt-in read paths: `TaskStore.listExhaustedInReviewTasks({ includeDeleted })`, `GET /api/tasks/exhausted-in-review`, `GET /api/tasks/:id?includeDeleted=true`, CLI `fn_task_show` soft-delete fallback marker, CLI `fn_task_list({ includeDeleted: true })`, and the dashboard ReliabilityView "Exhausted in-review (hidden blockers)" panel. This complements FN-5488/FN-5496 downstream blocker healing by surfacing the upstream blocker without mutating lifecycle state.
- **Soft-delete stream verification gate (FN-5153)**: `docs/soft-delete-verification-matrix.md` is the authoritative checklist for the FN-5105 → FN-5143 soft-delete stream. Every scenario × layer cell must be GREEN (or have a linked follow-up FN) before the stream is closed; `packages/engine/src/__tests__/reliability-interactions/soft-delete-end-to-end.test.ts` is the cross-layer regression backstop.

View File

@@ -0,0 +1,160 @@
// @vitest-environment node
import { EventEmitter } from "node:events";
import { afterEach, describe, expect, it, vi } from "vitest";
import { deleteTaskImpl } from "../task-store/archive-lifecycle.js";
import type { Task } from "../types.js";
function deferred<T>() {
let resolve!: (value: T) => void;
const promise = new Promise<T>((innerResolve) => {
resolve = innerResolve;
});
return { promise, resolve };
}
function createTask(overrides: Partial<Task> & { id: string }): Task {
const now = "2026-07-15T09:00:00.000Z";
return {
id: overrides.id,
title: overrides.title ?? overrides.id,
description: overrides.description ?? overrides.id,
column: overrides.column ?? "todo",
dependencies: overrides.dependencies ?? [],
createdAt: overrides.createdAt ?? now,
updatedAt: overrides.updatedAt ?? now,
size: "M",
subtasks: [],
log: overrides.log ?? [],
tags: [],
blockedBy: [],
source: { sourceType: "api" },
...overrides,
} as Task;
}
function makeDeleteStore(input: {
task: Task;
dependentIds?: string[];
lineageChildIds?: string[];
cleanupBranchForTask?: (task: Task) => Promise<string[]>;
}) {
const events = new EventEmitter();
const tasks = new Map<string, Task>([[input.task.id, { ...input.task, log: [...(input.task.log ?? [])] }]]);
const auditEvents: Array<{ mutationType: string; taskId?: string }> = [];
const prepareRun = vi.fn((sql: string, args: unknown[]) => {
if (sql.includes("UPDATE tasks SET \"column\" = 'archived'")) {
const [deletedAt, allowResurrection, updatedAt, id] = args as [string, number, string, string];
const task = tasks.get(id)!;
task.column = "archived";
task.deletedAt = deletedAt;
task.allowResurrection = allowResurrection === 1;
task.updatedAt = updatedAt;
return;
}
if (sql.includes("UPDATE tasks SET log = ?")) {
const [logJson, updatedAt, id] = args as [string, string, string];
const task = tasks.get(id)!;
task.log = JSON.parse(logJson) as Task["log"];
task.updatedAt = updatedAt;
}
});
const store = {
backendMode: false,
agentLogBuffer: [],
isWatching: true,
taskCache: new Map<string, Task>([[input.task.id, input.task]]),
missionStore: undefined,
db: {
transaction: (fn: () => void) => fn(),
prepare: (sql: string) => ({
run: (...args: unknown[]) => prepareRun(sql, args),
}),
bumpLastModified: vi.fn(),
},
withTaskLock: vi.fn(async (_id: string, fn: () => Promise<Task>) => fn()),
flushAgentLogBuffer: vi.fn(),
readTaskFromDb: vi.fn((id: string) => tasks.get(id) ?? null),
findLiveDependents: vi.fn(() => input.dependentIds ?? []),
findLiveLineageChildren: vi.fn(async () => input.lineageChildIds ?? []),
cleanupBranchForTask: vi.fn(input.cleanupBranchForTask ?? (async () => [])),
rewriteDependentsForRemoval: vi.fn(() => []),
rewriteBlockedByResidueDependentsForRemoval: vi.fn(() => []),
rewriteLineageChildrenForRemoval: vi.fn(() => []),
recordRunAuditEvent: vi.fn(async (event: { mutationType: string; taskId?: string }) => {
auditEvents.push(event);
}),
makeSyntheticDeleteRunId: vi.fn((id: string) => `synthetic-delete-${id}`),
clearLinkedAgentTaskIds: vi.fn(),
clearNearDuplicateReferencesToFailSoft: vi.fn(async () => undefined),
emit: vi.fn((event: string, ...args: unknown[]) => events.emit(event, ...args)),
on: events.on.bind(events),
getStoredTask: (id: string) => tasks.get(id),
getAuditEvents: () => auditEvents,
prepareRun,
};
return store;
}
describe("deleteTask non-blocking cleanup", () => {
afterEach(() => {
vi.restoreAllMocks();
});
it("soft-deletes before delayed branch cleanup finishes and still records cleanup", async () => {
const task = createTask({ id: "FN-7968", branch: "fusion/fn-7968" });
const cleanup = deferred<string[]>();
const store = makeDeleteStore({
task,
cleanupBranchForTask: async () => cleanup.promise,
});
const deletedEvents: string[] = [];
store.on("task:deleted", (deleted: Task) => {
deletedEvents.push(deleted.id);
});
let resolved = false;
const deletePromise = deleteTaskImpl(store as never, task.id).then((deleted) => {
resolved = true;
return deleted;
});
await vi.waitFor(() => expect(resolved).toBe(true), { timeout: 100 });
const deleted = await deletePromise;
expect(deleted).toMatchObject({ id: task.id, column: "archived" });
expect(deleted.deletedAt).toEqual(expect.any(String));
expect(store.cleanupBranchForTask).toHaveBeenCalledWith(expect.objectContaining({ id: task.id }));
expect(store.getStoredTask(task.id)?.log).toEqual([]);
expect(deletedEvents).toEqual([task.id]);
expect(store.getAuditEvents().filter((event) => event.mutationType === "task:deleted")).toHaveLength(1);
cleanup.resolve(["fusion/fn-7968"]);
await vi.waitFor(() => {
expect(store.getStoredTask(task.id)?.log?.some((entry) => entry.action === "Cleaned up branch: fusion/fn-7968")).toBe(true);
});
});
it("keeps idempotent and gated deletes fast without scheduling branch cleanup", async () => {
const deletedTask = createTask({ id: "FN-DELETED", deletedAt: "2026-07-15T09:01:00.000Z", column: "archived" });
const deletedStore = makeDeleteStore({ task: deletedTask });
await expect(deleteTaskImpl(deletedStore as never, deletedTask.id)).resolves.toMatchObject({ id: deletedTask.id });
expect(deletedStore.cleanupBranchForTask).not.toHaveBeenCalled();
expect(deletedStore.getAuditEvents()).toHaveLength(0);
const dependentParent = createTask({ id: "FN-DEPENDENT-PARENT", branch: "fusion/dependent-parent" });
const dependentStore = makeDeleteStore({ task: dependentParent, dependentIds: ["FN-DEPENDENT-CHILD"] });
await expect(deleteTaskImpl(dependentStore as never, dependentParent.id)).rejects.toMatchObject({ name: "TaskHasDependentsError" });
expect(dependentStore.cleanupBranchForTask).not.toHaveBeenCalled();
expect(dependentStore.getAuditEvents()).toHaveLength(0);
const lineageParent = createTask({ id: "FN-LINEAGE-PARENT", branch: "fusion/lineage-parent" });
const lineageStore = makeDeleteStore({ task: lineageParent, lineageChildIds: ["FN-LINEAGE-CHILD"] });
await expect(deleteTaskImpl(lineageStore as never, lineageParent.id)).rejects.toMatchObject({ name: "TaskHasLineageChildrenError" });
expect(lineageStore.cleanupBranchForTask).not.toHaveBeenCalled();
expect(lineageStore.getAuditEvents()).toHaveLength(0);
});
});

View File

@@ -6,12 +6,49 @@
* behavior-preserving refactor. Each function receives the TaskStore
* instance as its first parameter and performs byte-identical work.
*/
import {TaskStore} from "../store.js";
import {TaskStore, storeLog} from "../store.js";
import {MissionStore} from "../mission-store.js";
import {TaskHasDependentsError, TaskHasLineageChildrenError, TaskSelfDeleteError} from "./errors.js";
import type {Task, Column, GithubIssueAction} from "../types.js";
import "../builtin-traits.js";
import {__setTaskActivityLogLimitsForTesting} from "../task-store/comments.js";
import {toJson} from "../db-helpers.js";
import {getErrorMessage} from "../error-message.js";
function scheduleDeleteBranchCleanup(store: TaskStore, task: Task): void {
/*
FNXC:TaskDeletion 2026-07-15-09:45:
Soft-delete latency must be bounded by the database mutation, audit, and event emission; branch cleanup can spawn serialized git subprocesses and must not hold withTaskLock or the returned deleteTask Promise. Schedule the cleanup after the task is already soft-deleted, but keep the existing cleanup guarantees by still clearing stale execution-start branch references and persisting the cleaned-branch log entry on the deleted row.
*/
void (async () => {
try {
const cleanedBranches = await store.cleanupBranchForTask(task);
if (cleanedBranches.length === 0) {
return;
}
const deletedTask = store.readTaskFromDb(task.id, { includeDeleted: true });
if (!deletedTask) {
return;
}
const updatedAt = new Date().toISOString();
const nextLog = [
...(deletedTask.log ?? []),
{
timestamp: updatedAt,
action: `Cleaned up branch: ${cleanedBranches.join(", ")}`,
},
];
store.db.prepare("UPDATE tasks SET log = ?, updatedAt = ? WHERE id = ?").run(toJson(nextLog), updatedAt, task.id);
store.db.bumpLastModified();
} catch (error) {
storeLog.warn("Deferred task-delete branch cleanup failed", {
taskId: task.id,
error: getErrorMessage(error),
});
}
})();
}
export async function deleteTaskImpl(store: TaskStore, id: string, options?: { removeDependencyReferences?: boolean; removeLineageReferences?: boolean; allowResurrection?: boolean; githubIssueAction?: GithubIssueAction; auditContext?: { agentId: string; runId: string; sessionId?: string; taskId?: string }; },): Promise<Task> {
// FNXC:RuntimeLifecycleAsync 2026-06-24-12:00:
@@ -59,24 +96,15 @@ export async function deleteTaskImpl(store: TaskStore, id: string, options?: { r
throw new TaskHasLineageChildrenError(id, lineageChildIds);
}
// Clean up the task's branch before deleting from DB
const cleanedBranches = await store.cleanupBranchForTask(task);
if (cleanedBranches.length > 0) {
if (!task.log) task.log = [];
task.log.push({
timestamp: new Date().toISOString(),
action: `Cleaned up branch: ${cleanedBranches.join(", ")}`,
});
}
let rewrittenDependents: Task[] = [];
let rewrittenBlockedByResidueDependents: Task[] = [];
let rewrittenLineageChildren: Task[] = [];
let deletedAt = "";
store.db.transaction(() => {
rewrittenDependents = store.rewriteDependentsForRemoval(id, dependentIds);
rewrittenBlockedByResidueDependents = store.rewriteBlockedByResidueDependentsForRemoval(id, new Set(dependentIds));
rewrittenLineageChildren = store.rewriteLineageChildrenForRemoval(id, lineageChildIds);
const deletedAt = new Date().toISOString();
deletedAt = new Date().toISOString();
const allowResurrection = options?.allowResurrection === true ? 1 : 0;
store.db.prepare("UPDATE tasks SET \"column\" = 'archived', deletedAt = ?, allowResurrection = ?, updatedAt = ? WHERE id = ?").run(deletedAt, allowResurrection, deletedAt, id);
void store.recordRunAuditEvent({
@@ -104,6 +132,11 @@ export async function deleteTaskImpl(store: TaskStore, id: string, options?: { r
store.db.bumpLastModified();
});
task.column = "archived";
task.deletedAt = deletedAt;
task.updatedAt = deletedAt;
scheduleDeleteBranchCleanup(store, task);
// FN-5143 defense-in-depth: drop any in-memory buffer entries for this
// task. flushAgentLogBuffer() above already ran inside the lock, but a
// concurrent appendAgentLog from another async path could re-buffer

View File

@@ -0,0 +1,139 @@
// @vitest-environment node
import { describe, expect, it, vi } from "vitest";
import express from "express";
import type { Task, TaskStore } from "@fusion/core";
import { registerTaskWorkflowRoutes } from "../routes/register-task-workflow-routes.js";
import { request as performRequest } from "../test-request.js";
import { ApiError, sendErrorResponse } from "../api-error.js";
function deferred<T>() {
let resolve!: (value: T) => void;
const promise = new Promise<T>((innerResolve) => {
resolve = innerResolve;
});
return { promise, resolve };
}
function mkTask(overrides: Partial<Task> & { id: string }): Task {
const now = "2026-07-15T09:00:00.000Z";
return {
id: overrides.id,
title: overrides.title ?? overrides.id,
description: overrides.description ?? overrides.id,
column: overrides.column ?? "todo",
dependencies: [],
createdAt: now,
updatedAt: now,
size: "M",
subtasks: [],
log: [],
tags: [],
blockedBy: [],
source: { sourceType: "api" },
...overrides,
} as Task;
}
function buildApp(input: {
store: Partial<TaskStore>;
engine?: { getAgentStore?: () => unknown };
runtimeLogger?: { warn: ReturnType<typeof vi.fn>; error: ReturnType<typeof vi.fn> };
}) {
const runtimeLogger = input.runtimeLogger ?? { warn: vi.fn(), error: vi.fn() };
const router = express.Router();
registerTaskWorkflowRoutes({
router,
store: input.store as TaskStore,
options: {},
runtimeLogger: runtimeLogger as never,
planningLogger: runtimeLogger as never,
chatLogger: runtimeLogger as never,
getProjectIdFromRequest: () => undefined,
getScopedStore: async () => input.store as TaskStore,
getProjectContext: async () => ({ store: input.store as TaskStore, engine: input.engine as never, projectId: "p-1" }),
prioritizeProjectsForCurrentDirectory: (projects) => projects,
emitRemoteRouteDiagnostic: () => {},
emitAuthSyncAuditLog: () => {},
parseScopeParam: () => undefined,
resolveAutomationStore: () => ({}) as never,
resolveRoutineStore: () => ({}) as never,
resolveRoutineRunner: () => ({}) as never,
registerDispose: () => {},
dispose: () => {},
rethrowAsApiError: (error: unknown): never => {
if (error instanceof ApiError) throw error;
throw new ApiError(500, error instanceof Error ? error.message : "Internal server error");
},
}, {
runtimeLogger,
upload: { single: () => (_req: unknown, _res: unknown, next: () => void) => next() },
taskDetailActivityLogLimit: 100,
validateOptionalModelField: (value) => (typeof value === "string" ? value : undefined),
normalizeModelSelectionPair: (provider, modelId) => ({ provider: provider ?? null, modelId: modelId ?? null }),
runGitCommand: async () => "",
isGitRepo: async () => true,
resolveIntegrationBranch: async () => "main",
trimTaskDetailActivityLog: (task) => task,
triggerCommentWakeForAssignedAgent: async () => {},
resolveSelfHealingManager: () => undefined,
});
const app = express();
app.use(express.json());
app.use("/api", router);
app.use((error: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
if (error instanceof ApiError) {
sendErrorResponse(res, error.statusCode, error.message, { details: error.details });
return;
}
sendErrorResponse(res, 500, error instanceof Error ? error.message : "Internal server error");
});
return app;
}
describe("DELETE /api/tasks/:id", () => {
it("responds after soft-delete without waiting for delayed agent-binding release", async () => {
const deletedTask = mkTask({
id: "FN-7968",
column: "archived",
deletedAt: "2026-07-15T09:02:00.000Z",
});
const store: Partial<TaskStore> = {
deleteTask: vi.fn().mockResolvedValue(deletedTask),
};
const listAgents = deferred<Array<{ id: string; name: string; role: string; reportsTo: string; taskId: string }>>();
const agentStore = {
listAgents: vi.fn(() => listAgents.promise),
syncExecutionTaskLink: vi.fn().mockResolvedValue(undefined),
deleteAgent: vi.fn().mockResolvedValue(undefined),
};
const app = buildApp({
store,
engine: { getAgentStore: () => agentStore },
});
let responseResolved = false;
const responsePromise = performRequest(app, "DELETE", "/api/tasks/FN-7968").then((response) => {
responseResolved = true;
return response;
});
await vi.waitFor(() => expect(responseResolved).toBe(true), { timeout: 100 });
const response = await responsePromise;
expect(response.status).toBe(200);
expect(response.body).toMatchObject({ id: "FN-7968", deletedAt: "2026-07-15T09:02:00.000Z" });
expect(store.deleteTask).toHaveBeenCalledWith("FN-7968", expect.objectContaining({
auditContext: expect.objectContaining({ agentId: "system" }),
}));
expect(agentStore.listAgents).toHaveBeenCalledWith({ includeEphemeral: true });
expect(agentStore.syncExecutionTaskLink).not.toHaveBeenCalled();
listAgents.resolve([{ id: "agent-1", name: "durable", role: "executor", reportsTo: "manager", taskId: "FN-7968" }]);
await vi.waitFor(() => {
expect(agentStore.syncExecutionTaskLink).toHaveBeenCalledWith("agent-1", undefined);
});
expect(agentStore.deleteAgent).not.toHaveBeenCalled();
});
});

View File

@@ -489,6 +489,23 @@ async function releaseExecutionAgentBindings(
}
}
function scheduleReleaseExecutionAgentBindings(
engine: Parameters<typeof releaseExecutionAgentBindings>[0],
taskId: string,
runtimeLogger: { warn: (message: string, data?: Record<string, unknown>) => void },
): void {
/*
FNXC:TaskDeletion 2026-07-15-09:52:
The DELETE /tasks/:id response must not wait for an includeEphemeral agent-store scan or per-agent unlink/delete calls after the DB soft-delete has committed. Keep releaseExecutionAgentBindings as the reliable cleanup implementation, but run it off the HTTP critical path and log failures so agent-binding cleanup remains observable instead of silently dropped.
*/
void releaseExecutionAgentBindings(engine, taskId).catch((error: unknown) => {
runtimeLogger.warn("Deferred task-delete agent binding release failed", {
taskId,
error: error instanceof Error ? error.message : String(error),
});
});
}
function buildDuplicateQuery(title: string | undefined, description: string): string {
const tokens = `${title ?? ""} ${description}`
.toLowerCase()
@@ -5324,7 +5341,7 @@ export function registerTaskWorkflowRoutes(ctx: ApiRoutesContext, deps: TaskWork
runId: `synthetic-dashboard-delete-${req.params.id}-${Date.now()}`,
},
});
await releaseExecutionAgentBindings(engine, req.params.id);
scheduleReleaseExecutionAgentBindings(engine, req.params.id, runtimeLogger);
res.json(task);
} catch (err: unknown) {
if (err instanceof ApiError) {