feat(FN-5624): suppress transient task.json ENOENT with guard, logging, and
Implements graceful suppression of transient `task.json` ENOENT errors in the executor, logging a suppression signal and surfacing a banner in the UI, with test coverage for both the executor behavior and notification service. Documentation in `docs/architecture.md` and a changeset for `@runfusion/f Fusion-Task-Id: FN-5624 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai> Fusion-Task-Id: FN-5624
This commit is contained in:
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Suppress a misleading transient failure state when a worktree-local `.fusion/tasks/<id>/task.json` read briefly returns ENOENT during executor session startup. Fusion now treats this as recoverable, routes through existing auto-recovery, and avoids persisting `status: "failed"`/`error` so the red task-card error banner and failed notification are not shown for self-healed runs.
|
||||
@@ -685,6 +685,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
||||
- FN-4285 decision: add a follow-up for a tree-equality recovery strategy (`rev-parse <base>^{tree}` == `<task-branch>^{tree}`) in `findAlreadyMergedTaskCommit`. This closes stranded already-merged branches that evade trailer/ancestry/patch-id matching, with guardrails limited to retry-exhausted review tasks to avoid false positives during transient post-rebase parity windows.
|
||||
- No-`fn_task_done` recovery classification is normalized across executor, restart recovery, and self-healing: detection keys on executor-emitted `"without calling fn_task_done"` strings (while still tolerating legacy `task_done` wording), then applies the bounded ladder deterministically (in-session retries → bounded todo requeues with preserved progress when appropriate → terminal surfaced failure when budget is exhausted).
|
||||
- `clearStaleBlockedBy()` clears `blockedBy` (and transient `status`) on todo tasks when their blocker is missing, done, archived, paused in-review, or failed in-review with merge retries exhausted. FN-3924 extends this with a dependency-integrity guard: if a task has explicit dependencies and `blockedBy` is not one of the currently unresolved deps, the stale marker is cleared. FN-4091 broadens the sweep to active `in-progress` and un-paused `in-review` tasks as well, but those repairs only null `blockedBy` (they do not rewrite scheduler-owned queued state). FN-5488 adds two fast paths: (1) failed in-review blockers at/above `MAX_AUTO_MERGE_RETRIES` always fan out unblock recovery with explicit reason codes, and (2) `status="merging"|"merging-pr"` blockers with no active merger owner are treated as unbacked after a short grace window (`unbackedMergingFanoutGraceMs`, default 60s) so manual retry/unpause `updatedAt` refreshes cannot deadlock downstream todos indefinitely. Recovery logs now use `Auto-recovered (FN-5488): ... reason=<code>` for auditability while preserving FN-4538 overlap-blocking invariants.
|
||||
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
|
||||
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
||||
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
||||
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
||||
|
||||
@@ -1464,6 +1464,23 @@ describe("TaskCard", () => {
|
||||
expect(screen.queryByRole("button", { name: "Retry" })).toBeNull();
|
||||
});
|
||||
|
||||
it("does not render card-error banner for auto-recovered transient row", () => {
|
||||
const { container } = render(
|
||||
<TaskCard
|
||||
task={makeTask({
|
||||
column: "todo",
|
||||
status: undefined,
|
||||
error: undefined,
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Auto-recovered: retry/verification session targeted unusable worktree" }],
|
||||
})}
|
||||
onOpenDetail={noop}
|
||||
addToast={noop}
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(container.querySelector(".card-error")).toBeNull();
|
||||
});
|
||||
|
||||
it("calls onRetryTask with task id", async () => {
|
||||
const onRetryTask = vi.fn(async () => ({}) as Task);
|
||||
render(
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { TaskDeletedError } from "@fusion/core";
|
||||
import { isTransientMissingTaskJsonError } from "../executor.js";
|
||||
|
||||
describe("isTransientMissingTaskJsonError", () => {
|
||||
const task = {
|
||||
id: "FN-5624",
|
||||
worktree: "/tmp/worktrees/fn-5624",
|
||||
};
|
||||
|
||||
it("matches ENOENT on worktree-scoped .fusion/tasks/<id>/task.json", () => {
|
||||
const err = `ENOENT: no such file or directory, open '/tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/task.json'`;
|
||||
expect(isTransientMissingTaskJsonError(err, task)).toBe(true);
|
||||
});
|
||||
|
||||
it("does not match task.json parse failures", () => {
|
||||
const err = "Failed to parse task.json at /tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/task.json: Unexpected token";
|
||||
expect(isTransientMissingTaskJsonError(err, task)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not match TaskDeletedError", () => {
|
||||
expect(isTransientMissingTaskJsonError(new TaskDeletedError("FN-5624", new Date().toISOString()), task)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not match ENOENT for non-task.json paths", () => {
|
||||
const err = `ENOENT: no such file or directory, open '/tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/PROMPT.md'`;
|
||||
expect(isTransientMissingTaskJsonError(err, task)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -9,7 +9,7 @@ import { delimiter, isAbsolute, join, relative, resolve as resolvePath } from "n
|
||||
import { existsSync, realpathSync } from "node:fs";
|
||||
import { readFile, rm, writeFile } from "node:fs/promises";
|
||||
import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings } from "@fusion/core";
|
||||
import { RetryStormError, serializeRetryStormError } from "@fusion/core";
|
||||
import { RetryStormError, TaskDeletedError, serializeRetryStormError } from "@fusion/core";
|
||||
import {
|
||||
ApprovalRequestStore,
|
||||
buildExecutionMemoryInstructions,
|
||||
@@ -222,6 +222,33 @@ const WORKFLOW_RERUN_WATCHDOG_MS = 15_000;
|
||||
|
||||
const TASK_DONE_REFUSAL_SUFFIX = "Either finish the work and resubmit, or do not call fn_task_done — exit the session and the engine will requeue.";
|
||||
|
||||
const TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN = /ENOENT:\s+no such file or directory,\s+open\s+'([^']+\/\.fusion\/tasks\/([^/]+)\/task\.json)'/;
|
||||
|
||||
export function isTransientMissingTaskJsonError(error: unknown, task: Pick<Task, "id" | "worktree">): boolean {
|
||||
if (error instanceof TaskDeletedError) {
|
||||
return false;
|
||||
}
|
||||
const message = typeof error === "string"
|
||||
? error
|
||||
: error instanceof Error
|
||||
? error.message
|
||||
: "";
|
||||
const match = TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN.exec(message);
|
||||
if (!match) {
|
||||
return false;
|
||||
}
|
||||
const [, filePath, taskIdFromPath] = match;
|
||||
if (taskIdFromPath !== task.id) {
|
||||
return false;
|
||||
}
|
||||
if (typeof task.worktree !== "string" || task.worktree.length === 0) {
|
||||
return false;
|
||||
}
|
||||
const normalizedWorktree = resolvePath(task.worktree);
|
||||
const normalizedTaskJsonPath = resolvePath(filePath);
|
||||
return normalizedTaskJsonPath.startsWith(`${normalizedWorktree}/`);
|
||||
}
|
||||
|
||||
export const DISSENT_PATTERNS: RegExp[] = [
|
||||
/\btask (is|was)(?: not|n['’]?t) complete\b/i,
|
||||
/\b(?:i (?:could|can)(?:not|n['’]?t)|unable to|failed to) (?:complete|finish|implement)\b/i,
|
||||
@@ -9027,10 +9054,19 @@ Backward compat fallback: if JSON is unavailable, you may still begin output wit
|
||||
audit: RunAuditor,
|
||||
): Promise<boolean> {
|
||||
const errorText = error instanceof Error ? error.message : String(error);
|
||||
if (!isMissingWorktreeSessionStartFailure(errorText)) return false;
|
||||
const missingWorktreeFailure = isMissingWorktreeSessionStartFailure(errorText);
|
||||
const missingTaskJsonFailure = isTransientMissingTaskJsonError(error, task);
|
||||
if (!missingWorktreeFailure && !missingTaskJsonFailure) return false;
|
||||
|
||||
const classification = classifyMissingWorktreeSessionStartFailure(errorText);
|
||||
const staleWorktreePath = extractMissingWorktreePathFromSessionStartFailure(errorText) ?? worktreePath;
|
||||
const missingTaskJsonPath = errorText.match(TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN)?.[1] ?? null;
|
||||
const staleWorktreePath = extractMissingWorktreePathFromSessionStartFailure(errorText)
|
||||
?? (missingTaskJsonPath ? resolvePath(missingTaskJsonPath, "..", "..", "..") : null)
|
||||
?? worktreePath;
|
||||
|
||||
if (missingTaskJsonFailure) {
|
||||
executorLog.log(`[transient-task-json-suppressed] taskId=${task.id} elapsedMs=0 reason=missing-task-json-under-worktree path=${missingTaskJsonPath ?? "unknown"}`);
|
||||
}
|
||||
|
||||
await audit.git({
|
||||
type: "worktree:incomplete-detected",
|
||||
|
||||
@@ -113,6 +113,32 @@ describe("NotificationService deferred failure notifications", () => {
|
||||
await service.stop();
|
||||
});
|
||||
|
||||
it("suppresses transient missing task.json failure after Auto-recovered clear", async () => {
|
||||
const { store, service, sendNotification } = await setup();
|
||||
store.setTask(task({
|
||||
id: "FN-1",
|
||||
status: "failed",
|
||||
error: "ENOENT: no such file or directory, open '/tmp/worktrees/fn-1/.fusion/tasks/FN-1/task.json'",
|
||||
}));
|
||||
store.emit("task:updated", task({ id: "FN-1", status: "failed" }));
|
||||
|
||||
const recoveredTask = task({
|
||||
id: "FN-1",
|
||||
status: undefined,
|
||||
error: undefined,
|
||||
column: "todo",
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Auto-recovered: retry/verification session targeted unusable worktree" }],
|
||||
});
|
||||
store.setTask(recoveredTask);
|
||||
store.emit("task:moved", { task: recoveredTask, from: "in-progress", to: "todo" });
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
expect(sendNotification).not.toHaveBeenCalledWith("failed", expect.anything());
|
||||
expect((await store.getTask("FN-1"))?.status).not.toBe("failed");
|
||||
expect(service.getMetrics().failureNotificationSuppressedCount).toBe(1);
|
||||
await service.stop();
|
||||
});
|
||||
|
||||
it("Recovery via task:moved to done suppresses failed notification", async () => {
|
||||
const { store, service, sendNotification } = await setup();
|
||||
store.setTask(task({ id: "FN-1", status: "failed", column: "in-review" }));
|
||||
|
||||
Reference in New Issue
Block a user