feat(FN-5624): suppress transient task.json ENOENT with guard, logging, and
Implements graceful suppression of transient `task.json` ENOENT errors in the executor, logging a suppression signal and surfacing a banner in the UI, with test coverage for both the executor behavior and notification service. Documentation in `docs/architecture.md` and a changeset for `@runfusion/f Fusion-Task-Id: FN-5624 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai> Fusion-Task-Id: FN-5624
This commit is contained in:
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
Suppress a misleading transient failure state when a worktree-local `.fusion/tasks/<id>/task.json` read briefly returns ENOENT during executor session startup. Fusion now treats this as recoverable, routes through existing auto-recovery, and avoids persisting `status: "failed"`/`error` so the red task-card error banner and failed notification are not shown for self-healed runs.
|
||||||
@@ -685,6 +685,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
|||||||
- FN-4285 decision: add a follow-up for a tree-equality recovery strategy (`rev-parse <base>^{tree}` == `<task-branch>^{tree}`) in `findAlreadyMergedTaskCommit`. This closes stranded already-merged branches that evade trailer/ancestry/patch-id matching, with guardrails limited to retry-exhausted review tasks to avoid false positives during transient post-rebase parity windows.
|
- FN-4285 decision: add a follow-up for a tree-equality recovery strategy (`rev-parse <base>^{tree}` == `<task-branch>^{tree}`) in `findAlreadyMergedTaskCommit`. This closes stranded already-merged branches that evade trailer/ancestry/patch-id matching, with guardrails limited to retry-exhausted review tasks to avoid false positives during transient post-rebase parity windows.
|
||||||
- No-`fn_task_done` recovery classification is normalized across executor, restart recovery, and self-healing: detection keys on executor-emitted `"without calling fn_task_done"` strings (while still tolerating legacy `task_done` wording), then applies the bounded ladder deterministically (in-session retries → bounded todo requeues with preserved progress when appropriate → terminal surfaced failure when budget is exhausted).
|
- No-`fn_task_done` recovery classification is normalized across executor, restart recovery, and self-healing: detection keys on executor-emitted `"without calling fn_task_done"` strings (while still tolerating legacy `task_done` wording), then applies the bounded ladder deterministically (in-session retries → bounded todo requeues with preserved progress when appropriate → terminal surfaced failure when budget is exhausted).
|
||||||
- `clearStaleBlockedBy()` clears `blockedBy` (and transient `status`) on todo tasks when their blocker is missing, done, archived, paused in-review, or failed in-review with merge retries exhausted. FN-3924 extends this with a dependency-integrity guard: if a task has explicit dependencies and `blockedBy` is not one of the currently unresolved deps, the stale marker is cleared. FN-4091 broadens the sweep to active `in-progress` and un-paused `in-review` tasks as well, but those repairs only null `blockedBy` (they do not rewrite scheduler-owned queued state). FN-5488 adds two fast paths: (1) failed in-review blockers at/above `MAX_AUTO_MERGE_RETRIES` always fan out unblock recovery with explicit reason codes, and (2) `status="merging"|"merging-pr"` blockers with no active merger owner are treated as unbacked after a short grace window (`unbackedMergingFanoutGraceMs`, default 60s) so manual retry/unpause `updatedAt` refreshes cannot deadlock downstream todos indefinitely. Recovery logs now use `Auto-recovered (FN-5488): ... reason=<code>` for auditability while preserving FN-4538 overlap-blocking invariants.
|
- `clearStaleBlockedBy()` clears `blockedBy` (and transient `status`) on todo tasks when their blocker is missing, done, archived, paused in-review, or failed in-review with merge retries exhausted. FN-3924 extends this with a dependency-integrity guard: if a task has explicit dependencies and `blockedBy` is not one of the currently unresolved deps, the stale marker is cleared. FN-4091 broadens the sweep to active `in-progress` and un-paused `in-review` tasks as well, but those repairs only null `blockedBy` (they do not rewrite scheduler-owned queued state). FN-5488 adds two fast paths: (1) failed in-review blockers at/above `MAX_AUTO_MERGE_RETRIES` always fan out unblock recovery with explicit reason codes, and (2) `status="merging"|"merging-pr"` blockers with no active merger owner are treated as unbacked after a short grace window (`unbackedMergingFanoutGraceMs`, default 60s) so manual retry/unpause `updatedAt` refreshes cannot deadlock downstream todos indefinitely. Recovery logs now use `Auto-recovered (FN-5488): ... reason=<code>` for auditability while preserving FN-4538 overlap-blocking invariants.
|
||||||
|
- FN-5624 suppresses transient worktree-local `.fusion/tasks/<id>/task.json` ENOENT session-start failures. When the missing file path is under `task.worktree`, executor routes through unusable-worktree auto-recovery, skips persisting `status: "failed"`/`error` on the task row, and emits `[transient-task-json-suppressed] ... reason=missing-task-json-under-worktree`. The corresponding self-healing `Auto-recovered:` log entry keeps notification suppression aligned with the existing `/^Auto-recovered:/` grace-window rule.
|
||||||
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
- `inspectBranchConflict()` now treats self-owned zero-attribution collisions as reclaimable (instead of foreign) when ownership is proven by task/worktree identity, so stranded self-branches do not enter unrecoverable loops.
|
||||||
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
- `reclaimSelfOwnedBranchConflicts()` includes paused `branch-conflict-unrecoverable` tasks (not just todo/in-progress), clearing paused/error state in one update and requeueing only when parked in `in-review`.
|
||||||
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
- Together, `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and paused-aware in-review scheduling prevent merge-deadlock loops by finalizing already-landed work, clearing stale dependency blockers, reclaiming self-owned conflicts, and avoiding paused review cards re-blocking overlap dispatch.
|
||||||
|
|||||||
@@ -1464,6 +1464,23 @@ describe("TaskCard", () => {
|
|||||||
expect(screen.queryByRole("button", { name: "Retry" })).toBeNull();
|
expect(screen.queryByRole("button", { name: "Retry" })).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("does not render card-error banner for auto-recovered transient row", () => {
|
||||||
|
const { container } = render(
|
||||||
|
<TaskCard
|
||||||
|
task={makeTask({
|
||||||
|
column: "todo",
|
||||||
|
status: undefined,
|
||||||
|
error: undefined,
|
||||||
|
log: [{ timestamp: new Date().toISOString(), action: "Auto-recovered: retry/verification session targeted unusable worktree" }],
|
||||||
|
})}
|
||||||
|
onOpenDetail={noop}
|
||||||
|
addToast={noop}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(container.querySelector(".card-error")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
it("calls onRetryTask with task id", async () => {
|
it("calls onRetryTask with task id", async () => {
|
||||||
const onRetryTask = vi.fn(async () => ({}) as Task);
|
const onRetryTask = vi.fn(async () => ({}) as Task);
|
||||||
render(
|
render(
|
||||||
|
|||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { TaskDeletedError } from "@fusion/core";
|
||||||
|
import { isTransientMissingTaskJsonError } from "../executor.js";
|
||||||
|
|
||||||
|
describe("isTransientMissingTaskJsonError", () => {
|
||||||
|
const task = {
|
||||||
|
id: "FN-5624",
|
||||||
|
worktree: "/tmp/worktrees/fn-5624",
|
||||||
|
};
|
||||||
|
|
||||||
|
it("matches ENOENT on worktree-scoped .fusion/tasks/<id>/task.json", () => {
|
||||||
|
const err = `ENOENT: no such file or directory, open '/tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/task.json'`;
|
||||||
|
expect(isTransientMissingTaskJsonError(err, task)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match task.json parse failures", () => {
|
||||||
|
const err = "Failed to parse task.json at /tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/task.json: Unexpected token";
|
||||||
|
expect(isTransientMissingTaskJsonError(err, task)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match TaskDeletedError", () => {
|
||||||
|
expect(isTransientMissingTaskJsonError(new TaskDeletedError("FN-5624", new Date().toISOString()), task)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match ENOENT for non-task.json paths", () => {
|
||||||
|
const err = `ENOENT: no such file or directory, open '/tmp/worktrees/fn-5624/.fusion/tasks/FN-5624/PROMPT.md'`;
|
||||||
|
expect(isTransientMissingTaskJsonError(err, task)).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -9,7 +9,7 @@ import { delimiter, isAbsolute, join, relative, resolve as resolvePath } from "n
|
|||||||
import { existsSync, realpathSync } from "node:fs";
|
import { existsSync, realpathSync } from "node:fs";
|
||||||
import { readFile, rm, writeFile } from "node:fs/promises";
|
import { readFile, rm, writeFile } from "node:fs/promises";
|
||||||
import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings } from "@fusion/core";
|
import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings } from "@fusion/core";
|
||||||
import { RetryStormError, serializeRetryStormError } from "@fusion/core";
|
import { RetryStormError, TaskDeletedError, serializeRetryStormError } from "@fusion/core";
|
||||||
import {
|
import {
|
||||||
ApprovalRequestStore,
|
ApprovalRequestStore,
|
||||||
buildExecutionMemoryInstructions,
|
buildExecutionMemoryInstructions,
|
||||||
@@ -222,6 +222,33 @@ const WORKFLOW_RERUN_WATCHDOG_MS = 15_000;
|
|||||||
|
|
||||||
const TASK_DONE_REFUSAL_SUFFIX = "Either finish the work and resubmit, or do not call fn_task_done — exit the session and the engine will requeue.";
|
const TASK_DONE_REFUSAL_SUFFIX = "Either finish the work and resubmit, or do not call fn_task_done — exit the session and the engine will requeue.";
|
||||||
|
|
||||||
|
const TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN = /ENOENT:\s+no such file or directory,\s+open\s+'([^']+\/\.fusion\/tasks\/([^/]+)\/task\.json)'/;
|
||||||
|
|
||||||
|
export function isTransientMissingTaskJsonError(error: unknown, task: Pick<Task, "id" | "worktree">): boolean {
|
||||||
|
if (error instanceof TaskDeletedError) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const message = typeof error === "string"
|
||||||
|
? error
|
||||||
|
: error instanceof Error
|
||||||
|
? error.message
|
||||||
|
: "";
|
||||||
|
const match = TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN.exec(message);
|
||||||
|
if (!match) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const [, filePath, taskIdFromPath] = match;
|
||||||
|
if (taskIdFromPath !== task.id) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (typeof task.worktree !== "string" || task.worktree.length === 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const normalizedWorktree = resolvePath(task.worktree);
|
||||||
|
const normalizedTaskJsonPath = resolvePath(filePath);
|
||||||
|
return normalizedTaskJsonPath.startsWith(`${normalizedWorktree}/`);
|
||||||
|
}
|
||||||
|
|
||||||
export const DISSENT_PATTERNS: RegExp[] = [
|
export const DISSENT_PATTERNS: RegExp[] = [
|
||||||
/\btask (is|was)(?: not|n['’]?t) complete\b/i,
|
/\btask (is|was)(?: not|n['’]?t) complete\b/i,
|
||||||
/\b(?:i (?:could|can)(?:not|n['’]?t)|unable to|failed to) (?:complete|finish|implement)\b/i,
|
/\b(?:i (?:could|can)(?:not|n['’]?t)|unable to|failed to) (?:complete|finish|implement)\b/i,
|
||||||
@@ -9027,10 +9054,19 @@ Backward compat fallback: if JSON is unavailable, you may still begin output wit
|
|||||||
audit: RunAuditor,
|
audit: RunAuditor,
|
||||||
): Promise<boolean> {
|
): Promise<boolean> {
|
||||||
const errorText = error instanceof Error ? error.message : String(error);
|
const errorText = error instanceof Error ? error.message : String(error);
|
||||||
if (!isMissingWorktreeSessionStartFailure(errorText)) return false;
|
const missingWorktreeFailure = isMissingWorktreeSessionStartFailure(errorText);
|
||||||
|
const missingTaskJsonFailure = isTransientMissingTaskJsonError(error, task);
|
||||||
|
if (!missingWorktreeFailure && !missingTaskJsonFailure) return false;
|
||||||
|
|
||||||
const classification = classifyMissingWorktreeSessionStartFailure(errorText);
|
const classification = classifyMissingWorktreeSessionStartFailure(errorText);
|
||||||
const staleWorktreePath = extractMissingWorktreePathFromSessionStartFailure(errorText) ?? worktreePath;
|
const missingTaskJsonPath = errorText.match(TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN)?.[1] ?? null;
|
||||||
|
const staleWorktreePath = extractMissingWorktreePathFromSessionStartFailure(errorText)
|
||||||
|
?? (missingTaskJsonPath ? resolvePath(missingTaskJsonPath, "..", "..", "..") : null)
|
||||||
|
?? worktreePath;
|
||||||
|
|
||||||
|
if (missingTaskJsonFailure) {
|
||||||
|
executorLog.log(`[transient-task-json-suppressed] taskId=${task.id} elapsedMs=0 reason=missing-task-json-under-worktree path=${missingTaskJsonPath ?? "unknown"}`);
|
||||||
|
}
|
||||||
|
|
||||||
await audit.git({
|
await audit.git({
|
||||||
type: "worktree:incomplete-detected",
|
type: "worktree:incomplete-detected",
|
||||||
|
|||||||
@@ -113,6 +113,32 @@ describe("NotificationService deferred failure notifications", () => {
|
|||||||
await service.stop();
|
await service.stop();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("suppresses transient missing task.json failure after Auto-recovered clear", async () => {
|
||||||
|
const { store, service, sendNotification } = await setup();
|
||||||
|
store.setTask(task({
|
||||||
|
id: "FN-1",
|
||||||
|
status: "failed",
|
||||||
|
error: "ENOENT: no such file or directory, open '/tmp/worktrees/fn-1/.fusion/tasks/FN-1/task.json'",
|
||||||
|
}));
|
||||||
|
store.emit("task:updated", task({ id: "FN-1", status: "failed" }));
|
||||||
|
|
||||||
|
const recoveredTask = task({
|
||||||
|
id: "FN-1",
|
||||||
|
status: undefined,
|
||||||
|
error: undefined,
|
||||||
|
column: "todo",
|
||||||
|
log: [{ timestamp: new Date().toISOString(), action: "Auto-recovered: retry/verification session targeted unusable worktree" }],
|
||||||
|
});
|
||||||
|
store.setTask(recoveredTask);
|
||||||
|
store.emit("task:moved", { task: recoveredTask, from: "in-progress", to: "todo" });
|
||||||
|
await vi.advanceTimersByTimeAsync(100);
|
||||||
|
|
||||||
|
expect(sendNotification).not.toHaveBeenCalledWith("failed", expect.anything());
|
||||||
|
expect((await store.getTask("FN-1"))?.status).not.toBe("failed");
|
||||||
|
expect(service.getMetrics().failureNotificationSuppressedCount).toBe(1);
|
||||||
|
await service.stop();
|
||||||
|
});
|
||||||
|
|
||||||
it("Recovery via task:moved to done suppresses failed notification", async () => {
|
it("Recovery via task:moved to done suppresses failed notification", async () => {
|
||||||
const { store, service, sendNotification } = await setup();
|
const { store, service, sendNotification } = await setup();
|
||||||
store.setTask(task({ id: "FN-1", status: "failed", column: "in-review" }));
|
store.setTask(task({ id: "FN-1", status: "failed", column: "in-review" }));
|
||||||
|
|||||||
Reference in New Issue
Block a user