FN-7173: preserve stuck triage drafts on requeue
Stuck triage retries now continue from saved planning drafts instead of restarting. - Detect non-empty PROMPT.md drafts or saved plan task documents after stuck-detector aborts. - Requeue draft-backed stuck triage runs as replans with explicit resume feedback and bounded retry escalation. - Cover prompt-vs-plan recovery, cold retries, approval recovery, and exhaustion behavior in engine tests. - Document the stuck-triage preservation invariant and add a patch changeset. Files changed: ...-7173-preserve-triage-draft-on-stuck-requeue.md | 7 + docs/architecture.md | 6 +- .../triage-stuck-requeue-preserve-draft.test.ts | 348 +++++++++++++++++++++ packages/engine/src/__tests__/triage.test.ts | 2 +- packages/engine/src/triage.ts | 153 ++++++++- 5 files changed, 499 insertions(+), 17 deletions(-) Fusion-Task-Id: FN-7173 Fusion-Task-Lineage: 8e59b025-92cc-4902-bf88-b203b65c2c7d Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Stuck triage re-queues now resume from the drafted plan instead of restarting planning from scratch.
|
||||
category: fix
|
||||
dev: triage.ts stuck-abort paths seed buildSpecificationPrompt with the on-disk PROMPT.md draft, or a non-empty plan task document when PROMPT.md is absent, and bound consecutive triage stuck-retries by settings.maxStuckKills before escalating to failed/paused.
|
||||
@@ -598,7 +598,7 @@ See [Memory Plugin Contract](./memory-plugin-contract.md) for the full plan.
|
||||
`@fusion/engine` executes the autonomous workflow.
|
||||
|
||||
### Agent roles
|
||||
- **Planning**: the planning processor generates task plans (`PROMPT.md`) and selects eligible planning tasks by priority first, then FIFO (`createdAt` ascending) within each priority tier.
|
||||
- **Planning**: the planning processor generates task plans (`PROMPT.md`) and selects eligible planning tasks by priority first, then FIFO (`createdAt` ascending) within each priority tier. If the stuck-task detector kills a not-yet-approved planning session after a non-empty `PROMPT.md` draft exists, the retry is requeued as `needs-replan` and seeds the next prompt in revision mode from that draft instead of cold-starting. When `PROMPT.md` is absent, a non-empty `plan` task document written through `fn_task_document_write` is the fallback seed; missing or whitespace-only drafts still cold-start.
|
||||
- **Executor**: `TaskExecutor` (`executor.ts`) implements tasks in worktrees
|
||||
- **Reviewer**: `reviewStep()` (`reviewer.ts`) performs plan/code/spec reviews
|
||||
- **Merger**: `aiMergeTask()` (`merger.ts`) merges approved work
|
||||
@@ -688,7 +688,9 @@ Runtime action-gate flow (v1):
|
||||
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.
|
||||
|
||||
#### Stuck-loop exhaustion terminal contract
|
||||
When stuck-kill retries are exhausted, `checkStuckBudget()` marks the task `status: "failed"`, moves it to `in-review`, and writes an error that starts with `STUCK_LOOP_EXHAUSTED:`. The error and final task-log line both include the kill count/max and last stuck reason (`loop` or `inactivity`). `StuckTaskDetector` also untracks the task and refuses to re-track it while that failed terminal error remains, preventing further automatic kill/requeue churn. The final log line explicitly states that no further automatic retries will run and directs operators to manually retry, pause, or move the task back to triage to resume work.
|
||||
When stuck-kill retries are exhausted, `checkStuckBudget()` marks executor-phase tasks `status: "failed"`, moves them to `in-review`, and writes an error that starts with `STUCK_LOOP_EXHAUSTED:`. The error and final task-log line both include the kill count/max and last stuck reason (`loop` or `inactivity`). `StuckTaskDetector` also untracks the task and refuses to re-track it while that failed terminal error remains, preventing further automatic kill/requeue churn. The final log line explicitly states that no further automatic retries will run and directs operators to manually retry, pause, or move the task back to triage to resume work.
|
||||
|
||||
Planning-phase stuck kills use the same `stuckKillCount` / `settings.maxStuckKills` budget before execution starts. While under budget, a stuck triage requeue resumes from a non-empty on-disk `PROMPT.md` draft in revision mode and logs resume feedback; if `PROMPT.md` is absent, it falls back to a non-empty `plan` task document. Absent or whitespace-only drafts preserve cold-start behavior, and already-approved drafts continue through approved-spec recovery. At budget exhaustion, triage parks the task as `status: "failed"`, `paused: true` with a `STUCK_LOOP_EXHAUSTED:` error so a reasoning-looping planner cannot restart indefinitely.
|
||||
|
||||
Active `fn_run_verification` subprocesses are a bounded progress signal (FN-6598). `createRunVerificationTool()` brackets each command with `StuckTaskDetector.beginVerification()` / `endVerification()`; while the command is active and still inside its own timeout plus cleanup grace, the detector suppresses `loop` and `no-progress-churn` classification so healthy marathon verification output cannot consume stuck-kill budget. `inactivity` is not suppressed: the verification runner must continue emitting line output or synthetic heartbeats, and if the process overruns its recorded deadline or never sends an end signal, normal detection resumes.
|
||||
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Settings, Task, TaskDetail, TaskStore } from "@fusion/core";
|
||||
import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { TriageProcessor } from "../triage.js";
|
||||
|
||||
const { mockCreateResolvedAgentSession, mockPromptWithFallback } = vi.hoisted(() => ({
|
||||
mockCreateResolvedAgentSession: vi.fn(),
|
||||
mockPromptWithFallback: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../agent-session-helpers.js", () => ({
|
||||
createResolvedAgentSession: mockCreateResolvedAgentSession,
|
||||
extractRuntimeHint: vi.fn(),
|
||||
resolvePlanningSessionModel: vi.fn().mockReturnValue({ provider: "mock", modelId: "mock-model" }),
|
||||
}));
|
||||
|
||||
vi.mock("../pi.js", () => ({
|
||||
describeModel: vi.fn().mockReturnValue("mock-model"),
|
||||
promptWithFallback: mockPromptWithFallback,
|
||||
}));
|
||||
|
||||
function createTask(overrides: Partial<Task> = {}): Task {
|
||||
return {
|
||||
id: "FN-7173-T",
|
||||
title: "Preserve draft",
|
||||
description: "Preserve an existing draft after stuck triage requeue",
|
||||
column: "triage",
|
||||
status: null,
|
||||
dependencies: [],
|
||||
steps: [],
|
||||
currentStep: 0,
|
||||
log: [],
|
||||
createdAt: "2026-06-27T00:00:00.000Z",
|
||||
updatedAt: "2026-06-27T00:00:00.000Z",
|
||||
...overrides,
|
||||
} as Task;
|
||||
}
|
||||
|
||||
function toDetail(task: Task): TaskDetail {
|
||||
return {
|
||||
...task,
|
||||
attachments: [],
|
||||
comments: [],
|
||||
log: task.log ?? [],
|
||||
} as TaskDetail;
|
||||
}
|
||||
|
||||
function createMutableStore(initialTask: Task, settings: Partial<Settings> = {}, documents: Record<string, string> = {}) {
|
||||
let currentTask: Task = { ...initialTask, log: [...(initialTask.log ?? [])] };
|
||||
const store = {
|
||||
getTask: vi.fn(async () => toDetail(currentTask)),
|
||||
listTasks: vi.fn().mockResolvedValue([]),
|
||||
getSettings: vi.fn().mockResolvedValue({
|
||||
pollIntervalMs: 60_000,
|
||||
maxConcurrent: 1,
|
||||
maxWorktrees: 1,
|
||||
autoMerge: true,
|
||||
groupOverlappingFiles: false,
|
||||
maxStuckKills: 6,
|
||||
requirePlanApproval: false,
|
||||
...settings,
|
||||
} as Settings),
|
||||
getTaskDocument: vi.fn(async (_id: string, key: string) => {
|
||||
const content = documents[key];
|
||||
return content === undefined
|
||||
? null
|
||||
: {
|
||||
id: `doc-${key}`,
|
||||
taskId: currentTask.id,
|
||||
key,
|
||||
content,
|
||||
revision: 1,
|
||||
author: "agent",
|
||||
metadata: {},
|
||||
createdAt: "2026-06-27T00:00:00.000Z",
|
||||
updatedAt: "2026-06-27T00:00:00.000Z",
|
||||
};
|
||||
}),
|
||||
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => {
|
||||
currentTask = { ...currentTask, ...updates, updatedAt: "2026-06-27T00:01:00.000Z" } as Task;
|
||||
return currentTask;
|
||||
}),
|
||||
moveTask: vi.fn(async (_id: string, column: Task["column"]) => {
|
||||
currentTask = { ...currentTask, column, status: null } as Task;
|
||||
return currentTask;
|
||||
}),
|
||||
logEntry: vi.fn(async (_id: string, action: string, outcome?: string) => {
|
||||
currentTask = {
|
||||
...currentTask,
|
||||
log: [...(currentTask.log ?? []), { timestamp: new Date().toISOString(), action, outcome }],
|
||||
} as Task;
|
||||
}),
|
||||
appendAgentLog: vi.fn().mockResolvedValue(undefined),
|
||||
parseDependenciesFromPrompt: vi.fn().mockResolvedValue([]),
|
||||
parseStepsFromPrompt: vi.fn().mockResolvedValue([]),
|
||||
parseFileScopeFromPrompt: vi.fn().mockResolvedValue([]),
|
||||
on: vi.fn(),
|
||||
off: vi.fn(),
|
||||
} as unknown as TaskStore;
|
||||
|
||||
return {
|
||||
store,
|
||||
get currentTask() {
|
||||
return currentTask;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function createRoot(taskId: string, draft?: string): Promise<string> {
|
||||
const rootDir = await mkdtemp(join(tmpdir(), "fusion-triage-stuck-draft-"));
|
||||
const taskDir = join(rootDir, ".fusion", "tasks", taskId);
|
||||
await mkdir(taskDir, { recursive: true });
|
||||
if (draft !== undefined) {
|
||||
await writeFile(join(taskDir, "PROMPT.md"), draft, "utf8");
|
||||
}
|
||||
return rootDir;
|
||||
}
|
||||
|
||||
function mockSession() {
|
||||
mockCreateResolvedAgentSession.mockResolvedValue({
|
||||
session: {
|
||||
state: {},
|
||||
sessionManager: { getLeafId: vi.fn().mockReturnValue(null) },
|
||||
prompt: vi.fn().mockResolvedValue(undefined),
|
||||
dispose: vi.fn(),
|
||||
navigateTree: vi.fn(),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function cleanup(rootDir: string | undefined) {
|
||||
if (rootDir) {
|
||||
await rm(rootDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
describe("triage stuck requeue preserves existing PROMPT.md drafts", () => {
|
||||
let rootDir: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockSession();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await cleanup(rootDir);
|
||||
rootDir = undefined;
|
||||
});
|
||||
|
||||
it("reproduces the cold-start symptom and asserts the retry resumes from a non-empty draft", async () => {
|
||||
const draft = "# Task: FN-7173-T\n\n## Mission\n\nContinue from this already drafted plan.";
|
||||
const task = createTask();
|
||||
rootDir = await createRoot(task.id, draft);
|
||||
const harness = createMutableStore(task);
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
let retryPrompt = "";
|
||||
|
||||
mockPromptWithFallback
|
||||
.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
})
|
||||
.mockImplementationOnce(async (_session: unknown, prompt: string) => {
|
||||
retryPrompt = prompt;
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
expect(harness.currentTask.status).toBe("needs-replan");
|
||||
expect(harness.store.logEntry).toHaveBeenCalledWith(
|
||||
task.id,
|
||||
"Triage stuck re-queue will resume existing planning draft",
|
||||
expect.stringContaining("Resume from the existing draft"),
|
||||
);
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(retryPrompt).toContain("Revise this task");
|
||||
expect(retryPrompt).toContain("## Existing Specification");
|
||||
expect(retryPrompt).toContain(draft);
|
||||
expect(retryPrompt).toContain("instead of restarting planning from scratch");
|
||||
});
|
||||
|
||||
it("resumes from a saved plan task document when PROMPT.md is absent", async () => {
|
||||
const planDocument = "# Plan document draft\n\n## Mission\n\nResume from the saved task document.";
|
||||
const task = createTask({ id: "FN-7173-PLAN-DOC" });
|
||||
rootDir = await createRoot(task.id);
|
||||
const harness = createMutableStore(task, {}, { plan: planDocument });
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
let retryPrompt = "";
|
||||
|
||||
mockPromptWithFallback
|
||||
.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
})
|
||||
.mockImplementationOnce(async (_session: unknown, prompt: string) => {
|
||||
retryPrompt = prompt;
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
expect(harness.currentTask.status).toBe("needs-replan");
|
||||
expect(harness.store.logEntry).toHaveBeenCalledWith(
|
||||
task.id,
|
||||
"Triage stuck re-queue will resume existing planning draft",
|
||||
expect.stringContaining("Resume from the existing draft"),
|
||||
);
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(retryPrompt).toContain("Revise this task");
|
||||
expect(retryPrompt).toContain("## Existing Specification");
|
||||
expect(retryPrompt).toContain(planDocument);
|
||||
expect(retryPrompt).toContain("instead of restarting planning from scratch");
|
||||
});
|
||||
|
||||
it("prefers PROMPT.md over the plan task document when both drafts exist", async () => {
|
||||
const promptDraft = "# Prompt draft\n\n## Mission\n\nPrefer the executable prompt draft.";
|
||||
const planDocument = "# Plan document draft\n\nThis older plan document should not be the seed.";
|
||||
const task = createTask({ id: "FN-7173-PROMPT-WINS" });
|
||||
rootDir = await createRoot(task.id, promptDraft);
|
||||
const harness = createMutableStore(task, {}, { plan: planDocument });
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
let retryPrompt = "";
|
||||
|
||||
mockPromptWithFallback
|
||||
.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
})
|
||||
.mockImplementationOnce(async (_session: unknown, prompt: string) => {
|
||||
retryPrompt = prompt;
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(retryPrompt).toContain(promptDraft);
|
||||
expect(retryPrompt).not.toContain(planDocument);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["absent", undefined],
|
||||
["whitespace-only", " \n\t "],
|
||||
])("preserves cold-start behavior when the draft is %s", async (_label, draft) => {
|
||||
const task = createTask({ id: `FN-7173-${_label}` });
|
||||
rootDir = await createRoot(task.id, draft);
|
||||
const harness = createMutableStore(task);
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
let retryPrompt = "";
|
||||
|
||||
mockPromptWithFallback
|
||||
.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
})
|
||||
.mockImplementationOnce(async (_session: unknown, prompt: string) => {
|
||||
retryPrompt = prompt;
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
expect(harness.currentTask.status ?? null).toBeNull();
|
||||
expect(harness.store.logEntry).not.toHaveBeenCalledWith(
|
||||
task.id,
|
||||
"Triage stuck re-queue will resume existing planning draft",
|
||||
expect.anything(),
|
||||
);
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(retryPrompt).toContain("Specify this task");
|
||||
expect(retryPrompt).not.toContain("## Existing Specification");
|
||||
});
|
||||
|
||||
it("uses the same resume behavior for the outer catch stuck-abort path", async () => {
|
||||
const draft = "# Task: FN-7173-CATCH\n\n## Mission\n\nCatch path draft.";
|
||||
const task = createTask({ id: "FN-7173-CATCH" });
|
||||
rootDir = await createRoot(task.id, draft);
|
||||
const harness = createMutableStore(task);
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
let retryPrompt = "";
|
||||
|
||||
mockPromptWithFallback
|
||||
.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
throw new Error("disposed by stuck detector");
|
||||
})
|
||||
.mockImplementationOnce(async (_session: unknown, prompt: string) => {
|
||||
retryPrompt = prompt;
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
expect(harness.currentTask.status).toBe("needs-replan");
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(retryPrompt).toContain("Revise this task");
|
||||
expect(retryPrompt).toContain(draft);
|
||||
});
|
||||
|
||||
it("bounds repeated stuck retries by maxStuckKills and pauses failed tasks", async () => {
|
||||
const task = createTask({ id: "FN-7173-BOUND", stuckKillCount: 1 });
|
||||
rootDir = await createRoot(task.id, "# Task: FN-7173-BOUND\n\n## Mission\n\nDraft.");
|
||||
const harness = createMutableStore(task, { maxStuckKills: 2 });
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
|
||||
mockPromptWithFallback.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(harness.currentTask.stuckKillCount).toBe(2);
|
||||
expect(harness.currentTask.status).toBe("failed");
|
||||
expect(harness.currentTask.paused).toBe(true);
|
||||
expect(harness.currentTask.error).toContain("STUCK_LOOP_EXHAUSTED");
|
||||
});
|
||||
|
||||
it("leaves already-approved drafts on the approved-spec recovery path", async () => {
|
||||
const task = createTask({
|
||||
id: "FN-7173-APPROVED",
|
||||
status: "planning",
|
||||
log: [{ timestamp: new Date().toISOString(), action: "Spec review: APPROVE" }],
|
||||
});
|
||||
rootDir = await createRoot(
|
||||
task.id,
|
||||
"# Task: FN-7173-APPROVED\n\n## Mission\n\nApproved draft.\n\n## File Scope\n\n- packages/engine/src/triage.ts\n",
|
||||
);
|
||||
const harness = createMutableStore(task, { requirePlanApproval: false });
|
||||
const processor = new TriageProcessor(harness.store, rootDir);
|
||||
|
||||
mockPromptWithFallback.mockImplementationOnce(async () => {
|
||||
processor.markStuckAborted(task.id);
|
||||
});
|
||||
|
||||
await processor.specifyTask(harness.currentTask);
|
||||
|
||||
expect(harness.store.moveTask).toHaveBeenCalledWith(task.id, "todo");
|
||||
expect(harness.store.logEntry).not.toHaveBeenCalledWith(
|
||||
task.id,
|
||||
"Triage stuck re-queue will resume existing planning draft",
|
||||
expect.anything(),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -3935,7 +3935,7 @@ describe("specifyTask — status restore failure diagnostics", () => {
|
||||
|
||||
await expect(specifyPromise).resolves.toBeUndefined();
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining("FN-002: failed to restore status to 'null' during stuck-detector abort cleanup"),
|
||||
expect.stringContaining("FN-002: failed to restore status to 'null' during stuck-detector in-loop cleanup"),
|
||||
);
|
||||
|
||||
warnSpy.mockRestore();
|
||||
|
||||
@@ -39,6 +39,9 @@ type TaskListFormatter = (
|
||||
opts?: { maxChars?: number; clamp?: TaskListClamp },
|
||||
) => string;
|
||||
|
||||
const TRIAGE_STUCK_RESUME_LOG_ACTION = "Triage stuck re-queue will resume existing planning draft";
|
||||
const TRIAGE_STUCK_RESUME_FEEDBACK = "The previous triage session was killed by the stuck-task detector after writing a non-empty planning draft. Resume from the existing draft below: preserve useful structure and decisions, fill gaps, and continue toward review instead of restarting planning from scratch.";
|
||||
|
||||
export function inlineTaskListFallback(
|
||||
lines: string[],
|
||||
opts: { maxChars?: number } = {},
|
||||
@@ -567,6 +570,127 @@ export class TriageProcessor {
|
||||
return true;
|
||||
}
|
||||
|
||||
private async readNonEmptyPromptDraft(taskId: string, context: string): Promise<string | undefined> {
|
||||
/*
|
||||
FNXC:Triage 2026-06-27-00:00:
|
||||
Stuck triage re-queues prefer a non-empty on-disk PROMPT.md draft. Match scheduler filesystem validation and approved recovery semantics (`trim().length > 0`) so empty or whitespace-only drafts cold-start safely instead of seeding a bogus revision.
|
||||
*/
|
||||
const promptPath = join(this.rootDir, ".fusion", "tasks", taskId, "PROMPT.md");
|
||||
const written = await readFile(promptPath, "utf-8").catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${taskId}: failed to read PROMPT.md during ${context} (${promptPath}): ${msg}`);
|
||||
return "";
|
||||
});
|
||||
return written.trim().length > 0 ? written : undefined;
|
||||
}
|
||||
|
||||
private async readNonEmptyPlanDocument(taskId: string, context: string): Promise<string | undefined> {
|
||||
/*
|
||||
FNXC:Triage 2026-06-27-16:18:
|
||||
Some triage agents persist the draft through fn_task_document_write key="plan" before PROMPT.md exists. Stuck re-queue must still resume from that non-empty plan document when the file draft is absent, while preserving PROMPT.md as the preferred executable draft when both are present.
|
||||
*/
|
||||
const readTaskDocument = (this.store as unknown as { getTaskDocument?: (taskId: string, key: string) => Promise<{ content?: unknown } | null> }).getTaskDocument;
|
||||
if (typeof readTaskDocument !== "function") {
|
||||
return undefined;
|
||||
}
|
||||
const document = await readTaskDocument.call(this.store, taskId, "plan").catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${taskId}: failed to read plan task document during ${context}: ${msg}`);
|
||||
return null;
|
||||
});
|
||||
const content = typeof document?.content === "string" ? document.content : "";
|
||||
return content.trim().length > 0 ? content : undefined;
|
||||
}
|
||||
|
||||
private async readNonEmptyPlanningDraft(taskId: string, context: string): Promise<{ content: string; source: "prompt" | "plan-document" } | undefined> {
|
||||
const promptDraft = await this.readNonEmptyPromptDraft(taskId, context);
|
||||
if (promptDraft) {
|
||||
return { content: promptDraft, source: "prompt" };
|
||||
}
|
||||
const planDocument = await this.readNonEmptyPlanDocument(taskId, context);
|
||||
return planDocument ? { content: planDocument, source: "plan-document" } : undefined;
|
||||
}
|
||||
|
||||
private async handleStuckAbortRequeue(task: Task, context: "in-loop" | "catch"): Promise<void> {
|
||||
/*
|
||||
FNXC:Triage 2026-06-27-00:00:
|
||||
A stuck-killed planning session that already wrote a usable PROMPT.md or plan task document must resume in revision mode on the next poll, not re-triage from scratch. Reuse stuckKillCount and maxStuckKills for the triage retry budget so repeated stuck resumes escalate to manual intervention instead of looping forever.
|
||||
*/
|
||||
const freshTask = await this.store.getTask(task.id).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to refresh task during stuck-detector ${context} cleanup: ${msg}`);
|
||||
return task;
|
||||
});
|
||||
|
||||
if (hasLatestSpecReviewApproval(freshTask)) {
|
||||
const recovered = await this.recoverApprovedTask(freshTask).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: approved-spec recovery failed during stuck-detector ${context} cleanup: ${msg}`);
|
||||
return false;
|
||||
});
|
||||
if (recovered) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const maxStuckSettings = await this.store.getSettings().catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to read maxStuckKills during stuck-detector ${context} cleanup, using default 6: ${msg}`);
|
||||
return {} as Settings;
|
||||
});
|
||||
const maxKills = Math.max(1, maxStuckSettings.maxStuckKills ?? 6);
|
||||
const nextStuckKillCount = (freshTask.stuckKillCount ?? task.stuckKillCount ?? 0) + 1;
|
||||
const draft = await this.readNonEmptyPlanningDraft(task.id, `stuck-detector ${context} cleanup`);
|
||||
|
||||
if (nextStuckKillCount >= maxKills) {
|
||||
const exhaustedError = `STUCK_LOOP_EXHAUSTED: triage stuck detector killed ${task.id} ${nextStuckKillCount}/${maxKills} times without planning completion; task paused for manual intervention.`;
|
||||
planLog.error(exhaustedError);
|
||||
await this.store.logEntry(task.id, exhaustedError).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to log stuck-loop exhaustion: ${msg}`);
|
||||
});
|
||||
await this.store.updateTask(task.id, {
|
||||
stuckKillCount: nextStuckKillCount,
|
||||
status: "failed",
|
||||
error: exhaustedError,
|
||||
paused: true,
|
||||
pausedReason: "stuck-loop-exhausted-manual-intervention-required",
|
||||
pausedByAgentId: "triage",
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to persist stuck-loop exhaustion during stuck-detector ${context} cleanup: ${msg}`);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (draft) {
|
||||
const sourceLabel = draft.source === "prompt" ? "PROMPT.md draft" : "plan task document";
|
||||
planLog.log(`${task.id} killed by stuck detector — requeueing to resume existing ${sourceLabel} (${nextStuckKillCount}/${maxKills})`);
|
||||
await this.store.logEntry(task.id, TRIAGE_STUCK_RESUME_LOG_ACTION, TRIAGE_STUCK_RESUME_FEEDBACK).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to log stuck-resume feedback: ${msg}`);
|
||||
});
|
||||
await this.store.updateTask(task.id, {
|
||||
status: "needs-replan",
|
||||
stuckKillCount: nextStuckKillCount,
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to restore status to 'needs-replan' during stuck-detector ${context} cleanup: ${msg}`);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
planLog.log(`${task.id} killed by stuck detector — clearing status for cold retry (${nextStuckKillCount}/${maxKills})`);
|
||||
const restoreStatus = (freshTask.status ?? task.status) === "needs-replan" ? "needs-replan" : null;
|
||||
await this.store.updateTask(task.id, {
|
||||
status: restoreStatus,
|
||||
stuckKillCount: nextStuckKillCount,
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to restore status to '${restoreStatus}' during stuck-detector ${context} cleanup: ${msg}`);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* If `newIntervalMs` differs from the currently active timer, restart
|
||||
* the `setInterval` so the new cadence takes effect immediately.
|
||||
@@ -1046,9 +1170,22 @@ export class TriageProcessor {
|
||||
entry.action === "User comment requested re-specification of planned task"
|
||||
|| entry.action === "User comment invalidated spec approval — task needs re-specification"
|
||||
|| entry.action === "AI spec revision requested"
|
||||
|| entry.action === TRIAGE_STUCK_RESUME_LOG_ACTION
|
||||
);
|
||||
feedback = feedbackLogEntry?.outcome;
|
||||
|
||||
if (feedbackLogEntry?.action === TRIAGE_STUCK_RESUME_LOG_ACTION) {
|
||||
/*
|
||||
FNXC:Triage 2026-06-27-16:18:
|
||||
Stuck-resume replans must load the existing PROMPT.md draft, or the saved plan task document when PROMPT.md is absent, into buildSpecificationPrompt so `isRevision` is reachable for either persisted planning surface.
|
||||
*/
|
||||
const planningDraft = await this.readNonEmptyPlanningDraft(task.id, "stuck-resume replan seed");
|
||||
existingPrompt = planningDraft?.content;
|
||||
if (!existingPrompt) {
|
||||
feedback = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the latest user feedback is always actionable for re-plans.
|
||||
if (!feedback) {
|
||||
const latestUserComment = [...(detail.comments || [])]
|
||||
@@ -1092,12 +1229,7 @@ export class TriageProcessor {
|
||||
|
||||
if (this.stuckAborted.has(task.id)) {
|
||||
this.stuckAborted.delete(task.id);
|
||||
planLog.log(`${task.id} killed by stuck detector — clearing status for retry`);
|
||||
const restoreStatus = task.status === "needs-replan" ? "needs-replan" : null;
|
||||
await this.store.updateTask(task.id, { status: restoreStatus }).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to restore status to '${restoreStatus}' during stuck-detector abort cleanup: ${msg}`);
|
||||
});
|
||||
await this.handleStuckAbortRequeue(task, "in-loop");
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1424,15 +1556,8 @@ export class TriageProcessor {
|
||||
planLog.warn(`${task.id}: failed to restore status to '${restoreStatus}' during pause-abort error cleanup: ${msg}`);
|
||||
});
|
||||
} else if (this.stuckAborted.has(task.id)) {
|
||||
// Stuck task detector killed this session — clear planning status so the
|
||||
// next poll retries the task from scratch without reporting an error.
|
||||
this.stuckAborted.delete(task.id);
|
||||
planLog.log(`${task.id} killed by stuck detector — clearing status for retry`);
|
||||
const restoreStatus = task.status === "needs-replan" ? "needs-replan" : null;
|
||||
await this.store.updateTask(task.id, { status: restoreStatus }).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
planLog.warn(`${task.id}: failed to restore status to '${restoreStatus}' during stuck-detector error cleanup: ${msg}`);
|
||||
});
|
||||
await this.handleStuckAbortRequeue(task, "catch");
|
||||
} else {
|
||||
// Check if the error is a usage-limit error and trigger global pause
|
||||
if (this.options.usageLimitPauser && isUsageLimitError(errorMessage)) {
|
||||
|
||||
Reference in New Issue
Block a user