fix(FN-7965): let the overseer see executor-stage failures
`deriveSignalAndSources`'s executor branch never read `task.status`, so a row parked `status: "failed"` — e.g. the terminal fn_task_done refusal/invariant park — reported `signal: "progressing"` with the reason "Task is actively executing in-progress work". The overseer observed a dead task as healthy and took no action. `failed` was only ever derived for the merger/pull-request stages, so the sole backstop was the FN-7743 2h stall proxy firing hours later. This is exactly what FN-7965's audit trail shows: every intervention on a terminally-parked task was action="observe", reason="Task is actively executing in-progress work". Report `failed` so recovery engages on the next poll. This adds no new policy: a failed executor observation already routes to `retry_step` (executor sources are `agent-log`, never an ERROR_SOURCE_KIND), bounded by PLANNER_RECOVERY_MAX_ATTEMPTS and escalated on exhaustion. Precedence and dedup preserved: `paused` still wins, so an operator/user-paused row stays `blocked` and is never routed into autonomous recovery; and the reason is a constant (never interpolating task.error/status) so the FN-7577 `stage|signal|reason` feed dedup still suppresses repeat observations. Verified: the repro test fails with the branch disabled; paused-precedence, healthy-card (FN-7577) and dedup guards added; overseer/recovery surfaces 93 passed + core planner-recovery 20/20; engine + dashboard typecheck clean; `pnpm test:gate` green (294+122+63). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
7
.changeset/fn-7965-overseer-executor-failure-signal.md
Normal file
7
.changeset/fn-7965-overseer-executor-failure-signal.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: The planner overseer now notices a failed in-progress task immediately instead of after two hours.
|
||||
category: fix
|
||||
dev: FN-7965. `deriveSignalAndSources`'s `executor` branch never read `task.status`, so a row parked `status: "failed"` (e.g. the terminal fn_task_done refusal/invariant park) reported `signal: "progressing"` with reason "Task is actively executing in-progress work". `failed` was only ever derived for the merger/pull-request stages, leaving the FN-7743 2h stall proxy (`columnMovedAt ?? updatedAt`) as the sole backstop. The branch now reports `failed`, which routes into the pre-existing failed-signal policy — `retry_step` at executor stage (sources are `agent-log`, never an ERROR_SOURCE_KIND), bounded by `PLANNER_RECOVERY_MAX_ATTEMPTS` and escalated on exhaustion. `paused` keeps precedence (operator/user-paused stays `blocked`), the reason is held constant so the FN-7577 `stage|signal|reason` feed dedup still holds, and `status` was added to `OverseerTaskRef`.
|
||||
@@ -464,3 +464,73 @@ describe("PlannerOverseerMonitor.observeTask — FN-7743 executor stall detectio
|
||||
expect(store.logEntry).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
|
||||
// FNXC:PlannerOversight 2026-07-15-17:05:
|
||||
// FN-7965: a task parked `status: "failed"` at the executor stage reported
|
||||
// `progressing` ("Task is actively executing in-progress work") because the
|
||||
// derivation never read `status` — so the overseer treated a dead task as healthy
|
||||
// and only reacted once the FN-7743 stall proxy fired hours later. `failed` had
|
||||
// been derived for the merger/pull-request stages only. These tests lock the
|
||||
// invariant across the enumerated surfaces: failed (failed), failed-but-paused
|
||||
// (blocked — a human owns it), healthy (progressing, unchanged), and the
|
||||
// terminal columns that must stay unmonitored.
|
||||
describe("PlannerOverseerMonitor.observeTask — FN-7965 executor failure detection", () => {
|
||||
it("reports failed for a non-paused in-progress task parked status=failed", async () => {
|
||||
const monitor = new PlannerOverseerMonitor();
|
||||
const task = taskFixture({
|
||||
column: "in-progress",
|
||||
status: "failed",
|
||||
updatedAt: new Date().toISOString(),
|
||||
columnMovedAt: new Date().toISOString(),
|
||||
});
|
||||
|
||||
const observation = await monitor.observeTask(task, "autonomous");
|
||||
|
||||
// Must NOT wait for the 2h stall proxy: the failure is visible immediately.
|
||||
expect(observation?.stage).toBe("executor");
|
||||
expect(observation?.signal).toBe("failed");
|
||||
});
|
||||
|
||||
it("keeps paused precedence: a paused failed task is blocked, not failed", async () => {
|
||||
// A human owns an operator/user-paused row — it must never be routed into
|
||||
// autonomous bounded recovery just because status is failed.
|
||||
const monitor = new PlannerOverseerMonitor();
|
||||
const task = taskFixture({
|
||||
column: "in-progress",
|
||||
status: "failed",
|
||||
paused: true,
|
||||
pausedReason: "error-unrecoverable",
|
||||
});
|
||||
|
||||
const observation = await monitor.observeTask(task, "autonomous");
|
||||
|
||||
expect(observation?.signal).toBe("blocked");
|
||||
});
|
||||
|
||||
it.each([undefined, null, "queued"])("still reports progressing for a healthy in-progress task (status=%s)", async (status) => {
|
||||
// Negative control: FN-7577 — a healthy card must never flip to a problem
|
||||
// signal, or every in-progress task shows the "recovering" badge.
|
||||
const monitor = new PlannerOverseerMonitor();
|
||||
const task = taskFixture({
|
||||
column: "in-progress",
|
||||
status: status as never,
|
||||
updatedAt: new Date().toISOString(),
|
||||
columnMovedAt: new Date().toISOString(),
|
||||
});
|
||||
|
||||
const observation = await monitor.observeTask(task, "autonomous");
|
||||
|
||||
expect(observation?.signal).toBe("progressing");
|
||||
});
|
||||
|
||||
it("keeps the failed reason constant so the FN-7577 feed dedup holds", async () => {
|
||||
// The reason must not interpolate task.error/status, or an observation would
|
||||
// be re-emitted every poll for the same unchanged failure.
|
||||
const monitor = new PlannerOverseerMonitor();
|
||||
const a = await monitor.observeTask(taskFixture({ id: "FN-1000", status: "failed" }), "autonomous");
|
||||
const b = await monitor.observeTask(taskFixture({ id: "FN-1000", status: "failed" }), "autonomous");
|
||||
|
||||
expect(a?.reason).toBe(b?.reason);
|
||||
expect(a?.reason).not.toMatch(/failed:|error/i);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -55,6 +55,7 @@ export type OverseerTaskRef = Pick<
|
||||
Task,
|
||||
| "id"
|
||||
| "column"
|
||||
| "status"
|
||||
| "prInfo"
|
||||
| "reviewState"
|
||||
| "paused"
|
||||
@@ -180,6 +181,21 @@ function deriveSignalAndSources(
|
||||
};
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:PlannerOversight 2026-07-15-17:05:
|
||||
FN-7965: an executor-stage row parked `status: "failed"` (e.g. the terminal fn_task_done refusal/invariant park) reported `progressing` — "Task is actively executing in-progress work" — because nothing here read `status`. The overseer observed a dead task as healthy and took no action; `failed` was only ever derived for the merger/pull-request stages, so the sole backstop was the FN-7743 stall proxy below firing HOURS later. Read the status so bounded recovery engages on the next poll instead.
|
||||
`paused` deliberately keeps precedence above: an operator/user-paused row is `blocked` because a human owns it, not an autonomously recoverable failure.
|
||||
Downstream this yields `retry_step` (executor sources are `agent-log`, never an ERROR_SOURCE_KIND), bounded by `PLANNER_RECOVERY_MAX_ATTEMPTS` and then escalated on exhaustion — the pre-existing failed-signal policy, not a new one.
|
||||
The reason must stay CONSTANT per (stage|signal): the FN-7577 feed dedup keys on `stage|signal|reason`, so never interpolate `task.error`/`status` here — a per-failure string would re-emit an observation every poll.
|
||||
*/
|
||||
if (task.status === "failed") {
|
||||
return {
|
||||
signal: "failed",
|
||||
reason: "Executor stage parked failed with work incomplete",
|
||||
sources: [{ kind: "agent-log", ref: taskId }],
|
||||
};
|
||||
}
|
||||
|
||||
// FNXC:PlannerOversight 2026-07-09-00:00:
|
||||
// FN-7743: a non-paused in-progress task whose executor session has gone
|
||||
// silent (dead/hung agent, no commits/heartbeat) was previously ALWAYS
|
||||
|
||||
Reference in New Issue
Block a user