FN-7743: detect and recover stalled in-progress executor tasks in overseer
Fix the planner overseer's executor-stage stall detection so hung in-progress tasks get unstuck instead of being reported as progressing forever. - Add configurable stuck-detection: the executor-stage overseer observation now emits signal: "stuck" once an in-progress task has been inactive past a new plannerOverseerExecutorStuckAfterMs threshold, feeding the existing decidePlannerRecovery -> bounded inject_guidance recovery path. - Register the new plannerOverseerExecutorStuckAfterMs setting in builtin-workflow-settings.ts and export it via core index.ts/index.gate.ts. - Preserve human-control withholds (user-paused / approval-blocked / autoMerge-off) taking precedence over stuck detection. - Add/extend tests covering planner-overseer, planner-recovery-controller, planner-recovery, and builtin-workflow-settings-triage. - Document the new setting in docs/architecture.md and docs/settings-reference.md. - Add changeset fn-7743-overseer-executor-stall.md (patch). Files changed: .changeset/fn-7743-overseer-executor-stall.md | 7 + docs/architecture.md | 31 +++++ docs/settings-reference.md | 3 +- .../builtin-workflow-settings-triage.test.ts | 22 ++++ .../core/src/__tests__/planner-recovery.test.ts | 12 ++ packages/core/src/builtin-workflow-settings.ts | 26 ++++ packages/core/src/index.gate.ts | 1 + packages/core/src/index.ts | 1 + .../engine/src/__tests__/planner-overseer.test.ts | 146 +++++++++++++++++++++ .../__tests__/planner-recovery-controller.test.ts | 52 +++++++- packages/engine/src/planner-overseer.ts | 88 ++++++++++++- packages/engine/src/project-engine.ts | 11 +- 12 files changed, 391 insertions(+), 9 deletions(-) Fusion-Task-Id: FN-7743 Fusion-Task-Lineage: 791852a2-5b77-48de-955a-1b3916616dfa Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
7
.changeset/fn-7743-overseer-executor-stall.md
Normal file
7
.changeset/fn-7743-overseer-executor-stall.md
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
summary: The planner overseer now detects and recovers stalled in-progress tasks instead of leaving hung executors stuck.
|
||||||
|
category: fix
|
||||||
|
dev: FN-7743 — the executor-stage overseer observation now emits `signal: "stuck"` once an in-progress task has been inactive past a configurable threshold (`plannerOverseerExecutorStuckAfterMs`), feeding the existing `decidePlannerRecovery` → bounded `inject_guidance` path. Previously a non-paused in-progress task was always reported `progressing`, so a hung executor was never recovered. Human-control withholds (user-paused / approval-blocked / autoMerge-off) still take precedence.
|
||||||
@@ -1386,6 +1386,37 @@ call happens here, and it emits no run-audit events or dashboard UI. Steering/re
|
|||||||
gates, human-control safeguards, and dashboard/UI/run-audit surfaces are deferred to FN-7512 through
|
gates, human-control safeguards, and dashboard/UI/run-audit surfaces are deferred to FN-7512 through
|
||||||
FN-7520; this module is the seam those subtasks read observations from.
|
FN-7520; this module is the seam those subtasks read observations from.
|
||||||
|
|
||||||
|
#### Executor-stage stall detection (FN-7743)
|
||||||
|
|
||||||
|
/*
|
||||||
|
FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
FN-7743 requirement: an ordinary in-progress task (FN-7732) sat stuck for hours with no recovery action
|
||||||
|
because `deriveSignalAndSources`'s `case "executor"` had NO staleness check — a non-paused in-progress
|
||||||
|
task always reported `signal: "progressing"` regardless of how long it had been idle, so
|
||||||
|
`decidePlannerRecovery` always returned `action: "none"`.
|
||||||
|
*/
|
||||||
|
|
||||||
|
The executor stage now detects a stalled/idle non-paused `in-progress` task and reports `signal: "stuck"`
|
||||||
|
instead of always `"progressing"`. `PlannerOverseerMonitor#observeTask` accepts an optional
|
||||||
|
`{ now?: () => number; executorStuckAfterMs?: number }` (both default: `Date.now` and the declared
|
||||||
|
`plannerOverseerExecutorStuckAfterMs` workflow-setting default) and threads them into the still-pure,
|
||||||
|
still-never-reads-settings `deriveSignalAndSources`. The staleness input is `task.columnMovedAt ??
|
||||||
|
task.updatedAt` (mirroring the existing "age is measured from columnMovedAt when present, otherwise
|
||||||
|
updatedAt" convention already used by `stalePausedReviewThresholdMs`/`stalePausedTodoThresholdMs`) — the
|
||||||
|
best available store-backed proxy for "last execution activity" at this poll seam, since the live
|
||||||
|
in-session `StuckTaskDetector` heartbeat state (`packages/engine/src/stuck-task-detector.ts`) is
|
||||||
|
in-memory-only and unavailable here. A missing or unparseable timestamp degrades to `"progressing"`
|
||||||
|
(fail-safe — never fabricate a stall). The `stuck` reason buckets inactivity to whole hours so the FN-7577
|
||||||
|
`stage|signal|reason` feed dedup stays effective (it must never embed an ever-changing millisecond value).
|
||||||
|
|
||||||
|
`ProjectEngine.pollPlannerOverseer` resolves `plannerOverseerExecutorStuckAfterMs` from the task's
|
||||||
|
already-fetched effective workflow settings (`resolveExecutorStuckAfterMs`, fail-safe default 2 hours) once
|
||||||
|
per task per poll cycle — the same `workflowEffective` fetch already used for `plannerOversightLevel` — and
|
||||||
|
passes it into `observeTask`. `decidePlannerRecovery` already mapped executor `stuck` → `inject_guidance`
|
||||||
|
(no change needed there); the human-control withhold guard in `evaluateOverseerHumanControl` still runs
|
||||||
|
BEFORE any recovery decision, so a stuck-but-user-paused/approval-blocked/autoMerge-off task is still fully
|
||||||
|
withheld.
|
||||||
|
|
||||||
### Planner overseer bounded autonomous recovery (FN-7512)
|
### Planner overseer bounded autonomous recovery (FN-7512)
|
||||||
|
|
||||||
/*
|
/*
|
||||||
|
|||||||
@@ -327,7 +327,7 @@ These groups moved out of project settings and into workflow settings (built-in
|
|||||||
|---|---|
|
|---|---|
|
||||||
| **Step execution** | `workflowStepTimeoutMs`, `runStepsInNewSessions`, `maxParallelSteps`, `workflowStepScopeEnforcement`, `strictScopeEnforcement`, `verificationFixRetries`, `maxPostReviewFixes`, `buildRetryCount` |
|
| **Step execution** | `workflowStepTimeoutMs`, `runStepsInNewSessions`, `maxParallelSteps`, `workflowStepScopeEnforcement`, `strictScopeEnforcement`, `verificationFixRetries`, `maxPostReviewFixes`, `buildRetryCount` |
|
||||||
| **Review / approval** | Workflow values: `requirePrApproval`, `requirePlanApproval`, `reviewHandoffPolicy`, `maxReviewerContextRetries`, `maxReviewerFallbackRetries`, `planReviewMaxRevisions`, `codeReviewMaxRevisions`; project override: `planApprovalMode` |
|
| **Review / approval** | Workflow values: `requirePrApproval`, `requirePlanApproval`, `reviewHandoffPolicy`, `maxReviewerContextRetries`, `maxReviewerFallbackRetries`, `planReviewMaxRevisions`, `codeReviewMaxRevisions`; project override: `planApprovalMode` |
|
||||||
| **Planner oversight** | `plannerOversightLevel` (workflow-native; values: `off`, `observe`, `steer`, `autonomous`); `plannerOversightNotificationLevel` (workflow-native; values: `silent`, `errors`, `important`, `all`) |
|
| **Planner oversight** | `plannerOversightLevel` (workflow-native; values: `off`, `observe`, `steer`, `autonomous`); `plannerOversightNotificationLevel` (workflow-native; values: `silent`, `errors`, `important`, `all`); `plannerOverseerExecutorStuckAfterMs` (workflow-native; number, default `7200000` = 2h) |
|
||||||
| **Per-phase model lanes** | `executionProvider`/`executionModelId`, `planningProvider`/`planningModelId` (+ fallbacks), `validatorProvider`/`validatorModelId` (+ fallbacks) |
|
| **Per-phase model lanes** | `executionProvider`/`executionModelId`, `planningProvider`/`planningModelId` (+ fallbacks), `validatorProvider`/`validatorModelId` (+ fallbacks) |
|
||||||
|
|
||||||
### Workflow-native triage policy settings
|
### Workflow-native triage policy settings
|
||||||
@@ -363,6 +363,7 @@ The built-in workflows also declare triage/spec policy settings that were **not*
|
|||||||
| `codeReviewMaxRevisions` | unset | Workflow-native Code Review remediation cap. Unset/empty means unbounded automatic code-fix passes; a non-negative integer caps attempts; `0` disables automatic Code Review remediation. |
|
| `codeReviewMaxRevisions` | unset | Workflow-native Code Review remediation cap. Unset/empty means unbounded automatic code-fix passes; a non-negative integer caps attempts; `0` disables automatic Code Review remediation. |
|
||||||
| `plannerOversightLevel` | `autonomous` | Workflow-native planner oversight mode. `off` disables oversight; `observe` watches only; `steer` injects guidance or suggests revisions; `autonomous` enables bounded retry and targeted-fix recovery — but merge/PR progression and any destructive or external-service side effect ALWAYS require an explicit, recorded human confirmation before they run, even at `autonomous` (FN-7513's confirmation gate; see `docs/architecture.md` → "Planner overseer confirmation gate"). Tasks may set a nullable `Task.plannerOversightLevel` override (same four values) that wins over this workflow value when present; `null`/unset means "inherit the workflow value". `resolveEffectivePlannerOversightLevel` in `@fusion/core` computes the effective level (task override → workflow effective → `autonomous`). The per-task override is exposed in the dashboard as a "Planner oversight" selector (Inherit from workflow / Off / Observe / Steer / Autonomous recovery) in both the New Task dialog and Task Detail edit form, threaded through `createTask`/`updateTask` (FN-7515); the project/global default is set via the **Workflow Editor → Values** tab on the default workflow's `plannerOversightLevel` value, not in Project Settings. FN-7517 additionally exposes a quick inline oversight-level select in the Task Detail modal's meta-controls cluster (same `updateTask` override plumbing, no parallel path) plus manual nudge/stop-oversight/explain-current-action controls that call the overseer runtime directly — see `docs/dashboard-guide.md`. Engine read-site behavior beyond the FN-7513 confirmation gate remains follow-up work (FN-7510+). |
|
| `plannerOversightLevel` | `autonomous` | Workflow-native planner oversight mode. `off` disables oversight; `observe` watches only; `steer` injects guidance or suggests revisions; `autonomous` enables bounded retry and targeted-fix recovery — but merge/PR progression and any destructive or external-service side effect ALWAYS require an explicit, recorded human confirmation before they run, even at `autonomous` (FN-7513's confirmation gate; see `docs/architecture.md` → "Planner overseer confirmation gate"). Tasks may set a nullable `Task.plannerOversightLevel` override (same four values) that wins over this workflow value when present; `null`/unset means "inherit the workflow value". `resolveEffectivePlannerOversightLevel` in `@fusion/core` computes the effective level (task override → workflow effective → `autonomous`). The per-task override is exposed in the dashboard as a "Planner oversight" selector (Inherit from workflow / Off / Observe / Steer / Autonomous recovery) in both the New Task dialog and Task Detail edit form, threaded through `createTask`/`updateTask` (FN-7515); the project/global default is set via the **Workflow Editor → Values** tab on the default workflow's `plannerOversightLevel` value, not in Project Settings. FN-7517 additionally exposes a quick inline oversight-level select in the Task Detail modal's meta-controls cluster (same `updateTask` override plumbing, no parallel path) plus manual nudge/stop-oversight/explain-current-action controls that call the overseer runtime directly — see `docs/dashboard-guide.md`. Engine read-site behavior beyond the FN-7513 confirmation gate remains follow-up work (FN-7510+). |
|
||||||
| `plannerOversightNotificationLevel` | `important` | Workflow-native planner-overseer notification verbosity (FN-7518). `silent` suppresses overseer notifications; `errors` notifies only on failures/escalations; `important` (the default) notifies on interventions/recovery actions and errors; `all` notifies on every observation. Resolves through the generic `resolveEffectiveSettings` default path with no special-casing, alongside `plannerOversightLevel`. This is a declaration-only setting: the notification-emission gating that reads it lands downstream in FN-7519 (intervention timeline) and FN-7520 (run-audit/activity events). |
|
| `plannerOversightNotificationLevel` | `important` | Workflow-native planner-overseer notification verbosity (FN-7518). `silent` suppresses overseer notifications; `errors` notifies only on failures/escalations; `important` (the default) notifies on interventions/recovery actions and errors; `all` notifies on every observation. Resolves through the generic `resolveEffectiveSettings` default path with no special-casing, alongside `plannerOversightLevel`. This is a declaration-only setting: the notification-emission gating that reads it lands downstream in FN-7519 (intervention timeline) and FN-7520 (run-audit/activity events). |
|
||||||
|
| `plannerOverseerExecutorStuckAfterMs` | `7200000` (2h) | Workflow-native executor-stage stall threshold (FN-7743). Milliseconds of executor-stage inactivity — no execution activity since the task's last column move/update (`columnMovedAt ?? updatedAt`) — before a non-paused `in-progress` task is reported `signal: "stuck"` instead of `"progressing"`, feeding the existing `decidePlannerRecovery` → bounded `inject_guidance` recovery path at the `autonomous` oversight level (no effect at `off`/`observe`/`steer`). Fixes the class of bug where a genuinely hung/idle executor (dead session, silent agent) was indistinguishable from a healthy one and was never nudged, retried, or escalated. A missing/malformed activity timestamp degrades to `"progressing"` (fail-safe — never fabricates a stall), and a user-paused/approval-blocked/`autoMerge:false` task is still fully withheld from any autonomous action regardless of this threshold. Resolves through the generic `resolveEffectiveSettings` default path alongside `plannerOversightLevel`. See `docs/architecture.md` → "Executor-stage stall detection (FN-7743)". |
|
||||||
|
|
||||||
When `triageProactiveSubtaskSplittingEnabled` is `true` (the default), triage may proactively replace a large task with 2-5 child tasks when the size, step-count, package breadth, file-scope, or remediation-batch signals justify the coordination overhead. When it is `false`, those automatic oversized-task signals are advisory only for writing a realistic single-task spec; triage must not split solely because the task is large. The per-task `breakIntoSubtasks: true` flag is separate and remains mandatory: if a user explicitly asks for subtask breakdown, triage still evaluates and creates child tasks when the work is meaningfully decomposable.
|
When `triageProactiveSubtaskSplittingEnabled` is `true` (the default), triage may proactively replace a large task with 2-5 child tasks when the size, step-count, package breadth, file-scope, or remediation-batch signals justify the coordination overhead. When it is `false`, those automatic oversized-task signals are advisory only for writing a realistic single-task spec; triage must not split solely because the task is large. The per-task `breakIntoSubtasks: true` flag is separate and remains mandatory: if a user explicitly asks for subtask breakdown, triage still evaluates and creates child tasks when the work is meaningfully decomposable.
|
||||||
|
|
||||||
|
|||||||
@@ -89,6 +89,7 @@ describe("workflow-native built-in workflow settings", () => {
|
|||||||
expect(BUILTIN_OVERSIGHT_SETTINGS.map((setting) => setting.id)).toEqual([
|
expect(BUILTIN_OVERSIGHT_SETTINGS.map((setting) => setting.id)).toEqual([
|
||||||
"plannerOversightLevel",
|
"plannerOversightLevel",
|
||||||
"plannerOversightNotificationLevel",
|
"plannerOversightNotificationLevel",
|
||||||
|
"plannerOverseerExecutorStuckAfterMs",
|
||||||
]);
|
]);
|
||||||
const oversight = BUILTIN_OVERSIGHT_SETTINGS[0];
|
const oversight = BUILTIN_OVERSIGHT_SETTINGS[0];
|
||||||
expect(oversight).toMatchObject({
|
expect(oversight).toMatchObject({
|
||||||
@@ -144,6 +145,27 @@ describe("workflow-native built-in workflow settings", () => {
|
|||||||
movedKeyIds.has("plannerOversightNotificationLevel"),
|
movedKeyIds.has("plannerOversightNotificationLevel"),
|
||||||
"plannerOversightNotificationLevel should not be in MOVED_SETTINGS_KEYS",
|
"plannerOversightNotificationLevel should not be in MOVED_SETTINGS_KEYS",
|
||||||
).toBe(false);
|
).toBe(false);
|
||||||
|
|
||||||
|
// FN-7743: executor-stall recovery threshold, declared alongside the other
|
||||||
|
// workflow-native oversight settings.
|
||||||
|
const executorStuckAfterMs = BUILTIN_OVERSIGHT_SETTINGS[2];
|
||||||
|
expect(executorStuckAfterMs).toMatchObject({
|
||||||
|
id: "plannerOverseerExecutorStuckAfterMs",
|
||||||
|
type: "number",
|
||||||
|
default: 2 * 60 * 60 * 1000,
|
||||||
|
});
|
||||||
|
expect(
|
||||||
|
fullIds.has("plannerOverseerExecutorStuckAfterMs"),
|
||||||
|
"plannerOverseerExecutorStuckAfterMs should be in the full built-in catalog",
|
||||||
|
).toBe(true);
|
||||||
|
expect(
|
||||||
|
movedIds.has("plannerOverseerExecutorStuckAfterMs"),
|
||||||
|
"plannerOverseerExecutorStuckAfterMs should not be in the moved-key catalog",
|
||||||
|
).toBe(false);
|
||||||
|
expect(
|
||||||
|
movedKeyIds.has("plannerOverseerExecutorStuckAfterMs"),
|
||||||
|
"plannerOverseerExecutorStuckAfterMs should not be in MOVED_SETTINGS_KEYS",
|
||||||
|
).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("renders placeholders from resolved settings and rejects dangling tokens", () => {
|
it("renders placeholders from resolved settings and rejects dangling tokens", () => {
|
||||||
|
|||||||
@@ -212,4 +212,16 @@ describe("decidePlannerRecovery", () => {
|
|||||||
expect(decision.exhausted).toBe(true);
|
expect(decision.exhausted).toBe(true);
|
||||||
expect(decision.attemptLimit).toBe(1);
|
expect(decision.attemptLimit).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// FN-7743 invariant lock: a stalled non-paused in-progress task (the FN-7732
|
||||||
|
// symptom) is surfaced by the overseer as `stage: "executor", signal: "stuck"`.
|
||||||
|
// This asserts the downstream mapping this fix depends on — executor `stuck`
|
||||||
|
// → `inject_guidance`, not `none` — already holds and stays locked, so a
|
||||||
|
// future regression here is caught even though FN-7743 itself only changes
|
||||||
|
// the observation INPUT, never this mapping.
|
||||||
|
it("FN-7743: maps an executor stuck signal to inject_guidance (the hung-executor recovery path)", () => {
|
||||||
|
const decision = decidePlannerRecovery({ snapshot: observation({ stage: "executor", signal: "stuck" }) });
|
||||||
|
expect(decision.action).toBe("inject_guidance");
|
||||||
|
expect(decision.requiresConfirmation).toBe(false);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -439,6 +439,24 @@ export const BUILTIN_REVIEW_REVISION_SETTINGS: WorkflowSettingDefinition[] = [
|
|||||||
* FNXC:PlannerOversight 2026-07-04-12:00:
|
* FNXC:PlannerOversight 2026-07-04-12:00:
|
||||||
* FN-7518 adds `plannerOversightNotificationLevel`, a sibling workflow-native enum letting operators configure how noisy planner-overseer notifications are: Silent suppresses all, Errors only notifies on failures/escalations, Important (the default) notifies on interventions/recovery actions plus errors, and All notifies on every observation. Default is `important` (not `all`) to avoid noisy-by-default behavior. This setting stays workflow-native (out of project/global settings schemas and `MOVED_SETTINGS_KEYS`) and resolves through the generic `resolveEffectiveSettings` default path with no special-casing. This task only declares the setting — the emission gating that reads it lands downstream in FN-7519 (intervention timeline) and FN-7520 (run-audit/activity events).
|
* FN-7518 adds `plannerOversightNotificationLevel`, a sibling workflow-native enum letting operators configure how noisy planner-overseer notifications are: Silent suppresses all, Errors only notifies on failures/escalations, Important (the default) notifies on interventions/recovery actions plus errors, and All notifies on every observation. Default is `important` (not `all`) to avoid noisy-by-default behavior. This setting stays workflow-native (out of project/global settings schemas and `MOVED_SETTINGS_KEYS`) and resolves through the generic `resolveEffectiveSettings` default path with no special-casing. This task only declares the setting — the emission gating that reads it lands downstream in FN-7519 (intervention timeline) and FN-7520 (run-audit/activity events).
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
* FN-7743 requirement: an ordinary in-progress task (FN-7732) sat stuck for hours
|
||||||
|
* with no recovery action because the executor-stage overseer observation had no
|
||||||
|
* staleness detection — it always reported `signal: "progressing"` regardless of
|
||||||
|
* inactivity. `plannerOverseerExecutorStuckAfterMs` is the configurable inactivity
|
||||||
|
* threshold: once a non-paused in-progress task's last execution activity
|
||||||
|
* (`columnMovedAt ?? updatedAt`) is older than this, the executor stage reports
|
||||||
|
* `signal: "stuck"` instead, which already flows through `decidePlannerRecovery`
|
||||||
|
* into bounded `inject_guidance` recovery. Default 2 hours (7,200,000ms): long
|
||||||
|
* enough that a healthy, actively-working step (the vast majority of which finish
|
||||||
|
* well under 2h) is never nagged, short enough to actually recover a task that has
|
||||||
|
* gone dark for "hours" (the FN-7732 symptom) — mirrors the existing 2-hour
|
||||||
|
* convention `metaTaskStallAutoCloseMs` already uses for a comparable stall
|
||||||
|
* judgment call elsewhere in this codebase.
|
||||||
|
*/
|
||||||
|
export const DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS = 2 * 60 * 60 * 1000;
|
||||||
|
|
||||||
export const BUILTIN_OVERSIGHT_SETTINGS: WorkflowSettingDefinition[] = [
|
export const BUILTIN_OVERSIGHT_SETTINGS: WorkflowSettingDefinition[] = [
|
||||||
{
|
{
|
||||||
id: "plannerOversightLevel",
|
id: "plannerOversightLevel",
|
||||||
@@ -468,6 +486,14 @@ export const BUILTIN_OVERSIGHT_SETTINGS: WorkflowSettingDefinition[] = [
|
|||||||
description:
|
description:
|
||||||
"Planner overseer notification verbosity: Silent suppresses overseer notifications; Errors only notifies on failures/escalations; Important notifies on interventions/recovery actions and errors; All notifies on every observation. Notification-emission gating that reads this value is follow-up work (FN-7519/FN-7520).",
|
"Planner overseer notification verbosity: Silent suppresses overseer notifications; Errors only notifies on failures/escalations; Important notifies on interventions/recovery actions and errors; All notifies on every observation. Notification-emission gating that reads this value is follow-up work (FN-7519/FN-7520).",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
id: "plannerOverseerExecutorStuckAfterMs",
|
||||||
|
name: "Executor stall threshold (ms)",
|
||||||
|
type: "number",
|
||||||
|
default: DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS,
|
||||||
|
description:
|
||||||
|
"Milliseconds of executor-stage inactivity (no progress since the task's last column move/update) before the planner overseer reports the in-progress task as stuck, triggering bounded autonomous recovery (Autonomous level only). Default 7200000 (2 hours). Set higher to avoid nagging long-running steps; set lower to recover hung executors faster.",
|
||||||
|
},
|
||||||
];
|
];
|
||||||
|
|
||||||
export const BUILTIN_WORKFLOW_SETTINGS: WorkflowSettingDefinition[] = [
|
export const BUILTIN_WORKFLOW_SETTINGS: WorkflowSettingDefinition[] = [
|
||||||
|
|||||||
@@ -238,6 +238,7 @@ export {
|
|||||||
BUILTIN_MOVED_WORKFLOW_SETTINGS,
|
BUILTIN_MOVED_WORKFLOW_SETTINGS,
|
||||||
BUILTIN_TRIAGE_POLICY_SETTINGS,
|
BUILTIN_TRIAGE_POLICY_SETTINGS,
|
||||||
BUILTIN_OVERSIGHT_SETTINGS,
|
BUILTIN_OVERSIGHT_SETTINGS,
|
||||||
|
DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS,
|
||||||
renderTriagePolicyPlaceholders,
|
renderTriagePolicyPlaceholders,
|
||||||
} from "./builtin-workflow-settings.js";
|
} from "./builtin-workflow-settings.js";
|
||||||
export {
|
export {
|
||||||
|
|||||||
@@ -207,6 +207,7 @@ export {
|
|||||||
BUILTIN_MOVED_WORKFLOW_SETTINGS,
|
BUILTIN_MOVED_WORKFLOW_SETTINGS,
|
||||||
BUILTIN_TRIAGE_POLICY_SETTINGS,
|
BUILTIN_TRIAGE_POLICY_SETTINGS,
|
||||||
BUILTIN_OVERSIGHT_SETTINGS,
|
BUILTIN_OVERSIGHT_SETTINGS,
|
||||||
|
DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS,
|
||||||
renderTriagePolicyPlaceholders,
|
renderTriagePolicyPlaceholders,
|
||||||
} from "./builtin-workflow-settings.js";
|
} from "./builtin-workflow-settings.js";
|
||||||
export {
|
export {
|
||||||
|
|||||||
@@ -318,3 +318,149 @@ describe("PlannerOverseerMonitor.observeTask", () => {
|
|||||||
await expect(monitor.observeTask(undefined as unknown as OverseerTaskRef, "autonomous")).resolves.toBeNull();
|
await expect(monitor.observeTask(undefined as unknown as OverseerTaskRef, "autonomous")).resolves.toBeNull();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// FN-7743: executor-stage stall detection. FN-7732 was a non-paused in-progress
|
||||||
|
// task that sat stuck for hours while the overseer always reported
|
||||||
|
// `signal: "progressing"` because there was no staleness check. These tests lock
|
||||||
|
// the invariant across the enumerated data states: stale (stuck), recent
|
||||||
|
// (progressing, unchanged), paused (blocked, unchanged), and missing/malformed
|
||||||
|
// timestamp (fail-safe progressing).
|
||||||
|
describe("PlannerOverseerMonitor.observeTask — FN-7743 executor stall detection", () => {
|
||||||
|
const THRESHOLD_MS = 2 * 60 * 60 * 1000; // 2h, mirrors the declared setting default
|
||||||
|
const NOW = Date.UTC(2026, 6, 9, 12, 0, 0);
|
||||||
|
|
||||||
|
function isoMsAgo(ms: number): string {
|
||||||
|
return new Date(NOW - ms).toISOString();
|
||||||
|
}
|
||||||
|
|
||||||
|
it("reports stuck for a non-paused in-progress task whose last activity is older than the threshold", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({
|
||||||
|
column: "in-progress",
|
||||||
|
updatedAt: isoMsAgo(THRESHOLD_MS + 60 * 60 * 1000), // 3h idle
|
||||||
|
});
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("stuck");
|
||||||
|
expect(observation?.stage).toBe("executor");
|
||||||
|
expect(observation?.reason).toMatch(/inactive for over \d+h/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("prefers columnMovedAt over updatedAt when both are present", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({
|
||||||
|
column: "in-progress",
|
||||||
|
// updatedAt looks fresh, but columnMovedAt (the more specific signal) is stale.
|
||||||
|
updatedAt: isoMsAgo(1000),
|
||||||
|
columnMovedAt: isoMsAgo(THRESHOLD_MS + 60 * 60 * 1000),
|
||||||
|
});
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("stuck");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("remains progressing for a non-paused in-progress task with recent activity", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({
|
||||||
|
column: "in-progress",
|
||||||
|
updatedAt: isoMsAgo(5 * 60 * 1000), // 5 minutes ago — well under threshold
|
||||||
|
});
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("progressing");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("remains progressing at exactly the threshold boundary minus one ms, and flips to stuck at/after the boundary", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const justUnder = taskFixture({ column: "in-progress", updatedAt: isoMsAgo(THRESHOLD_MS - 1) });
|
||||||
|
const atThreshold = taskFixture({ column: "in-progress", updatedAt: isoMsAgo(THRESHOLD_MS) });
|
||||||
|
|
||||||
|
const obsUnder = await monitor.observeTask(justUnder, "autonomous", { now: () => NOW, executorStuckAfterMs: THRESHOLD_MS });
|
||||||
|
expect(obsUnder?.signal).toBe("progressing");
|
||||||
|
|
||||||
|
const obsAt = await monitor.observeTask(atThreshold, "autonomous", { now: () => NOW, executorStuckAfterMs: THRESHOLD_MS });
|
||||||
|
expect(obsAt?.signal).toBe("stuck");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still reports blocked (unchanged) for a paused in-progress task even with a stale timestamp", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({
|
||||||
|
column: "in-progress",
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "some-engine-park-reason",
|
||||||
|
updatedAt: isoMsAgo(THRESHOLD_MS + 60 * 60 * 1000),
|
||||||
|
});
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("blocked");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fails safe to progressing when both updatedAt and columnMovedAt are missing", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({ column: "in-progress", updatedAt: undefined, columnMovedAt: undefined });
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("progressing");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fails safe to progressing when the timestamp is malformed/unparseable", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const task = taskFixture({ column: "in-progress", updatedAt: "not-a-real-date" });
|
||||||
|
|
||||||
|
const observation = await monitor.observeTask(task, "autonomous", {
|
||||||
|
now: () => NOW,
|
||||||
|
executorStuckAfterMs: THRESHOLD_MS,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(observation?.signal).toBe("progressing");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("defaults executorStuckAfterMs to the declared 2h default when options are omitted", async () => {
|
||||||
|
const monitor = new PlannerOverseerMonitor();
|
||||||
|
const staleTask = taskFixture({ column: "in-progress", updatedAt: isoMsAgo(3 * 60 * 60 * 1000) });
|
||||||
|
const freshTask = taskFixture({ column: "in-progress", updatedAt: isoMsAgo(60 * 1000) });
|
||||||
|
|
||||||
|
const staleObs = await monitor.observeTask(staleTask, "autonomous", { now: () => NOW });
|
||||||
|
const freshObs = await monitor.observeTask(freshTask, "autonomous", { now: () => NOW });
|
||||||
|
|
||||||
|
expect(staleObs?.signal).toBe("stuck");
|
||||||
|
expect(freshObs?.signal).toBe("progressing");
|
||||||
|
});
|
||||||
|
|
||||||
|
// FN-7577: the dedup key must stay stable within an hour bucket so an ever-
|
||||||
|
// changing millisecond-precise duration in the reason does not defeat dedup.
|
||||||
|
it("keeps the stuck reason stable within the same inactivity-hour bucket for feed dedup", async () => {
|
||||||
|
const store = { logEntry: vi.fn().mockResolvedValue(undefined) };
|
||||||
|
const monitor = new PlannerOverseerMonitor({ store });
|
||||||
|
const task = taskFixture({ column: "in-progress", updatedAt: isoMsAgo(THRESHOLD_MS + 5 * 60 * 1000) });
|
||||||
|
|
||||||
|
// Two polls a few seconds apart within the same inactivity-hour bucket.
|
||||||
|
await monitor.observeTask(task, "observe", { now: () => NOW, executorStuckAfterMs: THRESHOLD_MS });
|
||||||
|
await monitor.observeTask(task, "observe", { now: () => NOW + 5000, executorStuckAfterMs: THRESHOLD_MS });
|
||||||
|
expect(store.logEntry).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
// An hour-boundary crossing is a real state change — re-logs once.
|
||||||
|
await monitor.observeTask(task, "observe", { now: () => NOW + 60 * 60 * 1000, executorStuckAfterMs: THRESHOLD_MS });
|
||||||
|
expect(store.logEntry).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { describe, expect, it, vi } from "vitest";
|
import { describe, expect, it, vi } from "vitest";
|
||||||
import { PLANNER_RECOVERY_MAX_ATTEMPTS } from "@fusion/core";
|
import { AWAITING_APPROVAL_PAUSE_REASON, PLANNER_RECOVERY_MAX_ATTEMPTS } from "@fusion/core";
|
||||||
import type { Task } from "@fusion/core";
|
import type { Task } from "@fusion/core";
|
||||||
import { PlannerRecoveryController, type PlannerRecoveryHandlers } from "../planner-recovery-controller.js";
|
import { PlannerRecoveryController, type PlannerRecoveryHandlers } from "../planner-recovery-controller.js";
|
||||||
import type { OverseerStageObservation, OverseerWatchedStage } from "../planner-overseer.js";
|
import type { OverseerStageObservation, OverseerWatchedStage } from "../planner-overseer.js";
|
||||||
@@ -189,4 +189,54 @@ describe("PlannerRecoveryController.tick", () => {
|
|||||||
expect(decision?.action).toBe("retry_step");
|
expect(decision?.action).toBe("retry_step");
|
||||||
expect(retryStep).toHaveBeenCalledTimes(1);
|
expect(retryStep).toHaveBeenCalledTimes(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// FN-7743: a `stuck` executor observation (a genuinely hung/idle in-progress
|
||||||
|
// task, the FN-7732 symptom) must reach bounded recovery exactly like a
|
||||||
|
// `failed`/`blocked` one — `inject_guidance`, dispatched once per tick, with
|
||||||
|
// the attempt budget incrementing — and a withheld (user-paused) task must
|
||||||
|
// still dispatch nothing even though its stage is `stuck`.
|
||||||
|
describe("FN-7743 stuck executor observation", () => {
|
||||||
|
it("dispatches inject_guidance exactly once and increments the attempt budget for a stuck executor stage", async () => {
|
||||||
|
const injectGuidance = vi.fn().mockResolvedValue(undefined);
|
||||||
|
const controller = makeController(observation({ stage: "executor", signal: "stuck", reason: "Executor stage inactive for over 3h with no execution activity" }), {
|
||||||
|
injectGuidance,
|
||||||
|
});
|
||||||
|
|
||||||
|
const decision = await controller.tick(task());
|
||||||
|
expect(decision?.action).toBe("inject_guidance");
|
||||||
|
expect(injectGuidance).toHaveBeenCalledTimes(1);
|
||||||
|
expect(controller.getAttemptCount("FN-1", "executor")).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dispatches nothing for a stuck executor stage when the task is user-paused (human-control withhold wins)", async () => {
|
||||||
|
const injectGuidance = vi.fn().mockResolvedValue(undefined);
|
||||||
|
const controller = makeController(observation({ stage: "executor", signal: "stuck" }), { injectGuidance });
|
||||||
|
|
||||||
|
const decision = await controller.tick(task({ userPaused: true }));
|
||||||
|
expect(decision).toBeNull();
|
||||||
|
expect(injectGuidance).not.toHaveBeenCalled();
|
||||||
|
expect(controller.getAttemptCount("FN-1", "executor")).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dispatches nothing for a stuck executor stage when the task is approval-blocked (human-control withhold wins)", async () => {
|
||||||
|
const injectGuidance = vi.fn().mockResolvedValue(undefined);
|
||||||
|
const controller = makeController(observation({ stage: "executor", signal: "stuck" }), { injectGuidance });
|
||||||
|
|
||||||
|
const decision = await controller.tick(task({ paused: true, pausedReason: AWAITING_APPROVAL_PAUSE_REASON }));
|
||||||
|
expect(decision).toBeNull();
|
||||||
|
expect(injectGuidance).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is inert for a stuck executor stage when effectiveLevel is off/observe/steer (no autonomous dispatch)", async () => {
|
||||||
|
for (const level of ["off", "observe", "steer"] as const) {
|
||||||
|
const injectGuidance = vi.fn().mockResolvedValue(undefined);
|
||||||
|
const controller = makeController(observation({ stage: "executor", signal: "stuck", oversightLevel: level }), {
|
||||||
|
injectGuidance,
|
||||||
|
});
|
||||||
|
const decision = await controller.tick(task());
|
||||||
|
expect(decision?.action, `level=${level}`).toBe("none");
|
||||||
|
expect(injectGuidance).not.toHaveBeenCalled();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -14,7 +14,7 @@
|
|||||||
* the seam every later planner-oversight subtask reads from.
|
* the seam every later planner-oversight subtask reads from.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { PlannerOversightLevel, PrInfo, Task } from "@fusion/core";
|
import { DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS, type PlannerOversightLevel, type PrInfo, type Task } from "@fusion/core";
|
||||||
|
|
||||||
/** Alias for the `Task.reviewState` shape without requiring a separate core export. */
|
/** Alias for the `Task.reviewState` shape without requiring a separate core export. */
|
||||||
type OverseerTaskReviewState = NonNullable<Task["reviewState"]>;
|
type OverseerTaskReviewState = NonNullable<Task["reviewState"]>;
|
||||||
@@ -53,7 +53,15 @@ export interface OverseerStageObservation {
|
|||||||
* full `Task` interface. */
|
* full `Task` interface. */
|
||||||
export type OverseerTaskRef = Pick<
|
export type OverseerTaskRef = Pick<
|
||||||
Task,
|
Task,
|
||||||
"id" | "column" | "prInfo" | "reviewState" | "paused" | "pausedReason" | "workflowTransitionNotification"
|
| "id"
|
||||||
|
| "column"
|
||||||
|
| "prInfo"
|
||||||
|
| "reviewState"
|
||||||
|
| "paused"
|
||||||
|
| "pausedReason"
|
||||||
|
| "workflowTransitionNotification"
|
||||||
|
| "updatedAt"
|
||||||
|
| "columnMovedAt"
|
||||||
>;
|
>;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -127,10 +135,40 @@ export function resolveWatchedStage(task: Partial<OverseerTaskRef> | null | unde
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
* FN-7743 stall-detection inputs for `deriveSignalAndSources`'s `executor` branch.
|
||||||
|
* Passed in already-resolved (never read from settings inside this pure function,
|
||||||
|
* per the FN-7743 "keep derivation pure and testable" requirement): `now` is an
|
||||||
|
* injectable clock (defaults to `Date.now` at the `observeTask` call site) and
|
||||||
|
* `executorStuckAfterMs` is the resolved `plannerOverseerExecutorStuckAfterMs`
|
||||||
|
* workflow setting value (or its declaration default).
|
||||||
|
*/
|
||||||
|
export interface ExecutorStallSignalInput {
|
||||||
|
now: () => number;
|
||||||
|
executorStuckAfterMs: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
* Validates a raw `plannerOverseerExecutorStuckAfterMs` workflow-setting value
|
||||||
|
* (which may be missing/malformed/legacy-orphaned per `resolveEffectiveSettings`'s
|
||||||
|
* never-throw contract) into a safe threshold, degrading to
|
||||||
|
* `DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS` for anything that is not a
|
||||||
|
* finite positive number. Pure, never throws.
|
||||||
|
*/
|
||||||
|
export function resolveExecutorStuckAfterMs(raw: unknown): number {
|
||||||
|
if (typeof raw === "number" && Number.isFinite(raw) && raw > 0) {
|
||||||
|
return raw;
|
||||||
|
}
|
||||||
|
return DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS;
|
||||||
|
}
|
||||||
|
|
||||||
function deriveSignalAndSources(
|
function deriveSignalAndSources(
|
||||||
taskId: string,
|
taskId: string,
|
||||||
stage: OverseerWatchedStage,
|
stage: OverseerWatchedStage,
|
||||||
task: Partial<OverseerTaskRef>,
|
task: Partial<OverseerTaskRef>,
|
||||||
|
stallInput: ExecutorStallSignalInput,
|
||||||
): { signal: OverseerObservationSignal; reason: string; sources: OverseerSourceLink[] } {
|
): { signal: OverseerObservationSignal; reason: string; sources: OverseerSourceLink[] } {
|
||||||
switch (stage) {
|
switch (stage) {
|
||||||
case "executor": {
|
case "executor": {
|
||||||
@@ -141,6 +179,32 @@ function deriveSignalAndSources(
|
|||||||
sources: [{ kind: "agent-log", ref: taskId }],
|
sources: [{ kind: "agent-log", ref: taskId }],
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
// FN-7743: a non-paused in-progress task whose executor session has gone
|
||||||
|
// silent (dead/hung agent, no commits/heartbeat) was previously ALWAYS
|
||||||
|
// reported "progressing" forever (FN-7732 symptom) since nothing here
|
||||||
|
// checked staleness. Use `columnMovedAt ?? updatedAt` as the best
|
||||||
|
// available store-backed "last execution activity" proxy at this poll
|
||||||
|
// seam (the live in-session `StuckTaskDetector` heartbeat state is
|
||||||
|
// in-memory-only and not available here). A missing/malformed timestamp
|
||||||
|
// degrades to "progressing" — never fabricate a stall. The reason is
|
||||||
|
// bucketed to whole hours so the FN-7577 `stage|signal|reason` feed dedup
|
||||||
|
// stays effective (it must not embed an ever-changing millisecond value).
|
||||||
|
const activityTimestamp = task.columnMovedAt ?? task.updatedAt;
|
||||||
|
const activityAtMs = activityTimestamp ? Date.parse(activityTimestamp) : NaN;
|
||||||
|
if (Number.isFinite(activityAtMs) && stallInput.executorStuckAfterMs > 0) {
|
||||||
|
const inactiveMs = stallInput.now() - activityAtMs;
|
||||||
|
if (inactiveMs >= stallInput.executorStuckAfterMs) {
|
||||||
|
const inactiveHours = Math.max(1, Math.floor(inactiveMs / 3_600_000));
|
||||||
|
return {
|
||||||
|
signal: "stuck",
|
||||||
|
reason: `Executor stage inactive for over ${inactiveHours}h with no execution activity`,
|
||||||
|
sources: [{ kind: "agent-log", ref: taskId }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
signal: "progressing",
|
signal: "progressing",
|
||||||
reason: "Task is actively executing in-progress work",
|
reason: "Task is actively executing in-progress work",
|
||||||
@@ -280,8 +344,20 @@ export class PlannerOverseerMonitor {
|
|||||||
* Observe a task's current watched stage and record a gated observation.
|
* Observe a task's current watched stage and record a gated observation.
|
||||||
* Returns `null` when the level is `"off"` or when no stage is currently
|
* Returns `null` when the level is `"off"` or when no stage is currently
|
||||||
* monitorable. Never throws.
|
* monitorable. Never throws.
|
||||||
|
*
|
||||||
|
* FNXC:PlannerOversight 2026-07-09-00:00:
|
||||||
|
* FN-7743: `options.now`/`options.executorStuckAfterMs` thread the clock and
|
||||||
|
* the resolved executor-stall threshold into the pure `deriveSignalAndSources`
|
||||||
|
* derivation. Both default (`Date.now`, `DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS`)
|
||||||
|
* so existing callers keep working unchanged; the poll seam
|
||||||
|
* (`project-engine.ts#pollPlannerOverseer`) resolves the real workflow-setting
|
||||||
|
* value once per cycle and passes it in explicitly.
|
||||||
*/
|
*/
|
||||||
async observeTask(task: OverseerTaskRef, level: PlannerOversightLevel): Promise<OverseerStageObservation | null> {
|
async observeTask(
|
||||||
|
task: OverseerTaskRef,
|
||||||
|
level: PlannerOversightLevel,
|
||||||
|
options?: { now?: () => number; executorStuckAfterMs?: number },
|
||||||
|
): Promise<OverseerStageObservation | null> {
|
||||||
try {
|
try {
|
||||||
if (level === "off") {
|
if (level === "off") {
|
||||||
return null;
|
return null;
|
||||||
@@ -292,13 +368,15 @@ export class PlannerOverseerMonitor {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
const { signal, reason, sources } = deriveSignalAndSources(task.id, stage, task);
|
const now = options?.now ?? Date.now;
|
||||||
|
const executorStuckAfterMs = options?.executorStuckAfterMs ?? DEFAULT_PLANNER_OVERSEER_EXECUTOR_STUCK_AFTER_MS;
|
||||||
|
const { signal, reason, sources } = deriveSignalAndSources(task.id, stage, task, { now, executorStuckAfterMs });
|
||||||
const observation: OverseerStageObservation = {
|
const observation: OverseerStageObservation = {
|
||||||
taskId: task.id,
|
taskId: task.id,
|
||||||
stage,
|
stage,
|
||||||
signal,
|
signal,
|
||||||
oversightLevel: level,
|
oversightLevel: level,
|
||||||
observedAt: Date.now(),
|
observedAt: now(),
|
||||||
reason,
|
reason,
|
||||||
sources,
|
sources,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ import { InProcessRuntime } from "./runtimes/in-process-runtime.js";
|
|||||||
import type { WorktreePool } from "./worktree-pool.js";
|
import type { WorktreePool } from "./worktree-pool.js";
|
||||||
import type { ProjectRuntimeConfig } from "./project-runtime.js";
|
import type { ProjectRuntimeConfig } from "./project-runtime.js";
|
||||||
import { PrMonitor } from "./pr-monitor.js";
|
import { PrMonitor } from "./pr-monitor.js";
|
||||||
import { PlannerOverseerMonitor } from "./planner-overseer.js";
|
import { PlannerOverseerMonitor, resolveExecutorStuckAfterMs } from "./planner-overseer.js";
|
||||||
import { PlannerRecoveryController, type PlannerRecoveryHandlers } from "./planner-recovery-controller.js";
|
import { PlannerRecoveryController, type PlannerRecoveryHandlers } from "./planner-recovery-controller.js";
|
||||||
import { evaluateOverseerHumanControl } from "./overseer-human-control-policy.js";
|
import { evaluateOverseerHumanControl } from "./overseer-human-control-policy.js";
|
||||||
import type { PrNodeGithubOps } from "./pr-nodes.js";
|
import type { PrNodeGithubOps } from "./pr-nodes.js";
|
||||||
@@ -2371,7 +2371,14 @@ export class ProjectEngine {
|
|||||||
if (level === "off") {
|
if (level === "off") {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
await overseer.observeTask(task, level);
|
// FN-7743: resolve the executor-stall threshold from the task's
|
||||||
|
// effective workflow settings (same `workflowEffective` fetch used
|
||||||
|
// for `plannerOversightLevel` above — no extra store round-trip) and
|
||||||
|
// pass it into `observeTask` so a genuinely idle non-paused
|
||||||
|
// in-progress task reports `signal: "stuck"` instead of always
|
||||||
|
// `progressing` (the FN-7732 symptom).
|
||||||
|
const executorStuckAfterMs = resolveExecutorStuckAfterMs(workflowEffective.plannerOverseerExecutorStuckAfterMs);
|
||||||
|
await overseer.observeTask(task, level, { executorStuckAfterMs });
|
||||||
|
|
||||||
// FN-7512: one guarded, autonomous-only bounded recovery tick at the
|
// FN-7512: one guarded, autonomous-only bounded recovery tick at the
|
||||||
// same passive seam FN-7511 uses for observation. Inert for every
|
// same passive seam FN-7511 uses for observation. Inert for every
|
||||||
|
|||||||
Reference in New Issue
Block a user