root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(FN-5223): anchor staleness and stall detectors to engine activation ti

Browse Source 
The merge introduces an engine-activation timestamp as the staleness floor for task age calculations, replacing arbitrary wall-clock thresholds with a runtime-relative anchor. Step 1 adds settings defaults, Steps 2–4 wire the floor helper through project engine, in-process runtime, and task store hy

Fusion-Task-Id: FN-5223
This commit is contained in:
Fusion (runfusion.ai)
2026-05-20 14:02:16 -07:00
committed by gsxdsm
parent 62f11e69d3
commit c60045df22
25 changed files with 626 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
AGENTS.md
View File

@@ -540,5 +540,6 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
- FN-5168 backstop: `packages/engine/src/__tests__/reliability-interactions/non-progress-churn.test.ts` covers loop→compact recovery followed by ignored-step-update churn escalation, terminal `beforeRequeue(false)` behavior, audit/log payloads, and FN-5147 autoMerge-off composition.
- FN-5219 backstop: `packages/engine/src/__tests__/reliability-interactions/in-progress-limbo-recovery.test.ts` covers `recoverInProgressLimbo` composition with `recoverOrphanedExecutions` (no double-recovery), `reconcile-task-worktree-metadata` (live rebindable worktree wins), `recoverMissingWorktreeReviewFailures` (in-review vs in-progress disjoint), and executor task-id claim skip, plus an explicit FN-5149 reproduction case.
- FN-5325 backstop: `packages/engine/src/__tests__/reliability-interactions/scheduler-overlap-priority-inversion.test.ts` covers queued-overlap priority/age deferral, equal-priority age ordering, FN-4969 fanout composition, and one-shot per-pass `scheduler:overlap-priority-inversion` audit surfacing against running lower-priority blockers.
- FN-5223 backstop: `packages/engine/src/__tests__/reliability-interactions/engine-active-since-floor.test.ts` covers engine-activation floor + grace composition across startup, pause/unpause, global-pause gating, and StuckTaskDetector lifecycle interactions.
The auto-recovery dispatcher at `packages/engine/src/auto-recovery.ts` (FN-4533) composes on top of existing layers (FN-4500 fast-path, FN-4508 deterministic branch-conflict, FN-4499 bootstrap-misbinding, FN-4428 contamination, `mergeAuditAutoRecovery` Stages 1–5, self-healing) to handle six residual classes: file-scope violation at squash, branch misbinding / ghost worktree, verification-fix scope leak, contamination, `branch-conflict-unrecoverable` residuals, and room-post/message-send failures. Invocation is additive — no existing layer's behavior changes.

2
docs/diagnostics.md
View File

@@ -47,6 +47,8 @@ The process supervisor logs when it registers a supervised child, starts teardow
## No-progress churn stuck-task escalation (`[executor]`, `[stuck-detector]`, `[self-healing]`)
Time-based stuck/stalled/stale surfaces now floor activity timestamps using `settings.engineActiveSinceMs` plus `settings.engineActivationGraceMs` (default `300000`). The runtime stamps `engineActiveSinceMs` on startup and each unpause transition so engine pause/downtime does not count as quiet time.
- Trigger shape: one loop classification/compact-and-resume has already fired for the current `execute()` lifecycle, then ignored `fn_task_update` rebuffs accumulate to `ignoredStepUpdateCount >= 25` without intervening progress.
- Executor diagnostic: `[executor] <taskId>: no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — escalating to STUCK_NO_PROGRESS_CHURN`.
- Self-healing diagnostic: `<taskId> no-progress churn detected (ignoredStepUpdates=N, stuckKillStreak=M) — marking failed`.

2
docs/settings-reference.md
View File

@@ -320,6 +320,8 @@ Default notes:
| `specStalenessEnabled` | `boolean` | `false` | Enforce automatic re-planning for stale plans. |
| `specStalenessMaxAgeMs` | `number` | `21600000` | Spec staleness threshold in ms (6 hours). |
| `taskStuckTimeoutMs` | `number` | `undefined` | Inactivity timeout for stuck-task recovery. |
| `engineActiveSinceMs` | `number` | `undefined` | Epoch ms when the in-process runtime last became active (startup or unpause). Time-based stuck/stalled/stale surfaces floor their activity anchor at this timestamp so paused/stopped downtime is not counted as quiet age. Runtime-managed; typically not set manually. |
| `engineActivationGraceMs` | `number` | `300000` | Extra grace window (ms) added after `engineActiveSinceMs` before time-based stuck/stalled/stale surfaces can fire. Set `0` to disable warmup. |
| `inReviewStallDeadlockThreshold` | `number` | `3` | Minimum number of identical consecutive in-review stall log entries (same stall code + reason) before self-healing auto-disposes the task by pausing it with `pausedReason="in-review-stall-deadlock"` and marking status `failed`. Set to `0` to disable. |
| `stalePausedReviewThresholdMs` | `number` | `86400000` | Threshold in ms for surfacing paused `in-review` tasks as stale paused review diagnostics (24 hours). `0` or `undefined` disables stale paused review surfacing/logging. |
| `inReviewStalledThresholdMs` | `number` | `86400000` | When `> 0`, enables surfacing of unpaused `in-review` tasks quiet beyond threshold via the `surface-in-review-stalled` self-healing pass; `0` disables. See **Backlog health alerts** below. |

39
packages/core/src/__tests__/in-review-stall.test.ts
View File

@@ -172,6 +172,45 @@ describe("getInReviewStallReason", () => {
expect(getInReviewStallReason({ ...baseTask }, { now: NOW })).toBeUndefined();
});
it("preserves status-driven merge-blocker even when activation floor is recent", () => {
const signal = getInReviewStallReason({
...baseTask,
status: "merging",
updatedAt: new Date(NOW - DEFAULT_STALE_MERGING_MIN_AGE_MS - 60_000).toISOString(),
}, {
now: NOW,
engineActiveSinceMs: NOW - (DEFAULT_STALE_MERGING_MIN_AGE_MS - 60_000),
engineActivationGraceMs: 90_000,
});
expect(signal?.code).toBe("merge-blocker");
});
it("fires transient status once engine activation floor is old", () => {
const signal = getInReviewStallReason({
...baseTask,
status: "merging",
updatedAt: new Date(NOW - DEFAULT_STALE_MERGING_MIN_AGE_MS - 60_000).toISOString(),
}, {
now: NOW,
engineActiveSinceMs: NOW - DEFAULT_STALE_MERGING_MIN_AGE_MS - 120_000,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("transient-merge-status-no-owner");
});
it("with zero grace, counts from activation timestamp immediately", () => {
const signal = getInReviewStallReason({
...baseTask,
status: "merging",
updatedAt: new Date(NOW - DEFAULT_STALE_MERGING_MIN_AGE_MS - 1).toISOString(),
}, {
now: NOW,
engineActiveSinceMs: NOW,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("merge-blocker");
});
it("prioritizes transient merge status over retries exhausted", () => {
const signal = getInReviewStallReason({
...baseTask,

32
packages/core/src/__tests__/in-review-stalled.test.ts
View File

@@ -120,4 +120,36 @@ describe("getInReviewStalledSignal", () => {
}, { now: NOW, thresholdMs: 10_000 });
expect(signal?.lastActivitySource).toBe("updated");
});
it("suppresses signal during activation grace warmup", () => {
const signal = getInReviewStalledSignal(baseTask, {
now: NOW,
thresholdMs: 10_000,
engineActiveSinceMs: NOW - 5_000,
engineActivationGraceMs: 30_000,
});
expect(signal).toBeUndefined();
});
it("adds effectiveLastActivityAt when floor changes activity anchor", () => {
const signal = getInReviewStalledSignal(baseTask, {
now: NOW,
thresholdMs: 10_000,
engineActiveSinceMs: NOW - 20_000,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("in-review-stalled");
expect(signal?.effectiveLastActivityAt).toBe(iso(NOW - 20_000));
expect(signal?.lastActivityAt).not.toBe(signal?.effectiveLastActivityAt);
});
it("fires normally when activation floor is far in the past", () => {
const signal = getInReviewStalledSignal(baseTask, {
now: NOW,
thresholdMs: 10_000,
engineActiveSinceMs: NOW - 100_000,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("in-review-stalled");
});
});

5
packages/core/src/__tests__/settings-parity.test.ts
View File

@@ -150,6 +150,11 @@ describe("settings key parity", () => {
expect(DEFAULT_PROJECT_SETTINGS.workflowStepTimeoutMs).toBe(360_000);
});
it("defaults engine activation grace and leaves engine active clock undefined", () => {
expect(DEFAULT_PROJECT_SETTINGS.engineActivationGraceMs).toBe(300_000);
expect(DEFAULT_PROJECT_SETTINGS.engineActiveSinceMs).toBeUndefined();
});
it("defaults autoRecovery and normalizes overrides", () => {
expect(DEFAULT_PROJECT_SETTINGS.autoRecovery).toEqual({ mode: "deterministic-only", maxRetries: 3 });
expect(normalizeAutoRecovery({ mode: "off", perClass: { "branch-conflict-unrecoverable": "ai-assisted" }, maxRetries: 2 })).toEqual({

19
packages/core/src/__tests__/stale-paused-review.test.ts
View File

@@ -50,4 +50,23 @@ describe("getStalePausedReviewSignal", () => {
}, { now: NOW });
expect(signal?.ageMs).toBe(DEFAULT_STALE_PAUSED_REVIEW_THRESHOLD_MS + 1_000);
});
it("suppresses signal during activation grace warmup", () => {
const signal = getStalePausedReviewSignal({ ...baseTask }, {
now: NOW,
engineActiveSinceMs: NOW - 60_000,
engineActivationGraceMs: 5 * 60_000,
});
expect(signal).toBeUndefined();
});
it("fires once activation floor is sufficiently in the past", () => {
const signal = getStalePausedReviewSignal({ ...baseTask }, {
now: NOW,
engineActiveSinceMs: NOW - DEFAULT_STALE_PAUSED_REVIEW_THRESHOLD_MS - 5_000,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("stale-paused-review");
expect(signal?.ageMs).toBe(DEFAULT_STALE_PAUSED_REVIEW_THRESHOLD_MS);
});
});

19
packages/core/src/__tests__/stale-paused-todo.test.ts
View File

@@ -46,4 +46,23 @@ describe("getStalePausedTodoSignal", () => {
expect(getStalePausedTodoSignal({ ...baseTask }, { now: NOW, thresholdMs: 0 })).toBeUndefined();
expect(getStalePausedTodoSignal({ ...baseTask }, { now: NOW, thresholdMs: -1 })).toBeUndefined();
});
it("suppresses signal during activation grace warmup", () => {
const signal = getStalePausedTodoSignal({ ...baseTask }, {
now: NOW,
engineActiveSinceMs: NOW - 60_000,
engineActivationGraceMs: 5 * 60_000,
});
expect(signal).toBeUndefined();
});
it("fires once activation floor is sufficiently in the past", () => {
const signal = getStalePausedTodoSignal({ ...baseTask }, {
now: NOW,
engineActiveSinceMs: NOW - DEFAULT_STALE_PAUSED_TODO_THRESHOLD_MS - 5_000,
engineActivationGraceMs: 0,
});
expect(signal?.code).toBe("stale-paused-todo");
expect(signal?.ageMs).toBe(DEFAULT_STALE_PAUSED_TODO_THRESHOLD_MS);
});
});

71
packages/core/src/__tests__/store-engine-active-since.test.ts Normal file
View File

@@ -0,0 +1,71 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { mkdtemp, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { TaskStore } from "../store.js";
describe("TaskStore engineActiveSinceMs hydration floor", () => {
let rootDir: string;
let globalDir: string;
let store: TaskStore;
beforeEach(async () => {
rootDir = await mkdtemp(join(tmpdir(), "store-engine-active-since-"));
globalDir = join(rootDir, ".fusion-global-settings");
store = new TaskStore(rootDir, globalDir, { inMemoryDb: true });
await store.init();
});
afterEach(async () => {
await store.close();
await rm(rootDir, { recursive: true, force: true });
});
async function seedTask(id: string, column: "in-review" | "todo" | "in-progress", paused: boolean, ageMs: number) {
const movedAt = new Date(Date.now() - ageMs).toISOString();
await store.createTaskWithReservedId(
{ description: id, column },
{ taskId: id, createdAt: movedAt, updatedAt: movedAt, applyDefaultWorkflowSteps: true },
);
const db = (store as unknown as { db: { prepare: (sql: string) => { run: (...params: unknown[]) => unknown } } }).db;
db.prepare(`UPDATE tasks SET paused = ?, mergeDetails = ?, log = ?, columnMovedAt = ?, updatedAt = ? WHERE id = ?`).run(
paused ? 1 : 0,
JSON.stringify({}),
JSON.stringify([]),
movedAt,
movedAt,
id,
);
}
it("FN-5223 suppresses stale signals until activation floor ages out", async () => {
const ageMs = 14 * 24 * 60 * 60_000;
await seedTask("FN-5223-REVIEW", "in-review", true, ageMs);
await seedTask("FN-5223-STALLED", "in-review", false, ageMs);
await seedTask("FN-5223-AGE", "in-progress", false, ageMs);
await seedTask("FN-5223-TODO", "todo", true, ageMs);
await store.updateSettings({
engineActiveSinceMs: Date.now(),
engineActivationGraceMs: 5 * 60_000,
stalePausedReviewThresholdMs: 60_000,
inReviewStalledThresholdMs: 60_000,
staleInProgressWarningMs: 60_000,
staleInProgressCriticalMs: 120_000,
stalePausedTodoThresholdMs: 60_000,
});
let tasks = await store.listTasks();
expect(tasks.find((task) => task.id === "FN-5223-REVIEW")?.stalePausedReview).toBeUndefined();
expect(tasks.find((task) => task.id === "FN-5223-STALLED")?.inReviewStalled).toBeUndefined();
expect(tasks.find((task) => task.id === "FN-5223-AGE")?.ageStaleness).toBeUndefined();
expect(tasks.find((task) => task.id === "FN-5223-TODO")?.stalePausedTodo).toBeUndefined();
await store.updateSettings({ engineActiveSinceMs: Date.now() - 20 * 60_000 });
tasks = await store.listTasks();
const restoredAge = tasks.find((task) => task.id === "FN-5223-AGE")?.ageStaleness?.code === "task-age-staleness";
const restoredTodo = tasks.find((task) => task.id === "FN-5223-TODO")?.stalePausedTodo?.code === "stale-paused-todo";
expect(restoredAge || restoredTodo).toBe(true);
});
});

24
packages/core/src/__tests__/task-age-staleness.test.ts
View File

@@ -126,4 +126,28 @@ describe("getTaskAgeStalenessSignal", () => {
)
).toThrowError(new RangeError("critical threshold must be >= warning threshold"));
});
it("suppresses age signal during activation grace warmup", () => {
const signal = getTaskAgeStalenessSignal({
...baseTask,
columnMovedAt: new Date(NOW - DEFAULT_TASK_AGE_STALENESS_THRESHOLDS.inProgressCriticalMs).toISOString(),
}, {
now: NOW,
engineActiveSinceMs: NOW - 60_000,
engineActivationGraceMs: 5 * 60_000,
});
expect(signal).toBeUndefined();
});
it("fires age signal when activation floor is far in the past", () => {
const signal = getTaskAgeStalenessSignal({
...baseTask,
columnMovedAt: new Date(NOW - DEFAULT_TASK_AGE_STALENESS_THRESHOLDS.inProgressCriticalMs).toISOString(),
}, {
now: NOW,
engineActiveSinceMs: NOW - DEFAULT_TASK_AGE_STALENESS_THRESHOLDS.inProgressCriticalMs - 60_000,
engineActivationGraceMs: 0,
});
expect(signal?.level).toBe("critical");
});
});

16
packages/core/src/in-review-stall.ts
View File

@@ -30,6 +30,8 @@ export interface InReviewStallContext {
executingTaskIds?: ReadonlySet<string>;
staleMergingMinAgeMs?: number;
maxAutoMergeRetries?: number;
engineActiveSinceMs?: number;
engineActivationGraceMs?: number;
}
/** Keep aligned with engine DEFAULT_STALE_MERGING_STATUS_MIN_AGE_MS. */
@@ -100,7 +102,11 @@ export function getInReviewStallReason(
if (task.status && TRANSIENT_MERGE_STATUSES.has(task.status)) {
const updatedAtMs = Date.parse(task.updatedAt);
if (Number.isFinite(updatedAtMs) && now - updatedAtMs >= staleMergingMinAgeMs) {
const activationFloorMs = getActivationFloorMs(context);
const effectiveUpdatedAtMs = Number.isFinite(updatedAtMs)
? activationFloorMs !== undefined ? Math.max(updatedAtMs, activationFloorMs) : updatedAtMs
: Number.NaN;
if (Number.isFinite(effectiveUpdatedAtMs) && Math.max(0, now - effectiveUpdatedAtMs) >= staleMergingMinAgeMs) {
const minutes = Math.max(1, Math.floor(staleMergingMinAgeMs / 60_000));
return {
code: "transient-merge-status-no-owner",
@@ -138,3 +144,11 @@ export function getInReviewStallReason(
return undefined;
}
function getActivationFloorMs(context: InReviewStallContext): number | undefined {
if (typeof context.engineActiveSinceMs !== "number" || !Number.isFinite(context.engineActiveSinceMs)) {
return undefined;
}
return context.engineActiveSinceMs + Math.max(0, context.engineActivationGraceMs ?? 0);
}

20
packages/core/src/in-review-stalled.ts
View File

@@ -11,6 +11,7 @@ export interface InReviewStalledSignal {
quietMs: number;
thresholdMs: number;
lastActivityAt: string;
effectiveLastActivityAt?: string;
lastActivitySource: "log" | "column-moved" | "updated";
}
@@ -20,6 +21,8 @@ export interface InReviewStalledContext {
autoMerge?: boolean;
activeMergeTaskId?: string | null;
executingTaskIds?: ReadonlySet<string>;
engineActiveSinceMs?: number;
engineActivationGraceMs?: number;
}
export const DEFAULT_IN_REVIEW_STALLED_THRESHOLD_MS = 24 * 60 * 60_000;
@@ -54,7 +57,11 @@ export function getInReviewStalledSignal(
const lastActivity = getLastActivity(task);
if (!lastActivity) return undefined;
const quietMs = Math.max(0, now - lastActivity.time);
const activationFloorMs = getActivationFloorMs(context);
const effectiveLastActivityMs = activationFloorMs !== undefined
? Math.max(lastActivity.time, activationFloorMs)
: lastActivity.time;
const quietMs = Math.max(0, now - effectiveLastActivityMs);
if (quietMs < thresholdMs) return undefined;
const ageAnchor = Date.parse(task.columnMovedAt ?? task.updatedAt);
@@ -72,10 +79,21 @@ export function getInReviewStalledSignal(
quietMs,
thresholdMs,
lastActivityAt: new Date(lastActivity.time).toISOString(),
...(effectiveLastActivityMs !== lastActivity.time
? { effectiveLastActivityAt: new Date(effectiveLastActivityMs).toISOString() }
: {}),
lastActivitySource: lastActivity.source,
};
}
function getActivationFloorMs(context: InReviewStalledContext): number | undefined {
if (typeof context.engineActiveSinceMs !== "number" || !Number.isFinite(context.engineActiveSinceMs)) {
return undefined;
}
return context.engineActiveSinceMs + Math.max(0, context.engineActivationGraceMs ?? 0);
}
function hasRecentReasonDrivenStall(log: readonly Pick<Task["log"][number], "action" | "timestamp">[], floor: number): boolean {
let latestTime = Number.NEGATIVE_INFINITY;

2
packages/core/src/settings-schema.ts
View File

@@ -284,6 +284,8 @@ export const DEFAULT_PROJECT_SETTINGS = {
specStalenessEnabled: false,
specStalenessMaxAgeMs: 6 * 60 * 60 * 1000,
taskStuckTimeoutMs: 600_000,
engineActiveSinceMs: undefined,
engineActivationGraceMs: 5 * 60_000,
inReviewStallDeadlockThreshold: 3,
stalePausedReviewThresholdMs: 24 * 60 * 60_000,
inReviewStalledThresholdMs: 24 * 60 * 60_000,

14
packages/core/src/stale-paused-review.ts
View File

@@ -15,6 +15,8 @@ export interface StalePausedReviewSignal {
export interface StalePausedReviewContext {
now?: number;
thresholdMs?: number;
engineActiveSinceMs?: number;
engineActivationGraceMs?: number;
}
export const DEFAULT_STALE_PAUSED_REVIEW_THRESHOLD_MS = 24 * 60 * 60_000;
@@ -33,7 +35,9 @@ export function getStalePausedReviewSignal(
const anchor = Date.parse(task.columnMovedAt ?? task.updatedAt);
if (!Number.isFinite(anchor)) return undefined;
const ageMs = now - anchor;
const activationFloorMs = getActivationFloorMs(context);
const effectiveAnchor = activationFloorMs !== undefined ? Math.max(anchor, activationFloorMs) : anchor;
const ageMs = Math.max(0, now - effectiveAnchor);
if (ageMs < thresholdMs) return undefined;
return {
@@ -46,3 +50,11 @@ export function getStalePausedReviewSignal(
pausedByAgentId: task.pausedByAgentId,
};
}
function getActivationFloorMs(context: StalePausedReviewContext): number | undefined {
if (typeof context.engineActiveSinceMs !== "number" || !Number.isFinite(context.engineActiveSinceMs)) {
return undefined;
}
return context.engineActiveSinceMs + Math.max(0, context.engineActivationGraceMs ?? 0);
}

14
packages/core/src/stale-paused-todo.ts
View File

@@ -15,6 +15,8 @@ export interface StalePausedTodoSignal {
export interface StalePausedTodoContext {
now?: number;
thresholdMs?: number;
engineActiveSinceMs?: number;
engineActivationGraceMs?: number;
}
export const DEFAULT_STALE_PAUSED_TODO_THRESHOLD_MS = 24 * 60 * 60_000;
@@ -32,7 +34,9 @@ export function getStalePausedTodoSignal(
const anchor = Date.parse(task.columnMovedAt ?? task.updatedAt);
if (!Number.isFinite(anchor)) return undefined;
const ageMs = now - anchor;
const activationFloorMs = getActivationFloorMs(context);
const effectiveAnchor = activationFloorMs !== undefined ? Math.max(anchor, activationFloorMs) : anchor;
const ageMs = Math.max(0, now - effectiveAnchor);
if (ageMs < thresholdMs) return undefined;
return {
@@ -45,3 +49,11 @@ export function getStalePausedTodoSignal(
pausedByAgentId: task.pausedByAgentId,
};
}
function getActivationFloorMs(context: StalePausedTodoContext): number | undefined {
if (typeof context.engineActiveSinceMs !== "number" || !Number.isFinite(context.engineActiveSinceMs)) {
return undefined;
}
return context.engineActiveSinceMs + Math.max(0, context.engineActivationGraceMs ?? 0);
}

60
packages/core/src/store.ts
View File

@@ -3922,23 +3922,39 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
let disableAgeStalenessHydration = false;
const activeTasks = await Promise.all((rows as unknown as TaskRow[]).map(async (row) => {
const task = this.rowToTask(row);
task.inReviewStall = getInReviewStallReason(task, { now, autoMerge: settings.autoMerge });
task.inReviewStall = getInReviewStallReason(task, {
now,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedReview = getStalePausedReviewSignal(task, {
now,
thresholdMs: settings.stalePausedReviewThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.inReviewStalled = getInReviewStalledSignal(task, {
now,
thresholdMs: settings.inReviewStalledThresholdMs,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedTodo = getStalePausedTodoSignal(task, {
now,
thresholdMs: settings.stalePausedTodoThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!disableAgeStalenessHydration) {
try {
task.ageStaleness = getTaskAgeStalenessSignal(task, { now, thresholds: staleThresholds });
task.ageStaleness = getTaskAgeStalenessSignal(task, {
now,
thresholds: staleThresholds,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
} catch (error) {
if (error instanceof RangeError) {
disableAgeStalenessHydration = true;
@@ -4113,23 +4129,39 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
let disableAgeStalenessHydration = false;
const tasks = rows.slice(0, resolvedLimit).map((row) => {
const task = this.rowToTask(row);
task.inReviewStall = getInReviewStallReason(task, { now, autoMerge: settings.autoMerge });
task.inReviewStall = getInReviewStallReason(task, {
now,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedReview = getStalePausedReviewSignal(task, {
now,
thresholdMs: settings.stalePausedReviewThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.inReviewStalled = getInReviewStalledSignal(task, {
now,
thresholdMs: settings.inReviewStalledThresholdMs,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedTodo = getStalePausedTodoSignal(task, {
now,
thresholdMs: settings.stalePausedTodoThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!disableAgeStalenessHydration) {
try {
task.ageStaleness = getTaskAgeStalenessSignal(task, { now, thresholds: staleThresholds });
task.ageStaleness = getTaskAgeStalenessSignal(task, {
now,
thresholds: staleThresholds,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
} catch (error) {
if (error instanceof RangeError) {
disableAgeStalenessHydration = true;
@@ -4260,23 +4292,39 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
let disableAgeStalenessHydration = false;
const activeMatches = await Promise.all(rows.map(async (row) => {
const task = this.rowToTask(row);
task.inReviewStall = getInReviewStallReason(task, { now, autoMerge: settings.autoMerge });
task.inReviewStall = getInReviewStallReason(task, {
now,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedReview = getStalePausedReviewSignal(task, {
now,
thresholdMs: settings.stalePausedReviewThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.inReviewStalled = getInReviewStalledSignal(task, {
now,
thresholdMs: settings.inReviewStalledThresholdMs,
autoMerge: settings.autoMerge,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
task.stalePausedTodo = getStalePausedTodoSignal(task, {
now,
thresholdMs: settings.stalePausedTodoThresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!disableAgeStalenessHydration) {
try {
task.ageStaleness = getTaskAgeStalenessSignal(task, { now, thresholds: staleThresholds });
task.ageStaleness = getTaskAgeStalenessSignal(task, {
now,
thresholds: staleThresholds,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
} catch (error) {
if (error instanceof RangeError) {
disableAgeStalenessHydration = true;

14
packages/core/src/task-age-staleness.ts
View File

@@ -30,6 +30,8 @@ export const DEFAULT_TASK_AGE_STALENESS_THRESHOLDS: Required<TaskAgeStalenessThr
interface TaskAgeStalenessContext {
now?: number;
thresholds?: TaskAgeStalenessThresholds;
engineActiveSinceMs?: number;
engineActivationGraceMs?: number;
}
type TaskAgeStalenessTask = Pick<Task, "column" | "paused" | "columnMovedAt" | "updatedAt" | "mergeDetails">;
@@ -81,7 +83,9 @@ export function getTaskAgeStalenessSignal(
if (!Number.isFinite(ageAnchorMs)) {
return undefined;
}
const ageMs = Math.max(0, now - ageAnchorMs);
const activationFloorMs = getActivationFloorMs(context);
const effectiveAgeAnchorMs = activationFloorMs !== undefined ? Math.max(ageAnchorMs, activationFloorMs) : ageAnchorMs;
const ageMs = Math.max(0, now - effectiveAgeAnchorMs);
let level: TaskAgeStalenessLevel | undefined;
if (criticalThresholdMs !== undefined && ageMs >= criticalThresholdMs) {
@@ -105,3 +109,11 @@ export function getTaskAgeStalenessSignal(
paused: task.paused === true,
};
}
function getActivationFloorMs(context: TaskAgeStalenessContext): number | undefined {
if (typeof context.engineActiveSinceMs !== "number" || !Number.isFinite(context.engineActiveSinceMs)) {
return undefined;
}
return context.engineActiveSinceMs + Math.max(0, context.engineActivationGraceMs ?? 0);
}

9
packages/core/src/types.ts
View File

@@ -3045,6 +3045,15 @@ export interface ProjectSettings {
* than this duration, the task is considered stuck and will be terminated and retried.
* Default: 600000 (10 minutes). Set to 0 to disable. */
taskStuckTimeoutMs?: number;
/** Epoch ms when the in-process runtime last became active (startup or transition
* out of globalPause/enginePaused). Time-based stuck/stalled/stale detectors floor
* their activity anchor at this value so engine downtime is not counted as quiet time.
* Stamped by the runtime; undefined when no runtime has come up yet. */
engineActiveSinceMs?: number;
/** Extra grace period in milliseconds added to engineActiveSinceMs before any
* time-based stuck/stalled/stale signal may fire after activation.
* Default: 300000 (5 minutes). Set to 0 to disable the grace period. */
engineActivationGraceMs?: number;
/** Minimum number of identical consecutive in-review stall log entries (same code + reason)
* before the task is auto-disposed with `pausedReason='in-review-stall-deadlock'`.
* Default: 3. Set to 0 to disable. */

29
packages/engine/src/__tests__/project-engine.test.ts
View File

@@ -1696,6 +1696,35 @@ describe("ProjectEngine paused in-review auto-merge behavior", () => {
await engine.stop();
});
it("stamps engineActiveSinceMs on global and engine unpause transitions", async () => {
const mockStore = createMockStore({ ...baseSettings, autoMerge: true });
mocks.currentStore = mockStore.store;
const engine = createEngine();
await engine.start();
mockStore.store.updateSettings.mockClear();
await mockStore.emitSettingsUpdated(
{ ...baseSettings, autoMerge: true, globalPause: false },
{ ...baseSettings, autoMerge: true, globalPause: true },
);
await mockStore.emitSettingsUpdated(
{ ...baseSettings, autoMerge: true, enginePaused: false },
{ ...baseSettings, autoMerge: true, enginePaused: true },
);
const activationStampCalls = mockStore.store.updateSettings.mock.calls.filter(
([patch]) => patch && typeof patch === "object" && "engineActiveSinceMs" in patch,
);
expect(activationStampCalls).toHaveLength(2);
for (const [patch] of activationStampCalls) {
expect((patch as { engineActiveSinceMs: unknown }).engineActiveSinceMs).toEqual(expect.any(Number));
}
await engine.stop();
});
it("resumes deferred startup recovery on engine unpause", async () => {
const mockStore = createMockStore(baseSettings);
mocks.currentStore = mockStore.store;

134
packages/engine/src/__tests__/reliability-interactions/engine-active-since-floor.test.ts Normal file
View File

@@ -0,0 +1,134 @@
import { EventEmitter } from "node:events";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { Task, TaskStore } from "@fusion/core";
import { SelfHealingManager } from "../../self-healing.js";
import { StuckTaskDetector } from "../../stuck-task-detector.js";
function makeTask(overrides: Partial<Task> = {}): Task {
return {
id: "FN-5223-RI",
title: "t",
description: "d",
column: "in-review",
paused: false,
status: undefined,
steps: [{ name: "s", status: "done" as const }],
workflowStepResults: [],
dependencies: [],
log: [],
updatedAt: "2026-01-01T00:00:00.000Z",
columnMovedAt: "2026-01-01T00:00:00.000Z",
...overrides,
} as Task;
}
function createStore(tasks: Task[], settingsOverrides: Record<string, unknown> = {}): TaskStore & EventEmitter {
const emitter = new EventEmitter() as TaskStore & EventEmitter;
(emitter as any).getSettings = vi.fn().mockResolvedValue({
autoMerge: true,
globalPause: false,
enginePaused: false,
inReviewStalledThresholdMs: 60_000,
stalePausedReviewThresholdMs: 60_000,
stalePausedTodoThresholdMs: 60_000,
taskStuckTimeoutMs: 60_000,
engineActivationGraceMs: 300_000,
...settingsOverrides,
});
(emitter as any).updateSettings = vi.fn().mockImplementation(async (patch: Record<string, unknown>) => {
const current = await (emitter as any).getSettings();
(emitter as any).getSettings = vi.fn().mockResolvedValue({ ...current, ...patch });
});
(emitter as any).listTasks = vi.fn().mockImplementation(async ({ column }: { column?: string } = {}) => {
if (!column) return tasks;
return tasks.filter((t) => t.column === column);
});
(emitter as any).logEntry = vi.fn().mockResolvedValue(undefined);
(emitter as any).updateTask = vi.fn().mockImplementation(async (taskId: string, updates: Partial<Task>) => {
const task = tasks.find((t) => t.id === taskId);
if (task) Object.assign(task, updates);
return task;
});
(emitter as any).moveTask = vi.fn().mockResolvedValue(undefined);
(emitter as any).recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
return emitter;
}
describe("FN-5223 reliability interactions: engineActiveSince floor", () => {
beforeEach(() => vi.useFakeTimers());
afterEach(() => vi.useRealTimers());
it("FN-5223 suppresses in-review stalled signal immediately after activation and restores after grace+threshold", async () => {
const now = new Date("2026-01-01T01:00:00.000Z");
vi.setSystemTime(now);
const task = makeTask({ id: "FN-5223-Q1" });
const store = createStore([task], { engineActiveSinceMs: now.getTime() });
const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
expect(await manager.surfaceInReviewStalled()).toBe(0);
vi.setSystemTime(new Date(now.getTime() + 7 * 60_000));
expect(await manager.surfaceInReviewStalled()).toBe(1);
manager.stop();
});
it("FN-5223 pause/unpause stamping prevents immediate stale paused review surfacing", async () => {
const now = new Date("2026-01-01T01:00:00.000Z");
vi.setSystemTime(now);
const task = makeTask({ id: "FN-5223-Q2", paused: true });
const store = createStore([task], { engineActiveSinceMs: now.getTime() });
const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
await (store as any).updateSettings({ globalPause: true });
await (store as any).updateSettings({ globalPause: false, engineActiveSinceMs: Date.now() });
expect(await manager.surfaceStalePausedReviews()).toBe(0);
manager.stop();
});
it("FN-5223 composes with globalPause/enginePaused cycle gate", async () => {
vi.setSystemTime(new Date("2026-01-01T01:00:00.000Z"));
const task = makeTask({ id: "FN-5223-Q3" });
const store = createStore([task], { globalPause: true, engineActiveSinceMs: Date.now() - 60_000 });
const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
expect(await manager.surfaceInReviewStalled()).toBe(0);
manager.stop();
});
it("FN-5223 grace 0 disables warmup", async () => {
const now = new Date("2026-01-01T01:00:00.000Z");
vi.setSystemTime(now);
const task = makeTask({ id: "FN-5223-Q4" });
const store = createStore([task], {
inReviewStalledThresholdMs: 1,
engineActivationGraceMs: 0,
engineActiveSinceMs: now.getTime() - 10_000,
});
const manager = new SelfHealingManager(store, { rootDir: "/tmp/repo" });
expect(await manager.surfaceInReviewStalled()).toBe(1);
manager.stop();
});
it("FN-5223 stuck detector resume hook refreshes tracked timestamps independent of persisted clock", async () => {
const now = new Date("2026-01-01T01:00:00.000Z");
vi.setSystemTime(now);
const store = createStore([]);
const detector = new StuckTaskDetector(store);
const session = { dispose: vi.fn() };
detector.trackTask("FN-5223-Q5", session);
const trackedBeforePause = (detector as any).tracked.get("FN-5223-Q5");
expect(trackedBeforePause.lastActivity).toBe(now.getTime());
detector.pause();
vi.setSystemTime(new Date(now.getTime() + 2 * 60_000));
detector.resume();
const trackedAfterResume = (detector as any).tracked.get("FN-5223-Q5");
expect(trackedAfterResume.lastActivity).toBe(now.getTime() + 2 * 60_000);
expect(trackedAfterResume.lastProgressAt).toBe(now.getTime() + 2 * 60_000);
expect(trackedAfterResume.activitySinceProgress).toBe(0);
});
});

64
packages/engine/src/__tests__/self-healing.test.ts
View File

@@ -4702,6 +4702,24 @@ describe("SelfHealingManager", () => {
managerWithRecovery.stop();
});
it("suppresses transient-merge stall surfacing when engine activation floor is recent", async () => {
vi.setSystemTime(new Date("2026-01-01T00:10:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
(store.getSettings as ReturnType<typeof vi.fn>).mockResolvedValue({
taskStuckTimeoutMs: 60_000,
autoMerge: true,
engineActiveSinceMs: Date.parse("2026-01-01T00:10:00.000Z"),
engineActivationGraceMs: 300_000,
});
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([
staleMergingTask({ mergeDetails: { mergeConfirmed: true } }),
]);
expect(await managerWithRecovery.surfaceInReviewStalls()).toBe(0);
expect(store.logEntry).not.toHaveBeenCalled();
managerWithRecovery.stop();
});
it("skips entirely when autoMerge is disabled", async () => {
vi.setSystemTime(new Date("2026-01-01T00:10:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
@@ -4998,6 +5016,22 @@ describe("SelfHealingManager", () => {
managerWithRecovery.stop();
});
it("suppresses quiet in-review surfacing when engine activation floor is recent", async () => {
vi.setSystemTime(new Date("2026-01-02T01:00:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
(store.getSettings as ReturnType<typeof vi.fn>).mockResolvedValue({
inReviewStalledThresholdMs: 24 * 60 * 60_000,
autoMerge: true,
engineActiveSinceMs: Date.parse("2026-01-02T01:00:00.000Z"),
engineActivationGraceMs: 300_000,
});
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([inReviewTask()]);
expect(await managerWithRecovery.surfaceInReviewStalled()).toBe(0);
expect(store.logEntry).not.toHaveBeenCalled();
managerWithRecovery.stop();
});
it("skips for recent activity, paused, global pause, engine pause, autoMerge off, threshold off, executing, and active merge", async () => {
vi.setSystemTime(new Date("2026-01-02T01:00:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, {
@@ -5089,6 +5123,21 @@ describe("SelfHealingManager", () => {
managerWithRecovery.stop();
});
it("suppresses stale paused review surfacing when engine activation floor is recent", async () => {
vi.setSystemTime(new Date("2026-01-02T01:00:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
(store.getSettings as ReturnType<typeof vi.fn>).mockResolvedValue({
stalePausedReviewThresholdMs: 24 * 60 * 60_000,
engineActiveSinceMs: Date.parse("2026-01-02T01:00:00.000Z"),
engineActivationGraceMs: 300_000,
});
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([pausedReviewTask()]);
expect(await managerWithRecovery.surfaceStalePausedReviews()).toBe(0);
expect(store.logEntry).not.toHaveBeenCalled();
managerWithRecovery.stop();
});
it("logs disposition recommendation when threshold met", async () => {
vi.setSystemTime(new Date("2026-01-02T01:00:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
@@ -5187,6 +5236,21 @@ describe("SelfHealingManager", () => {
managerWithRecovery.stop();
});
it("suppresses stale paused todo surfacing when engine activation floor is recent", async () => {
vi.setSystemTime(new Date("2026-01-02T01:00:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });
(store.getSettings as ReturnType<typeof vi.fn>).mockResolvedValue({
stalePausedTodoThresholdMs: 24 * 60 * 60_000,
engineActiveSinceMs: Date.parse("2026-01-02T01:00:00.000Z"),
engineActivationGraceMs: 300_000,
});
(store.listTasks as ReturnType<typeof vi.fn>).mockResolvedValue([pausedTodoTask()]);
expect(await managerWithRecovery.surfaceStalePausedTodos()).toBe(0);
expect(store.logEntry).not.toHaveBeenCalled();
managerWithRecovery.stop();
});
it("skips under threshold and for unpaused/non-todo tasks", async () => {
vi.setSystemTime(new Date("2026-01-01T00:10:00.000Z"));
const managerWithRecovery = new SelfHealingManager(store, { rootDir: "/tmp/test-project" });

8
packages/engine/src/project-engine.ts
View File

@@ -2416,6 +2416,14 @@ export class ProjectEngine {
);
}
try {
await store.updateSettings({ engineActiveSinceMs: Date.now() });
} catch (err: unknown) {
runtimeLog.warn(
`${source}: failed to stamp engineActiveSinceMs: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (settings.globalPause || settings.enginePaused || !settings.autoMerge) {
return;
}

16
packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts
View File

@@ -20,6 +20,7 @@ const {
mockResumeOrphaned,
mockTaskStoreSettings,
mockTaskStoreGetTask,
mockTaskStoreUpdateSettings,
mockMessageStoreSetHook,
mockSchedulerConfigurePrMonitoring,
mockIsGitRepository,
@@ -37,6 +38,7 @@ const {
mockResumeOrphaned: vi.fn().mockResolvedValue(undefined),
mockTaskStoreSettings: {} as Record<string, unknown>,
mockTaskStoreGetTask: vi.fn().mockResolvedValue(null),
mockTaskStoreUpdateSettings: vi.fn().mockResolvedValue(undefined),
mockMessageStoreSetHook: vi.fn(),
mockSchedulerConfigurePrMonitoring: vi.fn(),
mockIsGitRepository: vi.fn().mockResolvedValue(true),
@@ -72,6 +74,7 @@ vi.mock("@fusion/core", async () => {
self.updateTask = vi.fn().mockImplementation(async (taskId: string, patch: Record<string, unknown>) => ({ id: taskId, ...patch }));
self.moveTask = vi.fn().mockResolvedValue(undefined);
self.getSettings = vi.fn().mockImplementation(async () => structuredClone(mockTaskStoreSettings));
self.updateSettings = mockTaskStoreUpdateSettings;
self.getMissionStore = vi.fn().mockReturnValue({
listMissions: vi.fn().mockReturnValue([]),
getMissionWithHierarchy: vi.fn().mockReturnValue(null),
@@ -286,6 +289,19 @@ describe("InProcessRuntime", () => {
expect(runtime.getStatus()).toBe("active");
}, 30000);
it("stamps engineActiveSinceMs during runtime start", async () => {
const before = Date.now();
await runtime.start();
const after = Date.now();
expect(mockTaskStoreUpdateSettings).toHaveBeenCalledWith(
expect.objectContaining({ engineActiveSinceMs: expect.any(Number) }),
);
const stamp = (mockTaskStoreUpdateSettings.mock.calls.at(-1)?.[0] as { engineActiveSinceMs: number }).engineActiveSinceMs;
expect(stamp).toBeGreaterThanOrEqual(before);
expect(stamp).toBeLessThanOrEqual(after);
});
it("does not spawn real git subprocesses during start()", async () => {
const execSpy = vi.spyOn(childProcess, "exec");
const execFileSpy = vi.spyOn(childProcess, "execFile");

7
packages/engine/src/runtimes/in-process-runtime.ts
View File

@@ -777,6 +777,13 @@ export class InProcessRuntime
// 14. Start MissionAutopilot background polling
this.missionAutopilot?.start();
try {
await this.taskStore.updateSettings({ engineActiveSinceMs: Date.now() });
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
runtimeLog.warn(`Failed to stamp engineActiveSinceMs on runtime start: ${message}`);
}
this.setStatus("active");
runtimeLog.log(`InProcessRuntime started for project ${this.config.projectId}`);
} catch (error) {

18
packages/engine/src/self-healing.ts
View File

@@ -4130,6 +4130,8 @@ export class SelfHealingManager {
executingTaskIds,
staleMergingMinAgeMs: this.options.staleMergingStatusMinAgeMs ?? DEFAULT_STALE_MERGING_STATUS_MIN_AGE_MS,
maxAutoMergeRetries: MAX_AUTO_MERGE_RETRIES,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!signal) continue;
@@ -4266,6 +4268,8 @@ export class SelfHealingManager {
autoMerge: true,
activeMergeTaskId,
executingTaskIds,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!signal) continue;
@@ -4315,7 +4319,12 @@ export class SelfHealingManager {
for (const task of tasks) {
if (task.paused !== true) continue;
const signal = getStalePausedReviewSignal(task, { now: cycleStartMs, thresholdMs });
const signal = getStalePausedReviewSignal(task, {
now: cycleStartMs,
thresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!signal) continue;
if (Date.parse(task.updatedAt) >= cycleStartMs) continue;
@@ -4361,7 +4370,12 @@ export class SelfHealingManager {
for (const task of tasks) {
if (task.paused !== true) continue;
const signal = getStalePausedTodoSignal(task, { now: cycleStartMs, thresholdMs });
const signal = getStalePausedTodoSignal(task, {
now: cycleStartMs,
thresholdMs,
engineActiveSinceMs: settings.engineActiveSinceMs,
engineActivationGraceMs: settings.engineActivationGraceMs,
});
if (!signal) continue;
if (Date.parse(task.updatedAt) >= cycleStartMs) continue;