Raise min heartbeat staleness floor 5m -> 10m
Agents stop heartbeating during long legitimate work (e.g. a verification step blocked on a multi-minute test command). The 5-minute floor could misread a busy agent as dead and reclaim its in-progress task mid-run. Raise MIN_HEARTBEAT_STALENESS_MS to 10 minutes and strengthen the floor test (7-minute-silent fast-interval agent stays healthy — would have read stale under the old 5-minute floor). Engine typecheck clean; heartbeat suite 148/148 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
5
.changeset/heartbeat-staleness-10min.md
Normal file
5
.changeset/heartbeat-staleness-10min.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
Raise the minimum agent heartbeat staleness floor from 5 to 10 minutes. Agents go silent during long-running but legitimate work (notably a verification step running a multi-minute test command, where the agent is blocked awaiting the command and cannot tick/heartbeat). The 5-minute floor could misread such a busy agent as dead and reclaim its in-progress task mid-run; 10 minutes gives long operations room before the liveness gate acts.
|
||||||
@@ -460,15 +460,18 @@ describe("executeHeartbeat", () => {
|
|||||||
expect(section).toMatch(/\| Long Interval \| active \| FN-209 \| .* \| healthy \|/);
|
expect(section).toMatch(/\| Long Interval \| active \| FN-209 \| .* \| healthy \|/);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("buildReportsHealthSection enforces a 5-minute minimum staleness floor", async () => {
|
it("buildReportsHealthSection enforces a 10-minute minimum staleness floor", async () => {
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
const store = createStoreWithAgentForExec();
|
const store = createStoreWithAgentForExec();
|
||||||
vi.mocked(store.getCachedAgent).mockImplementation((id: string) => ({
|
vi.mocked(store.getCachedAgent).mockImplementation((id: string) => ({
|
||||||
id,
|
id,
|
||||||
runtimeConfig: { heartbeatIntervalMs: 1_000 },
|
runtimeConfig: { heartbeatIntervalMs: 1_000 },
|
||||||
}) as unknown as Agent);
|
}) as unknown as Agent);
|
||||||
|
// 7 minutes silent with a 1s interval: under the old 5-minute floor this
|
||||||
|
// would read as stale, but the 10-minute floor must still treat a busy
|
||||||
|
// long-running agent as healthy (e.g. mid multi-minute test command).
|
||||||
vi.mocked(store.getAgentsByReportsTo).mockResolvedValue([
|
vi.mocked(store.getAgentsByReportsTo).mockResolvedValue([
|
||||||
{ id: "agent-fast", name: "Fast Poller", state: "active", taskId: "FN-210", lastHeartbeatAt: new Date(now - 2 * 60_000).toISOString(), updatedAt: new Date(now - 2 * 60_000).toISOString() } as Agent,
|
{ id: "agent-fast", name: "Fast Poller", state: "active", taskId: "FN-210", lastHeartbeatAt: new Date(now - 7 * 60_000).toISOString(), updatedAt: new Date(now - 7 * 60_000).toISOString() } as Agent,
|
||||||
]);
|
]);
|
||||||
const monitor = new HeartbeatMonitor({ store, taskStore: mockTaskStore, rootDir: "/tmp" });
|
const monitor = new HeartbeatMonitor({ store, taskStore: mockTaskStore, rootDir: "/tmp" });
|
||||||
|
|
||||||
|
|||||||
@@ -204,8 +204,11 @@ const REPORTS_STALE_INTERVAL_MULTIPLIER = 1.5;
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Minimum staleness threshold floor for very short heartbeat intervals.
|
* Minimum staleness threshold floor for very short heartbeat intervals.
|
||||||
|
* 10 minutes: long-running but legitimately-busy agents (e.g. a verification
|
||||||
|
* step running a multi-minute test command, during which the agent does not
|
||||||
|
* tick/heartbeat) must not be misread as dead and reclaimed mid-run.
|
||||||
*/
|
*/
|
||||||
const MIN_HEARTBEAT_STALENESS_MS = 5 * 60_000;
|
const MIN_HEARTBEAT_STALENESS_MS = 10 * 60_000;
|
||||||
|
|
||||||
/** Format milliseconds into a human-readable duration string (e.g. "5m", "1h 20m", "2h"). */
|
/** Format milliseconds into a human-readable duration string (e.g. "5m", "1h 20m", "2h"). */
|
||||||
export function formatDuration(ms: number): string {
|
export function formatDuration(ms: number): string {
|
||||||
|
|||||||
Reference in New Issue
Block a user