From 8e9e48881b6f284e5e1011056e7838ad32ab17c4 Mon Sep 17 00:00:00 2001 From: Fusion Date: Tue, 12 May 2026 20:11:20 -0700 Subject: [PATCH] docs(FN-4172): document expected heartbeat gaps - Clarify in docs/agents.md that sub-hour heartbeat ages are normal with the default 1-hour interval - Add FN-4172 investigation notes capturing persisted agent heartbeat measurements and classification rubric - Record the close recommendation and verification results for the FN-4119 follow-up Fusion-Task-Id: FN-4172 --- docs/agents.md | 1 + docs/fn-4172-heartbeat-investigation.md | 38 +++++++++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 docs/fn-4172-heartbeat-investigation.md diff --git a/docs/agents.md b/docs/agents.md index 212ef0629..2dd4d544c 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -1151,6 +1151,7 @@ Health status uses interval-based staleness evaluation: ### Key Behaviors - **Monitoring disabled**: Agents with `runtimeConfig.enabled === false` display "Disabled" — they are NOT falsely labeled as "Unresponsive" +- **Interval-sized gaps are normal**: With the default `heartbeatIntervalMs = 3600000` (1 hour), an agent can legitimately go tens of minutes without a new heartbeat. Ages like 16–50 minutes are expected and should not be treated as unhealthy on interval age alone. - **Consistent across views**: All dashboard surfaces use the same centralized utility, ensuring consistent health labels everywhere - **Auto-refresh**: Health status is refreshed every 30 seconds while views are open to keep status current - **State-first evaluation**: Explicit non-idle states (error, paused, running) take priority over timeout-based evaluation diff --git a/docs/fn-4172-heartbeat-investigation.md b/docs/fn-4172-heartbeat-investigation.md new file mode 100644 index 000000000..14834e059 --- /dev/null +++ b/docs/fn-4172-heartbeat-investigation.md @@ -0,0 +1,38 @@ +# FN-4172 heartbeat investigation + +User steering comment from FN-4119 at `2026-05-12T18:01:35.010Z`: + +> Heartbeats are set to 1 hour so this isn’t a problem + +## Hypothesis + +The six CTO direct reports only look stale because their durable heartbeat interval is `3600000` ms (1 hour), so sub-hour heartbeat ages are expected rather than unhealthy. + +## Current persisted measurements + +Generated from `../../.fusion/fusion.db` at `2026-05-13T02:21:43.854057+00:00`. + +| Agent | State | Interval ms | Timeout ms | Last heartbeat | Age min | Active run | Classification | +|---|---|---:|---:|---|---:|---|---| +| Frontend Engineer | active | 3600000 | 60000 | 2026-05-13T02:14:35.060Z | 7.15 | — | expected | +| Backend Engineer | active | 3600000 | 60000 | 2026-05-13T02:04:04.380Z | 17.66 | — | expected | +| Executor | running | 3600000 | 300000 | 2026-05-13T02:17:51.250Z | 3.88 | run-23a05323 (`active`) | expected | +| Technical Writer | active | 3600000 | 60000 | 2026-05-13T02:11:05.740Z | 10.64 | — | expected | +| QA Engineer | active | 3600000 | 60000 | 2026-05-13T01:30:35.273Z | 51.14 | — | expected | +| CI Engineer | active | 3600000 | 60000 | 2026-05-13T02:18:03.528Z | 3.67 | — | expected | + +Classification rubric: + +- `expected`: `ageOfLastHeartbeat < heartbeatIntervalMs × 1.5` +- `borderline`: `1.5×` to `3×` +- `stale`: `> 3× interval` or an active run older than `heartbeatTimeoutMs × 2` + +## Decision + +**Case A** applies: all six agents are within the expected range for a 1-hour heartbeat interval, and the lone active run (Executor) is well inside the stale-run timeout threshold. Recommendation: **close FN-4119 follow-up for this symptom** and do not pursue additional engine work based on age-of-last-heartbeat alone. Re-open only if a future incident produces concrete evidence of a real orphaned run, lost timer, or dashboard misclassification. + +## Verification + +- `pnpm lint` ✅ +- `pnpm test` ✅ +- `pnpm build` ✅