## Why
FN-7996 sat in a dispatch→park loop **all day** (06:06→16:35): its
`worktree` metadata pointed at recycled pool worktrees (`coral-badger`,
`grand-ridge` — the latter actually belonged to FN-8069), Plan Review
refused to start in the missing directory, and the task terminal-parked
`failed` every cycle while the planner overseer blindly retried.
Root-cause chain:
1. **`graphFailureValue()` couldn't read optional-group results.**
`runOptionalGroup` publishes context under the group id
(`node:plan-review:value`) and the unqualified template id, but the
failed node is recorded as the materialized
`plan-review::plan-review-step` — the lookup only understood `#` foreach
ids. FN-7977's provider-failure hold *did* classify this failure, but
its hold value was invisible to routing.
2. **No graph-failure router handled the `assertValidWorktreeSession`
refusal**, so it fell to the terminal sink, which parked the task and
*overwrote* `task.error` with a generic message — erasing the signature
the missing-worktree self-healing sweep (in-review-only anyway)
classifies on.
3. **Plan Review didn't need the worktree at all** — its spec is
store-injected (FN-7561) — yet it launched its reviewer in whatever
stale `task.worktree` said.
## What
- `handleGraphFailure` routes unusable-worktree node failures (any node,
any error key, `::`/`#` materialized ids) into the existing bounded
worktree-session recovery: clear stale worktree/branch/session metadata,
requeue to todo, budgeted by `worktreeSessionRetryCount`. An exhausted
budget still falls through to the visible terminal park for human
inspection.
- `graphFailureValue` resolves `group::template` ids (group value first
— it carries post-classification routing intent — then the unqualified
template value). Foreach `#` behavior unchanged.
- Plan Review falls back to the repo root when its recorded worktree is
missing on disk; other read-only gates intentionally keep failing fast
into the new recovery (silently retargeting them to root would review
the wrong tree).
- `recoverMissingWorktreeSessionStartFailure` returns its outcome so the
graph router can distinguish requeue from escalate-exhausted; existing
truthy callers unchanged.
## Symptom Verification
- **Original symptom:** graph-node session-start refusal → `Workflow
graph terminated with failure at node 'plan-review::plan-review-step'`,
task parked failed with stale metadata intact, no recovery.
- **Reproduction:** `graph-node-missing-worktree-recovery.test.ts`
drives `handleGraphFailure` with the exact FN-7996 result shape
(optional-group materialized id + `Refusing to start coding agent in
missing worktree` node error).
- **Assertion it is gone:** the task is requeued to `todo` with
`worktree`/`branch`/`sessionFile` cleared and retry budget incremented —
and is *not* marked `failed`; budget exhaustion still parks visibly.
## Surface Enumeration
- Optional-group template nodes (Plan Review — the repro), write-capable
review gates, and any custom graph node: covered by the
`handleGraphFailure` router (scans exact/materialized/unqualified
`:error` keys).
- Execute-seam session start: already covered by the pre-existing
recovery (unchanged, still passes).
- In-review / merge-active columns: already covered by self-healing
sweeps (unchanged).
- Paused / user-paused / deleted / done tasks: explicitly left to their
owning machinery (guard tests).
- Budget exhaustion: falls through to the visible terminal park (test).
## Testing
- `pnpm --filter @fusion/engine exec vitest run
src/__tests__/reliability-interactions/graph-node-missing-worktree-recovery.test.ts`
— 13 passed
- Adjacent suites (`worktree-incomplete-session-start`,
`executor-graph-requeue-gate`, `workflow-graph-optional-group`,
`executor-paused-abort-todo-benign`) — 78 passed
- `tsc --noEmit` on `@fusion/engine` — clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved recovery when workflow tasks encounter missing or recycled
worktrees.
* Automatically retries affected tasks with stale worktree details
cleared, up to the configured retry limit.
* Escalates tasks after recovery attempts are exhausted.
* Improved failure routing for optional workflow groups and template
instances.
* Plan Review now falls back to the repository root when its recorded
worktree is unavailable.
* **Tests**
* Added regression coverage for recovery, routing, retry limits, and
repository-root fallback behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Tasks piled up in the Planning column and never moved. hasAdvancedPastPlanning
counted steps.length > 0 as proof a card had advanced past planning, but a
replan card legitimately retains the steps its previous planning pass
materialized. The still-in-planning guard therefore failed for every card
Plan Review sent back, so triage's specifyTask claim silently skipped its
status:"planning" write and re-claimed the same cards every poll — never
planning them, and starving healthy cards out of the maxTriageConcurrent
slots they held.
Steps are no longer advancement evidence while a card sits in a planner lane:
the "triage" column, and the merged "todo" planner lane used by plan-in-place
workflows when the card carries a planning status. Worktrees and
execution/terminal columns remain durable advancement evidence, preserving
FN-7977's protection against a recovery write clobbering a card that raced
ahead into execution.
The primary claim path now warns instead of returning silently; recovery-write
skips stay silent by design. The silence is why this stalled the planner for
hours undiagnosed.
Regression coverage asserts the invariant across both planner surfaces rather
than the reported repro alone: triage cards with and without an explicit
needs-replan status, plan-in-place todo replans, every parked-for-planning
status, and the advancement signals that must still fire.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two follow-ups to FN-8004. Both were found by watching FN-8004's *own*
merge livelock for 40 minutes — it turned out to be blocked by the very
class of bug it was filed to fix.
## 1. AI merge rejections lost their reasons
The reviewer prompt said **both** of these:
> "**End with a single decision line**: `REVIEW_VERDICT:
approve|reject`"
> "**Then list each concrete reason as a bullet.**"
Those are impossible to satisfy at once. Reviewers obeyed "End with" and
wrote their reasoning *above* the verdict — but `extractRejectReasons`
only scanned lines *after* it. So every such rejection collapsed to the
placeholder `reviewer rejected the merge without a stated reason`, and
that placeholder was then handed to the corrective re-merge pass **as
its instruction**. The pass got no actionable feedback and just
re-rolled the merge.
The evidence, from FN-8004's own merge — the pattern repeated across
*both* attempts:
| | Attempt A | Attempt B |
|---|---|---|
| review pass 1 | rejected, no reason (03:46) | rejected, no reason
(03:57) |
| corrective pass | 1/3 | 1/3 |
| review pass 2 | **approved** `a3a3cc6a8` (03:49) | approved |
A reviewer that rejects and then approves identical content isn't
objecting — the reason was being thrown away. Each wasted cycle cost ~7
minutes, stretching the merge past main's ~8-minute churn window so
every attempt lost to a concurrent advance and rebuilt. **The livelock
was caused by the lost-reason bug.**
Fix: the parser recovers reasons from either side of the verdict (inline
→ after → before, nearest-first so the closing argument leads, capped at
8 so a long transcript can't flood the corrective prompt), skipping
severity/verdict/markdown scaffolding. The prompt ordering is now
unambiguous — reasons first, verdict last, nothing after it.
## 2. An orphaned merge-active stamp was un-retryable by hand
The Retry gate refused **every** merge-active status (`Task is not in a
retryable state (current status: landing)`), while self-healing cleared
stale stamps automatically minutes later. So a merger killed mid-flight
— crash, engine restart, operator SIGTERM — blocked the operator's own
escape hatch at exactly the moment they'd reach for it. FN-8004 hit
this: a killed merge left `landing` stamped and Retry 400'd for the full
sweep delay.
`isStaleMergeActiveStatus` now lives in the leaf
`merge-active-status.ts`, shared by `recoverStaleMergingStatus` and the
Retry gate — so **the manual path can never be stricter than the
automatic one**. This is the same one-concept-two-definitions bug as
FN-8004's transient classifier, which is why it's worth fixing
structurally rather than adding another special case.
A live merge stays protected by two independent signals: it holds the
in-process lease **and** refreshes `updatedAt` each phase. Staleness
fails closed on an unparseable timestamp.
One subtlety worth reviewing: the bypass feeds `isInReviewRetry` rather
than only the gate. A bare gate bypass would fall through to the generic
branch and move fully-executed work to `todo`, **re-running finished
work** — a bug this fix could easily have introduced.
## Verification
- Gate green (294 + 122 + 63) · lint clean · engine + dashboard
typecheck clean · `verify:fast` PASS
- 70 merger-suite tests green; all 7 pre-existing verdict-parser tests
still pass (backward compatible — none of them covered the verdict-last
layout, which is exactly why this shipped)
- **The route regression test was confirmed non-vacuous**: neutralizing
the fix fails the two "now retryable" cases while the three
live-merge-protection cases still pass, proving they guard real behavior
rather than the new code.
- Regression tests assert the invariant across every surface per *Fix
the Invariant, Not the Repro*: all five `ACTIVE_MERGE_STATUSES` (a
merger can die in any phase, not just the reported `landing`), both
live-merge signals, boundary conditions, fail-closed paths, and that
pre-existing retry paths are unchanged. Test files carry the required
`## Symptom Verification` and `## Surface Enumeration` sections.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* AI merge rejections now reliably include concrete, correctly ordered
reasons, even when provided before the verdict line.
* Manual retry can recover tasks stuck in stale merge-processing states.
* Retry is still blocked for tasks tied to active merge activity or
recently updated/advancing merges.
* Existing failed-merge retry behavior remains unchanged.
* **Reliability**
* Improved shared handling of “orphaned” merge-active detection across
the engine and dashboard.
* **Tests**
* Added/expanded coverage for merge-active staleness, retry eligibility,
and verdict/reason parsing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A task soft-deleted concurrently with a heartbeat-driven moveTask raised
TaskDeletedError from the engine's own board path, leaving the agent in `error`
with a non-empty lastError and requiring a stop/start cycle to recover.
The race is benign by construction: the task is gone, so the move is a no-op.
The heartbeat now classifies it via isConcurrentSoftDeleteRaceError (matching the
canonical message and serialized/typed forms), keeps the agent active, clears
stale error/recovery state, and emits agent:heartbeat-move-skipped-soft-delete
with ids/counts-only metadata. Concurrent operator pauses are preserved.
Squash-merged by hand from fusion/fn-8004. The engine's AI merge approved this
content twice (squash a3a3cc6a8) but could not land it: main advances every ~8
minutes and each merge cycle took ~10, so every attempt lost to a concurrent
advance and rebuilt. Each cycle also burned a corrective pass on a first-pass
review rejection with no stated reason — the issue #1946 class of bug that this
task's own report cites as a sibling.
Reconciled against #2157, which refactored transient-error-detector.ts: the new
classifier coexists with the extracted transient-error-patterns.ts leaf. Verified
on the merged tree — 123 tests green across FN-8004's suites and #2157's,
engine typecheck clean.
Fusion-Task-Id: FN-8004
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rate-limited Plan Review re-ran every 30s for hours (~1,900 requests
per 5h window, reviewerFallbackRetryCount observed past 100), which is
the request volume that trips a provider's low-interactivity throttle —
so the retry storm prolonged the very outage it was retrying.
Root cause: runPlanReviewBeforeExecution catches every reviewStep throw
inline to keep triage alive, which converts them all to an UNAVAILABLE
verdict. That laundering had two consequences the earlier fixes missed:
FN-8006 terminalized RetryStormError and the reviewer started throwing
ReviewerProviderError for 429s, but a ReviewerProviderError still landed
in the UNAVAILABLE park — a FIXED 30s nextRecoveryAt with no attempt
counter and no cap. The reviewer's own escalation contract ("escalate so
UsageLimitPauser pauses every lane") held only on the executor path,
because the inline catch hid the error from triage's usage-limit handler
in specifyTask.
- triage: fire usageLimitPauser.onUsageLimitHit for usage-limit reviewer
failures, so a 429 pauses every lane instead of re-parking one task.
- triage: re-park via computeRecoveryDecision (60s/120s/240s, ±10%
jitter) and terminalize at MAX_RECOVERY_RETRIES. A reviewer that never
yields a verdict is a real failure and must surface, not spin.
- triage: clear the borrowed recoveryRetryCount budget on any real
verdict, so surviving an outage cannot shorten the executor's later
transient budget.
- core: RetryStormError takes an optional cause, surfaced as
underlyingError in serializeRetryStormError and folded into the
message, so a cap no longer masks the real error. recordRetry threads
it from the reviewer's error path.
Surface enumeration: the park is driven by a thrown provider error, a
thrown generic error, and a plain UNAVAILABLE verdict with no throw.
All three are covered — a repro pinned only to the reported 429 would
leave the other two spinning on the old fixed timer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What happened
FN-8004's implementation work finished and passed review. The auto-merge
then failed with `Grok ACP turn failed: Internal error` — a ~20 second
provider blip — and the task was parked `status: "failed"` with 8 files
of complete, reviewed work stranded on its branch.
The park is the interesting part: `status: "failed"` is precisely what
tells recovery to stop. So a misclassification here isn't a missed
retry, it's **terminal**. Both recovery paths were disabled by the same
wrong verdict:
- `maybeRetryTransientMerge` (inline, 3 retries w/ backoff) — never
fired once (`mergeTransientRetryCount: 0`).
- `recoverTransientMergeFailures` (self-healing sweep, exists exactly to
rescue parked in-review tasks) — skipped it, gated on the same
classifier.
## Three defects fixed
**1. No AI-provider failure class existed.** The AI merge drives a real
LLM turn, but `classifyTransientMergeError` only modeled git/lease/spawn
faults. Adds `ai-provider-turn-failure`.
**2. ACP dropped the error detail.** `promptAcpSession` rethrew the SDK
error unchanged, discarding the JSON-RPC `code`/`data` — the only
evidence the fault was provider-side. ("Internal error" is just the
standard text for `-32603`.) It now preserves them, keeping the original
as `cause`:
```
Internal error (acp rpc code -32603, retryable)
```
Classification anchors on that envelope, **not** on the bare `"Internal
error"` — matching that unanchored would disguise genuine application
defects as retryable blips. Only provider-fault codes (`-32603`,
`-32000`..`-32003`) are retryable; caller-fault codes
(`-32600`..`-32602`) stay permanent, since retrying just repeats the
failing call.
**3. Sweep/inline asymmetry** (found while tracing; latent and
unreported). The inline gate accepted `isTransientError(msg) ||
classify(msg)`, but the sweep consulted **only** the classifier. So
`ECONNRESET` / `socket hang up` during a merge earned inline retries and
then went **invisible to the sweep** once parked — stranded forever. The
classifier now delegates to `isTransientError`, so both gates agree by
construction.
To keep that delegation from importing the detector's
`usage-limit-detector → logger` chain (the chain FN-5627 split the
classifier out to avoid, which would break
`notification-service.test.ts`'s partial `vi.mock`), the pure predicates
moved to the import-free leaf `transient-error-patterns.ts`, re-exported
from `transient-error-detector.ts`. All 13 exports preserved, verified
programmatically.
## Loosened budgets
Per request, so more self-heals. Both apply **only** to errors already
proven transient; the ceiling and
`merger:transient-failure-budget-exhausted` audit path remain.
| Budget | Before | After |
|---|---|---|
| `MAX_AUTO_MERGE_TRANSIENT_RETRIES` | 3 | 5 (backoff
5s/10s/20s/40s/80s) |
| `MAX_TRANSIENT_MERGE_RECOVERIES` | 2 | 5 |
The bump broke two suites that had hardcoded the old `3`. Rather than
swap in another magic number, both now derive the cap from the constant
so future tuning doesn't re-break them.
## Verification
- `pnpm test:gate` green · `pnpm lint` clean · engine + ACP typecheck
clean · `pnpm verify:fast` PASS (5/5)
- ACP plugin 230 tests green · Grok plugin 64 green · engine
transient/merge suites 136 green
- Regression tests assert the **invariant across every surface** (per
*Fix the Invariant, Not the Repro*), not just the reported Grok string:
both ACP runtime prefixes, all retryable/non-retryable rpc codes, both
SDK error shapes, network delegation, class-ordering, and negative cases
proving bare `"Internal error"` and real defects stay permanent.
- A test caught a genuine bug in my own code mid-review (nested-shape
message shadowing), now fixed.
- `notifier.test.ts > "awaiting approval"` fails — **confirmed
pre-existing on clean main**, unrelated.
## Note
FN-8004's own branch (`fusion/fn-8004`) is still unmerged and its work
looks complete. Once this lands, its merge should be retried separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merge node could not observe a graph abort. WorkflowPrimitiveContext
carried no signal, so requestMerge raced the merge only against its own
30-minute GRAPH_MERGE_TIMEOUT_MS using a controller it owned. A hard-cancel
(user cancel, engine restart, pause/resume) aborted the graph controller and
the walk kept sitting inside the merge node for the full timeout. When the
timeout finally fired it aborted the still-running AI merge -- surfacing as
"Manual-merge failed: Request was aborted" -- and the walk reported
value=merge-timeout for a cancellation it had missed half an hour earlier.
An abort landing between merger-ai's `worktree: null` write and
mergeConfirmed then stranded the card as no-worktree-no-merge-confirmed.
Thread the graph AbortSignal from WorkflowNodeExecutionContext (where it
already existed) through primitiveNodeContext/primitiveContextForNode into
the primitives, and honor it on both merge surfaces:
- requestMerge fails fast when the walk is already cancelled, before
ensureWorkflowMergeBoundaryTask mutates the row or the requester enqueues
a merge, and links the graph signal into its timeout controller via
AbortSignal.any -- raced separately so the walk returns on the abort
rather than waiting on a requester that may never settle.
- The legacy merge seam had the identical unguarded race and gets the same
treatment.
The timeout stays: it bounds a wedged merge queue, which is a different
failure from cancellation. Both signals must stay live -- dropping either
silently restores the stall with no type error.
Cancellation returns a distinct `merge-cancelled` rather than reusing
merge-timeout. Returning `data.status: "failed"` would let classifyMergeFailure
read the unknown reason as merge-failed and route the cancellation into
bounded auto-merge retry, re-requesting the merge the operator just cancelled.
Regression test covers both merge surfaces, both cancel timings (pre-flight
and mid-flight), the no-signal back-compat path, the signal plumbing itself,
and the classification boundary. Verified by removing the fix: 7 of 9 cases
fail, with the mid-flight cases hanging until timeout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipping a stale planning-state write is the expected outcome of a normal
scheduler advancement, not an anomaly, so the warn was pure log noise.
Behavior is unchanged; only the two planLog.warn emissions are removed.
Fusion-Task-Id: FN-8024
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`deriveSignalAndSources`'s executor branch never read `task.status`, so a row
parked `status: "failed"` — e.g. the terminal fn_task_done refusal/invariant
park — reported `signal: "progressing"` with the reason "Task is actively
executing in-progress work". The overseer observed a dead task as healthy and
took no action. `failed` was only ever derived for the merger/pull-request
stages, so the sole backstop was the FN-7743 2h stall proxy firing hours later.
This is exactly what FN-7965's audit trail shows: every intervention on a
terminally-parked task was action="observe", reason="Task is actively executing
in-progress work".
Report `failed` so recovery engages on the next poll. This adds no new policy:
a failed executor observation already routes to `retry_step` (executor sources
are `agent-log`, never an ERROR_SOURCE_KIND), bounded by
PLANNER_RECOVERY_MAX_ATTEMPTS and escalated on exhaustion.
Precedence and dedup preserved: `paused` still wins, so an operator/user-paused
row stays `blocked` and is never routed into autonomous recovery; and the reason
is a constant (never interpolating task.error/status) so the FN-7577
`stage|signal|reason` feed dedup still suppresses repeat observations.
Verified: the repro test fails with the branch disabled; paused-precedence,
healthy-card (FN-7577) and dedup guards added; overseer/recovery surfaces
93 passed + core planner-recovery 20/20; engine + dashboard typecheck clean;
`pnpm test:gate` green (294+122+63).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The in-session `fn_task_done` handler parks a task terminally (status=failed,
worktree/branch/sessionFile cleared) once its refusal/invariant retry budget is
exhausted. That write happens inside the live agent session, so the executor's
no-fn_task_done retry loop never observed it and spawned a fresh session anyway.
The retry completed, marked the task done, and dragged a worktree-less row into
the pre-merge graph, where the first write-capable node failed on
`no-worktree-for-write-node` — surfacing as a misleading "Workflow graph
terminated with failure at node 'code-review-remediation'" instead of the real
refusal. Observed on FN-7965 and again live on FN-7981.
Re-read state at the top of the retry loop and honor the park. The status probe
covers all three park sites (invariant-check, explicit refusal, implicit
refusal) rather than the single reported repro.
Deliberately not routed through the FN-4806 reclaim branch: its silent todo
requeue would clear the park and, with the budget already spent, re-park on the
next pickup in a todo->execute->park loop.
The pre-existing reclaim probes could not catch this — they test
`worktree === null`, but the store maps a cleared column to `undefined`
(`task-store/serialization.ts`: `row.worktree || undefined`), so the existing
test only passed because its mock returned a value production never emits.
Tightening that probe regressed 7 fixtures and is left as separate work.
Verified: new tests fail with the guard disabled; engine reliability surfaces
show zero regressions vs baseline (17 pre-existing failures unchanged, 495->499
passing); engine-core gate suite 294/294.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Speeds up **time-to-HTTP-ready** for `fn dashboard` and `fn serve` after
the PostgreSQL cutover without reintroducing the historical 3s
cwd-engine race that degraded webhooks.
- **Dashboard store share (serve parity):** inject the factory-booted
`TaskStore` as `externalTaskStore` so cwd `ensureEngine` does not open a
second pool; share only when store root matches project working
directory (multi-project safe).
- **Serve multi-project:** stop awaiting `startAll()` before listen;
await only the primary engine; background the rest + reconciliation.
- **Defer non-route-critical engine work:** ordered OAuth (refresh →
monitor), automation schedule syncs, and auto-merge **enqueue** after
the engine handle is returnable.
- **Critical-path merge status clear:** still clear stale
`merging`/`merging-pr` before ready so manual merge is not blocked after
crash.
- **Serve `--paused`:** apply `enginePaused` before
`ensureEngine`/`startAll` (dashboard ordering).
- **Stop safety:** generation counter so deferred tails cannot resume
after `stop()` clears `shuttingDown`.
- **Phase timing:** shared `phaseTime` helper, factory substep logs,
serve time-to-listen.
Plan: `docs/plans/2026-07-14-001-feat-faster-startup-plan.md`
## Test plan
- [x] `packages/engine` — `project-engine-manager.test.ts` (path-matched
external store)
- [x] `packages/engine` — `project-engine-deferred-startup.test.ts`
(status clear, OAuth order, stop generation)
- [x] `packages/cli` — `startup-phase.test.ts`
- [x] `packages/cli` — `serve.test.ts` (60 tests, including `--paused`)
- [ ] Local: warm `fn dashboard` / `fn serve` and compare `startup phase
*` / `time-to-listen` logs
- [ ] `pnpm smoke:boot` (real serve `/api/health` on ephemeral port)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance**
* Improved dashboard and serve startup times, including faster
time-to-listen and time-to-ready.
* Moved non-essential background initialization off the critical startup
path.
* Parallelized dashboard service initialization where possible.
* **Reliability**
* Improved multi-project startup handling and project selection.
* Prevented cross-project task-store sharing.
* Added safer shutdown behavior for partially completed startup.
* **Diagnostics**
* Added startup phase timing logs to help identify performance
bottlenecks.
* **Tests**
* Expanded coverage for deferred startup, shutdown, project isolation,
and startup timing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Treat **shared PostgreSQL** (`DATABASE_URL`) as the multi-node durable
data plane; mesh HTTP is membership + optional auth, not task/settings
replication.
- **Peer exchange**: under Postgres backend mode, write queue is
**topology/auth-only**; non-topology pending rows fail rather than
replaying multi-leader task/settings payloads.
- **Mesh routes**: task-ID reserve/commit/abort always hit local shared
allocator rows (ignore remote `coordinatorNodeId`); mesh sync ignores
settings and only exchanges `authMaterial`.
- **Docs**: rewrite multi-project runbook, shared cluster protocol, and
architecture mesh sections for shared-Postgres + claims/leases.
## Context
Follows the SQLite→Postgres cutover. Multiple Fusion nodes can share one
external Postgres while keeping **per-node execution** (worktrees,
processes, claims via `central.task_claims`). Explicit non-goals remain:
scheduler failover and live process migration.
Plan:
`docs/plans/2026-07-15-001-refactor-mesh-shared-postgres-multinode-plan.md`
## Test plan
- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/peer-exchange-service.test.ts`
- [x] `pnpm --filter @fusion/dashboard exec vitest run
src/__tests__/mesh-routes.test.ts`
- [x] `pnpm --filter @fusion/core exec vitest run
src/__tests__/shared-mesh-state.test.ts`
- [ ] CI gate (lint/typecheck/build/gate)
- [ ] Manual (optional): two processes, same `DATABASE_URL`, create task
on A visible on B; settings change without mesh settings sync; claim
exclusivity
## Operator note
Multi-node shared board requires **external** `DATABASE_URL` on every
node. Default embedded Postgres is still single-host.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved multi-node deployments using shared PostgreSQL as the durable
source of execution state.
* Task ID reservation/commit/abort now run locally (no remote
coordinator forwarding).
* Mesh syncing now prioritizes topology visibility and authentication
material; settings replication is disabled in shared-Postgres mode.
* **Bug Fixes**
* Prevented task/settings replication over mesh HTTP in shared-Postgres
deployments.
* Refined lease ownership, recovery, and reconciliation to converge via
shared-database primitives.
* **Documentation**
* Updated architecture and shared-mesh protocol guidance, including
multi-node setup and lease/task-ID allocation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
merger-ai's local-checkout sync stashed under fusion-ai-merge-sync-<taskId>,
a label none of merger.ts's reclamation machinery matches — every path keys
off the fusion-merger-autostash: prefix. Those entries were never classified,
never subsumed-dropped, never age-swept, and never surfaced as orphans holding
work, so they accumulated indefinitely: six entries dating back a month were
found on one working tree, and their age made real lost work indistinguishable
from litter. merger-ai now labels through buildAutostashLabel, and the legacy
prefix stays recognized so already-leaked entries are reclaimed rather than
stranded in developers' stash lists.
Routing them into that machinery first required fixing what it does with
untracked files. A stash created with --include-untracked keeps them in a
third parent (<sha>^3) that git stash show omits, so an untracked-only stash
read as empty — and all three copies of the liveness check treated empty as
"subsumed, safe to drop". Every leaked ai-sync stash carried untracked files,
so the fix would otherwise have destroyed the work it was meant to reclaim.
Liveness now resolves through one authority, classifyStashContent, which reads
both sides, diffs untracked paths against <sha>^3 rather than the stash commit
(whose tree never contained them), and treats unreadable state as unknown and
therefore undroppable.
Age-based sweeping is left alone: it drops by timestamp without consulting
content, which is deliberate bounded retention and the backstop against this
same accumulation, not a safety gap.
Regression test uses real git — the defect lives in git's stash object model,
so a mocked git can neither express nor catch it — and asserts the invariant
across tracked-only, untracked-only, and mixed stashes in both live and
subsumed states. The mixed shape (tracked subsumed, untracked live) is the one
that silently lost work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An operator was re-asked to approve a plan they had already approved and
that had not changed.
POST /tasks/:id/approve-plan fingerprints the on-disk PROMPT.md, so a plan
approved before the `## Original Description` hygiene injection
(applyOriginalDescription) shipped carries a hash over PRE-injection
content. On the task's next pass the injection rewrites PROMPT.md, the
fingerprint moves, and FN-7569's idempotency short-circuit misses — so the
manual gate re-parks an unchanged, already-approved plan.
finalizeApprovedTask now also compares the recorded fingerprint against the
as-read (pre-injection) content. This does not weaken the gate: `written`
diverges from `writtenInput` only via that injection, so both arms hash
bytes the operator actually approved — only the representation differs. A
genuinely changed plan matches neither arm and still parks.
On a legacy match the stored fingerprint is migrated forward, so the
reconciliation is one-time per task rather than a comparison carried
forever. The migration is a direct updateTask — the taskUpdates batch is
flushed well before this gate runs.
Covers both finalizeApprovedTask callers (direct + recoverApprovedTask),
asserts the changed-plan safety edge still parks, and asserts no redundant
fingerprint write when the approval is already post-hygiene.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recoverApprovedTask idempotency test failed deterministically, and its
siblings passed for the wrong reason. Both traced to the same stale fixture,
not a product defect.
finalizeApprovedTask injects `## Original Description` into PROMPT.md
(applyOriginalDescription) BEFORE computing the approval fingerprint, and
POST /tasks/:id/approve-plan fingerprints the on-disk file — so the
fingerprint an approval records is always over post-injection content. The
fixtures wrote RAW planner text and fingerprinted that, modelling a state
approve-plan can never produce: the injection then rewrote the content, the
fingerprint moved, and the short-circuit looked broken.
Verified the product is correct: the injection is idempotent, so the real
approve -> recover round-trip fingerprint matches (checked end to end).
- recoverApprovedTask test: write and fingerprint the approved on-disk
content. It now exercises the real short-circuit — the run logs "plan
unchanged since prior approval" then "recovered and moved to todo",
where before it logged "awaiting manual approval".
- same-plan test: it only passed because the injection's rewrite ENOENT'd
(no task dir), the failure was swallowed, and `written` stayed raw — so
the fingerprint matched by accident. Feed it the approved content so the
injection is a genuine no-op and the assertion means something.
Fixtures derive from applyOriginalDescription rather than hard-coding
post-injection text, so they keep meaning "the content the operator
approved" if the hygiene injection changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rate-limited reviewer filled a task's Chat tab with 14 identical
"Reviewer using model: ..." markers and no review text, hammering an
already-limited provider.
Root cause: the reviewer was the only AI lane that never classified
provider errors, so a 429 became an UNAVAILABLE verdict. With no
validator fallback configured the fallback ladder re-ran the SAME model
instantly, and fn_review_step answered with "code review remains
blocking; retry once" — bounding the loop with prompt text rather than
code. The tool's catch-all also swallowed the error into tool output, so
withRateLimitRetry, UsageLimitPauser and RetryStormError never fired.
- reviewer: throw ReviewerProviderError for usage-limit/transient errors
instead of laundering them into UNAVAILABLE, and never spend the
fallback budget (which bounds bad reviews) on an outage.
- reviewer: absorb flaky-network blips in-lane via withRetry with
jittered backoff; rate limits still escalate immediately.
- executor: re-raise the fatal after the prompt via
throwDeferredReviewerFatal — pi-agent-core converts tool throws into
tool_error results, so a tool cannot throw out of session.prompt().
- executor: give code review a real MAX_CODE_REVIEW_UNAVAILABLE_RETRIES
counter, mirroring the plan/spec limiter.
- reviewer: dedupe the model marker on text, so same-model retries stay
silent while a genuine model switch still emits.
Also fixes the run-on rendering: AgentLogType gains `status` for complete
engine messages. `text` means "streamed delta" and is re-glued with
join(""), which is why N standalone markers rendered as one string. The
split is at the type, not a separator — a separator would reintroduce the
FN-5787/5789/5803 streamed-spacing regression.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both lines fired on every scheduler poll while nothing changed: a held
card re-attempts release each sweep, and every dispatch candidate logged
its resolved node. On a busy board that filled the operator log pane with
"Hold release for FN-XXXX deferred" and "routed to node=local" within
seconds, burying real scheduler events.
Add a Logger.debug() level, off by default and opted into per subsystem
via FUSION_DEBUG, and demote both lines to it. Routing to a remote node
stays at info since it explains where work actually went; only the local
default is demoted. Lines reporting a real transition (capacity
rejection, racing sweep, release failure) are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile active task segment anchors on full Global/Engine unpause so stopped-engine wall-clock does not inflate execution time, reusing the FN-7011 downtime path with a transition-captured heartbeat.
- Pass optional engineLastActiveAtOverride into reconcileActiveTimingForEngineDowntime so unpause callers freeze the stopped-window proof against racing scheduler heartbeats
- Await downtime reconciliation in resumeAfterUnpauseAndSweepInReview before resuming agentic work or sweeping in-review tasks
- Fold Global/Engine unpause into the unified pause-lifecycle listener (single reconcile when both clear together; no-op while either pause remains)
- Soft-fail reconcile errors so unpause resume still proceeds
- Add store and project-engine coverage for override, await-before-resume, dual-source clear, and fail-soft paths; document FN-7975 in AGENTS.md run-audit notes
- Add patch changeset for the operator-facing timing fix
Files changed:
.changeset/fn-7975-engine-pause-active-timing.md | 7 ++
AGENTS.md | 2 +-
.../core/src/__tests__/store-active-timing.test.ts | 86 +++++++++++++
packages/core/src/store.ts | 23 ++--
.../project-engine-unpause-active-timing.test.ts | 94 ++++++++++++++
.../engine/src/__tests__/project-engine.test.ts | 139 +++++++++++++++++++++
packages/engine/src/project-engine.ts | 64 +++++-----
packages/engine/src/self-healing.ts | 6 +-
8 files changed, 378 insertions(+), 43 deletions(-)
Fusion-Task-Id: FN-7975
Fusion-Task-Lineage: 84a46e6f-92bf-452a-ab67-c25ba85cbffb
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Address review findings on the FN-7956 hang fix:
- Per-tool outer timeouts so fn_research_run(wait_for_completion) is not clipped by a flat 60s budget
- Longer budgets for skills install, import/browse, and web_fetch
- Boot-failure cooldown + orphan-boot log when store boot times out
- Log timeout/abort/errors from the extension wrap; clearer host-extension skip reason
- Tests for budgets, research wait, and sessionPurpose forwarding
Plan Review REVISE previously fed feedback without the rejected plan body, so triage rewrote from title/description and looped. Seed the draft for surgical revision, use reviewType spec for the pre-execution gate, and tighten planner/reviewer prompts toward blocking-only REVISE with concrete edits.
Give planner/reviewer pairs more room to converge before escalating, and surface why a task is parked for plan approval—especially plan-review-replan-cap non-convergence—on cards, detail, and notifications.
AI merge review could park forever when the host fusion extension loaded
fn_task_show and booted a second TaskStore without a tool timeout (FN-7956).
- Skip host @runfusion/fusion extensions for sessionPurpose "merger"
- Forward sessionPurpose into createFnAgent for that policy
- Coalesce + 30s-bound extension TaskStore boots; ALS-propagate AbortSignal
- Wrap every extension registerTool execute with 60s timeout/abort fail-closed
- Unit tests for merger host-extension skip and tool timeout helpers
