A task soft-deleted concurrently with a heartbeat-driven moveTask raised
TaskDeletedError from the engine's own board path, leaving the agent in `error`
with a non-empty lastError and requiring a stop/start cycle to recover.
The race is benign by construction: the task is gone, so the move is a no-op.
The heartbeat now classifies it via isConcurrentSoftDeleteRaceError (matching the
canonical message and serialized/typed forms), keeps the agent active, clears
stale error/recovery state, and emits agent:heartbeat-move-skipped-soft-delete
with ids/counts-only metadata. Concurrent operator pauses are preserved.
Squash-merged by hand from fusion/fn-8004. The engine's AI merge approved this
content twice (squash a3a3cc6a8) but could not land it: main advances every ~8
minutes and each merge cycle took ~10, so every attempt lost to a concurrent
advance and rebuilt. Each cycle also burned a corrective pass on a first-pass
review rejection with no stated reason — the issue #1946 class of bug that this
task's own report cites as a sibling.
Reconciled against #2157, which refactored transient-error-detector.ts: the new
classifier coexists with the extracted transient-error-patterns.ts leaf. Verified
on the merged tree — 123 tests green across FN-8004's suites and #2157's,
engine typecheck clean.
Fusion-Task-Id: FN-8004
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rate-limited Plan Review re-ran every 30s for hours (~1,900 requests
per 5h window, reviewerFallbackRetryCount observed past 100), which is
the request volume that trips a provider's low-interactivity throttle —
so the retry storm prolonged the very outage it was retrying.
Root cause: runPlanReviewBeforeExecution catches every reviewStep throw
inline to keep triage alive, which converts them all to an UNAVAILABLE
verdict. That laundering had two consequences the earlier fixes missed:
FN-8006 terminalized RetryStormError and the reviewer started throwing
ReviewerProviderError for 429s, but a ReviewerProviderError still landed
in the UNAVAILABLE park — a FIXED 30s nextRecoveryAt with no attempt
counter and no cap. The reviewer's own escalation contract ("escalate so
UsageLimitPauser pauses every lane") held only on the executor path,
because the inline catch hid the error from triage's usage-limit handler
in specifyTask.
- triage: fire usageLimitPauser.onUsageLimitHit for usage-limit reviewer
failures, so a 429 pauses every lane instead of re-parking one task.
- triage: re-park via computeRecoveryDecision (60s/120s/240s, ±10%
jitter) and terminalize at MAX_RECOVERY_RETRIES. A reviewer that never
yields a verdict is a real failure and must surface, not spin.
- triage: clear the borrowed recoveryRetryCount budget on any real
verdict, so surviving an outage cannot shorten the executor's later
transient budget.
- core: RetryStormError takes an optional cause, surfaced as
underlyingError in serializeRetryStormError and folded into the
message, so a cap no longer masks the real error. recordRetry threads
it from the reviewer's error path.
Surface enumeration: the park is driven by a thrown provider error, a
thrown generic error, and a plain UNAVAILABLE verdict with no throw.
All three are covered — a repro pinned only to the reported 429 would
leave the other two spinning on the old fixed timer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What happened
FN-8004's implementation work finished and passed review. The auto-merge
then failed with `Grok ACP turn failed: Internal error` — a ~20 second
provider blip — and the task was parked `status: "failed"` with 8 files
of complete, reviewed work stranded on its branch.
The park is the interesting part: `status: "failed"` is precisely what
tells recovery to stop. So a misclassification here isn't a missed
retry, it's **terminal**. Both recovery paths were disabled by the same
wrong verdict:
- `maybeRetryTransientMerge` (inline, 3 retries w/ backoff) — never
fired once (`mergeTransientRetryCount: 0`).
- `recoverTransientMergeFailures` (self-healing sweep, exists exactly to
rescue parked in-review tasks) — skipped it, gated on the same
classifier.
## Three defects fixed
**1. No AI-provider failure class existed.** The AI merge drives a real
LLM turn, but `classifyTransientMergeError` only modeled git/lease/spawn
faults. Adds `ai-provider-turn-failure`.
**2. ACP dropped the error detail.** `promptAcpSession` rethrew the SDK
error unchanged, discarding the JSON-RPC `code`/`data` — the only
evidence the fault was provider-side. ("Internal error" is just the
standard text for `-32603`.) It now preserves them, keeping the original
as `cause`:
```
Internal error (acp rpc code -32603, retryable)
```
Classification anchors on that envelope, **not** on the bare `"Internal
error"` — matching that unanchored would disguise genuine application
defects as retryable blips. Only provider-fault codes (`-32603`,
`-32000`..`-32003`) are retryable; caller-fault codes
(`-32600`..`-32602`) stay permanent, since retrying just repeats the
failing call.
**3. Sweep/inline asymmetry** (found while tracing; latent and
unreported). The inline gate accepted `isTransientError(msg) ||
classify(msg)`, but the sweep consulted **only** the classifier. So
`ECONNRESET` / `socket hang up` during a merge earned inline retries and
then went **invisible to the sweep** once parked — stranded forever. The
classifier now delegates to `isTransientError`, so both gates agree by
construction.
To keep that delegation from importing the detector's
`usage-limit-detector → logger` chain (the chain FN-5627 split the
classifier out to avoid, which would break
`notification-service.test.ts`'s partial `vi.mock`), the pure predicates
moved to the import-free leaf `transient-error-patterns.ts`, re-exported
from `transient-error-detector.ts`. All 13 exports preserved, verified
programmatically.
## Loosened budgets
Per request, so more self-heals. Both apply **only** to errors already
proven transient; the ceiling and
`merger:transient-failure-budget-exhausted` audit path remain.
| Budget | Before | After |
|---|---|---|
| `MAX_AUTO_MERGE_TRANSIENT_RETRIES` | 3 | 5 (backoff
5s/10s/20s/40s/80s) |
| `MAX_TRANSIENT_MERGE_RECOVERIES` | 2 | 5 |
The bump broke two suites that had hardcoded the old `3`. Rather than
swap in another magic number, both now derive the cap from the constant
so future tuning doesn't re-break them.
## Verification
- `pnpm test:gate` green · `pnpm lint` clean · engine + ACP typecheck
clean · `pnpm verify:fast` PASS (5/5)
- ACP plugin 230 tests green · Grok plugin 64 green · engine
transient/merge suites 136 green
- Regression tests assert the **invariant across every surface** (per
*Fix the Invariant, Not the Repro*), not just the reported Grok string:
both ACP runtime prefixes, all retryable/non-retryable rpc codes, both
SDK error shapes, network delegation, class-ordering, and negative cases
proving bare `"Internal error"` and real defects stay permanent.
- A test caught a genuine bug in my own code mid-review (nested-shape
message shadowing), now fixed.
- `notifier.test.ts > "awaiting approval"` fails — **confirmed
pre-existing on clean main**, unrelated.
## Note
FN-8004's own branch (`fusion/fn-8004`) is still unmerged and its work
looks complete. Once this lands, its merge should be retried separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merge node could not observe a graph abort. WorkflowPrimitiveContext
carried no signal, so requestMerge raced the merge only against its own
30-minute GRAPH_MERGE_TIMEOUT_MS using a controller it owned. A hard-cancel
(user cancel, engine restart, pause/resume) aborted the graph controller and
the walk kept sitting inside the merge node for the full timeout. When the
timeout finally fired it aborted the still-running AI merge -- surfacing as
"Manual-merge failed: Request was aborted" -- and the walk reported
value=merge-timeout for a cancellation it had missed half an hour earlier.
An abort landing between merger-ai's `worktree: null` write and
mergeConfirmed then stranded the card as no-worktree-no-merge-confirmed.
Thread the graph AbortSignal from WorkflowNodeExecutionContext (where it
already existed) through primitiveNodeContext/primitiveContextForNode into
the primitives, and honor it on both merge surfaces:
- requestMerge fails fast when the walk is already cancelled, before
ensureWorkflowMergeBoundaryTask mutates the row or the requester enqueues
a merge, and links the graph signal into its timeout controller via
AbortSignal.any -- raced separately so the walk returns on the abort
rather than waiting on a requester that may never settle.
- The legacy merge seam had the identical unguarded race and gets the same
treatment.
The timeout stays: it bounds a wedged merge queue, which is a different
failure from cancellation. Both signals must stay live -- dropping either
silently restores the stall with no type error.
Cancellation returns a distinct `merge-cancelled` rather than reusing
merge-timeout. Returning `data.status: "failed"` would let classifyMergeFailure
read the unknown reason as merge-failed and route the cancellation into
bounded auto-merge retry, re-requesting the merge the operator just cancelled.
Regression test covers both merge surfaces, both cancel timings (pre-flight
and mid-flight), the no-signal back-compat path, the signal plumbing itself,
and the classification boundary. Verified by removing the fix: 7 of 9 cases
fail, with the mid-flight cases hanging until timeout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipping a stale planning-state write is the expected outcome of a normal
scheduler advancement, not an anomaly, so the warn was pure log noise.
Behavior is unchanged; only the two planLog.warn emissions are removed.
Fusion-Task-Id: FN-8024
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`deriveSignalAndSources`'s executor branch never read `task.status`, so a row
parked `status: "failed"` — e.g. the terminal fn_task_done refusal/invariant
park — reported `signal: "progressing"` with the reason "Task is actively
executing in-progress work". The overseer observed a dead task as healthy and
took no action. `failed` was only ever derived for the merger/pull-request
stages, so the sole backstop was the FN-7743 2h stall proxy firing hours later.
This is exactly what FN-7965's audit trail shows: every intervention on a
terminally-parked task was action="observe", reason="Task is actively executing
in-progress work".
Report `failed` so recovery engages on the next poll. This adds no new policy:
a failed executor observation already routes to `retry_step` (executor sources
are `agent-log`, never an ERROR_SOURCE_KIND), bounded by
PLANNER_RECOVERY_MAX_ATTEMPTS and escalated on exhaustion.
Precedence and dedup preserved: `paused` still wins, so an operator/user-paused
row stays `blocked` and is never routed into autonomous recovery; and the reason
is a constant (never interpolating task.error/status) so the FN-7577
`stage|signal|reason` feed dedup still suppresses repeat observations.
Verified: the repro test fails with the branch disabled; paused-precedence,
healthy-card (FN-7577) and dedup guards added; overseer/recovery surfaces
93 passed + core planner-recovery 20/20; engine + dashboard typecheck clean;
`pnpm test:gate` green (294+122+63).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The in-session `fn_task_done` handler parks a task terminally (status=failed,
worktree/branch/sessionFile cleared) once its refusal/invariant retry budget is
exhausted. That write happens inside the live agent session, so the executor's
no-fn_task_done retry loop never observed it and spawned a fresh session anyway.
The retry completed, marked the task done, and dragged a worktree-less row into
the pre-merge graph, where the first write-capable node failed on
`no-worktree-for-write-node` — surfacing as a misleading "Workflow graph
terminated with failure at node 'code-review-remediation'" instead of the real
refusal. Observed on FN-7965 and again live on FN-7981.
Re-read state at the top of the retry loop and honor the park. The status probe
covers all three park sites (invariant-check, explicit refusal, implicit
refusal) rather than the single reported repro.
Deliberately not routed through the FN-4806 reclaim branch: its silent todo
requeue would clear the park and, with the budget already spent, re-park on the
next pickup in a todo->execute->park loop.
The pre-existing reclaim probes could not catch this — they test
`worktree === null`, but the store maps a cleared column to `undefined`
(`task-store/serialization.ts`: `row.worktree || undefined`), so the existing
test only passed because its mock returned a value production never emits.
Tightening that probe regressed 7 fixtures and is left as separate work.
Verified: new tests fail with the guard disabled; engine reliability surfaces
show zero regressions vs baseline (17 pre-existing failures unchanged, 495->499
passing); engine-core gate suite 294/294.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Speeds up **time-to-HTTP-ready** for `fn dashboard` and `fn serve` after
the PostgreSQL cutover without reintroducing the historical 3s
cwd-engine race that degraded webhooks.
- **Dashboard store share (serve parity):** inject the factory-booted
`TaskStore` as `externalTaskStore` so cwd `ensureEngine` does not open a
second pool; share only when store root matches project working
directory (multi-project safe).
- **Serve multi-project:** stop awaiting `startAll()` before listen;
await only the primary engine; background the rest + reconciliation.
- **Defer non-route-critical engine work:** ordered OAuth (refresh →
monitor), automation schedule syncs, and auto-merge **enqueue** after
the engine handle is returnable.
- **Critical-path merge status clear:** still clear stale
`merging`/`merging-pr` before ready so manual merge is not blocked after
crash.
- **Serve `--paused`:** apply `enginePaused` before
`ensureEngine`/`startAll` (dashboard ordering).
- **Stop safety:** generation counter so deferred tails cannot resume
after `stop()` clears `shuttingDown`.
- **Phase timing:** shared `phaseTime` helper, factory substep logs,
serve time-to-listen.
Plan: `docs/plans/2026-07-14-001-feat-faster-startup-plan.md`
## Test plan
- [x] `packages/engine` — `project-engine-manager.test.ts` (path-matched
external store)
- [x] `packages/engine` — `project-engine-deferred-startup.test.ts`
(status clear, OAuth order, stop generation)
- [x] `packages/cli` — `startup-phase.test.ts`
- [x] `packages/cli` — `serve.test.ts` (60 tests, including `--paused`)
- [ ] Local: warm `fn dashboard` / `fn serve` and compare `startup phase
*` / `time-to-listen` logs
- [ ] `pnpm smoke:boot` (real serve `/api/health` on ephemeral port)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance**
* Improved dashboard and serve startup times, including faster
time-to-listen and time-to-ready.
* Moved non-essential background initialization off the critical startup
path.
* Parallelized dashboard service initialization where possible.
* **Reliability**
* Improved multi-project startup handling and project selection.
* Prevented cross-project task-store sharing.
* Added safer shutdown behavior for partially completed startup.
* **Diagnostics**
* Added startup phase timing logs to help identify performance
bottlenecks.
* **Tests**
* Expanded coverage for deferred startup, shutdown, project isolation,
and startup timing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Treat **shared PostgreSQL** (`DATABASE_URL`) as the multi-node durable
data plane; mesh HTTP is membership + optional auth, not task/settings
replication.
- **Peer exchange**: under Postgres backend mode, write queue is
**topology/auth-only**; non-topology pending rows fail rather than
replaying multi-leader task/settings payloads.
- **Mesh routes**: task-ID reserve/commit/abort always hit local shared
allocator rows (ignore remote `coordinatorNodeId`); mesh sync ignores
settings and only exchanges `authMaterial`.
- **Docs**: rewrite multi-project runbook, shared cluster protocol, and
architecture mesh sections for shared-Postgres + claims/leases.
## Context
Follows the SQLite→Postgres cutover. Multiple Fusion nodes can share one
external Postgres while keeping **per-node execution** (worktrees,
processes, claims via `central.task_claims`). Explicit non-goals remain:
scheduler failover and live process migration.
Plan:
`docs/plans/2026-07-15-001-refactor-mesh-shared-postgres-multinode-plan.md`
## Test plan
- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/peer-exchange-service.test.ts`
- [x] `pnpm --filter @fusion/dashboard exec vitest run
src/__tests__/mesh-routes.test.ts`
- [x] `pnpm --filter @fusion/core exec vitest run
src/__tests__/shared-mesh-state.test.ts`
- [ ] CI gate (lint/typecheck/build/gate)
- [ ] Manual (optional): two processes, same `DATABASE_URL`, create task
on A visible on B; settings change without mesh settings sync; claim
exclusivity
## Operator note
Multi-node shared board requires **external** `DATABASE_URL` on every
node. Default embedded Postgres is still single-host.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved multi-node deployments using shared PostgreSQL as the durable
source of execution state.
* Task ID reservation/commit/abort now run locally (no remote
coordinator forwarding).
* Mesh syncing now prioritizes topology visibility and authentication
material; settings replication is disabled in shared-Postgres mode.
* **Bug Fixes**
* Prevented task/settings replication over mesh HTTP in shared-Postgres
deployments.
* Refined lease ownership, recovery, and reconciliation to converge via
shared-database primitives.
* **Documentation**
* Updated architecture and shared-mesh protocol guidance, including
multi-node setup and lease/task-ID allocation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
merger-ai's local-checkout sync stashed under fusion-ai-merge-sync-<taskId>,
a label none of merger.ts's reclamation machinery matches — every path keys
off the fusion-merger-autostash: prefix. Those entries were never classified,
never subsumed-dropped, never age-swept, and never surfaced as orphans holding
work, so they accumulated indefinitely: six entries dating back a month were
found on one working tree, and their age made real lost work indistinguishable
from litter. merger-ai now labels through buildAutostashLabel, and the legacy
prefix stays recognized so already-leaked entries are reclaimed rather than
stranded in developers' stash lists.
Routing them into that machinery first required fixing what it does with
untracked files. A stash created with --include-untracked keeps them in a
third parent (<sha>^3) that git stash show omits, so an untracked-only stash
read as empty — and all three copies of the liveness check treated empty as
"subsumed, safe to drop". Every leaked ai-sync stash carried untracked files,
so the fix would otherwise have destroyed the work it was meant to reclaim.
Liveness now resolves through one authority, classifyStashContent, which reads
both sides, diffs untracked paths against <sha>^3 rather than the stash commit
(whose tree never contained them), and treats unreadable state as unknown and
therefore undroppable.
Age-based sweeping is left alone: it drops by timestamp without consulting
content, which is deliberate bounded retention and the backstop against this
same accumulation, not a safety gap.
Regression test uses real git — the defect lives in git's stash object model,
so a mocked git can neither express nor catch it — and asserts the invariant
across tracked-only, untracked-only, and mixed stashes in both live and
subsumed states. The mixed shape (tracked subsumed, untracked live) is the one
that silently lost work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An operator was re-asked to approve a plan they had already approved and
that had not changed.
POST /tasks/:id/approve-plan fingerprints the on-disk PROMPT.md, so a plan
approved before the `## Original Description` hygiene injection
(applyOriginalDescription) shipped carries a hash over PRE-injection
content. On the task's next pass the injection rewrites PROMPT.md, the
fingerprint moves, and FN-7569's idempotency short-circuit misses — so the
manual gate re-parks an unchanged, already-approved plan.
finalizeApprovedTask now also compares the recorded fingerprint against the
as-read (pre-injection) content. This does not weaken the gate: `written`
diverges from `writtenInput` only via that injection, so both arms hash
bytes the operator actually approved — only the representation differs. A
genuinely changed plan matches neither arm and still parks.
On a legacy match the stored fingerprint is migrated forward, so the
reconciliation is one-time per task rather than a comparison carried
forever. The migration is a direct updateTask — the taskUpdates batch is
flushed well before this gate runs.
Covers both finalizeApprovedTask callers (direct + recoverApprovedTask),
asserts the changed-plan safety edge still parks, and asserts no redundant
fingerprint write when the approval is already post-hygiene.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recoverApprovedTask idempotency test failed deterministically, and its
siblings passed for the wrong reason. Both traced to the same stale fixture,
not a product defect.
finalizeApprovedTask injects `## Original Description` into PROMPT.md
(applyOriginalDescription) BEFORE computing the approval fingerprint, and
POST /tasks/:id/approve-plan fingerprints the on-disk file — so the
fingerprint an approval records is always over post-injection content. The
fixtures wrote RAW planner text and fingerprinted that, modelling a state
approve-plan can never produce: the injection then rewrote the content, the
fingerprint moved, and the short-circuit looked broken.
Verified the product is correct: the injection is idempotent, so the real
approve -> recover round-trip fingerprint matches (checked end to end).
- recoverApprovedTask test: write and fingerprint the approved on-disk
content. It now exercises the real short-circuit — the run logs "plan
unchanged since prior approval" then "recovered and moved to todo",
where before it logged "awaiting manual approval".
- same-plan test: it only passed because the injection's rewrite ENOENT'd
(no task dir), the failure was swallowed, and `written` stayed raw — so
the fingerprint matched by accident. Feed it the approved content so the
injection is a genuine no-op and the assertion means something.
Fixtures derive from applyOriginalDescription rather than hard-coding
post-injection text, so they keep meaning "the content the operator
approved" if the hygiene injection changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rate-limited reviewer filled a task's Chat tab with 14 identical
"Reviewer using model: ..." markers and no review text, hammering an
already-limited provider.
Root cause: the reviewer was the only AI lane that never classified
provider errors, so a 429 became an UNAVAILABLE verdict. With no
validator fallback configured the fallback ladder re-ran the SAME model
instantly, and fn_review_step answered with "code review remains
blocking; retry once" — bounding the loop with prompt text rather than
code. The tool's catch-all also swallowed the error into tool output, so
withRateLimitRetry, UsageLimitPauser and RetryStormError never fired.
- reviewer: throw ReviewerProviderError for usage-limit/transient errors
instead of laundering them into UNAVAILABLE, and never spend the
fallback budget (which bounds bad reviews) on an outage.
- reviewer: absorb flaky-network blips in-lane via withRetry with
jittered backoff; rate limits still escalate immediately.
- executor: re-raise the fatal after the prompt via
throwDeferredReviewerFatal — pi-agent-core converts tool throws into
tool_error results, so a tool cannot throw out of session.prompt().
- executor: give code review a real MAX_CODE_REVIEW_UNAVAILABLE_RETRIES
counter, mirroring the plan/spec limiter.
- reviewer: dedupe the model marker on text, so same-model retries stay
silent while a genuine model switch still emits.
Also fixes the run-on rendering: AgentLogType gains `status` for complete
engine messages. `text` means "streamed delta" and is re-glued with
join(""), which is why N standalone markers rendered as one string. The
split is at the type, not a separator — a separator would reintroduce the
FN-5787/5789/5803 streamed-spacing regression.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both lines fired on every scheduler poll while nothing changed: a held
card re-attempts release each sweep, and every dispatch candidate logged
its resolved node. On a busy board that filled the operator log pane with
"Hold release for FN-XXXX deferred" and "routed to node=local" within
seconds, burying real scheduler events.
Add a Logger.debug() level, off by default and opted into per subsystem
via FUSION_DEBUG, and demote both lines to it. Routing to a remote node
stays at info since it explains where work actually went; only the local
default is demoted. Lines reporting a real transition (capacity
rejection, racing sweep, release failure) are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile active task segment anchors on full Global/Engine unpause so stopped-engine wall-clock does not inflate execution time, reusing the FN-7011 downtime path with a transition-captured heartbeat.
- Pass optional engineLastActiveAtOverride into reconcileActiveTimingForEngineDowntime so unpause callers freeze the stopped-window proof against racing scheduler heartbeats
- Await downtime reconciliation in resumeAfterUnpauseAndSweepInReview before resuming agentic work or sweeping in-review tasks
- Fold Global/Engine unpause into the unified pause-lifecycle listener (single reconcile when both clear together; no-op while either pause remains)
- Soft-fail reconcile errors so unpause resume still proceeds
- Add store and project-engine coverage for override, await-before-resume, dual-source clear, and fail-soft paths; document FN-7975 in AGENTS.md run-audit notes
- Add patch changeset for the operator-facing timing fix
Files changed:
.changeset/fn-7975-engine-pause-active-timing.md | 7 ++
AGENTS.md | 2 +-
.../core/src/__tests__/store-active-timing.test.ts | 86 +++++++++++++
packages/core/src/store.ts | 23 ++--
.../project-engine-unpause-active-timing.test.ts | 94 ++++++++++++++
.../engine/src/__tests__/project-engine.test.ts | 139 +++++++++++++++++++++
packages/engine/src/project-engine.ts | 64 +++++-----
packages/engine/src/self-healing.ts | 6 +-
8 files changed, 378 insertions(+), 43 deletions(-)
Fusion-Task-Id: FN-7975
Fusion-Task-Lineage: 84a46e6f-92bf-452a-ab67-c25ba85cbffb
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Address review findings on the FN-7956 hang fix:
- Per-tool outer timeouts so fn_research_run(wait_for_completion) is not clipped by a flat 60s budget
- Longer budgets for skills install, import/browse, and web_fetch
- Boot-failure cooldown + orphan-boot log when store boot times out
- Log timeout/abort/errors from the extension wrap; clearer host-extension skip reason
- Tests for budgets, research wait, and sessionPurpose forwarding
Plan Review REVISE previously fed feedback without the rejected plan body, so triage rewrote from title/description and looped. Seed the draft for surgical revision, use reviewType spec for the pre-execution gate, and tighten planner/reviewer prompts toward blocking-only REVISE with concrete edits.
Give planner/reviewer pairs more room to converge before escalating, and surface why a task is parked for plan approval—especially plan-review-replan-cap non-convergence—on cards, detail, and notifications.
AI merge review could park forever when the host fusion extension loaded
fn_task_show and booted a second TaskStore without a tool timeout (FN-7956).
- Skip host @runfusion/fusion extensions for sessionPurpose "merger"
- Forward sessionPurpose into createFnAgent for that policy
- Coalesce + 30s-bound extension TaskStore boots; ALS-propagate AbortSignal
- Wrap every extension registerTool execute with 60s timeout/abort fail-closed
- Unit tests for merger host-extension skip and tool timeout helpers
AI merge spends most of its time in reviewing and landing, not merging.
Treat the full merge pipeline as active so cards, workflow switcher, and
stall suppression show Merging… while the pump owns a task.
Prevent concurrent orphan merge after abort, protect long merging-phase
tools from false reclaim, emit run-audit on wedged reclaim, and race PR
merge dispatch the same way as direct AI merge.
AI-merge review hangs left activeMergeTaskId/mergeRunning set while
status=reviewing and overseer logEntry noise kept updatedAt fresh, so
self-healing never reclaimed the owner and the board showed no merging
badge. Race merge work with abort, force-abort on pause/reclaim, treat
reviewing as merge-active, and recover on merger agent silence; also
forward PluginRunner into AI merge so grok-cli merger matches chat.
Admit under-budget paused/heartbeat-model-unavailable agents to the shared
heartbeatErrorRecovery budget so timer, self-healing, and startup paths
retry without a manual Retry. Keep the pause reason when the budget is
exhausted so operators still see credential guidance.
## Summary
- Extend the workspace content-hash skip cache to **all** packages (not
just plugins), with `--force` / `--full` flags
- Default local CLI packaging to a **fast mode** (bin/extension +
migrations only); full desktop/plugin/DTS staging runs on CI or `pnpm
build:full`
- Enable TypeScript `incremental` builds for warm recompiles
- Add `maxConcurrentVerifications` (default **1**) so concurrent tasks
cannot stack monorepo typecheck/build and peg CPU
Warm `pnpm build` measured ~**126s → ~0.8s** when nothing changed.
## Test plan
- [x] `node --test scripts/__tests__/build-workspace.test.mjs` (12 pass)
- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/verification-concurrency.test.ts`
- [x] `pnpm --filter @fusion/core exec vitest run
src/__tests__/settings-parity.test.ts`
- [x] Local: first `pnpm build` rebuilds as needed; second warm `pnpm
build` skips all packages (~0.8s)
- [x] Fast CLI packaging logs skip of desktop/plugin staging without
`FUSION_CLI_FULL_PACKAGE`
- [ ] CI: `pnpm build` still full-packages under `CI=true` (plugin
staging / release surfaces)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Scheduling setting to limit concurrent verification tasks from
1–8, with a default of 1.
* Verification tasks now support cancellation while waiting or running.
* Added options for forced and full workspace builds.
* **Performance**
* Local builds can skip unchanged packages and use incremental
compilation for faster rebuilds.
* Local CLI packaging is faster by default, while full packaging remains
available when needed.
* **Documentation**
* Updated the settings reference with the new verification concurrency
option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Generated PROMPT.md (after triage/planning — not the bootstrap stub) now
keeps the operator's original task description near the top under `##
Original Description`, so executors always see the source request even
after Mission/Steps rewrites.
- **AI-planned path:** planning templates (standard/fast/concise)
require a verbatim `## Original Description` section;
`buildSpecificationPrompt` instructs the planner; `finalizeApprovedTask`
deterministically injects/rewrites it as hygiene.
- **Non-AI path:** `generateSpecifiedPrompt` uses the same pure helper
so direct creates into non-intake columns get the same contract.
- **Description edits:** real specs keep `## Original Description` in
sync when `task.description` changes.
- **Unchanged:** bootstrap stubs and `isUnplannedSeedPrompt` equality
detection.
## Surfaces
| Surface | Change |
|--------|--------|
| `original-description-policy.ts` | Shared inject/rewrite helper |
| `agent-prompts.ts` | Template + requirement text |
| `triage.ts` finalize + `buildSpecificationPrompt` | Instructions +
post-write pin |
| `generateSpecifiedPromptImpl` | Non-AI specified PROMPT.md |
| `task-update.ts` | Description sync on real specs |
## Test plan
- [x] `pnpm --filter @fusion/core exec vitest run
src/__tests__/original-description-policy.test.ts
src/__tests__/agent-prompts.test.ts
src/__tests__/mesh-task-replication.test.ts
src/__tests__/store-create-intake-column.test.ts --silent=passed-only
--reporter=dot`
- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/triage.test.ts -t "Original Description|injects ##
Original" --silent=passed-only --reporter=dot`
- [ ] CI gate (Lint / Typecheck / Build / Gate)
## How to verify manually
1. Create a task with a distinctive description, let triage plan it (or
finalize a mock plan).
2. Open `.fusion/tasks/<id>/PROMPT.md` and confirm `## Original
Description` appears after title/metadata with the raw description,
before Mission / Before → After.
3. Direct-create into `todo` (non-intake) and confirm the non-AI
generated prompt also has the section.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Generated `PROMPT.md` specifications now include an `## Original
Description` section near the top.
- Operator task descriptions are preserved verbatim for AI-planned and
specified prompts.
- Updated prompts remain synchronized when task descriptions change.
- **Bug Fixes**
- Replaced paraphrased original descriptions with the correct task
description.
- Preserved existing prompt content during review and retry workflows.
- **Tests**
- Added coverage for placement, formatting, replacement, and idempotent
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Engine and dashboard traffic now stays on the authoritative PostgreSQL
layer across execution, recovery, project discovery, planning sessions,
analytics, and shutdown. The dashboard no longer presents a migration
notice for a cutover that is already mandatory.
## Design decisions
- Runtime composition requires an async data layer instead of
constructing a hidden SQLite fallback.
- Engine workflow, mission, claim, and self-healing reads await their
PostgreSQL-backed store contracts.
- Project-scoped dashboard stores retain and close their backend owner
exactly once.
- The dashboard test quarantine entry remains paired with its Vitest
exclusion, preserving the repository’s deletion-ratchet policy.
## Validation
- Core, Engine, Dashboard, CLI, and Desktop typechecks pass on the
stacked branch.
- `pnpm test:gate` passes all 478 gate tests.
- This PR changes 62 files.
## Stack
- Depends on #2108.
- CLI/desktop/ops, plugins, and docs/release follow in later PRs.
Related: #2105
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Project discovery now recognizes projects using the
`.fusion/project.json` marker.
* Knowledge indexing and search are more reliable across project-scoped
storage.
* **Bug Fixes**
* Improved session, audit timeline, approval, monitoring, and analytics
data consistency.
* Prevented stale planning-session updates and project-store shutdown
races.
* Ensured chat usage and CLI session status are saved before continuing.
* **UI Changes**
* Removed the storage migration notice banner now that the PostgreSQL
transition is complete.
* **Reliability**
* Improved shutdown handling, workflow execution, and worktree behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Fusion’s core runtime now treats PostgreSQL as the authoritative
metadata store without leaving current CLI, dashboard, desktop, or
engine composition roots uncompilable between stack layers. This is the
99-file foundation for the larger cutover: subsequent PRs migrate the
remaining consumers, plugins, and operator surfaces.
## Design decisions
- Runtime store construction fails closed when an asynchronous
PostgreSQL layer is unavailable; SQLite remains readable only at
explicit migration and identity-recovery boundaries.
- Project ownership is enforced across active, archived, workflow,
mission, analytics, and plugin-schema data.
- The small set of cross-package files in this layer are
compatibility-critical call sites required for a green intermediate
commit, not the complete consumer migration.
- Schema migration 0008 remains assigned to session-advisor state from
current `main`; mission lineage idempotency advances to 0009 so neither
invariant can be skipped.
## Validation
- All affected package typechecks pass: Core, Engine, Dashboard, CLI,
and Desktop.
- `pnpm test:gate` passes: 478 tests across the engine gate, PostgreSQL
core gate, and CLI workflow shape.
- The PR changes exactly 99 files.
## Stack
This is the base PR. Engine/dashboard, CLI/desktop/ops, plugins, and
docs/release follow as stacked PRs, each below 100 changed files.
Related: #2105
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* PostgreSQL is now the standard runtime backend, with embedded
PostgreSQL enabled by default.
* Added project-scoped storage for tasks, archives, chat sessions,
missions, knowledge pages, and operational data.
* Improved archived-task search, filtering, pagination, and restoration.
* Added safer plugin schema initialization with validation and project
isolation.
* Added PostgreSQL-backed workflow, mission, validator, and dashboard
capabilities.
* **Bug Fixes**
* Improved startup timeout cancellation and resource cleanup.
* Prevented cross-project data access and phantom reservation cleanup
errors.
* Ensured archived tasks remain read-only and asynchronous writes
complete reliably.
* Retired SQLite opt-out settings with clear startup errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Operators could see more agents running than Global Max Concurrent
(e.g. 5 running with cap 4: 4 planners + 1 executor).
- Scheduler now `tryAcquire`s a shared semaphore slot before
todo→in-progress and hands that pre-held slot to the executor/graph run.
- Triage admits planners against the live top-level running-agent claim
(planning + in-progress + active in-review), not only
`semaphore.availableCount`.
- Executor claims the pre-held slot for the full run and avoids a second
top-level acquire on step/seam re-entry (deadlock under a full cap).
## Test plan
- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/concurrency.test.ts src/__tests__/triage.test.ts`
- [x] Regression: triage leaves room when 1 in-progress agent is live
under global cap 4
- [x] Regression: pre-held executor slot register/take/drop handoff
- [ ] Manual: set Global Max Concurrent and Max triage concurrent to 4,
fill Planning + run 1 In Progress; footer should not show 5 running
under a full steady state
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved global concurrency enforcement so the scheduler and executor
never start more agents than the configured limit, including tighter
top-level “claimed capacity” accounting.
- Updated triage admission control to consider global top-level
utilization, factoring processing tasks and agents already running to
prevent over-admitting planners.
- Added safer pre-held concurrency-slot handoff behavior to avoid
capacity leaks and drift during graph routing, step execution, and
legacy fallback.
- Ensured reserved capacity is reliably released on early exits, failed
dispatches, and other aborted paths (with idempotent cleanup).
- Refreshed concurrency diagnostics to better explain whether throttling
is due to project or global limits, with clearer claimed/processing
visibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What & why
Workflow graph merge failures classified `implementation-incomplete`
(i.e. the merge node reports there is no implementation proof — no
branch / no committed work) could still be routed to the no-op merge
requester and false-complete the task as **done**. This hides genuinely
unlanded work behind a green "merge" and is the merge-side sibling of
the "(no feedback captured)" no-verdict dispatch defect.
Closes the truthfulness gap: an `implementation-incomplete` merge-graph
failure now **fails closed** when there is no executable proof to
resume, or **requeues resumable parsed steps** back to `todo` for
execution — it is never handed to a no-branch no-op merge requester.
Refs #1991 (no-op merge truthfulness). Sibling of #1946 (no-verdict "(no
feedback captured)" dispatch defect).
## Change
- New classifier `routeImplementationIncompleteMergeGraphFailure(live,
failedNode)`:
- clears paused-aborted state + active worktree,
- requeues resumable parsed steps via the existing execution-resume
router when the task still has non-terminal workflow steps,
- otherwise fails closed (`status: "failed"` with a logged, explicit
reason).
- Defense-in-depth: `isRetryableBenignMergePauseAbort` and the
merge-requester route both short-circuit (`return false`) for
`implementation-incomplete`, so this value can never reach the no-op
merge requester.
- `handleGraphFailure` routes genuine (non-global-pause,
non-completion-finalize, non-user-paused) `implementation-incomplete`
merge-graph failures through the new classifier.
- Resume-eligibility predicate treats an `implementation-incomplete`
merge failure with **no** incomplete steps as fail-closed, and keeps the
premature-merge-with-incomplete-steps requeue path.
Legitimate `noCommitsExpected` no-op merges are explicitly preserved
(regression test included).
## Tests
New regression coverage in:
-
`packages/engine/src/__tests__/reliability-interactions/merge-node-paused-abort-retryable.test.ts`
— parametrized across merge node ids: (a) no-proof
`implementation-incomplete` fails closed without requesting a no-op
merge; (b) resumable parsed steps are requeued to `todo` for execution
resume, not no-op-merged.
- `packages/engine/src/__tests__/executor-fast-mode-workflows.test.ts` —
a legitimate `noCommitsExpected` builtin:coding merge is still allowed
(guard does not over-block).
Verification (engine package):
pnpm --filter @fusion/engine exec vitest run \
src/__tests__/executor-fast-mode-workflows.test.ts \
src/__tests__/reliability-interactions/merge-node-paused-abort-retryable.test.ts
# => 2 files, 69 tests, 0 failures
pnpm check:changesets # pass
pnpm --filter @fusion/engine typecheck # 0 errors
A `patch` changeset for `@runfusion/fusion` is included.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented “implementation-incomplete” workflow merge failures from
being treated as successful no-op merges.
* Ensured tasks with resumable implementation steps move back to
execution to continue where they left off.
* Ensured tasks without sufficient implementation evidence fail safely
rather than entering misleading retry/no-op paths.
* Improved paused/aborted merge-failure handling to avoid incorrect
completion states.
* **Tests**
* Added/expanded coverage for fast-mode coding merges and
implementation-incomplete pause/abort retry classification.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com>
Use PostgreSQL workflow selections in the dashboard TUI, authoritative driver, and graph-runner adapter so migrated tasks cannot silently fall back to the coding workflow.
Route workflow selections, model lanes, goals, skills, and reliability reads through project-scoped async stores. Recover heartbeat agents parked against an unrelated project model and preserve workflow JSONB patches atomically.
Preserve authenticated CLI usage after migration, surface OAuth remediation, and use a single distinct model fallback before parking permanent failures. Keep transient credential errors retryable and confirm each OAuth expiry notification independently.
Fusion-Task-Id: FN-7952
Use canonical Anthropic OAuth refresh, keep CLI-backed providers out of API-key auth rows, parse Grok's omitted zero usage, and carry board workflow context into task creation.
## Summary
Fixes shard 4 full-suite failures: chat_sessions schema baseline gap +
two remaining PG auth bugs missed by PR #2086.
**Scope: shard 4 only.** Shards 1/2 (engine timeouts) and shard 3
(compound-engineering CI-only failure) are separate issues not addressed
here.
## Changes
### Schema baseline gap — `chat_sessions` missing columns (42703 error)
- **`0000_initial.sql`**: Added `validator_thinking_level` and
`planning_thinking_level` columns to `CREATE TABLE
project.chat_sessions`. These exist in the Drizzle schema
(`project.ts:1492-1493`) but were missing from the SQL baseline, causing
`column does not exist` on all chat_sessions inserts in fresh test
databases.
- **`postgres-health.ts`**: Added both columns to
`EXPECTED_PROJECT_COLUMNS` self-heal list so existing databases also get
them via ALTER TABLE.
**Fixes**: `chat-store-content-search-edit.pg.test.ts` (5 tests),
`satellite-db-injected-stores.test.ts` (2 tests)
### Remaining auth bugs (password auth failed for user "runner")
- **`allocator-cross-project.test.ts`**: Still had `process.env.USER` in
inline adminExec — missed by PR #2086's batch fix. Replaced with
`PG_TEST_URL_BASE` connection string.
- **`connection.test.ts`**: Used `FUSION_PG_TEST_URL` (not set on CI)
with a bare default URL lacking credentials. `postgres.js` fell back to
OS user `runner`. Changed to derive from `FUSION_PG_TEST_URL_BASE` which
includes credentials.
**Fixes**: `allocator-cross-project.test.ts` (2 tests),
`connection.test.ts` (3 tests)
## Verification
| Check | Result |
|---|---|
| Merge gate (`pnpm test:gate`) | ✅ 294 + 114 + 63 = 471 passed |
| chat-store-content-search-edit | ✅ 5 passed |
| satellite-db-injected-stores | ✅ 10 passed |
| allocator-cross-project | ✅ 2 passed |
| connection | ✅ 13 passed |
| Lint | ✅ exit 0 |
| Typecheck | ✅ clean |
## Not in scope
- **Shards 1/2**: Engine test suite timeouts with
`getAsyncLayer`/`updateSettings` mock warnings. Pre-existing.
- **Shard 3**: `compound-engineering stage-skill-loading.test.ts` — 14
tests fail on CI (`TypeError: Cannot read properties of undefined
(reading 'close')`), pass locally. Likely CI-specific teardown issue.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added separate `validator_thinking_level` and
`planning_thinking_level` fields to chat session data, including
database schema and health-check recognition.
* **Bug Fixes**
* Improved PostgreSQL test connectivity by using configured connection
URL settings instead of hardcoded local defaults.
* Made Postgres-related test teardown null-safe to avoid failures when
setup doesn’t complete.
* **Tests**
* Updated automated test quarantine/exclusions for known failing engine
and reliability-interaction cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Preserve legacy-only tables, recover partial migration ownership, and enforce project-local keys, relationships, agents, merge queues, task IDs, archives, and monitor state with PostgreSQL RLS.
Report successful cutovers once in the dashboard and system inbox with retained SQLite paths and Discord support details.
