Two unrelated production failures with a shared symptom of an opaque error.
Chat: FN-8869 hoisted the agent-existence check out of its else branch, so it ran
even when the client supplied an explicit model pair. Model-target chats send the
client-only sentinel `__fn_agent__`, which is never an agent row, so every one of
them 404'd behind the generic "Failed to create chat session" toast. The agent is
now required only when it is the source of model resolution.
Self-healing: a failed `tip-already-merged` cleanup was rethrown and classified
`branch-conflict-unrecoverable`, failing and pausing tasks whose branch was already
an ancestor of the integration ref. Every one of the 78 logged parks carried a
`git worktree remove --force` / `ENOTEMPTY rmdir node_modules` message -- a pnpm
race, not a conflict. Cleanup failure now retries on the next sweep, and prune runs
before removal so a stale registration stops causing the failure it would prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent wedges kept cards silently stuck on the board.
1. Workflow principals were capped. `WorkflowAgentCapacity.acquire` enforced
`settings.maxConcurrent` as a project session budget plus an optional
per-agent `maxWorkflowSessions`, and `routeWorkflowPrincipal`'s availability
test applied the same per-agent ceiling. The workflow roles stand in for
STAGES, not workers, and there is typically one agent per role - so the cap
serialized the entire board behind a single Workflow Executor regardless of
maxConcurrent/maxWorktrees. Admission now always succeeds; the lease survives
as bookkeeping (it is what activeSessions counts and what the renewal timer
keeps warm). `maxProjectSessions` is removed from the input rather than
defaulted, so it cannot be reintroduced without deleting the contract, and
the agent-capacity re-route loops in triage and graph admission are deleted
with the refusal they existed to work around.
2. Continuations that stop in `running` or `held` were never re-polled. The
scheduler's due-poll takes only `runnable`/`retrying`; a row claimed through
a path that leaves `leaseExpiresAt` NULL keeps `state: "running"` forever
after its process dies, and `acquireWorkflowWorkItemLease` can only re-take a
`held` row whose blockedReason matches workflow-principal-%. Observed live:
seven cards `running` behind leases from a process that exited ~9h earlier,
two `held` with a NULL blockedReason for 46h, none emitting a single
run-audit row while stranded. A further 33 active-state rows belonged to
archived+soft-deleted tasks (the FK cascade only fires on hard delete).
New sweep `reconcileStrandedWorkflowContinuations` (startup + periodic)
re-queues both stranded shapes and retires dead tasks' rows, gated by the
canonical liveness triple, a 10-minute grace matching the capacity lease
duration, and a compare-and-set on the scanned state so a real claim wins.
The decision is the pure `evaluateStrandedContinuationReclaim`, shared with
its tests so coverage cannot drift from behavior - the drift that let the
FN-8923 sweep ship covering one ninth of this problem.
Verified: pnpm lint, engine typecheck, pnpm test:gate (606 tests), verify:fast,
and the new suite under mutation (removing either guard fails 3 cases). The two
pre-existing failures in self-healing-orphaned-pending-step-results.test.ts
reproduce identically at HEAD without these changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
## What
Adds a typed, queryable catalogue for engine run-audit events as the
first step of the delivery-pipeline reliability & observability effort.
- **New module** `packages/engine/src/run-audit/run-audit-catalogue.ts`
— a typed registry describing run-audit event kinds (scheduler,
self-healing, merger, worktree, symbol-lock, …) so pipeline
observability can ingest and reason about them consistently.
- **Parity test** `run-audit-catalogue.test.ts` — asserts the catalogue
matches the emitted run-audit event space.
- **Docs** `docs/run-audit.md` + index pointer.
## Why
Run-audit events are currently emitted ad-hoc without a typed contract.
A catalogue gives:
- a single source of truth for event kinds/names,
- a parity guard so any new or renamed event is caught,
- a foundation for delivery-pipeline reliability dashboards.
## Verification
- `@fusion/engine` `tsc` build → **PASS**
- `vitest run run-audit-catalogue.test.ts` → **3 tests passed**
- No production behavior change outside the new module.
## Scope
New isolated module + its test + docs. No changesets/release artifacts.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added a run-audit catalogue covering delivery-pipeline finalization,
self-healing reconciliation, and durable-agent error events.
* Documented recorded outcomes, emission conditions, audit-store
querying, and event catalogue maintenance.
* Added a documentation index entry linking to the new catalogue.
* **Tests**
* Added validation to ensure documented audit events remain complete,
consistently formatted, and synchronized with the supported event
catalogue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
## Summary
Supersedes #3405 — the fork head is not writable from maintainers, so
this branch carries the same fix rebased/merged onto current `main`.
## Conflict resolution
- Main already landed the equivalent fail-open path as **FN-8919**
(`readLinkedTaskOrUndefined` + per-agent try/catch).
- Kept the additional `recoverAgentsRunningOnInactiveTasks` regression
that covers task-gone races plus transient lookup isolation.
- Dropped the duplicate changeset (main already has
`fn-8919-agent-link-sweep-fail-open`).
## Test plan
- [x] `git merge-tree` clean against `main`
- [ ] CI green
Closes context from #3405.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved recovery handling when task lookups fail.
* Agents linked to deleted or missing tasks are now unlinked, while
agents affected by temporary errors remain preserved.
* Recovery continues for other eligible agents instead of stopping after
an individual lookup failure.
* **Tests**
* Added regression coverage for deleted, missing, and temporarily
unavailable tasks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Codex <codex@openai.com>
Convert the SettingsModal.general 'saves X via settings payload' tests from
real-timer waitFor (each burning a real ~500ms auto-save debounce) to the
FN-7506 fake-timer pattern already used by the auto-save tests in this file.
Payload-routing/coalescing assertions are unchanged; only the artificial
wall-clock wait is removed (Standing Rule: prefer fake timers over real time
waits). Isolated test-execution time drops ~21.7s -> ~15.0s (82 tests, all
green across repeated runs).
## Problem
Dashboard typecheck fails with **TS2416** in `@fusion/core`'s
`TaskStore`:
```
Property 'emit' in type 'TaskStore' is not assignable to the same property in base type 'EventEmitter<TaskStoreEvents>'.
```
The `override emit<E extends string | symbol>(event, ...args)` generic
conflicts with the base class's generic `emit<K>(eventName: keyof
TaskStoreEvents | K, ...)`. This breaks the dashboard typecheck / CI
merge gate.
## Fix
Change the override to:
```ts
override emit(event: unknown, ...args: any[]): boolean {
return EventEmitter.prototype.emit.call(this, event as string, ...args);
}
```
`event: unknown` remains assignable to the base's generic signature
while still forwarding non-typed runtime keys (`agent:log`,
`settings:updated`, …). Internal `EventEmitter.prototype.emit` calls
cast `event as string`. Behavior-preserving.
## Verification
- `@fusion/dashboard` `tsc --noEmit` → **PASS** (previously failed with
TS2416)
- `eslint` on touched file → clean
- Single-file change (`packages/core/src/store.ts`, +6/−3)
## Scope
No behavior change, no changesets required.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved task event handling to support a broader range of event
identifiers.
* Preserved cached-lane information for single-argument task update
events.
* Maintained support for custom and arbitrary event names without
disrupting existing behavior.
* Improved classification of workflow roles, session purposes, and
outcome-related status checks in lifecycle analysis, producing more
accurate findings and reducing misleading results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fusion wedged on "starting" and never brought the engine up. The dashboard bound
the migration holding server on 4040, then every query behind it failed with
"sorry, too many clients already", so the card never progressed and the
supervisor crash-looped.
Root cause: each spec-drift reconcile costs a DEDICATED PostgreSQL connection.
persist -> appendSpecDriftReport -> withPlanningLifecycleLock opens its own
postgres(directUrl, { max: 1 }) session, because the planning advisory lock is
session-scoped and deliberately fences a stale report against a newer plan.
enqueue() released every id straight into its own microtask, and project-engine
enqueues every task at runtime-boundary setup (listTasks includeArchived). On a
1,082-task project that opened ~1,082 lock sessions simultaneously against
max_connections = 500. The cluster saturated ~25s into boot and stayed saturated.
The flat 1s retry then made it self-sustaining rather than transient: once
saturated, every task failed for the same shared reason and re-armed in lockstep
once per second, re-opening the whole fleet of sessions and pinning the very
resource it was waiting on. Measured 4,777 lock sessions in 17 seconds.
Fix, contained to the reconciler — the advisory lock and its fencing semantics
are load-bearing and unchanged:
- concurrency bound (maxConcurrent, default 4) drained by a fair
insertion-ordered pump, so fan-out can no longer exceed a known connection cost
- per-task in-flight dedupe; two passes on one task would contend on that task's
own advisory lock while holding two connections
- exponential backoff with jitter capped at 60s, and retries re-enter through
enqueue so a retry storm is bounded by the same limit as a first pass
Verified against the real 1,082-task project: connections stay flat at 3-10
across a 70s boot that previously reached 1,109 and saturated, and the engine
boots through to executing tasks and shuts down cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`.fusion-knowledge/` is `fn knowledge-graph build` output (FN-8920 memory epic):
~85MB of generated JSON, rewritten on every build. edges.json alone is 43MB,
past GitHub's 50MB warning threshold.
It is deterministic and fully regenerable from source — the manifest's SHA-256
fingerprints drive incremental rebuilds — so tracking it preserves nothing a
rebuild cannot reproduce, while adding irreversible, compounding history bloat to
every clone and CI checkout.
docs/knowledge-graph.md previously left this to the operator ("it is not
ignored"); that guidance is updated here so the doc and .gitignore agree. An
operator who explicitly wants a snapshot in history can still force one with
`git add -f .fusion-knowledge/graph`.
Nothing under the path was ever tracked, so no index purge is needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Investigating the "dead cap" turned up the opposite of what it looked like, plus a
worse problem next to it.
The Plan Review replan loop was NOT unbounded. U3 re-owned the cap-park in the
graph: requestPreMergeOptionalStepFix parks via parkPlanReviewReplanCapExhausted
at awaiting-approval with reason plan-review-replan-cap, on both an explicit
finite budget and the unbounded default. That capability has been live throughout.
What was actually dead:
1. PLAN_REVIEW_GATE_REPLAN_CAP = 8 — an unread constant belonging to the
out-of-graph triage gate (runPlanReviewBeforeExecution) that U10/R4 deleted. Its
companion column Task.planReviewReplanCount was persisted, serialized and reset
but never incremented or compared. A constant and a column that look like a live
safety ceiling while enforcing nothing are worse than no ceiling: they answer "is
this loop bounded?" with a confident yes. Deleted, ratcheted in
legacy-tombstones.test.ts, and the column documented as legacy/never-written with
the live owner named.
2. planReviewReplanCap — an operator-facing setting, declared, validated,
documented in settings-reference.md and editable in the Workflow Editor, that
NOTHING read. Lowering it changed nothing. The unbounded backstop was instead
hardcoded to PLAN_REVIEW_FEEDBACK_HISTORY_LIMIT — a bound on how much reviewer
PROSE is replayed into the next planning prompt, whose own comment says it is
"bounded independently of persistence and retry accounting". Two unrelated
concerns shared one number, so trimming prompt history would have silently
tightened a safety ceiling.
The backstop now resolves from the setting, defaulting to the new
DEFAULT_PLAN_REVIEW_REPLAN_CAP = 15 — the previously-effective value, so this is a
pure re-wiring rather than a silent behavior change. The existing 15-attempt
regression test passes unchanged, which is the evidence for that. 0 is honored as
park-on-first-REVISE. An explicit planReviewMaxRevisions / node maxRevisions
budget remains a stricter, earlier gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes to the triage planning path.
1. Unclassified planning failures are bounded. specifyTask's catch-all branch —
the one reached by every error the classifiers above do not recognize — restored
the card's claimable status and wrote nothing else: no counter, no
nextRecoveryAt, no park. Triage rediscovery re-admitted the card on the very next
poll, and replaceActiveTaskWorkflowContinuation replaced the terminal work item
with a fresh one carrying no attempt count, so nothing recorded that the task had
already failed N times. It now consumes the same recoveryRetryCount/nextRecoveryAt
budget the transient branch uses (MAX_RECOVERY_RETRIES = 3, 60s/120s/300s jittered
backoff) and parks status:"failed" with a PLANNING_FAILED_EXHAUSTED: error once
spent — status:"failed" is what suppresses rediscovery. Classifying one error
string fixes one symptom; this budget is what makes the NEXT unrecognized error
fail safely instead of looping for a day.
2. The planning turn has a ceiling. Fusion set no timeout on it at all:
workflowStepTimeoutMs covers pre-merge workflow steps only, and the provider SDK's
300s APIConnectionTimeoutError caps time-to-first-byte and is cleared once headers
arrive, after which the stream is uncapped. configureHttpDispatcher, which would
install undici idle timeouts, is only called from pi's CLI entrypoints and never
in the in-process engine. Observed consequence: single attempts ran to 126 minutes,
with failed-attempt durations spread smoothly from 1 to 126 min and no clustering —
the signature of nothing enforcing a bound. New workflow-native planningTimeoutMs
(default 90 min) aborts the session; the failure consumes one bounded attempt.
The default is deliberately generous rather than tight. Successful planning work
items measured over 7 days ran p50 12.7 / p90 39.5 / p99 105.7 minutes, so a
tighter bound would abort legitimate plans and pay for the restart — the churn
this work exists to remove. It bounds hung turns, not slow ones.
3. [event:task:moved] executor tracing dropped from log to debug. It fires on
every dispatch, rebound, requeue, archive and self-healing move across every task,
which made it the loudest line in engine output and buried operator-actionable
events. No test pins the level; the information remains at debug.
Also fixes a test break shipped in 963dba6f80: the review blocking-severity
settings landed inside BUILTIN_REVIEW_REVISION_SETTINGS, whose contents
builtin-workflow-settings-triage.test.ts asserts exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Request timed out." is the literal default message of the Anthropic and OpenAI
SDKs' APIConnectionTimeoutError, surfaced to Fusion by checkSessionError after
pi-coding-agent exhausts its in-session retries. It matched none of the
connection-scoped timeout patterns, which deliberately excluded "general
timeouts", so it fell through to specifyTask's generic failure branch — the one
that restores status: null and writes no counter, no nextRecoveryAt, and no park.
Triage rediscovery then re-admitted the card on the very next poll, forever.
Measured before this change: 48 "Specification failed: Request timed out." events
across 10 tasks in 30 hours with zero backoff between attempts. FN-8950 alone
burned 8 consecutive attempts over ~8 hours and never reached implementation.
Across 2 days, 91 failed planning attempts averaged 33 minutes each — ~50 hours of
wall-clock producing nothing, 24% of all planning time.
Classifying these as transient routes them into the bounded recovery policy
(MAX_RECOVERY_RETRIES = 3, 60s/120s/300s jittered backoff) already used by the
connection-level patterns, so a provider blip costs three spaced retries instead
of an unbounded loop.
The pattern is anchored to "request timed out" rather than a bare timeout match:
agent log prose and verification output legitimately contain "timed out"
("BuildKit timed out", "stuck-kill unwind timeout"), and a broad pattern would
reclassify real permanent failures as retryable — the mistake the connection-only
rule was written to avoid. Regression tests pin both directions.
This does not affect model fallback, which pi decides internally and Fusion only
observes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review remediation loops were the dominant cost of task wall-clock: over 14 days,
tasks with >=5 post-review fix rounds were 22% of tasks but consumed 78% of all
task active time, and 311 of 331 recorded findings were spec-internal-consistency
complaints that changed no delivered behavior.
Two causes compounded. Plan/Code Review remediation was unbounded by default, and
the review policy ordered a full re-derivation of the artifact after every edit
("distrust the edit ... fresh holistic pass"), so each round surfaced a fresh crop
of previously-acceptable observations as new blockers.
Make the already-persisted WorkflowReviewFinding.severity load-bearing instead of
decorative: a REVISE only blocks when it carries a finding at or above the review
kind's threshold (plan: P0+P1, code: P0). Non-blocking findings are still parsed,
persisted, and handed to the implementer as advisory notes in PROMPT.md. Fails
closed — a REVISE with no findings, or with any unclassified finding, still blocks,
so prose-only and custom reviewers keep full blocking power. The gate only ever
relaxes a verdict, never promotes one.
Reviewer prompts now request the structured findings schema (Plan Review emitted
none before), define severity by consequence as P0/P1/P2, omit nits entirely rather
than filing them as low-severity findings, and use an incremental re-review contract.
Remediation renders findings grouped by priority and sanctions an explicit decline
with rationale, so a disputed finding has a terminal state.
Also preserve the implementation session across a review bounce: sendTaskBackForFix
no longer nulls sessionFile when preserving resume state, and the executor's finally
no longer clears it on a review handoff. Remediation rounds continue the conversation
instead of re-reading the repo and re-deriving the change they just wrote. The resume
prompt now directs a PROMPT.md re-read, without which a resumed agent would never see
the new findings.
New per-workflow settings planReviewBlockingSeverity / codeReviewBlockingSeverity;
set either to "any" to restore the previous behavior.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renewal runs every poll for every implementation-column task with declared
symbols, and a lost lock never recovers by renewing — renewSymbolLocks reports
the same lost set on each pass, so the warning and its store.logEntry companion
repeated forever: log-pane spam plus unbounded activityLog growth for a stuck
task. The two error paths had the same shape on any persistent failure.
Extract the executor's suppression into a shared createRepeatSuppressedLog and
use it in both: first occurrence per task/signature logs at full level, repeats
drop to debug(), a changed lost set or error message logs again, and a clean
renewal clears the memo. The logEntry write is gated on the same decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The unmet-dependency and ephemeral-disabled pre-dispatch gates re-run on every
dispatch attempt for a blocked task but only change state on the first, so every
later pass re-logged the same line at default level and drowned the log pane.
Route both through logDispatchBlockedOnce: first block per task/reason logs at
log(), identical repeats drop to debug() (FUSION_DEBUG=executor), a changed
reason logs again, and the marker clears when the gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
