## What
Adds a typed, queryable catalogue for engine run-audit events as the
first step of the delivery-pipeline reliability & observability effort.
- **New module** `packages/engine/src/run-audit/run-audit-catalogue.ts`
— a typed registry describing run-audit event kinds (scheduler,
self-healing, merger, worktree, symbol-lock, …) so pipeline
observability can ingest and reason about them consistently.
- **Parity test** `run-audit-catalogue.test.ts` — asserts the catalogue
matches the emitted run-audit event space.
- **Docs** `docs/run-audit.md` + index pointer.
## Why
Run-audit events are currently emitted ad-hoc without a typed contract.
A catalogue gives:
- a single source of truth for event kinds/names,
- a parity guard so any new or renamed event is caught,
- a foundation for delivery-pipeline reliability dashboards.
## Verification
- `@fusion/engine` `tsc` build → **PASS**
- `vitest run run-audit-catalogue.test.ts` → **3 tests passed**
- No production behavior change outside the new module.
## Scope
New isolated module + its test + docs. No changesets/release artifacts.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added a run-audit catalogue covering delivery-pipeline finalization,
self-healing reconciliation, and durable-agent error events.
* Documented recorded outcomes, emission conditions, audit-store
querying, and event catalogue maintenance.
* Added a documentation index entry linking to the new catalogue.
* **Tests**
* Added validation to ensure documented audit events remain complete,
consistently formatted, and synchronized with the supported event
catalogue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
## Summary
Supersedes #3405 — the fork head is not writable from maintainers, so
this branch carries the same fix rebased/merged onto current `main`.
## Conflict resolution
- Main already landed the equivalent fail-open path as **FN-8919**
(`readLinkedTaskOrUndefined` + per-agent try/catch).
- Kept the additional `recoverAgentsRunningOnInactiveTasks` regression
that covers task-gone races plus transient lookup isolation.
- Dropped the duplicate changeset (main already has
`fn-8919-agent-link-sweep-fail-open`).
## Test plan
- [x] `git merge-tree` clean against `main`
- [ ] CI green
Closes context from #3405.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved recovery handling when task lookups fail.
* Agents linked to deleted or missing tasks are now unlinked, while
agents affected by temporary errors remain preserved.
* Recovery continues for other eligible agents instead of stopping after
an individual lookup failure.
* **Tests**
* Added regression coverage for deleted, missing, and temporarily
unavailable tasks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Codex <codex@openai.com>
Convert the SettingsModal.general 'saves X via settings payload' tests from
real-timer waitFor (each burning a real ~500ms auto-save debounce) to the
FN-7506 fake-timer pattern already used by the auto-save tests in this file.
Payload-routing/coalescing assertions are unchanged; only the artificial
wall-clock wait is removed (Standing Rule: prefer fake timers over real time
waits). Isolated test-execution time drops ~21.7s -> ~15.0s (82 tests, all
green across repeated runs).
## Problem
Dashboard typecheck fails with **TS2416** in `@fusion/core`'s
`TaskStore`:
```
Property 'emit' in type 'TaskStore' is not assignable to the same property in base type 'EventEmitter<TaskStoreEvents>'.
```
The `override emit<E extends string | symbol>(event, ...args)` generic
conflicts with the base class's generic `emit<K>(eventName: keyof
TaskStoreEvents | K, ...)`. This breaks the dashboard typecheck / CI
merge gate.
## Fix
Change the override to:
```ts
override emit(event: unknown, ...args: any[]): boolean {
return EventEmitter.prototype.emit.call(this, event as string, ...args);
}
```
`event: unknown` remains assignable to the base's generic signature
while still forwarding non-typed runtime keys (`agent:log`,
`settings:updated`, …). Internal `EventEmitter.prototype.emit` calls
cast `event as string`. Behavior-preserving.
## Verification
- `@fusion/dashboard` `tsc --noEmit` → **PASS** (previously failed with
TS2416)
- `eslint` on touched file → clean
- Single-file change (`packages/core/src/store.ts`, +6/−3)
## Scope
No behavior change, no changesets required.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved task event handling to support a broader range of event
identifiers.
* Preserved cached-lane information for single-argument task update
events.
* Maintained support for custom and arbitrary event names without
disrupting existing behavior.
* Improved classification of workflow roles, session purposes, and
outcome-related status checks in lifecycle analysis, producing more
accurate findings and reducing misleading results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fusion wedged on "starting" and never brought the engine up. The dashboard bound
the migration holding server on 4040, then every query behind it failed with
"sorry, too many clients already", so the card never progressed and the
supervisor crash-looped.
Root cause: each spec-drift reconcile costs a DEDICATED PostgreSQL connection.
persist -> appendSpecDriftReport -> withPlanningLifecycleLock opens its own
postgres(directUrl, { max: 1 }) session, because the planning advisory lock is
session-scoped and deliberately fences a stale report against a newer plan.
enqueue() released every id straight into its own microtask, and project-engine
enqueues every task at runtime-boundary setup (listTasks includeArchived). On a
1,082-task project that opened ~1,082 lock sessions simultaneously against
max_connections = 500. The cluster saturated ~25s into boot and stayed saturated.
The flat 1s retry then made it self-sustaining rather than transient: once
saturated, every task failed for the same shared reason and re-armed in lockstep
once per second, re-opening the whole fleet of sessions and pinning the very
resource it was waiting on. Measured 4,777 lock sessions in 17 seconds.
Fix, contained to the reconciler — the advisory lock and its fencing semantics
are load-bearing and unchanged:
- concurrency bound (maxConcurrent, default 4) drained by a fair
insertion-ordered pump, so fan-out can no longer exceed a known connection cost
- per-task in-flight dedupe; two passes on one task would contend on that task's
own advisory lock while holding two connections
- exponential backoff with jitter capped at 60s, and retries re-enter through
enqueue so a retry storm is bounded by the same limit as a first pass
Verified against the real 1,082-task project: connections stay flat at 3-10
across a 70s boot that previously reached 1,109 and saturated, and the engine
boots through to executing tasks and shuts down cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`.fusion-knowledge/` is `fn knowledge-graph build` output (FN-8920 memory epic):
~85MB of generated JSON, rewritten on every build. edges.json alone is 43MB,
past GitHub's 50MB warning threshold.
It is deterministic and fully regenerable from source — the manifest's SHA-256
fingerprints drive incremental rebuilds — so tracking it preserves nothing a
rebuild cannot reproduce, while adding irreversible, compounding history bloat to
every clone and CI checkout.
docs/knowledge-graph.md previously left this to the operator ("it is not
ignored"); that guidance is updated here so the doc and .gitignore agree. An
operator who explicitly wants a snapshot in history can still force one with
`git add -f .fusion-knowledge/graph`.
Nothing under the path was ever tracked, so no index purge is needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Investigating the "dead cap" turned up the opposite of what it looked like, plus a
worse problem next to it.
The Plan Review replan loop was NOT unbounded. U3 re-owned the cap-park in the
graph: requestPreMergeOptionalStepFix parks via parkPlanReviewReplanCapExhausted
at awaiting-approval with reason plan-review-replan-cap, on both an explicit
finite budget and the unbounded default. That capability has been live throughout.
What was actually dead:
1. PLAN_REVIEW_GATE_REPLAN_CAP = 8 — an unread constant belonging to the
out-of-graph triage gate (runPlanReviewBeforeExecution) that U10/R4 deleted. Its
companion column Task.planReviewReplanCount was persisted, serialized and reset
but never incremented or compared. A constant and a column that look like a live
safety ceiling while enforcing nothing are worse than no ceiling: they answer "is
this loop bounded?" with a confident yes. Deleted, ratcheted in
legacy-tombstones.test.ts, and the column documented as legacy/never-written with
the live owner named.
2. planReviewReplanCap — an operator-facing setting, declared, validated,
documented in settings-reference.md and editable in the Workflow Editor, that
NOTHING read. Lowering it changed nothing. The unbounded backstop was instead
hardcoded to PLAN_REVIEW_FEEDBACK_HISTORY_LIMIT — a bound on how much reviewer
PROSE is replayed into the next planning prompt, whose own comment says it is
"bounded independently of persistence and retry accounting". Two unrelated
concerns shared one number, so trimming prompt history would have silently
tightened a safety ceiling.
The backstop now resolves from the setting, defaulting to the new
DEFAULT_PLAN_REVIEW_REPLAN_CAP = 15 — the previously-effective value, so this is a
pure re-wiring rather than a silent behavior change. The existing 15-attempt
regression test passes unchanged, which is the evidence for that. 0 is honored as
park-on-first-REVISE. An explicit planReviewMaxRevisions / node maxRevisions
budget remains a stricter, earlier gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes to the triage planning path.
1. Unclassified planning failures are bounded. specifyTask's catch-all branch —
the one reached by every error the classifiers above do not recognize — restored
the card's claimable status and wrote nothing else: no counter, no
nextRecoveryAt, no park. Triage rediscovery re-admitted the card on the very next
poll, and replaceActiveTaskWorkflowContinuation replaced the terminal work item
with a fresh one carrying no attempt count, so nothing recorded that the task had
already failed N times. It now consumes the same recoveryRetryCount/nextRecoveryAt
budget the transient branch uses (MAX_RECOVERY_RETRIES = 3, 60s/120s/300s jittered
backoff) and parks status:"failed" with a PLANNING_FAILED_EXHAUSTED: error once
spent — status:"failed" is what suppresses rediscovery. Classifying one error
string fixes one symptom; this budget is what makes the NEXT unrecognized error
fail safely instead of looping for a day.
2. The planning turn has a ceiling. Fusion set no timeout on it at all:
workflowStepTimeoutMs covers pre-merge workflow steps only, and the provider SDK's
300s APIConnectionTimeoutError caps time-to-first-byte and is cleared once headers
arrive, after which the stream is uncapped. configureHttpDispatcher, which would
install undici idle timeouts, is only called from pi's CLI entrypoints and never
in the in-process engine. Observed consequence: single attempts ran to 126 minutes,
with failed-attempt durations spread smoothly from 1 to 126 min and no clustering —
the signature of nothing enforcing a bound. New workflow-native planningTimeoutMs
(default 90 min) aborts the session; the failure consumes one bounded attempt.
The default is deliberately generous rather than tight. Successful planning work
items measured over 7 days ran p50 12.7 / p90 39.5 / p99 105.7 minutes, so a
tighter bound would abort legitimate plans and pay for the restart — the churn
this work exists to remove. It bounds hung turns, not slow ones.
3. [event:task:moved] executor tracing dropped from log to debug. It fires on
every dispatch, rebound, requeue, archive and self-healing move across every task,
which made it the loudest line in engine output and buried operator-actionable
events. No test pins the level; the information remains at debug.
Also fixes a test break shipped in 963dba6f80: the review blocking-severity
settings landed inside BUILTIN_REVIEW_REVISION_SETTINGS, whose contents
builtin-workflow-settings-triage.test.ts asserts exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Request timed out." is the literal default message of the Anthropic and OpenAI
SDKs' APIConnectionTimeoutError, surfaced to Fusion by checkSessionError after
pi-coding-agent exhausts its in-session retries. It matched none of the
connection-scoped timeout patterns, which deliberately excluded "general
timeouts", so it fell through to specifyTask's generic failure branch — the one
that restores status: null and writes no counter, no nextRecoveryAt, and no park.
Triage rediscovery then re-admitted the card on the very next poll, forever.
Measured before this change: 48 "Specification failed: Request timed out." events
across 10 tasks in 30 hours with zero backoff between attempts. FN-8950 alone
burned 8 consecutive attempts over ~8 hours and never reached implementation.
Across 2 days, 91 failed planning attempts averaged 33 minutes each — ~50 hours of
wall-clock producing nothing, 24% of all planning time.
Classifying these as transient routes them into the bounded recovery policy
(MAX_RECOVERY_RETRIES = 3, 60s/120s/300s jittered backoff) already used by the
connection-level patterns, so a provider blip costs three spaced retries instead
of an unbounded loop.
The pattern is anchored to "request timed out" rather than a bare timeout match:
agent log prose and verification output legitimately contain "timed out"
("BuildKit timed out", "stuck-kill unwind timeout"), and a broad pattern would
reclassify real permanent failures as retryable — the mistake the connection-only
rule was written to avoid. Regression tests pin both directions.
This does not affect model fallback, which pi decides internally and Fusion only
observes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review remediation loops were the dominant cost of task wall-clock: over 14 days,
tasks with >=5 post-review fix rounds were 22% of tasks but consumed 78% of all
task active time, and 311 of 331 recorded findings were spec-internal-consistency
complaints that changed no delivered behavior.
Two causes compounded. Plan/Code Review remediation was unbounded by default, and
the review policy ordered a full re-derivation of the artifact after every edit
("distrust the edit ... fresh holistic pass"), so each round surfaced a fresh crop
of previously-acceptable observations as new blockers.
Make the already-persisted WorkflowReviewFinding.severity load-bearing instead of
decorative: a REVISE only blocks when it carries a finding at or above the review
kind's threshold (plan: P0+P1, code: P0). Non-blocking findings are still parsed,
persisted, and handed to the implementer as advisory notes in PROMPT.md. Fails
closed — a REVISE with no findings, or with any unclassified finding, still blocks,
so prose-only and custom reviewers keep full blocking power. The gate only ever
relaxes a verdict, never promotes one.
Reviewer prompts now request the structured findings schema (Plan Review emitted
none before), define severity by consequence as P0/P1/P2, omit nits entirely rather
than filing them as low-severity findings, and use an incremental re-review contract.
Remediation renders findings grouped by priority and sanctions an explicit decline
with rationale, so a disputed finding has a terminal state.
Also preserve the implementation session across a review bounce: sendTaskBackForFix
no longer nulls sessionFile when preserving resume state, and the executor's finally
no longer clears it on a review handoff. Remediation rounds continue the conversation
instead of re-reading the repo and re-deriving the change they just wrote. The resume
prompt now directs a PROMPT.md re-read, without which a resumed agent would never see
the new findings.
New per-workflow settings planReviewBlockingSeverity / codeReviewBlockingSeverity;
set either to "any" to restore the previous behavior.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renewal runs every poll for every implementation-column task with declared
symbols, and a lost lock never recovers by renewing — renewSymbolLocks reports
the same lost set on each pass, so the warning and its store.logEntry companion
repeated forever: log-pane spam plus unbounded activityLog growth for a stuck
task. The two error paths had the same shape on any persistent failure.
Extract the executor's suppression into a shared createRepeatSuppressedLog and
use it in both: first occurrence per task/signature logs at full level, repeats
drop to debug(), a changed lost set or error message logs again, and a clean
renewal clears the memo. The logEntry write is gated on the same decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The unmet-dependency and ephemeral-disabled pre-dispatch gates re-run on every
dispatch attempt for a blocked task but only change state on the first, so every
later pass re-logged the same line at default level and drowned the log pane.
Route both through logDispatchBlockedOnce: first block per task/reason logs at
log(), identical repeats drop to debug() (FUSION_DEBUG=executor), a changed
reason logs again, and the marker clears when the gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FN-8923 sat silent in Todo for 7+ hours with zero run-audit rows. Its plan node
held on principal routing, triage correctly recorded `needs-replan`, and then
dependency auto-unblock nulled that status when its blocker completed. From that
moment the card was invisible to both lanes: triage saw a fully-written spec with
no replan flag and skipped it, while the executor's `isUnplannedForExecution`
refused to dispatch because no capacity-boundary continuation existed. Not stuck
in a retry loop -- unowned.
- Dependency auto-unblock clears only the `queued` marker it owns, at all four
sites (scheduler.ts plus three in self-healing.ts). `status` is a shared
lifecycle channel and `needs-replan` is the only signal that re-admits a
hold-column card whose PROMPT.md is already a real spec.
- New self-healing sweep `reconcilePrincipalHeldPlanningContinuations` re-queues
planning for a card whose sole active continuation is a principal-routing hold.
A planning hold otherwise has no retry owner at all. Gated on the planning
lane, effective auto-merge, an owned (null) status, and the shared planning
lifecycle lock, so it cannot clobber a triage claim or launder a `failed` /
`stuck-killed` / `queued` card into a replan.
- Workflow node-instance-id materialization is idempotent across foreach, loop,
and optional-group containers. It re-wrapped its own output on every dispatch,
so FN-8869 grew a ~1.8 KB `run_id` of ~30 repeated segments on a hot indexed
column and every retry read as a distinct run.
- An unresolvable node instance or absent IR now fails closed instead of being
treated as an edited-away override -- the previous shape would have discarded a
real reviewer fence and handed a named review to the pool.
- Mirror the routing exports into the gate-safe core barrel; the reduced barrel
resolved them to `undefined`, a latent trap for any suite reaching the router.
Findings from a multi-reviewer pass; 9 of 11 confirmed by an independent
validator. Each fix carries a regression asserting the invariant across its
surfaces, not the single reported case -- the optional-group accretion test was
verified to fail without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Workflow principal routing conflated two different questions: whether an agent
CAN run a node, and whether it can run it RIGHT NOW. Both produced a hold, and a
named principal never falls through to the role pool — so an agent that could
never satisfy the node wedged the task permanently.
FN-8869, FN-8928, and FN-8845 were each explicitly assigned to a permanent
engineer-role agent (which the assignment policy allows). Their `step-execute`
nodes took that owner as `task-assignee` authority, found no `executor` tag, and
held closed. Each card re-dispatched and re-held every ~15 minutes for hours
while two idle `Workflow Executor` pool agents were never consulted. The only
thing still touching them was the owner's hourly heartbeat, which logged
"progressing, no blockers" and exited: heartbeat observation had replaced
execution.
- Structural incapability (wrong role, agent deleted, authority edited away) is
no longer authority for the node. Routing continues to the column binding and
then the role pool.
- A resumed continuation whose fence proves stale discards it and re-routes,
instead of re-asserting a dead principal on every dispatch.
- Availability is unchanged and still fail-closed: a role-capable principal that
is paused, disabled, or at session capacity holds, and is never silently
replaced by a pool member.
- An explicitly assigned engineer-role agent is now valid task-assignee
authority for an executor node, so the assigned agent executes its own task
continuously under graph dispatch. The role pool stays strict, since automatic
backlog pickup by engineers is a separate opt-in.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
