Follow-up to #2785. Nine engine files to **zero**, all one question —
*"is this task finished?"* — asked in nine places, wrong in every one on
a renamed board.
## Census, per file (measured, `--strict` verified)
| file | main | here |
| --- | ---: | ---: |
| `agent-reflection.ts` | 2 | **0** |
| `merger-scope-auto-widen.ts` | 2 | **0** |
| `worktree-pool.ts` | 2 | **0** |
| `cli-agent/state-machine.ts` | 2 | **0** (reclassified — see below) |
| `auto-recovery-handlers/branch-worktree.ts` | 1 | **0** |
| `cli-agent/task-session.ts` | 1 | **0** (reclassified) |
| `merger-integration-worktree.ts` | 1 | **0** |
| `merger-orphan-rehome.ts` | 1 | **0** |
| `plugin-runner.ts` | 1 | **0** |
| **net** | | **−13** |
## The one worth reading: `branch-worktree` had TWO defects, and the
census could only see one
```ts
if (task.column === "in-progress") { …clear branch… } // counted
await this.deps.taskStore.moveTask(task.id, "todo", { … }); // INVISIBLE
```
The census scores **comparisons**. The requeue *destination* is a call
argument, so nothing in the backlog ever pointed at it — and it is the
worse of the two: a board with no `todo` column was requeued into a lane
**that does not exist**. The counted literal is the smaller half (a
renamed wip lane meant the stale branch was never cleared, so the card
carried a dead branch back into execution).
Converting the comparison alone would have dropped a census count and
left the board requeuing into nowhere. Destination now resolves through
`resolveReboundTarget` (KTD-10 ordering: hold → intake → first column).
Reverted **independently**: destination restored → 2 fail (`moveTask`
called with `"todo"`, not `"backlog"`); wip test restored → 1 fail
(`updateTask` never called).
## The rest
- **`plugin-runner`** — `onTaskCompleted` never fired on a renamed
board. Every plugin that closes an issue, posts a notification, or
records a metric on completion **silently stopped**, with nothing
logged. Resolved *asynchronously* inside the existing fire-and-forget
seam, not via `resolveTaskWorkflowIrSync` — per
`sync-workflow-ir-callsite-allowlist` that reader returns the DEFAULT
workflow for every task in production, so a sync guard here would read
as converted and still be wrong. The listener is already
`void`-dispatched, so awaiting inside it changes no observable ordering
(the shape `NotificationService` already uses).
- **`merger-orphan-rehome`** — a renamed complete lane made every source
task read as unfinished, so orphaned commits were never rehomed and
stayed stranded off the integration branch. Resolves by the **trailer
id**, not `sourceTask.id`, which the fake store does not populate.
- **`agent-reflection`** — `classifyOutcome` returned `null` for every
finished task, so both callers treated completed work as nothing to
reflect on: one recorded `reflection:skipped` with reason
`"not-completed"`, the other silently `continue`d. Reflection captured
**nothing at all** on a custom board.
- **`worktree-pool`** — shipped tasks' worktrees stayed in the ACTIVE
set, so the reclaim pass never returned them and the board walks into
worktree exhaustion — a stall whose cause is invisible from the symptom.
- **`merger-scope-auto-widen`** — finished cards counted as active
claimants, so a merge was blocked by a task that no longer exists in any
meaningful sense.
- **`merger-integration-worktree`** — a shipped task still counted as a
live worktree user, so the integration worktree could never be reused
and the merge path took the slower rebuild every time.
## The census OVERSTATED the engine backlog by 3
`cli-agent/state-machine.ts` and `cli-agent/task-session.ts` compare
against `done` — but that is a **`CliMachineState`**
(`ready`/`busy`/`waitingOnInput`/`done`/`resuming`/`idle`) tracking one
CLI agent process. It never reads a board column. The census matches the
bare string.
Marked `DELIBERATE-LITERAL` rather than left for a later sweep to
"convert" a process state into a workflow role. Worth flagging
fleet-wide: the backlog total includes at least these three non-columns.
## Revert results (measured, each run)
| conversion | reverted → |
| --- | --- |
| `plugin-runner` complete gate | RENAMED case fails — `onTaskCompleted`
never invoked |
| `merger-orphan-rehome` source gate | RENAMED case fails —
`orphan:false, reason:"source-task-not-done"` |
| `branch-worktree` destination | 2 fail — `moveTask` called with
`"todo"` |
| `branch-worktree` wip test | 1 fail — `updateTask` never called |
Each has a **non-vacuous companion** (renamed board, non-complete lane /
mid-flight source / non-wip column) so a guard that fired
unconditionally would not pass.
**Four are NOT revert-proven, and I am not claiming otherwise:**
`agent-reflection`, `merger-scope-auto-widen`,
`merger-integration-worktree`, `worktree-pool`. Their suites omit a
workflow and therefore assert the legacy fallback — they pass before and
after. `merger-scope-auto-widen` has no test file at all;
`scanIdleWorktrees` is mocked in every suite that touches it and driving
it for real needs git worktrees on disk. All four strictly **widen** the
finished set (resolved roles ∪ the legacy ids), so default boards are
byte-identical. That is the argument for shipping them, not a substitute
for coverage.
## Examined and deliberately NOT converted
- **`backlog-pressure-reporter:173`** — fed by `listTasks({ column:
"todo" })`, a hardcoded **query** filter. On a renamed board `todoFull`
is empty and the predicate never runs. Converting it drops a census
count and changes nothing observable; the fix belongs at the query
layer.
- **`auto-merge-finalization:28`** — the catch-arm legacy fallback,
which must stay for the same reason `columnRoles.ts` keeps its id
fallback.
- **`auto-merge-finalization:84`** — only selects between two diagnostic
reason strings that are **both** `ok: false`, on a pure validator with
no store in scope. Converting it would thread a store through a pure
function to change a label.
## Merge resolution note
Merging main brought conflicts in `agent-assignment.ts` and
`ephemeral-worker-manager.ts`. **Main's versions won both** and mine are
dropped: main threads an optional `activeColumns` from
`scheduler.ts:2340` (a cleaner seam than widening the store type to
resolve internally), and its `isAgentIdle` carries a greptile P1 fix
mine lacked — `columnsWithFlag` membership rather than first-per-role,
so a workflow declaring two wip lanes has both recognised. That is the
fourth time in this program main's version of a contested file was the
better one.
## Verification
- `pnpm test:gate` — 161 + 487 + 13 + 71, green
- `npx tsc -p packages/engine/tsconfig.json --noEmit` — clean
- `pnpm lint` — clean
- `--strict` exits 0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Workflow-dependent task completion now recognizes custom lifecycle
columns, including renamed boards.
* Recovery requeues tasks to the configured destination and clears
branch details only from the appropriate work-in-progress column.
* Improved handling of completed tasks, orphaned work, shared worktrees,
and scope evaluation across custom workflows.
* Plugin completion hooks now trigger for any column configured as
complete.
* **Tests**
* Added coverage for renamed workflow columns and custom completion,
recovery, and rehoming behavior.
* **Documentation**
* Clarified CLI state terminology in internal developer comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
Engine and dashboard traffic now stays on the authoritative PostgreSQL
layer across execution, recovery, project discovery, planning sessions,
analytics, and shutdown. The dashboard no longer presents a migration
notice for a cutover that is already mandatory.
## Design decisions
- Runtime composition requires an async data layer instead of
constructing a hidden SQLite fallback.
- Engine workflow, mission, claim, and self-healing reads await their
PostgreSQL-backed store contracts.
- Project-scoped dashboard stores retain and close their backend owner
exactly once.
- The dashboard test quarantine entry remains paired with its Vitest
exclusion, preserving the repository’s deletion-ratchet policy.
## Validation
- Core, Engine, Dashboard, CLI, and Desktop typechecks pass on the
stacked branch.
- `pnpm test:gate` passes all 478 gate tests.
- This PR changes 62 files.
## Stack
- Depends on #2108.
- CLI/desktop/ops, plugins, and docs/release follow in later PRs.
Related: #2105
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Project discovery now recognizes projects using the
`.fusion/project.json` marker.
* Knowledge indexing and search are more reliable across project-scoped
storage.
* **Bug Fixes**
* Improved session, audit timeline, approval, monitoring, and analytics
data consistency.
* Prevented stale planning-session updates and project-store shutdown
races.
* Ensured chat usage and CLI session status are saved before continuing.
* **UI Changes**
* Removed the storage migration notice banner now that the PostgreSQL
transition is complete.
* **Reliability**
* Improved shutdown handling, workflow execution, and worktree behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Fusion’s core runtime now treats PostgreSQL as the authoritative
metadata store without leaving current CLI, dashboard, desktop, or
engine composition roots uncompilable between stack layers. This is the
99-file foundation for the larger cutover: subsequent PRs migrate the
remaining consumers, plugins, and operator surfaces.
## Design decisions
- Runtime store construction fails closed when an asynchronous
PostgreSQL layer is unavailable; SQLite remains readable only at
explicit migration and identity-recovery boundaries.
- Project ownership is enforced across active, archived, workflow,
mission, analytics, and plugin-schema data.
- The small set of cross-package files in this layer are
compatibility-critical call sites required for a green intermediate
commit, not the complete consumer migration.
- Schema migration 0008 remains assigned to session-advisor state from
current `main`; mission lineage idempotency advances to 0009 so neither
invariant can be skipped.
## Validation
- All affected package typechecks pass: Core, Engine, Dashboard, CLI,
and Desktop.
- `pnpm test:gate` passes: 478 tests across the engine gate, PostgreSQL
core gate, and CLI workflow shape.
- The PR changes exactly 99 files.
## Stack
This is the base PR. Engine/dashboard, CLI/desktop/ops, plugins, and
docs/release follow as stacked PRs, each below 100 changed files.
Related: #2105
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* PostgreSQL is now the standard runtime backend, with embedded
PostgreSQL enabled by default.
* Added project-scoped storage for tasks, archives, chat sessions,
missions, knowledge pages, and operational data.
* Improved archived-task search, filtering, pagination, and restoration.
* Added safer plugin schema initialization with validation and project
isolation.
* Added PostgreSQL-backed workflow, mission, validator, and dashboard
capabilities.
* **Bug Fixes**
* Improved startup timeout cancellation and resource cleanup.
* Prevented cross-project data access and phantom reservation cleanup
errors.
* Ensured archived tasks remain read-only and asynchronous writes
complete reliably.
* Retired SQLite opt-out settings with clear startup errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover
Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.
## Status — every surface works in embedded-PG mode
Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).
| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |
## Approach
Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.
Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.
## Sync with main
The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.
## Residual Review Findings
Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).
- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.
~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.
---
## Update — 2026-07-12: production-readiness hardening & live acceptance
Everything below landed on this branch since the description above was
written:
**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).
**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.
**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.
**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.
**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
Mount CliChatSurface in ChatView for cli-backed chat sessions (sessions carrying
cliExecutorAdapterId): the message-pane + composer region is delegated to the
surface (transcript/raw-terminal toggle for hybrid/native adapters, terminal-only
for the generic adapter), while regular sessions keep the standard composer. The
existing message list and composer JSX are captured once as render thunks and
passed through, so there is no parallel message/composer UI.
Add a narrow telemetry seam: TelemetryHub gains an optional onEvent tap (also
settable post-construction via setEventListener) invoked with each sanitized
event after routing — best-effort, a throwing listener never breaks ingest. This
is the seam the CliChatSessionRunner uses to build the durable transcript from
the same sanitized events the hook route already feeds the hub, without the hub
becoming a general subscriber bus.
Fix the stale @fusion/engine vi.mocks across dashboard tests: object-literal
mocks that fully replace the module now also return listCliAdapterDescriptors
(added by U15's cli-agent-settings route, evaluated at module load). Mocks that
spread importOriginal/importActual already pick it up.
Tests: new ChatView.cli-mount.test.tsx (cli session → CliChatSurface, regular
session → normal composer, generic → terminal-only); telemetry-hub onEvent tap
coverage. chat-attachment-routes, chat-cli-sessions, cli-agent-hooks-route,
ChatView.cli-toggle all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `cli-agent` as a task-execute executor kind. A workflow node with
`config.executor === "cli-agent"` drives an engine-owned CLI coding agent
through the execute step via the new cli-agent/task-session.ts orchestration:
spawn in the worktree, issue the hook token + write hook scripts, inject the
prompt after readiness, subscribe to the state machine, and resolve on a
positive completion signal (R20 gating). Config is snapshotted at launch; the
PTY is reaped (completed) at the in-review handoff.
Executor seam: runGraphCustomNode gains a cli-agent branch delegating to
runCliAgentNode; the hard-cancel/abort path (awaitAbortInFlightTaskWork +
abortAllInFlight) claims and SIGKILLs the CLI session as a first-class surface,
marking it killed (never resume-eligible). Re-entry kills any prior live
session and launches fresh; follow-up resumes the recorded native session id
when supported. A PTY-pool ceiling surfaces as a typed task value, not a stall.
Node-config typing extended minimally (WorkflowNodeExecutorKind /
WorkflowNodeExecutorConfig in @fusion/core).
Tests: cli-agent/__tests__/task-session.test.ts (12) and
__tests__/cli-agent-executor.test.ts (9) cover AE1/AE5, hard cancel, re-entry,
follow-up, config snapshot, ceiling, and the generic-tier confirm-advance path,
using scripted adapters + a mock PTY seam. Engine typecheck clean; full
src/cli-agent suite + new tests green (144 passing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pure engine code (no HTTP) for the CLI agent executor:
- state-machine.ts: authoritative per-session state machine implementing the
HTD diagram (starting→ready→busy⇄waitingOnInput→done; done→busy follow-up;
dead-classification choice → killed/userExited/authFailed/resuming; resume cap
of 2 with backoff → needsAttention). Positive completion is distinct from idle
(idle never produces done); inactivity stall backstop re-armed by output/
telemetry events (no fixed turn timeout); termination classification helper for
all five paths; per-turn latches reset between turns. Persists every transition
via CliSessionStore (the transient `resuming` machine state maps onto the U1
`dead` store enum) and exposes a throttled `onStateChange` subscription for the
later SSE bridge — no dashboard imports.
- telemetry-hub.ts: in-process ingestion contract (ingest(sessionId, event)) for
the U17 route and log-tailing adapters. Mints high-entropy per-session hook
tokens (issueToken/validateToken/invalidate); rebuilds the registry only from
live sessions in CliSessionStore so stale tokens for non-live sessions never
validate; a token validates only for its own session. Bounds everything
ingested: per-event size caps, per-turn count caps (lifecycle events exempt),
ANSI/control stripping before pattern matching, and secret redaction that
survives chunk boundaries via a held-back carry window (redactSecrets from
@fusion/core).
Tests: 35 new (state-machine.test.ts, telemetry-hub.test.ts) covering AE1/AE2,
stall backstop, all termination paths, resume caps, token registry, two-turn
latch reset, oversized capping, ANSI stripping, and cross-chunk redaction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Engine-owned PTY lifecycle for CLI agent sessions:
- adapter.ts: CliAgentAdapter interface (launch/env-allowlist builders,
capability flags, readiness detection, injection formatter, resume builder,
telemetry wiring) + CliAdapterRegistry with typed unknown/duplicate errors.
- session-manager.ts: CliSessionManager owning node-pty processes via the U16
shared loader. Byte-bounded scrollback ring (default ~512KB), single
serialized write queue shared by injections + user input (FIFO, deferral in
quiet windows), latest-active-client resize, scoped-SIGKILL process registry
on process exit (never port 4040), explicit async attach interface
(scrollback + AsyncIterable<Uint8Array> + write/resize/detach),
requestPause/requestResume watermark hooks, separate concurrency pool with
typed CliConcurrencyLimitError at the ceiling.
- Security: bracketed paste only when ?2004h observed; unconditional control-char
neutralization on the raw path; user keystrokes bypass neutralization.
- Persists lifecycle into the U1 CliSessionStore (create on spawn, update
state/termination).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
