## Summary
Bundled plugins now persist shared runtime state in project-scoped
PostgreSQL tables instead of maintaining independent SQLite authority.
Reports, CLI Printing Press, Compound Engineering, Roadmap, Even
Realities, and WhatsApp all follow the same ownership and startup
contract as Fusion core.
## Design decisions
- Plugin schema hooks run through the host’s PostgreSQL owner and
enforce project isolation.
- The SDK exposes the host contract needed by bundled plugins without
importing engine internals.
- Legacy Roadmap ownership fixtures use the supported empty-owner
sentinel, preserving current composite primary/foreign keys while
exercising backfill behavior.
- The lockfile travels with the Even Realities PostgreSQL dependency so
packaged installs remain reproducible.
## Validation
- All six affected plugin builds pass.
- Affected plugin suites pass: 773 tests across Printing Press, Compound
Engineering, Even Realities, Reports, Roadmap, and WhatsApp.
- `pnpm test:gate` passes all 478 gate tests.
- This PR changes 40 files.
## Stack
- Depends on #2110 → #2109 → #2108.
- The documentation/release PR completes the stack.
Related: #2105
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Breaking Changes**
* PostgreSQL is now required for runtime storage; SQLite files are used
only as one-time migration inputs.
* The legacy `FUSION_NO_EMBEDDED_PG` fallback has been removed.
* **New Features**
* Added project-isolated PostgreSQL storage for plugins, reports, tasks,
notifications, and other plugin data.
* Added agent tools for reports and CLI service drafts.
* Added PostgreSQL schema initialization support for plugin authors.
* **Bug Fixes**
* Improved migration and recovery of legacy plugin state.
* Prevented cross-project data access and strengthened transactional
schema updates.
* **Documentation**
* Updated storage, migration, deployment, plugin authoring, CLI, and
dashboard guidance for PostgreSQL.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- add asynchronous PostgreSQL parity to research commands and engine
execution paths
- persist Roadmap, Compound Engineering sessions, and WhatsApp state in
PostgreSQL
- harden cancellation, concurrency, reconnect, replay-claim, and
detached-promise behavior
- bundle the PostgreSQL-backed integration implementations in the
published CLI
This is PR 2 of 2 and is intentionally stacked on #2088. It contains 44
changed files; merge #2088 first, then retarget this PR to `main` if
GitHub does not do so automatically.
## Verification
- `pnpm check:changesets --strict`
- `pnpm lint`
- `pnpm test:gate`: 463 tests passed
- Compound Engineering plugin: 299 tests passed
- Roadmap plugin: 144 tests passed
- WhatsApp plugin: 27 tests passed
- research CLI: 18 tests passed
- `pnpm verify:fast`: all scoped typechecks, builds, CLI build, and boot
smoke passed
## Post-Deploy Monitoring & Validation
- deploy only after #2088 and verify schema migration `0002` is present
- monitor research cancellation, automation claims, agent execution,
plugin schema initialization, and unhandled rejections
- validate Roadmap ownership, Compound Engineering session recovery, and
WhatsApp reconnect/replay deduplication
- compare per-project plugin and workflow counts after cutover
- restore the pre-deploy backup for data rollback; avoid an in-place
schema downgrade
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover
Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.
## Status — every surface works in embedded-PG mode
Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).
| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |
## Approach
Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.
Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.
## Sync with main
The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.
## Residual Review Findings
Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).
- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.
~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.
---
## Update — 2026-07-12: production-readiness hardening & live acceptance
Everything below landed on this branch since the description above was
written:
**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).
**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.
**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.
**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.
**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
- scope session collections to the active project\n- clean up runtime stage registration in discovery tests\n- model brainstorms and plans as repeatable artifact collections\n- compact the singleton Strategy presentation
Users can now watch everything the agent does while a CE stage works, steer
it mid-stage, and read the whole conversation as a proper chat surface.
Live output:
- New host capability: CreateInteractiveAiSessionOptions.onProgress — the
engine adapter streams thinking/text deltas + tool start/end markers from
the pi agent hooks (any plugin can use this).
- Orchestrator buffers per-session live activity (merged deltas, discrete
tool lines, capped), emits throttled progress events over SSE, and
GET /sessions/:id attaches it as liveActivity for the polling fallback.
- Routes detach turn execution: start/answer/resume return immediately
(status active) and clients converge via push/poll — the turn is watchable
instead of hidden inside a blocking POST.
- Turn timeout is now INACTIVITY-based: an actively-working long turn is
never killed; a quiet one interrupts with its working trace preserved.
- On settle the trace persists into history as a condensed record.
Steering:
- Stage protocol: responses may be a direct answer, {value, comment}
(answer + guidance), or {feedback} (guidance without answering); the
system prompt instructs agents to treat steering as first-class input.
- CeFlow: guidance textarea alongside selectable questions — attach to the
clicked answer, or "Send guidance" on its own.
Q&A UI:
- Transcript no longer hides control records: past questions/answers render
as chat bubbles (option ids → labels), steering turns marked, working
traces as collapsible "Agent work" blocks, completion marker.
- Live working pane (pulse + streaming thinking/tool lines) while a turn runs.
Tests: 130 plugin tests green (14 new: live buffer/flush ordering, inactivity
watchdog survives active work, detached convergence, steering payload shapes,
transcript rendering, live pane). Engine seam tests green; plugin/core/
engine/dashboard tsc clean. Core full suite OOMs locally (known orchestrator-
shell issue) — covered by CI shards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The store/orchestrator were already multi-session (independent rows + live
handles per session); this surfaces it end to end:
- Sessions panel in the dashboard view: lists every session with stage,
status badge ("needs your input" for awaiting_input), and last activity;
stays visible while a flow is open so switching is one click. Closing a
flow returns to the overview without stopping the session.
- useCeSession.open(): adopt an existing session (pins its projectId for
answer/resume/poll); useCeSessions list hook with push-event refresh and
poll fallback while any session is mid-turn.
- DELETE /sessions/:id + orchestrator.discard(): dispose the live handle
before deleting the row (pipeline-link rows kept for task provenance);
Discard affordance on settled sessions.
- Tests: cross-session independence through one orchestrator, store delete,
route list/delete, hook open/list/remove/push/poll, view panel
open/switch/discard. 116 tests green; plugin + dashboard tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
safeParse() previously only caught JSON syntax errors, so a semantically-wrong
but valid column ('null', '{}', a string) would rehydrate a non-array
conversationHistory that later crashed appendHistory's spread (and a bogus
currentQuestion). safeParse now takes a shape validator and falls back to []/null
on invalid shapes too. + regression test covering 'null'/'{}'.
- session-store: rowToSession parses JSON columns via a safeParse helper, so a
corrupted currentQuestion/conversationHistory column degrades to null/[] instead
of throwing and crashing reads of an otherwise-valid row (+regression test)
- session-routes: validate the ?status= list filter against CE_SESSION_STATUSES
(asCeSessionStatus) instead of casting an arbitrary query string with 'as never'
- _harness: makeScriptedSession throws on an empty script rather than yielding
undefined, surfacing test mistakes loudly
- useCeSession tests: use fake timers (advanceTimersByTimeAsync) instead of real
setTimeout waits for deterministic poll-interval assertions
Skipped: 3 doc nits in src/skills/ce-*/references/** — those are pinned upstream
ce-* skill copies (KTD5 vendored snapshot), not this repo's content.
Closes the U2/U5 skill-discovery carry-forward so the plugin's interactive ce-*
sessions actually load the stage's bundled skill in a live agent (not just in
scripted-fake tests).
Root cause: createFnAgent built its DefaultResourceLoader without forwarding any
skill-discovery path, and the interactive seam options couldn't carry one. The
loader's skillsOverride only *filters* skills already discovered from cwd's
standard roots, so the plugin-local .fusion-ce-skills/<id>/SKILL.md was never
discoverable.
Fix (end-to-end):
- AgentOptions.additionalSkillPaths forwarded into DefaultResourceLoader
- CreateInteractiveAiSessionOptions gains requestedSkillNames + additionalSkillPaths
- the interactive engine adapter forwards them to createFnAgent (skills +
additionalSkillPaths)
- the orchestrator runs the session with cwd on the real project root and hands
it [stage.skillId] + the install root
Proven: a real DefaultResourceLoader with additionalSkillPaths discovers ce-plan
and filters out ce-work; the orchestrator passes the right id/path/cwd. Plugin 96,
engine 136, core 99 tests green.
- reconciler: a deleted current-stage board task no longer wedges the pipeline
in 'running' forever (terminality computed over existing tasks only; all-deleted
is a no-op, not a wedge)
- session-store: a human-slow awaiting_input session is no longer misclassified
stale (interval rubric applies only to in-flight active/launching turns)
- stage-registry: pipeline progression uses an explicit order ordinal instead of
registry insertion order, so out-of-order registration can't corrupt advancement
- orchestrator.answer(): validate questionId before mutating state, so a stale id
can't destroy the persisted currentQuestion recovery anchor
- orchestrator.resume(): rehydrate a live interactive session by replaying
persisted history (side effects suppressed) so a resumed session is actually
answerable instead of dead-ending; honest interrupted+error fallback when no
factory is available
95 tests (6 new regression tests, each confirmed failing pre-fix).
Quality cleanup across the 9-unit build (behavior-preserving, 89 tests green):
- extract createCeTaskWithLink so the work bridge and reconciler share one
provenance+link contract (prevents drift)
- discovery list scan probes readability via accessSync instead of reading and
discarding full file bytes
- makeError helper replaces ~7 duplicated CeArtifactError literals
- shared asString route helper; drop dead pipelineIds set; collapse a double
pipeline-state write and a redundant link re-query in advance
- resolveStageSkillCwd no longer takes params it ignores
Add settingsSchema (default session provider/model, enabled stages, sync
reconcile-on-hooks toggle, reconcile cadence hint) aligned across manifest.json
and the runtime manifest, with typed getters. Wire the consumed getters:
orchestrator passes defaultProvider/defaultModelId into sessions and rejects
disabled stages; hooks gate their reconcile drain on reconcileOnHooks. Add the
plugin README and docs/plugins/compound-engineering.md documenting the hub,
interactive sessions, work bridge, and the sync ownership model.
reconcileIntervalMinutes is exposed as an operator cadence hint but not yet
consumed (no host scheduler; reconcile is on-demand by design).
Add a ce_pipeline_state machine (currentStage/status) kept distinct from
board-task ownership (task column) — separate tables, no shared column, per
FN-5719. onTaskMoved/onTaskCompleted hooks do only an indexed lookup + enqueue
and return well under the 5s budget; advancement happens in an on-demand
reconciler sweep that re-derives correct pipeline state from board truth, so a
dropped hook event still converges (no tight poll loop). Outbound CE-flow
changes create the next-stage board task. Conflict policy: board authoritative
for task state, CE flow authoritative for artifact/pipeline content (inbound
read and outbound write target different rows, so they cannot contend).
Host-scheduler note: no host timer wired; sweeps run on hook-drain and on the
dashboard/route refresh surface (tracked for the host event-publish follow-up).
When the work stage completes with a derived task list, create Fusion tasks via
ctx.taskStore.createTask tagged CE-originated (sourceType workflow_step +
sourceMetadata marker) and record an authoritative ce_pipeline_links row per
task (back-reference lives in the link table, not task-row JSON, per FN-5719).
Tasks then run the normal lifecycle untouched. Zero derived tasks is a clean
no-op. The link store is intentionally minimal for U8 to extend with the
bidirectional pipeline-state machine.
Extend the stage registry with presentation metadata (icon/label/glob) so adding
a stage stays data-only. Add CeFlow renderer for text/single_select/multi_select/
confirm questions over the U5 polling session routes, with a visibly-marked chat
fallback (AE1) that still completes the stage. Wire the artifact-hub launcher to
list and start registered stages. Add the skill-interaction audit test producing
a measured rich-vs-chat coverage ratio (declared classification: 8/8=100% across
brainstorm/ideate/plan), failing on unclassified interactions.
Add ce_sessions schema (onSchemaInit), CeSessionStore (via
ctx.taskStore.getDatabase()), and CeOrchestrator driving a stage's skill on the
U4 interactive seam: streams thinking/text, persists questions as
awaiting_input, writes the stage artifact on complete, and auto-saves +
emits an observable event on interrupt/error (never silent loss). Lifecycle:
launching/active/awaiting_input/completed/error/interrupted with resume and an
interval-relative staleness rubric. Start/answer/resume/get-state routes.
Streaming transport is client polling of the get-session-state route for v1:
plugin routes have no native SSE and the loader emitEvent is a logging stub, so
true server push needs a host publish-to-/api/events seam (tracked follow-up).
Skill discovery is cwd+prompt-based; forwarding install paths through the seam
is a tracked follow-up.
