Files
fusion/plugins/fusion-plugin-compound-engineering
gsxdsm c15c78feeb feat: migrate storage from SQLite to PostgreSQL (#1793)
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover

Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.

## Status — every surface works in embedded-PG mode

Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).

| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |

## Approach

Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.

Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.

## Sync with main

The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.

## Residual Review Findings

Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).

- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.

~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.







---

## Update — 2026-07-12: production-readiness hardening & live acceptance

Everything below landed on this branch since the description above was
written:

**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).

**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.

**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.

**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.

**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
2026-07-13 19:07:58 -07:00
..
2026-07-13 10:32:12 -07:00

Compound Engineering Plugin for Fusion

A dedicated dashboard surface for the compound-engineering (CE) workflow: an artifact hub, interactive in-dashboard ce-* skill sessions, a work→board bridge, and event-driven bidirectional sync between the Fusion board and a plugin-local CE-pipeline state model. It runs alongside Fusion's native pipeline — it does not replace or bypass it.

Install (one-click)

  1. Open Settings → Plugins → Fusion Plugins.
  2. In Bundled Plugins, click Install on Compound Engineering.
  3. Enable the plugin if prompted.

Once installed and enabled, Fusion registers the Compound Engineering dashboard destination automatically and installs the bundled ce-* skills into a plugin-local, discoverable directory.

What it does

Compound engineering normally runs as terminal slash-commands whose artifacts scatter across docs/, with no unified surface and no link between a finished plan and the board work that follows. This plugin surfaces the whole flow inside Fusion while reusing the real skills so the plugin improves as they do.

Artifact hub

The primary dashboard view (viewId: "compound-engineering") discovers and renders CE artifacts from their conventional locations (STRATEGY.md, docs/ideation/, legacy docs/brainstorms/, unified docs/plans/, docs/work/, docs/debug/, CONCEPTS.md, docs/solutions/) and groups them by stage. Artifacts are read through a plugin route and rendered self-contained (sandboxed preview). The hub renders explicit empty / partial / error states rather than crashing or silently dropping an unreadable artifact.

brainstorm and plan are an alias pair at the artifact layer: Fusion keeps the separate brainstorm / plan stage IDs and bundled ce-brainstorm / ce-plan skill IDs for session history, pipeline ordering, and board back-compat, while both stages now operate on upstream unified plan artifacts in docs/plans/. A brainstorm writes a requirements-only plan (artifact_contract: ce-unified-plan/v1, artifact_readiness: requirements-only, product_contract_source: ce-brainstorm); ce-plan enriches that same file in place to artifact_readiness: implementation-ready. The hub still discovers historical docs/brainstorms/*.md files as legacy input, but new brainstorm output targets docs/plans/.

Artifact HTTP endpoints live under /api/plugins/fusion-plugin-compound-engineering/ and back the hub list/read.

Interactive ce-* sessions

Each pipeline stage maps to a bundled skill via the stage registry (src/session/stage-registry.ts): { stageId, skillId, artifactLocation, icon, label }. The default launchable stages are Strategy, Ideate, Brainstorm, Plan, Work, and Debug. The Brainstorm registry entry intentionally points at docs/plans/ even though the durable stage ID remains brainstorm, matching the unified-plan alias described above. Adding a stage is a data entry — no new route, store, or screen.

The launcher lists the registered (and operator-enabled) stages. Launching a stage starts an interactive agent session driven by the host's createInteractiveAiSession seam (a foundational extension added by this plan, because the existing createAiSession is one-shot and cannot pause on a mid-agent question). The session orchestrator (src/session/orchestrator.ts):

  • streams thinking / text turns,
  • surfaces a structured question and pauses in awaiting_input,
  • accepts a structured answer and continues,
  • on complete, writes the artifact to the stage's conventional location (for Brainstorm and Plan, the shared unified docs/plans/ plan path).

Lifecycle states are launching → active → awaiting_input → completed, plus error and interrupted. On interrupt or error the orchestrator auto-saves progress and emits an observable event — never silent loss — and an interrupted/error session can be resumed/retried back to its current question. If the server restarts while a session is already awaiting_input, submitting the pending answer rehydrates the live interactive handle from the persisted conversation history before continuing, so old answerable sessions do not require a separate resume action.

Multiple sessions

Sessions are independent pipeline runs — the store, routes, and orchestrator all hold many at once (each with its own live agent handle). The dashboard's Sessions panel lists every session with its stage, status, and last activity; from there you can:

  • open any session and keep working on it (an awaiting_input session is flagged "needs your input"),
  • switch between sessions — the panel stays visible while a flow is open, and a session you switch away from keeps running server-side,
  • resume an interrupted/error session from where it stopped,
  • cancel an in-flight (launching/active/awaiting_input) session via POST /sessions/:id/cancel, which stops any live in-process handle, flushes live progress into history, and keeps the row as interrupted with a Cancelled by user marker for inspection/resume,
  • discard a settled (completed/error/interrupted) session via DELETE /sessions/:id, which disposes any live handle before deleting the row (pipeline-link rows are kept — board-task provenance survives).

Cancel and discard are intentionally different: cancel stops work but preserves conversation/progress; discard removes the row entirely.

The list refreshes on any CE push event and falls back to polling GET /sessions while any session has a turn in flight.

Live working output, steering, and the Q&A surface

Turn execution is detached: POST /sessions, /answer, and /resume return as soon as the session row reflects the request, with the agent turn running in the background. Closing the flow does not cancel the server-side agent; use POST /sessions/:id/cancel (or the dashboard cancel icon button) to stop an in-flight turn while preserving the session as interrupted. While it runs:

  • The engine streams live progress through the seam's onProgress option (thinking/text deltas + tool start/end markers — a host capability any plugin can use). The orchestrator accumulates it per session and GET /sessions/:id attaches it as liveActivity, so the flow renders a live working pane (pulsing indicator, muted thinking, per-tool ✓/✗ lines).
  • The per-turn timeout is inactivity-based: a long but actively-working turn is never killed; only a turn with no progress for turnIntervalMs is interrupted (its working trace is preserved in the transcript).
  • On settle, the working trace is persisted into the conversation history as a condensed collapsible "Agent work" block — the transcript keeps the full story: opening message, every past question and answer (option ids rendered as labels), steering turns, working traces, and completion.

Steering: alongside any selectable question the user can type free-text guidance — attached to their answer as {value, comment}, or sent WITHOUT answering as {feedback}. The stage system prompt instructs the agent to treat both as first-class input (incorporate, adjust course, re-ask or proceed).

Transport

Session updates are pushed over the shared /api/events SSE stream. The orchestrator emits observable events via ctx.emitEvent (turn / question / completed / error / interrupted, plus throttled mid-turn progress); the host forwards them to connected clients as project-scoped plugin:custom events, and the view subscribes through the subscribePluginEvents context capability — refetching the session on each event (no raw EventSource; no deep dashboard import). Client polling of GET /sessions/:id remains as a fallback while a turn is mid-flight, so a missed event still converges. Session identity is project-scoped: the projectId used at start is threaded through every later answer/resume/poll so they resolve the same store and live handle.

Work → board bridge

When a stage reaches its work phase (ce-work, stage id work), its complete payload may carry a derived task list. The orchestrator creates each as a Fusion board task via ctx.taskStore.createTask, tagged CE-originated (source workflow_step with CE markers in sourceMetadata) and recorded as a pipeline-link row. The link row — not task-row JSON — is the authoritative back-reference from a board task to its originating pipeline/stage/artifact (per the FN-5719 pattern). Created tasks then run the normal lifecycle with no plugin interference. Zero derived tasks is a clean no-op.

Bidirectional sync model

Two separate state machines are kept in sync, never merged:

  • Board-task ownership → the task's column. The board is authoritative for task state.
  • CE-pipeline ownership → ce_pipeline_state.{currentStage, status}. The CE flow is authoritative for artifact/pipeline content.

Inbound (board → pipeline). The onTaskMoved / onTaskCompleted lifecycle hooks do the minimum under the 5s hook budget: resolve the link and enqueueSync(...), then return. Heavy advancement is not done inline.

Reconcile (the convergence guarantee). reconcileCePipelines(ctx) is a single on-demand sweep — not a tight interval poll. It (1) drains the queue and (2) independently re-derives transitions by comparing live board state against pipeline state. Step (2) is why a dropped or never-enqueued hook event still converges: the queue is an optimization; the board↔state comparison is the source of truth.

Outbound (pipeline → board). When a pipeline advances to a stage that produces board work, the reconciler creates the next-stage board task via ctx.taskStore.createTask and links it. lastArtifactPath is carried forward when Brainstorm advances to Plan, so the Plan board task continues the same requirements-only unified file and later enriches it rather than creating a second artifact.

Conflict policy. The reconciler only reads the already-terminal board task columns (board-authoritative) and only writes CE-owned fields plus a brand-new board task — the two writers never contend over the same cell.

Bundled-skills isolation model

The ce-* skills are bundled and pinned inside the plugin (src/skills/<skillId>/SKILL.md), declared via PluginSkillContribution with plugin-root-relative skillFiles. On load they are physically installed (cpSync, provenance-aware refresh) into a plugin-local, discoverable directory so an agent session can resolve them. The install is guarded to never touch a global ~/.claude/skills path an operator's own compound-engineering install owns — registering the bundled copy can never clobber a global install. Existing plugin-local installs with no marker or an older upstream marker are refreshed from the bundled source so enabled plugins actually run the pinned release after an upgrade; installs whose marker already matches are left alone.

Vendored from upstream

The bundled ce-* skills and src/agents/ce-*.md personas are reconciled from EveryInc/compound-engineering-plugin, pinned at compound-engineering-v3.15.0 (commit 2bbdbfb1d4287db95af407808b53266988ada974; tarball SHA-256 fce13e71bd709f8f572bf167c6af3753fc3fde0309c8f878498c78cb391c0b14, also recorded in src/upstream-provenance.ts). Refreshes are reconcile-not-overwrite: pull upstream prompt/reference improvements, but preserve Fusion-local adaptations such as stage separation, plugin-local install paths, Fusion await-input handling, and fn_spawn_agent persona dispatch.

Settings

Operator-facing settings render in Settings → Plugins → Compound Engineering, grouped as follows. Every setting has a real consumption point in the plugin.

Sessions

Setting Type Default Effect
Default Session Provider (defaultProvider) string (host default) Passed to the interactive-session factory as defaultProvider. Blank → host picks.
Default Session Model (defaultModelId) string (host default) Passed to the factory as defaultModelId. Blank → host picks.
Disabled Stages (disabledStages) string[] [] Explicit opt-out list. Registered stages launch by default, and the orchestrator rejects only IDs listed here.

Sync

Setting Type Default Effect
Reconcile on Board Changes (reconcileOnHooks) boolean true When on, the reconcile sweep auto-fires after task move/complete hooks. When off, the hook still enqueues so an on-demand sweep converges later.
Reconcile Cadence (minutes) (reconcileIntervalMinutes) number 15 Cadence hint for an on-demand refresh surface. Not a continuous poll loop.

Getters live in src/settings.ts (getDefaultProvider, getDefaultModelId, getDisabledStages, getEnabledStages, getReconcileOnHooks, getReconcileIntervalMinutes), each returning its default when the setting is absent. getEnabledStages remains a derived helper for the live registry minus explicit disabledStages opt-outs; stale persisted enabledStages snapshots are ignored.