# Migrate storage from SQLite to PostgreSQL — full dashboard cutover
Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.
## Status — every surface works in embedded-PG mode
Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).
| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |
## Approach
Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.
Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.
## Sync with main
The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.
## Residual Review Findings
Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).
- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.
~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.
---
## Update — 2026-07-12: production-readiness hardening & live acceptance
Everything below landed on this branch since the description above was
written:
**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).
**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.
**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.
**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.
**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
The desktop-pack job broke the CI-shape invariant (ci-workflow.test.ts) that
pr-checks.yml contains exactly [build, gate, lint, typecheck] — the gate's
job set maps 1:1 to branch-protection required checks. Extract the advisory
desktop-packaging validation into desktop-packaging.yml, still PR-triggered
and non-required, so the thin gate stays pure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Windows desktop installer failed to package: electron-builder's production-dependency walk rejected
`@aws-sdk/core@3.974.26` because `@aws-sdk/credential-provider-env` (resolved in the `--legacy` deploy
closure) requires `^3.974.27`. Root cause: an incidental `pnpm.overrides` entry pinning
`@aws-sdk/core` to the exact version `3.974.26` (added without rationale in an unrelated commit) which
force-held core below what its consumers now demand — the classic stale-exact-pin trap.
Fixes / prevention:
- Remove the `@aws-sdk/core` override so the deploy closure resolves core to 3.974.27 (satisfies all
consumers). The main lockfile still resolves core to 3.974.26 for its own consistent graph, so the
published @runfusion/fusion closure is unchanged (no changeset needed). Verified locally: a fresh
`@fusion/desktop build` + `electron-builder --dir` now passes the dependency walk with no manual patch.
- Add an advisory, path-gated `Desktop packaging` job to pr-checks.yml that reproduces electron-builder's
production-dependency walk (`--dir`, no NSIS/signing) plus a `pnpm dedupe --check` early-warning. This
is the only check that validates the packageable closure, which previously ran only in release/manual
workflows — so any future dependency skew now fails at PR time, for ANY dependency, instead of at
release/local-build time. Kept OUT of the required set so the thin merge gate [Lint, Typecheck, Build,
Gate] and branch protection are untouched; promote to blocking by adding it to required checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- boot-smoke: shutdown verdict now requires SIGTERM actually delivered and a
clean exit (code 0 or SIGTERM); EADDRINUSE port race retries with a fresh
port (3 attempts)
- test-changed: rename shouldForceFullSuite -> isSharedInfraChange (it routes
to gate mode, not full); run the changed-mode gate under the isolation guard
- workflows: least-privilege permissions (contents: read) on pr-checks and
full-suite
- pr-checks.yml: gate job = boot smoke + pnpm test:gate; shards/slow/inventory removed
- full-suite.yml (new): 4-way shards, engine slow tier, inventory guard on push to main
- ci.yml deleted (trigger-disabled since FN-1541; was dead config)
- ci-workflow.test.ts rewritten to pin the new gate shape; release/signing blocks preserved
- docs/contributing.md: pnpm test:gate is the merge gate; verify:workspace repositioned as deep opt-in
BRANCH-PROTECTION CUTOVER: set required checks to exactly [Lint, Typecheck, Build, Gate] after merging; open PRs must rebase.
Every shard + the curated-guard job paid ~71s rebuilding 8 packages' dist
from scratch. actions/cache now restores dist on exact content-hash match
(--print-source-hash; branch-switch stable, pure git-based), with a
--seed-artifact-cache step on cache-hit that defeats the mtime trap
(restored dist looks older than checkout-time src mtimes). No restore-keys
partial fallback: stale dist is a known failure mode here. node_modules is
never cached (Windows junction policy). ensure-test-artifacts still runs as
the authority and rebuilds anything genuinely missing or changed.
- ci-test-shard: timing outputFile is now RELATIVE — one pnpm invocation
fans out to several packages whose vitests all received the same
absolute path, so every package overwrote the same timings file (last
writer wins). Each package now writes <pkgDir>/.timings/; discovery
(discoverWorkspaceTimingFiles) and the CI artifact globs scan the tree
- acp event-bridge-bounds: 20s timeout on the CPU-bound plan-flood test
(timed out at default 5s under loaded CI shard, passes in isolation)
- acp process-manager: port-4040-allowlist marker for its doc comments
(main-side; local guard flagged it after merging main)
- 427 orphaned dashboard test files ran in NO gate; 395 now gated via self-maintaining backfill lanes (glob minus curated minus skip-list), 31 pre-existing failures + build-output skip-listed with reasons
- settings -t name-filter lanes replaced by one unfiltered lane (describe blocks can no longer fall through filters)
- scripts/check-test-inventory.mjs: --capture/--diff superset harness + --dashboard-curated completeness guard
- pr-checks.yml: engine-slow CI gate (non-empty assertion) + inventory guard job
- docs/testing.md: guard, skip-list policy, harness usage
Test shards 3 and 4 were silently failing on every open PR because vitest's
CLI parser was treating `--shard X/Y` as positional file filters whenever the
arg arrived after a `--` separator. Removing the `--` in ci-test-shard.mjs
restores per-shard slicing; verified locally that shard 1/4 and 2/4 now run
distinct subsets.
The two consistently-failing engine tests:
1. self-healing in-review-branch-rebind ambiguous case-variant detection:
dedup keyed on lowercase branch name collapsed two physically distinct
refs (allowed on Linux ext4) into one candidate, so the "applied" path
ran instead of "ambiguous-candidates". Dedup now keys on the resolved
SHA — macOS APFS still collapses (same ref, same SHA), Linux keeps both
(distinct SHAs) and the ambiguity skip path fires as designed.
2. worktree-acquisition resume-misbinding spy: the production
verifyResumeBranchNotMisbound returns early when `git merge-base HEAD main`
fails, which is exactly what happens on shallow checkouts. Bumping the
test-shards checkout to fetch-depth: 0 makes CI mirror the local git
state these engine tests rely on.
Also adds `push: branches: [main]` to PR Checks so regressions like this
(which slipped into v0.33.0 with no post-merge run) go red immediately
on landing instead of being discovered on the next PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a test block covering authentication and ownership validation for node-settings sync routes, ensuring the API enforces proper authorization before settings or credentials are exchanged between nodes.
Fusion-Task-Id: FN-4862
- Remove the Bun install step from the deterministic test shard job in .github/workflows/ci.yml
- Remove the matching Bun install step from the PR checks shard job in .github/workflows/pr-checks.yml
- Keep shard test execution on the shared setup-node-pnpm action only
Fusion-Task-Id: FN-4105
FN-4093 introduced a reusable `setup-node-pnpm` composite GitHub Action and refactored the PR checks workflow to use it, reducing the workflow file by over 60 lines while adding a contract test for the action. A small follow-up stabilizes workspace quality gates by adding an export in the core store
Fusion-Task-Id: FN-4093
Removes the PR lint pre-build step from the GitHub Actions workflow and adds a new test suite (`ci-workflow.test.ts`) covering CI workflow behavior, with a small documentation update.
Fusion-Task-Id: FN-3986
The test clicked 'Medium' and 'Continue' synchronously without waiting
for the scope question options to fully render. Use findByText/findByRole
to wait for elements to appear before interacting, and add a timeout to
the final waitFor for the second question.
- Add `pnpm build` step to test-shards job so plugin dist/ and core
packages are compiled before tests run (fixes plugin-loader test)
- Add cli-alias/index.js to ESLint node scripts pattern so Node globals
(process, AbortController, fetch) are recognized
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merges origin/main to bring in the droid runtime plugin (needed by
plugin-loader test) and all recent changes. Adds a "Build plugins"
step to pr-checks.yml so plugin dist/ directories are compiled before
tests run — fixes ERR_MODULE_NOT_FOUND for hermes and droid plugins.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This merge refactors the test-changed script and adds new CLI tests (FN-3234), keeping automation on the full-suite path. It includes a new test file for the root test command, updates to the CI workflow, and documentation improvements.
Fusion-Task-Id: FN-3234
- Convert internal @fusion workspace aliases in packages/cli/vitest.config.ts to exact anchored regex entrypoint mappings
- Preserve subpath-before-root alias order so @fusion/core/gh-cli and @fusion/dashboard/planning resolve correctly
- Add vitest-workspace-resolution regression coverage for alias definitions and ordering
- Simulate clean worktrees by temporarily hiding internal dist/ directories and verify dynamic imports resolve from source
