# Migrate storage from SQLite to PostgreSQL — full dashboard cutover
Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.
## Status — every surface works in embedded-PG mode
Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).
| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |
## Approach
Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.
Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.
## Sync with main
The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.
## Residual Review Findings
Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).
- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.
~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.
---
## Update — 2026-07-12: production-readiness hardening & live acceptance
Everything below landed on this branch since the description above was
written:
**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).
**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.
**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.
**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.
**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
## Summary
Fixes ALL failing shards from the latest full-suite run (29225946428)
AND adds a structural gate check to prevent the recurring mock-export
drift pattern that has caused every full-suite failure across rounds
1–9.
## What broke (run 29225946428, commit 504b0f8b0)
| Shard | Root cause | Tests fixed |
|---|---|---|
| **3 (CLI)** | `workflowValidateParams` (FN-7911) missing from
`@fusion/engine` mock | 8 files |
| **3 (CLI)** | `skill-sync.test.ts` — `fn_workflow_validate` missing
from engine-tools.md | 1 file |
| **4 (dashboard)** | 6 chat default settings keys missing from
description allowlist | 1 file |
| **1+2 (engine)** | `additionalSkillPaths` missing from
`buildSessionSkillContext` mocks (FN-1510/1511) | 10 tests |
| **1+2 (engine)** | heartbeat FN-7878 changed paused→error for generic
run failures | 1 test |
| **1+2 (engine)** | executor `updateTask` exact-match →
`objectContaining` (new fields) | 2 tests |
| **1+2 (engine)** | `connectMcpSessionTools` mock missing for pi.test
MCP forwarding | 1 test |
## Structural fix — `scripts/check-mock-completeness.mjs` (the "fix for
good")
**New gate check** added to `pnpm test:gate`. Statically validates every
hardcoded `vi.mock("@fusion/dashboard")` and `vi.mock("@fusion/engine")`
factory covers all named imports the source file uses. Runs in <0.2s, no
module evaluation.
**How it works:**
1. Extracts named exports from each barrel
(`packages/dashboard/src/index.ts`, `packages/engine/src/index.ts`)
2. For each test file with a hardcoded `vi.mock` factory (no
`importOriginal`/`importActual` spread):
- Resolves source files the test covers (static + dynamic imports,
convention mapping)
- Extracts what those source files named-import from the barrel
- Resolves spread helpers (e.g. `...workflowAuthoringEngineMock`) by
reading the helper's exported keys
- Reports any barrel exports that are named-imported by source but
absent from the mock
**Why this fixes the recurring pattern:** Every round 1–9 failure was a
new barrel export imported by source but missing from a test mock. This
check catches it at gate time, before merge — not after the full-suite
fails on main.
Also completed all 15 latent mock gaps the guard found on first run (9
dashboard + 6 engine), including expanding the centralized
`workflowAuthoringEngineMock` helper with all `extension.ts` named
imports.
## Verification
- Gate (with new check): exit 0 ✅
- CLI: 355/355 passed ✅
- Engine (6 fixed files): 250/250 passed ✅
- i18n + settings: verified ✅
- Mock completeness guard: ✅ (0 issues)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Documented a new non-destructive workflow validation tool that
performs a dry-run and returns typed validation errors.
* **Tests**
* Updated and strengthened CLI, dashboard, extension, and engine tests
with more accurate mock exports and more resilient assertions.
* Adjusted expectations for session/heartbeat and retry-related
behaviors.
* **Chores**
* Added an automated mock-completeness gate and integrated it into the
test quality gate to keep mocks aligned with available platform exports.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
**Structural fix** for the recurring full-suite failure pattern where a
new `@fusion/dashboard` barrel export is imported by CLI source code but
missing from the hardcoded `vi.mock("@fusion/dashboard")` factory in CLI
tests.
## What's new
### Gate check script:
`scripts/check-cli-dashboard-mock-completeness.mjs`
Added to the merge gate (`pnpm test:gate`). Statically validates that
every hardcoded `vi.mock("@fusion/dashboard")` factory in CLI tests
includes all `@fusion/dashboard` exports that the corresponding source
files import.
- Pure static analysis (regex + depth-aware brace tracking) — no module
evaluation, <0.1s
- Handles named imports (`import { foo } from "@fusion/dashboard"`) AND
namespace imports (`import * as dashboard from "@fusion/dashboard"` →
scans `dashboard.X` usages)
- Filters against the real barrel exports to avoid false positives from
typos
- Resolves test→source mapping by parsing static/dynamic imports in the
test file (not just naming convention)
**Result:** the next time someone adds `export { newFunc } from
"./mod.js"` to `dashboard/src/index.ts` and `cli/src/commands/daemon.ts`
imports it, the gate catches the missing mock before merge instead of
the full-suite failing on main.
### Completed all 9 incomplete CLI dashboard mocks
Added the missing exports identified by the check:
| File | Missing exports added |
|---|---|
| `daemon.test.ts` | `registerGithubTrackingHook` |
| `serve.test.ts` | `registerGithubTrackingHook` |
| `dashboard.test.ts` | `AttachTicketStore`, `CliInputAttributionLog`,
`CliConfirmAdvanceRegistry`, `CliRelaunchRegistry`,
`registerGithubTrackingHook` |
| `task.test.ts` | `registerGithubTrackingHook`, `GitLabClient`,
`resolveGitlabAuth`, `buildGitLabTaskProvenance`,
`isGitLabAlreadyImported`, `buildGitLabTaskDescription` |
| `extension-*.test.ts` (×4) | `GitLabClient`, `resolveGitlabAuth`,
`buildGitLabTaskProvenance`, `isGitLabAlreadyImported`,
`buildGitLabTaskDescription` |
| `task-command-github-import-tracking.test.ts` | Same GitLab exports |
These were latent issues — the mocks were incomplete but tests passed
because the missing exports weren't called during test execution. Any
test change that exercises those code paths would have broken.
## Why not `importActual` spread?
Tried converting daemon.test.ts to `vi.mock("@fusion/dashboard", async
(importOriginal) => { ... })` — fails because the barrel's `export *
from "./plugins/index.js"` transitively imports
`@agentclientprotocol/sdk` which isn't available at test evaluation
time. The static check approach avoids this entirely.
## Verification
- `pnpm test:gate`: exit 0 (includes new check)
- `pnpm lint`: exit 0
- CLI tests: daemon 21/21, serve 58/58, dashboard 91/91, task 149/149 ✅
- Gate script: `✅ CLI dashboard mock completeness: all hardcoded mocks
cover source imports.`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added automated validation to ensure CLI test mocks remain aligned
with available dashboard functionality.
- Updated test coverage setup so GitHub, GitLab, daemon, dashboard,
server, and task scenarios use complete dashboard mocks.
- Test verification now reports missing mocked functionality and blocks
the release gate when inconsistencies are detected.
- **Chores**
- Improved reliability and maintainability of automated verification for
CLI and dashboard integrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add a report-only script that surfaces flaky-test quarantine entries approaching their 14-day deletion clock, so maintainers can make deliberate rescue-or-expire decisions before entries silently expire.
- Add scripts/check-quarantine-ledger.mjs: reads scripts/lib/test-quarantine.json, computes days-remaining against the existing 14-day deletion clock (shared DELETION_CLOCK_DAYS from scripts/test-velocity-baseline.mjs), and buckets each entry as expired/near/healthy/unknown
- Support --warn-within=<days> (default 5) to tune the near-deadline window, --json for machine-readable output, and --strict as an opt-in local/CI gate (exits 1 on expired/near entries) while default mode stays exit-0 and non-blocking
- Wire pnpm check:quarantine-ledger script in package.json
- Add scripts/__tests__/check-quarantine-ledger.test.mjs covering deadline bucketing/sorting, empty/missing ledger handling, --strict behavior, and --json output shape
- Document the new command and its flags in docs/testing.md under the quarantine ledger/deletion ratchet section
Files changed:
docs/testing.md | 10 +
package.json | 1 +
scripts/__tests__/check-quarantine-ledger.test.mjs | 159 ++++++++++++++++
scripts/check-quarantine-ledger.mjs | 202 +++++++++++++++++++++
4 files changed, 372 insertions(+)
Fusion-Task-Id: FN-7912
Fusion-Task-Lineage: c08e2e09-473a-4ad0-8c27-43cbc3355168
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
- pnpm dev / new pnpm start default to the dashboard command
- fn dashboard (and bare fn/fusion/npx, incl. packaged binaries) now runs
supervised by default via an attached foreground child (TUI-safe);
--no-supervise opts out; FUSION_RESTART_EXIT_CODE=86 = intentional restart
- New /api/system routes: info, restart, rebuild jobs with SSE output,
engine restart, agents restart-all, plugins reload-all, log tail
- System tab: rebuild & restart (source checkouts only, hidden elsewhere),
restart server/engine/agents, backup DB, live server logs, copy
diagnostics, report bug; new Plugins tab reusing PluginManager
- Desktop restart via Electron app.relaunch(); DashboardLogSink now keeps a
bounded history + listener feed for the log viewer
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prototype and land a rebuilt-every-run esbuild bundle of the @fusion/core gate-safe barrel closure, collapsing the engine-core gate's per-fork Vite SSR import-phase cost (18 forks x ~430-file closure re-resolved from scratch) into a single file load per fork.
- Add scripts/build-engine-core-gate-bundle.mjs: esbuild-bundles packages/core/src/index.gate.ts (220 first-party files, packages:"external" so third-party/node: imports stay external, treeShaking:false to preserve side effects) into packages/core/.gate-bundle/core.mjs + core.meta.json
- Wire the builder into packages/engine/vitest.config.ts's engine-core project globalSetup (alongside the existing vitest-teardown hook) so the bundle is rebuilt fresh before every gate invocation, and repoint the @fusion/core resolve.alias at the bundled output instead of index.gate.ts source
- Place the bundle output at packages/core/.gate-bundle/ as a sibling of packages/core/node_modules/ (not nested inside it) to avoid Vite SSR's external-dep heuristic, which would otherwise silently defeat vi.mock interception for imports nested in the bundle
- Gitignore packages/core/.gate-bundle/ and add a matching ESLint ignore entry so the generated bundle text is never linted or committed
- Add esbuild ^0.25.12 as a root devDependency (pnpm-lock.yaml updated accordingly)
- Document the pre-bundling rationale, placement constraints, and measured A/B wall-time results in docs/testing.md
Verified: pnpm test:gate passes (335/335 engine-core tests, 63/63 CLI ci-shape tests), engine package typecheck clean, eslint clean on touched files.
Files changed:
.gitignore | 11 ++
docs/testing.md | 3 +
eslint.config.mjs | 10 ++
package.json | 1 +
packages/engine/vitest.config.ts | 50 ++++++++-
pnpm-lock.yaml | 3 +
scripts/build-engine-core-gate-bundle.mjs | 174 ++++++++++++++++++++++++++++++
7 files changed, 247 insertions(+), 5 deletions(-)
Fusion-Task-Id: FN-7669
Fusion-Task-Lineage: 62b06b2a-4ac6-45ae-ac79-9771132bc303
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
The Windows desktop installer failed to package: electron-builder's production-dependency walk rejected
`@aws-sdk/core@3.974.26` because `@aws-sdk/credential-provider-env` (resolved in the `--legacy` deploy
closure) requires `^3.974.27`. Root cause: an incidental `pnpm.overrides` entry pinning
`@aws-sdk/core` to the exact version `3.974.26` (added without rationale in an unrelated commit) which
force-held core below what its consumers now demand — the classic stale-exact-pin trap.
Fixes / prevention:
- Remove the `@aws-sdk/core` override so the deploy closure resolves core to 3.974.27 (satisfies all
consumers). The main lockfile still resolves core to 3.974.26 for its own consistent graph, so the
published @runfusion/fusion closure is unchanged (no changeset needed). Verified locally: a fresh
`@fusion/desktop build` + `electron-builder --dir` now passes the dependency walk with no manual patch.
- Add an advisory, path-gated `Desktop packaging` job to pr-checks.yml that reproduces electron-builder's
production-dependency walk (`--dir`, no NSIS/signing) plus a `pnpm dedupe --check` early-warning. This
is the only check that validates the packageable closure, which previously ran only in release/manual
workflows — so any future dependency skew now fails at PR time, for ANY dependency, instead of at
release/local-build time. Kept OUT of the required set so the thin merge gate [Lint, Typecheck, Build,
Gate] and branch protection are untouched; promote to blocking by adding it to required checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds scripts/verify-fast.mjs + root `pnpm verify:fast`, an opt-in, flake-free
verification path that runs typecheck + build scoped to the changed packages
(reusing test-changed.mjs git-diff / changed-package resolution) plus the
existing boot smoke once, with no test suite. Each step is bounded by the
shared runWithWatchdog (class "changed"); exits nonzero on the first failure.
No default changed: pnpm test, the merge gate, and CI are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chain distillation into release:version so both local and CI versioning
flows get distilled notes. Update release.yml to use curated CHANGELOG
notes instead of GitHub's auto-generated release notes.
Add a pretest guard that caps new source files at 2,000 lines to stop
god-files from being born, following the existing check-no-* guard pattern.
Existing oversized files (106 of them) are grandfathered via a ratchet
baseline (scripts/line-count-baseline.json): each is pinned to its current
line count and may shrink but never grow. Files refactored under the cap
drop out of the baseline and cannot regress. Generated, lock, locale, and
.d.ts files are out of scope via the source-extension filter.
Wired into pretest and pretest:full; covered by 11 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a weekly baseline workflow for tracking test feedback-loop speed and flake pressure.
- Add a stdlib-only script to record gate and pnpm test timings alongside slowest test files and quarantine counts.
- Publish generated markdown and JSON baseline artifacts for #leads reporting.
- Document the weekly refresh process and add package scripts plus coverage for the baseline helper.
Files changed:
docs/test-feedback-loop-baseline.md | 48 ++++++
docs/test-feedback-loop-baselines.json | 122 ++++++++++++++
docs/testing.md | 12 ++
package.json | 1 +
scripts/__tests__/test-feedback-baseline.test.mjs | 90 ++++++++++
scripts/test-feedback-baseline.mjs | 192 ++++++++++++++++++++++
6 files changed, 465 insertions(+)
Fusion-Task-Id: FN-6612
Fusion-Task-Lineage: 72bdc070-50e3-4e6b-a07d-3b8aeb9c2e92
- Add scripts/mobile-run-android.sh + pnpm mobile:run:android (auto-detects
Android SDK + JDK 21, writes local.properties, reconnects network ADB
before deploy, supports remote backend via FUSION_SERVER_URL).
- Header.css: reserve env(safe-area-inset-top) so top app chrome no longer
draws under the OS status bar on edge-to-edge native shells (Capacitor
Android API 35+, iOS notch, PWA standalone). No-op on web/desktop.
- .gitignore: ignore generated packages/mobile/{ios,android} (stale paths
pointed at packages/dashboard/).
Add a fast guard that rejects Vitest timeout bumps in tracked test files.
- Add a test-timeout appeasement scanner with a temporary allowlist for legacy exemptions.
- Run the scanner in pretest, pretest:full, and test:gate so merge gates catch timeout bumps.
- Cover the scanner behavior with node:test cases and document the policy/remediation path.
Files changed:
docs/testing.md | 6 ++
package.json | 6 +-
.../check-no-test-timeout-appeasement.test.mjs | 49 +++++++++
scripts/check-no-test-timeout-appeasement.mjs | 119 +++++++++++++++++++++
.../lib/test-timeout-appeasement-allowlist.json | 10 ++
5 files changed, 187 insertions(+), 3 deletions(-)
Fusion-Task-Id: FN-6434
Fusion-Task-Lineage: deb46a27-b0c9-4644-b8bb-34ce98e7acde
