# Migrate storage from SQLite to PostgreSQL — full dashboard cutover
Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.
## Status — every surface works in embedded-PG mode
Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).
| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |
## Approach
Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.
Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.
## Sync with main
The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.
## Residual Review Findings
Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).
- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.
~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.
---
## Update — 2026-07-12: production-readiness hardening & live acceptance
Everything below landed on this branch since the description above was
written:
**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).
**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.
**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.
**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.
**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
Rename the settings navigation entry so Remote Access is no longer conflated with Node Sync.
- Update the remote settings nav label to "Remote Access" while keeping the separate Node Sync section intact.
- Add coverage that rejects the old combined label and confirms both standalone entries render.
- Align the workflow settings plan and changeset with the clarified settings IA.
Files changed:
.changeset/fn-7062-remote-access-rename.md | 7 +++++++
...26-06-04-002-feat-workflow-settings-mechanism-plan.md | 2 +-
packages/dashboard/app/components/SettingsModal.tsx | 6 +++++-
.../__tests__/SettingsModal.scheduling-merge.test.tsx | 16 ++++++++++++++++
4 files changed, 29 insertions(+), 2 deletions(-)
Fusion-Task-Id: FN-7062
Fusion-Task-Lineage: d50d47eb-aa15-4bf6-8f26-edaa79c8de9e
Six-reviewer doc review reshaped two foundational decisions:
- Results model: graph writes the existing task.workflowStepResults field
keyed by node id (no new table) — avoids upgrade data-loss for in-review
tasks and collapses U2/U3. Sibling-table spike reverted.
- Migration: reconcile with the existing migration-109 fragment scheme
instead of infeasible node-injection into read-only built-in workflows.
Also folds in: store-fallback fail-closed (legacy execute calls not all dead),
watchdog re-entry contract, plugin-row migration from persisted fields,
full progress-bar render-state spec, and circular-dependency fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
optionalGroupIdSet falls back to builtin:coding (mirroring the executor's
unselected-task resolution) so a toggled built-in group id like
browser-verification is no longer downgraded to a legacy WS-xxx step row the
graph executor never matches. Create-time optional-step controls resolve
builtin:coding when no project default workflow is set so the toggles render.
First unit of the graph-native workflow-step refactor (see
docs/plans/2026-06-25-001-refactor-workflow-steps-graph-native-plan.md).
Fusion-Task-Id: FN-7039
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix 6 failing tests caused by intentional source changes that landed
without updating dependent test assertions:
- Core test-project: taskPrefix default changed from "FN" to undefined
(commit 800f845e1, derived from project name at runtime)
- Dashboard ScriptsModal.css: replace banned --text-primary with --text
- CLI package-config: update expected pi dep version ^0.79.1 -> ^0.79.9
- CLI skill-sync: document 4 new engine tools in engine-tools.md
- CLI version: update expected release:version script to include
run-ci-distill.mjs
- CLI bundled-plugin-freshness: rebuild stale dist directories
## Summary
Replaces dense, agent-authored technical changeset paragraphs with a
**structured, concise changeset schema** (end-user summary + category +
optional dev detail), enforced by a linter. Adds a **deterministic
distillation step** at version time that transforms a release's
collected changesets into clean, grouped, end-user-facing release notes.
Unifies both release paths (local `release.mjs` and CI `version.yml` /
`release.yml`) behind a single distilled artifact so the root
`CHANGELOG.md` and GitHub Release both carry the same user-facing notes.
## Changes
### Changeset format (`scripts/lib/changeset-schema.mjs`)
- Each changeset body now uses labeled fields: `summary` (required,
user-facing, max 120 chars), `category` (required:
feature/fix/breaking/security/performance/internal), `dev` (optional
developer detail)
- Legacy freeform changesets are detected and flagged for the transition
period
### Linter (`scripts/check-changeset-format.mjs`)
- Validates structured schema, summary length, category enum, and
frontmatter package scope
- Wired into `test:gate`, `pretest`, `pretest:full`, and `pr-checks.yml`
- Legacy changesets warn (exit 0) during transition; `--strict` flag
fails on them
### Distillation (`scripts/lib/distill-release-notes.mjs`)
- `distillDeterministic` builds grouped, end-user-facing release notes
by category (New, Fixed, Breaking, Security, Performance, Internal)
- `buildDistillationPrompt` and `DISTILLATION_SYSTEM_PROMPT` ready for
AI distillation via `createFnAgent` when model credentials are available
- Graceful fallback: deterministic bullet list when no model is
configured
### Release integration
- **Local path** (`scripts/release.mjs`): captures changeset entries
before `changeset version` deletes them, distills notes post-version,
replaces the version's CHANGELOG section with curated notes
- **CI path** (`scripts/ci-distill-release-notes.mjs`,
`scripts/run-ci-distill.mjs`): chained into `release:version` so both
flows get distilled notes
- **GitHub Release** (`release.yml`): uses curated CHANGELOG notes
instead of `generate_release_notes: true`
### Documentation
- `AGENTS.md`, `RELEASING.md`, `docs/contributing.md` updated with the
structured format guide
- `.changeset/README.md` template for `pnpm changeset` consumers
## Testing
- 54 new unit tests across changeset-schema, check-changeset-format,
distill-release-notes, and extract-version-notes
- `pnpm lint` clean
- `pnpm test:gate` green (371 tests)
- `pnpm check:changesets` passes (13 legacy warnings expected during
transition)
## Plan
Full implementation plan:
`docs/plans/2026-06-24-001-feat-better-changelog-plan.md`
<!-- stage-review-badge-begin -->
---
<a href="https://stagereview.app/Runfusion/Fusion/pull/1750">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://stagereview.app/assets/gh-open-in-stage-dark.svg">
<img src="https://stagereview.app/assets/gh-open-in-stage-light.svg"
alt="Open in Stage">
</picture>
</a>
<!-- stage-review-badge-end -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added structured release-note guidance and validation for changeset
entries.
* Release notes now use curated, grouped changelog content instead of
auto-generated text.
* Added a fallback process to keep release notes consistent when
automated distillation isn’t available.
* **Bug Fixes**
* Improved handling of legacy changesets and malformed entries.
* Ensured version-specific changelog sections are updated without
affecting older releases.
* **Tests**
* Added coverage for changeset validation, note grouping, and changelog
section replacement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Update AGENTS.md, RELEASING.md, contributing.md with the structured
changeset format guide. Add .changeset/README.md template. Add changeset
for this change. Update distill-release-notes.mjs with final version.
Add the reviewed, completed plan for the dashboard App.tsx module-breakup
refactor (planning + two doc-review rounds). Status: completed — all 8
implementation units shipped on this branch.
- base-commit-capture: POSIX single-quote integration branch refs instead of
JSON.stringify (double quotes are subject to $-expansion in the shell)
- executor: add per-repo no_commits guard to the workspace verifyWorktreeInvariants
branch (parity with the singular path), gated by the same task-wide no-commit
eligibility
- executor: reviewWorkspacePerRepo failure message now states the per-repo verdict
list is partial (evaluation stops at first failure)
- worktree-acquisition: defensively wrap non-fatal/outer-catch logEntry/audit so a
logging throw cannot promote a non-fatal error to fatal or mask the original error
- docs/plans: add code-fence language tags and fix MD028 blank-line-in-blockquote
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Reject failure-condition edges inside optional-group templates (the single-pass
walk surfaces template failures as the group's outcome, so an internal failure
edge was silently dead) — Greptile P2.
- flowToIr: a container/group node (foreach/loop/optional-group) is v2-only — its
presence now forces v2 serialization (an inserted optional-group on a plain
workflow no longer serializes as invalid v1) — CodeRabbit.
- Disabled optional-group bypass routes a plain success with no distinguishing
value, so an outcome:* edge can't preempt success routing (inertness) — CodeRabbit.
- Downgrade heuristic: presence of a legacy optionalSteps key (incl. []) keeps v2.
- Resolver docblock corrected (config-less groups resolve to a fallback entry).
- Strengthen tests: assert both inserted groups + v2 round-trip; failure-edge
rejection case.
- Changeset: bump to major (removed exported WorkflowOptionalStep type).
- Plan: record U7a as delivered in this cohort; only the workflow-step seam
infra removal remains deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the minor changeset for the optional-group feature (U1-U6) and document in
the plan why the full legacy optional-steps retirement (U7) is deferred: the
workflow-step seam is shared engine infrastructure across ~9 files and the
dashboard still carries the prior declaration authoring surface (~10 files), so
removing the core type without that cleanup breaks the build. The new model is
the live path; the legacy declaration surface is inert pending a focused follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make runAiMerge (the FN-5633 clean-room AI merge path, already the default)
the sole merge path; soft-deprecate the legacy aiMergeTask pipeline. Phase 0
of the workspace-mode master plan — a standalone merge-consolidation refactor
that lands first so downstream workspace work targets one merge function.
U1 — collapse the dispatch: project-engine.ts now calls runAiMerge
unconditionally; the two direct callers that bypassed the dispatch
(dashboard.ts onMergeImpl in --no-engine mode, task.ts runTaskMerge /
`fn task merge`) now route to runAiMerge too. Export runAiMerge from
@fusion/engine.
U2 — soft-deprecate: aiMergeTask is @deprecated (body retained for a later
deletion pass; shared helpers runAiMerge imports, e.g.
captureSingleCommitLandedMetadata, left intact). merger.mode "deterministic"
is annotated deprecated and made inert (type + field kept — published
@runfusion/fusion surface); the dispatch logs a one-time deprecation warning
and routes to runAiMerge. Changeset added (minor).
U3 — R7 workspace merge-boundary guard: shared @fusion/core predicate
assertNotWorkspaceTaskMerge(task) rejects tasks with populated
workspaceWorktrees at all four merge entry points (dispatch, store.mergeTask,
onMergeImpl, runTaskMerge) with an error naming master-plan U6. Covers the
window until per-repo merge support lands; U6 removes it.
U4 — deterministic-mode blast-radius audit: no production project, CI config,
or seeded setting pins merger.mode "deterministic"; only four engine tests
used it to drive the dispatch to aiMergeTask as a mockable seam — migrated to
mock runAiMerge instead. Other module-level aiMergeTask mocks were dead under
the default "ai" mode or test aiMergeTask directly (body retained), so they
are unaffected.
Also removes an unused acquireWorkspaceRepoWorktree import inherited from the
foundation branch (executor.ts) that was failing lint; master-plan U1 re-adds
it with its per-repo usage.
Merge gate green: lint, typecheck (29 projects), build, test:gate
(649 + 58), plus the migrated (114) and new predicate (4) tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce the `optional-group` node kind and `WorkflowOptionalGroupConfig`
(defaultOn + single-pass template subgraph), validated by mirroring the loop
template rules: non-empty template, single entry/exit, no nested groups, no
rework edges (single-pass guarantee), no template/top-level id collisions,
boolean defaultOn. Foundation for graph-native optional steps; additive and
inert until producers/consumers land in later units. Includes the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts from main's analytics schema additions (plugin
activations, per-model token buckets) against the PR's contract-assertion
type column:
- db.ts: renumber behavioral-verification migration 124 -> 126, bump
SCHEMA_VERSION to 126 so it follows main's migrations 124/125
- core/roadmap tests: adopt main's SCHEMA_VERSION-constant assertions
instead of stale literal 124
- test-quarantine.json: keep all four quarantine entries from both sides
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stepwise-coding IR had no workflow-step seam node, so a per-task
enabledWorkflowSteps entry (e.g. browser verification) would never execute.
Add the seam node on the success path (steps -> workflow-step -> review,
once post-foreach) and declare browser-verification as an optional step,
matching the coding workflow. Covers the dead-toggle gap with resolver and
engine execution-divergence tests.
