2e4fcfcaea09b9dcbccd4443e2f67eb6fed04487
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
gsxdsm
|
2e4fcfcaea |
fix(FN-7952): establish PostgreSQL core authority (#2108)
## Summary Fusion’s core runtime now treats PostgreSQL as the authoritative metadata store without leaving current CLI, dashboard, desktop, or engine composition roots uncompilable between stack layers. This is the 99-file foundation for the larger cutover: subsequent PRs migrate the remaining consumers, plugins, and operator surfaces. ## Design decisions - Runtime store construction fails closed when an asynchronous PostgreSQL layer is unavailable; SQLite remains readable only at explicit migration and identity-recovery boundaries. - Project ownership is enforced across active, archived, workflow, mission, analytics, and plugin-schema data. - The small set of cross-package files in this layer are compatibility-critical call sites required for a green intermediate commit, not the complete consumer migration. - Schema migration 0008 remains assigned to session-advisor state from current `main`; mission lineage idempotency advances to 0009 so neither invariant can be skipped. ## Validation - All affected package typechecks pass: Core, Engine, Dashboard, CLI, and Desktop. - `pnpm test:gate` passes: 478 tests across the engine gate, PostgreSQL core gate, and CLI workflow shape. - The PR changes exactly 99 files. ## Stack This is the base PR. Engine/dashboard, CLI/desktop/ops, plugins, and docs/release follow as stacked PRs, each below 100 changed files. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the standard runtime backend, with embedded PostgreSQL enabled by default. * Added project-scoped storage for tasks, archives, chat sessions, missions, knowledge pages, and operational data. * Improved archived-task search, filtering, pagination, and restoration. * Added safer plugin schema initialization with validation and project isolation. * Added PostgreSQL-backed workflow, mission, validator, and dashboard capabilities. * **Bug Fixes** * Improved startup timeout cancellation and resource cleanup. * Prevented cross-project data access and phantom reservation cleanup errors. * Ensured archived tasks remain read-only and asynchronous writes complete reliably. * Retired SQLite opt-out settings with clear startup errors. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
gsxdsm
|
c15c78feeb |
feat: migrate storage from SQLite to PostgreSQL (#1793)
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover Migrates Fusion's storage layer to the embedded PostgreSQL `AsyncDataLayer` (the default backend) and **completes the satellite-store + feature cutover** so every dashboard and Command Center surface works in PG mode. ## Status — every surface works in embedded-PG mode Verified live against a running embedded-Postgres dashboard (all **200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate; core/engine/cli/dashboard typecheck clean). | Area | Surfaces | State | |---|---|---| | Satellite stores | workflows, todos, insights, research, missions, goals, mailbox | ✅ | | Views | artifacts, documents, evals | ✅ | | Command Center | activity, productivity, team, tokens, tools, **workflows**, **github**, **signals**, **plugin-activations**, **live** (all 10) | ✅ | | Run execution | insight generation, research run execution | ✅ (store-path; AI step needs a provider) | | Live updates | SSE push for mission/research/insight events | ✅ | | Workflow editing | create / update / delete / select (+ id counter) | ✅ | | Engine | mission autopilot, incident-signal ingestion, regression storm-guard, agent wake-on-message | ✅ | | Core | tasks, agents, secrets, automations, memory, chat, usage, PRs, git | ✅ | ## Approach Each satellite store gets an `Async<Store>` wrapper exposing the sync store's method names over the existing `async-*-store.ts` helpers; `get<Store>Store()` returns a `Sync | Async` union; consumers `await` (harmless on sync), and engine/CLI paths that can't convert use `instanceof Sync` graceful fallback. Analytics aggregators branch on `"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*` (snake_case) in PG. Executors/orchestrators/autopilot are await-converted to drive the union store; the async store wrappers extend `EventEmitter` so SSE live-push fires in both backends. Not-yet-ported capabilities degrade gracefully (never 500) and are individually called out in commits. ## Sync with main The branch is kept continuously merged with `main` (currently through FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer applies. Use **Create a merge commit** (or squash) to land it — GitHub's rebase-merge cannot replay a merge-maintained branch. ## Residual Review Findings Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5) applied 3 safe fixes (see `fix(review): apply autofix feedback`). The following are **real but gated** — recorded here as follow-up work rather than auto-applied. All are SQLite→PostgreSQL **concurrency/atomicity regressions**: the sync stores were immune only by SQLite's single-writer, single-threaded-handler execution; the async ports open multi-await read-modify-write windows. **Reachability is low today** because the execution engines that generate concurrent same-run mutations (insight run executor, research orchestrator/dispatcher) are `instanceof`-gated to sync mode in PG. No process-crash class survived (all engine fallbacks correctly guard the sync store). - **[P1] Research `appendResearchEvent` dual-write is non-atomic** (`packages/core/src/async-research-store.ts`, corroborated: adversarial + reliability). The `research_run_events` insert (own transaction) and the `run.events` jsonb update are separate writes — a crash between them, or two concurrent appends, splits the table count from the jsonb array. **Fix:** perform the seq-insert and the jsonb update in one `layer.transactionImmediate`. - **[P1] Research run terminal-reversion via stale full-row persist** (`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`). Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert a terminal run to `running` by overwriting the whole row, bypassing the transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status …` guard, or optimistic version column. - **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU** — concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:** `SELECT … FOR UPDATE` / enclosing transaction. - **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race** (`async-insight-store.ts`) — two callers can each create an "active" run. **Fix:** partial unique index on `(projectId, trigger) WHERE status IN ('pending','running')`. - **[P3] `createResearchRetryRun` return-value divergence** — sync returns the pre-update `queued` snapshot; async returns the reloaded `retry_waiting` run (persisted state is identical). Pick one side for cross-backend parity. - **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1 fan-out** — O(milestones×slices) sequential round-trips hold one pool slot per request; can starve the pool for large hierarchies. **Fix:** batched/joined reads. - **Testing gaps:** no PG-mode concurrency tests (interleaved status/event mutations), no sync↔async parity assertion for the lifecycle-error codes, and no mission status/health rollup parity test vs the sync `MissionStore`. ~~Out of scope (deferred): AI run *execution* (insight/research) + mission autopilot + live SSE mission events remain sync-gated/degraded in PG mode.~~ **Since ported** — insight/research run execution, mission autopilot, and SSE live push all run on the async layer now, which also makes the concurrency findings above genuinely reachable; they remain open follow-ups. --- ## Update — 2026-07-12: production-readiness hardening & live acceptance Everything below landed on this branch since the description above was written: **Production blockers from review — fixed** - `recoverStaleTransitionPending` ported to the async layer (backend moves write + clear the crash-safe marker; startup/maintenance sweeps no longer throw). - Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write changed columns only (full-row upserts silently resurrected stale fields across concurrent store instances — the "task stuck unplanned forever" bug). - First-boot **auto-migration**: booting the PG backend over a project with a legacy `fusion.db` migrates it automatically (loud failure, SQLite kept as backup), and the dashboard shows a one-time **"your data was migrated" banner** with the backup paths and a Need-help Discord link. - `pg_dump`/`pg_restore` discovered from common install locations for embedded-mode backups. - The PG suite is part of the blocking merge gate (`test:pg-gate`). **Multi-project isolation (PR #2007, merged into this branch)** - `project_id` partition key on tasks / archived tasks / config, `taskProjectScope` threaded through every scan/claim/count, per-project config rows, layer bound to the project at startup. - Review P1 follow-up: the shared cold-storage `archive.archived_tasks` table is also partitioned and all archived-board reads/counts/searches are scoped. - Schema drift self-heal generalized to schema-qualified columns so existing databases upgrade in place. **Other changes** - Node settings sync **removed** in PG mode (409 `settings-sync-disabled-postgres`) — nodes share state by connecting to the same database; auth sync kept (per-machine file). - Perf (review findings): `listTasks` pushes column filter + ORDER BY + LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200 messages. - Fixed a false "operator action required" pause-abort log fired on every successfully auto-merged task. **Live acceptance — PASSED (2026-07-12)** A sandboxed instance (isolated HOME, embedded PG, real Opus executor) ran a task through the complete cycle: create → triage (AI spec) → execute → in-review → AI squash-merge landed on the project's `main` → done. A write+read sweep of every data surface (settings, comments, documents, attachments + artifact bridge + artifact edit, chat with real generation, goals, missions, agent mail, secrets, workflows, memory, CC analytics) was green on embedded PG. **Known remaining work** - The per-project `config` PK re-key has no upgrade path for pre-isolation embedded-PG databases (needs a real `DROP CONSTRAINT`/re-key migration; fresh databases are fine). - `pg_dump`/`pg_restore` binaries are not yet bundled in release artifacts (PATH/common-location discovery only). - The satellite-store concurrency findings listed above. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Phil Larson <hello@phillarson.xyz> Co-authored-by: fusion-merge <fusion-merge@local> |
||
|
gsxdsm
|
e90a9c4f51 |
fix: auto-heal wedged SQLite connections in place instead of wedging until process restart
A long-lived connection can go SQLITE_NOTADB ('file is not a database' on
every query) while the on-disk file stays intact — observed 2026-07-10 on the
live dashboard, which then failed every API request and poll cycle until the
process was restarted, because all corruption recovery ran at open time only.
The sqlite adapter now detects connection-corruption errors, closes the dead
handle, reopens the same path, replays connection-scoped PRAGMAs, verifies
with quick_check, and retries the failed operation once when outside an
explicit transaction. Prepared statements are generation-tracked and
re-prepare transparently after a reopen; a lost transaction's unwind is
absorbed so the original error propagates cleanly. Reopens are rate-limited,
and real on-disk corruption still defers to the open-time recovery machinery.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
gsxdsm
|
6dc8eb7161 |
perf: snapshot migrated in-memory DB to skip per-test migrations
db.init() replays SCHEMA_SQL + ~129 migrations on every fresh in-memory DB (~40ms each), which is minutes of pure setup across thousands of DB-backed tests. Add a test-only migrated-schema snapshot: migrate ONE in-memory DB per test file, serialize it, and deserialize a fresh copy per test instead of re-migrating. Each test still gets a brand-new, fully-isolated in-memory DB; only the migration cost is amortized. - sqlite-adapter: expose serialize()/deserialize() (node:sqlite + bun) - db.ts: setInMemoryTemplateSnapshot() hook (test-only, null in prod) + serializeSnapshot(); constructor deserializes the snapshot for in-memory DBs so init() short-circuits migrate()+compat at v129 - store-test-helpers: install/clearInMemoryDbSnapshot harness - dashboard: db-snapshot-helper mirror (core __tests__ is cross-package) - convert agent-store, mission-store, workflow-routes suites Measured (raw db.init(): 43ms -> 5ms, 8x): - agent-store 13.12s -> 3.32s - mission-store 17.62s -> 5.69s (min of 3) - workflow-routes tests 4.38s -> 2.79s (min of 5; not init-dominated) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
gsxdsm
|
822bde7829 | test(FN-2360): harden test isolation around repo fusion state | ||
|
gsxdsm
|
14e41b0ebe |
fix(core,cli): make standalone binary actually run
Bun's --compile binary previously crashed at startup because:
1. node:sqlite isn't implemented in Bun 1.3.8 (require returns
undefined; import throws "No such built-in module")
2. ink imports react-devtools-core inside its reconciler; even though
gated by isDev(), the bundled module path failed to resolve at
runtime
Fixes:
- Add packages/core/src/sqlite-adapter.ts: a thin DatabaseSync wrapper
that picks bun:sqlite under Bun and node:sqlite under Node via
createRequire (so the bundler doesn't statically pull in either).
Drop-in for the three core files that import DatabaseSync.
- Install react-devtools-core as a workspace devDependency so it
resolves at bundle time. The dev-only code path is still gated by
DEV=true, so it stays inert in production.
- Revert the prior --external react-devtools-core flag (no longer
needed and was causing a different runtime error).
- Mark node-pty external in tsup so esbuild stops choking on the
homebridge fork's conditional native require()s
(build/Release/conpty.node etc.) when bundling for the npm package.
- Update bundle-output test: the bundle now contains both
bun:sqlite and node:sqlite specifiers (loaded via createRequire).
Verified end-to-end: dist/fn dashboard -p 0 starts cleanly (no PTY,
sqlite, or devtools errors). Core tests 3038/3038, CLI tests 826/826
(up from 822/826 baseline).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|