643a4098abb7bf5b966e70586d2911f2d2e71270
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
gsxdsm
|
3a016b1f17 |
fix(scripts): four FNXC stamps carried hour 26, and main has been red on them (#3010)
## `main` is currently red on `check-fnxc-future-dates` Four stamps read `2026-07-30-26:10` — an hour that cannot exist. They're exactly what #2995 taught this gate to catch. That PR landed the hour validation (`00-23`) *after* #2999 had already merged these four, so the gate started reporting a defect that was already sitting there rather than one introduced afterwards. **The guard is working**; nothing was checking before it. ``` scripts/lib/backend-db.mjs:41 scripts/reconcile-task-state-consistency.mjs:8, :51 scripts/__tests__/reconcile-task-state-consistency.test.mjs:109 ``` Corrected by **literal normalisation** — 26:10 on the 30th *is* 02:10 on the 31st — rather than flattening them to an arbitrary in-range hour. AGENTS.md specifies `yyyy-MM-dd-hh:mm`, and the stamp exists to give a readable why-does-this-exist trail, so the ordering is the part worth preserving. ## The baseline tightening rides along, and it's a date rollover Stamps written yesterday as `2026-07-31` were future *then* and were baselined as such. Today they're past, so **176 files ratchet to zero**. Nobody did anything. The gate rewrites the baseline as a side effect and exits 0, so leaving it uncommitted dirties the tree on every subsequent run **for everyone** — which is why it belongs in this commit rather than a later one. Re-recording on a decrease is the rule this gate and its siblings already state. Worth knowing about the design, since I wrote it: this churn recurs whenever a day boundary passes with future-dated stamps in the baseline, and it shrinks only as people stop writing them — which is the behaviour the gate exists to produce. **93 files still carry a non-zero allowance**, so the drain isn't finished. If it stays noisy once those clear, the gate's fail-on-tighten contract is the thing to revisit, not the stamps. ## Measured | check | result | |---|---| | gate | red before, **exit 0 after**, stable across two consecutive runs | | baseline | −176/+25 entries, all date-rollover | | inert-seam · sql-literal · lane-wiring · census | all green | | reconciler's own suite | green | ## One correction to a claim I made earlier this session While investigating I reported the gate as hanging for 600s. It wasn't — the harness killed the process (exit 144) and the empty output made it look like a stall. The gate completes in seconds. Noting it because I nearly filed a performance bug against a healthy script. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
gsxdsm
|
52a66297fc |
fix(gate): main is red — normalize #2994's four impossible-hour stamps (#3006)
**`main` is currently red on the FNXC gate.** ``` $ node scripts/check-fnxc-future-dates.mjs # on origin/main scripts/reconcile-task-state-consistency.mjs: 2 future-dated FNXC stamp(s), baseline allows 0 scripts/lib/backend-db.mjs: 1 scripts/__tests__/reconcile-task-state-consistency.test.mjs: 1 exit 1 ``` #2994 carried four `2026-07-30-26:10` stamps. I flagged them on that PR before it merged; #2995 (the hour check) landed first, so the merge order turned the warning into a red gate rather than a red PR. Clamped to `23:10` — same rule as the nine before it: hour to `23`, minutes preserved, so ordering within each file survives. This is a normalization with a stated rule, not a claim about the true minute. **Verified:** FNXC gate exit 0, `reconcile-task-state-consistency` 8 pass / 0 fail. Comment-text only. ### Worth fixing at the source Thirteen impossible-hour stamps across six PRs in two days, and the hours climb — `24:40` → `25:30` → `26:10`. They are being written as a continuing sequence past midnight rather than read off a clock, which is a reasonable instinct and produces an invalid stamp every time. The trap is that the honest spelling does not work either: a genuine post-midnight stamp needs *tomorrow's* date, and the gate compares against the **local** calendar — so `2026-07-31-00:40` written from UTC-7 is future-dated and fails for a different reason. Clamping to `23:xx` is currently the only spelling that satisfies both, which is not obvious and is why this keeps recurring. If it recurs again, the fix is probably in the error message rather than more normalization PRs: the gate could name the valid range and the timezone it compares against, so the next author sees the constraint at the moment they hit it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
gsxdsm
|
ac67b8d585 |
fix(scripts): the FN-4000 consistency reconciler failed in BOTH directions on a renamed board (#2994)
## The FN-4000 consistency reconciler failed in *both* directions
`findTaskStateInconsistencies` keyed both checks on legacy lane
literals, and they break in opposite ways:
```js
const hasDoneTransient = task.column === "done" && (status failed || error || worktree || blockedBy || …);
if (task.status === "failed" && task.column !== "in-review") { … }
```
| check | on a renamed board | effect |
| --- | --- | --- |
| `hasDoneTransient` | **never fires** | a finished card still holding
`status:"failed"`, a worktree, a blockedBy or live recovery counters is
never reported and never normalized — precisely the stale state FN-4000
exists to clear |
| `failed-status-outside-in-review` | **fires for every failed card** |
no column equals the literal, so the report lists the whole board |
The second is the more dangerous of the two: a tool that reports nothing
looks broken, but a tool that reports everything looks like it is
working.
## Wiring, and why the resolver is injected rather than built inline
Lanes are resolved **per task** (a board can span workflows) and passed
in. Resolving inside the loop would drag `importCore()` — and therefore
a built `packages/core/dist` — into every unit test of a pure
reconciliation loop.
`main` wires the real resolver whenever it opened a real backend, so
this is **not** the inert optional-parameter shape this migration keeps
finding. A caller injecting its own store (tests) has no staged dist and
falls back to the documented legacy literals, which is exactly today's
behaviour.
`importCore` is now exported from `scripts/lib/backend-db.mjs` so
operator scripts reach core helpers through the **same staged-dist seam
`openBackend` already uses**, rather than each growing its own dist path
— `@fusion/core` is not resolvable from repo-root `scripts/`, which is
what made the obvious import fail.
The normalization move now targets the card's **own** column: naming
`"done"` was only ever a way of spelling *"where it already is"*, since
the move exists to trigger the store's done-normalization.
## One of my test expectations was wrong before the code was
My first version asserted that a card in a renamed complete lane with
`status:"failed"` yields only the transient-state finding. It yields
**both** — and that is correct, because a failed card outside the review
lane genuinely is flagged. I isolated the case (dropping
`status:"failed"`, keeping the worktree) so it pins one behaviour
instead of blurring two, rather than "fixing" the expectation to match
whatever came out.
## Revert proof
Restoring the four literals:
```
✖ reports stale transient state in a RENAMED complete lane
✖ does NOT flag a failed card that is sitting in the board's own review lane
✖ runReconciliation normalizes a renamed complete lane by moving the card to its OWN column
ℹ pass 5 ℹ fail 3
```
The remaining two new cases pass both ways by design — "still flags a
failed card outside the resolved review lane" and "unresolved lanes keep
exactly the legacy behaviour" guard against over-correction, so I am not
counting them as coverage of the defect.
## Verification (measured)
- `node --test` — **8 passed / 0 failed** (3 pre-existing + 5 new)
- sibling script suites (`recover-stale-blocked-by`,
`reconcile-leaked-soft-deletes`) — **7 passed**, unaffected by the
shared-lib export
- `node --check`, `eslint` — clean
- `lifecycle-column-census --strict`, `check-sql-column-literals`,
`check-lane-wiring`, `check-fnxc-future-dates` — green
No changeset: root `scripts/` is repo tooling, not part of the published
package.
## Still not addressed in this territory
`reconcile-leaked-soft-deletes.mjs` carries a raw `UPDATE
project."tasks" SET "column" = 'archived'` — on a renamed board that
writes a column the workflow does not declare, creating the
undeclared-column state this migration keeps repairing elsewhere. It
holds a raw backend rather than a store, so it needs the same
`importCore` seam this PR exports; left for a follow-up rather than
bundled here.
|
||
|
gsxdsm
|
97172fdcf2 |
fix(FN-7952): require PostgreSQL in CLI and desktop (#2110)
## Summary CLI commands, daemon/dashboard startup, packaged desktop startup, and live-data maintenance scripts now share the mandatory PostgreSQL lifecycle. Operators no longer risk a command silently reading or writing a disconnected SQLite shadow when PostgreSQL setup fails. ## Design decisions - Every startup owner retains and awaits its PostgreSQL shutdown callback, including partial-startup failure paths. - CLI project context and lock-retry flows resolve through asynchronous project stores. - Maintenance scripts use the shared backend helper; explicit database migration/inspection remains the only CLI surface allowed to read legacy SQLite sources. ## Validation - CLI and Desktop typechecks pass on the stacked branch. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 54 files. ## Stack - Depends on #2109, which depends on #2108. - Bundled plugins and docs/release follow in later PRs. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the authoritative store for structured project and task metadata. * Projects can be recognized and initialized using `.fusion/project.json`, without creating a legacy SQLite database. * CLI commands now retry transient PostgreSQL contention errors. * **Bug Fixes** * Improved cleanup when commands complete, fail, or run in the background, preventing lingering resources. * Improved desktop, server, and session shutdown reliability. * **Documentation** * Updated storage and standalone binary guidance to reflect PostgreSQL and legacy SQLite compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
gsxdsm
|
c25f8b796d |
Harden PostgreSQL migration foundation (#2088)
## Summary - make SQLite-to-PostgreSQL cutover retryable, fail-closed, versioned, and transactionally serialized - isolate migration sessions from runtime traffic and apply schema upgrades through `0002` - enforce tenant ownership across automations, analytics, activity, usage, agent runs, evals, and todos - replace expired SQLite-only coverage with PostgreSQL parity and concurrency coverage This is PR 1 of 2. The stacked follow-up restores PostgreSQL parity for CLI, engine, dashboard, and bundled integrations. ## Verification - `pnpm check:changesets --strict` - `pnpm --filter @fusion/core typecheck` - migration schema, connection, and SQLite cutover suite: 57 tests passed - `pnpm test:gate`: 463 tests passed ## Post-Deploy Monitoring & Validation - take a restorable PostgreSQL backup before deploy - confirm `fusion_schema_migrations` contains `0002` - confirm each expected project has a complete `fusion_sqlite_migrations` row - verify no null or empty tenant ownership in automations, activity logs, agent runs, and usage events - monitor for ownership inference failures, cutover verification failures, and migration session errors - restore the backup for data rollback; do not downgrade the tenant-isolation schema in place <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL-backed analytics and live dashboard metrics are now project-scoped (activity, tools, monitor, signals, and live snapshots). * Evaluation runs and scheduled eval batches received lifecycle improvements (ordering, updates, and execution flow). * Todo list changes now emit events; WhatsApp persistence and project-scoped roadmap data are supported. * **Bug Fixes** * SQLite-to-PostgreSQL cutovers now fail safely with stronger verification, serialized cutover handling, and safer project ownership. * PostgreSQL backend writes and reads are now strictly project-isolated and fail closed when project context is missing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
gsxdsm
|
c15c78feeb |
feat: migrate storage from SQLite to PostgreSQL (#1793)
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover Migrates Fusion's storage layer to the embedded PostgreSQL `AsyncDataLayer` (the default backend) and **completes the satellite-store + feature cutover** so every dashboard and Command Center surface works in PG mode. ## Status — every surface works in embedded-PG mode Verified live against a running embedded-Postgres dashboard (all **200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate; core/engine/cli/dashboard typecheck clean). | Area | Surfaces | State | |---|---|---| | Satellite stores | workflows, todos, insights, research, missions, goals, mailbox | ✅ | | Views | artifacts, documents, evals | ✅ | | Command Center | activity, productivity, team, tokens, tools, **workflows**, **github**, **signals**, **plugin-activations**, **live** (all 10) | ✅ | | Run execution | insight generation, research run execution | ✅ (store-path; AI step needs a provider) | | Live updates | SSE push for mission/research/insight events | ✅ | | Workflow editing | create / update / delete / select (+ id counter) | ✅ | | Engine | mission autopilot, incident-signal ingestion, regression storm-guard, agent wake-on-message | ✅ | | Core | tasks, agents, secrets, automations, memory, chat, usage, PRs, git | ✅ | ## Approach Each satellite store gets an `Async<Store>` wrapper exposing the sync store's method names over the existing `async-*-store.ts` helpers; `get<Store>Store()` returns a `Sync | Async` union; consumers `await` (harmless on sync), and engine/CLI paths that can't convert use `instanceof Sync` graceful fallback. Analytics aggregators branch on `"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*` (snake_case) in PG. Executors/orchestrators/autopilot are await-converted to drive the union store; the async store wrappers extend `EventEmitter` so SSE live-push fires in both backends. Not-yet-ported capabilities degrade gracefully (never 500) and are individually called out in commits. ## Sync with main The branch is kept continuously merged with `main` (currently through FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer applies. Use **Create a merge commit** (or squash) to land it — GitHub's rebase-merge cannot replay a merge-maintained branch. ## Residual Review Findings Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5) applied 3 safe fixes (see `fix(review): apply autofix feedback`). The following are **real but gated** — recorded here as follow-up work rather than auto-applied. All are SQLite→PostgreSQL **concurrency/atomicity regressions**: the sync stores were immune only by SQLite's single-writer, single-threaded-handler execution; the async ports open multi-await read-modify-write windows. **Reachability is low today** because the execution engines that generate concurrent same-run mutations (insight run executor, research orchestrator/dispatcher) are `instanceof`-gated to sync mode in PG. No process-crash class survived (all engine fallbacks correctly guard the sync store). - **[P1] Research `appendResearchEvent` dual-write is non-atomic** (`packages/core/src/async-research-store.ts`, corroborated: adversarial + reliability). The `research_run_events` insert (own transaction) and the `run.events` jsonb update are separate writes — a crash between them, or two concurrent appends, splits the table count from the jsonb array. **Fix:** perform the seq-insert and the jsonb update in one `layer.transactionImmediate`. - **[P1] Research run terminal-reversion via stale full-row persist** (`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`). Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert a terminal run to `running` by overwriting the whole row, bypassing the transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status …` guard, or optimistic version column. - **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU** — concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:** `SELECT … FOR UPDATE` / enclosing transaction. - **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race** (`async-insight-store.ts`) — two callers can each create an "active" run. **Fix:** partial unique index on `(projectId, trigger) WHERE status IN ('pending','running')`. - **[P3] `createResearchRetryRun` return-value divergence** — sync returns the pre-update `queued` snapshot; async returns the reloaded `retry_waiting` run (persisted state is identical). Pick one side for cross-backend parity. - **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1 fan-out** — O(milestones×slices) sequential round-trips hold one pool slot per request; can starve the pool for large hierarchies. **Fix:** batched/joined reads. - **Testing gaps:** no PG-mode concurrency tests (interleaved status/event mutations), no sync↔async parity assertion for the lifecycle-error codes, and no mission status/health rollup parity test vs the sync `MissionStore`. ~~Out of scope (deferred): AI run *execution* (insight/research) + mission autopilot + live SSE mission events remain sync-gated/degraded in PG mode.~~ **Since ported** — insight/research run execution, mission autopilot, and SSE live push all run on the async layer now, which also makes the concurrency findings above genuinely reachable; they remain open follow-ups. --- ## Update — 2026-07-12: production-readiness hardening & live acceptance Everything below landed on this branch since the description above was written: **Production blockers from review — fixed** - `recoverStaleTransitionPending` ported to the async layer (backend moves write + clear the crash-safe marker; startup/maintenance sweeps no longer throw). - Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write changed columns only (full-row upserts silently resurrected stale fields across concurrent store instances — the "task stuck unplanned forever" bug). - First-boot **auto-migration**: booting the PG backend over a project with a legacy `fusion.db` migrates it automatically (loud failure, SQLite kept as backup), and the dashboard shows a one-time **"your data was migrated" banner** with the backup paths and a Need-help Discord link. - `pg_dump`/`pg_restore` discovered from common install locations for embedded-mode backups. - The PG suite is part of the blocking merge gate (`test:pg-gate`). **Multi-project isolation (PR #2007, merged into this branch)** - `project_id` partition key on tasks / archived tasks / config, `taskProjectScope` threaded through every scan/claim/count, per-project config rows, layer bound to the project at startup. - Review P1 follow-up: the shared cold-storage `archive.archived_tasks` table is also partitioned and all archived-board reads/counts/searches are scoped. - Schema drift self-heal generalized to schema-qualified columns so existing databases upgrade in place. **Other changes** - Node settings sync **removed** in PG mode (409 `settings-sync-disabled-postgres`) — nodes share state by connecting to the same database; auth sync kept (per-machine file). - Perf (review findings): `listTasks` pushes column filter + ORDER BY + LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200 messages. - Fixed a false "operator action required" pause-abort log fired on every successfully auto-merged task. **Live acceptance — PASSED (2026-07-12)** A sandboxed instance (isolated HOME, embedded PG, real Opus executor) ran a task through the complete cycle: create → triage (AI spec) → execute → in-review → AI squash-merge landed on the project's `main` → done. A write+read sweep of every data surface (settings, comments, documents, attachments + artifact bridge + artifact edit, chat with real generation, goals, missions, agent mail, secrets, workflows, memory, CC analytics) was green on embedded PG. **Known remaining work** - The per-project `config` PK re-key has no upgrade path for pre-isolation embedded-PG databases (needs a real `DROP CONSTRAINT`/re-key migration; fresh databases are fine). - `pg_dump`/`pg_restore` binaries are not yet bundled in release artifacts (PATH/common-location discovery only). - The satellite-store concurrency findings listed above. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Phil Larson <hello@phillarson.xyz> Co-authored-by: fusion-merge <fusion-merge@local> |