Repair CLI test mocks and retry-reset expectations for the current store contracts.
- Add the backend store factory to the experiment-finalize mock.
- Provide global settings directory access in backup test stores.
- Assert all manual retry reset fields in task command tests.
Files changed:
.../extension-experiment-finalize.test.ts | 13 ++++++++++
.../commands/__tests__/backup-lock-retry.test.ts | 2 ++
packages/cli/src/commands/__tests__/backup.test.ts | 17 ++++++++++++-
packages/cli/src/commands/__tests__/task.test.ts | 28 +++++++++++++++++-----
4 files changed, 53 insertions(+), 7 deletions(-)
Fusion-Task-Id: FN-8222
Fusion-Task-Lineage: 033ce6a6-c699-409c-a59e-2d1f5e041cfc
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Align daemon startup mocks with the model runtime initialization path.
- Add the auth-storage setModelRuntime mock.
- Stub the Fusion model registry factory to use the shared test registry.
Files changed:
packages/cli/src/commands/__tests__/daemon.test.ts | 9 +++++++++
1 file changed, 9 insertions(+)
Fusion-Task-Id: FN-8220
Fusion-Task-Lineage: 6e00980f-2e24-4502-a4d8-e91849df33b9
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
## Summary
An audit of the SQLite→PostgreSQL store migration found data-store paths
still reaching the removed SQLite stub in backend (PG) mode. In backend
mode `store.db`/`getDatabase()` throw the removed-SQLite error, so each
of these either threw on every run or — worse — had the throw swallowed
into a silent wrong result. This PR routes all of them through the
`AsyncDataLayer` (and removes one dead primitive).
## The 6 live bugs fixed
| Fix | Was |
|-----|-----|
| `executor.ts` authoritative assigned-agent fallback now inherits the
TaskStore `asyncLayer` | silently returned `null` → model drift to the
pi built-in (the exact thing its comment guards) |
| `pruneAgentLogFilesAsync` replaces the sync self-healing prune call |
threw `SQLite Database is not available` every maintenance sweep →
agent-log pruning never ran |
| `cleanupOrphanedMaterializedSteps` deletes PG `workflow_steps` rows on
a failed create | swallowed the throw → leaked rows |
| `deleteTaskBackendImpl` now runs the async mission feature/task-link
unlink | PG hard delete left orphaned mission links |
| `getWorkflowSettingsProjectId` returns `rootDir` in backend mode
without touching the stub | swallowed throw for unscoped backend stores
|
| `fn plugin` unregistered-project fallback bootstraps a `CentralCore`
`AsyncDataLayer` | layerless `PluginStore` threw in PG |
## The 4 latent traps, fixed properly
- **`cleanupArchivedTasks`** — real async port (enumerate archived
soft-deleted rows, guarantee cold snapshot, hard-delete project row +
purge selection rows + rm dir).
- **`deleteWorkflowStep`** — real async port (delete `workflow_steps`
via the layer with `.returning()` to preserve the not-found contract).
- **`applyTaskPatch`** — **removed** (zero-caller SQLite column-patch
primitive with no backend analogue; impl + facade + import deleted).
- **`AgentStore.importLegacyFileRuns`** — clean backend no-op (no legacy
SQLite run-files exist in a PG deployment; its only `init()` caller
early-returns in backend mode).
## Symptom Verification
New PG regression suite
`packages/core/src/__tests__/postgres/store-sqlite-residue-fixes.pg.test.ts`
reproduces the original failures against real embedded Postgres and
asserts they're gone:
- orphaned `workflow_steps` are actually deleted (no swallowed throw)
- `pruneAgentLogFilesAsync` resolves and prunes inactive-task log files
- hard delete unlinks the mission feature from the task
- `deleteWorkflowStep` removes the row / reports not-found
- `cleanupArchivedTasks` hard-deletes the project row while retaining
the cold snapshot
## Verification
- `@fusion/core`, `@fusion/engine`, `@runfusion/fusion` typecheck clean
- ~50 existing + 5 new PG tests pass; lint clean; changeset validates
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented PostgreSQL backend maintenance from hitting removed legacy
SQLite code paths, avoiding datastore failures and residue cleanup
issues.
* Fixed workflow-step deletion and “not found” behavior in backend mode.
* Ensured backend hard-deletes correctly unlink related mission
feature/task links and clean orphaned materialized steps.
* Prevented legacy file-run imports from incorrectly reporting success
in backend mode.
* **New Features**
* Added async agent-log pruning for inactive tasks and updated
maintenance to use it.
* **Tests**
* Added PostgreSQL regression coverage for residue fixes and
archive/workflow cleanup.
* **Refactor**
* Removed an unused task patch operation and updated task-store cleanup
methods to be async where needed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Problem
Permanent (durable) agent heartbeats went silent while the rest of the
engine kept running. Investigation of the live DB showed **every**
permanent agent's `heartbeatTimerRepair` metadata carrying
`nonAdvancingEscalated: true` with **3–63 consecutive** zombie re-arms —
the audit re-arming a timer every 60s for hours while emitting
`heartbeat-rearm-nonadvancing-escalated` warnings that never recovered
anything.
## Root cause
The heartbeat trigger audit classified a "zombie" (dead) timer **solely
from a stale `lastHeartbeatAt`**. But that column advances *only* on a
successful `"ok"` delivery (`agent-store.ts` `recordHeartbeat`). It
stays frozen whenever a heartbeat is intentionally skipped or no-op'd:
- agent over budget / over budget threshold
- `globalPause` / `enginePaused`
- `skipHeartbeatWhenIdle` on an idle agent
- idle "org" agents whose runs complete as `no_assignment_identity_run`
In all of these the interval keeps firing perfectly — the timer is
alive, delivery is just (correctly) skipped. Keying zombie detection off
`lastHeartbeatAt` misread those healthy timers as dead, re-armed them
every 60s, and escalated forever. Re-arming a live timer is a no-op, so
the loop could never recover — it only produced churn and phantom
warnings.
## Fix (the invariant)
Key zombie detection off **whether the interval physically fired**, not
whether delivery advanced.
- New `lastTimerFireAtMs` map, stamped at the top of `onTimerTick`
**before any gate** — a fired-but-skipped tick still counts as proof of
liveness.
- In the audit: a present + stale timer that fired within its stale
window is **left untouched** (no re-arm, no escalation, non-advancing
counter reset). Only a timer with **no recent fire** (a genuinely dead
interval) falls through to the existing re-arm/escalation path.
- Map cleaned up in `unregisterAgent()` / `stop()`.
This preserves the FN-7645 zombie repair (a timer that stops firing goes
stale in lockstep on both clocks and is still re-armed) and the FN-7939
watchdog, while eliminating the phantom churn for live-but-skipping
timers.
Why not "force a heartbeat" or "park the agent": forcing delivery would
bypass budget/pause governance, and parking a healthy idle agent would
be wrong. The correct action for a live-but-skipping timer is to leave
it alone — its next real tick delivers once the skip condition clears.
## Tests
- Rewrote the old `skipHeartbeatWhenIdle` test that codified the buggy
escalation → now asserts a **live** idle-skipping timer is left
untouched (no zombie re-arm, no escalation).
- Added a budget/no-assignment surface: a live timer that dispatches but
leaves `lastHeartbeatAt` frozen must not be misclassified.
`heartbeat-scheduler.test.ts` 120/120; broader heartbeat + concurrency
suites 349/349; `@fusion/engine` typecheck 0 errors.
## Review
Self-reviewed at medium effort. Two acknowledged, bounded trade-offs
(kept intentionally): a genuinely-dead-but-recently-fired timer's repair
latency is bounded at ~2× interval (same as the original FN-7645
latency), and the escalation warning is suppressed for live timers (it
only ever fired because of the churn this removes; per-tick error logs +
a new "left live-but-skipping timer" log retain visibility). One trivial
cleanup applied (single `Date.now()` sample).
## Notes
- Engine is a private package → no changeset.
- Complementary to a separate in-flight fix for the
agentStore/scheduler-not-constructed bug (why heartbeats stopped
*entirely*); this PR ensures that once the scheduler runs again, the
audit stops the phantom churn/escalation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved heartbeat timer monitoring to distinguish healthy timers from
genuinely stopped timers.
* Prevented unnecessary timer re-registration and warning escalation
when heartbeats are intentionally skipped due to idle, paused,
budget-limited, or unassigned states.
* Improved recovery when a replacement timer stops firing, ensuring it
is detected and repaired reliably.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Keep the Settings default-description coverage aligned with surfaced and global-only settings.
- Add missing Notifications and Scheduling description keys to the coverage map
- Allowlist the global-only LAN discovery setting so the guard does not require a UI description
Files changed:
.../sections/__tests__/settings-default-descriptions.test.tsx | 9 +++++++++
1 file changed, 9 insertions(+)
Fusion-Task-Id: FN-8216
Fusion-Task-Lineage: a5449429-c5b2-4d88-915b-26baba3fb60b
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Ensure task chat role icons reflect each agent's actual runtime model.
- Parse runtime model markers from status and text log entries.
- Prefer runtime and effective models over stale task provider overrides.
- Cover provider icon precedence and fallback behavior for all chat roles.
Files changed:
packages/dashboard/app/components/TaskChatTab.tsx | 8 +-
packages/dashboard/app/components/__tests__/TaskChatTab.test.tsx | 88 ++++++++++++++++++----
2 files changed, 76 insertions(+), 20 deletions(-)
Fusion-Task-Id: FN-8214
Fusion-Task-Lineage: 1eb02ca5-469f-4a9d-932c-5cf02383b7fe
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Restore the CLI package configuration test to the active test lane with current build expectations.
- Allowlist WhatsApp plugin-only tsup externals as non-runtime CLI dependencies.
- Assert the full workspace build command in the verification contract.
- Remove the stale package-config test quarantine.
Files changed:
packages/cli/src/__tests__/package-config.test.ts | 12 +++++++++++-
packages/cli/vitest.config.ts | 9 ++++-----
2 files changed, 15 insertions(+), 6 deletions(-)
Fusion-Task-Id: FN-8210
Fusion-Task-Lineage: aa29d866-430f-4097-affa-a89d107474b2
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Make pinned model provider headers sit flush against the fixed dropdown header stack.
- Remove top padding that exposed scrolling rows above sticky provider headers.
- Cover the no-seam invariant across desktop and mobile layouts with and without thinking controls.
- Add a patch changeset for the dropdown scrolling fix.
Files changed:
.changeset/fn-8212-model-dropdown-sticky-gap.md | 7 ++++
.../app/components/CustomModelDropdown.css | 5 ++-
.../__tests__/CustomModelDropdown.test.tsx | 46 ++++++++++++++++++++++
3 files changed, 57 insertions(+), 1 deletion(-)
Fusion-Task-Id: FN-8212
Fusion-Task-Lineage: d7ce42d1-bcbe-4000-8107-794ec8e05307
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
WhatsApp rejects handshakes advertising Baileys' baked-in stale protocol
version with a 405 close, so the plugin cycled starting->disconnected and
never issued a QR. connect() now fetches the current WA Web version per
socket build. /status also exposes lastError so this failure mode is
diagnosable from the documented troubleshooting surface.
The published bundled.js also failed to load entirely ("Dynamic require
of 'crypto' is not supported"): plugin bundles are ESM but Baileys is
CJS. bundlePluginEntry now injects the same createRequire banner as
dist/bin.js.
Verified end-to-end against an isolated fn serve: bundled.js loads,
status reaches awaiting-qr, /qr serves a scannable QR data URL,
/pair-code validates input, /logout clears auth state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full-screen mobile task-detail sheet hides all resize handles, so
FN-8015's `margin-inline-end: var(--space-lg)` gutter on the shared
`.floating-window__body` (added to keep the scrollbar clear of desktop
resize hot zones) only added dead space on the right and shifted the
entire panel left. Zero it for `.floating-window--task-detail` inside
the mobile breakpoint so `.detail-body`'s own padding defines both
insets equally; desktop resize-handle clearance is untouched.
Refined the FN-8015 invariant test to enforce the desktop hot-zone gutter
media-aware (strips @media blocks) and added a regression guard for the
mobile zeroing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tokenized spacing below the Settings current-theme selector before Font Size.
- Scope the spacing to the Settings current-theme row without affecting compact dropdowns.
- Add a regression test for the established spacing token.
- Add a patch changeset for the Settings layout fix.
Files changed:
.changeset/fn-8200-theme-selector-spacing.md | 6 ++++++
packages/dashboard/app/components/ThemeDropdown.css | 8 ++++++++
.../dashboard/app/components/__tests__/ThemeDropdown.test.tsx | 6 ++++++
3 files changed, 20 insertions(+)
Fusion-Task-Id: FN-8200
Fusion-Task-Lineage: 6b72d0f7-759f-49aa-a8f9-f788cc9ec5c5
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
## fix(postgres): scope the cross-process merge guard to the project
The guard's own comment (`project-engine.ts`) says it checks whether
another process is merging a task **"for this project"** — and in SQLite
mode the per-project DB file made that scoping implicit.
`getActiveMergingTaskImpl`'s `backendMode` branch queries the shared PG
`tasks` table with **no `project_id` filter**, so one merging task
anywhere serializes merges across **all** projects.
### Production evidence
6-project embedded-PG deployment: **697 cross-project `Merge deferred …
is already merging (cross-process guard)` retries in 10 minutes** — six
independent repos waiting on each other's serialized merger, collapsing
merge throughput ~6x and letting `in-review` pile up to 95 tasks.
### Fix
Add the existing `taskProjectScope(layer)` filter to the query's
conditions (one line + import). It is a no-op when the layer carries no
`projectId`, so single-project deployments and the SQLite path are
unchanged. Same pattern as the other project-scoped task queries.
Deployed on the affected instance: cross-project merges now proceed in
parallel; per-project serialization (the guard's documented intent) is
preserved.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved merge task handling so activity in one project no longer
unnecessarily blocks merge operations in other projects.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: TrinaryCompute <fusion-merge@trinarycompute.dev>
## Summary
- Follow-up after #2229: full suite on main still failed on dashboard
curated inventory (21 ungated files) and mass engine failures
(`this.store.getAgentLogCount is not a function`).
- Harden executor tool-failure cursor capture for minimal/test
`TaskStore` adapters (same optional-API pattern as `project-engine`),
keep mock fixtures in lockstep, and quarantine inventory-only dashboard
files with ledger + vitest exclude.
## Changes
- **Executor**: optional `getAgentLogCount` / `getAgentLogs` /
`updateTask` at graph entry and trailing-failure detection.
- **Mocks**: `createMockStore`, soft-delete guard, post-done
continuation, cron `getGlobalSettingsDir`, executor-prompt
`bulkCompletionRefusalAt` (FN-8141).
- **i18n** (prior commit): es/fr/ko/zh-CN/zh-TW triage-duplicate keys.
- **Inventory**: 21 dashboard files → `test-quarantine.json` +
`vitest.config.ts` lockstep (VAL-REMOVAL SQLite / load flakes /
build-only dist assert).
## Test plan
- [x] `node scripts/check-test-inventory.mjs --dashboard-curated`
- [x] `pnpm test:gate`
- [x] engine: soft-delete, prompt, cron, post-done, tool-failure-retry,
and related samples
- [x] `@fusion/core` schema-applier + `@fusion/i18n` parity
- [ ] Full Suite (non-blocking) on this PR / main after merge
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added localized text for triage duplicate-resolution settings and
near-duplicate task actions in Spanish, French, Korean, Simplified
Chinese, and Traditional Chinese.
- Users can now see translated options and confirmations to keep or
delete detected duplicate tasks.
- **Bug Fixes**
- Improved resilience during task execution and recovery when optional
activity-log services are unavailable, preventing avoidable failures
during error handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
pi >=0.80.8 moved session request auth to ModelRuntime.getAuth -> pi-ai
resolveProviderAuth, which reads credentials.read("anthropic") and refreshes an
OAuth credential via credentials.modify("anthropic"). Fusion stores the
subscription login under `anthropic-subscription` with no raw `anthropic` row,
so the refresh callback saw current===undefined, bailed, and auth resolved to
undefined -> "Provider is not configured: anthropic" (then fell back).
Resolve read("anthropic") through fusion's getApiKey (refresh + raw/legacy/
subscription/fallback precedence) and hand pi-ai a ready api_key credential;
pi-ai routes it as OAuth by the sk-ant-oat token prefix. Supersedes the prior
read-alias, which fixed lookup but not the broken OAuth refresh-via-modify path.
Verified end-to-end: ModelRuntime.getAuth(anthropic/opus) now resolves the
subscription token instead of returning undefined.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pi-ai >=0.80 resolves provider auth via credentials.read(provider.id) and
performs OAuth refresh/derivation itself, bypassing fusion's getApiKey()
where the anthropic-subscription -> anthropic alias lived. A subscription-only
login surfaced as "Provider is not configured: anthropic" at prompt time even
though the status card showed connected.
- Alias the subscription OAuth credential into read("anthropic") at the
credential-store layer (createFusionCredentialStore); raw/legacy rows still win.
- Match "not configured" in isRetryableModelSelectionError so an unresolved
provider triggers the configured fallback model instead of hard-failing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## fix(engine): reclaim leaked semaphore slots when the system is busy,
not only at total idle
### The bug
`recoverIdleSemaphoreLeakCandidate` only reclaims stale `AgentSemaphore`
slots when the system is **completely** idle (`persistedActive === 0 &&
inFlightCount === 0` → reconcile to 0). If even one in-progress row
persists — e.g. a zombie task whose agent session died without its
`finally` release — the valve never opens, and slots leaked by abnormal
teardown accumulate monotonically until `activeCount` pins the limit.
At that point the engine deadlocks in a distinctive way:
- every hold/release sweep logs `Hold release for <task> deferred — no
reservable slot for in-progress`
- triage/plan report `planning=0 … processing=0, semaphore
active=<limit>/<limit>, available=0`
- the merge queue grows unboundedly (merges also need a slot)
- only a process restart recovers
`reapLeakedConcurrencySlots` (FN-6782) doesn't help — it reconciles
**worktree** slots, not the shared semaphore.
### Production evidence
Observed twice on a 6-project embedded-PG deployment driving a local
model:
- After ~5 days of continuous operation: `semaphore active=24/24`,
`planning=0/24, processing=0`, 5 persisted in-progress rows (dead
sessions), merge queue at 88, **zero merges for >24h**. Restart
immediately restored merging.
- Same signature earlier at `active=40/40` with both LLM backends idle
(`kvcache≈0`).
The handful of zombie in-progress rows kept `persistedActive` nonzero
indefinitely, so the idle-only valve could never fire.
### The fix
Generalize the valve: clamp `activeCount` down to the **persisted +
in-flight bound** whenever the semaphore over-holds **continuously** for
a repair window.
- The strict-idle case (`bound === 0`) keeps its existing fast 5s window
— behavior unchanged, existing tests pass as-is.
- The non-idle case uses a deliberately conservative new window
(`STALE_SEMAPHORE_EXCESS_REPAIR_MS = 600_000`, 10 min): nested helper
agents (`runNested`) legitimately push `activeCount` above the persisted
top-level count for the duration of a nested run, so the excess must
outlive any plausible nested session before it is treated as leaked. The
candidate timestamp resets the moment the excess clears.
- `reconcileActiveCount` only ever lowers the count, so the clamp cannot
inflate capacity; a late release from a genuinely live agent after a
(worst-case, mis-timed) clamp is absorbed by the existing excess-release
guard (FN-6423).
Call-site changes are limited to the two log messages (the old
parenthetical claimed "no persisted … agent work", which is no longer
the only repair case).
### Tests
- existing idle-valve tests pass unchanged (same window, same
reconcile-to-0)
- new: stale excess above a nonzero persisted bound is repaired only
after the long window, and clamps exactly to the bound
- new: candidate resets when the excess clears (nested overshoot ending)
- new: caller in-flight sessions count into the bound (no false
candidate)
### Files
- `packages/engine/src/concurrency.ts` — generalized valve +
`STALE_SEMAPHORE_EXCESS_REPAIR_MS`
- `packages/engine/src/scheduler.ts`, `packages/engine/src/triage.ts` —
log message accuracy
- `packages/engine/src/__tests__/concurrency.test.ts` — 3 new tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved stale “semaphore excess” recovery by using a configurable
repair window when excess persists.
* Prevented premature capacity corrections by accounting for in-flight
top-level work during reconciliation.
* Correctly handles nested helper activity so only leaked excess is
reclaimed, preserving legitimate nested runs.
* Updated reconciliation to clamp excess to the appropriate reclaim
floor instead of waiting indefinitely.
* **Improvements**
* Refreshed diagnostic warning text to clarify the over-held vs
persisted+in-flight work comparison.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: TrinaryCompute <fusion-merge@trinarycompute.dev>
Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
PR #2260 added project.tasks.bulk_completion_refusal_at to the Drizzle model
and the 0000 baseline but shipped no forward migration. Databases created
before #2260 already carry the 0000 marker, so the applier skips the baseline
and they never gained the column — every such cluster crashed on the first
TaskStore SELECT ("column bulk_completion_refusal_at does not exist"), taking
down dashboard/app boot.
Adds forward migration 0018 (wired via BULK_COMPLETION_REFUSAL_AT_VERSION;
SCHEMA_BASELINE_VERSION -> "0018") so existing clusters heal on next startup.
Prevention:
- Per-column upgrade regression test reproducing the exact existing-DB failure.
- Migration-wiring-integrity guard (no PostgreSQL): SCHEMA_BASELINE_VERSION must
equal the highest migration file, and every .sql must be registered in the
applier so none silently never runs.
- Repairs 6 pre-existing schema-applier tests left stale by the 0017 addition
(baseline-marker identity + version-list enumerations).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
