feat: faster dashboard and serve startup (#2132)
## Summary Speeds up **time-to-HTTP-ready** for `fn dashboard` and `fn serve` after the PostgreSQL cutover without reintroducing the historical 3s cwd-engine race that degraded webhooks. - **Dashboard store share (serve parity):** inject the factory-booted `TaskStore` as `externalTaskStore` so cwd `ensureEngine` does not open a second pool; share only when store root matches project working directory (multi-project safe). - **Serve multi-project:** stop awaiting `startAll()` before listen; await only the primary engine; background the rest + reconciliation. - **Defer non-route-critical engine work:** ordered OAuth (refresh → monitor), automation schedule syncs, and auto-merge **enqueue** after the engine handle is returnable. - **Critical-path merge status clear:** still clear stale `merging`/`merging-pr` before ready so manual merge is not blocked after crash. - **Serve `--paused`:** apply `enginePaused` before `ensureEngine`/`startAll` (dashboard ordering). - **Stop safety:** generation counter so deferred tails cannot resume after `stop()` clears `shuttingDown`. - **Phase timing:** shared `phaseTime` helper, factory substep logs, serve time-to-listen. Plan: `docs/plans/2026-07-14-001-feat-faster-startup-plan.md` ## Test plan - [x] `packages/engine` — `project-engine-manager.test.ts` (path-matched external store) - [x] `packages/engine` — `project-engine-deferred-startup.test.ts` (status clear, OAuth order, stop generation) - [x] `packages/cli` — `startup-phase.test.ts` - [x] `packages/cli` — `serve.test.ts` (60 tests, including `--paused`) - [ ] Local: warm `fn dashboard` / `fn serve` and compare `startup phase *` / `time-to-listen` logs - [ ] `pnpm smoke:boot` (real serve `/api/health` on ephemeral port) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Improved dashboard and serve startup times, including faster time-to-listen and time-to-ready. * Moved non-essential background initialization off the critical startup path. * Parallelized dashboard service initialization where possible. * **Reliability** * Improved multi-project startup handling and project selection. * Prevented cross-project task-store sharing. * Added safer shutdown behavior for partially completed startup. * **Diagnostics** * Added startup phase timing logs to help identify performance bottlenecks. * **Tests** * Expanded coverage for deferred startup, shutdown, project isolation, and startup timing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
gsxdsm
7
.changeset/faster-startup-p0.md
Normal file
7
.changeset/faster-startup-p0.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Speed up dashboard and serve startup by sharing the PostgreSQL store and deferring non-route work.
|
||||
category: performance
|
||||
dev: Dashboard injects externalTaskStore for cwd engine (serve parity); multi-project engines only share when working directories match. ProjectEngine defers notifiers/OAuth (refresh-before-monitor), automation syncs, and merge sweep. Serve no longer awaits startAll before listen. Phase timing logs on both surfaces.
|
||||
447
docs/plans/2026-07-14-001-feat-faster-startup-plan.md
Normal file
447
docs/plans/2026-07-14-001-feat-faster-startup-plan.md
Normal file
@@ -0,0 +1,447 @@
|
||||
---
|
||||
title: "feat: Faster dashboard and serve startup"
|
||||
status: completed
|
||||
date: 2026-07-14
|
||||
completed: 2026-07-15
|
||||
pr: https://github.com/Runfusion/Fusion/pull/2132
|
||||
type: feat
|
||||
origin: conversation (startup path analysis on feature/faster-startup)
|
||||
---
|
||||
|
||||
# feat: Faster dashboard and serve startup
|
||||
|
||||
## Summary
|
||||
|
||||
Shorten **time-to-HTTP-ready / TUI-usable** for `fn dashboard` and `fn serve` by eliminating dual TaskStore boots, moving non-route-critical engine work off the pre-listen path, parallelizing independent satellite inits, and extending phase timing — without reintroducing the historical 3s cwd-engine race that degraded webhooks, and without weakening PostgreSQL migration integrity.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
After the PostgreSQL cutover, local startup is dominated by:
|
||||
|
||||
1. **Backend boot** — embedded PG (cold `initdb` / warm `pg_ctl`) plus schema baseline and optional SQLite→PG auto-migration.
|
||||
2. **Dual store construction on dashboard** — dashboard already boots a PostgreSQL `TaskStore`, then `ensureEngine(cwd)` boots a **second** factory path because `externalTaskStore` is not wired (serve already shares one store).
|
||||
3. **Serial engine bring-up** — `ProjectEngine.start()` awaits notifiers, OAuth refresh/monitor, automation syncs, and `startupMergeSweep` before returning, which blocks `createServer` / listen on the dashboard critical path.
|
||||
4. **Extension resolution** — `packageManager.resolve()` is called out in-code as a slow dashboard phase; serve blocks on the same walk before listen.
|
||||
5. **Incomplete phase metrics** — dashboard logs coarse `startup phase *` labels; serve and engine internals lack matching substep timing, so regressions are hard to attribute.
|
||||
|
||||
Primary success metric: **time from process start to HTTP listening (and TUI `setReady`)** on warm multi-project and single-project boots. Full orchestration readiness may lag slightly if deferred work remains fail-soft and does not leave route closures unbound.
|
||||
|
||||
## Requirements
|
||||
|
||||
- R1. `fn dashboard` with engine on reuses a single PostgreSQL-backed `TaskStore` (and connection pool) for the cwd project HTTP layer and cwd engine — parity with serve’s `externalTaskStore` wiring.
|
||||
- R2. Multi-project engines must not receive a cwd-bound store for a different project root; store injection is cwd/path-matched (or projectId-matched), not “one store for every registered project.”
|
||||
- R3. `createServer` always receives a live cwd `options.engine` when engine mode is on — no timed race / partial-undefined engine that unbinds webhook, automation, mission, or routine routes.
|
||||
- R4. PostgreSQL schema baseline and first-boot SQLite auto-migration remain pre-first-write, single-owner, and fatal on verification failure.
|
||||
- R5. Non-route-critical engine startup work (merge sweep, notification/OAuth stack, automation schedule syncs) may complete after the engine object is returned / after HTTP listen, with fail-soft logging and no silent permanent skip.
|
||||
- R6. OAuth refresh-before-expiry-monitor order is preserved whenever both run (avoids false “token expired” ntfy on restart).
|
||||
- R7. Plugin module load may remain pre-`createServer` while plugin routes depend on a loaded `pluginLoader`; schema init failures stay integrity-critical (serve fatal; dashboard must not leave a half-bound plugin schema silently).
|
||||
- R8. Operators and developers can see per-phase wall times for backend factory substeps, engine blocks, and total time-to-listen on both dashboard and serve (extend existing `phaseTime` style; optional EL lag via `FUSION_TRACE_EL_LAG`).
|
||||
- R9. Boot smoke and thin merge gate remain green: real `fn serve` → `/api/health` on ephemeral port, clean SIGTERM; no binding/killing port 4040.
|
||||
- R10. Regression tests prove single factory boot for dashboard engine-on cwd path and that deferred work still eventually runs; tests stay file-scoped and free of real-network / slow full-suite habits (FN-5048).
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### In scope
|
||||
|
||||
- `fn dashboard` and `fn serve` process startup critical path.
|
||||
- `ProjectEngine` / `InProcessRuntime` pre-return awaits that are not required for route closures.
|
||||
- Satellite store init ordering on dashboard after shared layer boot.
|
||||
- Phase timing instrumentation and lightweight extension-path improvements (cache/defer only where chat/provider readiness is not blocked incorrectly).
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- Desktop embedded runtime dual-store parity (`packages/cli/src/commands/desktop.ts`, Electron local-runtime) — same seams, not primary.
|
||||
- Embedded PG keep-alive daemon / always-on external `DATABASE_URL` as the default operator workflow (document as operational tip; no product redesign here).
|
||||
- Full CLI bundle code-split / 14MB `bin.js` parse cost.
|
||||
- Deep SQLite→PG migration throughput rewrite (bulk copy algorithms) beyond ensuring progress remains visible.
|
||||
- Capturing the 3s webhook race as a dedicated `docs/solutions/` entry (recommended after land via ce-compound).
|
||||
|
||||
### Out of scope
|
||||
|
||||
- Re-enabling HybridExecutor for ordinary local multi-project (prior ~7s duplicate runtime cost).
|
||||
- Changing engine singleton lock semantics (`has` vs `hasRunningEngine`).
|
||||
- Dashboard query/index load work already covered by `docs/performance/dashboard-load.md`.
|
||||
- Product UX redesign of TUI loading copy beyond accurate phase status.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Confirmed primary metric is **time-to-HTTP-ready / TUI usable**, not full orchestration quiet-state.
|
||||
- Surfaces: **dashboard + serve**; desktop only if a unit can reuse the same API without expanding risk.
|
||||
- Aggressiveness includes **backgrounding non-route-critical engine work** after a correct engine handle exists, not only store-sharing.
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
1. **Share store via existing `externalTaskStore` seam, not a new abstraction.** Serve already proves the pattern: factory once → CentralCore `asyncLayer` → `ProjectEngineManager({ externalTaskStore })` → single shutdown. Dashboard adopts the same seam.
|
||||
|
||||
2. **Prefer cwd-scoped injection over manager-global blind share.** Manager today forwards one `externalTaskStore` to every engine in `buildEngineOptions`. For multi-project correctness under project-partitioned PG, inject only when the engine’s working directory / projectId matches the booted store’s root (cwd-only override on `ensureEngine`, or clear/undefined for other projects so they factory-boot their own bound store). Do not leave multi-project engines silently writing through the cwd partition.
|
||||
|
||||
3. **Engine handle before `createServer` remains a hard invariant.** Defer work **inside** `ProjectEngine.start()` / post-`start()` background phases so `options.engine` is non-null and route closures bind real subsystem getters. Do **not** reintroduce `Promise.race(ensureEngine, 3s)`.
|
||||
|
||||
4. **Deferral allowlist (post engine object, preferably post-listen for pure side effects):**
|
||||
- `startupMergeSweep` (self-healing / periodic merge retry already cover related cases; accept brief stale `merging*` window)
|
||||
- NotificationService + OAuth refresh/monitor/validity + NtfyNotifier (preserve refresh→monitor order inside the deferred chain)
|
||||
- Automation schedule sync helpers (CronRunner can start; syncs fill in with degraded→ready health)
|
||||
- Already allowed: custom provider `/models` refresh, mDNS (needs bound port), Claude skill FS backfill, runtime startup recovery sequence
|
||||
|
||||
5. **Keep pre-listen:** PostgreSQL factory + migration, store init/watch (or equivalent ownership), plugin load required for `getPluginRoutes()`, cwd `ensureEngine` completion for engine mode, HybridExecutor only when gate enables multi-node.
|
||||
|
||||
6. **Serve: stop awaiting `startAll()` before listen.** Match dashboard: `void startAll()` + reconciliation + await only primary/cwd engine needed for `createServer`’s primary store/engine. Multi-project engines warm in background.
|
||||
|
||||
7. **Instrumentation first-class but cheap.** Shared helper (or duplicated minimal `phaseTime`) for serve; add factory substeps (`embedded.start`, `schema.baseline`, `sqlite.migrate?`, `taskStore.construct`) and engine blocks (`runtime.start`, `notifiers`, `automations`, `mergeSweep`) behind normal log lines — not a new metrics product.
|
||||
|
||||
8. **Tests stay mock-first.** Assert factory call count / `externalTaskStore` wiring in CLI unit tests (extend serve/dashboard tests). Use `pnpm smoke:boot` / `verify:fast` for real process proof; avoid new real embedded-PG cold boots in the CLI vitest suite.
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
### Target boot topology (cwd project)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[bin / command load] --> B[createTaskStoreForBackend once]
|
||||
B --> C[CentralCore + asyncLayer]
|
||||
B --> D[Dashboard satellite inits parallel-safe]
|
||||
C --> E[ProjectEngineManager externalTaskStore for cwd only]
|
||||
D --> E
|
||||
E --> F[await ensureEngine cwd]
|
||||
F --> G[runtime.start core path]
|
||||
G --> H[return engine handle]
|
||||
H --> I[createServer with engine]
|
||||
I --> J[listen + TUI ready]
|
||||
H --> K[background: notifiers OAuth order]
|
||||
H --> L[background: automation syncs]
|
||||
H --> M[background: merge sweep]
|
||||
J --> N[background: custom providers mDNS skills]
|
||||
```
|
||||
|
||||
### Critical-path vs deferred
|
||||
|
||||
| Phase | Pre-listen / pre-engine-return | Deferred |
|
||||
|-------|--------------------------------|----------|
|
||||
| Embedded PG + schema + migrate | Yes | Never |
|
||||
| TaskStore + CentralCore layer | Yes | Never |
|
||||
| Satellite inits (agent/plugin/automation for HTTP) | Yes (parallelize) | — |
|
||||
| Plugin load for routes | Yes | Module-only defer only with route readiness redesign (out of scope) |
|
||||
| `InProcessRuntime` scheduler/executor/self-healing construct | Yes | Recovery sequence already deferred |
|
||||
| Notifiers / OAuth stack | No (after handle) | Yes |
|
||||
| Automation sync ×4 | No | Yes |
|
||||
| `startupMergeSweep` | No | Yes |
|
||||
| `packageManager.resolve` | Prefer overlap; cache if safe | Non-essential providers post-listen |
|
||||
| Non-cwd `startAll` engines | Background | Yes |
|
||||
|
||||
### Ownership rules when sharing store
|
||||
|
||||
- **One** `createTaskStoreForBackend` for cwd on dashboard engine-on path.
|
||||
- Engine `InProcessRuntime` takes `externalTaskStore` and skips factory + does not own `backendShutdown`.
|
||||
- Process teardown: engines stop → then factory `shutdown()` once (serve pattern).
|
||||
- Dashboard continues to pass the **same** store instance into `createServer(store, …)` so HTTP and engine see one writer.
|
||||
|
||||
## Sequenced Delivery (phases)
|
||||
|
||||
Land as dependency-ordered commits/PRs. Each phase is independently shippable and measurable.
|
||||
|
||||
| Phase | Units | Outcome | Risk |
|
||||
|-------|-------|---------|------|
|
||||
| **P0 — Measure & share store** | U1, U2 | Factory substep + serve timing; dashboard single store for cwd engine | Low |
|
||||
| **P1 — Serve multi-project non-block** | U3 | Serve listens without awaiting every project engine | Medium |
|
||||
| **P2 — Defer non-route-critical engine work** | U4 | Shorter `ensureEngine` / `ProjectEngine.start` critical path | Medium–High |
|
||||
| **P3 — Parallel satellite inits** | U5 | Shorter dashboard pre-engine path | Low |
|
||||
| **P4 — Extension path** | U6 | Reduce `packageManager.resolve` wall time / blocking | Medium |
|
||||
| **P5 — Verification & docs** | U7 | Boot smoke, gate, FNXC, optional solutions capture | Low |
|
||||
|
||||
Do not start P2 until P0 is green: store sharing is the largest structural win and simplifies measuring engine-internal deferrals.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. Extend startup phase instrumentation (dashboard factory + serve + engine hooks)
|
||||
|
||||
**Goal:** Make time-to-listen and bottleneck attribution first-class on both surfaces so later units can be validated with numbers, not guesswork.
|
||||
|
||||
**Requirements:** R8, R9
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/cli/src/commands/dashboard.ts` (extend labels / wrap factory)
|
||||
- `packages/cli/src/commands/serve.ts` (add phase logging)
|
||||
- `packages/core/src/postgres/startup-factory.ts` (optional substep logs)
|
||||
- `packages/engine/src/project-engine.ts` and/or `packages/engine/src/runtimes/in-process-runtime.ts` (block timings)
|
||||
- `packages/cli/src/commands/__tests__/dashboard.test.ts` or a small new helper test if extracting `phaseTime`
|
||||
- Prefer extracting a tiny shared helper under `packages/cli/src/` only if both commands can import without cycles
|
||||
|
||||
**Approach:**
|
||||
|
||||
- Keep cheap wall-clock logs (`startup phase <label>: Nms`).
|
||||
- Cover at least: `backend.factory` (and if easy: embedded start, schema, migrate, construct), `engine.ensureEngine`, total time-to-listen on serve (mirror dashboard `startupDurationMs`).
|
||||
- Engine: log durations for `runtime.start`, `notifiers+oauth`, `automations`, `mergeSweep` even before deferral so before/after is comparable.
|
||||
|
||||
**Patterns to follow:** Existing `phaseTime` in `dashboard.ts`; `FUSION_TRACE_EL_LAG` for optional EL stalls.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: helper or command test that a phase label is emitted around a stubbed async step (mock logger/logSink).
|
||||
- Edge: phase logs still emit when the wrapped function throws (use `finally`).
|
||||
- Test expectation if pure log wiring without extract: cover via existing dashboard/serve tests that still boot under mocks without asserting every label.
|
||||
|
||||
**Verification:** Manual or scripted dashboard/serve log shows labeled phases; no behavior change.
|
||||
|
||||
---
|
||||
|
||||
### U2. Dashboard cwd TaskStore sharing (`externalTaskStore` serve parity)
|
||||
|
||||
**Goal:** One factory boot and one connection pool for dashboard HTTP + cwd engine.
|
||||
|
||||
**Requirements:** R1, R2, R3, R4, R10
|
||||
|
||||
**Dependencies:** U1 helpful but not hard-required
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/cli/src/commands/dashboard.ts` — pass `externalTaskStore: store` (or cwd-only ensure override); align shutdown with `dashboardBackendShutdown`
|
||||
- `packages/engine/src/project-engine-manager.ts` — if needed, support **cwd/project-matched** injection so non-cwd engines do not inherit the wrong store
|
||||
- `packages/cli/src/commands/__tests__/dashboard.test.ts` — assert single `createTaskStoreForBackend` (or equivalent) when engine on; assert manager receives external store for cwd
|
||||
- `packages/cli/src/commands/__tests__/serve.test.ts` — keep/extend as regression for serve single-owner pattern
|
||||
- FNXC comments on ownership / multi-project matching
|
||||
|
||||
**Approach:**
|
||||
|
||||
- Mirror serve’s `ProjectEngineManager(..., { externalTaskStore: boot.taskStore })` for the dashboard-owned store.
|
||||
- Fix multi-project hazard: either document that the shared store is multi-tenant-safe for all projects under current PG binding **only if proven**, or inject store only for the matching projectId/path (preferred if binding is per-store).
|
||||
- Ensure `store.init()` / `watch()` ownership remains single: engine must not re-init/close the shared store destructively; teardown order matches serve (engines → backend shutdown once).
|
||||
- `--no-engine` path unchanged (no manager share required).
|
||||
|
||||
**Patterns to follow:** `serve.ts` externalTaskStore + `serve.test.ts` single-owner factory mock; `InProcessRuntime` external branch.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: dashboard engine-on path constructs factory once for cwd; engine receives external store; `createServer` still gets engine + same store.
|
||||
- Multi-project: second registered project engine does **not** use cwd store when roots differ (assert factory called for other project or explicit projectId bind — depending on implementation choice).
|
||||
- Error path: factory failure still aborts boot; no orphan engine without shutdown.
|
||||
- Integration with mocks: engine `ensureEngine(cwd)` does not call `createTaskStoreForBackend` again for cwd when share is wired.
|
||||
- `--no-engine`: factory still once; no engine share requirements.
|
||||
|
||||
**Verification:** Unit tests green; optional local dashboard log shows shorter `engine: ensureEngine(cwd)` and no second embedded “starting” for same data dir from a second owner path; `pnpm smoke:boot` still green.
|
||||
|
||||
---
|
||||
|
||||
### U3. Serve: non-blocking multi-project `startAll`
|
||||
|
||||
**Goal:** `fn serve` becomes HTTP-ready after primary/cwd engine is ready, not after every registered project engine finishes.
|
||||
|
||||
**Requirements:** R3, R5, R8, R9
|
||||
|
||||
**Dependencies:** U1 recommended; U2 independent
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/cli/src/commands/serve.ts` — `void engineManager.startAll()` (or await only primary), keep reconciliation; resolve primary store/engine for `createServer`
|
||||
- `packages/cli/src/commands/__tests__/serve.test.ts`
|
||||
- FNXC: serve readiness vs multi-project engine warmup
|
||||
|
||||
**Approach:**
|
||||
|
||||
- Align with dashboard: background `startAll`, await only the engine whose store backs `createServer`.
|
||||
- Preserve hybrid executor gate behavior; do not enable HybridExecutor for local-only multi-project.
|
||||
- Health endpoint must still report engine availability correctly for external singleton ownership (`hasRunningEngine`).
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: serve listen path does not await slow secondary project engines (mock second `ensureEngine` delayed; listen proceeds after primary).
|
||||
- Edge: zero projects / cwd unregistered — fail-soft or existing serve registration behavior preserved.
|
||||
- Error path: primary engine failure still fails boot (or existing policy); secondary failure is logged, not fatal to listen.
|
||||
- Regression: single-project serve still one factory + externalTaskStore.
|
||||
|
||||
**Verification:** Serve unit tests; boot smoke.
|
||||
|
||||
---
|
||||
|
||||
### U4. Defer non-route-critical `ProjectEngine` startup work
|
||||
|
||||
**Goal:** Shrink wall time of `ProjectEngine.start()` / `ensureEngine` by moving merge sweep, notifier/OAuth stack, and automation syncs off the return-critical path while keeping route-bound engine subsystems constructed.
|
||||
|
||||
**Requirements:** R3, R5, R6, R10
|
||||
|
||||
**Dependencies:** U2 (preferred so measurements isolate engine internals)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/engine/src/project-engine.ts` — split critical vs deferred phases; readiness/health signals if needed
|
||||
- `packages/engine/src/__tests__/project-engine.test.ts` (and related soft-delete/merge tests that use `skipNotifier`)
|
||||
- Possibly notification/oauth unit tests if start timing changes
|
||||
- FNXC comments documenting deferral allowlist and OAuth order
|
||||
|
||||
**Approach:**
|
||||
|
||||
- After `runtime.start()` and wiring required for routes (PR monitor config, settings listeners, auto-merge wiring as required), return-capable engine may complete deferred chain via `void this.startDeferredSubsystems().catch(...)`.
|
||||
- Deferred chain order for OAuth: **refresh scheduler start → expiry monitor → validity logger** (same relative order as today).
|
||||
- `startupMergeSweep`: background; document stale merging status window; keep unconditional stale status clear eventually.
|
||||
- Automation: prefer `cronRunner.start()` then background the four `sync*` calls; keep degraded health messaging if sync fails.
|
||||
- Do **not** defer plugin schema integrity or TaskStore factory/migration.
|
||||
- Do **not** use a wall-clock race on `ensureEngine` from dashboard/serve.
|
||||
|
||||
**Execution note:** Characterization-first for `ProjectEngine.start` ordering tests — capture what routes need from `options.engine` at construction time before moving awaits.
|
||||
|
||||
**Patterns to follow:** Existing deferred `resumeStartupRecoverySequence` in `InProcessRuntime`; post-listen custom provider refresh; `skipNotifier` test pattern for isolating notifier-free starts.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: `start()` resolves before deferred notifiers complete (mock delayed OAuth start); engine methods used by routes remain defined.
|
||||
- OAuth order: when deferred chain runs, refresh is invoked before expiry monitor check (spy call order).
|
||||
- Merge sweep: deferred sweep still clears stale `merging` / `merging-pr` on in-review tasks.
|
||||
- Automation: sync failures leave degraded health but do not reject `start()`.
|
||||
- Error path: deferred chain failure logs and does not crash process / does not leave unhandled rejection.
|
||||
- Regression: `skipNotifier: true` tests still pass; auto-merge wiring still present for task:moved paths after start.
|
||||
|
||||
**Verification:** Engine unit tests; dashboard phase log for `ensureEngine` drops vs baseline; boot smoke.
|
||||
|
||||
---
|
||||
|
||||
### U5. Parallelize independent dashboard satellite inits
|
||||
|
||||
**Goal:** Reduce serial `store.init → automation → plugin → agent → watch` wall time where dependencies allow.
|
||||
|
||||
**Requirements:** R1, R8
|
||||
|
||||
**Dependencies:** U2 recommended (shared store already init’d once)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/cli/src/commands/dashboard.ts`
|
||||
- `packages/cli/src/commands/__tests__/dashboard.test.ts` if ordering assertions exist
|
||||
- FNXC on init dependency order
|
||||
|
||||
**Approach:**
|
||||
|
||||
- After core `store.init()` (and layer available): `Promise.all` for independent `automationStore.init`, `pluginStore.init`, `agentStore.init` if safe.
|
||||
- `store.watch()` can overlap with extension resolution / plugin loading where event races are acceptable (today watch is early; keep correctness if TUI/subscriptions need watch before events).
|
||||
- Do not parallelize with factory/migration.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: dashboard boot under mocks still initializes all satellites.
|
||||
- Failure path: one satellite init rejection still surfaces (no swallowed partial boot unless existing fail-soft policy).
|
||||
|
||||
**Verification:** Phase logs show overlapping satellite durations; unit tests green.
|
||||
|
||||
---
|
||||
|
||||
### U6. Extension / packageManager critical-path reduction
|
||||
|
||||
**Goal:** Cut dashboard/serve wall time spent in `packageManager.resolve` and extension discovery without breaking provider registration for chat.
|
||||
|
||||
**Requirements:** R5, R8
|
||||
|
||||
**Dependencies:** U1 (measure first); can ship after P0–P2
|
||||
|
||||
**Files:**
|
||||
|
||||
- `packages/cli/src/commands/dashboard.ts`
|
||||
- `packages/cli/src/commands/serve.ts`
|
||||
- `packages/cli/src/commands/claude-cli-extension.ts` / droid / llama path resolvers (single settings read)
|
||||
- Optional small cache module under `packages/cli/src/`
|
||||
- Tests colocated under `packages/cli/src/commands/__tests__/`
|
||||
|
||||
**Approach:**
|
||||
|
||||
- Collapse triple `getGlobalSettings()` for Claude/Droid/Llama into one read.
|
||||
- Overlap `packageManager.resolve` with work that does not depend on its result (already partly true for plugins).
|
||||
- Optional: fingerprint cache of resolved extension paths (agentDir + settings mtime/hash) with safe invalidation.
|
||||
- Defer only provider registration that is already safe post-listen (custom providers already post-listen); do not defer self-extension `fn_*` tools required for agent sessions if chat can start immediately.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Happy path: extensions still load; self-extension path still set via `setHostExtensionPaths`.
|
||||
- Cache (if implemented): second resolve with unchanged fingerprint skips full walk; settings change busts cache.
|
||||
- Failure path: resolve failure still fails soft as today (dashboard creates empty extension runtime).
|
||||
|
||||
**Verification:** Phase log `packageManager.resolve` reduced on warm restart; unit tests for cache invalidation if added.
|
||||
|
||||
---
|
||||
|
||||
### U7. Gate verification, changesets, and operator-facing notes
|
||||
|
||||
**Goal:** Prove the sequenced work does not break boot smoke or published CLI contracts; document measurement how-to.
|
||||
|
||||
**Requirements:** R9, R10
|
||||
|
||||
**Dependencies:** U2–U6 as landed
|
||||
|
||||
**Files:**
|
||||
|
||||
- `.changeset/*.md` when `@runfusion/fusion` user-visible performance/behavior changes (category `performance`)
|
||||
- Optional short note in `docs/cli-reference.md` or `docs/getting-started.md` on reading startup phase logs / preferring `DATABASE_URL` for faster warm ops (only if product-facing)
|
||||
- FNXC updates kept current
|
||||
|
||||
**Approach:**
|
||||
|
||||
- Run file-scoped vitest for touched packages + `pnpm smoke:boot` / `pnpm verify:fast`.
|
||||
- Changeset body: labeled `summary` / `category: performance` / optional `dev`.
|
||||
- Do not quarantine flakes by widening timeouts.
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Boot smoke: CLI help + serve health + clean shutdown.
|
||||
- Gate-shaped: no new dependence on port 4040.
|
||||
|
||||
**Verification:** `pnpm verify:fast` and scoped tests green on the branch.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Dual writers / split HTTP vs engine store | U2 single store; single shutdown; tests on factory call count |
|
||||
| Multi-project wrong partition via global externalTaskStore | U2 matching rule; tests for second project |
|
||||
| Webhook/route degradation if engine undefined | Hard invariant: await cwd engine; no 3s race |
|
||||
| Stale merging status if sweep deferred | Background sweep ASAP after start; self-healing / retry still run |
|
||||
| False OAuth expiry ntfy | Preserve refresh→monitor order in deferred chain |
|
||||
| Plugin routes empty if load deferred naively | Keep plugin load pre-createServer in this plan |
|
||||
| Serve multi-project race on first request to cold project | `onProjectFirstAccessed` + reconciliation already exist |
|
||||
| Measurement noise | U1 before claiming wins |
|
||||
|
||||
**Dependencies:** PostgreSQL embedded binaries available for default boot smoke; existing serve single-store pattern.
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
- **Operators:** Faster dashboard/serve start; logs gain more phase lines.
|
||||
- **Multi-project:** Background engine warmup for non-cwd projects on serve may mean first touch is slightly colder — acceptable under existing on-access ensure.
|
||||
- **Health banner:** Must keep `hasRunningEngine` semantics when another process owns the lock.
|
||||
- **Published CLI:** performance changeset if user-visible.
|
||||
|
||||
## Open Questions (implementation-time)
|
||||
|
||||
- Exact multi-tenant safety of a single TaskStore instance across projectIds under current RLS/bind model — resolve in U2 by reading `TaskStore` bind behavior; default to path/projectId-matched injection if ambiguous.
|
||||
- Which `createServer` closures require notifier/cron **instances** at construction vs lazy getters — inventory during U4 characterization.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- Warm single-project dashboard: measurable drop in `engine: ensureEngine(cwd)` and total `startupDurationMs` after U2+U4.
|
||||
- Serve: time-to-listen no longer scales linearly with registered project count after U3.
|
||||
- Zero dual `createTaskStoreForBackend` for same cwd on dashboard engine-on path after U2.
|
||||
- Boot smoke remains green; no increase in false OAuth expiry notifications.
|
||||
|
||||
## Alternative Approaches Considered
|
||||
|
||||
| Approach | Why not chosen |
|
||||
|----------|----------------|
|
||||
| Race `ensureEngine` with deadline, createServer with optional engine | Previously shipped and regressed webhooks/automations; hard ban |
|
||||
| “UI shell first” without engine handle; rebind routes later | Requires large `createServer` redesign; out of scope |
|
||||
| Always require external `DATABASE_URL` | Good ops tip; not a code fix for default embedded path |
|
||||
| Full min-engine / full-engine process split | Higher complexity; defer after P0–P2 wins measured |
|
||||
|
||||
## Sources & Research
|
||||
|
||||
- Local: `packages/cli/src/commands/dashboard.ts`, `serve.ts`, `packages/core/src/postgres/startup-factory.ts`, `embedded-lifecycle.ts`, `packages/engine/src/project-engine.ts`, `project-engine-manager.ts`, `runtimes/in-process-runtime.ts`, `packages/dashboard/src/server.ts`
|
||||
- Tests: `packages/cli/src/commands/__tests__/serve.test.ts`, `dashboard.test.ts`
|
||||
- Learnings: `docs/solutions/integration-issues/engine-already-running-is-not-no-engine.md`, `docs/solutions/logic-errors/terminal-bootstrap-list-serialized-before-auto-create.md`, `docs/solutions/architecture-patterns/thin-trusted-merge-gate.md`, in-code FNXC on 3s race and CustomProviders post-listen
|
||||
- Prior perf note (query-level only): `docs/performance/dashboard-load.md`
|
||||
- External research: skipped — strong in-repo serve parity pattern and documented readiness constraints
|
||||
|
||||
## Execution Posture
|
||||
|
||||
- U2/U3/U4: characterization-first around store ownership and engine start ordering before moving awaits.
|
||||
- Prefer file-scoped vitest + `pnpm smoke:boot` / `verify:fast` over full-suite.
|
||||
- FNXC comments required for ownership, deferral allowlist, and multi-project injection rules.
|
||||
29
packages/cli/src/__tests__/startup-phase.test.ts
Normal file
29
packages/cli/src/__tests__/startup-phase.test.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { phaseTime } from "../startup-phase.js";
|
||||
|
||||
describe("phaseTime", () => {
|
||||
it("logs duration on success", async () => {
|
||||
const log = vi.fn();
|
||||
const result = await phaseTime("demo", async () => {
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
return 42;
|
||||
}, log, "test");
|
||||
|
||||
expect(result).toBe(42);
|
||||
expect(log).toHaveBeenCalledOnce();
|
||||
expect(log.mock.calls[0][0]).toMatch(/^startup phase demo: \d+ms$/);
|
||||
expect(log.mock.calls[0][1]).toBe("test");
|
||||
});
|
||||
|
||||
it("logs duration when the phase throws", async () => {
|
||||
const log = vi.fn();
|
||||
await expect(
|
||||
phaseTime("boom", async () => {
|
||||
throw new Error("nope");
|
||||
}, log),
|
||||
).rejects.toThrow("nope");
|
||||
|
||||
expect(log).toHaveBeenCalledOnce();
|
||||
expect(log.mock.calls[0][0]).toMatch(/^startup phase boom: \d+ms$/);
|
||||
});
|
||||
});
|
||||
@@ -382,6 +382,10 @@ const mocks = vi.hoisted(() => {
|
||||
return pluginLoader;
|
||||
});
|
||||
|
||||
const pluginRunner = {
|
||||
getRuntimeById: vi.fn(),
|
||||
};
|
||||
|
||||
const authStorage = {
|
||||
getApiKey: vi.fn().mockResolvedValue(undefined),
|
||||
reload: vi.fn(),
|
||||
@@ -536,6 +540,11 @@ const mocks = vi.hoisted(() => {
|
||||
at: new Date().toISOString(),
|
||||
provider: null,
|
||||
})),
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-12:41:
|
||||
Keep the serve startup fixture aligned with the HTTP host contract: createServer receives the engine PluginRunner so model routes can resolve runtime-backed providers while startup remains non-blocking.
|
||||
*/
|
||||
getPluginRunner: vi.fn(() => pluginRunner),
|
||||
startRemoteTunnel: vi.fn(async () => remoteStatus),
|
||||
stopRemoteTunnel: vi.fn(async () => ({ ...remoteStatus, state: "stopped" as const, provider: null, pid: null, url: null })),
|
||||
onMerge: vi.fn().mockResolvedValue(undefined),
|
||||
@@ -681,17 +690,48 @@ vi.mock("@fusion/engine", async (importOriginal) => {
|
||||
ProjectEngine: mocks.projectEngineCtor,
|
||||
ProjectEngineManager: vi.fn().mockImplementation(function (centralCore: any, options: any) {
|
||||
const engines = new Map<string, any>();
|
||||
const starting = new Map<string, Promise<any>>();
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:20:
|
||||
Serve no longer awaits startAll before primary ensureEngine. The mock must
|
||||
create-on-ensure (matching real ProjectEngineManager) so primary resolution
|
||||
works when background startAll has not finished yet.
|
||||
*/
|
||||
const ensureEngine = async (id: string) => {
|
||||
const existing = engines.get(id);
|
||||
if (existing) return existing;
|
||||
const pending = starting.get(id);
|
||||
if (pending) return pending;
|
||||
const promise = (async () => {
|
||||
// Prefer listProjects path (authoritative registry fixture) over getProject,
|
||||
// which some multi-project suite stubs invent as `/repo/${id}`.
|
||||
const listed = await centralCore.listProjects();
|
||||
const fromList = listed.find((p: { id: string }) => p.id === id);
|
||||
const project = fromList ?? (await centralCore.getProject(id));
|
||||
const engine = mocks.projectEngineCtor(
|
||||
{
|
||||
projectId: id,
|
||||
workingDirectory: project?.path ?? `/tmp/${id}`,
|
||||
isolationMode: "in-process",
|
||||
maxConcurrent: 4,
|
||||
maxWorktrees: 10,
|
||||
},
|
||||
centralCore,
|
||||
{ ...options, projectId: id },
|
||||
);
|
||||
await engine.start();
|
||||
engines.set(id, engine);
|
||||
starting.delete(id);
|
||||
return engine;
|
||||
})();
|
||||
starting.set(id, promise);
|
||||
return promise;
|
||||
};
|
||||
return {
|
||||
startAll: vi.fn(async () => {
|
||||
const projects = await centralCore.listProjects();
|
||||
for (const project of projects) {
|
||||
const engine = mocks.projectEngineCtor(
|
||||
{ projectId: project.id, workingDirectory: project.path, isolationMode: "in-process", maxConcurrent: 4, maxWorktrees: 10 },
|
||||
centralCore,
|
||||
{ ...options, projectId: project.id },
|
||||
);
|
||||
await engine.start();
|
||||
engines.set(project.id, engine);
|
||||
await ensureEngine(project.id);
|
||||
}
|
||||
}),
|
||||
// Track which engine is used to verify correct cwd/default routing
|
||||
@@ -701,11 +741,12 @@ vi.mock("@fusion/engine", async (importOriginal) => {
|
||||
}),
|
||||
getAllEngines: vi.fn(() => engines),
|
||||
getStore: vi.fn((id: string) => engines.get(id)?.getTaskStore()),
|
||||
has: vi.fn((id: string) => engines.has(id)),
|
||||
ensureEngine: vi.fn(async (id: string) => engines.get(id)),
|
||||
has: vi.fn((id: string) => engines.has(id) || starting.has(id)),
|
||||
ensureEngine: vi.fn(async (id: string) => ensureEngine(id)),
|
||||
stopAll: vi.fn(async () => {
|
||||
for (const engine of engines.values()) await engine.stop();
|
||||
engines.clear();
|
||||
starting.clear();
|
||||
}),
|
||||
onProjectAccessed: vi.fn(),
|
||||
startReconciliation: vi.fn(),
|
||||
@@ -1202,7 +1243,7 @@ describe("runServe — Plugin wiring", () => {
|
||||
expect(serverOpts).toHaveProperty("pluginStore");
|
||||
expect(serverOpts).toHaveProperty("pluginLoader");
|
||||
expect(serverOpts).toHaveProperty("pluginRunner");
|
||||
expect(serverOpts.pluginRunner).toBe(serverOpts.pluginLoader);
|
||||
expect(serverOpts.pluginRunner).toBe(mocks.projectEngineInstances[0].getPluginRunner());
|
||||
|
||||
await triggerSignal("SIGINT");
|
||||
});
|
||||
|
||||
@@ -108,6 +108,7 @@ import { registerCustomProviders, reregisterCustomProviders } from "./custom-pro
|
||||
import { handleOpencodeGoApiKeySaved, syncStartupModels } from "./startup-model-sync.js";
|
||||
import { DashboardTUI, DashboardLogSink, isTTYAvailable, type SystemInfo, type GitStatus, type GitCommit, type GitCommitDetail, type GitBranch, type GitWorktree, type FileEntry, type FileReadResult, type TaskStep as TUITaskStep, type TaskLogEntry as TUITaskLogEntry, type TaskDetailData, type TaskEvent } from "./dashboard-tui/index.js";
|
||||
import { DASHBOARD_STARTUP_STATUS, runTuiStartupPrelude } from "./dashboard-startup-chain.js";
|
||||
import { phaseTime } from "../startup-phase.js";
|
||||
|
||||
// Re-export for backward compatibility with tests
|
||||
export { promptForPort };
|
||||
@@ -885,7 +886,17 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
// existing store.close() (which closes the AsyncDataLayer) plus the
|
||||
// dashboardBackendShutdown
|
||||
// registered below for embedded-cluster teardown.
|
||||
const dashboardBackendBoot = await createTaskStoreForBackend({ rootDir: cwd });
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Phase timing is permanent and cheap. Factory wall time (embedded PG, schema,
|
||||
optional SQLite migrate) is the first large bucket after the PostgreSQL cutover.
|
||||
*/
|
||||
const logPhase = (message: string, scope = "dashboard") => logSink.log(message, scope);
|
||||
const dashboardBackendBoot = await phaseTime(
|
||||
"backend.factory",
|
||||
() => createTaskStoreForBackend({ rootDir: cwd }),
|
||||
logPhase,
|
||||
);
|
||||
// FNXC:PostgresFinalCutover 2026-07-14-17:20: Dashboard runtime storage is
|
||||
// PostgreSQL-only; factory failure is surfaced instead of creating a dead store.
|
||||
store = dashboardBackendBoot.taskStore;
|
||||
@@ -926,42 +937,27 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
})()
|
||||
: undefined;
|
||||
|
||||
// Phase timing instrumentation — each step logs its wall-clock duration so
|
||||
// we can see at-a-glance which startup phase is the actual bottleneck.
|
||||
// Cheap enough (microsecond reads, one log per phase) to leave on
|
||||
// permanently; lands in the dashboard log buffer and can be diffed across
|
||||
// restarts to spot regressions.
|
||||
const phaseTime = async <T>(label: string, fn: () => Promise<T> | T): Promise<T> => {
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
logSink.log(`startup phase ${label}: ${Date.now() - t0}ms`, "dashboard");
|
||||
}
|
||||
};
|
||||
|
||||
// FNXC:PostgresFinalCutover 2026-07-14-17:20: Initialize the PostgreSQL-backed
|
||||
// store and satellite adapters in dependency order so each receives the live
|
||||
// AsyncDataLayer before watchers and engines begin dispatching work.
|
||||
await phaseTime("store.init", () => store.init());
|
||||
await phaseTime("automationStore.init", () => automationStore.init());
|
||||
// store and satellite adapters so each receives the live AsyncDataLayer before
|
||||
// watchers and engines begin dispatching work.
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
After store.init(), automation / plugin / agent satellite inits are independent
|
||||
of each other and run in parallel. watch() still follows so filesystem events
|
||||
do not race half-initialized satellites.
|
||||
*/
|
||||
await phaseTime("store.init", () => store.init(), logPhase);
|
||||
const pluginStore = store.getPluginStore();
|
||||
await phaseTime("pluginStore.init", () => pluginStore.init());
|
||||
|
||||
// FNXC:PhysicalDeleteSqliteClass 2026-06-26-15:10:
|
||||
// Propagate the backend mode (asyncLayer) from the resolved TaskStore so
|
||||
// AgentStore does not construct a SQLite file under PostgreSQL. Without
|
||||
// this, AgentStore falls into the legacy SQLite path in backend mode and
|
||||
// throws "SQLite Database is not available in backend mode" the first time
|
||||
// any getter touches `this.db`. Mirrors the AutomationStore fix on line ~893
|
||||
// (VAL-CROSS-008 dashboard boot on embedded PostgreSQL). The `?? undefined`
|
||||
// coerces `AsyncDataLayer | null` to the optional option shape.
|
||||
agentStore = new AgentStore({ rootDir: store.getFusionDir(), asyncLayer: dashboardLayer });
|
||||
if (tui) tui.setLoadingStatus(DASHBOARD_STARTUP_STATUS.initializingAgentStore);
|
||||
await phaseTime("agentStore.init", () => agentStore!.init());
|
||||
await phaseTime("satellites.init", () => Promise.all([
|
||||
automationStore.init(),
|
||||
pluginStore.init(),
|
||||
agentStore!.init(),
|
||||
]), logPhase);
|
||||
// store.watch() is filesystem-watcher setup — no DB schema work, safe to
|
||||
// overlap with anything coming after.
|
||||
await phaseTime("store.watch", () => store.watch());
|
||||
await phaseTime("store.watch", () => store.watch(), logPhase);
|
||||
if (tui) tui.setLoadingStatus(DASHBOARD_STARTUP_STATUS.startingAgents);
|
||||
|
||||
// Set up database health check for diagnostics
|
||||
@@ -1667,64 +1663,50 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
agentDir,
|
||||
settingsManager: createReadOnlyProviderSettingsView(cwd, agentDir) as unknown as SettingsManager,
|
||||
});
|
||||
const resolvedPaths = await phaseTime("packageManager.resolve", () => packageManager!.resolve());
|
||||
const resolvedPaths = await phaseTime("packageManager.resolve", () => packageManager!.resolve(), logPhase);
|
||||
const packageExtensionPaths = resolvedPaths.extensions
|
||||
.filter((r) => r.enabled)
|
||||
.map((r) => r.path);
|
||||
|
||||
const claudeCliPaths = await (async () => {
|
||||
try {
|
||||
const globalSettings = await store.getGlobalSettingsStore().getSettings();
|
||||
const result = resolveClaudeCliExtensionPaths(globalSettings);
|
||||
setCachedClaudeCliResolution(result.resolution);
|
||||
if (result.warning) {
|
||||
console.warn(`[extensions] pi-claude-cli: ${result.warning}`);
|
||||
}
|
||||
return result.paths;
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[extensions] Unable to evaluate useClaudeCli setting: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
setCachedClaudeCliResolution(null);
|
||||
return [];
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Claude / Droid / Llama extension toggles all read the same global settings
|
||||
document. One getSettings() replaces three sequential store reads on the
|
||||
dashboard extension critical path.
|
||||
*/
|
||||
let claudeCliPaths: string[] = [];
|
||||
let droidCliPaths: string[] = [];
|
||||
let llamaCppPaths: string[] = [];
|
||||
try {
|
||||
const globalSettings = await store.getGlobalSettingsStore().getSettings();
|
||||
const claude = resolveClaudeCliExtensionPaths(globalSettings);
|
||||
setCachedClaudeCliResolution(claude.resolution);
|
||||
if (claude.warning) {
|
||||
console.warn(`[extensions] pi-claude-cli: ${claude.warning}`);
|
||||
}
|
||||
})();
|
||||
claudeCliPaths = claude.paths;
|
||||
|
||||
const droidCliPaths = await (async () => {
|
||||
try {
|
||||
const globalSettings = await store.getGlobalSettingsStore().getSettings();
|
||||
const result = resolveDroidCliExtensionPaths(globalSettings);
|
||||
setCachedDroidCliResolution(result.resolution);
|
||||
if (result.warning) {
|
||||
console.warn(`[extensions] droid-cli: ${result.warning}`);
|
||||
}
|
||||
return result.paths;
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[extensions] Unable to evaluate useDroidCli setting: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
setCachedDroidCliResolution(null);
|
||||
return [];
|
||||
const droid = resolveDroidCliExtensionPaths(globalSettings);
|
||||
setCachedDroidCliResolution(droid.resolution);
|
||||
if (droid.warning) {
|
||||
console.warn(`[extensions] droid-cli: ${droid.warning}`);
|
||||
}
|
||||
})();
|
||||
droidCliPaths = droid.paths;
|
||||
|
||||
const llamaCppPaths = await (async () => {
|
||||
try {
|
||||
const globalSettings = await store.getGlobalSettingsStore().getSettings();
|
||||
const result = resolveLlamaCppExtensionPaths(globalSettings);
|
||||
setCachedLlamaCppResolution(result.resolution);
|
||||
if (result.warning) {
|
||||
console.warn(`[extensions] llama-cpp: ${result.warning}`);
|
||||
}
|
||||
return result.paths;
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[extensions] Unable to evaluate useLlamaCpp setting: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
setCachedLlamaCppResolution(null);
|
||||
return [];
|
||||
const llama = resolveLlamaCppExtensionPaths(globalSettings);
|
||||
setCachedLlamaCppResolution(llama.resolution);
|
||||
if (llama.warning) {
|
||||
console.warn(`[extensions] llama-cpp: ${llama.warning}`);
|
||||
}
|
||||
})();
|
||||
llamaCppPaths = llama.paths;
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[extensions] Unable to evaluate CLI extension settings: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
setCachedClaudeCliResolution(null);
|
||||
setCachedDroidCliResolution(null);
|
||||
setCachedLlamaCppResolution(null);
|
||||
}
|
||||
|
||||
// Always inject the cli's own extension (`@runfusion/fusion`) so its
|
||||
// `fn_*` tools register globally even when the user hasn't run
|
||||
@@ -1752,7 +1734,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
],
|
||||
cwd,
|
||||
join(cwd, ".fusion", "disabled-auto-extension-discovery"),
|
||||
));
|
||||
), logPhase);
|
||||
|
||||
for (const { path, error } of extensionsResult.errors) {
|
||||
logSink.log(`Failed to load ${path}: ${error}`, "extensions");
|
||||
@@ -1984,7 +1966,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
//
|
||||
const githubClient = new GitHubClient();
|
||||
|
||||
const centralCoreForEngine = await phaseTime("centralCore.init (await)", () => centralCoreInitPromise!);
|
||||
const centralCoreForEngine = await phaseTime("centralCore.init (await)", () => centralCoreInitPromise!, logPhase);
|
||||
|
||||
try {
|
||||
registerGithubTrackingHook?.();
|
||||
@@ -1995,6 +1977,13 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
const resolvedCliPackageVersion = getCliPackageVersion(import.meta.url);
|
||||
const cliPackageVersion = isUnresolvedCliPackageVersion(resolvedCliPackageVersion) ? undefined : resolvedCliPackageVersion;
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Serve parity: inject the dashboard-booted TaskStore so ensureEngine(cwd)
|
||||
reuses the same PostgreSQL pool instead of factory-booting a second store.
|
||||
ProjectEngineManager only applies this store when the project's working
|
||||
directory matches the store root (multi-project safety).
|
||||
*/
|
||||
const engineManager = new ProjectEngineManager(centralCoreForEngine, {
|
||||
cliPackageVersion,
|
||||
getMergeStrategy,
|
||||
@@ -2005,6 +1994,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
prNodeGithubOps: createPrNodeGithubOps(githubClient),
|
||||
prReconcileGithubOps: createPrReconcileGithubOps(githubClient),
|
||||
getTaskMergeBlocker,
|
||||
externalTaskStore: store,
|
||||
});
|
||||
|
||||
// Start engines for all registered projects in the background. The
|
||||
@@ -2069,7 +2059,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
logSink.warn(`Failed to start peer exchange service: ${message}`, "dashboard");
|
||||
}
|
||||
})(),
|
||||
]));
|
||||
]), logPhase);
|
||||
|
||||
logSink.log(
|
||||
`hybrid executor gate: enabled=${hybridGate.enabled} reason=${hybridGate.reason}`,
|
||||
@@ -2084,7 +2074,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
const x = new HybridExecutor(centralCoreForEngine);
|
||||
await x.initialize();
|
||||
return x;
|
||||
});
|
||||
}, logPhase);
|
||||
hybridExecutor = he;
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
@@ -2109,13 +2099,19 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
// duplicate-runtime issue that previously made this 7s+ is gone (see
|
||||
// hybrid-executor-gate change), so warmup typically runs in ~3-5s with
|
||||
// engineManager.startAll() already in flight in parallel.
|
||||
//
|
||||
// FNXC:FasterStartup 2026-07-14-23:55: Do not reintroduce Promise.race with
|
||||
// a deadline. Defer non-route-critical work inside ProjectEngine.start instead.
|
||||
const cwdEngine = cwdRegistered
|
||||
? await phaseTime("engine: ensureEngine(cwd)", () =>
|
||||
engineManager.ensureEngine(cwdRegistered.id).catch((err) => {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
logSink.warn(`Failed to warm cwd project engine: ${message}`, "engine");
|
||||
return undefined;
|
||||
}),
|
||||
? await phaseTime(
|
||||
"engine: ensureEngine(cwd)",
|
||||
() =>
|
||||
engineManager.ensureEngine(cwdRegistered.id).catch((err) => {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
logSink.warn(`Failed to warm cwd project engine: ${message}`, "engine");
|
||||
return undefined;
|
||||
}),
|
||||
logPhase,
|
||||
)
|
||||
: undefined;
|
||||
|
||||
@@ -2138,7 +2134,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
|
||||
// Ensure plugin loading has completed before pluginLoader is handed off
|
||||
// to createServer — routes derived from getPluginRoutes() rely on it.
|
||||
await phaseTime("pluginLoadingPromise (await)", () => pluginLoadingPromise);
|
||||
await phaseTime("pluginLoadingPromise (await)", () => pluginLoadingPromise, logPhase);
|
||||
|
||||
// ── CLI Agent Executor: hub resolver + session transport ─────────────
|
||||
//
|
||||
@@ -2479,7 +2475,7 @@ export async function runDashboard(port: number, opts: { paused?: boolean; dev?:
|
||||
|
||||
// Ensure plugin loading has completed before pluginLoader is handed off
|
||||
// to createServer — routes derived from getPluginRoutes() rely on it.
|
||||
await phaseTime("pluginLoadingPromise (await)", () => pluginLoadingPromise);
|
||||
await phaseTime("pluginLoadingPromise (await)", () => pluginLoadingPromise, logPhase);
|
||||
|
||||
// UI-only mode: no engine, pass individual proxy objects to createServer.
|
||||
//
|
||||
|
||||
@@ -82,6 +82,7 @@ import { registerCustomProviders, reregisterCustomProviders } from "./custom-pro
|
||||
import { handleOpencodeGoApiKeySaved, syncStartupModels } from "./startup-model-sync.js";
|
||||
import { ensureBundledDependencyGraphPluginInstalled, ensureBundledGrokRuntimePluginInstalled, ensureBundledPluginInstalled, isBundledPluginId } from "../plugins/bundled-plugin-install.js";
|
||||
import { ensureCwdProjectRegistered } from "./ensure-project-registered.js";
|
||||
import { phaseTime } from "../startup-phase.js";
|
||||
|
||||
const DIAGNOSTIC_INTERVAL_MS = 30 * 60 * 1000; // 30 minutes
|
||||
let diagnosticIntervalHandle: ReturnType<typeof setInterval> | null = null;
|
||||
@@ -292,11 +293,20 @@ export async function runServe(
|
||||
/*
|
||||
* FNXC:PostgresFinalCutover 2026-07-14-17:20:
|
||||
* Serve must share one successfully booted PostgreSQL layer between CentralCore and the cwd engine. A backend boot error is fatal; constructing a layerless CentralCore would make project discovery appear empty and split control-plane state.
|
||||
*
|
||||
* FNXC:FasterStartup 2026-07-14-23:55:
|
||||
* Phase labels mirror dashboard so time-to-listen can be compared across surfaces.
|
||||
*/
|
||||
const centralBootResult = await createTaskStoreForBackend({ rootDir: cwd });
|
||||
const serveStartedAt = Date.now();
|
||||
const logPhase = (message: string, scope = "serve") => console.log(`[${scope}] ${message}`);
|
||||
const centralBootResult = await phaseTime(
|
||||
"backend.factory",
|
||||
() => createTaskStoreForBackend({ rootDir: cwd }),
|
||||
logPhase,
|
||||
);
|
||||
/*
|
||||
FNXC:MergeQueue 2026-07-15-11:40:
|
||||
Share the serve TaskStore with the host pi extension so agent fn_* tools reuse the engine pool (no dual-boot).
|
||||
FNXC:FasterStartup 2026-07-15-12:38:
|
||||
Preserve both the timed backend-factory phase and the host-store injection during rebases. The serve command must report its critical-path duration without allowing host fn_* tools to boot a second TaskStore pool.
|
||||
*/
|
||||
setHostTaskStore(cwd, centralBootResult.taskStore);
|
||||
let centralBackendShutdownPromise: Promise<void> | undefined;
|
||||
@@ -306,7 +316,7 @@ export async function runServe(
|
||||
};
|
||||
sharedCentralCore = new CentralCore(undefined, { asyncLayer: centralBootResult.asyncLayer });
|
||||
try {
|
||||
await sharedCentralCore.init();
|
||||
await phaseTime("centralCore.init", () => sharedCentralCore!.init(), logPhase);
|
||||
} catch (error) {
|
||||
/* FNXC:PostgresServeLifecycle 2026-07-14-18:03: A failed shared CentralCore boot occurs before serve installs signal teardown, so release the sole shared TaskStore pool and embedded lifecycle here. */
|
||||
await shutdownCentralBackendOnce().catch(() => undefined);
|
||||
@@ -402,11 +412,35 @@ export async function runServe(
|
||||
// FNXC:SqliteFinalRemoval 2026-06-26-11:15: share the central boot's TaskStore
|
||||
// as the externalTaskStore so the cwd engine reuses the same connection pool
|
||||
// (no second embedded PG).
|
||||
// FNXC:FasterStartup 2026-07-14-23:55: Manager only injects this store when
|
||||
// the project's working directory matches the store root (multi-project safe).
|
||||
externalTaskStore: centralBootResult.taskStore,
|
||||
});
|
||||
|
||||
// Start engines for all registered projects eagerly
|
||||
await engineManager.startAll();
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:20:
|
||||
Apply --paused before any ensureEngine/startAll so recovery, merge enqueue,
|
||||
and deferred OAuth side effects observe enginePaused on the shared factory
|
||||
store (cwd path). Idempotent re-apply on the primary store after ensure.
|
||||
Matches dashboard's pause-before-engine ordering.
|
||||
*/
|
||||
if (opts.paused) {
|
||||
await centralBootResult.taskStore.updateSettings({ enginePaused: true });
|
||||
console.log("[engine] Starting in paused mode — automation disabled");
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Do not await startAll() before listen — multi-project count must not gate
|
||||
HTTP readiness. Warm the primary project (cwd / flag / default) fully so
|
||||
createServer receives a live engine; other projects continue via startAll
|
||||
+ reconciliation in the background.
|
||||
*/
|
||||
void engineManager.startAll().catch((err) => {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
console.warn(`[serve] Background startAll failed: ${message}`);
|
||||
});
|
||||
engineManager.startReconciliation();
|
||||
|
||||
let hybridExecutor: HybridExecutor | null = null;
|
||||
const hybridGate = await shouldUseHybridExecutor(sharedCentralCore);
|
||||
@@ -435,14 +469,6 @@ export async function runServe(
|
||||
}
|
||||
})();
|
||||
|
||||
// Start background reconciliation to detect and start engines for projects
|
||||
// registered after startup (without requiring headless node API access).
|
||||
// This ensures project task execution starts from backend runtime alone.
|
||||
// The onProjectFirstAccessed callback in createServer remains as a fast-path
|
||||
// fallback for immediate engine startup on project access, but it is NOT
|
||||
// required for correctness — reconciliation handles all cases.
|
||||
engineManager.startReconciliation();
|
||||
|
||||
// ── PeerExchangeService: gossip protocol for mesh peer discovery ──────
|
||||
//
|
||||
// Periodically exchanges peer information with connected remote nodes
|
||||
@@ -460,52 +486,61 @@ export async function runServe(
|
||||
}
|
||||
}
|
||||
|
||||
const startedEngines = [...engineManager.getAllEngines().values()];
|
||||
const projects = sharedCentralCore ? await sharedCentralCore.listProjects() : [];
|
||||
|
||||
const resolvePrimaryEngine = async (): Promise<{
|
||||
engine: (typeof startedEngines)[number];
|
||||
engine: import("@fusion/engine").ProjectEngine;
|
||||
source: "cli-flag" | "default-setting" | "cwd" | "fallback";
|
||||
} | null> => {
|
||||
const ensure = async (projectId: string, source: "cli-flag" | "default-setting" | "cwd" | "fallback") => {
|
||||
const engine = await phaseTime(
|
||||
`engine: ensureEngine(${source})`,
|
||||
() => engineManager.ensureEngine(projectId),
|
||||
logPhase,
|
||||
);
|
||||
return { engine, source };
|
||||
};
|
||||
|
||||
if (opts.project) {
|
||||
const byId = startedEngines.find((engine) => engine.getProjectId() === opts.project);
|
||||
if (byId) {
|
||||
return { engine: byId, source: "cli-flag" };
|
||||
const byIdProject = projects.find((project) => project.id === opts.project);
|
||||
if (byIdProject) {
|
||||
return ensure(byIdProject.id, "cli-flag");
|
||||
}
|
||||
|
||||
const projectMatch = projects.find((project) => project.name === opts.project);
|
||||
if (projectMatch) {
|
||||
const byName = engineManager.getEngine(projectMatch.id);
|
||||
if (byName) {
|
||||
return { engine: byName, source: "cli-flag" };
|
||||
}
|
||||
return ensure(projectMatch.id, "cli-flag");
|
||||
}
|
||||
|
||||
console.error(`[serve] --project "${opts.project}" did not match any started engine`);
|
||||
console.error(`[serve] --project "${opts.project}" did not match any registered project`);
|
||||
process.exit(1);
|
||||
return null;
|
||||
}
|
||||
|
||||
const defaultProjectId = await sharedCentralCore?.getDefaultProjectId?.();
|
||||
if (defaultProjectId) {
|
||||
const defaultEngine = engineManager.getEngine(defaultProjectId);
|
||||
if (defaultEngine) {
|
||||
return { engine: defaultEngine, source: "default-setting" };
|
||||
try {
|
||||
return await ensure(defaultProjectId, "default-setting");
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
console.warn(`[serve] defaultProjectId ${defaultProjectId} failed to start — falling through: ${message}`);
|
||||
}
|
||||
console.warn(`[serve] defaultProjectId ${defaultProjectId} is set but no engine started for it — falling through`);
|
||||
}
|
||||
|
||||
const cwdEngine = ntfyProjectId ? engineManager.getEngine(ntfyProjectId) : undefined;
|
||||
if (cwdEngine) {
|
||||
return { engine: cwdEngine, source: "cwd" };
|
||||
if (ntfyProjectId) {
|
||||
try {
|
||||
return await ensure(ntfyProjectId, "cwd");
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
console.warn(`[serve] cwd project engine failed to start — falling through: ${message}`);
|
||||
}
|
||||
}
|
||||
|
||||
const fallback = startedEngines[0];
|
||||
if (!fallback) {
|
||||
const fallbackProject = projects[0];
|
||||
if (!fallbackProject) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return { engine: fallback, source: "fallback" };
|
||||
return ensure(fallbackProject.id, "fallback");
|
||||
};
|
||||
|
||||
const primarySelection = await resolvePrimaryEngine();
|
||||
@@ -534,9 +569,10 @@ export async function runServe(
|
||||
// Set up database health check for diagnostics
|
||||
setServeDbHealthCheck(() => store.healthCheck());
|
||||
|
||||
// Re-apply pause on the primary store when it is not the factory/cwd share
|
||||
// (non-cwd --project / fallback). No-op when already set on the shared store.
|
||||
if (opts.paused) {
|
||||
await store.updateSettings({ enginePaused: true });
|
||||
console.log("[engine] Starting in paused mode — automation disabled");
|
||||
}
|
||||
|
||||
// ── PluginStore: plugin installation management ─────────────────────
|
||||
@@ -1017,6 +1053,7 @@ export async function runServe(
|
||||
});
|
||||
|
||||
const actualPort = (server.address() as AddressInfo).port;
|
||||
logPhase(`startup phase time-to-listen: ${Date.now() - serveStartedAt}ms`);
|
||||
|
||||
/*
|
||||
FNXC:CustomProviders 2026-06-30-00:00:
|
||||
|
||||
29
packages/cli/src/startup-phase.ts
Normal file
29
packages/cli/src/startup-phase.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* Shared startup phase timing for CLI surfaces (dashboard, serve).
|
||||
*
|
||||
* FNXC:FasterStartup 2026-07-14-23:55:
|
||||
* Operators and developers need wall-clock labels for each boot phase so
|
||||
* time-to-listen regressions are attributable. Dashboard already had an
|
||||
* inline phaseTime helper; serve and factory/engine paths need the same
|
||||
* cheap pattern without inventing a separate metrics product.
|
||||
*/
|
||||
|
||||
export type StartupPhaseLogger = (message: string, scope?: string) => void;
|
||||
|
||||
/**
|
||||
* Time an async or sync startup phase and log `startup phase <label>: Nms`.
|
||||
* Always logs in `finally` so failures still surface their duration.
|
||||
*/
|
||||
export async function phaseTime<T>(
|
||||
label: string,
|
||||
fn: () => Promise<T> | T,
|
||||
log: StartupPhaseLogger,
|
||||
scope = "startup",
|
||||
): Promise<T> {
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
log(`startup phase ${label}: ${Date.now() - t0}ms`, scope);
|
||||
}
|
||||
}
|
||||
@@ -422,9 +422,17 @@ export async function createTaskStoreForBackend(
|
||||
}
|
||||
const rootDir = options.rootDir ?? "";
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Factory substep timings feed CLI phase logs so operators can separate embedded
|
||||
PG start + schema work from engine bring-up. Cheap permanent diagnostics.
|
||||
*/
|
||||
const factoryT0 = Date.now();
|
||||
let boot: SchemaBackendBootResult;
|
||||
try {
|
||||
const schemaT0 = Date.now();
|
||||
boot = await bootSchemaBackend(options);
|
||||
log.log(`startup phase backend.schemaBackend: ${Date.now() - schemaT0}ms`);
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`startup-factory: failed to initialize PostgreSQL schema backend: ${err instanceof Error ? err.message : String(err)}`,
|
||||
@@ -721,6 +729,7 @@ export async function createTaskStoreForBackend(
|
||||
*/
|
||||
let taskStore: TaskStore;
|
||||
try {
|
||||
const constructT0 = Date.now();
|
||||
if (options.projectId && !options.rootDir) {
|
||||
taskStore = await TaskStore.getOrCreateForProject(
|
||||
options.projectId,
|
||||
@@ -734,6 +743,7 @@ export async function createTaskStoreForBackend(
|
||||
});
|
||||
await taskStore.init();
|
||||
}
|
||||
log.log(`startup phase backend.taskStore.construct: ${Date.now() - constructT0}ms`);
|
||||
} catch (err) {
|
||||
await asyncLayer.close().catch(() => undefined);
|
||||
if (embeddedLifecycle) {
|
||||
@@ -745,6 +755,7 @@ export async function createTaskStoreForBackend(
|
||||
}`,
|
||||
);
|
||||
}
|
||||
log.log(`startup phase backend.factory.total: ${Date.now() - factoryT0}ms`);
|
||||
|
||||
/*
|
||||
FNXC:PluginPostgresContract 2026-07-14-18:32:
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
/**
|
||||
* FNXC:FasterStartup 2026-07-14-23:55 / 2026-07-15-00:20:
|
||||
* ProjectEngine.start returns before notifiers/OAuth/merge enqueue finish, but
|
||||
* OAuth refresh must still start before the expiry monitor when deferred work runs.
|
||||
* Stale merging/merging-pr statuses clear on the critical path.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
|
||||
const oauthRefreshStart = vi.fn(async () => undefined);
|
||||
const oauthExpiryStart = vi.fn(async () => undefined);
|
||||
const notificationStart = vi.fn(async () => undefined);
|
||||
const notifierStart = vi.fn(async () => undefined);
|
||||
const oauthValidityStart = vi.fn(async () => undefined);
|
||||
const updateTask = vi.fn(async () => undefined);
|
||||
const listTasks = vi.fn(async () => [] as Array<{ id: string; column: string; status: string | null }>);
|
||||
|
||||
vi.mock("../notification/index.js", () => ({
|
||||
NotificationService: vi.fn().mockImplementation(function () {
|
||||
return { start: notificationStart, stop: vi.fn() };
|
||||
}),
|
||||
OAuthAlertStateStore: vi.fn().mockImplementation(function () {
|
||||
return {};
|
||||
}),
|
||||
OAuthExpiryMonitor: vi.fn().mockImplementation(function () {
|
||||
return { start: oauthExpiryStart, stop: vi.fn() };
|
||||
}),
|
||||
OAuthRefreshScheduler: vi.fn().mockImplementation(function () {
|
||||
return { start: oauthRefreshStart, stop: vi.fn() };
|
||||
}),
|
||||
OAuthValidityLogger: vi.fn().mockImplementation(function () {
|
||||
return { start: oauthValidityStart, stop: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../notifier.js", () => ({
|
||||
NtfyNotifier: vi.fn().mockImplementation(function () {
|
||||
return { start: notifierStart, stop: vi.fn(), notifyGridlock: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../auth-storage.js", () => ({
|
||||
createFusionAuthStorage: vi.fn(() => ({})),
|
||||
getFusionOAuthAlertStatePath: () => "/tmp/fusion-oauth-alert-state-test",
|
||||
}));
|
||||
|
||||
vi.mock("../runtimes/in-process-runtime.js", () => ({
|
||||
InProcessRuntime: vi.fn().mockImplementation(function () {
|
||||
return {
|
||||
start: vi.fn(async () => undefined),
|
||||
stop: vi.fn(async () => undefined),
|
||||
getTaskStore: vi.fn(() => ({
|
||||
getSettings: vi.fn(async () => ({})),
|
||||
getAsyncLayer: vi.fn(() => ({})),
|
||||
listTasks,
|
||||
updateTask,
|
||||
on: vi.fn(),
|
||||
off: vi.fn(),
|
||||
})),
|
||||
getMessageStore: vi.fn(() => undefined),
|
||||
getAgentStore: vi.fn(() => undefined),
|
||||
getPluginRunner: vi.fn(() => undefined),
|
||||
configurePrMonitoring: vi.fn(),
|
||||
getExecutor: vi.fn(() => undefined),
|
||||
};
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../gridlock-detector.js", () => ({
|
||||
GridlockDetector: vi.fn().mockImplementation(function () {
|
||||
return { start: vi.fn(), stop: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../cron-runner.js", () => ({
|
||||
CronRunner: vi.fn().mockImplementation(function () {
|
||||
return { start: vi.fn(), stop: vi.fn() };
|
||||
}),
|
||||
createAiPromptExecutor: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
vi.mock("@fusion/core", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("@fusion/core")>();
|
||||
return {
|
||||
...actual,
|
||||
AutomationStore: vi.fn().mockImplementation(function () {
|
||||
return { init: vi.fn(async () => undefined) };
|
||||
}),
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock("../pr-monitor.js", () => ({
|
||||
PrMonitor: vi.fn().mockImplementation(function () {
|
||||
return { onNewComments: vi.fn(), start: vi.fn(), stop: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../pr-comment-handler.js", () => ({
|
||||
PrCommentHandler: vi.fn().mockImplementation(function () {
|
||||
return { handleNewComments: vi.fn(), createFollowUpTask: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../pr-reconcile.js", () => ({
|
||||
PrReconciler: vi.fn().mockImplementation(function () {
|
||||
return { start: vi.fn(), stop: vi.fn() };
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../planner-overseer.js", () => ({
|
||||
PlannerOverseerMonitor: vi.fn().mockImplementation(function () {
|
||||
return {};
|
||||
}),
|
||||
resolveExecutorStuckAfterMs: vi.fn(() => 60_000),
|
||||
}));
|
||||
|
||||
vi.mock("../planner-recovery-controller.js", () => ({
|
||||
PlannerRecoveryController: vi.fn().mockImplementation(function () {
|
||||
return {};
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("../postgres-migration-notice.js", () => ({
|
||||
deliverPostgresMigrationNoticeIfNeeded: vi.fn(async () => undefined),
|
||||
deliverPostgresMigrationCompleteNoticeIfNeeded: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
vi.mock("../merger.js", () => ({
|
||||
sweepStaleAutostashes: vi.fn(async () => 0),
|
||||
VerificationError: class VerificationError extends Error {},
|
||||
}));
|
||||
|
||||
import { ProjectEngine } from "../project-engine.js";
|
||||
|
||||
function makeEngine(projectId: string, skipNotifier: boolean) {
|
||||
return new ProjectEngine(
|
||||
{
|
||||
projectId,
|
||||
workingDirectory: `/tmp/${projectId}`,
|
||||
isolationMode: "in-process",
|
||||
maxConcurrent: 1,
|
||||
maxWorktrees: 1,
|
||||
},
|
||||
{ on: vi.fn(), off: vi.fn() } as any,
|
||||
{ skipNotifier, projectId },
|
||||
);
|
||||
}
|
||||
|
||||
describe("ProjectEngine deferred startup", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
listTasks.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
it("clears stale merging statuses during start critical path", async () => {
|
||||
listTasks.mockResolvedValue([
|
||||
{ id: "FN-1", column: "in-review", status: "merging" },
|
||||
{ id: "FN-2", column: "in-review", status: "merging-pr" },
|
||||
{ id: "FN-3", column: "in-review", status: null },
|
||||
]);
|
||||
|
||||
const engine = makeEngine("proj_stale", true);
|
||||
await engine.start();
|
||||
|
||||
expect(updateTask).toHaveBeenCalledWith("FN-1", { status: null });
|
||||
expect(updateTask).toHaveBeenCalledWith("FN-2", { status: null });
|
||||
expect(updateTask).not.toHaveBeenCalledWith("FN-3", expect.anything());
|
||||
});
|
||||
|
||||
it("returns from start before OAuth refresh completes, then runs refresh before expiry monitor", async () => {
|
||||
let resolveRefresh!: () => void;
|
||||
const refreshGate = new Promise<void>((resolve) => {
|
||||
resolveRefresh = resolve;
|
||||
});
|
||||
oauthRefreshStart.mockImplementation(async () => {
|
||||
await refreshGate;
|
||||
});
|
||||
|
||||
const engine = makeEngine("proj_deferred", false);
|
||||
|
||||
const startPromise = engine.start();
|
||||
await expect(startPromise).resolves.toBeUndefined();
|
||||
|
||||
// Deferred work is scheduled after start resolves; wait for refresh to be entered.
|
||||
await vi.waitFor(() => {
|
||||
expect(oauthRefreshStart).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// Expiry monitor must not start until refresh finishes
|
||||
expect(oauthExpiryStart).not.toHaveBeenCalled();
|
||||
|
||||
resolveRefresh();
|
||||
await vi.waitFor(() => {
|
||||
expect(oauthExpiryStart).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
expect(notificationStart).toHaveBeenCalled();
|
||||
expect(oauthRefreshStart.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
oauthExpiryStart.mock.invocationCallOrder[0]!,
|
||||
);
|
||||
});
|
||||
|
||||
it("stop sets shuttingDown so deferred work exits without throwing", async () => {
|
||||
let resolveRefresh!: () => void;
|
||||
oauthRefreshStart.mockImplementation(
|
||||
() =>
|
||||
new Promise<void>((resolve) => {
|
||||
resolveRefresh = resolve;
|
||||
}),
|
||||
);
|
||||
|
||||
const engine = makeEngine("proj_stop", false);
|
||||
await engine.start();
|
||||
await vi.waitFor(() => {
|
||||
expect(oauthRefreshStart).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
await engine.stop();
|
||||
resolveRefresh();
|
||||
// Deferred chain should observe shuttingDown and not reject after stop.
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
expect(oauthExpiryStart).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -110,6 +110,45 @@ describe("ProjectEngineManager", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("injects externalTaskStore only when project working directory matches store root", async () => {
|
||||
const sharedStore = {
|
||||
getRootDir: () => "/mapped/proj_aaa",
|
||||
} as any;
|
||||
const manager = new ProjectEngineManager(centralCore, {
|
||||
externalTaskStore: sharedStore,
|
||||
});
|
||||
|
||||
await manager.ensureEngine("proj_aaa");
|
||||
expect(ProjectEngine).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ workingDirectory: "/mapped/proj_aaa" }),
|
||||
centralCore,
|
||||
expect.objectContaining({ externalTaskStore: sharedStore }),
|
||||
);
|
||||
|
||||
await manager.ensureEngine("proj_bbb");
|
||||
expect(ProjectEngine).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ workingDirectory: "/mapped/proj_bbb" }),
|
||||
centralCore,
|
||||
expect.not.objectContaining({ externalTaskStore: sharedStore }),
|
||||
);
|
||||
});
|
||||
|
||||
it("shares externalTaskStore when roots differ only by trailing slash", async () => {
|
||||
const sharedStore = {
|
||||
getRootDir: () => "/mapped/proj_aaa/",
|
||||
} as any;
|
||||
const manager = new ProjectEngineManager(centralCore, {
|
||||
externalTaskStore: sharedStore,
|
||||
});
|
||||
|
||||
await manager.ensureEngine("proj_aaa");
|
||||
expect(ProjectEngine).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ workingDirectory: "/mapped/proj_aaa" }),
|
||||
centralCore,
|
||||
expect.objectContaining({ externalTaskStore: sharedStore }),
|
||||
);
|
||||
});
|
||||
|
||||
it("returns existing engine on repeated calls", async () => {
|
||||
const manager = new ProjectEngineManager(centralCore);
|
||||
const engine1 = await manager.ensureEngine("proj_aaa");
|
||||
|
||||
@@ -13,6 +13,8 @@
|
||||
* - Graceful shutdown of all engines via `stopAll()`
|
||||
*/
|
||||
|
||||
import { realpathSync } from "node:fs";
|
||||
import { resolve as pathResolve } from "node:path";
|
||||
import type {
|
||||
CentralCore,
|
||||
TaskStore,
|
||||
@@ -47,8 +49,16 @@ export interface EngineManagerOptions {
|
||||
prReconcileGithubOps?: ProjectEngineOptions["prReconcileGithubOps"];
|
||||
getTaskMergeBlocker?: ProjectEngineOptions["getTaskMergeBlocker"];
|
||||
onInsightRunProcessed?: ProjectEngineOptions["onInsightRunProcessed"];
|
||||
// FNXC:SqliteFinalRemoval 2026-06-26-11:20: shared TaskStore from the central
|
||||
// backend boot so all engines reuse one connection pool (no second embedded PG).
|
||||
/**
|
||||
* FNXC:SqliteFinalRemoval 2026-06-26-11:20: shared TaskStore from the central
|
||||
* backend boot so engines reuse one connection pool (no second embedded PG).
|
||||
*
|
||||
* FNXC:FasterStartup 2026-07-14-23:55:
|
||||
* Inject only for the project whose resolved working directory matches this
|
||||
* store's rootDir. Multi-project engines must factory-boot (or receive) their
|
||||
* own bound store — a cwd-partitioned TaskStore must never back a different
|
||||
* project root. Callers may still pass per-call overrides via ensureEngine.
|
||||
*/
|
||||
externalTaskStore?: ProjectEngineOptions["externalTaskStore"];
|
||||
}
|
||||
|
||||
@@ -469,7 +479,7 @@ export class ProjectEngineManager {
|
||||
}
|
||||
|
||||
const runtimeConfig = await this.buildRuntimeConfig(project);
|
||||
const engineOptions = this.buildEngineOptions(project, overrides);
|
||||
const engineOptions = this.buildEngineOptions(project, runtimeConfig.workingDirectory, overrides);
|
||||
|
||||
// Acquire the per-machine singleton guard before spinning up any engine
|
||||
// subsystems. This prevents two fusion processes from running engines for
|
||||
@@ -548,8 +558,21 @@ export class ProjectEngineManager {
|
||||
|
||||
private buildEngineOptions(
|
||||
project: RegisteredProject,
|
||||
workingDirectory: string,
|
||||
overrides?: Partial<ProjectEngineOptions>,
|
||||
): ProjectEngineOptions {
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55 / 2026-07-15-00:40:
|
||||
Share the CLI-booted TaskStore only when the engine's working directory is
|
||||
the same project root as the store. Compare realpath when available so a
|
||||
symlinked CLI cwd and a registry-canonical path still share one pool
|
||||
(Greptile: path.resolve alone double-boots symlink aliases).
|
||||
*/
|
||||
const sharedStore = this.options.externalTaskStore;
|
||||
const shareForThisProject = Boolean(
|
||||
sharedStore
|
||||
&& sameProjectRoot(sharedStore.getRootDir(), workingDirectory),
|
||||
);
|
||||
return {
|
||||
projectId: project.id,
|
||||
cliPackageVersion: this.options.cliPackageVersion,
|
||||
@@ -561,11 +584,25 @@ export class ProjectEngineManager {
|
||||
prReconcileGithubOps: this.options.prReconcileGithubOps,
|
||||
getTaskMergeBlocker: this.options.getTaskMergeBlocker,
|
||||
onInsightRunProcessed: this.options.onInsightRunProcessed,
|
||||
// FNXC:SqliteFinalRemoval 2026-06-26-11:20: forward the shared external
|
||||
// TaskStore so engines reuse the central boot's connection pool instead
|
||||
// of starting a second embedded PostgreSQL on the same data dir.
|
||||
...(this.options.externalTaskStore ? { externalTaskStore: this.options.externalTaskStore } : {}),
|
||||
...(shareForThisProject && sharedStore ? { externalTaskStore: sharedStore } : {}),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:FasterStartup 2026-07-15-00:40:
|
||||
* Path identity for externalTaskStore matching: resolve then realpath so
|
||||
* symlinked project roots compare equal to their canonical registry path.
|
||||
*/
|
||||
function sameProjectRoot(a: string, b: string): boolean {
|
||||
const normalize = (p: string): string => {
|
||||
const resolved = pathResolve(p);
|
||||
try {
|
||||
return realpathSync(resolved);
|
||||
} catch {
|
||||
return resolved;
|
||||
}
|
||||
};
|
||||
return normalize(a) === normalize(b);
|
||||
}
|
||||
|
||||
@@ -497,6 +497,13 @@ export class ProjectEngine {
|
||||
// would let a second caller overwrite the first, stranding its promise.
|
||||
private manualMergeResolvers = new Map<string, Array<MergeResolver>>();
|
||||
private shuttingDown = false;
|
||||
/**
|
||||
* FNXC:FasterStartup 2026-07-15-00:20:
|
||||
* stop() clears shuttingDown so the engine can restart, which would otherwise
|
||||
* let in-flight deferred startup work resume after stop. Bump this generation
|
||||
* on stop (and capture it when scheduling deferred work) so post-stop tails abort.
|
||||
*/
|
||||
private startupGeneration = 0;
|
||||
|
||||
private addMergeResolver(taskId: string, r: MergeResolver): void {
|
||||
const list = this.manualMergeResolvers.get(taskId);
|
||||
@@ -880,10 +887,21 @@ export class ProjectEngine {
|
||||
this.prReconciler.start();
|
||||
}
|
||||
|
||||
// 3. Initialize notification services (unless caller manages them externally)
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-14-23:55:
|
||||
Route-critical path constructs AutomationStore/CronRunner and wires merge
|
||||
listeners so createServer closures bind real subsystems. Notifiers, OAuth
|
||||
(refresh-before-monitor order preserved), automation schedule syncs, and
|
||||
startupMergeSweep run in the background so ensureEngine returns sooner.
|
||||
Do not reintroduce a timed race that hands createServer an undefined engine.
|
||||
*/
|
||||
const engineStartT0 = Date.now();
|
||||
|
||||
// 3. Construct notification services (start deferred — see startDeferredStartupWork)
|
||||
let deferredAgentNameResolver: ((agentId: string) => Promise<string | null>) | undefined;
|
||||
if (!this.options.skipNotifier) {
|
||||
const agentStore = this.runtime.getAgentStore();
|
||||
const agentNameResolver = agentStore
|
||||
deferredAgentNameResolver = agentStore
|
||||
? async (agentId: string): Promise<string | null> => {
|
||||
const agent = await agentStore.getAgent(agentId);
|
||||
const name = typeof agent?.name === "string" ? agent.name.trim() : "";
|
||||
@@ -895,59 +913,20 @@ export class ProjectEngine {
|
||||
projectId: this.options.projectId,
|
||||
ntfyBaseUrl: this.options.ntfyBaseUrl,
|
||||
messageStore: this.runtime.getMessageStore(),
|
||||
agentNameResolver,
|
||||
agentNameResolver: deferredAgentNameResolver,
|
||||
});
|
||||
await this.notificationService.start();
|
||||
const authStorage = createFusionAuthStorage();
|
||||
this.authStorage = authStorage;
|
||||
const oauthAlertState = new OAuthAlertStateStore({
|
||||
statePath: getFusionOAuthAlertStatePath(),
|
||||
});
|
||||
/*
|
||||
FNXC:ClaudeOAuth 2026-07-05-00:00:
|
||||
FN-7574: proactively refresh OAuth access tokens ahead of expiry (widened window,
|
||||
see OAUTH_REFRESH_BUFFER_MS in auth-storage.ts) so a healthy subscription session
|
||||
never lapses waiting for something else to request a runtime API key. Reuses the
|
||||
same authStorage instance as OAuthExpiryMonitor below so detection/notification and
|
||||
proactive refresh observe a consistent, single credential source.
|
||||
|
||||
FNXC:ClaudeOAuth 2026-07-08-12:10:
|
||||
Start the proactive refresher BEFORE OAuthExpiryMonitor. Both are awaited on startup
|
||||
and share this authStorage; the monitor's OAuthExpiryMonitor.start() runs its first
|
||||
check() synchronously, and that check is refresh-blind (it only reads the stored
|
||||
`expires` timestamp, with no refresh token or getApiKey in its interface). If the
|
||||
monitor ran first, a stale-but-refreshable access token (the normal state after the
|
||||
app has been closed a while) fired a false "OAuth token expired" ntfy push even
|
||||
though the connection was fine — the refresher would silently renew the token moments
|
||||
later. Refreshing first means the scheduler's awaited initial tick() renews the token
|
||||
and the monitor (which reload()s authStorage at the top of check()) sees the fresh
|
||||
`expires`, so the alarm only fires when refresh genuinely fails.
|
||||
*/
|
||||
this.oauthRefreshScheduler = new OAuthRefreshScheduler({ authStorage });
|
||||
await this.oauthRefreshScheduler.start();
|
||||
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: this.notificationService,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthExpiryMonitor.start();
|
||||
this.oauthValidityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthValidityLogger.start();
|
||||
|
||||
// Backward-compatibility shim for gridlock notifications.
|
||||
// Backward-compatibility shim for gridlock notifications (started in deferred work).
|
||||
this.notifier = new NtfyNotifier(
|
||||
store,
|
||||
{
|
||||
projectId: this.options.projectId,
|
||||
ntfyBaseUrl: this.options.ntfyBaseUrl,
|
||||
agentNameResolver,
|
||||
agentNameResolver: deferredAgentNameResolver,
|
||||
},
|
||||
this.notificationService,
|
||||
);
|
||||
await this.notifier.start();
|
||||
}
|
||||
|
||||
this.gridlockDetector = new GridlockDetector(store, {
|
||||
@@ -956,13 +935,14 @@ export class ProjectEngine {
|
||||
});
|
||||
this.gridlockDetector.start();
|
||||
|
||||
// 4. Initialize AutomationStore + CronRunner
|
||||
// 4. Initialize AutomationStore + CronRunner (syncs deferred)
|
||||
this.setAutomationSubsystemHealth(
|
||||
"initializing",
|
||||
"Initializing AutomationStore and CronRunner",
|
||||
);
|
||||
let coreAutomationModule: typeof import("@fusion/core") | undefined;
|
||||
try {
|
||||
const coreAutomationModule = await import("@fusion/core");
|
||||
coreAutomationModule = await import("@fusion/core");
|
||||
const { AutomationStore } = coreAutomationModule;
|
||||
// FNXC:PhysicalDeleteSqliteClass 2026-06-26-14:05:
|
||||
// Propagate the backend mode (asyncLayer) from the owning TaskStore so
|
||||
@@ -984,76 +964,17 @@ export class ProjectEngine {
|
||||
scope: "project", // Project-scoped execution — global schedules run separately
|
||||
});
|
||||
|
||||
const settings = await store.getSettings();
|
||||
const startupSyncFailures: string[] = [];
|
||||
|
||||
// Sync insight extraction automation on startup
|
||||
if (typeof coreAutomationModule.syncInsightExtractionAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncInsightExtractionAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`insight extraction: ${message}`);
|
||||
runtimeLog.warn(`Insight extraction automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncInsightExtractionAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
// Sync auto-summarize automation on startup
|
||||
if (typeof coreAutomationModule.syncAutoSummarizeAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncAutoSummarizeAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`auto-summarize: ${message}`);
|
||||
runtimeLog.warn(`Auto-summarize automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncAutoSummarizeAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
// Sync memory dreams automation on startup
|
||||
if (typeof coreAutomationModule.syncMemoryDreamsAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncMemoryDreamsAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`memory dreams: ${message}`);
|
||||
runtimeLog.warn(`Memory dreams automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncMemoryDreamsAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
// Sync scheduled eval batch automation on startup
|
||||
if (typeof coreAutomationModule.syncScheduledEvalBatchAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncScheduledEvalBatchAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`scheduled eval: ${message}`);
|
||||
runtimeLog.warn(`Scheduled eval automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncScheduledEvalBatchAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
this.cronRunner.start();
|
||||
|
||||
if (startupSyncFailures.length > 0) {
|
||||
this.setAutomationSubsystemHealth(
|
||||
"degraded",
|
||||
`CronRunner started with startup sync warnings: ${startupSyncFailures.join("; ")}`,
|
||||
);
|
||||
} else {
|
||||
this.setAutomationSubsystemHealth(
|
||||
"ready",
|
||||
"CronRunner initialized and startup automation sync completed",
|
||||
);
|
||||
}
|
||||
|
||||
runtimeLog.log("CronRunner initialized and started");
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:40:
|
||||
Do not start CronRunner until deferred automation schedule syncs finish.
|
||||
start() ticks immediately; running it before sync can fire one overdue
|
||||
schedule with stale settings from the previous process (Greptile P1).
|
||||
*/
|
||||
this.setAutomationSubsystemHealth(
|
||||
"initializing",
|
||||
"AutomationStore ready; CronRunner starts after schedule sync",
|
||||
);
|
||||
runtimeLog.log("AutomationStore initialized; CronRunner start deferred until schedule sync");
|
||||
} catch (err) {
|
||||
// Non-fatal — automations are optional
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
@@ -1085,10 +1006,25 @@ export class ProjectEngine {
|
||||
this.wireTaskPauseMergeInterruption(store);
|
||||
this.wireAutostashOrphanRecovery(store);
|
||||
|
||||
// 7. Auto-merge startup sweep
|
||||
await this.startupMergeSweep(store);
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:20:
|
||||
Clear crash-leftover merging/merging-pr statuses on the critical path so
|
||||
manual merge is not blocked while deferred work finishes. Auto-merge enqueue
|
||||
stays deferred (pause-aware) after the engine handle is returnable.
|
||||
*/
|
||||
const statusClearT0 = Date.now();
|
||||
await this.clearStaleMergingStatuses(store);
|
||||
runtimeLog.log(`ProjectEngine stale merging status clear: ${Date.now() - statusClearT0}ms`);
|
||||
|
||||
// 8. Start periodic merge retry sweep
|
||||
// 7–9. Deferred: notifiers/OAuth (ordered), automation syncs, merge enqueue
|
||||
const deferredGeneration = this.startupGeneration;
|
||||
void this.startDeferredStartupWork(store, coreAutomationModule, deferredGeneration).catch((err) => {
|
||||
if (this.shuttingDown || this.startupGeneration !== deferredGeneration) return;
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
runtimeLog.error(`Deferred ProjectEngine startup work failed: ${message}`);
|
||||
});
|
||||
|
||||
// 8. Start periodic merge retry sweep (does not require merge enqueue to have finished)
|
||||
this.scheduleMergeRetry(store);
|
||||
this.scheduleMergeActiveReconciliation(settings.maintenanceIntervalMs ?? 900_000);
|
||||
|
||||
@@ -1097,7 +1033,171 @@ export class ProjectEngine {
|
||||
this.scheduleStaleAutostashSweep(store);
|
||||
|
||||
this.started = true;
|
||||
runtimeLog.log(`ProjectEngine started for ${this.config.projectId}`);
|
||||
runtimeLog.log(
|
||||
`ProjectEngine started for ${this.config.projectId} (critical path ${Date.now() - engineStartT0}ms; deferred work in background)`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-route-critical startup work. Runs after the engine handle is returnable.
|
||||
* OAuth refresh must still complete before the expiry monitor's first check.
|
||||
* Aborts cleanly when stop() sets shuttingDown.
|
||||
*/
|
||||
private deferredStartupAborted(generation: number): boolean {
|
||||
return this.shuttingDown || this.startupGeneration !== generation;
|
||||
}
|
||||
|
||||
private async startDeferredStartupWork(
|
||||
store: TaskStore,
|
||||
coreAutomationModule: typeof import("@fusion/core") | undefined,
|
||||
generation: number,
|
||||
): Promise<void> {
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
const t0 = Date.now();
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:40:
|
||||
Isolate notifier/OAuth failures so automation schedule sync and merge
|
||||
enqueue still run (Greptile: one reject must not skip reconciliation).
|
||||
OAuth refresh still precedes expiry monitor inside the try block.
|
||||
*/
|
||||
if (!this.options.skipNotifier && this.notificationService && this.authStorage) {
|
||||
try {
|
||||
const notifiersT0 = Date.now();
|
||||
await this.notificationService.start();
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
const oauthAlertState = new OAuthAlertStateStore({
|
||||
statePath: getFusionOAuthAlertStatePath(),
|
||||
});
|
||||
/*
|
||||
FNXC:ClaudeOAuth 2026-07-05-00:00 / 2026-07-08-12:10 / FNXC:FasterStartup 2026-07-14-23:55:
|
||||
FN-7574: proactively refresh OAuth before expiry. Refresh scheduler still
|
||||
starts BEFORE OAuthExpiryMonitor so a stale-but-refreshable token does not
|
||||
fire a false "OAuth token expired" ntfy on restart. Only the await moved
|
||||
off the ensureEngine critical path — relative order is unchanged.
|
||||
*/
|
||||
this.oauthRefreshScheduler = new OAuthRefreshScheduler({ authStorage: this.authStorage });
|
||||
await this.oauthRefreshScheduler.start();
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
|
||||
authStorage: this.authStorage,
|
||||
notificationService: this.notificationService,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthExpiryMonitor.start();
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
this.oauthValidityLogger = new OAuthValidityLogger({
|
||||
authStorage: this.authStorage,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthValidityLogger.start();
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
if (this.notifier) {
|
||||
await this.notifier.start();
|
||||
}
|
||||
runtimeLog.log(`ProjectEngine deferred notifiers+oauth: ${Date.now() - notifiersT0}ms`);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
runtimeLog.error(`Deferred notifiers/OAuth failed (continuing automation/merge startup): ${message}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
if (this.automationStore && coreAutomationModule) {
|
||||
const syncT0 = Date.now();
|
||||
const settings = await store.getSettings();
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
const startupSyncFailures: string[] = [];
|
||||
|
||||
if (typeof coreAutomationModule.syncInsightExtractionAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncInsightExtractionAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`insight extraction: ${message}`);
|
||||
runtimeLog.warn(`Insight extraction automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncInsightExtractionAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
if (typeof coreAutomationModule.syncAutoSummarizeAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncAutoSummarizeAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`auto-summarize: ${message}`);
|
||||
runtimeLog.warn(`Auto-summarize automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncAutoSummarizeAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
if (typeof coreAutomationModule.syncMemoryDreamsAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncMemoryDreamsAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`memory dreams: ${message}`);
|
||||
runtimeLog.warn(`Memory dreams automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncMemoryDreamsAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
if (typeof coreAutomationModule.syncScheduledEvalBatchAutomation === "function") {
|
||||
try {
|
||||
await coreAutomationModule.syncScheduledEvalBatchAutomation(this.automationStore, settings);
|
||||
} catch (err) {
|
||||
const { message, detail } = formatErrorDetails(err);
|
||||
startupSyncFailures.push(`scheduled eval: ${message}`);
|
||||
runtimeLog.warn(`Scheduled eval automation startup sync failed:\n${detail}`);
|
||||
}
|
||||
} else {
|
||||
runtimeLog.warn("syncScheduledEvalBatchAutomation is unavailable; skipping startup sync");
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
// Start CronRunner only after schedule sync so the first tick is not stale.
|
||||
if (this.cronRunner && !this.deferredStartupAborted(generation)) {
|
||||
this.cronRunner.start();
|
||||
runtimeLog.log("CronRunner started after schedule sync");
|
||||
}
|
||||
|
||||
if (startupSyncFailures.length > 0) {
|
||||
this.setAutomationSubsystemHealth(
|
||||
"degraded",
|
||||
`CronRunner started with startup sync warnings: ${startupSyncFailures.join("; ")}`,
|
||||
);
|
||||
} else {
|
||||
this.setAutomationSubsystemHealth(
|
||||
"ready",
|
||||
"CronRunner initialized and startup automation sync completed",
|
||||
);
|
||||
}
|
||||
runtimeLog.log(`ProjectEngine deferred automation syncs: ${Date.now() - syncT0}ms`);
|
||||
} else if (this.cronRunner && !this.deferredStartupAborted(generation)) {
|
||||
// No sync module/store — still start the runner so schedules are not stuck offline.
|
||||
this.cronRunner.start();
|
||||
this.setAutomationSubsystemHealth("ready", "CronRunner started without schedule sync module");
|
||||
}
|
||||
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
|
||||
const mergeT0 = Date.now();
|
||||
await this.startupMergeEnqueue(store);
|
||||
if (this.deferredStartupAborted(generation)) return;
|
||||
runtimeLog.log(
|
||||
`ProjectEngine deferred mergeEnqueue: ${Date.now() - mergeT0}ms (total deferred ${Date.now() - t0}ms)`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1108,14 +1208,17 @@ export class ProjectEngine {
|
||||
* promptly without continuing git/verification work after shutdown starts.
|
||||
*/
|
||||
async stop(): Promise<void> {
|
||||
if (!this.started) {
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:20:
|
||||
Always raise shuttingDown first so deferred startup work (OAuth, automation
|
||||
sync, merge enqueue) observes the flag even if start() has not flipped
|
||||
started yet — prevents unhandled post-stop side effects on fast recycle.
|
||||
*/
|
||||
this.shuttingDown = true;
|
||||
this.startupGeneration += 1;
|
||||
|
||||
// FNXC:VerificationConcurrency 2026-07-15-09:05: Drop this project's cap so it no longer pins process min.
|
||||
unregisterProjectVerificationLimit(this.config.projectId);
|
||||
|
||||
// Stop merge retry timer
|
||||
if (this.mergeRetryTimer) {
|
||||
clearTimeout(this.mergeRetryTimer);
|
||||
@@ -1131,6 +1234,28 @@ export class ProjectEngine {
|
||||
}
|
||||
this.stopPlannerOverseerPoll();
|
||||
|
||||
/*
|
||||
FNXC:FasterStartup 2026-07-15-00:40:
|
||||
Even when start() never flipped started (partial/failed start), stop any
|
||||
critical-path timers already running (gridlock, cron) so abandon/stop does
|
||||
not leak intervals (Greptile partial-start cleanup).
|
||||
*/
|
||||
if (!this.started) {
|
||||
try {
|
||||
this.gridlockDetector?.stop();
|
||||
this.cronRunner?.stop();
|
||||
this.oauthExpiryMonitor?.stop();
|
||||
this.oauthRefreshScheduler?.stop();
|
||||
this.oauthValidityLogger?.stop();
|
||||
this.notificationService?.stop();
|
||||
this.notifier?.stop();
|
||||
this.setAutomationSubsystemHealth("not-initialized", "Automation subsystem stopped (partial start)");
|
||||
} catch {
|
||||
// Best-effort partial cleanup
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Abort active/pending merge work before tearing down sessions.
|
||||
this.mergeAbortController?.abort();
|
||||
this.mergeAbortController = null;
|
||||
@@ -4706,32 +4831,56 @@ export class ProjectEngine {
|
||||
store.on("task:deleted", this.taskDeletedHandler);
|
||||
}
|
||||
|
||||
private async startupMergeSweep(store: TaskStore): Promise<void> {
|
||||
try {
|
||||
const tasks = await store.listTasks({ column: "in-review" });
|
||||
|
||||
// Clear stale "merging"/"merging-pr" statuses left by a prior crash.
|
||||
// No merge is actually running at startup, so any task still marked
|
||||
// as merging is a leftover from a previous engine lifecycle.
|
||||
// This runs unconditionally (regardless of autoMerge setting) because
|
||||
// stale statuses block manual merges too.
|
||||
const staleStatuses = new Set(["merging", "merging-pr"]);
|
||||
for (const t of tasks) {
|
||||
if (t.status && staleStatuses.has(t.status)) {
|
||||
runtimeLog.log(`Startup sweep: clearing stale '${t.status}' status on ${t.id}`);
|
||||
await store.updateTask(t.id, { status: null });
|
||||
// Update in-memory object so canMergeTask sees the cleared status
|
||||
|
||||
(t as any).status = null;
|
||||
}
|
||||
/**
|
||||
* Clear crash-leftover merging statuses so manual merge is unblocked.
|
||||
* Unconditional (not gated on autoMerge). Safe to run on the critical path.
|
||||
*/
|
||||
private async clearStaleMergingStatuses(store: TaskStore): Promise<Task[]> {
|
||||
const tasks = await store.listTasks({ column: "in-review" });
|
||||
// No merge is actually running at startup, so any task still marked
|
||||
// as merging is a leftover from a previous engine lifecycle.
|
||||
const staleStatuses = new Set(["merging", "merging-pr"]);
|
||||
for (const t of tasks) {
|
||||
if (t.status && staleStatuses.has(t.status)) {
|
||||
runtimeLog.log(`Startup sweep: clearing stale '${t.status}' status on ${t.id}`);
|
||||
await store.updateTask(t.id, { status: null });
|
||||
// Update in-memory object so canMergeTask sees the cleared status
|
||||
(t as any).status = null;
|
||||
}
|
||||
}
|
||||
return tasks as Task[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Enqueue auto-merge-eligible in-review tasks. Pause-aware; deferred after
|
||||
* status clear so ensureEngine is not blocked by enqueue work.
|
||||
*/
|
||||
private async startupMergeEnqueue(store: TaskStore): Promise<void> {
|
||||
if (this.shuttingDown) return;
|
||||
try {
|
||||
const settings = await store.getSettings();
|
||||
|
||||
if (settings.globalPause || settings.enginePaused) {
|
||||
runtimeLog.log("Auto-merge startup enqueue skipped: pause active");
|
||||
return;
|
||||
}
|
||||
const tasks = await store.listTasks({ column: "in-review" });
|
||||
if (this.shuttingDown) return;
|
||||
const enqueued = await this.enqueueEligibleInReviewTasks(tasks as Task[], settings);
|
||||
if (enqueued > 0) {
|
||||
runtimeLog.log(`Auto-merge startup sweep: enqueueing ${enqueued} task(s)`);
|
||||
}
|
||||
} catch (err: unknown) {
|
||||
runtimeLog.warn(
|
||||
`Auto-merge startup enqueue failed: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Full startup merge sweep (status clear + enqueue). Kept for tests/callers. */
|
||||
private async startupMergeSweep(store: TaskStore): Promise<void> {
|
||||
try {
|
||||
await this.clearStaleMergingStatuses(store);
|
||||
await this.startupMergeEnqueue(store);
|
||||
} catch (err: unknown) {
|
||||
runtimeLog.warn(
|
||||
`Auto-merge startup sweep failed: ${err instanceof Error ? err.message : String(err)}`,
|
||||
|
||||
Reference in New Issue
Block a user