FN-8453: unify concurrency accounting and indicators
Unify live-agent capacity accounting across engine and dashboard. - Derive Running and Waiting from workflow traits and durable agent liveness. - Apply unified limits to planner, executor, and merge admission while updating dashboard indicators. - Remove duplicate concurrency controls and document the unified operator model. Files changed: .changeset/fn-8453-unified-concurrency.md | 7 + docs/agent-tool-surface-full-loop.md | 4 +- docs/architecture.md | 2 +- docs/dashboard-guide.md | 4 +- docs/settings-reference.md | 4 +- .../skill/fusion/references/fusion-capabilities.md | 4 +- .../core/src/__tests__/live-agent-count.test.ts | 91 ++++---- packages/core/src/index.gate.ts | 6 + packages/core/src/index.ts | 6 + packages/core/src/live-agent-count.ts | 107 ++++++--- packages/dashboard/app/App.tsx | 28 ++- packages/dashboard/app/api/board-workflows.ts | 2 + packages/dashboard/app/components/Column.tsx | 6 +- .../dashboard/app/components/EngineControlMenu.tsx | 26 --- .../dashboard/app/components/ExecutorStatusBar.tsx | 38 ++- .../dashboard/app/components/SettingsModal.tsx | 1 - .../app/components/__tests__/Column.test.tsx | 6 +- .../__tests__/EngineControlMenu.test.tsx | 10 +- .../__tests__/ExecutorStatusBar.test.tsx | 32 ++- .../command-center/CommandCenterControls.tsx | 26 --- .../settings/sections/SchedulingSection.search.ts | 9 - .../settings/sections/SchedulingSection.tsx | 13 -- .../app/hooks/__tests__/useExecutorStats.test.ts | 12 +- packages/dashboard/app/hooks/useExecutorStats.ts | 50 ++-- .../src/__tests__/project-store-resolver.test.ts | 11 +- packages/dashboard/src/project-store-resolver.ts | 14 +- .../register-config-mcp-pi-settings-routes.ts | 3 +- packages/engine/src/__tests__/concurrency.test.ts | 123 +++++++++- .../engine/src/__tests__/project-engine.test.ts | 34 +++ packages/engine/src/__tests__/triage.test.ts | 7 +- packages/engine/src/concurrency.ts | 207 ++++++++++++++++- packages/engine/src/project-engine.ts | 151 ++++++++++-- packages/engine/src/scheduler.ts | 82 ++++++- packages/engine/src/triage.ts | 254 +++++++++++++-------- .../lib/dashboard-browser-safe-core-modules.json | 5 + 35 files changed, 991 insertions(+), 394 deletions(-) Fusion-Task-Id: FN-8453 Fusion-Task-Lineage: 12cfa5df-675d-4fce-b17e-932376544239 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -63,9 +63,9 @@ FR-07/FR-08 require one canonical operation beneath UI and tool callers. The par
|
||||
Fusion does not have a single-workflow-only scheduler by design. The scheduler can dispatch independent runnable tasks, but admission is bounded and serialized at specific safety gates:
|
||||
|
||||
- `packages/engine/src/scheduler.ts` computes dispatch capacity from **`maxConcurrent`**, **`maxWorktrees`**, and the shared **`semaphore`** in `computeConcurrencyGateDiagnostic()`. The settings default to `maxConcurrent` 2 and `maxWorktrees` 4 when unspecified (scheduler dispatch path).
|
||||
- `AgentSemaphore` in `packages/engine/src/concurrency.ts` gates all top-level triage, execution, and merge agents. Its priority queue serves merge before execute before specification (`PRIORITY_MERGE`, `PRIORITY_EXECUTE`, `PRIORITY_SPECIFY`). A full semaphore can therefore make work appear serialized even with multiple runnable cards.
|
||||
- `AgentSemaphore` in `packages/engine/src/concurrency.ts` gates all top-level planning, execution, and merge/review agents. Per-project admission ranks eligible candidates by `createdAt` then task ID across lanes before claiming the shared capacity; lane priority is not a free-slot rank key. Nested helpers remain parent-internal and intentionally soft-breach this displayed top-level cap to avoid parent/child deadlocks.
|
||||
- The workflow hold/release sweep in `scheduler.ts` reserves worktree and semaphore capacity with `tryAcquire()` **before** moving a task to `in-progress`, then transfers that pre-held slot to the executor. This prevents a race but means the available minimum of all gates is authoritative.
|
||||
- `maxWorktrees` counts only `in-progress` tasks; in-review worktrees do not consume that execution-worktree limit. `maxConcurrent` counts execution slots, while the global semaphore also accounts for planning and active review top-level holders.
|
||||
- `maxWorktrees` counts only `in-progress` tasks; in-review worktrees do not consume that execution-worktree limit. `maxConcurrent` caps per-project top-level working agents across planning, execution, and active review/merge, while the host semaphore remains the process-global pool.
|
||||
- Runnable candidates are additionally filtered for paused state, unmet dependencies, recovery backoff, workflow hold/release state, and file-scope overlap via `isRunnableQueuedOverlapCandidate()` / `pathsOverlap()` in `scheduler.ts`. This is coarse path-scope serialization, not FR-48 symbol locking.
|
||||
- `packages/engine/src/workflow-work-scheduler.ts` claims one due workflow work item per call with a lease. The surrounding scheduler’s repeated dispatch and capacity gates determine aggregate concurrency; this helper alone does not fan out a batch.
|
||||
- `packages/engine/src/verification-concurrency.ts` separately defaults expensive verification subprocesses to one concurrent project-wide slot. Task execution can be parallel while heavy E2E verification intentionally queues.
|
||||
|
||||
@@ -1134,7 +1134,7 @@ The run-audit system records every mutation performed by the engine across four
|
||||
|
||||
Events are tied to specific run IDs for end-to-end traceability.
|
||||
|
||||
For scheduler concurrency diagnostics, the queued reason now names the active limiter(s) and usage (for example `gate=maxConcurrent ...`). The reason includes the `bindingGates` (`maxConcurrent`/`maxWorktrees`/`semaphore`), per-gate `{ used, limit, slack }`, `holders`, and computed `available`. `holders.maxConcurrent` and `holders.maxWorktrees` are current `in-progress` task IDs; `holders.semaphore` mirrors that set but semaphore slots can also be consumed by triage/merge agents outside `in-progress`. So if `semaphore.used` exceeds the visible holder list, that usually indicates non-execution agents are legitimately consuming shared capacity (not stale accounting). `maxWorktrees` is also enforced inside `TaskStore.moveTaskInternal` when committing an allocated move into `in-progress`, making it a hard active execution worktree cap even when workflow WIP/`maxConcurrent` would allow more tasks. These queued-reason logs are transition-only: a newly emitted line indicates the limiter signature changed or the condition cleared and later reappeared, not that a poll loop simply observed the same blocked state again.
|
||||
For scheduler concurrency diagnostics, the queued reason names the active limiter(s) and usage (for example `gate=maxConcurrent ...`). The reason includes the `bindingGates` (`maxConcurrent`/`maxWorktrees`/`semaphore`), per-gate `{ used, limit, slack }`, `holders`, and computed `available`. `maxConcurrent` is a per-project cap on enriched live top-level planning, execution, and review/merge agents; the host semaphore is the separate process-global pool. Free project capacity is admitted oldest-first across lanes, rather than by lane priority. `maxWorktrees` is also enforced inside `TaskStore.moveTaskInternal` when committing an allocated move into `in-progress`, making it a hard active execution worktree cap even when workflow WIP/`maxConcurrent` would allow more tasks. These queued-reason logs are transition-only: a newly emitted line indicates the limiter signature changed or the condition cleared and later reappeared, not that a poll loop simply observed the same blocked state again.
|
||||
|
||||
**Run audit endpoints:**
|
||||
- `GET /api/agents/:id/runs/:runId/audit` — Returns audit trail for a specific agent run
|
||||
|
||||
@@ -1258,7 +1258,7 @@ Features:
|
||||
<!-- FNXC:CommandCenter 2026-06-27-10:03: Tokens detail charts must show every model bucket returned by analytics for accurate spend attribution; Overview remains a compact top-model summary because its copy explicitly frames those cards as top consumers/share. -->
|
||||
<!-- FNXC:CommandCenterActivity 2026-06-30-00:00: Activity active-agent counts include both durable-agent usage events and ephemeral task-worker execution runs from agentRuns, because task execution can be visible without a matching usage_events row. -->
|
||||
<!-- FNXC:CommandCenterActivity 2026-07-01-00:00: Graph-owned workflow step sessions publish active-to-terminal agentRuns lifecycle rows with task lineage and step metadata, so daily activity and Activity throughput charts include new workflow execution without dashboard-side recounting. -->
|
||||
- **Overview controls dashboard** sits at the top of the Overview landing surface on desktop and mobile. It includes AI engine stop/start backed by `globalPause`, live scheduler status from executor stats, the shared Global Max Concurrent slider backed by `/api/global-concurrency`, range sliders for `maxConcurrent`, `maxTriageConcurrent`, and `maxWorktrees` that persist through `/api/settings`, and a compact theme dropdown with the same color-chip swatches and Shadcn variant list as Settings → Appearance. The four concurrency sliders ask for confirmation after a changed value settles; confirming persists the new cap, while cancel, backdrop, or Escape dismissal reverts to the last persisted value without saving. The global and current-project max-concurrent sliders show running-agent counts plus a current-use dot on the track once utilization data loads; triage and worktree sliders remain cap-only. These controls reuse existing APIs and App-level theme setters; they do not add a new backend route or second theme owner.
|
||||
- **Overview controls dashboard** includes AI engine stop/start backed by `globalPause`, the shared Global Max Concurrent slider, and current-project **Max concurrency** plus **Max worktrees** controls. Max concurrency caps top-level working agents across planning, execution, and review/merge; free capacity is admitted oldest-first within the project. The footer reports **Waiting**, **Running (N/max)**, and **Blocked**; column headers report active/total. Nested helper agents remain parent-internal and may temporarily exceed the displayed top-level count.
|
||||
<!-- FNXC:TeamArea 2026-07-18-12:30: FN-8351 moves organization export and import to the Team tab so team-level portability controls are not presented as Overview dashboard controls. -->
|
||||
- **Team tab — Org export / import** lets an operator download a portable organization JSON bundle or paste one for a dry-run preview before confirming the apply step. Exports are secret-scrubbed by default: credentials and tokens are never included, while safe secret references can remain for setup in the destination project.
|
||||
- **Configuration versions** lives in **Settings → Project → Configuration Versions**. It lists recorded project-setting revisions newest first; select **Roll back** on any revision and confirm once to restore it. The restore is recorded as a new forward revision, so it can itself be undone without manually reconstructing settings.
|
||||
@@ -1451,7 +1451,7 @@ Use this panel when upgrading a project with pre-FN-6245/FN-6277 in-review rows
|
||||
<!-- FNXC:ExecutorStatusBar 2026-06-29-19:09: FN-7248 makes footer concurrency edits confirmation-gated like Command Center. Closing the popover, outside-clicking, pressing Escape, dismissing the backdrop, or unmounting must revert unconfirmed slider edits instead of saving them. -->
|
||||
<!-- FNXC:ExecutorStatusBar 2026-06-30-16:42: FN-7273 keeps the footer Engine Controls popover usable on mobile, narrow tablets, and tablet landscape by documenting that constrained screens use a full-width bottom panel above both fixed bottom bars instead of the compact desktop anchor. -->
|
||||
<!-- FNXC:GlobalConcurrencyControls 2026-07-15-17:30: FN-8007 keeps footer and dashboard current-use marker geometry aligned with the native range thumb, including its desktop and mobile thumb-size edge inset. -->
|
||||
The global AI engine stop/start control and triage pause/resume control live in the executor footer status bar rather than the header. Select the small engine-controls button beside the executor state badge, or select the state text such as **Running**, to open the footer popover. The popover includes **Stop AI engine** / **Start AI engine**, **Pause triage** / **Resume scheduling**, and live scheduler sliders for max concurrent tasks, max triage concurrency, and max worktrees. On mobile, narrow tablets, and tablet landscape, the same controls open as a full-width bottom panel above the executor footer and mobile navigation so the close button and sliders remain reachable. Use the visible **Close engine controls** X button, Escape, or outside-click to dismiss it. The global and current-project concurrency sliders also show how many agents are running, including actively-triaging planners (`triage` + `planning`, not paused), and a dot on the slider track for current use. The dot uses the same min-relative range coordinates and thumb-size edge inset as the native slider: it aligns to the cap-clamped running count, so one running agent at the slider minimum stays visible at the start and over-cap usage pins to the cap thumb instead of the expanded track end. Changed concurrency slider values ask for confirmation after the value settles. Confirming saves the global cap through `/api/global-concurrency` and project caps through `/api/settings`; cancel, backdrop dismissal, Escape, close, outside-click, or unmount reverts unconfirmed slider edits without saving. Multiple changed project sliders within one debounce window are summarized in one confirmation dialog, matching Command Center behavior.
|
||||
The global AI engine stop/start control and triage pause/resume control live in the executor footer status bar rather than the header. Select the small engine-controls button beside the executor state badge, or select the state text such as **Running**, to open the footer popover. The popover includes **Stop AI engine** / **Start AI engine**, **Pause triage** / **Resume scheduling**, and live scheduler sliders for **Max concurrency** and max worktrees. Max concurrency is the per-project top-level working-agent cap across planning, execution, and review/merge; nested helpers remain parent-internal and can temporarily exceed the displayed count. On mobile, narrow tablets, and tablet landscape, the same controls open as a full-width bottom panel above the executor footer and mobile navigation so the close button and sliders remain reachable. Use the visible **Close engine controls** X button, Escape, or outside-click to dismiss it. The global and current-project concurrency sliders show the shared live top-level agent count and a dot on the slider track for current use. The dot uses the same min-relative range coordinates and thumb-size edge inset as the native slider: it aligns to the cap-clamped running count, so one running agent at the slider minimum stays visible at the start and over-cap usage pins to the cap thumb instead of the expanded track end. Changed concurrency slider values ask for confirmation after the value settles. Confirming saves the global cap through `/api/global-concurrency` and project caps through `/api/settings`; cancel, backdrop dismissal, Escape, close, outside-click, or unmount reverts unconfirmed slider edits without saving. Multiple changed project sliders within one debounce window are summarized in one confirmation dialog, matching Command Center behavior.
|
||||
|
||||
<!-- FNXC:ExecutorStatusBar 2026-06-27-00:00: FN-7163 makes footer stats loading initial-only so routine heartbeat refreshes keep the populated footer and open concurrency popover mounted instead of blinking to the loading branch. -->
|
||||
Brief, single-poll executor stats fetch blips keep showing the last good footer stats instead of flashing **Connecting…**. Routine executor stats heartbeats also keep the populated footer mounted after initial load, so an open engine/concurrency popover stays open while counts refresh. The footer only switches to **Connecting…** for sustained suspension-like stats failures, or to an explicit error state for non-transient failures.
|
||||
|
||||
@@ -430,9 +430,9 @@ Security-sensitive file-browser escape hatches are project-only. `allowAbsoluteF
|
||||
| `globalPause` | `boolean` | `false` | Hard stop: terminate active engine sessions and pause scheduling immediately. |
|
||||
| `globalPauseReason` | `string` | `undefined` | Optional reason for `globalPause` (`"rate-limit"` for automatic pauses, `"manual"` for user-triggered pauses). Cleared on unpause. |
|
||||
| `enginePaused` | `boolean` | `false` | Soft pause: stop dispatching new work while letting active sessions finish. While paused (including shared pause windows with `globalPause`), stuck-task polling/timers are suspended so paused wall-clock time does not count against `taskStuckTimeoutMs`. Clearing pause state resumes runtime scheduling and gives tracked active sessions a fresh stuck-task grace window before normal detection resumes; when `autoMerge` is enabled, eligible `in-review` tasks are re-swept into the auto-merge queue (paused/blocked/failed review tasks remain skipped). |
|
||||
| `maxConcurrent` | `number` | `2` | Max concurrent task-lane AI agents (planning, executor, merge). Editable from Settings and the Command Center Overview controls dashboard. |
|
||||
| `maxConcurrent` | `number` | `2` | Max concurrency for top-level working agents per project across planning, execution, and review/merge. Nested helper agents remain parent-internal and may temporarily exceed this displayed count. Editable from Settings, Command Center, and Engine Control. |
|
||||
| `maxConcurrentVerifications` | `number` | `1` | Max concurrent verification subprocesses (`fn_run_verification`, merge test/build commands) process-wide. Caps stacked monorepo typecheck/build so concurrent tasks do not peg host CPU. Range **1–8** (clamped at runtime and in Settings). Editable from Settings → Scheduling. Each project engine registers its cap; the effective process limit is the **minimum** of registered project caps. |
|
||||
| `maxTriageConcurrent` | `number` | `2` | Max concurrent planning agents. Editable from Settings and the Command Center Overview controls dashboard. |
|
||||
| `maxTriageConcurrent` | `number` | `2` | Legacy persisted value; ignored. Planning shares `maxConcurrent` and no Max Triage control is displayed. |
|
||||
| `globalMaxConcurrent` | `number` | `4` | System-wide max concurrent agents across all projects. |
|
||||
| `maxWorktrees` | `number` | `4` | Max git worktrees. Editable from Settings and the Command Center Overview controls dashboard. |
|
||||
| `pollIntervalMs` | `number` | `15000` | Scheduler poll interval (ms). |
|
||||
|
||||
Reference in New Issue
Block a user