Files
fusion/packages/engine/src/project-manager.ts
gsxdsm c15c78feeb feat: migrate storage from SQLite to PostgreSQL (#1793)
# Migrate storage from SQLite to PostgreSQL — full dashboard cutover

Migrates Fusion's storage layer to the embedded PostgreSQL
`AsyncDataLayer` (the default backend) and **completes the
satellite-store + feature cutover** so every dashboard and Command
Center surface works in PG mode.

## Status — every surface works in embedded-PG mode

Verified live against a running embedded-Postgres dashboard (all
**200**, zero 5xx) and gate-tested (**23 files / 99 tests** on embedded
PG, plus engine-core 294 and ci-shape 63 in the blocking merge gate;
core/engine/cli/dashboard typecheck clean).

| Area | Surfaces | State |
|---|---|---|
| Satellite stores | workflows, todos, insights, research, missions,
goals, mailbox | ✅ |
| Views | artifacts, documents, evals | ✅ |
| Command Center | activity, productivity, team, tokens, tools,
**workflows**, **github**, **signals**, **plugin-activations**, **live**
(all 10) | ✅ |
| Run execution | insight generation, research run execution | ✅
(store-path; AI step needs a provider) |
| Live updates | SSE push for mission/research/insight events | ✅ |
| Workflow editing | create / update / delete / select (+ id counter) |
✅ |
| Engine | mission autopilot, incident-signal ingestion, regression
storm-guard, agent wake-on-message | ✅ |
| Core | tasks, agents, secrets, automations, memory, chat, usage, PRs,
git | ✅ |

## Approach

Each satellite store gets an `Async<Store>` wrapper exposing the sync
store's method names over the existing `async-*-store.ts` helpers;
`get<Store>Store()` returns a `Sync | Async` union; consumers `await`
(harmless on sync), and engine/CLI paths that can't convert use
`instanceof Sync` graceful fallback. Analytics aggregators branch on
`"ping" in dbOrLayer` to run schema-qualified raw SQL over `project.*`
(snake_case) in PG. Executors/orchestrators/autopilot are
await-converted to drive the union store; the async store wrappers
extend `EventEmitter` so SSE live-push fires in both backends.

Not-yet-ported capabilities degrade gracefully (never 500) and are
individually called out in commits.

## Sync with main

The branch is kept continuously merged with `main` (currently through
FN-7845, 2026-07-12); the earlier "final rebase deferred" note no longer
applies. Use **Create a merge commit** (or squash) to land it — GitHub's
rebase-merge cannot replay a merge-maintained branch.

## Residual Review Findings

Multi-agent code review of the PostgreSQL satellite-store ports (U1–U5)
applied 3 safe fixes (see `fix(review): apply autofix feedback`). The
following are **real but gated** — recorded here as follow-up work
rather than auto-applied. All are SQLite→PostgreSQL
**concurrency/atomicity regressions**: the sync stores were immune only
by SQLite's single-writer, single-threaded-handler execution; the async
ports open multi-await read-modify-write windows. **Reachability is low
today** because the execution engines that generate concurrent same-run
mutations (insight run executor, research orchestrator/dispatcher) are
`instanceof`-gated to sync mode in PG. No process-crash class survived
(all engine fallbacks correctly guard the sync store).

- **[P1] Research `appendResearchEvent` dual-write is non-atomic**
(`packages/core/src/async-research-store.ts`, corroborated: adversarial
+ reliability). The `research_run_events` insert (own transaction) and
the `run.events` jsonb update are separate writes — a crash between
them, or two concurrent appends, splits the table count from the jsonb
array. **Fix:** perform the seq-insert and the jsonb update in one
`layer.transactionImmediate`.
- **[P1] Research run terminal-reversion via stale full-row persist**
(`async-research-store.ts` `persistResearchRun`/`updateResearchStatus`).
Concurrent `PATCH /runs/:id/status` + `POST /runs/:id/events` can revert
a terminal run to `running` by overwriting the whole row, bypassing the
transition guard. **Fix:** scoped column `UPDATE`s with a `WHERE status
…` guard, or optimistic version column.
- **[P2] `updateResearchRun`/`updateInsightRun` read-then-write TOCTOU**
— concurrent PATCHes last-writer-wins on the lifecycle merge. **Fix:**
`SELECT … FOR UPDATE` / enclosing transaction.
- **[P2] `upsertRun`/`createRunOrThrowConflict` check-then-create race**
(`async-insight-store.ts`) — two callers can each create an "active"
run. **Fix:** partial unique index on `(projectId, trigger) WHERE status
IN ('pending','running')`.
- **[P3] `createResearchRetryRun` return-value divergence** — sync
returns the pre-update `queued` snapshot; async returns the reloaded
`retry_waiting` run (persisted state is identical). Pick one side for
cross-backend parity.
- **[P2/perf] Mission `getMissionWithHierarchy`/`getMissionHealth` N+1
fan-out** — O(milestones×slices) sequential round-trips hold one pool
slot per request; can starve the pool for large hierarchies. **Fix:**
batched/joined reads.
- **Testing gaps:** no PG-mode concurrency tests (interleaved
status/event mutations), no sync↔async parity assertion for the
lifecycle-error codes, and no mission status/health rollup parity test
vs the sync `MissionStore`.

~~Out of scope (deferred): AI run *execution* (insight/research) +
mission autopilot + live SSE mission events remain sync-gated/degraded
in PG mode.~~ **Since ported** — insight/research run execution, mission
autopilot, and SSE live push all run on the async layer now, which also
makes the concurrency findings above genuinely reachable; they remain
open follow-ups.







---

## Update — 2026-07-12: production-readiness hardening & live acceptance

Everything below landed on this branch since the description above was
written:

**Production blockers from review — fixed**
- `recoverStaleTransitionPending` ported to the async layer (backend
moves write + clear the crash-safe marker; startup/maintenance sweeps no
longer throw).
- Lost-update class fixed: `atomicWriteTaskJson`/`WithAudit` write
changed columns only (full-row upserts silently resurrected stale fields
across concurrent store instances — the "task stuck unplanned forever"
bug).
- First-boot **auto-migration**: booting the PG backend over a project
with a legacy `fusion.db` migrates it automatically (loud failure,
SQLite kept as backup), and the dashboard shows a one-time **"your data
was migrated" banner** with the backup paths and a Need-help Discord
link.
- `pg_dump`/`pg_restore` discovered from common install locations for
embedded-mode backups.
- The PG suite is part of the blocking merge gate (`test:pg-gate`).

**Multi-project isolation (PR #2007, merged into this branch)**
- `project_id` partition key on tasks / archived tasks / config,
`taskProjectScope` threaded through every scan/claim/count, per-project
config rows, layer bound to the project at startup.
- Review P1 follow-up: the shared cold-storage `archive.archived_tasks`
table is also partitioned and all archived-board reads/counts/searches
are scoped.
- Schema drift self-heal generalized to schema-qualified columns so
existing databases upgrade in place.

**Other changes**
- Node settings sync **removed** in PG mode (409
`settings-sync-disabled-postgres`) — nodes share state by connecting to
the same database; auth sync kept (per-machine file).
- Perf (review findings): `listTasks` pushes column filter + ORDER BY +
LIMIT/OFFSET into SQL; `getConversation` capped to the most recent 200
messages.
- Fixed a false "operator action required" pause-abort log fired on
every successfully auto-merged task.

**Live acceptance — PASSED (2026-07-12)**
A sandboxed instance (isolated HOME, embedded PG, real Opus executor)
ran a task through the complete cycle: create → triage (AI spec) →
execute → in-review → AI squash-merge landed on the project's `main` →
done. A write+read sweep of every data surface (settings, comments,
documents, attachments + artifact bridge + artifact edit, chat with real
generation, goals, missions, agent mail, secrets, workflows, memory, CC
analytics) was green on embedded PG.

**Known remaining work**
- The per-project `config` PK re-key has no upgrade path for
pre-isolation embedded-PG databases (needs a real `DROP
CONSTRAINT`/re-key migration; fresh databases are fine).
- `pg_dump`/`pg_restore` binaries are not yet bundled in release
artifacts (PATH/common-location discovery only).
- The satellite-store concurrency findings listed above.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Phil Larson <hello@phillarson.xyz>
Co-authored-by: fusion-merge <fusion-merge@local>
2026-07-13 19:07:58 -07:00

570 lines
19 KiB
TypeScript

import { EventEmitter } from "node:events";
import type { Task, CentralCore } from "@fusion/core";
import { InProcessRuntime } from "./runtimes/in-process-runtime.js";
import { ChildProcessRuntime } from "./runtimes/child-process-runtime.js";
import { RemoteNodeRuntime } from "./runtimes/remote-node-runtime.js";
import { AgentSemaphore } from "./concurrency.js";
import type {
ProjectRuntime,
ProjectRuntimeConfig,
RuntimeStatus,
GlobalMetrics,
} from "./project-runtime.js";
import { projectManagerLog } from "./logger.js";
/**
* Events emitted by ProjectManager with project attribution.
*/
export interface ProjectManagerEvents {
/** Emitted when a task is created in any project */
"task:created": [data: { projectId: string; projectName: string; task: Task }];
/** Emitted when a task is deleted in any project */
"task:deleted": [data: { projectId: string; projectName: string; task: Task; meta?: { githubIssueAction?: import("@fusion/core").GithubIssueAction } }];
/** Emitted when a task is moved in any project */
"task:moved": [
data: {
projectId: string;
projectName: string;
task: Task;
from: string;
to: string;
}
];
/** Emitted when a task is updated in any project */
"task:updated": [data: { projectId: string; projectName: string; task: Task }];
/** Emitted when an error occurs in any project */
"error": [data: { projectId: string; projectName: string; error: Error }];
/** Emitted when project health status changes */
"health:changed": [
data: {
projectId: string;
projectName: string;
status: RuntimeStatus;
previous: RuntimeStatus;
}
];
/** Emitted when a runtime is added */
"runtime:added": [data: { projectId: string; projectName: string }];
/** Emitted when a runtime is removed */
"runtime:removed": [data: { projectId: string; projectName: string }];
/** Emitted when a runtime is restarted */
"project:runtime-restarted": [data: { projectId: string; projectName: string; isolationMode: import("@fusion/core").IsolationMode; reason?: string }];
}
export interface ProjectRuntimeRestartBlockedError {
kind: "active_tasks";
count: number;
}
export function isProjectRuntimeRestartBlockedError(error: unknown): error is ProjectRuntimeRestartBlockedError {
if (!error || typeof error !== "object") return false;
const candidate = error as { kind?: unknown; count?: unknown };
return candidate.kind === "active_tasks" && typeof candidate.count === "number";
}
/**
* ProjectManager orchestrates all project runtimes and enforces global
* concurrency limits from CentralCore.
*
* This is the main entry point for multi-project support in the engine.
* It manages multiple ProjectRuntime instances (one per project) and:
* - Creates appropriate runtime type based on isolation mode
* - Forwards runtime events with project attribution
* - Enforces global concurrency limits via CentralCore
* - Updates project health in CentralCore
* - Logs activity to CentralCore's unified feed
*
* **Project Registration Assumption:**
* ProjectManager assumes projects are already registered in CentralCore.
* Call `centralCore.registerProject()` before `addProject()`.
*
* @example
* ```typescript
* const central = new CentralCore();
* await central.init();
*
* const manager = new ProjectManager(central);
*
* // Register project in CentralCore first
* const project = await central.registerProject({
* name: "My Project",
* path: "/path/to/project"
* });
*
* // Then add runtime
* const runtime = await manager.addProject({
* projectId: project.id,
* workingDirectory: await central.resolveLocalProjectWorkingDirectory(project.id),
* isolationMode: "in-process",
* maxConcurrent: 2,
* maxWorktrees: 4,
* });
*
* // Listen to all project events
* manager.on("task:created", ({ projectId, projectName, task }) => {
* console.log(`${projectName}: Task ${task.id} created`);
* });
* ```
*/
export class ProjectManager extends EventEmitter<ProjectManagerEvents> {
private runtimes = new Map<string, ProjectRuntime>();
private projectNames = new Map<string, string>();
private globalSemaphore: AgentSemaphore;
/** Mutable limit read by the shared semaphore's getter function. */
private currentGlobalLimit = 4;
private globalLimitRefreshInterval: ReturnType<typeof setInterval>;
/**
* FNXC:LiveGlobalConcurrency 2026-07-11:
* Listener that applies a live global-concurrency change to the shared
* semaphore's limit immediately, so a PUT /api/global-concurrency takes effect
* without waiting for the 30s refresh poll (or a process restart). This is the
* binding semaphore: it is injected into every InProcessRuntime (which only
* installs its OWN concurrency:changed listener when it has to create a local
* semaphore, i.e. NOT in the shared-semaphore path), so the subscription must
* live here. Mirrors project-engine-manager.ts.
*/
private concurrencyListener = (state: unknown): void => {
const s = state as { globalMaxConcurrent?: number };
if (typeof s.globalMaxConcurrent === "number") {
this.currentGlobalLimit = s.globalMaxConcurrent;
projectManagerLog.log(`Global concurrency limit updated to ${this.currentGlobalLimit}`);
}
};
/**
* @param centralCore - CentralCore reference for global coordination
*/
constructor(private centralCore: CentralCore) {
super();
this.setMaxListeners(100);
// Initialize global semaphore with a getter that reads the mutable limit.
// This single semaphore instance is shared across all runtimes so
// cross-project concurrency is enforced correctly.
this.globalSemaphore = new AgentSemaphore(() => this.currentGlobalLimit);
// Subscribe to live concurrency changes so limit updates take effect
// immediately (no restart / no 30s poll wait) on the shared semaphore.
if (typeof centralCore.on === "function") {
centralCore.on("concurrency:changed", this.concurrencyListener);
}
// Refresh the global limit periodically (fallback reconciliation).
this.refreshGlobalLimit();
this.globalLimitRefreshInterval = setInterval(() => this.refreshGlobalLimit(), 30000);
this.globalLimitRefreshInterval.unref?.();
projectManagerLog.log("ProjectManager initialized");
}
/**
* Refresh the global concurrency limit from CentralCore.
*/
private async refreshGlobalLimit(): Promise<void> {
try {
const state = await this.centralCore.getGlobalConcurrencyState();
this.currentGlobalLimit = state.globalMaxConcurrent;
} catch (error) {
projectManagerLog.warn("Failed to refresh global concurrency limit:", error);
}
}
/**
* Add a project runtime and start it.
*
* **Prerequisite:** Project must already be registered in CentralCore.
*
* @param config - Runtime configuration (must match a registered project)
* @returns The created and started ProjectRuntime
* @throws Error if project not found in CentralCore or runtime already exists
*/
async addProject(config: ProjectRuntimeConfig): Promise<ProjectRuntime> {
// Validate project exists in CentralCore
const project = await this.centralCore.getProject(config.projectId);
if (!project) {
throw new Error(
`Project ${config.projectId} not found in CentralCore. ` +
"Register the project first with centralCore.registerProject()."
);
}
// Check if runtime already exists
if (this.runtimes.has(config.projectId)) {
throw new Error(`Runtime already exists for project ${config.projectId}`);
}
projectManagerLog.log(`Adding project runtime for ${config.projectId} (${project.name})`);
// Store project name for event attribution
this.projectNames.set(config.projectId, project.name);
// Create appropriate runtime based on isolation mode
let runtime: ProjectRuntime;
// Inject the shared global semaphore so all runtimes share one concurrency pool.
const configWithSemaphore = { ...config, globalSemaphore: this.globalSemaphore };
if (config.isolationMode === "child-process") {
runtime = new ChildProcessRuntime(configWithSemaphore, this.centralCore);
} else {
let assignedNode = undefined;
if (project.nodeId) {
assignedNode = await this.centralCore.getNode(project.nodeId);
if (!assignedNode) {
projectManagerLog.warn(
`Assigned node ${project.nodeId} not found for project ${project.id}; falling back to InProcessRuntime`
);
}
}
if (assignedNode?.type === "remote") {
runtime = new RemoteNodeRuntime({
nodeConfig: assignedNode,
projectId: config.projectId,
projectName: project.name,
});
} else {
// Default to local in-process runtime (includes unassigned + local-node assigned)
runtime = new InProcessRuntime(configWithSemaphore, this.centralCore);
}
}
// Set up event forwarding with project attribution
this.setupEventForwarding(runtime, config.projectId, project.name);
// Start the runtime
await runtime.start();
// Update project health to active
await this.centralCore.updateProjectHealth(config.projectId, {
status: "active",
activeTaskCount: 0,
inFlightAgentCount: 0,
totalTasksCompleted: 0,
totalTasksFailed: 0,
updatedAt: new Date().toISOString(),
});
// Update project status from "initializing" to "active"
await this.centralCore.updateProject(project.id, {
status: "active",
});
// Store runtime
this.runtimes.set(config.projectId, runtime);
// Log to activity feed
await this.centralCore.logActivity({
type: "task:created", // Using task:created as a generic activity type
projectId: config.projectId,
projectName: project.name,
timestamp: new Date().toISOString(),
details: `Project runtime started for ${project.name}`,
});
this.emit("runtime:added", { projectId: config.projectId, projectName: project.name });
projectManagerLog.log(`Project runtime added for ${config.projectId}`);
return runtime;
}
/**
* Remove a project runtime and stop it.
*
* @param id - Project ID to remove
* @throws Error if runtime not found
*/
async removeProject(id: string): Promise<void> {
const runtime = this.runtimes.get(id);
if (!runtime) {
throw new Error(`Runtime not found for project ${id}`);
}
const projectName = this.projectNames.get(id) ?? id;
projectManagerLog.log(`Removing project runtime for ${id}`);
// Stop the runtime
await runtime.stop();
// Remove from maps
this.runtimes.delete(id);
this.projectNames.delete(id);
// Update project health to paused
await this.centralCore.updateProjectHealth(id, {
status: "paused",
activeTaskCount: 0,
inFlightAgentCount: 0,
updatedAt: new Date().toISOString(),
});
// Log to activity feed
await this.centralCore.logActivity({
type: "task:created",
projectId: id,
projectName: projectName,
timestamp: new Date().toISOString(),
details: `Project runtime stopped for ${projectName}`,
});
this.emit("runtime:removed", { projectId: id, projectName });
projectManagerLog.log(`Project runtime removed for ${id}`);
}
/**
* Get a runtime by project ID.
*/
getRuntime(id: string): ProjectRuntime | undefined {
return this.runtimes.get(id);
}
/**
* List all managed runtimes.
*/
listRuntimes(): ProjectRuntime[] {
return Array.from(this.runtimes.values());
}
/**
* Get all project IDs.
*/
getProjectIds(): string[] {
return Array.from(this.runtimes.keys());
}
/**
* Get global metrics aggregated across all runtimes.
*/
async getGlobalMetrics(): Promise<GlobalMetrics> {
const statusCounts: Record<RuntimeStatus, number> = {
active: 0,
paused: 0,
errored: 0,
stopped: 0,
starting: 0,
stopping: 0,
};
let totalInFlight = 0;
let totalActiveAgents = 0;
for (const [id, runtime] of this.runtimes) {
const status = runtime.getStatus();
statusCounts[status]++;
try {
const metrics = runtime.getMetrics();
totalInFlight += metrics.inFlightTasks;
totalActiveAgents += metrics.activeAgents;
// Update health in CentralCore
const project = await this.centralCore.getProject(id);
if (project) {
await this.centralCore.updateProjectHealth(id, {
status: status as "active" | "paused" | "errored" | "initializing",
activeTaskCount: metrics.inFlightTasks,
inFlightAgentCount: metrics.activeAgents,
lastActivityAt: metrics.lastActivityAt,
updatedAt: new Date().toISOString(),
});
}
} catch (error) {
projectManagerLog.warn(`Failed to get metrics for ${id}:`, error);
}
}
return {
totalInFlightTasks: totalInFlight,
totalActiveAgents: totalActiveAgents,
runtimeCountByStatus: statusCounts,
totalRuntimes: this.runtimes.size,
};
}
/**
* Set up event forwarding from runtime to ProjectManager listeners.
*/
private setupEventForwarding(
runtime: ProjectRuntime,
projectId: string,
projectName: string
): void {
// Forward task:created
runtime.on("task:created", (task: Task) => {
this.emit("task:created", { projectId, projectName, task });
this.logActivity("task:created", projectId, projectName, `Task ${task.id} created`, task.id, task.title).catch((err: unknown) => {
projectManagerLog.warn(`Failed to log task:created activity for ${projectId}:`, err);
});
});
// Forward task:moved
runtime.on("task:moved", (data: { task: Task; from: string; to: string }) => {
this.emit("task:moved", { projectId, projectName, task: data.task, from: data.from, to: data.to });
this.logActivity(
"task:moved",
projectId,
projectName,
`Task ${data.task.id} moved: ${data.from} → ${data.to}`,
data.task.id,
data.task.title,
{ from: data.from, to: data.to }
).catch((err: unknown) => {
projectManagerLog.warn(`Failed to log task:moved activity for ${projectId}:`, err);
});
});
// Forward task:updated
runtime.on("task:updated", (task: Task) => {
this.emit("task:updated", { projectId, projectName, task });
});
// Forward task:deleted
runtime.on("task:deleted", (task: Task, meta?: { githubIssueAction?: import("@fusion/core").GithubIssueAction }) => {
this.emit("task:deleted", { projectId, projectName, task, meta });
});
// Forward errors
runtime.on("error", (error: Error) => {
this.emit("error", { projectId, projectName, error });
this.logActivity(
"task:failed",
projectId,
projectName,
`Error in ${projectName}: ${error.message}`
).catch((err: unknown) => {
projectManagerLog.warn(`Failed to log task:failed activity for ${projectId}:`, err);
});
});
// Forward health changes
runtime.on("health-changed", (data: { status: RuntimeStatus; previous: RuntimeStatus }) => {
this.emit("health:changed", { projectId, projectName, status: data.status, previous: data.previous });
// Update health in CentralCore
this.centralCore.updateProjectHealth(projectId, {
status: data.status as "active" | "paused" | "errored" | "initializing",
updatedAt: new Date().toISOString(),
}).catch((err: unknown) => {
projectManagerLog.warn(`Failed to update health for ${projectId}:`, err);
});
});
}
/**
* Log activity to CentralCore's unified feed.
*/
private async logActivity(
type: "task:created" | "task:moved" | "task:failed",
projectId: string,
projectName: string,
details: string,
taskId?: string,
taskTitle?: string,
metadata?: Record<string, unknown>
): Promise<void> {
try {
await this.centralCore.logActivity({
type,
projectId,
projectName,
timestamp: new Date().toISOString(),
details,
taskId,
taskTitle,
metadata,
});
} catch (error) {
// Best-effort logging
projectManagerLog.warn("Failed to log activity:", error);
}
}
/**
* Acquire a global concurrency slot.
* Call this before starting task execution.
*/
async acquireGlobalSlot(projectId: string): Promise<boolean> {
try {
return await this.centralCore.acquireGlobalSlot(projectId);
} catch (error) {
projectManagerLog.error(`Failed to acquire global slot for ${projectId}:`, error);
return false;
}
}
/**
* Release a global concurrency slot.
* Call this after task execution completes.
*/
async releaseGlobalSlot(projectId: string): Promise<void> {
try {
await this.centralCore.releaseGlobalSlot(projectId);
} catch (error) {
projectManagerLog.error(`Failed to release global slot for ${projectId}:`, error);
}
}
async restartProjectRuntime(projectId: string, options?: { reason?: string; force?: boolean }): Promise<void> {
const runtime = this.runtimes.get(projectId);
if (!runtime) {
throw new Error(`Runtime not found for project ${projectId}`);
}
const metrics = runtime.getMetrics();
if (!options?.force && metrics.inFlightTasks > 0) {
throw { kind: "active_tasks", count: metrics.inFlightTasks } satisfies ProjectRuntimeRestartBlockedError;
}
const project = await this.centralCore.getProject(projectId);
if (!project) {
throw new Error(`Project not found: ${projectId}`);
}
const workingDirectory = await this.centralCore.resolveLocalProjectWorkingDirectory(projectId);
await this.removeProject(projectId);
await this.addProject({
projectId,
workingDirectory,
isolationMode: project.isolationMode,
maxConcurrent: project.settings?.maxConcurrent ?? 2,
maxWorktrees: project.settings?.maxWorktrees ?? 4,
settings: project.settings,
});
this.emit("project:runtime-restarted", {
projectId,
projectName: project.name,
isolationMode: project.isolationMode,
reason: options?.reason,
});
}
/**
* Stop all runtimes and clean up.
*/
async stopAll(): Promise<void> {
projectManagerLog.log("Stopping all project runtimes...");
const stopPromises = Array.from(this.runtimes.keys()).map((id) =>
this.removeProject(id).catch((error) => {
projectManagerLog.error(`Failed to stop runtime ${id}:`, error);
})
);
await Promise.all(stopPromises);
clearInterval(this.globalLimitRefreshInterval);
if (typeof this.centralCore.off === "function") {
this.centralCore.off("concurrency:changed", this.concurrencyListener);
}
projectManagerLog.log("All project runtimes stopped");
this.removeAllListeners();
}
}