diff --git a/.changeset/fn-7723-cross-process-agent-notify.md b/.changeset/fn-7723-cross-process-agent-notify.md new file mode 100644 index 0000000000..816e366254 --- /dev/null +++ b/.changeset/fn-7723-cross-process-agent-notify.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": patch +--- + +summary: The engine now reacts to CLI `fn agent stop`/`start` promptly instead of waiting up to a minute for the audit sweep. +category: fix +dev: AgentStore gains opt-in cross-process change detection (fs.watch + poll fallback, modeled on TaskStore) that re-emits the existing agent:updated/agent:stateChanged events in the engine process when another process (the fn CLI) mutates an agent row, so HeartbeatTriggerScheduler's listeners fire without waiting for the 60s auditTimerRegistrations sweep. The audit sweep is retained as the durable backstop (FN-7723, follow-up from FN-7718). diff --git a/docs/agents.md b/docs/agents.md index 6a7192d1f8..877c26c31e 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -309,6 +309,7 @@ Layered enforcement: - **Heartbeat reconciliation (`packages/engine/src/agent-heartbeat.ts`)**: `reconcileOrphanedRunningAgents()` repairs persisted `state="running"` drift when no active run exists, or when a persisted active run is untracked and older than `heartbeatTimeoutMs × 3` (work-budget grace; see inline FN-4278/FN-4255 comment near that check). - **Heartbeat scheduler stale-run reap (`packages/engine/src/agent-heartbeat.ts`)**: `HeartbeatTriggerScheduler.maybeReapStaleActiveRun()` repairs stale persisted `status="active"` heartbeat runs using `heartbeatTimeoutMs × heartbeatRepairStaleMultiplier` (default multiplier `2`). - **Self-healing reconciler (`packages/engine/src/self-healing.ts`)**: `recoverAgentsRunningOnInactiveTasks()` and `recoverStaleHeartbeatRuns()` use task-column mismatch checks plus PID/young-run/age guards (including the 6h stale active-run max age) rather than a simple timeout multiplier. +- **Cross-process CLI stop/start notification (FN-7723, follow-up from FN-7718):** `fn agent stop`/`start` mutate the agent row from a separate short-lived CLI process. The long-lived engine `AgentStore` now opts into a bounded `fs.watch`+poll change-detection fast-path (`startWatching()`/`checkForChanges()`, modeled on `TaskStore`'s own watcher — see `docs/architecture.md`'s FN-7723 note) that re-emits the existing `agent:updated`/`agent:stateChanged` events, so `HeartbeatTriggerScheduler` typically observes a CLI-driven stop/start within one poll interval (~2s) instead of waiting on the 60s `auditTimerRegistrations` sweep, which remains the durable backstop. Manual recovery for pre-fix stuck rows: 1. Open the agent in Dashboard → Agent Detail. diff --git a/docs/architecture.md b/docs/architecture.md index 11eac9c829..cbbdf09a2f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1354,6 +1354,7 @@ Limits are controlled by project settings (`maxSpawnedAgentsPerParent`, `maxSpaw - Assignment triggers skipped because a heartbeat run is already active are deferred and re-fired from `HeartbeatMonitor.onRunCompleted`, preserving the existing completion recovery path while avoiding timer-dependent stalls. - FN-7645: `HeartbeatTriggerScheduler.auditTimerRegistrations` (60s cadence) repairs both MISSING timer registrations and "zombie" ones — a timer map entry that stays present after its underlying `setInterval` silently stops firing. When a tickable, heartbeat-managed agent's `lastHeartbeatAt` exceeds the repair-stale threshold (`heartbeatIntervalMs * heartbeatRepairStaleMultiplier`, default 2x) even though a timer entry already exists, the audit clears and re-registers it (phase-aligned via `computeInitialDelayMs`), logging `reason=zombie-timer-rearmed`. This closes the gap where long-interval (~1h) agents could silently drift stale for hours because their sparse cadence meant a single lost tick was never re-armed by the previous missing-registration-only repair; short-interval agents were unaffected because their frequent ticks self-heal within minutes. All existing guards are preserved: pause suppression (`globalPause`/`enginePaused`) still gates dispatch in `onTimerTick`, FN-4119 stale active-run reaping still runs first for agents with a live run, ephemeral/task-worker agents stay excluded via `isTimerEligibleAgent`, and `registrationEpochs` staleness protection is untouched. - FN-7718: CLI-driven `fn agent stop`/`start` mutate the agent row from a SEPARATE process, so the in-process `agent:updated` listener never fires for those transitions — the 60s audit is the ONLY cross-process reconciliation path. The audit now invalidates a stopped/non-eligible agent's lingering timer entry (state made non-tickable, `runtimeConfig.enabled === false`, or ephemeral/`!isHeartbeatManaged`) instead of bare-`continue`ing past it, so the entry never survives to become an orphaned/"zombie" registration. `syncTimerForAgent` mirrors this for the in-process start seam: an eligible agent whose present timer entry is already stale beyond the same repair threshold is force-cleared and re-armed rather than left in place by the "already ticking" no-op. Net effect: a `stop`/`start` cycle durably clears the zombie-timer condition in one audit cycle instead of deferring repair to the FN-7645 stale-repair path minutes later. +- FN-7723 (follow-up from FN-7718): `AgentStore` (`packages/core/src/agent-store.ts`) now supports an opt-in cross-process change-detection fast-path over the FN-7645/FN-7718 audit backstop — `startWatching()`/`stopWatching()`/`checkForChanges()`, modeled directly on `TaskStore`'s `fs.watch`+poll pattern (`packages/core/src/store.ts`): an `fs.watch` on the project's `.fusion` dir as a fail-soft fast-path nudge, plus an always-on poll fallback (default 2s) gated by `db.getLastModified()` so an unchanged DB costs one cheap `__meta` read. On a detected change it diffs current agent rows against a last-seen per-instance snapshot (comparing `state` explicitly, not just `updatedAt`, since two rapid writes can land in the same ISO-millisecond and mask a genuine transition) and re-emits the EXISTING `agent:updated`/`agent:stateChanged` events — no new event names, so `HeartbeatTriggerScheduler.watchAgentLifecycle`'s current listener reacts unchanged, funneling through the same `syncTimerForAgent` seam (including FN-7718's stale-present-entry force-re-arm). Only the long-lived engine `AgentStore` instance opts in (started/stopped alongside `HeartbeatTriggerScheduler` in `packages/engine/src/runtimes/in-process-runtime.ts`); the CLI's short-lived `AgentStore` (`packages/cli/src/commands/agent.ts`) and per-request dashboard stores never call `startWatching()`. The 60s `auditTimerRegistrations` sweep is UNCHANGED and remains the durable backstop — this is a purely additive latency improvement, not a replacement: a `fn agent stop`/`start` is now typically observed within one poll interval (~2s) instead of up to 60s. ### Custom instructions `packages/engine/src/agent-instructions.ts` resolves per-agent instruction text/path with path-traversal and extension validation. diff --git a/packages/core/src/__tests__/agent-store.test.ts b/packages/core/src/__tests__/agent-store.test.ts index 5f48d8ae8d..800c0888d4 100644 --- a/packages/core/src/__tests__/agent-store.test.ts +++ b/packages/core/src/__tests__/agent-store.test.ts @@ -3056,4 +3056,181 @@ describe("AgentStore", () => { expect([run1.id, run2.id]).toContain(limitedRunId); expect(applyRun.applied + applyRun.skipped).toBeGreaterThanOrEqual(0); }); + + /* + * FN-7723: cross-process change detection. `fn agent stop`/`start` mutate + * agent rows from a SEPARATE process (a separate short-lived AgentStore), + * so a long-lived engine AgentStore's in-process `agent:updated` listener + * never fires for them without this watch/poll re-emit path. These tests + * drive `checkForChanges()` directly (per docs/testing.md's "no real + * polling waits" rule) rather than waiting on the real setInterval. + */ + describe("cross-process change detection (FN-7723)", () => { + it("a second AgentStore watching the same DB re-emits agent:updated and agent:stateChanged for a state change it did not write", async () => { + // Cross-instance persistence requires disk-backed stores (see the + // "SQLite persistence" describe block above for the same swap). + store.close(); + store = new AgentStore({ rootDir }); + await store.init(); + + const agent = await store.createAgent({ name: "WatchedAgent", role: "executor" }); + expect(agent.state).toBe("active"); + + const reader = new AgentStore({ rootDir }); + await reader.init(); + const readerUpdated = vi.fn(); + const readerStateChanged = vi.fn(); + reader.on("agent:updated", readerUpdated); + reader.on("agent:stateChanged", readerStateChanged); + + try { + await reader.startWatching(); + expect(reader.isWatching()).toBe(true); + + // Writer (a different AgentStore instance) mutates the agent state — + // this is the exact cross-process shape of `fn agent stop`. + const updated = await store.updateAgentState(agent.id, "paused"); + + // Drive the reader's poll cycle directly instead of waiting 2s. + await reader.checkForChanges(); + + expect(readerStateChanged).toHaveBeenCalledTimes(1); + expect(readerStateChanged).toHaveBeenCalledWith(agent.id, "active", "paused"); + expect(readerUpdated).toHaveBeenCalledTimes(1); + expect(readerUpdated).toHaveBeenCalledWith( + expect.objectContaining({ id: agent.id, state: "paused", updatedAt: updated.updatedAt }), + "active", + ); + + // A second poll cycle with no further writes must not re-emit. + readerUpdated.mockClear(); + readerStateChanged.mockClear(); + await reader.checkForChanges(); + expect(readerUpdated).not.toHaveBeenCalled(); + expect(readerStateChanged).not.toHaveBeenCalled(); + } finally { + reader.close(); + } + }); + + it("does not double-emit for a write this same watching instance originated", async () => { + store.close(); + store = new AgentStore({ rootDir }); + await store.init(); + + const agent = await store.createAgent({ name: "SelfWriteAgent", role: "executor" }); + + const updated = vi.fn(); + store.on("agent:updated", updated); + + await store.startWatching(); + updated.mockClear(); // startWatching() itself does not emit + + await store.updateAgentState(agent.id, "paused"); + // In-process write already emitted synchronously inside updateAgentState. + expect(updated).toHaveBeenCalledTimes(1); + + // The next poll cycle must not find a "new" change — the write already + // updated this instance's own snapshot cache via writeAgent(). + updated.mockClear(); + await store.checkForChanges(); + expect(updated).not.toHaveBeenCalled(); + }); + + it("stopWatching() and close() clear the watcher and poll interval handles", async () => { + store.close(); + store = new AgentStore({ rootDir }); + await store.init(); + + await store.startWatching(); + expect(store.isWatching()).toBe(true); + + store.stopWatching(); + expect(store.isWatching()).toBe(false); + + // stopWatching() is idempotent. + expect(() => store.stopWatching()).not.toThrow(); + + // close() also stops watching for callers that skip the explicit call. + await store.startWatching(); + expect(store.isWatching()).toBe(true); + store.close(); + // Re-open to assert cleanly without leaking a handle across tests; the + // outer afterEach() will close() this fresh instance. + store = new AgentStore({ rootDir }); + await store.init(); + expect(store.isWatching()).toBe(false); + }); + + it("logs a watcher error and keeps the poll fallback operational", async () => { + store.close(); + store = new AgentStore({ rootDir }); + await store.init(); + + const agent = await store.createAgent({ name: "PollFallbackAgent", role: "executor" }); + + const reader = new AgentStore({ rootDir }); + await reader.init(); + const readerUpdated = vi.fn(); + reader.on("agent:updated", readerUpdated); + const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {}); + + try { + await reader.startWatching(); + const readerAny = reader as unknown as { watcher: { emit: (event: string, err: Error) => void } | null }; + + // Simulate a degraded fs.watch (or its unavailability, matching + // TaskStore's own fail-soft test shape) — the poll fallback must + // still detect the change regardless of the watcher's health. + if (readerAny.watcher) { + readerAny.watcher.emit("error", new Error("watcher degraded (simulated)")); + const watcherErrorCall = warnSpy.mock.calls.find( + (call) => typeof call[0] === "string" && call[0].includes("fs.watch emitted an error; polling will continue"), + ); + expect(watcherErrorCall).toBeDefined(); + } else { + const fallbackCall = warnSpy.mock.calls.find( + (call) => typeof call[0] === "string" && call[0].includes("fs.watch unavailable; falling back to polling-only updates"), + ); + expect(fallbackCall).toBeDefined(); + } + + expect(reader.isWatching()).toBe(true); + + await store.updateAgentState(agent.id, "paused"); + await reader.checkForChanges(); + + expect(readerUpdated).toHaveBeenCalledTimes(1); + expect(readerUpdated).toHaveBeenCalledWith( + expect.objectContaining({ id: agent.id, state: "paused" }), + "active", + ); + } finally { + reader.close(); + warnSpy.mockRestore(); + } + }); + + it("checkForChanges() failures are logged and non-fatal", async () => { + store.close(); + store = new AgentStore({ rootDir }); + await store.init(); + await store.startWatching(); + + const listAgentsSpy = vi.spyOn(store, "listAgents").mockImplementationOnce(() => { + throw new Error("simulated listAgents failure"); + }); + const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {}); + + // Force a change to be detected so checkForChanges() attempts the + // (failing) listAgents() call rather than short-circuiting on an + // unchanged lastModified. + await store.createAgent({ name: "TriggerChange", role: "executor" }); + + await expect(store.checkForChanges()).resolves.toBeUndefined(); + + listAgentsSpy.mockRestore(); + warnSpy.mockRestore(); + }); + }); }); diff --git a/packages/core/src/agent-store.ts b/packages/core/src/agent-store.ts index 86c64358ef..ec1f9c0e31 100644 --- a/packages/core/src/agent-store.ts +++ b/packages/core/src/agent-store.ts @@ -8,7 +8,7 @@ */ import { mkdir, readFile, writeFile, readdir, unlink, rename, access, appendFile } from "node:fs/promises"; -import { constants as fsConstants } from "node:fs"; +import { constants as fsConstants, watch, type FSWatcher } from "node:fs"; import { basename, dirname, join, resolve } from "node:path"; import { randomUUID, randomBytes, createHash } from "node:crypto"; import { EventEmitter } from "node:events"; @@ -59,6 +59,9 @@ import { canAgentTakeImplementationTask, canAgentTakeImplementationTaskForExplic import { normalizeAgentPermissionPolicy } from "./agent-permission-policy.js"; import { Database } from "./db.js"; import { createAgentRunSnapshot, createAgentSnapshot, validateSnapshotEnvelope, type AgentRunSnapshot, type AgentSnapshot } from "./shared-mesh-state.js"; +import { createLogger } from "./logger.js"; + +const agentStoreLog = createLogger("agent-store"); /** Database row shape returned by SELECT on agentRatings. */ interface AgentRatingRow { @@ -252,6 +255,30 @@ export class AgentStore extends EventEmitter { private readonly defaultNodeId?: string; private readonly inMemoryDb: boolean; + /* + * FNXC:AgentStore 2026-07-09-08:15: + * FN-7723 — `fn agent stop`/`start` mutate the agent row from a SEPARATE + * process (the CLI opens its own AgentStore, writes, and exits), so the + * engine's long-lived in-process `agent:updated`/`agent:stateChanged` + * listeners (HeartbeatTriggerScheduler.watchAgentLifecycle) never fire for + * those transitions — the only prior reconciliation was the 60s + * `auditTimerRegistrations` sweep. This opt-in fs.watch+poll lifecycle + * (started ONLY by the long-lived engine store, never the CLI/dashboard + * short-lived stores) mirrors TaskStore's proven watch()/checkForChanges() + * pattern: watch the `.fusion` dir for a fast-path nudge, always run a + * poll fallback gated by db.getLastModified(), diff agent rows against a + * last-seen snapshot, and re-emit the EXISTING `agent:updated`/ + * `agent:stateChanged` events (no new event names) so the engine's current + * listeners react within a bounded latency instead of waiting up to 60s. + * The audit sweep is retained unmodified as the durable backstop. + */ + private watcher: FSWatcher | null = null; + private pollInterval: ReturnType | null = null; + /** Last-seen (state, updatedAt) per agent id, used to diff external changes. Populated on startWatching() and kept current by every in-process write via touchWatchSnapshot(). */ + private agentSnapshotCache: Map = new Map(); + private lastKnownModified = 0; + private pollingInProgress = false; + constructor(options: AgentStoreOptions = {}) { super(); @@ -1870,6 +1897,11 @@ export class AgentStore extends EventEmitter { this.db.prepare("DELETE FROM agents WHERE id = ?").run(agentId); this.db.bumpLastModified(); + // FN-7723: keep this instance's own change-detection snapshot in sync + // with its own delete so a later poll never mistakes the row's absence + // for an external delete (deletes are pruned from the cache, not + // re-emitted, since agent:deleted already fires below). + this.agentSnapshotCache.delete(agentId); this.emit("agent:deleted", agentId); }); @@ -2942,12 +2974,188 @@ export class AgentStore extends EventEmitter { JSON.stringify(data), ); this.db.bumpLastModified(); + + /* + * FNXC:AgentStore 2026-07-09-08:15: + * FN-7723 — update this instance's own change-detection snapshot cache + * synchronously with every in-process write (createAgent/updateAgent/ + * updateAgentState/etc. all funnel through writeAgent()). This is the + * self-write suppression mechanism: when this instance's own poll tick + * later diffs against the cache, its own write is already reflected, so + * it is never mistaken for an external change and never double-emitted. + * Only meaningful once startWatching() has populated the cache; a no-op + * write to a Map before watching starts is negligible cost. + */ + this.agentSnapshotCache.set(agent.id, this.watchSnapshotOf(agent)); + } + + /** + * FN-7723: build the change-detection snapshot for an agent. + * + * Deliberately does NOT key equality on `updatedAt` alone: `updatedAt` is a + * millisecond-resolution ISO string, and two writes issued in quick + * succession (e.g. `createAgent()` immediately followed by + * `updateAgentState()` in a test, or two rapid CLI mutations) can land in + * the SAME millisecond, producing an identical `updatedAt` even though + * `state` genuinely changed. Comparing `state` explicitly (in addition to + * `updatedAt`, which still catches non-state metadata churn) closes that + * race so a stop/start transition is never silently missed. + */ + private watchSnapshotOf(agent: Agent): { state: AgentState; updatedAt: string } { + return { state: agent.state, updatedAt: agent.updatedAt }; + } + + // ── Cross-process change detection (FN-7723) ──────────────────────────────────── + + /** + * Whether this store is actively watching for cross-process changes + * (fs.watch and/or poll fallback registered). + */ + isWatching(): boolean { + return this.watcher !== null || this.pollInterval !== null; + } + + /** + * Start opt-in cross-process change detection. + * + * Only the long-lived engine `AgentStore` instance should call this (see + * packages/engine/src/runtimes/in-process-runtime.ts). The CLI's + * short-lived `AgentStore` (packages/cli/src/commands/agent.ts) and + * per-request dashboard stores must NOT call this — they mutate and exit + * (or are ephemeral) so there is nothing for them to watch. + * + * Mirrors TaskStore.watch(): a sentinel `fs.watch` on the rootDir for a + * fast-path nudge (fail-soft if unavailable), plus an always-on poll + * fallback gated by `db.getLastModified()` so a no-op tick is one cheap + * `__meta` read. + */ + async startWatching(pollIntervalMs = 2000): Promise { + if (this.watcher || this.pollInterval) return; // already watching + + const agents = await this.listAgents({ includeEphemeral: true }); + this.agentSnapshotCache.clear(); + for (const agent of agents) { + this.agentSnapshotCache.set(agent.id, this.watchSnapshotOf(agent)); + } + this.lastKnownModified = this.db.getLastModified(); + + try { + this.watcher = watch(this.rootDir, (_event, _filename) => { + // No-op — the poll fallback below does the actual diffing; fs.watch + // here only exists as a fast-path nudge candidate and for API/close + // symmetry, matching TaskStore's own watch() implementation. + }); + this.watcher.on("error", (err) => { + agentStoreLog.warn("fs.watch emitted an error; polling will continue", { + phase: "watch:fs-watch-error", + error: err instanceof Error ? err.message : String(err), + rootDir: this.rootDir, + }); + }); + } catch (err) { + // fs.watch may not be available on this platform/filesystem — that's + // fine, the poll fallback below is the reliable path. + agentStoreLog.warn("fs.watch unavailable; falling back to polling-only updates", { + phase: "watch:fs-watch-setup", + error: err instanceof Error ? err.message : String(err), + rootDir: this.rootDir, + }); + } + + this.pollInterval = setInterval(() => { + void this.checkForChanges(); + }, pollIntervalMs); + } + + /** + * Diff current agent rows against the last-seen snapshot and re-emit the + * EXISTING `agent:updated`/`agent:stateChanged` events for any agent whose + * `state` or `updatedAt` advanced since this instance last observed it. + * Gated by `db.getLastModified()` so an unchanged DB costs one `__meta` + * SELECT. Compares `state` explicitly (not just `updatedAt`) so a genuine + * transition is never masked by an `updatedAt` millisecond collision + * between two rapid writes (see `watchSnapshotOf()`). + * + * Exposed (not private) so tests can drive a poll cycle directly instead + * of waiting on the real setInterval, per docs/testing.md's "no real + * polling waits" rule — mirrors TaskStore's own testable checkForChanges(). + */ + async checkForChanges(): Promise { + if (this.pollingInProgress) return; + this.pollingInProgress = true; + try { + const currentModified = this.db.getLastModified(); + if (currentModified <= this.lastKnownModified) return; + this.lastKnownModified = currentModified; + + const agents = await this.listAgents({ includeEphemeral: true }); + const seenIds = new Set(); + for (const agent of agents) { + seenIds.add(agent.id); + const cached = this.agentSnapshotCache.get(agent.id); + if (!cached) { + // A brand-new agent row this instance has never seen. createAgent() + // already emits agent:created in-process; a cross-process create is + // out of scope for this task (agents are created via the dashboard/ + // CLI in the same flow that would also emit agent:created there). + // Just seed the cache so future updates to it diff correctly. + this.agentSnapshotCache.set(agent.id, this.watchSnapshotOf(agent)); + continue; + } + const stateChanged = cached.state !== agent.state; + const updatedAtChanged = cached.updatedAt !== agent.updatedAt; + if (!stateChanged && !updatedAtChanged) continue; + + const previousState = cached.state; + this.agentSnapshotCache.set(agent.id, this.watchSnapshotOf(agent)); + + if (stateChanged) { + this.emit("agent:stateChanged", agent.id, previousState, agent.state); + this.emit("agent:updated", agent, previousState); + } else { + this.emit("agent:updated", agent); + } + } + + // Prune cache entries for agents deleted by another process. deleteAgent() + // already emits agent:deleted in-process for the originating instance; + // cross-process delete detection is not required by this task's scope + // (delete is a rare, operator-driven action, unlike stop/start), so we + // just keep the cache from growing unbounded. + for (const id of this.agentSnapshotCache.keys()) { + if (!seenIds.has(id)) this.agentSnapshotCache.delete(id); + } + } catch (err) { + agentStoreLog.warn("checkForChanges poll cycle failed", { + lastKnownModified: this.lastKnownModified, + error: err instanceof Error ? err.message : String(err), + }); + } finally { + this.pollingInProgress = false; + } + } + + /** + * Stop cross-process change detection and clear all handles. + */ + stopWatching(): void { + if (this.watcher) { + this.watcher.close(); + this.watcher = null; + } + if (this.pollInterval) { + clearInterval(this.pollInterval); + this.pollInterval = null; + } + this.agentSnapshotCache.clear(); } /** * Close the underlying SQLite connection and release resources. */ close(): void { + this.stopWatching(); + if (!this._db) { return; } diff --git a/packages/engine/src/__tests__/heartbeat-scheduler.test.ts b/packages/engine/src/__tests__/heartbeat-scheduler.test.ts index 17ded011ac..f91b502ac8 100644 --- a/packages/engine/src/__tests__/heartbeat-scheduler.test.ts +++ b/packages/engine/src/__tests__/heartbeat-scheduler.test.ts @@ -1002,6 +1002,168 @@ describe("HeartbeatTriggerScheduler", () => { expect(scheduler.getRegisteredAgents()).not.toContain("agent-repeat-stop"); }); }); + + /* + * FNXC:AgentHeartbeat 2026-07-09-08:15: + * FN-7723 — the cross-process notification fast-path over the FN-7718/ + * FN-7645 audit backstop. `AgentStore.checkForChanges()` re-emits the + * SAME `agent:updated`/`agent:stateChanged` events exercised by the + * "agent lifecycle seam registration" tests above; this describe proves + * `watchAgentLifecycle`'s existing listener reacts to a re-emitted + * EXTERNAL event (simulated here by emitting directly on the + * EventEmitter-backed store, exactly as AgentStore.checkForChanges() + * would after diffing a cross-process write) WITHOUT the 60s audit ever + * running, and that the audit still works as the backstop when no event + * arrives at all. No new engine-side listener/seam was added for this + * task — syncTimerForAgent handles the re-emitted event identically to + * an in-process one, so this is a pure regression/characterization + * suite over the existing wiring plus FN-7718's stale-repair guard. + */ + describe("FN-7723: re-emitted external agent:updated drives the timer without the audit", () => { + // Self-contained EventEmitter-backed store mirroring `createLifecycleStore` + // from the "agent lifecycle seam registration" describe above (out of + // scope here since this sits inside "scheduler timer audit"). Emitting + // directly on this store simulates AgentStore.checkForChanges()'s + // re-emit of the EXISTING `agent:updated` event after diffing a + // cross-process write — no new event names, no engine-side seam. + type CrossProcessStore = EventEmitter & Pick & { + agents: Map; + }; + + function buildCrossProcessAgent(overrides: Partial & { id: string; heartbeatIntervalMs: number }): Agent { + const { heartbeatIntervalMs, ...rest } = overrides; + return { + name: rest.id, + role: "executor", + state: "active", + lastHeartbeatAt: "2026-01-01T00:00:00.000Z", + runtimeConfig: { enabled: true, heartbeatIntervalMs }, + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:00:00.000Z", + metadata: {}, + ...rest, + } as Agent; + } + + function createCrossProcessStore(initialAgents: Agent[] = []): CrossProcessStore { + return Object.assign(new EventEmitter(), { + agents: new Map(initialAgents.map((agent) => [agent.id, agent])), + getAgent: vi.fn(async function(this: CrossProcessStore, agentId: string) { + return this.agents.get(agentId) ?? null; + }), + getActiveHeartbeatRun: vi.fn().mockResolvedValue(null), + getBudgetStatus: vi.fn().mockResolvedValue(createBudgetStatus()), + listAgents: vi.fn(async function(this: CrossProcessStore) { + return Array.from(this.agents.values()); + }), + getRecentRuns: vi.fn().mockResolvedValue([]), + updateAgent: vi.fn(), + }) as CrossProcessStore; + } + + it("a re-emitted external stop clears the timer and a re-emitted external start re-arms it, with zero audit cycles elapsed", async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z")); + + const agent = buildCrossProcessAgent({ id: "agent-cross-process", heartbeatIntervalMs: 300_000 }); + const eventStore = createCrossProcessStore([agent]); + scheduler = new HeartbeatTriggerScheduler(eventStore as unknown as AgentStore, callback); + scheduler.start(); + eventStore.emit("agent:created", agent); + expect(scheduler.getRegisteredAgents()).toContain(agent.id); + + // Simulate AgentStore.checkForChanges() re-emitting agent:updated for a + // CLI `fn agent stop` it detected on its next poll tick — well before + // the 60s audit interval elapses. + const stopped = { ...eventStore.agents.get(agent.id)!, state: "paused" as const }; + eventStore.agents.set(agent.id, stopped); + await vi.advanceTimersByTimeAsync(2_000); // one AgentStore poll tick (2s default), zero audit cycles (60s) + eventStore.emit("agent:updated", stopped, "active"); + + expect(scheduler.getRegisteredAgents()).not.toContain(agent.id); + + // Simulate the re-emitted external `fn agent start`, still with no + // audit cycle having elapsed. + const started = { ...eventStore.agents.get(agent.id)!, state: "active" as const }; + eventStore.agents.set(agent.id, started); + await vi.advanceTimersByTimeAsync(2_000); + eventStore.emit("agent:updated", started, "paused"); + + expect(scheduler.getRegisteredAgents()).toContain(agent.id); + const timers = (scheduler as unknown as { timers: Map }).timers; + expect(timers.has(agent.id)).toBe(true); + + // Total elapsed time (4s) never reached a single 60s audit cycle. + expect(callback).not.toHaveBeenCalled(); + }); + + it("honors FN-7718's stale-present-entry force-re-arm when the re-emitted event arrives after the timer entry went stale", async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z")); + + const agent = buildCrossProcessAgent({ id: "agent-cross-process-stale", heartbeatIntervalMs: 3_600_000 }); + const eventStore = createCrossProcessStore([agent]); + scheduler = new HeartbeatTriggerScheduler(eventStore as unknown as AgentStore, callback); + scheduler.start(); + eventStore.emit("agent:created", agent); + expect(scheduler.getRegisteredAgents()).toContain(agent.id); + + // Advance well past the 2x stale threshold (7.2M ms) with the entry + // still present — the audit is never driven here, only the poll-detected + // re-emit path. + await vi.advanceTimersByTimeAsync(8 * 60 * 60 * 1000); + vi.mocked(heartbeatLog.warn).mockClear(); + + // A re-emitted external start (CLI `fn agent start`, surfaced by + // AgentStore.checkForChanges()) while the stale entry is still present + // must force re-arm exactly like an in-process start would (FN-7718). + const started = { ...eventStore.agents.get(agent.id)!, state: "active" as const }; + eventStore.agents.set(agent.id, started); + eventStore.emit("agent:updated", started, "paused"); + + expect(heartbeatLog.warn).toHaveBeenCalledWith(expect.stringContaining("Timer sync force re-armed stale present entry")); + expect(scheduler.getRegisteredAgents()).toContain(agent.id); + }); + + it("the 60s audit still reconciles a stop/start when NO re-emitted event ever arrives (backstop retained)", async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z")); + + // Use the plain (non-EventEmitter) mocked store so no agent:updated + // event can ever fire — the ONLY path that can reconcile this stop is + // the audit's listAgents() sweep, proving the backstop is untouched by + // this task's additive fast-path. + let agent: Agent = { + id: "agent-audit-backstop", + name: "agent-audit-backstop", + role: "executor", + state: "active", + lastHeartbeatAt: "2026-01-01T00:00:00.000Z", + runtimeConfig: { enabled: true, heartbeatIntervalMs: 300_000 }, + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:00:00.000Z", + metadata: {}, + }; + vi.mocked(store.listAgents).mockImplementation(async () => [agent]); + vi.mocked(store.getActiveHeartbeatRun).mockResolvedValue(null); + + scheduler = new HeartbeatTriggerScheduler(store, callback); + scheduler.start(); + await vi.advanceTimersByTimeAsync(0); + expect(scheduler.getRegisteredAgents()).toContain("agent-audit-backstop"); + + // Mutate the DB row out-of-process (no event, ever) — same shape as the + // FN-7718 "clears an orphaned timer entry" test, kept here to assert the + // audit backstop is unchanged by FN-7723's additive fast-path. + agent = { ...agent, state: "paused" }; + await vi.advanceTimersByTimeAsync(3 * 60_000); // several audit cycles + expect(scheduler.getRegisteredAgents()).not.toContain("agent-audit-backstop"); + + agent = { ...agent, state: "active", lastHeartbeatAt: "2026-01-01T00:00:00.000Z" }; + await vi.advanceTimersByTimeAsync(60_000); + expect(scheduler.getRegisteredAgents()).toContain("agent-audit-backstop"); + }); + }); }); describe("registerAgent", () => { diff --git a/packages/engine/src/runtimes/in-process-runtime.ts b/packages/engine/src/runtimes/in-process-runtime.ts index 9c14c98de4..fd802f030f 100644 --- a/packages/engine/src/runtimes/in-process-runtime.ts +++ b/packages/engine/src/runtimes/in-process-runtime.ts @@ -391,6 +391,23 @@ export class InProcessRuntime agentStoreForReflection = new AgentStoreClass({ rootDir: this.taskStore.getFusionDir(), taskStore: this.taskStore }); await agentStoreForReflection.init(); runtimeLog.log("AgentStore initialized for reflection service"); + + /* + * FNXC:AgentStore 2026-07-09-08:15: + * FN-7723 — this is the ONE long-lived engine AgentStore instance for + * this project/runtime, so it is the only one that should opt into + * cross-process change detection (fs.watch + poll re-emitting + * agent:updated/agent:stateChanged). The CLI's short-lived AgentStore + * (packages/cli/src/commands/agent.ts) and dashboard per-request + * stores never call startWatching(). Failure here is non-fatal — the + * 60s auditTimerRegistrations sweep remains the durable backstop. + */ + try { + await agentStoreForReflection.startWatching(); + runtimeLog.log("AgentStore cross-process change detection started"); + } catch (watchErr) { + runtimeLog.warn(`AgentStore.startWatching() failed (falling back to the 60s audit sweep only):`, watchErr instanceof Error ? watchErr.message : watchErr); + } } catch (agentErr) { runtimeLog.warn(`AgentStore initialization failed (reflection service will be unavailable):`, agentErr instanceof Error ? agentErr.message : agentErr); } @@ -1121,6 +1138,19 @@ export class InProcessRuntime runtimeLog.log("TriggerScheduler stopped"); } + // 4a. Stop AgentStore cross-process change detection (FN-7723). This + // engine store is the only AgentStore instance in this process that + // ever called startWatching(); stopping it here clears the fs.watch + // handle and poll interval so shutdown does not leak them. + if (this.agentStore) { + try { + this.agentStore.stopWatching(); + runtimeLog.log("AgentStore cross-process change detection stopped"); + } catch (watchStopErr) { + runtimeLog.warn(`AgentStore.stopWatching() failed:`, watchStopErr instanceof Error ? watchStopErr.message : watchStopErr); + } + } + // 4. Stop stuck task detector if (this.stuckTaskDetector) { this.stuckTaskDetector.stop();