FN-9109: add resilient cross-runtime Cursor fallback
Route eligible CLI fallback failures into one bounded, auditable Cursor runtime handoff. - defer cursor-cli fallback selection until a retryable primary prompt failure - serialize concurrent swaps, retry the primary after failed handoffs, and dispose replacements safely - transfer text-only conversation context within strict turn and total character limits - document routing behavior and cover provider, runtime, concurrency, and failure cases Files changed: .../fn-9109-cross-runtime-cursor-fallback.md | 7 + AGENTS.md | 2 + docs/cursor-cli-contract.md | 8 + .../src/__tests__/cli-provider-routing.test.ts | 18 +- .../cli-runtime-routing-conformance.test.ts | 64 ++++- .../src/__tests__/cross-runtime-fallback.test.ts | 146 +++++++++++ .../engine/src/agents/agent-session-helpers.ts | 211 ++++++---------- packages/engine/src/agents/cli-provider-routing.ts | 4 +- .../engine/src/agents/cross-runtime-fallback.ts | 277 +++++++++++++++++++++ packages/engine/src/util/run-audit.ts | 8 + 10 files changed, 602 insertions(+), 143 deletions(-) Fusion-Task-Id: FN-9109 Fusion-Task-Lineage: 30664371-8410-4d6f-a263-cc86c8e0dc72 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-9109-cross-runtime-cursor-fallback.md
Normal file
7
.changeset/fn-9109-cross-runtime-cursor-fallback.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
summary: Enable Cursor CLI as a retryable fallback for other AI runtimes.
|
||||
category: feature
|
||||
dev: Adds the cross-runtime fallback dispatcher and the cursor-cli defer-cross-runtime policy.
|
||||
@@ -276,6 +276,8 @@ Scoped exception (FN-5819/FN-8823): while project auto-merge is On, shared-branc
|
||||
`testMode?: boolean` is now available in both project and global settings. If project `testMode === true` (or the resolved default provider is `"mock"` at any tier), every AI lane is forced to `mock/scripted`, overriding per-task and per-lane model selections. The dashboard exposes this via the Settings Modal "Enable test mode" toggle and a persistent "Test mode — no real AI calls" banner.
|
||||
|
||||
### Run Audit
|
||||
- FN-9109: `session:cross-runtime-fallback-engaged` records a single retryable-failure handoff from a primary runtime to a deferred CLI runtime. Metadata is ids/outcomes-only (`sessionPurpose`, primary/fallback provider and model IDs, trigger point, failure category, `contextTransferred`); never record error prose or transferred transcript text.
|
||||
|
||||
- FN-8958: `merge:orphan-write-fenced` is emitted once per orphan merge body at its fence's first interaction. Metadata is ids/counts/outcomes-only: `{ taskId, category, interaction, suppressedCount }`; `suppressedCount` is the emit-time count (`1` for `interaction:"suppressed"`, `0` for `interaction:"rejected"`), never a cumulative body total.
|
||||
|
||||
- Store-open provenance: every `TaskStore.init()` emits `store:open` with ids/paths-only metadata (`pid`, `ppid`, `execPath`, `entry`, `cwd`, `nodeVersion`). Purpose: attribute shared-DB mutations to the process that opened the store (the FN-7910 Ideas-evacuation writer was unidentifiable without it). Tests reading unfiltered `runAuditEvents` must filter out `store:open` rather than assert exact counts.
|
||||
|
||||
@@ -6,6 +6,14 @@ tool-name prefix, MCP approval semantics, and negotiated protocol. Fusion stages
|
||||
for that verified session contract; it never assumes a fallback CLI flag or config location.
|
||||
-->
|
||||
|
||||
## Cross-runtime fallback
|
||||
|
||||
When a non-Cursor primary model has a configured `cursor-cli` fallback, Fusion withholds that pair from the primary runtime and arms it only when the Cursor runtime plugin is registered. The first retryable model-selection failure creates one Cursor session, re-issues the failed prompt there, and routes all later prompts through that replacement; non-retryable failures and swap-time failures preserve the original primary error. Disposing the primary session also disposes the replacement.
|
||||
|
||||
Cursor has no cross-runtime resume token. On the first swapped prompt only, Fusion can prepend a clearly labeled, non-authoritative text-only transcript from the primary session: at most the last 10 turns, 2,000 characters per turn, and 12,000 characters total. Tool and thinking blocks are excluded; missing or malformed session state transfers nothing and leaves the prompt unchanged. If the Cursor runtime plugin is unavailable or cannot be looked up, Fusion keeps the primary runtime and drops the fallback with a warning instead of crashing.
|
||||
|
||||
A successful handoff emits `session:cross-runtime-fallback-engaged`. Its metadata is ids/outcomes-only (`sessionPurpose`, primary/fallback provider and model ids, trigger point, failure category, and whether context transferred); it never records error prose or transcript text.
|
||||
|
||||
## MCP staging and cleanup
|
||||
|
||||
Fusion creates a unique `fusion-custom-tools-<uuid>` server key per Cursor session. The `.cursor/.fusion-mcp-state.json` manifest retains the complete `{ command, args, env }` entry for every lease, allowing one process to recompose a peer process's live entry. Operator content is taken from current bytes; Fusion content is taken from that manifest.
|
||||
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
dropUnsupportedCliFallback,
|
||||
stripCliProviderPrefix,
|
||||
} from "../agents/cli-provider-routing.js";
|
||||
import { deferCrossRuntimeCliFallback } from "../agents/cross-runtime-fallback.js";
|
||||
|
||||
const options = (overrides: Record<string, unknown> = {}) => ({
|
||||
defaultProvider: undefined,
|
||||
@@ -66,9 +67,20 @@ describe("CLI provider routing census", () => {
|
||||
expect(applyCliRuntimeOptions(options({ defaultProvider: "openai", fallbackProvider: "omp-cli", fallbackModelId: "omp-cli/model" }), "omp")).toMatchObject({ defaultProvider: "omp-cli", defaultModelId: "model", fallbackProvider: undefined });
|
||||
});
|
||||
|
||||
it("uses a provider-named Cursor failure in both injected support states", () => {
|
||||
for (const _support of [false, true]) {
|
||||
expect(() => deriveCliRuntimeHint({ runtimeOptions: options({ defaultProvider: "cursor-cli" }), pluginRunner: runner(true) as never, grokApiKeyVisible: false })).toThrow(/Cursor CLI/);
|
||||
it("defers an available Cursor fallback and drops it when the runtime cannot be looked up", () => {
|
||||
const runtimeOptions = options({ defaultProvider: "openai", defaultModelId: "gpt", fallbackProvider: "cursor-cli", fallbackModelId: "cursor-cli/small", fallbackThinkingLevel: "high" });
|
||||
const deferred = deferCrossRuntimeCliFallback(runtimeOptions, runner(true) as never);
|
||||
expect(deferred.options).toMatchObject({ fallbackProvider: undefined, fallbackModelId: undefined, fallbackThinkingLevel: undefined });
|
||||
expect(deferred.deferred).toMatchObject({ providerId: "cursor-cli", runtimeId: "cursor", modelId: "small", thinkingLevel: "high" });
|
||||
expect(dropUnsupportedCliFallback(deferred.options).droppedProvider).toBeUndefined();
|
||||
for (const unavailable of [undefined, runner(false), runner(false, true)]) {
|
||||
expect(deferCrossRuntimeCliFallback(runtimeOptions, unavailable as never).dropped).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("uses a provider-named Cursor failure when its required runtime is unavailable", () => {
|
||||
for (const pluginRunner of [undefined, runner(false), runner(false, true)]) {
|
||||
expect(() => deriveCliRuntimeHint({ runtimeOptions: options({ defaultProvider: "cursor-cli" }), pluginRunner: pluginRunner as never, grokApiKeyVisible: false })).toThrow(/Cursor CLI/);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -5,11 +5,14 @@ import { CLI_PROVIDER_ROUTING_CENSUS, type CliProviderRouting } from "../agents/
|
||||
import type { PluginRunner } from "../plugins/plugin-runner.js";
|
||||
import type { PluginRuntimeRegistration } from "@fusion/core";
|
||||
|
||||
const mockCreateFnAgent = vi.hoisted(() => vi.fn());
|
||||
const { mockCreateFnAgent, mockPromptWithFallback } = vi.hoisted(() => ({
|
||||
mockCreateFnAgent: vi.fn(),
|
||||
mockPromptWithFallback: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../pi.js", () => ({
|
||||
createFnAgent: mockCreateFnAgent,
|
||||
promptWithFallback: vi.fn().mockResolvedValue(undefined),
|
||||
promptWithFallback: mockPromptWithFallback,
|
||||
describeModel: vi.fn().mockReturnValue("pi/default"),
|
||||
wrapToolsWithActionGate: vi.fn((tools) => tools),
|
||||
wrapToolsWithPermanentAgentGating: vi.fn((tools) => tools),
|
||||
@@ -74,6 +77,7 @@ describe("CLI provider routing conformance", () => {
|
||||
vi.restoreAllMocks();
|
||||
vi.spyOn(fusionCore, "isGrokApiKeyFusionVisible").mockReturnValue(false);
|
||||
mockCreateFnAgent.mockReset().mockResolvedValue({ session: { runtimeId: "pi" } });
|
||||
mockPromptWithFallback.mockReset().mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
it.each(CLI_PROVIDER_ROUTING_CENSUS.filter((entry) => entry.classification === "runtime-routed"))(
|
||||
@@ -185,6 +189,13 @@ describe("CLI provider routing conformance", () => {
|
||||
}));
|
||||
const absent = await createResolvedAgentSession(options(undefined, { pluginRunner }));
|
||||
expect(fallback.runtimeId).toBe(entry.fallbackPolicy === "promote-to-primary" ? "omp" : "pi");
|
||||
if (entry.fallbackPolicy === "defer-cross-runtime") {
|
||||
expect(mockCreateFnAgent).toHaveBeenCalledWith(expect.objectContaining({
|
||||
defaultProvider: "openai",
|
||||
fallbackProvider: undefined,
|
||||
fallbackModelId: undefined,
|
||||
}));
|
||||
}
|
||||
expect(absent.runtimeId).toBe("pi");
|
||||
},
|
||||
);
|
||||
@@ -199,6 +210,55 @@ describe("CLI provider routing conformance", () => {
|
||||
expect(pluginRunner.getRuntimeById).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("swaps a deferred Cursor fallback after a retryable primary failure with portable context", async () => {
|
||||
const cursorPrompt = vi.fn().mockResolvedValue("cursor result");
|
||||
const cursorCreateSession = vi.fn().mockResolvedValue({ session: {} });
|
||||
const pluginRunner = {
|
||||
getRuntimeById: vi.fn(() => ({
|
||||
pluginId: "fusion-plugin-cursor-runtime",
|
||||
runtime: {
|
||||
metadata: { runtimeId: "cursor", name: "Cursor" },
|
||||
factory: vi.fn().mockResolvedValue({
|
||||
id: "cursor", name: "Cursor", createSession: cursorCreateSession,
|
||||
promptWithFallback: cursorPrompt, describeModel: vi.fn(),
|
||||
}),
|
||||
},
|
||||
})),
|
||||
createRuntimeContext: vi.fn().mockResolvedValue({ pluginId: "fusion-plugin-cursor-runtime", taskStore: {}, settings: {}, logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, emitEvent: vi.fn() }),
|
||||
} as unknown as PluginRunner;
|
||||
mockCreateFnAgent.mockResolvedValue({ session: { state: { messages: [{ role: "assistant", content: "prior answer" }] } } });
|
||||
vi.mocked((await import("../pi.js")).isRetryableModelSelectionError).mockReturnValue(true);
|
||||
mockPromptWithFallback.mockRejectedValueOnce(new Error("rate limit"));
|
||||
const database = vi.fn();
|
||||
const result = await createResolvedAgentSession(options(undefined, {
|
||||
defaultProvider: "openai", defaultModelId: "primary", fallbackProvider: "cursor-cli", fallbackModelId: "cursor-cli/small",
|
||||
pluginRunner, runAuditor: { database },
|
||||
}));
|
||||
await expect((result.session as { promptWithFallback: (prompt: string) => Promise<unknown> }).promptWithFallback("current prompt")).resolves.toBe("cursor result");
|
||||
expect(cursorCreateSession).toHaveBeenCalledOnce();
|
||||
expect(cursorPrompt).toHaveBeenCalledWith(expect.anything(), expect.stringContaining("prior answer"), undefined);
|
||||
expect(database).toHaveBeenCalledWith(expect.objectContaining({
|
||||
type: "session:runtime-resolved",
|
||||
metadata: expect.objectContaining({ crossRuntimeFallbackDeferred: true }),
|
||||
}));
|
||||
expect(database).toHaveBeenCalledWith(expect.objectContaining({ type: "session:cross-runtime-fallback-engaged" }));
|
||||
});
|
||||
|
||||
it("drops an unavailable Cursor fallback without creating a replacement runtime", async () => {
|
||||
const pluginRunner = runner("cursor", "missing");
|
||||
const database = vi.fn();
|
||||
const result = await createResolvedAgentSession(options(undefined, {
|
||||
defaultProvider: "openai", defaultModelId: "primary", fallbackProvider: "cursor-cli", fallbackModelId: "cursor-cli/small",
|
||||
pluginRunner, runAuditor: { database },
|
||||
}));
|
||||
await expect((result.session as { promptWithFallback: (prompt: string) => Promise<unknown> }).promptWithFallback("current prompt")).resolves.toBeUndefined();
|
||||
expect(database).toHaveBeenCalledWith(expect.objectContaining({
|
||||
type: "session:runtime-resolved",
|
||||
metadata: expect.objectContaining({ crossRuntimeFallbackDropped: true }),
|
||||
}));
|
||||
expect(pluginRunner.getRuntimeById).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("routes Cursor primary selection through its installed runtime", async () => {
|
||||
const entry = CLI_PROVIDER_ROUTING_CENSUS.find((candidate) => candidate.providerId === "cursor-cli");
|
||||
const result = await createResolvedAgentSession(options(entry, { runtimeHint: "cursor", pluginRunner: runner("cursor") }));
|
||||
|
||||
146
packages/engine/src/__tests__/cross-runtime-fallback.test.ts
Normal file
146
packages/engine/src/__tests__/cross-runtime-fallback.test.ts
Normal file
@@ -0,0 +1,146 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const { resolveRuntime } = vi.hoisted(() => ({ resolveRuntime: vi.fn() }));
|
||||
vi.mock("../execution/runtime-resolution.js", async (importOriginal) => ({
|
||||
...await importOriginal<typeof import("../execution/runtime-resolution.js")>(),
|
||||
resolveRuntime,
|
||||
}));
|
||||
|
||||
import {
|
||||
armDeferredCrossRuntimeFallback,
|
||||
captureTransferableConversationContext,
|
||||
TRANSFERABLE_CONVERSATION_LIMITS,
|
||||
} from "../agents/cross-runtime-fallback.js";
|
||||
|
||||
const primaryError = new Error("rate limit exceeded");
|
||||
|
||||
function createSession(prompt = vi.fn(async () => { throw primaryError; })) {
|
||||
return {
|
||||
promptWithFallback: prompt,
|
||||
dispose: vi.fn(),
|
||||
state: { messages: [{ role: "user", content: "Earlier user context" }, { role: "assistant", content: [{ type: "text", text: "Earlier answer" }, { type: "thinking", text: "private" }] }] },
|
||||
};
|
||||
}
|
||||
|
||||
function arm(session: ReturnType<typeof createSession>, overrides: Record<string, unknown> = {}) {
|
||||
armDeferredCrossRuntimeFallback({
|
||||
session: session as never,
|
||||
sessionPurpose: "executor",
|
||||
pluginRunner: {} as never,
|
||||
runAuditor: undefined,
|
||||
deferred: { providerId: "cursor-cli", runtimeId: "cursor", modelId: "cursor-small", thinkingLevel: undefined },
|
||||
createOptions: { cwd: "/tmp", systemPrompt: "system", defaultProvider: "cursor-cli", defaultModelId: "cursor-small" },
|
||||
primaryProvider: "openai",
|
||||
primaryModelId: "gpt",
|
||||
onFallbackModelUsed: undefined,
|
||||
taskId: undefined,
|
||||
taskTitle: undefined,
|
||||
auditEventType: "session:cross-runtime-fallback-engaged",
|
||||
preserveConversationContext: true,
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
describe("cross-runtime fallback", () => {
|
||||
beforeEach(() => resolveRuntime.mockReset());
|
||||
|
||||
it("leaves non-retryable primary errors untouched", async () => {
|
||||
const error = new Error("prompt rejected");
|
||||
const session = createSession(vi.fn(async () => { throw error; }));
|
||||
arm(session);
|
||||
await expect((session.promptWithFallback as (prompt: string) => Promise<unknown>)("hello")).rejects.toBe(error);
|
||||
expect(resolveRuntime).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("swaps once, transfers portable context once, and disposes the replacement", async () => {
|
||||
const fallbackPrompt = vi.fn(async () => "swapped");
|
||||
const fallbackSession = { dispose: vi.fn() };
|
||||
resolveRuntime.mockResolvedValue({ runtimeId: "cursor", runtime: { createSession: vi.fn(async () => ({ session: fallbackSession })), promptWithFallback: fallbackPrompt } });
|
||||
const session = createSession();
|
||||
arm(session);
|
||||
const prompt = session.promptWithFallback as (prompt: string) => Promise<unknown>;
|
||||
await expect(prompt("current request")).resolves.toBe("swapped");
|
||||
await expect(prompt("later request")).resolves.toBe("swapped");
|
||||
expect(resolveRuntime).toHaveBeenCalledTimes(1);
|
||||
expect(fallbackPrompt.mock.calls[0]?.[1]).toContain("Transferred prior conversation from openai/gpt");
|
||||
expect(fallbackPrompt.mock.calls[0]?.[1]).toContain("Earlier answer");
|
||||
expect(fallbackPrompt.mock.calls[0]?.[1]).not.toContain("private");
|
||||
expect(fallbackPrompt.mock.calls[1]?.[1]).toBe("later request");
|
||||
await session.dispose();
|
||||
expect(fallbackSession.dispose).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["resolves another runtime", { runtimeId: "pi", runtime: {} }],
|
||||
["cannot create the replacement", { runtimeId: "cursor", runtime: { createSession: vi.fn(async () => { throw new Error("create failed"); }) } }],
|
||||
])("rethrows the original error when it %s", async (_label, resolved) => {
|
||||
resolveRuntime.mockResolvedValue(resolved);
|
||||
const session = createSession();
|
||||
arm(session);
|
||||
await expect((session.promptWithFallback as (prompt: string) => Promise<unknown>)("hello")).rejects.toBe(primaryError);
|
||||
});
|
||||
|
||||
it("shares one handoff across concurrent primary failures", async () => {
|
||||
let releaseCreation!: (value: { session: { dispose: ReturnType<typeof vi.fn> } }) => void;
|
||||
const fallbackSession = { dispose: vi.fn() };
|
||||
const createFallbackSession = vi.fn(() => new Promise<{ session: typeof fallbackSession }>((resolve) => { releaseCreation = resolve; }));
|
||||
const fallbackPrompt = vi.fn(async (_session: unknown, prompt: string) => prompt);
|
||||
const observer = vi.fn();
|
||||
const database = vi.fn();
|
||||
resolveRuntime.mockResolvedValue({ runtimeId: "cursor", runtime: { createSession: createFallbackSession, promptWithFallback: fallbackPrompt } });
|
||||
const session = createSession();
|
||||
arm(session, { onFallbackModelUsed: observer, runAuditor: { database } });
|
||||
const prompt = session.promptWithFallback as (prompt: string) => Promise<unknown>;
|
||||
const first = prompt("first");
|
||||
const second = prompt("second");
|
||||
await vi.waitFor(() => expect(createFallbackSession).toHaveBeenCalledOnce());
|
||||
releaseCreation({ session: fallbackSession });
|
||||
await Promise.all([first, second]);
|
||||
expect(resolveRuntime).toHaveBeenCalledOnce();
|
||||
expect(observer).toHaveBeenCalledOnce();
|
||||
expect(database).toHaveBeenCalledOnce();
|
||||
expect(fallbackPrompt).toHaveBeenCalledTimes(2);
|
||||
expect(fallbackPrompt.mock.calls.filter((call) => String(call[1]).includes("Transferred prior conversation"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("retries the primary after a failed handoff and preserves each primary error", async () => {
|
||||
const firstPrimaryError = new Error("rate limit first");
|
||||
const secondPrimaryError = new Error("rate limit second");
|
||||
const primary = vi.fn()
|
||||
.mockRejectedValueOnce(firstPrimaryError)
|
||||
.mockRejectedValueOnce(secondPrimaryError);
|
||||
resolveRuntime
|
||||
.mockRejectedValueOnce(new Error("cursor unavailable"))
|
||||
.mockResolvedValueOnce({ runtimeId: "cursor", runtime: { createSession: vi.fn(async () => { throw new Error("create failed"); }) } });
|
||||
const session = createSession(primary);
|
||||
arm(session);
|
||||
const prompt = session.promptWithFallback as (prompt: string) => Promise<unknown>;
|
||||
await expect(prompt("first")).rejects.toBe(firstPrimaryError);
|
||||
await expect(prompt("second")).rejects.toBe(secondPrimaryError);
|
||||
expect(primary).toHaveBeenCalledTimes(2);
|
||||
expect(resolveRuntime).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("does not let observer or audit failures block the swapped prompt", async () => {
|
||||
const fallbackPrompt = vi.fn(async () => "swapped");
|
||||
resolveRuntime.mockResolvedValue({ runtimeId: "cursor", runtime: { createSession: vi.fn(async () => ({ session: {} })), promptWithFallback: fallbackPrompt } });
|
||||
const session = createSession();
|
||||
arm(session, { onFallbackModelUsed: vi.fn(async () => { throw new Error("observer"); }), runAuditor: { database: vi.fn(async () => { throw new Error("audit"); }) } });
|
||||
await expect((session.promptWithFallback as (prompt: string) => Promise<unknown>)("hello")).resolves.toBe("swapped");
|
||||
});
|
||||
|
||||
it("extracts bounded text-only context across supported message shapes", () => {
|
||||
expect(captureTransferableConversationContext({ getMessages: () => [{ role: "assistant", content: [{ type: "tool", text: "ignore" }, { type: "text", text: "keep" }] }] })).toBe("assistant: keep");
|
||||
expect(captureTransferableConversationContext({ state: { messages: [] } })).toBeUndefined();
|
||||
expect(captureTransferableConversationContext({ messages: [{ role: "user", content: "string content" }] })).toBe("user: string content");
|
||||
const oversized = "x".repeat(TRANSFERABLE_CONVERSATION_LIMITS.maxCharsPerTurn + 1);
|
||||
expect(captureTransferableConversationContext({ agent: { state: { messages: [{ role: "user", content: oversized }] } } })).toContain("[truncated]");
|
||||
const manyTurns = Array.from({ length: 12 }, (_, index) => ({ role: "user", content: `turn-${index}` }));
|
||||
const recent = captureTransferableConversationContext({ messages: manyTurns });
|
||||
expect(recent).not.toContain("turn-0");
|
||||
expect(recent).toContain("turn-11");
|
||||
const total = captureTransferableConversationContext({ messages: Array.from({ length: 10 }, () => ({ role: "user", content: "x".repeat(2_000) })) });
|
||||
expect(total?.length).toBeLessThanOrEqual(TRANSFERABLE_CONVERSATION_LIMITS.maxCharsTotal);
|
||||
expect(total?.split("\n").every((turn) => turn.length <= TRANSFERABLE_CONVERSATION_LIMITS.maxCharsPerTurn)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -37,7 +37,6 @@ import { createLogger } from "../logger.js";
|
||||
import {
|
||||
promptWithFallback,
|
||||
describeModel,
|
||||
isRetryableModelSelectionError,
|
||||
wrapToolsWithActionGate,
|
||||
wrapToolsWithPermanentAgentGating,
|
||||
wrapToolsWithOutputBudget,
|
||||
@@ -53,6 +52,11 @@ import {
|
||||
deriveCliRuntimeHint,
|
||||
dropUnsupportedCliFallback,
|
||||
} from "./cli-provider-routing.js";
|
||||
import {
|
||||
armDeferredCrossRuntimeFallback,
|
||||
deferCrossRuntimeCliFallback,
|
||||
type DeferredCrossRuntimeFallback,
|
||||
} from "./cross-runtime-fallback.js";
|
||||
|
||||
/** Logger for agent session helpers */
|
||||
const sessionLog = createLogger("agent-session");
|
||||
@@ -152,8 +156,8 @@ export interface ResolvedSessionOptions extends AgentRuntimeOptions {
|
||||
/** Optional runtime hint from task/agent configuration */
|
||||
runtimeHint?: string;
|
||||
/**
|
||||
* Injected Cursor support status for routing conformance. Production leaves it
|
||||
* unset while the bundled Cursor adapter remains a non-executable stub.
|
||||
* Runtime routing conformance injects plugin availability through `pluginRunner`.
|
||||
* The bundled Cursor adapter is executable when its plugin is registered.
|
||||
*/
|
||||
/**
|
||||
* Optional run-audit emitter; when provided, a `session:runtime-resolved`
|
||||
@@ -389,33 +393,14 @@ function hasCompleteRuntimeModel(
|
||||
return Boolean(model.provider && model.modelId);
|
||||
}
|
||||
|
||||
function stripGrokCliModelProviderPrefix(modelId: string | undefined): string | undefined {
|
||||
const normalized = modelId?.trim();
|
||||
if (!normalized) return normalized;
|
||||
const grokCliPrefix = `${GROK_CLI_PROVIDER_ID}/`;
|
||||
return normalized.startsWith(grokCliPrefix)
|
||||
? normalized.slice(grokCliPrefix.length)
|
||||
: normalized;
|
||||
}
|
||||
|
||||
/** The deferred grok-cli fallback pair a session swaps to when its primary fails (see below). */
|
||||
interface DeferredGrokCliFallback {
|
||||
/** Concrete model id for the Grok CLI runtime (provider prefix stripped). */
|
||||
modelId: string | undefined;
|
||||
thinkingLevel: AgentRuntimeOptions["fallbackThinkingLevel"];
|
||||
}
|
||||
/** The deferred Grok CLI fallback is routed through the shared cross-runtime dispatcher. */
|
||||
type DeferredGrokCliFallback = DeferredCrossRuntimeFallback;
|
||||
|
||||
/*
|
||||
FNXC:GrokCliRouting 2026-07-22-15:10:
|
||||
A grok-cli fallback behind a non-grok primary cannot ride pi's in-session swap without a
|
||||
Fusion-visible GROK_API_KEY: pi resolves fallback swaps through the key-requiring provider
|
||||
registry (the original FN-7758 failure mode). Instead of promoting the fallback over a
|
||||
healthy primary (the old FN-7758 behavior — it silently replaced the configured planning
|
||||
model on every session) or discarding it, DEFER it: withhold the pair from the primary
|
||||
runtime's options and, when the Grok CLI runtime plugin is available, arm a prompt-time
|
||||
swap that recreates the session on the Grok CLI runtime only after the primary actually
|
||||
fails with a retryable model-selection error. When the Grok runtime plugin is unavailable
|
||||
the pair is dropped with a warning + audit flag so the drift is operator-visible.
|
||||
FNXC:CliRuntimeRouting 2026-08-16-01:25:
|
||||
Grok deliberately does not replay prior transcript text. Its historical fallback behavior is
|
||||
kept byte-for-byte through the shared dispatcher; only Cursor opts into portable text context
|
||||
because Cursor has no cross-runtime resume token and the new contract explicitly preserves it.
|
||||
*/
|
||||
function deferGrokCliFallbackForNoVisibleKey(
|
||||
runtimeOptions: AgentRuntimeOptions,
|
||||
@@ -432,34 +417,23 @@ function deferGrokCliFallbackForNoVisibleKey(
|
||||
fallbackModelId: undefined,
|
||||
fallbackThinkingLevel: undefined,
|
||||
};
|
||||
let grokRuntimeAvailable = false;
|
||||
try {
|
||||
grokRuntimeAvailable = Boolean(pluginRunner?.getRuntimeById("grok"));
|
||||
if (!pluginRunner?.getRuntimeById("grok")) return { options, dropped: true };
|
||||
} catch {
|
||||
grokRuntimeAvailable = false;
|
||||
}
|
||||
if (!grokRuntimeAvailable) {
|
||||
return { options, dropped: true };
|
||||
}
|
||||
return {
|
||||
options,
|
||||
deferred: {
|
||||
modelId: stripGrokCliModelProviderPrefix(runtimeOptions.fallbackModelId),
|
||||
providerId: GROK_CLI_PROVIDER_ID,
|
||||
runtimeId: "grok",
|
||||
modelId: runtimeOptions.fallbackModelId?.replace(/^grok-cli\//, ""),
|
||||
thinkingLevel: runtimeOptions.fallbackThinkingLevel,
|
||||
},
|
||||
dropped: false,
|
||||
};
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:GrokCliRouting 2026-07-22-15:10:
|
||||
Prompt-time engagement of a deferred grok-cli fallback (see deferGrokCliFallbackForNoVisibleKey).
|
||||
Wraps the session's promptWithFallback: the first retryable model-selection failure of the
|
||||
primary creates a fresh session on the Grok CLI runtime with the deferred fallback model and
|
||||
re-issues the failed prompt there; every later prompt stays on the Grok CLI session. The swap
|
||||
happens at most once, non-retryable errors propagate unchanged, and the engagement is reported
|
||||
through onFallbackModelUsed plus an ids-only `session:grok-cli-fallback-engaged` audit event.
|
||||
*/
|
||||
function armDeferredGrokCliFallback(args: {
|
||||
session: AgentSession & { promptWithFallback?: unknown };
|
||||
sessionPurpose: SessionPurpose;
|
||||
@@ -473,97 +447,13 @@ function armDeferredGrokCliFallback(args: {
|
||||
taskId: string | undefined;
|
||||
taskTitle: string | undefined;
|
||||
}): void {
|
||||
const {
|
||||
session, sessionPurpose, pluginRunner, runAuditor, deferred, grokCreateOptions,
|
||||
primaryProvider, primaryModelId, onFallbackModelUsed, taskId, taskTitle,
|
||||
} = args;
|
||||
const original = session.promptWithFallback as (prompt: string, options?: unknown) => Promise<unknown>;
|
||||
const primaryDescription = `${primaryProvider ?? "unknown"}/${primaryModelId ?? "unknown"}`;
|
||||
const fallbackDescription = `${GROK_CLI_PROVIDER_ID}/${deferred.modelId ?? "unknown"}`;
|
||||
let grokSwap: { runtime: Awaited<ReturnType<typeof resolveRuntime>>["runtime"]; session: AgentSession } | undefined;
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(session as any).promptWithFallback = async (prompt: string, promptOptions?: unknown) => {
|
||||
if (grokSwap) {
|
||||
return grokSwap.runtime.promptWithFallback(grokSwap.session, prompt, promptOptions);
|
||||
}
|
||||
try {
|
||||
return await original(prompt, promptOptions);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
if (!isRetryableModelSelectionError(message)) throw err;
|
||||
sessionLog.warn(
|
||||
`[${sessionPurpose}] primary "${primaryDescription}" failed retryably (${message}); engaging deferred grok-cli fallback "${fallbackDescription}" on the Grok CLI runtime`,
|
||||
);
|
||||
let resolvedGrok: Awaited<ReturnType<typeof resolveRuntime>>;
|
||||
let grokSession: AgentSession;
|
||||
try {
|
||||
resolvedGrok = await resolveRuntime(buildRuntimeResolutionContext(sessionPurpose, pluginRunner, "grok"));
|
||||
if (resolvedGrok.runtimeId !== "grok") throw new Error("Grok CLI runtime unavailable at swap time");
|
||||
grokSession = (await resolvedGrok.runtime.createSession(grokCreateOptions)).session;
|
||||
} catch (swapErr) {
|
||||
const swapMessage = swapErr instanceof Error ? swapErr.message : String(swapErr);
|
||||
sessionLog.warn(
|
||||
`[${sessionPurpose}] deferred grok-cli fallback engagement failed (${swapMessage}); propagating primary failure`,
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
grokSwap = { runtime: resolvedGrok.runtime, session: grokSession };
|
||||
// Dispose the Grok CLI ACP session alongside the primary session so the
|
||||
// swapped-in subprocess cannot outlive the session the engine tracks.
|
||||
const disposable = session as unknown as { dispose?: () => Promise<void> | void };
|
||||
const originalDispose = typeof disposable.dispose === "function" ? disposable.dispose.bind(session) : undefined;
|
||||
disposable.dispose = async () => {
|
||||
try {
|
||||
await (grokSession as unknown as { dispose?: () => Promise<void> | void }).dispose?.();
|
||||
} catch {
|
||||
// best-effort: the primary session's dispose must still run
|
||||
}
|
||||
return originalDispose?.();
|
||||
};
|
||||
const normalizedFailure = message.toLowerCase();
|
||||
const failureCategory: FallbackModelUsedPayload["failureCategory"] =
|
||||
normalizedFailure.includes("auth")
|
||||
|| normalizedFailure.includes("api key")
|
||||
|| normalizedFailure.includes("credential")
|
||||
|| normalizedFailure.includes("401")
|
||||
|| normalizedFailure.includes("403")
|
||||
? "authentication"
|
||||
: normalizedFailure.includes("rate limit") || normalizedFailure.includes("429") || normalizedFailure.includes("quota")
|
||||
? "rate-limit"
|
||||
: "model-selection";
|
||||
try {
|
||||
await onFallbackModelUsed?.({
|
||||
primaryModel: primaryDescription,
|
||||
fallbackModel: fallbackDescription,
|
||||
triggerPoint: "prompt-time",
|
||||
taskId,
|
||||
taskTitle,
|
||||
timestamp: new Date().toISOString(),
|
||||
failureCategory,
|
||||
});
|
||||
} catch {
|
||||
// observer failures must not break the swapped prompt
|
||||
}
|
||||
try {
|
||||
await runAuditor?.database({
|
||||
type: "session:grok-cli-fallback-engaged",
|
||||
target: "grok",
|
||||
metadata: {
|
||||
sessionPurpose,
|
||||
primaryProvider: primaryProvider ?? null,
|
||||
primaryModelId: primaryModelId ?? null,
|
||||
fallbackModelId: deferred.modelId ?? null,
|
||||
triggerPoint: "prompt-time",
|
||||
failureCategory,
|
||||
},
|
||||
});
|
||||
} catch (auditErr) {
|
||||
sessionLog.warn(`[${sessionPurpose}] failed to record session:grok-cli-fallback-engaged audit: ${String(auditErr)}`);
|
||||
}
|
||||
return grokSwap.runtime.promptWithFallback(grokSwap.session, prompt, promptOptions);
|
||||
}
|
||||
};
|
||||
armDeferredCrossRuntimeFallback({
|
||||
...args,
|
||||
createOptions: args.grokCreateOptions,
|
||||
auditEventType: "session:grok-cli-fallback-engaged",
|
||||
preserveConversationContext: false,
|
||||
engagementLabel: "grok-cli",
|
||||
});
|
||||
}
|
||||
|
||||
function pickSettingsThenRuntimeModel(
|
||||
@@ -930,12 +820,19 @@ export async function createResolvedAgentSession(
|
||||
const grokFallbackDeferral = !useMockRuntime && !usesAutoGrokRuntime && effectiveRuntimeHint !== "grok"
|
||||
? deferGrokCliFallbackForNoVisibleKey(effectiveRuntimeOptions, pluginRunner)
|
||||
: { options: effectiveRuntimeOptions, dropped: false as const };
|
||||
const droppedCliFallback = dropUnsupportedCliFallback(grokFallbackDeferral.options);
|
||||
const crossRuntimeFallbackDeferral = !useMockRuntime && !("deferred" in grokFallbackDeferral)
|
||||
&& effectiveRuntimeHint !== "cursor"
|
||||
? deferCrossRuntimeCliFallback(grokFallbackDeferral.options, pluginRunner)
|
||||
: { options: grokFallbackDeferral.options, dropped: false as const };
|
||||
const droppedCliFallback = dropUnsupportedCliFallback(crossRuntimeFallbackDeferral.options);
|
||||
const effectiveRuntimeOptionsWithModel = applyCliRuntimeOptions(
|
||||
droppedCliFallback.options,
|
||||
effectiveRuntimeHint,
|
||||
);
|
||||
const deferredGrokFallback = "deferred" in grokFallbackDeferral ? grokFallbackDeferral.deferred : undefined;
|
||||
const deferredCrossRuntimeFallback = "deferred" in crossRuntimeFallbackDeferral
|
||||
? crossRuntimeFallbackDeferral.deferred
|
||||
: undefined;
|
||||
if (droppedCliFallback.droppedProvider) {
|
||||
sessionLog.warn(
|
||||
`[${sessionPurpose}] configured ${droppedCliFallback.droppedProvider} fallback "${runtimeOptions.fallbackModelId ?? "unknown"}" dropped: primary "${runtimeOptions.defaultProvider}/${runtimeOptions.defaultModelId}" is unchanged because the fallback requires its own CLI runtime.`,
|
||||
@@ -944,6 +841,14 @@ export async function createResolvedAgentSession(
|
||||
sessionLog.warn(
|
||||
`[${sessionPurpose}] configured grok-cli fallback "${runtimeOptions.fallbackModelId ?? "unknown"}" dropped: no Fusion-visible GROK_API_KEY and the Grok CLI runtime plugin is unavailable; primary "${runtimeOptions.defaultProvider}/${runtimeOptions.defaultModelId}" is unchanged. Install/enable the Grok CLI runtime plugin or set GROK_API_KEY.`,
|
||||
);
|
||||
} else if (crossRuntimeFallbackDeferral.dropped) {
|
||||
sessionLog.warn(
|
||||
`[${sessionPurpose}] configured ${runtimeOptions.fallbackProvider ?? "cross-runtime"} fallback "${runtimeOptions.fallbackModelId ?? "unknown"}" dropped: its runtime plugin is unavailable; primary "${runtimeOptions.defaultProvider}/${runtimeOptions.defaultModelId}" is unchanged.`,
|
||||
);
|
||||
} else if (deferredCrossRuntimeFallback) {
|
||||
sessionLog.debug(
|
||||
`[${sessionPurpose}] ${deferredCrossRuntimeFallback.providerId} fallback "${deferredCrossRuntimeFallback.modelId ?? "unknown"}" deferred to the ${deferredCrossRuntimeFallback.runtimeId} runtime: it engages only if primary "${runtimeOptions.defaultProvider}/${runtimeOptions.defaultModelId}" fails with a retryable model error.`,
|
||||
);
|
||||
} else if (deferredGrokFallback) {
|
||||
/*
|
||||
FNXC:EngineDiagnostics 2026-08-01-18:11:
|
||||
@@ -1036,6 +941,8 @@ export async function createResolvedAgentSession(
|
||||
testModeActive,
|
||||
...(grokFallbackDeferral.dropped ? { grokCliFallbackDropped: true } : {}),
|
||||
...(deferredGrokFallback ? { grokCliFallbackDeferred: true } : {}),
|
||||
...(crossRuntimeFallbackDeferral.dropped ? { crossRuntimeFallbackDropped: true } : {}),
|
||||
...(deferredCrossRuntimeFallback ? { crossRuntimeFallbackDeferred: true } : {}),
|
||||
...(noModelResolved ? { noModelResolved: true, runtimeBuiltInFallbackModel } : {}),
|
||||
...(credentialResolution?.ref.instanceId ? { credentialInstanceId: credentialResolution.ref.instanceId } : {}),
|
||||
...(credentialResolution?.missing ? {
|
||||
@@ -1063,7 +970,9 @@ export async function createResolvedAgentSession(
|
||||
...(autoCliRuntimeHint === "omp" ? { reason: "omp-cli-runtime" } : {}),
|
||||
...(grokFallbackDeferral.dropped ? { reason: "grok-cli-fallback-dropped-no-visible-key" } : {}),
|
||||
...(deferredGrokFallback ? { reason: "grok-cli-fallback-deferred-no-visible-key" } : {}),
|
||||
...(!autoCliRuntimeHint && !grokFallbackDeferral.dropped && !deferredGrokFallback && "fallbackReason" in resolved && resolved.fallbackReason ? { reason: resolved.fallbackReason } : {}),
|
||||
...(crossRuntimeFallbackDeferral.dropped ? { reason: "cross-runtime-fallback-dropped-runtime-unavailable" } : {}),
|
||||
...(deferredCrossRuntimeFallback ? { reason: "cross-runtime-fallback-deferred" } : {}),
|
||||
...(!autoCliRuntimeHint && !grokFallbackDeferral.dropped && !deferredGrokFallback && !crossRuntimeFallbackDeferral.dropped && !deferredCrossRuntimeFallback && "fallbackReason" in resolved && resolved.fallbackReason ? { reason: resolved.fallbackReason } : {}),
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
@@ -1128,6 +1037,36 @@ export async function createResolvedAgentSession(
|
||||
});
|
||||
}
|
||||
|
||||
if (deferredCrossRuntimeFallback) {
|
||||
const fallbackBaseOptions: AgentRuntimeOptions = {
|
||||
...effectiveRuntimeOptionsWithModel,
|
||||
sessionPurpose,
|
||||
defaultProvider: deferredCrossRuntimeFallback.providerId,
|
||||
defaultModelId: deferredCrossRuntimeFallback.modelId,
|
||||
defaultThinkingLevel: deferredCrossRuntimeFallback.thinkingLevel ?? effectiveRuntimeOptionsWithModel.defaultThinkingLevel,
|
||||
};
|
||||
armDeferredCrossRuntimeFallback({
|
||||
session,
|
||||
sessionPurpose,
|
||||
pluginRunner,
|
||||
runAuditor,
|
||||
deferred: deferredCrossRuntimeFallback,
|
||||
createOptions: shouldWrapCustomToolsForRuntime(deferredCrossRuntimeFallback.runtimeId)
|
||||
? {
|
||||
...fallbackBaseOptions,
|
||||
...wrapPluginRuntimeToolOptions(fallbackBaseOptions, { runtimeId: deferredCrossRuntimeFallback.runtimeId, sessionPurpose }),
|
||||
}
|
||||
: fallbackBaseOptions,
|
||||
primaryProvider: runtimeOptions.defaultProvider,
|
||||
primaryModelId: runtimeOptions.defaultModelId,
|
||||
onFallbackModelUsed: runtimeOptions.onFallbackModelUsed,
|
||||
taskId: runtimeOptions.taskId,
|
||||
taskTitle: runtimeOptions.taskTitle,
|
||||
auditEventType: "session:cross-runtime-fallback-engaged",
|
||||
preserveConversationContext: true,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
session: result.session,
|
||||
sessionFile: result.sessionFile,
|
||||
|
||||
@@ -3,7 +3,7 @@ import type { PluginRunner } from "../plugins/plugin-runner.js";
|
||||
|
||||
export type CliProviderClassification = "registry-native" | "runtime-routed" | "non-cli" | "withheld-unsupported";
|
||||
export type CliPathPolicy = "fail-fast" | "pinned-pi-fallback" | "assert-available" | "defer-to-resolve-runtime" | "n/a";
|
||||
export type CliFallbackPolicy = "promote-to-primary" | "defer-to-runtime" | "drop-with-warning" | "none";
|
||||
export type CliFallbackPolicy = "promote-to-primary" | "defer-to-runtime" | "defer-cross-runtime" | "drop-with-warning" | "none";
|
||||
|
||||
export interface CliProviderRouting {
|
||||
providerId: string;
|
||||
@@ -70,7 +70,7 @@ export const CLI_PROVIDER_ROUTING_CENSUS: readonly CliProviderRouting[] = [
|
||||
{ providerId: "grok-cli", classification: "runtime-routed", runtimeId: "grok", autoDerive: "fail-fast", guardNotApplicable: "pinned-pi-fallback", onExplicitHint: "defer-to-resolve-runtime", fallbackPolicy: "defer-to-runtime", missingRuntimeError: buildMissingGrokRuntimeError, rationale: "Visible-key, fallback-only, and explicit-hint paths intentionally preserve the shipped direct xAI/pi fallback." },
|
||||
{ providerId: "hermes", classification: "runtime-routed", runtimeId: "hermes", autoDerive: "fail-fast", guardNotApplicable: "pinned-pi-fallback", onExplicitHint: "assert-available", fallbackPolicy: "drop-with-warning", missingRuntimeError: buildMissingHermesRuntimeError, rationale: "Fallback-only Hermes cannot be resolved by a healthy primary pi runtime." },
|
||||
{ providerId: "claude-cli", classification: "runtime-routed", runtimeId: "claude", autoDerive: "fail-fast", guardNotApplicable: "pinned-pi-fallback", onExplicitHint: "assert-available", fallbackPolicy: "drop-with-warning", missingRuntimeError: buildMissingClaudeRuntimeError, rationale: "Fallback-only Claude CLI cannot be resolved by a healthy primary pi runtime." },
|
||||
{ providerId: "cursor-cli", classification: "runtime-routed", runtimeId: "cursor", autoDerive: "fail-fast", guardNotApplicable: "pinned-pi-fallback", onExplicitHint: "assert-available", fallbackPolicy: "drop-with-warning", missingRuntimeError: buildMissingCursorRuntimeError, rationale: "FN-9097 verified Cursor's supervised stream-json transport and session resume contract." },
|
||||
{ providerId: "cursor-cli", classification: "runtime-routed", runtimeId: "cursor", autoDerive: "fail-fast", guardNotApplicable: "pinned-pi-fallback", onExplicitHint: "assert-available", fallbackPolicy: "defer-cross-runtime", missingRuntimeError: buildMissingCursorRuntimeError, rationale: "Cursor fallback is withheld from a healthy foreign runtime and armed for a single prompt-time cross-runtime swap." },
|
||||
] as const;
|
||||
|
||||
export function getCliProviderRouting(providerId: string | undefined): CliProviderRouting | undefined {
|
||||
|
||||
277
packages/engine/src/agents/cross-runtime-fallback.ts
Normal file
277
packages/engine/src/agents/cross-runtime-fallback.ts
Normal file
@@ -0,0 +1,277 @@
|
||||
import type { AgentRuntimeOptions } from "./agent-runtime.js";
|
||||
import type { PluginRunner } from "../plugins/plugin-runner.js";
|
||||
import type { AgentSession } from "@earendil-works/pi-coding-agent";
|
||||
import { buildRuntimeResolutionContext, resolveRuntime, type SessionPurpose } from "../execution/runtime-resolution.js";
|
||||
import { createLogger } from "../logger.js";
|
||||
import { isRetryableModelSelectionError, type FallbackModelUsedPayload } from "../pi.js";
|
||||
import type { RunAuditor } from "../util/run-audit.js";
|
||||
import { getCliProviderRouting, stripCliProviderPrefix } from "./cli-provider-routing.js";
|
||||
|
||||
const fallbackLog = createLogger("cross-runtime-fallback");
|
||||
|
||||
export interface DeferredCrossRuntimeFallback {
|
||||
providerId: string;
|
||||
runtimeId: string;
|
||||
modelId: string | undefined;
|
||||
thinkingLevel: AgentRuntimeOptions["fallbackThinkingLevel"];
|
||||
}
|
||||
|
||||
export interface TransferableConversationLimits {
|
||||
maxTurns: number;
|
||||
maxCharsPerTurn: number;
|
||||
maxCharsTotal: number;
|
||||
}
|
||||
|
||||
export const TRANSFERABLE_CONVERSATION_LIMITS: TransferableConversationLimits = {
|
||||
maxTurns: 10,
|
||||
maxCharsPerTurn: 2_000,
|
||||
maxCharsTotal: 12_000,
|
||||
};
|
||||
|
||||
/*
|
||||
FNXC:CliRuntimeRouting 2026-08-16-01:25:
|
||||
Cross-runtime CLI fallbacks are census-driven because a primary runtime cannot safely interpret a
|
||||
fallback owned by a different runtime. The routing census declares which fallback pairs may defer;
|
||||
this seam only withholds and arms an available target, otherwise it preserves the established
|
||||
warning/drop behavior rather than silently falling through to an unrelated runtime.
|
||||
*/
|
||||
export function deferCrossRuntimeCliFallback(
|
||||
runtimeOptions: AgentRuntimeOptions,
|
||||
pluginRunner: PluginRunner | undefined,
|
||||
): { options: AgentRuntimeOptions; deferred?: DeferredCrossRuntimeFallback; dropped: boolean } {
|
||||
const entry = getCliProviderRouting(runtimeOptions.fallbackProvider);
|
||||
if (!entry || entry.fallbackPolicy !== "defer-cross-runtime" || runtimeOptions.defaultProvider === entry.providerId) {
|
||||
return { options: runtimeOptions, dropped: false };
|
||||
}
|
||||
const options: AgentRuntimeOptions = {
|
||||
...runtimeOptions,
|
||||
fallbackProvider: undefined,
|
||||
fallbackModelId: undefined,
|
||||
fallbackThinkingLevel: undefined,
|
||||
};
|
||||
try {
|
||||
if (!entry.runtimeId || !pluginRunner?.getRuntimeById(entry.runtimeId)) {
|
||||
return { options, dropped: true };
|
||||
}
|
||||
} catch {
|
||||
return { options, dropped: true };
|
||||
}
|
||||
return {
|
||||
options,
|
||||
deferred: {
|
||||
providerId: entry.providerId,
|
||||
runtimeId: entry.runtimeId,
|
||||
modelId: stripCliProviderPrefix(entry.providerId, runtimeOptions.fallbackModelId),
|
||||
thinkingLevel: runtimeOptions.fallbackThinkingLevel,
|
||||
},
|
||||
dropped: false,
|
||||
};
|
||||
}
|
||||
|
||||
function extractTextContent(content: unknown): string | undefined {
|
||||
if (typeof content === "string") return content.trim() || undefined;
|
||||
if (!Array.isArray(content)) return undefined;
|
||||
const text = content
|
||||
.flatMap((block) => {
|
||||
if (!block || typeof block !== "object") return [];
|
||||
const record = block as { type?: unknown; text?: unknown };
|
||||
return record.type === "text" && typeof record.text === "string" ? [record.text] : [];
|
||||
})
|
||||
.join("\n")
|
||||
.trim();
|
||||
return text || undefined;
|
||||
}
|
||||
|
||||
function extractMessageText(message: unknown): string | undefined {
|
||||
if (!message || typeof message !== "object") return undefined;
|
||||
const record = message as { content?: unknown; text?: unknown; role?: unknown };
|
||||
const text = extractTextContent(record.content) ?? (typeof record.text === "string" ? record.text.trim() : undefined);
|
||||
if (!text) return undefined;
|
||||
const role = typeof record.role === "string" && record.role.trim() ? record.role.trim() : "message";
|
||||
return `${role}: ${text}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture only portable text from common pi/runtime conversation shapes. This never fabricates a
|
||||
* resume token or exports tool/thinking blocks, which are runtime-specific execution state.
|
||||
*/
|
||||
export function captureTransferableConversationContext(
|
||||
session: unknown,
|
||||
limits: TransferableConversationLimits = TRANSFERABLE_CONVERSATION_LIMITS,
|
||||
): string | undefined {
|
||||
try {
|
||||
const candidate = session as {
|
||||
messages?: unknown;
|
||||
state?: { messages?: unknown };
|
||||
agent?: { state?: { messages?: unknown } };
|
||||
getMessages?: () => unknown;
|
||||
};
|
||||
const messages = [
|
||||
candidate?.messages,
|
||||
candidate?.state?.messages,
|
||||
candidate?.agent?.state?.messages,
|
||||
typeof candidate?.getMessages === "function" ? candidate.getMessages() : undefined,
|
||||
].find(Array.isArray);
|
||||
if (!Array.isArray(messages) || messages.length === 0) return undefined;
|
||||
const truncate = (text: string, maxChars: number): string => {
|
||||
const boundedMax = Math.max(0, Math.floor(maxChars));
|
||||
if (text.length <= boundedMax) return text;
|
||||
const marker = " [truncated]";
|
||||
if (boundedMax <= marker.length) return text.slice(0, boundedMax);
|
||||
return `${text.slice(0, boundedMax - marker.length)}${marker}`;
|
||||
};
|
||||
const turns = messages.slice(-Math.max(0, Math.floor(limits.maxTurns))).flatMap((message) => {
|
||||
const text = extractMessageText(message);
|
||||
if (!text) return [];
|
||||
return [truncate(text, limits.maxCharsPerTurn)];
|
||||
});
|
||||
if (turns.length === 0) return undefined;
|
||||
const maxCharsTotal = Math.max(0, Math.floor(limits.maxCharsTotal));
|
||||
let used = 0;
|
||||
const bounded: string[] = [];
|
||||
/*
|
||||
FNXC:CliRuntimeRouting 2026-08-16-02:05:
|
||||
The transferable-context total includes separators as well as turn text. Account for each newline
|
||||
before clipping so the serialized transcript never exceeds the documented total character cap.
|
||||
*/
|
||||
for (const turn of turns) {
|
||||
const separatorLength = bounded.length > 0 ? 1 : 0;
|
||||
const available = maxCharsTotal - used - separatorLength;
|
||||
if (available <= 0) break;
|
||||
bounded.push(truncate(turn, available));
|
||||
used += separatorLength + bounded.at(-1)!.length;
|
||||
}
|
||||
return bounded.length > 0 ? bounded.join("\n") : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function failureCategory(message: string): FallbackModelUsedPayload["failureCategory"] {
|
||||
const normalized = message.toLowerCase();
|
||||
if (normalized.includes("auth") || normalized.includes("api key") || normalized.includes("credential") || normalized.includes("401") || normalized.includes("403")) return "authentication";
|
||||
if (normalized.includes("rate limit") || normalized.includes("429") || normalized.includes("quota")) return "rate-limit";
|
||||
return "model-selection";
|
||||
}
|
||||
|
||||
function withTranscript(args: {
|
||||
prompt: string;
|
||||
transcript: string | undefined;
|
||||
primaryProvider: string | undefined;
|
||||
primaryModelId: string | undefined;
|
||||
}): string {
|
||||
if (!args.transcript) return args.prompt;
|
||||
return `[Transferred prior conversation from ${args.primaryProvider ?? "unknown"}/${args.primaryModelId ?? "unknown"}; partial, text-only, and non-authoritative]\n${args.transcript}\n\n[Current prompt]\n${args.prompt}`;
|
||||
}
|
||||
|
||||
export function armDeferredCrossRuntimeFallback(args: {
|
||||
session: AgentSession & { promptWithFallback?: unknown };
|
||||
sessionPurpose: SessionPurpose;
|
||||
pluginRunner: PluginRunner | undefined;
|
||||
runAuditor: RunAuditor | undefined;
|
||||
deferred: DeferredCrossRuntimeFallback;
|
||||
createOptions: AgentRuntimeOptions;
|
||||
primaryProvider: string | undefined;
|
||||
primaryModelId: string | undefined;
|
||||
onFallbackModelUsed: ((payload: FallbackModelUsedPayload) => Promise<void> | void) | undefined;
|
||||
taskId: string | undefined;
|
||||
taskTitle: string | undefined;
|
||||
auditEventType: "session:grok-cli-fallback-engaged" | "session:cross-runtime-fallback-engaged";
|
||||
preserveConversationContext: boolean;
|
||||
engagementLabel?: string;
|
||||
}): void {
|
||||
const { session, sessionPurpose, pluginRunner, runAuditor, deferred, createOptions, primaryProvider, primaryModelId, onFallbackModelUsed, taskId, taskTitle, auditEventType, preserveConversationContext } = args;
|
||||
const original = session.promptWithFallback as (prompt: string, options?: unknown) => Promise<unknown>;
|
||||
const primaryDescription = `${primaryProvider ?? "unknown"}/${primaryModelId ?? "unknown"}`;
|
||||
const fallbackDescription = `${deferred.providerId}/${deferred.modelId ?? "unknown"}`;
|
||||
type Swap = {
|
||||
runtime: Awaited<ReturnType<typeof resolveRuntime>>["runtime"];
|
||||
session: AgentSession;
|
||||
transferredContext: string | undefined;
|
||||
};
|
||||
let swap: Swap | undefined;
|
||||
let swapPromise: Promise<Swap> | undefined;
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(session as any).promptWithFallback = async (prompt: string, promptOptions?: unknown) => {
|
||||
if (swap) return swap.runtime.promptWithFallback(swap.session, prompt, promptOptions);
|
||||
try {
|
||||
return await original(prompt, promptOptions);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (!isRetryableModelSelectionError(message)) throw error;
|
||||
let initiatedSwap = false;
|
||||
if (!swapPromise) {
|
||||
initiatedSwap = true;
|
||||
fallbackLog.warn(`[${sessionPurpose}] primary "${primaryDescription}" failed retryably (${message}); engaging deferred ${args.engagementLabel ?? deferred.providerId} fallback "${fallbackDescription}"`);
|
||||
/*
|
||||
FNXC:CliRuntimeRouting 2026-08-16-02:05:
|
||||
A session owns one shared handoff attempt. Concurrent retryable primary failures await the same
|
||||
replacement creation, context capture, observer notification, and audit instead of leaking
|
||||
duplicate Cursor sessions. A failed attempt clears the fence so later prompts retry the primary;
|
||||
every waiter still receives its own original primary error when that handoff cannot be created.
|
||||
*/
|
||||
const attempt = (async (): Promise<Swap> => {
|
||||
const resolved = await resolveRuntime(buildRuntimeResolutionContext(sessionPurpose, pluginRunner, deferred.runtimeId));
|
||||
if (resolved.runtimeId !== deferred.runtimeId) throw new Error(`${deferred.runtimeId} runtime unavailable at swap time`);
|
||||
const fallbackSession = (await resolved.runtime.createSession(createOptions)).session;
|
||||
const transferredContext = preserveConversationContext
|
||||
? captureTransferableConversationContext(session)
|
||||
: undefined;
|
||||
const createdSwap = { runtime: resolved.runtime, session: fallbackSession, transferredContext };
|
||||
const disposable = session as unknown as { dispose?: () => Promise<void> | void };
|
||||
const originalDispose = typeof disposable.dispose === "function" ? disposable.dispose.bind(session) : undefined;
|
||||
disposable.dispose = async () => {
|
||||
try { await (fallbackSession as unknown as { dispose?: () => Promise<void> | void }).dispose?.(); } catch { /* best effort */ }
|
||||
return originalDispose?.();
|
||||
};
|
||||
const category = failureCategory(message);
|
||||
try {
|
||||
await onFallbackModelUsed?.({ primaryModel: primaryDescription, fallbackModel: fallbackDescription, triggerPoint: "prompt-time", taskId, taskTitle, timestamp: new Date().toISOString(), failureCategory: category });
|
||||
} catch { /* observer failures must not break the swapped prompt */ }
|
||||
try {
|
||||
await runAuditor?.database({
|
||||
type: auditEventType,
|
||||
target: deferred.runtimeId,
|
||||
metadata: {
|
||||
sessionPurpose,
|
||||
primaryProvider: primaryProvider ?? null,
|
||||
primaryModelId: primaryModelId ?? null,
|
||||
fallbackProvider: deferred.providerId,
|
||||
fallbackModelId: deferred.modelId ?? null,
|
||||
triggerPoint: "prompt-time",
|
||||
failureCategory: category,
|
||||
...(auditEventType === "session:cross-runtime-fallback-engaged" ? { contextTransferred: Boolean(transferredContext) } : {}),
|
||||
},
|
||||
});
|
||||
} catch (auditError) {
|
||||
fallbackLog.warn(`[${sessionPurpose}] failed to record ${auditEventType} audit: ${String(auditError)}`);
|
||||
}
|
||||
swap = createdSwap;
|
||||
return createdSwap;
|
||||
})();
|
||||
swapPromise = attempt;
|
||||
void attempt.catch(() => {
|
||||
if (swapPromise === attempt) swapPromise = undefined;
|
||||
});
|
||||
}
|
||||
let engagedSwap: Swap;
|
||||
try {
|
||||
engagedSwap = await swapPromise;
|
||||
} catch (swapError) {
|
||||
fallbackLog.warn(`[${sessionPurpose}] deferred ${args.engagementLabel ?? deferred.providerId} fallback engagement failed (${String(swapError)}); propagating primary failure`);
|
||||
throw error;
|
||||
}
|
||||
return engagedSwap.runtime.promptWithFallback(
|
||||
engagedSwap.session,
|
||||
withTranscript({
|
||||
prompt,
|
||||
transcript: initiatedSwap ? engagedSwap.transferredContext : undefined,
|
||||
primaryProvider,
|
||||
primaryModelId,
|
||||
}),
|
||||
promptOptions,
|
||||
);
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -739,6 +739,14 @@ export type DatabaseMutationType =
|
||||
* — never error prose.
|
||||
*/
|
||||
| "session:grok-cli-fallback-engaged"
|
||||
/**
|
||||
* FNXC:CliRuntimeRouting 2026-08-16-01:25:
|
||||
* A configured cross-runtime CLI fallback engaged once after a retryable primary failure.
|
||||
* Metadata is ids/outcomes-only: `{ sessionPurpose, primaryProvider, primaryModelId,
|
||||
* fallbackProvider, fallbackModelId, triggerPoint, failureCategory, contextTransferred }`.
|
||||
* It never stores the primary error or transferred conversation text.
|
||||
*/
|
||||
| "session:cross-runtime-fallback-engaged"
|
||||
/**
|
||||
* FNXC:AgentReflectionTelemetry 2026-06-27-00:00:
|
||||
* Agent performance reflection attempts must emit durable telemetry for every generated, skipped, or failed outcome. Metadata carries ids, trigger taxonomy, counts, and outcomes only; never persist reflection summaries, insight strings, suggested-improvement text, triggerDetail, or prompt text.
|
||||
|
||||
Reference in New Issue
Block a user