root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FN-043: add canonical thinking levels for custom providers

Browse Source 
Custom-provider models now expose the full thinking-level selector and faithfully transmit the selected effort.

- Register all canonical thinking levels and preserve provider-specific API dialects.
- Remove the redundant reasoning toggle and update provider status and settings documentation.
- Add dashboard, engine, and provider registration coverage for thinking-level behavior.
- Add a minor changeset for the published CLI package.

Files changed:
 .../fn-043-custom-provider-thinking-levels.md      |  7 ++
 docs/settings-reference.md                         |  2 +
 .../dashboard/app/api/settings/provider-status.ts  |  1 -
 .../app/components/CustomProviderForm.css          |  5 +-
 .../app/components/CustomProviderForm.tsx          | 18 ++---
 .../__tests__/CustomProviderForm.test.tsx          | 35 ++++++---
 ...r-model-routes-custom-provider-thinking.test.ts | 91 ++++++++++++++++++++++
 .../custom-provider-thinking-levels.test.ts        | 49 ++++++++++++
 .../custom-providers-openai-completions.test.ts    |  2 +-
 .../custom-providers-openai-responses.test.ts      |  3 +-
 .../src/__tests__/pi-create-fn-agent.test.ts       | 31 +++++++-
 .../src/__tests__/provider-registration.test.ts    | 20 ++++-
 .../engine/src/auth/custom-provider-registry.ts    | 14 +++-
 packages/engine/src/pi.ts                          |  9 ++-
 14 files changed, 251 insertions(+), 36 deletions(-)

Fusion-Task-Id: FN-043

Fusion-Task-Lineage: fdd3308a-d5bd-4f54-911a-226c2b46a927

Co-authored-by: Fusion <noreply@runfusion.ai>
This commit is contained in:
Fusion Agent
2026-08-19 15:34:45 +00:00
parent a81c9b80d9
commit b1893a63af
14 changed files with 251 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.changeset/fn-043-custom-provider-thinking-levels.md Normal file
View File

@@ -0,0 +1,7 @@
---
"@runfusion/fusion": minor
---
summary: Custom-provider models now offer all thinking levels (Off → Max) and actually send the selected effort.
category: feature
dev: buildCustomProviderModels registers reasoning: true with an identity thinkingLevelMap for xhigh/max.

2
docs/settings-reference.md
View File

@@ -99,6 +99,8 @@ Fallback thinking-level values are applied at runtime when Fusion swaps from the
Fusion persists one canonical ordered vocabulary: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`. Saved settings and runtime validation accept every canonical value so a model switch never silently rewrites a user's choice. Model-bound dashboard controls use the optional `supportedThinkingLevels` metadata from `/api/models` when pi documents a model's `thinkingLevelMap`; `null` entries are excluded, string entries are included, and `xhigh`/`max` are opt-in. Rows without capability metadata retain the full canonical fallback, including `max`, rather than guessing from provider or model names.
Custom-provider models are presumed thinking-capable and expose all seven levels by default; no capability checkbox or other configuration is required. Fusion registers these levels with pi as transmissible, while pi owns API-specific `off` translation and up-then-down clamping. Consequently, a global or inherited thinking level that custom providers previously ignored is now sent to the gateway. A strict gateway can reject an unsupported effort; select **Off** for that gateway, which pi translates to its explicit non-reasoning form or omits when the API has no such form.
| `ntfyEnabled` | `boolean` | `false` | Enable ntfy push notifications. |
| `agentClarificationEnabled` | `boolean` | `false` | Legacy default for programmatic Planning Mode session notification eligibility. Dashboard Planning Mode always starts its infinite, user-validated interview with follow-up questions enabled; this setting no longer suppresses questions or creates a final summary. |
| `failureNotificationMode` | `"sticky-only" \| "terminal-only" \| "all"` | `"sticky-only"` | Failure notification behavior. `sticky-only` defers failed-task notifications by `failureNotificationDelayMs` and suppresses transient self-recoveries. `terminal-only` suppresses while auto-retry is still active and only dispatches when `paused === true` or `column === "in-review"` with `status === "failed"`. `all` restores legacy immediate failure notifications. |

1
packages/dashboard/app/api/settings/provider-status.ts
View File

@@ -703,7 +703,6 @@ export function refreshProviderModels(id: string): Promise<RefreshProviderModels
export interface CustomProviderModelInput {
id: string;
name?: string;
reasoning?: boolean;
contextWindow?: number;
maxTokens?: number;
}

5
packages/dashboard/app/components/CustomProviderForm.css
View File

@@ -18,14 +18,11 @@
.custom-provider-form__model-row {
display: grid;
grid-template-columns: repeat(5, minmax(0, 1fr)) auto;
grid-template-columns: repeat(4, minmax(0, 1fr)) auto;
gap: var(--space-sm);
align-items: center;
}
.custom-provider-form__toggle {
justify-content: flex-start;
}
.custom-provider-form__actions {
display: flex;

18
packages/dashboard/app/components/CustomProviderForm.tsx
View File

@@ -33,7 +33,7 @@ type Props = {
};
function emptyModel(): CustomProviderModelInput {
return { id: "", name: "", reasoning: false };
return { id: "", name: "" };
}
export function CustomProviderForm({ initialConfig, onSave, onCancel, saving = false, error }: Props) {
@@ -100,7 +100,6 @@ export function CustomProviderForm({ initialConfig, onSave, onCancel, saving = f
.map((m) => ({
id: m.id,
name: m.name || m.id,
reasoning: Boolean(m.reasoning),
contextWindow: m.contextWindow,
maxTokens: m.maxTokens,
}));
@@ -159,7 +158,6 @@ export function CustomProviderForm({ initialConfig, onSave, onCancel, saving = f
models: models.map((model) => ({
id: model.id.trim(),
name: model.name?.trim() || undefined,
reasoning: Boolean(model.reasoning),
contextWindow: model.contextWindow,
maxTokens: model.maxTokens,
})),
@@ -199,6 +197,11 @@ export function CustomProviderForm({ initialConfig, onSave, onCancel, saving = f
<label>{t("providers.fields.models", "Models")}</label>
<div className="custom-provider-form__models">
{models.map((model, index) => (
/*
FNXC:CustomProviders 2026-08-19-15:13:
There is no reasoning capability control because custom models are presumed thinking-capable.
Pi derives selector options and execution behavior from their shared server registration.
*/
<div key={`${index}-model`} className="custom-provider-form__model-row">
<input
className="input"
@@ -216,15 +219,6 @@ export function CustomProviderForm({ initialConfig, onSave, onCancel, saving = f
onChange={(e) => updateModel(index, { name: e.target.value })}
disabled={saving}
/>
<label className="checkbox-label custom-provider-form__toggle">
<input
type="checkbox"
checked={Boolean(model.reasoning)}
onChange={(e) => updateModel(index, { reasoning: e.target.checked })}
disabled={saving}
/>
{t("providers.fields.reasoning", "Reasoning")}
</label>
<input
className="input"
aria-label={`${t("providers.fields.contextWindow", "Context window")} ${index + 1}`}

35
packages/dashboard/app/components/__tests__/CustomProviderForm.test.tsx
View File

@@ -44,14 +44,25 @@ describe("CustomProviderForm", () => {
await user.click(screen.getByRole("button", { name: "Save Provider" }));
expect(onSave).toHaveBeenCalledWith(expect.objectContaining({
expect(onSave).toHaveBeenCalledWith({
id: "my-proxy",
name: "My Proxy",
baseUrl: "https://proxy.example.com/v1",
api: "openai-responses",
apiKey: "MY_API_KEY",
models: [expect.objectContaining({ id: "gpt-4.1-mini", name: "GPT 4.1 Mini" })],
}));
models: [{
id: "gpt-4.1-mini",
name: "GPT 4.1 Mini",
contextWindow: undefined,
maxTokens: undefined,
}],
});
});
it("does not render a reasoning capability toggle", () => {
render(<CustomProviderForm onSave={vi.fn()} />);
expect(screen.queryByRole("checkbox")).not.toBeInTheDocument();
expect(screen.queryByText("Reasoning")).not.toBeInTheDocument();
});
it("shows external error state", () => {
@@ -132,8 +143,8 @@ describe("Detect Models", () => {
it("calls probeProviderModels and adds discovered models", async () => {
const mockProbe = vi.spyOn(api, "probeProviderModels").mockResolvedValue({
models: [
{ id: "gpt-4o", name: "GPT 4o", reasoning: false },
{ id: "gpt-4", name: "GPT 4", reasoning: false },
{ id: "gpt-4o", name: "GPT 4o" },
{ id: "gpt-4", name: "GPT 4" },
],
count: 2,
});
@@ -146,7 +157,7 @@ describe("Detect Models", () => {
baseUrl: "https://api.example.com/v1",
api: "openai-completions",
apiKey: "sk-test",
models: [{ id: "", name: "", reasoning: false }],
models: [{ id: "", name: "" }],
}}
/>
);
@@ -170,8 +181,8 @@ describe("Detect Models", () => {
it("deduplicates models when detecting", async () => {
const mockProbe = vi.spyOn(api, "probeProviderModels").mockResolvedValue({
models: [
{ id: "gpt-4o", name: "GPT 4o", reasoning: false },
{ id: "gpt-4", name: "GPT 4", reasoning: false },
{ id: "gpt-4o", name: "GPT 4o" },
{ id: "gpt-4", name: "GPT 4" },
],
count: 2,
});
@@ -185,8 +196,8 @@ describe("Detect Models", () => {
api: "openai-completions",
apiKey: "sk-test",
models: [
{ id: "gpt-4o", name: "GPT 4o", reasoning: false },
{ id: "", name: "", reasoning: false },
{ id: "gpt-4o", name: "GPT 4o" },
{ id: "", name: "" },
],
}}
/>
@@ -215,7 +226,7 @@ describe("Detect Models", () => {
baseUrl: "https://api.example.com/v1",
api: "openai-completions",
apiKey: "sk-invalid",
models: [{ id: "", name: "", reasoning: false }],
models: [{ id: "", name: "" }],
}}
/>
);
@@ -235,7 +246,7 @@ describe("Detect Models", () => {
baseUrl: "",
api: "openai-completions",
apiKey: "sk-test",
models: [{ id: "", name: "", reasoning: false }],
models: [{ id: "", name: "" }],
}}
/>
);

91
packages/dashboard/src/__tests__/register-model-routes-custom-provider-thinking.test.ts Normal file
View File

@@ -0,0 +1,91 @@
import { THINKING_LEVELS } from "@fusion/core";
import type { Router } from "express";
import { describe, expect, it, vi } from "vitest";
import { registerModelRoutes } from "../routes/register-model-routes.js";
type RegistryModel = {
provider: string;
id: string;
name: string;
reasoning: boolean;
thinkingLevelMap?: { xhigh?: string | null; max?: string | null };
contextWindow: number;
};
function createModelsHandler(models: RegistryModel[]) {
const getHandlers = new Map<string, (req: unknown, res: { json: (body: unknown) => void }) => Promise<void>>();
const router = {
post: vi.fn(),
get: vi.fn((path: string, handler: (req: unknown, res: { json: (body: unknown) => void }) => Promise<void>) => {
getHandlers.set(path, handler);
}),
} as unknown as Router;
const registry = {
refresh: vi.fn(),
getAvailable: vi.fn(() => models),
getAll: vi.fn(() => models),
};
const customProviders = [...new Set(models.map((model) => model.provider))].map((provider) => ({
id: provider,
name: provider,
apiType: "openai-compatible" as const,
baseUrl: "https://custom.example/v1",
models: [],
}));
registerModelRoutes({
router,
store: {
getGlobalSettingsStore: () => ({ getSettings: vi.fn().mockResolvedValue({ customProviders }) }),
getSettingsFast: vi.fn().mockResolvedValue({}),
} as never,
runtimeLogger: { child: vi.fn(() => ({ warn: vi.fn() })) } as never,
options: { modelRegistry: registry } as never,
} as never);
return getHandlers.get("/models")!;
}
async function getModels(handler: ReturnType<typeof createModelsHandler>) {
const json = vi.fn();
await handler({}, { json });
return json.mock.calls[0][0] as { models: Array<RegistryModel & { supportedThinkingLevels?: string[] }> };
}
describe("custom-provider thinking levels on /api/models", () => {
it("emits the complete canonical list from a custom-provider registration", async () => {
const response = await getModels(createModelsHandler([{
provider: "custom-provider",
id: "custom-model",
name: "Custom Model",
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
contextWindow: 128_000,
}]));
expect(response.models).toContainEqual(expect.objectContaining({
provider: "custom-provider",
id: "custom-model",
reasoning: true,
supportedThinkingLevels: THINKING_LEVELS,
}));
});
it("keeps catalog non-thinking models off-only", async () => {
const response = await getModels(createModelsHandler([{
provider: "catalog",
id: "non-thinking-model",
name: "Non-thinking Model",
reasoning: false,
contextWindow: 128_000,
}]));
expect(response.models).toContainEqual(expect.objectContaining({
provider: "catalog",
id: "non-thinking-model",
reasoning: false,
supportedThinkingLevels: ["off"],
}));
});
});

49
packages/engine/src/__tests__/custom-provider-thinking-levels.test.ts Normal file
View File

@@ -0,0 +1,49 @@
import { getSupportedThinkingLevels } from "@earendil-works/pi-ai";
import { THINKING_LEVELS, type CustomProvider } from "@fusion/core";
import { describe, expect, it } from "vitest";
import { buildCustomProviderModels, resolveApiType } from "../auth/custom-provider-registry.js";
const API_TYPES = [
"openai-compatible",
"openai-responses",
"anthropic-compatible",
"google-generative-ai",
] as const;
describe("custom-provider thinking levels", () => {
it.each(API_TYPES)("registers every %s model with the canonical transmissible thinking levels", (apiType) => {
const provider: CustomProvider = {
id: "550e8400-e29b-41d4-a716-446655440000",
name: "Custom Provider",
apiType,
baseUrl: "https://custom.example/v1",
models: [{ id: "custom-model", name: "Custom Model" }],
};
const resolvedApi = resolveApiType(apiType);
const [model] = buildCustomProviderModels(provider, resolvedApi);
expect(resolvedApi).toBe(
apiType === "openai-compatible" ? "openai-completions"
: apiType === "anthropic-compatible" ? "anthropic-messages"
: apiType,
);
expect(model).toMatchObject({
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
});
expect(getSupportedThinkingLevels(model)).toEqual(THINKING_LEVELS);
});
it("keeps providers with absent or empty model lists empty", () => {
const provider = {
id: "550e8400-e29b-41d4-a716-446655440000",
name: "Custom Provider",
apiType: "openai-compatible",
baseUrl: "https://custom.example/v1",
} as CustomProvider;
expect(buildCustomProviderModels(provider, "openai-completions")).toEqual([]);
expect(buildCustomProviderModels({ ...provider, models: [] }, "openai-completions")).toEqual([]);
});
});

2
packages/engine/src/__tests__/custom-providers-openai-completions.test.ts
View File

@@ -47,7 +47,7 @@ describe("custom providers openai-completions regression", () => {
baseUrl: provider.baseUrl,
api: "openai-completions",
apiKey: provider.apiKey,
models: [{ id: "my-model", name: "My Model", reasoning: false, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 128000, maxTokens: 16384 }],
models: [{ id: "my-model", name: "My Model", reasoning: true, thinkingLevelMap: { xhigh: "xhigh", max: "max" }, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 128000, maxTokens: 16384 }],
});
await modelRegistry.refresh();

3
packages/engine/src/__tests__/custom-providers-openai-responses.test.ts
View File

@@ -53,7 +53,8 @@ describe("custom providers openai-responses regression", () => {
models: [{
id: "gpt-5.4",
name: "GPT 5.4",
reasoning: false,
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 128000,

31
packages/engine/src/__tests__/pi-create-fn-agent.test.ts
View File

@@ -2041,6 +2041,14 @@ describe("createFnAgent", () => {
anthropicPromptCaching: true,
models: [{ id: "anthropic-model", name: "Anthropic Model" }],
},
{
id: "880e8400-e29b-41d4-a716-446655440003",
name: "Custom Google Provider",
apiType: "google-generative-ai",
baseUrl: "https://google.example",
apiKey: "GOOGLE_API_KEY",
models: [{ id: "google-model", name: "Google Model" }],
},
] as any);
const { createPiAgentSessionRaw: createFnAgent } = await import("../pi.js");
@@ -2058,6 +2066,8 @@ describe("createFnAgent", () => {
api: "openai-completions",
models: [expect.objectContaining({
id: "custom-model",
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
compat: expect.objectContaining({ cacheControlFormat: "anthropic" }),
})],
}));
@@ -2065,16 +2075,33 @@ describe("createFnAgent", () => {
// Opted-out (default) openai-compatible provider: no forced cache_control marker.
const noCacheCall = registerProviderMock.mock.calls.find(([key]: [string]) => key === "custom-openai-no-caching");
expect(noCacheCall).toBeDefined();
const [, noCacheConfig] = noCacheCall as [string, { models: Array<{ compat?: Record<string, unknown> }> }];
const [, noCacheConfig] = noCacheCall as [string, { models: Array<{ reasoning: boolean; thinkingLevelMap: Record<string, string>; compat?: Record<string, unknown> }> }];
expect(noCacheConfig.models[0]).toMatchObject({
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
});
expect(noCacheConfig.models[0].compat).not.toHaveProperty("cacheControlFormat");
// anthropic-compatible provider: opt-in is a documented no-op — pi-ai's anthropic path already
// auto-caches without this compat flag, and openai-completions-only compat must not leak in.
const anthropicCall = registerProviderMock.mock.calls.find(([key]: [string]) => key === "custom-anthropic-caching-opt-in");
expect(anthropicCall).toBeDefined();
const [, anthropicConfig] = anthropicCall as [string, { api: string; models: Array<{ compat?: Record<string, unknown> }> }];
const [, anthropicConfig] = anthropicCall as [string, { api: string; models: Array<{ reasoning: boolean; thinkingLevelMap: Record<string, string>; compat?: Record<string, unknown> }> }];
expect(anthropicConfig.api).toBe("anthropic-messages");
expect(anthropicConfig.models[0]).toMatchObject({
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
});
expect(anthropicConfig.models[0].compat).toBeUndefined();
const googleCall = registerProviderMock.mock.calls.find(([key]: [string]) => key === "custom-google-provider");
expect(googleCall).toBeDefined();
const [, googleConfig] = googleCall as [string, { api: string; models: Array<{ reasoning: boolean; thinkingLevelMap: Record<string, string> }> }];
expect(googleConfig.api).toBe("google-generative-ai");
expect(googleConfig.models[0]).toMatchObject({
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
});
});
it("avoids lock-based SettingsManager.create when loading extension providers", async () => {

20
packages/engine/src/__tests__/provider-registration.test.ts
View File

@@ -146,7 +146,7 @@ describe("seedDashboardProviders", () => {
baseUrl: provider.baseUrl,
api: "openai-completions",
apiKey: provider.apiKey,
models: [{ id: "acme-1", name: "Acme Model 1", reasoning: false, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 128000, maxTokens: 16384 }],
models: [{ id: "acme-1", name: "Acme Model 1", reasoning: true, thinkingLevelMap: { xhigh: "xhigh", max: "max" }, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 128000, maxTokens: 16384 }],
});
const registryB = await createInMemoryModelRegistry();
@@ -184,6 +184,24 @@ describe("seedDashboardProviders", () => {
expect(providerIds).toEqual(expect.arrayContaining(["zai", "openrouter", "acme-one", "acme-two"]));
});
it("registers a google-generative-ai custom provider under the Google api key", async () => {
const modelRegistry = makeModelRegistry();
const provider = customProvider({ apiType: "google-generative-ai", baseUrl: "https://google.test" });
await registerCustomProviders(modelRegistry, [provider], vi.fn());
expect(modelRegistry.registerProvider).toHaveBeenCalledWith(
"acme-ai",
expect.objectContaining({
api: "google-generative-ai",
models: [expect.objectContaining({
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
})],
}),
);
});
it("registers an anthropic-compatible custom provider under the anthropic-messages api key (FN-7690)", async () => {
const store = makeStore([
customProvider({

14
packages/engine/src/auth/custom-provider-registry.ts
View File

@@ -17,6 +17,7 @@ interface ModelRegistryLike extends RefreshableModelRegistry {
id: string;
name: string;
reasoning: boolean;
thinkingLevelMap?: { xhigh?: string | null; max?: string | null };
input: ("text" | "image")[];
cost: { input: number; output: number; cacheRead: number; cacheWrite: number };
contextWindow: number;
@@ -50,6 +51,10 @@ export function resolveApiType(apiType: string): string {
if (apiType === "openai-responses") {
return "openai-responses";
}
// FNXC:CustomProviders 2026-08-19-15:28: Google-compatible custom providers must retain pi's Google API dialect so its shared thinking translation handles Off and every selected effort.
if (apiType === "google-generative-ai") {
return "google-generative-ai";
}
return "openai-completions";
}
@@ -66,6 +71,11 @@ export function resolveApiType(apiType: string): string {
* anthropic path already auto-caches without any flag, and `openai-responses` uses OpenAI's
* native `prompt_cache_key`/`prompt_cache_retention` mechanism (no `cache_control` marker concept
* per pi-ai's `OpenAIResponsesCompat`), so the opt-in is inert there by construction.
*
* FNXC:CustomProviders 2026-08-19-15:13:
* Custom-provider models are presumed thinking-capable without a user-declared capability. Register
* all seven canonical levels as transmissible so pi owns Off translation and up-then-down clamping;
* the same registration is the source for selector display and execution.
*/
export function buildCustomProviderModels(
provider: CustomProvider,
@@ -74,6 +84,7 @@ export function buildCustomProviderModels(
id: string;
name: string;
reasoning: boolean;
thinkingLevelMap: { xhigh: string; max: string };
input: ("text" | "image")[];
cost: { input: number; output: number; cacheRead: number; cacheWrite: number };
contextWindow: number;
@@ -86,7 +97,8 @@ export function buildCustomProviderModels(
return (provider.models ?? []).map((model) => ({
id: model.id,
name: model.name,
reasoning: false,
reasoning: true,
thinkingLevelMap: { xhigh: "xhigh", max: "max" },
input: ["text" as const],
cost: {
input: 0,

9
packages/engine/src/pi.ts
View File

@@ -1147,16 +1147,23 @@ export interface AgentOptions {
* provider register but threw "No API provider registered for api: anthropic"
* the moment a task tried to stream.
*
* FNXC:CustomProviders 2026-08-19-15:28:
* Google-compatible custom providers keep the registered Google dialect. This path and dashboard
* registration must agree so pi performs the same thinking-level translation for every custom model.
*
* @param apiType - the custom provider's declared compatibility type.
* @returns the registered pi-ai api key to stream against.
*/
function resolveCustomProviderApiType(apiType: string): "anthropic-messages" | "openai-responses" | "openai-completions" {
function resolveCustomProviderApiType(apiType: string): "anthropic-messages" | "openai-responses" | "google-generative-ai" | "openai-completions" {
if (apiType === "anthropic-compatible") {
return "anthropic-messages";
}
if (apiType === "openai-responses") {
return "openai-responses";
}
if (apiType === "google-generative-ai") {
return "google-generative-ai";
}
return "openai-completions";
}