FN-6665: group token analytics by runtime model
Record runtime model snapshots so token analytics group usage by the model that actually generated it. - Add token-usage provider/model snapshot columns, store mapping, and migration support. - Preserve actually-used session model metadata during executor/session token accumulation without changing task model overrides. - Prefer runtime model snapshots in token provider/model aggregation and cover the behavior with regression tests and docs. Files changed: .changeset/fn-6665-tokens-by-model.md | 5 ++ docs/dashboard-guide.md | 2 +- docs/storage.md | 2 + .../core/src/__tests__/store-token-usage.test.ts | 4 ++ .../core/src/__tests__/token-analytics.test.ts | 54 ++++++++++++++++++++- packages/core/src/db.ts | 12 ++++- packages/core/src/store.ts | 10 +++- packages/core/src/token-analytics.ts | 12 ++++- packages/core/src/types.ts | 10 ++++ .../src/__tests__/session-token-usage.test.ts | 55 ++++++++++++++++++++-- packages/engine/src/executor.ts | 37 +++++++++++++-- packages/engine/src/session-token-usage.ts | 7 +++ 12 files changed, 194 insertions(+), 16 deletions(-) Fusion-Task-Id: FN-6665 Fusion-Task-Lineage: e103de14-6298-4a9e-93af-4dd15798fde5
This commit is contained in:
gsxdsm
5
.changeset/fn-6665-tokens-by-model.md
Normal file
5
.changeset/fn-6665-tokens-by-model.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Fix Command Center token analytics so Tokens by model and the per-model table group tasks by the actually-used runtime model instead of collapsing resolved-via-settings usage into `(unknown)`.
|
||||
@@ -664,7 +664,7 @@ Navigation:
|
||||
Features:
|
||||
- Global date-range picker in the header scopes the analytics tabs; **Mission Control** remains live rather than historical.
|
||||
- **Overview** summarizes token usage/cost, autonomy, active nodes, agent runs, tasks done, model breadth, and open signals, and includes the SDLC throughput funnel for the selected range. Its token total and Live activity snapshot token metric refresh on a bounded live cadence and animate number changes while preserving reduced-motion preferences. The Live activity snapshot also shows the current board-state count for tasks in progress, independent of the selected analytics date range. Overview includes a graph-rich software-factory snapshot with tokens-by-model, tool-category, and daily activity trend charts that reuse the already-loaded tokens, tools, and activity analytics. The chart reveal/glow accents are decorative and disabled when reduced-motion preferences are active. The SDLC completion rate is shown as a radial gauge and is calculated as cohort conversion from in-range triage entrants, so the rate is capped at 100% even when older tasks finish during the range.
|
||||
- **Tokens** breaks down token totals, estimated cost, tasks, and per-model usage. It also includes a live token-usage-over-time chart backed by per-task token timestamps; use the granularity control to switch the chart between hourly, daily, and weekly buckets. The token total and chart poll on a bounded cadence, keep the previous data visible during refresh, animate decorative count/bar transitions, and disable those animations for reduced-motion users.
|
||||
- **Tokens** breaks down token totals, estimated cost, tasks, and per-model usage. Per-model and per-provider breakdowns use the task's analytics-only actually-used model snapshot when available, so usage from settings-resolved runs appears under the real runtime model instead of `(unknown)` without changing future model resolution. It also includes a live token-usage-over-time chart backed by per-task token timestamps; use the granularity control to switch the chart between hourly, daily, and weekly buckets. The token total and chart poll on a bounded cadence, keep the previous data visible during refresh, animate decorative count/bar transitions, and disable those animations for reduced-motion users.
|
||||
- **Tools** shows autonomy ratio, tool-call volume, intervention counts, sessions, and tool categories.
|
||||
- **Activity** tracks sessions, messages, active nodes, active agents, agent heartbeat runs, and stickiness. Agent-run sheets show total, active, completed, and failed runs for the selected range, and the Agent runs/day sparkline trends runs by `agentRuns.startedAt`. The area also renders live animated line charts for messages/day, active agents/day, active nodes/day, and combined throughput/day (`messages + active agents + active nodes`). These charts reuse the existing activity analytics endpoint, refresh on a bounded 15-second cadence while mounted, keep the previous data visible during refreshes, and disable decorative draw-on motion for reduced-motion users.
|
||||
- **Productivity** separates outcome counters (commits and pull requests) from volume proxies such as modified files, lines changed, and files by language.
|
||||
|
||||
@@ -387,6 +387,8 @@ Backups in `.fusion/backups/` now capture the project DB and (when present) the
|
||||
FN-5240/FN-5241/FN-5242 establish the handoff invariant: the only legal executor/self-healing path into `in-review` after execution finishes is `TaskStore.handoffToReview(...)`. That helper runs the column move, `mergeQueue` insert, and handoff audit fan-out inside one `BEGIN IMMEDIATE` transaction so observers never see `column = "in-review"` without the matching queue row. Direct `moveTask(taskId, "in-review")` writes remain allowed for explicit non-handoff/test paths but emit `task:handoff-invariant-violation` run-audit events unless the caller opts into the narrow allowlist flag.
|
||||
|
||||
The `tasks.githubTracking` JSON column stores per-task GitHub tracking state (`enabled`, optional `repoOverride`, linked issue metadata, and `unlinkedAt`). It is additive and default-off; imported-source issue metadata remains in `issueInfo` / `sourceIssue`. Behavior wiring (issue creation/lifecycle sync and UI surfacing) lands in FN-3870/FN-3873/FN-3874.
|
||||
|
||||
The `tasks.tokenUsage*` columns store cumulative per-task token usage for analytics. `tokenUsageModelProvider` and `tokenUsageModelId` are analytics-only snapshots of the actually-used runtime model recorded when usage is accumulated; they let Command Center group resolved-via-settings usage by provider/model without writing the task-level `modelProvider` / `modelId` own-model override fields that control future model resolution.
|
||||
| `config` | Single-row project configuration (`nextId`, settings payload, workflow step counters). |
|
||||
| `workflow_steps` | Workflow step definitions (`prompt`/`script`) with phase, template metadata, and model overrides. |
|
||||
| `activityLog` | Per-project activity/event log with timestamp/type/task indexes. |
|
||||
|
||||
@@ -28,6 +28,8 @@ describe("TaskStore", () => {
|
||||
totalTokens: 204,
|
||||
firstUsedAt: "2026-04-23T10:00:00.000Z",
|
||||
lastUsedAt: "2026-04-23T10:05:00.000Z",
|
||||
modelProvider: "anthropic",
|
||||
modelId: "claude-sonnet-4-5",
|
||||
};
|
||||
|
||||
const task = await harness.store().createTask({
|
||||
@@ -52,6 +54,8 @@ describe("TaskStore", () => {
|
||||
totalTokens: 345,
|
||||
firstUsedAt: "2026-04-23T12:00:00.000Z",
|
||||
lastUsedAt: "2026-04-23T12:30:00.000Z",
|
||||
modelProvider: "openai",
|
||||
modelId: "gpt-5",
|
||||
};
|
||||
|
||||
const updated = await harness.store().updateTask(task.id, { tokenUsage });
|
||||
|
||||
@@ -17,6 +17,8 @@ interface TaskSeed {
|
||||
lastUsedAt: string | null;
|
||||
modelProvider?: string | null;
|
||||
modelId?: string | null;
|
||||
tokenUsageModelProvider?: string | null;
|
||||
tokenUsageModelId?: string | null;
|
||||
nodeId?: string | null;
|
||||
agentId?: string | null;
|
||||
}
|
||||
@@ -27,9 +29,9 @@ function insertTask(db: Database, t: TaskSeed): void {
|
||||
(id, description, "column", createdAt, updatedAt,
|
||||
tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens,
|
||||
tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageLastUsedAt,
|
||||
modelProvider, modelId, checkoutNodeId, assignedAgentId)
|
||||
modelProvider, modelId, tokenUsageModelProvider, tokenUsageModelId, checkoutNodeId, assignedAgentId)
|
||||
VALUES (?, 'desc', 'todo', '2026-01-01T00:00:00.000Z', '2026-01-01T00:00:00.000Z',
|
||||
?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
t.id,
|
||||
t.inputTokens ?? null,
|
||||
@@ -40,6 +42,8 @@ function insertTask(db: Database, t: TaskSeed): void {
|
||||
t.lastUsedAt,
|
||||
t.modelProvider ?? null,
|
||||
t.modelId ?? null,
|
||||
t.tokenUsageModelProvider ?? null,
|
||||
t.tokenUsageModelId ?? null,
|
||||
t.nodeId ?? null,
|
||||
t.agentId ?? null,
|
||||
);
|
||||
@@ -90,6 +94,52 @@ describe("token-analytics", () => {
|
||||
expect(result.groups[0].key).toBe("model-A");
|
||||
});
|
||||
|
||||
it("groups resolved-via-settings token usage by the actually-used model snapshot", () => {
|
||||
insertTask(db, { id: "t1", inputTokens: 100, outputTokens: 50, totalTokens: 150, lastUsedAt: "2026-03-01T00:00:00.000Z", modelId: null, modelProvider: null, tokenUsageModelId: "claude-sonnet-4-5", tokenUsageModelProvider: "anthropic" });
|
||||
insertTask(db, { id: "t2", inputTokens: 25, outputTokens: 25, totalTokens: 50, lastUsedAt: "2026-03-02T00:00:00.000Z", modelId: null, modelProvider: null, tokenUsageModelId: "gpt-5", tokenUsageModelProvider: "openai" });
|
||||
insertTask(db, { id: "t3", inputTokens: 30, outputTokens: 20, totalTokens: 50, lastUsedAt: "2026-03-03T00:00:00.000Z", modelId: null, modelProvider: null, tokenUsageModelId: "gpt-5", tokenUsageModelProvider: "openai" });
|
||||
|
||||
const result = aggregateTokenAnalytics(db, { groupBy: "model" });
|
||||
|
||||
const groups = new Map(result.groups.map((g) => [g.key, g]));
|
||||
expect([...groups.keys()].sort()).toEqual(["claude-sonnet-4-5", "gpt-5"]);
|
||||
expect(groups.get("claude-sonnet-4-5")).toMatchObject({ totalTokens: 150, inputTokens: 100, outputTokens: 50, nTasks: 1 });
|
||||
expect(groups.get("gpt-5")).toMatchObject({ totalTokens: 100, inputTokens: 55, outputTokens: 45, nTasks: 2 });
|
||||
expect(groups.has(null)).toBe(false);
|
||||
});
|
||||
|
||||
it("groups providers by the token-usage snapshot before task own-provider", () => {
|
||||
insertTask(db, { id: "t1", inputTokens: 100, totalTokens: 100, lastUsedAt: "2026-03-01T00:00:00.000Z", modelProvider: null, tokenUsageModelProvider: "anthropic", tokenUsageModelId: "claude-sonnet-4-5" });
|
||||
insertTask(db, { id: "t2", inputTokens: 200, totalTokens: 200, lastUsedAt: "2026-03-02T00:00:00.000Z", modelProvider: null, tokenUsageModelProvider: "openai", tokenUsageModelId: "gpt-5" });
|
||||
insertTask(db, { id: "t3", inputTokens: 25, totalTokens: 25, lastUsedAt: "2026-03-03T00:00:00.000Z", modelProvider: "legacy-provider", tokenUsageModelProvider: "openai", tokenUsageModelId: "gpt-5" });
|
||||
|
||||
const result = aggregateTokenAnalytics(db, { groupBy: "provider" });
|
||||
|
||||
expect(new Map(result.groups.map((g) => [g.key, g.totalTokens]))).toEqual(
|
||||
new Map([["anthropic", 100], ["openai", 225]]),
|
||||
);
|
||||
});
|
||||
|
||||
it("falls back to legacy task model columns when no token snapshot exists", () => {
|
||||
insertTask(db, { id: "legacy", inputTokens: 40, totalTokens: 40, lastUsedAt: "2026-03-01T00:00:00.000Z", modelProvider: "anthropic", modelId: "legacy-model" });
|
||||
|
||||
const result = aggregateTokenAnalytics(db, { groupBy: "model" });
|
||||
|
||||
expect(result.groups).toHaveLength(1);
|
||||
expect(result.groups[0]).toMatchObject({ key: "legacy-model", totalTokens: 40, nTasks: 1 });
|
||||
});
|
||||
|
||||
it("keeps own-model and resolved-model token snapshots as distinct model groups", () => {
|
||||
insertTask(db, { id: "own", inputTokens: 100, totalTokens: 100, lastUsedAt: "2026-03-01T00:00:00.000Z", modelProvider: "anthropic", modelId: "own-model", tokenUsageModelProvider: "anthropic", tokenUsageModelId: "own-model" });
|
||||
insertTask(db, { id: "resolved", inputTokens: 75, totalTokens: 75, lastUsedAt: "2026-03-02T00:00:00.000Z", modelProvider: null, modelId: null, tokenUsageModelProvider: "openai", tokenUsageModelId: "resolved-model" });
|
||||
|
||||
const result = aggregateTokenAnalytics(db, { groupBy: "model" });
|
||||
|
||||
expect(new Map(result.groups.map((g) => [g.key, g.totalTokens]))).toEqual(
|
||||
new Map([["own-model", 100], ["resolved-model", 75]]),
|
||||
);
|
||||
});
|
||||
|
||||
it("groups by provider, node, and agent", () => {
|
||||
insertTask(db, { id: "t1", inputTokens: 100, totalTokens: 100, lastUsedAt: "2026-03-01T00:00:00.000Z", modelProvider: "anthropic", nodeId: "node-1", agentId: "agent-x" });
|
||||
insertTask(db, { id: "t2", inputTokens: 200, totalTokens: 200, lastUsedAt: "2026-03-02T00:00:00.000Z", modelProvider: "openai", nodeId: "node-1", agentId: "agent-y" });
|
||||
|
||||
@@ -162,7 +162,7 @@ export function isFts5CorruptionError(error: unknown): boolean {
|
||||
|
||||
// ── Schema Definition ────────────────────────────────────────────────
|
||||
|
||||
const SCHEMA_VERSION = 120;
|
||||
const SCHEMA_VERSION = 121;
|
||||
|
||||
const TASKS_FTS_AUTOMERGE = 8;
|
||||
const TASKS_FTS_CRISISMERGE = 16;
|
||||
@@ -285,6 +285,8 @@ CREATE TABLE IF NOT EXISTS tasks (
|
||||
tokenUsageTotalTokens INTEGER,
|
||||
tokenUsageFirstUsedAt TEXT,
|
||||
tokenUsageLastUsedAt TEXT,
|
||||
tokenUsageModelProvider TEXT,
|
||||
tokenUsageModelId TEXT,
|
||||
tokenBudgetSoftAlertedAt TEXT,
|
||||
tokenBudgetHardAlertedAt TEXT,
|
||||
tokenBudgetOverride TEXT,
|
||||
@@ -4945,6 +4947,14 @@ export class Database {
|
||||
});
|
||||
}
|
||||
|
||||
// Migration 121: Token-usage model snapshot for Command Center analytics.
|
||||
if (version < 121) {
|
||||
this.applyMigration(121, () => {
|
||||
this.addColumnIfMissing("tasks", "tokenUsageModelProvider", "TEXT");
|
||||
this.addColumnIfMissing("tasks", "tokenUsageModelId", "TEXT");
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -233,6 +233,8 @@ interface TaskRow {
|
||||
tokenUsageTotalTokens: number | null;
|
||||
tokenUsageFirstUsedAt: string | null;
|
||||
tokenUsageLastUsedAt: string | null;
|
||||
tokenUsageModelProvider: string | null;
|
||||
tokenUsageModelId: string | null;
|
||||
tokenBudgetSoftAlertedAt: string | null;
|
||||
tokenBudgetHardAlertedAt: string | null;
|
||||
tokenBudgetOverride: string | null;
|
||||
@@ -381,6 +383,8 @@ const TASK_COLUMN_DESCRIPTORS: TaskColumnDescriptor[] = [
|
||||
defineTaskColumn("tokenUsageTotalTokens", (task) => task.tokenUsage?.totalTokens ?? null),
|
||||
defineTaskColumn("tokenUsageFirstUsedAt", (task) => task.tokenUsage?.firstUsedAt ?? null),
|
||||
defineTaskColumn("tokenUsageLastUsedAt", (task) => task.tokenUsage?.lastUsedAt ?? null),
|
||||
defineTaskColumn("tokenUsageModelProvider", (task) => task.tokenUsage?.modelProvider ?? null),
|
||||
defineTaskColumn("tokenUsageModelId", (task) => task.tokenUsage?.modelId ?? null),
|
||||
defineTaskColumn("tokenBudgetSoftAlertedAt", (task) => task.tokenBudgetSoftAlertedAt ?? null),
|
||||
defineTaskColumn("tokenBudgetHardAlertedAt", (task) => task.tokenBudgetHardAlertedAt ?? null),
|
||||
defineTaskColumn("tokenBudgetOverride", (task) => toJsonNullable(task.tokenBudgetOverride)),
|
||||
@@ -2014,6 +2018,8 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
totalTokens: row.tokenUsageTotalTokens,
|
||||
firstUsedAt: row.tokenUsageFirstUsedAt,
|
||||
lastUsedAt: row.tokenUsageLastUsedAt,
|
||||
modelProvider: row.tokenUsageModelProvider ?? undefined,
|
||||
modelId: row.tokenUsageModelId ?? undefined,
|
||||
};
|
||||
})(),
|
||||
attachments: (() => { const a = fromJson<TaskAttachment[]>(row.attachments); return a && a.length > 0 ? a : undefined; })(),
|
||||
@@ -2479,7 +2485,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
|
||||
"dependencies", "steps", "customFields", "comments", "review", "reviewState", "workflowStepResults", "steeringComments",
|
||||
"attachments", "prInfo", "prInfos", "issueInfo", "githubTracking", "sourceIssueProvider", "sourceIssueRepository", "sourceIssueExternalIssueId", "sourceIssueNumber", "sourceIssueUrl", "mergeDetails",
|
||||
@@ -2528,7 +2534,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
|
||||
"dependencies", "steps", "customFields", "attachments", "steeringComments",
|
||||
"comments", "review", "reviewState", "workflowStepResults", "prInfo", "prInfos", "issueInfo", "githubTracking", "sourceIssueProvider", "sourceIssueRepository", "sourceIssueExternalIssueId", "sourceIssueNumber", "sourceIssueUrl", "mergeDetails",
|
||||
|
||||
@@ -105,6 +105,8 @@ interface TaskTokenRow {
|
||||
totalTokens: number | null;
|
||||
modelProvider: string | null;
|
||||
modelId: string | null;
|
||||
tokenUsageModelProvider: string | null;
|
||||
tokenUsageModelId: string | null;
|
||||
checkoutNodeId: string | null;
|
||||
assignedAgentId: string | null;
|
||||
tokenUsageLastUsedAt: string;
|
||||
@@ -113,9 +115,13 @@ interface TaskTokenRow {
|
||||
function groupKeyFor(row: TaskTokenRow, groupBy: TokenGroupBy): string | null {
|
||||
switch (groupBy) {
|
||||
case "model":
|
||||
return row.modelId;
|
||||
/*
|
||||
* FNXC:TokenAnalytics 2026-06-18-16:23:
|
||||
* By-model analytics must prefer the analytics-only actually-used model snapshot because task.modelId is only an own-model override. Fall back to legacy task.modelId so pre-snapshot rows keep their historical grouping and never throw.
|
||||
*/
|
||||
return row.tokenUsageModelId ?? row.modelId;
|
||||
case "provider":
|
||||
return row.modelProvider;
|
||||
return row.tokenUsageModelProvider ?? row.modelProvider;
|
||||
case "node":
|
||||
return row.checkoutNodeId;
|
||||
case "agent":
|
||||
@@ -243,6 +249,8 @@ export function aggregateTokenAnalytics(
|
||||
tokenUsageTotalTokens AS totalTokens,
|
||||
modelProvider,
|
||||
modelId,
|
||||
tokenUsageModelProvider,
|
||||
tokenUsageModelId,
|
||||
checkoutNodeId,
|
||||
assignedAgentId,
|
||||
tokenUsageLastUsedAt
|
||||
|
||||
@@ -1814,6 +1814,16 @@ export interface TaskTokenUsage {
|
||||
firstUsedAt: string;
|
||||
/** ISO-8601 timestamp of the most recent recorded usage event for this task. */
|
||||
lastUsedAt: string;
|
||||
/**
|
||||
* FNXC:TokenAnalytics 2026-06-18-16:23:
|
||||
* Snapshot the provider of the actually-used model for analytics only. This is intentionally distinct from task.modelProvider, which is an own-model override used by model resolution and must not be written by token bookkeeping.
|
||||
*/
|
||||
modelProvider?: string;
|
||||
/**
|
||||
* FNXC:TokenAnalytics 2026-06-18-16:23:
|
||||
* Snapshot the id of the actually-used model for analytics only. This is intentionally distinct from task.modelId, which is an own-model override used by model resolution and must not be written by token bookkeeping.
|
||||
*/
|
||||
modelId?: string;
|
||||
}
|
||||
|
||||
export interface TaskTokenBudget {
|
||||
|
||||
@@ -7,8 +7,11 @@ interface MockSessionStats {
|
||||
tokens?: { input?: number; output?: number; cacheRead?: number; cacheWrite?: number; total?: number };
|
||||
}
|
||||
|
||||
function createSession(stats: MockSessionStats | undefined) {
|
||||
return { getSessionStats: vi.fn(() => stats) } as unknown as Parameters<typeof accumulateSessionTokenUsage>[2];
|
||||
function createSession(
|
||||
stats: MockSessionStats | undefined,
|
||||
model?: { provider?: string; id?: string },
|
||||
) {
|
||||
return { getSessionStats: vi.fn(() => stats), ...(model ? { model } : {}) } as unknown as Parameters<typeof accumulateSessionTokenUsage>[2];
|
||||
}
|
||||
|
||||
function createStore(initial: Task["tokenUsage"]): TaskStore & { _task: Task; updateTask: ReturnType<typeof vi.fn> } {
|
||||
@@ -61,6 +64,47 @@ describe("accumulateSessionTokenUsage", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("persists the actually-used session model snapshot with token usage", async () => {
|
||||
const store = createStore(undefined);
|
||||
const session = createSession(
|
||||
{ tokens: { input: 20, output: 10, cacheRead: 0, cacheWrite: 0 } },
|
||||
{ provider: "anthropic", id: "claude-sonnet-4-5" },
|
||||
);
|
||||
|
||||
await accumulateSessionTokenUsage(store, "FN-1", session);
|
||||
|
||||
const call = store.updateTask.mock.calls[0]![1] as { tokenUsage: Task["tokenUsage"] };
|
||||
expect(call.tokenUsage).toMatchObject({
|
||||
modelProvider: "anthropic",
|
||||
modelId: "claude-sonnet-4-5",
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves an existing model snapshot when the session has no model", async () => {
|
||||
const store = createStore({
|
||||
inputTokens: 50,
|
||||
outputTokens: 20,
|
||||
cachedTokens: 0,
|
||||
cacheWriteTokens: 0,
|
||||
totalTokens: 70,
|
||||
firstUsedAt: "2024-01-01T00:00:00.000Z",
|
||||
lastUsedAt: "2024-01-01T00:00:00.000Z",
|
||||
modelProvider: "openai",
|
||||
modelId: "gpt-5",
|
||||
});
|
||||
const session = createSession({ tokens: { input: 55, output: 25, cacheRead: 0, cacheWrite: 0 } });
|
||||
|
||||
await accumulateSessionTokenUsage(store, "FN-1", session);
|
||||
|
||||
const call = store.updateTask.mock.calls[0]![1] as { tokenUsage: Task["tokenUsage"] };
|
||||
expect(call.tokenUsage).toMatchObject({
|
||||
inputTokens: 105,
|
||||
outputTokens: 45,
|
||||
modelProvider: "openai",
|
||||
modelId: "gpt-5",
|
||||
});
|
||||
});
|
||||
|
||||
it("does nothing when delta is zero (no write, no metrics log)", async () => {
|
||||
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
const store = createStore({
|
||||
@@ -89,10 +133,15 @@ describe("accumulateSessionTokenUsage", () => {
|
||||
executor.tokenUsageBaselines = new Map();
|
||||
executor.activeSessions = new Map();
|
||||
|
||||
await executor.persistTokenUsage("FN-1", { getSessionStats: () => ({ tokens: { input: 3, output: 2, cacheRead: 1, cacheWrite: 0, total: 6 } }) });
|
||||
await executor.persistTokenUsage("FN-1", {
|
||||
getSessionStats: () => ({ tokens: { input: 3, output: 2, cacheRead: 1, cacheWrite: 0, total: 6 } }),
|
||||
model: { provider: "mock", id: "scripted" },
|
||||
});
|
||||
|
||||
const cacheLogCall = errorSpy.mock.calls.find((entry) => String(entry[0]).includes("[token-cache-metrics]"));
|
||||
expect(cacheLogCall).toBeTruthy();
|
||||
const call = store.updateTask.mock.calls[0]![1] as { tokenUsage: Task["tokenUsage"] };
|
||||
expect(call.tokenUsage).toMatchObject({ modelProvider: "mock", modelId: "scripted" });
|
||||
});
|
||||
|
||||
it("swallows store errors instead of throwing", async () => {
|
||||
|
||||
@@ -3332,6 +3332,23 @@ export class TaskExecutor {
|
||||
return merged;
|
||||
}
|
||||
|
||||
private tokenUsageWithModelSnapshot(
|
||||
tokenUsage: TaskTokenUsage,
|
||||
session: AgentSession | undefined,
|
||||
existing: TaskTokenUsage | undefined,
|
||||
): TaskTokenUsage {
|
||||
const model = (session as { model?: { provider?: string; id?: string } } | undefined)?.model;
|
||||
return {
|
||||
...tokenUsage,
|
||||
/*
|
||||
* FNXC:TokenAnalytics 2026-06-18-16:23:
|
||||
* Persist the actually-used session model as an analytics snapshot while leaving task.modelProvider/task.modelId untouched so normal model-resolution hierarchy is not pinned by usage bookkeeping.
|
||||
*/
|
||||
modelProvider: model?.provider ?? existing?.modelProvider,
|
||||
modelId: model?.id ?? existing?.modelId,
|
||||
};
|
||||
}
|
||||
|
||||
private async extractSessionTokenUsage(
|
||||
session: AgentSession | undefined,
|
||||
): Promise<Pick<TaskTokenUsage, "inputTokens" | "outputTokens" | "cachedTokens" | "cacheWriteTokens" | "totalTokens"> | undefined> {
|
||||
@@ -3414,18 +3431,19 @@ export class TaskExecutor {
|
||||
const task = await this.store.getTask(taskId);
|
||||
const merged = this.accumulateTokenUsage(task.tokenUsage, delta);
|
||||
if (!merged) return;
|
||||
const tokenUsage = this.tokenUsageWithModelSnapshot(merged, activeSession, task.tokenUsage);
|
||||
|
||||
tokenCacheMetricsLog.log(JSON.stringify({
|
||||
taskId,
|
||||
agentId: task.assignedAgentId ?? undefined,
|
||||
role: "executor",
|
||||
inputTokens: merged.inputTokens,
|
||||
cachedTokens: merged.cachedTokens,
|
||||
cacheWriteTokens: merged.cacheWriteTokens,
|
||||
hitRatio: merged.inputTokens + merged.cachedTokens > 0 ? merged.cachedTokens / (merged.inputTokens + merged.cachedTokens) : 0,
|
||||
inputTokens: tokenUsage.inputTokens,
|
||||
cachedTokens: tokenUsage.cachedTokens,
|
||||
cacheWriteTokens: tokenUsage.cacheWriteTokens,
|
||||
hitRatio: tokenUsage.inputTokens + tokenUsage.cachedTokens > 0 ? tokenUsage.cachedTokens / (tokenUsage.inputTokens + tokenUsage.cachedTokens) : 0,
|
||||
}));
|
||||
|
||||
await this.store.updateTask(taskId, { tokenUsage: merged });
|
||||
await this.store.updateTask(taskId, { tokenUsage });
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -7303,7 +7321,12 @@ export class TaskExecutor {
|
||||
return;
|
||||
}
|
||||
|
||||
const previousStepTokenUsage = accumulatedStepTokenUsage;
|
||||
accumulatedStepTokenUsage = this.accumulateTokenUsage(accumulatedStepTokenUsage, result.tokenUsage);
|
||||
if (accumulatedStepTokenUsage) {
|
||||
// FNXC:TokenAnalytics 2026-06-18-16:23: Step-scoped token writes must not clear the analytics-only actually-used model snapshot captured by the central session seams.
|
||||
accumulatedStepTokenUsage = this.tokenUsageWithModelSnapshot(accumulatedStepTokenUsage, undefined, previousStepTokenUsage);
|
||||
}
|
||||
tokenUsageRecordedSteps.add(stepIndex);
|
||||
if (!accumulatedStepTokenUsage) {
|
||||
return;
|
||||
@@ -7348,7 +7371,11 @@ export class TaskExecutor {
|
||||
if (!result.tokenUsage || tokenUsageRecordedSteps.has(result.stepIndex)) {
|
||||
continue;
|
||||
}
|
||||
const previousStepTokenUsage = accumulatedStepTokenUsage;
|
||||
accumulatedStepTokenUsage = this.accumulateTokenUsage(accumulatedStepTokenUsage, result.tokenUsage);
|
||||
if (accumulatedStepTokenUsage) {
|
||||
accumulatedStepTokenUsage = this.tokenUsageWithModelSnapshot(accumulatedStepTokenUsage, undefined, previousStepTokenUsage);
|
||||
}
|
||||
}
|
||||
|
||||
if (accumulatedStepTokenUsage) {
|
||||
|
||||
@@ -81,6 +81,7 @@ export async function accumulateSessionTokenUsage(
|
||||
const newCacheWrite = (task.tokenUsage?.cacheWriteTokens ?? 0) + cacheWriteDelta;
|
||||
|
||||
const role = options?.role ?? "executor";
|
||||
const model = (session as { model?: { provider?: string; id?: string } }).model;
|
||||
const tokenUsage = {
|
||||
inputTokens: newInput,
|
||||
outputTokens: newOutput,
|
||||
@@ -89,6 +90,12 @@ export async function accumulateSessionTokenUsage(
|
||||
totalTokens: newInput + newOutput + newCached + newCacheWrite,
|
||||
firstUsedAt: task.tokenUsage?.firstUsedAt ?? now,
|
||||
lastUsedAt: now,
|
||||
/*
|
||||
* FNXC:TokenAnalytics 2026-06-18-16:23:
|
||||
* Token accumulation must snapshot the actually-used session model for by-model analytics without touching task.modelProvider/task.modelId, which would pin future model resolution.
|
||||
*/
|
||||
modelProvider: model?.provider ?? task.tokenUsage?.modelProvider,
|
||||
modelId: model?.id ?? task.tokenUsage?.modelId,
|
||||
};
|
||||
|
||||
cacheMetricsLog.log(JSON.stringify({
|
||||
|
||||
Reference in New Issue
Block a user