From 4b306f10bd11075dacba2d5419c2dfb5b6f990da Mon Sep 17 00:00:00 2001 From: gsxdsm Date: Sat, 1 Aug 2026 04:41:50 -0700 Subject: [PATCH] FN-8689: document Grok CLI source provenance gap Record the unrecoverable Grok CLI provenance chain and preserve the unmeterable usage state. - Document installed asset identity, attempted provenance retrievals, and the static-blocked verdict. - Clarify that the legacy billing request is not verified CLI /usage behavior. - Add a regression test for the provenance finding and credential-safe documentation. Files changed: docs/solutions/integration-issues/grok-cli-usage-data-source.md | 172 +++++++++++---------- packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts | 45 ++++++ packages/dashboard/src/usage.ts | 8 +- 3 files changed, 143 insertions(+), 82 deletions(-) Fusion-Task-Id: FN-8689 Fusion-Task-Lineage: 473fc008-2e08-47f6-8f53-152da5b2c31a Co-authored-by: Fusion (runfusion.ai) --- .../grok-cli-usage-data-source.md | 150 ++++++++++-------- .../__tests__/grok-usage-finding-doc.test.ts | 45 ++++++ packages/dashboard/src/usage.ts | 8 +- 3 files changed, 132 insertions(+), 71 deletions(-) create mode 100644 packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts diff --git a/docs/solutions/integration-issues/grok-cli-usage-data-source.md b/docs/solutions/integration-issues/grok-cli-usage-data-source.md index 95be29173d..1339e66b34 100644 --- a/docs/solutions/integration-issues/grok-cli-usage-data-source.md +++ b/docs/solutions/integration-issues/grok-cli-usage-data-source.md @@ -10,84 +10,96 @@ applies_when: investigating a third-party CLI usage display whose installed rele ## Symptom -Fusion's Grok card was authenticated but empty because `GET -https://cli-chat-proxy.grok.com/v1/billing?format=credits` did not provide -`config.creditUsagePercent`. The installed `grok` command documents `/usage` -(and alias `/cost`) as credit usage, but the request and arithmetic behind that -display were not available from the source trees named in the original -integration evidence. +Fusion's Grok card is authenticated but empty because its existing `GET +https://cli-chat-proxy.grok.com/v1/billing?format=credits` request does not +provide a finite `config.creditUsagePercent`. The installed `grok` command +advertises `/usage` (and alias `/cost`), but that display cannot be used as +evidence until the installed binary is mapped to source. -## Evidence recorded on 2026-08-01 +## Historical FN-8688 evidence -The local command resolves to a standalone `grok-0.2.118-macos-aarch64` Mach-O -binary. It was never executed or treated as source. +FN-8688 replayed Fusion's existing request through a disposable redacted +in-memory harness. It observed HTTP status `200`; `config.creditUsagePercent` +was absent, and the only numeric `*.val` leaves were unclassified. The harness +did not print a response body, credential, string value, or dynamic key. -Two disposable, source-only retrievals were inspected without installing or -executing code: +VERDICT FN-8688: BLOCKED — Fusion's existing request does not identify the CLI request or a meter formula. -1. `npm pack @vibe-kit/grok-cli@0.0.34` supplied package version `0.0.34`. Its - source contains no `/usage`, `/cost`, billing, `creditUsagePercent`, - `productUsage`, or `cli-chat-proxy` reference. -2. A read-only shallow clone of `https://github.com/superagent-ai/grok-cli.git` - at `fb97af83f06dca873281d60168430f06c8de6324` identifies itself as - `grok-dev@1.1.7`. It has no `/usage` or `/cost` slash-command handler and no - Grok billing-endpoint or response-field reference. No upstream tag matching - `0.2.118` was available during the inspection. +## Source provenance (FN-8689) -These are version-skewed from the installed binary and cannot establish its -request, response schema, or display formula. The original package/repository -references therefore do not currently supply auditable source provenance for -this installed CLI release. +### Installed asset identity -## Redacted replay of Fusion's existing request +- Version: `0.2.118` (`~/.grok/version.json`; local `CHANGELOG.md` dates that + version 2026-07-31). +- Asset filename: `grok-0.2.118-macos-aarch64`. +- Installed binary path: `~/.grok/downloads/grok-0.2.118-macos-aarch64`. +- Installed SHA-256: + `2de5b9609a03492dd6b9e4cca9637d651fe998bb8371bf9f852e7b28b38c034e`. -A disposable in-memory harness read the credential locally, made only Fusion's -existing request, parsed the response in memory, sanitized every key segment, -and self-deleted. It printed status `200` and only this allow-listed shape: +### Retrieval method and attempted provenance chain -```text -config.currentPeriod.type: string= -config.currentPeriod.start: string= -config.currentPeriod.end: string= -config.onDemandCap.val: number= -config.onDemandUsed.val: number= -config.: boolean= -config.prepaidBalance.val: number= -config.topUpMethod: string= -config.billingPeriodStart: string= -config.billingPeriodEnd: string= -``` +1. Local installation metadata (`version.json`, `CHANGELOG.md`, downloads, + bundled manifest, README, and the shim target) names the version and asset, + but publishes neither a binary digest nor a source commit. +2. The official installer source at `https://x.ai/cli/install.sh` maps version + `0.2.118` to asset filename `grok-0.2.118-macos-aarch64` and downloads it + from `https://x.ai/cli` or + `https://storage.googleapis.com/grok-build-public-artifacts/cli`. It does + not provide a checksum, an asset manifest, or a source tag/commit mapping. + The official stable channel pointer at the latter base returned `0.2.118`, + which is only a version pointer. +3. Bounded, non-executing binary inspection confirmed the embedded proxy base + URL and candidate build identifier `0.2.118 (1e1687c1cf6a)`. A build + identifier without a published asset/digest mapping is candidate-only, not + source provenance. +4. The public `xai-org-shared/grok-build` repository and release-tag metadata, + the GCS `.sha256`/`.sha256sum` sidecars, and the x.ai `.sha256` sidecar each + returned not found. A read-only remote reference query for that repository + also returned repository-not-found. +5. FN-8688's `@vibe-kit/grok-cli@0.0.34` and + `superagent-ai/grok-cli` commit `fb97af83f06dca873281d60168430f06c8de6324` + remain version-skewed candidate material. They are not the installed + binary's source and were not used to infer a request or formula. -`config.creditUsagePercent` was absent. The generic `*.val` leaves remained -redacted because their names do not prove that they form credits, usage, limits, -or a valid used/total pair. No raw response body, string value, credential, or -dynamic account key was recorded. +No official `version → asset filename → published digest → source tag/commit` +chain is available. Therefore the installed digest cannot be linked to an +inspectable source archive or commit, and the candidate build identifier is not +sufficient provenance. -## Verdict: BLOCKED — actual CLI request and meter derivation unconfirmed +## Source-identified handler, request, and arithmetic -This is not a conclusion that no meter exists. The authenticated replay proves -only that Fusion's current request does not expose the percentage it needs. It -does not prove which request the installed CLI issues, nor support a percentage +Unavailable. Because the provenance chain is unrecoverable, no `/usage` or +`/cost` handler was read as the installed CLI's handler. Consequently this +finding records no source-identified method, URL, query parameters, headers, +response fields, formatter operands, rounding, or clamping. Fusion's existing +billing request is historical context only; it is not asserted to be the CLI's +request. + +## Redacted replay + +Not performed for FN-8689. A parseable local credential was present, but the +redacted harness contract permits live traffic only for a request proven by +provenance-chained source. Replaying Fusion's legacy request again, guessing an +endpoint, or widening numeric output for unproven fields would not meet that +contract. + +## Surface enumeration confirmation + +A repository search for `cli-chat-proxy`, `creditUsagePercent`, and +`format=credits` found the executable request only in +`packages/dashboard/src/usage.ts` and its unit tests. Changelog and changeset +hits are historical text. No other provider, shared UI hook, component, +breakpoint, or affordance is changed. Evidence states remain distinct: +credential absence/unparseability, field absence, non-numeric fields, and +unrecoverable provenance never imply a usage value. + +VERDICT FN-8689: STATIC-BLOCKED — BLOCKED, source provenance unrecoverable. No provenance chain proven; no source-identified request and no live capture performed. + +## Narrow next action + +Request from xAI the release checksum or updater manifest for +`grok-0.2.118-macos-aarch64` that maps its SHA-256 to the source commit or +source archive. Once that chain is supplied, read the mapped handler and replay +only its exact request through the redacted in-process harness. Until then, +FN-8668 may implement nothing: there is no confirmed formatter operand or formula. - -## Blocked pending operator capture - -Before any implementation, obtain source provenance for the installed -`grok-0.2.118-macos-aarch64` release: the exact source archive or repository -commit that built it, plus its `/usage`/`/cost` handler. From that source, -record the exact method, URL, query parameters, non-secret headers, response -fields, and display arithmetic. - -Then an operator must replay that **source-identified exact request** through a -harness equivalent to the one above: read the credential in-process; never -print it or a raw body; print only HTTP status, sanitized structural paths, -allow-listed meter numbers, and redacted non-meter values. Report the -source-read fields and every numeric input to the formatter. Do not use a CLI -screen, `curl`, or an unsanitized proxy capture as evidence. - -## What FN-8668 should implement - -Nothing yet. FN-8668 must remain blocked until the installed CLI's source and -harness-backed response identify meterable fields and an exact formula. In -particular, it must not infer a percentage from missing -`config.creditUsagePercent` or from the unclassified `*.val` fields. diff --git a/packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts b/packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts new file mode 100644 index 0000000000..523869346f --- /dev/null +++ b/packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts @@ -0,0 +1,45 @@ +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; + +const findingPath = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + "../../../../docs/solutions/integration-issues/grok-cli-usage-data-source.md", +); + +const canonicalVerdicts = [ + "VERDICT FN-8689: SOURCE+LIVE — provenance chain proven; source-identified request and formatter arithmetic recorded; numeric formatter inputs confirmed by redacted harness capture.", + "VERDICT FN-8689: SOURCE-ONLY — BLOCKED, live confirmation unavailable. Provenance chain proven and source-identified request/arithmetic recorded; no harness-confirmed numeric inputs.", + "VERDICT FN-8689: STATIC-BLOCKED — BLOCKED, source provenance unrecoverable. No provenance chain proven; no source-identified request and no live capture performed.", +] as const; + +describe("Grok CLI usage source finding", () => { + const finding = readFileSync(findingPath, "utf8"); + + it("keeps the required solution frontmatter and provenance record", () => { + expect(finding).toMatch(/^---\ncategory: integration-issues\nmodule: packages\/dashboard\/src\/usage\.ts\ntags: \[[^\]]+\]\nproblem_type: upstream-provenance-mismatch\napplies_when: .+\n---/); + expect(finding).toContain("## Source provenance (FN-8689)"); + expect(finding).toContain("Retrieval method"); + expect(finding).toContain("0.2.118"); + expect(finding).toContain("Installed SHA-256"); + expect(finding).toMatch(/candidate-only|No official `version → asset filename → published digest → source tag\/commit` chain/i); + expect(finding).toContain("repository-not-found"); + }); + + it("records exactly one canonical FN-8689 verdict", () => { + const verdictLines = finding + .split("\n") + .filter((line) => line.startsWith("VERDICT FN-8689:")); + + expect(verdictLines).toHaveLength(1); + expect(canonicalVerdicts).toContain(verdictLines[0] as (typeof canonicalVerdicts)[number]); + }); + + it("does not commit credential-shaped material", () => { + expect(finding).not.toMatch(/Bearer\s+/); + expect(finding).not.toContain("eyJ"); + expect(finding).not.toMatch(/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/); + expect(finding).not.toMatch(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/i); + }); +}); diff --git a/packages/dashboard/src/usage.ts b/packages/dashboard/src/usage.ts index d6a50f6653..0f23c50e33 100644 --- a/packages/dashboard/src/usage.ts +++ b/packages/dashboard/src/usage.ts @@ -1676,8 +1676,12 @@ async function readGrokUserSettingsApiKey(): Promise { } /* -FNXC:UsageProviders 2026-08-01-11:04: -FN-8688 disproved the assumption that Fusion's credits-format billing request is the installed Grok CLI's `/usage` source. A harness replay returned 200 without `config.creditUsagePercent`, while the available `@vibe-kit/grok-cli@0.0.34` and `grok-dev@1.1.7` source trees cannot be mapped to the installed `grok-0.2.118` binary or its slash-command handler. Keep field absence authenticated-but-unmeterable until source provenance and a harness-backed meter formula are confirmed; see `docs/solutions/integration-issues/grok-cli-usage-data-source.md`. +FNXC:GrokUsage 2026-08-01-11:30: +FN-8689 recovered no provenance chain from the installed Grok 0.2.118 asset to +inspectable source, so this legacy billing request must not be described as the +CLI's verified `/usage` behavior. Keep its credential handling local and emit a +usage window only when this endpoint itself supplies a finite percentage; absent +or unclassified fields remain authenticated but unmeterable. */ async function readGrokCliOidcToken(): Promise { try {