FN-8689: document Grok CLI source provenance gap
Record the unrecoverable Grok CLI provenance chain and preserve the unmeterable usage state. - Document installed asset identity, attempted provenance retrievals, and the static-blocked verdict. - Clarify that the legacy billing request is not verified CLI /usage behavior. - Add a regression test for the provenance finding and credential-safe documentation. Files changed: docs/solutions/integration-issues/grok-cli-usage-data-source.md | 172 +++++++++++---------- packages/dashboard/src/__tests__/grok-usage-finding-doc.test.ts | 45 ++++++ packages/dashboard/src/usage.ts | 8 +- 3 files changed, 143 insertions(+), 82 deletions(-) Fusion-Task-Id: FN-8689 Fusion-Task-Lineage: 473fc008-2e08-47f6-8f53-152da5b2c31a Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -10,84 +10,96 @@ applies_when: investigating a third-party CLI usage display whose installed rele
|
||||
|
||||
## Symptom
|
||||
|
||||
Fusion's Grok card was authenticated but empty because `GET
|
||||
https://cli-chat-proxy.grok.com/v1/billing?format=credits` did not provide
|
||||
`config.creditUsagePercent`. The installed `grok` command documents `/usage`
|
||||
(and alias `/cost`) as credit usage, but the request and arithmetic behind that
|
||||
display were not available from the source trees named in the original
|
||||
integration evidence.
|
||||
Fusion's Grok card is authenticated but empty because its existing `GET
|
||||
https://cli-chat-proxy.grok.com/v1/billing?format=credits` request does not
|
||||
provide a finite `config.creditUsagePercent`. The installed `grok` command
|
||||
advertises `/usage` (and alias `/cost`), but that display cannot be used as
|
||||
evidence until the installed binary is mapped to source.
|
||||
|
||||
## Evidence recorded on 2026-08-01
|
||||
## Historical FN-8688 evidence
|
||||
|
||||
The local command resolves to a standalone `grok-0.2.118-macos-aarch64` Mach-O
|
||||
binary. It was never executed or treated as source.
|
||||
FN-8688 replayed Fusion's existing request through a disposable redacted
|
||||
in-memory harness. It observed HTTP status `200`; `config.creditUsagePercent`
|
||||
was absent, and the only numeric `*.val` leaves were unclassified. The harness
|
||||
did not print a response body, credential, string value, or dynamic key.
|
||||
|
||||
Two disposable, source-only retrievals were inspected without installing or
|
||||
executing code:
|
||||
VERDICT FN-8688: BLOCKED — Fusion's existing request does not identify the CLI request or a meter formula.
|
||||
|
||||
1. `npm pack @vibe-kit/grok-cli@0.0.34` supplied package version `0.0.34`. Its
|
||||
source contains no `/usage`, `/cost`, billing, `creditUsagePercent`,
|
||||
`productUsage`, or `cli-chat-proxy` reference.
|
||||
2. A read-only shallow clone of `https://github.com/superagent-ai/grok-cli.git`
|
||||
at `fb97af83f06dca873281d60168430f06c8de6324` identifies itself as
|
||||
`grok-dev@1.1.7`. It has no `/usage` or `/cost` slash-command handler and no
|
||||
Grok billing-endpoint or response-field reference. No upstream tag matching
|
||||
`0.2.118` was available during the inspection.
|
||||
## Source provenance (FN-8689)
|
||||
|
||||
These are version-skewed from the installed binary and cannot establish its
|
||||
request, response schema, or display formula. The original package/repository
|
||||
references therefore do not currently supply auditable source provenance for
|
||||
this installed CLI release.
|
||||
### Installed asset identity
|
||||
|
||||
## Redacted replay of Fusion's existing request
|
||||
- Version: `0.2.118` (`~/.grok/version.json`; local `CHANGELOG.md` dates that
|
||||
version 2026-07-31).
|
||||
- Asset filename: `grok-0.2.118-macos-aarch64`.
|
||||
- Installed binary path: `~/.grok/downloads/grok-0.2.118-macos-aarch64`.
|
||||
- Installed SHA-256:
|
||||
`2de5b9609a03492dd6b9e4cca9637d651fe998bb8371bf9f852e7b28b38c034e`.
|
||||
|
||||
A disposable in-memory harness read the credential locally, made only Fusion's
|
||||
existing request, parsed the response in memory, sanitized every key segment,
|
||||
and self-deleted. It printed status `200` and only this allow-listed shape:
|
||||
### Retrieval method and attempted provenance chain
|
||||
|
||||
```text
|
||||
config.currentPeriod.type: string=<redacted>
|
||||
config.currentPeriod.start: string=<redacted>
|
||||
config.currentPeriod.end: string=<redacted>
|
||||
config.onDemandCap.val: number=<redacted>
|
||||
config.onDemandUsed.val: number=<redacted>
|
||||
config.<key1>: boolean=<redacted>
|
||||
config.prepaidBalance.val: number=<redacted>
|
||||
config.topUpMethod: string=<redacted>
|
||||
config.billingPeriodStart: string=<redacted>
|
||||
config.billingPeriodEnd: string=<redacted>
|
||||
```
|
||||
1. Local installation metadata (`version.json`, `CHANGELOG.md`, downloads,
|
||||
bundled manifest, README, and the shim target) names the version and asset,
|
||||
but publishes neither a binary digest nor a source commit.
|
||||
2. The official installer source at `https://x.ai/cli/install.sh` maps version
|
||||
`0.2.118` to asset filename `grok-0.2.118-macos-aarch64` and downloads it
|
||||
from `https://x.ai/cli` or
|
||||
`https://storage.googleapis.com/grok-build-public-artifacts/cli`. It does
|
||||
not provide a checksum, an asset manifest, or a source tag/commit mapping.
|
||||
The official stable channel pointer at the latter base returned `0.2.118`,
|
||||
which is only a version pointer.
|
||||
3. Bounded, non-executing binary inspection confirmed the embedded proxy base
|
||||
URL and candidate build identifier `0.2.118 (1e1687c1cf6a)`. A build
|
||||
identifier without a published asset/digest mapping is candidate-only, not
|
||||
source provenance.
|
||||
4. The public `xai-org-shared/grok-build` repository and release-tag metadata,
|
||||
the GCS `.sha256`/`.sha256sum` sidecars, and the x.ai `.sha256` sidecar each
|
||||
returned not found. A read-only remote reference query for that repository
|
||||
also returned repository-not-found.
|
||||
5. FN-8688's `@vibe-kit/grok-cli@0.0.34` and
|
||||
`superagent-ai/grok-cli` commit `fb97af83f06dca873281d60168430f06c8de6324`
|
||||
remain version-skewed candidate material. They are not the installed
|
||||
binary's source and were not used to infer a request or formula.
|
||||
|
||||
`config.creditUsagePercent` was absent. The generic `*.val` leaves remained
|
||||
redacted because their names do not prove that they form credits, usage, limits,
|
||||
or a valid used/total pair. No raw response body, string value, credential, or
|
||||
dynamic account key was recorded.
|
||||
No official `version → asset filename → published digest → source tag/commit`
|
||||
chain is available. Therefore the installed digest cannot be linked to an
|
||||
inspectable source archive or commit, and the candidate build identifier is not
|
||||
sufficient provenance.
|
||||
|
||||
## Verdict: BLOCKED — actual CLI request and meter derivation unconfirmed
|
||||
## Source-identified handler, request, and arithmetic
|
||||
|
||||
This is not a conclusion that no meter exists. The authenticated replay proves
|
||||
only that Fusion's current request does not expose the percentage it needs. It
|
||||
does not prove which request the installed CLI issues, nor support a percentage
|
||||
Unavailable. Because the provenance chain is unrecoverable, no `/usage` or
|
||||
`/cost` handler was read as the installed CLI's handler. Consequently this
|
||||
finding records no source-identified method, URL, query parameters, headers,
|
||||
response fields, formatter operands, rounding, or clamping. Fusion's existing
|
||||
billing request is historical context only; it is not asserted to be the CLI's
|
||||
request.
|
||||
|
||||
## Redacted replay
|
||||
|
||||
Not performed for FN-8689. A parseable local credential was present, but the
|
||||
redacted harness contract permits live traffic only for a request proven by
|
||||
provenance-chained source. Replaying Fusion's legacy request again, guessing an
|
||||
endpoint, or widening numeric output for unproven fields would not meet that
|
||||
contract.
|
||||
|
||||
## Surface enumeration confirmation
|
||||
|
||||
A repository search for `cli-chat-proxy`, `creditUsagePercent`, and
|
||||
`format=credits` found the executable request only in
|
||||
`packages/dashboard/src/usage.ts` and its unit tests. Changelog and changeset
|
||||
hits are historical text. No other provider, shared UI hook, component,
|
||||
breakpoint, or affordance is changed. Evidence states remain distinct:
|
||||
credential absence/unparseability, field absence, non-numeric fields, and
|
||||
unrecoverable provenance never imply a usage value.
|
||||
|
||||
VERDICT FN-8689: STATIC-BLOCKED — BLOCKED, source provenance unrecoverable. No provenance chain proven; no source-identified request and no live capture performed.
|
||||
|
||||
## Narrow next action
|
||||
|
||||
Request from xAI the release checksum or updater manifest for
|
||||
`grok-0.2.118-macos-aarch64` that maps its SHA-256 to the source commit or
|
||||
source archive. Once that chain is supplied, read the mapped handler and replay
|
||||
only its exact request through the redacted in-process harness. Until then,
|
||||
FN-8668 may implement nothing: there is no confirmed formatter operand or
|
||||
formula.
|
||||
|
||||
## Blocked pending operator capture
|
||||
|
||||
Before any implementation, obtain source provenance for the installed
|
||||
`grok-0.2.118-macos-aarch64` release: the exact source archive or repository
|
||||
commit that built it, plus its `/usage`/`/cost` handler. From that source,
|
||||
record the exact method, URL, query parameters, non-secret headers, response
|
||||
fields, and display arithmetic.
|
||||
|
||||
Then an operator must replay that **source-identified exact request** through a
|
||||
harness equivalent to the one above: read the credential in-process; never
|
||||
print it or a raw body; print only HTTP status, sanitized structural paths,
|
||||
allow-listed meter numbers, and redacted non-meter values. Report the
|
||||
source-read fields and every numeric input to the formatter. Do not use a CLI
|
||||
screen, `curl`, or an unsanitized proxy capture as evidence.
|
||||
|
||||
## What FN-8668 should implement
|
||||
|
||||
Nothing yet. FN-8668 must remain blocked until the installed CLI's source and
|
||||
harness-backed response identify meterable fields and an exact formula. In
|
||||
particular, it must not infer a percentage from missing
|
||||
`config.creditUsagePercent` or from the unclassified `*.val` fields.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
const findingPath = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
"../../../../docs/solutions/integration-issues/grok-cli-usage-data-source.md",
|
||||
);
|
||||
|
||||
const canonicalVerdicts = [
|
||||
"VERDICT FN-8689: SOURCE+LIVE — provenance chain proven; source-identified request and formatter arithmetic recorded; numeric formatter inputs confirmed by redacted harness capture.",
|
||||
"VERDICT FN-8689: SOURCE-ONLY — BLOCKED, live confirmation unavailable. Provenance chain proven and source-identified request/arithmetic recorded; no harness-confirmed numeric inputs.",
|
||||
"VERDICT FN-8689: STATIC-BLOCKED — BLOCKED, source provenance unrecoverable. No provenance chain proven; no source-identified request and no live capture performed.",
|
||||
] as const;
|
||||
|
||||
describe("Grok CLI usage source finding", () => {
|
||||
const finding = readFileSync(findingPath, "utf8");
|
||||
|
||||
it("keeps the required solution frontmatter and provenance record", () => {
|
||||
expect(finding).toMatch(/^---\ncategory: integration-issues\nmodule: packages\/dashboard\/src\/usage\.ts\ntags: \[[^\]]+\]\nproblem_type: upstream-provenance-mismatch\napplies_when: .+\n---/);
|
||||
expect(finding).toContain("## Source provenance (FN-8689)");
|
||||
expect(finding).toContain("Retrieval method");
|
||||
expect(finding).toContain("0.2.118");
|
||||
expect(finding).toContain("Installed SHA-256");
|
||||
expect(finding).toMatch(/candidate-only|No official `version → asset filename → published digest → source tag\/commit` chain/i);
|
||||
expect(finding).toContain("repository-not-found");
|
||||
});
|
||||
|
||||
it("records exactly one canonical FN-8689 verdict", () => {
|
||||
const verdictLines = finding
|
||||
.split("\n")
|
||||
.filter((line) => line.startsWith("VERDICT FN-8689:"));
|
||||
|
||||
expect(verdictLines).toHaveLength(1);
|
||||
expect(canonicalVerdicts).toContain(verdictLines[0] as (typeof canonicalVerdicts)[number]);
|
||||
});
|
||||
|
||||
it("does not commit credential-shaped material", () => {
|
||||
expect(finding).not.toMatch(/Bearer\s+/);
|
||||
expect(finding).not.toContain("eyJ");
|
||||
expect(finding).not.toMatch(/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/);
|
||||
expect(finding).not.toMatch(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/i);
|
||||
});
|
||||
});
|
||||
@@ -1676,8 +1676,12 @@ async function readGrokUserSettingsApiKey(): Promise<string | null> {
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:UsageProviders 2026-08-01-11:04:
|
||||
FN-8688 disproved the assumption that Fusion's credits-format billing request is the installed Grok CLI's `/usage` source. A harness replay returned 200 without `config.creditUsagePercent`, while the available `@vibe-kit/grok-cli@0.0.34` and `grok-dev@1.1.7` source trees cannot be mapped to the installed `grok-0.2.118` binary or its slash-command handler. Keep field absence authenticated-but-unmeterable until source provenance and a harness-backed meter formula are confirmed; see `docs/solutions/integration-issues/grok-cli-usage-data-source.md`.
|
||||
FNXC:GrokUsage 2026-08-01-11:30:
|
||||
FN-8689 recovered no provenance chain from the installed Grok 0.2.118 asset to
|
||||
inspectable source, so this legacy billing request must not be described as the
|
||||
CLI's verified `/usage` behavior. Keep its credential handling local and emit a
|
||||
usage window only when this endpoint itself supplies a finite percentage; absent
|
||||
or unclassified fields remain authenticated but unmeterable.
|
||||
*/
|
||||
async function readGrokCliOidcToken(): Promise<string | null> {
|
||||
try {
|
||||
|
||||
Reference in New Issue
Block a user