FN-9031: enforce snapshot consumption after computer actions
Computer-use actions now consume the captured app snapshot and require a fresh state capture before another element-targeted action. - Add app-scoped, serialized snapshot-pointer consumption with stale-state diagnostics. - Mark successful action results as consuming their snapshot and cover action/replay behavior. - Document the enforced snapshot → act → snapshot loop and add a CLI changeset. Files changed: .../fn-9031-computer-use-snapshot-consumption.md | 7 ++ docs/cli-reference.md | 2 +- docs/computer-use.md | 11 +-- .../commands/__tests__/computer-commands.test.ts | 88 ++++++++++++++++++++-- .../commands/__tests__/computer-contract.test.ts | 7 +- .../__tests__/computer-snapshot-index.test.ts | 52 ++++++++++++- .../commands/__tests__/computer-use-guide.test.ts | 4 + packages/cli/src/commands/computer.ts | 37 ++++++--- .../cli/src/commands/computer/adapter-macos.ts | 4 +- packages/cli/src/commands/computer/contract.ts | 17 ++++- packages/cli/src/commands/computer/guide.ts | 4 +- .../cli/src/commands/computer/snapshot-store.ts | 79 +++++++++++++++++-- 12 files changed, 272 insertions(+), 40 deletions(-) Fusion-Task-Id: FN-9031 Fusion-Task-Lineage: f196b7ca-ce3f-4a50-98d2-d6ac8bf184ac Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-9031-computer-use-snapshot-consumption.md
Normal file
7
.changeset/fn-9031-computer-use-snapshot-consumption.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
summary: Require a fresh computer-use snapshot after each action so indexes cannot go stale.
|
||||
category: feature
|
||||
dev: Adds `SNAPSHOT_STALE` reason `consumed-by-action` and `snapshotConsumed` action field.
|
||||
@@ -6,7 +6,7 @@ Fusion’s command-line interface is exposed through the `fn` command.
|
||||
|
||||
## `fn computer` — local desktop automation
|
||||
|
||||
`fn computer` discovers and operates local desktop application windows through a **snapshot → act → snapshot** loop. It is supported on macOS only; other platforms report an honest unsupported capability. Every command supports `--json` and returns the versioned computer-use envelope. See the full [Computer Use reference](./computer-use.md) for setup, permissions, output shapes, snapshot safety, and error handling.
|
||||
`fn computer` discovers and operates local desktop application windows through an enforced **snapshot → act → snapshot** loop: each successful action consumes its capture, so a fresh `get-app-state` is required before the next element action. It is supported on macOS only; other platforms report an honest unsupported capability. Every command supports `--json` and returns the versioned computer-use envelope. See the full [Computer Use reference](./computer-use.md) for setup, permissions, output shapes, snapshot safety, and error handling.
|
||||
|
||||
```bash
|
||||
fn computer capabilities --json
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
[← Docs index](./README.md) · [CLI reference](./cli-reference.md)
|
||||
|
||||
`fn computer` inspects and operates local desktop application windows. It is designed for a safe, repeatable **snapshot → act → snapshot** loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or rendering.
|
||||
`fn computer` inspects and operates local desktop application windows. It enforces a safe, repeatable **snapshot → act → snapshot** loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or re-rendering.
|
||||
|
||||
> **Platform support:** macOS is supported using only operating-system-provided `osascript` and `screencapture`. Linux, Windows, and other platforms are honestly unsupported: `capabilities` and `permissions` succeed with `supported: false`; all other commands fail with `UNSUPPORTED_PLATFORM`. Fusion does not download a helper, native module, or automation dependency.
|
||||
|
||||
@@ -34,10 +34,11 @@ fn computer list-apps --json
|
||||
fn computer get-app-state --app com.apple.Safari --json
|
||||
# Inspect result.snapshot.elements[].index and save result.snapshot.snapshotId.
|
||||
fn computer click --app com.apple.Safari --element-index 42 --snapshot-id cs_01HABCDE123 --json
|
||||
# The successful action consumed that capture; inspect the fresh tree before another action.
|
||||
fn computer get-app-state --app com.apple.Safari --json
|
||||
```
|
||||
|
||||
The first command persists the snapshot before it prints its `snapshotId`. The later `click` may run in a completely separate `fn` process; `--snapshot-id` is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app.
|
||||
The first command persists the snapshot before it prints its `snapshotId`. The later `click` may run in a completely separate `fn` process; `--snapshot-id` is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app. Every successful action consumes the app's latest capture, so the next element action fails with `SNAPSHOT_STALE` / `consumed-by-action` until `get-app-state` captures again.
|
||||
|
||||
Element indexes are snapshot-scoped and sparse. Read indexes from `snapshot.elements[].index`; never derive an index from `elementCount`, position, or bounds. Refresh state after navigation, focus change, scrolling, or re-rendering. Semantic actions (`click`, `set-value`) are preferred over raw keys because they survive focus changes better.
|
||||
|
||||
@@ -109,7 +110,7 @@ Remediation is required for unsupported platform, permission denied/unverified,
|
||||
- `list-apps`: `{ apps }`, sorted by name. An app is `{ bundleId, name, pid }`.
|
||||
- `list-windows`: `{ app, windows }`. A window is `{ windowId, windowIndex, title, bounds, minimized }`.
|
||||
- `get-app-state`: `{ app, window, snapshot, screenshot, screenshotError? }`. `snapshot` has `{ snapshotId, targetKey, windowKey, capturedAt, expiresAt, treeText, elementCount, truncated, elements }`. An element is `{ index, role, title, value, label, enabled, focused, bounds, actions, locator }`; its locator is `{ kind: "ax-path", path, role, subrole, identifier, title }`.
|
||||
- Actions return `{ action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true }` only after the OS action succeeds. Single-endpoint actions use `elementIndex`; element-form drag uses `fromElementIndex` and `toElementIndex` with `elementIndex: null`; hotkey reports all index fields and `snapshotId` as `null`.
|
||||
- Actions return `{ action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true, snapshotConsumed: true }` only after the OS action succeeds. `snapshotConsumed` confirms the app's latest capture was burned and a fresh capture is required before another element action. Single-endpoint actions use `elementIndex`; element-form drag uses `fromElementIndex` and `toElementIndex` with `elementIndex: null`; hotkey reports all index fields and `snapshotId` as `null`.
|
||||
|
||||
Screenshots are always paths, never base64, `data:` URLs, or byte arrays. `--no-screenshot` produces `screenshot: null` with no `screenshotError`; successful capture has `screenshot` and no error; failed/not-captured has `screenshot: null` and `screenshotError`. Screenshot `verifiedPermission` is true only for a preflight-confirmed Screen Recording grant.
|
||||
|
||||
@@ -132,14 +133,14 @@ Screenshots are always paths, never base64, `data:` URLs, or byte arrays. `--no-
|
||||
|
||||
Snapshots are stored under the resolved Fusion project root, not the invoking working directory: `<projectRoot>/.fusion/computer-use/snapshots/<snapshotId>.json`; the app's latest pointer is `<projectRoot>/.fusion/computer-use/latest/<targetKeySlug>.json`. Each `fn computer` invocation resolves that root once by walking upward from its starting directory, and falls back to that resolved directory when no project root is found. Snapshots, pointers, and screenshots use the same root, so they are never shared across projects. Capture atomically persists the record and pointer before returning `snapshotId`.
|
||||
|
||||
A resolved app has app-scoped `targetKey` (`bundle:<bundleId>`, or `pid:<pid>`) and window-scoped `windowKey` (`<targetKey>#<windowId>`). There is one latest pointer per app, not per window. An action with no `--snapshot-id` uses that latest snapshot. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces `SNAPSHOT_STALE` / `window-mismatch`.
|
||||
A resolved app has app-scoped `targetKey` (`bundle:<bundleId>`, or `pid:<pid>`) and window-scoped `windowKey` (`<targetKey>#<windowId>`). There is one latest pointer per app, not per window. An action with no `--snapshot-id` uses that latest snapshot. A successful action consumes that app-scoped pointer rather than deleting its record, so both implicit and matching explicit IDs fail with `SNAPSHOT_STALE` / `consumed-by-action` until a new capture re-arms the pointer. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces `SNAPSHOT_STALE` / `window-mismatch`.
|
||||
|
||||
Snapshots expire after five minutes by default; `expiresAt` publishes the exact deadline. An explicit snapshot ID is a concurrency fence, not a way to revive old UI: a superseded ID fails. Before acting, Fusion re-resolves the recorded window and then the locator rooted in it, verifying role, subrole, and recorded identifier. It never acts on a new occupant of the old path and never falls back to saved bounds/coordinates.
|
||||
|
||||
| Failure | When | Recovery |
|
||||
| --- | --- | --- |
|
||||
| `SNAPSHOT_REQUIRED` | No latest snapshot exists for the target app | Run `get-app-state`. |
|
||||
| `SNAPSHOT_STALE` | `not-found`, `superseded`, `expired`, `pid-changed`, `window-mismatch`, or `window-gone` | Run `get-app-state`; use the current window and snapshot. |
|
||||
| `SNAPSHOT_STALE` | `not-found`, `superseded`, `consumed-by-action`, `expired`, `pid-changed`, `window-mismatch`, or `window-gone` | Run `get-app-state`; use the current window and snapshot. |
|
||||
| `ELEMENT_INDEX_NOT_FOUND` | Index is absent from sparse map | Read the current `elements[].index` after re-snapshotting. |
|
||||
| `ELEMENT_UNRESOLVABLE` | Locator path fails or identity no longer matches | Re-snapshot; do not retry with coordinates. |
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ const adapter: ComputerAdapter = {
|
||||
listApps: async () => ({ apps: [app] }), listWindows: async () => ({ app, windows: [{ windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }] }),
|
||||
captureState: async () => ({ app, window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, snapshot: { snapshotId: "", targetKey: "", windowKey: "", capturedAt: new Date(0).toISOString(), expiresAt: "", treeText: "tree", elementCount: 1, truncated: false, elements: [{ index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator: { kind: "ax-path", path: "button[0]", role: "AXButton", subrole: null, identifier: null, title: "Go" } }] }, screenshot: null }),
|
||||
resolveWindow: async () => ({ window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, handle: "w" }), resolveLocator: async (_w, locator) => ({ element: { index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator }, handle: "e" }),
|
||||
click: async (x) => ({ action: "click", app: x.app, snapshotId: x.snapshotId, elementIndex: 7, fromElementIndex: null, toElementIndex: null, performed: true }), "set-value": async () => { throw new Error("unused"); }, "type-text": async () => { throw new Error("unused"); }, "press-key": async () => { throw new Error("unused"); }, hotkey: async () => { throw new Error("unused"); }, scroll: async () => { throw new Error("unused"); }, drag: async () => { throw new Error("unused"); },
|
||||
click: async (x) => ({ action: "click", app: x.app, snapshotId: x.snapshotId, elementIndex: 7, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }), "set-value": async () => { throw new Error("unused"); }, "type-text": async () => { throw new Error("unused"); }, "press-key": async () => { throw new Error("unused"); }, hotkey: async () => { throw new Error("unused"); }, scroll: async () => { throw new Error("unused"); }, drag: async () => { throw new Error("unused"); },
|
||||
};
|
||||
describe("computer commands", () => {
|
||||
it("emits one JSON envelope and persists a snapshot", async () => { const output: string[] = []; const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-")); try { expect(await runComputer(["get-app-state", "--app", "App", "--no-screenshot", "--json"], { adapter, projectRoot: root, stdout: (x) => output.push(x) })).toBe(0); const envelope = JSON.parse(output[0]); expect(envelope).toMatchObject({ schemaVersion: 1, ok: true, command: "computer.get-app-state" }); expect(envelope.result.snapshot.snapshotId).toMatch(/^cs_/); } finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); } });
|
||||
@@ -70,8 +70,8 @@ describe("computer commands", () => {
|
||||
expect(calls).toEqual(["window", "locator", "click"]);
|
||||
expect(click).toHaveBeenLastCalledWith(expect.objectContaining({ snapshotId: latest.snapshotId, element: expect.objectContaining({ element: expect.objectContaining({ index: 7, locator: snapshotElement(7).locator, bounds: { x: 10, y: 20, width: 30, height: 40 } }) }) }));
|
||||
calls.length = 0;
|
||||
expect(await runComputer(["set-value", "--app", "App", "--element-index", "7", "--value", "safe", "--json"], { adapter: replayAdapter, store, projectRoot: root, stdout: () => undefined })).toBe(0);
|
||||
expect(calls).toEqual(["window", "locator", "set-value"]);
|
||||
expect(await runComputer(["set-value", "--app", "App", "--element-index", "7", "--value", "safe", "--json"], { adapter: replayAdapter, store, projectRoot: root, stdout: () => undefined })).toBe(1);
|
||||
expect(calls).toEqual([]);
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
@@ -96,7 +96,7 @@ describe("computer commands", () => {
|
||||
let typed = 0;
|
||||
const replayAdapter: ComputerAdapter = { ...adapter,
|
||||
resolveLocator: async (_window, locator) => { resolved += 1; return { element: { index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator }, handle: "live" }; },
|
||||
"type-text": async (input) => { typed += 1; return { action: "type-text", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true }; },
|
||||
"type-text": async (input) => { typed += 1; return { action: "type-text", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }; },
|
||||
};
|
||||
try {
|
||||
await runComputer(["get-app-state", "--app", "App", "--no-screenshot", "--json"], { adapter: replayAdapter, projectRoot: root, clock: { now: () => new Date(0) }, stdout: () => undefined });
|
||||
@@ -106,7 +106,7 @@ describe("computer commands", () => {
|
||||
});
|
||||
it("accepts complete coordinate drag without a snapshot", async () => {
|
||||
let input: Parameters<ComputerAdapter["drag"]>[0] | undefined;
|
||||
const dragAdapter: ComputerAdapter = { ...adapter, drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true }; } };
|
||||
const dragAdapter: ComputerAdapter = { ...adapter, drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }; } };
|
||||
expect(await runComputer(["drag", "--app", "App", "--from-x", "1", "--from-y", "2", "--to-x", "3", "--to-y", "4", "--json"], { adapter: dragAdapter, stdout: () => undefined })).toBe(0);
|
||||
expect(input).toMatchObject({ snapshotId: null, fromX: 1, fromY: 2, toX: 3, toY: 4 });
|
||||
});
|
||||
@@ -118,13 +118,85 @@ describe("computer commands", () => {
|
||||
let input: Parameters<ComputerAdapter["drag"]>[0] | undefined;
|
||||
const dragAdapter: ComputerAdapter = { ...adapter,
|
||||
resolveLocator: async (_window, locator) => ({ element: locator.path === from.locator.path ? from : to, handle: locator.path }),
|
||||
drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: value.snapshotId, elementIndex: null, fromElementIndex: value.from?.element.index ?? null, toElementIndex: value.to?.element.index ?? null, performed: true }; },
|
||||
drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: value.snapshotId, elementIndex: null, fromElementIndex: value.from?.element.index ?? null, toElementIndex: value.to?.element.index ?? null, performed: true, snapshotConsumed: false }; },
|
||||
};
|
||||
const store = { resolve, getElement: (_record: typeof record, index: number) => _record.elements[String(index)] } as unknown as import("../computer/snapshot-store.js").ComputerSnapshotStore;
|
||||
const store = { resolve, consume: vi.fn(), getElement: (_record: typeof record, index: number) => _record.elements[String(index)] } as unknown as import("../computer/snapshot-store.js").ComputerSnapshotStore;
|
||||
expect(await runComputer(["drag", "--app", "App", "--from-element-index", "7", "--to-element-index", "9", "--json"], { adapter: dragAdapter, store, stdout: () => undefined })).toBe(0);
|
||||
expect(resolve).toHaveBeenCalledTimes(1);
|
||||
expect(input).toMatchObject({ snapshotId: record.snapshotId, from: { element: { index: 7 } }, to: { element: { index: 9 } } });
|
||||
});
|
||||
it("enforces snapshot → act → snapshot and re-arms after a fresh capture", async () => {
|
||||
const output: string[] = [];
|
||||
const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-"));
|
||||
const clock = { now: () => new Date(0) };
|
||||
const run = async (args: string[]) => {
|
||||
output.length = 0;
|
||||
const code = await runComputer([...args, "--json"], { adapter, projectRoot: root, clock, stdout: (text) => output.push(text) });
|
||||
return { code, envelope: JSON.parse(output[0]!) };
|
||||
};
|
||||
try {
|
||||
await run(["get-app-state", "--app", "App", "--no-screenshot"]);
|
||||
expect((await run(["click", "--app", "App", "--element-index", "7"])).envelope.result).toMatchObject({ snapshotConsumed: true });
|
||||
const stale = await run(["click", "--app", "App", "--element-index", "7"]);
|
||||
expect(stale.code).toBe(1);
|
||||
expect(stale.envelope).toMatchObject({ ok: false, error: { code: "SNAPSHOT_STALE", details: { reason: "consumed-by-action" } } });
|
||||
expect(stale.envelope.error.remediation).toContain("fn computer get-app-state");
|
||||
await run(["get-app-state", "--app", "App", "--no-screenshot"]);
|
||||
expect((await run(["click", "--app", "App", "--element-index", "7"])).code).toBe(0);
|
||||
} finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("consumes a captured app pointer after every action form", async () => {
|
||||
const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-"));
|
||||
const clock = { now: () => new Date(0) };
|
||||
const action = (name: string, input: { app: typeof app; snapshotId?: string | null; element?: { element: { index: number } }; from?: { element: { index: number } }; to?: { element: { index: number } } }) => ({ action: name, app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: input.from?.element.index ?? null, toElementIndex: input.to?.element.index ?? null, performed: true as const, snapshotConsumed: false });
|
||||
const actions: ComputerAdapter = { ...adapter,
|
||||
click: async (input) => action("click", input), "set-value": async (input) => action("set-value", input), "type-text": async (input) => action("type-text", input), "press-key": async (input) => action("press-key", input), hotkey: async (input) => action("hotkey", input), scroll: async (input) => action("scroll", input), drag: async (input) => action("drag", input),
|
||||
};
|
||||
const cases = [
|
||||
["click", "--app", "App", "--element-index", "7"], ["set-value", "--app", "App", "--element-index", "7", "--value", "x"],
|
||||
["type-text", "--app", "App", "--element-index", "7", "--text", "x"], ["press-key", "--app", "App", "--element-index", "7", "--key", "enter"],
|
||||
["scroll", "--app", "App", "--element-index", "7", "--direction", "down"], ["hotkey", "--app", "App", "--keys", "cmd+k"],
|
||||
["drag", "--app", "App", "--from-x", "1", "--from-y", "2", "--to-x", "3", "--to-y", "4"],
|
||||
["drag", "--app", "App", "--from-element-index", "7", "--to-element-index", "7"],
|
||||
];
|
||||
try {
|
||||
for (const args of cases) {
|
||||
const output: string[] = [];
|
||||
await runComputer(["get-app-state", "--app", "App", "--no-screenshot", "--json"], { adapter: actions, projectRoot: root, clock, stdout: () => undefined });
|
||||
expect(await runComputer([...args, "--json"], { adapter: actions, projectRoot: root, clock, stdout: (text) => output.push(text) })).toBe(0);
|
||||
expect(JSON.parse(output[0]!).result.snapshotConsumed).toBe(true);
|
||||
output.length = 0;
|
||||
expect(await runComputer(["click", "--app", "App", "--element-index", "7", "--json"], { adapter: actions, projectRoot: root, clock, stdout: (text) => output.push(text) })).toBe(1);
|
||||
expect(JSON.parse(output[0]!).error.details.reason).toBe("consumed-by-action");
|
||||
}
|
||||
} finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("consumes after targetless actions but not after failed actions or reads", async () => {
|
||||
const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-"));
|
||||
const clock = { now: () => new Date(0) };
|
||||
let failClick = true;
|
||||
const resilientAdapter: ComputerAdapter = { ...adapter,
|
||||
click: async (input) => { if (failClick) { failClick = false; throw new (await import("../computer/contract.js")).ComputerUseError("ACTION_FAILED", "failed"); } return { action: "click", app: input.app, snapshotId: input.snapshotId, elementIndex: 7, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }; },
|
||||
"type-text": async (input) => ({ action: "type-text", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }),
|
||||
"press-key": async (input) => ({ action: "press-key", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }),
|
||||
scroll: async (input) => ({ action: "scroll", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }),
|
||||
};
|
||||
const run = (args: string[]) => runComputer([...args, "--json"], { adapter: resilientAdapter, projectRoot: root, clock, stdout: () => undefined });
|
||||
try {
|
||||
await run(["get-app-state", "--app", "App", "--no-screenshot"]);
|
||||
await run(["list-apps"]); await run(["list-windows", "--app", "App"]);
|
||||
expect(await run(["click", "--app", "App", "--element-index", "7"])).toBe(1);
|
||||
expect(await run(["click", "--app", "App", "--element-index", "7"])).toBe(0);
|
||||
for (const args of [["type-text", "--app", "App", "--text", "x"], ["press-key", "--app", "App", "--key", "enter"], ["scroll", "--app", "App", "--direction", "down"]]) {
|
||||
await run(["get-app-state", "--app", "App", "--no-screenshot"]);
|
||||
expect(await run(args)).toBe(0);
|
||||
expect(await run(["click", "--app", "App", "--element-index", "7"])).toBe(1);
|
||||
}
|
||||
} finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("uses group-level INVALID_ARGUMENTS for unknown commands", async () => { const output: string[] = []; expect(await runComputer(["nope", "--json"], { adapter, stdout: (x) => output.push(x) })).toBe(1); expect(JSON.parse(output[0])).toMatchObject({ command: "computer", error: { code: "INVALID_ARGUMENTS" } }); });
|
||||
it("returns the required JSON envelope for a missing subcommand", async () => {
|
||||
const output: string[] = [];
|
||||
@@ -141,7 +213,7 @@ describe("computer commands", () => {
|
||||
it("falls back from an unmatched dotted bundle spelling to an exact app name", async () => {
|
||||
const dotted = { ...app, bundleId: "com.example.Other", name: "Foo.Bar" };
|
||||
const output: string[] = [];
|
||||
expect(await runComputer(["hotkey", "--app", "Foo.Bar", "--keys", "cmd+k", "--json"], { adapter: { ...adapter, listApps: async () => ({ apps: [dotted] }), hotkey: async (input) => ({ action: "hotkey", app: input.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true }) }, stdout: (text) => output.push(text) })).toBe(0);
|
||||
expect(await runComputer(["hotkey", "--app", "Foo.Bar", "--keys", "cmd+k", "--json"], { adapter: { ...adapter, listApps: async () => ({ apps: [dotted] }), hotkey: async (input) => ({ action: "hotkey", app: input.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }) }, stdout: (text) => output.push(text) })).toBe(0);
|
||||
expect(JSON.parse(output[0]!)).toMatchObject({ result: { app: { name: "Foo.Bar" } } });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
COMPUTER_ACTIONS, COMPUTER_ERROR_CODES, COMPUTER_SUBCOMMANDS, COMPUTER_TIMEOUTS, SNAPSHOT_STALE_REASONS,
|
||||
failureEnvelope, isActionResult, isAppStateResult, isPermissionsResult, isValidSnapshotId, secretValue,
|
||||
failureEnvelope, isActionResult, isAppStateResult, isPermissionsResult, isValidSnapshotId, secretValue, validateResult,
|
||||
successEnvelope, targetKeyForApp, targetKeySlug, windowKeyFor,
|
||||
} from "../computer/contract.js";
|
||||
|
||||
@@ -18,11 +18,16 @@ describe("computer contract", () => {
|
||||
});
|
||||
it("validates ids, permissions, state and action index shapes", () => {
|
||||
expect(isValidSnapshotId("cs_AbCdEf1234")).toBe(true); expect(isValidSnapshotId("cs_../evil")).toBe(false);
|
||||
expect(SNAPSHOT_STALE_REASONS).toEqual(["not-found", "superseded", "expired", "pid-changed", "window-mismatch", "window-gone", "consumed-by-action"]);
|
||||
expect(isPermissionsResult({ allGranted: false, checks: [{ status: "granted", granted: true, probed: false }] })).toBe(false);
|
||||
const state = { snapshot: { elements: [{ index: 7, locator }] }, screenshot: { path: "x", width: null, height: null, verifiedPermission: true }, screenshotError: { code: "SCREENSHOT_FAILED", message: "no" } };
|
||||
expect(isAppStateResult(state)).toBe(false);
|
||||
expect(isActionResult({ action: "drag", performed: true, snapshotId: null, elementIndex: 1, fromElementIndex: 0, toElementIndex: 1 })).toBe(false);
|
||||
expect(isActionResult({ action: "hotkey", performed: true, snapshotId: "cs_AbCdEf1234", elementIndex: null, fromElementIndex: null, toElementIndex: null })).toBe(false);
|
||||
const actionWithoutConsumption = { action: "click", performed: true, snapshotId: "cs_AbCdEf1234", elementIndex: 7, fromElementIndex: null, toElementIndex: null };
|
||||
expect(isActionResult(actionWithoutConsumption)).toBe(false);
|
||||
expect(validateResult("click", actionWithoutConsumption)).toBe(false);
|
||||
expect(validateResult("click", { ...actionWithoutConsumption, snapshotConsumed: true })).toBe(true);
|
||||
});
|
||||
it("keeps app and window addressing separate", () => {
|
||||
const app = { bundleId: "com.apple.Safari", name: "Safari", pid: 2 };
|
||||
|
||||
@@ -2,7 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { ComputerUseError, type AppRef, type Element, type WindowRef } from "../computer/contract.js";
|
||||
import { ComputerUseError, targetKeyForApp, targetKeySlug, type AppRef, type Element, type WindowRef } from "../computer/contract.js";
|
||||
import { ComputerSnapshotStore } from "../computer/snapshot-store.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
@@ -84,6 +84,56 @@ describe("ComputerSnapshotStore", () => {
|
||||
await expect(live.store.resolve({ app, snapshotId: record.snapshotId, assertedWindowId: "other-window" })).rejects.toMatchObject({ details: { reason: "window-mismatch", snapshotId: record.snapshotId } });
|
||||
});
|
||||
|
||||
it("consumes the latest pointer until a fresh capture re-arms it", async () => {
|
||||
const { store } = await fixture();
|
||||
await expect(store.consume(app)).resolves.toBeUndefined();
|
||||
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
await store.consume(app);
|
||||
await expect(store.resolve({ app })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "consumed-by-action", snapshotId: first.snapshotId } });
|
||||
await expect(store.resolve({ app, snapshotId: first.snapshotId })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "consumed-by-action", snapshotId: first.snapshotId } });
|
||||
|
||||
const second = await store.persist({ app, window, elementCount: 1, elements: [element(8)] });
|
||||
await expect(store.resolve({ app, snapshotId: first.snapshotId })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "superseded", snapshotId: first.snapshotId } });
|
||||
await expect(store.resolve({ app, snapshotId: second.snapshotId })).resolves.toMatchObject({ snapshotId: second.snapshotId });
|
||||
});
|
||||
|
||||
it("preserves a re-capture when consumption completes before the new pointer write", async () => {
|
||||
const { store } = await fixture();
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
await store.consume(app, first.snapshotId);
|
||||
const second = await store.persist({ app, window, elementCount: 1, elements: [element(8)] });
|
||||
|
||||
await expect(store.resolve({ app, snapshotId: second.snapshotId })).resolves.toMatchObject({ snapshotId: second.snapshotId });
|
||||
});
|
||||
|
||||
it("does not let an earlier action consume a newer re-capture", async () => {
|
||||
const { store } = await fixture();
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
const second = await store.persist({ app, window, elementCount: 1, elements: [element(8)] });
|
||||
await store.consume(app, first.snapshotId);
|
||||
|
||||
await expect(store.resolve({ app, snapshotId: second.snapshotId })).resolves.toMatchObject({ snapshotId: second.snapshotId });
|
||||
});
|
||||
|
||||
it("serializes concurrent consumption and re-capture without losing the re-arm", async () => {
|
||||
const { store } = await fixture();
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
const recapture = store.persist({ app, window, elementCount: 1, elements: [element(8)] });
|
||||
const consumption = store.consume(app, first.snapshotId);
|
||||
const [second] = await Promise.all([recapture, consumption]);
|
||||
|
||||
await expect(store.resolve({ app, snapshotId: second.snapshotId })).resolves.toMatchObject({ snapshotId: second.snapshotId });
|
||||
});
|
||||
|
||||
it("keeps legacy pointers without consumedAt usable", async () => {
|
||||
const { store, root } = await fixture();
|
||||
const record = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
const pointerPath = join(root, ".fusion", "computer-use", "latest", `${targetKeySlug(targetKeyForApp(app))}.json`);
|
||||
await writeFile(pointerPath, `${JSON.stringify({ snapshotId: record.snapshotId })}\n`);
|
||||
await expect(store.resolve({ app })).resolves.toMatchObject({ snapshotId: record.snapshotId });
|
||||
});
|
||||
|
||||
it("treats absent or unparsable named records as not-found", async () => {
|
||||
const { store, root } = await fixture();
|
||||
const record = await store.persist({ app, window, elementCount: 1, elements: [element(0)] });
|
||||
|
||||
@@ -20,6 +20,10 @@ describe("computer-use guide", () => {
|
||||
expect(guide).toMatch(/### fn computer press-key[\s\S]*?Rules:\n- --snapshot-id and window flags require --element-index\./);
|
||||
expect(guide).toContain("Choose exactly one form: all four coordinate flags, or both element-index flags.");
|
||||
expect(guide).toContain("Coordinate drag takes no --snapshot-id or window flags.");
|
||||
for (const trigger of ["focus changes", "navigation", "scrolling", "re-rendering"]) expect(guide).toContain(trigger);
|
||||
expect(guide).toContain("elementCount");
|
||||
expect(guide).toContain("consumed-by-action");
|
||||
expect(guide).toContain("snapshotConsumed");
|
||||
});
|
||||
|
||||
it("is the complete rendering link after descriptor anchors establish the live surface", () => {
|
||||
|
||||
@@ -4,7 +4,7 @@ import { resolveComputerAdapter, type ComputerClock } from "./computer/adapter-r
|
||||
import type { ComputerAdapter, ResolvedComputerElement, ResolvedComputerWindow } from "./computer/adapter.js";
|
||||
import { createComputerSnapshotStore, type ComputerSnapshotStore } from "./computer/snapshot-store.js";
|
||||
import { resolveComputerStateRoot } from "./computer/state-root.js";
|
||||
import { COMPUTER_COMMAND_SURFACE, COMPUTER_SUBCOMMANDS, ComputerUseError, failureEnvelope, isValidSnapshotId, parseAppTarget, successEnvelope, validateResult, type AppRef, type CommandName, type ComputerSubcommand } from "./computer/contract.js";
|
||||
import { COMPUTER_COMMAND_SURFACE, COMPUTER_SUBCOMMANDS, ComputerUseError, failureEnvelope, isValidSnapshotId, parseAppTarget, successEnvelope, validateResult, type ActionResult, type AppRef, type CommandName, type ComputerSubcommand } from "./computer/contract.js";
|
||||
|
||||
export interface ComputerCommandOptions { platform?: string; projectRoot?: string; adapter?: ComputerAdapter; store?: ComputerSnapshotStore; clock?: ComputerClock; stdout?: (text: string) => void; stderr?: (text: string) => void; stdin?: () => Promise<string>; }
|
||||
export type ComputerHandler = (args: string[], options: ComputerCommandOptions) => Promise<unknown>;
|
||||
@@ -53,15 +53,27 @@ async function requireElement(args: string[], adapter: ComputerAdapter, store: C
|
||||
return { record: resolved.record, window: resolved.window, element: resolved.elements[0]! };
|
||||
}
|
||||
async function optionalElement(args: string[], adapter: ComputerAdapter, store: ComputerSnapshotStore, app: AppRef) { return value(args, "--element-index") === undefined ? undefined : requireElement(args, adapter, store, app); }
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-13-22:02:
|
||||
* This is the sole action completion seam: only a successful adapter call may burn the app fence.
|
||||
* Requiring a new capture prevents a second sparse index from silently targeting a different element
|
||||
* after the first action changed focus, navigation, scrolling, or the rendered accessibility tree.
|
||||
*/
|
||||
async function finishAction(result: ActionResult, store: ComputerSnapshotStore, app: AppRef): Promise<ActionResult> {
|
||||
await store.consume(app, result.snapshotId ?? undefined);
|
||||
return { ...result, snapshotConsumed: true };
|
||||
}
|
||||
|
||||
export const COMPUTER_HANDLERS: Record<ComputerSubcommand, ComputerHandler> = {
|
||||
capabilities: async (_args, o) => adapterFor(o).capabilities(), permissions: async (_args, o) => adapterFor(o).permissions(),
|
||||
"list-apps": async (_args, o) => adapterFor(o).listApps(),
|
||||
"list-windows": async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); const adapter = adapterFor(o); return adapter.listWindows(parseAppTarget(raw)); },
|
||||
"get-app-state": async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); if (value(args, "--window-id") && value(args, "--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Window flags are mutually exclusive."); const adapter = adapterFor(o); const state = await adapter.captureState(parseAppTarget(raw), { windowId: value(args, "--window-id"), windowIndex: number(args, "--window-index"), screenshot: !args.includes("--no-screenshot"), restoreWindow: args.includes("--restore-window") }); const record = await storeFor(o).persist({ app: state.app, window: state.window, elementCount: state.snapshot.elementCount, elements: state.snapshot.elements, capturedAt: state.snapshot.capturedAt }); state.snapshot.snapshotId = record.snapshotId; state.snapshot.targetKey = record.targetKey; state.snapshot.windowKey = record.windowKey; state.snapshot.capturedAt = record.capturedAt; state.snapshot.expiresAt = record.expiresAt; return state; },
|
||||
click: async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await requireElement(args, adapter, storeFor(o), app); return adapter.click({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId }); },
|
||||
"set-value": async (args, o) => { const raw = value(args, "--app"), text = value(args, "--value"); if (!raw || (!text && !args.includes("--value-stdin")) || (text && args.includes("--value-stdin"))) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and exactly one value source are required."); const secret = args.includes("--value-stdin") ? await (o.stdin ?? (async () => ""))() : text!; const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await requireElement(args, adapter, storeFor(o), app); return adapter["set-value"]({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId, value: secret }); },
|
||||
click: async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); const adapter = adapterFor(o), app = await appFor(adapter, raw), store = storeFor(o), item = await requireElement(args, adapter, store, app); return finishAction(await adapter.click({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId }), store, app); },
|
||||
"set-value": async (args, o) => { const raw = value(args, "--app"), text = value(args, "--value"); if (!raw || (!text && !args.includes("--value-stdin")) || (text && args.includes("--value-stdin"))) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and exactly one value source are required."); const secret = args.includes("--value-stdin") ? await (o.stdin ?? (async () => ""))() : text!; const adapter = adapterFor(o), app = await appFor(adapter, raw), store = storeFor(o), item = await requireElement(args, adapter, store, app); return finishAction(await adapter["set-value"]({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId, value: secret }), store, app); },
|
||||
"type-text": async (args, o) => targetedOrUntargeted("type-text", args, o), "press-key": async (args, o) => targetedOrUntargeted("press-key", args, o), scroll: async (args, o) => targetedOrUntargeted("scroll", args, o),
|
||||
hotkey: async (args, o) => { const raw = value(args, "--app"), keys = value(args, "--keys"); if (!raw || !keys) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and --keys are required."); const adapter = adapterFor(o), app = await appFor(adapter, raw); return adapter.hotkey({ app, keys: keys.split("+") }); },
|
||||
hotkey: async (args, o) => { const raw = value(args, "--app"), keys = value(args, "--keys"); if (!raw || !keys) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and --keys are required."); const adapter = adapterFor(o), app = await appFor(adapter, raw), store = storeFor(o); return finishAction(await adapter.hotkey({ app, keys: keys.split("+") }), store, app); },
|
||||
drag: async (args, o) => {
|
||||
const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required.");
|
||||
const coordinateFlags = ["--from-x", "--from-y", "--to-x", "--to-y"];
|
||||
@@ -73,12 +85,13 @@ export const COMPUTER_HANDLERS: Record<ComputerSubcommand, ComputerHandler> = {
|
||||
if (hasCoordinates) {
|
||||
const coordinates = coordinateFlags.map((flag) => number(args, flag));
|
||||
if (coordinates.some((item) => item === undefined) || value(args, "--snapshot-id") || value(args, "--window-id") || value(args, "--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate drag requires all coordinates and takes no snapshot or window flags.");
|
||||
return adapter.drag({ app, snapshotId: null, fromX: coordinates[0]!, fromY: coordinates[1]!, toX: coordinates[2]!, toY: coordinates[3]! });
|
||||
return finishAction(await adapter.drag({ app, snapshotId: null, fromX: coordinates[0]!, fromY: coordinates[1]!, toX: coordinates[2]!, toY: coordinates[3]! }), storeFor(o), app);
|
||||
}
|
||||
if (from === undefined || to === undefined) throw new ComputerUseError("INVALID_ARGUMENTS", "Drag requires either all coordinates or both element indexes.");
|
||||
// Both endpoints share one resolved record and one replayed window, even if another capture updates latest mid-action.
|
||||
const resolved = await requireElements(args, [from, to], adapter, storeFor(o), app);
|
||||
return adapter.drag({ app, snapshotId: resolved.record.snapshotId, window: resolved.window, from: resolved.elements[0]!, to: resolved.elements[1]! });
|
||||
const store = storeFor(o);
|
||||
const resolved = await requireElements(args, [from, to], adapter, store, app);
|
||||
return finishAction(await adapter.drag({ app, snapshotId: resolved.record.snapshotId, window: resolved.window, from: resolved.elements[0]!, to: resolved.elements[1]! }), store, app);
|
||||
},
|
||||
};
|
||||
function validateFlags(name: ComputerSubcommand, args: string[]): void {
|
||||
@@ -144,19 +157,19 @@ function validateFlags(name: ComputerSubcommand, args: string[]): void {
|
||||
}
|
||||
}
|
||||
|
||||
async function targetedOrUntargeted(kind: "type-text" | "press-key" | "scroll", args: string[], o: ComputerCommandOptions): Promise<unknown> {
|
||||
async function targetedOrUntargeted(kind: "type-text" | "press-key" | "scroll", args: string[], o: ComputerCommandOptions): Promise<ActionResult> {
|
||||
const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required.");
|
||||
const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await optionalElement(args, adapter, storeFor(o), app);
|
||||
const adapter = adapterFor(o), app = await appFor(adapter, raw), store = storeFor(o), item = await optionalElement(args, adapter, store, app);
|
||||
if (!item && (value(args, "--snapshot-id") || value(args, "--window-id") || value(args, "--window-index"))) throw new ComputerUseError("INVALID_ARGUMENTS", "Snapshot and window flags require --element-index.");
|
||||
if (kind === "type-text") {
|
||||
const direct = value(args, "--text"), fromStdin = args.includes("--text-stdin");
|
||||
if ((direct === undefined && !fromStdin) || (direct !== undefined && fromStdin)) throw new ComputerUseError("INVALID_ARGUMENTS", "Exactly one text source is required.");
|
||||
const text = fromStdin ? await (o.stdin ?? (async () => ""))() : direct!;
|
||||
return adapter["type-text"]({ app, text, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) });
|
||||
return finishAction(await adapter["type-text"]({ app, text, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) }), store, app);
|
||||
}
|
||||
if (kind === "press-key") { const key = value(args, "--key"); if (!key) throw new ComputerUseError("INVALID_ARGUMENTS", "--key is required."); return adapter["press-key"]({ app, key, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) }); }
|
||||
if (kind === "press-key") { const key = value(args, "--key"); if (!key) throw new ComputerUseError("INVALID_ARGUMENTS", "--key is required."); return finishAction(await adapter["press-key"]({ app, key, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) }), store, app); }
|
||||
const direction = value(args, "--direction"); if (!direction || !["up", "down", "left", "right"].includes(direction)) throw new ComputerUseError("INVALID_ARGUMENTS", "A valid --direction is required.");
|
||||
return adapter.scroll({ app, direction: direction as "up" | "down" | "left" | "right", amount: number(args, "--amount") ?? 3, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) });
|
||||
return finishAction(await adapter.scroll({ app, direction: direction as "up" | "down" | "left" | "right", amount: number(args, "--amount") ?? 3, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) }), store, app);
|
||||
}
|
||||
export async function runComputer(args: string[], options: ComputerCommandOptions = {}): Promise<number> {
|
||||
const json = args.includes("--json");
|
||||
|
||||
@@ -155,7 +155,7 @@ export class MacosComputerAdapter implements ComputerAdapter {
|
||||
await this.callJson(["drag-coordinates", input.app.name, String(from.x), String(from.y), String(to.x), String(to.y)], COMPUTER_TIMEOUTS.action);
|
||||
} else if (input.fromX !== undefined && input.fromY !== undefined && input.toX !== undefined && input.toY !== undefined) { await this.callJson(["drag-coordinates", input.app.name, String(input.fromX), String(input.fromY), String(input.toX), String(input.toY)], COMPUTER_TIMEOUTS.action); }
|
||||
else throw new ComputerUseError("ACTION_FAILED", "Drag requires coordinates or resolved elements.");
|
||||
return { action: "drag", app: input.app, snapshotId: input.snapshotId, elementIndex: null, fromElementIndex: input.from?.element.index ?? null, toElementIndex: input.to?.element.index ?? null, performed: true };
|
||||
return { action: "drag", app: input.app, snapshotId: input.snapshotId, elementIndex: null, fromElementIndex: input.from?.element.index ?? null, toElementIndex: input.to?.element.index ?? null, performed: true, snapshotConsumed: false };
|
||||
}
|
||||
|
||||
private async assertAccessibility(): Promise<void> {
|
||||
@@ -208,4 +208,4 @@ function isWindowRef(value: unknown): value is ResolvedComputerWindow["window"]
|
||||
function isElement(value: unknown): value is Element { const element = value as Element; return isLiveElement(value) && typeof element.index === "number"; }
|
||||
function isLiveElement(value: unknown): value is Omit<Element, "index"> { const element = value as Element; return !!element && typeof element.role === "string" && !!element.locator && typeof element.locator.path === "string"; }
|
||||
function center(bounds: Element["bounds"]): { x: number; y: number } | undefined { return bounds ? { x: bounds.x + bounds.width / 2, y: bounds.y + bounds.height / 2 } : undefined; }
|
||||
function singleAction(action: string, app: AppRef, snapshotId: string | null, elementIndex: number | null): ActionResult { return { action, app, snapshotId: action === "hotkey" ? null : snapshotId, elementIndex, fromElementIndex: null, toElementIndex: null, performed: true }; }
|
||||
function singleAction(action: string, app: AppRef, snapshotId: string | null, elementIndex: number | null): ActionResult { return { action, app, snapshotId: action === "hotkey" ? null : snapshotId, elementIndex, fromElementIndex: null, toElementIndex: null, performed: true, snapshotConsumed: false }; }
|
||||
|
||||
@@ -73,7 +73,13 @@ export const COMPUTER_COMMAND_SURFACE = Object.freeze({
|
||||
drag: { description: "Drag between coordinates or two captured elements.", flags: [{ flag: "--app", valueKind: "string", required: true, description: "Application target." }, { flag: "--from-x", valueKind: "integer", required: false, description: "Starting x coordinate." }, { flag: "--from-y", valueKind: "integer", required: false, description: "Starting y coordinate." }, { flag: "--to-x", valueKind: "integer", required: false, description: "Ending x coordinate." }, { flag: "--to-y", valueKind: "integer", required: false, description: "Ending y coordinate." }, { flag: "--from-element-index", valueKind: "integer", required: false, description: "Starting element index." }, { flag: "--to-element-index", valueKind: "integer", required: false, description: "Ending element index." }, { flag: "--snapshot-id", valueKind: "string", required: false, description: "Snapshot fence for element drag." }, { flag: "--window-id", valueKind: "string", required: false, mutuallyExclusiveWith: "--window-index", description: "Window identifier." }, { flag: "--window-index", valueKind: "integer", required: false, mutuallyExclusiveWith: "--window-id", description: "Window position." }], requirements: ["Choose exactly one form: all four coordinate flags, or both element-index flags.", "Coordinate drag takes no --snapshot-id or window flags."] },
|
||||
} as const satisfies Record<ComputerSubcommand, ComputerCommandSurfaceEntry>);
|
||||
export type CommandName = `computer.${ComputerSubcommand}` | "computer";
|
||||
export const SNAPSHOT_STALE_REASONS = Object.freeze(["not-found", "superseded", "expired", "pid-changed", "window-mismatch", "window-gone"] as const);
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-13-22:02:
|
||||
* A successful action can re-render or refocus the window, so the capture that supplied sparse
|
||||
* element indexes is no longer authoritative. Keep this append-only reason so stale replays fail
|
||||
* closed and callers must capture again rather than risk acting on a different element.
|
||||
*/
|
||||
export const SNAPSHOT_STALE_REASONS = Object.freeze(["not-found", "superseded", "expired", "pid-changed", "window-mismatch", "window-gone", "consumed-by-action"] as const);
|
||||
export type SnapshotStaleReason = (typeof SNAPSHOT_STALE_REASONS)[number];
|
||||
export const COMPUTER_TIMEOUTS = Object.freeze({ permissionProbe: 5_000, discovery: 10_000, stateCapture: 20_000, screenshotCapture: 15_000, locatorReplay: 10_000, action: 10_000 });
|
||||
|
||||
@@ -109,7 +115,12 @@ export interface ListAppsResult { apps: AppRef[]; }
|
||||
export interface ListWindowsResult { app: AppRef; windows: WindowRef[]; }
|
||||
export interface Screenshot { path: string; width: number | null; height: number | null; verifiedPermission: boolean; }
|
||||
export interface AppStateResult { app: AppRef; window: WindowRef; snapshot: { snapshotId: string; targetKey: string; windowKey: string; capturedAt: string; expiresAt: string; treeText: string; elementCount: number; truncated: boolean; elements: Element[] }; screenshot: Screenshot | null; screenshotError?: { code: "SCREENSHOT_FAILED" | "PERMISSION_DENIED" | "PERMISSION_UNVERIFIED" | "TIMEOUT"; message: string }; }
|
||||
export interface ActionResult { action: string; app: AppRef; snapshotId: string | null; elementIndex: number | null; fromElementIndex: number | null; toElementIndex: number | null; performed: true; }
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-13-22:02:
|
||||
* Successful actions burn the app-scoped snapshot fence because they may change the accessible
|
||||
* tree. Publishing this field lets callers confirm they must capture again before replaying indexes.
|
||||
*/
|
||||
export interface ActionResult { action: string; app: AppRef; snapshotId: string | null; elementIndex: number | null; fromElementIndex: number | null; toElementIndex: number | null; performed: true; snapshotConsumed: boolean; }
|
||||
export interface SnapshotRecord { snapshotId: string; targetKey: string; windowKey: string; capturedAt: string; expiresAt: string; app: AppRef; window: WindowRef; elementCount: number; elements: Record<string, Element>; }
|
||||
export const SNAPSHOT_ID_PATTERN = /^cs_[A-Za-z0-9]{10,40}$/;
|
||||
export const isValidSnapshotId = (value: string | undefined): value is string => typeof value === "string" && SNAPSHOT_ID_PATTERN.test(value);
|
||||
@@ -127,7 +138,7 @@ function validLocator(value: unknown): value is ElementLocator { const x = value
|
||||
export function isCapabilitiesResult(value: unknown): value is CapabilitiesResult { const x = value as CapabilitiesResult; return !!x && typeof x.platform === "string" && Array.isArray(x.actions) && Array.isArray(x.unsupportedActions) && !!x.features; }
|
||||
export function isPermissionsResult(value: unknown): value is PermissionsResult { const x = value as PermissionsResult; return !!x && Array.isArray(x.checks) && typeof x.allGranted === "boolean" && x.checks.every((c) => c.granted === (c.status === "granted") && (!c.granted || c.probed)) && (!x.allGranted || x.checks.length > 0 && x.checks.every((c) => c.status === "granted")); }
|
||||
export function isAppStateResult(value: unknown, screenshotSkipped = false): value is AppStateResult { const x = value as AppStateResult; return !!x && !!x.snapshot && Array.isArray(x.snapshot.elements) && x.snapshot.elements.every((e) => Number.isInteger(e.index) && validLocator(e.locator)) && !(x.screenshot && x.screenshotError) && !(!x.screenshot && !x.screenshotError && !screenshotSkipped); }
|
||||
export function isActionResult(value: unknown): value is ActionResult { const x = value as ActionResult; if (!x || x.performed !== true) return false; if (x.action === "drag") return x.elementIndex === null && ((x.fromElementIndex === null && x.toElementIndex === null) || (typeof x.fromElementIndex === "number" && typeof x.toElementIndex === "number")); if (x.action === "hotkey") return x.snapshotId === null && x.elementIndex === null && x.fromElementIndex === null && x.toElementIndex === null; return x.fromElementIndex === null && x.toElementIndex === null; }
|
||||
export function isActionResult(value: unknown): value is ActionResult { const x = value as ActionResult; if (!x || x.performed !== true || typeof x.snapshotConsumed !== "boolean") return false; if (x.action === "drag") return x.elementIndex === null && ((x.fromElementIndex === null && x.toElementIndex === null) || (typeof x.fromElementIndex === "number" && typeof x.toElementIndex === "number")); if (x.action === "hotkey") return x.snapshotId === null && x.elementIndex === null && x.fromElementIndex === null && x.toElementIndex === null; return x.fromElementIndex === null && x.toElementIndex === null; }
|
||||
export function validateResult(subcommand: ComputerSubcommand, value: unknown, screenshotSkipped = false): boolean { if (subcommand === "capabilities") return isCapabilitiesResult(value); if (subcommand === "permissions") return isPermissionsResult(value); if (subcommand === "get-app-state") return isAppStateResult(value, screenshotSkipped); if (COMPUTER_ACTIONS.includes(subcommand as ComputerAction)) return isActionResult(value); return !!value && typeof value === "object"; }
|
||||
export const SECRET_VALUE: unique symbol = Symbol("computer-secret");
|
||||
export type SecretValue = { readonly [SECRET_VALUE]: true; readonly value: string };
|
||||
|
||||
@@ -38,13 +38,13 @@ export function renderComputerUseGuide(
|
||||
macOS is first-class. Other platforms report \`supported: false\` from capabilities and actions fail with \`UNSUPPORTED_PLATFORM\`; stop rather than retrying.
|
||||
|
||||
## The snapshot → act → snapshot loop
|
||||
Capture state, act on that snapshot, then capture again after navigation, focus changes, scrolling, or rendering.
|
||||
Capture state, act on that snapshot, then capture again. A successful action consumes its capture, so the next element action requires a fresh \`fn computer get-app-state\`; re-capture after focus changes, navigation, scrolling, or re-rendering.
|
||||
|
||||
## App selection precedence
|
||||
Targets resolve by exact bundle id, then exact unambiguous app name, then \`pid:<n>\`. Ambiguous names return \`AMBIGUOUS_APP\`.
|
||||
|
||||
## Element indexes are snapshot-scoped and sparse
|
||||
Indexes are valid only for their capture and can be sparse. Never derive an index from an element count; \`--snapshot-id\` fences a capture and stale fences return \`SNAPSHOT_STALE\`.
|
||||
Indexes are valid only for their capture and can be sparse. Read them from \`snapshot.elements[].index\`; never derive one from \`elementCount\`, position, or bounds. \`--snapshot-id\` fences a capture, stale fences return \`SNAPSHOT_STALE\` including \`consumed-by-action\`, and successful action results report \`snapshotConsumed: true\`.
|
||||
|
||||
## Command reference
|
||||
${commands}
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
type AppRef,
|
||||
type Element,
|
||||
type SnapshotRecord,
|
||||
type SnapshotStaleReason,
|
||||
type WindowRef,
|
||||
} from "./contract.js";
|
||||
|
||||
@@ -88,11 +89,27 @@ export class ComputerSnapshotStore {
|
||||
await mkdir(this.snapshotsDirectory, { recursive: true });
|
||||
await mkdir(this.latestDirectory, { recursive: true });
|
||||
await writeJsonAtomically(this.snapshotPath(record.snapshotId), record);
|
||||
await writeJsonAtomically(this.latestPath(targetKey), { snapshotId: record.snapshotId });
|
||||
await this.mutateLatestPointer(targetKey, async () => {
|
||||
await writeJsonAtomically(this.latestPath(targetKey), { snapshotId: record.snapshotId });
|
||||
});
|
||||
await this.prune({ preserveSnapshotId: record.snapshotId });
|
||||
return record;
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-13-22:02:
|
||||
* A successful action consumes only this app's latest pointer, retaining the record for clear
|
||||
* stale details and pruning. The next element replay must capture a new accessibility tree.
|
||||
*/
|
||||
async consume(app: AppRef, expectedSnapshotId?: string): Promise<void> {
|
||||
const targetKey = targetKeyForApp(app);
|
||||
await this.mutateLatestPointer(targetKey, async () => {
|
||||
const latest = await this.readLatest(targetKey);
|
||||
if (!latest || (expectedSnapshotId !== undefined && latest.snapshotId !== expectedSnapshotId)) return;
|
||||
await writeJsonAtomically(this.latestPath(targetKey), { snapshotId: latest.snapshotId, consumedAt: this.now().toISOString() });
|
||||
});
|
||||
}
|
||||
|
||||
/** Read the current record for the resolved app and enforce the C9 fence order. */
|
||||
async resolve(input: ResolveSnapshotInput): Promise<SnapshotRecord> {
|
||||
const targetKey = targetKeyForApp(input.app);
|
||||
@@ -113,6 +130,15 @@ export class ComputerSnapshotStore {
|
||||
if (input.snapshotId && latest.snapshotId !== input.snapshotId) {
|
||||
throw snapshotStale("superseded", input.snapshotId);
|
||||
}
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-13-22:02:
|
||||
* Resolve checks consumption after supersession so an explicit old ID preserves the published
|
||||
* superseded diagnosis. A consumed latest pointer fails closed before expiry because its sparse
|
||||
* indexes may already name different live elements after the preceding action.
|
||||
*/
|
||||
if (latest.consumedAt !== undefined) {
|
||||
throw snapshotStale("consumed-by-action", latest.snapshotId);
|
||||
}
|
||||
if (this.now().getTime() >= Date.parse(record.expiresAt)) {
|
||||
throw snapshotStale("expired", record.snapshotId);
|
||||
}
|
||||
@@ -165,6 +191,18 @@ export class ComputerSnapshotStore {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-13-22:29:
|
||||
* Latest-pointer updates from separate CLI processes must serialize. An action that resolved S
|
||||
* must not overwrite a later get-app-state capture N, so consumption checks its expected ID only
|
||||
* while holding the same per-app lock that persist uses to re-arm the pointer.
|
||||
*/
|
||||
private async mutateLatestPointer(targetKey: string, mutation: () => Promise<void>): Promise<void> {
|
||||
await mkdir(this.latestDirectory, { recursive: true });
|
||||
const lockPath = `${this.latestPath(targetKey)}.lock`;
|
||||
await withDirectoryLock(lockPath, mutation);
|
||||
}
|
||||
|
||||
private snapshotPath(snapshotId: string): string {
|
||||
return join(this.snapshotsDirectory, `${snapshotId}.json`);
|
||||
}
|
||||
@@ -173,7 +211,7 @@ export class ComputerSnapshotStore {
|
||||
return join(this.latestDirectory, `${targetKeySlug(targetKey)}.json`);
|
||||
}
|
||||
|
||||
private async readLatest(targetKey: string): Promise<{ snapshotId: string } | undefined> {
|
||||
private async readLatest(targetKey: string): Promise<LatestPointer | undefined> {
|
||||
const value = await readJson(this.latestPath(targetKey));
|
||||
return isLatestPointer(value) ? value : undefined;
|
||||
}
|
||||
@@ -207,7 +245,7 @@ function snapshotRequired(): ComputerUseError {
|
||||
);
|
||||
}
|
||||
|
||||
function snapshotStale(reason: "not-found" | "superseded" | "expired" | "pid-changed" | "window-mismatch", snapshotId: string): ComputerUseError {
|
||||
function snapshotStale(reason: SnapshotStaleReason, snapshotId: string): ComputerUseError {
|
||||
return new ComputerUseError(
|
||||
"SNAPSHOT_STALE",
|
||||
`Snapshot ${snapshotId} is ${reason}; re-run fn computer get-app-state.`,
|
||||
@@ -238,8 +276,39 @@ async function readDirectory(path: string): Promise<string[]> {
|
||||
}
|
||||
}
|
||||
|
||||
function isLatestPointer(value: unknown): value is { snapshotId: string } {
|
||||
return !!value && typeof value === "object" && typeof (value as { snapshotId?: unknown }).snapshotId === "string";
|
||||
const POINTER_LOCK_RETRY_MS = 5;
|
||||
const POINTER_LOCK_STALE_MS = 60_000;
|
||||
|
||||
/** Serialize a tiny pointer rewrite across independently-invoked CLI processes. */
|
||||
async function withDirectoryLock(lockPath: string, operation: () => Promise<void>): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await mkdir(lockPath);
|
||||
break;
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
||||
const lockInfo = await stat(lockPath).catch(() => undefined);
|
||||
const age = lockInfo === undefined ? Number.NaN : Date.now() - lockInfo.mtimeMs;
|
||||
if (Number.isFinite(age) && age > POINTER_LOCK_STALE_MS) {
|
||||
await rm(lockPath, { recursive: true, force: true });
|
||||
continue;
|
||||
}
|
||||
await new Promise<void>((resolveRetry) => setTimeout(resolveRetry, POINTER_LOCK_RETRY_MS));
|
||||
}
|
||||
}
|
||||
try {
|
||||
await operation();
|
||||
} finally {
|
||||
await rm(lockPath, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
interface LatestPointer { snapshotId: string; consumedAt?: string; }
|
||||
|
||||
function isLatestPointer(value: unknown): value is LatestPointer {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const pointer = value as { snapshotId?: unknown; consumedAt?: unknown };
|
||||
return typeof pointer.snapshotId === "string" && (pointer.consumedAt === undefined || typeof pointer.consumedAt === "string");
|
||||
}
|
||||
|
||||
function isSnapshotRecord(value: unknown): value is SnapshotRecord {
|
||||
|
||||
Reference in New Issue
Block a user