FN-8960: add macOS computer automation CLI
Add a snapshot-backed macOS computer-use surface to the Fusion CLI. - Register fn computer commands for app state, permission, capability, and UI actions. - Implement macOS JXA automation with locator replay and persisted snapshot safety checks. - Document the CLI surface, add contract coverage, and include a release changeset. Files changed: .changeset/fn-8960-computer-use-cli.md | 7 + docs/README.md | 1 + docs/cli-reference.md | 21 ++ docs/computer-use.md | 156 +++++++++++++ packages/cli/src/bin.ts | 11 + .../__tests__/computer-adapter-registry.test.ts | 154 ++++++++++++ .../commands/__tests__/computer-commands.test.ts | 71 ++++++ .../commands/__tests__/computer-contract.test.ts | 34 +++ .../__tests__/computer-snapshot-index.test.ts | 87 +++++++ packages/cli/src/commands/computer.ts | 182 +++++++++++++++ .../cli/src/commands/computer/adapter-macos.ts | 211 +++++++++++++++++ .../cli/src/commands/computer/adapter-registry.ts | 42 ++++ .../src/commands/computer/adapter-unsupported.ts | 44 ++++ packages/cli/src/commands/computer/adapter.ts | 97 ++++++++ packages/cli/src/commands/computer/contract.ts | 83 +++++++ packages/cli/src/commands/computer/exec-seam.ts | 61 +++++ .../cli/src/commands/computer/scripts/macos-jxa.ts | 37 +++ .../cli/src/commands/computer/snapshot-store.ts | 257 +++++++++++++++++++++ 18 files changed, 1556 insertions(+) Fusion-Task-Id: FN-8960 Fusion-Task-Lineage: 5870bf69-b751-4879-bf9d-5f5396cc0f55 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-8960-computer-use-cli.md
Normal file
7
.changeset/fn-8960-computer-use-cli.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
summary: Add macOS `fn computer` desktop-app discovery, snapshots, actions, and permission reporting.
|
||||
category: feature
|
||||
dev: Adds capabilities, permissions, discovery, state, and action subcommands with a versioned JSON envelope, preflight outcome matrix, durable cross-invocation snapshots, deterministic error ordering, and an honest macOS-only support matrix.
|
||||
@@ -22,6 +22,7 @@ For a full walkthrough (installation, onboarding, first task, and daily workflow
|
||||
| [Getting Started](./getting-started.md) | Installation, first-run, first task, and daily workflow basics |
|
||||
| [Dashboard Guide](./dashboard-guide.md) | Board/list views, left/right sidebar navigation, Artifacts, Import Tasks, chat, workflow selection/editor, terminal, git manager, files, planning, and UI tools |
|
||||
| [CLI Reference](./cli-reference.md) | Complete `fn` command reference with subcommands, flags, and examples |
|
||||
| [Computer Use](./computer-use.md) | `fn computer` desktop-app discovery, snapshots, actions, permissions, and JSON contract |
|
||||
| [Remote Access](./remote-access.md) | Operator runbook for Tailscale/Cloudflare setup, tokenized login links, security caveats, and troubleshooting |
|
||||
| [Native Shell Connection Guide](./native-shell.md) | Canonical mobile/desktop shell onboarding, profile management, QR/manual setup, and remote handoff behavior |
|
||||
|
||||
|
||||
@@ -4,6 +4,27 @@
|
||||
|
||||
Fusion’s command-line interface is exposed through the `fn` command.
|
||||
|
||||
## `fn computer` — local desktop automation
|
||||
|
||||
`fn computer` discovers and operates local desktop application windows through a **snapshot → act → snapshot** loop. It is supported on macOS only; other platforms report an honest unsupported capability. Every command supports `--json` and returns the versioned computer-use envelope. See the full [Computer Use reference](./computer-use.md) for setup, permissions, output shapes, snapshot safety, and error handling.
|
||||
|
||||
```bash
|
||||
fn computer capabilities --json
|
||||
fn computer permissions --json
|
||||
fn computer list-apps --json
|
||||
fn computer list-windows --app <app> --json
|
||||
fn computer get-app-state --app <app> [--window-id <id> | --window-index <n>] [--no-screenshot] [--restore-window] --json
|
||||
fn computer click --app <app> --element-index <n> [--snapshot-id <id>] [--window-id <id> | --window-index <n>] --json
|
||||
fn computer set-value --app <app> --element-index <n> (--value <text> | --value-stdin) [--snapshot-id <id>] --json
|
||||
fn computer type-text --app <app> (--text <text> | --text-stdin) [--element-index <n>] [--snapshot-id <id>] --json
|
||||
fn computer press-key --app <app> --key <name> [--element-index <n>] [--snapshot-id <id>] --json
|
||||
fn computer hotkey --app <app> --keys <combo> --json
|
||||
fn computer scroll --app <app> --direction <up|down|left|right> [--amount <n>] [--element-index <n>] [--snapshot-id <id>] --json
|
||||
fn computer drag --app <app> (--from-x <n> --from-y <n> --to-x <n> --to-y <n> | --from-element-index <n> --to-element-index <n>) [--snapshot-id <id>] --json
|
||||
```
|
||||
|
||||
Element actions use the latest persisted app snapshot unless `--snapshot-id` supplies a current-snapshot concurrency fence. Element indexes are sparse and snapshot-scoped; read `snapshot.elements[].index`, then re-snapshot after UI changes. Use `--value-stdin` and `--text-stdin` for secrets. Screenshots return a filesystem path, never image bytes.
|
||||
|
||||
<!--
|
||||
FNXC:AgentTools 2026-06-29-22:31:
|
||||
The published CLI/pi extension must document its agent-facing workflow authoring surface so operators know agents can inspect, create, update, configure, and delete custom workflows without using the dashboard editor.
|
||||
|
||||
156
docs/computer-use.md
Normal file
156
docs/computer-use.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Computer Use CLI
|
||||
|
||||
[← Docs index](./README.md) · [CLI reference](./cli-reference.md)
|
||||
|
||||
`fn computer` inspects and operates local desktop application windows. It is designed for a safe, repeatable **snapshot → act → snapshot** loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or rendering.
|
||||
|
||||
> **Platform support:** macOS is supported using only operating-system-provided `osascript` and `screencapture`. Linux, Windows, and other platforms are honestly unsupported: `capabilities` and `permissions` succeed with `supported: false`; all other commands fail with `UNSUPPORTED_PLATFORM`. Fusion does not download a helper, native module, or automation dependency.
|
||||
|
||||
## Setup and permissions
|
||||
|
||||
Run the following first:
|
||||
|
||||
```bash
|
||||
fn computer capabilities --json
|
||||
fn computer permissions --json
|
||||
```
|
||||
|
||||
On macOS, grant the application/process running `fn` both permissions in **System Settings**:
|
||||
|
||||
1. **Privacy & Security → Accessibility**: enable the terminal application, IDE, or host process that launches `fn`.
|
||||
2. **Privacy & Security → Screen Recording**: enable that same host when screenshots are needed.
|
||||
3. Quit and reopen the host application if macOS asks for it, then rerun `fn computer permissions --json`.
|
||||
|
||||
`permissions` reports each check as `granted`, `denied`, or `unknown`, not a guessed yes/no. It uses non-mutating preflights through the JXA Objective-C bridge: `AXIsProcessTrusted` for Accessibility and `CGPreflightScreenCaptureAccess` for Screen Recording. A probe that cannot run is `unknown`, never a claimed grant.
|
||||
|
||||
Do **not** use a successful `screencapture` as proof that Screen Recording is granted. macOS can return a normal-looking, desktop-only image when permission is denied. A denied Screen Recording grant therefore does not capture; an unknown grant may capture, but marks `verifiedPermission: false`.
|
||||
|
||||
## Quick workflow
|
||||
|
||||
Use a bundle ID when available; it is the least ambiguous target:
|
||||
|
||||
```bash
|
||||
fn computer list-apps --json
|
||||
fn computer get-app-state --app com.apple.Safari --json
|
||||
# Inspect result.snapshot.elements[].index and save result.snapshot.snapshotId.
|
||||
fn computer click --app com.apple.Safari --element-index 42 --snapshot-id cs_01HABCDE123 --json
|
||||
fn computer get-app-state --app com.apple.Safari --json
|
||||
```
|
||||
|
||||
The first command persists the snapshot before it prints its `snapshotId`. The later `click` may run in a completely separate `fn` process; `--snapshot-id` is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app.
|
||||
|
||||
Element indexes are snapshot-scoped and sparse. Read indexes from `snapshot.elements[].index`; never derive an index from `elementCount`, position, or bounds. Refresh state after navigation, focus change, scrolling, or re-rendering. Semantic actions (`click`, `set-value`) are preferred over raw keys because they survive focus changes better.
|
||||
|
||||
App resolution for a bare `--app` value is: exact bundle ID, then exact unambiguous name, then `pid:<n>`. Ambiguous names fail with `AMBIGUOUS_APP`.
|
||||
|
||||
## Commands and flags
|
||||
|
||||
All commands support `--json`.
|
||||
|
||||
| Command | Required flags | Optional flags |
|
||||
| --- | --- | --- |
|
||||
| `capabilities` | — | — |
|
||||
| `permissions` | — | — |
|
||||
| `list-apps` | — | — |
|
||||
| `list-windows` | `--app <target>` | — |
|
||||
| `get-app-state` | `--app <target>` | `--window-id <id>` or `--window-index <n>`, `--no-screenshot`, `--restore-window` |
|
||||
| `click` | `--app`, `--element-index <n>` | `--snapshot-id <id>`, `--window-id <id>` or `--window-index <n>` |
|
||||
| `set-value` | `--app`, `--element-index <n>`, one of `--value <text>` or `--value-stdin` | snapshot/window flags |
|
||||
| `type-text` | `--app`, one of `--text <text>` or `--text-stdin` | `--element-index`, snapshot/window flags |
|
||||
| `press-key` | `--app`, `--key <name>` | `--element-index`, snapshot/window flags |
|
||||
| `hotkey` | `--app`, `--keys <combo>` (repeatable or `+`-joined) | — |
|
||||
| `scroll` | `--app`, `--direction up\|down\|left\|right` | `--amount <n>` (default 3), `--element-index`, snapshot/window flags |
|
||||
| `drag` | `--app`, either all `--from-x --from-y --to-x --to-y` or both `--from-element-index --to-element-index` | snapshot/window flags for element form only |
|
||||
|
||||
`--window-id` and `--window-index` are mutually exclusive. The value of `--snapshot-id` must match `^cs_[A-Za-z0-9]{10,40}$`; it is validated before filesystem access. Numeric indexes, coordinates, and amounts must be non-negative. Do not mix or partially provide drag forms. `hotkey` and coordinate-only `drag` take neither snapshot nor window flags.
|
||||
|
||||
Use `--value-stdin` and `--text-stdin` for secrets. Fusion does not put stdin values in output, process arguments, snapshot records, logs, audit metadata, or error messages.
|
||||
|
||||
Targetless `type-text`, `press-key`, and `scroll` act on the target app's currently focused UI element and never activate or raise the app implicitly. If nothing is focused, they fail with `ACTION_FAILED`; focus the app yourself or pass `--element-index`.
|
||||
|
||||
## JSON contract (schema version 1)
|
||||
|
||||
Every `--json` invocation prints exactly one object to stdout and nothing else. Success has `result` only; failure has `error` only:
|
||||
|
||||
```json
|
||||
{"schemaVersion":1,"ok":true,"command":"computer.get-app-state","result":{}}
|
||||
```
|
||||
|
||||
```json
|
||||
{"schemaVersion":1,"ok":false,"command":"computer.click","error":{"code":"SNAPSHOT_STALE","message":"Snapshot expired.","remediation":"Re-run `fn computer get-app-state --app <app>`.","details":{"reason":"expired"}}}
|
||||
```
|
||||
|
||||
`schemaVersion` is literal `1` and changes only for breaking reshapes. `command` is `computer.<subcommand>` using a runtime-inventory token. Group-routing failures (unknown or missing subcommand in JSON mode) uniquely use `command: "computer"`. `error.details`, when present, is a flat map of string, number, or boolean values and never contains a secret. Diagnostics go to stderr; in human mode successful summaries go to stdout and failures are `error: <code>: <message>` on stderr.
|
||||
|
||||
The runtime exports frozen `COMPUTER_SUBCOMMANDS`, `COMPUTER_ACTIONS`, and `COMPUTER_ERROR_CODES` tuples; their TypeScript unions and dispatcher derive from them. Adding a command requires updating this runtime inventory. Exit code is `0` for success and `1` for every failure. Error codes are append-only:
|
||||
|
||||
| Code | Meaning |
|
||||
| --- | --- |
|
||||
| `UNSUPPORTED_PLATFORM` | No supported adapter (except capability/permission reporting) |
|
||||
| `PERMISSION_DENIED` | Definitively denied required permission or missing required built-in |
|
||||
| `PERMISSION_UNVERIFIED` | Unknown probe followed by a permission-shaped OS failure |
|
||||
| `INVALID_ARGUMENTS` | Invalid command, flags, number, enum, or snapshot ID |
|
||||
| `APP_NOT_FOUND` / `AMBIGUOUS_APP` | No app match / more than one name match |
|
||||
| `WINDOW_NOT_FOUND` | Supplied live window selector did not match |
|
||||
| `SNAPSHOT_REQUIRED` | An element action has no snapshot for the app |
|
||||
| `SNAPSHOT_STALE` | Snapshot is no longer usable; see below |
|
||||
| `ELEMENT_INDEX_NOT_FOUND` | Sparse index is absent from the snapshot |
|
||||
| `ELEMENT_UNRESOLVABLE` | Recorded locator no longer resolves or identity differs |
|
||||
| `ACTION_UNSUPPORTED` / `ACTION_FAILED` | Adapter does not support action / OS action failed |
|
||||
| `SCREENSHOT_FAILED` | Screenshot-only failure, reported in `screenshotError` rather than top-level error |
|
||||
| `TIMEOUT` / `INTERNAL` | Timed-out seam / unexpected, redacted failure |
|
||||
|
||||
Remediation is required for unsupported platform, permission denied/unverified, snapshot required/stale, missing/unresolvable element, ambiguous app, and unsupported action.
|
||||
|
||||
### Result values
|
||||
|
||||
- `capabilities`: `{ platform, adapterId, supported, actions, unsupportedActions, features: { screenshot, restoreWindow, stdinSecrets, crossInvocationSnapshots } }`. Every action is in exactly one action list.
|
||||
- `permissions`: `{ platform, adapterId, supported, allGranted, checks }`, where each check is `{ id: "accessibility"|"screen-recording", status: "granted"|"denied"|"unknown", granted, probed, probe, detail, remediation }`. `granted` is exactly `status === "granted"`; `allGranted` is the AND of checks.
|
||||
- `list-apps`: `{ apps }`, sorted by name. An app is `{ bundleId, name, pid }`.
|
||||
- `list-windows`: `{ app, windows }`. A window is `{ windowId, windowIndex, title, bounds, minimized }`.
|
||||
- `get-app-state`: `{ app, window, snapshot, screenshot, screenshotError? }`. `snapshot` has `{ snapshotId, targetKey, windowKey, capturedAt, expiresAt, treeText, elementCount, truncated, elements }`. An element is `{ index, role, title, value, label, enabled, focused, bounds, actions, locator }`; its locator is `{ kind: "ax-path", path, role, subrole, identifier, title }`.
|
||||
- Actions return `{ action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true }` only after the OS action succeeds. Single-endpoint actions use `elementIndex`; element-form drag uses `fromElementIndex` and `toElementIndex` with `elementIndex: null`; hotkey reports all index fields and `snapshotId` as `null`.
|
||||
|
||||
Screenshots are always paths, never base64, `data:` URLs, or byte arrays. `--no-screenshot` produces `screenshot: null` with no `screenshotError`; successful capture has `screenshot` and no error; failed/not-captured has `screenshot: null` and `screenshotError`. Screenshot `verifiedPermission` is true only for a preflight-confirmed Screen Recording grant.
|
||||
|
||||
## Permissions outcome matrix
|
||||
|
||||
| Check/status | Command behavior |
|
||||
| --- | --- |
|
||||
| Accessibility granted | Proceed. |
|
||||
| Accessibility denied | Before automation, fail `PERMISSION_DENIED` with `details.permission: "accessibility"`. |
|
||||
| Accessibility unknown | Attempt the call. A permission-shaped failure becomes `PERMISSION_UNVERIFIED`; other failures are `ACTION_FAILED` or `TIMEOUT`. This applies to window/locator replay too. |
|
||||
| `capabilities` or `permissions` | Never fail because of permission status; return status data. |
|
||||
| Screen Recording granted | Capture; successful image has `verifiedPermission: true`; a failed capture is `SCREENSHOT_FAILED` or `TIMEOUT`. |
|
||||
| Screen Recording denied | Do not capture; return screenshot `null` and `screenshotError.code: "PERMISSION_DENIED"`. |
|
||||
| Screen Recording unknown | Attempt capture; success has `verifiedPermission: false`; failure is `PERMISSION_UNVERIFIED` or `TIMEOUT`. |
|
||||
| Any Screen Recording status with `--no-screenshot` | Do not probe or capture; screenshot is null without an error. |
|
||||
| Missing `osascript` | Checks are unknown; non-reporting commands fail `PERMISSION_DENIED` with `details.missingBinary: "osascript"`. |
|
||||
| Missing `screencapture` | Only screenshot degrades with `SCREENSHOT_FAILED`. |
|
||||
|
||||
## Durable snapshots and safe element replay
|
||||
|
||||
Snapshots are stored per project at `.fusion/computer-use/snapshots/<snapshotId>.json`; the app's latest pointer is `.fusion/computer-use/latest/<targetKeySlug>.json`. They are never shared across project roots. Capture atomically persists the record and pointer before returning `snapshotId`.
|
||||
|
||||
A resolved app has app-scoped `targetKey` (`bundle:<bundleId>`, or `pid:<pid>`) and window-scoped `windowKey` (`<targetKey>#<windowId>`). There is one latest pointer per app, not per window. An action with no `--snapshot-id` uses that latest snapshot. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces `SNAPSHOT_STALE` / `window-mismatch`.
|
||||
|
||||
Snapshots expire after five minutes by default; `expiresAt` publishes the exact deadline. An explicit snapshot ID is a concurrency fence, not a way to revive old UI: a superseded ID fails. Before acting, Fusion re-resolves the recorded window and then the locator rooted in it, verifying role, subrole, and recorded identifier. It never acts on a new occupant of the old path and never falls back to saved bounds/coordinates.
|
||||
|
||||
| Failure | When | Recovery |
|
||||
| --- | --- | --- |
|
||||
| `SNAPSHOT_REQUIRED` | No latest snapshot exists for the target app | Run `get-app-state`. |
|
||||
| `SNAPSHOT_STALE` | `not-found`, `superseded`, `expired`, `pid-changed`, `window-mismatch`, or `window-gone` | Run `get-app-state`; use the current window and snapshot. |
|
||||
| `ELEMENT_INDEX_NOT_FOUND` | Index is absent from sparse map | Read the current `elements[].index` after re-snapshotting. |
|
||||
| `ELEMENT_UNRESOLVABLE` | Locator path fails or identity no longer matches | Re-snapshot; do not retry with coordinates. |
|
||||
|
||||
## Deterministic failure order and timeouts
|
||||
|
||||
Exactly one error is emitted: the first failing stage wins. The order is group routing; flag parsing; platform resolution; action support; built-in and permission gate; stdin read; app resolution; supplied-window resolution; snapshot resolution; sparse element lookup; window/locator replay; OS call. Thus malformed flags beat unsupported platform, and a permission-shaped replay error is `PERMISSION_UNVERIFIED`, not an element error. Unexpected failures become a redacted `INTERNAL` envelope, never a stack trace on stdout.
|
||||
|
||||
No user timeout flag is provided. Default seam timeouts are: permission probe 5 seconds; app/window listing 10 seconds; state capture 20 seconds; screenshot 15 seconds; locator replay 10 seconds; action 10 seconds.
|
||||
|
||||
## Orca prior art and Fusion differences
|
||||
|
||||
This surface was informed by [Orca](https://github.com/stablyai/orca) and its [computer-use CLI documentation](https://www.onorca.dev/docs/cli/computer-use). Orca uses two layers: a native-backed `orca computer` CLI and a thin skill that loads a version-matched guide. Fusion follows the CLI-plus-skill separation; the skill is deliberately out of scope for this command and is delivered separately.
|
||||
|
||||
Fusion deliberately differs by using macOS OS built-ins instead of native helpers; publishing non-mutating preflight permission checks, an `unknown` status, and the outcome matrix; documenting durable on-disk cross-process snapshots with identity-verified window/locator replay; distinguishing stale, missing, and unresolvable indexes; publishing deterministic failure precedence; and using a versioned envelope with a fixed append-only error enum. `paste-text` and `perform-secondary-action` are not implemented in this release; they remain absent from capabilities rather than being stubs.
|
||||
@@ -154,6 +154,7 @@ async function loadCommandHandlers() {
|
||||
const { runPluginDev } = await import("./commands/plugin-dev.js");
|
||||
const { runPluginPublish } = await import("./commands/plugin-publish.js");
|
||||
const { runSkillsSearch, runSkillsInstall } = await import("./commands/skills.js");
|
||||
const { runComputer } = await import("./commands/computer.js");
|
||||
const { runResearchCreate, runResearchList, runResearchShow, runResearchExport, runResearchCancel, runResearchRetry } = await import("./commands/research.js");
|
||||
const { runExperimentFinalize } = await import("./commands/experiment-finalize.js");
|
||||
const { dispatchUpdateCliArgs } = await import("./commands/update.js");
|
||||
@@ -287,6 +288,7 @@ async function loadCommandHandlers() {
|
||||
runPluginPublish,
|
||||
runSkillsSearch,
|
||||
runSkillsInstall,
|
||||
runComputer,
|
||||
runResearchCreate,
|
||||
runResearchList,
|
||||
runResearchShow,
|
||||
@@ -495,6 +497,8 @@ PR:
|
||||
fn skills install <owner/repo> Install skills from a source
|
||||
fn skills install <owner/repo> --skill <name>
|
||||
Install a specific skill
|
||||
fn computer <subcommand> [--json] Inspect and automate supported desktop applications
|
||||
See fn computer --help for snapshot → act → snapshot commands
|
||||
|
||||
Options:
|
||||
--project, -P <name> Target a specific project (bypasses CWD detection)
|
||||
@@ -842,6 +846,7 @@ async function main() {
|
||||
runPluginPublish,
|
||||
runSkillsSearch,
|
||||
runSkillsInstall,
|
||||
runComputer,
|
||||
runResearchCreate,
|
||||
runResearchList,
|
||||
runResearchShow,
|
||||
@@ -2292,6 +2297,12 @@ async function main() {
|
||||
break;
|
||||
}
|
||||
|
||||
case "computer": {
|
||||
const exitCode = await runComputer(args.slice(1), { projectRoot: process.cwd() });
|
||||
if (exitCode !== 0) process.exit(exitCode);
|
||||
break;
|
||||
}
|
||||
|
||||
case "skills": {
|
||||
const subcommand = args[1];
|
||||
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { COMPUTER_ACTIONS, type CapabilitiesResult } from "../computer/contract.js";
|
||||
import type { ComputerAdapter } from "../computer/adapter.js";
|
||||
import { resolveComputerAdapter } from "../computer/adapter-registry.js";
|
||||
import { MacosComputerAdapter } from "../computer/adapter-macos.js";
|
||||
|
||||
const projectRoot = "/not-a-real-project";
|
||||
const seam = { run: vi.fn() };
|
||||
const clock = { now: () => new Date("2026-08-11T03:34:00.000Z") };
|
||||
|
||||
function fakeMacosAdapter(): ComputerAdapter {
|
||||
return {
|
||||
platform: "darwin", id: "macos", supported: true,
|
||||
capabilities: async (): Promise<CapabilitiesResult> => ({
|
||||
platform: "darwin", adapterId: "macos", supported: true,
|
||||
actions: [...COMPUTER_ACTIONS], unsupportedActions: [],
|
||||
features: { screenshot: true, restoreWindow: true, stdinSecrets: true, crossInvocationSnapshots: true },
|
||||
}),
|
||||
} as ComputerAdapter;
|
||||
}
|
||||
|
||||
function resolve(platform: string, macosAdapterFactory = vi.fn(fakeMacosAdapter)) {
|
||||
return {
|
||||
adapter: resolveComputerAdapter({ platform, seam, clock, projectRoot, macosAdapterFactory }),
|
||||
macosAdapterFactory,
|
||||
};
|
||||
}
|
||||
|
||||
describe("computer adapter registry", () => {
|
||||
it("resolves the injected macOS adapter with the injected I/O dependencies", () => {
|
||||
const { adapter, macosAdapterFactory } = resolve("darwin");
|
||||
expect(adapter.supported).toBe(true);
|
||||
expect(adapter.id).toBe("macos");
|
||||
expect(macosAdapterFactory).toHaveBeenCalledWith({ platform: "darwin", seam, clock, projectRoot });
|
||||
});
|
||||
|
||||
it.each(["linux", "win32"])("uses an honest unsupported adapter on %s", async (platform) => {
|
||||
const { adapter, macosAdapterFactory } = resolve(platform);
|
||||
expect(macosAdapterFactory).not.toHaveBeenCalled();
|
||||
await expect(adapter.capabilities()).resolves.toMatchObject({
|
||||
platform, supported: false, actions: [], unsupportedActions: [...COMPUTER_ACTIONS],
|
||||
features: { crossInvocationSnapshots: false },
|
||||
});
|
||||
await expect(adapter.permissions()).resolves.toMatchObject({
|
||||
platform, supported: false, allGranted: false, checks: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("posts a real CoreGraphics press-move-release program for coordinate drag", async () => {
|
||||
const dragSeam = { run: vi.fn()
|
||||
.mockResolvedValueOnce({ stdout: '{"accessibility":true,"screenRecording":true}', stderr: "", exitCode: 0, timedOut: false })
|
||||
.mockResolvedValueOnce({ stdout: '{"ok":true}', stderr: "", exitCode: 0, timedOut: false }) };
|
||||
const adapter = new MacosComputerAdapter({ seam: dragSeam, clock, projectRoot });
|
||||
await adapter.drag({ app: { bundleId: "com.example.App", name: "App", pid: 4 }, snapshotId: null, fromX: 1, fromY: 2, toX: 3, toY: 4 });
|
||||
const [file, args, options] = dragSeam.run.mock.calls[1]!;
|
||||
expect(file).toBe("osascript");
|
||||
expect(args).toEqual(expect.arrayContaining(["drag-coordinates", "App", "1", "2", "3", "4"]));
|
||||
expect(args[3]).toContain("CGEventCreateMouseEvent");
|
||||
expect(args[3]).toContain("CGEventPost");
|
||||
expect(args[3]).not.toContain("se.mouseMove");
|
||||
expect(options).toMatchObject({ timeoutMs: 10_000 });
|
||||
});
|
||||
|
||||
it.each(["hotkey", "drag"])("does not automate %s when Accessibility is denied", async (action) => {
|
||||
const denied = { stdout: '{"accessibility":false,"screenRecording":false}', stderr: "", exitCode: 0, timedOut: false };
|
||||
const deniedSeam = { run: vi.fn().mockResolvedValue(denied) };
|
||||
const adapter = new MacosComputerAdapter({ seam: deniedSeam, clock, projectRoot });
|
||||
const app = { bundleId: "com.example.App", name: "App", pid: 4 };
|
||||
const run = action === "hotkey"
|
||||
? adapter.hotkey({ app, keys: ["cmd", "k"] })
|
||||
: adapter.drag({ app, snapshotId: null, fromX: 1, fromY: 2, toX: 3, toY: 4 });
|
||||
await expect(run).rejects.toMatchObject({ code: "PERMISSION_DENIED" });
|
||||
expect(deniedSeam.run).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("maps a vanished locator path to ELEMENT_UNRESOLVABLE without automating", async () => {
|
||||
const locatorSeam = { run: vi.fn()
|
||||
.mockResolvedValueOnce({ stdout: '{"accessibility":true,"screenRecording":true}', stderr: "", exitCode: 0, timedOut: false })
|
||||
.mockResolvedValueOnce({ stdout: "", stderr: "locator-not-found", exitCode: 1, timedOut: false }) };
|
||||
const adapter = new MacosComputerAdapter({ seam: locatorSeam, clock, projectRoot });
|
||||
await expect(adapter.resolveLocator({ window: { windowId: "window-7", windowIndex: 0, title: "Window", bounds: null, minimized: false }, handle: { appTarget: "App", windowId: "window-7", windowIndex: 0 } }, { kind: "ax-path", path: "window[0]/AXButton[3]", role: "AXButton", subrole: null, identifier: null, title: "Go" })).rejects.toMatchObject({ code: "ELEMENT_UNRESOLVABLE", remediation: expect.stringContaining("get-app-state") });
|
||||
expect(locatorSeam.run.mock.calls).toHaveLength(2);
|
||||
expect(locatorSeam.run.mock.calls[1]![1]).toEqual(expect.arrayContaining(["resolve-locator", "window-7", "window[0]/AXButton[3]"]));
|
||||
});
|
||||
|
||||
it.each([["left", 123], ["right", 124], ["up", 126], ["down", 125]] as const)("passes %s scroll direction to the static four-way JXA mapping", async (direction, keyCode) => {
|
||||
const scrollSeam = { run: vi.fn()
|
||||
.mockResolvedValueOnce({ stdout: '{"accessibility":true,"screenRecording":true}', stderr: "", exitCode: 0, timedOut: false })
|
||||
.mockResolvedValueOnce({ stdout: '{"ok":true}', stderr: "", exitCode: 0, timedOut: false }) };
|
||||
const adapter = new MacosComputerAdapter({ seam: scrollSeam, clock, projectRoot });
|
||||
const element = { element: { index: 3, role: "AXScrollArea", title: null, value: null, label: null, enabled: true, focused: true, bounds: null, actions: [], locator: { kind: "ax-path" as const, path: "window[0]", role: "AXScrollArea", subrole: null, identifier: null, title: null } }, handle: "window[0]" };
|
||||
await adapter.scroll({ app: { bundleId: "com.example.App", name: "App", pid: 4 }, window: { window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, handle: { appTarget: "App", windowId: "w", windowIndex: 0 } }, element, snapshotId: "cs_0123456789", direction, amount: 2 });
|
||||
const args = scrollSeam.run.mock.calls[1]![1] as string[];
|
||||
expect(args).toEqual(expect.arrayContaining(["scroll", "App", `${direction}:2`]));
|
||||
expect(args[3]).toContain(`direction==='${direction}'`);
|
||||
expect(args[3]).toContain(`return ${keyCode}`);
|
||||
});
|
||||
|
||||
it("degrades a rejected screencapture spawn without discarding the state tree", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "fusion-computer-screenshot-"));
|
||||
const missingCapture = Object.assign(new Error("spawn screencapture ENOENT"), { code: "ENOENT" });
|
||||
const captureSeam = {
|
||||
run: vi.fn(async (file: string, args: readonly string[]) => {
|
||||
if (file === "screencapture") throw missingCapture;
|
||||
const operation = args[4];
|
||||
if (operation === "list-apps") return { stdout: '{"apps":[{"bundleId":"com.example.App","name":"App","pid":4}]}', stderr: "", exitCode: 0, timedOut: false };
|
||||
if (operation === "list-windows") return { stdout: '{"windows":[{"windowId":"w","windowIndex":0,"title":"Window","bounds":null,"minimized":false,"captureWindowId":42}]}', stderr: "", exitCode: 0, timedOut: false };
|
||||
if (operation === "state") return { stdout: '{"treeText":"captured tree","elements":[],"truncated":false}', stderr: "", exitCode: 0, timedOut: false };
|
||||
return { stdout: '{"accessibility":true,"screenRecording":true}', stderr: "", exitCode: 0, timedOut: false };
|
||||
}),
|
||||
};
|
||||
try {
|
||||
const adapter = new MacosComputerAdapter({ seam: captureSeam, clock, projectRoot: root });
|
||||
const state = await adapter.captureState({ kind: "name", raw: "App", value: "App" }, { screenshot: true, restoreWindow: false });
|
||||
expect(state.snapshot.treeText).toBe("captured tree");
|
||||
expect(state.screenshot).toBeNull();
|
||||
expect(state.screenshotError).toEqual({ code: "SCREENSHOT_FAILED", message: "Screenshot capture failed." });
|
||||
expect(captureSeam.run).toHaveBeenCalledWith("screencapture", expect.any(Array), { timeoutMs: 15_000 });
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("reports a missing osascript as unknown permissions and blocks automation", async () => {
|
||||
const missing = Object.assign(new Error("spawn osascript ENOENT"), { code: "ENOENT" });
|
||||
const missingSeam = { run: vi.fn().mockRejectedValue(missing) };
|
||||
const adapter = new MacosComputerAdapter({ seam: missingSeam, clock, projectRoot });
|
||||
await expect(adapter.permissions()).resolves.toMatchObject({ allGranted: false, checks: [{ id: "accessibility", status: "unknown", granted: false, probed: false }, { id: "screen-recording", status: "unknown", granted: false, probed: false }] });
|
||||
await expect(adapter.listApps()).rejects.toMatchObject({ code: "PERMISSION_DENIED", details: { missingBinary: "osascript" } });
|
||||
expect(missingSeam.run).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("rejects every unsupported operation, including both replay resolvers", async () => {
|
||||
const { adapter } = resolve("linux");
|
||||
const expectUnsupported = async (operation: Promise<unknown>) => {
|
||||
await expect(operation).rejects.toMatchObject({
|
||||
code: "UNSUPPORTED_PLATFORM",
|
||||
remediation: expect.stringContaining("macOS"),
|
||||
});
|
||||
};
|
||||
|
||||
await expectUnsupported(adapter.listApps());
|
||||
await expectUnsupported(adapter.listWindows({ kind: "name", raw: "Notes", value: "Notes" }));
|
||||
await expectUnsupported(adapter.captureState({ kind: "name", raw: "Notes", value: "Notes" }, { screenshot: false, restoreWindow: false }));
|
||||
await expectUnsupported(adapter.resolveWindow({ kind: "name", raw: "Notes", value: "Notes" }, {}));
|
||||
await expectUnsupported(adapter.resolveLocator({} as never, {} as never));
|
||||
for (const action of COMPUTER_ACTIONS) {
|
||||
await expectUnsupported((adapter as unknown as Record<string, (input: unknown) => Promise<unknown>>)[action]({}));
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,71 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { runComputer } from "../computer.js";
|
||||
import type { ComputerAdapter } from "../computer/adapter.js";
|
||||
|
||||
const app = { bundleId: "com.example.App", name: "App", pid: 1 };
|
||||
const adapter: ComputerAdapter = {
|
||||
platform: "darwin", id: "fake", supported: true,
|
||||
capabilities: async () => ({ platform: "darwin", adapterId: "fake", supported: true, actions: ["click", "set-value", "type-text", "press-key", "hotkey", "scroll", "drag"], unsupportedActions: [], features: { screenshot: false, restoreWindow: false, stdinSecrets: true, crossInvocationSnapshots: true } }),
|
||||
permissions: async () => ({ platform: "darwin", adapterId: "fake", supported: true, allGranted: true, checks: [] }),
|
||||
listApps: async () => ({ apps: [app] }), listWindows: async () => ({ app, windows: [{ windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }] }),
|
||||
captureState: async () => ({ app, window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, snapshot: { snapshotId: "", targetKey: "", windowKey: "", capturedAt: new Date(0).toISOString(), expiresAt: "", treeText: "tree", elementCount: 1, truncated: false, elements: [{ index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator: { kind: "ax-path", path: "button[0]", role: "AXButton", subrole: null, identifier: null, title: "Go" } }] }, screenshot: null }),
|
||||
resolveWindow: async () => ({ window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, handle: "w" }), resolveLocator: async (_w, locator) => ({ element: { index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator }, handle: "e" }),
|
||||
click: async (x) => ({ action: "click", app: x.app, snapshotId: x.snapshotId, elementIndex: 7, fromElementIndex: null, toElementIndex: null, performed: true }), "set-value": async () => { throw new Error("unused"); }, "type-text": async () => { throw new Error("unused"); }, "press-key": async () => { throw new Error("unused"); }, hotkey: async () => { throw new Error("unused"); }, scroll: async () => { throw new Error("unused"); }, drag: async () => { throw new Error("unused"); },
|
||||
};
|
||||
describe("computer commands", () => {
|
||||
it("emits one JSON envelope and persists a snapshot", async () => { const output: string[] = []; const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-")); try { expect(await runComputer(["get-app-state", "--app", "App", "--no-screenshot", "--json"], { adapter, projectRoot: root, stdout: (x) => output.push(x) })).toBe(0); const envelope = JSON.parse(output[0]); expect(envelope).toMatchObject({ schemaVersion: 1, ok: true, command: "computer.get-app-state" }); expect(envelope.result.snapshot.snapshotId).toMatch(/^cs_/); } finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); } });
|
||||
it("uses the snapshot replay path for element-scoped typing", async () => {
|
||||
const root = await import("node:fs/promises").then((fs) => fs.mkdtemp("/tmp/fusion-computer-"));
|
||||
let resolved = 0;
|
||||
let typed = 0;
|
||||
const replayAdapter: ComputerAdapter = { ...adapter,
|
||||
resolveLocator: async (_window, locator) => { resolved += 1; return { element: { index: 7, role: "AXButton", title: "Go", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator }, handle: "live" }; },
|
||||
"type-text": async (input) => { typed += 1; return { action: "type-text", app: input.app, snapshotId: input.snapshotId ?? null, elementIndex: input.element?.element.index ?? null, fromElementIndex: null, toElementIndex: null, performed: true }; },
|
||||
};
|
||||
try {
|
||||
await runComputer(["get-app-state", "--app", "App", "--no-screenshot", "--json"], { adapter: replayAdapter, projectRoot: root, clock: { now: () => new Date(0) }, stdout: () => undefined });
|
||||
expect(await runComputer(["type-text", "--app", "App", "--element-index", "7", "--text", "safe", "--json"], { adapter: replayAdapter, projectRoot: root, clock: { now: () => new Date(0) }, stdout: () => undefined })).toBe(0);
|
||||
expect(resolved).toBe(1); expect(typed).toBe(1);
|
||||
} finally { await (await import("node:fs/promises")).rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
it("accepts complete coordinate drag without a snapshot", async () => {
|
||||
let input: Parameters<ComputerAdapter["drag"]>[0] | undefined;
|
||||
const dragAdapter: ComputerAdapter = { ...adapter, drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true }; } };
|
||||
expect(await runComputer(["drag", "--app", "App", "--from-x", "1", "--from-y", "2", "--to-x", "3", "--to-y", "4", "--json"], { adapter: dragAdapter, stdout: () => undefined })).toBe(0);
|
||||
expect(input).toMatchObject({ snapshotId: null, fromX: 1, fromY: 2, toX: 3, toY: 4 });
|
||||
});
|
||||
it("resolves both element-drag endpoints from one snapshot fence during a concurrent latest update", async () => {
|
||||
const from = { index: 7, role: "AXButton", title: "From", value: null, label: null, enabled: true, focused: false, bounds: null, actions: [], locator: { kind: "ax-path" as const, path: "window[0]/AXButton[0]", role: "AXButton", subrole: null, identifier: null, title: "From" } };
|
||||
const to = { ...from, index: 9, title: "To", locator: { ...from.locator, path: "window[0]/AXButton[1]", title: "To" } };
|
||||
const record = { snapshotId: "cs_0123456789", window: { windowId: "w", windowIndex: 0, title: "w", bounds: null, minimized: false }, app, elements: { "7": from, "9": to } };
|
||||
const resolve = vi.fn().mockImplementation(async () => record);
|
||||
let input: Parameters<ComputerAdapter["drag"]>[0] | undefined;
|
||||
const dragAdapter: ComputerAdapter = { ...adapter,
|
||||
resolveLocator: async (_window, locator) => ({ element: locator.path === from.locator.path ? from : to, handle: locator.path }),
|
||||
drag: async (value) => { input = value; return { action: "drag", app: value.app, snapshotId: value.snapshotId, elementIndex: null, fromElementIndex: value.from?.element.index ?? null, toElementIndex: value.to?.element.index ?? null, performed: true }; },
|
||||
};
|
||||
const store = { resolve, getElement: (_record: typeof record, index: number) => _record.elements[String(index)] } as unknown as import("../computer/snapshot-store.js").ComputerSnapshotStore;
|
||||
expect(await runComputer(["drag", "--app", "App", "--from-element-index", "7", "--to-element-index", "9", "--json"], { adapter: dragAdapter, store, stdout: () => undefined })).toBe(0);
|
||||
expect(resolve).toHaveBeenCalledTimes(1);
|
||||
expect(input).toMatchObject({ snapshotId: record.snapshotId, from: { element: { index: 7 } }, to: { element: { index: 9 } } });
|
||||
});
|
||||
it("uses group-level INVALID_ARGUMENTS for unknown commands", async () => { const output: string[] = []; expect(await runComputer(["nope", "--json"], { adapter, stdout: (x) => output.push(x) })).toBe(1); expect(JSON.parse(output[0])).toMatchObject({ command: "computer", error: { code: "INVALID_ARGUMENTS" } }); });
|
||||
it("returns the required JSON envelope for a missing subcommand", async () => {
|
||||
const output: string[] = [];
|
||||
expect(await runComputer(["--json"], { adapter, stdout: (text) => output.push(text) })).toBe(1);
|
||||
expect(JSON.parse(output[0]!)).toMatchObject({ ok: false, command: "computer", error: { code: "INVALID_ARGUMENTS" } });
|
||||
});
|
||||
it("validates mutually exclusive text flags before app discovery", async () => {
|
||||
const listApps = vi.fn(adapter.listApps);
|
||||
const output: string[] = [];
|
||||
expect(await runComputer(["type-text", "--app", "Missing", "--text", "a", "--text-stdin", "--json"], { adapter: { ...adapter, listApps }, stdout: (text) => output.push(text) })).toBe(1);
|
||||
expect(JSON.parse(output[0]!)).toMatchObject({ error: { code: "INVALID_ARGUMENTS" } });
|
||||
expect(listApps).not.toHaveBeenCalled();
|
||||
});
|
||||
it("falls back from an unmatched dotted bundle spelling to an exact app name", async () => {
|
||||
const dotted = { ...app, bundleId: "com.example.Other", name: "Foo.Bar" };
|
||||
const output: string[] = [];
|
||||
expect(await runComputer(["hotkey", "--app", "Foo.Bar", "--keys", "cmd+k", "--json"], { adapter: { ...adapter, listApps: async () => ({ apps: [dotted] }), hotkey: async (input) => ({ action: "hotkey", app: input.app, snapshotId: null, elementIndex: null, fromElementIndex: null, toElementIndex: null, performed: true }) }, stdout: (text) => output.push(text) })).toBe(0);
|
||||
expect(JSON.parse(output[0]!)).toMatchObject({ result: { app: { name: "Foo.Bar" } } });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,34 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
COMPUTER_ACTIONS, COMPUTER_ERROR_CODES, COMPUTER_SUBCOMMANDS, COMPUTER_TIMEOUTS, SNAPSHOT_STALE_REASONS,
|
||||
failureEnvelope, isActionResult, isAppStateResult, isPermissionsResult, isValidSnapshotId, secretValue,
|
||||
successEnvelope, targetKeyForApp, targetKeySlug, windowKeyFor,
|
||||
} from "../computer/contract.js";
|
||||
|
||||
const locator = { kind: "ax-path" as const, path: "window[0]/button[1]", role: "AXButton", subrole: null, identifier: null, title: "OK" };
|
||||
describe("computer contract", () => {
|
||||
it("has frozen, runtime inventories and stable envelope", () => {
|
||||
expect(COMPUTER_SUBCOMMANDS).toEqual(["capabilities", "permissions", "list-apps", "list-windows", "get-app-state", "click", "set-value", "type-text", "press-key", "hotkey", "scroll", "drag"]);
|
||||
for (const item of [COMPUTER_ACTIONS, COMPUTER_ERROR_CODES, COMPUTER_SUBCOMMANDS, SNAPSHOT_STALE_REASONS]) {
|
||||
expect(Object.isFrozen(item)).toBe(true); expect(new Set(item).size).toBe(item.length);
|
||||
}
|
||||
expect(successEnvelope("computer.click", { x: 1 })).toMatchObject({ schemaVersion: 1, ok: true, command: "computer.click" });
|
||||
expect(failureEnvelope("computer", { code: "INVALID_ARGUMENTS", message: "Bad arguments" })).toMatchObject({ schemaVersion: 1, ok: false, command: "computer" });
|
||||
expect(COMPUTER_TIMEOUTS).toMatchObject({ permissionProbe: 5000, discovery: 10000, stateCapture: 20000, screenshotCapture: 15000, locatorReplay: 10000, action: 10000 });
|
||||
});
|
||||
it("validates ids, permissions, state and action index shapes", () => {
|
||||
expect(isValidSnapshotId("cs_AbCdEf1234")).toBe(true); expect(isValidSnapshotId("cs_../evil")).toBe(false);
|
||||
expect(isPermissionsResult({ allGranted: false, checks: [{ status: "granted", granted: true, probed: false }] })).toBe(false);
|
||||
const state = { snapshot: { elements: [{ index: 7, locator }] }, screenshot: { path: "x", width: null, height: null, verifiedPermission: true }, screenshotError: { code: "SCREENSHOT_FAILED", message: "no" } };
|
||||
expect(isAppStateResult(state)).toBe(false);
|
||||
expect(isActionResult({ action: "drag", performed: true, snapshotId: null, elementIndex: 1, fromElementIndex: 0, toElementIndex: 1 })).toBe(false);
|
||||
expect(isActionResult({ action: "hotkey", performed: true, snapshotId: "cs_AbCdEf1234", elementIndex: null, fromElementIndex: null, toElementIndex: null })).toBe(false);
|
||||
});
|
||||
it("keeps app and window addressing separate", () => {
|
||||
const app = { bundleId: "com.apple.Safari", name: "Safari", pid: 2 };
|
||||
const key = targetKeyForApp(app); expect(key).toBe("bundle:com.apple.Safari");
|
||||
expect(windowKeyFor(key, "one")).not.toBe(windowKeyFor(key, "two"));
|
||||
expect(targetKeySlug("bundle:../../x")).not.toContain("/");
|
||||
expect(secretValue("very-secret")).toBeTruthy();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,87 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { ComputerUseError, type AppRef, type Element, type WindowRef } from "../computer/contract.js";
|
||||
import { ComputerSnapshotStore } from "../computer/snapshot-store.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); });
|
||||
|
||||
const app: AppRef = { bundleId: "com.example.Editor", name: "Editor", pid: 41 };
|
||||
const window: WindowRef = { windowId: "w-1", windowIndex: 1, title: "Editor", bounds: null, minimized: false };
|
||||
const element = (index: number): Element => ({
|
||||
index, role: "AXButton", title: `button-${index}`, value: null, label: null, enabled: true, focused: false,
|
||||
bounds: null, actions: ["AXPress"], locator: { kind: "ax-path", path: `AXWindow[0]/AXButton[${index}]`, role: "AXButton", subrole: null, identifier: null, title: `button-${index}` },
|
||||
});
|
||||
|
||||
async function fixture(now = new Date("2026-08-11T03:34:00.000Z")) {
|
||||
const root = await mkdtemp(join(tmpdir(), "computer-snapshot-"));
|
||||
roots.push(root);
|
||||
let clock = now;
|
||||
return {
|
||||
root,
|
||||
setNow(value: Date) { clock = value; },
|
||||
store: new ComputerSnapshotStore({ projectRoot: root, now: () => clock }),
|
||||
};
|
||||
}
|
||||
|
||||
function expectComputerError(error: unknown, code: string, reason?: string) {
|
||||
expect(error).toBeInstanceOf(ComputerUseError);
|
||||
const computerError = error as ComputerUseError;
|
||||
expect(computerError.code).toBe(code);
|
||||
if (reason) expect(computerError.details?.reason).toBe(reason);
|
||||
expect(computerError.message).toContain("get-app-state");
|
||||
}
|
||||
|
||||
describe("ComputerSnapshotStore", () => {
|
||||
it("persists sparse indexes and rejects indexes derived from elementCount", async () => {
|
||||
const { store, root } = await fixture();
|
||||
const record = await store.persist({ app, window, elementCount: 3, elements: [element(0), element(3), element(7)] });
|
||||
|
||||
expect(store.getElement(record, 7).index).toBe(7);
|
||||
try { store.getElement(record, 2); } catch (error) { expectComputerError(error, "ELEMENT_INDEX_NOT_FOUND"); }
|
||||
expect(JSON.parse(await readFile(join(root, ".fusion/computer-use/snapshots", `${record.snapshotId}.json`), "utf8")).elements["7"].index).toBe(7);
|
||||
});
|
||||
|
||||
it("uses the required C9 freshness order and remediation", async () => {
|
||||
const { store, setNow } = await fixture();
|
||||
await expect(store.resolve({ app })).rejects.toMatchObject({ code: "SNAPSHOT_REQUIRED" });
|
||||
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
const second = await store.persist({ app, window, elementCount: 1, elements: [element(7)] });
|
||||
await expect(store.resolve({ app, snapshotId: first.snapshotId })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "superseded", snapshotId: first.snapshotId } });
|
||||
await expect(store.resolve({ app, snapshotId: "cs_0000000000missing" })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "not-found", snapshotId: "cs_0000000000missing" } });
|
||||
|
||||
setNow(new Date(Date.parse(second.expiresAt) + 1));
|
||||
await expect(store.resolve({ app, snapshotId: second.snapshotId })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "expired", snapshotId: second.snapshotId } });
|
||||
});
|
||||
|
||||
it("orders superseded before expiry and checks pid and asserted window after freshness", async () => {
|
||||
const { store, setNow } = await fixture();
|
||||
const first = await store.persist({ app, window, elementCount: 1, elements: [element(0)] });
|
||||
const second = await store.persist({ app, window, elementCount: 1, elements: [element(0)] });
|
||||
setNow(new Date(Date.parse(second.expiresAt) + 1));
|
||||
await expect(store.resolve({ app, snapshotId: first.snapshotId })).rejects.toMatchObject({ details: { reason: "superseded" } });
|
||||
|
||||
const live = await fixture();
|
||||
const record = await live.store.persist({ app, window, elementCount: 1, elements: [element(0)] });
|
||||
await expect(live.store.resolve({ app: { ...app, pid: 99 }, snapshotId: record.snapshotId })).rejects.toMatchObject({ details: { reason: "pid-changed", snapshotId: record.snapshotId } });
|
||||
await expect(live.store.resolve({ app, snapshotId: record.snapshotId, assertedWindowId: "other-window" })).rejects.toMatchObject({ details: { reason: "window-mismatch", snapshotId: record.snapshotId } });
|
||||
});
|
||||
|
||||
it("treats absent or unparsable named records as not-found", async () => {
|
||||
const { store, root } = await fixture();
|
||||
const record = await store.persist({ app, window, elementCount: 1, elements: [element(0)] });
|
||||
await writeFile(join(root, ".fusion/computer-use/snapshots", `${record.snapshotId}.json`), "not json");
|
||||
await expect(store.resolve({ app, snapshotId: record.snapshotId })).rejects.toMatchObject({ code: "SNAPSHOT_STALE", details: { reason: "not-found", snapshotId: record.snapshotId } });
|
||||
});
|
||||
|
||||
it("prunes expired records with a single-level snapshots directory read", async () => {
|
||||
const { store, root, setNow } = await fixture();
|
||||
const old = await store.persist({ app, window, elementCount: 1, elements: [element(0)], expiresAt: "2026-08-11T03:33:00.000Z" });
|
||||
setNow(new Date("2026-08-11T03:35:00.000Z"));
|
||||
await store.prune();
|
||||
await expect(readFile(join(root, ".fusion/computer-use/snapshots", `${old.snapshotId}.json`), "utf8")).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
182
packages/cli/src/commands/computer.ts
Normal file
182
packages/cli/src/commands/computer.ts
Normal file
@@ -0,0 +1,182 @@
|
||||
import { result } from "../output.js";
|
||||
import { MacosComputerAdapter } from "./computer/adapter-macos.js";
|
||||
import { resolveComputerAdapter, type ComputerClock } from "./computer/adapter-registry.js";
|
||||
import type { ComputerAdapter, ResolvedComputerElement, ResolvedComputerWindow } from "./computer/adapter.js";
|
||||
import { createComputerSnapshotStore, type ComputerSnapshotStore } from "./computer/snapshot-store.js";
|
||||
import { COMPUTER_SUBCOMMANDS, ComputerUseError, failureEnvelope, isValidSnapshotId, parseAppTarget, successEnvelope, validateResult, type AppRef, type CommandName, type ComputerSubcommand } from "./computer/contract.js";
|
||||
|
||||
export interface ComputerCommandOptions { platform?: string; projectRoot?: string; adapter?: ComputerAdapter; store?: ComputerSnapshotStore; clock?: ComputerClock; stdout?: (text: string) => void; stderr?: (text: string) => void; stdin?: () => Promise<string>; }
|
||||
export type ComputerHandler = (args: string[], options: ComputerCommandOptions) => Promise<unknown>;
|
||||
const command = (subcommand: ComputerSubcommand): CommandName => `computer.${subcommand}`;
|
||||
const emit = (value: unknown, json: boolean, options: ComputerCommandOptions): void => { const write = options.stdout ?? result; if (json) write(`${JSON.stringify(value)}\n`); else write(`${JSON.stringify(value, null, 2)}\n`); };
|
||||
const fail = (name: CommandName, error: unknown, json: boolean, options: ComputerCommandOptions): number => { const e = error instanceof ComputerUseError ? error : new ComputerUseError("INTERNAL", "Computer command failed unexpectedly."); const envelope = failureEnvelope(name, e); if (json) emit(envelope, true, options); else (options.stderr ?? console.error)(`error: ${e.code}: ${e.message}${e.remediation ? `\n${e.remediation}` : ""}`); return 1; };
|
||||
function value(args: string[], flag: string): string | undefined { const i = args.indexOf(flag); return i >= 0 && args[i + 1] && !args[i + 1].startsWith("--") ? args[i + 1] : undefined; }
|
||||
function number(args: string[], flag: string): number | undefined { const raw = value(args, flag); if (raw === undefined) return undefined; const parsed = Number(raw); if (!Number.isInteger(parsed) || parsed < 0) throw new ComputerUseError("INVALID_ARGUMENTS", `Invalid ${flag}.`); return parsed; }
|
||||
function adapterFor(options: ComputerCommandOptions): ComputerAdapter { if (options.adapter) return options.adapter; const root = options.projectRoot ?? process.cwd(); return resolveComputerAdapter({ platform: options.platform, projectRoot: root, clock: options.clock, macosAdapterFactory: ({ seam, clock, projectRoot }) => new MacosComputerAdapter({ seam, clock, projectRoot }) }); }
|
||||
function storeFor(options: ComputerCommandOptions): ComputerSnapshotStore { return options.store ?? createComputerSnapshotStore({ projectRoot: options.projectRoot, now: options.clock ? () => options.clock!.now() : undefined }); }
|
||||
async function appFor(adapter: ComputerAdapter, raw: string): Promise<AppRef> { const target = parseAppTarget(raw); const apps = (await adapter.listApps()).apps; const matches = target.kind === "pid" ? apps.filter((x) => String(x.pid) === target.value) : (() => { const byBundle = apps.filter((x) => x.bundleId === target.value); return byBundle.length ? byBundle : apps.filter((x) => x.name === target.value); })(); if (!matches.length) throw new ComputerUseError("APP_NOT_FOUND", `No running app matches ${raw}.`); if (matches.length > 1) throw new ComputerUseError("AMBIGUOUS_APP", `More than one app matches ${raw}.`, "Use a bundle id or pid target.", { candidateCount: matches.length }); return matches[0]; }
|
||||
async function requireElements(args: string[], indexes: readonly number[], adapter: ComputerAdapter, store: ComputerSnapshotStore, app: AppRef): Promise<{ record: Awaited<ReturnType<ComputerSnapshotStore["resolve"]>>; window: ResolvedComputerWindow; elements: ResolvedComputerElement[] }> {
|
||||
if (!indexes.length) throw new ComputerUseError("INVALID_ARGUMENTS", "At least one --element-index is required.");
|
||||
const id = value(args, "--snapshot-id");
|
||||
if (id !== undefined && !isValidSnapshotId(id)) throw new ComputerUseError("INVALID_ARGUMENTS", "Invalid --snapshot-id.", undefined, { reason: "snapshot-id-format" });
|
||||
const windowId = value(args, "--window-id");
|
||||
const windowIndex = number(args, "--window-index");
|
||||
if (windowId && windowIndex !== undefined) throw new ComputerUseError("INVALID_ARGUMENTS", "Window flags are mutually exclusive.");
|
||||
// A supplied window index is resolved before snapshot lookup, then asserted against its recorded window.
|
||||
const target = parseAppTarget(app.bundleId ?? `pid:${app.pid}`);
|
||||
const assertedWindowId = windowId ?? (windowIndex !== undefined ? (await adapter.resolveWindow(target, { windowIndex })).window.windowId : undefined);
|
||||
// Resolve the optimistic snapshot fence once. Pair actions must never mix endpoints from concurrent latest-pointer updates.
|
||||
const record = await store.resolve({ app, snapshotId: id, assertedWindowId });
|
||||
const rawElements = indexes.map((index) => store.getElement(record, index));
|
||||
let window: ResolvedComputerWindow;
|
||||
try { window = await adapter.resolveWindow(target, { windowId: record.window.windowId }); }
|
||||
catch (error) { if (error instanceof ComputerUseError && error.code === "WINDOW_NOT_FOUND") throw new ComputerUseError("SNAPSHOT_STALE", "Snapshot window no longer exists; re-run fn computer get-app-state.", "Re-run fn computer get-app-state.", { reason: "window-gone", snapshotId: record.snapshotId }); throw error; }
|
||||
const elements = await Promise.all(rawElements.map(async (raw) => {
|
||||
try {
|
||||
const replayed = await adapter.resolveLocator(window, raw.locator);
|
||||
// Preserve sparse snapshot identity while using bounds observed during this replay.
|
||||
return { ...replayed, element: { ...raw, ...replayed.element, index: raw.index, locator: raw.locator } };
|
||||
} catch (error) {
|
||||
if (error instanceof ComputerUseError && error.code === "ELEMENT_UNRESOLVABLE") {
|
||||
throw new ComputerUseError(error.code, error.message, error.remediation, { snapshotId: record.snapshotId, elementIndex: raw.index });
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}));
|
||||
return { record, window, elements };
|
||||
}
|
||||
async function requireElement(args: string[], adapter: ComputerAdapter, store: ComputerSnapshotStore, app: AppRef): Promise<{ record: Awaited<ReturnType<ComputerSnapshotStore["resolve"]>>; window: ResolvedComputerWindow; element: ResolvedComputerElement }> {
|
||||
const index = number(args, "--element-index");
|
||||
if (index === undefined) throw new ComputerUseError("INVALID_ARGUMENTS", "--element-index is required.");
|
||||
const resolved = await requireElements(args, [index], adapter, store, app);
|
||||
return { record: resolved.record, window: resolved.window, element: resolved.elements[0]! };
|
||||
}
|
||||
async function optionalElement(args: string[], adapter: ComputerAdapter, store: ComputerSnapshotStore, app: AppRef) { return value(args, "--element-index") === undefined ? undefined : requireElement(args, adapter, store, app); }
|
||||
export const COMPUTER_HANDLERS: Record<ComputerSubcommand, ComputerHandler> = {
|
||||
capabilities: async (_args, o) => adapterFor(o).capabilities(), permissions: async (_args, o) => adapterFor(o).permissions(),
|
||||
"list-apps": async (_args, o) => adapterFor(o).listApps(),
|
||||
"list-windows": async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); const adapter = adapterFor(o); return adapter.listWindows(parseAppTarget(raw)); },
|
||||
"get-app-state": async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); if (value(args, "--window-id") && value(args, "--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Window flags are mutually exclusive."); const adapter = adapterFor(o); const state = await adapter.captureState(parseAppTarget(raw), { windowId: value(args, "--window-id"), windowIndex: number(args, "--window-index"), screenshot: !args.includes("--no-screenshot"), restoreWindow: args.includes("--restore-window") }); const record = await storeFor(o).persist({ app: state.app, window: state.window, elementCount: state.snapshot.elementCount, elements: state.snapshot.elements, capturedAt: state.snapshot.capturedAt }); state.snapshot.snapshotId = record.snapshotId; state.snapshot.targetKey = record.targetKey; state.snapshot.windowKey = record.windowKey; state.snapshot.capturedAt = record.capturedAt; state.snapshot.expiresAt = record.expiresAt; return state; },
|
||||
click: async (args, o) => { const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await requireElement(args, adapter, storeFor(o), app); return adapter.click({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId }); },
|
||||
"set-value": async (args, o) => { const raw = value(args, "--app"), text = value(args, "--value"); if (!raw || (!text && !args.includes("--value-stdin")) || (text && args.includes("--value-stdin"))) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and exactly one value source are required."); const secret = args.includes("--value-stdin") ? await (o.stdin ?? (async () => ""))() : text!; const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await requireElement(args, adapter, storeFor(o), app); return adapter["set-value"]({ app, window: item.window, element: item.element, snapshotId: item.record.snapshotId, value: secret }); },
|
||||
"type-text": async (args, o) => targetedOrUntargeted("type-text", args, o), "press-key": async (args, o) => targetedOrUntargeted("press-key", args, o), scroll: async (args, o) => targetedOrUntargeted("scroll", args, o),
|
||||
hotkey: async (args, o) => { const raw = value(args, "--app"), keys = value(args, "--keys"); if (!raw || !keys) throw new ComputerUseError("INVALID_ARGUMENTS", "--app and --keys are required."); const adapter = adapterFor(o), app = await appFor(adapter, raw); return adapter.hotkey({ app, keys: keys.split("+") }); },
|
||||
drag: async (args, o) => {
|
||||
const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required.");
|
||||
const coordinateFlags = ["--from-x", "--from-y", "--to-x", "--to-y"];
|
||||
const hasCoordinates = coordinateFlags.some((flag) => value(args, flag) !== undefined);
|
||||
const from = number(args, "--from-element-index"), to = number(args, "--to-element-index");
|
||||
const hasElements = from !== undefined || to !== undefined;
|
||||
if (hasCoordinates && hasElements) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate and element drag forms are mutually exclusive.");
|
||||
const adapter = adapterFor(o), app = await appFor(adapter, raw);
|
||||
if (hasCoordinates) {
|
||||
const coordinates = coordinateFlags.map((flag) => number(args, flag));
|
||||
if (coordinates.some((item) => item === undefined) || value(args, "--snapshot-id") || value(args, "--window-id") || value(args, "--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate drag requires all coordinates and takes no snapshot or window flags.");
|
||||
return adapter.drag({ app, snapshotId: null, fromX: coordinates[0]!, fromY: coordinates[1]!, toX: coordinates[2]!, toY: coordinates[3]! });
|
||||
}
|
||||
if (from === undefined || to === undefined) throw new ComputerUseError("INVALID_ARGUMENTS", "Drag requires either all coordinates or both element indexes.");
|
||||
// Both endpoints share one resolved record and one replayed window, even if another capture updates latest mid-action.
|
||||
const resolved = await requireElements(args, [from, to], adapter, storeFor(o), app);
|
||||
return adapter.drag({ app, snapshotId: resolved.record.snapshotId, window: resolved.window, from: resolved.elements[0]!, to: resolved.elements[1]! });
|
||||
},
|
||||
};
|
||||
function validateFlags(name: ComputerSubcommand, args: string[]): void {
|
||||
const present = (flag: string) => args.includes(flag);
|
||||
const supplied = (flag: string) => {
|
||||
if (!present(flag)) return undefined;
|
||||
const parsed = value(args, flag);
|
||||
if (parsed === undefined) throw new ComputerUseError("INVALID_ARGUMENTS", `${flag} requires a value.`);
|
||||
return parsed;
|
||||
};
|
||||
const requiredApp = () => { if (!supplied("--app")) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required."); };
|
||||
const validateWindow = () => {
|
||||
supplied("--window-id");
|
||||
if (present("--window-index")) number(args, "--window-index");
|
||||
if (present("--window-id") && present("--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Window flags are mutually exclusive.");
|
||||
};
|
||||
const validateSnapshot = () => { if (present("--snapshot-id") && !isValidSnapshotId(supplied("--snapshot-id"))) throw new ComputerUseError("INVALID_ARGUMENTS", "Invalid --snapshot-id.", undefined, { reason: "snapshot-id-format" }); };
|
||||
const validateElement = (flag = "--element-index") => { if (!present(flag)) throw new ComputerUseError("INVALID_ARGUMENTS", `${flag} is required.`); number(args, flag); };
|
||||
if (name === "capabilities" || name === "permissions" || name === "list-apps") return;
|
||||
requiredApp();
|
||||
if (name === "list-windows") return;
|
||||
if (name === "get-app-state") { validateWindow(); return; }
|
||||
if (name === "click") { validateElement(); validateWindow(); validateSnapshot(); return; }
|
||||
if (name === "set-value") {
|
||||
validateElement(); validateWindow(); validateSnapshot();
|
||||
if (present("--value")) supplied("--value");
|
||||
if (present("--value") === present("--value-stdin")) throw new ComputerUseError("INVALID_ARGUMENTS", "Exactly one value source is required.");
|
||||
return;
|
||||
}
|
||||
if (name === "type-text" || name === "press-key" || name === "scroll") {
|
||||
const hasElement = present("--element-index");
|
||||
if (hasElement) number(args, "--element-index");
|
||||
validateWindow(); validateSnapshot();
|
||||
if (!hasElement && (present("--snapshot-id") || present("--window-id") || present("--window-index"))) throw new ComputerUseError("INVALID_ARGUMENTS", "Snapshot and window flags require --element-index.");
|
||||
if (name === "type-text") {
|
||||
if (present("--text")) supplied("--text");
|
||||
if (present("--text") === present("--text-stdin")) throw new ComputerUseError("INVALID_ARGUMENTS", "Exactly one text source is required.");
|
||||
} else if (name === "press-key") { if (!supplied("--key")) throw new ComputerUseError("INVALID_ARGUMENTS", "--key is required."); }
|
||||
else {
|
||||
const direction = supplied("--direction");
|
||||
if (!direction || !["up", "down", "left", "right"].includes(direction)) throw new ComputerUseError("INVALID_ARGUMENTS", "A valid --direction is required.");
|
||||
if (present("--amount")) number(args, "--amount");
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (name === "hotkey") {
|
||||
if (!supplied("--keys")) throw new ComputerUseError("INVALID_ARGUMENTS", "--keys is required.");
|
||||
if (present("--snapshot-id") || present("--window-id") || present("--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Hotkey takes no snapshot or window flags.");
|
||||
return;
|
||||
}
|
||||
// Drag's mutually exclusive forms must be validated before resolving the app.
|
||||
const coordinateFlags = ["--from-x", "--from-y", "--to-x", "--to-y"];
|
||||
const anyCoordinates = coordinateFlags.some(present);
|
||||
const anyElements = present("--from-element-index") || present("--to-element-index");
|
||||
if (anyCoordinates && anyElements) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate and element drag forms are mutually exclusive.");
|
||||
if (anyCoordinates) {
|
||||
if (!coordinateFlags.every(present)) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate drag requires all coordinates.");
|
||||
coordinateFlags.forEach((flag) => number(args, flag));
|
||||
if (present("--snapshot-id") || present("--window-id") || present("--window-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Coordinate drag takes no snapshot or window flags.");
|
||||
} else {
|
||||
if (!present("--from-element-index") || !present("--to-element-index")) throw new ComputerUseError("INVALID_ARGUMENTS", "Drag requires either all coordinates or both element indexes.");
|
||||
number(args, "--from-element-index"); number(args, "--to-element-index"); validateWindow(); validateSnapshot();
|
||||
}
|
||||
}
|
||||
|
||||
async function targetedOrUntargeted(kind: "type-text" | "press-key" | "scroll", args: string[], o: ComputerCommandOptions): Promise<unknown> {
|
||||
const raw = value(args, "--app"); if (!raw) throw new ComputerUseError("INVALID_ARGUMENTS", "--app is required.");
|
||||
const adapter = adapterFor(o), app = await appFor(adapter, raw), item = await optionalElement(args, adapter, storeFor(o), app);
|
||||
if (!item && (value(args, "--snapshot-id") || value(args, "--window-id") || value(args, "--window-index"))) throw new ComputerUseError("INVALID_ARGUMENTS", "Snapshot and window flags require --element-index.");
|
||||
if (kind === "type-text") {
|
||||
const direct = value(args, "--text"), fromStdin = args.includes("--text-stdin");
|
||||
if ((direct === undefined && !fromStdin) || (direct !== undefined && fromStdin)) throw new ComputerUseError("INVALID_ARGUMENTS", "Exactly one text source is required.");
|
||||
const text = fromStdin ? await (o.stdin ?? (async () => ""))() : direct!;
|
||||
return adapter["type-text"]({ app, text, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) });
|
||||
}
|
||||
if (kind === "press-key") { const key = value(args, "--key"); if (!key) throw new ComputerUseError("INVALID_ARGUMENTS", "--key is required."); return adapter["press-key"]({ app, key, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) }); }
|
||||
const direction = value(args, "--direction"); if (!direction || !["up", "down", "left", "right"].includes(direction)) throw new ComputerUseError("INVALID_ARGUMENTS", "A valid --direction is required.");
|
||||
return adapter.scroll({ app, direction: direction as "up" | "down" | "left" | "right", amount: number(args, "--amount") ?? 3, ...(item ? { window: item.window, element: item.element, snapshotId: item.record.snapshotId } : {}) });
|
||||
}
|
||||
export async function runComputer(args: string[], options: ComputerCommandOptions = {}): Promise<number> {
|
||||
const json = args.includes("--json");
|
||||
const subIndex = args.findIndex((item) => item !== "--json");
|
||||
const sub = subIndex < 0 ? undefined : args[subIndex];
|
||||
if (args.includes("--help") || args.includes("-h")) {
|
||||
emit("fn computer <capabilities|permissions|list-apps|list-windows|get-app-state|click|set-value|type-text|press-key|hotkey|scroll|drag>\nUse snapshot → act → snapshot; --snapshot-id fences the latest capture.", false, options);
|
||||
return 0;
|
||||
}
|
||||
if (!sub) return json
|
||||
? fail("computer", new ComputerUseError("INVALID_ARGUMENTS", "A computer subcommand is required."), true, options)
|
||||
: (emit("fn computer <capabilities|permissions|list-apps|list-windows|get-app-state|click|set-value|type-text|press-key|hotkey|scroll|drag>\nUse snapshot → act → snapshot; --snapshot-id fences the latest capture.", false, options), 0);
|
||||
if (!(COMPUTER_SUBCOMMANDS as readonly string[]).includes(sub)) return fail("computer", new ComputerUseError("INVALID_ARGUMENTS", `Unknown computer subcommand: ${sub}.`), json, options);
|
||||
const name = sub as ComputerSubcommand;
|
||||
const handlerArgs = args.filter((_item, index) => index !== subIndex && _item !== "--json");
|
||||
try {
|
||||
// C10 stage 2 is deliberately complete and precedes adapter construction, filesystem, and OS discovery.
|
||||
validateFlags(name, handlerArgs);
|
||||
const payload = await COMPUTER_HANDLERS[name](handlerArgs, options);
|
||||
if (!validateResult(name, payload, handlerArgs.includes("--no-screenshot"))) throw new ComputerUseError("INTERNAL", "Computer command produced an invalid contract result.");
|
||||
emit(successEnvelope(command(name), payload), json, options);
|
||||
return 0;
|
||||
} catch (error) { return fail(command(name), error, json, options); }
|
||||
}
|
||||
211
packages/cli/src/commands/computer/adapter-macos.ts
Normal file
211
packages/cli/src/commands/computer/adapter-macos.ts
Normal file
@@ -0,0 +1,211 @@
|
||||
import { mkdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import type { ClickInput, ComputerAdapter, ComputerStateCaptureOptions, DragInput, HotkeyInput, PressKeyInput, ResolvedComputerElement, ResolvedComputerWindow, ScrollInput, SetValueInput, TypeTextInput } from "./adapter.js";
|
||||
import type { ComputerClock } from "./adapter-registry.js";
|
||||
import type { ComputerExecSeam } from "./exec-seam.js";
|
||||
import { COMPUTER_ACTIONS, COMPUTER_TIMEOUTS, ComputerUseError, type ActionResult, type AppRef, type AppStateResult, type AppTarget, type CapabilitiesResult, type Element, type ElementLocator, type ListAppsResult, type ListWindowsResult, type PermissionCheck, type PermissionsResult, type WindowRef } from "./contract.js";
|
||||
import { MACOS_AUTOMATION, MACOS_PERMISSION_PREFLIGHT } from "./scripts/macos-jxa.js";
|
||||
|
||||
const REMEDIATION_ACCESSIBILITY = "Grant the host terminal or app in System Settings → Privacy & Security → Accessibility, then re-run fn computer permissions.";
|
||||
const REMEDIATION_SCREEN = "Grant the host terminal or app in System Settings → Privacy & Security → Screen Recording, then re-run fn computer permissions.";
|
||||
const MISSING_OSASCRIPT_DETAIL = "The macOS osascript built-in is unavailable.";
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-11-03:54:
|
||||
* Permission reporting uses only non-mutating TCC preflight. A denied screencapture can still save
|
||||
* wallpaper pixels, so capture success is not proof of Screen Recording permission. Denied access
|
||||
* refuses before automation while unknown attempts the operation and classifies an authorization failure.
|
||||
*/
|
||||
export class MacosComputerAdapter implements ComputerAdapter {
|
||||
readonly platform = "darwin";
|
||||
readonly id = "macos-osascript-system-events";
|
||||
readonly supported = true;
|
||||
constructor(private readonly context: { seam: ComputerExecSeam; clock: ComputerClock; projectRoot: string }) {}
|
||||
|
||||
async capabilities(): Promise<CapabilitiesResult> {
|
||||
return { platform: this.platform, adapterId: this.id, supported: true, actions: [...COMPUTER_ACTIONS], unsupportedActions: [], features: { screenshot: true, restoreWindow: true, stdinSecrets: true, crossInvocationSnapshots: true } };
|
||||
}
|
||||
|
||||
async permissions(): Promise<PermissionsResult> {
|
||||
const fallback = (detail: string): PermissionsResult => ({ platform: this.platform, adapterId: this.id, supported: true, allGranted: false, checks: [check("accessibility", "unknown", false, detail, REMEDIATION_ACCESSIBILITY), check("screen-recording", "unknown", false, detail, REMEDIATION_SCREEN)] });
|
||||
let run;
|
||||
try {
|
||||
run = await this.context.seam.run("osascript", ["-l", "JavaScript", "-e", MACOS_PERMISSION_PREFLIGHT], { timeoutMs: COMPUTER_TIMEOUTS.permissionProbe });
|
||||
} catch (error) {
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-11-04:29:
|
||||
* A missing osascript cannot prove either TCC grant, so permissions remains unknown. Every
|
||||
* automation path then fails C7.4 before acting instead of exposing a spawn failure as INTERNAL.
|
||||
*/
|
||||
return fallback(isMissingBinary(error, "osascript") ? MISSING_OSASCRIPT_DETAIL : "Permission preflight could not run.");
|
||||
}
|
||||
if (run.timedOut) return fallback("Permission preflight timed out.");
|
||||
if (run.exitCode !== 0) return fallback("Permission preflight could not run.");
|
||||
try {
|
||||
const values = JSON.parse(run.stdout) as { accessibility: boolean; screenRecording: boolean };
|
||||
if (typeof values.accessibility !== "boolean" || typeof values.screenRecording !== "boolean") return fallback("Permission preflight returned invalid output.");
|
||||
const checks = [check("accessibility", values.accessibility ? "granted" : "denied", true, null, REMEDIATION_ACCESSIBILITY), check("screen-recording", values.screenRecording ? "granted" : "denied", true, null, REMEDIATION_SCREEN)];
|
||||
return { platform: this.platform, adapterId: this.id, supported: true, allGranted: checks.every((item) => item.granted), checks };
|
||||
} catch { return fallback("Permission preflight returned invalid output."); }
|
||||
}
|
||||
|
||||
async listApps(): Promise<ListAppsResult> {
|
||||
await this.assertAccessibility();
|
||||
const value = await this.callJson(["list-apps"], COMPUTER_TIMEOUTS.discovery);
|
||||
const apps = Array.isArray(value.apps) ? value.apps.filter(isAppRef) : [];
|
||||
return { apps: apps.sort((a, b) => a.name.localeCompare(b.name)) };
|
||||
}
|
||||
|
||||
async listWindows(target: AppTarget): Promise<ListWindowsResult> {
|
||||
await this.assertAccessibility();
|
||||
const app = await this.resolveApp(target);
|
||||
const value = await this.callJson(["list-windows", target.raw], COMPUTER_TIMEOUTS.discovery);
|
||||
return { app, windows: Array.isArray(value.windows) ? value.windows.filter(isWindowRef) : [] };
|
||||
}
|
||||
|
||||
async captureState(target: AppTarget, options: ComputerStateCaptureOptions): Promise<AppStateResult> {
|
||||
await this.assertAccessibility();
|
||||
const app = await this.resolveApp(target);
|
||||
const resolved = await this.resolveWindow(target, { windowId: options.windowId, windowIndex: options.windowIndex });
|
||||
if (options.restoreWindow) await this.callJson(["restore-window", target.raw, resolved.window.windowId, String(resolved.window.windowIndex)], COMPUTER_TIMEOUTS.stateCapture);
|
||||
const value = await this.callJson(["state", target.raw, resolved.window.windowId, String(resolved.window.windowIndex)], COMPUTER_TIMEOUTS.stateCapture);
|
||||
const elements = Array.isArray(value.elements) ? value.elements.filter(isElement) : [];
|
||||
const snapshot = { snapshotId: "", targetKey: "", windowKey: "", capturedAt: this.context.clock.now().toISOString(), expiresAt: "", treeText: typeof value.treeText === "string" ? value.treeText : "", elementCount: elements.length, truncated: value.truncated === true, elements };
|
||||
let screenshot: AppStateResult["screenshot"] = null;
|
||||
let screenshotError: AppStateResult["screenshotError"];
|
||||
if (options.screenshot) {
|
||||
const screen = (await this.permissions()).checks.find((item) => item.id === "screen-recording");
|
||||
if (screen?.status === "denied") screenshotError = { code: "PERMISSION_DENIED", message: REMEDIATION_SCREEN };
|
||||
else {
|
||||
const directory = join(this.context.projectRoot, ".fusion", "computer-use", "screenshots");
|
||||
const file = join(directory, `computer-${this.context.clock.now().getTime()}.png`);
|
||||
// FNXC:ComputerUse 2026-08-11-04:42: Screenshot I/O is optional state decoration. A
|
||||
// filesystem or screencapture spawn failure must preserve the captured accessibility tree
|
||||
// so the command can persist its snapshot and report the failure only as screenshotError.
|
||||
try {
|
||||
await mkdir(directory, { recursive: true });
|
||||
// FNXC:ComputerUse 2026-08-11-04:12: screencapture -l accepts only a verified Quartz
|
||||
// CGWindowID, never an AX id or a System Events index. Without the identity-mapped ID this
|
||||
// optional capture degrades rather than selecting an unrelated window on a numeric collision.
|
||||
const captureWindowId = (resolved.handle as { captureWindowId?: unknown }).captureWindowId;
|
||||
if (typeof captureWindowId !== "number" || !Number.isSafeInteger(captureWindowId) || captureWindowId < 0) {
|
||||
screenshotError = { code: "SCREENSHOT_FAILED", message: "The selected window cannot be captured by screencapture." };
|
||||
} else {
|
||||
const output = await this.context.seam.run("screencapture", ["-x", "-l", String(captureWindowId), file], { timeoutMs: COMPUTER_TIMEOUTS.screenshotCapture });
|
||||
if (output.timedOut) screenshotError = { code: "TIMEOUT", message: "Screenshot capture timed out." };
|
||||
else if (output.exitCode !== 0) screenshotError = { code: screen?.status === "unknown" ? "PERMISSION_UNVERIFIED" : "SCREENSHOT_FAILED", message: "Screenshot capture failed." };
|
||||
else screenshot = { path: file, width: null, height: null, verifiedPermission: screen?.status === "granted" };
|
||||
}
|
||||
} catch (error) {
|
||||
screenshotError = {
|
||||
code: isMissingBinary(error, "screencapture") ? "SCREENSHOT_FAILED" : screen?.status === "unknown" ? "PERMISSION_UNVERIFIED" : "SCREENSHOT_FAILED",
|
||||
message: "Screenshot capture failed.",
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
return { app, window: resolved.window, snapshot, screenshot, ...(screenshotError ? { screenshotError } : {}) };
|
||||
}
|
||||
|
||||
async resolveWindow(target: AppTarget, selector: { windowId?: string; windowIndex?: number }): Promise<ResolvedComputerWindow> {
|
||||
const listed = await this.listWindows(target);
|
||||
const window = selector.windowId ? listed.windows.find((item) => item.windowId === selector.windowId) : selector.windowIndex !== undefined ? listed.windows.find((item) => item.windowIndex === selector.windowIndex) : listed.windows[0];
|
||||
if (!window) throw new ComputerUseError("WINDOW_NOT_FOUND", "The requested window was not found.");
|
||||
const captureWindowId = (window as WindowRef & { captureWindowId?: unknown }).captureWindowId;
|
||||
return { window, handle: { appTarget: target.raw, windowId: window.windowId, windowIndex: window.windowIndex, ...(typeof captureWindowId === "number" ? { captureWindowId } : {}) } };
|
||||
}
|
||||
|
||||
/** FNXC:ComputerUse 2026-08-11-03:54: A fresh CLI process must replay and identity-check the locator. It reports window-gone separately and never falls back to stale bounds or a replacement path occupant. */
|
||||
async resolveLocator(window: ResolvedComputerWindow, locator: ElementLocator): Promise<ResolvedComputerElement> {
|
||||
await this.assertAccessibility();
|
||||
const windowHandle = window.handle as { appTarget?: string; windowId?: string; windowIndex?: number };
|
||||
let result: Record<string, unknown>;
|
||||
try {
|
||||
result = await this.callJson(["resolve-locator", windowHandle.appTarget ?? "", windowHandle.windowId ?? window.window.windowId, String(windowHandle.windowIndex ?? window.window.windowIndex), locator.path], COMPUTER_TIMEOUTS.locatorReplay);
|
||||
} catch (error) {
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-11-04:29:
|
||||
* Locator replay is the cross-process identity fence. Preserve transport and permission
|
||||
* outcomes, but report a vanished AX path as ELEMENT_UNRESOLVABLE rather than ACTION_FAILED.
|
||||
*/
|
||||
if (error instanceof ComputerUseError && error.code === "ACTION_FAILED") {
|
||||
throw new ComputerUseError("ELEMENT_UNRESOLVABLE", "The captured element can no longer be resolved; re-run fn computer get-app-state.", "Re-run fn computer get-app-state.");
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const element = result.element;
|
||||
if (!isLiveElement(element) || element.role !== locator.role || (locator.subrole !== null && element.locator.subrole !== locator.subrole) || (locator.identifier !== null && element.locator.identifier !== locator.identifier)) {
|
||||
throw new ComputerUseError("ELEMENT_UNRESOLVABLE", "The captured element no longer matches; re-run fn computer get-app-state.", "Re-run fn computer get-app-state.");
|
||||
}
|
||||
return { element: { ...element, index: 0 } as Element, handle: locator.path };
|
||||
}
|
||||
|
||||
async click(input: ClickInput): Promise<ActionResult> { await this.perform("click", input.app, input.window, input.element); return singleAction("click", input.app, input.snapshotId, input.element.element.index); }
|
||||
async "set-value"(input: SetValueInput): Promise<ActionResult> { await this.perform("set-value", input.app, input.window, input.element, undefined, input.value); return singleAction("set-value", input.app, input.snapshotId, input.element.element.index); }
|
||||
async "type-text"(input: TypeTextInput): Promise<ActionResult> { await this.perform("type-text", input.app, input.window, input.element, undefined, input.text); return singleAction("type-text", input.app, input.snapshotId ?? null, input.element?.element.index ?? null); }
|
||||
async "press-key"(input: PressKeyInput): Promise<ActionResult> { await this.perform("press-key", input.app, input.window, input.element, input.key); return singleAction("press-key", input.app, input.snapshotId ?? null, input.element?.element.index ?? null); }
|
||||
async hotkey(input: HotkeyInput): Promise<ActionResult> { await this.assertAccessibility(); await this.callJson(["hotkey", input.app.name, input.keys[input.keys.length - 1] ?? "", ...input.keys.slice(0, -1)], COMPUTER_TIMEOUTS.action); return singleAction("hotkey", input.app, null, null); }
|
||||
async scroll(input: ScrollInput): Promise<ActionResult> { await this.perform("scroll", input.app, input.window, input.element, `${input.direction}:${input.amount}`); return singleAction("scroll", input.app, input.snapshotId ?? null, input.element?.element.index ?? null); }
|
||||
async drag(input: DragInput): Promise<ActionResult> {
|
||||
await this.assertAccessibility();
|
||||
if (input.from && input.to) {
|
||||
const from = center(input.from.element.bounds), to = center(input.to.element.bounds);
|
||||
if (!from || !to) throw new ComputerUseError("ACTION_FAILED", "Resolved drag elements have no current bounds.");
|
||||
// Bounds come from locator replay in this invocation, never the persisted snapshot fallback.
|
||||
await this.callJson(["drag-coordinates", input.app.name, String(from.x), String(from.y), String(to.x), String(to.y)], COMPUTER_TIMEOUTS.action);
|
||||
} else if (input.fromX !== undefined && input.fromY !== undefined && input.toX !== undefined && input.toY !== undefined) { await this.callJson(["drag-coordinates", input.app.name, String(input.fromX), String(input.fromY), String(input.toX), String(input.toY)], COMPUTER_TIMEOUTS.action); }
|
||||
else throw new ComputerUseError("ACTION_FAILED", "Drag requires coordinates or resolved elements.");
|
||||
return { action: "drag", app: input.app, snapshotId: input.snapshotId, elementIndex: null, fromElementIndex: input.from?.element.index ?? null, toElementIndex: input.to?.element.index ?? null, performed: true };
|
||||
}
|
||||
|
||||
private async assertAccessibility(): Promise<void> {
|
||||
const access = (await this.permissions()).checks.find((check) => check.id === "accessibility");
|
||||
if (access?.detail === MISSING_OSASCRIPT_DETAIL) {
|
||||
throw new ComputerUseError("PERMISSION_DENIED", "macOS osascript is unavailable.", "macOS with the osascript built-in is required for fn computer.", { missingBinary: "osascript" });
|
||||
}
|
||||
if (access?.status === "denied") throw new ComputerUseError("PERMISSION_DENIED", "Accessibility permission is denied.", REMEDIATION_ACCESSIBILITY, { permission: "accessibility" });
|
||||
}
|
||||
|
||||
private async resolveApp(target: AppTarget): Promise<AppRef> {
|
||||
const apps = (await this.listApps()).apps;
|
||||
const matches = target.kind === "pid"
|
||||
? apps.filter((app) => String(app.pid) === target.value)
|
||||
: (() => { const byBundle = apps.filter((app) => app.bundleId === target.value); return byBundle.length ? byBundle : apps.filter((app) => app.name === target.value); })();
|
||||
if (!matches.length) throw new ComputerUseError("APP_NOT_FOUND", `No running app matches ${target.raw}.`);
|
||||
if (matches.length > 1) throw new ComputerUseError("AMBIGUOUS_APP", `More than one app matches ${target.raw}.`, "Use a bundle identifier or pid target.", { candidateCount: matches.length });
|
||||
return matches[0];
|
||||
}
|
||||
|
||||
private async perform(operation: string, app: AppRef, window?: ResolvedComputerWindow, element?: ResolvedComputerElement, argument?: string, stdin?: string): Promise<void> {
|
||||
await this.assertAccessibility();
|
||||
// Element paths and names are argv values, never JXA source interpolation.
|
||||
const args = [operation, app.name, window?.window.windowId ?? "", String(window?.window.windowIndex ?? ""), element ? String(element.handle) : "", argument ?? ""];
|
||||
await this.callJson(args, COMPUTER_TIMEOUTS.action, stdin);
|
||||
}
|
||||
|
||||
private async callJson(args: string[], timeoutMs: number, stdin?: string): Promise<Record<string, unknown>> {
|
||||
let output;
|
||||
try {
|
||||
output = await this.context.seam.run("osascript", ["-l", "JavaScript", "-e", MACOS_AUTOMATION, ...args], { timeoutMs, stdin });
|
||||
} catch (error) {
|
||||
if (isMissingBinary(error, "osascript")) throw new ComputerUseError("PERMISSION_DENIED", "macOS osascript is unavailable.", "macOS with the osascript built-in is required for fn computer.", { missingBinary: "osascript" });
|
||||
throw error;
|
||||
}
|
||||
if (output.timedOut) throw new ComputerUseError("TIMEOUT", "OS automation timed out.");
|
||||
if (output.exitCode !== 0) throw new ComputerUseError(permissionShaped(output.stderr) ? "PERMISSION_UNVERIFIED" : "ACTION_FAILED", "OS automation failed.", permissionShaped(output.stderr) ? REMEDIATION_ACCESSIBILITY : undefined);
|
||||
try { const parsed: unknown = JSON.parse(output.stdout); return parsed && typeof parsed === "object" ? parsed as Record<string, unknown> : {}; } catch { throw new ComputerUseError("ACTION_FAILED", "OS automation returned invalid output."); }
|
||||
}
|
||||
}
|
||||
|
||||
function check(id: PermissionCheck["id"], status: PermissionCheck["status"], probed: boolean, detail: string | null, remediation: string): PermissionCheck { return { id, status, granted: status === "granted", probed, probe: id === "accessibility" ? "AXIsProcessTrusted" : "CGPreflightScreenCaptureAccess", detail, remediation }; }
|
||||
export function permissionShaped(stderr: string): boolean { return /-1719|-25211|not allowed assistive access|not authorized|accessibility/i.test(stderr); }
|
||||
function isMissingBinary(error: unknown, binary: string): boolean {
|
||||
const candidate = error as NodeJS.ErrnoException | undefined;
|
||||
return candidate?.code === "ENOENT" || new RegExp(`\\b${binary}\\b.*\\bnot found\\b|\\bENOENT\\b`, "i").test(candidate?.message ?? "");
|
||||
}
|
||||
function isAppRef(value: unknown): value is AppRef { const app = value as AppRef; return !!app && typeof app.name === "string" && typeof app.pid === "number" && (typeof app.bundleId === "string" || app.bundleId === null); }
|
||||
function isWindowRef(value: unknown): value is ResolvedComputerWindow["window"] { const window = value as ResolvedComputerWindow["window"]; return !!window && typeof window.windowId === "string" && typeof window.windowIndex === "number" && typeof window.title === "string"; }
|
||||
function isElement(value: unknown): value is Element { const element = value as Element; return isLiveElement(value) && typeof element.index === "number"; }
|
||||
function isLiveElement(value: unknown): value is Omit<Element, "index"> { const element = value as Element; return !!element && typeof element.role === "string" && !!element.locator && typeof element.locator.path === "string"; }
|
||||
function center(bounds: Element["bounds"]): { x: number; y: number } | undefined { return bounds ? { x: bounds.x + bounds.width / 2, y: bounds.y + bounds.height / 2 } : undefined; }
|
||||
function singleAction(action: string, app: AppRef, snapshotId: string | null, elementIndex: number | null): ActionResult { return { action, app, snapshotId: action === "hotkey" ? null : snapshotId, elementIndex, fromElementIndex: null, toElementIndex: null, performed: true }; }
|
||||
42
packages/cli/src/commands/computer/adapter-registry.ts
Normal file
42
packages/cli/src/commands/computer/adapter-registry.ts
Normal file
@@ -0,0 +1,42 @@
|
||||
import type { ComputerAdapter } from "./adapter.js";
|
||||
import { UnsupportedComputerAdapter } from "./adapter-unsupported.js";
|
||||
import { defaultComputerExecSeam, type ComputerExecSeam } from "./exec-seam.js";
|
||||
|
||||
export interface ComputerClock {
|
||||
now(): Date;
|
||||
}
|
||||
|
||||
export interface MacosComputerAdapterContext {
|
||||
platform: "darwin";
|
||||
seam: ComputerExecSeam;
|
||||
clock: ComputerClock;
|
||||
projectRoot: string;
|
||||
}
|
||||
|
||||
export type MacosComputerAdapterFactory = (context: MacosComputerAdapterContext) => ComputerAdapter;
|
||||
|
||||
export interface ComputerAdapterRegistryOptions {
|
||||
platform?: string;
|
||||
seam?: ComputerExecSeam;
|
||||
clock?: ComputerClock;
|
||||
projectRoot: string;
|
||||
/** Injection keeps registry tests and non-macOS hosts away from OS automation. */
|
||||
macosAdapterFactory: MacosComputerAdapterFactory;
|
||||
}
|
||||
|
||||
const systemClock: ComputerClock = { now: () => new Date() };
|
||||
|
||||
/**
|
||||
* Resolves the sole first-class platform adapter. The macOS factory is injected until the concrete
|
||||
* adapter owns the OS built-in implementation, while unsupported platforms always get a rejecting adapter.
|
||||
*/
|
||||
export function resolveComputerAdapter(options: ComputerAdapterRegistryOptions): ComputerAdapter {
|
||||
const platform = options.platform ?? process.platform;
|
||||
if (platform !== "darwin") return new UnsupportedComputerAdapter(platform);
|
||||
return options.macosAdapterFactory({
|
||||
platform: "darwin",
|
||||
seam: options.seam ?? defaultComputerExecSeam,
|
||||
clock: options.clock ?? systemClock,
|
||||
projectRoot: options.projectRoot,
|
||||
});
|
||||
}
|
||||
44
packages/cli/src/commands/computer/adapter-unsupported.ts
Normal file
44
packages/cli/src/commands/computer/adapter-unsupported.ts
Normal file
@@ -0,0 +1,44 @@
|
||||
import { COMPUTER_ACTIONS, ComputerUseError, type CapabilitiesResult, type PermissionsResult } from "./contract.js";
|
||||
import type {
|
||||
ClickInput, ComputerAdapter, ComputerStateCaptureOptions, DragInput, HotkeyInput, PressKeyInput,
|
||||
ResolvedComputerElement, ResolvedComputerWindow, ScrollInput, SetValueInput, TypeTextInput,
|
||||
} from "./adapter.js";
|
||||
import type { ActionResult, AppTarget, AppStateResult, ElementLocator, ListAppsResult, ListWindowsResult } from "./contract.js";
|
||||
|
||||
const remediation = "Computer use is supported on macOS only. Run this command on macOS with Accessibility permission.";
|
||||
|
||||
export class UnsupportedComputerAdapter implements ComputerAdapter {
|
||||
readonly id = "unsupported";
|
||||
readonly supported = false;
|
||||
|
||||
constructor(readonly platform: string) {}
|
||||
|
||||
async capabilities(): Promise<CapabilitiesResult> {
|
||||
return {
|
||||
platform: this.platform, adapterId: this.id, supported: false,
|
||||
actions: [], unsupportedActions: [...COMPUTER_ACTIONS],
|
||||
features: { screenshot: false, restoreWindow: false, stdinSecrets: false, crossInvocationSnapshots: false },
|
||||
};
|
||||
}
|
||||
|
||||
async permissions(): Promise<PermissionsResult> {
|
||||
return { platform: this.platform, adapterId: this.id, supported: false, allGranted: false, checks: [] };
|
||||
}
|
||||
|
||||
private unsupported(): never {
|
||||
throw new ComputerUseError("UNSUPPORTED_PLATFORM", `Computer use is unsupported on ${this.platform}.`, remediation);
|
||||
}
|
||||
|
||||
async listApps(): Promise<ListAppsResult> { return this.unsupported(); }
|
||||
async listWindows(_target: AppTarget): Promise<ListWindowsResult> { return this.unsupported(); }
|
||||
async captureState(_target: AppTarget, _options: ComputerStateCaptureOptions): Promise<AppStateResult> { return this.unsupported(); }
|
||||
async resolveWindow(_target: AppTarget, _selector: { windowId?: string; windowIndex?: number }): Promise<ResolvedComputerWindow> { return this.unsupported(); }
|
||||
async resolveLocator(_window: ResolvedComputerWindow, _locator: ElementLocator): Promise<ResolvedComputerElement> { return this.unsupported(); }
|
||||
async click(_input: ClickInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async "set-value"(_input: SetValueInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async "type-text"(_input: TypeTextInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async "press-key"(_input: PressKeyInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async hotkey(_input: HotkeyInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async scroll(_input: ScrollInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
async drag(_input: DragInput): Promise<ActionResult> { return this.unsupported(); }
|
||||
}
|
||||
97
packages/cli/src/commands/computer/adapter.ts
Normal file
97
packages/cli/src/commands/computer/adapter.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
import type {
|
||||
ActionResult,
|
||||
AppRef,
|
||||
AppStateResult,
|
||||
AppTarget,
|
||||
CapabilitiesResult,
|
||||
ComputerAction,
|
||||
Element,
|
||||
ElementLocator,
|
||||
ListAppsResult,
|
||||
ListWindowsResult,
|
||||
PermissionsResult,
|
||||
WindowRef,
|
||||
} from "./contract.js";
|
||||
|
||||
/** A serializable window reference paired with an adapter-private live handle. */
|
||||
export interface ResolvedComputerWindow {
|
||||
readonly window: WindowRef;
|
||||
readonly handle: unknown;
|
||||
}
|
||||
|
||||
/** A serializable element paired with an adapter-private live accessibility handle. */
|
||||
export interface ResolvedComputerElement {
|
||||
readonly element: Element;
|
||||
readonly handle: unknown;
|
||||
}
|
||||
|
||||
export interface ComputerStateCaptureOptions {
|
||||
windowId?: string;
|
||||
windowIndex?: number;
|
||||
screenshot: boolean;
|
||||
restoreWindow: boolean;
|
||||
}
|
||||
|
||||
export interface ElementActionInput {
|
||||
app: AppRef;
|
||||
window: ResolvedComputerWindow;
|
||||
element: ResolvedComputerElement;
|
||||
snapshotId: string | null;
|
||||
}
|
||||
|
||||
export interface UntargetedActionInput {
|
||||
app: AppRef;
|
||||
}
|
||||
|
||||
export type ClickInput = ElementActionInput;
|
||||
export interface SetValueInput extends ElementActionInput { value: string; }
|
||||
export interface TypeTextInput extends Omit<Partial<ElementActionInput>, "app">, UntargetedActionInput { text: string; }
|
||||
export interface PressKeyInput extends Omit<Partial<ElementActionInput>, "app">, UntargetedActionInput { key: string; }
|
||||
export interface HotkeyInput extends UntargetedActionInput { keys: readonly string[]; }
|
||||
export interface ScrollInput extends Omit<Partial<ElementActionInput>, "app">, UntargetedActionInput { direction: "up" | "down" | "left" | "right"; amount: number; }
|
||||
export interface DragInput extends UntargetedActionInput {
|
||||
snapshotId: string | null;
|
||||
window?: ResolvedComputerWindow;
|
||||
from?: ResolvedComputerElement;
|
||||
to?: ResolvedComputerElement;
|
||||
fromX?: number;
|
||||
fromY?: number;
|
||||
toX?: number;
|
||||
toY?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-11-03:34:
|
||||
* Desktop automation must resolve unsupported platforms honestly: a clear failure is safer than a
|
||||
* silent no-op that makes callers believe an action happened. Element-taking methods receive the
|
||||
* command layer's resolved window and locator-backed element, never an index, because indexes are
|
||||
* sparse and snapshot-scoped across separate CLI processes.
|
||||
*/
|
||||
export interface ComputerAdapter {
|
||||
readonly platform: string;
|
||||
readonly id: string;
|
||||
readonly supported: boolean;
|
||||
|
||||
capabilities(): Promise<CapabilitiesResult>;
|
||||
permissions(): Promise<PermissionsResult>;
|
||||
listApps(): Promise<ListAppsResult>;
|
||||
listWindows(target: AppTarget): Promise<ListWindowsResult>;
|
||||
captureState(target: AppTarget, options: ComputerStateCaptureOptions): Promise<AppStateResult>;
|
||||
resolveWindow(target: AppTarget, selector: { windowId?: string; windowIndex?: number }): Promise<ResolvedComputerWindow>;
|
||||
resolveLocator(window: ResolvedComputerWindow, locator: ElementLocator): Promise<ResolvedComputerElement>;
|
||||
|
||||
click(input: ClickInput): Promise<ActionResult>;
|
||||
"set-value"(input: SetValueInput): Promise<ActionResult>;
|
||||
"type-text"(input: TypeTextInput): Promise<ActionResult>;
|
||||
"press-key"(input: PressKeyInput): Promise<ActionResult>;
|
||||
hotkey(input: HotkeyInput): Promise<ActionResult>;
|
||||
scroll(input: ScrollInput): Promise<ActionResult>;
|
||||
drag(input: DragInput): Promise<ActionResult>;
|
||||
}
|
||||
|
||||
/** Ensures the adapter surface remains exhaustive when the runtime action inventory grows. */
|
||||
export type ComputerAdapterActionMethods = {
|
||||
[Action in ComputerAction]: ComputerAdapter[Action];
|
||||
};
|
||||
|
||||
export type ComputerAdapterAction = ComputerAction;
|
||||
83
packages/cli/src/commands/computer/contract.ts
Normal file
83
packages/cli/src/commands/computer/contract.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-11-03:34:
|
||||
* This is the versioned contract consumed by the follow-up skill. Fields and codes are append-only;
|
||||
* schemaVersion changes only for a breaking reshape. Runtime tuples are intentional because erased
|
||||
* TypeScript unions cannot protect the downstream conformance test. Locators persist because each
|
||||
* CLI invocation is a separate process, and deterministic one-code failures let callers branch safely.
|
||||
*/
|
||||
|
||||
export const COMPUTER_ERROR_CODES = Object.freeze([
|
||||
"UNSUPPORTED_PLATFORM", "PERMISSION_DENIED", "PERMISSION_UNVERIFIED", "INVALID_ARGUMENTS",
|
||||
"APP_NOT_FOUND", "AMBIGUOUS_APP", "WINDOW_NOT_FOUND", "SNAPSHOT_REQUIRED", "SNAPSHOT_STALE",
|
||||
"ELEMENT_INDEX_NOT_FOUND", "ELEMENT_UNRESOLVABLE", "ACTION_UNSUPPORTED", "ACTION_FAILED",
|
||||
"SCREENSHOT_FAILED", "TIMEOUT", "INTERNAL",
|
||||
] as const);
|
||||
export type ComputerErrorCode = (typeof COMPUTER_ERROR_CODES)[number];
|
||||
export const COMPUTER_ACTIONS = Object.freeze(["click", "set-value", "type-text", "press-key", "hotkey", "scroll", "drag"] as const);
|
||||
export type ComputerAction = (typeof COMPUTER_ACTIONS)[number];
|
||||
export const COMPUTER_SUBCOMMANDS = Object.freeze([
|
||||
"capabilities", "permissions", "list-apps", "list-windows", "get-app-state",
|
||||
"click", "set-value", "type-text", "press-key", "hotkey", "scroll", "drag",
|
||||
] as const);
|
||||
export type ComputerSubcommand = (typeof COMPUTER_SUBCOMMANDS)[number];
|
||||
export type CommandName = `computer.${ComputerSubcommand}` | "computer";
|
||||
export const SNAPSHOT_STALE_REASONS = Object.freeze(["not-found", "superseded", "expired", "pid-changed", "window-mismatch", "window-gone"] as const);
|
||||
export type SnapshotStaleReason = (typeof SNAPSHOT_STALE_REASONS)[number];
|
||||
export const COMPUTER_TIMEOUTS = Object.freeze({ permissionProbe: 5_000, discovery: 10_000, stateCapture: 20_000, screenshotCapture: 15_000, locatorReplay: 10_000, action: 10_000 });
|
||||
|
||||
export type FlatDetails = Record<string, string | number | boolean>;
|
||||
export interface ComputerError { code: ComputerErrorCode; message: string; remediation?: string; details?: FlatDetails; }
|
||||
export type ComputerEnvelope<T = unknown> =
|
||||
| { schemaVersion: 1; ok: true; command: CommandName; result: T }
|
||||
| { schemaVersion: 1; ok: false; command: CommandName; error: ComputerError };
|
||||
const REMEDIATION_REQUIRED = new Set<ComputerErrorCode>(["UNSUPPORTED_PLATFORM", "PERMISSION_DENIED", "PERMISSION_UNVERIFIED", "SNAPSHOT_REQUIRED", "SNAPSHOT_STALE", "ELEMENT_INDEX_NOT_FOUND", "ELEMENT_UNRESOLVABLE", "AMBIGUOUS_APP", "ACTION_UNSUPPORTED"]);
|
||||
export class ComputerUseError extends Error {
|
||||
constructor(public readonly code: ComputerErrorCode, message: string, public readonly remediation?: string, public readonly details?: FlatDetails) { super(message); }
|
||||
}
|
||||
export const successEnvelope = <T>(command: CommandName, result: T): ComputerEnvelope<T> => ({ schemaVersion: 1, ok: true, command, result });
|
||||
export function failureEnvelope(command: CommandName, error: ComputerError | ComputerUseError): ComputerEnvelope<never> {
|
||||
const value: ComputerError = error instanceof ComputerUseError ? { code: error.code, message: error.message, remediation: error.remediation, details: error.details } : error;
|
||||
if (REMEDIATION_REQUIRED.has(value.code) && !value.remediation) throw new Error(`Remediation required for ${value.code}`);
|
||||
if (value.details && !isFlatDetails(value.details)) throw new Error("Computer error details must be flat");
|
||||
return { schemaVersion: 1, ok: false, command, error: value };
|
||||
}
|
||||
export function isFlatDetails(value: unknown): value is FlatDetails { return !!value && typeof value === "object" && Object.values(value as object).every((entry) => ["string", "number", "boolean"].includes(typeof entry)); }
|
||||
|
||||
export interface Bounds { x: number; y: number; width: number; height: number; }
|
||||
export interface AppTarget { kind: "bundleId" | "name" | "pid"; raw: string; value: string; }
|
||||
export interface AppRef { bundleId: string | null; name: string; pid: number; }
|
||||
export interface WindowRef { windowId: string; windowIndex: number; title: string; bounds: Bounds | null; minimized: boolean; }
|
||||
export interface ElementLocator { kind: "ax-path"; path: string; role: string; subrole: string | null; identifier: string | null; title: string | null; }
|
||||
export interface Element { index: number; role: string; title: string | null; value: string | null; label: string | null; enabled: boolean; focused: boolean; bounds: Bounds | null; actions: string[]; locator: ElementLocator; }
|
||||
export type PermissionCheckStatus = "granted" | "denied" | "unknown";
|
||||
export interface PermissionCheck { id: "accessibility" | "screen-recording"; status: PermissionCheckStatus; granted: boolean; probed: boolean; probe: string; detail: string | null; remediation: string | null; }
|
||||
export interface CapabilitiesResult { platform: string; adapterId: string; supported: boolean; actions: string[]; unsupportedActions: string[]; features: { screenshot: boolean; restoreWindow: boolean; stdinSecrets: boolean; crossInvocationSnapshots: boolean }; }
|
||||
export interface PermissionsResult { platform: string; adapterId: string; supported: boolean; allGranted: boolean; checks: PermissionCheck[]; }
|
||||
export interface ListAppsResult { apps: AppRef[]; }
|
||||
export interface ListWindowsResult { app: AppRef; windows: WindowRef[]; }
|
||||
export interface Screenshot { path: string; width: number | null; height: number | null; verifiedPermission: boolean; }
|
||||
export interface AppStateResult { app: AppRef; window: WindowRef; snapshot: { snapshotId: string; targetKey: string; windowKey: string; capturedAt: string; expiresAt: string; treeText: string; elementCount: number; truncated: boolean; elements: Element[] }; screenshot: Screenshot | null; screenshotError?: { code: "SCREENSHOT_FAILED" | "PERMISSION_DENIED" | "PERMISSION_UNVERIFIED" | "TIMEOUT"; message: string }; }
|
||||
export interface ActionResult { action: string; app: AppRef; snapshotId: string | null; elementIndex: number | null; fromElementIndex: number | null; toElementIndex: number | null; performed: true; }
|
||||
export interface SnapshotRecord { snapshotId: string; targetKey: string; windowKey: string; capturedAt: string; expiresAt: string; app: AppRef; window: WindowRef; elementCount: number; elements: Record<string, Element>; }
|
||||
export const SNAPSHOT_ID_PATTERN = /^cs_[A-Za-z0-9]{10,40}$/;
|
||||
export const isValidSnapshotId = (value: string | undefined): value is string => typeof value === "string" && SNAPSHOT_ID_PATTERN.test(value);
|
||||
export const targetKeyForApp = (app: AppRef): string => app.bundleId ? `bundle:${app.bundleId}` : `pid:${app.pid}`;
|
||||
export const windowKeyFor = (targetKey: string, windowId: string): string => `${targetKey}#${windowId}`;
|
||||
export const targetKeySlug = (targetKey: string): string => targetKey.replace(/[^A-Za-z0-9._-]/g, "_");
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-11-04:19:
|
||||
* A dotted display name is valid, so punctuation cannot choose the target kind. Resolution always
|
||||
* tries an exact bundle id before an exact unambiguous name, preserving the published precedence.
|
||||
*/
|
||||
export function parseAppTarget(raw: string): AppTarget { return raw.startsWith("pid:") ? { kind: "pid", raw, value: raw.slice(4) } : { kind: "name", raw, value: raw }; }
|
||||
function validLocator(value: unknown): value is ElementLocator { const x = value as ElementLocator; return !!x && x.kind === "ax-path" && typeof x.path === "string" && !!x.path && typeof x.role === "string" && typeof x.subrole !== "undefined" && typeof x.identifier !== "undefined" && typeof x.title !== "undefined"; }
|
||||
export function isCapabilitiesResult(value: unknown): value is CapabilitiesResult { const x = value as CapabilitiesResult; return !!x && typeof x.platform === "string" && Array.isArray(x.actions) && Array.isArray(x.unsupportedActions) && !!x.features; }
|
||||
export function isPermissionsResult(value: unknown): value is PermissionsResult { const x = value as PermissionsResult; return !!x && Array.isArray(x.checks) && typeof x.allGranted === "boolean" && x.checks.every((c) => c.granted === (c.status === "granted") && (!c.granted || c.probed)) && (!x.allGranted || x.checks.length > 0 && x.checks.every((c) => c.status === "granted")); }
|
||||
export function isAppStateResult(value: unknown, screenshotSkipped = false): value is AppStateResult { const x = value as AppStateResult; return !!x && !!x.snapshot && Array.isArray(x.snapshot.elements) && x.snapshot.elements.every((e) => Number.isInteger(e.index) && validLocator(e.locator)) && !(x.screenshot && x.screenshotError) && !(!x.screenshot && !x.screenshotError && !screenshotSkipped); }
|
||||
export function isActionResult(value: unknown): value is ActionResult { const x = value as ActionResult; if (!x || x.performed !== true) return false; if (x.action === "drag") return x.elementIndex === null && ((x.fromElementIndex === null && x.toElementIndex === null) || (typeof x.fromElementIndex === "number" && typeof x.toElementIndex === "number")); if (x.action === "hotkey") return x.snapshotId === null && x.elementIndex === null && x.fromElementIndex === null && x.toElementIndex === null; return x.fromElementIndex === null && x.toElementIndex === null; }
|
||||
export function validateResult(subcommand: ComputerSubcommand, value: unknown, screenshotSkipped = false): boolean { if (subcommand === "capabilities") return isCapabilitiesResult(value); if (subcommand === "permissions") return isPermissionsResult(value); if (subcommand === "get-app-state") return isAppStateResult(value, screenshotSkipped); if (COMPUTER_ACTIONS.includes(subcommand as ComputerAction)) return isActionResult(value); return !!value && typeof value === "object"; }
|
||||
export const SECRET_VALUE: unique symbol = Symbol("computer-secret");
|
||||
export type SecretValue = { readonly [SECRET_VALUE]: true; readonly value: string };
|
||||
export const secretValue = (value: string): SecretValue => ({ [SECRET_VALUE]: true, value });
|
||||
export function redact(value: unknown): string { if (typeof value === "object" && value && SECRET_VALUE in value) return "[REDACTED]"; const text = value instanceof Error ? value.message : String(value); return text.replace(/[^\s]{8,}/g, "[REDACTED]"); }
|
||||
61
packages/cli/src/commands/computer/exec-seam.ts
Normal file
61
packages/cli/src/commands/computer/exec-seam.ts
Normal file
@@ -0,0 +1,61 @@
|
||||
import { superviseSpawn } from "@fusion/core";
|
||||
|
||||
export interface ComputerExecOptions {
|
||||
/** Every OS command is bounded; callers select a named COMPUTER_TIMEOUTS value. */
|
||||
timeoutMs: number;
|
||||
stdin?: string;
|
||||
}
|
||||
|
||||
export interface ComputerExecResult {
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
exitCode: number | null;
|
||||
timedOut: boolean;
|
||||
}
|
||||
|
||||
export interface ComputerExecSeam {
|
||||
run(file: string, args: readonly string[], options: ComputerExecOptions): Promise<ComputerExecResult>;
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-11-03:34:
|
||||
* OS automation is asynchronous, shell-free, and bounded by the fixed C10.2 timeout defaults.
|
||||
* The seam keeps tests off the host desktop while preventing synchronous exec, shell interpolation,
|
||||
* or a child that can wait forever; stdin is written once and then closed.
|
||||
*/
|
||||
export const defaultComputerExecSeam: ComputerExecSeam = {
|
||||
async run(file, args, options) {
|
||||
if (!Number.isFinite(options.timeoutMs) || options.timeoutMs <= 0) {
|
||||
throw new Error("Computer exec timeoutMs must be a positive finite number");
|
||||
}
|
||||
|
||||
const supervised = superviseSpawn(file, args, {
|
||||
shell: false,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
maxLifetimeMs: options.timeoutMs,
|
||||
});
|
||||
const { child } = supervised;
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout?.setEncoding("utf8");
|
||||
child.stderr?.setEncoding("utf8");
|
||||
child.stdout?.on("data", (chunk: string) => { stdout += chunk; });
|
||||
child.stderr?.on("data", (chunk: string) => { stderr += chunk; });
|
||||
|
||||
if (options.stdin !== undefined) child.stdin?.write(options.stdin);
|
||||
child.stdin?.end();
|
||||
|
||||
let timedOut = false;
|
||||
const timeout = setTimeout(() => {
|
||||
timedOut = true;
|
||||
supervised.kill("SIGTERM");
|
||||
}, options.timeoutMs);
|
||||
timeout.unref();
|
||||
try {
|
||||
const exited = await supervised.waitExit();
|
||||
return { stdout, stderr, exitCode: exited.code, timedOut };
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
},
|
||||
};
|
||||
37
packages/cli/src/commands/computer/scripts/macos-jxa.ts
Normal file
37
packages/cli/src/commands/computer/scripts/macos-jxa.ts
Normal file
@@ -0,0 +1,37 @@
|
||||
/*
|
||||
* FNXC:ComputerUse 2026-08-11-03:54:
|
||||
* Values are passed through argv or stdin rather than interpolated into this source. OS automation
|
||||
* receives untrusted app names, window IDs, and locator paths, so keeping this program static is
|
||||
* the injection boundary.
|
||||
*/
|
||||
export const MACOS_PERMISSION_PREFLIGHT = `ObjC.import('ApplicationServices'); ObjC.import('CoreGraphics'); ObjC.bindFunction('AXIsProcessTrusted', ['bool', []]); ObjC.bindFunction('CGPreflightScreenCaptureAccess', ['bool', []]); function run(argv) { return JSON.stringify({accessibility: Boolean($.AXIsProcessTrusted()), screenRecording: Boolean($.CGPreflightScreenCaptureAccess())}); }`;
|
||||
|
||||
/** System Events operations; argv is [operation, ...parameters] and is never source-interpolated. */
|
||||
export const MACOS_AUTOMATION = `
|
||||
ObjC.import('Foundation');
|
||||
ObjC.import('CoreGraphics');
|
||||
function text(v) { try { return String(v); } catch (_) { return ''; } }
|
||||
function nullable(getter) { try { var v = getter(); return v === undefined ? null : v; } catch (_) { return null; } }
|
||||
function appRef(p) { return { bundleId: nullable(function(){ return p.bundleIdentifier(); }), name: text(p.name()), pid: Number(p.unixId()) }; }
|
||||
function allApps(se) { var ps = se.applicationProcesses(); var out=[]; for (var i=0;i<ps.length;i++) { try { out.push(appRef(ps[i])); } catch (_) {} } return out; }
|
||||
function processFor(se, raw) { var ps=se.applicationProcesses(); for (var i=0;i<ps.length;i++) { var p=ps[i]; var id=nullable(function(){return p.bundleIdentifier();}); if (text(p.name())===raw || id===raw || ('pid:'+String(p.unixId()))===raw) return p; } throw new Error('app-not-found'); }
|
||||
function rect(v) { if (!v || v.length < 4) return null; return {x:Number(v[0]),y:Number(v[1]),width:Number(v[2]),height:Number(v[3])}; }
|
||||
function sameBounds(a,b) { var frame=ObjC.unwrap(b); return a && frame && Math.abs(a.x-Number(frame.X))<2 && Math.abs(a.y-Number(frame.Y))<2 && Math.abs(a.width-Number(frame.Width))<2 && Math.abs(a.height-Number(frame.Height))<2; }
|
||||
function captureWindowId(p,title,bounds) { try { var options=1|16, rows=$.CGWindowListCopyWindowInfo(options,0), pid=Number(p.unixId()), match=null; for(var i=0;i<rows.count;i++) { var row=rows.objectAtIndex(i), owner=Number(ObjC.unwrap(row.objectForKey('kCGWindowOwnerPID'))), name=String(ObjC.unwrap(row.objectForKey('kCGWindowName')) || ''), frame=row.objectForKey('kCGWindowBounds'); if(owner===pid && name===title && sameBounds(bounds, frame)) { if(match!==null) return null; match=Number(ObjC.unwrap(row.objectForKey('kCGWindowNumber'))); } } return match; } catch (_) {} return null; }
|
||||
function windowsFor(p) { var ws=p.windows(), out=[]; for(var i=0;i<ws.length;i++) { var w=ws[i], position=nullable(function(){return w.position();}), size=nullable(function(){return w.size();}), bounds=position&&size?{x:Number(position[0]),y:Number(position[1]),width:Number(size[0]),height:Number(size[1])}:null, title=text(nullable(function(){return w.name();})); out.push({windowId: String(nullable(function(){return w.id();}) || i), windowIndex:i, title:title, bounds:bounds, minimized:Boolean(nullable(function(){return w.attributes.byName('AXMinimized').value();})), captureWindowId:captureWindowId(p,title,bounds)}); } return out; }
|
||||
function windowFor(p, id, index) { var ws=p.windows(); for(var i=0;i<ws.length;i++) { var w=ws[i], current=String(nullable(function(){return w.id();}) || i); if ((id && current===id) || (!id && index!=='' && i===Number(index)) || (!id && index==='' && i===0)) return w; } throw new Error('window-not-found'); }
|
||||
function children(el) { try { return el.uiElements(); } catch (_) { return []; } }
|
||||
function role(el) { return text(nullable(function(){return el.role();})); }
|
||||
function elementValue(el) { return nullable(function(){ return text(el.value()); }); }
|
||||
function attrs(el, path) { var r=role(el), sub=nullable(function(){return text(el.subrole());}), ident=nullable(function(){return text(el.identifier());}), title=nullable(function(){return text(el.name());}), pos=nullable(function(){return el.position();}), size=nullable(function(){return el.size();}), actions=[]; try { var as=el.actions(); for(var i=0;i<as.length;i++) actions.push(text(as[i].name())); } catch (_) {} return {role:r,title:title,value:elementValue(el),label:nullable(function(){return text(el.description());}),enabled:!Boolean(nullable(function(){return el.attributes.byName('AXEnabled').value();})===false),focused:Boolean(nullable(function(){return el.attributes.byName('AXFocused').value();})),bounds:pos&&size?{x:Number(pos[0]),y:Number(pos[1]),width:Number(size[0]),height:Number(size[1])}:null,actions:actions,locator:{kind:'ax-path',path:path,role:r,subrole:sub,identifier:ident,title:title}}; }
|
||||
function capture(w) { var elements=[], lines=[], count=0, truncated=false, max=500, depthMax=12; function walk(el,path,depth) { if(count>=max||depth>depthMax){truncated=true;return;} var a=attrs(el,path); a.index=count++; elements.push(a); lines.push(path+' '+a.role+(a.title?' '+a.title:'')); var cs=children(el); for(var i=0;i<cs.length;i++) walk(cs[i],path+'/'+role(cs[i])+'['+i+']',depth+1); } walk(w,'window[0]',0); return {treeText:lines.join('\n'),elements:elements,truncated:truncated}; }
|
||||
function resolve(w,path) { var parts=String(path).split('/'); var el=w; for(var i=1;i<parts.length;i++){ var m=/^(.*)\\[(\\d+)\\]$/.exec(parts[i]); if(!m) throw new Error('locator-not-found'); var cs=children(el), n=Number(m[2]); if(n>=cs.length || role(cs[n])!==m[1]) throw new Error('locator-not-found'); el=cs[n]; } return attrs(el,path); }
|
||||
function focused(p) { try { return p.attributes.byName('AXFocusedUIElement').value(); } catch (_) { throw new Error('no-focused-element'); } }
|
||||
function stdin() { var d=$.NSFileHandle.fileHandleWithStandardInput.readDataToEndOfFile; return ObjC.unwrap($.NSString.alloc.initWithDataEncoding(d,$.NSUTF8StringEncoding)); }
|
||||
function scrollKeyCode(direction) { if(direction==='up') return 126; if(direction==='down') return 125; if(direction==='left') return 123; if(direction==='right') return 124; throw new Error('invalid-scroll-direction'); }
|
||||
function perform(p,w,path,kind,arg) { var el=path ? resolve(w,path) : null; if(kind==='click') { el.actions.byName('AXPress').perform(); } else if(kind==='set-value') { el.value = stdin(); } else if(kind==='type-text') { el=el||focused(p); try { el.attributes.byName('AXFocused').value=true; } catch (_) {} Application('System Events').keystroke(stdin()); } else if(kind==='press-key') { el=el||focused(p); try { el.attributes.byName('AXFocused').value=true; } catch (_) {} Application('System Events').keystroke(arg); } else if(kind==='scroll') { el=el||focused(p); if(!el) throw new Error('no-focused-element'); var n=Number(arg.split(':')[1]); var keyCode=scrollKeyCode(arg.split(':')[0]); Application('System Events').keyCode(keyCode,{using:['option']}); for(var i=1;i<n;i++) Application('System Events').keyCode(keyCode,{using:['option']}); } else if(kind==='drag') { throw new Error('drag-requires-coordinate-event'); } return true; }
|
||||
/* FNXC:ComputerUse 2026-08-11-04:12: System Events has no mouseMove/mouseDown/mouseUp API. Drag must post the real CoreGraphics down, dragged, and up events so this advertised action never becomes a silent scripting no-op. */
|
||||
function postMouse(type, x, y) { var event=$.CGEventCreateMouseEvent(null,type,$.CGPointMake(x,y),0); if(!event) throw new Error('cg-event-create-failed'); $.CGEventPost(0,event); }
|
||||
function dragCoordinates(_se, fromX, fromY, toX, toY) { var values=[Number(fromX),Number(fromY),Number(toX),Number(toY)]; for(var i=0;i<values.length;i++) if(!isFinite(values[i])) throw new Error('invalid-drag-coordinate'); postMouse(1,values[0],values[1]); postMouse(6,values[2],values[3]); postMouse(2,values[2],values[3]); }
|
||||
function run(argv) { var se=Application('System Events'), op=argv[0]; if(op==='list-apps') return JSON.stringify({apps:allApps(se)}); var p=processFor(se,argv[1]); if(op==='list-windows') return JSON.stringify({windows:windowsFor(p)}); if(op==='hotkey'){ se.keystroke(argv[2],{using:argv.slice(3)}); return JSON.stringify({ok:true}); } if(op==='drag-coordinates'){ dragCoordinates(se,argv[2],argv[3],argv[4],argv[5]); return JSON.stringify({ok:true}); } var w=windowFor(p,argv[2],argv[3]); if(op==='state') return JSON.stringify(capture(w)); if(op==='resolve-locator') return JSON.stringify({element:resolve(w,argv[4])}); if(op==='restore-window'){ try { w.attributes.byName('AXMinimized').value=false; } catch (_) {} return JSON.stringify({ok:true}); } return JSON.stringify({ok:perform(p,w,argv[4],op,argv[5])}); }
|
||||
`;
|
||||
257
packages/cli/src/commands/computer/snapshot-store.ts
Normal file
257
packages/cli/src/commands/computer/snapshot-store.ts
Normal file
@@ -0,0 +1,257 @@
|
||||
import { mkdir, readFile, readdir, rename, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import {
|
||||
ComputerUseError,
|
||||
isValidSnapshotId,
|
||||
targetKeyForApp,
|
||||
targetKeySlug,
|
||||
windowKeyFor,
|
||||
type AppRef,
|
||||
type Element,
|
||||
type SnapshotRecord,
|
||||
type WindowRef,
|
||||
} from "./contract.js";
|
||||
|
||||
const SNAPSHOT_TTL_MS = 5 * 60_000;
|
||||
const DEFAULT_MAX_RECORDS = 50;
|
||||
let lastSnapshotTimestamp = 0;
|
||||
let snapshotSequence = 0;
|
||||
|
||||
export interface SnapshotStoreOptions {
|
||||
/** Project root shared by all CLI invocations; tests inject an isolated root. */
|
||||
projectRoot?: string;
|
||||
now?: () => Date;
|
||||
ttlMs?: number;
|
||||
maxRecords?: number;
|
||||
}
|
||||
|
||||
export interface PersistSnapshotInput {
|
||||
app: AppRef;
|
||||
window: WindowRef;
|
||||
elementCount: number;
|
||||
elements: readonly Element[];
|
||||
capturedAt?: string;
|
||||
expiresAt?: string;
|
||||
}
|
||||
|
||||
export interface ResolveSnapshotInput {
|
||||
app: AppRef;
|
||||
snapshotId?: string;
|
||||
assertedWindowId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:ComputerUse 2026-08-11-03:34:
|
||||
* Every computer command is a new process, so get-app-state must atomically persist a serializable
|
||||
* locator map before returning its snapshot ID. Actions only replay the current app-scoped record;
|
||||
* sparse indexes are map keys rather than positions or elementCount-derived values to prevent a
|
||||
* stale UI from turning an intended action into a wrong-element action.
|
||||
*/
|
||||
export class ComputerSnapshotStore {
|
||||
private readonly projectRoot: string;
|
||||
private readonly now: () => Date;
|
||||
private readonly ttlMs: number;
|
||||
private readonly maxRecords: number;
|
||||
|
||||
constructor(options: SnapshotStoreOptions = {}) {
|
||||
this.projectRoot = resolve(options.projectRoot ?? process.cwd());
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.ttlMs = options.ttlMs ?? SNAPSHOT_TTL_MS;
|
||||
this.maxRecords = options.maxRecords ?? DEFAULT_MAX_RECORDS;
|
||||
}
|
||||
|
||||
get snapshotsDirectory(): string {
|
||||
return join(this.projectRoot, ".fusion", "computer-use", "snapshots");
|
||||
}
|
||||
|
||||
get latestDirectory(): string {
|
||||
return join(this.projectRoot, ".fusion", "computer-use", "latest");
|
||||
}
|
||||
|
||||
async persist(input: PersistSnapshotInput): Promise<SnapshotRecord> {
|
||||
const capturedAt = input.capturedAt ?? this.now().toISOString();
|
||||
const expiresAt = input.expiresAt ?? new Date(Date.parse(capturedAt) + this.ttlMs).toISOString();
|
||||
const targetKey = targetKeyForApp(input.app);
|
||||
const record: SnapshotRecord = {
|
||||
snapshotId: this.nextSnapshotId(),
|
||||
targetKey,
|
||||
windowKey: windowKeyFor(targetKey, input.window.windowId),
|
||||
capturedAt,
|
||||
expiresAt,
|
||||
app: input.app,
|
||||
window: input.window,
|
||||
elementCount: input.elementCount,
|
||||
elements: Object.fromEntries(input.elements.map((element) => [String(element.index), element])),
|
||||
};
|
||||
|
||||
await mkdir(this.snapshotsDirectory, { recursive: true });
|
||||
await mkdir(this.latestDirectory, { recursive: true });
|
||||
await writeJsonAtomically(this.snapshotPath(record.snapshotId), record);
|
||||
await writeJsonAtomically(this.latestPath(targetKey), { snapshotId: record.snapshotId });
|
||||
await this.prune({ preserveSnapshotId: record.snapshotId });
|
||||
return record;
|
||||
}
|
||||
|
||||
/** Read the current record for the resolved app and enforce the C9 fence order. */
|
||||
async resolve(input: ResolveSnapshotInput): Promise<SnapshotRecord> {
|
||||
const targetKey = targetKeyForApp(input.app);
|
||||
const latest = await this.readLatest(targetKey);
|
||||
if (!latest) {
|
||||
throw snapshotRequired();
|
||||
}
|
||||
|
||||
const requestedId = input.snapshotId ?? latest.snapshotId;
|
||||
// Command parsing validates this too; the store repeats the guard so an injected caller cannot
|
||||
// turn an id into a path outside the snapshots directory.
|
||||
const record = isValidSnapshotId(requestedId) ? await this.readRecord(requestedId) : undefined;
|
||||
if (!record) {
|
||||
if (input.snapshotId) throw snapshotStale("not-found", requestedId);
|
||||
throw snapshotRequired();
|
||||
}
|
||||
|
||||
if (input.snapshotId && latest.snapshotId !== input.snapshotId) {
|
||||
throw snapshotStale("superseded", input.snapshotId);
|
||||
}
|
||||
if (this.now().getTime() >= Date.parse(record.expiresAt)) {
|
||||
throw snapshotStale("expired", record.snapshotId);
|
||||
}
|
||||
if (record.app.pid !== input.app.pid) {
|
||||
throw snapshotStale("pid-changed", record.snapshotId);
|
||||
}
|
||||
if (input.assertedWindowId !== undefined && record.window.windowId !== input.assertedWindowId) {
|
||||
throw snapshotStale("window-mismatch", record.snapshotId);
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
/** Look up the actual sparse index emitted by the captured accessibility traversal. */
|
||||
getElement(record: SnapshotRecord, elementIndex: number): Element {
|
||||
const element = record.elements[String(elementIndex)];
|
||||
if (element) return element;
|
||||
throw new ComputerUseError(
|
||||
"ELEMENT_INDEX_NOT_FOUND",
|
||||
`Element index ${elementIndex} is not in snapshot ${record.snapshotId}; re-run fn computer get-app-state.`,
|
||||
"Re-run fn computer get-app-state and use an element index from that snapshot.",
|
||||
{ snapshotId: record.snapshotId, elementIndex },
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Keep the store bounded with single-level reads only. A caller holding a resolved record passes
|
||||
* it as preserveSnapshotId so a concurrent capture cannot remove that action's replay source.
|
||||
*/
|
||||
async prune(options: { preserveSnapshotId?: string } = {}): Promise<void> {
|
||||
const entries = await readDirectory(this.snapshotsDirectory);
|
||||
const records: Array<{ id: string; expiresAt: number; modifiedAt: number }> = [];
|
||||
const now = this.now().getTime();
|
||||
for (const name of entries) {
|
||||
if (!name.endsWith(".json")) continue;
|
||||
const id = name.slice(0, -5);
|
||||
const record = await this.readRecord(id);
|
||||
if (!record) continue;
|
||||
const expiresAt = Date.parse(record.expiresAt);
|
||||
if (id !== options.preserveSnapshotId && Number.isFinite(expiresAt) && expiresAt <= now) {
|
||||
await rm(this.snapshotPath(id), { force: true });
|
||||
continue;
|
||||
}
|
||||
const info = await stat(this.snapshotPath(id)).catch(() => undefined);
|
||||
records.push({ id, expiresAt, modifiedAt: info?.mtimeMs ?? 0 });
|
||||
}
|
||||
|
||||
const overflow = Math.max(0, records.length - this.maxRecords);
|
||||
for (const record of records.sort((a, b) => a.modifiedAt - b.modifiedAt).slice(0, overflow)) {
|
||||
if (record.id !== options.preserveSnapshotId) await rm(this.snapshotPath(record.id), { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
private snapshotPath(snapshotId: string): string {
|
||||
return join(this.snapshotsDirectory, `${snapshotId}.json`);
|
||||
}
|
||||
|
||||
private latestPath(targetKey: string): string {
|
||||
return join(this.latestDirectory, `${targetKeySlug(targetKey)}.json`);
|
||||
}
|
||||
|
||||
private async readLatest(targetKey: string): Promise<{ snapshotId: string } | undefined> {
|
||||
const value = await readJson(this.latestPath(targetKey));
|
||||
return isLatestPointer(value) ? value : undefined;
|
||||
}
|
||||
|
||||
private async readRecord(snapshotId: string): Promise<SnapshotRecord | undefined> {
|
||||
const value = await readJson(this.snapshotPath(snapshotId));
|
||||
return isSnapshotRecord(value) ? value : undefined;
|
||||
}
|
||||
|
||||
private nextSnapshotId(): string {
|
||||
const timestamp = this.now().getTime();
|
||||
if (timestamp > lastSnapshotTimestamp) {
|
||||
lastSnapshotTimestamp = timestamp;
|
||||
snapshotSequence = 0;
|
||||
} else {
|
||||
snapshotSequence += 1;
|
||||
}
|
||||
return `cs_${lastSnapshotTimestamp.toString(36).padStart(10, "0")}${snapshotSequence.toString(36).padStart(3, "0")}${randomBytes(4).toString("hex")}`;
|
||||
}
|
||||
}
|
||||
|
||||
export function createComputerSnapshotStore(options: SnapshotStoreOptions = {}): ComputerSnapshotStore {
|
||||
return new ComputerSnapshotStore(options);
|
||||
}
|
||||
|
||||
function snapshotRequired(): ComputerUseError {
|
||||
return new ComputerUseError(
|
||||
"SNAPSHOT_REQUIRED",
|
||||
"No current snapshot exists for this app; re-run fn computer get-app-state.",
|
||||
"Re-run fn computer get-app-state before using an element index.",
|
||||
);
|
||||
}
|
||||
|
||||
function snapshotStale(reason: "not-found" | "superseded" | "expired" | "pid-changed" | "window-mismatch", snapshotId: string): ComputerUseError {
|
||||
return new ComputerUseError(
|
||||
"SNAPSHOT_STALE",
|
||||
`Snapshot ${snapshotId} is ${reason}; re-run fn computer get-app-state.`,
|
||||
"Re-run fn computer get-app-state before retrying this action.",
|
||||
{ reason, snapshotId },
|
||||
);
|
||||
}
|
||||
|
||||
async function writeJsonAtomically(path: string, value: unknown): Promise<void> {
|
||||
const temporary = `${path}.${process.pid}.${randomBytes(6).toString("hex")}.tmp`;
|
||||
await writeFile(temporary, `${JSON.stringify(value)}\n`, "utf8");
|
||||
await rename(temporary, path);
|
||||
}
|
||||
|
||||
async function readJson(path: string): Promise<unknown | undefined> {
|
||||
try {
|
||||
return JSON.parse(await readFile(path, "utf8")) as unknown;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async function readDirectory(path: string): Promise<string[]> {
|
||||
try {
|
||||
return await readdir(path);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
function isLatestPointer(value: unknown): value is { snapshotId: string } {
|
||||
return !!value && typeof value === "object" && typeof (value as { snapshotId?: unknown }).snapshotId === "string";
|
||||
}
|
||||
|
||||
function isSnapshotRecord(value: unknown): value is SnapshotRecord {
|
||||
const record = value as Partial<SnapshotRecord> | null;
|
||||
return !!record
|
||||
&& typeof record.snapshotId === "string"
|
||||
&& typeof record.targetKey === "string"
|
||||
&& typeof record.windowKey === "string"
|
||||
&& typeof record.capturedAt === "string"
|
||||
&& typeof record.expiresAt === "string"
|
||||
&& !!record.app && typeof record.app.pid === "number"
|
||||
&& !!record.window && typeof record.window.windowId === "string"
|
||||
&& typeof record.elementCount === "number"
|
||||
&& !!record.elements && typeof record.elements === "object";
|
||||
}
|
||||
Reference in New Issue
Block a user