fusion/docs/computer-use.md

# Computer Use CLI

[← Docs index](./README.md) · [CLI reference](./cli-reference.md)

`fn computer` inspects and operates local desktop application windows. It is designed for a safe, repeatable **snapshot → act → snapshot** loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or rendering.

> **Platform support:** macOS is supported using only operating-system-provided `osascript` and `screencapture`. Linux, Windows, and other platforms are honestly unsupported: `capabilities` and `permissions` succeed with `supported: false`; all other commands fail with `UNSUPPORTED_PLATFORM`. Fusion does not download a helper, native module, or automation dependency.

## Setup and permissions

Run the following first:

```bash
fn computer capabilities --json
fn computer permissions --json
```

On macOS, grant the application/process running `fn` both permissions in **System Settings**:

1. **Privacy & Security → Accessibility**: enable the terminal application, IDE, or host process that launches `fn`.
2. **Privacy & Security → Screen Recording**: enable that same host when screenshots are needed.
3. Quit and reopen the host application if macOS asks for it, then rerun `fn computer permissions --json`.

`permissions` reports each check as `granted`, `denied`, or `unknown`, not a guessed yes/no. It uses non-mutating preflights through the JXA Objective-C bridge: `AXIsProcessTrusted` for Accessibility and `CGPreflightScreenCaptureAccess` for Screen Recording. A probe that cannot run is `unknown`, never a claimed grant.

Do **not** use a successful `screencapture` as proof that Screen Recording is granted. macOS can return a normal-looking, desktop-only image when permission is denied. A denied Screen Recording grant therefore does not capture; an unknown grant may capture, but marks `verifiedPermission: false`.

## Quick workflow

Use a bundle ID when available; it is the least ambiguous target:

```bash
fn computer list-apps --json
fn computer get-app-state --app com.apple.Safari --json
# Inspect result.snapshot.elements[].index and save result.snapshot.snapshotId.
fn computer click --app com.apple.Safari --element-index 42 --snapshot-id cs_01HABCDE123 --json
fn computer get-app-state --app com.apple.Safari --json
```

The first command persists the snapshot before it prints its `snapshotId`. The later `click` may run in a completely separate `fn` process; `--snapshot-id` is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app.

Element indexes are snapshot-scoped and sparse. Read indexes from `snapshot.elements[].index`; never derive an index from `elementCount`, position, or bounds. Refresh state after navigation, focus change, scrolling, or re-rendering. Semantic actions (`click`, `set-value`) are preferred over raw keys because they survive focus changes better.

App resolution for a bare `--app` value is: exact bundle ID, then exact unambiguous name, then `pid:<n>`. Ambiguous names fail with `AMBIGUOUS_APP`.

## Commands and flags

All commands support `--json`.

| Command | Required flags | Optional flags |
| --- | --- | --- |
| `capabilities` | — | — |
| `permissions` | — | — |
| `list-apps` | — | — |
| `list-windows` | `--app <target>` | — |
| `get-app-state` | `--app <target>` | `--window-id <id>` or `--window-index <n>`, `--no-screenshot`, `--restore-window` |
| `click` | `--app`, `--element-index <n>` | `--snapshot-id <id>`, `--window-id <id>` or `--window-index <n>` |
| `set-value` | `--app`, `--element-index <n>`, one of `--value <text>` or `--value-stdin` | snapshot/window flags |
| `type-text` | `--app`, one of `--text <text>` or `--text-stdin` | `--element-index`, snapshot/window flags |
| `press-key` | `--app`, `--key <name>` | `--element-index`, snapshot/window flags |
| `hotkey` | `--app`, `--keys <combo>` (repeatable or `+`-joined) | — |
| `scroll` | `--app`, `--direction up\|down\|left\|right` | `--amount <n>` (default 3), `--element-index`, snapshot/window flags |
| `drag` | `--app`, either all `--from-x --from-y --to-x --to-y` or both `--from-element-index --to-element-index` | snapshot/window flags for element form only |

`--window-id` and `--window-index` are mutually exclusive. The value of `--snapshot-id` must match `^cs_[A-Za-z0-9]{10,40}$`; it is validated before filesystem access. Numeric indexes, coordinates, and amounts must be non-negative. Do not mix or partially provide drag forms. `hotkey` and coordinate-only `drag` take neither snapshot nor window flags.

Use `--value-stdin` and `--text-stdin` for secrets. Fusion does not put stdin values in output, process arguments, snapshot records, logs, audit metadata, or error messages.

Targetless `type-text`, `press-key`, and `scroll` act on the target app's currently focused UI element and never activate or raise the app implicitly. If nothing is focused, they fail with `ACTION_FAILED`; focus the app yourself or pass `--element-index`.

## JSON contract (schema version 1)

Every `--json` invocation prints exactly one object to stdout and nothing else. Success has `result` only; failure has `error` only:

```json
{"schemaVersion":1,"ok":true,"command":"computer.get-app-state","result":{}}
```

```json
{"schemaVersion":1,"ok":false,"command":"computer.click","error":{"code":"SNAPSHOT_STALE","message":"Snapshot expired.","remediation":"Re-run `fn computer get-app-state --app <app>`.","details":{"reason":"expired"}}}
```

`schemaVersion` is literal `1` and changes only for breaking reshapes. `command` is `computer.<subcommand>` using a runtime-inventory token. Group-routing failures (unknown or missing subcommand in JSON mode) uniquely use `command: "computer"`. `error.details`, when present, is a flat map of string, number, or boolean values and never contains a secret. Diagnostics go to stderr; in human mode successful summaries go to stdout and failures are `error: <code>: <message>` on stderr.

The runtime exports frozen `COMPUTER_SUBCOMMANDS`, `COMPUTER_ACTIONS`, and `COMPUTER_ERROR_CODES` tuples; their TypeScript unions and dispatcher derive from them. Adding a command requires updating this runtime inventory. Exit code is `0` for success and `1` for every failure. Error codes are append-only:

| Code | Meaning |
| --- | --- |
| `UNSUPPORTED_PLATFORM` | No supported adapter (except capability/permission reporting) |
| `PERMISSION_DENIED` | Definitively denied required permission or missing required built-in |
| `PERMISSION_UNVERIFIED` | Unknown probe followed by a permission-shaped OS failure |
| `INVALID_ARGUMENTS` | Invalid command, flags, number, enum, or snapshot ID |
| `APP_NOT_FOUND` / `AMBIGUOUS_APP` | No app match / more than one name match |
| `WINDOW_NOT_FOUND` | Supplied live window selector did not match |
| `SNAPSHOT_REQUIRED` | An element action has no snapshot for the app |
| `SNAPSHOT_STALE` | Snapshot is no longer usable; see below |
| `ELEMENT_INDEX_NOT_FOUND` | Sparse index is absent from the snapshot |
| `ELEMENT_UNRESOLVABLE` | Recorded locator no longer resolves or identity differs |
| `ACTION_UNSUPPORTED` / `ACTION_FAILED` | Adapter does not support action / OS action failed |
| `SCREENSHOT_FAILED` | Screenshot-only failure, reported in `screenshotError` rather than top-level error |
| `TIMEOUT` / `INTERNAL` | Timed-out seam / unexpected, redacted failure |

Remediation is required for unsupported platform, permission denied/unverified, snapshot required/stale, missing/unresolvable element, ambiguous app, and unsupported action.

### Result values

- `capabilities`: `{ platform, adapterId, supported, actions, unsupportedActions, features: { screenshot, restoreWindow, stdinSecrets, crossInvocationSnapshots } }`. Every action is in exactly one action list.
- `permissions`: `{ platform, adapterId, supported, allGranted, checks }`, where each check is `{ id: "accessibility"|"screen-recording", status: "granted"|"denied"|"unknown", granted, probed, probe, detail, remediation }`. `granted` is exactly `status === "granted"`; `allGranted` is the AND of checks.
- `list-apps`: `{ apps }`, sorted by name. An app is `{ bundleId, name, pid }`.
- `list-windows`: `{ app, windows }`. A window is `{ windowId, windowIndex, title, bounds, minimized }`.
- `get-app-state`: `{ app, window, snapshot, screenshot, screenshotError? }`. `snapshot` has `{ snapshotId, targetKey, windowKey, capturedAt, expiresAt, treeText, elementCount, truncated, elements }`. An element is `{ index, role, title, value, label, enabled, focused, bounds, actions, locator }`; its locator is `{ kind: "ax-path", path, role, subrole, identifier, title }`.
- Actions return `{ action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true }` only after the OS action succeeds. Single-endpoint actions use `elementIndex`; element-form drag uses `fromElementIndex` and `toElementIndex` with `elementIndex: null`; hotkey reports all index fields and `snapshotId` as `null`.

Screenshots are always paths, never base64, `data:` URLs, or byte arrays. `--no-screenshot` produces `screenshot: null` with no `screenshotError`; successful capture has `screenshot` and no error; failed/not-captured has `screenshot: null` and `screenshotError`. Screenshot `verifiedPermission` is true only for a preflight-confirmed Screen Recording grant.

## Permissions outcome matrix

| Check/status | Command behavior |
| --- | --- |
| Accessibility granted | Proceed. |
| Accessibility denied | Before automation, fail `PERMISSION_DENIED` with `details.permission: "accessibility"`. |
| Accessibility unknown | Attempt the call. A permission-shaped failure becomes `PERMISSION_UNVERIFIED`; other failures are `ACTION_FAILED` or `TIMEOUT`. This applies to window/locator replay too. |
| `capabilities` or `permissions` | Never fail because of permission status; return status data. |
| Screen Recording granted | Capture; successful image has `verifiedPermission: true`; a failed capture is `SCREENSHOT_FAILED` or `TIMEOUT`. |
| Screen Recording denied | Do not capture; return screenshot `null` and `screenshotError.code: "PERMISSION_DENIED"`. |
| Screen Recording unknown | Attempt capture; success has `verifiedPermission: false`; failure is `PERMISSION_UNVERIFIED` or `TIMEOUT`. |
| Any Screen Recording status with `--no-screenshot` | Do not probe or capture; screenshot is null without an error. |
| Missing `osascript` | Checks are unknown; non-reporting commands fail `PERMISSION_DENIED` with `details.missingBinary: "osascript"`. |
| Missing `screencapture` | Only screenshot degrades with `SCREENSHOT_FAILED`. |

## Durable snapshots and safe element replay

Snapshots are stored per project at `.fusion/computer-use/snapshots/<snapshotId>.json`; the app's latest pointer is `.fusion/computer-use/latest/<targetKeySlug>.json`. They are never shared across project roots. Capture atomically persists the record and pointer before returning `snapshotId`.

A resolved app has app-scoped `targetKey` (`bundle:<bundleId>`, or `pid:<pid>`) and window-scoped `windowKey` (`<targetKey>#<windowId>`). There is one latest pointer per app, not per window. An action with no `--snapshot-id` uses that latest snapshot. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces `SNAPSHOT_STALE` / `window-mismatch`.

Snapshots expire after five minutes by default; `expiresAt` publishes the exact deadline. An explicit snapshot ID is a concurrency fence, not a way to revive old UI: a superseded ID fails. Before acting, Fusion re-resolves the recorded window and then the locator rooted in it, verifying role, subrole, and recorded identifier. It never acts on a new occupant of the old path and never falls back to saved bounds/coordinates.

| Failure | When | Recovery |
| --- | --- | --- |
| `SNAPSHOT_REQUIRED` | No latest snapshot exists for the target app | Run `get-app-state`. |
| `SNAPSHOT_STALE` | `not-found`, `superseded`, `expired`, `pid-changed`, `window-mismatch`, or `window-gone` | Run `get-app-state`; use the current window and snapshot. |
| `ELEMENT_INDEX_NOT_FOUND` | Index is absent from sparse map | Read the current `elements[].index` after re-snapshotting. |
| `ELEMENT_UNRESOLVABLE` | Locator path fails or identity no longer matches | Re-snapshot; do not retry with coordinates. |

## Deterministic failure order and timeouts

Exactly one error is emitted: the first failing stage wins. The order is group routing; flag parsing; platform resolution; action support; built-in and permission gate; stdin read; app resolution; supplied-window resolution; snapshot resolution; sparse element lookup; window/locator replay; OS call. Thus malformed flags beat unsupported platform, and a permission-shaped replay error is `PERMISSION_UNVERIFIED`, not an element error. Unexpected failures become a redacted `INTERNAL` envelope, never a stack trace on stdout.

No user timeout flag is provided. Default seam timeouts are: permission probe 5 seconds; app/window listing 10 seconds; state capture 20 seconds; screenshot 15 seconds; locator replay 10 seconds; action 10 seconds.

## Orca prior art and Fusion differences

This surface was informed by [Orca](https://github.com/stablyai/orca) and its [computer-use CLI documentation](https://www.onorca.dev/docs/cli/computer-use). Orca uses two layers: a native-backed `orca computer` CLI and a thin skill that loads a version-matched guide. Fusion follows the CLI-plus-skill separation; the skill is deliberately out of scope for this command and is delivered separately.

Fusion deliberately differs by using macOS OS built-ins instead of native helpers; publishing non-mutating preflight permission checks, an `unknown` status, and the outcome matrix; documenting durable on-disk cross-process snapshots with identity-verified window/locator replay; distinguishing stale, missing, and unresolvable indexes; publishing deterministic failure precedence; and using a versioned envelope with a fixed append-only error enum. `paste-text` and `perform-secondary-action` are not implemented in this release; they remain absent from capabilities rather than being stubs.