fusion/docs/computer-use.md

Computer Use CLI

← Docs index · CLI reference

fn computer inspects and operates local desktop application windows. It is designed for a safe, repeatable snapshot → act → snapshot loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or rendering.

Platform support: macOS is supported using only operating-system-provided osascript and screencapture. Linux, Windows, and other platforms are honestly unsupported: capabilities and permissions succeed with supported: false; all other commands fail with UNSUPPORTED_PLATFORM. Fusion does not download a helper, native module, or automation dependency.

Setup and permissions

Run the following first:

fn computer capabilities --json
fn computer permissions --json

On macOS, grant the application/process running fn both permissions in System Settings:

1. Privacy & Security → Accessibility: enable the terminal application, IDE, or host process that launches fn.
2. Privacy & Security → Screen Recording: enable that same host when screenshots are needed.
3. Quit and reopen the host application if macOS asks for it, then rerun fn computer permissions --json.

permissions reports each check as granted, denied, or unknown, not a guessed yes/no. It uses non-mutating preflights through the JXA Objective-C bridge: AXIsProcessTrusted for Accessibility and CGPreflightScreenCaptureAccess for Screen Recording. A probe that cannot run is unknown, never a claimed grant.

Do not use a successful screencapture as proof that Screen Recording is granted. macOS can return a normal-looking, desktop-only image when permission is denied. A denied Screen Recording grant therefore does not capture; an unknown grant may capture, but marks verifiedPermission: false.

Quick workflow

Use a bundle ID when available; it is the least ambiguous target:

fn computer list-apps --json
fn computer get-app-state --app com.apple.Safari --json
# Inspect result.snapshot.elements[].index and save result.snapshot.snapshotId.
fn computer click --app com.apple.Safari --element-index 42 --snapshot-id cs_01HABCDE123 --json
fn computer get-app-state --app com.apple.Safari --json

The first command persists the snapshot before it prints its snapshotId. The later click may run in a completely separate fn process; --snapshot-id is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app.

Element indexes are snapshot-scoped and sparse. Read indexes from snapshot.elements[].index; never derive an index from elementCount, position, or bounds. Refresh state after navigation, focus change, scrolling, or re-rendering. Semantic actions (click, set-value) are preferred over raw keys because they survive focus changes better.

App resolution for a bare --app value is: exact bundle ID, then exact unambiguous name, then pid:<n>. Ambiguous names fail with AMBIGUOUS_APP.

Commands and flags

All commands support --json.

Command	Required flags	Optional flags
capabilities	—	—
permissions	—	—
list-apps	—	—
list-windows	--app <target>	—
get-app-state	--app <target>	--window-id <id> or --window-index <n>, --no-screenshot, --restore-window
click	--app, --element-index <n>	--snapshot-id <id>, --window-id <id> or --window-index <n>
set-value	--app, --element-index <n>, one of --value <text> or --value-stdin	snapshot/window flags
type-text	--app, one of --text <text> or --text-stdin	--element-index, snapshot/window flags
press-key	--app, --key <name>	--element-index, snapshot/window flags
hotkey	--app, --keys <combo> (repeatable or +-joined)	—
scroll	--app, --direction up|down|left|right	--amount <n> (default 3), --element-index, snapshot/window flags
drag	--app, either all --from-x --from-y --to-x --to-y or both --from-element-index --to-element-index	snapshot/window flags for element form only

--window-id and --window-index are mutually exclusive. The value of --snapshot-id must match ^cs_[A-Za-z0-9]{10,40}$; it is validated before filesystem access. Numeric indexes, coordinates, and amounts must be non-negative. Do not mix or partially provide drag forms. hotkey and coordinate-only drag take neither snapshot nor window flags.

Use --value-stdin and --text-stdin for secrets. Fusion does not put stdin values in output, process arguments, snapshot records, logs, audit metadata, or error messages.

Targetless type-text, press-key, and scroll act on the target app's currently focused UI element and never activate or raise the app implicitly. If nothing is focused, they fail with ACTION_FAILED; focus the app yourself or pass --element-index.

JSON contract (schema version 1)

Every --json invocation prints exactly one object to stdout and nothing else. Success has result only; failure has error only:

{"schemaVersion":1,"ok":true,"command":"computer.get-app-state","result":{}}

{"schemaVersion":1,"ok":false,"command":"computer.click","error":{"code":"SNAPSHOT_STALE","message":"Snapshot expired.","remediation":"Re-run `fn computer get-app-state --app <app>`.","details":{"reason":"expired"}}}

schemaVersion is literal 1 and changes only for breaking reshapes. command is computer.<subcommand> using a runtime-inventory token. Group-routing failures (unknown or missing subcommand in JSON mode) uniquely use command: "computer". error.details, when present, is a flat map of string, number, or boolean values and never contains a secret. Diagnostics go to stderr; in human mode successful summaries go to stdout and failures are error: <code>: <message> on stderr.

The runtime exports frozen COMPUTER_SUBCOMMANDS, COMPUTER_ACTIONS, and COMPUTER_ERROR_CODES tuples; their TypeScript unions and dispatcher derive from them. Adding a command requires updating this runtime inventory. Exit code is 0 for success and 1 for every failure. Error codes are append-only:

Code	Meaning
UNSUPPORTED_PLATFORM	No supported adapter (except capability/permission reporting)
PERMISSION_DENIED	Definitively denied required permission or missing required built-in
PERMISSION_UNVERIFIED	Unknown probe followed by a permission-shaped OS failure
INVALID_ARGUMENTS	Invalid command, flags, number, enum, or snapshot ID
APP_NOT_FOUND / AMBIGUOUS_APP	No app match / more than one name match
WINDOW_NOT_FOUND	Supplied live window selector did not match
SNAPSHOT_REQUIRED	An element action has no snapshot for the app
SNAPSHOT_STALE	Snapshot is no longer usable; see below
ELEMENT_INDEX_NOT_FOUND	Sparse index is absent from the snapshot
ELEMENT_UNRESOLVABLE	Recorded locator no longer resolves or identity differs
ACTION_UNSUPPORTED / ACTION_FAILED	Adapter does not support action / OS action failed
SCREENSHOT_FAILED	Screenshot-only failure, reported in screenshotError rather than top-level error
TIMEOUT / INTERNAL	Timed-out seam / unexpected, redacted failure

Remediation is required for unsupported platform, permission denied/unverified, snapshot required/stale, missing/unresolvable element, ambiguous app, and unsupported action.

Result values

• capabilities: { platform, adapterId, supported, actions, unsupportedActions, features: { screenshot, restoreWindow, stdinSecrets, crossInvocationSnapshots } }. Every action is in exactly one action list.
• permissions: { platform, adapterId, supported, allGranted, checks }, where each check is { id: "accessibility"|"screen-recording", status: "granted"|"denied"|"unknown", granted, probed, probe, detail, remediation }. granted is exactly status === "granted"; allGranted is the AND of checks.
• list-apps: { apps }, sorted by name. An app is { bundleId, name, pid }.
• list-windows: { app, windows }. A window is { windowId, windowIndex, title, bounds, minimized }.
• get-app-state: { app, window, snapshot, screenshot, screenshotError? }. snapshot has { snapshotId, targetKey, windowKey, capturedAt, expiresAt, treeText, elementCount, truncated, elements }. An element is { index, role, title, value, label, enabled, focused, bounds, actions, locator }; its locator is { kind: "ax-path", path, role, subrole, identifier, title }.
• Actions return { action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true } only after the OS action succeeds. Single-endpoint actions use elementIndex; element-form drag uses fromElementIndex and toElementIndex with elementIndex: null; hotkey reports all index fields and snapshotId as null.

Screenshots are always paths, never base64, data: URLs, or byte arrays. --no-screenshot produces screenshot: null with no screenshotError; successful capture has screenshot and no error; failed/not-captured has screenshot: null and screenshotError. Screenshot verifiedPermission is true only for a preflight-confirmed Screen Recording grant.

Permissions outcome matrix

Check/status	Command behavior
Accessibility granted	Proceed.
Accessibility denied	Before automation, fail PERMISSION_DENIED with details.permission: "accessibility".
Accessibility unknown	Attempt the call. A permission-shaped failure becomes PERMISSION_UNVERIFIED; other failures are ACTION_FAILED or TIMEOUT. This applies to window/locator replay too.
capabilities or permissions	Never fail because of permission status; return status data.
Screen Recording granted	Capture; successful image has verifiedPermission: true; a failed capture is SCREENSHOT_FAILED or TIMEOUT.
Screen Recording denied	Do not capture; return screenshot null and screenshotError.code: "PERMISSION_DENIED".
Screen Recording unknown	Attempt capture; success has verifiedPermission: false; failure is PERMISSION_UNVERIFIED or TIMEOUT.
Any Screen Recording status with --no-screenshot	Do not probe or capture; screenshot is null without an error.
Missing osascript	Checks are unknown; non-reporting commands fail PERMISSION_DENIED with details.missingBinary: "osascript".
Missing screencapture	Only screenshot degrades with SCREENSHOT_FAILED.

Durable snapshots and safe element replay

Snapshots are stored per project at .fusion/computer-use/snapshots/<snapshotId>.json; the app's latest pointer is .fusion/computer-use/latest/<targetKeySlug>.json. They are never shared across project roots. Capture atomically persists the record and pointer before returning snapshotId.

A resolved app has app-scoped targetKey (bundle:<bundleId>, or pid:<pid>) and window-scoped windowKey (<targetKey>#<windowId>). There is one latest pointer per app, not per window. An action with no --snapshot-id uses that latest snapshot. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces SNAPSHOT_STALE / window-mismatch.

Snapshots expire after five minutes by default; expiresAt publishes the exact deadline. An explicit snapshot ID is a concurrency fence, not a way to revive old UI: a superseded ID fails. Before acting, Fusion re-resolves the recorded window and then the locator rooted in it, verifying role, subrole, and recorded identifier. It never acts on a new occupant of the old path and never falls back to saved bounds/coordinates.

Failure	When	Recovery
SNAPSHOT_REQUIRED	No latest snapshot exists for the target app	Run get-app-state.
SNAPSHOT_STALE	not-found, superseded, expired, pid-changed, window-mismatch, or window-gone	Run get-app-state; use the current window and snapshot.
ELEMENT_INDEX_NOT_FOUND	Index is absent from sparse map	Read the current elements[].index after re-snapshotting.
ELEMENT_UNRESOLVABLE	Locator path fails or identity no longer matches	Re-snapshot; do not retry with coordinates.

Deterministic failure order and timeouts

Exactly one error is emitted: the first failing stage wins. The order is group routing; flag parsing; platform resolution; action support; built-in and permission gate; stdin read; app resolution; supplied-window resolution; snapshot resolution; sparse element lookup; window/locator replay; OS call. Thus malformed flags beat unsupported platform, and a permission-shaped replay error is PERMISSION_UNVERIFIED, not an element error. Unexpected failures become a redacted INTERNAL envelope, never a stack trace on stdout.

No user timeout flag is provided. Default seam timeouts are: permission probe 5 seconds; app/window listing 10 seconds; state capture 20 seconds; screenshot 15 seconds; locator replay 10 seconds; action 10 seconds.

Orca prior art and Fusion differences

This surface was informed by Orca and its computer-use CLI documentation. Orca uses two layers: a native-backed orca computer CLI and a thin skill that loads a version-matched guide. Fusion follows the CLI-plus-skill separation; the skill is deliberately out of scope for this command and is delivered separately.

Fusion deliberately differs by using macOS OS built-ins instead of native helpers; publishing non-mutating preflight permission checks, an unknown status, and the outcome matrix; documenting durable on-disk cross-process snapshots with identity-verified window/locator replay; distinguishing stale, missing, and unresolvable indexes; publishing deterministic failure precedence; and using a versioned envelope with a fixed append-only error enum. paste-text and perform-secondary-action are not implemented in this release; they remain absent from capabilities rather than being stubs.