fusion/docs/computer-use.md

Computer Use CLI

← Docs index · CLI reference

fn computer inspects and operates local desktop application windows. It is designed for a safe, repeatable snapshot → act → snapshot loop: capture the current accessibility state, perform one deliberate action against that capture, then capture again before relying on the UI after navigation, focus changes, scrolling, or rendering.

Platform support: macOS is supported using only operating-system-provided osascript and screencapture. Linux, Windows, and other platforms are honestly unsupported: capabilities and permissions succeed with supported: false; all other commands fail with UNSUPPORTED_PLATFORM. Fusion does not download a helper, native module, or automation dependency.

Setup and permissions

Run the following first:

fn computer capabilities --json
fn computer permissions --json

On macOS, grant the application/process running fn both permissions in System Settings:

1. Privacy & Security → Accessibility: enable the terminal application, IDE, or host process that launches fn.
2. Privacy & Security → Screen Recording: enable that same host when screenshots are needed.
3. Quit and reopen the host application if macOS asks for it, then rerun fn computer permissions --json.

permissions reports each check as granted, denied, or unknown, not a guessed yes/no. It uses non-mutating preflights through the JXA Objective-C bridge: AXIsProcessTrusted for Accessibility and CGPreflightScreenCaptureAccess for Screen Recording. A probe that cannot run is unknown, never a claimed grant.

Do not use a successful screencapture as proof that Screen Recording is granted. macOS can return a normal-looking, desktop-only image when permission is denied. A denied Screen Recording grant therefore does not capture; an unknown grant may capture, but marks verifiedPermission: false.

Quick workflow

Use a bundle ID when available; it is the least ambiguous target:

fn computer list-apps --json
fn computer get-app-state --app com.apple.Safari --json
# Inspect result.snapshot.elements[].index and save result.snapshot.snapshotId.
fn computer click --app com.apple.Safari --element-index 42 --snapshot-id cs_01HABCDE123 --json
fn computer get-app-state --app com.apple.Safari --json

The first command persists the snapshot before it prints its snapshotId. The later click may run in a completely separate fn process; --snapshot-id is an optimistic-concurrency fence that confirms the snapshot is still the current capture for that app.

Element indexes are snapshot-scoped and sparse. Read indexes from snapshot.elements[].index; never derive an index from elementCount, position, or bounds. Refresh state after navigation, focus change, scrolling, or re-rendering. Semantic actions (click, set-value) are preferred over raw keys because they survive focus changes better.

App resolution for a bare --app value is: exact bundle ID, then exact unambiguous name, then pid:<n>. Ambiguous names fail with AMBIGUOUS_APP.

Commands and flags

All commands support --json.

Command	Required flags	Optional flags
capabilities	—	—
permissions	—	—
list-apps	—	—
list-windows	--app <target>	—
get-app-state	--app <target>	--window-id <id> or --window-index <n>, --no-screenshot, --restore-window
click	--app, --element-index <n>	--snapshot-id <id>, --window-id <id> or --window-index <n>
set-value	--app, --element-index <n>, one of --value <text> or --value-stdin	snapshot/window flags
type-text	--app, one of --text <text> or --text-stdin	--element-index, snapshot/window flags
press-key	--app, --key <name>	--element-index, snapshot/window flags
hotkey	--app, --keys <combo> (repeatable or +-joined)	—
scroll	--app, --direction up|down|left|right	--amount <n> (default 3), --element-index, snapshot/window flags
drag	--app, either all --from-x --from-y --to-x --to-y or both --from-element-index --to-element-index	snapshot/window flags for element form only

--window-id and --window-index are mutually exclusive. The value of --snapshot-id must match ^cs_[A-Za-z0-9]{10,40}$; it is validated before filesystem access. Numeric indexes, coordinates, and amounts must be non-negative. Do not mix or partially provide drag forms. hotkey and coordinate-only drag take neither snapshot nor window flags.

Use --value-stdin and --text-stdin for secrets. Fusion does not put stdin values in output, process arguments, snapshot records, logs, audit metadata, or error messages.

Targetless type-text, press-key, and scroll act on the target app's currently focused UI element and never activate or raise the app implicitly. If nothing is focused, they fail with ACTION_FAILED; focus the app yourself or pass --element-index.

JSON contract (schema version 1)

Every --json invocation prints exactly one object to stdout and nothing else. Success has result only; failure has error only:

{"schemaVersion":1,"ok":true,"command":"computer.get-app-state","result":{}}

{"schemaVersion":1,"ok":false,"command":"computer.click","error":{"code":"SNAPSHOT_STALE","message":"Snapshot expired.","remediation":"Re-run `fn computer get-app-state --app <app>`.","details":{"reason":"expired"}}}

schemaVersion is literal 1 and changes only for breaking reshapes. command is computer.<subcommand> using a runtime-inventory token. Group-routing failures (unknown or missing subcommand in JSON mode) uniquely use command: "computer". error.details, when present, is a flat map of string, number, or boolean values and never contains a secret. Diagnostics go to stderr; in human mode successful summaries go to stdout and failures are error: <code>: <message> on stderr.

The runtime exports frozen COMPUTER_SUBCOMMANDS, COMPUTER_ACTIONS, and COMPUTER_ERROR_CODES tuples; their TypeScript unions and dispatcher derive from them. Adding a command requires updating this runtime inventory. Exit code is 0 for success and 1 for every failure. Error codes are append-only:

Code	Meaning
UNSUPPORTED_PLATFORM	No supported adapter (except capability/permission reporting)
PERMISSION_DENIED	Definitively denied required permission or missing required built-in
PERMISSION_UNVERIFIED	Unknown probe followed by a permission-shaped OS failure
INVALID_ARGUMENTS	Invalid command, flags, number, enum, or snapshot ID
APP_NOT_FOUND / AMBIGUOUS_APP	No app match / more than one name match
WINDOW_NOT_FOUND	Supplied live window selector did not match
SNAPSHOT_REQUIRED	An element action has no snapshot for the app
SNAPSHOT_STALE	Snapshot is no longer usable; see below
ELEMENT_INDEX_NOT_FOUND	Sparse index is absent from the snapshot
ELEMENT_UNRESOLVABLE	Recorded locator no longer resolves or identity differs
ACTION_UNSUPPORTED / ACTION_FAILED	Adapter does not support action / OS action failed
SCREENSHOT_FAILED	Screenshot-only failure, reported in screenshotError rather than top-level error
TIMEOUT / INTERNAL	Timed-out seam / unexpected, redacted failure

Remediation is required for unsupported platform, permission denied/unverified, snapshot required/stale, missing/unresolvable element, ambiguous app, and unsupported action.

Result values

• capabilities: { platform, adapterId, supported, actions, unsupportedActions, features: { screenshot, restoreWindow, stdinSecrets, crossInvocationSnapshots } }. Every action is in exactly one action list.
• permissions: { platform, adapterId, supported, allGranted, checks }, where each check is { id: "accessibility"|"screen-recording", status: "granted"|"denied"|"unknown", granted, probed, probe, detail, remediation }. granted is exactly status === "granted"; allGranted is the AND of checks.
• list-apps: { apps }, sorted by name. An app is { bundleId, name, pid }.
• list-windows: { app, windows }. A window is { windowId, windowIndex, title, bounds, minimized }.
• get-app-state: { app, window, snapshot, screenshot, screenshotError? }. snapshot has { snapshotId, targetKey, windowKey, capturedAt, expiresAt, treeText, elementCount, truncated, elements }. An element is { index, role, title, value, label, enabled, focused, bounds, actions, locator }; its locator is { kind: "ax-path", path, role, subrole, identifier, title }.
• Actions return { action, app, snapshotId, elementIndex, fromElementIndex, toElementIndex, performed: true } only after the OS action succeeds. Single-endpoint actions use elementIndex; element-form drag uses fromElementIndex and toElementIndex with elementIndex: null; hotkey reports all index fields and snapshotId as null.

Screenshots are always paths, never base64, data: URLs, or byte arrays. --no-screenshot produces screenshot: null with no screenshotError; successful capture has screenshot and no error; failed/not-captured has screenshot: null and screenshotError. Screenshot verifiedPermission is true only for a preflight-confirmed Screen Recording grant.

Permissions outcome matrix

Check/status	Command behavior
Accessibility granted	Proceed.
Accessibility denied	Before automation, fail PERMISSION_DENIED with details.permission: "accessibility".
Accessibility unknown	Attempt the call. A permission-shaped failure becomes PERMISSION_UNVERIFIED; other failures are ACTION_FAILED or TIMEOUT. This applies to window/locator replay too.
capabilities or permissions	Never fail because of permission status; return status data.
Screen Recording granted	Capture; successful image has verifiedPermission: true; a failed capture is SCREENSHOT_FAILED or TIMEOUT.
Screen Recording denied	Do not capture; return screenshot null and screenshotError.code: "PERMISSION_DENIED".
Screen Recording unknown	Attempt capture; success has verifiedPermission: false; failure is PERMISSION_UNVERIFIED or TIMEOUT.
Any Screen Recording status with --no-screenshot	Do not probe or capture; screenshot is null without an error.
Missing osascript	Checks are unknown; non-reporting commands fail PERMISSION_DENIED with details.missingBinary: "osascript".
Missing screencapture	Only screenshot degrades with SCREENSHOT_FAILED.

Durable snapshots and safe element replay

Snapshots are stored per project at .fusion/computer-use/snapshots/<snapshotId>.json; the app's latest pointer is .fusion/computer-use/latest/<targetKeySlug>.json. They are never shared across project roots. Capture atomically persists the record and pointer before returning snapshotId.

A resolved app has app-scoped targetKey (bundle:<bundleId>, or pid:<pid>) and window-scoped windowKey (<targetKey>#<windowId>). There is one latest pointer per app, not per window. An action with no --snapshot-id uses that latest snapshot. Action window flags are optional assertions: a supplied selector that differs from the recorded window produces SNAPSHOT_STALE / window-mismatch.

Snapshots expire after five minutes by default; expiresAt publishes the exact deadline. An explicit snapshot ID is a concurrency fence, not a way to revive old UI: a superseded ID fails. Before acting, Fusion re-resolves the recorded window and then the locator rooted in it, verifying role, subrole, and recorded identifier. It never acts on a new occupant of the old path and never falls back to saved bounds/coordinates.

Failure	When	Recovery
SNAPSHOT_REQUIRED	No latest snapshot exists for the target app	Run get-app-state.
SNAPSHOT_STALE	not-found, superseded, expired, pid-changed, window-mismatch, or window-gone	Run get-app-state; use the current window and snapshot.
ELEMENT_INDEX_NOT_FOUND	Index is absent from sparse map	Read the current elements[].index after re-snapshotting.
ELEMENT_UNRESOLVABLE	Locator path fails or identity no longer matches	Re-snapshot; do not retry with coordinates.

Deterministic failure order and timeouts

Exactly one error is emitted: the first failing stage wins. The order is group routing; flag parsing; platform resolution; action support; built-in and permission gate; stdin read; app resolution; supplied-window resolution; snapshot resolution; sparse element lookup; window/locator replay; OS call. Thus malformed flags beat unsupported platform, and a permission-shaped replay error is PERMISSION_UNVERIFIED, not an element error. Unexpected failures become a redacted INTERNAL envelope, never a stack trace on stdout.

No user timeout flag is provided. Default seam timeouts are: permission probe 5 seconds; app/window listing 10 seconds; state capture 20 seconds; screenshot 15 seconds; locator replay 10 seconds; action 10 seconds.

Orca prior art and Fusion differences

This surface was informed by Orca and its computer-use CLI documentation. Orca uses two layers: a native-backed orca computer CLI and a thin skill that loads a version-matched guide. Fusion follows the CLI-plus-skill separation; the skill is deliberately out of scope for this command and is delivered separately.

Fusion deliberately differs by using macOS OS built-ins instead of native helpers; publishing non-mutating preflight permission checks, an unknown status, and the outcome matrix; documenting durable on-disk cross-process snapshots with identity-verified window/locator replay; distinguishing stale, missing, and unresolvable indexes; publishing deterministic failure precedence; and using a versioned envelope with a fixed append-only error enum. paste-text and perform-secondary-action are not implemented in this release; they remain absent from capabilities rather than being stubs.

Use it from an agent

Fusion ships a thin computer-use skill alongside the Fusion skill. When Claude-compatible skill installation is configured, both are reconciled into project .claude/skills/ directories; fn init also copies both into supported home skill directories.

The discovery stub intentionally contains no command reference. An agent resolves one fn executable for its session and runs fn skills get computer-use before automation. That command renders its complete guide in-process from the same binary's command-surface descriptor and reports the same package version as fn --version.

This is an anti-drift contract: dispatch keys, parser flag literals, required/mutually-exclusive validation, and emitted error codes are independently checked against the descriptor; the rendered guide then includes every command, flag, and error code without truncation. The delivered parser still does not reject unknown flags or invalid enum choices, and a flag literal's source presence does not itself prove a handler honors it.