diff --git a/.changeset/fn-8960-computer-use-cli.md b/.changeset/fn-8960-computer-use-cli.md new file mode 100644 index 0000000000..082d15300f --- /dev/null +++ b/.changeset/fn-8960-computer-use-cli.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": minor +--- + +summary: Add macOS `fn computer` desktop-app discovery, snapshots, actions, and permission reporting. +category: feature +dev: Adds capabilities, permissions, discovery, state, and action subcommands with a versioned JSON envelope, preflight outcome matrix, durable cross-invocation snapshots, deterministic error ordering, and an honest macOS-only support matrix. diff --git a/docs/README.md b/docs/README.md index 8a180e0cc3..ffdd4b646d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,6 +22,7 @@ For a full walkthrough (installation, onboarding, first task, and daily workflow | [Getting Started](./getting-started.md) | Installation, first-run, first task, and daily workflow basics | | [Dashboard Guide](./dashboard-guide.md) | Board/list views, left/right sidebar navigation, Artifacts, Import Tasks, chat, workflow selection/editor, terminal, git manager, files, planning, and UI tools | | [CLI Reference](./cli-reference.md) | Complete `fn` command reference with subcommands, flags, and examples | +| [Computer Use](./computer-use.md) | `fn computer` desktop-app discovery, snapshots, actions, permissions, and JSON contract | | [Remote Access](./remote-access.md) | Operator runbook for Tailscale/Cloudflare setup, tokenized login links, security caveats, and troubleshooting | | [Native Shell Connection Guide](./native-shell.md) | Canonical mobile/desktop shell onboarding, profile management, QR/manual setup, and remote handoff behavior | diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 99c8ddd610..fac0f752a8 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -4,6 +4,27 @@ Fusion’s command-line interface is exposed through the `fn` command. +## `fn computer` — local desktop automation + +`fn computer` discovers and operates local desktop application windows through a **snapshot → act → snapshot** loop. It is supported on macOS only; other platforms report an honest unsupported capability. Every command supports `--json` and returns the versioned computer-use envelope. See the full [Computer Use reference](./computer-use.md) for setup, permissions, output shapes, snapshot safety, and error handling. + +```bash +fn computer capabilities --json +fn computer permissions --json +fn computer list-apps --json +fn computer list-windows --app --json +fn computer get-app-state --app [--window-id | --window-index ] [--no-screenshot] [--restore-window] --json +fn computer click --app --element-index [--snapshot-id ] [--window-id | --window-index ] --json +fn computer set-value --app --element-index (--value | --value-stdin) [--snapshot-id ] --json +fn computer type-text --app (--text | --text-stdin) [--element-index ] [--snapshot-id ] --json +fn computer press-key --app --key [--element-index ] [--snapshot-id ] --json +fn computer hotkey --app --keys --json +fn computer scroll --app --direction [--amount ] [--element-index ] [--snapshot-id ] --json +fn computer drag --app (--from-x --from-y --to-x --to-y | --from-element-index --to-element-index ) [--snapshot-id ] --json +``` + +Element actions use the latest persisted app snapshot unless `--snapshot-id` supplies a current-snapshot concurrency fence. Element indexes are sparse and snapshot-scoped; read `snapshot.elements[].index`, then re-snapshot after UI changes. Use `--value-stdin` and `--text-stdin` for secrets. Screenshots return a filesystem path, never image bytes. +