From 1da61f35f7d902542729949c1688444e31e54ea5 Mon Sep 17 00:00:00 2001 From: gsxdsm Date: Mon, 10 Aug 2026 21:56:18 -0700 Subject: [PATCH] FN-8960: add macOS computer automation CLI Add a snapshot-backed macOS computer-use surface to the Fusion CLI. - Register fn computer commands for app state, permission, capability, and UI actions. - Implement macOS JXA automation with locator replay and persisted snapshot safety checks. - Document the CLI surface, add contract coverage, and include a release changeset. Files changed: .changeset/fn-8960-computer-use-cli.md | 7 + docs/README.md | 1 + docs/cli-reference.md | 21 ++ docs/computer-use.md | 156 +++++++++++++ packages/cli/src/bin.ts | 11 + .../__tests__/computer-adapter-registry.test.ts | 154 ++++++++++++ .../commands/__tests__/computer-commands.test.ts | 71 ++++++ .../commands/__tests__/computer-contract.test.ts | 34 +++ .../__tests__/computer-snapshot-index.test.ts | 87 +++++++ packages/cli/src/commands/computer.ts | 182 +++++++++++++++ .../cli/src/commands/computer/adapter-macos.ts | 211 +++++++++++++++++ .../cli/src/commands/computer/adapter-registry.ts | 42 ++++ .../src/commands/computer/adapter-unsupported.ts | 44 ++++ packages/cli/src/commands/computer/adapter.ts | 97 ++++++++ packages/cli/src/commands/computer/contract.ts | 83 +++++++ packages/cli/src/commands/computer/exec-seam.ts | 61 +++++ .../cli/src/commands/computer/scripts/macos-jxa.ts | 37 +++ .../cli/src/commands/computer/snapshot-store.ts | 257 +++++++++++++++++++++ 18 files changed, 1556 insertions(+) Fusion-Task-Id: FN-8960 Fusion-Task-Lineage: 5870bf69-b751-4879-bf9d-5f5396cc0f55 Co-authored-by: Fusion (runfusion.ai) --- .changeset/fn-8960-computer-use-cli.md | 7 + docs/README.md | 1 + docs/cli-reference.md | 21 ++ docs/computer-use.md | 156 +++++++++++ packages/cli/src/bin.ts | 11 + .../computer-adapter-registry.test.ts | 154 +++++++++++ .../__tests__/computer-commands.test.ts | 71 +++++ .../__tests__/computer-contract.test.ts | 34 +++ .../__tests__/computer-snapshot-index.test.ts | 87 ++++++ packages/cli/src/commands/computer.ts | 182 +++++++++++++ .../src/commands/computer/adapter-macos.ts | 211 ++++++++++++++ .../src/commands/computer/adapter-registry.ts | 42 +++ .../commands/computer/adapter-unsupported.ts | 44 +++ packages/cli/src/commands/computer/adapter.ts | 97 +++++++ .../cli/src/commands/computer/contract.ts | 83 ++++++ .../cli/src/commands/computer/exec-seam.ts | 61 +++++ .../commands/computer/scripts/macos-jxa.ts | 37 +++ .../src/commands/computer/snapshot-store.ts | 257 ++++++++++++++++++ 18 files changed, 1556 insertions(+) create mode 100644 .changeset/fn-8960-computer-use-cli.md create mode 100644 docs/computer-use.md create mode 100644 packages/cli/src/commands/__tests__/computer-adapter-registry.test.ts create mode 100644 packages/cli/src/commands/__tests__/computer-commands.test.ts create mode 100644 packages/cli/src/commands/__tests__/computer-contract.test.ts create mode 100644 packages/cli/src/commands/__tests__/computer-snapshot-index.test.ts create mode 100644 packages/cli/src/commands/computer.ts create mode 100644 packages/cli/src/commands/computer/adapter-macos.ts create mode 100644 packages/cli/src/commands/computer/adapter-registry.ts create mode 100644 packages/cli/src/commands/computer/adapter-unsupported.ts create mode 100644 packages/cli/src/commands/computer/adapter.ts create mode 100644 packages/cli/src/commands/computer/contract.ts create mode 100644 packages/cli/src/commands/computer/exec-seam.ts create mode 100644 packages/cli/src/commands/computer/scripts/macos-jxa.ts create mode 100644 packages/cli/src/commands/computer/snapshot-store.ts diff --git a/.changeset/fn-8960-computer-use-cli.md b/.changeset/fn-8960-computer-use-cli.md new file mode 100644 index 0000000000..082d15300f --- /dev/null +++ b/.changeset/fn-8960-computer-use-cli.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": minor +--- + +summary: Add macOS `fn computer` desktop-app discovery, snapshots, actions, and permission reporting. +category: feature +dev: Adds capabilities, permissions, discovery, state, and action subcommands with a versioned JSON envelope, preflight outcome matrix, durable cross-invocation snapshots, deterministic error ordering, and an honest macOS-only support matrix. diff --git a/docs/README.md b/docs/README.md index 8a180e0cc3..ffdd4b646d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,6 +22,7 @@ For a full walkthrough (installation, onboarding, first task, and daily workflow | [Getting Started](./getting-started.md) | Installation, first-run, first task, and daily workflow basics | | [Dashboard Guide](./dashboard-guide.md) | Board/list views, left/right sidebar navigation, Artifacts, Import Tasks, chat, workflow selection/editor, terminal, git manager, files, planning, and UI tools | | [CLI Reference](./cli-reference.md) | Complete `fn` command reference with subcommands, flags, and examples | +| [Computer Use](./computer-use.md) | `fn computer` desktop-app discovery, snapshots, actions, permissions, and JSON contract | | [Remote Access](./remote-access.md) | Operator runbook for Tailscale/Cloudflare setup, tokenized login links, security caveats, and troubleshooting | | [Native Shell Connection Guide](./native-shell.md) | Canonical mobile/desktop shell onboarding, profile management, QR/manual setup, and remote handoff behavior | diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 99c8ddd610..fac0f752a8 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -4,6 +4,27 @@ Fusion’s command-line interface is exposed through the `fn` command. +## `fn computer` — local desktop automation + +`fn computer` discovers and operates local desktop application windows through a **snapshot → act → snapshot** loop. It is supported on macOS only; other platforms report an honest unsupported capability. Every command supports `--json` and returns the versioned computer-use envelope. See the full [Computer Use reference](./computer-use.md) for setup, permissions, output shapes, snapshot safety, and error handling. + +```bash +fn computer capabilities --json +fn computer permissions --json +fn computer list-apps --json +fn computer list-windows --app --json +fn computer get-app-state --app [--window-id | --window-index ] [--no-screenshot] [--restore-window] --json +fn computer click --app --element-index [--snapshot-id ] [--window-id | --window-index ] --json +fn computer set-value --app --element-index (--value | --value-stdin) [--snapshot-id ] --json +fn computer type-text --app (--text | --text-stdin) [--element-index ] [--snapshot-id ] --json +fn computer press-key --app --key [--element-index ] [--snapshot-id ] --json +fn computer hotkey --app --keys --json +fn computer scroll --app --direction [--amount ] [--element-index ] [--snapshot-id ] --json +fn computer drag --app (--from-x --from-y --to-x --to-y | --from-element-index --to-element-index ) [--snapshot-id ] --json +``` + +Element actions use the latest persisted app snapshot unless `--snapshot-id` supplies a current-snapshot concurrency fence. Element indexes are sparse and snapshot-scoped; read `snapshot.elements[].index`, then re-snapshot after UI changes. Use `--value-stdin` and `--text-stdin` for secrets. Screenshots return a filesystem path, never image bytes. +