fusion/docs/research/sandbox-options.md

FN-4635: Pluggable sandbox options for executor command isolation

1) Threat model

Fusion executor lanes currently run shell commands (testCommand, buildCommand, workflow script steps) as the host user inside a git worktree. This exposes the host if task content or agent-generated commands are malicious or buggy.

Primary threats:

• Host filesystem access/exfiltration: command reads/writes outside task worktree, including secrets in $HOME, SSH keys, cloud creds, and unrelated repos.
• Secret exfiltration over network: command uploads tokens, source, or .fusion metadata.
• Persistence / host tampering: command installs startup hooks, modifies shell rc, mutates global package/tool state.
• Runaway resource abuse: fork bombs, memory exhaustion, or very long-running scripts.
• Cross-task contamination: command mutates shared caches or shared worktree state in unsafe ways.
• Dashboard safety: accidental conflict with production dashboard process/port conventions (port 4040 must not be disrupted).

Non-goals for this phase:

• Perfect VM-grade multi-tenant isolation.
• Replacing host OS hardening strategy end-to-end.
• Sandboxing every agent role immediately (triage/reviewer may remain unchanged initially).

2) Requirements

Functional/operational requirements:

• Platform first target: macOS + Linux; document Windows path as follow-up.
• Low startup overhead: ideally sub-second to a few seconds for frequent executor invocations.
• Pluggable with WorktreeBackend plan: sandbox layer composes with current worktree lifecycle, not replacing it.
• Opt-in settings model: project/global defaults with task-level override.
• Action-gate friendly: policy-gated privileged/provisioning operations.
• Command compatibility: must still run realistic pnpm test, pnpm build, git operations, and workflow scripts.
• Filesystem policy controls: allowlist worktree + required caches + minimal runtime dirs.
• Network policy controls: allow default egress profile for git/npm unless explicitly restricted.
• Graceful availability handling: when backend is missing/unavailable, use configurable fail-hard vs fallback-native behavior.
• Auditability: emit run-audit events for backend prepare/run/failure/fallback.

3) Candidate survey

3.1 Baseline matrix

Option	Platforms	Isolation primitives	Startup cost	Fit with worktrees	Operational notes
Native (status quo)	macOS/Linux/Windows	None beyond cwd/user perms	~0	Excellent	No added protection
macOS sandbox-exec	macOS	Seatbelt profile (fs/process/network constraints)	Low	Good	Deprecated tool; still available on many macOS versions
Apple App Sandbox	macOS app model	Entitlement-driven app containerization	N/A for CLI	Poor	Not realistic for arbitrary CLI shell commands
bubblewrap	Linux	mount/user/pid/net/uts/cgroup namespaces; seccomp via composition	Low	Excellent	Strong for per-command filesystem isolation
firejail	Linux	namespaces + seccomp + profiles	Low	Good	SUID/profile complexity trade-offs
Landlock	Linux (kernel feature)	unprivileged LSM policy (fs + TCP port control by ABI)	Very low	Good	Additive policy layer; kernel/ABI variability
nsjail	Linux	namespaces+cgroups+rlimits+seccomp	Low-med	Good	Powerful, but more ops complexity
Docker rootless	Linux/macOS (Desktop VM on macOS)	OCI containers + userns/rootless mode	Med	Good	Daemon/service dependency; image lifecycle overhead
Podman rootless	Linux/macOS (machine)	OCI containers daemonless + userns	Med	Good	Good local dev ergonomics; fewer daemon assumptions
gVisor (runsc)	Linux (container runtime)	userspace kernel boundary for containers	Med-high	Indirect	Strong isolation, but compatibility/perf overhead
Firecracker microVM	Linux + KVM	microVM boundary	High	Indirect	Strongest isolation, highest complexity/latency
WASM/WASI (Wasmtime)	macOS/Linux/Windows	runtime sandbox for wasm modules	Low-med	Limited	Only works for specifically-built wasm workloads, not arbitrary shell

3.2 Evidence notes (sources)

• sandbox-exec is explicitly marked DEPRECATED and points developers to App Sandbox: https://keith.github.io/xcode-man-pages/sandbox-exec.1.html
• Bubblewrap provides unprivileged namespace-based sandbox construction and empty-rootfs model: https://manpages.debian.org/testing/bubblewrap/bwrap.1.en.html
• Firejail uses Linux namespaces and seccomp with profile-driven isolation and optional SUID model: https://firejail.wordpress.com/documentation-2/basic-usage/
• Landlock is an unprivileged, stackable LSM for filesystem and (ABI v4+) TCP access controls, with ABI/version caveats: https://www.kernel.org/doc/html/latest/userspace-api/landlock.html
• nsjail feature set includes namespaces, cgroups, rlimits, seccomp policy support: https://github.com/google/nsjail
• Docker rootless mode runs daemon + containers as non-root in user namespaces; requires uid/gid mappings: https://docs.docker.com/engine/security/rootless/
• Podman is daemonless and supports rootless operation with subuid/subgid and rootless network caveats: https://docs.podman.io/en/latest/markdown/podman.1.html
• gVisor introduces a userspace kernel (runsc) isolation layer and compatibility/perf trade-offs: https://gvisor.dev/docs/
• Firecracker provides KVM-backed microVMs with stronger isolation and lightweight VM design: https://firecracker-microvm.github.io/
• Wasmtime provides secure WASM runtime isolation, but requires wasm-targeted binaries/workloads: https://wasmtime.dev/
• OCI runtime spec context (container backend interoperability): https://github.com/opencontainers/runtime-spec

3.3 Fusion-specific operational considerations

• pnpm/node_modules/caches: sandbox policy must preserve usable package manager cache paths and workspace artifacts (or mount explicit cache volumes).
• .fusion/fusion.db and task files: executor may need read/write to project-local .fusion metadata depending on callsite.
• Git access: must permit git process execution and network egress for fetch/push where configured.
• Network defaults: practical default is allow egress; stricter policy should be explicit opt-in to avoid breaking test/build workflows.
• Port policy: no backend should force use of port 4040 or manage unrelated host processes.

4) Proposed pluggable abstraction

export type SandboxBackendId =
  | "native"
  | "sandbox-exec"
  | "bubblewrap"
  | "docker"
  | "podman"
  | "custom";

export type SandboxFailureMode = "fail-hard" | "fallback-native";

export interface SandboxPolicy {
  allowNetwork: boolean;
  allowedPaths: string[]; // absolute host paths mounted/readable in sandbox
  writablePaths?: string[];
  envAllowlist?: string[];
  cpuLimit?: { maxCores?: number; maxMs?: number };
  memoryLimitMb?: number;
}

export interface SandboxRunOptions {
  cwd: string;
  timeoutMs: number;
  maxBuffer: number;
  env?: NodeJS.ProcessEnv;
}

export interface SandboxPrepareContext {
  taskId: string;
  runId: string;
  worktreePath: string;
  policy: SandboxPolicy;
}

export interface SandboxRunResult {
  stdout: string;
  stderr: string;
  exitCode: number | null;
  signal: NodeJS.Signals | null;
}

export interface SandboxCapabilities {
  filesystemIsolation: "none" | "basic" | "strong";
  networkPolicy: "none" | "coarse" | "fine";
  resourceLimits: "none" | "basic" | "advanced";
  requiresInstall: boolean;
  supportsMac: boolean;
  supportsLinux: boolean;
  supportsWindows: boolean;
}

export interface SandboxBackend {
  id: SandboxBackendId;
  capabilities(): SandboxCapabilities;
  prepare(ctx: SandboxPrepareContext): Promise<void>;
  run(command: string, opts: SandboxRunOptions): Promise<SandboxRunResult>;
  dispose(): Promise<void>;
}

Integration shape:

• Keep existing async command model (promisify(exec) / spawn-based command execution) and route through selected backend wrapper.
• native backend is default for backward compatibility.
• Backend selected from merged settings + optional per-task prompt override.

4.1 Settings proposal (not implemented here)

sandbox:
  backend: native | sandbox-exec | bubblewrap | docker | podman | custom
  policy:
    allowNetwork: true
    allowedPaths: []
    writablePaths: []
  failureMode: fail-hard | fallback-native

Per-task prompt override example:

• **Sandbox:** podman

4.2 Audit proposal

Emit run-audit events:

• sandbox:prepare
• sandbox:run
• sandbox:failure
• sandbox:fallback

Include taskId, runId, backend id, policy hash/summary, and failure reason.

4.3 Action-gate / governance proposal

• Reuse network_api gating for backends requiring outbound fetch/pull/bootstrap.
• Add optional sandbox_provision gate for first-time install/provision actions (container runtime pull/setup, helper binary bootstrap).
• Keep default native path ungated to preserve current behavior for existing projects.

4.4 Self-healing interaction notes

• Sandbox startup failure should be classified similarly to recoverable/unrecoverable session-start failures:
  • backend missing and fallback-native => record fallback + continue.
  • backend missing and fail-hard => task failure/pause with explicit reason.
• Must not break worktree liveness invariants; sandbox prepare should consume existing worktree path validated by current worktree-pool logic.
• Stuck-task recovery remains authoritative; sandbox runs must preserve timeout/abort semantics so watchdog and restart recovery can reason about active subprocesses.

5) Recommendation

Recommended platform defaults (when backend family is enabled in future work):

• macOS: native default, optional sandbox-exec backend for teams accepting deprecated-but-available Seatbelt CLI behavior.
• Linux: bubblewrap recommended first non-native backend (best balance of strong fs isolation + low startup + worktree compatibility).
• Windows: keep native initially; evaluate Windows-native sandboxing/container path in later phase.

Rationale:

• bubblewrap is lightweight, namespace-native, and aligns with per-command short-lived executor workflow.
• Container and microVM backends are valuable as optional high-isolation modes but carry higher latency and ops complexity.
• sandbox-exec can be an interim macOS hardening option, but deprecation risk means it should not be mandatory default.

6) Phased rollout plan

1. Phase A: land SandboxBackend abstraction + native only; no behavior change.
2. Phase B: add macOS sandbox-exec backend behind explicit opt-in.
3. Phase C: add Linux bubblewrap backend with policy templates for Fusion workloads.
4. Phase D: optional container backend (podman first, docker optional) for stronger but heavier isolation.
5. Phase E: docs, telemetry dashboards, and hardening guidance; then consider default changes.

7) Follow-up tasks and open questions

Follow-up implementation tasks (created from this research)

• Add SandboxBackend abstraction and native passthrough backend (depends on FN-4635)
• Implement macOS sandbox-exec backend with policy templates + fallback behavior (depends on abstraction task)
• Implement Linux bubblewrap backend with path/network policy adapter (depends on abstraction task)
• Add sandbox settings schema + prompt override parsing + validation (depends on abstraction task)
• Add sandbox run-audit events and dashboard visibility for sandbox telemetry (depends on abstraction task)
• Add action-gate support for sandbox provisioning/privileged backend setup (depends on abstraction task)

Open questions

• Should sandboxing apply only to executor command lanes, or also merger verification commands/workflow post-merge scripts?
• Is network default allow acceptable, with explicit deny profiles, or should stricter defaults be project-specific?
• Should fallback from sandbox to native be allowed by default, or require explicit failureMode=fallback-native?
• Are teams willing to accept sandbox-exec deprecation risk on macOS, or should macOS remain native until another path exists?
• Should container backends be first-class, or treated as external/custom backend plugins only?
• For Windows support, do we target native isolation primitives first or container runtime parity?