diff --git a/.gitignore b/.gitignore index 8b5e0a0c2f..fc21833572 100644 --- a/.gitignore +++ b/.gitignore @@ -89,6 +89,6 @@ packages/dashboard/android/ # Plugin hot-reload scratch artifacts **/.index.reload-*.ts -# Dolt database files (added by bd init) +# Dolt database files .dolt/ *.db diff --git a/AGENTS.md b/AGENTS.md index 1ad9ecc886..27bf33809b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -224,116 +224,3 @@ Keep this AGENTS inventory in sync with App lazy imports and `packages/dashboard - `PluginManager` - `PiExtensionsManager` - `AgentDetailView` - - -## Issue Tracking with bd (beads) - -**IMPORTANT**: This project uses **bd (beads)** for ALL issue tracking. Do NOT use markdown TODOs, task lists, or other tracking methods. - -### Why bd? - -- Dependency-aware: Track blockers and relationships between issues -- Git-friendly: Dolt-powered version control with native sync -- Agent-optimized: JSON output, ready work detection, discovered-from links -- Prevents duplicate tracking systems and confusion - -### Quick Start - -**Check for ready work:** - -```bash -bd ready --json -``` - -**Create new issues:** - -```bash -bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json -bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --json -``` - -**Claim and update:** - -```bash -bd update --claim --json -bd update bd-42 --priority 1 --json -``` - -**Complete work:** - -```bash -bd close bd-42 --reason "Completed" --json -``` - -### Issue Types - -- `bug` - Something broken -- `feature` - New functionality -- `task` - Work item (tests, docs, refactoring) -- `epic` - Large feature with subtasks -- `chore` - Maintenance (dependencies, tooling) - -### Priorities - -- `0` - Critical (security, data loss, broken builds) -- `1` - High (major features, important bugs) -- `2` - Medium (default, nice-to-have) -- `3` - Low (polish, optimization) -- `4` - Backlog (future ideas) - -### Workflow for AI Agents - -1. **Check ready work**: `bd ready` shows unblocked issues -2. **Claim your task atomically**: `bd update --claim` -3. **Work on it**: Implement, test, document -4. **Discover new work?** Create linked issue: - - `bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:` -5. **Complete**: `bd close --reason "Done"` - -### Auto-Sync - -bd automatically syncs via Dolt: - -- Each write auto-commits to Dolt history -- Use `bd dolt push`/`bd dolt pull` for remote sync -- No manual export/import needed! - -### Important Rules - -- ✅ Use bd for ALL task tracking -- ✅ Always use `--json` flag for programmatic use -- ✅ Link discovered work with `discovered-from` dependencies -- ✅ Check `bd ready` before asking "what should I work on?" -- ❌ Do NOT create markdown TODO lists -- ❌ Do NOT use external issue trackers -- ❌ Do NOT duplicate tracking systems - -For more details, see README.md and docs/QUICKSTART.md. - -## Landing the Plane (Session Completion) - -**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds. - -**MANDATORY WORKFLOW:** - -1. **File issues for remaining work** - Create issues for anything that needs follow-up -2. **Run quality gates** (if code changed) - Tests, linters, builds -3. **Update issue status** - Close finished work, update in-progress items -4. **PUSH TO REMOTE** - This is MANDATORY: - ```bash - git pull --rebase - bd sync - git push - git status # MUST show "up to date with origin" - ``` -5. **Clean up** - Clear stashes, prune remote branches -6. **Verify** - All changes committed AND pushed -7. **Hand off** - Provide context for next session - -**CRITICAL RULES:** -- Work is NOT complete until `git push` succeeds -- NEVER stop before pushing - that leaves work stranded locally -- NEVER say "ready to push when you are" - YOU must push -- If push fails, resolve and retry until it succeeds - - diff --git a/docs/README.md b/docs/README.md index ec80d6291a..7896ab59ad 100644 --- a/docs/README.md +++ b/docs/README.md @@ -55,7 +55,6 @@ For a full walkthrough (installation, onboarding, first task, and daily workflow | [Storage](./storage.md) | Storage architecture, migration, archive system, and SQLite schema | | [DAG Architecture Deliverables](./dag/) | Milestone A DAG architecture documents plus Milestone B prototype scaffold docs (schema migration plan, DagCoordinator design, implementation checklist) | | [Dev Server Module Audit](./dev-server-modules.md) | Analysis of parallel dashboard dev-server module families, production wiring, and consolidation guidance | -| [Beads and Dolt Evaluation for Fusion Node Sync](./beads-dolt-sync-evaluation.md) | Evaluation of Beads and Dolt for node sync, with a recommendation for Fusion-native sync design | | [Shared Mesh Replication Protocol](./shared-mesh-protocol.md) | Canonical multi-leader replication/write-coordination contract (versioning, quorum, leases/fencing, queue/replay, reconciliation, and degraded-read semantics) | | [Multi-Project Sequencing and Dependency Analysis](./multi-project-sequencing.md) | Sequencing guidance for FN-3448/FN-3449/FN-3503/FN-3182, including identity boundaries and recommended board dependency edges | | [Contributing](./contributing.md) | Local development setup, testing, release flow, and contributor conventions | diff --git a/docs/beads-dolt-sync-evaluation.md b/docs/beads-dolt-sync-evaluation.md deleted file mode 100644 index b2f8c00967..0000000000 --- a/docs/beads-dolt-sync-evaluation.md +++ /dev/null @@ -1,501 +0,0 @@ -# Beads and Dolt Evaluation for Fusion Node Sync - -[← Docs index](./README.md) - -## Summary - -Recommendation: **do not switch Fusion wholesale to either Beads or Dolt for node sync right now**. - -Use them as design references or optional experiments, but keep Fusion’s current SQLite + filesystem hybrid model and add an explicit Fusion-native sync layer. - -| Option | Recommendation | -|---|---| -| Beads | Useful inspiration for local-first issue/task sync, but too domain-specific to become Fusion’s persistence or sync substrate. | -| Dolt | Technically interesting for versioned relational data and SQL merge semantics, but too heavy and operationally different from Fusion’s embedded SQLite model. Consider only as an experimental backend or audit/export target. | -| Best path | Keep SQLite. Add a Fusion-native sync protocol based on append-only change events, per-record revisions, deterministic conflict policies, and blob transfer for `.fusion/tasks/*`. | - -## Fusion’s Current Sync Requirements - -Fusion is more than a task board. Current persistence includes: - -- Project DB: `.fusion/fusion.db` -- Central DB: `~/.fusion/fusion-central.db` -- Filesystem blobs: `.fusion/tasks/{ID}/PROMPT.md`, logs, attachments -- Project tables for tasks, agents, activity logs, missions, roadmaps, workflow steps, chat, insights, audit events, and more -- Central multi-node metadata including `nodes`, `peerNodes`, `settingsSyncState`, project routing, health, and global concurrency - -Node sync needs to handle: - -1. Task metadata -2. Task lifecycle transitions -3. Agent state and heartbeats -4. Settings sync -5. Mission, roadmap, and todo data -6. Attachments and task files -7. Conflict detection -8. Offline edits -9. Partial peer availability -10. Security and authentication per node - -The sync layer needs to be more general than an issue tracker sync model. - -## Beads Evaluation - -Beads is attractive because it appears philosophically aligned with Fusion: - -- Local-first -- CLI-friendly -- Task/issue oriented -- Git-friendly or file/db-backed sync model -- Human-readable workflows -- Good fit for small project issue tracking - -### Potential Uses - -Beads could be useful as inspiration for: - -- Task IDs -- Dependency graph semantics -- Syncing issue-like records -- Minimal local-first UX -- Conflict-tolerant issue updates -- Import/export interoperability - -### Problems as Fusion’s Backend - -#### Domain mismatch - -Fusion tasks are only one part of the data model. Fusion also has: - -- Agents -- Agent ratings -- Heartbeat runs -- AI sessions/messages -- Workflow steps -- Missions, milestones, slices, and features -- Roadmaps -- Settings -- Node registry -- Run audit events -- Attachments and logs - -Beads is likely optimized around issues, not full distributed orchestration state. - -#### Schema constraints - -If Fusion uses Beads as the substrate, Fusion either: - -- Adopts Beads’ issue model and loses domain expressiveness, or -- Stores Fusion-specific JSON payloads inside Beads, turning Beads into an awkward blob store. - -Neither is ideal. - -#### Sync granularity mismatch - -Fusion needs record-level and event-level sync with explicit lifecycle semantics. - -Example: - -- Node A moves `FN-123` from `todo` to `in-progress` -- Node B edits title/description -- Node C assigns `nodeId` -- An agent heartbeat writes progress -- A reviewer moves the task to `in-review` - -Fusion needs deterministic merge rules per field and per event type. Beads is unlikely to provide that across Fusion’s full schema. - -#### Runtime state should not sync like tasks - -Some Fusion data is operational and ephemeral: - -- Agent heartbeat timestamps -- In-progress run state -- Local worktree paths -- Scheduler locks -- Checkout leases - -A general issue tracker sync model may accidentally replicate data that should remain node-local. - -### Beads Verdict - -Do not use Beads as Fusion’s storage backend. - -Potential uses: - -- Study its local-first task model. -- Build an importer/exporter if useful. -- Reuse similar concepts for task dependency syncing. -- Do not couple Fusion core storage or node sync to it. - -## Dolt Evaluation - -Dolt is effectively “Git for SQL databases”: relational tables with branches, diffs, commits, remotes, merges, conflicts, and SQL access. - -### What Dolt Is Good At - -Dolt provides: - -- Versioned relational data -- SQL access -- Branch/merge workflow -- Diff/commit history -- Conflict detection -- Remote push/pull semantics -- MySQL-compatible protocol -- Data provenance - -This sounds compelling because Fusion already stores structured metadata in SQLite. - -### Why Dolt Is Tempting - -Fusion needs distributed relational sync. Dolt provides many adjacent primitives: - -- Nodes could have branches. -- Sync could be pull/merge/push. -- Conflicts could be represented explicitly. -- Settings/task diffs could be inspected. -- History could be queryable. -- Multi-node sync could use Dolt remotes. - -For structured metadata, Dolt is more relevant than Beads. - -### Major Tradeoffs - -#### Dolt is not an embedded SQLite replacement - -Fusion currently uses SQLite through `node:sqlite`. - -This is simple: - -- No server -- No external daemon -- Small operational footprint -- Works inside a published npm CLI -- Easy local project DB -- WAL mode concurrency -- Files live in `.fusion/fusion.db` - -Dolt is operationally different: - -- Usually accessed as a Dolt database/server or CLI-managed repo -- MySQL-compatible, not SQLite-compatible -- Requires different drivers and query behavior -- Adds a heavier binary dependency -- Is harder to bundle into `@runfusion/fusion` - -For a globally installed npm CLI, this matters. - -#### Migration cost is high - -Fusion’s persistence layer is deeply SQLite-oriented: - -- `packages/core/src/db.ts` -- `TaskStore` -- `CentralCore` -- migrations -- tests using real SQLite -- project-local DB assumptions -- `.fusion/fusion.db` file layout - -Switching to Dolt would likely require a storage abstraction layer or a major rewrite. - -#### SQL merge is not domain merge - -Dolt can tell you there is a data conflict. It does not know what Fusion should do. - -Example conflict: - -```text -task.status: -Node A: in-progress -Node B: done -``` - -Dolt can surface a conflict, but Fusion still needs to decide: - -- Is `done` allowed if `in-progress` happened elsewhere? -- Was review skipped? -- Which transition wins? -- Should this create a merge-resolution task? -- Should the losing transition be preserved in activity log? - -Fusion has domain-level lifecycle rules. SQL merge does not replace those rules. - -#### Some tables should not be globally merged - -Fusion has mixed data classes: - -| Data type | Sync behavior | -|---|---| -| Tasks | Sync | -| Task documents | Sync | -| Settings | Selectively sync | -| Missions/roadmaps | Sync | -| Activity log | Append-only | -| Run audit | Append-only or local-origin | -| Agent heartbeats | Usually local-only or summarized | -| Checkout leases | Local-only or TTL-based | -| Local worktree paths | Local-only | -| Auth secrets | Special encrypted sync only | - -A generic database merge risks syncing the wrong things unless carefully partitioned. - -#### Filesystem blobs remain unsolved - -Fusion stores large task artifacts in `.fusion/tasks/{ID}/`. - -Dolt can store data in tables, but putting logs, prompts, attachments, and large blobs into Dolt would be undesirable. Fusion would still need a blob sync protocol. - -#### Operational burden - -Dolt introduces product questions: - -- How is Dolt installed? -- Is it bundled with the npm package? -- What about Windows/macOS/Linux binaries? -- How are DB upgrades handled? -- Does the dashboard start a Dolt SQL server? -- What port does it use? -- How does this work with `fn serve`? -- How does it interact with user Git repos? -- How are backups handled? -- How are conflicts exposed in the dashboard? - -That is a large product surface. - -### Dolt Verdict - -Dolt is technically promising, but should not become Fusion’s primary storage engine now. - -Better uses: - -1. **Experimental sync backend** - - Add a prototype adapter for selected tables. - - Try syncing `tasks`, `activityLog`, and `settingsSyncState`. - - Measure complexity. -2. **External export format** - - Export Fusion state into Dolt for audit/history/diff. - - Do not make runtime depend on it. -3. **Admin/enterprise mode** - - Potentially useful later for teams wanting SQL history and data provenance. - -## Recommended Fusion-Native Sync Design - -Keep the current SQLite architecture and add a dedicated sync layer. - -### Core Idea - -Use **append-only change events** plus **materialized SQLite state**. - -Fusion already has adjacent concepts: - -- `activityLog` -- `runAuditEvents` -- node registry -- settings sync state -- task lifecycle transitions -- central DB node metadata - -Build on those instead of replacing storage. - -### Stable Node Identity - -Each node should have: - -```ts -nodeId -publicKey? -apiKey / auth credentials -lastSeen -syncCursorByPeer -``` - -This is already partially represented by `nodes` and `peerNodes`. - -### Change Log Table - -Add a project-level sync log: - -```sql -CREATE TABLE sync_events ( - id TEXT PRIMARY KEY, - originNodeId TEXT NOT NULL, - seq INTEGER NOT NULL, - entityType TEXT NOT NULL, - entityId TEXT NOT NULL, - operation TEXT NOT NULL, - payloadJson TEXT NOT NULL, - baseRevision TEXT, - resultingRevision TEXT NOT NULL, - createdAt TEXT NOT NULL -); -``` - -This becomes the canonical stream nodes exchange. - -### Per-Entity Revision Metadata - -For synced tables: - -```sql -CREATE TABLE entity_revisions ( - entityType TEXT NOT NULL, - entityId TEXT NOT NULL, - revision TEXT NOT NULL, - updatedAt TEXT NOT NULL, - updatedByNodeId TEXT NOT NULL, - PRIMARY KEY (entityType, entityId) -); -``` - -Revision options: - -- Lamport timestamp -- Hybrid logical clock -- Compact vector clock -- Content hash plus origin sequence - -### Conflict Policies by Entity and Field - -Do not rely on generic last-write-wins everywhere. - -| Entity | Conflict policy | -|---|---| -| Task title/description | Last-write-wins or field-level merge | -| Task status | Lifecycle-aware transition merge | -| Task labels | Set union | -| Task dependencies | Set union with cycle detection | -| Activity log | Append-only | -| Run audit | Append-only | -| Checkout lease | Local-only or TTL conflict | -| Agent heartbeat | Node-local by default | -| Settings | Scoped, field-level, explicit push/pull | -| Auth | Encrypted explicit sync only | -| Attachments | Content-addressed blob transfer | - -### Blob Sync - -For files under `.fusion/tasks/{ID}/`, use content-addressed metadata: - -```sql -CREATE TABLE sync_blobs ( - digest TEXT PRIMARY KEY, - taskId TEXT, - relativePath TEXT NOT NULL, - size INTEGER NOT NULL, - mimeType TEXT, - createdAt TEXT NOT NULL -); -``` - -Then transfer blobs separately: - -```text -GET /api/sync/blobs/:digest -PUT /api/sync/blobs/:digest -``` - -Avoid stuffing large logs and attachments into relational sync. - -### Peer Protocol - -Potential endpoints: - -```text -GET /api/sync/summary -GET /api/sync/events?since= -POST /api/sync/events -GET /api/sync/blobs/:digest -POST /api/sync/blobs -POST /api/sync/resolve-conflict -``` - -### Conflict Visibility - -Conflicts should become first-class Fusion objects: - -- Shown in the dashboard -- Resolvable by user or agent -- Optionally converted into `FN-*` tasks -- Include both versions and proposed resolution - -## Dolt vs Fusion-Native Sync - -### Dolt Advantages - -- Existing versioned SQL system -- Built-in diff/merge/push/pull -- Strong data history -- Good for auditable relational datasets -- Mature conceptual model - -### Dolt Disadvantages for Fusion - -- Heavy runtime dependency -- Not SQLite-compatible -- Requires major persistence refactor -- Does not handle Fusion domain conflicts automatically -- Does not solve file/blob sync cleanly -- Harder npm distribution story -- Risky for local CLI UX - -### Fusion-Native Advantages - -- Keeps current SQLite -- Minimal disruption -- Tailored conflict semantics -- Can sync only the right tables -- Easier dashboard integration -- Easier to bundle and test -- Works with current `.fusion/` layout -- Can evolve incrementally - -### Fusion-Native Disadvantages - -- More custom code -- Conflict logic must be carefully designed -- Sync cursor correctness is hard -- Requires robust test coverage -- More engineering effort than delegating to an existing system - -However, Fusion’s domain is specialized enough that custom sync semantics are likely unavoidable either way. - -## Decision - -Do not switch to Beads. - -Do not replace SQLite with Dolt yet. - -Build native sync first. - -Recommended phased plan: - -1. **Classify all tables** - - synced - - append-only - - local-only - - encrypted/explicit - - blob-backed -2. **Add sync event log** - - append-only - - origin node - - sequence/cursor - - entity revision -3. **Implement task/settings sync first** - - tasks - - task documents - - settings - - activity log -4. **Add blob sync** - - content-addressed task files -5. **Add dashboard conflict UI** - - task conflicts - - settings conflicts - - agent-assisted resolution -6. **Prototype Dolt separately** - - feature flag or branch only - - measure install size, query compatibility, performance, and conflict ergonomics - -Final recommendation: - -> Keep SQLite as Fusion’s embedded source of truth. Build a Fusion-native sync layer. Treat Dolt as an optional future backend or audit/export target. Treat Beads as product inspiration, not infrastructure. diff --git a/docs/contributing.md b/docs/contributing.md index 3263e2b694..f97cec31a2 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -220,63 +220,6 @@ Use task-ID-scoped conventional commits: - `test(FN-XXX): ...` - `docs(FN-XXX): ...` (for documentation-only changes) -## Issue Tracking with bd (Beads) - -This repository uses `bd` (Beads) for **all** local issue tracking: do not create markdown TODO lists, task lists, or external tracker records for repository work; see [AGENTS.md](../AGENTS.md) for the authoritative agent-specific policy. - -Use this end-to-end workflow: - -1. **Check ready, unblocked work.** Expected outcome: you see the issue IDs that are ready to claim, with dependency-blocked work filtered out. - - ```bash - bd ready --json - ``` - -2. **Claim a task atomically before changing files.** Expected outcome: the selected issue is assigned/claimed so another contributor or agent does not start the same work. - - ```bash - bd update --claim --json - ``` - -3. **Create discovered follow-up work with a dependency link.** Expected outcome: separate work is tracked as its own issue and linked back to the issue where it was discovered. Use the appropriate type and priority rather than leaving an inline TODO. - - ```bash - bd create "Title" --description="What this issue is about" -t bug|feature|task -p 0-4 --deps discovered-from: --json - ``` - -4. **Close completed work with a reason.** Expected outcome: the issue history records why the work is done. - - ```bash - bd close --reason "Completed" --json - ``` - -5. **Sync and push before ending the session.** Expected outcome: both Beads/Dolt issue state and Git commits are available remotely, and `git status` reports that the branch is up to date with origin. - - ```bash - git pull --rebase - bd sync - git push - git status # MUST show "up to date with origin" - ``` - -Issue types match the Beads policy in `AGENTS.md`: - -- `bug` — something broken -- `feature` — new functionality -- `task` — work item such as tests, docs, or refactoring -- `epic` — large feature with subtasks -- `chore` — maintenance - -Priorities use a `0`–`4` scale: - -- `0` — critical: security, data loss, or broken builds -- `1` — high: major features or important bugs -- `2` — medium: default, nice-to-have work -- `3` — low: polish or optimization -- `4` — backlog: future ideas - -This workflow is only for this repository's local issue tracking. It is separate from the product evaluation in [Beads and Dolt Evaluation for Fusion Node Sync](./beads-dolt-sync-evaluation.md), which assesses Beads/Dolt as a possible Fusion node-sync substrate and recommends **against** switching Fusion's storage/sync backend to Beads or Dolt. - ## Project Memory When enabled, Fusion uses OpenClaw-style memory files: diff --git a/packages/cli/src/__tests__/docs-readme-index.test.ts b/packages/cli/src/__tests__/docs-readme-index.test.ts index a1ccdba7c3..e596db7a29 100644 --- a/packages/cli/src/__tests__/docs-readme-index.test.ts +++ b/packages/cli/src/__tests__/docs-readme-index.test.ts @@ -6,7 +6,6 @@ const workspaceRoot = resolve(import.meta.dirname, "../../../.."); const docsReadmePath = resolve(workspaceRoot, "docs", "README.md"); const requiredDocs = [ - "docs/beads-dolt-sync-evaluation.md", "docs/dev-server-modules.md", "docs/research/pi-autoresearch-analysis.md", "docs/research/research-hardening-preflight.md",