feat(FN-4355): add MVP spec for experiment session feature
Added the experiment-session MVP specification document (`docs/research/experiment-session-mvp-spec.md`) as the final delivery for FN-4355, completing the spec work for the experiment session feature. Fusion-Task-Id: FN-4355
This commit is contained in:
Fusion
154
docs/research/experiment-session-mvp-spec.md
Normal file
154
docs/research/experiment-session-mvp-spec.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# Problem
|
||||
|
||||
The current Experiment Session initiative is split across four large tasks (FN-4218, FN-4219, FN-4220, FN-4222) that together mix foundational schema work, executor lifecycle, hardening, and finalize workflows. Shipping all four at once increases coordination risk and delays learning.
|
||||
|
||||
A smaller MVP gives stakeholders a usable try-measure-keep-revert loop quickly: persistent session storage, benchmark execution, metric parsing, and keep/discard git behavior. That enables real operator feedback before adding hardening and branch-finalize surfaces.
|
||||
|
||||
# Capability Overlap Matrix
|
||||
|
||||
| Capability | FN-4218 | FN-4219 | FN-4220 | FN-4222 | Overlap class |
|
||||
|---|---|---|---|---|---|
|
||||
| Session domain types/statuses/records | ✅ | ✅ (consumes) | ✅ (consumes) | ✅ (consumes) | Core overlap (must-have) |
|
||||
| Persistent store + schema (`experiment_sessions`, `experiment_session_records`) | ✅ | ✅ (uses) | ✅ (uses) | ✅ (uses) | Core overlap (must-have) |
|
||||
| `initExperiment` config/session bootstrap | ⚪ | ✅ | ✅ (extends lifecycle) | ⚪ | Overlap (must-have) |
|
||||
| `runExperiment` benchmark execution + metric parse | ⚪ | ✅ | ✅ (adds checks/hooks/cap) | ⚪ | Overlap (must-have) |
|
||||
| `logExperiment` keep/discard outcome + git policy | ⚪ | ✅ | ✅ (adds checks_failed/revert paths) | ⚪ | Overlap (must-have) |
|
||||
| Post-benchmark checks (`autoresearch.checks.sh`) | ⚪ | ⚪ | ✅ | ⚪ | Single-task (deferrable) |
|
||||
| Before/after hooks + steer messages | ⚪ | ⚪ | ✅ | ⚪ | Single-task (deferrable) |
|
||||
| Resume from artifacts/compaction | ⚪ | ⚪ | ✅ | ⚪ | Single-task (deferrable) |
|
||||
| `maxIterations` hard cap to `finalizing` | ⚪ | ⚪ | ✅ | ⚪ | Single-task (deferrable) |
|
||||
| Finalize plan/group/split branches | ⚪ | ⚪ | ⚪ | ✅ | Single-task (deferrable) |
|
||||
| CLI command surface (`fn experiment ...`) | ⚪ | ⚪ | ⚪ | ✅ | Single-task (deferrable) |
|
||||
| Pi-extension experiment tools | ⚪ | ⚪ | ⚪ | ✅ | Single-task (deferrable) |
|
||||
| Dashboard experiment finalize/session APIs/UI | ⚪ | ⚪ | ⚪ | ✅ | Single-task (deferrable) |
|
||||
|
||||
# MVP Scope
|
||||
|
||||
## In scope (ship in MVP)
|
||||
|
||||
1. **FN-4218 minimal domain surface**: types, store APIs, and SQLite schema for `experiment_sessions` and `experiment_session_records` with tests.
|
||||
2. **FN-4219 minimal executor slice**:
|
||||
- `initExperiment`
|
||||
- `runExperiment`
|
||||
- `logExperiment({ outcome: "keep" | "discard" })`
|
||||
- METRIC parser (`METRIC name=value`)
|
||||
- async benchmark runner (non-blocking process execution)
|
||||
- keep/revert git policy for run outcomes
|
||||
|
||||
No CLI, pi-extension, or dashboard experiment UI/API shipping in this increment.
|
||||
|
||||
## MVP user flow (end-to-end)
|
||||
|
||||
1. Operator initializes an experiment session (name + metric definition + working directory).
|
||||
2. Executor persists session row in `experiment_sessions` and appends a `config` record in `experiment_session_records`.
|
||||
3. Operator runs a benchmark command through `runExperiment`.
|
||||
4. Executor runs the command asynchronously, captures stdout/stderr, and parses first valid `METRIC` line as primary metric.
|
||||
5. Operator reviews run output and chooses `keep` or `discard` via `logExperiment`.
|
||||
6. Executor appends a `run` record to `experiment_session_records` with metric/result payload.
|
||||
7. If `keep`: executor commits changes and records the kept run linkage (`bestRunId` / kept run IDs). If `discard`: executor reverts to baseline commit while preserving `autoresearch.*` artifacts.
|
||||
8. Session remains `active` for additional manual iterations.
|
||||
|
||||
## Minimal API/domain surface (verbatim contract names)
|
||||
|
||||
### `@fusion/core`
|
||||
- `ExperimentSessionStore`
|
||||
- `ExperimentSession`
|
||||
- `ExperimentSessionRecord`
|
||||
- `ExperimentSessionCreateInput`
|
||||
- `ExperimentSessionUpdateInput`
|
||||
- `ExperimentSessionRecordAppendInput`
|
||||
- `ExperimentSessionListOptions`
|
||||
- `ExperimentMetricDefinition`
|
||||
- `ExperimentRunRecordPayload`
|
||||
- `ExperimentConfigRecordPayload`
|
||||
- `EXPERIMENT_SESSION_STATUSES`
|
||||
- `EXPERIMENT_RECORD_TYPES`
|
||||
- `EXPERIMENT_RUN_OUTCOMES`
|
||||
|
||||
### `@fusion/engine`
|
||||
- `ExperimentExecutor`
|
||||
- `initExperiment`
|
||||
- `runExperiment`
|
||||
- `logExperiment`
|
||||
- `parseMetricLines`
|
||||
- `runBenchmark`
|
||||
- `commitKept`
|
||||
- `revertDiscarded`
|
||||
|
||||
## MVP “shipped” definition
|
||||
|
||||
MVP is shipped when a Fusion operator can complete one persisted experiment iteration (init → run → parse metric → keep/discard log → commit/revert) with no manual DB edits and with passing lint/test/build gates.
|
||||
|
||||
# Non-Goals
|
||||
|
||||
1. **FN-4220 checks/hook/resume/maxIterations hardening** — defer to a post-MVP hardening slice because these behaviors add failure-mode complexity and are not required to validate the base operator loop.
|
||||
2. **FN-4222 finalize-into-branches workflow** — defer because branch grouping/cherry-pick orchestration is a second-phase workflow after teams first validate that kept/discarded runs are being produced consistently.
|
||||
3. **Dashboard experiment session UI (views/modals/cards)** — defer to FN-4221/FN-4222 follow-up surface work; MVP can be validated from engine/store level without adding new frontend maintenance burden.
|
||||
4. **Pi-extension `fn_experiment_*` tools** — defer to FN-4221 so API/tool ergonomics are designed after core runtime behavior stabilizes.
|
||||
5. **CLI `fn experiment ...` command surface** — defer to FN-4222 after finalize semantics settle, to avoid reworking command contracts twice.
|
||||
6. **Steer-message and advanced observability enrichment** — defer to FN-4220 hardening where hook execution is introduced.
|
||||
|
||||
# Metrics
|
||||
|
||||
1. **Adoption metric (schema-verifiable):** At least **15 `experiment_sessions` rows** created within **8 weeks** of MVP release.
|
||||
- **Source:** `experiment_sessions.createdAt` (FN-4218 schema), counted from opt-in telemetry or per-node periodic export.
|
||||
- **Threshold rationale:** conservative first target because MVP intentionally excludes CLI/UI surfaces; we expect lower initial volume than existing multi-surface research flows documented in the audit (`docs/research/pi-autoresearch-audit-2026-05.md`, “Dashboard / CLI / extension are out of sync”).
|
||||
|
||||
2. **Reliability metric (schema-verifiable):** At least **99% of appended `run` records** reach terminal outcomes in `{keep, discard, errored}` (no stuck `pending`) within **10 minutes** of record creation.
|
||||
- **Source:** `experiment_session_records` (`type='run'`, payload status, `createdAt`), plus executor logs from `createLogger("experiment-executor")` for timeout/error diagnostics.
|
||||
- **Threshold rationale:** AGENTS.md Engine Process Rules require async execution paths (no blocking `execSync` for user commands), so terminal completion rates should stay high under normal engine load.
|
||||
|
||||
3. **Operational reliability metric:** At least **99% of `runExperiment` invocations** return a structured result object (including parse warnings when needed) without unhandled rejection.
|
||||
- **Source:** `experiment-executor` logger error/info counts and run invocation counters.
|
||||
- **Threshold rationale:** minimal MVP loop has limited branches; high structured-return rate is expected before hardening features are introduced.
|
||||
|
||||
# Dependencies
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[FN-4218 Domain model + schema] -->|Hard dependency| B[FN-4219 Executor MVP lifecycle]
|
||||
B -->|Hard dependency| C[Post-MVP FN-4220 Hardening]
|
||||
B -->|Hard dependency| D[Post-MVP FN-4222 Finalize workflow]
|
||||
C -. Soft/informational only .-> D
|
||||
D -. Soft/informational only .-> C
|
||||
```
|
||||
|
||||
- **Hard dependency:** FN-4218 → FN-4219 (executor imports core types/store contracts).
|
||||
- **Hard dependency:** FN-4220 depends on FN-4218 + FN-4219.
|
||||
- **Hard dependency:** FN-4222 depends on FN-4218 + FN-4219.
|
||||
- **Independence note:** FN-4220 and FN-4222 are independent of each other and can run in parallel after MVP lands.
|
||||
|
||||
# Follow-ups
|
||||
|
||||
1. **Slice A — Core experiment schema + store baseline (FN-4218, Size M)**
|
||||
Scope: land only the minimum session/record types, store CRUD, and SQLite migration for `experiment_sessions` + `experiment_session_records`, plus focused core tests. This intentionally excludes extra convenience APIs not required by `init/run/log`.
|
||||
Touched packages: `@fusion/core`
|
||||
Parent: FN-4218
|
||||
|
||||
2. **Slice B — Executor lifecycle skeleton (FN-4219, Size M)**
|
||||
Scope: implement `ExperimentExecutor` with `initExperiment`, `runExperiment`, and `logExperiment` for keep/discard only, including session status guards and run-record append behavior. Exclude checks/hooks/resume/max-iteration.
|
||||
Touched packages: `@fusion/engine` (and `@fusion/core` only if a tiny additive store contract patch is unavoidable)
|
||||
Parent: FN-4219
|
||||
|
||||
3. **Slice C — Metric parser + benchmark runner hard contract (FN-4219, Size S)**
|
||||
Scope: isolate parser grammar (`METRIC ...`) and async benchmark runner semantics (timeout, truncation, warning behavior) with dedicated unit tests; wire into Slice B. This is materially smaller than the full FN-4219 parent.
|
||||
Touched packages: `@fusion/engine`
|
||||
Parent: FN-4219
|
||||
|
||||
4. **Slice D — Keep/discard git policy module (FN-4219, Size S)**
|
||||
Scope: implement and test `commitKept`/`revertDiscarded` and `GitOps` wrapper behavior for baseline MVP outcomes only. Defer conflict-specialization and finalize branch splitting.
|
||||
Touched packages: `@fusion/engine`
|
||||
Parent: FN-4219
|
||||
|
||||
5. **Slice E — Post-MVP hardening pack (FN-4220, Size M)**
|
||||
Scope: add checks runner, before/after hooks, compaction-resume helpers, and `maxIterations` enforcement after the MVP loop is validated in production-like usage.
|
||||
Touched packages: `@fusion/engine`
|
||||
Parent: FN-4220
|
||||
|
||||
# Open Questions
|
||||
|
||||
1. Should MVP include a minimal API endpoint for operations teams, or remain engine-internal only until FN-4221/FN-4222 surfaces are ready?
|
||||
2. Should `logExperiment` persist explicit baseline commit provenance per run in MVP, or leave provenance solely to git history + kept run IDs?
|
||||
3. Does product want `pending` run statuses visible to users in MVP, or hidden behind internal executor/state APIs until hardening lands?
|
||||
4. If adoption is below the 15-session/8-week target, should next investment go first to CLI ergonomics or dashboard UI?
|
||||
5. Audit reference file `docs/research/pi-autoresearch-audit-2026-05.md` is present; no missing-audit fallback needed.
|
||||
Reference in New Issue
Block a user