feat(FN-4219): complete Steps 5-7 — docs, changeset, and verification fixes
Fusion-Task-Id: FN-4219 Fusion-Task-Lineage: 8d9a9ba6-6729-4376-b935-549e28a7fa35
This commit is contained in:
74
docs/research/experiment-executor.md
Normal file
74
docs/research/experiment-executor.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Experiment Executor (`@fusion/engine`)
|
||||
|
||||
`ExperimentExecutor` provides engine-side parity with pi-autoresearch's init/run/log loop.
|
||||
|
||||
## Public API
|
||||
|
||||
- `initExperiment(input)`
|
||||
- Creates a new active session and appends a `config` record.
|
||||
- If an active/finalizing session with the same `name` + `projectId` exists, starts a new segment instead.
|
||||
- `runExperiment(input, opts?)`
|
||||
- Runs benchmark command asynchronously, parses `METRIC` lines, and returns a transient run result.
|
||||
- Does not persist run records.
|
||||
- `logExperiment(input)`
|
||||
- Appends a persisted `run` record with selected outcome.
|
||||
- `keep` commits git changes.
|
||||
- `discard` / `checks_failed` / `errored` can revert to a baseline commit.
|
||||
- `getStatus(sessionId)` returns current session status, runs in segment, active handles, and limits.
|
||||
- `cancel(runHandle)` aborts an in-flight benchmark run.
|
||||
|
||||
## Store Composition (FN-4218)
|
||||
|
||||
Executor uses `ExperimentSessionStore` for:
|
||||
- session creation/reuse (`createSession`, `startNewSegment`)
|
||||
- record append (`appendRecord`)
|
||||
- best/kept pointers (`setBestRun`, `recordKept`)
|
||||
- run payload commit patching (`updateRecordPayload` additive method)
|
||||
|
||||
## METRIC Grammar
|
||||
|
||||
Parser accepts:
|
||||
|
||||
```regex
|
||||
^METRIC\s+([A-Za-z_][A-Za-z0-9_.-]*)\s*=\s*(-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)\s*(?:\(([^)]+)\))?\s*$
|
||||
```
|
||||
|
||||
- first valid metric = primary
|
||||
- later metrics = secondary
|
||||
- dedup by metric name (last-write-wins)
|
||||
- denylist: `__proto__`, `constructor`, `prototype`
|
||||
- non-finite values are ignored with warnings
|
||||
|
||||
## Benchmark Execution Contract
|
||||
|
||||
`runBenchmark()` uses non-blocking child process execution (`spawn`, async path):
|
||||
- default timeout: 10 minutes
|
||||
- default max buffer: 10MB
|
||||
- supports `AbortSignal`
|
||||
- throttled progress callback (<= every 500ms)
|
||||
- when stdout exceeds buffer, full output is written to temp file and returned stdout is truncated to last 64KB
|
||||
|
||||
## Keep/Revert Git Policy
|
||||
|
||||
Preserved artifacts on revert:
|
||||
- `autoresearch.jsonl`
|
||||
- `autoresearch.md`
|
||||
- `autoresearch.ideas.md`
|
||||
- `autoresearch.checks.sh`
|
||||
- `autoresearch.config.json`
|
||||
- `autoresearch.hooks/`
|
||||
|
||||
Behavior:
|
||||
- `keep`: stage all + commit (`experiment(<session>): keep <run>` message default)
|
||||
- discard/check failures/errors: reset hard to baseline while preserving autoresearch artifacts via stash roundtrip
|
||||
|
||||
## Error Taxonomy
|
||||
|
||||
- `ExperimentMaxIterationsError` — run attempted after reaching max iterations.
|
||||
- `ExperimentGitNotConfiguredError` — keep/revert path requested without configured `GitOps`.
|
||||
- `ExperimentRevertConflictError` — stash-pop conflict while restoring preserved artifacts after revert.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- FN-4221: CLI/dashboard/pi-extension wiring.
|
||||
- FN-4222: finalize workflow (branch/finalization orchestration).
|
||||
Reference in New Issue
Block a user