Commit Graph

14 Commits

Author SHA1 Message Date
ischindl
8fcf4bdbaa feat: Stash memory backend — session capture, per-chat backfill, opt-in vector search (#3494)
## Summary

Adds the **Stash memory backend** (`memory.backendType=stash`) that
connects Fusion's agent memory to the
[Stash](https://github.com/Fergana-Labs/stash) product — *knowledge
bases for the agent era* ([product site:
joinstash.ai](https://joinstash.ai)). Fusion becomes a first-class Stash
client: complete chat sessions and finished tasks are captured into
Stash, memory is recalled during chat, and Stash sessions are kept in
sync with the dashboard (including deletes and archival).

**Product:** <https://github.com/Fergana-Labs/stash> ·
[joinstash.ai](https://joinstash.ai)

## What's included

### 1. Stash memory backend (RUFU-068 / RUFU-121)
- New `StashMemoryBackend` (`memory.backendType=stash`) with `stashUrl`
/ `stashApiKey` settings (global secrets-store `stash-api-key` +
per-project override).
- **Complete-chat-session capture** keyed by ChatSession id.
- Sessions are classified into **per-project folders** (get-or-create,
`external_key fusion-<projectId>`, 1h per-process cache) and
**soft-deleted with their chat** via `DELETE /api/chat/sessions/:id`.
- Per-conversation **memory-focus** read-time scoping (new
`0066_chat_session_memory_focus.sql` migration — sequence renumbered
0059→0060→0061→0065→0066 as origin/main claimed the lower numbers);
event metadata enriched with `project` / `project_name` / `chat_title`.
- Recall queries normalized to single-keyword / explicit-OR ASCII (≤100
chars); shared normalizer export reused by per-turn recall.

### 2. Per-task executor transcript capture (RUFU-122)
Finished or failed tasks upload their executor transcript
(`agent-log.jsonl`) to Stash as a task session.

### 3. Bulk archive Stash sync (RUFU-125)
Archived task-planner chats soft-delete their Stash sessions on bulk
archival (paged). The snapshot of doomed session ids is taken *before*
the local bulk delete, and the Stash sync runs fire-and-forget so a
Stash stall can never delay local archival.

### 4. Per-chat "Preserve to Stash" backfill (RUFU-136)
A per-chat action that backfills a chat's full history into Stash, with
client-side idempotency and a pre-check that skips already-uploaded
content (fail-closed, no duplicate upload on transport failure).
- **Session-folder naming fix:** the first project folder is now named
"Fusion — &lt;project name&gt;" instead of the bare "Fusion" fallback
(the backfill now resolves the central-registry project name,
best-effort, never blocking the upload).

### 5. Opt-in semantic (vector) recall (RUFU-126)
`stashVectorSearch` setting (default `false` — **zero behavior change
until enabled**). For multi-word queries the backend tries Stash's
semantic-search endpoint first, then falls back byte-identically to the
keyword path. Definitive 404/405/501/503 responses are negatively cached
per process. Requires a patched Stash server (new endpoint +
`sentence-transformers` + embedding backfill); unpatched servers are
transparently bypassed after the first 404.

## Safety
- **Opt-in / inert by default:** the default backend remains `qmd`; the
Stash backend is inert until `memoryBackendType=stash` + `stashUrl` are
set.
- All Stash I/O is **best-effort, fail-closed, and non-blocking** — a
Stash outage never blocks chat, task completion, or archival. No
run-audit content is emitted.

## Testing
- Backfill + delete-sync suites (20/20), Stash backend suite (68/68),
executor memory / session capture suites, `memory-focus-recalling`,
description-guard — all green.
- `tsc` clean across core / engine / dashboard.
- Live verification: bulk backfill of 21/24 chats completed; the
"Preserve to Stash" action is idempotent on re-run.

## Changesets
- `@runfusion/fusion` **minor** — Stash memory backend + capture
(RUFU-068/121), per-task transcript (RUFU-122), bulk archive sync
(RUFU-125), per-chat backfill (RUFU-136), opt-in vector search
(RUFU-126)
- `@runfusion/fusion` **patch** — backfill session-folder naming fix


## Rebase Note (2026-08-23)

Rebased onto `origin/main` `3f448f7292` (v0.77.0-beta.7). Conflicts
resolved additively:
- `packages/core/src/postgres/schema-applier.ts` + test — upstream's
0062-0065 migrations (task/subtask splitting removal, AI merge review
reconciliation, task repository scope, FN-149 review convergence)
unioned with this PR's `chat_sessions.memory_focus` migration, which is
**renumbered 0065 → 0066** (upstream's FN-149 shipped 0065 canonically
on origin/main); `SCHEMA_BASELINE_VERSION` advances to `0066`.
- `packages/dashboard/app/components/ChatView.tsx` — upstream's docked
chat sidebar resize handlers unioned with the RUFU-136 "Preserve to
Stash" backfill handler.
- New commit: `settings.memory.*` stash-backend i18n keys added to all 6
secondary locales (RUFU-121/122 parity fix; `pnpm i18n:status` no longer
reports any violation introduced by this PR).

**Deploy note (operator environments that already ran a pre-rebase build
of this PR):** the memory-focus SQL may already be in the schema under
ledger row `0065`. Remap that row to `0066` (`UPDATE
fusion_schema_migrations SET version = '0066' WHERE version = '0065';`)
*before* first boot of a 0066-ceiling binary — otherwise the fresh
upstream `0065_fn_149_review_convergence_stage.sql` would be skipped as
"already applied". Clean databases (no prior memory-focus row) need no
action.

**CI note — Lint (lifecycle-column census) is red on the merge base:**
`pnpm check:lifecycle-columns --strict` fails identically on pure
`origin/main` `3f448f7292` with
`packages/core/src/db/legacy-adoption.ts: 0 -> 3` (3 column guards in
the U9b legacy-adoption table without a baseline entry or
`DELIBERATE-LITERAL` marker). Verified by running the census on a clean
origin/main checkout — inherited from the base, not introduced by this
PR. Fix belongs upstream; tracked separately.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Stash memory integration with project configuration and optional
semantic search.
  * Added per-chat memory focus controls and a `/focus` command.
  * Added “Preserve to Stash” for uploading complete chat history.
  * Added automatic chat, task transcript, and completion-event capture.
* Added project-specific Stash session folders and archive/delete
synchronization.
* **Bug Fixes**
* Improved Stash folder naming and handling of missing branches during
no-commit tasks.
* **Documentation**
* Added setup, configuration, integration, vector-search, and
performance guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Fusion <noreply@runfusion.ai>
Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com>
2026-08-23 16:46:14 -07:00
gsxdsm
5573d042a9 FN-8922: add persistent memory recall store
Add PostgreSQL-backed recall records for durable project memory retrieval.

- add recall record types, storage, search, deduplication, and prompt instructions
- migrate project schemas and expose recall APIs from core memory modules
- document plugin and storage contracts, with unit and PostgreSQL coverage

Files changed:
 .changeset/fn-8922-memory-recall-store.md          |   7 +
 docs/memory-plugin-contract.md                     |  10 +
 docs/storage.md                                    |   4 +
 .../__tests__/memory-recall-dedup-search.test.ts   |  10 +
 .../__tests__/memory-recall-instructions.test.ts   |  62 +++++
 .../postgres/memory-recall-store.pg.test.ts        | 296 +++++++++++++++++++++
 .../src/__tests__/postgres/schema-applier.test.ts  |  15 +-
 packages/core/src/index.ts                         |   1 +
 packages/core/src/memory/index.ts                  |   1 +
 packages/core/src/memory/project-memory.ts         |  33 ++-
 packages/core/src/memory/recall/index.ts           |  10 +
 packages/core/src/memory/recall/recall-dedup.ts    |  35 +++
 .../core/src/memory/recall/recall-instructions.ts  |  59 ++++
 packages/core/src/memory/recall/recall-search.ts   |  20 ++
 packages/core/src/memory/recall/recall-store.ts    |  51 ++++
 packages/core/src/memory/recall/recall-types.ts    |  14 +
 .../0052_fn_8922_memory_recall_records.sql         |  17 ++
 packages/core/src/postgres/schema-applier.ts       |  13 +-
 packages/core/src/postgres/schema/project.ts       |  10 +-
 19 files changed, 648 insertions(+), 20 deletions(-)

Fusion-Task-Id: FN-8922

Fusion-Task-Lineage: bb98f048-6fcf-46d9-bd1f-09294c2dc005

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
2026-08-10 04:46:10 -07:00
Fusion
cfdc72e511 feat(FN-3199): align memory insight docs with canonical paths
Updates three documentation files to align memory insight and plugin path references with their canonical locations.

Fusion-Task-Id: FN-3199
2026-05-04 13:43:22 -07:00
Fusion
69aa1a550a feat(FN-2605): merge fusion/fn-2605 (auto-resolved)
- test(FN-2605): complete Step 4 — align tests with planning labels
- docs(FN-2605): complete Step 3 — update demo and script terminology
- docs(FN-2605): complete Step 2 — update docs terminology
- docs(FN-2605): complete Step 1 — update README terminology
2026-04-26 12:34:02 -07:00
Fusion
8cd5936bb0 refactor(FN-2134): remove legacy memory path exports and bootstrap aliases
- Remove legacy memory path constants/helpers from core exports and backend contract surface
- Simplify memory bootstrap to initialize canonical layered memory files without seeding from .fusion/memory.md
- Update core memory tests to assert canonical .fusion/memory/MEMORY.md behavior and legacy-path rejection
- Refresh dashboard and docs naming/references to reflect canonical long-term memory path semantics
2026-04-19 10:13:34 -07:00
Fusion
08224f97f7 refactor(FN-2126): align memory handling with canonical .fusion/memory paths
- Update memory docs and contracts to reference canonical .fusion/memory files while treating the legacy top-level memory file as compatibility-only
- Tighten memory backend path normalization messaging and map stale qmd legacy top-level memory results back to .fusion/memory/MEMORY.md
- Remove legacy memory read/write fallback branches from backend initialization paths and rely on ensureOpenClawMemoryFiles() migration behavior
- Preserve migration-seeded legacy content during ensureMemoryFile() bootstrap and add regression coverage for seeded long-term memory creation
2026-04-19 10:13:34 -07:00
Fusion
e0f6647d11 feat(FN-2118): merge fusion/fn-2118 2026-04-19 10:13:33 -07:00
Fusion
91abafc11c docs(FN-2097): clarify layered memory layout and legacy fallback semantics
- Update memory architecture and contributing docs to describe .fusion/memory/ as the canonical layered workspace
- Reframe .fusion/memory.md references as a deprecated legacy fallback used only for migration/alias compatibility
- Expand the memory plugin contract with explicit layered layout, migration behavior, and invariant language
- Align settings reference memory-insight file descriptions with the canonical long-term memory model
2026-04-19 10:13:33 -07:00
Fusion
c920ecb6e5 docs(FN-2094): align memory plugin contract with OpenClaw layout
- Rewrite memory backend contract docs to match shipped MemoryBackend interfaces, capabilities, and registry behavior
- Clarify .fusion/memory/ as the canonical layered workspace and keep .fusion/memory.md compatibility-only
- Document backend resolution through memoryBackendType with qmd as the default and qmd-to-file search fallback behavior
- Synchronize architecture, contributing, and settings reference docs with the updated memory model terminology
2026-04-19 10:13:33 -07:00
Fusion
049f356f88 feat(FN-2087): finalize canonical memory path migration
- Remove legacy .fusion/memory.md fallback references and normalize prompts/docs to .fusion/memory/MEMORY.md
- Stop legacy mirror writes and fallback reads in core memory backend and project memory flows
- Update engine worktree boundary checks and tests for canonical memory file handling
- Align dashboard memory/settings surfaces and route tests with canonical memory behavior
- Add model-favorites persistence test coverage for mission interview and new agent dialogs
2026-04-18 22:47:17 -07:00
Fusion
53cad3ba9d fix(FN-2078): adopt canonical .fusion/memory paths
- Update core and engine prompt text to reference .fusion/memory/ with MEMORY.md and daily-note guidance
- Allow worktree tool boundary access to .fusion/memory/ files while preserving legacy .fusion/memory.md compatibility
- Revise memory compaction/insight extraction messaging and related type/docs comments to use canonical memory paths
- Refresh dashboard and test expectations across core/engine/dashboard/docs for the new memory path wording
2026-04-18 22:02:46 -07:00
gsxdsm
5b0b0591b3 feat(FN-1772): add backend-aware memory instruction context API
- Add MemoryBackendContext interface for backend-aware memory integration
- Implement getMemoryInstructions() method with backend-specific logic in project-memory.ts
- Add QMD memory backend type detection and instruction context for QMD-backed projects
- Update memory-plugin-contract.md with backend-variant instruction requirements (section 3.8.7)
- Update settings reference for memoryBackendType with custom backend support
- Add comprehensive regression tests for backend-variant memory instruction behavior
- Add changeset for @gsxdsm/fusion package
2026-04-13 22:03:42 -07:00
gsxdsm
27c02d33c7 feat(FN-1477): add memory pruning to daily insight extraction
- Add pruneStaleMemoryEntities() function to memory-insights.ts
- Integrate memory pruning into dailyInsightExtraction() workflow
- Add maxMemoryAgeDays setting (default: 30 days) for configurable pruning threshold
- Include pruned count in daily insight report summary
- Add comprehensive unit tests for memory pruning logic
- Update docs: architecture, memory-plugin-contract, settings-reference, contributing
2026-04-09 21:07:48 -07:00
gsxdsm
e8db0de099 feat(FN-1397): replace MultiProjectFlow smoke tests with behavior tests 2026-04-09 14:58:47 -07:00