The fingerprint hash (even after a7fe80f) still over-splits: the same root
problem produces different tag-sets / first-error / first-failed-endpoint
across sessions, so it hashes to a new fingerprint and creates a duplicate
insight. And generic ux_friction/frustrated_session sessions (category-tree
vs parts-panel vs schema-image) carry no structured signal to separate
sub-problems — only the LLM's reading of the timeline does. Dismissed insights
were only suppressed on an EXACT fingerprint repeat; a near-variant got a fresh
fingerprint and re-opened the theme. Dismissed rows were also excluded from the
fast-path cache, so every recurring session re-ran the LLM for no reason.
Two-layer dedup so (1) duplicates can't be created and (2) nothing similar to a
dismissed/duplicate theme is re-opened:
Layer 1 — fingerprint (or aliased fingerprint) fast-path, no LLM:
- dismissed/duplicate → SUPPRESS (bump occ/lastSeen, keep status)
- validated/shipped → flag REGRESSION (cheap, no re-analysis)
- active & fresh → attach occurrence
- active & stale → fall through to refresh re-analysis (unchanged)
Layer 2 — semantic dedup gate before creating a NEW insight (flash, no new infra):
- build a bounded catalog: all active insights + recently-touched
dismissed/duplicate/validated/shipped (lookback-windowed, capped)
- ask the model whether the session is the SAME underlying problem as a
catalog entry (anti-hallucination: only accept catalog ids; conf >= 0.72)
- on match: alias this fingerprint onto that insight + attach/suppress, so the
next identical session fast-paths in Layer 1 (no LLM, no duplicate row)
- no match: create a new insight as before
Schema: Insight.aliasFingerprints String[] (additive; applied via prisma db push
on deploy). Dedup-gate cost is logged to costLedger as promptTag=dedup_gate.
Tunables: INSIGHT_DEDUP_GATE (default on), INSIGHT_DEDUP_THRESHOLD (0.72),
INSIGHT_DEDUP_SUPPRESS_DAYS (90), INSIGHT_DEDUP_CATALOG_MAX (120).
Pure decision helpers (classifyExistingAction, shouldAcceptMatch) extracted and
covered by a smoke test (apps/worker `pnpm test`, 18/18). tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b61d955256