The fingerprint hash (even after a7fe80f) still over-splits: the same root
problem produces different tag-sets / first-error / first-failed-endpoint
across sessions, so it hashes to a new fingerprint and creates a duplicate
insight. And generic ux_friction/frustrated_session sessions (category-tree
vs parts-panel vs schema-image) carry no structured signal to separate
sub-problems — only the LLM's reading of the timeline does. Dismissed insights
were only suppressed on an EXACT fingerprint repeat; a near-variant got a fresh
fingerprint and re-opened the theme. Dismissed rows were also excluded from the
fast-path cache, so every recurring session re-ran the LLM for no reason.
Two-layer dedup so (1) duplicates can't be created and (2) nothing similar to a
dismissed/duplicate theme is re-opened:
Layer 1 — fingerprint (or aliased fingerprint) fast-path, no LLM:
- dismissed/duplicate → SUPPRESS (bump occ/lastSeen, keep status)
- validated/shipped → flag REGRESSION (cheap, no re-analysis)
- active & fresh → attach occurrence
- active & stale → fall through to refresh re-analysis (unchanged)
Layer 2 — semantic dedup gate before creating a NEW insight (flash, no new infra):
- build a bounded catalog: all active insights + recently-touched
dismissed/duplicate/validated/shipped (lookback-windowed, capped)
- ask the model whether the session is the SAME underlying problem as a
catalog entry (anti-hallucination: only accept catalog ids; conf >= 0.72)
- on match: alias this fingerprint onto that insight + attach/suppress, so the
next identical session fast-paths in Layer 1 (no LLM, no duplicate row)
- no match: create a new insight as before
Schema: Insight.aliasFingerprints String[] (additive; applied via prisma db push
on deploy). Dedup-gate cost is logged to costLedger as promptTag=dedup_gate.
Tunables: INSIGHT_DEDUP_GATE (default on), INSIGHT_DEDUP_THRESHOLD (0.72),
INSIGHT_DEDUP_SUPPRESS_DAYS (90), INSIGHT_DEDUP_CATALOG_MAX (120).
Pure decision helpers (classifyExistingAction, shouldAcceptMatch) extracted and
covered by a smoke test (apps/worker `pnpm test`, 18/18). tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same root cause was producing one Insight row per user because URL carried
vehicleId/categoryId UUIDs and the raw URL fed the fingerprint hash. Result:
6 active "new" insights describing the same kategori-bouncing problem with
slightly different LLM phrasing, none deduplicated, only one (cmpclg7b9 →
sase.tr#76) had been triaged.
apps/worker/src/lib/compress.ts:
- new normalizePath() — collapses UUID / ULID / CUID2 / numeric path segments
to ":id", conservative on plain words. Mirrors what the LLM already sees in
the timeline.
- fingerprint inputs now:
tags (sorted) | normalizePath(url) | normalizeError(errors[0]) | normalizePath(failedEndpoints[0])
- removed header.severity from the hash — severity is a property of the
Insight bucket, not its identity; rage-click counts pushing the same root
cause across P1/P2/P3 was forcing extra rows.
Smoke (12/12 pass via tsx /tmp/check_fingerprint.ts):
- 6 historical kategori-bouncing URLs → 1 fingerprint
- severity changes don't move the hash
- distinct tag sets / distinct errors still split
Historical rows are untouched — only new compressed sessions get the new
hash. Old near-duplicate insights can be merged manually via the panel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reproducing insight cmpvfrjgc000114fzc7cdyh66 (a P1 "PL24 timeout" false
positive): trial user typed VW part numbers ("500 907 521", "5Q0 907 521")
into the VIN field on /; client-side regex rejected them with "Geçersiz şase
numarası. 17 karakter olmalı". No provider was called. The pipeline still
tagged the session as `vin_decode_fail_pattern`, routed to `provider_quality`,
and the LLM dutifully invented a PL24 outage.
Root cause spans three files:
1. tagger.ts grouped vin_decode_failed by `provider_attempted ?? source`. When
`provider_attempted` is missing, `source: "landing"` (a UI location) was
treated as a provider name, so a 1-provider set was synthesized and
`vin_decode_fail_pattern` (P1) was emitted.
2. compress.ts formatCustom whitelist excluded `error`, `source`, `vin`. The
LLM therefore never saw "Geçersiz şase numarası" or the offending input.
Pattern 3 mechanical hypothesis told it "check provider health" regardless.
3. prompts.ts pickPromptTag routed any `vin_decode_fail_pattern` straight to
`provider_quality` with no input-quality check, and the v3 system prompt
had no guardrail for client-side validation rejects.
Fix:
- tagger: detect client-side rejects by `error` regex (Turkish + English) and
by VIN shape (length != 17 or contains I/O/Q). When all fails are client
rejects, emit new tag `vin_decode_client_validation_fail` at P3 instead of
`vin_decode_fail_pattern` at P1. Real provider failures now require
`provider_attempted` to be set (no more `source` fallback).
- compress: add `error`, `source`, `vin` to the formatCustom property
whitelist so the LLM can see the actual failure context. Split Pattern 3
into client-reject vs. real-provider-failure branches with distinct
Turkish hypotheses.
- prompts: route `vin_decode_client_validation_fail` to `ux_friction` before
the provider rule. Ship provider_quality v4 with an explicit guardrail
instructing the model to return confidence ≤0.15 and reclassify when the
inlined event properties show client-side rejection.
The seed-runtime upsert path deactivates the active v3 template on next
worker boot and inserts v4 in its place — no manual SQL needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sentry-archive job was hardcoded to a single SENTRY_PROJECT slug.
Now that sase has a second Sentry project (sase-web, browser SDK), the
worker has to pull both — otherwise frontend events live only in
Sentry's UI and never reach the panel's sentry_events table for the
behavioral-insight pipeline to pick up.
Changes:
- SENTRY_PROJECT becomes a CSV (e.g. "python,sase-web") parsed once at
module load into PROJECTS[]. Backward-compatible: a single value
keeps working exactly as before.
- Issues are fetched org-wide in a single loop (unchanged endpoint)
but each issue's Sentry project slug is now persisted via the new
`sentrySourceProject` column on sentry_issues.
- Events are project-scoped on Sentry's side, so the job iterates
projects and calls listEventsPage(slug, cursor) for each. Per-project
pagination + dedup is preserved. A project-level failure is recorded
in `error` but doesn't abort the other projects.
- New result shape: `perProject: { [slug]: { fetched, inserted, duplicate } }`
Pipeline log gains a `[python=N/M sase-web=N/M]` breakdown so it's
obvious at a glance which project produced what.
- Schema: nullable `sentrySourceProject` on both SentryIssue and
SentryEvent (+ composite index per projectKey for queries that want
to filter "panel project = sase AND sentry project = sase-web").
Migration is a pure nullable add — `prisma db push` on container
start is safe. Existing rows stay null until next sync touches them.
After merge, set `SENTRY_PROJECT=python,sase-web` on the panel-worker
Coolify app and redeploy.
Mirrors Sentry into our DB before the free tier prunes events (~30d).
- `SentryIssue` (aggregate state, upserted to latest) + `SentryEvent`
(raw occurrences, lossless full payload JSONB, dedup by eventId, cold
dump to MinIO `sentry-archive/{project}/YYYY/MM/DD.jsonl.gz`).
- `lib/sentry.ts`: read-only client, Link-header cursor pagination,
listIssuesPage / listEventsPage. EU-region aware (SENTRY_API_BASE).
- `sentry-archive` job @hourly: issues upsert + events newest-first with
skipDuplicates, stops once a page is all-duplicates (caught up).
- Config via env: SENTRY_AUTH_TOKEN / SENTRY_ORG / SENTRY_PROJECT /
SENTRY_API_BASE.
Verified live against otolog/python (EU): a test event archived on run 1,
0 inserts / 1 duplicate on run 2 (dedup), issue upserted idempotently.
Note: Sase API currently sends to an inaccessible org's DSN (hardcoded
fallback); repointing SENTRY_DSN to otolog/python is a separate deploy step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append-only history of person properties and cohort definitions, since
the free tier overwrites the live person / prunes data. A new row is
written only when the payload changes (hash compare), so the tables stay
a compact change-timeline rather than a daily full copy.
- `PosthogPersonSnapshot` (personId join key to PosthogEvent.personId,
all distinct_ids, properties JSONB, propertiesHash).
- `PosthogCohortSnapshot` (cohortId, name, count, filters, stateHash).
- `listPersons` / `listCohorts` REST helpers (next-pagination) in posthog.ts.
- `stableHash` (sorted-key JSON hash) in hash.ts for change detection.
- `posthog-identity-archive` job wired @03:00 daily. Latest-hash lookup
via one DISTINCT ON query, inserts only changed rows via createMany.
Verified against prod: 242 persons → 242 snapshots on first run, 0 on
immediate re-run (change detection). 0 cohorts currently → graceful no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live verification surfaced two issues in archive-recordings:
- blob_v2 rejects wide blob-key ranges (a 23-key span 400s; ~5 is fine).
Fetch in contiguous chunks of BLOB_CHUNK (default 10) and concatenate.
- PostHog's snapshot API is aggressively rate-limited; back-to-back
sessions tripped 429. Pace requests (per-session + per-chunk delays)
and stop the run on sustained 429 — unarchived rows keep
rrwebArchivedAt=null and retry next cycle (oldest-first ordering
protects soon-to-be-deleted recordings first). Result now carries
rateLimited.
Verified against prod: 87- and 23-blob recordings reconstruct fully
(no 400), failed=0, graceful 429 stop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coupled fixes that together stop frustrated sessions from being
mislabelled as `power_user_path` and make click timelines actually
diagnosable.
1) Counter propagation (compress.ts, compress-sessions.ts):
compress now returns rageClickCount / network5xxCount / network4xxCount
and writes them back to SessionMeta. Previously these stayed at 0
forever because PostHog recording metadata doesn't expose them and
nothing updated the row after compress ran.
2) Tag rule hardening (tagger.ts, compress-sessions.ts):
- power_user_path v1 fallback now requires rageClickCount===0; without
this guard any auth user with 20+ clicks (rage clusters included)
was labelled a power user.
- New frustrated_session tag (rageClickCount>=3, P2) for sustained
friction beyond a single cluster.
- compress-sessions re-runs tagSession+scoreSession after writing the
fresh counters, so the corrected tags land on the row.
3) Click target enrichment (compress.ts):
describeTarget used to read tagName/attributes off MouseInteraction
events — fields that don't exist on rrweb type=3 source=2 — so every
click rendered as `[el]`. Now compress builds a node map from
FullSnapshot (type=2) and keeps it in sync with mutations
(type=3 source=0), then resolves clicks via `d.id`. Clicks now show
real tag + visible text / aria-label / name / data-testid.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HYPOTHESES block was empty on most sessions because the only rules
covered backend failure patterns (5xx + rage). Add 7 behavioral patterns
derived from canonical custom events — these give the LLM analyzer
concrete starting points instead of reasoning from raw timeline alone.
Patterns (all conservative — fire only on clear evidence):
1. Catalog evaluated, no checkout (parts_panel_viewed>=3 + on upgrade
page + !checkout_started)
2. Fresh trial + successful decode (happy onboarding)
3. VIN decode failure pattern (failures>=2 or fallback>=1)
4. Rage click on subscription UI (upgrade friction)
5. Cancel flow entered without resume (churn risk)
6. Trial urgency banner ignored (banner_viewed without CTA)
7. Search opened but no decode attempted (funnel drop)
The previous 500-token threshold fired on virtually every authenticated
session because rrweb masks inputs at source and stripQuery() removes
URL params before sanitize() runs — so 0 matches is the normal case,
not an anomaly. Confirmed on session 019e455e: 9m32s authed session
with ~1093 tokens, 0 matches, no actual PII leak.
- Raise broad threshold 500→2000 tokens
- Add precise leak detector: any unmasked \`input [tag] "text"\` line
in the compressed timeline AND authed AND 0 matches → alert. This
catches the actual failure mode (PostHog source-side masking broken)
instead of guessing from total token volume.
- Sync UI warning threshold to match (500→2000).
inferSubscriptionTier only checked subscription_tier|tier|plan_tier|plan,
but Sase.tr's setPeopleProperties writes subscription_plan. Result: every
authed user showed tier=— in /insights/sessions/* and the tagger's
userValue boost (full=1.0, brand=0.7) collapsed to the 0.5 default.
- Add subscription_plan to the key list (priority first since it matches
Sase.tr's actual property name)
- Extract into shared extractTierFromProps(props) helper
- In tag-sessions, re-evaluate tier from enrichment.userProperties and
backfill when null. Tag step runs ~2min after ingest, by which point
PostHog has the user's post-signup $set props that weren't on the
recording row at ingest time.
Drop successful (2xx/3xx) asset and telemetry GETs from the semantic
timeline — these were eating MAX_LINES (80) before any user-behavior
signal could surface. In a 9m32s session we observed only the first
1:18 reached the LLM input because PostHog /collect/* pings filled
the budget. Failures (4xx/5xx) are still emitted as signal.
Filtered prefixes: /collect/, /flags, /array/, /static/, /assets/,
/api/surveys, /css2. Also raise MAX_LINES 80→120.
Add optional { sessionId } to POST /api/insights/reprocess so a single
session can be re-run end-to-end without resetting a whole batch.
Closes the alert side of the deploy-regression view added in Faz 2b.
The dashboard table already flagged regressed deploys; this commit
pushes a Telegram when one happens, so MTTD doesn't depend on the
founder checking the dashboard.
Detection (panel)
- detectVinRegressions() pulls the last 20 Coolify deploys for the
Sase.tr app, filters to those whose post-window has elapsed (≥30min
since finishedAt) and isn't too old (≤180min since finishedAt), and
reuses analyzeDeployRegressions to compute the 30min before/after
success-rate slices. A row is flagged when:
- both before and after have ≥5 samples, and
- success rate dropped ≥10pp (severity 'high'; ≥15pp → 'critical').
- Returns a RegressionHit per flagged deploy with a 24h dedupe TTL
keyed on deploymentUuid so each deploy alerts exactly once ever
(regardless of how often the 5-min cron checks).
Endpoint
- GET /api/internal/vin-anomaly-check now returns
{ ok, current, baseline, anomalies, regressions }.
Worker
- sendTelegram() accepts an optional dedupeTtlSeconds override so
per-call long-TTL dedupes (like deploy alerts) don't have to go
through the global env default.
- New alertVinRegression() formats severity icon + before/after %
+ deploy commit/timestamp + dashboard link.
- runVinAnomalyDetect now also walks the regressions array and
fires Telegram for each. Returns { anomalies, regressions,
alertsFired, alertsDeduped }; pipeline log prints when either
count is non-zero.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the active half of Phase 8: dashboard was passive, this pushes
when something breaks. Targets MTTD < 10 minutes from the PRD.
Detection (panel-web)
- detectVinAnomalies() compares a 15-minute current window against a
baseline built from the same 15-minute slot across the previous 7
days (same-hour-of-day, average across days that had ≥ 1 sample).
- Five hits, two severities:
* success_rate_drop — high at ≥ 5pp drop, critical at ≥ 15pp
* p95_latency_spike — high at 2× baseline (and > 1s), critical at 3×
* volume_drop — high at ≥ 80% below baseline, critical at ≥ 95%
* volume_spike — high at ≥ 5× baseline
* timeout_dominance — high when ≥ 50% of failures are timeouts
- Minimum-volume guards on both current and baseline so quiet hours
don't generate noise (MIN_CURRENT_VOLUME=5, MIN_BASELINE_VOLUME=10).
- Each anomaly carries a 15-min-bucket dedupe key — same anomaly type
fires at most once per bucket regardless of cron cadence.
Endpoint
- GET /api/internal/vin-anomaly-check, gated by x-internal-worker-token
header (constant-time compare against INTERNAL_WORKER_TOKEN env).
Returns { current, baseline, anomalies[] }.
Worker
- New job vin-anomaly-detect, BullMQ scheduler */5 * * * *.
Fetches the panel endpoint, then for each anomaly calls
alertVinAnomaly() — sendTelegram with the bucket dedupe key, so
Redis SETEX NX dedupes across the 1h cooldown window.
- alertVinAnomaly() in worker/lib/telegram.ts formats the message with
severity icon + baseline/observed/volume context + dashboard link.
- runVinAnomalyDetect returns { ok, checked, anomalies, alertsFired,
alertsDeduped }; pipeline logs only when something happened or the
check failed.
Env
- INTERNAL_WORKER_TOKEN set on both panel-web and panel-worker
(32-byte hex, generated in Coolify).
- PANEL_INTERNAL_URL on panel-worker → coolify-network UUID hostname
for panel-web, no Cloudflare/Tailscale hop on internal calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
apps/worker/src/lib/prompts.ts:
- SASE_CONTEXT'e Türkçe çıktı kuralı eklendi (tüm doğal dil alanları Türkçe,
enum/const/teknik tanımlayıcılar İngilizce kalır)
- 7 seed prompt v2'ye bump edildi (bug_triage, ux_friction, payment_issue,
onboarding_stuck, upgrade_hesitation, pattern_bundle, provider_quality)
- name suffix "v2 (TR)"
apps/worker/src/lib/seed-runtime.ts:
- Yeni version insert ederken, aynı tag'in eski active versiyonlarını otomatik
deactivate eder (publishPromptVersion ile aynı davranış)
apps/web/src/lib/gitea.ts (buildIssueBody):
- Tüm bölüm başlıkları Türkçe: Bağlam, Hipotez, Kullanıcı niyeti, Friction
noktası, Tekrarlama adımları, Etkilenen alan, Hata imzası, Şüpheli provider,
Hata modu, Hızlı çözüm, Uzun vadeli çözüm, Önerilen aksiyon, Araştırma
önerisi, Tıkanma noktası, Anlaşılmayan kavram, Dokümantasyon eksiği,
Önerilen in-app yardım, Tahmini efor, Kanıt, Definition of Done
- Önem/Tekrar/İlk/Son görüldü etiketleri
Sonraki analyze döngüsü itibarıyla yeni insight'ların metin alanları Türkçe.
Mevcut insight #cmp4o*, #cmp5e* (v1 ile üretilmiş) değişmez.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bug: previous logic advanced watermark to the latest session start_time including
ongoing sessions. PostHog session_recordings filters by start_time, so once a
session was 'seen' as ongoing the watermark moved past its start time and the
session was never re-fetched after it ended. Today 4 auth sessions on
/dashboard/vehicles/* and /dashboard/search (07:17-07:40 UTC) were lost this way.
Fix:
1. ROLLING_LOOKBACK_MINUTES (default 60): every cycle queries date_from =
min(watermark, now - 60min). Sessions that just finished get re-fetched
regardless of watermark drift. Upsert dedupes.
2. Track earliestOngoingStart; cap watermark to (earliestOngoingStart - 1s)
so subsequent cycles re-read that range.
Also added GET/DELETE /api/insights/watermark for manual reset (used to
trigger 24h backfill after this deploy).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
7 new event names added to TRACKED_EVENTS so enrich.ts fetches them for compression
timeline interleaving + tagger context. No new tag rules yet (Faz 2 is enrichment,
not signal extraction).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Webhook can't reach sp.semih.ai from internet (Tailscale-only DNS). Switched to a
worker job that polls GitHub for tracked issue states every 10 minutes.
apps/worker/src/lib/github.ts: read-only getIssue() client
apps/worker/src/jobs/github-sync.ts: scan insights with githubIssueNumber + not validated/dismissed/duplicate,
fetch remote state, reflect transitions:
open → closed = shipped + shippedAt + validationStartedAt
closed → open = in_progress + clear validation state
Scheduler: github-sync@*/10min added.
The /api/webhooks/github route stays (works if reachable in future) but is no longer
the source of truth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- thinking={type:disabled} skips reasoning trace → output_tokens drop 10x
- find first content block of type='text' (V4 may emit thinking before text)
- DEEPSEEK_THINKING=true env to re-enable if needed for hard problems
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
