Goodwill bonus'ta 360 gün gibi süreler gerekiyor; 90 gün sınırı iş
ihtiyacını karşılamıyordu. UI + route guard 3650'ye (10 yıl, parmak
kayması koruması) çekildi. Spoke tarafındaki eş limit sase.tr PR'ı ile
gevşiyor — o merge olana kadar spoke 90 üstünü 400 ile reddeder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Çeviri artefaktları gerçek parçayı gizliyor ("Charger Management"→"Şarjör
Yönetimi") — LLM v3 pasında orijinal ad 985 kova + 1.764 alt düzeltmesi
getirdi. Reviewer da aynı sinyali görsün: örnek adın altında amber renkte
"⟵ <orijinal>" satırı (name_original, çeviriden farklıysa).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kök neden: super_panel_reader read-only idi → saveMapping "permission denied
for table category_canonical_map" (42501) fırlatıyordu → unhandled 500, boş
gövde → client "Unexpected end of JSON input". Grant verildi (INSERT/UPDATE
category_canonical_map + UPDATE(canonical_category_id,canonical_subcategory_id)
categories). Route artık DB hatasını yakalayıp {ok:false,error} döner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Debriyaj 1" gibi patlak-şema sayfaları (set+disk+bilya karışık) tek alt-
kategoriye atanamaz; reviewer atlayınca kuyrukta tekrar beliriyor. Yeni buton:
is_manual=true + sub=NULL kaydeder → kuyruk sorgusu bu adları kalıcı hariç
tutar; Birleşik'te kova → Genel altında kalır (doğru). Kaydet artık no-sub
modunda alt seçilmeden yanlışlıkla genel-kayıt yapmaz (koruma + yönlendirme).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
'Sase 24h success 48.5% (-20.7pp)' alarmı yine coverage-gap artefaktıydı:
99 sorgunun 51'i katalog-boşluğu (41'i sadece 4 Tofaş-Fiat NM4* VIN'inin
5-17× retry'ı), gerçek infra hatası 0 — decode health %100.
- getSaseHealthSnapshot: getOperationalHealth ile aynı partition'la
decodable/decodeHealth/coverageGap alanları eklendi (ham alanlar duruyor)
- Ana sayfa KPI: başlık metrik decode health; ham oran + katalog-boşluğu
sayısı hint'te
- buildActionItems: düşüş uyarısı artık decodeHealthDeltaPp üzerinden
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P1 insight 6153c350 yanlış çıktı: 'VIN çözümleme sonuç dönmüyor, parça
listesi render edilemiyor' dediği session'da decode 3.3sn'de 'No catalog —
identified as Opel 2020' döndü, kullanıcı browse-CTA'ya tıklayıp Opel
kataloğunu gezdi. Panel vin_decode_no_catalog ve
vin_no_catalog_browse_clicked event'lerini PostHog'dan hiç çekmediği için
tagger 'initiated ama sonuç yok' gördü (vin_decode_no_outcome) ve analyzer
bunu P1 sessiz-hata hikayesine çevirdi.
- TRACKED_EVENTS: vin_decode_no_catalog, vin_no_catalog_browse_clicked,
vin_decode_vinpin_pending/resolved, catalog_recent_brand_clicked eklendi
- tagger: no_catalog ve vinpin_pending artık no-outcome saymaz; yeni
vin_decode_no_catalog_gap tag'i truthful hikayeyi analyzer'a taşır
- prompts: yeni tag ux_friction rotasına eklendi
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"(1901- )" gibi kod/tarih yaprakları (inherit ile kova almış, isimde sub sinyali
yok) ve Aksesuar gibi alt-yaprağı olmayan kovalar kuyruğa girmesin. REVIEWABLE'a
eklendi: NOT STRUCTURAL + isimde ≥3 harfli kelime var (NORM ~ '[a-z]{3,}').
no-sub'a: bucket'ın alt-yaprağı olmalı. Artık her satır gerçekten sub-atanabilir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Şemsiye/dal kategoriler ("Fren Sistemi", "Frenler", "Yakıt Sistemi") kuyruğa
girmesin — bunların tek alt-yaprağı yok, üst-seviyede doğrular. REVIEWABLE
filtresi: çocuğu olmayan (leaf) VE parça tutan kategoriler. Şemsiyeler 0-1
yaprak-parçalı olduğu için HAVING count>1 ile düşer. Reviewer yalnızca
sub-classify edilebilir spesifik parça gruplarını görür. Fast index-probe
(parent_id/category_id), tüm-kovalar worst-case 0.75s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Otomatik keyword pası sonrası kalan boşlukları SP'den kapatmak için. İki kuyruk:
"alt-kategori eksik" (mapped ama sub yok — asıl iş) + "kova eksik" (unmapped).
İsim-seviyesinde (source+normalized_name) review — karar o adı paylaşan tüm
araçlara yayılır. Kaydet → category_canonical_map is_manual=true (backfill
korumalı) + anında categories'e materialize. Kaynak/kova filtresi, etki-sıralı
kuyruk, kova-başına ilerleme. saseDb raw query (canonical tabloları prisma
şemasında yok). /operations/category-mapping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
api-keys internal uçları success alanı döndürmediği için spoke'un global
interceptor'ı {success,data} zarfı ekliyor (lifecycle/billing uçları kendi
success'ini döndürdüğünden zarflanmıyor). unwrapEnvelope iki şekli de kabul
eder; AdminApiError artık {error:{message}} zarfındaki mesajı da okur.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
volume_drop iki nedenle false-positive üretiyordu (2026-06-14 Pazar, decode
tamamen çalışırken saatte bir −%100 "kesinti" bastı):
1. Tek 15dk pencere — düşük B2B decode trafiği patlamalı (bir kullanıcı birkaç
sorgu yapar, sonra bir saat sessizlik), tek bir 0 penceresi sağlıklıyken bile
normal.
2. Ardışık-7-gün baseline — decode talebinde güçlü haftalık mevsimsellik var
(hafta sonu sakin); 7 gün baseline yoğun hafta içini hafta sonu beklentisine
katlıyor → normal bir Pazar (~1-2/saat) hafta-içi-şişmiş ~18/saat baseline'a
karşı −%100 okunuyor.
Düzeltme: volume_drop artık SON 1 SAATİ, AYNI GÜN/AYNI SAAT son 4 haftanın
ortalamasına karşı değerlendiriyor; yalnız o gün/saat tarihsel olarak gerçek
hacim görüyorsa (≥12/saat) ve son saat bunun ≥%80 altındaysa tetikleniyor.
Hafta sonu saatleri aynı-gün baseline'ı eşiğin altında kaldığı için kendiliğinden
susuyor; gerçek bir hafta-içi-yoğun-saat çöküşü hâlâ tetikliyor (prod query_logs
ile doğrulandı: bugünün Pazar saatleri SUSUYOR, Çarşamba 13:00 ~13/saat hâlâ izleniyor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
p95 was computed over ALL successful rows: cache hits (~10-30ms) set the
baseline, a single lock_wait row (duplicate click waiting on the in-flight
decode of the same VIN) set the spike. On a healthy 3-query window this read
as 'P95 15ms → 2058ms (137× baseline)' (2026-06-11, 4th such false alert).
- p95 now over FRESH decodes only (source='cache', db_hit/redis_positive and
lock_wait rows excluded; null-safe via coalesce)
- both windows must have ≥4 fresh decodes — a 'p95' of fewer is one slow
request, not a tail
- absolute floor raised 1s → 8s (fresh decodes legitimately take seconds;
matches the dashboard's fresh-P95 ok≤8s tone)
Simulated on today's prod data: alert window had 0 fresh decodes → silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/analytics/proxy: Sase proxy_logs telemetrisinden (sase.tr dev 45f7e3f)
provider × servis karnesi (A-F: başarı/ban/transport/latency ağırlıklı),
banlı exit IP / sticky session tablosu, ≥400 status dağılımı (402 = proxy
bakiye), günlük seyir. Tablo prod'a promote edilene dek bilgilendirici boş
durum gösterir.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lists catalog_issue_reported PostHog events (one-click 'Bildir' on
sase.tr empty/error catalog states, sase.tr dev c972a17): time, reporter
email (distinct_id → Sase users), VIN, vehicle, category, source, and
the page it happened on. Category distribution + unique-reporter KPIs,
7/30/90d ranges, linked from /analytics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prod doesn't write pcat_skipped/pcat_transient yet (sase fd78957 is on
dev), so the 2026-06-11 15:31 run still raised false gaps for Audi
(10/10 rows cold-skip), Hyundai (2/3) and VW (1/3 + 2 untraceable cache
echoes). A real pcat answer needs a network round-trip, so pcat <= 10ms
means skipped — exclude those rows. Verified on prod data: the three
false brands drop below threshold, Renault/Fiat/Honda etc. unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'No catalog — identified as X' rows produced during a pcat cold-pool
skip, transport blip, or open circuit breaker are not evidence of a
coverage gap; counting them raises false brand-gap insights (Audi,
2026-06-11). Excludes rows carrying the new pcat_skipped /
pcat_transient markers (sase.tr dev fd78957) or pcat_circuit_open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raw succeeded/total read 61% on a healthy pipeline because coverage
gaps (no-catalog, unknown VIN) counted as failures, and latency
percentiles mixed ~10ms cache hits with ~5-40s provider chains
(P50 28ms next to P95 25.8s meant nothing).
- partition failures exactly: no_catalog / unknown / real (timeout,
provider) via one CASE bucket
- Başarı = decode health (coverage gaps excluded), Gerçek hata real
failures only, new Kapsama dışı card (katalog yok + unknown VIN)
- P50/P95/P99 now fresh decodes only, cache hits excluded
- evening brief gains realErrorRatePct/decodeHealthPct/noCatalog24h
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After the success_rate_drop fix, the SAME benign data (one Fiat VIN, "No catalog —
identified as Fiat", retried across a couple users) just moved the false positive to
unknown_vin_spike (live: "Tanınamadı 11.1%→62.5%, 5/8, critical"). A car we DID
identify but lack a catalog for is a coverage gap, not an unrecognized VIN.
- unknown_vins / unknown_vin_users now count only TRULY unrecognized VINs:
exclude identified_no_catalog=1 (the "No catalog — identified as X" case).
- Add distinct_unknown_lookups (deduped) + gate the check on ≥4 distinct
unrecognized lookups so a couple users' uncovered cars can't trip it.
Validated vs prod query_logs: all recent windows truly_unknown=0 → GATED; a real
"tanınamadı" surge (≥2 users, ≥4 distinct unrecognized VINs, rate ≥2× baseline)
still fires. typecheck clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes acce1fc (which gated volume_spike/unknown_vin_spike for retry noise but
left success_rate_drop on a raw row-rate). Today two 🔴 success_rate_drop alerts
fired (88.9%→60%, 61.9%→33.3%) on a single user retrying one catalogless Fiat VIN
6× in a low-volume window — 0 real decode failures.
Root cause: success_rate_drop used succeeded/total over raw rows with only a
MIN_CURRENT_VOLUME=5 gate — no distinct-user gate and no coverage-gap exclusion,
unlike the sibling checks. At ~2.6 decodes/15min, one user's retries collapse the
"rate".
Fix — bring it to the same standard:
- Measure REAL (infra) failures only: exclude coverage gaps (no-catalog / unknown
/ unsupported VIN; ~20-35% of traffic, driven by which cars users query, not
decode health — already tracked by unknown_vin_spike).
- Dedupe by DISTINCT (user,vin) lookup so retries count once.
- Gate on ≥2 distinct users + ≥6 decodable lookups; fire on ≥10pp rise in real
failure rate AND ≥20% absolute floor.
- Also added an absolute-volume floor (MIN_VOLUME_SPIKE_ABS=25) to volume_spike,
which fired at now=8 yesterday (below baseline).
Validated against prod query_logs: both alert windows now GATED (real_failed=0);
the 06-10 09:30 real-incident window (7 users, 5/8 real failures, DataImpulse
degradation) still FIRES. typecheck clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Night of 2026-06-11 one user retrying a catalogless Toyota VIN tripped
volume_spike + unknown_vin_spike, and ~2.5/window night baselines made
quiet windows fire volume_drop at -100%.
- volume_spike: require >=2 distinct users and >=5 distinct (user,vin)
lookups so same-VIN retries don't count as a spike
- unknown_vin_spike: require unknown failures from >=2 distinct users
- volume_drop: raise expected-per-window floor from 1 to 4
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recordings tab filters by signal (errors / long >=5m / active >=30 clicks)
with live counts, highlights error sessions, and adds a one-click PostHog
replay player link per recording. Turns the 5k+ archived recordings into a
triage surface for watching problem sessions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prisma serialized windowDays as bigint -> make_interval(days => bigint) does
not exist (42883), so detectCatalogCoverageGaps threw on every real call: the
evening brief silently dropped its Sase-DB half and catalog-gap-detect never
created insights. Use a JS Date param (matches the rest of the codebase).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- experiment results compute a two-proportion z-test p-value + 95% CI per
variant vs control (significant = p<0.05); shown in the archive detail view
and used to label the leader honestly.
- evening brief: getExperimentBriefSummary feeds active-experiment results
(leader, lift, p-value, significance) into the brief metrics; prompt tells
the LLM to surface only statistically significant winners, not noisy lifts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Survey detail now shows the response funnel (shown -> sent -> dismissed) +
per-question distributions (rating/choice bars, open-text list), keyed by
$survey_id -> $survey_response_<questionId> from the archived definition.
Feature-flag/experiment detail shows per-variant exposure -> conversion with
lift-vs-control + a goal-event selector (signup/decode/trial/checkout/payment).
All from posthog_events; no schema change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
40-page cap captured only the most recent ~13 days. Add a 150ms inter-page
delay (PostHog list API rate-limits a tight loop) and raise the default cap
to 200 pages / 120d window so a run can sweep the full retention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- groups: posthog-resource-archive now also snapshots group types + group
instances (groups_types/ + groups/?group_type_index) into
posthog_resource_snapshots (change-detected).
- recordings: new posthog-recording-archive job (@*/6h) captures complete
metadata for EVERY recording into posthog_recordings (skipDuplicates over a
rolling 90d window) + MinIO dump — a superset of the insight pipeline's
promoted SessionMeta subset. rrweb blobs still archived by archive-recordings.
- /posthog-archive browser: Groups / Group Types tabs + a Recordings tab.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read view over posthog_resource_snapshots: type tabs (feature flags,
experiments, surveys, insights, dashboards, annotations, actions) with
resource counts, latest-per-resource table, and a detail page showing the
full lossless payload + change history (version timeline). Nav entry added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends archival beyond events/persons/cohorts. New generic resource
archiver (posthog-resource-archive@*/6h) snapshots surveys, feature flags,
experiments, dashboards, insights, annotations, and actions into
posthog_resource_snapshots — change-detected (hash over non-volatile config),
lossless `data` payload — and dumps a full per-type daily copy to MinIO
(resources/<type>/<date>.json.gz). listResource() pages any PostHog REST
resource. Survey responses + flag calls are already captured as events.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daily Sase brief aggregating VIN decode health, catalog coverage gaps,
revenue (Stripe/iyzico TRY), users/subscriptions, PostHog behavior + UTM
source, Meta ads (env-gated), and the insight pipeline into one
critical-points-first report.
- Brief model (panel DB): one row/project/day (date, headline, summary,
health score, structured body, raw 360 metrics).
- Panel: getSaseBriefMetrics() (saseDb half) + /api/internal/sase-brief-data.
- Worker: sase-evening-brief job collects panel metrics + PostHog
(hogqlQuery) + Meta (lib/meta, env-gated) + pipeline stats -> DeepSeek
structured brief -> upsert Brief -> Telegram. Scheduled 17:00 UTC
(20:00 Europe/Istanbul).
- UI: /projects/sase/briefs (table of days) + /briefs/[id] (detail:
critical-first, sections by status, recommendations, raw metrics).
Meta dormant until META_ACCESS_TOKEN/META_AD_ACCOUNT_ID set on the worker.
Needs a `briefs` table migration on the panel DB before first run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces "vehicle resolved but no parts/categories" — Sase's #1 churn
signal, previously invisible to insights (a prospect emailed support
instead of any insight firing).
Client (PostHog session tagger):
- fetch the catalog_*/category_view_changed/empty_catalog_cta_clicked
event family (was never pulled) and tag parts_render_blocked (P1) /
catalog_empty_result (P2), scoped to the real /dashboard product (demo
excluded). Harden the search-affordance-misuse guard so a real parts
failure is never suppressed. Route both tags to bug_triage.
Server (decode-log catalog gaps):
- new catalog-gap-detect worker job (every 6h) -> panel internal API
/api/internal/catalog-gap-check aggregates "No catalog - identified as
X" query_logs failures by brand -> upserts catalog_coverage_gap
insights (Telegram on P1), respecting founder triage. Add NO_CATALOG
error bucket.
Validated: both apps typecheck clean; tagger smoke 21/21; aggregation
over 90d real data yields 13 brand gaps (Renault/Fiat/Honda P1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On SIGTERM/SIGINT the worker only disconnected prisma/redis and exited — the
BullMQ Workers were never closed, so in-flight job locks were never released. On
every deploy the orchestrator killed the worker mid-flight, the next container
saw the half-finished jobs as "stalled", re-ran them, and the dead worker's
pending moveToFinished surfaced as `Missing lock for job
repeat:nightly-refresh:… moveToFinished`, with duplicate nightly-refresh firings
during the container overlap.
- startScheduledJobs / startInsightPipeline / startContentPipeline now return
their Worker so index.ts can close them.
- shutdown() closes all workers FIRST (releases locks, drains in-flight),
bounded by a 15s race so an in-flight job can't block past the orchestrator's
stop grace period, then disconnects prisma/redis.
- nightly worker now uses lockDuration 5min + stalledInterval 60s (matching the
pipeline/content workers); the 30s default could expire during panel-backup
(pg_dump) and trip the same stalled → re-run → Missing-lock cycle.
The scheduler config itself was already correct (nightly@03:00, audit@03:30,
panel-backup@04:00 — verified against the Redis job-scheduler ZSET; next fires
were exactly 03:00/03:30/04:00). tsc --noEmit clean; dedup + tagger smoke 30/30.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Users routinely use the VIN-only /dashboard/search box to look for a part by
name (e.g. "Cam düğme") or fiddle with search/history and rage-click out of
affordance confusion — while nothing is actually broken. The generic
ux_friction / frustrated_session tags turned these pure-rage sessions into
insights (noise; ~40 of the first 67 open insights were exactly this, all
dismissed by the founder).
apps/worker/src/lib/tagger.ts: when a session used the search box
(search_input_focused) but carries NO concrete failure (errorCount=0,
network5xxCount=0, no vin_decode_failed, search_input_validation_failed<3, no
payment_initiated/failed, no checkout_started), do not emit the generic
ux_friction / frustrated_session tags. With no other actionable tag the session
becomes tag-less and tag-sessions discards it → no compress / analyze / insight.
Scoped narrowly to search-box sessions on purpose, to avoid hiding genuine
parts/category bugs. Every concrete signal stays actionable: JS errors
(bug_suspected), 5xx (server_error_impact), payment friction, VIN upstream
provider failures, and ≥3 client validation failures (search_validation_friction).
Smoke: apps/worker/src/lib/tagger.smoke.ts (12/12) covers the misuse case plus
six must-keep cases. tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fingerprint hash (even after a7fe80f) still over-splits: the same root
problem produces different tag-sets / first-error / first-failed-endpoint
across sessions, so it hashes to a new fingerprint and creates a duplicate
insight. And generic ux_friction/frustrated_session sessions (category-tree
vs parts-panel vs schema-image) carry no structured signal to separate
sub-problems — only the LLM's reading of the timeline does. Dismissed insights
were only suppressed on an EXACT fingerprint repeat; a near-variant got a fresh
fingerprint and re-opened the theme. Dismissed rows were also excluded from the
fast-path cache, so every recurring session re-ran the LLM for no reason.
Two-layer dedup so (1) duplicates can't be created and (2) nothing similar to a
dismissed/duplicate theme is re-opened:
Layer 1 — fingerprint (or aliased fingerprint) fast-path, no LLM:
- dismissed/duplicate → SUPPRESS (bump occ/lastSeen, keep status)
- validated/shipped → flag REGRESSION (cheap, no re-analysis)
- active & fresh → attach occurrence
- active & stale → fall through to refresh re-analysis (unchanged)
Layer 2 — semantic dedup gate before creating a NEW insight (flash, no new infra):
- build a bounded catalog: all active insights + recently-touched
dismissed/duplicate/validated/shipped (lookback-windowed, capped)
- ask the model whether the session is the SAME underlying problem as a
catalog entry (anti-hallucination: only accept catalog ids; conf >= 0.72)
- on match: alias this fingerprint onto that insight + attach/suppress, so the
next identical session fast-paths in Layer 1 (no LLM, no duplicate row)
- no match: create a new insight as before
Schema: Insight.aliasFingerprints String[] (additive; applied via prisma db push
on deploy). Dedup-gate cost is logged to costLedger as promptTag=dedup_gate.
Tunables: INSIGHT_DEDUP_GATE (default on), INSIGHT_DEDUP_THRESHOLD (0.72),
INSIGHT_DEDUP_SUPPRESS_DAYS (90), INSIGHT_DEDUP_CATALOG_MAX (120).
Pure decision helpers (classifyExistingAction, shouldAcceptMatch) extracted and
covered by a smoke test (apps/worker `pnpm test`, 18/18). tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same root cause was producing one Insight row per user because URL carried
vehicleId/categoryId UUIDs and the raw URL fed the fingerprint hash. Result:
6 active "new" insights describing the same kategori-bouncing problem with
slightly different LLM phrasing, none deduplicated, only one (cmpclg7b9 →
sase.tr#76) had been triaged.
apps/worker/src/lib/compress.ts:
- new normalizePath() — collapses UUID / ULID / CUID2 / numeric path segments
to ":id", conservative on plain words. Mirrors what the LLM already sees in
the timeline.
- fingerprint inputs now:
tags (sorted) | normalizePath(url) | normalizeError(errors[0]) | normalizePath(failedEndpoints[0])
- removed header.severity from the hash — severity is a property of the
Insight bucket, not its identity; rage-click counts pushing the same root
cause across P1/P2/P3 was forcing extra rows.
Smoke (12/12 pass via tsx /tmp/check_fingerprint.ts):
- 6 historical kategori-bouncing URLs → 1 fingerprint
- severity changes don't move the hash
- distinct tag sets / distinct errors still split
Historical rows are untouched — only new compressed sessions get the new
hash. Old near-duplicate insights can be merged manually via the panel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reproducing insight cmpvfrjgc000114fzc7cdyh66 (a P1 "PL24 timeout" false
positive): trial user typed VW part numbers ("500 907 521", "5Q0 907 521")
into the VIN field on /; client-side regex rejected them with "Geçersiz şase
numarası. 17 karakter olmalı". No provider was called. The pipeline still
tagged the session as `vin_decode_fail_pattern`, routed to `provider_quality`,
and the LLM dutifully invented a PL24 outage.
Root cause spans three files:
1. tagger.ts grouped vin_decode_failed by `provider_attempted ?? source`. When
`provider_attempted` is missing, `source: "landing"` (a UI location) was
treated as a provider name, so a 1-provider set was synthesized and
`vin_decode_fail_pattern` (P1) was emitted.
2. compress.ts formatCustom whitelist excluded `error`, `source`, `vin`. The
LLM therefore never saw "Geçersiz şase numarası" or the offending input.
Pattern 3 mechanical hypothesis told it "check provider health" regardless.
3. prompts.ts pickPromptTag routed any `vin_decode_fail_pattern` straight to
`provider_quality` with no input-quality check, and the v3 system prompt
had no guardrail for client-side validation rejects.
Fix:
- tagger: detect client-side rejects by `error` regex (Turkish + English) and
by VIN shape (length != 17 or contains I/O/Q). When all fails are client
rejects, emit new tag `vin_decode_client_validation_fail` at P3 instead of
`vin_decode_fail_pattern` at P1. Real provider failures now require
`provider_attempted` to be set (no more `source` fallback).
- compress: add `error`, `source`, `vin` to the formatCustom property
whitelist so the LLM can see the actual failure context. Split Pattern 3
into client-reject vs. real-provider-failure branches with distinct
Turkish hypotheses.
- prompts: route `vin_decode_client_validation_fail` to `ux_friction` before
the provider rule. Ship provider_quality v4 with an explicit guardrail
instructing the model to return confidence ≤0.15 and reclassify when the
inlined event properties show client-side rejection.
The seed-runtime upsert path deactivates the active v3 template on next
worker boot and inserts v4 in its place — no manual SQL needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sentry-archive job was hardcoded to a single SENTRY_PROJECT slug.
Now that sase has a second Sentry project (sase-web, browser SDK), the
worker has to pull both — otherwise frontend events live only in
Sentry's UI and never reach the panel's sentry_events table for the
behavioral-insight pipeline to pick up.
Changes:
- SENTRY_PROJECT becomes a CSV (e.g. "python,sase-web") parsed once at
module load into PROJECTS[]. Backward-compatible: a single value
keeps working exactly as before.
- Issues are fetched org-wide in a single loop (unchanged endpoint)
but each issue's Sentry project slug is now persisted via the new
`sentrySourceProject` column on sentry_issues.
- Events are project-scoped on Sentry's side, so the job iterates
projects and calls listEventsPage(slug, cursor) for each. Per-project
pagination + dedup is preserved. A project-level failure is recorded
in `error` but doesn't abort the other projects.
- New result shape: `perProject: { [slug]: { fetched, inserted, duplicate } }`
Pipeline log gains a `[python=N/M sase-web=N/M]` breakdown so it's
obvious at a glance which project produced what.
- Schema: nullable `sentrySourceProject` on both SentryIssue and
SentryEvent (+ composite index per projectKey for queries that want
to filter "panel project = sase AND sentry project = sase-web").
Migration is a pure nullable add — `prisma db push` on container
start is safe. Existing rows stay null until next sync touches them.
After merge, set `SENTRY_PROJECT=python,sase-web` on the panel-worker
Coolify app and redeploy.
Mirrors Sentry into our DB before the free tier prunes events (~30d).
- `SentryIssue` (aggregate state, upserted to latest) + `SentryEvent`
(raw occurrences, lossless full payload JSONB, dedup by eventId, cold
dump to MinIO `sentry-archive/{project}/YYYY/MM/DD.jsonl.gz`).
- `lib/sentry.ts`: read-only client, Link-header cursor pagination,
listIssuesPage / listEventsPage. EU-region aware (SENTRY_API_BASE).
- `sentry-archive` job @hourly: issues upsert + events newest-first with
skipDuplicates, stops once a page is all-duplicates (caught up).
- Config via env: SENTRY_AUTH_TOKEN / SENTRY_ORG / SENTRY_PROJECT /
SENTRY_API_BASE.
Verified live against otolog/python (EU): a test event archived on run 1,
0 inserts / 1 duplicate on run 2 (dedup), issue upserted idempotently.
Note: Sase API currently sends to an inaccessible org's DSN (hardcoded
fallback); repointing SENTRY_DSN to otolog/python is a separate deploy step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append-only history of person properties and cohort definitions, since
the free tier overwrites the live person / prunes data. A new row is
written only when the payload changes (hash compare), so the tables stay
a compact change-timeline rather than a daily full copy.
- `PosthogPersonSnapshot` (personId join key to PosthogEvent.personId,
all distinct_ids, properties JSONB, propertiesHash).
- `PosthogCohortSnapshot` (cohortId, name, count, filters, stateHash).
- `listPersons` / `listCohorts` REST helpers (next-pagination) in posthog.ts.
- `stableHash` (sorted-key JSON hash) in hash.ts for change detection.
- `posthog-identity-archive` job wired @03:00 daily. Latest-hash lookup
via one DISTINCT ON query, inserts only changed rows via createMany.
Verified against prod: 242 persons → 242 snapshots on first run, 0 on
immediate re-run (change detection). 0 cohorts currently → graceful no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live verification surfaced two issues in archive-recordings:
- blob_v2 rejects wide blob-key ranges (a 23-key span 400s; ~5 is fine).
Fetch in contiguous chunks of BLOB_CHUNK (default 10) and concatenate.
- PostHog's snapshot API is aggressively rate-limited; back-to-back
sessions tripped 429. Pace requests (per-session + per-chunk delays)
and stop the run on sustained 429 — unarchived rows keep
rrwebArchivedAt=null and retry next cycle (oldest-first ordering
protects soon-to-be-deleted recordings first). Result now carries
rateLimited.
Verified against prod: 87- and 23-blob recordings reconstruct fully
(no 400), failed=0, graceful 429 stop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
