The fingerprint hash (even after a7fe80f) still over-splits: the same root
problem produces different tag-sets / first-error / first-failed-endpoint
across sessions, so it hashes to a new fingerprint and creates a duplicate
insight. And generic ux_friction/frustrated_session sessions (category-tree
vs parts-panel vs schema-image) carry no structured signal to separate
sub-problems — only the LLM's reading of the timeline does. Dismissed insights
were only suppressed on an EXACT fingerprint repeat; a near-variant got a fresh
fingerprint and re-opened the theme. Dismissed rows were also excluded from the
fast-path cache, so every recurring session re-ran the LLM for no reason.
Two-layer dedup so (1) duplicates can't be created and (2) nothing similar to a
dismissed/duplicate theme is re-opened:
Layer 1 — fingerprint (or aliased fingerprint) fast-path, no LLM:
- dismissed/duplicate → SUPPRESS (bump occ/lastSeen, keep status)
- validated/shipped → flag REGRESSION (cheap, no re-analysis)
- active & fresh → attach occurrence
- active & stale → fall through to refresh re-analysis (unchanged)
Layer 2 — semantic dedup gate before creating a NEW insight (flash, no new infra):
- build a bounded catalog: all active insights + recently-touched
dismissed/duplicate/validated/shipped (lookback-windowed, capped)
- ask the model whether the session is the SAME underlying problem as a
catalog entry (anti-hallucination: only accept catalog ids; conf >= 0.72)
- on match: alias this fingerprint onto that insight + attach/suppress, so the
next identical session fast-paths in Layer 1 (no LLM, no duplicate row)
- no match: create a new insight as before
Schema: Insight.aliasFingerprints String[] (additive; applied via prisma db push
on deploy). Dedup-gate cost is logged to costLedger as promptTag=dedup_gate.
Tunables: INSIGHT_DEDUP_GATE (default on), INSIGHT_DEDUP_THRESHOLD (0.72),
INSIGHT_DEDUP_SUPPRESS_DAYS (90), INSIGHT_DEDUP_CATALOG_MAX (120).
Pure decision helpers (classifyExistingAction, shouldAcceptMatch) extracted and
covered by a smoke test (apps/worker `pnpm test`, 18/18). tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sentry-archive job was hardcoded to a single SENTRY_PROJECT slug.
Now that sase has a second Sentry project (sase-web, browser SDK), the
worker has to pull both — otherwise frontend events live only in
Sentry's UI and never reach the panel's sentry_events table for the
behavioral-insight pipeline to pick up.
Changes:
- SENTRY_PROJECT becomes a CSV (e.g. "python,sase-web") parsed once at
module load into PROJECTS[]. Backward-compatible: a single value
keeps working exactly as before.
- Issues are fetched org-wide in a single loop (unchanged endpoint)
but each issue's Sentry project slug is now persisted via the new
`sentrySourceProject` column on sentry_issues.
- Events are project-scoped on Sentry's side, so the job iterates
projects and calls listEventsPage(slug, cursor) for each. Per-project
pagination + dedup is preserved. A project-level failure is recorded
in `error` but doesn't abort the other projects.
- New result shape: `perProject: { [slug]: { fetched, inserted, duplicate } }`
Pipeline log gains a `[python=N/M sase-web=N/M]` breakdown so it's
obvious at a glance which project produced what.
- Schema: nullable `sentrySourceProject` on both SentryIssue and
SentryEvent (+ composite index per projectKey for queries that want
to filter "panel project = sase AND sentry project = sase-web").
Migration is a pure nullable add — `prisma db push` on container
start is safe. Existing rows stay null until next sync touches them.
After merge, set `SENTRY_PROJECT=python,sase-web` on the panel-worker
Coolify app and redeploy.
Mirrors Sentry into our DB before the free tier prunes events (~30d).
- `SentryIssue` (aggregate state, upserted to latest) + `SentryEvent`
(raw occurrences, lossless full payload JSONB, dedup by eventId, cold
dump to MinIO `sentry-archive/{project}/YYYY/MM/DD.jsonl.gz`).
- `lib/sentry.ts`: read-only client, Link-header cursor pagination,
listIssuesPage / listEventsPage. EU-region aware (SENTRY_API_BASE).
- `sentry-archive` job @hourly: issues upsert + events newest-first with
skipDuplicates, stops once a page is all-duplicates (caught up).
- Config via env: SENTRY_AUTH_TOKEN / SENTRY_ORG / SENTRY_PROJECT /
SENTRY_API_BASE.
Verified live against otolog/python (EU): a test event archived on run 1,
0 inserts / 1 duplicate on run 2 (dedup), issue upserted idempotently.
Note: Sase API currently sends to an inaccessible org's DSN (hardcoded
fallback); repointing SENTRY_DSN to otolog/python is a separate deploy step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append-only history of person properties and cohort definitions, since
the free tier overwrites the live person / prunes data. A new row is
written only when the payload changes (hash compare), so the tables stay
a compact change-timeline rather than a daily full copy.
- `PosthogPersonSnapshot` (personId join key to PosthogEvent.personId,
all distinct_ids, properties JSONB, propertiesHash).
- `PosthogCohortSnapshot` (cohortId, name, count, filters, stateHash).
- `listPersons` / `listCohorts` REST helpers (next-pagination) in posthog.ts.
- `stableHash` (sorted-key JSON hash) in hash.ts for change detection.
- `posthog-identity-archive` job wired @03:00 daily. Latest-hash lookup
via one DISTINCT ON query, inserts only changed rows via createMany.
Verified against prod: 242 persons → 242 snapshots on first run, 0 on
immediate re-run (change detection). 0 cohorts currently → graceful no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Suspend / Reactivate / Ban buttons on the user detail page; wired to
spoke endpoints (sase.tr#27).
Admin SDK
- suspendUser/reactivateUser/banUser on SaseAdmin + LifecycleResult.
- SASE_ADMIN_ENDPOINTS updated.
Route
- POST /api/sase/users/[id]/lifecycle (action + reason). Auth + spoke
wired + min reason length checks. Audit on both success and failure.
RO model
- Sase Prisma schema adds status/statusReason/statusChangedAt/
statusChangedBy. getUser() returns lifecycleStatus + statusReason.
UI
- LifecycleStatusBadge in header next to email reveal.
- LifecycleButtons renders the right actions for the current state.
- Modals with reason textarea; ban requires a double-confirm checkbox.
- Impersonate hidden when user is suspended/banned (AuthGuard rejects).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Panel-side only — first writable feature on the Sase user detail page.
Notes are stored in the panel database (sase_user_notes); the spoke is
never touched (KVKK minimize).
Model
- SaseUserNote { id, saseUserId, authorUserId, body, pinned, createdAt }
- Index on (saseUserId, pinned, createdAt) for the default render order
API
- GET /api/sase/users/[id]/notes — list (pinned first, then newest)
- POST /api/sase/users/[id]/notes — { body, pinned } create (audit'li)
- PATCH/DELETE /api/sase/notes/[noteId] — toggle pin / hard delete
UI
- New "Notlar" tab on user detail. Textarea + pinned checkbox + submit;
list shows TR-localized timestamps, per-row pin/unpin and delete.
Schema sync
- package.json `start` now runs `prisma db push --skip-generate` before
`next start`. Panel uses db-push style (no migrations dir); this lets
the new table land on next deploy without a separate manual step.
Future destructive changes will require a smarter migration approach.
teknikborc.md updated: super_panel_reader role still needs to be created
on the new sase-postgres (current panel uses sase superuser).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- prisma/sase/schema.prisma with User + UserSubscription (partial view)
- lib/db-sase.ts exports typed RO client
- /projects/sase shows live metrics (users count, new users 7d, active subs)
- seed marks sase as active
DATABASE_URL_SASE_RO wired to super_panel_reader (SELECT-only)
