Semih
|
b635b529e2
|
feat(phase6e): eval framework + prompt editor + daily brief + retention + bundle mode
Schema:
- eval_sets (promptTag, cases JSON: [{id, timeline, expected: {...}}])
- eval_runs (per-execution scoring: passedSchema/Severity/Rubric, cost, tokens, results)
Prompts:
- 2 new seed templates: upgrade_hesitation (flash), pattern_bundle (pro)
- pickPromptTag() routes 'upgrade_hesitation' tag to upgrade_hesitation prompt
- Editor UI at /insights/settings/prompts/[id]: edit system/user/schema/tier/temp,
publishPromptVersion() creates new version + deactivates old (CRUD with auto bump)
- setPromptActive() to toggle versions
Daily Brief (/insights/brief):
- Last 24h: sessions processed, insights produced (with severity breakdown),
cost, cache hit rate; today's top 5 priorities; week stats (shipped/validated/regressed)
Retention cron (04:15 UTC daily):
- Delete sessions_meta + session_custom_events older than 90d (unless referenced by
active insight)
- Delete compressed_sessions rows + MinIO timeline blobs older than 180d
- raw_metadata 30d (currently no-op; we don't persist raw metadata to MinIO)
Eval framework:
- apps/worker/src/jobs/eval-run.ts: runs cases against current/specified prompt version,
scores schema_pass + severity_match + rubric_substring; stores EvalRun
- apps/worker BullMQ queue handler for 'eval-run' job name
- apps/web installed bullmq; /lib/queue.ts thin Queue accessor
- Web actions: createEvalSet, triggerEvalRun (enqueues job to insight-pipeline queue)
- UI: /insights/settings/eval-sets list, /new create form (paste JSON cases),
/[id] detail with Run button + recent runs + per-case JSON
Bundle mode (analyze job):
- Pull 3x batch, group by fingerprint
- Groups ≥ INSIGHT_BUNDLE_THRESHOLD (default 3) → use 'pattern_bundle' prompt
- Timeline = primary rep + PATTERN BUNDLE summary block (occurrences, unique users, deltas)
- Insight stores ALL group session IDs as relatedSessionIds; all marked analyzed in one tx
- Cost amortized: 1 LLM call per group
Nav/Cmd+K:
- Inbox header links: Brief, Patterns, Eval sets
- Palette: Daily brief, Eval sets entries
Deferred to backlog: embedding-similarity cross-fingerprint clustering, Telegram brief delivery,
Sase.tr-side data-private audit (separate repo).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
2026-05-14 06:11:48 +00:00 |
|
Semih
|
909cacf7b3
|
feat(phase4): rate limit + audit archive + panel backup + DR docs
- src/lib/rate-limit.ts (Redis sliding window, fail-open)
- before-hook on /sign-in/email: 5 attempts/min per IP+email
- worker job audit-archive (daily 03:30, JSONL → MinIO, 90d retention)
- worker job panel-backup (daily 04:00, pg_dump -Fc -Z9 → MinIO)
- Dockerfile adds postgresql16-client
- scripts/restore-drill.sh restores latest dump into panel_drill
- docs/disaster-recovery.md + docs/phase4-deferred.md (mTLS + Infisical rationale)
|
2026-05-13 11:14:36 +00:00 |
|
Semih
|
d189206f06
|
feat(phase1a): admin-sdk base, audit helper, project seed, panel pages
- src/lib/audit.ts + src/lib/admin-sdk/base.ts (typed HTTP + auto audit)
- prisma/seed.ts seeds 5 projects (all inactive)
- pages: /projects, /projects/[key], /audit, /operations, /events, /settings
- components/panel-shell.tsx shared layout
- entrypoint runs seed after db push
|
2026-05-13 10:33:18 +00:00 |
|