root/sp
1
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(analytics): experiment significance + experiments in the evening brief

Browse Source 
- experiment results compute a two-proportion z-test p-value + 95% CI per
  variant vs control (significant = p<0.05); shown in the archive detail view
  and used to label the leader honestly.
- evening brief: getExperimentBriefSummary feeds active-experiment results
  (leader, lift, p-value, significance) into the brief metrics; prompt tells
  the LLM to surface only statistically significant winners, not noisy lifts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Semih
2026-06-10 22:33:05 +03:00
parent 99965f4088
commit ea63f4d167
4 changed files with 142 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
apps/web/src/app/posthog-archive/[type]/[id]/_results.tsx
View File

@@ -117,21 +117,33 @@ export async function ExperimentResultsView({
<div className="h-4 flex-1 rounded-sm bg-muted">
<div className="h-4 rounded-sm bg-primary/80" style={{ width: `${Math.min(100, v.convRate)}%` }} />
</div>
<div className="w-44 shrink-0 text-right tabular-nums text-xs text-muted-foreground">
{v.converted}/{v.exposed} = <b className="text-foreground">{v.convRate}%</b>
{v.liftVsControl !== null && (
<span className={v.liftVsControl > 0 ? "ml-1 text-emerald-600" : "ml-1 text-red-600"}>
{v.liftVsControl > 0 ? "+" : ""}
{v.liftVsControl}%
</span>
)}
<div className="w-56 shrink-0 text-right text-xs">
<div className="tabular-nums">
{v.converted}/{v.exposed} = <b className="text-foreground">{v.convRate}%</b>
{v.liftVsControl !== null && (
<span className={v.liftVsControl > 0 ? "ml-1 text-emerald-600" : "ml-1 text-red-600"}>
{v.liftVsControl > 0 ? "+" : ""}
{v.liftVsControl}%
</span>
)}
</div>
<div className="text-[10px] text-muted-foreground">
GA: {v.ciLow}–{v.ciHigh}%
{v.pValue !== null && (
<span className={v.significant ? "ml-1 font-medium text-emerald-600" : "ml-1"}>
· p={v.pValue}
{v.significant ? " ✓" : ""}
</span>
)}
</div>
</div>
</div>
))}
{variants.length === 0 && <div className="text-sm text-muted-foreground">exposure verisi yok</div>}
</div>
<p className="text-[11px] text-muted-foreground">
Lider = ≥30 exposure&apos;lı en yüksek dönüşüm. Lift = control&apos;e göre. (Basit dönüşüm; istatistiksel anlamlılık testi değil.)
Lider = ≥30 exposure&apos;lı en yüksek dönüşüm · Lift = control&apos;e göre · GA = %95 güven aralığı ·
p = two-proportion z-test (control&apos;e karşı), p&lt;0.05 → ✓ anlamlı.
</p>
</CardContent>
</Card>

121
apps/web/src/lib/analytics/experiment-results.ts
View File

@@ -74,8 +74,34 @@ export type VariantResult = {
converted: number;
convRate: number;
liftVsControl: number | null;
ciLow: number;
ciHigh: number;
pValue: number | null; // two-proportion z-test vs control
significant: boolean; // p < 0.05 vs control
};
// Standard-normal CDF via erf (Abramowitz-Stegun 7.1.26) for the z-test p-value.
function erf(x: number): number {
const t = 1 / (1 + 0.3275911 * Math.abs(x));
const y =
1 -
((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) *
t *
Math.exp(-x * x);
return x >= 0 ? y : -y;
}
const normCdf = (z: number): number => 0.5 * (1 + erf(z / Math.SQRT2));
function ci95(ratePct: number, n: number): [number, number] {
if (n <= 0) return [0, 0];
const p = ratePct / 100;
const se = Math.sqrt((p * (1 - p)) / n);
return [
Math.max(0, Math.round((p - 1.96 * se) * 1000) / 10),
Math.min(100, Math.round((p + 1.96 * se) * 1000) / 10),
];
}
export async function getExperimentResults(
projectKey: string,
flagKey: string,
@@ -106,11 +132,92 @@ export async function getExperimentResults(
return { variant: r.variant, exposed, converted, convRate: exposed > 0 ? Math.round((converted / exposed) * 1000) / 10 : 0 };
});
const control = base.find((b) => /control|baseline/i.test(b.variant));
return base.map((b) => ({
...b,
liftVsControl:
control && control.convRate > 0 && b.variant !== control.variant
? Math.round(((b.convRate - control.convRate) / control.convRate) * 1000) / 10
: null,
}));
return base.map((b) => {
const [ciLow, ciHigh] = ci95(b.convRate, b.exposed);
let pValue: number | null = null;
if (control && b.variant !== control.variant && b.exposed > 0 && control.exposed > 0) {
const p1 = b.converted / b.exposed;
const p2 = control.converted / control.exposed;
const pPool = (b.converted + control.converted) / (b.exposed + control.exposed);
const se = Math.sqrt(pPool * (1 - pPool) * (1 / b.exposed + 1 / control.exposed));
if (se > 0) {
const z = (p1 - p2) / se;
pValue = Math.round(2 * (1 - normCdf(Math.abs(z))) * 1000) / 1000;
}
}
return {
...b,
liftVsControl:
control && control.convRate > 0 && b.variant !== control.variant
? Math.round(((b.convRate - control.convRate) / control.convRate) * 1000) / 10
: null,
ciLow,
ciHigh,
pValue,
significant: pValue !== null && pValue < 0.05,
};
});
}
// ── Active experiments + brief summary (for the evening brief) ──
export type ExperimentBrief = {
name: string;
flagKey: string;
leader: string | null;
leaderRate: number | null;
controlRate: number | null;
lift: number | null;
pValue: number | null;
significant: boolean;
totalExposed: number;
};
export async function getActiveExperiments(
projectKey: string,
): Promise<Array<{ name: string; flagKey: string }>> {
const rows = await prisma.$queryRaw<Array<{ name: string | null; data: unknown }>>`
WITH ranked AS (
SELECT name, data, row_number() OVER (PARTITION BY "resourceId" ORDER BY "capturedAt" DESC) AS rn
FROM posthog_resource_snapshots
WHERE "projectKey"=${projectKey} AND "resourceType"='experiment'
)
SELECT name, data FROM ranked WHERE rn = 1
`;
return rows
.map((r) => {
const d = (r.data ?? {}) as Record<string, unknown>;
const flagKey = typeof d.feature_flag_key === "string" ? d.feature_flag_key : null;
const archived = d.archived === true || d.deleted === true;
return flagKey && !archived ? { name: r.name ?? flagKey, flagKey } : null;
})
.filter((x): x is { name: string; flagKey: string } => !!x);
}
export async function getExperimentBriefSummary(
projectKey: string,
goal: GoalEvent = "user_signed_up",
): Promise<ExperimentBrief[]> {
const exps = await getActiveExperiments(projectKey);
const out: ExperimentBrief[] = [];
for (const e of exps) {
const variants = await getExperimentResults(projectKey, e.flagKey, goal);
const totalExposed = variants.reduce((a, v) => a + v.exposed, 0);
if (totalExposed === 0) continue;
const control = variants.find((v) => /control|baseline/i.test(v.variant));
const eligible = variants.filter((v) => v.exposed >= 30);
const leader = eligible.length ? eligible.reduce((a, b) => (b.convRate > a.convRate ? b : a)) : null;
out.push({
name: e.name,
flagKey: e.flagKey,
leader: leader?.variant ?? null,
leaderRate: leader?.convRate ?? null,
controlRate: control?.convRate ?? null,
lift: leader?.liftVsControl ?? null,
pValue: leader?.pValue ?? null,
significant: leader?.significant ?? false,
totalExposed,
});
}
return out;
}

5
apps/web/src/lib/sase/brief-data.ts
View File

@@ -7,6 +7,7 @@ import {
getInsightSummary,
getRecentDeployStatuses,
} from "./daily-overview";
import { getExperimentBriefSummary } from "@/lib/analytics/experiment-results";
// The Sase-DB half of the evening brief — everything that needs `saseDb`
// (the panel is the only process with that connection). The worker fetches this
@@ -15,7 +16,7 @@ export async function getSaseBriefMetrics() {
const since24 = new Date(Date.now() - 24 * 3600_000);
const since7 = new Date(Date.now() - 7 * 24 * 3600_000);
const [health, op, gaps, users, insights, deploys, rev24, rev7, pend7, byMethod7] =
const [health, op, gaps, users, insights, deploys, rev24, rev7, pend7, byMethod7, experiments] =
await Promise.all([
getSaseHealthSnapshot(),
getOperationalHealth("24h"),
@@ -27,6 +28,7 @@ export async function getSaseBriefMetrics() {
saseDb.payment.aggregate({ where: { status: "completed", createdAt: { gte: since7 } }, _sum: { amount: true }, _count: true }),
saseDb.payment.aggregate({ where: { status: "pending", createdAt: { gte: since7 } }, _sum: { amount: true }, _count: true }),
saseDb.payment.groupBy({ by: ["method"], where: { status: "completed", createdAt: { gte: since7 } }, _sum: { amount: true }, _count: true }),
getExperimentBriefSummary("sase", "user_signed_up"),
]);
// amount is stored in kuruş (TRY cents).
@@ -68,6 +70,7 @@ export async function getSaseBriefMetrics() {
deltaPp: d.successRateDeltaPp,
startedAt: d.startedAt.toISOString(),
})),
experiments,
};
}

6
apps/worker/src/jobs/sase-evening-brief.ts
View File

@@ -82,7 +82,7 @@ async function fetchPipelineStats(): Promise<unknown> {
}
// ─── LLM ──────────────────────────────────────────────────────────────────
const SYSTEM_PROMPT = `Sen Sase.tr (B2B oto yedek-parça VIN/katalog SaaS'ı, TR pazarı) için AKŞAM yönetici brief'i yazıyorsun. Sana 360° metrikler verilecek: VIN decode sağlığı, katalog kapsama açıkları, gelir (Stripe/iyzico ₺), kullanıcı + abonelik durumu, PostHog davranışı + kaynak/UTM, Meta reklam, ve insight pipeline.
const SYSTEM_PROMPT = `Sen Sase.tr (B2B oto yedek-parça VIN/katalog SaaS'ı, TR pazarı) için AKŞAM yönetici brief'i yazıyorsun. Sana 360° metrikler verilecek: VIN decode sağlığı, katalog kapsama açıkları, gelir (Stripe/iyzico ₺), kullanıcı + abonelik durumu, PostHog davranışı + kaynak/UTM, Meta reklam, A/B test (experiment) sonuçları, ve insight pipeline.
ÇOK DETAYLI ama KRİTİK NOKTALARI ÖN PLANDA tutan bir rapor üret. Founder bunu akşam okuyup günü kavrasın ve yarın ne yapacağını bilsin.
@@ -93,12 +93,12 @@ SADECE şu şemada geçerli JSON döndür (markdown/code-fence yok):
"health_score": 0-100 arası tamsayı (genel sağlık),
"critical": ["en kritik 2-5 madde, ön planda, en acil önce"],
"sections": [
{"key": "vin_decode|revenue|users|behavior|ads|product|pipeline", "title": "...", "status": "good|warn|bad", "bullets": ["somut, rakamlı maddeler"]}
{"key": "vin_decode|revenue|users|behavior|ads|experiments|product|pipeline", "title": "...", "status": "good|warn|bad", "bullets": ["somut, rakamlı maddeler"]}
],
"recommendations": ["yarın için önerilen 2-4 aksiyon"]
}
Kurallar: Tüm metin TÜRKÇE. Rakamları SADECE verilen metrikten al, uydurma. Veri eksikse (örn. Meta 'configured:false') bunu dürüstçe belirt, kritik yapma. Para birimi ₺. Kritik noktalar: gelir durgunluğu, decode başarı düşüşü, katalog açıkları (Renault gibi), empty-handed payer, dormant payer, P0/P1 insight, deploy regresyonu. status alanını dürüst seç.`;
Kurallar: Tüm metin TÜRKÇE. Rakamları SADECE verilen metrikten al, uydurma. Veri eksikse (örn. Meta 'configured:false') bunu dürüstçe belirt, kritik yapma. Para birimi ₺. Kritik noktalar: gelir durgunluğu, decode başarı düşüşü, katalog açıkları (Renault gibi), empty-handed payer, dormant payer, P0/P1 insight, deploy regresyonu. Experiment'ler (sase.experiments): istatistiksel ANLAMLI (significant=true) bir kazanan varsa "X experiment'inde <varyant> kazandı (+%Y lift, p=<p>)" diye öne çıkar ve experiments bölümüne koy; anlamlı değilse "henüz anlamlı değil (N exposure)" de — uydurma kazanan ilan etme. status alanını dürüst seç.`;
async function generateBrief(metrics: unknown): Promise<{ parsed: any; model: string; cost: number } | null> {
const today = new Date().toISOString().slice(0, 10);