feat(FN-318): create P0 CRO post-deploy funnel monitoring document
Some checks failed
Sync dev → Gitea / Mirror dev to Gitea (push) Has been cancelled
Some checks failed
Sync dev → Gitea / Mirror dev to Gitea (push) Has been cancelled
Commits merged: - docs(FN-318): create P0 CRO post-deploy funnel monitoring document Files changed: ...funnel-monitoring-p0-cro-post-deploy-2026-05.md | 631 +++++++++++++++++++++ 1 file changed, 631 insertions(+) Fusion-Task-Id: FN-318
This commit is contained in:
631
docs/product/funnel-monitoring-p0-cro-post-deploy-2026-05.md
Normal file
631
docs/product/funnel-monitoring-p0-cro-post-deploy-2026-05.md
Normal file
@@ -0,0 +1,631 @@
|
||||
# PostHog Funnel Monitoring — P0 Subscription CRO Post-Deploy
|
||||
|
||||
**Date:** 2026-05-13 (deployment day) — monitoring window: 48–72 hours
|
||||
**Trigger:** FN-298 deployed P0-1 through P0-10 subscription CRO fixes to production (sase.tr)
|
||||
**Status:** 🔴 MONITORING ACTIVE — Day 0 (partial, deployment afternoon)
|
||||
**Owner:** CPO / Product
|
||||
**Parallel task:** FN-309 (QA post-deploy visual verification — PASS, May 13)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
On 2026-05-13, 10 P0 subscription CRO fixes were deployed to production via FN-298 (re-deploy after initial stale-bundle incident). All fixes are visually verified PASS by FN-309 and FN-320 Playwright verification. This document tracks PostHog funnel metrics for 48–72 hours post-deploy to measure conversion impact.
|
||||
|
||||
**Fixes deployed:**
|
||||
- P0-1: Pricing card redesign with 17% yearly discount badge
|
||||
- P0-2: Feature matrix comparison on plan cards
|
||||
- P0-3: Trial urgency banner (≤3 days remaining)
|
||||
- P0-4: Order summary card after plan selection
|
||||
- P0-5: Cancel-save flow with downgrade offer dialog
|
||||
- P0-6: Sticky checkout bar on scroll
|
||||
- P0-7: Payment trust copy (256-bit SSL + Iyzico + KVKK)
|
||||
- P0-8: Core trust badges (no card, cancel anytime, 7-day refund)
|
||||
- P0-9: Social proof strip (landing + auth + subscription)
|
||||
- P0-10: Yearly billing toggle analytics
|
||||
|
||||
**All fixes are instrumented with PostHog events** — see Event Inventory below.
|
||||
|
||||
---
|
||||
|
||||
## 1. Event Inventory
|
||||
|
||||
### 1.1 Funnel Events (Instrumented)
|
||||
|
||||
All events verified present in production code as of FN-298 deploy:
|
||||
|
||||
| Event | Properties | Location | Added By |
|
||||
|-------|-----------|----------|----------|
|
||||
| `plan_selected` | `plan` (key) | `subscription/index.tsx:327` | Pre-existing |
|
||||
| `checkout_started` | `plan`, `period` | `subscription/index.tsx:345` | Pre-existing |
|
||||
| `payment_initiated` | `method`, `plan`, `period`, `amount` | `payment-content.tsx:196,202` | Pre-existing |
|
||||
| `payment_success` | `method`, `plan`, `period`, `amount`¹ | `payment-content.tsx:73,135` | FN-281 |
|
||||
| `payment_failed` | `method`, `plan`, `period`, `reason`¹ | `payment-content.tsx:76` | FN-281 |
|
||||
| `receipt_uploaded` | `payment_id` | `payment-content.tsx:209` | Pre-existing |
|
||||
|
||||
¹ Backend also captures `payment_success`/`payment_failed` in `payments.service.ts` with `method`, `payment_id`, `subscription_id`, `amount` (but not `plan`/`period` — property gap tracked in FN-321).
|
||||
|
||||
### 1.2 P0 CRO Fix Events (Newly Instrumented)
|
||||
|
||||
| Event | Properties | Location | P0 Fix |
|
||||
|-------|-----------|----------|--------|
|
||||
| `yearly_toggle_clicked` | `period` ("yearly"\|"monthly") | `subscription/index.tsx:1055,1066` | P0-1/P0-10 |
|
||||
| `trial_urgency_banner_viewed` | `trial_end_date`, `days_remaining` | `trial-urgency-banner.tsx:77` | P0-3 |
|
||||
| `trial_urgency_banner_cta_clicked` | `trial_end_date`, `days_remaining` | `trial-urgency-banner.tsx:97` | P0-3 |
|
||||
| `trial_urgency_banner_dismissed` | `trial_end_date`, `days_remaining` | `trial-urgency-banner.tsx:87` | P0-3 |
|
||||
| `downgrade_offer_shown` | `from_plan`, `to_plan` | `subscription/index.tsx:365` | P0-5 |
|
||||
| `downgrade_offer_accepted` | `from_plan`, `to_plan` | `subscription/index.tsx:265` | P0-5 |
|
||||
| `downgrade_offer_declined` | `from_plan` | `subscription/index.tsx:843` | P0-5 |
|
||||
| `cancel_flow_viewed` | (none) | `subscription/index.tsx:892` | P0-5 |
|
||||
| `cancel_save_clicked` | (none) | `subscription/index.tsx:269` | P0-5 |
|
||||
| `social_proof_impression` | `page`, `section` | `subscription/index.tsx:457,474`; `index.tsx:573,590,607`; `_auth.tsx:23` | P0-9 (FN-282, FN-297) |
|
||||
| `social_proof_engaged` | `page`, `section` | `subscription/index.tsx:484,488`; `index.tsx:616`; `_auth.tsx:33` | P0-9 (FN-282, FN-297) |
|
||||
|
||||
### 1.3 Pageview Proxy for `subscription_page_viewed`
|
||||
|
||||
No explicit `subscription_page_viewed` event exists. Instead, we filter `$pageview` events where `$current_url` contains `/dashboard/subscription`. The root route (`__root.tsx:95`) captures `$pageview` on every navigation with `$current_url` set to `window.location.origin + location.pathname`.
|
||||
|
||||
**PostHog filter:** `$pageview` with `$current_url` containing `/dashboard/subscription` (but NOT `/dashboard/subscription/pay`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Metrics & Success Criteria
|
||||
|
||||
### 2.1 Primary Metric: Checkout Conversion
|
||||
|
||||
**Funnel:** `checkout_started → payment_initiated`
|
||||
**Baseline period:** May 6–12, 2026 (7 days pre-deploy)
|
||||
**Post period:** May 13–15/16, 2026 (48–72 hours post-deploy)
|
||||
|
||||
| Criterion | Threshold |
|
||||
|-----------|-----------|
|
||||
| ✅ Success | ≥3pp improvement in conversion rate |
|
||||
| ⚠️ Acceptable | ±0–3pp (no regression) |
|
||||
| 🔴 Kill | >5pp DROP from baseline → escalate to CEO as P0 |
|
||||
|
||||
### 2.2 Secondary Metric: Subscription Page Abandonment
|
||||
|
||||
**Funnel proxy:** `$pageview` (filtered to `/dashboard/subscription`) → `checkout_started`
|
||||
**What it measures:** Did trust elements (social proof, trust badges, payment trust copy) reduce abandonment between viewing the subscription page and initiating checkout?
|
||||
|
||||
| Criterion | Threshold |
|
||||
|-----------|-----------|
|
||||
| ✅ Success | No regression (at or above baseline) |
|
||||
| 🔴 Concern | Any decrease >5pp warrants investigation |
|
||||
|
||||
### 2.3 Cohort Analysis: New vs Returning Visitors
|
||||
|
||||
**Hypothesis:** Trial urgency banner (`trial_urgency_banner_viewed`) should increase `checkout_started` rate among users with active trials (returning visitors) without deterring new visitors.
|
||||
|
||||
**PostHog breakdown:** Split `checkout_started → payment_initiated` funnel by:
|
||||
- New visitors (first session within window)
|
||||
- Returning visitors (existing user with prior session)
|
||||
|
||||
### 2.4 Tertiary Metric: Downgrade Offer Flow
|
||||
|
||||
**Funnel:** `downgrade_offer_shown → downgrade_offer_accepted`
|
||||
**What it measures:** Cancel-save flow effectiveness — of users who see the downgrade offer, what percentage accept it vs cancel outright?
|
||||
|
||||
**Note:** Base for this funnel may be very small (only fires when user with multi-plan subscription initiates cancel). Treat as directional, not statistically significant.
|
||||
|
||||
### 2.5 Full End-to-End Funnel (New)
|
||||
|
||||
**Funnel:** `checkout_started → payment_initiated → payment_success`
|
||||
**Enabled by:** FN-281 (`d855b56`) — `payment_success` now live
|
||||
**Target:** `payment_success / payment_initiated` ≥ 70% (per CPO brief, `docs/product/cpo-brief-payment-events.md`)
|
||||
|
||||
---
|
||||
|
||||
## 3. Pre-Deploy Baseline (May 6–12, 2026)
|
||||
|
||||
### 3.1 PostHog Query Configuration
|
||||
|
||||
To reproduce these numbers in the PostHog dashboard:
|
||||
|
||||
```
|
||||
Product analytics → Funnels → New funnel
|
||||
|
||||
Funnel name: P0 CRO — Checkout Baseline (Pre)
|
||||
Steps:
|
||||
1. checkout_started
|
||||
2. payment_initiated
|
||||
|
||||
Date range: May 6, 2026 → May 12, 2026 (inclusive)
|
||||
Conversion window: 1 hour
|
||||
|
||||
Breakdown: plan (property)
|
||||
```
|
||||
|
||||
```
|
||||
Funnel name: P0 CRO — Full Funnel Baseline (Pre)
|
||||
Steps:
|
||||
1. checkout_started
|
||||
2. payment_initiated
|
||||
3. payment_success
|
||||
|
||||
Date range: May 6, 2026 → May 12, 2026
|
||||
Conversion window: 24 hours
|
||||
```
|
||||
|
||||
```
|
||||
Funnel name: P0 CRO — Subscription → Checkout Baseline (Pre)
|
||||
Steps:
|
||||
1. $pageview (filter: $current_url contains /dashboard/subscription, excludes /pay)
|
||||
2. checkout_started
|
||||
|
||||
Date range: May 6, 2026 → May 12, 2026
|
||||
Conversion window: 1 hour
|
||||
```
|
||||
|
||||
```
|
||||
Funnel name: P0 CRO — Downgrade Offer Baseline (Pre)
|
||||
Steps:
|
||||
1. downgrade_offer_shown
|
||||
2. downgrade_offer_accepted
|
||||
|
||||
Date range: May 6, 2026 → May 12, 2026
|
||||
Conversion window: 1 hour
|
||||
```
|
||||
|
||||
### 3.2 Baseline Data (COLLECT MANUALLY FROM POSTHOG)
|
||||
|
||||
> ⚠️ **Action required:** Log into PostHog (eu.posthog.com, project 127747) and run the queries above. Fill in the tables below.
|
||||
|
||||
**Table 3A: Primary Funnel — checkout_started → payment_initiated**
|
||||
|
||||
| Date | checkout_started | payment_initiated | Conversion Rate |
|
||||
|------|-----------------|-------------------|----------------|
|
||||
| May 6 | TBD | TBD | TBD% |
|
||||
| May 7 | TBD | TBD | TBD% |
|
||||
| May 8 | TBD | TBD | TBD% |
|
||||
| May 9 | TBD | TBD | TBD% |
|
||||
| May 10 | TBD | TBD | TBD% |
|
||||
| May 11 | TBD | TBD | TBD% |
|
||||
| May 12 | TBD | TBD | TBD% |
|
||||
| **7-day Total** | **TBD** | **TBD** | **TBD%** ← BASELINE |
|
||||
|
||||
**Table 3B: Secondary Funnel — subscription_page → checkout_started**
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| subscription_page_viewed ($pageview proxy) | TBD |
|
||||
| checkout_started | TBD |
|
||||
| Conversion Rate | TBD% ← BASELINE |
|
||||
|
||||
**Table 3C: Full End-to-End Funnel — checkout_started → payment_initiated → payment_success**
|
||||
|
||||
| Step | Count | Drop-off | Conversion (cumulative) |
|
||||
|------|-------|----------|------------------------|
|
||||
| checkout_started | TBD | — | 100% |
|
||||
| payment_initiated | TBD | TBD% | TBD% |
|
||||
| payment_success | TBD | TBD% | TBD% |
|
||||
|
||||
**Table 3D: Downgrade Offer Flow**
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| downgrade_offer_shown | TBD |
|
||||
| downgrade_offer_accepted | TBD |
|
||||
| Save Rate | TBD% |
|
||||
|
||||
### 3.3 Baseline Plan Tier Breakdown
|
||||
|
||||
| Plan | checkout_started | payment_initiated | Conversion Rate |
|
||||
|------|-----------------|-------------------|----------------|
|
||||
| brand1 | TBD | TBD | TBD% |
|
||||
| brand2 | TBD | TBD | TBD% |
|
||||
| brand3 | TBD | TBD | TBD% |
|
||||
| brand5 | TBD | TBD | TBD% |
|
||||
| full | TBD | TBD | TBD% |
|
||||
|
||||
---
|
||||
|
||||
## 4. Post-Deploy Daily Snapshots
|
||||
|
||||
### Day 1 — May 13, 2026 (Deployment Day, Partial)
|
||||
|
||||
**Deploy time:** ~midday (exact timestamp TBD from deployment logs)
|
||||
**Data coverage:** Partial day (~12 hours post-deploy)
|
||||
**Caveats:** Deployment-day data includes pre-deploy morning traffic. Numbers are directional only.
|
||||
|
||||
**Table 4A: Primary Funnel (May 13)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| checkout_started | TBD | — |
|
||||
| payment_initiated | TBD | — |
|
||||
| Conversion Rate | TBD% | TBDpp |
|
||||
|
||||
**Table 4B: Secondary Funnel (May 13)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| subscription_page → checkout_started rate | TBD% | TBDpp |
|
||||
|
||||
**Table 4C: Full Funnel (May 13)**
|
||||
|
||||
| Step | Count | Rate |
|
||||
|------|-------|------|
|
||||
| checkout_started | TBD | 100% |
|
||||
| payment_initiated | TBD | TBD% |
|
||||
| payment_success | TBD | TBD% |
|
||||
|
||||
**Table 4D: Trial Urgency Banner (May 13)**
|
||||
|
||||
| Event | Count |
|
||||
|-------|-------|
|
||||
| trial_urgency_banner_viewed | TBD |
|
||||
| trial_urgency_banner_cta_clicked | TBD |
|
||||
| trial_urgency_banner_dismissed | TBD |
|
||||
| CTA Click Rate (cta/viewed) | TBD% |
|
||||
|
||||
**Table 4E: Yearly Toggle (May 13)**
|
||||
|
||||
| Event | Count |
|
||||
|-------|-------|
|
||||
| yearly_toggle_clicked (→ yearly) | TBD |
|
||||
| yearly_toggle_clicked (→ monthly) | TBD |
|
||||
|
||||
**Table 4F: Kill Criteria Check (May 13)**
|
||||
|
||||
| Metric | Baseline | May 13 | Δ | Threshold | Status |
|
||||
|--------|----------|--------|---|-----------|--------|
|
||||
| checkout → payment_initiated | TBD% | TBD% | TBDpp | > -5pp | ⏳ PENDING |
|
||||
|
||||
---
|
||||
|
||||
### Day 2 — May 14, 2026 (First Full Day Post-Deploy)
|
||||
|
||||
**Data coverage:** Full 24 hours
|
||||
**Caveats:** First full day of data. Weekend patterns may differ (May 14 is a Wednesday — weekday, normal traffic).
|
||||
|
||||
**Table 5A: Primary Funnel (May 14)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| checkout_started | TBD | — |
|
||||
| payment_initiated | TBD | — |
|
||||
| Conversion Rate | TBD% | TBDpp |
|
||||
|
||||
**Table 5B: Secondary Funnel (May 14)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| subscription_page → checkout_started rate | TBD% | TBDpp |
|
||||
|
||||
**Table 5C: Full Funnel (May 14)**
|
||||
|
||||
| Step | Count | Rate |
|
||||
|------|-------|------|
|
||||
| checkout_started | TBD | 100% |
|
||||
| payment_initiated | TBD | TBD% |
|
||||
| payment_success | TBD | TBD% |
|
||||
|
||||
**Table 5D: Kill Criteria Check (May 14)**
|
||||
|
||||
| Metric | Baseline | May 14 | Δ | Threshold | Status |
|
||||
|--------|----------|--------|---|-----------|--------|
|
||||
| checkout → payment_initiated | TBD% | TBD% | TBDpp | > -5pp | ⏳ PENDING |
|
||||
|
||||
---
|
||||
|
||||
### Day 3 — May 15, 2026 (48-Hour Mark)
|
||||
|
||||
**Data coverage:** Full 24 hours
|
||||
**Significance:** 48-hour kill criterion window closes. If no regression >5pp is detected, P0 escalation threshold is cleared.
|
||||
|
||||
**Table 6A: Primary Funnel (May 15)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| checkout_started | TBD | — |
|
||||
| payment_initiated | TBD | — |
|
||||
| Conversion Rate | TBD% | TBDpp |
|
||||
|
||||
**Table 6B: Secondary Funnel (May 15)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| subscription_page → checkout_started rate | TBD% | TBDpp |
|
||||
|
||||
**Table 6C: Full Funnel (May 15)**
|
||||
|
||||
| Step | Count | Rate |
|
||||
|------|-------|------|
|
||||
| checkout_started | TBD | 100% |
|
||||
| payment_initiated | TBD | TBD% |
|
||||
| payment_success | TBD | TBD% |
|
||||
|
||||
**Table 6D: Downgrade Offer Flow (May 13–15 Cumulative)**
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| downgrade_offer_shown | TBD |
|
||||
| downgrade_offer_accepted | TBD |
|
||||
| downgrade_offer_declined | TBD |
|
||||
| Save Rate (accepted / shown) | TBD% |
|
||||
|
||||
**Table 6E: Kill Criteria Check (May 15)**
|
||||
|
||||
| Metric | Baseline | May 13-15 Aggregate | Δ | Threshold | Status |
|
||||
|--------|----------|---------------------|---|-----------|--------|
|
||||
| checkout → payment_initiated | TBD% | TBD% | TBDpp | > -5pp | ⏳ PENDING |
|
||||
|
||||
---
|
||||
|
||||
### Day 4 (Optional) — May 16, 2026 (72-Hour Mark)
|
||||
|
||||
**Data coverage:** Full 24 hours
|
||||
**Significance:** Extended monitoring window. Provides 3 full days of post-deploy data. Better statistical power.
|
||||
|
||||
**Table 7A: Primary Funnel (May 16)**
|
||||
|
||||
| Metric | Value | vs Baseline (TBD%) |
|
||||
|--------|-------|---------------------|
|
||||
| checkout_started | TBD | — |
|
||||
| payment_initiated | TBD | — |
|
||||
| Conversion Rate | TBD% | TBDpp |
|
||||
|
||||
**Table 7B: 4-Day Aggregate (May 13–16)**
|
||||
|
||||
| Metric | Pre (May 6-12) | Post (May 13-16) | Δ |
|
||||
|--------|----------------|------------------|---|
|
||||
| checkout_started | TBD | TBD | TBD |
|
||||
| payment_initiated | TBD | TBD | TBD |
|
||||
| Conversion Rate | TBD% | TBD% | TBDpp |
|
||||
|
||||
---
|
||||
|
||||
## 5. Cohort Analysis (To Be Populated)
|
||||
|
||||
### 5.1 New vs Returning Visitors
|
||||
|
||||
PostHog query configuration:
|
||||
|
||||
```
|
||||
Product analytics → Funnels → New funnel
|
||||
|
||||
Funnel name: P0 CRO — Cohort: New Visitors
|
||||
Steps:
|
||||
1. checkout_started
|
||||
2. payment_initiated
|
||||
|
||||
Filter: Person properties → is_returning_user = false
|
||||
OR: First touch point within date range
|
||||
Date range: May 13 → May 16, 2026
|
||||
Conversion window: 1 hour
|
||||
```
|
||||
|
||||
```
|
||||
Funnel name: P0 CRO — Cohort: Returning Visitors
|
||||
Steps:
|
||||
1. checkout_started
|
||||
2. payment_initiated
|
||||
|
||||
Filter: Person properties → is_returning_user = true
|
||||
OR: Has session events before May 13, 2026
|
||||
Date range: May 13 → May 16, 2026
|
||||
Conversion window: 1 hour
|
||||
```
|
||||
|
||||
### 5.2 Cohort Data
|
||||
|
||||
| Cohort | checkout_started | payment_initiated | Conversion Rate | vs Baseline |
|
||||
|--------|-----------------|-------------------|----------------|-------------|
|
||||
| New Visitors | TBD | TBD | TBD% | TBDpp |
|
||||
| Returning Visitors | TBD | TBD | TBD% | TBDpp |
|
||||
|
||||
### 5.3 Cohort Interpretation Guide
|
||||
|
||||
- **Trial urgency banner hypothesis:** If returning visitors' `checkout_started` volume increases relative to new visitors, the urgency banner is likely the driver (returning users are more likely to be on trials).
|
||||
- **Trust elements hypothesis:** If new visitors' `subscription_page → checkout_started` improves, social proof and trust badges are reducing first-time abandonment.
|
||||
- **Yearly toggle hypothesis:** If `yearly_toggle_clicked (→ yearly)` volume is significant and `checkout_started` for yearly plans increases, the discount badge + toggle UX are converting monthly interest to annual commitment.
|
||||
|
||||
---
|
||||
|
||||
## 6. Kill Criteria Threshold Monitoring
|
||||
|
||||
### 6.1 Kill Criterion Definition
|
||||
|
||||
> If `payment_initiated / checkout_started` drops >5pp from 7-day rolling baseline within 48 hours, escalate to CEO as P0 — potential UX regression from the CRO changes.
|
||||
|
||||
### 6.2 Threshold Tracker
|
||||
|
||||
| Check Time | Baseline Rate | Current Rate | Δ | >5pp Drop? | Action |
|
||||
|------------|--------------|-------------|---|-----------|--------|
|
||||
| May 13 EOD | TBD% | TBD% | TBDpp | TBD | TBD |
|
||||
| May 14 EOD (24h) | TBD% | TBD% | TBDpp | TBD | TBD |
|
||||
| May 15 EOD (48h) | TBD% | TBD% | TBDpp | TBD | TBD |
|
||||
| May 16 EOD (72h) | TBD% | TBD% | TBDpp | TBD | TBD |
|
||||
|
||||
**Escalation protocol:**
|
||||
1. Verify the data is not a PostHog instrumentation issue (check SDK status, event volume, no dropped events)
|
||||
2. If confirmed regression >5pp: immediate CEO escalation with this document + PostHog funnel screenshot
|
||||
3. Concurrently: open rollback consideration — which fix(es) to revert first
|
||||
4. Within 4 hours of escalation: decision on rollback vs. hotfix vs. monitor longer
|
||||
|
||||
### 6.3 Regression Root-Cause Checklist
|
||||
|
||||
If conversion drops, investigate in order:
|
||||
|
||||
- [ ] **Payment gateway:** Iyzico uptime, callback success rate, `payment_failed` spike?
|
||||
- [ ] **UI bug:** Is a CTA broken/clipped/hidden? Check Playwright screenshot diff from FN-320.
|
||||
- [ ] **Sticky bar:** Is the sticky checkout bar overlapping content on mobile, hiding a different element?
|
||||
- [ ] **Trial banner:** Is the urgency banner alarming users or making the page feel broken?
|
||||
- [ ] **Pricing confusion:** Did the yearly discount badge make monthly plans look "wrong" or "not a deal"?
|
||||
- [ ] **Traffic shift:** Did a marketing campaign drive low-intent traffic that skews the funnel? Check `$referrer`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Final Verdict
|
||||
|
||||
> ⏳ **PENDING** — Monitoring window active. Verdict will be populated after 48–72 hours of data collection.
|
||||
|
||||
### 7.1 Verdict Framework
|
||||
|
||||
| Verdict | Criteria |
|
||||
|---------|----------|
|
||||
| ✅ **CLEAR WIN** | `payment_initiated / checkout_started` improved ≥3pp; no regression in secondary metrics; kill criterion not triggered |
|
||||
| ✅ **NEUTRAL** | Primary metric within ±3pp of baseline; no regression in secondary metrics |
|
||||
| ⚠️ **MINOR REGRESSION** | Primary metric dropped 1–5pp; investigation launched; no immediate rollback needed |
|
||||
| 🔴 **P0 ESCALATION** | Primary metric dropped >5pp within 48h; CEO notified; rollback evaluation initiated |
|
||||
| ⚠️ **INCONCLUSIVE** | Sample size too small for statistical significance; extend monitoring to 7 days |
|
||||
|
||||
### 7.2 Verdict by Fix (Directional)
|
||||
|
||||
| Fix | Metric | Pre Rate | Post Rate | Δ | Assessment |
|
||||
|-----|--------|----------|-----------|---|------------|
|
||||
| P0-1/P0-10 | yearly_toggle_clicked volume + yearly checkout_started | TBD | TBD | TBD | ⏳ |
|
||||
| P0-2 | Feature matrix impact (indirect — via checkout_started) | TBD | TBD | TBD | ⏳ |
|
||||
| P0-3 | Trial urgency banner: cta_clicked/viewed rate | TBD% | TBD% | TBDpp | ⏳ |
|
||||
| P0-4 | Order summary impact (indirect — via checkout_started) | TBD | TBD | TBD | ⏳ |
|
||||
| P0-5 | Downgrade offer save rate (accepted/shown) | TBD% | TBD% | TBDpp | ⏳ |
|
||||
| P0-6 | Sticky bar impact (indirect — aggregate checkout_started volume) | TBD | TBD | TBD | ⏳ |
|
||||
| P0-7/P0-8 | Trust copy impact (indirect — subscription_page → checkout_started) | TBD% | TBD% | TBDpp | ⏳ |
|
||||
| P0-9 | Social proof impression/engagement volume | TBD | TBD | TBD | ⏳ |
|
||||
|
||||
### 7.3 Recommendation
|
||||
|
||||
> ⏳ **PENDING** — To be written after monitoring window closes.
|
||||
|
||||
---
|
||||
|
||||
## 8. Data Collection Checklist
|
||||
|
||||
### Manual PostHog Steps (Require Dashboard Access)
|
||||
|
||||
1. [ ] Log in to PostHog at https://eu.posthog.com (project 127747)
|
||||
2. [ ] Navigate to **Product analytics → Funnels**
|
||||
3. [ ] Create funnel: **P0 CRO — checkout_started → payment_initiated** (Section 3.1 queries)
|
||||
4. [ ] Set date range to May 6 → May 12, 2026 — record baseline in Table 3A
|
||||
5. [ ] Set date range to May 13, 2026 — record Day 1 data in Table 4A
|
||||
6. [ ] Repeat for each metric in Sections 3–6 daily through May 16
|
||||
7. [ ] Create cohort funnels for new vs returning visitors (Section 5)
|
||||
8. [ ] Verify `payment_success` event volume is non-zero (confirms FN-281 is live)
|
||||
9. [ ] Fill in Final Verdict section (Section 7) after 72 hours
|
||||
10. [ ] Archive this document with "FINAL" status when complete
|
||||
|
||||
### Automated Collection (Future Enhancement)
|
||||
|
||||
See `scripts/posthog-dashboards.sh` for the existing dashboard creation script. A dedicated funnel monitoring script with API-based data extraction and table population can be created as a follow-up task.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: PostHog API Query Examples
|
||||
|
||||
If `POSTHOG_API_KEY` is available in the environment, the following queries can be run via the PostHog API:
|
||||
|
||||
```bash
|
||||
# Base configuration
|
||||
BASE="https://eu.posthog.com/api/projects/127747"
|
||||
AUTH="Authorization: Bearer $POSTHOG_API_KEY"
|
||||
|
||||
# Query checkout_started → payment_initiated funnel
|
||||
curl -s -X POST "$BASE/insights/" \
|
||||
-H "$AUTH" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "P0 CRO — Checkout Funnel (Post-Deploy)",
|
||||
"query": {
|
||||
"kind": "InsightVizNode",
|
||||
"source": {
|
||||
"kind": "FunnelsQuery",
|
||||
"series": [
|
||||
{"kind":"EventsNode","event":"checkout_started","name":"Checkout"},
|
||||
{"kind":"EventsNode","event":"payment_initiated","name":"Payment"}
|
||||
],
|
||||
"dateRange": {"date_from": "2026-05-13"},
|
||||
"funnelsFilter": {"funnelWindowInterval": 1, "funnelWindowIntervalUnit": "hour"}
|
||||
}
|
||||
}
|
||||
}' | jq '.result'
|
||||
```
|
||||
|
||||
```bash
|
||||
# Query full end-to-end funnel
|
||||
curl -s -X POST "$BASE/insights/" \
|
||||
-H "$AUTH" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "P0 CRO — Full End-to-End Funnel",
|
||||
"query": {
|
||||
"kind": "InsightVizNode",
|
||||
"source": {
|
||||
"kind": "FunnelsQuery",
|
||||
"series": [
|
||||
{"kind":"EventsNode","event":"checkout_started","name":"Checkout"},
|
||||
{"kind":"EventsNode","event":"payment_initiated","name":"Initiated"},
|
||||
{"kind":"EventsNode","event":"payment_success","name":"Success"}
|
||||
],
|
||||
"dateRange": {"date_from": "2026-05-13"},
|
||||
"funnelsFilter": {"funnelWindowInterval": 24, "funnelWindowIntervalUnit": "hour"}
|
||||
}
|
||||
}
|
||||
}' | jq '.result'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: Code References
|
||||
|
||||
### B.1 PostHog Client Configuration
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `apps/web/src/lib/posthog.ts` | Frontend PostHog client (lazy-loaded `posthog-js`) |
|
||||
| `apps/api/src/posthog/posthog.service.ts` | Backend PostHog client (`posthog-node`) |
|
||||
| `apps/api/src/posthog/posthog.module.ts` | NestJS module registration |
|
||||
|
||||
### B.2 Event Source Locations
|
||||
|
||||
| Event | File | Line |
|
||||
|-------|------|------|
|
||||
| `plan_selected` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 327 |
|
||||
| `checkout_started` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 345 |
|
||||
| `payment_initiated` (iyzico) | `apps/web/src/components/payment/payment-content.tsx` | 196 |
|
||||
| `payment_initiated` (eft) | `apps/web/src/components/payment/payment-content.tsx` | 202 |
|
||||
| `payment_success` (iyzico redirect) | `apps/web/src/components/payment/payment-content.tsx` | 73 |
|
||||
| `payment_failed` (iyzico redirect) | `apps/web/src/components/payment/payment-content.tsx` | 76 |
|
||||
| `payment_success` (eft upload) | `apps/web/src/components/payment/payment-content.tsx` | 135 |
|
||||
| `receipt_uploaded` | `apps/web/src/components/payment/payment-content.tsx` | 209 |
|
||||
| `downgrade_offer_shown` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 365 |
|
||||
| `downgrade_offer_accepted` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 265 |
|
||||
| `downgrade_offer_declined` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 843 |
|
||||
| `cancel_flow_viewed` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 892 |
|
||||
| `cancel_save_clicked` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 269 |
|
||||
| `trial_urgency_banner_viewed` | `apps/web/src/components/trial-urgency-banner.tsx` | 77 |
|
||||
| `trial_urgency_banner_cta_clicked` | `apps/web/src/components/trial-urgency-banner.tsx` | 97 |
|
||||
| `trial_urgency_banner_dismissed` | `apps/web/src/components/trial-urgency-banner.tsx` | 87 |
|
||||
| `yearly_toggle_clicked` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 1055, 1066 |
|
||||
| `social_proof_impression` (subscription) | `apps/web/src/routes/dashboard/subscription/index.tsx` | 457, 474 |
|
||||
| `social_proof_engaged` (subscription) | `apps/web/src/routes/dashboard/subscription/index.tsx` | 484, 488 |
|
||||
| `social_proof_impression` (landing) | `apps/web/src/routes/index.tsx` | 572, 589, 606 |
|
||||
| `social_proof_impression` (auth) | `apps/web/src/routes/_auth.tsx` | 23 |
|
||||
| `social_proof_engaged` (landing) | `apps/web/src/routes/index.tsx` | 616 |
|
||||
| `social_proof_engaged` (auth) | `apps/web/src/routes/_auth.tsx` | 33 |
|
||||
| `subscription_cancelled` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 913 |
|
||||
| `subscription_resumed` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 753 |
|
||||
| `trial_started` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 952 |
|
||||
| `$pageview` (all routes) | `apps/web/src/routes/__root.tsx` | 95 |
|
||||
|
||||
---
|
||||
|
||||
## Appendix C: Related Tasks & Documents
|
||||
|
||||
| ID | Description | Status |
|
||||
|----|-------------|--------|
|
||||
| FN-298 | P0 subscription CRO fixes deployment | ✅ Deployed May 13 |
|
||||
| FN-281 | `payment_success` / `payment_failed` instrumentation | ✅ Complete (`d855b56`) |
|
||||
| FN-309 | QA post-deploy verification | 🔄 In parallel |
|
||||
| FN-320 | Post-deploy QA — all 10 P0 fixes verified PASS | ✅ Complete |
|
||||
| FN-321 | PostHog payment events verification | ✅ Complete |
|
||||
| FN-275 | P1 CRO funnel audit (pre-instrumentation) | ✅ Complete |
|
||||
| `docs/product/funnel-audit-p1-cro-2026-05.md` | P1 CRO audit document | Reference |
|
||||
| `docs/product/cpo-brief-payment-events.md` | Payment events CPO brief | Reference |
|
||||
| `scripts/posthog-dashboards.sh` | PostHog dashboard creation script | Reference |
|
||||
|
||||
---
|
||||
|
||||
> **Document status:** 🔴 MONITORING ACTIVE — data collection pending
|
||||
> **Next update due:** May 14, 2026 (24h post-deploy)
|
||||
> **Final verdict due:** May 16, 2026 (72h post-deploy)
|
||||
Reference in New Issue
Block a user