fix(parts): dedupe ingest path + clean ~108k duplicate rows
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Backfill + reactive prefetch can re-drill the same category multiple times, and the parts insert path had no dedupe guard. Result: 8.2% duplicate rows on pl24, 14.5% on parts-catalogs, and 33.7% on emex — ~108k extra rows across 7,571 categories on 224 vehicles. Every drilled catalog page rendered each part twice (the Tampon example: 32 rows for 19 distinct OEMs). * Migration 0010 — phase 1 deletes existing dupes preserving the oldest row per (vehicle_id, category_id, oem_code, name, position) group; phase 2 adds a UNIQUE INDEX over the same tuple with NULLS NOT DISTINCT (PG 15+) so null position/vehicle_id collapse like equal values rather than each counting as its own "distinct" row. Idempotent CREATE UNIQUE INDEX IF NOT EXISTS so the runner is safe to re-apply. * All five insert(parts).values(...).returning() call sites (parts.service, categories.service ×3, catalog.service) get .onConflictDoNothing() so future re-drills no-op instead of erroring on the new constraint. `.returning()` continues to surface only the newly-inserted rows; existing logs read `Stored N parts` as actual net insertions, which is what we want. Dry-run on dev DB: 524,540 → 425,186 parts (99,354 dupes deleted), index created cleanly. Same delta expected on prod (~108k drop). drizzle-orm 0.41 doesn't expose .nullsNotDistinct() on the index builder so the constraint is owned by raw SQL — see the inline comment in the parts schema and the migration file. Future schema generators should NOT try to drop or rewrite this index.
This commit is contained in:
21
apps/api/drizzle/0010_dedupe_parts.sql
Normal file
21
apps/api/drizzle/0010_dedupe_parts.sql
Normal file
@@ -0,0 +1,21 @@
|
||||
-- Phase 1: remove existing duplicate rows.
|
||||
-- "Duplicate" = same (vehicle_id, category_id, oem_code, name, position).
|
||||
-- Backfill + reactive prefetch can re-drill the same category multiple times;
|
||||
-- without a unique constraint that re-insert kept all rows, leaving 8–34%
|
||||
-- duplicate per source (~108k extra rows of ~588k). Keep the oldest row per
|
||||
-- group (preserves original created_at) and drop the rest.
|
||||
DELETE FROM "parts" WHERE id IN (
|
||||
SELECT id FROM (
|
||||
SELECT id, row_number() OVER (
|
||||
PARTITION BY vehicle_id, category_id, oem_code, name, position
|
||||
ORDER BY created_at ASC, id ASC
|
||||
) AS rn FROM "parts"
|
||||
) t WHERE rn > 1
|
||||
);
|
||||
--> statement-breakpoint
|
||||
|
||||
-- Phase 2: prevent future dupes. NULLS NOT DISTINCT (Postgres 15+) so NULL
|
||||
-- position / NULL vehicle_id collapse like equal values rather than each
|
||||
-- counting as a separate "distinct" row.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS "parts_dedup_idx" ON "parts"
|
||||
(vehicle_id, category_id, oem_code, name, position) NULLS NOT DISTINCT;
|
||||
5564
apps/api/drizzle/meta/0010_snapshot.json
Normal file
5564
apps/api/drizzle/meta/0010_snapshot.json
Normal file
File diff suppressed because it is too large
Load Diff
@@ -71,6 +71,13 @@
|
||||
"when": 1779703007129,
|
||||
"tag": "0009_referral_rewards_rework",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 10,
|
||||
"version": "7",
|
||||
"when": 1780281600000,
|
||||
"tag": "0010_dedupe_parts",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user