Semih Yesilyurt
10dcf2d6ae
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Backfill + reactive prefetch can re-drill the same category multiple times, and the parts insert path had no dedupe guard. Result: 8.2% duplicate rows on pl24, 14.5% on parts-catalogs, and 33.7% on emex — ~108k extra rows across 7,571 categories on 224 vehicles. Every drilled catalog page rendered each part twice (the Tampon example: 32 rows for 19 distinct OEMs). * Migration 0010 — phase 1 deletes existing dupes preserving the oldest row per (vehicle_id, category_id, oem_code, name, position) group; phase 2 adds a UNIQUE INDEX over the same tuple with NULLS NOT DISTINCT (PG 15+) so null position/vehicle_id collapse like equal values rather than each counting as its own "distinct" row. Idempotent CREATE UNIQUE INDEX IF NOT EXISTS so the runner is safe to re-apply. * All five insert(parts).values(...).returning() call sites (parts.service, categories.service ×3, catalog.service) get .onConflictDoNothing() so future re-drills no-op instead of erroring on the new constraint. `.returning()` continues to surface only the newly-inserted rows; existing logs read `Stored N parts` as actual net insertions, which is what we want. Dry-run on dev DB: 524,540 → 425,186 parts (99,354 dupes deleted), index created cleanly. Same delta expected on prod (~108k drop). drizzle-orm 0.41 doesn't expose .nullsNotDistinct() on the index builder so the constraint is owned by raw SQL — see the inline comment in the parts schema and the migration file. Future schema generators should NOT try to drop or rewrite this index.
22 lines
1009 B
SQL
22 lines
1009 B
SQL
-- Phase 1: remove existing duplicate rows.
|
||
-- "Duplicate" = same (vehicle_id, category_id, oem_code, name, position).
|
||
-- Backfill + reactive prefetch can re-drill the same category multiple times;
|
||
-- without a unique constraint that re-insert kept all rows, leaving 8–34%
|
||
-- duplicate per source (~108k extra rows of ~588k). Keep the oldest row per
|
||
-- group (preserves original created_at) and drop the rest.
|
||
DELETE FROM "parts" WHERE id IN (
|
||
SELECT id FROM (
|
||
SELECT id, row_number() OVER (
|
||
PARTITION BY vehicle_id, category_id, oem_code, name, position
|
||
ORDER BY created_at ASC, id ASC
|
||
) AS rn FROM "parts"
|
||
) t WHERE rn > 1
|
||
);
|
||
--> statement-breakpoint
|
||
|
||
-- Phase 2: prevent future dupes. NULLS NOT DISTINCT (Postgres 15+) so NULL
|
||
-- position / NULL vehicle_id collapse like equal values rather than each
|
||
-- counting as a separate "distinct" row.
|
||
CREATE UNIQUE INDEX IF NOT EXISTS "parts_dedup_idx" ON "parts"
|
||
(vehicle_id, category_id, oem_code, name, position) NULLS NOT DISTINCT;
|