fix(catalog-source): gate emex parts behind allowlist (safety)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
The catalog-wide bridge in EmexSourceDbService.fetchCategoryParts was
measured against vehicle_parts on 2026-06-01 and found to return 7-114x
more parts than belong to the requesting vehicle, with 49-98 wrong OEM
codes per 100 served. That directly violates the project rule that the
user must never see a wrong OEM.
Per-catalog noiseRatio sample (catalog-wide / per-vehicle):
RENAULT201910 51x | FFIAT84 45x | VOLVO201410 24x | MB201810 14x
AU1587 8x | BMW202501 70x (+ gid namespace mismatch ETK vs numeric)
GM_C201809 114x | MINI202501 12x | LRE201412 7x | MAZDA2020 54x
GM_OP201809 dump has only 1 wildcard vehicle (unique_key="_") so the
single Crossland X "owns" all 47k Opel parts — same firehose served
to any Opel sub-model in sase prod.
All alternative bridges were proven dead:
SSD eşleştirme - session-bound, 0/91 sase SSDs match dump
scrape_queue_v2.vehicle_ssd - same session SSD format
api_cache replay - table empty (0 rows)
wizard_parameters - table empty (0 rows)
VIN direct - no VIN column in dump
The only viable per-vehicle bridge is vehicles.unique_key reconstruction
from raw_data.parsedOptions, but sase currently stores the required 4
wizard fields on just 5/103 emex vehicles (all Renault). That work is
follow-up; this patch only stops the bleeding.
Change:
- Add EMEX_SOURCE_DB_ALLOWED_CATALOGS env (comma-separated, default "")
- EmexSourceDbService.fetchCategoryParts returns null unless catalogCode
is in the allowlist. Empty allowlist = service is effectively off for
parts, full fallthrough to live emex.
- Connection pool stays alive so the follow-up per-vehicle bridge /
schema-only path can use it without flipping env.
- Boot logs warn loudly when connected with an empty allowlist.
Prod was never affected — CATALOG_SOURCE_DB_ENABLED was unset there. This
fixes dev branch behaviour (default-on since commit 3a3a7d3) and keeps
prod safe by default once main is promoted.
Files:
- packages/config/src/index.ts env schema + audit notes
- apps/api/src/config/configuration.ts parse allowlist into string[]
- apps/api/src/integrations/catalog-source-db/emex-source-db.service.ts
allowlist field, init logging, fetchCategoryParts gate, class doc
- docker-compose.coolify.yml env injection for api + worker
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -77,6 +77,13 @@ export default () => ({
|
||||
// doesn't cover sase's TR-market vehicles; see service comment for details.
|
||||
emexEnabled: (process.env.EMEX_SOURCE_DB_ENABLED ?? "true") === "true",
|
||||
pcatEnabled: process.env.PCAT_SOURCE_DB_ENABLED === "true",
|
||||
// Per-catalog allowlist for emex parts lookup. Empty → service returns
|
||||
// null for every catalog (safe default after the 2026-06-01 noise audit).
|
||||
// Populate ONLY after a catalog's per-vehicle bridge is wired & verified.
|
||||
emexAllowedCatalogs: (process.env.EMEX_SOURCE_DB_ALLOWED_CATALOGS ?? "")
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean),
|
||||
},
|
||||
otel: {
|
||||
enabled: process.env.OTEL_ENABLED === "true",
|
||||
|
||||
@@ -11,21 +11,29 @@ import type { EmexHotspot, EmexHotspotArea, EmexPart, EmexPartsResult } from "..
|
||||
* Bridge: catalogs.code ↔ sase's vehicles.rawData.catalogCode,
|
||||
* part_groups.group_id (varchar) ↔ sase's categories.externalId (gid).
|
||||
*
|
||||
* Why catalog-wide (no per-vehicle filter): the dump's only stable per-vehicle
|
||||
* key is the EMEX session-state-descriptor (`ssd`), but SSD is regenerated
|
||||
* every decode session, so the SSD sase stored for its decoded vehicle never
|
||||
* matches the SSD the dump scraper recorded (0 / 10 sampled — verified). The
|
||||
* vehicle_id we'd need to join `vehicle_parts` is unreachable. So we return
|
||||
* all parts in (catalog, group) across every vehicle in the dump for that
|
||||
* catalog — over-returns variants that may not apply to the user's specific
|
||||
* vehicle, but parts overlap heavily and the cost is a slightly noisier list.
|
||||
* Hits: ~71% of sase's 8287 (catalogCode, gid) pairs (verified 2026-06-01).
|
||||
* SAFETY (2026-06-01 audit): the catalog-wide bridge below (no per-vehicle
|
||||
* filter) measured a 7-114x noiseRatio across every sampled catalog and
|
||||
* 49-98 wrong-OEM per 100 parts served, because vehicle_parts is shared
|
||||
* densely across all variants in a (catalog, group) pair (e.g. Renault
|
||||
* Mégane vs Duster vs Clio share the same gids). That directly violates the
|
||||
* "always correct OEM, no exceptions" rule. To prevent leakage, fetchCategoryParts
|
||||
* now returns null unless the catalogCode is in the EMEX_SOURCE_DB_ALLOWED_CATALOGS
|
||||
* allowlist — empty by default. A catalog should ONLY be added once a
|
||||
* per-vehicle bridge (vehicles.unique_key reconstruction from raw_data.parsedOptions)
|
||||
* lands and has been verified OEM-by-OEM vs live. All alternative bridges (SSD,
|
||||
* api_cache, wizard_parameters, scrape_queue_v2.vehicle_ssd) were verified
|
||||
* dead — see memory `sase-emex-source-db-safety.md` for the full inventory.
|
||||
*
|
||||
* The connection pool stays alive (master + emex switches default-on) so
|
||||
* follow-up code can use it for per-vehicle queries / schema-only diagrams
|
||||
* without having to flip env again.
|
||||
*/
|
||||
@Injectable()
|
||||
export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
private readonly logger = new Logger(EmexSourceDbService.name);
|
||||
private pool: Pool | null = null;
|
||||
private enabled = false;
|
||||
private allowedCatalogs: Set<string> = new Set();
|
||||
|
||||
constructor(private readonly config: ConfigService) {}
|
||||
|
||||
@@ -33,6 +41,8 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
const masterEnabled = this.config.get<boolean>("catalogSource.enabled");
|
||||
const emexEnabled = this.config.get<boolean>("catalogSource.emexEnabled");
|
||||
const url = this.config.get<string>("catalogSource.emexUrl");
|
||||
const allowed = this.config.get<string[]>("catalogSource.emexAllowedCatalogs") ?? [];
|
||||
this.allowedCatalogs = new Set(allowed);
|
||||
if (!masterEnabled || !emexEnabled || !url) {
|
||||
this.logger.log(
|
||||
`[emex-src] disabled (master=${masterEnabled}, emex=${emexEnabled}, urlSet=${Boolean(
|
||||
@@ -48,7 +58,15 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
waitForConnections: true,
|
||||
});
|
||||
this.enabled = true;
|
||||
this.logger.log("[emex-src] connected, lookup-first enabled");
|
||||
if (this.allowedCatalogs.size === 0) {
|
||||
this.logger.warn(
|
||||
"[emex-src] connected but ALLOWLIST EMPTY — all fetchCategoryParts calls return null until EMEX_SOURCE_DB_ALLOWED_CATALOGS is populated (safety: catalog-wide bridge has 7-114x noise; see sase-emex-source-db-safety memory)",
|
||||
);
|
||||
} else {
|
||||
this.logger.log(
|
||||
`[emex-src] connected, allowlist: [${[...this.allowedCatalogs].sort().join(",")}]`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async onModuleDestroy() {
|
||||
@@ -70,6 +88,10 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
): Promise<EmexPartsResult | null> {
|
||||
if (!this.enabled || !this.pool) return null;
|
||||
if (!catalogCode || !gid) return null;
|
||||
// Safety gate: the only path currently implemented (catalog-wide DISTINCT
|
||||
// on vehicle_parts.group_id, ignoring vehicle_id) is unsafe — see class
|
||||
// doc. Until a per-vehicle bridge lands, allow no catalogs.
|
||||
if (!this.allowedCatalogs.has(catalogCode)) return null;
|
||||
|
||||
try {
|
||||
// Resolve (catalog.id, part_group.id) in one round trip.
|
||||
|
||||
Reference in New Issue
Block a user