E-commerce Price & Catalog Sync

What it does

an electronics e-commerce store sources electronics from distributors whose prices move constantly. Without APIs, the only feed is HTML. Manual audits of hundreds of SKUs were slow and missed upward supplier moves that wiped thin margins.

The pipeline scrapes supplier catalogs, diffs against the live OpenCart catalog, applies margin rules, and emits an import manifest. Large price swings go to a review queue instead of publishing automatically.

Constraints

• No official supplier APIs
• Supplier HTML changes without notice — parsers must fail locally, not corrupt the whole batch
• Margin floors and category markups must be enforceable in code
• Output must match OpenCart import expectations

Decisions & tradeoffs

1. Three stages — extract → diff/margin → OpenCart manifest
2. Defensive parsing — layout breakage logs and skips; other SKUs continue
3. Threshold guards — e.g. >20% deltas need human confirmation
4. Rules separated from scrapers — pricing strategy can change without rewriting extractors

flowchart LR
  DistributorScraper[Distributor Scraping] --> CatalogSnapshot[Catalog Snapshot]
  StorefrontExport[Current OpenCart Catalog] --> DiffEngine[Diff and Price-Drift Detector]
  CatalogSnapshot --> DiffEngine
  DiffEngine --> MarginEngine[Margin Rule Engine]
  MarginEngine --> ThresholdGuard{Price Delta within Threshold?}
  ThresholdGuard -->|Pass| Manifest[OpenCart Import Manifest]
  ThresholdGuard -->|Flagged| ReviewQueue[Manual Review Queue]

Tradeoffs: human review of outliers slows publish slightly and prevents catastrophic mispricing. Scrapers need occasional maintenance when markup changes; that still beat weekly manual audits.

What it demonstrates

• Practical ETL under brittle web sources
• Separating extraction from business rules
• Human-in-the-loop controls where automation can lose money