Jul 2023Data Pipelines1 min read

E-commerce Price & Catalog Sync

An electronics store checked supplier prices by hand every day. I automated supplier catalog extraction and price-drift detection, with margin rules and outlier thresholds that produce ready-to-import OpenCart files.

Daily manual price checks replaced by generated OpenCart imports

Python / OpenCart / Web Scraping

Pipeline
Extract → Diff → Margin → Manifest
Safety
Large deltas queued for human review

What it does

an electronics e-commerce store sources electronics from distributors whose prices move constantly. Without APIs, the only feed is HTML. Manual audits of hundreds of SKUs were slow and missed upward supplier moves that wiped thin margins.

The pipeline scrapes supplier catalogs, diffs against the live OpenCart catalog, applies margin rules, and emits an import manifest. Large price swings go to a review queue instead of publishing automatically.

Constraints

  • No official supplier APIs
  • Supplier HTML changes without notice: parsers must fail locally, not corrupt the whole batch
  • Margin floors and category markups must be enforceable in code
  • Output must match OpenCart import expectations

Decisions & tradeoffs

  1. Three stages: extract → diff/margin → OpenCart manifest
  2. Defensive parsing: layout breakage logs and skips; other SKUs continue
  3. Threshold guards: e.g. >20% deltas need human confirmation
  4. Rules separated from scrapers: pricing strategy can change without rewriting extractors
Diagram
PassFlaggedDistributor ScrapingCatalog SnapshotCurrentOpenCart CatalogPrice-DriftDetectorMargin Rule EngineIn range?OpenCart Import ManifestManual Review Queue

Tradeoffs: human review of outliers slows publish slightly and prevents catastrophic mispricing. Scrapers need occasional maintenance when markup changes; that still beat weekly manual audits.

What it demonstrates

  • Practical ETL under brittle web sources
  • Separating extraction from business rules
  • Human-in-the-loop controls where automation can lose money