Plugin-Style ETL on AWS Lambda

Small-to-medium ETL often does not need Airflow. It needs: read A, transform, write B, on a schedule — and the ability to swap A/B without rewriting orchestration.

wirecept/importer does that with named provider modules selected from YAML.

Config-driven job

source:
  type: mysql
  connection: ${env:DB_URL}
  query: "SELECT * FROM leads WHERE synced_at IS NULL"

transforms:
  - type: map_fields
    mapping:
      email: contact_email
      name: full_name

destination:
  type: mongodb
  connection: ${env:MONGO_URI}
  collection: imported_leads

Lambda reads CONFIG_URL, fetches YAML, instantiates providers. Switching MySQL→MongoDB to console→MongoDB backfill is a config change.

Why Lambda

• Pay per nightly run
• Fresh process each invocation (no leaked connections)
• Serverless Framework handles packaging/CloudFormation

Limit: 15-minute ceiling — large slow batches need chunking or a container task.

Gaps worth naming

• No test suite around providers/transforms (schema drift risk)
• No metrics hooks (zero-row “success” looks like a healthy run)
• Idempotency is application-level (synced_at filters), not built into the pipeline

Useful when job shapes are simple and config churn beats code churn.