Build a Universal ETL Toolkit in Python in Under 100 Lines

What “universal ETL” means here

ETL is just three things done in order:

1. Extract
   Read rows from somewhere: a CSV, database cursor, API, or log file.
2. Transform
   Apply small, pure functions to each row: keep or drop keys, rename fields, cast types, clean values.
3. Load
   Write the transformed rows somewhere else: another CSV, database, queue, or in-memory buffer.

The “universal” part is not magic. It is just building your pipeline so only the extract and load ends change. The middle stays the same.

The 3-part pattern in under 100 lines

The GitHub repo ships a small ETLPipeline that takes three things:

• a generator that yields row dicts,
• a list of row transforms,
• a sink that consumes the cleaned rows.

from universal_etl import (
    ETLPipeline,
    dict_filter,
    dict_rename,
    iter_csv_rows,
    write_csv_rows,
)

# 1. Extract: read rows from a CSV
source = iter_csv_rows("data/example_sales_raw.csv")

# 2. Transform: keep and rename fields
transforms = [
    dict_filter(keys=["order_id", "customer", "quantity", "total_price"]),
    dict_rename(mapping={"total_price": "revenue"}),
]

# 3. Load: write cleaned rows to a new CSV
sink = write_csv_rows("data/example_sales_clean.csv")

pipeline = ETLPipeline(source=source, transforms=transforms, sink=sink)
pipeline.run()
    

Swap iter_csv_rows for a database cursor or API iterator and the transform list stays the same. Swap write_csv_rows for a bulk insert and you have a database loader. Same core, different edges.

How to adopt this in your own project

• Start with one ugly pipeline you already run by hand: a CSV export you clean every week.
• Wrap the steps into extract, a transform list, and load.
• Move those pieces into a tiny module (or reuse the one from the repo).
• Write one or two tests against a toy CSV so you can refactor safely.
• Only add complexity (logging, retries, scheduling) when you feel actual pain.

Once you are happy with the pattern on CSV-to-CSV, point it at a database or object store and keep the transforms exactly the same.

Related PLEX reading

References & further reading

• Python docs — csv module
  Official documentation for reading and writing CSV files in Python.
• pandas documentation
  Higher-level data manipulation library if you outgrow simple row-by-row transforms.
• Apache Airflow documentation
  A full orchestration framework for when simple scripts are no longer enough.
• dbt docs
  SQL-first transformation framework that pairs well with a lightweight Python-based extract and load layer.