Why Your Next Side Project Should Be a Data Pipeline, Not an App

The world does not need your 17th todo app. Recruiters and hiring managers have seen thousands of “simple CRUD projects” and can tell, in seconds, that most of them never touched messy data, scheduling, or monitoring.

If you want a side project that actually changes how people read your CV or GitHub, build a data pipeline instead of another app. In this article, we’ll look at why pipelines are a stronger signal, what a realistic solo-stack looks like, and how to design a small pipeline that still feels “production-flavoured”.

CRUD apps plateau fast; data products keep compounding

Traditional app side projects are great for your first encounter with HTTP, forms, and databases. After that, every extra clone teaches you less. You wire the same login flow, the same REST endpoints, the same “update profile” screen—useful, but not particularly rare.

A data pipeline, by contrast, keeps compounding:

• You add a new data source and suddenly your joins, quality checks, and backfills get interesting.
• You add scheduling and now failures, retries, and idempotency matter.
• You add a small dashboard and realise you’re shipping a data product, not just a script.

The end result is something you can confidently talk through in an interview as a system: where the data comes from, how it moves, how it’s validated, and how a consumer uses it.

What a portfolio-ready data pipeline side project looks like

At a high level, your pipeline should answer four simple questions:

• Source: What real-world signal are you capturing? (Public transport delays, crypto prices, property listings?)
• Transform: How do you clean, validate, and reshape that signal into something trustworthy?
• Store: Where does the cleaned data live, and how can someone query it?
• Serve: How does a human (or another service) actually consume the result?

You don’t need Kubernetes or a full cloud stack. For a solo developer, a realistic stack can be:

• Language: Python (requests, pandas or polars, standard library).
• Orchestrator: Prefect or Dagster for scheduling, retries, and runs history.
• Storage: SQLite or DuckDB on disk; Parquet files if you want columnar storage.
• Serving layer: A Streamlit page, a static report, or even a simple Flask endpoint.

Designing a small but production-flavoured pipeline

Think in terms of “mini production system”, not “one-off notebook”. Here’s a concrete pattern you can follow:

1. Pick a narrow, opinionated question
   Avoid generic “ETL anything” ideas. Instead, ask a sharp question like: “How often is my city’s bus network more than 10 minutes late on weekdays?” or “How much of my monthly spending goes to subscriptions I forgot about?”.
2. Choose one primary, stable data source
   Use a reasonably stable API or dataset (GTFS feeds, public finance APIs, open weather or energy data) and learn its quirks: rate limits, schema changes, occasional outages. Design your ingestion to handle those gracefully.
3. Make transformations explicit and testable
   Separate ingestion from transformation. Store raw data first, then transform into tidy tables. Add at least a few simple checks: row counts, non-null fields, valid ranges for key metrics, and duplicates detection.
4. Persist results in a queryable layer
   Use a small analytics store like DuckDB or SQLite. Create a handful of views that directly answer your core questions, so you can demo answers live in an interview without re-running the whole pipeline.
5. Expose a tiny, opinionated data product
   Ship something that non-engineers could understand: a simple report, a chart page, or a dashboard that refreshes when the pipeline runs. Include screenshots and example queries in your README so people can “see” the product immediately.

How hiring managers read your pipeline project in 30 seconds

When someone looks at your GitHub, they’re not reading every line of code. They are looking for quick signals:

• Structure: Is there a clear layout for ingest, transform, and serve code, or is everything dumped into one file?
• Story: Does the README explain what the data product is, why it exists, and how to run it end-to-end?
• Reliability: Do you mention retries, idempotency, or handling bad data—anything that suggests you’ve thought beyond the happy path?
• Evidence: Are there screenshots, example queries, or sample outputs that show the pipeline has actually run?

A simple, well-documented pipeline will often beat a flashy app that doesn’t show how it deals with reality.

Three concrete pipeline ideas you can start this month

If you’re stuck on ideas, here are three “small but real” projects that map nicely to the pattern above:

• City reliability tracker: Ingest public transport or traffic feeds, calculate lateness or congestion scores, and publish a weekly reliability index for your city.
• Subscription drain monitor: Pull bank transaction exports or budgeting app data, classify recurring payments, and chart which subscriptions consume most of your net income over time.
• Learning firehose curator: Aggregate new blog posts or videos from a small list of technical sources, tag them by topic, and email yourself a weekly “top 10” digest.

Each of these can stay small in scope while still touching realistic data and concerns you can talk about in interviews.

From “project” to career asset

The goal isn’t to collect side projects like badges. It’s to have one or two living data products that showcase how you think about systems, not just code.

Frame your pipeline on your resume and LinkedIn as a data product: mention the question it answers, the data volume you handle, how often it runs, and how you monitor it. Link to the repo, highlight the stack, and include a screenshot of the output.

Most developers will keep shipping small apps that look impressive for a weekend and are forgotten by Monday. If you ship a thoughtful data pipeline instead, you’ll quietly move into the group of people who can turn chaotic data into dependable decisions—and that’s what modern teams actually hire for.

Related PLEX reading

References & further reading

• Data Engineering: Role, Skills and Tools
  A succinct overview of what data engineers actually do and why pipelines sit at the centre of the role.
• Build Better Data Pipelines: Tools, Types & Real-Time Use Cases
  A practical guide to modern data pipeline types and design patterns.
• Top Data Engineering Projects for Hands-On Learning
  Examples of portfolio projects that emphasise end-to-end pipelines over simple apps.
• How to Build a Data Engineering Portfolio That Actually Gets You Hired
  Hiring-focused advice on structuring projects and GitHub repos for maximum impact.
• Build a Poor Man’s Data Lake from Scratch with DuckDB
  A concrete example of using Python, DuckDB and Dagster to build lightweight analytics pipelines.