Data Engineering & The Modern Data Stack · Chapter 1 · ~15 min · Beginner
What is Data Engineering? Building the Data Highways
Discover the critical foundation of modern data: learn how Data Engineers build the pipelines and storage highways that move terabytes of information reliably, master ETL, and compare OLTP vs OLAP databases.
- Learn what Data Engineers do and why big tech cannot function without them
- Trace the 3 stages of data movement: Extract, Transform, and Load (ETL)
- Understand the crucial difference between OLTP (App DBs) and OLAP (Warehouses)
- Learn why modern Big Data uses Parquet columnar files instead of CSV
1The Plumbing of Tech: What Data Engineers Actually Do
Data Scientists build AI models, and Data Analysts build executive dashboards. But where do they get their clean, reliable data from?
In a modern enterprise, raw data is scattered across 50 different places: user signups in PostgreSQL, payment receipts in Stripe, click logs in web servers, and mobile metrics in Firebase.
A Data Engineer builds the pipes, highways, and warehouses. They write automated software pipelines that extract data from all these messy sources, clean and standardize it, and deliver it reliably to an analytical data warehouse every morning.
What is the primary mission of a Data Engineer in a tech company?
2The Classic 3-Stage Pipeline: ETL
The most famous acronym in data engineering is ETL:
Extract
Pulling raw data out of production databases, REST APIs, payment providers, and log files.
Transform
Cleaning corrupt data, filtering out bot traffic, masking sensitive credit cards, and converting currencies.
Load
Writing the clean, structured data into an Analytical Warehouse (like Snowflake, BigQuery, or Redshift).
What does the 'T' in ETL stand for, and what happens during this stage?
3OLTP vs. OLAP: Two Different Worlds
Beginners often ask: "Why can't our data analysts just run their reports directly on our main production app database?"
Because running a heavy analytical query (like "calculate average customer spend across 50,000,000 orders") would freeze your production database, preventing real customers from checking out! That's why we separate OLTP and OLAP:
OLTP (App Databases)
PostgreSQL, MySQL, MongoDB
Designed for fast, single-row operations (e.g. inserting 1 order, updating 1 user's password in 2ms). Stores data row-by-row on disk.
OLAP (Data Warehouses)
Snowflake, BigQuery, DuckDB
Designed for massive aggregations over millions of rows simultaneously. Stores data column-by-column on disk for 100x faster math!
Why shouldn't data analysts run heavy analytical reports directly against an application's production OLTP database?
4Why Big Data Uses Parquet Instead of CSV
Spreadsheet files (CSV) store records row-by-row as plain text. If your table has 50 columns and you only want to calculate the sum of price, a CSV forces the computer to read all 50 columns off disk!
Modern data pipelines use Apache Parquet. Parquet is a binary columnar format that:
- Compresses data up to 90% smaller than CSV files.
- Allows the computer to read only the columns you ask for, skipping the other 49 columns completely!
5Hands-On: Run an ETL Pipeline Live
Click through the steps below to watch raw web logs get extracted, transformed to remove bots and convert currency, and loaded into an analytics warehouse:
Live 3-Stage ETL Pipeline Visualizer
Pipeline Ready.
Click "1. Extract Logs" above to pull raw transaction records from production servers.
6Chapter Knowledge Challenge
Test your data engineering foundations on ETL pipelines, OLTP vs. OLAP, and columnar storage:
What is the primary architectural difference between OLTP and OLAP systems?
Why do queries on Apache Parquet files run much faster than queries on CSV files?
7Recap
- Data Engineering builds the infrastructure that ingests, cleans, and delivers data to analysts and AI models.
- The ETL pipeline extracts raw data, transforms/cleans it, and loads it into a warehouse.
- OLTP databases handle rapid single-row transactions; OLAP warehouses handle massive multi-million row analytics.
- Parquet columnar storage reduces disk size and speeds up analytical queries by scanning only requested columns.
Ready for Chapter 2? Mark this chapter as complete to proceed: