Chapter 1 of 2
Course Progress0 of 2 completed (0%)

Data Engineering & The Modern Data Stack · Chapter 1 · ~15 min · Beginner

What is Data Engineering? Building the Data Highways

Discover the critical foundation of modern data: learn how Data Engineers build the pipelines and storage highways that move terabytes of information reliably, master ETL, and compare OLTP vs OLAP databases.

  • Learn what Data Engineers do and why big tech cannot function without them
  • Trace the 3 stages of data movement: Extract, Transform, and Load (ETL)
  • Understand the crucial difference between OLTP (App DBs) and OLAP (Warehouses)
  • Learn why modern Big Data uses Parquet columnar files instead of CSV

1The Plumbing of Tech: What Data Engineers Actually Do

Data Scientists build AI models, and Data Analysts build executive dashboards. But where do they get their clean, reliable data from?

In a modern enterprise, raw data is scattered across 50 different places: user signups in PostgreSQL, payment receipts in Stripe, click logs in web servers, and mobile metrics in Firebase.

A Data Engineer builds the pipes, highways, and warehouses. They write automated software pipelines that extract data from all these messy sources, clean and standardize it, and deliver it reliably to an analytical data warehouse every morning.

Concept Check

What is the primary mission of a Data Engineer in a tech company?

2The Classic 3-Stage Pipeline: ETL

The most famous acronym in data engineering is ETL:

Stage 1

Extract

Pulling raw data out of production databases, REST APIs, payment providers, and log files.

Stage 2

Transform

Cleaning corrupt data, filtering out bot traffic, masking sensitive credit cards, and converting currencies.

Stage 3

Load

Writing the clean, structured data into an Analytical Warehouse (like Snowflake, BigQuery, or Redshift).

Concept Check

What does the 'T' in ETL stand for, and what happens during this stage?

3OLTP vs. OLAP: Two Different Worlds

Beginners often ask: "Why can't our data analysts just run their reports directly on our main production app database?"

Because running a heavy analytical query (like "calculate average customer spend across 50,000,000 orders") would freeze your production database, preventing real customers from checking out! That's why we separate OLTP and OLAP:

OLTP (App Databases)

PostgreSQL, MySQL, MongoDB

Designed for fast, single-row operations (e.g. inserting 1 order, updating 1 user's password in 2ms). Stores data row-by-row on disk.

OLAP (Data Warehouses)

Snowflake, BigQuery, DuckDB

Designed for massive aggregations over millions of rows simultaneously. Stores data column-by-column on disk for 100x faster math!

Concept Check

Why shouldn't data analysts run heavy analytical reports directly against an application's production OLTP database?

4Why Big Data Uses Parquet Instead of CSV

Spreadsheet files (CSV) store records row-by-row as plain text. If your table has 50 columns and you only want to calculate the sum of price, a CSV forces the computer to read all 50 columns off disk!

Modern data pipelines use Apache Parquet. Parquet is a binary columnar format that:

  • Compresses data up to 90% smaller than CSV files.
  • Allows the computer to read only the columns you ask for, skipping the other 49 columns completely!

5Hands-On: Run an ETL Pipeline Live

Click through the steps below to watch raw web logs get extracted, transformed to remove bots and convert currency, and loaded into an analytics warehouse:

Live 3-Stage ETL Pipeline Visualizer

Ready to Start
Stage 1Extract (Ingest)
Stage 2Transform (Clean)
Stage 3Load (Warehouse)

Pipeline Ready.

Click "1. Extract Logs" above to pull raw transaction records from production servers.

6Chapter Knowledge Challenge

Test your data engineering foundations on ETL pipelines, OLTP vs. OLAP, and columnar storage:

Concept Check 1 of 2
1

What is the primary architectural difference between OLTP and OLAP systems?

Concept Check 2 of 2
2

Why do queries on Apache Parquet files run much faster than queries on CSV files?

7Recap

  • Data Engineering builds the infrastructure that ingests, cleans, and delivers data to analysts and AI models.
  • The ETL pipeline extracts raw data, transforms/cleans it, and loads it into a warehouse.
  • OLTP databases handle rapid single-row transactions; OLAP warehouses handle massive multi-million row analytics.
  • Parquet columnar storage reduces disk size and speeds up analytical queries by scanning only requested columns.

Ready for Chapter 2? Mark this chapter as complete to proceed:

Enjoying this free lesson? Your support helps us build more free animated chapters.

Support Us