---
title: Spark, Airflow & Data Pipelines
description: Modern data platforms move and transform data at scale. You will learn Spark for distributed processing, Airflow for orchestration, dbt for transformations and the design of reliable batch and streaming pipelines.
category: programming-tech
subcategory: data-science
difficulty: beginner, intermediate, advanced
url: /subject/apache-spark-and-data-pipelines
---

# Spark, Airflow & Data Pipelines

Modern data platforms move and transform data at scale. You will learn Spark for distributed processing, Airflow for orchestration, dbt for transformations and the design of reliable batch and streaming pipelines.

## Available Resources

1 Books • 1 Courses • 2 Websites

## Websites

### 1. Streaming 101: The World Beyond Batch

**Author:** Tyler Akidau

The foundational conceptual text for streaming pipeline design. Essay defining streaming as processing unbounded data and arguing it can be a superset of batch. Explains event time versus processing time, windowing strategies, watermarks and why correctness requires consistent state and exactly-once processing.

**Difficulty:** Advanced | **Language:** English | **Price:** Free

**Link:** https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/

**Tags:** stream-processing, event-time, windowing, watermarks

### 2. Apache Airflow Tutorial: Fundamental Concepts

Official Airflow 3 tutorial that builds a first DAG: default arguments, scheduling, operators and tasks, Jinja templating, task dependencies, documentation, and testing with the airflow CLI. Leads on to the TaskFlow API and a full pipeline tutorial.

**Difficulty:** Beginner | **Language:** English | **Price:** Free

**Link:** https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html

**Tags:** apache-airflow, dags, workflow-orchestration, data-pipelines

## Courses

### 1. dbt Fundamentals

Free, roughly five-hour course from dbt Labs: connecting dbt to a warehouse and Git, building models, declaring sources, writing data tests, generating documentation and deploying jobs. Hands-on exercises use a free dbt trial account. The canonical starting point for the transformation layer; free and maintained by the tool's creators.

**Difficulty:** Beginner | **Language:** English | **Price:** Free

**Link:** https://learn.getdbt.com/courses/dbt-fundamentals

**Tags:** dbt, analytics-engineering, data-transformation, data-testing

## Books

### 1. Learning Spark: Lightning-Fast Data Analytics (2nd Edition)

**Author:** Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee

Covers Spark 3.0's structured APIs: DataFrames, Datasets and Spark SQL, the Catalyst optimizer, data sources, tuning, Structured Streaming, Delta Lake and MLlib, with Python and Scala examples. Code lives in the companion databricks/LearningSparkV2 repository. Best book-length treatment of modern (DataFrame-first) Spark from the people who built it.

**Difficulty:** Intermediate | **Language:** English | **Price:** Paid

**Link:** https://www.amazon.com/dp/1492050040?tag=edmonddante07-20

**Tags:** apache-spark, spark-sql, structured-streaming, delta-lake, pyspark

---

*This content is part of Dantes.io - Your Treasure Map to Knowledge*

*Curated by humans at Dantes.io. Personal study use welcome; republishing this curation requires permission (team@dantes.io).*

View this page online: https://dantes.io/subject/apache-spark-and-data-pipelines