Streaming 101: The World Beyond Batch
by Tyler Akidau · O'Reilly Radar
The foundational conceptual text for streaming pipeline design. Essay defining streaming as processing unbounded data and arguing it can be a superset of batch. Explains event time versus processing time, windowing strategies, watermarks and why correctness requires consistent state and exactly-once processing.
More resources on Spark, Airflow & Data Pipelines
dbt Fundamentals
Free, roughly five-hour course from dbt Labs: connecting dbt to a warehouse and Git, building models, declaring sources, writing data tests, generating documentation and deploying jobs. Hands-on exercises use a free dbt trial account. The canonical starting point for the transformation layer; free and maintained by the tool's creators.
Apache Airflow Tutorial: Fundamental Concepts
Official Airflow 3 tutorial that builds a first DAG: default arguments, scheduling, operators and tasks, Jinja templating, task dependencies, documentation, and testing with the airflow CLI. Leads on to the TaskFlow API and a full pipeline tutorial.
Learning Spark: Lightning-Fast Data Analytics (2nd Edition)
Covers Spark 3.0's structured APIs: DataFrames, Datasets and Spark SQL, the Catalyst optimizer, data sources, tuning, Structured Streaming, Delta Lake and MLlib, with Python and Scala examples. Code lives in the companion databricks/LearningSparkV2 repository. Best book-length treatment of modern (DataFrame-first) Spark from the people who built it.