r/dataengineering
Unknown
Practitioner subreddit where working data engineers discuss tooling choices, architecture tradeoffs, hiring and day-to-day problems. Reading it gives you a sense of what stacks teams actually run and which technologies are hype versus standard practice.
More resources on Data Engineering
Designing Data-Intensive
Martin Kleppmann's O'Reilly survey of the systems behind modern databases: storage engines, replication, partitioning, transactions, consensus and stream processing. Readers finish able to reason about the tradeoffs when choosing or combining data stores at scale.
DBT Learn
dbt Labs' official training site, offering free self-paced courses on dbt fundamentals, Jinja and macros, incremental models, and refactoring SQL. Completing them prepares you to build and test transformation pipelines in a dbt project.
Data Engineering Roadmap
Interactive skill map from roadmap.sh laying out the data engineering path step by step: programming, SQL, warehouses, orchestration, streaming and infrastructure. It helps you see what to study next and track which areas you have already covered.
Introduction to Data Engineering
Start your journey in one of the fastest growing professions today with this beginner-friendly Data Engineering course! You will be introduced to the core concepts, processes, and tools you need to know in order to get a foundational knowledge of data engineering. as well as the roles that Data Engineers, Data Scientists, and Data Analysts play in the ecosystem. You will begin this course by understanding what is data engineering as well as the roles that Data Engineers, Data Scientists, and Data Analysts play in this exciting field. Next you will learn about the data engineering ecosystem, the different types of data structures, file formats, sources of data, and the languages data professionals use in their day-to-day tasks. You will become familiar with the components of a data platform and gain an understanding of several different types of data repositories such as Relational (RDBMS) and NoSQL databases, Data Warehouses, Data Marts, Data Lakes and Data Lakehouses. You’ll then learn about Big Data processing tools like Apache Hadoop and Spark. You will also become familiar with ETL, ELT, Data Pipelines and Data Integration. This course provides you with an understanding of a typical Data Engineering lifecycle which includes architecting data platforms, designing data stores, and gathering, importing, wrangling, querying, and analyzing data. You will also learn about security, governance, and compliance. You will learn about career opportunities in the field of Data Engineering and the different paths that you can take for getting skilled as a Data Engineer. You will hear from several experienced Data Engineers, sharing their insights and advice. By the end of this course, you will also have completed several hands-on labs and worked with a relational database, loaded data into the database, and performed some basic querying operations.
Data Engineering Zoomcamp
Become a data engineer! Learn data pipelines, cloud technologies, and more with Alexey Grigorev's Data Engineering Zoomcamp.
Data Engineer with Python
A DataCamp career track of interactive Python courses on data engineering: importing data from files, APIs, and databases, cleaning it with pandas, applying software engineering practices, and scheduling ETL pipelines with Airflow. Exercises run in the browser.