--- title: Theoretical Machine Learning description: Theoretical machine learning analyzes the mathematical foundations and statistical limits of learning algorithms. Learners will understand concepts like computational learning theory, generalization bounds, and Rademacher complexity to explain why and when machine learning models succeed. category: programming-tech subcategory: theoretical-computer-science difficulty: beginner, intermediate, advanced url: /subject/theoretical-machine-learning --- # Theoretical Machine Learning Theoretical machine learning analyzes the mathematical foundations and statistical limits of learning algorithms. Learners will understand concepts like computational learning theory, generalization bounds, and Rademacher complexity to explain why and when machine learning models succeed. ## Available Resources 2 Books • 4 Courses • 1 Websites ## Courses ### 1. High-Dimensional Statistics (MIT 18.S997) **Author:** Philippe Rigollet Finite-sample analysis of high-dimensional methods: concentration inequalities, sparse linear regression, matrix estimation, principal component analysis and minimax lower bounds, ending with open research questions. Lecture notes and problem sets present the proof techniques behind modern optimality guarantees. **Difficulty:** Advanced | **Price:** Free **Link:** https://ocw.mit.edu/courses/18-s997-high-dimensional-statistics-spring-2015/ **Tags:** high-dimensional-statistics, concentration-inequalities, sparse-regression, minimax-rates, matrix-estimation ### 2. Mathematics of Machine Learning (MIT 18.657) **Author:** Philippe Rigollet A mathematically rigorous introduction to statistical learning theory: generalization bounds, VC dimension, Rademacher complexity, convex surrogates and optimization, boosting, and online learning. 194 pages of lecture notes and problem sets with solutions teach learners to analyze why learning algorithms work. **Difficulty:** Advanced | **Price:** Free **Link:** https://ocw.mit.edu/courses/18-657-mathematics-of-machine-learning-fall-2015/ **Tags:** statistical-learning-theory, generalization-bounds, rademacher-complexity, convex-optimization, online-learning ### 3. Algorithmic Aspects of Machine Learning (MIT 18.409) **Author:** Ankur Moitra Designing machine learning algorithms with provable guarantees: nonnegative matrix factorization, topic models, tensor decomposition, sparse recovery, dictionary learning and learning mixture models. Moitra's detailed lecture notes, which read as a complete monograph, and problem sets teach rigorous analysis of learning algorithms. **Difficulty:** Advanced | **Price:** Free **Link:** https://ocw.mit.edu/courses/18-409-algorithmic-aspects-of-machine-learning-spring-2015/ **Tags:** provable-algorithms, matrix-factorization, tensor-decomposition, sparse-recovery, learning-theory ### 4. Machine Learning Course - CS 156 Master theoretical machine learning with Yaser Abu-Mostafa's renowned "Learning From Data" course. Explore fundamental concepts and algorithms. **Difficulty:** Advanced | **Price:** Free **Link:** https://www.youtube.com/playlist?list=PLD63A284B7615313A **Tags:** machine-learning, statistical-learning-theory, generalization, caltech ## Websites ### 1. Distill.pub Peer-reviewed web journal of machine learning explanations, publishing interactive articles on topics like feature visualisation, attention and neural network interpretability. Archive remains readable, though the journal went on indefinite hiatus in 2021 and no longer publishes. **Difficulty:** Intermediate | **Price:** Free **Link:** https://distill.pub/ **Tags:** deep-learning, neural-networks, interpretability, interactive-visualization, machine-learning ## Books ### 1. Information Theory, Inference, and Learning Algorithms **Author:** David J. C. MacKay David MacKay's textbook, free to read online, covering entropy, data compression, noisy-channel coding, error-correcting codes, Bayesian inference, Monte Carlo methods and neural networks. Readers come to understand Shannon's coding theorems and apply probabilistic reasoning to communication and learning problems. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://www.inference.org.uk/mackay/itila/ **Tags:** information-theory, coding-theory, data-compression, bayesian-inference, machine-learning ### 2. Understanding Machine Learning: From Theory to Algorithms **Author:** Shai Shalev-Shwartz, Shai Ben-David A rigorous textbook treatment of statistical learning theory: PAC learning, VC dimension, uniform convergence, regularization, boosting, and SVMs. Readers finish able to state and prove generalization bounds and reason about why learning algorithms work. **Difficulty:** Intermediate | **Language:** English | **Price:** Paid **Link:** https://www.amazon.com/dp/1107057132?tag=edmonddante07-20 **Tags:** books, technology-computer-science, theoretical-computer-science --- *This content is part of Dantes.io - Your Treasure Map to Knowledge* *Curated by humans at Dantes.io. Personal study use welcome; republishing this curation requires permission (team@dantes.io).* View this page online: https://dantes.io/subject/theoretical-machine-learning