Skip to main content
Dantes.io Logo
Dantes.io

Best resources to learnInformation Theory

Study sheet

Information theory quantifies information, compression and communication. You will learn entropy, mutual information, channel capacity and coding, and why these ideas underpin statistics and machine learning.

9 resources · · How we curate

📺YouTube(1)

youtube

Information Theory, Pattern Recognition, and Neural Networks (Cambridge, 16 lectures)

MacKay's complete 2012 Cambridge lecture course, sixteen sessions filmed in the Pippard Lecture Theatre. Covers entropy, compression, noisy-channel coding, error-correcting codes, Bayesian inference and neural networks, with the blackboard derivations and physical intuition the book compresses.

intermediate🇬🇧

🎓Courses(2)

course

Information Theory (MIT 6.441)

Graduate introduction to the mathematics of information: entropy, lossless compression, binary hypothesis testing, channel coding and lossy compression. Provides 29 lecture note files that together form a textbook-length treatment, plus problem sets. Prepares you to state and prove the core coding theorems.

advanced🇬🇧
course

Information and Entropy (MIT 6.050J)

The physical limits of communication and computation: digital signals, codes and compression, noise and error correction, probability, channel capacity, reversible and quantum computation, and entropy's link to the second law of thermodynamics. Includes an open textbook, programming assignments and 11 problem sets with solutions.

beginner🇬🇧

📚Books(4)

book

Information Theory: From Coding to Learning

Polyanskiy and Wu's modern graduate text, free in full from the author's MIT page. Extends classical coding theory into f-divergences, finite-blocklength bounds, minimax statistical estimation and information-theoretic lower bounds used across machine learning theory.

advanced🇬🇧
bookPaid

Elements of Information Theory (2nd Edition)

The canonical citation of the field and the reference a learner returns to for a precise statement of any theorem. Theorem-by-theorem development of the AEP, source and channel coding theorems, rate distortion, network information theory, hypothesis testing and the Kolmogorov complexity connection, with the field's most complete problem sets.

advanced🇬🇧
book

Information Theory, Inference, and Learning Algorithms

MacKay's Cambridge text develops entropy, source coding, channel capacity, Hamming and LDPC codes, then shows the same mathematics driving Bayesian inference, clustering and neural networks. MacKay refuses to separate coding from inference, so a learner leaves with one mental model instead of two disconnected ones. Full PDF is free from the author's site. Heavy exercise sets with worked solutions.

intermediate🇬🇧
book

Information Theory, Inference, and Learning Algorithms

David MacKay's textbook, free to read online, covering entropy, data compression, noisy-channel coding, error-correcting codes, Bayesian inference, Monte Carlo methods and neural networks. Readers come to understand Shannon's coding theorems and apply probabilistic reasoning to communication and learning problems.

advanced🇬🇧

🌐Websites(1)

website

Visual Information Theory

Christopher Olah builds entropy, cross-entropy, KL divergence and mutual information from variable-length codes using interactive diagrams. After reading you can picture why these quantities have their formulas rather than memorising them, and read ML loss functions fluently.

beginner🇬🇧

📄Papers(1)

paper

A Mathematical Theory of Communication (Shannon, 1948)

The paper that created the field, both Bell System Technical Journal parts in one 55-page PDF. Shannon derives entropy from three axioms, proves the source and noisy-channel coding theorems, and defines capacity. Shannon's own framing of what a 'bit' is and why entropy must take its form is clearer than most textbook restatements, and reading it inoculates a learner against the pop-science distortions of the subject. Startlingly readable primary source.

advanced🇬🇧
Share this topic

More in Applied Mathematics

See all 11 topics in Applied Mathematics →

Frequently Asked Questions

What are the best free resources to learn Information Theory?

Dantes has curated 9 resources for Information Theory, including 1 YouTube videos, 4 books, 2 courses, 1 websites, 1 papers. All resources are hand-picked for quality — no algorithmic filler. Browse the full list above to find the format that works best for you.

Is Information Theory hard to learn?

Information Theory is approachable at the beginner level — there are resources here specifically for those starting from scratch. As you progress, intermediate and advanced material is also available to take your skills further.

What types of Information Theory learning resources are available on Dantes?

For Information Theory, Dantes has curated 1 YouTube videos, 4 books, 2 courses, 1 websites, 1 papers. Each resource type serves a different learning style: videos and YouTube for visual learners, books for depth, courses for structured progression, and websites for quick reference.

How does Dantes select Information Theory resources?

Dantes is an algorithm-free learning directory. Resources are hand-curated based on quality, accuracy, and usefulness — not engagement metrics or paid placements. The goal is to surface the best learning material for Information Theory, whether it's a free YouTube series, a classic textbook, or an open courseware from a top university.

Go deeper

Test your understanding of Information Theory

Explain it out loud. An AI tutor listens and asks questions that expose gaps you didn't know you had.