Elements of Information Theory (2nd Edition)
by Thomas M. Cover, Joy A. Thomas
The canonical citation of the field and the reference a learner returns to for a precise statement of any theorem. Theorem-by-theorem development of the AEP, source and channel coding theorems, rate distortion, network information theory, hypothesis testing and the Kolmogorov complexity connection, with the field's most complete problem sets.
This link may earn us a small commission at no extra cost to you. Affiliate disclosure
More resources on Information Theory
Information Theory, Inference, and Learning Algorithms
David MacKay's textbook, free to read online, covering entropy, data compression, noisy-channel coding, error-correcting codes, Bayesian inference, Monte Carlo methods and neural networks. Readers come to understand Shannon's coding theorems and apply probabilistic reasoning to communication and learning problems.
Information Theory (MIT 6.441)
Graduate introduction to the mathematics of information: entropy, lossless compression, binary hypothesis testing, channel coding and lossy compression. Provides 29 lecture note files that together form a textbook-length treatment, plus problem sets. Prepares you to state and prove the core coding theorems.
Information and Entropy (MIT 6.050J)
The physical limits of communication and computation: digital signals, codes and compression, noise and error correction, probability, channel capacity, reversible and quantum computation, and entropy's link to the second law of thermodynamics. Includes an open textbook, programming assignments and 11 problem sets with solutions.
Information Theory: From Coding to Learning
Polyanskiy and Wu's modern graduate text, free in full from the author's MIT page. Extends classical coding theory into f-divergences, finite-blocklength bounds, minimax statistical estimation and information-theoretic lower bounds used across machine learning theory.
A Mathematical Theory of Communication (Shannon, 1948)
The paper that created the field, both Bell System Technical Journal parts in one 55-page PDF. Shannon derives entropy from three axioms, proves the source and noisy-channel coding theorems, and defines capacity. Shannon's own framing of what a 'bit' is and why entropy must take its form is clearer than most textbook restatements, and reading it inoculates a learner against the pop-science distortions of the subject. Startlingly readable primary source.
Information Theory, Pattern Recognition, and Neural Networks (Cambridge, 16 lectures)
MacKay's complete 2012 Cambridge lecture course, sixteen sessions filmed in the Pippard Lecture Theatre. Covers entropy, compression, noisy-channel coding, error-correcting codes, Bayesian inference and neural networks, with the blackboard derivations and physical intuition the book compresses.