Information Theory, Inference, and Learning Algorithms
by David J. C. MacKay
MacKay's Cambridge text develops entropy, source coding, channel capacity, Hamming and LDPC codes, then shows the same mathematics driving Bayesian inference, clustering and neural networks. MacKay refuses to separate coding from inference, so a learner leaves with one mental model instead of two disconnected ones. Full PDF is free from the author's site. Heavy exercise sets with worked solutions.
More resources on Information Theory
Information Theory: From Coding to Learning
Polyanskiy and Wu's modern graduate text, free in full from the author's MIT page. Extends classical coding theory into f-divergences, finite-blocklength bounds, minimax statistical estimation and information-theoretic lower bounds used across machine learning theory.
A Mathematical Theory of Communication (Shannon, 1948)
The paper that created the field, both Bell System Technical Journal parts in one 55-page PDF. Shannon derives entropy from three axioms, proves the source and noisy-channel coding theorems, and defines capacity. Shannon's own framing of what a 'bit' is and why entropy must take its form is clearer than most textbook restatements, and reading it inoculates a learner against the pop-science distortions of the subject. Startlingly readable primary source.
Elements of Information Theory (2nd Edition)
The canonical citation of the field and the reference a learner returns to for a precise statement of any theorem. Theorem-by-theorem development of the AEP, source and channel coding theorems, rate distortion, network information theory, hypothesis testing and the Kolmogorov complexity connection, with the field's most complete problem sets.
Information Theory, Pattern Recognition, and Neural Networks (Cambridge, 16 lectures)
MacKay's complete 2012 Cambridge lecture course, sixteen sessions filmed in the Pippard Lecture Theatre. Covers entropy, compression, noisy-channel coding, error-correcting codes, Bayesian inference and neural networks, with the blackboard derivations and physical intuition the book compresses.
Visual Information Theory
Christopher Olah builds entropy, cross-entropy, KL divergence and mutual information from variable-length codes using interactive diagrams. After reading you can picture why these quantities have their formulas rather than memorising them, and read ML loss functions fluently.