How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
by Simon Boehm · siboehm.com
The canonical concrete demonstration of how the GPU memory hierarchy decides kernel performance. Step-by-step worklog that takes a naive CUDA SGEMM kernel to about 94% of cuBLAS on an A6000 through global memory coalescing, shared-memory caching, 1D and 2D block tiling, vectorization, and autotuning, with benchmarks and code for each step.
More resources on GPU Programming & CUDA
Triton Tutorials
The modern alternative to hand-written CUDA for ML kernels. Primary source, maintained with the language, and it reuses the tiling ideas from PMPP. Official ordered tutorials for Triton, the Python-embedded GPU kernel language used by PyTorch's compiler. Covers vector addition, fused softmax, block-tiled matrix multiplication, low-memory dropout, layer normalization, fused attention, and grouped GEMM, with benchmarks against PyTorch.
GPU MODE Lectures
Covers what the textbook does not: applying CUDA to machine-learning workloads in PyTorch and Triton, taught by the engineers who build those systems. Community lecture series formerly called CUDA MODE. Practitioners cover PMPP chapter recaps, profiling and integrating CUDA kernels in PyTorch, reductions, a CUDA performance checklist, Triton, quantization, sparsity, and FlashAttention. Slides and notebooks are on GitHub.
GPU Puzzles
Fourteen interactive notebook exercises that teach CUDA kernel writing through Numba in Python, progressing from thread indexing and guards through shared memory to pooling, dot product, prefix sum, convolution, and matrix multiplication. Runs free in Google Colab. Best free active-practice resource: learners write real kernels with instant tests and no local GPU needed. Complements the textbook's patterns chapters.
An Even Easier Introduction to CUDA
Shortest path from zero to a working CUDA kernel, from the vendor itself. NVIDIA tutorial that turns a C++ array-addition loop into a parallel CUDA kernel. It covers kernel function specifiers, Unified Memory allocation, thread and block indexing, launching many blocks, and memory prefetching, with speedups measured on a T4 GPU.
GPU Computing Lectures (AUB Spring 2021) by Izzat El Hajj
Recorded 24-lecture university course from a PMPP co-author, following the book's chapters: data-parallel programming, GPU architecture, memory and tiling, profiling, convolution, stencil, reduction, scan, histogram, merge, sort, sparse matrices, graph processing, and streams.
Programming Massively Parallel Processors: A Hands-on Approach (4th Edition)
The anchor text for the topic: standard GPU computing textbook covering the CUDA programming model, memory hierarchy, and performance considerations, followed by parallel patterns such as convolution, stencil, reduction, prefix sum, histogram, merge, sort, sparse matrices, and graph traversal.