--- title: GPU Programming & CUDA description: GPU programming runs computations across thousands of parallel cores. You will learn the CUDA programming model, the GPU memory hierarchy, writing kernels, parallel patterns such as reduction and convolution, and how GPUs accelerate machine learning and scientific computing. category: programming-tech subcategory: computer-science difficulty: beginner, intermediate, advanced url: /subject/gpu-programming-and-cuda --- # GPU Programming & CUDA GPU programming runs computations across thousands of parallel cores. You will learn the CUDA programming model, the GPU memory hierarchy, writing kernels, parallel patterns such as reduction and convolution, and how GPUs accelerate machine learning and scientific computing. ## Available Resources 1 Books • 4 Websites ## Websites ### 1. Triton Tutorials **Author:** Philippe Tillet and Triton contributors The modern alternative to hand-written CUDA for ML kernels. Primary source, maintained with the language, and it reuses the tiling ideas from PMPP. Official ordered tutorials for Triton, the Python-embedded GPU kernel language used by PyTorch's compiler. Covers vector addition, fused softmax, block-tiled matrix multiplication, low-memory dropout, layer normalization, fused attention, and grouped GEMM, with benchmarks against PyTorch. **Difficulty:** Intermediate | **Language:** English | **Price:** Free **Link:** https://triton-lang.org/main/getting-started/tutorials/index.html **Tags:** triton, gpu-kernels, fused-attention, matrix-multiplication, pytorch ### 2. How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog **Author:** Simon Boehm The canonical concrete demonstration of how the GPU memory hierarchy decides kernel performance. Step-by-step worklog that takes a naive CUDA SGEMM kernel to about 94% of cuBLAS on an A6000 through global memory coalescing, shared-memory caching, 1D and 2D block tiling, vectorization, and autotuning, with benchmarks and code for each step. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://siboehm.com/articles/22/CUDA-MMM **Tags:** cuda, sgemm, matrix-multiplication, memory-coalescing, shared-memory, kernel-optimization ### 3. GPU Puzzles **Author:** Sasha Rush Fourteen interactive notebook exercises that teach CUDA kernel writing through Numba in Python, progressing from thread indexing and guards through shared memory to pooling, dot product, prefix sum, convolution, and matrix multiplication. Runs free in Google Colab. Best free active-practice resource: learners write real kernels with instant tests and no local GPU needed. Complements the textbook's patterns chapters. **Difficulty:** Beginner | **Language:** English | **Price:** Free **Link:** https://github.com/srush/GPU-Puzzles **Tags:** cuda, numba, gpu-kernels, shared-memory, parallel-patterns ### 4. An Even Easier Introduction to CUDA **Author:** Mark Harris Shortest path from zero to a working CUDA kernel, from the vendor itself. NVIDIA tutorial that turns a C++ array-addition loop into a parallel CUDA kernel. It covers kernel function specifiers, Unified Memory allocation, thread and block indexing, launching many blocks, and memory prefetching, with speedups measured on a T4 GPU. **Difficulty:** Beginner | **Language:** English | **Price:** Free **Link:** https://developer.nvidia.com/blog/even-easier-introduction-cuda/ **Tags:** cuda, cuda-kernels, unified-memory, thread-indexing, parallel-programming ## Youtubes ### 1. GPU MODE Lectures **Author:** Mark Saroufim, Andreas Köpf, et al. Covers what the textbook does not: applying CUDA to machine-learning workloads in PyTorch and Triton, taught by the engineers who build those systems. Community lecture series formerly called CUDA MODE. Practitioners cover PMPP chapter recaps, profiling and integrating CUDA kernels in PyTorch, reductions, a CUDA performance checklist, Triton, quantization, sparsity, and FlashAttention. Slides and notebooks are on GitHub. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://www.youtube.com/playlist?list=PLMxsBHP5XAAGY4fOPe-LB-GRB3shDzcx7 **Tags:** cuda, pytorch, triton, quantization, flashattention, gpu-profiling ### 2. GPU Computing Lectures (AUB Spring 2021) by Izzat El Hajj **Author:** Izzat El Hajj Recorded 24-lecture university course from a PMPP co-author, following the book's chapters: data-parallel programming, GPU architecture, memory and tiling, profiling, convolution, stencil, reduction, scan, histogram, merge, sort, sparse matrices, graph processing, and streams. **Difficulty:** Intermediate | **Language:** English | **Price:** Free **Link:** https://www.youtube.com/playlist?list=PLRRuQYjFhpmubuwx-w8X964ofVkW1T8O4 **Tags:** gpu-architecture, cuda, parallel-patterns, memory-tiling, parallel-programming ## Books ### 1. Programming Massively Parallel Processors: A Hands-on Approach (4th Edition) **Author:** Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj The anchor text for the topic: standard GPU computing textbook covering the CUDA programming model, memory hierarchy, and performance considerations, followed by parallel patterns such as convolution, stencil, reduction, prefix sum, histogram, merge, sort, sparse matrices, and graph traversal. **Difficulty:** Intermediate | **Language:** English | **Price:** Paid **Link:** https://www.amazon.com/dp/0323912311?tag=edmonddante07-20 **Tags:** cuda, gpu-architecture, parallel-programming, parallel-patterns, memory-hierarchy --- *This content is part of Dantes.io - Your Treasure Map to Knowledge* *Curated by humans at Dantes.io. Personal study use welcome; republishing this curation requires permission (team@dantes.io).* View this page online: https://dantes.io/subject/gpu-programming-and-cuda