GGUF (Hugging Face Hub Documentation)
Hugging Face
The best short primary source on model file formats. Its quant-type table explains the file names learners see when they download models. Hugging Face documentation for the GGUF model format. Explains how GGUF differs from safetensors, how to find and inspect GGUF files on the Hub, and how they load in llama.cpp, Ollama and LM Studio, with a reference table of quantization types and bits per weight.
More resources on Running LLMs Locally
LM Studio Documentation
The pending set has the low-level engine (llama.cpp) and a CLI runner FAQ (Ollama) but no GUI on-ramp for a non-terminal beginner, and nothing covering Apple Silicon's MLX path. LM Studio covers both in one place, including concrete RAM/VRAM requirements. Official docs for LM Studio, a free desktop app for downloading and running open-weight models offline through llama.cpp (GGUF) or Apple MLX engines. Covers system requirements, model downloads, chatting with documents, and serving models on an OpenAI-compatible local API.
LLM Inference Handbook
Fills the biggest gap in the pending set: hardware and memory sizing (weights x precision + KV cache + overhead, with worked examples and a calculator) plus model-type and GPU selection. Free open-source handbook on how LLM inference works and how to size it: CPU vs GPU, estimating GPU memory from parameters, precision and KV cache, choosing a GPU, choosing an open-weight model, and quantization, with interactive memory calculators.
Optimizing Your LLM in Production
Works through LLM memory from first principles: the rule that X billion parameters need about 2X GB in 16-bit, 8-bit and 4-bit quantization, Flash Attention, how the key-value cache grows with context, and how multi-query and grouped-query attention reduce it. Teaches how to estimate whether a model and context length will fit on your hardware, which tool docs assume you already know.
Ollama FAQ
The Ollama quickstart now mixes in cloud models and coding-agent setup. Official answers to the practical questions of running models with Ollama: setting context length, checking CPU/GPU placement with ollama ps, keep-alive and memory release, KV-cache quantization, flash attention, and exposing the server on a network.
A Visual Guide to Quantization
The clearest free explanation of how quantization works, and a good first read before choosing quant files. It dates from 2024, but the concepts have not changed. An illustrated explanation of why LLMs use so much memory and how quantization shrinks them. Covers FP16, BF16 and INT8, symmetric and asymmetric mapping, and post-training methods including GPTQ and GGUF's block-wise scheme, plus quantization-aware training and BitNet.
llama.cpp Quantize Tool: Quantization Types and Trade-offs
Answers the practical question of which quant to download using measured sizes and speeds. Maintainer documentation for llama-quantize. Measured tables for Llama-3.1-8B and 70B show file size, bits per weight and throughput for each quantization type from IQ1_S to F16, and explain importance-matrix (imatrix) quantization and memory needs.