AI Evals: Everything You Need to Know (LLM Evals FAQ)
by Hamel Husain, Shreya Shankar · hamel.dev
Long question-and-answer guide drawn from the authors' evals course. Covers error analysis, synthetic test data, binary versus Likert grading, building LLM judges, human annotation, custom review tools, CI versus production monitoring, and guardrails versus evaluators. The densest free operational reference in the field, and awesome-evals lists it as a must-read. It covers the full topic scope: error analysis, test sets, code and model graders, and production monitoring. Most of the paid Maven course's methodology is here for free, which is why the course was discarded.
More resources on LLM Evaluation (Evals)
Improving Accuracy of LLM Applications
Walks through a systematic loop for making LLM applications reliable: build evaluation metrics, apply prompting and self-reflection, then fine-tune with LoRA and memory tuning. The running example is a text-to-SQL agent that hallucinates less each iteration.
Inspect: Open-Source Framework for LLM Evaluations
Documentation and tutorial for Inspect, the UK AI Security Institute's open-source Python evaluation framework. Covers datasets, solvers, custom scorers, model grading, sandboxed agent evals, log analysis, and running evaluation sets across models, with over 200 prebuilt evals.
AI Engineering: Building Applications with Foundation Models
Chip Huyen's 2025 O'Reilly book on building foundation-model applications. Chapters 3 and 4 cover evaluation: language-modeling metrics, exact evaluation, AI as a judge, comparative ranking, evaluation criteria, model selection, and designing an evaluation pipeline. The most rigorous textbook treatment of LLM evaluation methodology. Its two evaluation chapters (about 95 pages) give the conceptual grounding that blog posts skip.
Holistic Evaluation of Language Models (HELM)
Stanford CRFM benchmark paper that evaluates 30 language models across 42 scenarios and seven metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Shows how benchmark coverage and standardized conditions change what model comparisons mean. HELM is the canonical argument that single-number leaderboards mislead and that evaluation needs defined scenarios and multiple metrics.
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
UIST 2024 paper introducing EvalGen, an interface that helps people align LLM-generated graders with their own judgments. Its user study identifies criteria drift: people refine their grading criteria as they grade outputs, so criteria cannot be fixed in advance.
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Literature review of about two dozen papers on using LLMs to grade LLM output. Covers use cases, prompting techniques, aligning judges to human labels, fine-tuned evaluator models, and documented weaknesses such as position, verbosity, and self-preference bias. The standard deep reference on model-graded checks, the most error-prone part of any eval stack.