Skip to main content
Dantes.io Logo
Dantes.io

Best resources to learnLLM Evaluation (Evals)

Study sheet

LLM evaluation measures whether a language-model application actually works. You will learn error analysis, building test sets, code-based and model-graded checks, benchmarks and their limits, and how to monitor quality once a system is in production.

9 resources · · How we curate

🎓Courses(1)

course

Improving Accuracy of LLM Applications

Walks through a systematic loop for making LLM applications reliable: build evaluation metrics, apply prompting and self-reflection, then fine-tune with LoRA and memory tuning. The running example is a text-to-SQL agent that hallucinates less each iteration.

intermediate🇬🇧

📚Books(1)

bookPaid

AI Engineering: Building Applications with Foundation Models

Chip Huyen's 2025 O'Reilly book on building foundation-model applications. Chapters 3 and 4 cover evaluation: language-modeling metrics, exact evaluation, AI as a judge, comparative ranking, evaluation criteria, model selection, and designing an evaluation pipeline. The most rigorous textbook treatment of LLM evaluation methodology. Its two evaluation chapters (about 95 pages) give the conceptual grounding that blog posts skip.

intermediate🇬🇧

🌐Websites(5)

scroll for more →
website

Inspect: Open-Source Framework for LLM Evaluations

Documentation and tutorial for Inspect, the UK AI Security Institute's open-source Python evaluation framework. Covers datasets, solvers, custom scorers, model grading, sandboxed agent evals, log analysis, and running evaluation sets across models, with over 200 prebuilt evals.

intermediate🇬🇧
website

Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)

Literature review of about two dozen papers on using LLMs to grade LLM output. Covers use cases, prompting techniques, aligning judges to human labels, fine-tuned evaluator models, and documented weaknesses such as position, verbosity, and self-preference bias. The standard deep reference on model-graded checks, the most error-prone part of any eval stack.

advanced🇬🇧
website

Demystifying Evals for AI Agents

Anthropic engineering guide to evaluating multi-step, tool-using agents. Defines tasks, trials, transcripts, and outcomes. Compares code-based, model-based, and human graders, explains pass@k versus pass^k, and separates capability evals from regression evals. Anthropic's newest and most substantive public writing on evals, and the clearest vocabulary for agent evaluation, where most new eval work now happens. It complements Hamel's product-centric material with grader taxonomy and trial statistics.

intermediate🇬🇧
website

AI Evals: Everything You Need to Know (LLM Evals FAQ)

Long question-and-answer guide drawn from the authors' evals course. Covers error analysis, synthetic test data, binary versus Likert grading, building LLM judges, human annotation, custom review tools, CI versus production monitoring, and guardrails versus evaluators. The densest free operational reference in the field, and awesome-evals lists it as a must-read. It covers the full topic scope: error analysis, test sets, code and model graders, and production monitoring. Most of the paid Maven course's methodology is here for free, which is why the course was discarded.

intermediate🇬🇧
website

Your AI Product Needs Evals

Hamel Husain's 2024 essay arguing that LLM products improve through a loop of evaluating, debugging, and changing the system. Lays out three levels of evaluation: assertion-style unit tests, human and model review of logged traces, and A/B testing. It's short and free, and it frames the whole topic before readers move to the denser FAQ.

beginner🇬🇧

📄Papers(2)

paper

Holistic Evaluation of Language Models (HELM)

Stanford CRFM benchmark paper that evaluates 30 language models across 42 scenarios and seven metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Shows how benchmark coverage and standardized conditions change what model comparisons mean. HELM is the canonical argument that single-number leaderboards mislead and that evaluation needs defined scenarios and multiple metrics.

advanced🇬🇧
paper

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

UIST 2024 paper introducing EvalGen, an interface that helps people align LLM-generated graders with their own judgments. Its user study identifies criteria drift: people refine their grading criteria as they grade outputs, so criteria cannot be fixed in advance.

advanced🇬🇧
Share this topic

More in Artificial Intelligence

See all 28 topics in Artificial Intelligence →

Frequently Asked Questions

What are the best free resources to learn LLM Evaluation (Evals)?

Dantes has curated 9 resources for LLM Evaluation (Evals), including 1 books, 1 courses, 5 websites, 2 papers. All resources are hand-picked for quality — no algorithmic filler. Browse the full list above to find the format that works best for you.

Is LLM Evaluation (Evals) hard to learn?

LLM Evaluation (Evals) is approachable at the beginner level — there are resources here specifically for those starting from scratch. As you progress, intermediate and advanced material is also available to take your skills further.

What types of LLM Evaluation (Evals) learning resources are available on Dantes?

For LLM Evaluation (Evals), Dantes has curated 1 books, 1 courses, 5 websites, 2 papers. Each resource type serves a different learning style: videos and YouTube for visual learners, books for depth, courses for structured progression, and websites for quick reference.

How does Dantes select LLM Evaluation (Evals) resources?

Dantes is an algorithm-free learning directory. Resources are hand-curated based on quality, accuracy, and usefulness — not engagement metrics or paid placements. The goal is to surface the best learning material for LLM Evaluation (Evals), whether it's a free YouTube series, a classic textbook, or an open courseware from a top university.

Go deeper

Test your understanding of LLM Evaluation (Evals)

Explain it out loud. An AI tutor listens and asks questions that expose gaps you didn't know you had.