Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

AI Evaluation

nounAI Evaluation and Reliabilityalso LLM evaluation, evals, AI evals

In one line

AI evaluation, often shortened to evals, is the practice of systematically testing the quality, safety and reliability of AI systems, especially generative models and agents, using test sets, scoring methods and human review.

AI evaluation, often shortened to evals, is the practice of systematically testing the quality, safety and reliability of AI systems. For classic predictive models, evaluation means computing metrics such as accuracy on labeled test data. For large language models, chatbots and AI agents, which produce open-ended text, take actions and can be right in many different ways, evaluation is harder and has become a discipline of its own. Evals are the main way teams know whether a change helped or hurt, and they are a core part of LLMOps and model evaluation.

Why generative AI is harder to evaluate

  • Open-ended outputs. There is no single correct answer to a request to draft an email or summarize a report, so exact-match scoring fails [1].
  • Many dimensions of quality. Accuracy, relevance, completeness, tone, safety, format and style all matter.
  • Non-determinism. The same prompt can give different answers.
  • Sensitivity to prompts and context. Small wording changes shift results.
  • Rapid change. Models, prompts and tools are updated often.
  • Hidden failures. Hallucination and subtle errors look fluent; see hallucination.
  • Long, multi-step tasks. Agents succeed or fail across many actions.

Components of an eval

An evaluation has three parts: a dataset of test cases, the system under test and a scoring method.

### Test cases

Good test cases come from real use: logged user requests, support tickets, documents and edge cases that caused problems. They should cover common tasks, hard cases, rare but important cases, adversarial inputs and different user groups, with reference answers or clear criteria where possible. See dataset and data labeling.

### Scoring methods

  • Exact and programmatic checks. Code compiles and passes unit tests, the output is valid JSON, a number matches, a required citation is present. Cheap and reliable when applicable.
  • Reference-based metrics. Overlap measures such as BLEU and ROUGE, and embedding similarity. Useful for narrow tasks, weak for open-ended quality.
  • Human evaluation. Experts or crowd raters score outputs or compare two. The gold standard for subjective qualities, but slow and costly.
  • LLM-as-a-judge. A strong model scores outputs against a rubric or compares two answers. It scales well and can match human preferences closely on many tasks, but it has biases, including a preference for longer answers and for answers written by itself, and it needs calibration against human labels [2].
  • User signals. Thumbs, edits, retries, task completion and retention in live use.
  • Task success in environments. For agents, whether the goal was achieved in a sandbox, such as a repaired software issue or a completed web task.

Types of evals

  • Capability evals. What can the system do? Includes public AI benchmarks and custom task suites.
  • Regression evals. Did a change break something that worked? Run on every update.
  • Safety and red-team evals. Does the system refuse harmful requests, resist jailbreaks and avoid leaking data? See red teaming and AI safety.
  • Fairness and bias evals. Do results differ unfairly across groups? See algorithmic bias.
  • Groundedness and factuality evals. Are claims supported by sources? Central for retrieval-augmented generation.
  • Component evals. Test retrieval, reranking, tools or prompts separately from the full pipeline.
  • End-to-end evals. Judge the whole user experience.
  • Online evals. A/B tests and monitoring on live traffic; see AI observability.

Building an eval program

  1. Define what good means with the product team: criteria, priorities and unacceptable failures.
  2. Start small. Even 50 to 100 well-chosen cases reveal many problems.
  3. Write rubrics with clear scales and examples, so that raters and judge models apply the same standard.
  4. Calibrate automatic judges against human ratings on a sample, and recheck periodically.
  5. Automate the runs in a pipeline, with results tracked by version.
  6. Examine failures by hand, group them into categories and feed them back as new test cases.
  7. Set release thresholds and block releases that regress.
  8. Monitor in production, since real traffic differs from tests.
  9. Keep a hidden test set that is not used to tune prompts.

Common pitfalls

  • Vibes-based testing. Trying a few prompts and judging by feel.
  • Overfitting to the eval set. Tuning prompts until the suite passes, without real improvement; see overfitting.
  • Contamination. Test items leaking into training data.
  • Biased judges. Uncalibrated model graders that favor verbosity.
  • Averages that hide failures. A 95 percent pass rate may conceal a critical failure type.
  • Stale suites. Test sets that no longer reflect real use.
  • Ignoring cost and latency. Quality must be weighed against speed and price.
  • Single-run noise. Non-determinism means results should be averaged over repeated runs.

Evaluating agents

Agents need evaluation of both outcomes and process: did they complete the task, how many steps and tool calls did they use, did they stay within permissions, and how did they recover from errors? Traces of each step are reviewed, and tests include failure injection, such as a tool returning an error; see AI agents and tool use.

Governance and standards

Organizations and regulators increasingly expect documented evaluation before high-risk systems are deployed, including evidence of testing for accuracy, robustness, bias and safety. Model and system cards report results to the public; see AI governance and AI regulation.

A worked example

A company builds an assistant that answers employee questions about benefits using company documents. The team collects 120 real questions from the help desk, adds 30 tricky ones, such as questions whose answers are not in the documents, and writes a short reference answer or a rubric for each. The rubric scores four things on a scale of one to five: factual accuracy against the documents, completeness, tone and whether the assistant correctly says “I don’t know” when the answer is absent. A strong judge model scores each response, and the team has a person check a random sample of thirty judgments, finding agreement in about 85 percent of cases and spotting that the judge is too lenient on long answers. They fix the rubric to penalize padding and rerun. Retrieval is evaluated separately by checking how often the right passage appears among the top five results. When a new prompt version raises average accuracy but increases wrong answers on questions with no answer in the documents, the release is blocked until that regression is fixed.

Tips for LLM judges

Judge models work best with specific rubrics, short scales and requests for brief reasoning before the score. Comparing two answers side by side tends to be more reliable than scoring one in isolation, but order effects exist, so the order should be randomized or both orders tested. Use a different, strong model as judge where possible to limit self-preference, avoid giving the judge information about which system produced which answer and keep human review in the loop for samples and for disagreements. For high-stakes criteria such as medical accuracy or legal compliance, human experts remain the final authority.

Cost, speed and sustainability of evals

Evaluations cost money and time, particularly when each run involves hundreds of model calls and a judge. Teams manage this with tiers: a small fast suite that runs on every change, a larger suite nightly and a full suite before major releases. They cache results for unchanged cases, sample from large sets and use cheaper models for first-pass screening. The investment pays back by catching regressions early and by allowing confident changes to prompts, models and retrieval settings. In mature teams, adding a new test case after each production failure becomes a habit, so that the suite grows into a living record of everything the system must get right; see model drift for why this record must be refreshed.

FAQs

What are AI evals? They are systematic tests of an AI system’s quality, safety and reliability, using test cases and scoring methods.

What is LLM-as-a-judge? Using a language model to score or compare the outputs of other models against a rubric.

How is evaluating an LLM different from a classifier? Language model outputs are open-ended, so scoring needs rubrics, model judges and human review instead of simple accuracy.

How many test cases do I need? Start with 50 to 100 representative cases and grow from real failures.

What is a regression test for AI? A rerun of saved cases after a change, to confirm that nothing got worse.

More in AI Evaluation and Reliability

Key facts

  • Evaluating generative models is harder than evaluating classifiers because outputs are open-ended and many answers can be correct1.
  • Using a strong language model as a judge to score other models' outputs can agree with human preferences more than 80 percent of the time on some tasks, but it has known biases such as favoring longer answers2.
  • Evaluation suites should be built from real use cases, rerun on every change and combined with monitoring after release1.

Related concepts

Quick checkWhat is AI evaluation?Show answer

AI evaluation, often called evals, is the practice of systematically testing the quality, safety and reliability of an AI system, especially a generative model or agent, using test sets, scoring methods and human review, before and after release.

Sources

  1. Chip Huyen, AI Engineering, O'Reilly Media, https://www.oreilly.com/library/view/ai-engineering/9781098166298/
  2. Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Advances in Neural Information Processing Systems, https://arxiv.org/abs/2306.05685

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue