Data labeling is the process of attaching a meaningful tag or target value to each example in a data set so that a machine learning model can learn the correct answer. An email is labeled “spam” or “not spam.” A photograph is labeled with the animal it shows. A house listing is labeled with the price it sold for. These labels, often called ground truth, are what supervised learning tries to predict [1]. Labeling is a core step in turning a raw dataset into training data, and it overlaps heavily with data annotation, which sometimes refers to richer markup.
Labels and tasks
The kind of label depends on the task the model must perform.
- Categories (classification). A single class from a fixed list, such as “positive,” “negative” or “neutral.”
- Multiple tags (multi-label). Several labels at once, such as the topics covered by a news article.
- Numbers (regression). A value such as a temperature, price or age.
- Structured labels. Boxes, masks, keypoints or entity spans, used in computer vision and natural language processing.
- Sequences. Transcripts for speech recognition or target translations for machine translation.
- Preferences. Rankings of which of two answers is better, used to tune large language models.
Where labels come from
- Human experts. Doctors labeling scans, lawyers marking contract clauses. Accurate but costly.
- Crowd workers. Many people performing simple tasks through online platforms, with redundancy to catch errors.
- Natural labels. Outcomes recorded in the course of normal operations, such as whether a customer actually cancelled a subscription or a loan defaulted. These cost little but may be delayed or biased by past decisions.
- Programmatic labeling. Rules or scripts that assign labels automatically, sometimes combined statistically in weak supervision.
- Model-generated labels. Existing models, including large language models, produce labels that people check.
- Self-supervision. Labels derived from the data itself, such as the next word in a sentence; see self-supervised learning.
The labeling process
- Define the label set. Make categories clear, mutually exclusive where needed, and include an “unsure” or “other” option.
- Create guidelines with examples. Include borderline cases.
- Sample data carefully. Make sure rare but important cases are included.
- Label a pilot batch. Measure agreement between labelers and fix confusing instructions.
- Scale up with quality checks. Use review layers, hidden test items and consensus.
- Audit and iterate. After training a first model, study its errors, because they often reveal labeling mistakes or missing categories.
Label quality and noise
Labels are rarely perfect. Sources of error include careless work, unclear definitions, genuinely ambiguous items and drift in standards over time. Label noise lowers model accuracy and can make evaluation misleading. Northcutt and colleagues found measurable rates of mislabeled examples even in the test sets of famous benchmark data sets, enough to change model rankings in some cases [2]. Tools for finding suspected label errors, which flag examples where a model strongly disagrees with the given label, can help clean data.
### Handling disagreement
When labelers disagree, teams can take a majority vote, escalate to an expert, keep the disagreement as a soft label (for example, 70 percent positive) or study the pattern to improve guidelines. For subjective tasks, preserving multiple viewpoints can be more truthful than forcing a single answer.
Efficiency techniques
- Active learning. Prioritize labeling items the model is least sure about; see active learning.
- Pre-labeling. Show model suggestions for people to accept or fix.
- Transfer learning. Reuse a pretrained model to need fewer labels; see transfer learning.
- Data augmentation. Create variants of labeled examples, covered in data augmentation.
- Semi-supervised learning. Learn from a small labeled set plus a large unlabeled one.
Risks and responsibilities
- Bias in labels. Labels that encode stereotypes or past discrimination, such as past hiring decisions, teach models to repeat them; see data bias.
- Privacy. Labelers may see personal data, so access controls, contracts and de-identification are needed.
- Fair work. Responsible organizations pay fair wages, set reasonable quotas and protect workers who review harmful content.
- Documentation. Recording who labeled what, under which guidelines, supports audits and reproducibility.
Labeling in the era of large language models
Large language models have changed labeling in two ways. First, they can act as labelers. Given a clear instruction and a few examples, a model can tag thousands of documents by topic, sentiment or intent in minutes, at a small fraction of the cost of human workers. Teams typically check a random sample against expert judgment, measure how often the model agrees, and send low-confidence items to people. This approach works best for tasks that are well defined, and it can fail quietly when the model has systematic blind spots, so human review is still needed.
Second, models themselves need new kinds of labels. Instead of a single correct answer, alignment work asks people to rank several model responses by helpfulness, accuracy and safety, or to write ideal answers. These preference labels train reward models and guide instruction tuning. Because such judgments are subjective, clear rubrics and diverse labeler pools are important.
Examples across industries
- Healthcare. Radiologists label tumors on scans, and pathologists mark cells on slides, to build diagnostic aids. Strict privacy rules apply.
- Automotive. Teams label lanes, signs, vehicles and pedestrians in camera and lidar data for driver-assistance and self-driving cars.
- Retail and finance. Transactions are labeled as fraudulent or legitimate, and customer messages as complaints or inquiries.
- Agriculture. Drone images are labeled for crop disease and weeds.
- Customer service. Conversations are tagged by intent and outcome to train assistants and routing systems.
Measuring the value of labels
Labeling should be managed like any other investment. Teams track agreement rates, error rates found in audits, throughput per labeler and cost per accepted label. They also test how model accuracy changes as the amount of labeled data grows, using a learning curve. If the curve has flattened, more labels of the same kind will add little, and effort is better spent on fixing errors or covering missing cases. If the curve is still rising steeply, more labeling is likely to pay off.
Finally, labeled data should be treated as a maintained asset. Standards change, products evolve and the world shifts, so labels may need review. Versioning label sets and recording changes helps teams reproduce earlier results and explain differences between model versions.
FAQs
What is data labeling? It is assigning tags or target values to raw data so supervised models can learn from correct examples.
What is ground truth? It is the trusted set of correct labels used to train and evaluate a model.
How do you improve label quality? With clear guidelines, trained labelers, multiple reviews, gold-standard test items and error analysis after training.
Is data labeling needed for all AI? No. Unsupervised and self-supervised methods learn from unlabeled data, but most applied systems still need some labeled data, especially for evaluation.
What is the difference between data labeling and annotation? They largely mean the same thing; annotation often implies richer markup such as outlines and relations.
More in AI Data and Infrastructure
Key facts
- In supervised learning each training example pairs an input with a label that represents the correct output1.
- Labels that serve as the trusted reference answers for training and evaluation are often called ground truth1.
- Research on benchmark data sets has found that even widely used test sets contain a measurable share of incorrectly labeled examples2.
Related concepts
Quick checkWhat is data labeling?Show answer
Data labeling is the process of assigning meaningful tags or target values to raw data, such as marking an email as spam or an image as a cat, so that supervised machine learning models have correct answers to learn from.
Sources
- Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, Pearson, https://aima.cs.berkeley.edu/
- Curtis Northcutt, Anish Athalye and Jonas Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, https://arxiv.org/abs/2103.14749
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
