Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Data Annotation

nounAI Data and Infrastructurealso data tagging, annotation, dataset annotation

In one line

Data annotation is the process of adding labels, tags or other descriptive information to raw data such as images, text, audio and video so that machine learning models can learn from it.

Data annotation is the process of adding labels, tags or other descriptive information to raw data so that machine learning models can learn from it. A photograph becomes useful for teaching an object detector once someone has drawn boxes around the cars and pedestrians in it. A paragraph becomes training material for a sentiment model once it is marked positive or negative. Annotation turns raw data into training data, and in supervised learning it is often the slowest and most expensive part of a project [1]. It is closely related to data labeling, and the two terms are often used interchangeably.

Why annotation matters

A model can only learn what its labels teach it. If annotations are inconsistent, mistaken or biased, the model will absorb those flaws. Many practitioners therefore treat annotation as an engineering discipline of its own. Clear guidelines, trained annotators, quality checks and good tools often improve results more than changes to the model, an idea at the heart of the data-centric approach to AI. Annotation also produces the reference answers used for testing data and evaluation.

Types of annotation

### Images and video

  • Classification. Assigning one or more tags to a whole image, such as “cat” or “damaged roof.”
  • Bounding boxes. Rectangles around objects, used for object detection.
  • Polygons and masks. Precise outlines or pixel-level labels for image segmentation.
  • Keypoints. Marked points such as joints on a body, used for pose estimation.
  • Tracking. Following the same object across video frames; see object tracking.
  • Captions. Sentences describing a scene, used for image captioning.

The COCO data set, with about 330,000 images carrying object outlines and captions, shows how much human effort large vision data sets require [2].

### Text

  • Classification. Labeling documents by topic, intent or sentiment.
  • Named entities. Marking people, places and organizations; see named entity recognition.
  • Part-of-speech and syntax. Grammatical labels for language research.
  • Relations and coreference. Links between entities and mentions.
  • Ranking and preference. Comparing model answers, used to align large language models with human preferences.

### Audio and speech

Transcription of speech to text, speaker identification, marking of sound events and labeling of emotion or intent, used in speech recognition.

### Sensor and 3D data

Lidar point clouds for autonomous vehicles are annotated with 3D boxes and labels for each point, a demanding task that supports self-driving cars.

The annotation workflow

  1. Define the task. Decide the label set, edge cases and what counts as correct.
  2. Write guidelines. Include examples and rules for ambiguous cases.
  3. Select and train annotators. Experts for specialized fields, such as radiologists for medical scans, and trained crowd workers for common tasks.
  4. Pilot. Annotate a small batch, measure agreement and revise the guidelines.
  5. Annotate at scale. Using a tool with shortcuts, pre-labeling and review queues.
  6. Review and quality control. Spot checks, second opinions and adjudication of disagreements.
  7. Deliver and version. Export in standard formats such as COCO or JSON and track versions.

Quality control

Quality is usually measured by inter-annotator agreement, the extent to which different annotators give the same label to the same item. Statistics such as Cohen’s kappa correct for agreement that would happen by chance [3]. Other methods include inserting items with known answers (gold standards), consensus voting among several annotators, and audits by senior reviewers. Low agreement often points to unclear guidelines instead of careless workers.

Automation and model-assisted labeling

Manual annotation does not scale easily, so teams reduce the work with several techniques.

  • Pre-labeling. A model proposes labels and humans correct them, which is typically faster than labeling from scratch.
  • Active learning. The model picks the most informative examples for people to label.
  • Weak supervision. Rules or heuristics generate noisy labels that are combined statistically.
  • Self-supervised learning. Models learn from structure in unlabeled data.
  • Synthetic data. Simulations produce examples with exact labels.
  • LLM-assisted labeling. Large language models label text, with human review to catch errors.

Tools and platforms

Annotation tools range from open-source programs, such as Label Studio and CVAT, to commercial platforms and managed services that supply both software and workforce. Features to look for include support for the needed data types, collaboration, review workflows, automation and secure handling of sensitive data.

Challenges and ethics

  • Cost and time. Large projects need thousands of hours of work.
  • Subjectivity. Tasks such as judging toxicity or emotion have no single right answer, and differences between annotators are meaningful.
  • Bias. Annotator backgrounds and instructions shape labels and, through them, model behavior; see AI bias.
  • Labor conditions. Much annotation is done by contract workers, some paid low wages, and some exposed to disturbing content when labeling harmful material. Responsible organizations pay fair wages and provide support.
  • Privacy and security. Images, recordings and documents may contain personal information requiring consent and protection.

Annotation guidelines in practice

The guideline document is the most important artifact in an annotation project. A good one defines every label with a plain-language description, shows positive and negative examples, and lists the awkward cases that come up again and again. For an object detection project, for example, it should say whether a partly hidden car gets a box, how tightly boxes must fit, whether reflections count and what to do when a crowd makes individual people impossible to separate. For a text project it should state whether sarcasm is labeled by its literal or intended meaning.

Guidelines are living documents. After each pilot round, teams review disagreements, decide on a rule and add it to the guide, then relabel affected items. Keeping a version history lets the team trace which rules were in force when each batch was labeled, which matters when model errors need to be explained later.

Costs and planning

Annotation cost depends on task difficulty, expertise needed, volume and quality targets. Drawing a bounding box can take seconds, while outlining every pixel of a complex street scene can take many minutes, and expert medical labeling is far more costly again. Teams usually estimate cost per item from a pilot, add a margin for review and rework, and compare it with the value of the model. Spending on a smaller, cleaner set of labels often beats collecting a larger, noisier one.

Increasingly, annotation is also used after deployment: reviewers check model outputs, correct mistakes and feed the corrections back as new training examples, forming a continuing loop of improvement. This links to human-in-the-loop AI practice and to broader data pipelines.

FAQs

What is data annotation? It is adding labels or tags to raw data such as images, text or audio so that machine learning models can learn from it.

What is the difference between data annotation and data labeling? The terms are largely interchangeable. Labeling often means assigning a category, while annotation can include richer markup such as outlines and relations.

Who does data annotation? Trained annotators, domain experts, crowd workers and increasingly AI models assisted by human reviewers.

How is annotation quality measured? Mainly by inter-annotator agreement, gold-standard checks and expert review.

Can annotation be automated? Partly. Pre-labeling, active learning and weak supervision reduce manual work, but human review remains important.

Key facts

  • Supervised machine learning depends on labeled examples, and creating those labels is often the most time-consuming part of building a model1.
  • The COCO data set provides about 330,000 images with object outlines and captions, produced through large-scale human annotation2.
  • Annotation quality is commonly measured by inter-annotator agreement, such as Cohen's kappa3.

Related concepts

Quick checkWhat is data annotation?Show answer

Data annotation is the process of adding labels, tags or other descriptive information to raw data, such as drawing boxes around objects in images or marking entities in text, so that machine learning models can learn from the labeled examples.

Sources

  1. Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, Pearson, https://aima.cs.berkeley.edu/
  2. Tsung-Yi Lin et al., Microsoft COCO: Common Objects in Context, European Conference on Computer Vision, https://arxiv.org/abs/1405.0312
  3. Ron Artstein and Massimo Poesio, Inter-Coder Agreement for Computational Linguistics, Computational Linguistics, https://doi.org/10.1162/coli.07-034-R2

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue