Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Object Detection

nounComputer Visionalso object recognition, object localization, bounding box detection

In one line

Object detection is a computer vision task that finds the objects in an image or video and gives each one a category label and a location, usually as a bounding box.

Object detection is a computer vision task in which a model finds the objects of interest in an image or video and reports both what each object is and where it is. The location is usually given as a bounding box, a rectangle drawn around the object, together with a class label such as “car” or “person” and a confidence score. Object detection goes a step beyond image classification, which assigns one label to a whole picture, and it differs from image segmentation, which outlines objects pixel by pixel. It is a core part of image recognition and underlies applications from driver assistance to retail analytics.

Overview

A detector must solve two problems at once. Classification decides what each object is. Localization determines its position and size. Real scenes may hold many objects of different sizes, partly hidden behind one another, so the number of outputs is not known in advance. This makes detection harder than classification, where a fixed set of scores is produced.

Detection is related to several other tasks.

  • Instance segmentation. Detects each object and also gives its exact pixel mask.
  • Object tracking. Follows detected objects over time in video; see object tracking.
  • Pose estimation. Finds the keypoints of a body or object; see pose estimation.
  • Open-vocabulary detection. Finds objects named in free text, even categories not seen in training.

History

Early detectors used hand-designed features. The Viola-Jones face detector in 2001 combined simple features with a cascade of classifiers and ran fast enough for cameras. Histogram of oriented gradients features with support vector machines later became a standard method for detecting pedestrians. After the deep learning breakthrough of 2012, researchers adapted convolutional neural networks to detection. R-CNN in 2014 proposed candidate regions and classified each with a network, greatly raising accuracy on the PASCAL VOC benchmark [1]. Fast R-CNN and Faster R-CNN made it much quicker. In 2016, YOLO treated detection as a single prediction problem and made real-time operation practical [2]. Transformer-based detectors such as DETR followed in 2020, and large vision-language models now enable detection guided by text prompts.

Main approaches

### Two-stage detectors

Two-stage methods first propose regions that may contain objects, then classify and refine each region. The R-CNN family is the best-known example. They tend to be accurate, particularly for small objects, but slower.

### One-stage detectors

One-stage methods predict boxes and classes directly from the image in a single pass. YOLO, SSD and RetinaNet belong to this group. They are fast and suit real-time use on video and embedded devices, and modern versions are close to two-stage models in accuracy.

### Transformer-based detectors

DETR uses a transformer to predict a set of objects directly, avoiding hand-made steps such as anchor boxes and some post-processing. Later variants improved training speed and small-object performance.

Key concepts

  • Bounding box. A rectangle given by coordinates, with a class label and confidence score.
  • Anchor boxes. Predefined boxes of various shapes that many detectors refine.
  • Intersection over Union (IoU). The area of overlap between a predicted and true box divided by the area of their union. A prediction usually counts as correct if IoU exceeds a threshold such as 0.5.
  • Non-maximum suppression. A step that removes duplicate boxes around the same object, keeping the most confident one.
  • Feature pyramids. Multi-scale features that help detect both large and small objects.

Evaluation

The standard measure is mean average precision (mAP). Average precision summarizes the trade-off between precision and recall for each class across confidence thresholds, and mAP averages this across classes. The COCO benchmark, with about 200,000 labeled images and 80 object categories, reports mAP averaged over several IoU thresholds, which rewards precise boxes [3]. Speed matters too, so papers report frames per second or latency along with accuracy.

Applications

  • Autonomous driving and driver assistance. Detecting vehicles, pedestrians, lights and signs; see self-driving cars.
  • Robotics. Finding objects to grasp or avoid; see robotic perception.
  • Surveillance and safety. Counting people, detecting intrusions or checking protective equipment, subject to privacy limits.
  • Retail and logistics. Counting stock, reading shelves and sorting parcels.
  • Healthcare. Locating lesions, cells or instruments in medical images with clinician review.
  • Agriculture and environment. Counting animals or fruit and spotting pests from drones; see drone AI.
  • Manufacturing. Finding defects and verifying assemblies.
  • Media and photography. Auto-focus, content moderation and editing tools.

Challenges

  • Small and crowded objects. Tiny or overlapping objects are easy to miss.
  • Occlusion and variation. Partial views, odd poses, lighting and weather reduce accuracy.
  • Class imbalance. Background regions far outnumber objects, which complicates training.
  • Data labeling cost. Drawing boxes on large image sets is expensive. Semi-automatic labeling and self-supervised learning help.
  • Real-time constraints. Embedded systems need small, efficient models, often produced with quantization and model distillation.
  • Robustness and bias. Detectors may perform differently across environments and populations, and are vulnerable to adversarial changes.

Training a detector

Training a modern detector needs images with annotated boxes. Developers usually begin with a model pretrained on a large data set and fine-tune it on their own images, which reduces the amount of labeling needed. Data augmentation, such as random scaling, cropping, flipping and color changes, increases variety. The loss function combines a term for classification errors with a term for box position errors. After training, the confidence threshold is tuned to balance missed objects against false alarms, since the right balance depends on the application: a safety system may accept more false alarms, while an automated checkout may not.

Detection in video

Video adds time. A detector can run on every frame, but consecutive frames are highly similar, so many systems detect on some frames and use tracking to carry objects between them, which saves computation and keeps object identities stable. Temporal models can use motion to detect partly hidden or blurred objects. Edge devices such as cameras and phones often run compact detectors, while heavier models run on servers where accuracy matters more than latency.

Ethics and governance

Detection technology can be used to count customers or protect workers, and it can also support intrusive monitoring. Responsible use includes limiting the data kept, telling people when cameras are in use, testing for uneven performance across settings and groups, and following local law, which in some places restricts the use of automated recognition in public spaces. Where a detection can lead to consequences for a person, a human should review it before action is taken.

FAQs

What is object detection? It is the task of finding objects in an image or video, labeling what they are and marking where they are, usually with bounding boxes.

What is the difference between object detection and image classification? Classification gives one label to a whole image. Detection finds multiple objects and their positions.

What is YOLO? YOLO, short for You Only Look Once, is a family of fast one-stage detectors that predict boxes and classes in a single pass.

What is IoU? Intersection over Union measures how much a predicted box overlaps a true box, and it is used to decide whether a detection is correct.

Where is object detection used? It is used in self-driving, robotics, retail, healthcare imaging, manufacturing inspection, security and many consumer apps.

Key facts

  • The R-CNN method, published in 2014, combined region proposals with convolutional network features and greatly improved detection accuracy on the PASCAL VOC benchmark1.
  • YOLO, introduced in 2016, framed detection as a single regression problem and made real-time detection practical2.
  • The COCO data set contains over 200,000 labeled images and 80 object categories and is a standard benchmark for detection3.

Related concepts

Quick checkWhat is object detection?Show answer

Object detection is a computer vision task in which a model finds each object of interest in an image or video, says what it is and marks its position, usually with a rectangular bounding box.

Sources

  1. Ross Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, https://arxiv.org/abs/1311.2524
  2. Joseph Redmon et al., You Only Look Once: Unified, Real-Time Object Detection, https://arxiv.org/abs/1506.02640
  3. Tsung-Yi Lin et al., Microsoft COCO: Common Objects in Context, https://arxiv.org/abs/1405.0312

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue