Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Object Tracking

nounComputer Visionalso video tracking, multi-object tracking, MOT

In one line

Object tracking is the computer vision task of following one or more objects across the frames of a video while keeping a consistent identity for each.

Object tracking is the computer vision task of following one or more objects through the frames of a video while keeping a consistent identity for each of them. A tracker answers questions such as where a particular car is in every frame, how a player moves across a field, or how many separate people entered a store. It builds on object detection, which finds objects in single images, and adds the dimension of time. Tracking is central to video analysis, robotic perception, driver assistance and sports analytics.

Overview

A video is a sequence of images, and the same object may look different in each one because of motion, lighting, viewpoint, blur and partial occlusion. A tracker must decide which detection in the current frame corresponds to which object in the previous one, which is called data association. It should also cope with objects entering and leaving the scene, disappearing behind obstacles and crossing paths with similar-looking objects.

Tracking tasks fall into two broad groups.

  • Single-object tracking. The user marks one target in the first frame, and the system follows it afterward. The target class is not fixed in advance, so the tracker learns its appearance on the fly.
  • Multi-object tracking. The system detects and follows many objects of known classes, such as pedestrians or vehicles, and gives each an identity number.

Related tasks include video object segmentation, which tracks pixel masks, and pose estimation over time, which follows body keypoints.

History

Early tracking relied on mathematical filters. The Kalman filter, introduced in 1960, estimates the state of a moving object, such as its position and velocity, from noisy measurements and predicts where it will be next [1]. Particle filters later allowed more flexible models. Appearance-based trackers followed, using color histograms and templates, with the mean-shift algorithm as a classic example. After 2010, correlation filters offered fast and accurate single-object tracking. The deep learning period brought trackers based on convolutional networks, Siamese networks that compare a target template with search regions, and later transformer trackers. For multiple objects, tracking by detection became the dominant approach.

Main approaches

### Tracking by detection

In this approach, a detector finds objects in each frame, and a separate step links them over time. SORT, published in 2016, used a Kalman filter to predict each object’s next position and the Hungarian algorithm to match predictions with new detections based on overlap, which made tracking simple and fast [2]. DeepSORT added a learned appearance descriptor, so that objects could be re-identified after being hidden. Later methods such as ByteTrack improved association by also using low-confidence detections, and others use graph or transformer models for matching.

### Joint detection and tracking

Some models predict detection and association together, sharing features to save computation and improve consistency. Transformer-based trackers keep a set of query vectors that carry each object’s identity from frame to frame.

### Siamese and transformer trackers for single objects

Siamese trackers learn a similarity function between the first-frame template and candidate regions in later frames. Transformer trackers use attention to relate the template to the search area with strong results on difficult sequences.

### Classical trackers

Methods such as Kalman filters, optical flow and correlation filters are still used where computing power is limited or where the motion model is simple, and they often form part of modern pipelines.

Key ideas

  • Motion model. A prediction of where an object will be, such as constant velocity.
  • Appearance model. A description of what the object looks like, used to tell similar objects apart. It is often an embedding.
  • Data association. The matching of detections to tracks, using motion, appearance and overlap, often solved as an assignment problem.
  • Track management. Rules for starting new tracks, keeping lost ones for a short time and ending them.
  • Re-identification. Recognizing an object that reappears after occlusion or in another camera view.

Evaluation

Single-object trackers are scored by overlap between predicted and true boxes over time, and by robustness, meaning how often they lose the target. Multi-object tracking uses metrics such as MOTA, which counts misses, false alarms and identity switches, IDF1, which measures how well identities are kept, and HOTA, which balances detection and association quality. The MOTChallenge benchmarks supply data and standard metrics for tracking pedestrians, including crowded scenes [3].

Applications

  • Autonomous driving. Following vehicles, cyclists and pedestrians to predict their paths; see self-driving cars.
  • Robotics and drones. Keeping a target in view or following a person; see drone AI.
  • Video surveillance and traffic analysis. Counting, measuring dwell time and detecting unusual movement, subject to privacy limits.
  • Sports analytics. Tracking players and the ball to produce statistics and tactical insight.
  • Retail and smart buildings. Understanding movement through stores and spaces.
  • Wildlife and biology. Following animals or cells in microscope videos.
  • Augmented reality and filmmaking. Anchoring graphics to moving objects.
  • Healthcare. Tracking surgical instruments or patient movement.

Challenges

  • Occlusion. Objects hidden behind others may lose their identity when they reappear.
  • Similar appearance. Crowds of similar-looking people or vehicles cause identity switches.
  • Camera motion and scale change. Moving cameras and fast zooms complicate prediction.
  • Real-time constraints. Tracking must keep up with the frame rate, often on embedded hardware.
  • Long-term tracking. Over minutes or hours, drift accumulates and appearance changes.
  • Privacy and misuse. Tracking people raises concerns about surveillance, and rules differ widely between places.

A typical tracking pipeline

A practical multi-object tracker usually runs in a loop. For each new frame, the detector produces boxes with class labels and confidence scores. The motion model predicts where each existing track should be. The system computes a cost for pairing each prediction with each detection, using box overlap and appearance similarity, and solves the assignment. Matched tracks are updated, unmatched detections start new tentative tracks, and tracks that stay unmatched for several frames are deleted. Output is a list of identity numbers with their boxes in every frame, which downstream software can use to count objects, measure speed or draw trajectories.

Tracking across cameras

In large spaces such as airports, campuses and cities, one camera is not enough, so systems attempt multi-camera tracking. They must recognize that an object leaving one view is the same one entering another. This uses re-identification features, geometry about where cameras are placed and timing constraints. The task is hard because lighting and viewpoint change between cameras, and it also raises particularly sharp privacy concerns when applied to people, which is why many deployments limit it to vehicles or to anonymous counts.

Design advice

Practitioners should match the method to the setting. A simple filter-based tracker with a good detector is often enough for a fixed camera and slow motion, and it is easy to debug. Crowded or fast scenes benefit from appearance models and better association. Whatever the choice, it is wise to evaluate on video that resembles the real deployment, to inspect identity switches visually and to log tracking failures so that detector or association settings can be tuned over time.

FAQs

What is object tracking? It is the task of following objects through a video, estimating their positions in each frame and keeping a consistent identity for each.

What is the difference between object detection and object tracking? Detection finds objects in a single image. Tracking connects detections across frames so that each object keeps the same identity.

What is multi-object tracking? It is the tracking of many objects at once, such as all the people in a scene, each with its own identity.

What is the Kalman filter used for in tracking? It predicts an object’s next position from its past motion and combines this with new measurements to reduce noise.

What is SORT? SORT is a fast tracking-by-detection method that uses a Kalman filter and the Hungarian algorithm to link detections across frames.

Key facts

  • The Kalman filter, published by Rudolf Kalman in 1960, is a recursive method for estimating the state of a moving system from noisy measurements and is widely used in tracking1.
  • The SORT algorithm, introduced in 2016, combined a Kalman filter with the Hungarian algorithm for fast online multi-object tracking by detection2.
  • The MOTChallenge benchmarks provide standard data sets and metrics for evaluating pedestrian tracking methods3.

Related concepts

Quick checkWhat is object tracking?Show answer

Object tracking is a computer vision task in which a system follows one or more objects across the frames of a video, estimating their positions over time and keeping a consistent identity for each object.

Sources

  1. Rudolf Kalman, A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering, https://doi.org/10.1115/1.3662552
  2. Alex Bewley et al., Simple Online and Realtime Tracking, https://arxiv.org/abs/1602.00763
  3. Patrick Dendorfer et al., MOT20: A benchmark for multi object tracking in crowded scenes, https://arxiv.org/abs/2003.09003

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue