Pose estimation is the computer vision task of locating the keypoints of a person, animal, hand or object in an image or video, so that its posture and movement can be described. For a human body, the keypoints are usually joints such as the shoulders, elbows, wrists, hips, knees and ankles, and facial points such as the eyes and nose. Connecting the points gives a stick-figure skeleton. Pose estimation turns pictures into compact geometric data that programs can analyze, and it supports fitness apps, animation, sports science, robotics and human-computer interaction. It is related to object detection and image segmentation, but it focuses on the structure inside an object, not just its location or outline.
Overview
A pose is described by a set of keypoints and the connections between them. Pose estimation comes in several forms.
- 2D pose estimation. Predicts keypoint positions in image coordinates, as pixel locations.
- 3D pose estimation. Predicts positions in three-dimensional space, which gives depth and body orientation. It may use several cameras, depth sensors or a single image with learned priors.
- Single-person and multi-person estimation. Multi-person methods must also decide which keypoints belong to which person.
- Hand, face and whole-body estimation. Dense sets of keypoints track fingers, facial expressions or the entire body together.
- Animal and object pose estimation. The same idea extends to animals, vehicles and tools, for example to find the 6D pose (position and orientation) of an object a robot must grasp.
History
Early approaches modeled the body as a collection of parts linked by springs, called pictorial structures, and searched for the arrangement that best fit the image. Motion capture in film and sports used reflective markers and special cameras. In 2010, Microsoft’s Kinect brought depth-based skeleton tracking to consumers, using a classifier trained on depth images to label body parts [3]. The deep learning era began around 2014 with networks that regressed or predicted heat maps of keypoints. In 2017, OpenPose showed that a network could detect many people in real time using part affinity fields, which encode the direction of limbs to link keypoints into skeletons [1]. Later systems improved accuracy with higher-resolution representations and transformer models, and lightweight models now run on phones and in browsers.
Methods
### Top-down
Top-down methods first detect each person with an object detector, crop the region and then predict keypoints within each box. They are usually accurate, but their running time grows with the number of people, and they depend on the detector.
### Bottom-up
Bottom-up methods predict all keypoints in the image at once, then group them into individuals. OpenPose belongs to this group. They run at a steady speed regardless of crowd size, but grouping can be difficult in crowded scenes.
### Heat maps and regression
Most modern networks output a heat map for each keypoint, a picture in which brightness shows the likelihood of the keypoint being at each location. The peak gives the position. Some methods instead regress coordinates directly, which is simpler and faster but sometimes less accurate.
### Lifting 2D to 3D
A common way to get 3D pose from one camera is to estimate the 2D keypoints first and then “lift” them to 3D using a second network trained on motion-capture data. Other methods fit a body model, such as a parametric mesh of the human shape, to the image. These approaches rely on convolutional networks or transformers and on large labeled data sets.
### Temporal models
In video, models use several frames to smooth results, handle occlusion and resolve depth ambiguity, and may combine pose estimation with object tracking.
Data sets and evaluation
The COCO keypoints data set, with 17 labeled points per person in everyday photos, is a standard benchmark for 2D pose [2]. Others cover sports, crowded scenes and 3D motion captured in studios. Accuracy is measured with the percentage of correct keypoints within a distance threshold, with object keypoint similarity-based average precision for COCO, and, for 3D, with the mean per-joint position error in millimeters. Real-world performance can be lower than benchmark results when clothing, camera angles or lighting differ from the training data.
Applications
- Fitness and rehabilitation. Counting repetitions and checking form with a phone camera, and measuring movement in physical therapy.
- Sports analytics. Studying technique and injury risk by measuring joint angles.
- Animation and gaming. Driving avatars and characters from a performer’s movement, related to AI avatars and digital humans.
- Human-robot interaction. Letting collaborative robots understand where workers are and what gestures mean.
- Healthcare and elder care. Detecting falls or analyzing gait, with strict privacy safeguards.
- Retail and safety. Studying shopper behavior or checking safe lifting posture in workplaces.
- Autonomous driving. Predicting what pedestrians will do next from their posture; see self-driving cars.
- Augmented and virtual reality. Tracking hands and bodies for control without controllers.
- Sign language and gesture recognition. Reading hand and body keypoints as input.
Challenges
- Occlusion and self-occlusion. Hidden or overlapping limbs are hard to locate.
- Depth ambiguity. A single 2D image can correspond to several 3D poses.
- Unusual poses and clothing. Yoga, dance, loose garments and uncommon viewpoints lower accuracy.
- Crowds. Grouping keypoints for many nearby people is difficult.
- Speed and hardware. Real-time mobile use needs compact models, often via quantization.
- Bias. Systems trained on limited body types, skin tones or ages may perform unevenly.
- Privacy. Skeleton data is less identifying than video, but movement patterns can still reveal identity and health information, so consent and data protection matter.
Body models and representations
Some systems describe a person not only with keypoints but with a full parametric body model. These models represent the shape of a body through a small number of parameters, and its pose through joint angles, so that a single set of numbers can generate a three-dimensional mesh of the whole body. Fitting such a model to an image gives richer information than a skeleton, including body shape and surface, and supports applications such as virtual try-on and realistic animation. Hands and faces have their own models, and whole-body versions combine all three.
Building a pose estimation application
Developers can use pretrained models that run on phones, in web browsers or on servers, and that return keypoints with confidence scores. A typical application adds a layer of logic on top: computing angles between joints, smoothing results over time to remove jitter, and applying rules, such as checking that a knee does not pass a certain angle during a squat. Testing with people of different sizes, clothing and environments is important, as is giving users clear feedback when the system is unsure instead of presenting a guess as fact.
Responsible use
Because pose data describes how a person moves, it can reveal health conditions, fatigue or identity, and it should be treated with care. Responsible deployments obtain clear consent, process data on the device where possible, keep only what is needed, and avoid using posture or movement to make high-stakes judgments about people, such as in hiring or insurance, without strong evidence and human oversight.
FAQs
What is pose estimation? It is the task of finding the positions of key points, such as body joints, in images or video to describe a person’s posture and movement.
What is the difference between 2D and 3D pose estimation? 2D estimation gives keypoint positions within the image. 3D estimation gives positions in space, including depth.
What is OpenPose? OpenPose is an open-source system from Carnegie Mellon University, released in 2017, that detects the poses of multiple people in real time.
What is the difference between top-down and bottom-up pose estimation? Top-down methods detect each person first and then find their keypoints. Bottom-up methods find all keypoints first and then group them into people.
Where is pose estimation used? It is used in fitness apps, sports analysis, animation, robotics, healthcare monitoring and augmented reality.
More in Computer Vision
Key facts
- OpenPose, released by Carnegie Mellon University researchers in 2017, was the first open-source system for real-time multi-person 2D pose detection1.
- The COCO keypoints data set labels 17 body keypoints such as shoulders, elbows and knees for people in everyday images2.
- Microsoft's Kinect, released in 2010, estimated the skeleton of players from depth images in real time for gaming3.
Related concepts
Quick checkWhat is pose estimation?Show answer
Pose estimation is a computer vision task in which a model finds the positions of keypoints, such as the joints of a person's body, in an image or video in order to describe posture and movement.
Sources
- Zhe Cao et al., Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, https://arxiv.org/abs/1611.08050
- Tsung-Yi Lin et al., Microsoft COCO: Common Objects in Context, https://arxiv.org/abs/1405.0312
- Jamie Shotton et al., Real-Time Human Pose Recognition in Parts from Single Depth Images, CVPR 2011, https://www.microsoft.com/en-us/research/publication/real-time-human-pose-recognition-in-parts-from-a-single-depth-image/
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
