Back to the journalKeypoint Annotation for Pose Estimation: A Practical Guide
Short answer: keypoint annotation marks specific points on a person or object, such as joints, fingertips or facial landmarks, and connects them into a skeleton. Pose estimation models learn from these points to track body movement, gestures and hand-object interaction. Each keypoint usually also gets a visibility flag: visible, hidden but inferable, or not labeled.
Common keypoint layouts
| Layout | Points | Used for |
|---|---|---|
| COCO body | 17 (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) | Human pose, sports, fitness, safety |
| Hand | 21 per hand is a common layout (wrist plus 4 points per finger) | Gestures, hand-object interaction, robotics |
| Face landmarks | Varies by project | Face alignment, expression, AR |
| Custom objects | Defined per project | Vehicle corners, tool tips, animal pose |
Visibility flags
Most formats store each keypoint with a flag. A common pattern:
- Visible: the point can be seen.
- Occluded: hidden, but its position can be reasonably inferred.
- Not labeled: outside the image or impossible to place.
Agree on when an annotator should guess a hidden point and when to leave it unlabeled. Inconsistent guessing is one of the biggest sources of noise in pose data.
Edge cases to write rules for
- Left vs right: always the person's own left and right, not the viewer's.
- Back views: face points are hidden; shoulders and hips still visible.
- Self-occlusion: an arm crossing the body.
- Crowds: which person a hand belongs to.
- Motion blur: minimum clarity before placing a point.
- Clothing: loose clothing hides joint positions.
Keypoints in video
In video, keypoints are tracked across frames. Check that points move smoothly and that left and right do not swap when a person turns.
QA checks for keypoint data
- Distance error: compare points between two annotators; large gaps flag unclear rules.
- Left/right swap check: a common, easy-to-miss error.
- Skeleton sanity: limb lengths should be consistent for the same person across frames.
- Visibility consistency: the same situation should get the same flag.
Sources
- COCO keypoint layout as documented in the Ultralytics COCO-Pose dataset docs (17 keypoints per person).
Have a dataset in mind?
Start with a Free Calibration Pilot