Back to the journalBounding Box vs Polygon vs Segmentation: Which to Use
Short answer: use bounding boxes when your model only needs to find and count objects; use polygons when object shape matters but you do not need every pixel; use segmentation masks when the model must know exactly which pixels belong to which object or surface. More detail means more labeling effort, so pick the simplest type your model actually needs.
The three types at a glance
| Bounding box | Polygon | Segmentation mask | |
|---|---|---|---|
| What it marks | A rectangle around the object | An outline made of points | Every pixel, assigned to a class or instance |
| Shape accuracy | Low (includes background) | Medium to high | Highest |
| Effort per object | Lowest | Medium | Highest |
| Typical models | Object detection | Detection and shape-aware models | Semantic, instance and panoptic segmentation |
| Good for | Counting, tracking, detection | Irregular objects, defects, rooftops | Drivable area, medical regions, precise boundaries |
Bounding boxes
A box is drawn with two corners. It is fast, easy to review and works well for most detection tasks.
Use it when:
- objects are roughly rectangular or compact (vehicles, people, packages);
- the model needs location and class, not exact shape;
- you need large volumes quickly.
Watch out for: thin, diagonal or overlapping objects. A box around a diagonal pipe is mostly background, which can confuse the model. Rotated (oriented) boxes help for aerial and document images where objects sit at angles.
Polygons
A polygon traces the object's outline with points. It captures shape without labeling every pixel.
Use it when:
- objects are irregular (defects, crops, building footprints, tools);
- overlapping objects need to be told apart;
- a box would include too much background.
Watch out for: point density. Rules should say how closely the outline must follow the edge, or two annotators will draw very different shapes for the same object.
Segmentation masks
Segmentation assigns pixels to classes.
- Semantic segmentation: every pixel gets a class (road, sidewalk, sky). Two cars are both just "car".
- Instance segmentation: each object gets its own mask, so car 1 and car 2 are separate.
- Panoptic segmentation: combines both: countable objects as instances, and background areas (road, sky) as classes.
Use it when: exact boundaries change the model's decision, such as drivable space, medical regions of interest, or surface defects measured by area.
Watch out for: effort and edge rules. Hair, glass, shadows and motion blur need written rules, or masks become inconsistent.
Keypoints: the fourth option
Keypoints mark specific points such as joints, hand landmarks or facial landmarks. Use them for pose, gesture and hand-object interaction tasks. They are often combined with boxes.
How to choose: four questions
- What decision will the model make? Detection needs boxes; area or shape decisions need polygons or masks.
- How irregular are the objects? The less rectangular, the more a polygon or mask helps.
- How much data do you need? Boxes scale fastest; consider masks only for the subset that needs them.
- How will you check quality? Boxes and masks can be compared with IoU (intersection over union); agree on a minimum overlap before work starts.
A common mixed approach
Many projects label everything with boxes, then add polygons or masks only to the classes or images that need shape detail. Starting with a short pilot on a few hundred images shows which type gives your model the most benefit for the effort.
Have a dataset in mind?
Start with a Free Calibration Pilot