Back to the journalSemantic vs Instance vs Panoptic Segmentation Explained
Short answer: semantic segmentation gives every pixel a class (road, car, sky) but does not separate individual objects. Instance segmentation draws a separate mask for each countable object (car 1, car 2) but usually ignores background areas. Panoptic segmentation combines both: every pixel gets a class, and every countable object also gets its own ID.
Things and stuff
The difference is easiest to understand with two terms from the panoptic segmentation paper (Kirillov et al., CVPR 2019):
- Things: countable objects such as people, cars and animals.
- Stuff: amorphous regions such as road, grass, sky and water.
| Semantic | Instance | Panoptic | |
|---|---|---|---|
| Labels every pixel | Yes | No (objects only) | Yes |
| Separates objects of the same class | No | Yes | Yes (things) |
| Handles stuff (road, sky) | Yes | Usually no | Yes |
| Annotation effort | High | High | Highest |
| Typical use | Drivable area, land cover, medical regions | Counting, robotic picking, cell counting | Full scene understanding for driving and robotics |
Semantic segmentation
Every pixel is painted with one class. Two touching cars become one "car" region.
Choose it when you care about where a class is, not how many: drivable surface, crop vs weed, tumor vs healthy tissue, land cover from satellites.
Instance segmentation
Each object gets its own mask and ID, so overlapping objects can be separated.
Choose it when you need to count, measure or act on individual objects: picking items from a bin, counting cells, measuring each defect's area.
Panoptic segmentation
Combines both: stuff regions get a class, things get class plus instance ID, and no pixel is left unlabeled or labeled twice.
Choose it when a system needs a complete picture of the scene, such as autonomous driving or mobile robots.
Annotation tips for all three
- Write boundary rules: what happens at edges, shadows, reflections, glass and hair.
- Define overlap order: which object "wins" a pixel when objects overlap.
- Set a minimum size: tiny objects below a size threshold may be ignored or grouped.
- Use tools that snap to edges or offer AI-assisted masks, then review every edge by eye.
- Measure agreement with mask IoU on a sample labeled by two annotators.
Can you convert between them?
Panoptic labels can be turned into semantic labels (drop the IDs) or instance labels (keep only things). Semantic labels cannot be turned into instance labels without extra work, because object boundaries were never recorded. If you may need instance-level data later, label at that level from the start.
Sources
- Kirillov, A., He, K., Girshick, R., Rother, C., & Dollár, P. (2019). Panoptic Segmentation. CVPR 2019. arXiv:1801.00868.
Have a dataset in mind?
Start with a Free Calibration Pilot