Back to the journalHow to Write Annotation Guidelines Annotators Follow
Short answer: good annotation guidelines define every class in one clear sentence, show correct and incorrect examples, give a rule for every common edge case, explain what to do when unsure, and are versioned so changes are tracked. Most labeling quality problems come from unclear guidelines, not from careless annotators.
Why guidelines matter
Two skilled annotators given a vague instruction will make different, reasonable choices. The model then learns both choices as noise. A clear rulebook turns many individual judgments into one consistent dataset.
A guideline outline you can copy
- Purpose: what the model will do with the labels (one paragraph).
- Output format: tool, label type (box, polygon, tag), export format.
- Class list: each class with:
- a one-sentence definition;
- 2 or 3 correct examples;
- 1 or 2 "looks similar but is not" examples.
- General rules: minimum size, how tight boxes must be, how to treat partly visible objects.
- Edge cases: one rule per situation, with a picture or text example.
- When unsure: a clear default ("label as uncertain", "skip", or "flag for review").
- Quality bar: how work will be checked and what counts as an error.
- Change log: version number, date and what changed.
Writing rules that work
- Use decisions, not descriptions. "If more than half the vehicle is hidden, still draw the box around the visible part" is better than "handle occlusion carefully".
- One rule, one situation. Long paragraphs hide the rule.
- Show pictures or real text samples for every tricky case.
- Prefer observable criteria (size in pixels, percentage visible) over opinions ("clearly visible").
- Define the default for anything not covered.
Edge cases to cover in most projects
| Data | Typical edge cases |
|---|---|
| Images | Occlusion, truncation at image border, reflections, blur, tiny objects, groups |
| Video | Objects leaving and re-entering, ID switches, action boundaries |
| Text | Sarcasm, mixed intents, abbreviations, multiple languages |
| Audio | Overlapping speech, background noise, inaudible words |
| 3D / LiDAR | Sparse distant points, heading of stopped objects, articulated vehicles |
Test the guidelines before scaling
- Have 3 to 5 annotators label the same 50 to 100 items.
- Compare results and measure agreement.
- Find where they disagree; each disagreement is either a missing rule or an unclear one.
- Update the guidelines and repeat until agreement is stable.
Keep guidelines alive
- Log every question annotators ask; repeated questions mean a missing rule.
- Version every change and tell the whole team.
- Re-check a sample of earlier work when a rule changes, so old and new labels stay consistent.
Have a dataset in mind?
Start with a Free Calibration Pilot