Inter-Annotator Agreement in Data Labeling, Explained

Why agreement matters

A model learns whatever the labels say. If two annotators disagree on 20 percent of items, the model receives mixed signals on those items. Low agreement usually points to unclear rules, hard edge cases or training gaps, not just careless work. Measuring it shows where to fix the rulebook.

The main measures

Percent agreement

The share of items where annotators chose the same label. Easy to understand, but it ignores agreement that happens by chance. If 95 percent of items are "no defect", two people guessing "no defect" every time will agree most of the time without being useful.

Cohen's kappa (two annotators)

Kappa corrects for chance:

κ = (pₒ − pₑ) / (1 − pₑ)

where pₒ is the observed agreement and pₑ is the agreement expected by chance. κ = 1 means perfect agreement; κ = 0 means no better than chance. A widely cited scale (Landis and Koch, 1977) calls 0.61 to 0.80 "substantial" and 0.81 to 1.00 "almost perfect", but acceptable values depend on the task.

Fleiss' kappa (three or more annotators)

Extends the same idea to several annotators who each label the same items.

Krippendorff's alpha

Handles missing labels and different data types (categories, ordered scales, numbers). Useful when not every annotator labels every item.

IoU for spatial and time labels

For boxes and masks, agreement is measured by overlap:

IoU = area of overlap / area of union

Two boxes with IoU of 0.5 overlap by half of their combined area. For video, temporal IoU (tIoU) does the same with start and end times of an action. Teams agree on a threshold (for example 0.5 or 0.75) that counts as a match.

How to read a vendor's agreement number

A single percentage on a website is not enough to compare vendors. Ask:

1. Which measure? Percent agreement, kappa and IoU are not comparable.
2. On what data? Easy items inflate agreement; hard edge cases lower it.
3. Measured how often, and on what sample size?
4. Agreement between whom? Two annotators, or annotator vs reviewer?
5. What threshold counts as a match for boxes, masks or time spans?
6. What happens to items that disagree? Are they reviewed and used to update the rules?

How to raise agreement

• Write rules with picture or text examples for every edge case.
• Run calibration rounds: everyone labels the same items, then discusses differences.
• Add gold items with known answers to every batch.
• Track agreement per class; low-agreement classes need clearer rules.
• Update the rulebook and re-calibrate when rules change.

Agreement is not accuracy

Two annotators can agree and both be wrong if the rules are wrong. Combine agreement checks with expert review against a gold set to measure both consistency and correctness.

Sources

• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.