How to Choose an AI Data Annotation Vendor

Define your methodology before you compare vendors

Vendors can only be compared fairly when they are working to the same definition of a correct label. If each one receives a slightly different brief, you end up comparing their interpretations rather than their execution. Your methodology is the fixed reference point.

Label taxonomy

List every class, attribute and relationship your model needs, with a short definition and a positive and negative example for each. Decide early whether classes are mutually exclusive, whether attributes such as occlusion or truncation are recorded, and how unknown or ambiguous objects are handled. A taxonomy that is clear on paper removes a large share of later disagreements.

Edge-case rubric

Your hardest data decides model quality, so write down how to treat it: partially visible objects, reflections, motion blur, crowded scenes, sarcasm or mixed sentiment in text, overlapping entities. A good rubric shows the case, the decision and the reason, so a new annotator can apply it without asking.

Versioned guidelines and rubric ownership

Guidelines change as you learn. Give every change a version number and a date, and record which batches were labeled under which version. Then agree who owns rubric changes: usually a named person on your side approves them, while the vendor proposes changes based on what annotators find. Ask each vendor how quickly an approved change reaches every annotator, and how they confirm it has been understood.

Calibration before scale

A calibration pilot is a small, scored batch of your own data labeled before any volume commitment. Whether a vendor offers it free or charges for it, it should come first. It is the cheapest point at which to find misunderstandings, and the only reliable way to see how a team handles your edge cases rather than a vendor's demo set.

What to measure in a pilot

Accuracy against a gold set: a subset you or a domain expert labeled carefully, used as the answer key.

Agreement between annotators: how consistently different people label the same items.

Edge-case handling: how the team treats the cases in your rubric, and which new ones they flag.

Question quality: whether the vendor asks precise questions about the guideline, which shows they read it closely.

Turnaround and communication: how long feedback takes to show up in the next batch.

Comparing vendors on the same sample

Give every shortlisted vendor the identical pilot sample, the same guideline version and the same timeline. Include your hardest items, not just typical ones. Score all returns with the same script and the same reviewer. A shared sample turns a subjective sales comparison into a measured one, and it also shows how each vendor responds to the same piece of feedback.

QA metrics that matter

Quality claims are only useful when you know how they were measured. Ask every vendor to explain each metric they report: what was compared, on which sample, at what threshold, and who did the checking.

Inter-annotator agreement

Inter-annotator agreement (IAA) measures how often independent annotators reach the same label. It can be reported as simple percentage agreement or as chance-corrected statistics such as Cohen's or Fleiss' kappa. Ask which one a vendor uses and treat any IAA figure as a measured quality target on a defined sample, not a promise about every item.

IoU for boxes and masks

Intersection over Union (IoU) compares the overlap between a submitted box or mask and a reference. Ask which IoU threshold counts as a match, whether it differs by class or object size, and how small or thin objects are treated. As with agreement, an IoU figure is a measured quality target that should be checked on your own pilot data.

Temporal IoU and keypoint error

For video events and actions, temporal IoU (tIoU) measures how well predicted start and end times overlap the reference boundaries. For pose and landmark work, keypoint error is usually expressed as a normalized distance between a placed point and the reference, alongside visibility flag accuracy. Ask how boundary rules are defined before these numbers are calculated.

Adjudication, sampling and audit rates

Ask how disagreements are resolved, who has the final word, and whether decisions are fed back into the guideline. Ask what share of each batch an independent reviewer audits, how the sample is drawn, and what happens when a batch fails. Finally, ask to see an example QA report so you know what you will receive with each delivery.

For a deeper look at why these checks shape model performance, see why training data quality matters.

Native tooling and data formats

Where the work happens affects speed, quality and your ability to move later. Some vendors work only in their own platform; others work directly in the tools your team already uses.

Working in your tools

Ask whether the vendor can work natively in CVAT, Roboflow, SuperAnnotate or Labelbox, inside your workspace and under your access rules. Working in your own instance keeps the data, the label history and the QA comments in one place you control.

Export formats and avoiding lock-in

Confirm the exact export formats you need, such as COCO JSON, YOLO, CVAT XML or a project-specific JSON schema, and ask for a sample export during the pilot. Check that class IDs, attributes, track IDs and timestamps survive the export. If labels can only leave a platform in a proprietary format, switching vendors later becomes expensive.

For format detail by modality, see our image annotation guide to boxes, polygons and segmentation.

Tracking and temporal work are covered on the video annotation page.

For NER, intent and LLM evaluation, see text annotation.

Security and workforce

Your training data is often sensitive and always valuable. Security and workforce questions deserve the same attention as quality metrics.

NDA, role-based access and data handling

Ask whether every annotator signs an NDA, how access is granted and removed, and whether permissions are role-based so people see only the projects they work on. Ask where data is stored, whether it can be downloaded to personal devices, and how it is deleted at the end of the project. A reasonable standard is strict NDA, role-based access, and your own compliance requirements respected.

Workforce stability and turnover

Annotators improve as they learn your rubric. When people rotate often, that learning is lost and calibration has to start again. Ask whether the same people stay on your project, how new members are trained and calibrated, and what happens to quality when someone leaves.

How to ask about it

Ask for specifics rather than assurances: who exactly will work on your data, how many of them were on the pilot, and how access is reviewed. Vague answers here are a signal in their own right.

If your project spans a particular domain, browse annotation by industry to see the edge cases that matter there.

Choosing the model that fits your data

There is no single right vendor model for every project. The right one is the one that performs on your hardest data, reports quality in a way you can verify, works in your tools and protects your data. A shared pilot sample and the questions above will show you which one that is.