Calibration pilot
Data Annotation Pilot Project: How the Calibration Pilot Works
Start smaller than the backlog. A pilot on your own data shows the process and the labels before the work scales.
01 / Definition
What the pilot is
A calibration pilot is a small first batch on your own data used to agree labelling rules and edge cases before a project scales.
A pilot usually needs a few hundred representative items—not your full dataset. We ask for a sample that includes your common cases and your hardest edge cases, so the calibration batch reflects the real work.
02 / Steps
How it works
- Run it small. We start with a pilot batch of your real data, so you can see the process and the labels before the work scales.
- Agree the process. We settle how data is shared, who can access it, how questions are handled, and how completed work is delivered.
- Calibrate on real examples. Annotators and reviewers label a representative sample, compare decisions, and resolve unclear guidance before scale.
- Review with two-stage QA. Every batch passes an annotator self-check, then an independent reviewer audit.
- Scale what worked. Once the pilot output meets your standard, the same reviewed process carries the full workload.
Typical length
A pilot is usually turned around in days to a couple of weeks depending on task complexity.
03 / Deliverables
What you receive
- Labelled output: a reviewed pilot batch
- A list of disagreements
- An updated decision guide
- A recommendation: lock and scale, revise the ontology, or stop because the task is still undefined
04 / Quality
Pilot QA rubric
| Criterion | What we check | How it is reported |
|---|---|---|
| Label accuracy against the written rules | Each label is checked against the written guide in two stages: an annotator self-check, then an independent reviewer audit. | Reviewed pilot batch |
| Class consistency | Annotators and reviewers label the same representative sample and compare their decisions. | List of disagreements |
| Edge-case handling | The sample includes your hardest edge cases; resolved questions are added to the working guide. | Updated decision guide |
| Turnaround and communication | How data is shared, who can access it and how questions are handled are settled before work starts. | Recommendation: lock and scale, revise the ontology, or stop |
Pilot FAQ
How much data do we need for a pilot?
A pilot usually needs a few hundred representative items—not your full dataset. We ask for a sample that includes your common cases and your hardest edge cases, so the calibration batch reflects the real work.
How long does a pilot take?
A pilot is usually turned around in days to a couple of weeks depending on task complexity. Production timelines depend on volume and review depth, and a schedule is agreed once the pilot output meets your standard.
What does a pilot include?
A representative slice of the real data, a first guide, labelled output, a list of disagreements, and a recommendation: lock and scale, revise the ontology, or stop because the task is still undefined. A pilot is a decision, not a discount on clicking.
Can you work with our existing guidelines and ontology?
Yes. We start from your existing guide, test it on a calibration batch, and surface conflicts or gaps with documented examples before scaling. If you have no guide yet, we help build one from your model's decisions.