Back to the journalRLHF Data Annotation: How Human Preference Data Is Made
Short answer: RLHF (reinforcement learning from human feedback) data annotation is the work of collecting human judgments about language model outputs. People write example answers, compare or rank several model responses to the same prompt, and score them against a rubric. These judgments train a reward model, which then guides further training so the language model produces answers people prefer.
Where RLHF comes from
The approach was described in OpenAI's paper Training language models to follow instructions with human feedback (Ouyang et al., 2022), the work behind InstructGPT. Its opening line makes the point: "Making language models bigger does not inherently make them better at following a user's intent." The paper describes three steps:
- Demonstrations: human labelers write examples of the desired behavior, and a pretrained model is fine-tuned on them.
- Comparisons: labelers compare several model outputs for the same prompt; these preferences train a reward model.
- Reinforcement learning: the model is further tuned (using PPO) to produce outputs the reward model scores highly.
Steps 1 and 2 are human annotation work.
Types of RLHF and LLM annotation tasks
| Task | What the annotator does |
|---|---|
| Demonstration writing | Writes an ideal answer to a prompt |
| Pairwise preference | Picks the better of two responses |
| Ranking | Orders several responses from best to worst |
| Rubric scoring | Rates a response on helpfulness, accuracy, safety, tone |
| Factuality checking | Verifies claims against sources and flags errors |
| Red-teaming | Writes prompts designed to expose unsafe or wrong behavior |
| Rewrite / critique | Explains what is wrong and writes an improved answer |
Why rubrics matter
Preference data is only as good as the shared definition of "better". Without a rubric, one rater rewards long answers, another rewards short ones, and the reward model learns noise. A strong rubric:
- defines each criterion (for example accuracy, helpfulness, safety, instruction following) with examples;
- says which criterion wins when they conflict;
- explains how to handle ties and "both bad" cases;
- includes worked examples of borderline cases.
Quality controls for preference data
- Calibration rounds: raters score the same items, discuss disagreements and update the rubric.
- Gold items: items with agreed answers mixed into batches to check accuracy.
- Agreement tracking: how often raters pick the same winner, per task type.
- Reviewer audits: senior reviewers re-check a sample, especially low-agreement items.
- Rater feedback loops: individual feedback when a rater drifts from the rubric.
- Domain matching: coding, legal or medical prompts go to raters with matching skills.
Common mistakes
- Treating preference work as simple clicking instead of careful reading.
- Letting response length bias choices.
- Not checking facts in confident-sounding answers.
- Changing the rubric mid-project without re-calibrating raters.
- Mixing very different task types in one queue.
RLHF vs other text annotation
Classic text annotation (entities, intent, sentiment) labels fixed data. RLHF annotation judges model behavior, so the data changes as the model improves, and rubrics often need updating between rounds.
Sources
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
Have a dataset in mind?
Start with a Free Calibration Pilot