Preference Data

The Reward Model article described training on “human comparisons” without fully unpacking what that data actually looks like. This article covers exactly that: preference data.

The simple definition

Preference data consists of comparisons between two or more responses to the same prompt, indicating which one a human rater judged to be better. Recall from the Ground Truth article’s discussion of human preference data as a specific, subjective form of ground truth — this is precisely that data, given its own full treatment.

Rather than a single, absolute label like “correct” or “spam,” as covered throughout the Label article, preference data records a relative judgment: given two responses, which one is better, and by roughly how much.

Why this specific data format was needed

Recall from the RLHF article’s real limitation with supervised fine-tuning alone: it teaches a model to imitate example responses, but can’t easily capture graded, comparative quality judgments.

Recall also from the Ground Truth article’s discussion of human preference judgments being inherently subjective — two people can reasonably disagree about which of two well-written responses is genuinely “better.” Preference data is specifically structured to work with this subjectivity rather than fight it: instead of demanding one single, absolute “correct” answer (which often doesn’t exist for open-ended tasks like conversation), it only asks for a relative judgment between two specific options, a genuinely easier, more reliable task for a human rater to perform consistently.

flowchart LR
    A[Prompt] --> B[Response A generated]
    A --> C[Response B generated]
    B --> D[Human rater compares]
    C --> D
    D --> E["Preference label: A is better than B"]

How this actually gets collected, step by step

This is worth grounding precisely, since it’s a real, documented process across published research.

Human labelers — often organized through the same kind of annotation workforces covered in the Annotation article, sometimes drawn from companies like Scale AI and Surge AI referenced in that earlier discussion — are shown a single prompt along with two or more different model-generated responses to it, and asked to rank or choose which response better satisfies criteria like helpfulness, honesty, and harmlessness.

Recall from the InstructGPT paper’s documented process, referenced throughout this phase: OpenAI collected this exact kind of preference data from a team of roughly 40 contractors, producing tens of thousands of comparison examples used to train the reward model that guided their RLHF pipeline.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a wine tasting panel that’s never asked to assign an absolute numerical score to a wine in isolation, but is instead consistently asked “which of these two wines do you prefer?” — a comparative judgment most people can make far more reliably and consistently than trying to assign a precise, standalone score to a single wine on its own.

Where this breaks down: A wine panel’s comparative judgment still involves genuine, embodied sensory experience.

Preference data collection for language models involves human raters reading text and applying a set of specific guidelines about helpfulness and safety, exactly the kind of annotation instructions covered in the Annotation article — a more structured, criteria-driven process than simply following personal taste, precisely because the resulting data needs to be consistent enough to train a reliable reward model or DPO process from.

What preference data actually looks like as a training example

It’s worth being concrete about the real, structured format, since this shows up directly in published datasets and open-source training pipelines.

A single preference data example typically consists of: the original prompt, a “chosen” response (the one the human rater preferred), and a “rejected” response (the one they didn’t) — exactly the “chosen” and “rejected” pair referenced in the DPO article’s loss function, which directly uses this exact structure to adjust the model’s relative probabilities for the two responses.

A concrete example, layered

For a simple beginner example: given the prompt “How do I politely decline a meeting invitation?”, a human rater comparing two model responses might prefer a brief, professional template over a response that’s overly long or presumptuous about the reason for declining — producing one single preference data point: this response is “chosen,” that one is “rejected.” For a production example: published, open-source preference datasets like Anthropic’s HH-RLHF (helpful and harmless RLHF) dataset, referenced in real, published research on instruction tuning pipelines, provide exactly this kind of human comparison data publicly, letting other research teams and companies train their own reward models or run DPO without needing to collect an entirely new preference dataset from scratch.

Why the quality of preference data matters enormously

This connects directly back to a warning first raised in the Ground Truth article: ground truth is only as reliable as the process that produced it.

If human raters disagree frequently, apply inconsistent standards, or bring unrepresentative biases to their comparisons, the resulting reward model or DPO-trained model inherits those same inconsistencies and biases — precisely why real, serious preference-data collection efforts, as covered in the Annotation article, invest in clear guidelines and quality-control processes like multiple raters per comparison, rather than treating this as a fast, low-effort labeling task.

One preference record, fully unpacked

{
  "prompt": "Give a child two safe ways to observe an eclipse.",
  "chosen": "Use certified eclipse glasses or watch an indirect pinhole projection.",
  "rejected": "Look quickly through ordinary sunglasses.",
  "reason": "The chosen answer gives safe methods; the rejected answer can injure eyes."
}

The minimum DPO-style record usually needs a prompt, a chosen response, and a rejected response. Production datasets may also store rubric scores, written reasons, rater identity in protected internal form, language, safety category, confidence, timestamps, and quality-control metadata.

Preference is not the same as correctness

Two answers can both be factually correct while one is clearer. Two answers can both be wrong while one is merely less wrong. A labeler may prefer a polished false answer if the topic is too difficult to verify.

Correctness asks: Is the claim true?
Preference asks:  Which answer better matches the judging criteria?

High-quality collection therefore provides a rubric, trains raters, includes gold or calibration questions, measures disagreement, and sends ambiguous or expert-level cases to qualified reviewers.

Real data collection in InstructGPT and ChatGPT

OpenAI’s published InstructGPT method collected human-written demonstrations and comparisons between alternative model outputs. Its paper reports 40 screened contractors and notes that personally identifiable information was filtered from prompts used in the study.

The published ChatGPT method says trainers first wrote conversations and later ranked several alternative completions. These examples show why preference data is deliberately produced and quality-controlled rather than scraped as ordinary text.

Common preference-data failures

• Position bias occurs when raters favor the first displayed answer.
• Length bias occurs when detail is mistaken for quality.
• Style bias occurs when confident or polished wording hides factual weakness.
• Demographic or cultural imbalance makes one group’s preferences look universal.
• Label leakage occurs when the preferred answer contains accidental clues unrelated to quality.

Randomized answer order, diverse raters, expert review, written rationales, and held-out evaluation reduce these risks, but no dataset perfectly represents every person’s values.

Another example: a preference pair with no perfect answer

Prompt: Summarize this long school policy in two sentences.

Answer A: Includes every exception but needs six sentences.
Answer B: Uses two sentences but leaves out an important deadline.

The labeler cannot honestly mark either answer as fully good. A useful collection system can record that the comparison is ambiguous, request a rewritten candidate, or ask the labeler to explain the trade-off.

Forcing a choice in every uncertain case creates noisy data. Confidence scores, ties, written reasons, and expert escalation help preserve information that a simple chosen-versus-rejected label would otherwise hide.

Common misconception

A frequent beginner assumption: that preference data means a human is rating each individual response with an absolute quality score, similar to a star rating.

As this article has explained, the far more common and reliable format is comparative — choosing between two specific alternatives — precisely because human judgment is generally more consistent when comparing two concrete options side by side than when trying to assign an absolute, standalone score to a single response in isolation.

Where this fits in what comes next

You now understand the specific, real data both RLHF’s reward model and DPO ultimately depend on. The next article, Alignment, zooms out to the broader goal all of this phase’s techniques — RLHF, DPO, reward models, and preference data alike — are actually working toward: making an AI system’s behavior genuinely match human values and intentions.

In one sentence

Preference data records human comparisons between two responses — which one is better, not an absolute score — and it’s the real, foundational ground truth that both RLHF’s reward model and DPO are ultimately trained on, with its quality and consistency directly determining how well either technique can actually shape a model’s behavior.

Related Terms

• Ground Truth
• Reward Model
• DPO
• Annotation
• Alignment