Reward Model

The RLHF and DPO articles both referenced this component directly, one building on it explicitly, the other showing a way to route around building it at all. This article covers exactly what it is: a reward model.

The simple definition

A reward model is a separate, trained model whose job is to predict how much a human would rate or prefer a given response, producing a single numerical score that stands in for direct human judgment. Recall from the RLHF article’s three-stage pipeline: after supervised fine-tuning, the second stage trains this exact reward model, using real human comparisons, so that the third stage’s reinforcement learning process has a fast, automated way to score enormous numbers of generated responses without needing a human to personally review every single one.

Why this component needs to exist at all

Recall from the Reinforcement Learning article’s core loop: an agent needs a reward signal for every action it takes, repeated potentially millions of times during training.

Having an actual human personally rate every single one of those millions of generated responses would be genuinely impossible — recall from the Annotation and Labeling articles’ discussion of how expensive and slow human labeling already is at much smaller scales.

A reward model solves this scaling problem directly: train it once, using a manageable amount of real human judgment, and then let it stand in for a human rater at massive scale, scoring as many generated responses as the reinforcement learning process needs.

flowchart LR
    A[Humans rank a manageable set of response comparisons] --> B[Reward Model trains to predict those human rankings]
    B --> C[Reward Model can now score millions of new responses automatically]

How this actually gets trained, step by step

This is worth walking through precisely, since it’s a real, specific training process, distinct from the language model training covered throughout the Training Mechanics phase.

Human labelers are shown a prompt along with several different candidate responses generated by the model being trained, and asked to rank those responses from best to worst — exactly the comparison-based preference data covered in the very next article, rather than a single, absolute quality score.

The reward model, itself typically built from the same kind of Transformer architecture covered in the Transformers phase, is then trained — using its own loss function and gradient descent, as covered throughout the Training Mechanics phase — to produce scores that correctly reflect these human-provided rankings: higher scores for responses humans ranked better, lower scores for the ones ranked worse.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a talent competition’s head judge, who has personally watched enough live performances to develop an accurate, well-calibrated sense of quality, and can then reliably predict how the full judging panel would score a new performance — even without asking every individual judge personally, because their prediction has been well-trained against real, past panel decisions.

Where this breaks down: A head judge’s prediction draws on genuine accumulated experience and taste.

A reward model’s “prediction” is a learned, numerical function — the exact weighted calculations covered throughout the Node and Neural Network articles — trained to output a score that statistically correlates with the human rankings it was trained on, with zero actual taste or experience involved, just a pattern learned from a specific, finite set of human comparisons.

What a reward model’s score actually represents, and its real limitation

It’s worth being precise and honest here, since this is a genuine, well-understood limitation, not a minor footnote. A reward model’s score is a proxy for human preference — a learned approximation, not a perfect, direct measurement of what a human would actually think.

Because the reinforcement learning process in RLHF specifically optimizes the language model to maximize this proxy score, a well-documented failure mode called reward hacking can occur — the language model finding ways to score highly according to the reward model’s specific learned patterns, without those responses genuinely being what a human would actually prefer, echoing the broader concern about optimizing a measurable proxy rather than the real underlying goal.

A concrete example, layered

For a simple beginner example: given a customer service question and three candidate responses, a reward model trained on human rankings might learn to score a warm, empathetic, and clearly-worded response higher than a curt or overly technical one, having learned this pattern from many similar human comparisons during its own training.

For a production example: OpenAI’s InstructGPT paper, referenced throughout this phase, describes training a reward model on human comparisons of GPT-3’s own outputs, then using that trained reward model to guide the reinforcement learning stage that ultimately produced InstructGPT — a real, published, and historically significant example of exactly this component in action.

Why reward models remain an active area of research

It’s worth being honest about this being a genuinely unresolved, ongoing challenge rather than a fully solved problem. Building a reward model that reliably captures nuanced, sometimes conflicting human preferences — helpfulness versus caution, brevity versus thoroughness — without being exploitable through reward hacking remains an active area of published AI safety and alignment research, directly connected to the broader Alignment and AI Safety topics covered later in this phase.

Train a reward model with simple numbers

For one prompt, a labeler prefers answer B over answer A:

Prompt: Explain why exercise is useful.
Answer A: Exercise is useful.               Human rank: lower
Answer B: Exercise strengthens the body...  Human rank: higher

The reward model may initially score them incorrectly:

Initial scores: A = 1.2, B = 0.8  ✗ wrong order
Later scores:   A = 0.4, B = 1.6  ✓ preferred order

Training loss encourages the score for the preferred answer to exceed the score for the rejected answer. The absolute value 1.6 has no universal meaning such as “80% correct”; the useful information is how scores compare under that reward model.

Reward model versus critic versus judge

Component	Main job
Reward model	Convert a prompt-response pair into a learned scalar preference score.
RL critic or value model	Estimate expected future reward from a state during an RL algorithm.
LLM judge	Produce an evaluation, score, or comparison using a prompted language model.
Rule-based checker	Apply explicit rules, tests, or schemas without learning human preference.

These components can be combined, but their scores should not be treated as objective truth.

Real GPT and Gemini use

InstructGPT trained a reward model from human comparisons and used its scores during PPO training. ChatGPT’s published training description similarly says human trainers ranked alternative answers to create comparison data for reward-model training.

The Gemini 1.0 technical report documents RLHF using a reward model as part of post-training and safety mitigation. Exact current proprietary reward-model architectures and datasets are not fully public, so the reliable connection is the documented pipeline rather than invented parameter counts.

Reward hacking

If the policy discovers a shortcut that earns a high score without satisfying the real goal, it is exploiting the reward model. For example, a reward model that mistakes confident language for correctness may reward a fluent false answer.

Teams limit this problem through fresh human evaluation, adversarial tests, multiple metrics, KL constraints, reward-model updates, and external tools that verify claims where possible.

Another example: ranking travel plans

A travel assistant proposes two plans for a family with a wheelchair user and a fixed budget:

Plan A: within budget, accessible transport, realistic travel times
Plan B: cheaper, but includes stairs and impossible connections

A well-trained reward model should score Plan A higher because it follows all important constraints, not simply because Plan B costs less. Training data must contain accessibility and feasibility examples, or the reward model may overvalue the easiest numeric feature.

Before deployment, evaluators can create challenge cases where the cheapest, longest, or most detailed plan is deliberately unsafe or impractical. These cases reveal whether the reward model learned the real preference or a shortcut.

Common misconception

A frequent beginner assumption: that a reward model and the main language model being trained are the same model, or that “the reward model” is just another name for the language model’s own confidence or probability output, as covered in the Probability Distribution article.

As this article has explained, a reward model is a genuinely separate model, trained on a different objective (predicting human preference rankings) using different training data (human comparisons) than the language model’s own next-token-prediction training, even though both are built using similar underlying neural network architecture.

Where this fits in what comes next

You now understand the specific model that lets RLHF scale human judgment across massive amounts of training. The next article, Preference Data, covers exactly what a reward model (and DPO) is actually trained on — the specific format and real, human-generated source of the comparison data both techniques fundamentally depend on.

In one sentence

A reward model is a separate, trained model that predicts how much a human would prefer a given response, letting the expensive, limited resource of direct human judgment scale automatically across the millions of training examples RLHF’s reinforcement learning process actually needs — though its status as an imperfect proxy for real human preference remains a genuine, actively researched limitation.

Related Terms

• RLHF
• Reinforcement Learning (RL)
• Preference Data
• DPO
• Loss Function