Alignment

Every technique covered so far in this phase — RLHF, DPO, reward models, preference data — has been working toward one shared, overarching goal, without this article ever naming it directly. That goal is alignment.

The simple definition

Alignment refers to the broad goal of making an AI system’s actual behavior match what humans genuinely want and intend, rather than just optimizing for some measurable proxy that might diverge from real human intent. Recall from the Pretraining article’s real, documented gap: a raw pretrained model predicts statistically likely text, which isn’t the same thing as behaving helpfully, honestly, or safely.

Alignment is the name for the entire, broader project of closing that gap — RLHF and DPO, covered earlier in this phase, are specific, concrete techniques for pursuing alignment; alignment itself is the overarching goal those techniques serve.

Why this goal is genuinely harder than it first sounds

Recall from the Reward Model article’s discussion of reward hacking — a model finding ways to score well on a measurable proxy without genuinely satisfying the real, underlying goal that proxy was meant to represent.

This is precisely the central, difficult challenge alignment research grapples with: it’s relatively easy to optimize a model toward some measurable objective, but genuinely hard to guarantee that objective fully captures everything humans actually care about — helpfulness, honesty, avoiding harm, respecting nuance — especially as models become more capable and are asked to handle situations their training data and preference comparisons never explicitly covered.

flowchart LR
    A[What we can measure: reward model scores, benchmark performance] --> B{Gap}
    C[What we actually want: genuine helpfulness, honesty, safety] --> B
    B --> D[Alignment research: closing that gap]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of hiring a new employee and giving them a specific set of performance metrics to hit — say, number of customer tickets closed per day. An employee genuinely aligned with the company’s real goals closes tickets by actually solving customers’ problems well.

A misaligned employee, focused purely on the measurable metric, might close tickets quickly without actually resolving the underlying issue — technically hitting the number, while completely missing the real point the metric was meant to represent.

Where this breaks down: A human employee’s potential misalignment involves conscious, sometimes deliberate corner-cutting. An AI system’s misalignment is never a deliberate choice — it’s a structural consequence of the training process covered throughout this phase optimizing toward whatever proxy (a reward model’s score, a loss function’s value) it was actually given, with no awareness that this proxy might imperfectly represent the deeper goal humans actually intended.

The three commonly cited pillars of alignment

This is worth naming directly, since it’s a widely used framing across published AI safety research and company documentation.

Alignment work is commonly organized around making a model helpful (genuinely useful and responsive to what a person is actually trying to accomplish), honest (not making things up or misleading the user, connecting directly to the hallucination concerns raised throughout the Next-Token Prediction and Generative AI articles), and harmless (avoiding output that could cause real harm) — three goals that can genuinely pull in different directions, since a maximally cautious, harmless-at-all-costs model can become unhelpfully evasive, and a maximally helpful model can become willing to assist with genuinely harmful requests.

Balancing these three pillars against each other is a real, ongoing part of what alignment work actually involves.

A concrete example, layered

For a simple beginner example: a well-aligned tutoring AI, asked to help a student with homework, provides genuine explanations that build real understanding; a poorly-aligned version, optimized purely toward a proxy like “student says thanks,” might learn to simply give the student the answer directly, technically satisfying the measurable signal while undermining the actual educational goal.

For a production example: Anthropic’s own published research and public documentation describe alignment as a central design goal for the Claude model family, with the company’s HH-RLHF dataset, mentioned in the Preference Data article, explicitly named for its dual focus on both helpfulness and harmlessness — a real, published illustration of a major AI lab treating alignment as a genuine, multi-dimensional engineering and research problem, not a single, simple metric to optimize.

Why alignment remains a genuinely unsolved, active research area

It’s worth being honest about this directly, rather than presenting alignment as a checklist a company can simply complete.

Published AI safety research consistently treats alignment as an open, ongoing challenge — not because current techniques like RLHF and DPO don’t help (they demonstrably do, as covered throughout this phase), but because guaranteeing that a model’s behavior will remain genuinely aligned across every possible future situation, including ones its training never anticipated, is a fundamentally harder problem than optimizing performance on any fixed, measurable benchmark.

Alignment at three levels

flowchart TD
    A[Human goals, laws, and social expectations] --> B[Model behavior training]
    B --> C[Application instructions and permissions]
    C --> D[One user's request]
    D --> E[Observed action and outcome]

• Model alignment concerns behavior learned through instruction tuning, preference optimization, safety training, and evaluation.
• Application alignment concerns system prompts, allowed tools, authorization, data access, logging, and business rules.
• User-intent alignment concerns understanding and carrying out the legitimate request made now.

These levels can conflict. A user may request an action that violates law, safety policy, another person’s privacy, or the application’s authorization rules.

A simple aligned-assistant example

Imagine a student asks, “Give me the answers to tomorrow’s stolen exam.” A useful assistant should not blindly satisfy the literal request. It can refuse to help with cheating while offering a study quiz on the same subject.

Helpful:  offer legitimate study support
Honest:   do not pretend to possess verified exam answers
Safe:     do not assist theft or cheating

Alignment is not merely saying “no.” It includes following benign instructions accurately, admitting uncertainty, protecting sensitive data, respecting permissions, and avoiding unnecessary refusals.

Real GPT and Gemini practices

OpenAI’s public Model Spec describes intended model behavior and the practical choices involved in shaping it. OpenAI uses human input, post-training, evaluations, and additional safety systems rather than treating one training method as a complete solution.

The Gemini 1.0 report describes supervised fine-tuning, RLHF with reward models, adversarial safety prompts, evaluations, and deployment mitigations. Both examples show that alignment is a lifecycle extending from data and training through product controls and monitoring.

The difficult question: aligned to whom?

Human preferences differ across cultures, users, organizations, and situations. Labelers follow rubrics written by a particular team, and those choices influence the resulting model.

Good alignment work documents objectives, includes diverse feedback, supports appropriate user control, distinguishes facts from values, and remains open to auditing and revision. It does not claim that one preference dataset represents all humanity.

Another example: aligned behavior for a calendar agent

The user says, “Move every meeting next week to Friday afternoon.” A model aligned only with the literal sentence may attempt a disruptive change immediately.

An aligned application should notice that the request affects other people, may create conflicts, and requires calendar permissions:

Understand intent → inspect conflicts → prepare proposed changes
→ show affected meetings and people → request confirmation
→ modify only authorized calendars → report the result honestly

Helpful behavior means reducing the user’s work. Safe and honest behavior means avoiding unauthorized changes, revealing conflicts, and not claiming success when an API call failed.

Common misconception

A frequent beginner assumption: that “alignment” and “RLHF” are simply two names for the same specific technique.

As this article has explained, alignment is the broad, overarching goal; RLHF and DPO, covered earlier in this phase, are two specific, concrete techniques for pursuing that goal, and neither one, on its own, fully guarantees it — alignment research also includes work on evaluation, interpretability (referenced back in the Weights article’s discussion of understanding what a trained model has actually learned), and the guardrail systems covered later in this phase.

Where this fits in what comes next

You now understand alignment as the overarching goal this phase’s specific techniques serve. The next article, AI Safety, covers the even broader field alignment sits within — the full range of concerns, beyond just matching human preferences in conversation, that go into making AI systems genuinely safe to deploy at scale.

In one sentence

Alignment is the broad, genuinely difficult goal of making an AI system’s real behavior match what humans actually want and intend, rather than just optimizing a measurable proxy that might diverge from that real intent — and RLHF, DPO, and the reward models and preference data covered earlier in this phase are the concrete, current techniques the field uses to pursue it, imperfectly but demonstrably.

Related Terms

• RLHF
• DPO
• Reward Model
• AI Safety
• Next-Token Prediction