AI Safety

The Alignment article covered making a model’s behavior genuinely match human intent. This article covers the broader field that goal sits within: AI safety.

The simple definition

AI safety is the broad field concerned with preventing AI systems from causing harm — whether through misuse, accident, or unintended behavior — spanning technical research, engineering practice, and policy. Recall from the Alignment article that alignment specifically addresses whether a model’s behavior matches human intent.

AI safety is the larger field alignment sits inside — it also includes concerns that go well beyond one model’s conversational behavior, like preventing a system from being misused for genuinely harmful purposes, ensuring systems fail gracefully rather than catastrophically, and considering the broader societal effects of deploying powerful AI at scale.

Why this needs to be a distinct, broader category from alignment

Recall from the Alignment article’s three pillars — helpful, honest, harmless.

A perfectly aligned model, in that narrow sense, could still contribute to real, serious harm through other paths entirely: a genuinely helpful, honest, harmless-in-conversation model could still be deliberately misused by a bad actor for a harmful purpose, or could produce genuinely dangerous information if not properly restricted, or could behave unpredictably in a real-world deployment scenario its training never anticipated.

AI safety is the broader umbrella that covers all of these concerns, not just the narrower question of whether a model’s own behavior, in isolation, matches human intent.

flowchart LR
    A[AI Safety: the broad field] --> B[Alignment: model behavior matches human intent]
    A --> C[Misuse prevention]
    A --> D[Robustness to unexpected situations]
    A --> E[Guardrails and deployment safeguards]

What AI safety work actually covers in real practice

This is worth being concrete about, since “AI safety” can sound abstract until you see its real, practical components. It includes red-teaming — deliberately trying to make a model produce harmful output, in order to find and fix weaknesses before real users encounter them.

It includes robustness research — studying how models behave on inputs genuinely different from anything in their training data, connecting back to the generalization concerns covered throughout the Generalization phase. It includes the guardrail systems covered in the very next article — practical, deployed safeguards layered around a model in production.

And it includes policy and governance work — questions about appropriate use, deployment restrictions, and regulatory frameworks, which extend beyond pure engineering into organizational and societal decision-making.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of aviation safety as a broad field. It’s not just about training pilots to fly correctly (analogous to alignment) — it also includes aircraft engineering redundancies, air traffic control systems, maintenance protocols, and regulatory oversight, all working together so that even if one specific safeguard fails, the overall system remains genuinely safe.

Where this breaks down: Aviation safety has decades of accumulated, well-tested engineering standards and regulatory frameworks. AI safety, as a field, is comparatively young and rapidly evolving — many of its practices and standards, including the specific techniques covered throughout this phase, are still being actively developed and refined, rather than representing settled, mature engineering practice the way aviation safety largely does today.

Real, published examples of AI safety work in practice

This is worth grounding concretely, since major AI labs have published real, documented safety practices worth knowing by name.

Anthropic, as a company, has published research specifically framing safety as central to its mission, including work on interpretability (understanding what’s actually happening inside a trained model, connecting back to the Weights article’s discussion of distributed, hard-to-interpret representations) and red-teaming practices before releasing new models.

OpenAI has published its own documented safety practices, including external red-teaming programs before major model releases and a published usage policy restricting certain categories of use.

Both companies, along with others across the field, publish model cards and safety evaluations alongside major model releases, a real, documented industry practice of transparency around a model’s tested capabilities and known limitations.

A concrete example, layered

For a simple beginner example: before releasing a new customer-facing chatbot, a company might run a structured red-teaming exercise, deliberately trying prompts designed to make the bot reveal confidential information or produce inappropriate content, specifically to catch and fix these weaknesses before real customers ever encounter them.

For a production example: major AI labs’ published practice of running extensive safety evaluations — testing a new model against categories like weapons information, self-harm content, and manipulation attempts — before public release is a real, documented part of how frontier models actually reach deployment, distinct from and layered on top of the alignment training (RLHF, DPO) covered earlier in this phase.

Why AI safety involves genuine, honest trade-offs

It’s worth being direct about a real tension here, not presenting AI safety as a straightforward, cost-free addition. Safety measures — extensive red-teaming, conservative guardrails, cautious default behaviors — can genuinely reduce a model’s usefulness or increase the friction of interacting with it, echoing the “harmless can become unhelpfully evasive” tension raised in the Alignment article.

Balancing genuine safety against genuine usefulness is a real, ongoing engineering and product decision, not a solved problem with one universally correct answer.

Start with a threat model

A threat model asks four practical questions:

1. What valuable people, data, or systems could be harmed?
2. What can the AI system do, including through tools?
3. How might mistakes or attackers cause harm?
4. Which prevention, detection, response, and recovery controls are needed?

For a homework tutor, important risks include unsafe advice, exposure of a child’s personal information, biased explanations, and fabricated facts. For a tool-using financial agent, the risk is much higher because it may read private records or initiate real actions.

Safety across the complete lifecycle

flowchart LR
    A[Data filtering and governance] --> B[Safety-aware training]
    B --> C[Capability and misuse evaluation]
    C --> D[Red teaming]
    D --> E[Restricted deployment]
    E --> F[Monitoring and incident response]
    F --> G[Updates and re-evaluation]

Safety is not a filter added at the final second. Training data, model capability, tool permissions, user interface, deployment scope, monitoring, and response plans all affect risk.

Real examples from GPT and Gemini systems

OpenAI has documented rule-based rewards as part of its safety stack since GPT-4, including use with GPT-4o mini. The method converts written safety rules into structured propositions used to judge model responses, reducing reliance on repeatedly collecting human labels for routine policy cases.

The Gemini 1.0 report describes safety policies, supervised fine-tuning, RLHF, adversarial harm-inducing prompts, evaluation across risk categories, and deployment mitigations. Public reports describe the approach at a high level; they do not reveal every current internal detector, threshold, or dataset.

Safety trade-offs and false positives

A strict system may block harmless educational questions, while a permissive system may allow dangerous assistance. Safety work measures both harmful content that slips through and benign content incorrectly blocked.

False negative: harmful request is incorrectly allowed
False positive: harmless request is incorrectly blocked

The correct balance depends on the application. A creative-writing tool and a medical decision-support system should not share identical risk thresholds or human-review requirements.

When humans must remain involved

Human approval is especially important for irreversible, high-impact, or legally sensitive actions such as financial transfers, medical decisions, employment decisions, publishing private data, or changing production infrastructure. AI can prepare information or recommendations, but authorization should remain outside the model and be enforced by the surrounding system.

Another example: safety for an AI hiring assistant

An AI system may summarize résumés and organize interview notes, but employment decisions can affect a person’s livelihood and may be governed by law.

Possible harm: qualified candidates are ranked lower because of biased patterns
Prevention:     exclude protected attributes and unsupported proxies where required
Evaluation:     compare error rates and outcomes across relevant groups
Oversight:      require trained humans to make and document final decisions
Response:       provide appeal, correction, and incident-review processes

Removing a person’s name does not automatically remove bias because schools, locations, career gaps, and writing patterns can act as proxies. Safety therefore requires careful feature choices, fairness evaluation, access controls, audit logs, legal review, and meaningful human accountability.

Common misconception

A frequent beginner assumption: that AI safety is primarily about preventing dramatic, science-fiction-style scenarios — a rogue AI taking over, and nothing more mundane. As this article’s real, practical examples have shown, the actual, present-day work of AI safety is overwhelmingly about concrete, near-term concerns — misuse prevention, robustness, honest evaluation, and deployment safeguards — the genuine, documented, everyday work happening at labs right now, not primarily speculative long-term scenarios.

Where this fits in what comes next

You now understand AI safety as the broad field alignment sits within. The final article in this phase, Guardrails, covers the specific, practical, deployed safeguards this article has referenced repeatedly — the real, concrete systems that sit around a model in production, actively catching and preventing harmful input or output in real time.

In one sentence

AI safety is the broad field concerned with preventing AI systems from causing genuine harm, spanning alignment, misuse prevention, robustness, and deployment safeguards — a comparatively young, rapidly evolving field where major labs have published real, documented practices like red-teaming and safety evaluations, even as the underlying engineering and policy standards continue actively developing.

Related Terms

• Alignment
• RLHF
• Guardrails
• Reward Model
• Generalization