LLM Evaluation Framework & Lifecycle

Begin with the problem

Passing one offline test does not prove a change is safe for every user. Evaluation is a lifecycle that moves cautiously from repeatable datasets to shadow traffic, canaries, monitoring, and rollback.

change → offline regression → shadow → canary → monitor → expand or roll back

What you will learn

Distinguish offline evaluation from online measurement.
Use regression, shadow, canary, and A/B stages appropriately.
Keep evaluation datasets current as production failures appear.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Module 10 gave you evaluation dimensions and a golden dataset. This module answers the next question: in what order, and through what stages, does a real change move from “an engineer’s idea” to “running for all production traffic”?

A single evaluation run before deployment is necessary but insufficient — real production behavior only fully reveals itself once traffic is involved, which is exactly why this lifecycle has multiple, distinct stages.

2. Offline vs. Online Evaluation

OFFLINE EVALUATION:      run against a GOLDEN DATASET (Module 10),
                        BEFORE any real user sees the change --
                        fast, cheap, repeatable, but limited to whatever cases the dataset covers

ONLINE EVALUATION:           run against, real, live traffic
                            -- catches issues offline evaluation
                            couldn't anticipate, but
                            carries real risk since real users are
                            involved

Neither replaces the other. Offline evaluation is your first, necessary gate — but real production traffic is more diverse and unpredictable than any golden dataset, which is exactly why online evaluation stages (canary, shadow) exist.

3. The Deployment Lifecycle Stages

1. OFFLINE EVALUATION (Module 10) -- run against golden dataset,
   BEFORE any real traffic

2. SHADOW TESTING -- the NEW version runs ALONGSIDE the current
   production version, on REAL traffic, but its output is NEVER
   shown to users -- only LOGGED and compared

3. CANARY DEPLOYMENT -- the new version SERVES a SMALL
   percentage of real traffic (e.g., 5%), with close monitoring

4. FULL ROLLOUT -- once canary metrics are healthy, the
   new version serves 100% of traffic

Each stage reduces risk before the next, larger exposure — exactly a graduated trust model, not a binary “deployed or not.”

4. A Real-World Analogy — The Power Grid

A power grid operator doesn't switch an ENTIRE region to a NEW
power source instantly -- they test it in isolation
(offline eval), run it in PARALLEL with the existing source while
monitoring output without depending on it (shadow testing), then
route a SMALL portion of real load to it (canary), and ONLY THEN
switch over completely (full rollout).

A sudden, ungraduated switch risks a REAL, large-scale outage if
something was missed -- exactly the risk this lifecycle protects
against for an AI system.

5. Regression Testing for Probabilistic Systems

Directly connecting to Module 2's determinism discussion: a
REGRESSION test for an AI system doesn't check "does output STILL
equal X" -- it checks "do EVALUATION SCORES stay at or above the
PREVIOUS baseline" across the golden dataset (Module 10).

A regression = evaluation scores dropping compared
to the current production baseline, not a single example changing
wording.

6. A/B Testing and Canary Testing — A Distinction

A/B TESTING:      comparing TWO versions with REAL users,
                 typically to measure a business metric (user
                 satisfaction, task completion) over a MEANINGFUL
                 time period

CANARY TESTING:       a SAFETY-focused, gradual rollout --
                     primarily watching for FAILURES (error rates,
                     safety violations) before expanding exposure,
                     not necessarily comparing business outcomes

Both are real, valid techniques — A/B testing answers “is this better,” canary testing answers “is this safe to roll out further.”

7. Evaluation Drift

EVALUATION DRIFT: your golden dataset and rubrics become
                  LESS representative over time, as real usage
                  patterns, user expectations, and the underlying
                  knowledge base (Module 7) evolve.

Incorrect idea: A golden dataset built at launch and never revisited will, eventually stop reflecting real production traffic — directly connecting to Module 7’s RAG-freshness discussion, now applied to evaluation datasets specifically.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect.

8. A worked developer example

TechCorp’s prompt-change deployment lifecycle:

Stage	What Happens	Result
Offline eval	Run against golden dataset, compare to current baseline	Faithfulness score dropped 5% — STOP HERE
(Shadow, canary, rollout)	Never reached	The regression was caught before any real user was exposed

Compare to a good change:

Stage	What Happens	Result
Offline eval	No regressions vs. baseline	✅ Proceed
Canary (5% of traffic)	Error rate within acceptable range vs. baseline	✅ Proceed
Full rollout	Change now serves 100% of traffic	✅ Complete

9. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production AI teams implement this exact staged lifecycle as an automated deployment pipeline (Module 26 covers the CI/CD mechanics) — a change that fails offline evaluation never reaches canary; a change with unhealthy canary metrics never reaches full rollout, all enforced automatically, not by manual judgment calls under deployment pressure.

10. Common Mistakes

Incorrect idea: Deploying directly to full rollout after only offline evaluation.

Why it is incorrect: As shown directly in Section 2-3, real production traffic reveals issues a golden dataset can’t anticipate.

Incorrect idea: Treating regression testing like traditional exact-match testing.

Why it is incorrect: As shown directly in Section 5, AI regression testing compares evaluation SCORES against a baseline, not exact output.

Incorrect idea: Never revisiting the golden dataset after initial launch.

Why it is incorrect: As shown directly in Section 7, this leads to evaluation drift and false confidence.

11. Code — A Staged Deployment Lifecycle Gate

What this shows: implementing Section 3’s staged lifecycle as, sequential gates — a change must pass offline evaluation before even reaching the canary stage, exactly Section 8’s real developer example made into working, automated logic.

from dataclasses import dataclass
from enum import Enum

class DeploymentStage(Enum):
    OFFLINE_EVAL = "offline_evaluation"
    CANARY = "canary_deployment"

@dataclass
class StageResult:
    stage: DeploymentStage
    passed: bool
    detail: str

def offline_eval_gate(new_scores: dict, baseline_scores: dict) -> StageResult:
    """OFFLINE evaluation gate (Section 3, stage 1) -- run BEFORE
    any real traffic sees the change, against a golden dataset
    (Module 10)."""
    regressions = {k: v for k, v in new_scores.items() if v < baseline_scores.get(k, 0) - 0.02}
    if regressions:
        return StageResult(DeploymentStage.OFFLINE_EVAL, False, f"Regressions detected: {regressions}")
    return StageResult(DeploymentStage.OFFLINE_EVAL, True, "No regressions vs baseline")

def canary_gate(canary_error_rate: float, baseline_error_rate: float, threshold_multiplier: float = 1.5) -> StageResult:
    """CANARY gate (Section 3, stage 3) -- a small percentage of
    REAL traffic sees the change first; error rate is compared
    against baseline BEFORE rolling out further."""
    if canary_error_rate > baseline_error_rate * threshold_multiplier:
        return StageResult(DeploymentStage.CANARY, False,
                            f"Canary error rate {canary_error_rate} exceeds {threshold_multiplier}x baseline {baseline_error_rate}")
    return StageResult(DeploymentStage.CANARY, True, "Canary error rate within acceptable range")

def run_deployment_lifecycle(new_scores, baseline_scores, canary_error_rate, baseline_error_rate) -> list:
    """The FULL lifecycle -- offline THEN canary, stopping at the
    FIRST failed gate, exactly Section 4's graduated-trust model."""
    results = [offline_eval_gate(new_scores, baseline_scores)]
    if not results[0].passed:
        return results  # STOP -- never reaches canary

    results.append(canary_gate(canary_error_rate, baseline_error_rate))
    return results

# A BAD change -- a faithfulness regression, caught at
# the FIRST gate, never reaching real users at all.
bad_results = run_deployment_lifecycle(
    new_scores={"faithfulness": 0.75, "relevance": 0.91},
    baseline_scores={"faithfulness": 0.93, "relevance": 0.90},
    canary_error_rate=0.02, baseline_error_rate=0.02,
)
for r in bad_results:
    print(f"[{r.stage.value}] passed={r.passed}: {r.detail}")

Expected Output:

[offline_evaluation] passed=False: Regressions detected:
{'faithfulness': 0.75}

What this confirms: the bad change is correctly stopped at the FIRST gate — the function never even reaches canary evaluation, meaning no real user was ever exposed to this regression — exactly Section 8’s worked developer example, made into working, automated gate logic rather than a manual deployment checklist someone could skip under pressure.

12. Production Considerations

Each stage’s specific thresholds (regression tolerance, error-rate multiplier) should be tuned to your system’s real risk tolerance — a customer-facing financial system warrants stricter gates than an internal tool
Automate this entire lifecycle in CI/CD (Module 26) — manual execution under deployment pressure is exactly when steps get skipped

13. Trade-offs

A fully staged lifecycle slows down deployment velocity compared to deploying directly — a real, worthwhile trade-off for systems where a regression has, real consequences
Shadow testing requires running two versions in parallel, adding real infrastructure cost during the testing window

14. Chapter Summary

An AI system change moves through a staged lifecycle — offline evaluation, shadow testing, canary deployment, and only then full rollout — with each stage reducing risk before the next, larger exposure. Regression testing for AI compares evaluation scores against a baseline, not exact output equality.

Golden datasets and rubrics drift out of relevance over time and need periodic revisiting. This entire lifecycle is what closes the loop Module 10 opened: a repeatable, automatable way to know whether a change is safe to ship.

15. Visual Cheat Sheet

Offline Eval (golden dataset) -> Shadow (real traffic, no exposure)
-> Canary (small % of real traffic) -> Full Rollout

Fail ANY stage -> STOP, never reach the next, larger exposure

16. Top Takeaways

Offline evaluation is necessary but insufficient — real production traffic reveals issues a golden dataset can’t anticipate.
Shadow testing observes real traffic without exposing users to the new version at all — a low-risk way to gather real-world signal.
Canary deployment gradually exposes a small percentage of real traffic, watching closely before a full rollout.
AI regression testing compares evaluation SCORES against a baseline, not exact output equality.
Golden datasets drift out of relevance over time and need periodic revisiting.

17. Interview Questions

Q: 1. Why is offline evaluation alone insufficient before a full production rollout?**

Ans: A golden dataset, however well-constructed, cannot anticipate every real-world query pattern, edge case, or user behavior — real production traffic is more diverse. Shadow testing and canary deployment expose the change to real traffic in progressively riskier stages, catching issues offline evaluation couldn’t have anticipated, before the entire user base is affected.

Why it matters: Skipping these stages risks a large-scale production incident from an issue that a limited golden dataset simply didn’t cover.
Real-world example: Section 8’s TechCorp lifecycle — a change passing offline evaluation still goes through canary before full rollout.
Common mistake: Treating a clean offline evaluation run as sufficient justification for immediate full deployment.
Interviewer is testing: Whether the candidate understands evaluation as a staged, risk-graduated process, not a single gate.
Likely follow-up: “What would you monitor during the canary stage specifically?” → Error rates, latency, and evaluation-relevant metrics (Module 12), compared against the current production baseline.

Q: 2. What’s the difference between A/B testing and canary testing for an AI system change?**

Ans: A/B testing compares two versions with real users over a meaningful period. It measures a business or quality metric to answer, “Is this version better?”

Canary testing is primarily safety-focused. It exposes a small percentage of traffic, watches for failures such as elevated error rates or safety violations, and answers, “Is this version safe to roll out further?” Both techniques are useful and complementary.

Why it matters: Conflating them can lead to under-monitoring safety during what’s framed as a pure A/B business-metric comparison.
Real-world example: A team might run an A/B test to measure whether a new response style increases user satisfaction, while ALSO applying canary-style safety monitoring during that same rollout.
Common mistake: Running only an A/B test with no safety-focused monitoring, missing a real regression because the team was only watching the business metric.
Interviewer is testing: Whether the candidate can distinguish these different (though related) evaluation purposes.
Likely follow-up: “Can you run both simultaneously?” → Yes, common — canary-style safety gates alongside an A/B business-metric comparison on the same gradual rollout.

18. Scenario-Based Question

Scenario: TechCorp’s team deploys a new retrieval configuration directly to 100% of production traffic after it passes offline evaluation with strong scores. Within hours, a subset of users in a specific region report consistently poor answers — a query pattern underrepresented in the golden dataset.

Problem Analysis: Section 10’s common mistake — skipping shadow and canary stages, going straight from offline evaluation to full rollout.
How to Think: The golden dataset didn’t represent this region’s query patterns — a canary stage would have surfaced this with a much smaller, contained blast radius.
Investigation: Compare the affected region’s actual query patterns against golden dataset coverage; confirm the gap.
Root Cause: No staged rollout (Section 3) — the full user base was exposed simultaneously with no graduated risk exposure.
Solution: Roll back immediately; add the affected region’s query patterns to the golden dataset (Section 7); reintroduce the change through the FULL staged lifecycle (offline → shadow → canary → full rollout) this time.
Trade-offs: The staged lifecycle takes longer to reach full rollout — a real, worthwhile cost given this exact incident is precisely what it’s designed to prevent.
Production Considerations: This scenario is a direct, concrete illustration of Section 2’s core point: offline evaluation and real production traffic are different tests, and skipping the staged exposure between them is a real production risk.

19. Next Step

Next: Module 12 — Observability — Level 5 begins here: AI-specific observability beyond traditional logs and metrics — prompt tracing, token usage, latency breakdown, and debugging one failed AI request end-to-end.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026