The Evaluator-Optimizer Pattern

What You Will Learn

How generation, evaluation, and improvement form a loop.
What measurable criteria mean.
How thresholds stop the loop.

Module 7 closed with a rule worth repeating exactly: if you can run a test, run the test. This module is that rule, built into a full pattern — replacing a model’s own subjective judgment of its work with something genuinely objective wherever possible.

Where this comes from

The evaluator-optimizer pattern appears in Anthropic’s Building Effective Agents (December 2024): one model call generates a response, another evaluates it against stated criteria, and feedback drives another attempt. Anthropic presents it alongside several workflow patterns; it does not define a formal “top-tier” ranking or say that only two useful shapes exist. The linked AgenticOrgChart page offers a secondary interpretation that groups Evaluator-Optimizer with Orchestrator-Workers. (Anthropic, Building Effective Agents; AgenticOrgChart.com, Evaluator-Optimiser Agent Pattern)

The architecture

Generator
   ↓
Candidate
   ↓
Evaluator
   ↓
Pass?
 ↙    ↘
No     Yes
↓       ↓
Feedback Finish
↓
Generator

Two agents alternate: the generator produces a candidate, the evaluator scores it against explicit quality criteria, and critique flows back to the generator if the candidate fails. The result only emits once the evaluator genuinely passes it. (AgenticOrgChart.com)

The evaluator is asked to score against a real rubric — factuality, citation coverage, whether code passes its tests, fluency, alignment with the original brief — not a vague sense of “is this good.”

Anthropic’s own guidance on when this fits

This is worth quoting precisely, because it’s the primary source’s own words, not an interpretation of them: this pattern is a genuine “workhorse for tasks where iteration genuinely improves the output” — drafting tasks, code generation, and content-against-brief tasks are paradigmatic. (AgenticOrgChart.com)

Just as precisely, Anthropic names where it doesn’t fit: tasks where the first attempt is usually correct, and the iteration cost is high — long-form synthesis, or work involving expensive tool calls per attempt. Forcing this pattern onto a task like that means paying real, repeated cost for iteration a task rarely actually needs.

Why this differs from Reflection

It’s worth being precise about the real distinction, since Module 7 already set this comparison up directly. Reflection uses a model’s own reasoning to judge quality — inherently subjective, and inherently biased toward validating output it just produced. Evaluator-Optimizer replaces that judgment with something as close to objective as the task allows: you can enforce specific, checkable business rules — “must be under 280 characters,” “must use JSON format,” “must pass this test suite” — by simply rejecting outputs that fail them. (DIY #19, Evaluator-Optimiser LLM Workflow Pattern)

This is exactly why the pattern is described as fitting code generation, legal document drafting, and complex math problems particularly well — domains where “almost right” is effectively “wrong,” and a genuine pass/fail check is possible rather than a matter of taste.

The problem this pattern doesn’t fully solve

This is worth taking as seriously as this module’s case for the pattern, because it’s real, peer-reviewed research, not a hypothetical caveat.

Even when the evaluator is genuinely objective — an actual test suite, not a model’s opinion — the resulting output can still be wrong. Real research studying test-suite-based program repair across 224 real bugs from the Defects4J benchmark found that generated patches routinely pass the test suite while still being incorrect — a phenomenon named overfitting patches: fixes overly specific to the exact tests they were checked against, failing to generalize to cases the test suite simply didn’t cover.

The same research found that adding more generated test cases could change which patch got produced, but was not effective at turning genuinely incorrect patches into correct ones. (Test Case Generation for Program Repair)

This is worth holding as a genuine correction to how this pattern gets talked about casually. “The tests pass” is a real, objective signal — and it is not the same claim as “the code is correct.” A test suite is a sample of the space of correct behavior, not a complete specification of it, and an evaluator-optimizer loop will happily converge on a candidate that satisfies the sample while genuinely missing the actual goal.

LLM-based evaluators have their own faithfulness problems

It’s worth knowing this applies even when the evaluator itself is an LLM, not a plain test suite — which is genuinely common for tasks (fluency, brief-alignment) that don’t reduce to a pass/fail check at all.

Current, rigorous 2026 research studying evaluator faithfulness — using a real model (Qwen-Plus) as the evaluator backbone — identified recurring, systematic failure patterns that undermine how faithfully an LLM evaluator actually reflects genuine quality, and designed targeted mitigations through progressive prompt refinement, measured against real correlation metrics between the evaluator’s score and actual outcomes. (The Verification Horizon, arXiv)

This is worth connecting directly to Module 7: an LLM-based evaluator genuinely avoids the self-validation bias, since it’s a separate call judging someone else’s output — but it doesn’t automatically become a perfectly faithful judge just because it’s structurally separate. Evaluator prompt quality is itself a real, measurable variable, not something to assume works correctly by default.

What “measurable” means here

It’s worth knowing precisely how a real evaluator’s faithfulness gets tested, rather than leaving “faithfulness problems” abstract. The Verification Horizon research measured evaluator quality through concrete metrics — how well an evaluator’s score correlates with actual, ground-truth outcomes, and how much regret a decision-maker would incur trusting that evaluator’s judgment over repeated iterations of prompt refinement.

This is genuinely the same discipline Module 7 already argued for reflection loops — measuring whether a judgment mechanism is actually trustworthy, empirically, rather than assuming it is because it sounds structurally sound on paper. An evaluator that hasn’t been measured this way is an unverified assumption sitting at the center of a pattern whose entire value proposition depends on that evaluator being trustworthy.

Production variant: Planner-Generator-Evaluator

It’s worth knowing a genuine, named extension used in real production coding systems. Inspired by the competitive feedback loop of generative adversarial networks, this variant splits the work into three distinct roles rather than two: a planner produces a spec, a generator writes code against that spec, and an evaluator reads both the spec and the code, producing a structured critique. (MindStudio, What Is the Planner-Generator-Evaluator Pattern?)

The reason this three-way split matters, stated directly: “The evaluator is the most underrated component. It reads both the spec from the planner and the code from the generator, then produces a structured critique. A useful evaluator does more than check syntax.” Checking code against tests alone catches whether it runs. Checking code against the original spec — not just whether it happens to pass whatever tests exist — is what catches the overfitting-patches problem above, at least partially, by giving the evaluator a genuine reference point beyond the test suite itself.

Why a quality threshold matters more than “better”

It’s worth knowing this distinction precisely, because it’s easy to build a loop that improves output without ever actually finishing. A real documented example: a system generating drafts through multiple parallel critics produced output genuinely better than the original — but “better” is not a number. Without an explicit quality threshold, there’s no way to know when to stop revising and actually publish. (Medium, The Evaluator-Optimizer Pattern: Quality-Gated Generation)

This is precisely why this pattern’s evaluator emits a genuine PASS/FAIL against an explicit rubric, not just a running commentary of feedback — the threshold is what turns “getting better” into “actually done,” a distinction Module 7’s own reflection loop, without an explicit rubric, can quietly lack.

What this looks like in code

Before reading the syntax, follow the execution flow: identify the incoming state, the component making the decision, the function doing the work, and the condition that returns a result or stops the loop. The code is a small teaching model of the pattern, not hidden framework magic.

def evaluator_optimizer(task: str, max_iterations: int = 3, pass_threshold: int = 85) -> str:
    candidate = generate(task)

    for i in range(max_iterations):
        result = evaluate(candidate, rubric=task.rubric)

        if result.score >= pass_threshold:
            return candidate

        candidate = generate(task, feedback=result.feedback)

    # Hit max_iterations without passing — surface this, don't ship silently
    return escalate_for_review(candidate, last_score=result.score)

Notice the loop’s failure path doesn’t quietly return the best candidate found — it escalates explicitly. Given this module’s overfitting-patches warning, silently shipping “the best attempt so far” when the evaluator never actually passed it is precisely the kind of decision that should require a human’s explicit sign-off, not an automatic default.

Applying this to a concrete scenario

It’s worth extending your Multi-Agent Systems coursework’s recurring legal-contract pipeline through this exact pattern’s lens, since the Critic role in that pipeline was, functionally, always an evaluator. Suppose the Executor’s clause-comparison output is scored against a genuine rubric — does it correctly identify the relevant policy clause, does it cite the actual contract language, does it flag deviations accurately — with a numeric pass threshold rather than a binary approve/reject.

Run this module’s overfitting-patches warning against that design honestly. A rubric-based evaluator can pass a comparison that technically satisfies every rubric item while still missing something the rubric never anticipated — a genuinely unusual clause structure the rubric’s authors never considered when they wrote it.

This is the same structural risk the Defects4J research demonstrated for code: passing an explicit, objective check is real evidence of quality, not proof of it. The Planner-Generator-Evaluator variant’s insight applies directly here too — an evaluator checking the Executor’s output against the original contract section itself, not just against the rubric’s checklist, catches a genuinely different class of error than rubric-checking alone ever could.

Interview-relevant framing

Q: Why is evaluator-optimizer generally more reliable than reflection?

Ans: Because it replaces the model’s own subjective judgment with something as close to objective as the task allows — a test suite, a schema validator, an explicit rubric — rather than asking the same reasoning process that generated an answer to also judge it. Anthropic’s own research names this one of only two top-tier multi-agent patterns, specifically fitting drafting, code generation, and content-against-brief tasks, where iteration genuinely improves the result and a real pass/fail check is possible.

Q: If a piece of code passes all its tests through an evaluator-optimizer loop, can you trust it’s correct?

Ans: Not automatically. Real research on program repair across 224 real bugs found generated patches routinely passing their test suite while still being genuinely incorrect — called overfitting patches, fixes narrowly tailored to the specific tests checked rather than the actual underlying goal. A test suite is a sample of correct behavior, not a complete specification of it. This is exactly why more sophisticated variants have the evaluator check the code against the original spec directly, not just against whatever tests happen to exist.

Q: When would evaluator-optimizer be the wrong choice, even for a quality-sensitive task?

Ans: When the first attempt is usually already correct and each iteration is genuinely expensive — long-form synthesis, or work involving costly tool calls per attempt. Anthropic’s own guidance names this directly as a poor fit. Paying for repeated generate-evaluate cycles on a task that rarely needed a second attempt in the first place is real, avoidable cost for a benefit the task’s actual failure rate doesn’t justify.

Common Misconception

Incorrect idea: An LLM evaluator provides objective truth.

Why it is incorrect: An LLM evaluator is another prediction. Use deterministic tests, references, or human review when correctness matters.

Key takeaways

Evaluator-Optimizer is defined in Anthropic’s own primary research as one of only two top-tier multi-agent patterns — a generator and a genuinely separate evaluator alternate until an explicit quality threshold passes.
Anthropic’s own guidance is precise: this pattern fits drafting, code generation, and content-against-brief tasks well, and fits poorly where the first attempt is usually correct and iteration is expensive.
The real distinction from Reflection: this pattern replaces subjective self-judgment with something as close to objective as the task allows — a test, a schema, an explicit rubric — rather than the model’s own reasoning about its own output.
Even genuinely objective test-based evaluation has a real, documented limit: research on 224 real bugs found generated patches routinely passing tests while remaining incorrect — “overfitting patches,” narrowly tailored to the specific tests checked rather than the true underlying goal.
LLM-based evaluators, even though structurally separate from the generator, have their own real, measurable faithfulness problems — current research found systematic failure patterns requiring deliberate prompt refinement, not something to assume works correctly by default.
A real, production three-role variant — Planner-Generator-Evaluator — has the evaluator check generated work against the original spec directly, not just against whatever tests exist, partially addressing the overfitting-patches problem.
An explicit quality threshold, not just “the output got better,” is what turns iteration into genuine completion — without one, a loop can keep improving output indefinitely without ever actually finishing.

Module 9 covers a genuinely simpler, more general variant of this same iterate-until-satisfied idea, one without a formal generator/evaluator role split at all: Iterative Refinement.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026