The Generator-Critic Pattern

What You Will Learn

How critics return actionable feedback.
How judge bias distorts criticism.
How rubrics and diverse checks help.

Modules 7 and 8 already gave you the generate-critique-revise shape in depth. This module has one specific, deep job: taking seriously the critic’s own fallibility, because “add a critic” is not automatically the same as “add a trustworthy judgment.”

The architecture

Generator
 ↓
Proposal
 ↓
Critic
 ↓
Feedback
 ↓
Generator

This is worth naming honestly upfront: structurally, this is the same shape as Module 7’s Producer-Critic variant of Reflection. What genuinely earns this its own module is the depth of research specifically on why the critic step itself can quietly fail — research this course has only touched briefly until now.

Five measured biases

This is worth knowing exactly, not as a vague warning that “critics can be biased.” Real, current research names five distinct, measured biases: position, verbosity, self-preference, format, and calibration drift — each with its own specific mechanism and its own specific fix. “‘Better prompts’ is the layer on top” — the mitigation that actually works is mechanical, not rhetorical. (FutureAGI, LLM-Judge Bias Mitigation)

These aren’t claims resting on one source.

Independent, peer-reviewed research corroborates each bias separately: position bias — where a judge’s verdict shifts based on which answer appears first or last in a comparison, regardless of actual quality — has its own dedicated, systematic study. (Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge, cited in Fairness or Fluency?, ACL) Verbosity bias — where longer responses score higher independent of genuine quality — has been separately quantified across GPT-4 and GPT-3.5-Turbo specifically, measuring the actual error rate difference between longer and shorter answer options. (Self-Preference Bias in LLM-as-a-Judge, arXiv)

The precise number worth knowing for self-preference specifically: a judge model scores its own family’s outputs 10 to 25 percent higher than equivalent outputs from other models. (FutureAGI)

Narrative production failure

This is worth reading slowly, because it’s a genuinely precise account of exactly how this failure mode manifests in practice — silently, until it doesn’t:

“You score helpfulness at 0.91 all quarter. The judge model bumps a minor version in March. The mean shifts four points, the distribution narrows, and your CI gate keeps passing. In May the agent quotes a refund off by an order of magnitude. The eval suite never flagged it. The judge changed, the rubric didn’t, and the signal stopped meaning what you thought it meant the day the model rolled.” — FutureAGI, LLM-Judge Bias Mitigation

Read this precisely. Nothing about the deployment pipeline looked broken. The score stayed high. The CI gate kept passing. The actual failure — a critic whose calibration had silently shifted underneath an unchanged rubric — was invisible until a real customer-facing error finally surfaced it, months after the drift actually began.

The mechanism behind self-preference

It’s worth knowing precisely why this bias exists, not just that it does. Research links self-preference bias directly to self-recognition ability — a model’s capacity to recognize its own outputs contributes significantly to how much it favors them. (Panickssery et al., cited in AI Self-preferencing in Algorithmic Hiring, arXiv)

This produces a genuinely counter-intuitive finding worth holding onto precisely: larger and more capable models often show stronger self-preference, not weaker. (UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge, arXiv) The intuitive assumption — a more capable model makes a more trustworthy judge — genuinely doesn’t hold for this specific bias. Better self-recognition, a genuine capability, directly fuels a genuine liability.

Finding that complicates Module 7 directly

This is worth taking seriously as a real extension of material you already learned. Research found self-preference bias amplified in self-refinement pipelines — the exact Reflection loop Module 7 taught — “which enhance fluency and coherence but reinforce the model’s preference for its own outputs.” (arXiv)

This deepens Module 7’s own self-validation-bias warning in a genuinely important way: it’s not just that a model tends to validate its own prior output. The more a self-refinement loop actually improves an output’s fluency and coherence, the more the model’s preference for that output compounds — a real, measured feedback loop between genuine quality improvement and genuine judgment distortion, running simultaneously, worth taking seriously precisely because it means the loop’s own apparent success is part of what makes the underlying bias harder to notice.

The balance: judges aren’t useless

Consistent with this course’s commitment to evenhanded evidence, it’s worth knowing the real positive case too. Strong models like GPT-4 achieve over 80% agreement with human preferences on multi-turn conversation evaluation. (AI Self-preferencing in Algorithmic Hiring, arXiv)

This is worth holding alongside the failure story above, not instead of it: an LLM critic is a genuinely useful, scalable evaluation tool with real, measured agreement with human judgment — and it carries genuine, measurable, specific biases that a team deploying it needs to actively design against, not assume away. Neither the enthusiasm nor the dismissal that dominate casual discussion of LLM-as-judge captures this honestly — the real picture is a genuinely useful tool with genuinely specific, named, measurable failure modes, and the engineering discipline this module describes exists precisely to keep the first true while managing the second.

Mitigations

It’s worth knowing the actual production discipline, not just the diagnosis. Pin a contract: judge_model_id, rubric_version, prompt_template_hash. Bump deliberately — a judge version change is a genuine event to review, not something that happens silently in the background. Calibrate against humans monthly. “Treat a judge swap as an eval-suite migration, not a config change.” (FutureAGI)

This isn’t one team’s isolated practice. A genuinely independent production account describes the identical discipline, down to real code: “Calibration drift. Judge scoring is not absolute — it reflects the judge model’s current behavior. If Anthropic or OpenAI updates the judge model, your score baselines shift even if your RAG system did not change. Pin your judge model to a specific version and treat model upgrades as a baseline reset event.” — with the concrete practice shown directly: JUDGE_MODEL = "claude-sonnet-4-20250514" # pinned — update intentionally, not automatically.

The same account independently confirms the self-preference mitigation too: “If you use Claude to judge output produced by Claude, the judge will systematically prefer its own generation style. For high-stakes eval, use a different model family as judge than the one producing answers.” (LLM Evaluation in Production, Alok)

The calibration-drift failure story itself is worth grounding in genuine, rigorous research too, not only narrative accounts.

A real, current academic paper treats this exact ambiguity formally: “the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores — so every drift alarm is ambiguous between a worse product and a changed judge.” The paper’s proposed resolution is a genuine statistical methodology — a fixed, human-labeled anchor set the current judge re-scores at a steady interval, combined with a formal statistical test distinguishing whether an observed score drop reflects the system, the judge, or neither. (Who Drifted: the System or the Judge?, arXiv)

A separate, real, structural mitigation worth knowing: replacing a single judge with a panel of diverse models — real, published research proposes exactly this, evaluating generations with a genuine jury rather than one judge. (Replacing Judges with Juries, cited in Judging the Judges, arXiv) This is worth connecting directly to your Multi-Agent Systems coursework’s own weighted-voting material — the same underlying logic, now applied specifically to defend against one critic’s idiosyncratic, measured biases rather than trusting any single judge’s calibration to hold indefinitely.

Why diversity in the panel specifically matters

It’s worth being precise about why this mitigation works, rather than assuming any group of judges automatically cancels out bias. If a panel is composed entirely of models from the same family, the self-preference bias this module measured doesn’t diversify away — every member of the panel shares the same underlying tendency to favor familiar generation patterns, and a majority vote among biased judges simply confirms the shared bias with more apparent authority, not less.

This is directly the same lesson your Multi-Agent Systems coursework taught about cognitive monoculture: a panel’s genuine value depends on its members having genuinely different underlying models, different training data, and different self-recognition capabilities, not just a higher raw count of judges nominally voting.

A real panel worth designing deliberately, then, pairs models from genuinely different families — not three variants of the same underlying model dressed up as three independent judges. The real cost of this is worth being honest about too: a genuine cross-family panel means paying for calls to multiple, genuinely different providers, and reconciling disagreement between judges that may have real, principled reasons to see a proposal differently, not just noise to average away.

What this looks like in code

Before reading the syntax, follow the execution flow: identify the incoming state, the component making the decision, the function doing the work, and the condition that returns a result or stops the loop. The code is a small teaching model of the pattern, not hidden framework magic.

CRITIC_CONTRACT = {
    "judge_model_id": "claude-sonnet-4-5-20250929",
    "rubric_version": "v3.2",
    "prompt_template_hash": "a1b2c3d4",
}

def generator_critic(task: str, contract: dict = CRITIC_CONTRACT) -> str:
    proposal = generator(task)

    critique = critic(
        proposal,
        model=contract["judge_model_id"],
        rubric=load_rubric(contract["rubric_version"]),
    )

    log_critic_call(contract, critique.score)  # feeds monthly calibration audit

    if critique.approved:
        return proposal

    return generator(task, feedback=critique.feedback)

The CRITIC_CONTRACT dictionary is the concrete, code-level answer to this module’s narrative failure story — pinning the exact model ID, rubric version, and prompt hash means a judge-model version bump becomes a visible, deliberate change to this contract, not a silent shift underneath an unchanged config.

Applying this to a concrete scenario

It’s worth running this module’s findings against your Multi-Agent Systems coursework’s recurring legal-contract pipeline directly, since the Critic role in that pipeline is precisely a Generator-Critic instance this module’s research applies to.

If that firm’s Critic is a Claude model reviewing Executor output also produced by a Claude model, this module’s self-preference finding applies exactly: the Critic may systematically favor comparisons phrased in ways stylistically familiar to its own family, inflating approval rates by roughly the same 10-to-25-percent range measured in real research — not because the comparisons are genuinely more accurate, but because they’re more familiar.

Given this course’s earlier discussion of Module 10’s cost tip — pairing a stronger model for high-stakes roles with a cheaper one for execution — it’s worth noting this module complicates that advice specifically for the Critic role: a stronger, more capable Critic model isn’t automatically a safer choice against self-preference bias, precisely because stronger models measure higher on this specific bias, not lower. The firm’s genuine mitigation isn’t simply “use the best available model as Critic” — it’s the pinned-contract discipline and periodic human calibration this module actually recommends, regardless of which specific model fills that role.

Interview-relevant framing

Q: Why can’t you just trust an LLM critic’s score without further scrutiny?

Ans: Because real, measured research names five distinct biases affecting LLM judges — position, verbosity, self-preference, format, and calibration drift — not a vague, generic unreliability. Self-preference alone measures at 10 to 25 percent inflation when a judge scores its own model family’s output. A real production account describes exactly how this fails silently: a judge model’s minor version bump shifted calibration underneath an unchanged rubric, and a CI gate kept passing for months before a real customer-facing error finally surfaced the drift.

Q: Wouldn’t using the most capable available model as the critic solve this?

Ans: No — it’s actually the opposite for self-preference specifically. Research found self-preference bias linked to self-recognition ability, and larger, more capable models tend to show stronger self-preference, not weaker. A more capable critic is genuinely better at some things and genuinely worse at this specific bias, precisely because it’s better at recognizing its own outputs as its own.

Q: How would you actually defend a production system against critic bias, concretely?

Ans: Pin an explicit contract — judge model ID, rubric version, and prompt template hash — so a version bump becomes a visible, deliberate event rather than a silent drift. Calibrate against real human judgment on a real cadence, monthly rather than once at launch. And where the stakes genuinely justify it, replace a single judge with a panel of diverse models — real published research proposes exactly this — so one judge’s idiosyncratic, measured bias doesn’t single-handedly determine what passes.

Common Misconception

Incorrect idea: A confident critic is a correct critic.

Why it is incorrect: Confidence is presentation, not proof. Criticism should point to evidence, criteria, tests, or verifiable defects.

Key takeaways

Generator-Critic is structurally the same shape as Module 7’s Producer-Critic Reflection variant — this module’s genuine contribution is depth on the critic’s own, specific, measured fallibility.
Five real, named biases affect LLM judges — position, verbosity, self-preference, format, and calibration drift — each with a distinct mechanism and a distinct, mechanical fix, not a single generic “better prompt” solution.
Self-preference bias measures at a real 10 to 25 percent inflation when a judge scores its own model family’s output, linked directly to self-recognition ability.
Counter-intuitively, larger and more capable models show stronger self-preference bias, not weaker — capability and this specific trustworthiness dimension move in opposite directions.
Self-preference bias is measurably amplified inside self-refinement loops specifically — Module 7’s Reflection pattern — meaning genuine fluency and coherence gains and genuine judgment distortion compound together, not separately.
The honest balance: strong models still achieve over 80% agreement with human judgment on real evaluation tasks — LLM critics are genuinely useful, not useless, and need active bias mitigation rather than blind trust or blanket dismissal.
Real, concrete mitigations exist: a pinned judge contract treating version changes as deliberate events, monthly human calibration, and replacing a single judge with a diverse panel for genuinely high-stakes decisions.

Module 17 turns from one critic’s judgment to several agents actively arguing toward a shared answer — and a genuinely important finding from your Multi-Agent Systems coursework about exactly when that helps, and when it measurably makes things worse: Multi-Agent Debate.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026