The Faithfulness article closed on a real, honest gap: a faithful answer can still be wrong if its source was wrong. This article covers the metric built specifically to catch that gap: answer correctness.
The simple definition
Answer correctness measures whether a generated response is factually accurate and semantically aligned with a known-correct reference answer, independent of whether it faithfully used its retrieved source material. Recall from the Faithfulness article’s core limitation — faithfulness checks only whether an answer matches its given sources, not whether those sources, or the resulting answer, were actually true. Answer correctness is the metric that closes precisely this gap, checking a response against ground truth rather than against whatever context it happened to be given.
Why this needs to be a genuinely separate metric from faithfulness
Recall from the Ground Truth article, back in the Data Handling phase, that a known-correct answer, established independently of any specific model’s output, is the real standard a prediction gets checked against. Faithfulness and answer correctness check a response against two fundamentally different things — one against its own retrieved context, the other against genuine, external ground truth — meaning a response can score well on one while failing the other entirely, and only measuring both together gives a complete, honest picture.
flowchart LR
A[Generated answer] --> B{Faithful to its retrieved source?}
A --> C{Correct against ground truth?}
B -->|Faithful but source was wrong| D[High faithfulness, low correctness]
C -->|Right answer, added unsupported detail| E[High correctness, lower faithfulness]How this actually gets measured, technically
This is worth being concrete about, since checking “correctness” for open-ended text is genuinely harder than it sounds, echoing the LLM Evaluation article’s opening challenge.
Real answer correctness metrics, including those in the RAGAS framework covered throughout this phase, typically combine two distinct checks: a semantic similarity component, measuring how closely the generated answer’s meaning aligns with the reference answer even if the exact wording differs — recall from the Cosine Similarity article’s technique for comparing embeddings — and a factual overlap component, checking whether the same specific facts appear in both, often using the same LLM-as-a-judge technique covered earlier in this phase to identify and compare individual factual claims.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of grading a student’s math homework where they showed correct, faithful work based on a textbook formula, but the textbook itself contained a printing error in that formula — the student followed their source material perfectly, yet arrived at a genuinely wrong final answer. Answer correctness is the grade that catches the wrong final number; faithfulness is the separate grade that confirms the student’s work matched what the textbook actually said.
Where this breaks down: A math teacher can verify a formula’s correctness with certainty, since mathematics has objectively verifiable rules. Answer correctness for open-ended language is considerably fuzzier — recall from the Semantic Search article’s core insight that meaning can be expressed many valid ways — meaning “correctness” here is a matter of degree and semantic judgment, not a clean, binary right-or-wrong the way a math answer usually is.
A concrete example, layered
For a simple beginner example: asked “what year was the company founded,” a response saying “the company was established in 2015” would score highly on answer correctness against a reference answer of “founded in 2015,” even though the exact wording differs, since the actual, meaningful fact matches.
For a production example: teams building RAG systems for high-stakes domains like healthcare or finance commonly track answer correctness against a carefully curated evaluation dataset of expert-verified question-answer pairs, treating a drop in this specific metric as a genuine, serious signal that something in the pipeline — retrieval, the underlying knowledge base, or the generation step itself — needs real investigation.
Why a system can be highly faithful yet score poorly on correctness
It’s worth being direct about this real, practical scenario, since it’s precisely the failure mode this metric exists to catch. Recall from the Knowledge Base article’s principle that a retrieval system’s answers are capped by the quality of its underlying source material.
A RAG system retrieving from an outdated internal policy document could generate a response that’s perfectly faithful to that document — using only information genuinely present in it — while still being flatly, factually wrong compared to the company’s actual, current policy, a real, serious gap that faithfulness alone would never catch, but answer correctness, checked against genuine ground truth, reliably would.
The real story: a government chatbot that told businesses to break the law
This deserves telling in full, because it’s a genuine, extensively documented, and ultimately consequential real-world failure of answer correctness at real scale. In October 2023, New York City launched the MyCity Chatbot, a Microsoft-powered tool meant to give small business owners “actionable and trusted information” on navigating city regulations.
Investigative reporting by The Markup in early 2024 found the chatbot regularly gave incorrect, and in several cases outright illegal, advice: it told business owners they could take a cut of workers’ tips (illegal under New York law), that landlords could discriminate against tenants using Section 8 housing vouchers (illegal), that businesses could refuse cash payments (required to accept cash in NYC since 2020), and, in one particularly striking case, that a restaurant could still serve cheese that had been partially eaten by a rat, as long as customers were informed.
Mayor Eric Adams initially defended keeping the tool online even after acknowledging its answers were “wrong in some areas.” The problem never fully went away — by 2025, incoming Mayor Zohran Mamdani cited the chatbot’s continued unreliability, and its roughly half-million-dollar cost, in ordering it shut down, calling it “functionally unusable.” Andrew Rigie, director of the NYC Hospitality Alliance, summarized exactly why answer correctness matters in real, practical terms: “If when I ask a question and then I have to go back to my lawyers to know whether or not the answer is correct, it defeats the purpose.”
Common misconception
Several answers can be correct
Question: “Name a programming language commonly used for machine learning.”
Valid answers include Python, R, Julia, and others. A reference containing only “Python” is an example, not a complete list of truth.
candidate differs from reference
does not automatically mean
candidate is incorrectA correctness rubric should say whether the task expects one example, a complete list, a calculation, or a specific fact.
References can be wrong or incomplete
Before trusting a low score, inspect the reference. Policies become outdated, experts disagree, and an expected answer may omit a valid interpretation.
For high-stakes domains, store the source, reviewer, verification date, and scope of each reference. Allow an evaluator to mark a case ambiguous or reference problem rather than forcing the candidate model to fail.
A current Gemini evaluation example
Google Vertex AI supports pointwise question-answering correctness and pairwise comparison. A team can compare Gemini 2.5 Flash with Gemini 2.5 Pro on its verified questions, but the resulting score describes that dataset and rubric—not universal correctness.
Compare claims with a reference
Question: “Which planet is closest to the Sun, and how many moons does it have?”
Reference answer: “Mercury is closest to the Sun and has zero moons.”
Candidate answer: “Mercury is closest to the Sun and has one moon.”
| Required fact | Candidate result |
|---|---|
| Closest planet is Mercury | Correct |
| Mercury has zero moons | Incorrect |
The answer is partly correct but should not receive full credit. A useful rubric can assign one point per required fact, giving this answer 1 out of 2.
Exact match is often too strict
“Mercury has no natural satellites” and “Mercury has zero moons” mean the same thing but do not contain identical words. Open-ended evaluation may therefore use claim matching, semantic similarity, or an LLM judge.
For calculations, identifiers, dates, or structured tool arguments, exact checks may be safer. The evaluation method should match the kind of answer.
Current platform connection
Google Vertex AI exposes question-answering correctness and customizable pointwise metrics. Ragas provides answer-correctness metrics that compare a response with a reference answer. Both approaches still depend on the reference being correct and complete.
Verified sources
A frequent beginner assumption: that once a RAG system has been confirmed faithful — not hallucinating beyond its sources — its answers can be trusted as correct. As this article’s textbook-error analogy demonstrated, this isn’t a safe assumption at all — faithfulness and correctness are genuinely independent measurements, and a real, responsible evaluation pipeline tracks both together, since either one alone can hide a real, serious quality problem the other would catch.
Where this fits in what comes next
You now understand both major dimensions of factual quality — sticking to sources (faithfulness) and being objectively right (correctness). The next article, Toxicity, shifts to a genuinely different concern entirely — not whether an answer is accurate, but whether it’s safe, covering a real, documented case of an evaluation tool itself being fooled by a surprisingly simple trick.
In one sentence
Answer correctness measures whether a response is factually accurate against genuine, independent ground truth, combining semantic similarity and factual overlap checks, and it exists specifically because faithfulness alone — checking only whether an answer matches its given source — can’t catch the real, serious case of an answer being perfectly faithful to source material that was itself wrong.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed