Confusion Matrix

The Accuracy article ended with a clear promise: a tool that reveals what accuracy hides, by breaking predictions down into their specific types rather than lumping every outcome into one number. That tool is the confusion matrix.

The simple definition

A confusion matrix is a simple grid that breaks a classification model’s predictions down into four specific categories: correct positive predictions, correct negative predictions, and the two distinct kinds of mistakes. Rather than saying “the model was right 90% of the time,” a confusion matrix says something much more specific and useful: exactly how many times, and in exactly which direction, the model was wrong.

The four cells, explained plainly

For any task where a model predicts “yes” or “no” — spam or not-spam, fraud or not-fraud, disease or no-disease — every single prediction falls into exactly one of four boxes:

• True Positive (TP) — the model predicted “yes,” and it actually was “yes.” A correctly caught spam email.
• True Negative (TN) — the model predicted “no,” and it actually was “no.” A correctly ignored legitimate email.
• False Positive (FP) — the model predicted “yes,” but it actually was “no.” A legitimate email wrongly flagged as spam.
• False Negative (FN) — the model predicted “no,” but it actually was “yes.” A real spam email that slipped through undetected.

flowchart LR
    A[Every prediction] --> B{Predicted Yes or No?}
    B -->|Predicted Yes| C{Actually Yes or No?}
    B -->|Predicted No| D{Actually Yes or No?}
    C -->|Actually Yes| E[True Positive]
    C -->|Actually No| F[False Positive]
    D -->|Actually Yes| G[False Negative]
    D -->|Actually No| H[True Negative]

Why breaking predictions into four categories matters so much

Recall from the Accuracy article’s disease-detection example: a model that always predicts “no disease” achieves 99% accuracy on a rare disease, but a confusion matrix would expose this immediately. Its True Positives would be zero — the model never once correctly caught an actual sick patient — and its False Negatives would equal every single real case, missed entirely. Accuracy, calculated from all four cells combined, hid this catastrophic failure inside one deceptively reassuring number. The confusion matrix doesn’t hide it at all; the zero in the True Positive box is impossible to miss.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a security guard reviewing camera footage for intruders. A simple pass/fail grade — “the guard was right 95% of the time” — doesn’t tell you whether the 5% of mistakes were false alarms (flagging an innocent passerby as an intruder) or missed break-ins (an actual intruder walking by unnoticed). Those two kinds of mistakes have wildly different consequences, and a confusion matrix is exactly the record that separates them out clearly.

Where this breaks down: A security guard’s mistakes involve human judgment calls under real-time pressure. A confusion matrix’s four cells are calculated mechanically, by directly comparing a model’s predictions to ground truth after the fact — no judgment involved in the counting itself, only in deciding, afterward, which of these four outcome types matters most for the task at hand.

Why the two error types genuinely matter differently

This is the whole point of breaking accuracy apart this way, and it’s worth being explicit: False Positives and False Negatives are not interchangeable, and different tasks care about them very differently. A spam filter with too many False Positives (legitimate emails wrongly blocked) frustrates users badly; a few False Negatives (occasional spam slipping through) is merely mildly annoying. A cancer-screening model has the relationship reversed — a False Negative (a real cancer case missed) can be life-threatening, while a False Positive (an unnecessary follow-up test for a healthy patient) is a much smaller cost, though not a trivial one. No single accuracy number can capture this asymmetry; the confusion matrix is what makes it visible in the first place.

Build the matrix from 100 predictions

Continue with the screening example: 20 people truly have the condition and 80 do not. The model flags 26 people in total.

	Actually positive	Actually negative	Row total
Predicted positive	TP = 18	FP = 8	26 flagged
Predicted negative	FN = 2	TN = 72	74 cleared
Column total	20 sick	80 healthy	100 people

Read each cell as a full sentence:

• TP = 18: 18 sick people were correctly flagged.
• FP = 8: 8 healthy people were incorrectly flagged.
• FN = 2: 2 sick people were incorrectly cleared.
• TN = 72: 72 healthy people were correctly cleared.

The words positive and negative mean the predicted or actual class; they do not mean good and bad. The words true and false tell us whether the prediction matched reality.

Every metric reads this same table differently

Accuracy  asks: How many predictions were correct overall?       (TP + TN) ÷ all
Precision asks: Of the positive predictions, how many were right? TP ÷ (TP + FP)
Recall    asks: Of the actual positives, how many did we find?     TP ÷ (TP + FN)

Before calculating any metric, decide which class is the “positive” class. In fraud detection it may be fraud; in screening it may be disease. Swapping the positive class changes precision and recall.

A concrete example, layered

For a simple beginner example: a spam filter evaluated against 100 emails (10 actually spam, 90 actually legitimate) might produce a confusion matrix showing 8 True Positives, 2 False Negatives, 85 True Negatives, and 5 False Positives — revealing that it caught 8 of 10 real spam emails, missed 2, and wrongly flagged 5 legitimate emails as spam, a far richer picture than a single 93% accuracy figure would convey. For a production example: a bank’s fraud detection model’s confusion matrix is reviewed directly by the fraud operations team, not just its overall accuracy, because the bank cares specifically about minimizing False Negatives (real fraud that slips through, costing real money) while keeping False Positives (legitimate transactions wrongly flagged, frustrating customers) at a tolerable level — a genuine business trade-off that only becomes visible once predictions are broken into these four specific categories.

Real AI-agent example: count the safety outcomes

The Operator System Card reports two useful evaluation slices for its prompt-injection monitor:

• On 77 attack attempts, it missed one borderline attempt. In that attack-only slice, this corresponds to 76 detected attacks and 1 missed attack.
• On 13,704 benign screens, it incorrectly flagged 46. In that benign-only slice, 13,658 passed without being flagged and 46 were false alarms.

Attack slice: TP = 76, FN = 1
Benign slice: TN = 13,658, FP = 46

These slices show why teams inspect the cells instead of reporting only “the monitor works.” One missed attack and 46 interrupted benign screens create different risks and require different engineering responses. Because the report presents multiple evaluation sets, their counts should be understood in their stated slices rather than silently treated as one randomly sampled production dataset.

Common misconception

A frequent beginner assumption: that a confusion matrix is itself a single metric, like accuracy, that produces one number to judge a model by. It isn’t — it’s a breakdown, a foundation for calculating several different, more specific metrics from, each emphasizing a different aspect of performance. The next two articles, Precision and Recall, are both calculated directly from the four cells introduced here, and F1 Score, covered after that, combines them into a single balanced number — meaning nearly everything left in this phase is really just different ways of reading the same underlying confusion matrix.

Where this fits in what comes next

You now have the foundational breakdown every remaining metric in this phase builds from. The next article, Precision, takes the confusion matrix’s four cells and asks a specific, practically important question: of everything the model flagged as positive, how much of it was actually correct?

In one sentence

A confusion matrix breaks a classifier’s predictions into four specific outcomes — True Positive, True Negative, False Positive, and False Negative — revealing exactly what kind of mistakes a model makes, which a single accuracy number can never show on its own.

Related Terms

• Accuracy
• Evaluation
• Precision
• Recall
• F1 Score