Explainability

Every phase of this glossary so far has explained how AI systems work, from the mathematics of gradient descent to the architecture of a Transformer. This final phase turns to a different, equally important question: can the systems themselves explain what they’re doing? It starts with explainability.

The simple definition

Explainability is the ability to provide a human-understandable account of why a model produced a specific output. Recall from the Reasoning article’s careful distinction between a model producing genuinely useful step-by-step output and a model possessing real understanding. Explainability sits in a related but distinct space — it’s not about whether a model understands itself, but about whether someone else can get a clear, honest account of why it did what it did.

Why this matters, connecting directly to real-world stakes

Recall from the Evaluation Basics phase’s discussion of measuring model performance — accuracy alone tells you whether a model got something right, not why. For a spam filter, that might not matter much. For a model denying someone a loan, flagging a medical scan, or recommending a prison sentence, the “why” is often just as important as the “what” — genuinely affecting whether an affected person can contest a decision, whether a regulator can audit a system, and whether an engineer can actually find and fix a real problem.

flowchart LR
    A[Model makes a decision] --> B[Explainability: after-the-fact explanation]
    B --> C["e.g., 'this loan was denied primarily due to high debt-to-income ratio'"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of asking an experienced doctor why they recommended a specific treatment. The doctor doesn’t walk you through every neuron firing in their brain — they give you a clear, human-level account: “your bloodwork showed elevated markers, combined with your family history, which points toward this specific treatment.” That account is genuinely useful and actionable, even though it’s a simplified summary of a much more complex underlying decision process.

Where this breaks down: A doctor’s explanation reflects genuine, conscious reasoning they can accurately introspect on. A model’s explanation, as covered in the next section, is often generated by a separate technique analyzing the model’s behavior from the outside — it’s a plausible, useful account of what likely mattered, not necessarily a transparent window into the actual computation that occurred, a distinction the next article, Interpretability, will make precise.

How explainability actually gets produced in practice

This is worth being concrete about, since real explainability techniques work by probing a model’s behavior rather than reading its internals directly. Two widely used, published methods illustrate this: LIME (Local Interpretable Model-agnostic Explanations) approximates a complex model’s behavior around one specific prediction with a much simpler, genuinely understandable model, then reports which input features that simple approximation weighted most heavily.

SHAP (SHapley Additive exPlanations) borrows a concept from cooperative game theory to fairly attribute a prediction’s outcome across each individual input feature, producing a clean, additive breakdown of “how much did each factor contribute.” Both are explicitly “model-agnostic” — they work by observing input-output behavior, not by reading the model’s actual weights.

A concrete example, layered

For a simple beginner example: a model denying a loan application might produce an explanation like “the three factors that most influenced this decision were: debt-to-income ratio (40%), credit history length (25%), and recent late payments (20%)” — a genuinely useful, actionable account, generated by a technique like SHAP analyzing the model’s behavior around that one specific application.

For a production example: financial institutions deploying credit-scoring models are, in many jurisdictions, legally required to provide exactly this kind of explanation to a rejected applicant — a real, regulatory-driven reason explainability techniques like SHAP have become standard, deployed infrastructure in real lending systems, not just a research curiosity.

Why explainability has a real, honest limitation

It’s worth being direct about a genuine trade-off, not presenting explainability as a fully solved problem. An explanation generated by LIME or SHAP is an approximation of the model’s behavior near one specific case — it can be a genuinely useful, accurate summary, but it isn’t a guaranteed, complete account of the model’s actual full decision process, and different explanation techniques applied to the exact same prediction can sometimes highlight different factors, a real, documented inconsistency actively studied in the field.

The real story: when nobody, including the bank, could explain a credit decision

This deserves to be told in full, because it’s a genuine, high-profile case of exactly what happens when explainability is missing. In November 2019, software developer David Heinemeier Hansson tweeted that his new Apple Card had given him a credit limit 20 times higher than his wife’s, despite her having a higher credit score and the couple filing joint tax returns — calling it “a fucking sexist program.” Apple co-founder Steve Wozniak publicly reported an identical experience.

The tweets went viral, and the New York State Department of Financial Services opened a formal investigation into Goldman Sachs, which managed the card’s underwriting. The core problem was explainability itself: neither Goldman Sachs nor Apple could offer customers a clear, specific account of why the algorithm had reached its decision — Goldman’s own spokesperson could only say credit decisions were “based on a customer’s creditworthiness,” with no ability to explain the individual case.

Under public and regulatory pressure, Goldman re-reviewed the credit files of women who had received lower limits than their spouses and raised many of them to match, and eliminated its six-month waiting period for appeals.

After reviewing hundreds of thousands of pages of records and underwriting data covering nearly 400,000 New York applicants, the NYDFS ultimately found no illegal discrimination in 2021 — but the case had already forced real, lasting changes to Goldman’s appeals process, and became a widely cited example of why regulators increasingly expect lenders to provide individual, case-specific explanations, not just a general assurance that a system is fair.

Follow one prediction from input to explanation

Suppose a small loan model calculates a risk score from three features: debt ratio, missed payments, and income stability. It returns a score of 72 out of 100, where a higher number means greater risk.

Applicant's numbers
        |
        v
Loan model ---------> Risk score: 72
        |
        v
Explanation tool ---> Debt ratio: +18
                      Missed payments: +11
                      Stable income: -7

The model produced 72. The explanation tool then estimated how the features pushed that result up or down compared with a reference applicant. It did not discover a sentence secretly stored inside the model. This distinction is important: an explanation can describe which inputs influenced an output, while a human still has to decide whether those inputs are appropriate, whether the model is accurate, and whether the decision is fair.

Local and global explanations

A local explanation answers, “Why did the model make this one prediction for this one person or image?”
A global explanation answers, “What patterns does the model usually rely on across many predictions?” A local explanation might say that one application was affected by recent missed payments. A global explanation might show that debt ratio is usually the strongest feature across 100,000 applications. Neither one automatically proves cause and effect.

A current production connection

Modern AI products use several layers of explanation rather than one magic “why” button. A customer-service application may show citations for retrieved facts, record which tools an agent called, and keep traces of the steps taken. These records explain the application’s observable path, but they are not a complete map of every internal calculation inside GPT or Gemini.

NIST lists explainability and interpretability among the characteristics of trustworthy AI and warns that explanations must suit the knowledge level of the person using them. That is why an engineer may need feature-attribution charts while a customer needs a short, plain-language reason and a way to appeal.

Common misconception

GPT and Gemini application examples

Imagine asking an AI research assistant, “Which planet has the shortest day?” The application may do this:

Your question
    -> model creates a search query
    -> search tool returns webpages
    -> model writes "Jupiter"
    -> application displays citations beside the answer

The citations explain where the answer’s factual support came from. A tool trace can also show which search was performed and what the tool returned. Neither one reveals every calculation inside the model.

OpenAI’s deep-research system and Gemini’s Google Search grounding use this application-level approach. They can return answers connected to web sources. This makes factual claims easier to check, although a citation can still be misunderstood or attached to a claim it does not fully support.

Remember: citations explain evidence, traces explain visible actions, and interpretability investigates internal mechanisms. These are related, but they are not the same thing.

A frequent beginner assumption: that an “explanation” a model provides is a literal, transparent readout of its internal computation — that the model is directly reporting what it actually did. As the LIME/SHAP mechanism above explained, this usually isn’t accurate — most explainability techniques generate a plausible, useful, after-the-fact account by observing behavior, not by directly reading the model’s genuine internal reasoning process, a distinction the next article makes central.

Where this fits in what comes next

You now understand explainability as producing a human-understandable account of a model’s output. The next article, Interpretability, covers the closely related but genuinely distinct concept — not explaining a decision after the fact, but actually being able to understand a model’s internal workings directly.

In one sentence

Explainability is the ability to provide a clear, human-understandable account of why a model produced a specific output, commonly generated through techniques like LIME and SHAP that observe a model’s behavior rather than read its internals directly — a genuinely practical, often legally required capability, even though it’s an approximation rather than a complete, guaranteed account.

Verified sources

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026