Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Model Interpretability

nounAI Evaluation and Reliabilityalso interpretable machine learning, interpretable AI, model transparency

In one line

Model interpretability is the degree to which a human can understand how a machine learning model makes its predictions, either because the model is simple enough to read directly or because tools explain its behavior.

Model interpretability is the degree to which a person can understand how a machine learning model arrives at its predictions. An interpretable model lets a human see what the model pays attention to, how its inputs combine and why it gave a particular answer. Interpretability supports debugging, trust, fairness checks, regulatory compliance and scientific insight. It is closely related to explainable AI, which emphasizes producing explanations for users, and the two terms are often used together, though some authors use them differently as described below.

Why interpretability matters

  • Debugging. Understanding why a model errs reveals data problems or shortcuts, such as relying on a background cue.
  • Trust and adoption. Doctors, judges and loan officers are reluctant to rely on advice they cannot examine.
  • Fairness. Seeing which features drive decisions helps detect bias; see algorithmic bias and fairness in AI.
  • Regulation and rights. Rules in some jurisdictions require meaningful information about automated decisions, and credit laws require reasons for adverse actions; see AI regulation.
  • Safety. Knowing how a system works helps predict where it will fail; see AI safety.
  • Science. Models that capture real relationships can generate hypotheses.

Interpretability and explainability

The terms overlap. A common distinction is that interpretability refers to how inherently understandable a model is, such as a short list of rules, while explainability refers to methods that produce explanations for models that are not transparent in themselves. Others use the words interchangeably. What matters in practice is who needs to understand what, and for which purpose.

Interpretable models by design

Some models can be read directly [1].

  • Linear and logistic regression. Each feature has a coefficient showing its effect on the prediction.
  • Decision trees. A flowchart of yes or no questions leads to a prediction. Small trees are easy to follow; large ones are not.
  • Rule lists and scorecards. Short sets of if-then rules, or point systems such as those used in medicine and credit.
  • Generalized additive models. Sum of simple curves, one per feature, which can be plotted.
  • Sparse models. Methods that use few features.

Rudin has argued that for high-stakes decisions in areas such as healthcare and criminal justice, such inherently interpretable models should be preferred, because explanations of black boxes can be unfaithful and because interpretable models often match their accuracy on tabular data [4].

Explaining complex models (post-hoc methods)

Deep networks and large ensembles are not readable directly, so tools approximate or probe them.

### Global explanations

  • Feature importance. Scores showing which inputs matter most overall, computed from the model’s structure or by permuting a feature and measuring the loss in accuracy.
  • Partial dependence and accumulated local effects. Plots of how predictions change as one feature varies.
  • Surrogate models. A simple model trained to imitate a complex one.

### Local explanations

  • LIME. Explains a single prediction by perturbing the input and fitting a simple model nearby [2].
  • SHAP. Assigns each feature a contribution for a prediction using Shapley values from cooperative game theory, with desirable consistency properties [3].
  • Counterfactual explanations. Show the smallest change that would flip the outcome, such as “the loan would be approved if income were 5,000 higher.”
  • Anchors and rules. Conditions that make a prediction stable.

### For images and text

  • Saliency maps and Grad-CAM. Highlight the pixels or regions that influenced a vision model’s output.
  • Attention visualization. Shows which words a transformer attends to, though attention is not always a faithful explanation.
  • Example-based explanations. Show similar training examples or influential data points.

### Inside neural networks

Mechanistic interpretability tries to reverse-engineer the internal circuits and features of networks, including large language models, by studying neurons, directions in activation space and how information flows. It is an active research area aimed at understanding and controlling model behavior; see AI alignment.

Limits and cautions

  • Faithfulness. An explanation may not reflect what the model actually does. Some methods are unstable or can be manipulated.
  • Correlated features. Importance scores can be misleading when inputs are related.
  • Plausible but wrong stories. People may trust a neat explanation more than is justified.
  • Accuracy trade-off. Simpler models sometimes lose accuracy, though less often than assumed on structured data.
  • Audience. An explanation for a data scientist differs from one for a patient.
  • Not causation. Feature importance shows association in the model, not causes in the world.
  • Generated explanations. When a language model writes its own explanation, it may not match its real computation.

Evaluating explanations

Good explanations should be faithful to the model, stable under small input changes, understandable to the intended audience and useful for a task, such as helping a human detect an error. Studies with real users, and checks such as removing the features an explanation calls important and seeing whether predictions change, help assess quality.

A loan example

A bank uses a model to decide on small loans. With a logistic regression, an applicant who is declined can be told exactly which factors lowered the score: a short credit history and a high ratio of debt to income, each with a measurable effect. If the bank instead uses a large gradient boosted ensemble for higher accuracy, no coefficient list exists, so it adds SHAP values to show that, for this applicant, the debt ratio pushed the score down most, while steady income pushed it up. A counterfactual explanation tells the applicant that reducing debt by a given amount would change the decision. Compliance staff check that the reasons given to applicants match what the model actually does, and that protected traits and close proxies are not driving outcomes. If the explanations prove unstable or confusing, the bank may decide to use the simpler model, accepting a small loss in accuracy in return for clarity.

Choosing an approach

The right level of interpretability depends on the stakes and the audience. For a music recommender, little is needed. For a medical triage tool or a parole risk score, people affected have a strong interest in understanding and contesting the decision, and regulators may require it. A practical rule is to start with the simplest model that meets accuracy needs, to test whether a complex model really adds value, and to document what each explanation method can and cannot show. Interpretability should be planned from the start of a project, with data features that people can understand, instead of added at the end.

Human factors

Explanations are communication, and communication can fail. People tend to over-trust any explanation, even a weak one, and to accept it when it matches their expectations. Good design presents explanations at the right level of detail, shows uncertainty, allows people to question the output and tests with real users whether explanations improve decisions. See human-in-the-loop AI and AI transparency.

FAQs

What is model interpretability? It is how well a human can understand how a machine learning model makes its predictions.

What is the difference between interpretability and explainability? Interpretability is the inherent understandability of a model, and explainability is producing explanations for models, though the terms are often used interchangeably.

What are SHAP and LIME? Popular methods that explain individual predictions by showing how much each input contributed.

Are deep neural networks interpretable? Not directly. They need post-hoc tools or research methods such as mechanistic interpretability.

Is there a trade-off between accuracy and interpretability? Sometimes, but on many structured data problems simple models perform nearly as well as complex ones.

More in AI Evaluation and Reliability

Key facts

  • Interpretable models such as linear models and small decision trees can be read directly, while complex models such as deep networks usually need post-hoc explanation methods1.
  • LIME explains an individual prediction by fitting a simple model around it, and SHAP assigns each feature a contribution based on Shapley values from game theory23.
  • Rudin argued that for high-stakes decisions, inherently interpretable models should be preferred to explanations of black boxes4.

Related concepts

Quick checkWhat is model interpretability?Show answer

Model interpretability is the degree to which a human can understand how a machine learning model produces its predictions, either because the model is simple enough to read directly or because tools can explain its behavior.

Sources

  1. Christoph Molnar, Interpretable Machine Learning, https://christophm.github.io/interpretable-ml-book/
  2. Marco Ribeiro, Sameer Singh and Carlos Guestrin, Why Should I Trust You? Explaining the Predictions of Any Classifier, ACM SIGKDD, https://arxiv.org/abs/1602.04938
  3. Scott Lundberg and Su-In Lee, A Unified Approach to Interpreting Model Predictions, Advances in Neural Information Processing Systems, https://arxiv.org/abs/1705.07874
  4. Cynthia Rudin, Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, Nature Machine Intelligence, https://arxiv.org/abs/1811.10154

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue