Interpretability: Feature Importance, SHAP & Their Failure Modes

“Why did the model predict this?” can mean several incompatible questions:

• Which inputs were necessary for this fitted output?
• How would the output change under plausible input changes?
• Which features support accuracy over a population?
• Why did the real outcome occur?
• What action would change the outcome?

Interpretability tools generally answer one of the first three. The last two are causal questions and need a causal model or experiment.

Start with the intended consumer and claim #

A developer debugging leakage needs different evidence from a reviewer contesting one decision or a scientist testing a mechanism. Specify whether the explanation is local or global, model-specific or model-agnostic, descriptive or actionable, and which reference population it uses.

An intrinsically simple model can expose its mechanics: a sparse linear score or shallow tree path. But coefficients depend on scaling and collinearity, and tree paths depend on unstable greedy splits. Simplicity makes inspection possible; it does not make causal interpretation correct.

Permutation importance asks about predictive reliance #

After measuring baseline held-out performance, permute feature $j$ and measure the decrease:

$$I_j=S(f,X,y)-S(f,\pi_j(X),y).$$

A large decrease means the fitted model relied on information in that feature under this dataset and scoring rule. It is not an intrinsic property of the variable.

Correlated features create two opposite hazards. Permuting one breaks realistic joint combinations and can exaggerate importance. Or its redundant partner preserves performance and makes both look unimportant. Grouped/conditional permutation can better match a specific question but requires a model of dependence.

Compute importance only after establishing that the model has useful held-out performance. A bad model's feature ranking explains a bad model.

Partial dependence and individual curves #

Partial dependence averages predictions while setting feature $x_j$ to value $z$:

$$PD_j(z)=\mathbb E_{X_{-j}}[f(z,X_{-j})].$$

Individual conditional expectation (ICE) keeps one curve per case. Variation among ICE curves exposes interactions that the average can hide.

If $x_j$ is correlated with other features, substituting $z$ while holding them fixed can create impossible cases—adult age with child grade, or incompatible sensor states. The plotted response is then off-support model behavior, not observed evidence. Restrict ranges and show data density.

SHAP is an additive allocation relative to a background #

SHAP methods seek feature attributions $\phi_j$ satisfying an additive explanation

$$f(x)=\phi_0+\sum_j\phi_j,$$

where $\phi_0$ is an expected output under a background/reference distribution. Shapley values average a feature's marginal contribution over all orders in which feature coalitions could be introduced. This gives principled symmetry and additivity properties for the chosen value function.

The crucial phrase is chosen value function. To evaluate a coalition with missing features, one must define what “missing” means. Interventional variants break dependence and sample missing features independently; conditional variants preserve correlations through an estimated conditional distribution. They answer different questions and allocate credit differently.

The output scale also matters. Attributions on log-odds add on the score scale; they are not percentage-point probability changes. The background population sets the baseline, so changing it can change every explanation without changing the model or case.

Duplicate-feature counterexample #

Let $x_2=x_1$ be an exact copy and let the model use either interchangeably. Any fair attribution rule may split credit between them. Remove one copy and the survivor receives all credit, though predictions are unchanged. Therefore a small individual attribution does not mean the underlying information is unimportant, and an attribution is not a uniquely identifiable causal contribution.

Explanations can be unfaithful or manipulable #

Model-agnostic local methods probe perturbed points around a case. If those points are off the data manifold, an adversarial model can behave benignly on probes and differently on real data, producing a plausible but false explanation. Surrogate fidelity must be measured in the relevant neighbourhood, not assumed from an attractive chart.

Attributions can also expose protected information or enable gaming. Explanation access is an interface with security and privacy consequences.

A validation protocol for explanations #

1. State the exact question, output scale, background, and dependence assumption.
2. Test on models with known synthetic mechanisms and realistic correlations.
3. Measure local surrogate fidelity or recompute the model after feature ablation.
4. Check stability across retraining, seeds, nearby cases, and background samples.
5. Inspect whether highlighted features are available, lawful, and semantically valid.
6. Compare explanation-driven hypotheses against raw data or interventions.

An explanation is evidence about a specified model query. It does not absolve the model from validation, and it should never be promoted silently into an explanation of the outcome itself.