Rational Shapley Values

David S. Watson
Department of Statistical Science, University College London
London, UK
[email protected]
ABSTRACT as interpretable machine learning or explainable artificial intelligence
Explaining the predictions of opaque machine learning algorithms (XAI). For good introductory surveys, see, e.g., [2, 17, 33].
is an important and challenging task, especially as complex models The most prominent methods in this burgeoning area are post-
are increasingly used to assist in high-stakes decisions such as those hoc, model-agnostic tools such as feature attributions and counter-
arising in healthcare and finance. Most popular tools for post-hoc factuals. Both approaches have recently come under fire for failing
explainable artificial intelligence (XAI) are either insensitive to to properly handle dependencies between covariates. Most algo-
context (e.g., feature attributions) or difficult to summarize (e.g., rithms for computing Shapley values – a popular feature attribution
counterfactuals). In this paper, I introduce rational Shapley values, technique, formally defined in Sect. 2 – implicitly treat predictors
a novel XAI method that synthesizes and extends these seemingly as mutually independent, assigning positive weight to potentially
incompatible approaches in a rigorous, flexible manner. I leverage impossible data permutations [16, 22]. Much recent work in coun-
tools from decision theory and causal modeling to formalize and terfactual explanation has focused on how to model the impact
implement a pragmatic approach that resolves a number of known of interventions on downstream variables, a reformulation that
challenges in XAI. By pairing the distribution of random variables requires an explicitly causal approach [21, 29].
with the appropriate reference class for a given explanation task, Others have pushed back against these critiques. Sundararajan
I illustrate through theory and experiments how user goals and and Najmi [42] demonstrate that conditioning on covariates can
knowledge can inform and constrain the solution set in an iterative misleadingly assign importance to irrelevant features. Janzing et al.
fashion. The method compares favorably to state of the art XAI [19] point out that standard supervised learning algorithms do not
tools in a range of quantitative and qualitative comparisons. explicitly model dependencies between features, and so intervening
to set predictors to some fixed value is not just permissible but
CCS CONCEPTS proper when computing Shapley values.
Often lost in these debates is the seemingly obvious realization
• Theory of computation → Machine learning theory; • Math-
that the “right” explanation depends crucially on who is asking and
ematics of computing → Probability and statistics.
why. Current explainability methods tend to treat explanations as
objective deliverables that can be computed from just two ingre-
KEYWORDS dients: a target function and an accompanying dataset of real or
Explainable artificial intelligence, Interpretable machine learning, simulated samples. This ignores a basic fact long acknowledged in
Shapley values, Counterfactuals, Decision theory philosophy and social science – that explanations are fundamentally
ACM Reference Format: pragmatic. To succeed, they must be tailored to the inquiring agent,
David S. Watson. 2022. Rational Shapley Values. In 2022 ACM Conference who requests an explanation on the basis of certain beliefs and
on Fairness, Accountability, and Transparency (FAccT ’22), June 21–24, 2022, interests.
Seoul, Republic of Korea. ACM, New York, NY, USA, 12 pages. https://doi. If pragmatic approaches are few and far between in the XAI
org/10.1145/3531146.3533170 literature, perhaps this is because it is not obvious how we should
formalize abstract, subjective notions like beliefs and interests. In
1 INTRODUCTION this paper, I use tools from Bayesian decision theory to do just that. I
Machine learning algorithms are increasingly used in a variety of make three main contributions: (1) I combine disparate work in fea-
high-stakes domains, from credit scoring to medical diagnosis. How- ture attributions and counterfactuals, demonstrating how the two
ever, many of the most popular and powerful models are opaque, can join forces to create more useful and flexible explanations. (2) I
insomuch as end users cannot understand the reasoning that goes extend the axiomatic guarantees of Shapley values, allowing user
into individual predictions. A subdiscipline of computational statis- beliefs and preferences to guide the search for a relevant subspace
tics has rapidly emerged to address this issue, variously referred to of contrastive baselines. (3) I implement an expected utility maxi-
mization procedure that optionally incorporates causal information
Permission to make digital or hard copies of all or part of this work for personal or to compute optimal attribution sets for individual agents.
classroom use is granted without fee provided that copies are not made or distributed The remainder of this paper is structured as follows. In Sect. 2,
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
I review important background material in XAI, with a focus on
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or Shapley values and counterfactuals, as well as the essential for-
republish, to post on servers or to redistribute to lists, requires prior specific permission malisms of decision theory and structural causal models. In Sect. 3,
and/or a fee. Request permissions from [email protected].
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea
I compare several different reference distributions, propose a new
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM. desideratum for Shapley values, and introduce an algorithm for
ACM ISBN 978-1-4503-9352-2/22/06. . . $15.00 computing them. In Sect. 4, I put this method to work on a number
https://doi.org/10.1145/3531146.3533170
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

of benchmark datasets. Following a discussion in Sect. 5, Sect. 6 of a conditional expectation, with x iS held fixed while x R varies.
concludes. Following Merrick and Taly [30], I consider a general form for the
value function, parametrized by a distribution DR :
2 BACKGROUND
v DR (i, S) = E [f (x iS , x R )]. (3)
In what follows, I use uppercase italics to represent variables, e.g. x R ∼DR
X ; lowercase italics to represent their values, e.g. x; uppercase bold-
Popular options for DR include the marginal P(X R ), as in Lundberg
face to represent matrices, e.g. X ; lowercase boldface to represent
and Lee [28]; the conditional P(X R |x iS ), as in Aas et al. [1]; and the
vectors, e.g. x; and calligraphic type to represent the support of a
interventional P(X R |do(x iS )), as in Heskes et al. [18]. I revisit the
distribution, e.g. X.
distinction between these distributions in Sect. 3.1.
2.1 XAI Methods Counterfactual explanations, unlike Shapley values, do not pro-
duce feature weights. Instead, they identify nearby datapoints with
Originally developed in the context of cooperative game theory alternative outcomes – the eponymous counterfactuals – intended
[38], Shapley values have been adapted for model interpretability by to explain predictions by highlighting some (preferably small) set
numerous authors. They represent the contribution of each feature of input perturbations. The optimization problem can be expressed
toward a particular prediction. Let x i ∈ X ⊆ Rd denote an input as
datapoint and f (x i ) = ŷi ∈ Y ⊆ R the corresponding output of our x ∗ = argmin d(x i , x̃), (4)
target function f for each sample i ∈ [n] = {1, . . . , n}.1 Shapley x̃ ∈CF(x i )
values express this value as a sum: where CF(x i ) denotes a counterfactual space, where f (x i ) , f (x̃)
d
Õ (for classification) or f (x i ) = f (x̃) + δ (for regression), where δ is
f (x i ) = ϕ j (i), (1) a hyperparameter and d is some distance measure (unspecified for
j=0 now). The method was originally introduced by Wachter et al. [45],
where ϕ 0 (i) represents i’s “baseline” (i.e., the default output of f ; who use generative adversarial networks (GANs) to solve Eq. 4.
more on this shortly) and ϕ j (i) the weight assigned to feature X j Others have proposed a variety of alternatives designed to ensure
at point x i . Feature attribution methods that satisfy Eq. 1 are said that counterfactuals are coherent and/or actionable [20, 35, 37, 43].
to be efficient. Let v : [n] × 2d → R be a value function such that
v(i, S) is the payoff associated with feature subset S ⊆ [d] and 2.2 Bayesian Decision Theory
v(i, {∅}) = 0 for all i ∈ [n]. The Shapley value ϕ j (i) is given by a Why do we use XAI tools in the first place? Watson and Floridi [47]
weighted average of j’s marginal contributions to all subsets that list three reasons: (1) to audit for potential bias; (2) to validate per-
exclude it: formance, guarding against unexpected errors; and (3) to discover
1 Õ underlying mechanisms of the data generating process. Another
ϕ j (i) = |S |!(d − |S | − 1)! v(i, S ∪ {j}) − v(i, S). (2)
d! use case, with elements of (1) and (3) yet distinctly analyzed in the
S ⊆[d ]\{j }
algorithmic recourse setting, is (4) to recommend actions so as to
Several properties of this value are worth noting. alter predicted outcomes. In all four cases, the aim is ultimately
• Linearity. For all j ∈ [d], all a, b ∈ R, and value functions to make some sort of decision – be it about whether to sue a firm,
v 1 , v 2 , we have: deploy an algorithm, or perform an experiment. These motivations
ϕ j (av 1 + bv 2 ) = aϕ j (v 1 ) + bϕ j (v 2 ), may overlap at the edges but they represent distinct tasks based on
different assumptions and requiring their own explanatory method-
where ϕ j (v) makes explicit the dependence of attributions ologies. This heterogeneity is largely ignored by current feature
on value functions. (Indeed, ϕ j only depends on i through attribution and counterfactual approaches, which implicitly assume
v.) a sort of explanatory objectivism. According to this view, the quality
• Sensitivity. If v(i, S ∪ {j}) = v(i, S) for all S ⊆ [d]\{j}, then of an explanation is independent of its context.
ϕ j (i) = 0. Decades of research in philosophy and social science has made
• Symmetry. If v(i, S ∪ {j}) = v(i, S ∪ {k}) for all S ⊆ [d]\{j, k }, such objectivism untenable [15, 26, 31]. If the view has any adher-
then ϕ j (i) = ϕ k (i). ents today, perhaps it is because the alternative is often miscon-
It can be shown that Shapley values are the unique solution to the strued as relativism, an anything-goes slippery slope that inevitably
attribution problem that satisfies efficiency, linearity, sensitivity, leads to what Feyerabend [13] (approvingly) refers to as “epistemo-
and symmetry [38]. Computing exact Shapley values is NP-hard, logical anarchism”. This, however, is a false dichotomy. I reject the
although numerous efficient approximations have been proposed polar extremes of objectivism and relativism in favor of pragmatism,
[42]. which holds neither that there exists some single ideal explanation
There is some ambiguity as to how one should calculate payoffs nor that all explanations are equally valid, but rather that various
on a proper subset of features, since f requires d-dimensional input. explanations may be more or less appropriate depending on who
Let R = [d]\S, such that we can rewrite any x i as a pair of subvec- requests them and why.
tors (x iS , x iR ). Then the payoff for feature subset S takes the form To operationalize this insight, I rely on the formal apparatus
1 The
of decision theory. Let A and H denote finite sets of actions and
model f may be a classifier, although in this case we typically require class
probabilities and work on the logit scale. For ease of exposition, in this section we will outcomes, respectively. An agent’s preferences over action-outcome
assume f is a regressor. pairs induce a partial ordering that we represent with a utility
Rational Shapley Values FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea

function u : A × H → R. Let p(·) denote a credence function potential outcomes, but instead by minimizing expected loss as
over outcomes such that p(H ) = 1 and p(h|E) denotes the agent’s computed over a set of observed data. A similar premise under-
subjective degree of belief in some h ∈ H given a body of evidence lies recent work in causal reinforcement learning [24, 25], where
E = {e 1 , . . . , en }. The expected utility of each action a ∈ A is given the goal is to find optimal interventions by maximizing expected
by a weighted average over hypotheses: rewards over trials.
Õ
E [u(a, h)|E] = p(h j |E) u(a, h j ). (5)
h∼p j
3 RATIONAL SHAPLEY VALUES
If u satisfies the utility axioms (i.e., completeness, transitivity, con- My basic strategy is to allow user inputs to inform the generation of
tinuity, and independence) and p satisfies the probability axioms counterfactuals and estimation of Shapley values, thereby resulting
(i.e., non-negativity, unit measure, and σ -additivity), then we say in custom feature attributions that are more useful than either
that the agent is rational, and it can be shown that she will tend method on its own. Intuitively, this gives agents the tools to pose
to maximize expected utility [44]. That is, she will always choose not just generic (i.e., objectivist) questions of the form “Why did
some optimal action: model f predict outcome ŷi for input x i ?”, but more targeted (i.e.,
pragmatic) questions, such as: (1) Why did f predict ŷi as opposed
a ∗ = argmax E [u(a, h)|E] (6) to ỹ for x i ? (2) Why did f predict ŷi as opposed to ỹ for x i , given
a ∈A h∼p
x iS ? (3) Why did f predict ŷi as opposed to ỹ for x i , given x iS
from a set of alternatives.
and certain constraints on X R ? Unlike counterfactuals, which may
This expected utility framework is very general, and applies to ex-
confuse users by generating a large number of synthetic datapoints
planation instances of all four types cited above. For instance, in an
with no clear takeaway message, the proposed method provides
auditing example where a data subject believes her loan application
unambiguous weights for all variables of interest. Unlike feature
was denied due to some sensitive attribute(s), we might have A =
attributions, which ignore pragmatic information by construction,
{sue, ¬sue} and H = {biased, ¬biased}, with evidence provided
the present approach is rational in the decision theoretic sense.
by some XAI tool and fairness criteria [4]. In algorithmic recourse,
I will make these claims more precise and apparent below. First,
we typically consider more diverse action sets. For example, a
however, I introduce the notion of a relevant subspace to formalize
credit scoring case may include A = {education, income, credit},
the hierarchy of specification implied by the questions above.
where increasing any subset of those variables incurs some cost
but may improve outcomes H = {approve, ¬approve}. Definition 1 (Relevant subspace). Let Y0 , Y1 be a partition of Y
A key consideration, often ignored by authors in XAI, is the into baseline and contrastive outcomes, respectively, with ŷi ∈ Y0 .
difference between model- and system-level explanations [46]. At Let S ⊆ [d] denote a (potentially empty) conditioning set with com-
the model level, for instance, it is typically appropriate to treat plement R = [d]\S. Let DS be the distribution of possible values
all features as mutually independent and observe outputs on real for the conditioning set. Let DR be a distribution encoding probable
and/or synthetic inputs that vary in some systematic fashion. At variation in X R as a function of x S . Define Z = X × Y. We say that
the system level, by contrast, we cannot afford to ignore causal counterfactual sample z̃ = (x̃ S , x̃ R , ỹ = f (x̃ S , x̃ R )) is relevant for
relationships between features, as an intervention on one variable the inquiring agent if and only if: (i) ỹ ∈ Y1 ; (ii) x̃ S ∼ DS ; and (iii)
may have downstream effects on others. In these cases, nature x̃ R ∼ DR . Any space Z̃ ⊆ Z that meets these criteria is a relevant
constrains the input space in accordance with certain structural subspace.
dependencies.
This distinction is formalized using tools from the causal mod- The issue with popular Shapley value approximators (e.g., SHAP
eling literature [34]. A structural causal model (SCM) is a tuple [28]) is that baselines are fixed by the data centroid. Attributions
M = ⟨U , V , F ⟩, where U is a set of exogenous variables, i.e. back- are only designed to account for the discrepancy between f (x i )
ground conditions; V is a set of endogenous variables, i.e. observed and the empirical mean ϕ 0 (i) = n−1 ni=1 f (x i ). Moreover, it is
Í

states; and F is a set of structural equations, one for each Vj ∈ V . common to use the marginal P(X ) as the reference distribution
R
These functions map causes to direct effects in a deterministic fash- DR , thereby breaking associations between the features in S and R,
ion, f j : pa(Vj ) → Vj , where pa(·) denotes the “parents” of some resulting in a value function that implicitly compares each x i j to
endogenous variable(s). The union of Vj ’s direct and indirect causes the mean of the corresponding variable X j . Yet these values might
(parents, grandparents, etc.) is called its “ancestors”, denoted an(Vj ). not fall within the relevant subspace for a given agent, who may,
Pearl’s do-calculus [2000, Ch. 3.4] provides a provably complete for instance, want to know why she received an average credit
set of rules for identifying causal effects from observational data score instead of a better-than-average one. Or perhaps she wants
where possible [39]. to compare her predictions only to those of others in a similar age
A deterministic SCM can be expanded to accommodate stochastic range and income bracket, effectively setting attributions for those
relationships by placing a joint probability distribution on back- features to zero.
ground conditions, p(U ). By equating this with the credence func- Counterfactuals naturally handle such cases, identifying real
tion introduced above, we may encode an agent’s uncertainty with or synthetic datapoints that differ from the explanans in specific
respect to a causal system, which has implications for expected ways. However, there are several issues with the counterfactual
utility if H ∈ V . This decision theoretic approach to causality is approach. First, the cost function in Eq. 4 is almost always equated
explicitly endorsed by Dawid [8, 9, 10], who argues that treatment with some distance metric in practice, despite much discussion of
policies should be designed not by considering differences between more generic alternatives [20, 43, 48]. This is an unsatisfying notion
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

of costs, which can and should reflect the preferences of inquiring more useful, or more broadly applicable. Rather, each is the right
agents much more flexibly. For instance, relevant action sets may answer to a different question. What follows is a brief discussion
range over interventions exogenous to the feature space, where the of the advantages and disadvantages associated with each value of
notion of distance is often inapplicable. DR , as well some heuristic guidelines on when to use which.
Second, there is the question of how many counterfactuals to Marginal. The marginal distribution is preferable when the goal
provide. Eq. 4 suggests that just a single sample would suffice; how- is a model-level explanation. This may be the case in certain in-
ever, such a procedure overestimates the certainty with which such stances of auditing or validation, where the analyst seeks merely
a datapoint is generated. Can we really be sure that counterfactual to discover what the model has learned without any further restric-
x ∗ is to be preferred to counterfactual x ′ just because the former is tions. In this case, we may not care too much about impossible data
marginally closer in feature space to the input x i ? Even if we are perturbations (e.g., teenage grandmothers) since we simply want to
willing to accept a purely distance-based notion of cost, it is not recreate a decision boundary in model space. The extent to which
at all obvious that very small distances should be taken seriously this model space matches reality is another question altogether,
when there are so many potential sources of noise – in the mea- presumably one that data scientists should take seriously, both be-
surements themselves, in the finite training data, and in whatever fore and throughout deployment. Once the model is in the wild,
approximations were necessary to efficiently generate candidate however, it may be of independent interest to users and regulators
counterfactuals. That is why some authors prefer to sample a large, how it performs on a wide range of hypothetical cases, including
diverse coalition of counterfactuals [32], so that users can survey those off the true data manifold.
the various paths through which a prediction may change. Yet this Conditional. The conditional distribution is better suited to ex-
approach raises new problems of intelligibility. How should an planations at the system level, where impossible data permutations
agent make sense of a relevant subspace with dozens or hundreds could lead to issues. This may be the case, for instance, in certain
of samples within a tolerable distance of x i ? How should she prior- instances of validation, where we do not wish to punish a model for
itize among different counterfactuals or integrate this information failing to extrapolate to data points far from its training set. Such
into a high-level summary? methods are especially valuable when joint distributions may be
Feature attribution methods are well suited to this task. Shapley estimated with reasonable accuracy and causal relationships are
values provide an efficient and intuitive account of the information unknown or potentially inapplicable (e.g., in image classification
encoded in a large set of counterfactuals, regardless of sample size. tasks). Estimating joint densities is a hard problem in general, es-
I therefore propose a hybrid method in which data are sampled pecially when data are high-dimensional and/or of mixed variable
from a relevant subspace, and Shapley values computed only with types.
respect to such samples. The crucial observation is that nothing Interventional. The interventional distribution is optimal when
in the definition of Shapley values (Eqs. 1, 2, 3) precludes us from seeking explanations at the system level for causal data generating
shifting the reference distribution from an entire observational processes. With access to the underlying SCM, there can be no
dataset to a particular region with certain desirable properties – i.e., more accurate estimator than that defined by (3). These methods
a relevant subspace. are required when seeking to use XAI for discovery and/or planning,
as both sorts of actions invariably rely on real-world mechanisms
that cannot be approximated by either a purely marginal approach
3.1 Selecting the Reference Distribution (which ignores all dependencies) or a conditional one (which fails to
I take it more or less for granted that agents requesting explana- distinguish between correlation and causation). Of course, complete
tions for algorithmic predictions have some contrastive outcome causal information is almost never available, which means that in
Y1 in mind. They are also likely to have some intuition about the practice analysts must make strong assumptions and/or make do
conditioning set S, which will presumably include immutable fea- with workable approximations.
tures and/or any variables that the agent does not want to enter I employ marginal, conditional, and interventional value func-
into the explanation if it can be avoided. The precise distribution tions in all experiments below (see Sect. 4). To compute inter-
for these features may not be available a priori, but conditioning ventional Shapley values, I rely on partial orderings, as these
on the observed x iS is probably a good start when the data allow are often the most readily available to analysts at test time.
for it. The trickiest component to evaluate then is the reference dis- A partial ordering of features implies a chain graph in which
tribution DR , which has implications both for how counterfactuals links are composed of all variables in a single causal group.
are computed (since this helps define the relevant subspace) and For instance, users may not know the complete graph struc-
for how Shapley values are estimated (since this helps define the ture of the set {age, sex, income, savings, credit} but they may
value function). be more confident about the partial ordering {age, sex} →
I consider three possibilities: (1) marginal: P(X R ); (2) conditional: {income, savings} → {credit}. Lauritzen and Richardson [23]
P(X R |x iS ); and (3) interventional: P(X R |do(x iS )). The first point to demonstrate how to factorize the probability distribution for a DAG
observe is that the three distributions are not equivalent. (1) comes of chain components. Heskes et al. [18] combine this procedure
apart from the other two whenever statistical dependencies are with a few steps of do-calculus to compute interventional Shapley
present, i.e. X R ̸⊥⊥ X S . Options (2) and (3) come apart whenever values from observational data.
causal dependencies are present, i.e. if an(X R ) ∩ X S , {∅}. I want
to be clear that the differences between these distributions are in
no sense normative. None is generally “better” than the others, or
Rational Shapley Values FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea

3.2 A New Axiom Algorithm 1: Rational Shapley Values

Say two agents are identical along all recorded variables for some Input: Input datapoint x i , set of candidate subspaces
credit scoring task, and both receive lower scores than they had {⟨DS , DR , Y1 ⟩k }k=1
m , utility function u, credence
hoped. Counterfactuals identify two minimal perturbations suffi- function p
cient to improve predictions: (1) increase education by one unit; or Output: ϕ(z̃ ∗ )
(2) increase savings by one unit. Let us stipulate that these actions Compute classical Shapley values ϕ(z) using mean
ϕ 0 = n−1 ni=1 f (x i ) and reference DR
Í
are equivalent in terms of time and money. However, they differ
Initialize r (z) ← j p(h j |ϕ(z)) u(a ∗ , h j )
Í
in another important respect – the first agent wants to go back to
school and the second wants to save more each month. In this case, for k = {1, . . . , m} do
preferences alone determine which of two explanations is optimal. Draw data z̃k ∼ ⟨DS , DR , Y1 ⟩k
We may devise a similar example to show the impact of differing if z̃k = {∅} then
credences. Say, for instance, that two labs are planning expensive Fail
gene knockout experiments to test the predictions of a gene regula- else
tory network inference algorithm. Their utilities may be identical – Compute ϕ(z̃k )
Record r (z̃k ) ← j p(h j |ϕ(z̃)) u(a ∗ , h j )
Í
the goal for both groups is to maximize true discoveries while min-
imizing false positives – but they are working with different sets of end
evidence after preliminary, as yet unpublished results indicated to end
the first lab that one set of pathways is a probable dead end. In this z̃ ∗ ← argmaxz̃ r (z̃)
case, beliefs alone determine which of two explanations is optimal.
What these examples demonstrate is that utilities and credences
matter when attempting to explain model predictions. With default
software like SHAP, feature attributions for these credit loan ap- rewards. That is:
plicants or research labs would not generally differ – same inputs,
same outputs. However, by explicitly incorporating pragmatic in- Z ∗ = argmax E [r (z̃)]. (8)
formation, we can secure custom explanations at no cost to the Z̃ ⊆Z z̃ ∼ Z̃
axiomatic guarantees of Shapley values. In fact, we can extend the
current desiderata. This differs from (and improves upon) Eq. 4 in several respects.
Observe that for fixed value function v, model f , and input First, it results not in a single point but in a region of space, which
x i , feature attributions ϕ = {ϕ j }dj=1 vary only as a function of is stabler and more informative. Second, it explicitly incorporates
preferences and credences via the functions that define r (·). Finally,
contrastive baseline z̃. I make this dependence explicit moving
since rewards are conditioned upon a feature attribution vector,
forward, writing ϕ(z̃). Define a reward function for the inquiring
this target combines elements of both counterfactuals and feature
agent as the expected utility of the utility-maximizing action a ∗ ,
attributions, which is preferable to either alternative alone for all
conditional on a given set of feature attributions:
the reasons argued above.
r (z̃) = E [u(a ∗ , h)|ϕ(z̃)]. (7) There are many ways one could go about computing this sub-
h∼p space in practice, and no single method is optimal in general. See
Karimi et al. [20] for a recent review. In the following experiments, I
The axiomatic constraint that characterizes pragmatic feature attri- follow Wexler et al. [49] in relying on observational samples. These
butions can now be stated as follows. For the proof of Thm. 3.1, see subspaces have the advantage of being fast to find and guaranteed
Appx. A. to lie on the true data manifold (up to potential sampling or mea-
surement errors). They avoid the assumptions and approximations
Definition 2 (Rationality). Let r (·) be the reward function for a of more convoluted optimization techniques. The main disadvan-
rational agent, i.e. one who behaves in accordance with the utility and tage of this method is that it may provide low coverage for regions
probability axioms. Let ϕ(z̃) be a feature attribution vector computed of the feature space where data are undersampled. However, this
with respect to subspace Z̃. Then ϕ(z̃) is rational if and only if, for poses issues for other approaches as well, as no method can confi-
any alternative subspace Z ′ , Ez̃ ∼Z̃ [r (z̃)] ≥ Ez ′ ∼Z′ [r (z ′ )]. In other dently draw realistic points from low-density regions of the feature
words, feature attributions are rational to the extent that they tend to space.
maximize expected rewards for the inquiring agent. The basic strategy is schematized in pseudocode (see Alg. 1).
The user postulates a set of candidate subspaces, computing a se-
Theorem 3.1. When the relevant subspace is nonempty, rational
quence of rational Shapley vectors and comparing relative rewards.
Shapley values are the unique additive feature attribution method that
Evaluating r (z̃) with precision requires explicit utilities and (condi-
satisfies efficiency, linearity, sensitivity, symmetry, and rationality.
tional) credences, which may not be generally available. This can
be done informally by comparing feature attribution vectors on
3.3 Rational Shapley Value Algorithm an ordinal basis, such that rewards are ranked rather than directly
I reframe the objective function of Eq. 4 with a slight twist. Our quantified. This procedure is equivalent to the cardinal alternative
goal is not to minimize the cost or distance between two individ- under reasonable assumptions about agentive rationality, as noted
ual datapoints, but rather to compute a subspace that maximizes above.
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

The algorithm is deliberately silent on how to select candidate Table 1: Utility matrix for a 21-year-old African American defen-
subspaces, as this is irreducibly context-dependent. Credence up- dant with a single prior, predicted to be high risk and deciding
whether to sue the makers of COMPAS.
dates may be calculated in closed form under some parametric
assumptions, or else via Monte Carlo. The examples below illus-
trate how the method works in practice. h 1 : biased h 2 : ¬biased
a 1 : sue 5 –1
a 2 : ¬sue 0 0
4 EXPERIMENTS
In this section, I describe results from a number of experiments
on benchmark datasets. I compare performance against baselines
using marginal, conditional, and interventional value functions, To compute rational Shapley values, I consider a relevant sub-
respectively labelled MSV, CSV, and ISV. Code for reproducing all space of defendants in the same age range (at most 33 years old)
results and figures can be found at https://github.com/dswatson/ with at least one prior, yet who fell in the bottom half of defendants
rational_shapley. by predicted risk score. All other features are allowed to vary uncon-
strained. 356 subjects in the data meet these criteria, and therefore
4.1 Auditing: COMPAS constitute the reference sample for this experiment. This is too large
a group of counterfactuals to “explain” anything on their own, even
The COMPAS algorithm is a statistical model used to assign risk
if they were ranked by some measure (e.g., distance from the target
scores to defendants awaiting trial. High risk scores are associated
input). What the task requires is a summary of differences across
with elevated probability of recidivism, which judges in nine US
features, a job for which Shapley values are uniquely well-suited.
states use to help decide whether to let defendants out on bail
The racial breakdown of defendants in the relevant subspace is
while awaiting trial. A 2016 report by ProPublica alleged that the
markedly different from our high-risk group – just 48% are African
COMPAS algorithm was racially biased against African Americans
American – further reinforcing concerns about potential bias in
[3]; subsequent analysis by independent researchers has not always
the COMPAS data. The resulting Shapley values look fairly similar
corroborated those findings [14, 36].
to their classical counterparts, although priors and age appear to
Though the creators of COMPAS have published technical re-
dominate the explanations under conditional and interventional
ports defending their model, they have made neither their train-
reference distributions. Using the marginal reference, however –
ing data nor code publicly available, so attempts to recreate the
which, as noted above, is most relevant for auditing scenarios – we
algorithm and audit for racial bias are typically based on data gath-
find greater evidence of racial bias using rational Shapley values.
ered by ProPublica on some 12,000 individuals arrested in Broward
The racial attribution gap between Black and white defendants is
County, Florida between 2013 and 2014. Using a random forest [5],
significantly greater in the relevant subspace than it is across the
I regress violent risk scores on the following features: age, number
complete dataset (t = 3.874, p < 0.001).
of priors, and whether the present charge is a felony (all deemed
Consider the case of a young Black defendant deciding whether
“admissible”); as well as race and sex (deemed “inadmissible”). For
to file suit against the makers of COMPAS. Despite having just a
the purposes of this experiment, I take the trained model as the
single prior offense, this 21-year-old was placed in the highest risk
true target.
group according to the algorithm. A utility matrix for this individual
MSV is computed via Monte Carlo [41], with 2000 simulations
is given in Table 1. Assume that the defendant in question assigns a
per attribution. CSV is computed using the TreeSHAP method [27],
uniform prior over H to begin with. We say that h 1 is corroborated
since random forests are composed of individual regression trees.
to the extent that race receives greater positive attribution for
For ISV, I presume a simple partial ordering in which demographic
African Americans than it does for defendants of other races. Then
variables (age, sex, and race) are root nodes, upon which all other
although both classical and rational Shapley values recommend the
predictors depend. The mean predicted response for all defendants
same action (a 1 ), the latter does so with higher expected reward,
in the dataset is ϕ 0 = 3.69.
since p(h 1 |ϕ(z̃)) > p(h 1 |ϕ(z)). Rational Shapley values are therefore
I focus specifically on high-risk defendants, namely the 264 sub-
preferable in this case, as theory suggests.
jects with risk scores at or above the 95th percentile according to the
COMPAS approximator. Some 94% of these defendants are African
American, compared to just 60% in the general dataset, which im- 4.2 Discovery: Medical Diagnosis
mediately raises questions about racial bias. MSV, CSV, and ISV all As an example of XAI for discovery, I consider the diabetes dataset
look fairly similar in this case (see Fig. 1). Age and priors generally originally described by [12]. The data consist of 442 patients and 10
receive the highest feature attributions in this group, consistent predictors – age, sex, body mass index (BMI), blood pressure (MAP),
with the findings of other researchers. Race also receives nonzero and six blood serum measurements, including three cholesterol-
attribution throughout. Notably, race receives higher attribution related variables (LDL, HDL, and TC). The goal is to predict disease
under the ISV reference than MSV, since MSV can only detect direct progression after one year, for which I use an elastic net regres-
effects, and race is presumed to have both direct and indirect effects sion [50]. Note that exact Shapley values can be analytically calcu-
given the partial ordering. In this example, I presume that auditors lated for linear models under the assumption of feature indepen-
are working at the model level – i.e., evaluating the patterns learned dence. However, exploratory data analysis reveals strong correla-
by the algorithm, rightly or wrongly – in which case the marginal tions between some features in the data (especially blood serum
distribution is best suited to their purpose. measurements), and causal dependencies between features such
Rational Shapley Values FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea

Marginal Conditional Interventional

sex

race

Classical
priors

felony

Feature
Value
age
Feature

2
sex
0

race

Rational
priors

felony

age

−1 0 1 2 3 4 −1 0 1 2 3 4 −1 0 1 2 3 4
Shapley Value

Figure 1: Classical and rational Shapley values for the Broward County COMPAS dataset, computed with marginal, conditional, and interven-
tional reference distributions. Continuous feature values are z-scored for visualization.

as BMI and MAP are well established. In such a case, the ana- explanations are provided primarily in terms of lifestyle variables
lytic formulae for local explanation in linear models are inappli- like BMI and MAP, which agents can work to minimize in order to
cable. For ISV, I assume the partial ordering {age, sex, bmi} → improve their disease prognosis.
{map, ldl, hdl, tc} → {tch, ltg, glt}. This treats age, sex, and Consider the case of two high-risk patients – call them Bert
BMI as root nodes; MAP and cholesterol as causally intermedi- and Ernie – who share identical feature vectors (see Table 2; note
ate; and remaining blood serum measurements are downstream. that the negative value of age is a quirk of the scaling procedure.)
Such a DAG admittedly oversimplifies several complex biochemical Despite their striking similarities, however, Bert and Ernie differ in
processes but is broadly consistent with known structural relation- their culinary tastes. Bert was raised on carbohydrate-rich foods,
ships. and cannot imagine living without bread, pasta, and potatoes; Ernie,
I focus on patients with especially poor prognoses, as these are by contrast, is a voracious carnivore who eats bacon for breakfast,
typically the subjects of greatest concern to clinicians. Specifically, cold cuts for lunch, and steak for dinner. Their respective utility
I examine the top decile of patients by disease activity (ŷ ≥ 265). matrices are given by Table 3, where we assume for the sake of this
Classical Shapley values for these patients are visualized under all example that only two dietary interventions are under considera-
three value functions in Fig. 2. CSV and ISV are fairly similar here, tion: a low-carb diet and a low-fat diet. Evidence suggests that the
although both are quite different from the marginal alternative. latter is slightly more effective for weight loss, while the former has
For instance, TC receives relatively large attributions under the the added benefit of reducing blood pressure [6]. Classical Shap-
marginal reference distribution, but nearly none under conditional ley values, which place the greatest weight on BMI, would tend
or interventional distributions, owing to its strong collinearity with to favour a 1 regardless of agentive preferences. However, rational
other cholesterol measures. Shapley values can distinguish between optimal explanations based
For rational Shapley analysis, I consider a subspace consisting entirely on utilities, since changes to either BMI or MAP are suffi-
of older men (i.e., those in the top quartile by age), who are gen- cient to bring about the desired outcome, albeit at differing costs to
erally at higher risk than other groups in this dataset. Since age Bert and Ernie. Thus we find the following two feature attribution
and sex are (more or less) unactionable, a rational agent will likely vectors (see Table 4), computed using the conditional reference
want prognostic explanations that place little or no weight on such distribution on inputs that differ only with respect to utilities u.
features. We find in Fig. 2 that BMI and MAP receive large ratio- Observe that, among the many differences between these two
nal attributions on average, reflecting their greater variance in the vectors, they flip the relative importance of BMI and MAP, sug-
relevant subspace. Conditional and interventional reference distri- gesting two alternative paths toward reducing disease risk. A coun-
butions are especially valuable here, as the analytical goal almost terfactual approach with a well-designed cost function could in
certainly requires a system-level approach and causal dependencies principle also identify different explanations for these two agents,
cannot therefore be ignored. Alternative structural assumptions but it could not additionally provide complete feature attributions
may lead to somewhat different attributions, but focusing on a summarising the relative impact of other predictors. Such a synthe-
relevant subspace has effectively driven the Shapley values for un- sis is unique to the rational Shapley approach.
actionable variables like sex and age toward zero. The result is that
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

Marginal Conditional Interventional

tch

tc

sex

map

Classical
ltg

ldl

hdl

glu
Feature
bmi Value

age
Feature

0.1

tch
0.0
tc

sex −0.1

map

Rational
ltg

ldl

hdl

glu

bmi

age

−200 −100 0 100 200 −200 −100 0 100 200 −200 −100 0 100 200
Shapley Value

Figure 2: Classical and rational Shapley values for the diabetes dataset, computed with marginal, conditional, and interventional reference
distributions. Continuous feature values are z-scored for visualization.

Table 2: Complete feature vector for both Bert and Ernie in the diabetes dataset.

sex age bmi map tc ldl hdl tch ltg glu

1 –0.064 0.096 0.105 –0.003 –0.005 –0.007 -0.003 0.023 0.073

Table 3: Utility matrices for Bert (left) and Ernie (right). Bert would like to reduce his risk of diabetes without lowering his carb intake; Ernie
wants the same outcome without lowering his fat intake.

Bert h 1 : ŷ < 265 h 2 : ŷ ≥ 265 Ernie h 1 : ŷ < 265 h 2 : ŷ ≥ 265

a 1 : low-fat 5 –1 a 1 : low-fat 1 –6
a 2 : low-carb 1 –6 a 2 : low-carb 5 –1

Table 4: Rational Shapley vectors for Bert (top) and Ernie (bottom), computed from the same input vector (see Table 2) but with respect to
different relevant subspaces.

sex age bmi map tc ldl hdl tch ltg glu

–3.789 –4.867 94.922 29.241 1.763 0.849 4.053 6.566 13.019 4.324
–1.291 0.114 42.604 66.942 2.187 1.129 –0.167 –3.532 11.136 9.903

4.3 Recourse: Credit Scoring rate of 70%. I train a support vector machine (SVM) [7] to predict Y
As a final example, I examine the recourse setting, where the goal using a radial basis kernel.
is to advise an agent what interventions are necessary and/or suffi- I assume a simple partial ordering in which demographic vari-
cient to secure a desired prediction. As Karimi et al. [21] have shown, ables are causally antecedent to both financial predictors and loan
this task requires causal information. More specifically, they prove application details: {A, G, M } → {L, D, J , S }. The UCI website notes
that noncausal recommendations, such as those provided by Ustun that false positives are more costly for banks than false negatives,
et al. [43], guarantee recourse if and only if the treatment variables and therefore recommends penalizing the former at five times the
have no descendants in the underlying causal graph. For this ex- rate of the latter. Post-processing the SVM, an optimal decision
periment, I use the German credit dataset from the UCI Machine threshold is obtained at 0.74, and I therefore label all and only
Learning Repository [11]. To simplify the presentation, I restrict those points with predicted probabilities at or above this value as
focus to just seven of the most informative features: age A, gender 1. The resulting model is about 69% accurate on the true positives
G, marital status M, job J , savings S, loan amount L, and duration and 73% accurate on the true negatives. I focus on the 189 border-
D. The outcome Y is binary (loan approved/denied), with a base line applicants with predicted success probabilities on the interval
Rational Shapley Values FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea

Marginal Conditional Interventional

savings

marital

job

Classical
gender

duration

amount Feature
Value
age 2
Feature

1
0
savings
−1
−2
marital

job

Rational
gender

duration

amount

age

−0.4 0.0 0.4 0.8 −0.4 0.0 0.4 0.8 −0.4 0.0 0.4 0.8
Shapley Value

Figure 3: Classical and rational Shapley values for the German credit dataset, computed with marginal, conditional, and interventional refer-
ence distributions. Continuous feature values are z-scored for visualization.

Table 5: Complete feature vector for Ruth in the German credit dataset.

gender marital age amount duration savings job

0 0 26 3181 26 290 2

[0.7, 0.74). These are subjects who could plausibly benefit from Table 6: Utility matrix for Ruth, whose demographic and financial
even a slight improvement in their model predictions. Classical information place her just below the loan approval threshold for a
hypothetical bank.
Shapley values are unable to help such applicants, since the mean
response in the full dataset is ϕ 0 = 0.7, i.e. the lower bound of
the interval. (Irrational) feature attributions for these subjects are h 1 : ŷ ≥ 0.74 h 2 : ŷ < 0.74
therefore guaranteed to sum to some value on [0, 0.04), explaining a 1 : Increase S 1 –2
why they did slightly better than average instead of why they were a 2 : Decrease L 2 –1
ultimately unsuccessful. a 3 : Decrease D 2 –1
The most important component of the relevant subspace is the
contrastive outcome Y1 , as the borderline applicants appear fairly
representative along all predictors of interest. I therefore stratify
by predicted response, zooming in on the top quartile of applicants
(ŷ ≥ 0.76). Bivariate analyses suggest that such applicants are for some changes if necessary – specifically, to increase her savings,
the most part financially secure and request small loans of relatively decrease the size of her requested loan, or pay it back sooner – but
brief duration. The classical and rational Shapley vectors look fairly in any case she would like to act quickly, since she fears the house
similar in this experiment, except for the crucial difference in offset. she wants to buy will not be on the market for long. Ruth’s utility
(See Fig. 3; note that the response variable is transformed to the matrix is given in Table 6. We find here that she prefers changing
logit scale for easier visualization.) the terms of her application to increasing her savings in general,
Consider the case of a loan applicant, call her Ruth, whose pre- as this is likely the faster route to bank approval. She is indifferent
dicted success probability is 0.73 – just under the decision boundary. between changing the loan amount or duration.
Her input feature vector can be found in Table 5, where values are As noted above, computing Shapley values from the entire
reported on the original (pre-transformation) scale. She is looking dataset is pointless in this case, as Ruth’s predicted outcome is
to buy a new home and believes her high-skilled job and decent already above the base rate. Counterfactuals could potentially aid
savings (both well above their respective median values in the her search for an explanation, but only by either providing an arti-
dataset) should make her a strong applicant. She is willing to make ficially narrow set of pathways to approval or overwhelming her
with an unnecessarily large reference class of successful applicants.
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

Table 7: Rational Shapley values for Ruth in the German credit dataset.

gender marital age amount duration savings job

0 0 –0.168 0.061 –0.115 0.013 0.128

Rational Shapley values, computed from a relevant subspace Such a failure is highly informative, as it demarcates a realm of
of successful loan applicants who share Ruth’s demographic char- (im)possibility for the inquiring agent. Thus it is simply false to
acteristics, provide a summary of feature attributions within this allege that users will always find what they are looking for. Second,
reference class (see Table 7; note that calculations are performed in in cases where the relevant subspace is nonempty but sparsely
logit space.) Interestingly, we find that her requested loan amount, populated, we should expect estimates to be unstable. Any good
though above average for the relevant subspace, did not hurt Ruth’s inference procedure should take such uncertainty into account,
application. On the contrary, it appears to boost her chances some- making it difficult for an opportunistic user’s desired outcome to
what, a result we might not expect from surveying the counter- pass severe tests on the basis of just a small handful of outlying
factuals directly. The loan duration, however – about double the points. I have not said much about testing here, but it is simple
average for the relevant subspace – brought her success probability in principle to extend the experiments above with frequentist or
down considerably. This is a highly actionable piece of information, Bayesian methods tailored to the particular task at hand [40]. Third,
insomuch as it guides Ruth toward algorithmic recourse that can there is no guarantee that agents will generally concur on proper
push her application over the decision boundary. Of course, if the values of DS , DR , and Y1 .
change is too onerous for Ruth, then she is free to resample from This makes it all the more important to be explicit about which
another subspace of successful applicants with longer loan dura- values of each input go into any given explanation.
tion on average. The procedure may continue like this indefinitely, Such transparency will make it harder for adversarial agents to
with agents testing out new hypotheses in an iterative, exploratory game the system and easier for those acting in good faith to come
fashion, updating their rewards accordingly. to consensus on particular cases.

5 DISCUSSION 6 CONCLUSION
The skeptical reader may plausibly object that the examples above The two most popular XAI tools available for local, post-hoc ex-
are fairly neat and straightforward, with their small utility matrices planations – feature attributions and counterfactuals – each have
of well-defined user preferences. Reality, of course, is far messier. certain advantages. However, they tend to provide different an-
Complete sets of actions and outcomes may not be known upfront, swers in particular cases that can confuse and/or mislead end users.
let alone utility and credence functions defined thereon. Potential In this paper, I have shown how the two can be synthesized into
interventions may be far more numerous than these experiments a single method that preserves the best elements of each. Ratio-
permit, and associated outcomes entirely uncertain. Can rational nal Shapley values preserve and extend the axiomatic guarantees
Shapley values scale to larger, more complicated instances of algo- of their classical forebears, giving users the flexibility to compute
rithmic explanation? additive feature attributions in a fast, flexible manner. Marginal,
In a word, yes. These cases are merely illustrative, following conditional, and interventional value functions were all evaluated,
on the back of theoretical results establishing the viability of a and the applicability of each was compared across different use
pragmatic synthesis between feature attributions and counterfactu- cases. By formalizing the task in an expected utility framework, I
als that works with various different reference distributions. The was able to demonstrate how and why individual agents may ratio-
limiting factors in these experiments were complications around nally seek different explanations for the same model predictions.
efficient Monte Carlo sampling, conditional probability estimation, The resulting explanations are concise, intuitive, and thoroughly
and reliable partial orderings. The first can be entirely resolved pragmatic.
with greater computational power. The second requires some care,
but may in principle be addressed with flexible methods such as
ACKNOWLEDGMENTS
variational autoencoders. The third relies on background knowl-
edge, which may be substantial in some areas, and should in any This research was supported by ONR grant N62909-19-1-2096.
case grow over time. Improvements in any or all of these areas can
help extend the rational Shapley method to larger, more complex REFERENCES
cases. [1] Kjersti Aas, Martin Jullum, and Anders Løland. 2021. Explaining individual
Another potential challenge to the proposed method is that it is predictions when features are dependent: More accurate approximations to
Shapley values. Artif. Intell. 298 (2021), 103502.
vulnerable to confirmation bias. If users can scan the data in search [2] Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A
of attribution vectors that make their preferred outcomes more survey on explainable artificial intelligence (XAI). IEEE Access 6 (2018), 52138–
52160.
likely, what’s to stop them from finding such vectors even when [3] J Angwin, J Larson, S Mattu, and L Kirchner. 2016. Machine bias. Technical Report.
do they do not exist? There are three answers to this charge. First, ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-
it is entirely possible that the relevant subspace be empty – that in-criminal-sentencing
[4] Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine
is, no observed or synthetic datapoint meets the threefold criteria Learning. fairmlbook.org. http://www.fairmlbook.org.
specified by ⟨DS , DR , Y1 ⟩. [5] Leo Breiman. 2001. Random Forests. Mach. Learn. 45, 1 (2001), 1–33.
Rational Shapley Values FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea

[6] Grant D Brinkworth, Manny Noakes, Jonathan D Buckley, Jennifer B Keogh, and [33] W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu.
Peter M Clifton. 2009. Long-term effects of a very-low-carbohydrate weight loss 2019. Definitions, methods, and applications in interpretable machine learning.
diet compared with an isocaloric low-fat diet after 12 mo. Am. J. Clin. Neutr. 90, Proceedings of the National Academy of Sciences 116, 44 (2019), 22071 – 22080.
1 (2009), 23–32. [34] Judea Pearl. 2000. Causality: Models, reasoning, and inference. Cambridge Univer-
[7] Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. sity Press, New York.
Learn. 20, 3 (1995), 273–297. [35] Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach.
[8] Philip Dawid. 2012. The Decision-Theoretic Approach to Causal Inference. In 2020. FACE: Feasible and actionable counterfactual explanations. In Proceedings
Causality: Statistical Perspectives and Applications, Carlo Berzuini, Philip Dawid, of the AAAI/ACM Conference on AI, Ethics, and Society. 344–350.
and Luisa Bernardinelli (Eds.). Wiley, London, Chapter 4, 25–42. [36] Cynthia Rudin, Caroline Wang, and Beau Coker. 2020. The Age of Secrecy and
[9] Philip Dawid. 2015. Statistical Causality from a Decision-Theoretic Perspective. Unfairness in Recidivism Prediction. Harvard Data Sci. Rev. 2, 1 (2020).
Annu. Rev. Stat. Appl. 2, 1 (2015), 273–303. [37] Chris Russell. 2019. Efficient Search for Diverse Coherent Explanations. In
[10] Philip Dawid. 2020. Decision-theoretic foundations for statistical causality. (2020). Proceedings of the International Conference on Fairness, Accountability, and Trans-
arXiv preprint, 2004.12493. parency. 20–28.
[11] Dheera Dua and Casey Graff. 2017. UCI Machine Learning Repository. University [38] Lloyd Shapley. 1953. A Value for n-Person Games. In Contributions to the Theory
of California, School of Information and Computer Science, Irvine, CA. of Games. Princeton University Press, Princeton, Chapter 17, 307–317.
[12] Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. 2004. Least [39] Ilya Shpitser and Judea Pearl. 2008. Complete Identification Methods for the
Angle Regression. Ann. Stat. 32, 2 (2004), 407–499. Causal Hierarchy. J. Mach. Learn. Res. 9 (2008), 1941–1979.
[13] Paul Feyerabend. 1975. Against Method. New Left Books, London. [40] Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. 2021. Reli-
[14] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. 2019. All Models are able Post hoc Explanations: Modeling Uncertainty in Explainability. In Advances
Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an in Neural Information Processing Systems, Vol. 34. 9391–9404.
Entire Class of Prediction Models Simultaneously. J. Mach. Learn. Res. 20, 177 [41] Erik Štrumbelj and Igor Kononenko. 2014. Explaining prediction models and
(2019), 1–81. individual predictions with feature contributions. Knowledge and Information
[15] Luciano Floridi. 2019. The Logic of Information. Oxford University Press, Oxford. Systems 41, 3 (2014), 647–665.
[16] Christopher Frye, Ilya Feige, and Colin Rowat. 2020. Asymmetric Shapley values: [42] Mukund Sundararajan and Amir Najmi. 2019. The many Shapley values for
Incorporating causal knowledge into model-agnostic explainability. In Advances model explanation. In Proceedings of the ACM Conference.
in Neural Information Processing Systems. [43] Berk Ustun, Alexander Spangher, and Yang Liu. 2019. Actionable Recourse in
[17] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Linear Classification. In Proceedings of the Conference on Fairness, Accountability,
Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black and Transparency. 10–19.
Box Models. Comput. Surveys 51, 5 (2018), 1–42. [44] John von Neumann and Oskar Morgenstern. 1944. Theory of Games and Economic
[18] Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. 2020. Causal Behavior. Princeton University Press, Princeton, NJ.
Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions [45] Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual
of Complex Models. In Advances in Neural Information Processing Systems. explanations without opening the black box: Automated decisions and the GDPR.
[19] Dominik Janzing, Lenon Minorics, and Patrick Bloebaum. 2020. Feature relevance Harvard J. Law Technol. 31, 2 (2018), 841–887.
quantification in explainable AI: A causal problem. In Proceedings of the 23rd [46] David S Watson. 2022. Conceptual challenges for interpretable machine learning.
International Conference on Artificial Intelligence and Statistics, Vol. 108. 2907– Synthese 200, 2 (2022), 65.
2916. [47] David S Watson and Luciano Floridi. 2021. The explanation game: a formal
[20] Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. framework for interpretable machine learning. Synthese 198, 10 (2021), 9211–
2020. A survey of algorithmic recourse: Definitions, formulations, solutions, 9242.
and prospects. (2020). arXiv preprint, 2010.04050. [48] David S Watson, Limor Gultchin, Ankur Taly, and Luciano Floridi. 2022. Local
[21] Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. 2021. Algorithmic Explanations via Necessity and Sufficiency: Unifying Theory and Practice. Minds
Recourse: From Counterfactual Explanations to Interventions. In Proceedings of Mach. 32, 1 (2022), 185–218.
the 2021 ACM Conference on Fairness, Accountability, and Transparency. 353–362. [49] J Wexler, M Pushkarna, T Bolukbasi, M Wattenberg, F Viégas, and J Wilson.
[22] Indra Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle 2020. The What-If Tool: Interactive Probing of Machine Learning Models. IEEE
Friedler. 2020. Problems with Shapley-value-based explanations as feature impor- Transactions on Visualization and Computer Graphics 26, 1 (2020), 56–65.
tance measures. In Proceedings of the 37th International Conference on Machine [50] Hui Zou and Trevor Hastie. 2005. Regularization and Variable Selection via the
Learning (ICML ’20). 1–10. Elastic Net. J. Royal Stat. Soc. Ser. B Methodol. 67, 2 (2005), 301–320.
[23] Steffen L Lauritzen and Thomas S Richardson. 2002. Chain graph models and their
causal interpretations. J. Royal Stat. Soc. Ser. B Methodol. 64, 3 (2002), 321–348.
[24] Sanghack Lee and Elias Bareinboim. 2018. Structural Causal Bandits: Where A APPENDIX
to Intervene?. In Advances in Neural Information Processing Systems, Vol. 31.
2568–2578. The following is a proof of Thm. 3.1, which states that rational Shap-
[25] Sanghack Lee and Elias Bareinboim. 2019. Structural Causal Bandits with Non- ley values are the unique additive feature attribution method that
Manipulable Variables. Proceedings of the AAAI Conference on Artificial Intelligence satisfies efficiency, linearity, sensitivity, symmetry, and rationality.
33, 1 (2019), 4164–4172.
[26] Catherine Legg and Christopher Hookway. 2020. Pragmatism. In The Stanford
Encyclopedia of Philosophy (fall 2020 ed.), Edward N Zalta (Ed.). Metaphysics Proof. Since uniqueness is already well established for the clas-
Research Lab, Stanford University. sical axioms, all that remains is to show that rational Shapley values
[27] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin,
Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020.
(a) satisfy the additional rationality axiom and (b) do not violate
From local explanations to global understanding with explainable AI for trees. any of the classical axioms.
Nat. Mach. Intell. 2, 1 (2020), 56–67. Take (a) first. Assume there exists some Z ′ that generates greater
[28] Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model
Predictions. In Advances in Neural Information Processing Systems. 4765–4774. expected rewards for an agent than the relevant subspace Z̃. Then
[29] Divyat Mahajan, Chenhao Tan, and Amit Sharma. 2019. Preserving Causal conditioning on the corresponding Shapley values ϕ(z ′ ) must lead
Constraints in Counterfactual Explanations for Machine Learning Classifiers. In
CausalML: Machine Learning and Causal Inference for Improved Decision Making
to an action of greater expected utility than the action recommended
Workshop, NeurIPS 2019. by ϕ(z̃). This means that the values for some ϕ j (z̃) are either too
[30] Luke Merrick and Ankur Taly. 2020. The Explanation Game: Explaining Machine large or too small. But this can only arise from a misspecification of
Learning Models Using Shapley Values. In Machine Learning and Knowledge
Extraction - 4th International Cross-Domain Conference (CD-MAKE). Springer, (i) Y1 , (ii) DS , or (iii) DR , which together permit either too much
17–38. variation in X j (resulting in inflated values of |ϕ j (z̃)|), or too little
[31] Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social
sciences. Artif. Intell. 267 (2019), 1–38.
(resulting in deflated values of |ϕ j (z̃)|). By Def. 1, any Z̃ that fails
[32] Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining to satisfy one of these criteria is not a relevant subspace. Thus we
Machine Learning Classifiers through Diverse Counterfactual Explanations. In have a contradiction.
Proceedings of the International Conference on Fairness, Accountability, and Trans-
parency in Machine Learning (Barcelona, Spain). 607–617. One can verify (b) by confirming that rational Shapley values
do not deviate from the classical formulae (see Eqs. 1, 2, 3). The
FAccT ’22, June 21–24, 2022, Seoul, Republic of Korea David S. Watson

only difference between the current proposal and more familiar work on Shapley values has been devoted to exploring alternati-
alternatives is their respective methods of specifying baseline ex- vere reference distributions; for more on the plurality of Shapley
pectations ϕ 0 and reference distributions DR . Shifting these two values, see [30, 42]. Any procedure that inputs valid values for ϕ 0
parameters changes the interpretation of resulting Shapley vectors, and DR and conforms to the characteristic equations enjoys the
but does not alter their essential character. Indeed, much recent same properties. □