Think About the Stakeholders First!

Towards an Algorithmic
Transparency Playbook for Regulatory Compliance
Andrew Bell Oded Nov Julia Stoyanovich
New York University New York University New York University
New York, United States New York, United States New York, United States
[email protected] [email protected] [email protected]

ABSTRACT Unfortunately, most strategies, directives and laws to date lack

Increasingly, laws are being proposed and passed by governments specificity on how AI regulation should be carried out in prac-
around the world to regulate Artificial Intelligence (AI) systems tice by technologists. Where there is specificity, there is a lack of
arXiv:2207.01482v1 [cs.CY] 10 Jun 2022

implemented into the public and private sectors. Many of these mechanisms for enforcing laws and holding institutions using AI
regulations address the transparency of AI systems, and related accountable. Documents on AI governance have focused on what
citizen-aware issues like allowing individuals to have the right to do (or what not to do) with respect to AI, but leave the brunt
to an explanation about how an AI system makes a decision that of the work to practitioners to figure out how things should be
impacts them. Yet, almost all AI governance documents to date done [32]. This tension plays out heavily in regulations govern-
have a significant drawback: they have focused on what to do (or ing the transparency of AI systems (called “explainability” by AI
what not to do) with respect to making AI systems transparent, but practitioners). The most prominent example of this is the “right
have left the brunt of the work to technologists to figure out how to explanation” of data use that is included in the EU’s General
to build transparent systems. We fill this gap by proposing a novel Data Protection Regulation (GDPR). Despite being passed into law
stakeholder-first approach that assists technologists in designing in 2016, the meaning and scope of the right is still being debated
transparent, regulatory compliant systems. We also describe a real- by legal scholars, with little of the discussion resulting in concrete
world case-study that illustrates how this approach can be used in benefits for citizens [64].
practice. While regulation can help weigh the benefits of new technology
against the risks, developing effective regulation is difficult, as is es-
CCS CONCEPTS tablishing effective mechanisms to comply with existing regulation.
This paper aims to fill a gap in the existing literature by writing to
• Human-centered computing → Interaction design theory, technologists and AI practitioners about the existing AI regulatory
concepts and paradigms. landscape, and speaks to their role in designing complaint systems.
We make a case for why AI practitioners should be leading efforts
KEYWORDS to ensure the transparency of AI systems, and to this end, we pro-
human-centered computing, transparency, explainability, regula- pose a novel framework for implementing regulatory-compliant
tion, regulatory compliance explanations for stakeholders. We also consider an instantiation
of our stakeholder-first approach in the context of a real-world
example using work done by a national employment agency.
1 INTRODUCTION
We make the following three contributions: (1) provide a sur-
In the past decade, there has been widespread proliferation of arti- vey of existing and proposed regulations on the transparency and
ficial intelligence (AI) systems into the private and public sectors. explainability of AI systems; (2) propose a novel framework for a
These systems have been implemented in a broad range of contexts, stakeholder-first approach to designing transparent AI systems; and
including employment, healthcare, lending, criminal justice, and (3) present a case-study that illustrates how this stakeholder-first
more. The rapid development and implementation of AI technolo- approach could be used in practice.
gies has greatly outpaced public oversight, creating a “wild-west”-
style regulatory environment. As policy makers struggle to catch
up, the issues of unregulated AI have become glaringly obvious,
especially for underprivileged and marginalized communities. Fa- 2 EXISTING AND EMERGING REGULATIONS
mously, ProPublica revealed that the AI-driven system COMPAS In recent years, countries around the world have increasingly been
used to assess the likelihood of a prisoner recidivating was highly drafting strategies, action plans, and policy directives to govern the
discriminatory against black individuals [5]. In another example, use of AI systems. To some extent, regulatory approaches vary by
Amazon built and implemented an automated resume screening country and region. For example, policy strategies in the US and
and hiring AI system–only to later find out that the system was the EU reflect their respective strengths: free-market ideas for the
biased against hiring women [53]. In an effort to address these former, and citizen voice for the latter [23]. Yet, despite country-
issues, countries around the world have begun regulating the use of level variation, many AI policies contain similar themes and ideas.
AI systems. Over 50 nations and intergovernmental organizations A meta-analysis of over 80 AI ethics guidelines and soft-laws found
have published AI strategies, actions plans, policy papers or direc- that 87% mention transparency, and include an effort to increase
tives [72]. A survey of existing and proposed regulation around AI the explainability of AI systems [32]. Unfortunately, all documents
transparency is given in Section 2. to date have one major limitation: they are filled with uncertainty
 Andrew Bell, Oded Nov, and Julia Stoyanovich

on how transparency and explainability should actually be imple- come from a standardized list of numeric reason codes, such as:
mented in a way that is compliant with the evolving regulatory “U4: Too many recently opened accounts with balances2 .”
landscape [22, 32, 32, 38]. This limitation has 3 main causes: (1)
it is difficult to design transparency regulations that can easily 2.2 European Union
be standardized across different fields of AI, such as self-driving In 2019 the EU published a white paper titled “Ethics Guidelines
cars, robotics, and predictive modeling [75]; (2) when it comes to for Trustworthy AI,” containing a legal framework that outlines
transparency, there is a strong information asymmetry between ethical principles and legal obligations for EU member states to
technologists and policymakers, and, ultimately, the individuals follow when deploying AI3 . While the white paper is non-binding,
who are impacted by AI systems [35]; (3) there is no normative it lays out expectations on how member-states should regulate the
consensus around AI transparency, and most policy debates are transparency of AI systems: “... data, system and AI business models
focused on the risks of AI rather than the opportunities [22]. For the should be transparent. Traceability mechanisms can help achieving
purposes of scope, we will focus on regulations in the United States this. Moreover, AI systems and their decisions should be explained
and Europe. However, its important noting that there is meaningful in a manner adapted to the stakeholder concerned. Humans need
AI regulation emerging in Latin and South America, Asia, Africa, to be aware that they are interacting with an AI system, and must
and beyond, and summarizing those regulations is an avenue for be informed of the system’s capabilities and limitations.”
future work. For example, in 2021, Chile presented it’s first national Currently, the European Commission is reviewing the Artificial
action plan on AI policy 1 . Intelligence Act4 , which would create a common legal framework
for governing all types of AI used in all non-military sectors in
2.1 United States Europe. The directive takes the position that AI systems pose a sig-
In 2019 the US took two major steps in the direction of AI regula- nificant risk to the health, safety and fundamental rights of persons,
tion. First, Executive Order 13859 was issued with the purpose of and governs from that perspective. With respect to transparency,
establishing federal principles for AI systems, and to promote AI the directive delineates between non-high-risk and high-risk AI sys-
research, economic competitiveness, and national security. Impor- tems (neither of which are rigorously defined at this time). It states
tantly, the order mandates that AI algorithms implemented for use that for “non-high-risk AI systems, only very limited transparency
by public bodies must be “understandable”, “transparent”, “responsi- obligations are imposed, for example in terms of the provision
ble”, and “accountable.” Second, the Algorithmic Accountability Act of information to flag the use of an AI system when interacting
of 2019 was introduced to the House of Representatives, and more with humans.” Yet, for high-risk systems, “the requirements of high
recently reintroduced under the name Algorithmic Accountability quality data, documentation and traceability, transparency, human
Act of 2022. If passed into law, the Algorithmic Accountability Act oversight, accuracy and robustness, are strictly necessary to mit-
would be a landmark legalisation for AI regulation in the US. The igate the risks to fundamental rights and safety posed by AI and
purpose of the bill is to create transparency and prevent disparate that are not covered by other existing legal frameworks.” Notably,
outcomes for AI systems, and it would require companies to assess as in the Algorithmic Accountability Act in the United States, the
the impacts of the AI systems they use and sell. The bill describes document contains explicit text mentioning recourse (referred to
the impact assessment in detail — which must be submitted to an as “redress”) for persons affected by AI systems.
oversight committee— and states that the assessment must address The EU has also passed Regulation (EU) 2019/1150 that sets
“the transparency and explainability of [an AI system] and the de- guidelines for the transparency of rankings for online search.5 In
gree to which a consumer may contest, correct, or appeal a decision practice, this means that online stores and search engines should
or opt out of such system or process”, which speaks directly to what be required to disclose the algorithmic parameters used to rank
AI practitioners refer to as “recourse”, or the ability of an individual goods and services on their site. The regulation also states that
to understand the outcome of an AI system and what they could explanations about rankings should contain redress mechanisms
do to change that outcome [73, 76]. for individuals and businesses affected by the rankings.
In 2019 the OPEN Government Data Act was passed into law, 2.2.1 Right to Explanation. The Right to Explanation is a proposed
requiring that federal agencies maintain and publish their infor- fundamental human right that would guarantee individuals access
mation online as open data. The data also must be cataloged on to an explanation for any AI system decision that affects them.
Data.gov, a public data repository created by the the US govern- The Right to Explanation was written into the EU’s 2016 GDPR
ment. While this law only applies to public data, it demonstrates regulations, and reads as follows: “[the data subject should have]
how policy can address transparency within the whole pipeline of the right ... to obtain an explanation of the decision reached.”6 The
an AI system, from the data to the algorithm to the system outcome. legal meaning and obligation of the text has been debated heavily by
There are also some industry-specific standards for transparency legal scholars, who are unsure under which circumstances it applies,
that could act as a model for future cross-industry regulations. Un- what constitutes an explanation [64], and how the right is applicable
der the Equal Credit Opportunity Act, creditors who deny loan
applicants must provide a specific reason for the denial. This in- 2 https://www.fico.com/en/latest-thinking/solution-sheet/us-fico-score-reason-

cludes denials made by AI systems. The explanations for a denial codes

3 https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-
ai
4 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
1 https://www.gob.cl/en/news/chile-presents-first-national-policy-artificial- 5 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32019R1150

intelligence/ 6 https://www.privacy-regulation.eu/en/r71.htm
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance

to different AI systems [19]. The Right to Explanation is an example include AI practitioners, researchers, designers, programmers, and
of how emerging AI technologies may “reveal” additional rights developers.
that need to be considered by lawmakers and legal experts [52]. Practitioners have the right technical expertise. Transparency
The EU’s recently proposed Artificial Intelligence Act simulta- has been a central topic of AI research for the past decade, and is
neously reinforces the idea that explanations about AI systems are motivated beyond just regulatory compliance by ideas like making
a human right, while slightly rolling back the Right to Explanation systems more efficient, debugging systems, and giving decision mak-
by acknowledging that there are both non-high-risk and high-risk ing agency to the data subjects (i.e., those affected by AI-assisted
AI systems. Discussions about the Right are likely to continue, and decisions) or to the users of AI systems (i.e., those making decisions
will be a central part of debates on regulating AI transparency. In with the help of AI). New technologies in transparent AI are being
fact, some local governing bodies have already taken steps to adopt created at a fast pace, and there is no indication that the rapid inno-
the Right to Explanation. France passed the Digital Republic Act in vation of explainable AI will slow any time soon [14, 14, 16, 41, 57],
2016, which gives the Right to Explanation for individuals affected meaning that of all the stakeholders involved in the socio-technical
by an AI system in the public sector [20]. Hungary also has a similar environment of AI systems, technologists are the most likely to
law [42]. be aware of available tools for creating transparent AI systems.
Furthermore, there are currently no objective measures for the
2.3 Local quality of transparency in AI systems [1, 25, 29, 40, 79], and so
technologists are necessary to discern the difference between a
There has been significant movement on the regulation of specific
“good explanation” and a “bad explanation” about a system.
forms of AI systems at local levels of government. In response to
Practitioners are the least-cost avoiders. This idea is based
the well-documented biases of facial recognition software when
on the principle of the least-cost avoider, which states that obliga-
identifying people of different races and ethnicities [11], Washing-
tions and liabilities should be allocated entirely to the party with the
ton State signed Senate Bill 6820 into law in 2020, which prohibits
lowest cost of care [68]. AI practitioners are the least-cost avoiders
the use of facial recognition software in surveillance and limits its
because they are already equipped with the technical know-how
use in criminal investigation.7 Detroit has also reacted to concerns
for building and implementing transparency tools into AI systems,
about facial recognition, and its City Council approved legislation
especially when compared to policymakers and the individuals af-
that mandates transparency and accountability for the procure-
fected by the outcome of the system. Notably, given the wide range
ment process of video and camera surveillance contracts used in
of existing transparency tools, implementing the “bare minimum”
the city.8 The New York City Council recently regulated the use
is trivially easy for most technologists.
of AI systems in relation to employment decisions (Local Law 144
One argument practitioners give against building transparent
of 2021).9 The bill requires that AI tools for hiring employees be
systems is that they may be less accurate than highly complex,
subject to yearly bias audits. An additional requirement is to notify
black-box systems [30]. However, there has been a growing amount
job seekers that they were screened by a tool, and to disclose to
of evidence suggesting that building transparent systems actually
them what “qualifications or characteristics” were used by the tool
results into little to no trade-off in the accuracy of AI systems [9,
as basis of decisions. Finally, in the Netherlands, the municipality
17, 61, 67]. In other words: building transparent systems is not a
of Rotterdam has created a Data-Driven Working program which
Pareto-reducing constraint for practitioners.
has been critical of transparency surrounding the algorithms used
Practitioners already bear the responsibility for implement-
for fraud detection.10
ing transparency into AI systems. A study interviewing AI prac-
titioners found that using AI responsibly in their work is viewed as
3 THE ROLE OF TECHNOLOGISTS the practitioner’s burden, not the institutions for which they work.
The continuously evolving regulatory landscape of AI, combined Practitioners noted that existing structures within institutions are
with the limitations of existing regulation in providing clarity on often antithetical to the goals of responsible AI, and that it is up to
how transparency should be implemented into AI systems, has them to push for structural change within that institution [55]. Sec-
left open questions concerning responsibilities for AI design and tion 2 shows that AI regulation is converging on requiring transpar-
implementation. We argue that (1) practitioners should bear the bulk ent AI systems that offer meaningful explanations to stakeholders.
of the responsibility for designing and implementing compliant, Therefore, it is in the best interest of practitioners to continue the
transparent AI systems (2) it is in the best interest of practitioners bottom-up approach of building transparent AI systems in the face
to bear this responsibility. Researchers have also shown that there of looming regulations.
may be risks of only partially complying with AI regulations, and
that fusll compliance is the best way forward [15]. Technologists
4 A STAKEHOLDER-FIRST APPROACH TO
7 https://app.leg.wa.gov/billsummary?BillNumber=6280&Initiative=false&Year=2019 DESIGNING TRANSPARENT ADS
8 https://www.detroitnews.com/story/news/local/detroit-city/2021/05/25/detroit-
council-approves-ordinance-boost-transparency-surveillance-camera-contracts/
4.1 Definitions
7433185002/ Technologists and AI researchers have not agreed on a definition of
9 https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=4344524&GUID=
transparency for AI systems. Instead, a number of terms have been
B051915D-A9AC-451E-81F8-6596032FA3F9&Options=Advanced&Search
10 https://nos.nl/artikel/2376810-rekenkamer-rotterdam-risico-op-vooringenomen- used, including explainability, interpretability, intelligibility, un-
uitkomsten-door-gebruik-algoritmes derstandability, and comprehensibility [43]. There is no consensus
 Andrew Bell, Oded Nov, and Julia Stoyanovich

on the meaning of these terms and they are often defined differ- 4.2 Running Example: Predicting
ently by different authors or used interchangeably. Furthermore, Unemployment in Portugal
transparency and its related terms cannot trivially be quantified or
To make the discussion concrete, we use a running example of an
measured, and transparency for one stakeholder does not automat-
ADS implemented in Portugal to try and prevent long-term unem-
ically imply the same for different stakeholders [27, 37].
ployment (being unemployed for 12 months or more). The long-
While having multiple definitions of transparency has been use-
term unemployed are particularly vulnerable persons, and tend to
ful for distinguishing nuance in a research setting, it also poses
earn less once they find new jobs, have poorer health and have chil-
a challenge for policy making. In contrast to technologists, poli-
dren with worse academic performance as compared to those who
cymakers favor definitions of transparency that are about human
had continuous employment [49]. The Portuguese national employ-
thought and behavior such as accountability or legibility [34]. Ta-
ment agency, the Institute for Employment and Vocational Training
ble 4.1 outlines terms related to transparency commonly used by
(IEFP), uses an ADS to allocate unemployment resources to at-risk
policymakers versus those used by technologists.
unemployed persons. The system is based on demographic data
Transparency. For the purposes of this paper, we choose to
about the individual, including their age, unemployment length,
use only the term “transparency,” in the broadest possible sense, so
and profession, along with other data on macroeconomic trends in
that it encompasses all the definitions above. This is most similar
Portugal.
to the way “explainability” is used by technologists. Here we use
The ADS is used by job counselors who work at the IEFP unem-
the definition adapted from work by Christoph Molnar and define
ployment centers spread across Portugal. This interaction model,
transparency as “the degree to which a human can understand an
where an ML system makes a prediction and a human ultimately
AI system [48].”
makes a final determination informed by the system’s predictions, is
Explanation. We use the term “explanation” to refer to an in-
referred to as having a “human-in-the-loop” (HITL). Having a HITL
stantiation of transparency. For example, to ensure transparency
is an increasingly common practice for implementing ADS [24, 56,
for a system, a technologist may create an explanation about the
77]. The ADS assigns unemployed persons as low, medium, or high
data it uses.
risk for remaining unemployed, and then job counselors have the
responsibility of assigning them to interventions such as re-skilling,
resume building, or job search training [80].
This is a useful case study for three reasons: (1) people’s access
to economic opportunity is at stake, and as a result, systems for
Terms used by policymakers Terms used by technologists predicting long-term unemployment are used widely around the
Explainability world [12, 39, 44, 59, 63, 70]; (2) the ADS exists in a dynamic set-
Transparency
Transparency ting which includes several stakeholders, like unemployed persons,
Accountability
Interpretablity job counselors who act as the human-in-the-loop, policymakers
Understandable
Intellegibility who oversee the implementation of the tool, and the technologists
Legibility
Understandability who developed the tool; (3) lessons from this case about design-
Traceability
Comprehensibility ing stakeholder-first transparent systems generalize well to other
Redress
Recourse
real-world uses cases of ADS.
Table 1: Discrepancies in the way policymakers and AI prac-
titioners communicate about the transparency of AI sys-
tems.
4.3 The Approach
There are many purposes, goals, use-cases and methods for the
transparency of ADS, which have been categorized in a number
of taxonomies and frameworks [6, 36, 43, 45, 48, 58, 60, 66, 74].
The approach we propose here has three subtle — yet important
— differences from much of the existing work in this area: (1) our
approach is stakeholder-first, capturing an emerging trend among
Automated Decision Systems. The approach described in this researchers in this field to reject existing method or use-case driven
paper applies to all Automated Decision Systems (ADS), which is approaches; (2) our approach is focused on improving the design of
any system that processes data to make decisions about people. transparent ADS, rather than attempting to categorize the entire
This means that AI systems are a subset of ADS, but there are two field of transparency; (3) our approach is aimed at designing ADS
key distinctions: (1) an ADS is underpinned by any algorithm and that comply with transparency regulations.
not just AI or machine learning, and (2) an ADS implies a context Our approach can be seen in Figure 1 and is made up of the
of use and some kind of impact. For a formal definition of ADS, following components: stakeholders, goals, purpose, and methods.
see [69]. Henceforth, we will use the term ADS. We describe each component in the remainder of this section, and
Notably, while many regulations are written to specifically men- explain how they apply to the running example.
tion “AI systems”, all the ideas they contain about transparency
could be applied to all ADS. It is likely that future regulations will 4.3.1 Stakeholders. Much of ADS transparency research is focused
focus broadly on ADS, as seen in NYC Local Law 144 of 2021 and on creating novel and innovative transparency methods for algo-
France’s Digital Republic Act. rithms, and then later trying to understand how these methods
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance

Goal Definition Example

Validity Making sure that an ADS is constructed correctly and An practitioner may use a transparency method to de-
is reasonable; encompasses ideas like making sure the bug an ADS; An auditor may gain intuition about how
ADS is reliable and robust [18] an ADS is making decisions through transparency
Trust Knowing “how often an ADS is right” and “for which A policymaker may use transparency to gain trust in
examples it is right” [37]; influences the adoption of an
the ADS; an affected individual may find through trans-
ADS [60] parency that they do not trust a particular ADS [62]
Fairness Ensuring that an ADS is fair An auditor may use an explanation about an ADS to
make sure it is fair to all groups of individuals; a practi-
tioner may use transparency tools to find bias in their
modeling pipeline
Privacy Ensuring that an ADS respects the data privacy of an An auditor individual may use an explanation of the
individual data used in an ADS to evaluate privacy concerns
Learning and To satisfy human curiosity, or increase understanding A doctor may use an explanation to understand an ADS
Support about how an ADS is supporting a real-world recom- recommendation of a certain treatment
mendation [48, 60]
Recourse Allowing a stakeholder to take some action against the An individual may use an explanation to appeal a loan
outcome of an ADS [10, 60] rejection; An individual may request to see an expla-
nation of an ADS output to understand why it was
made
Table 2: Definitions and examples of stakeholder goals for the 6 categories of ADS transparency goals.

can be used to meet stakeholders needs [10, 54]. Counter to this and significantly, affected individuals were not considered. Had
rationale, we propose a starting point that focuses on ADS stake- the practitioners adopted a robust stakeholder-first approach to
holders: assuming algorithmic transparency is intended to improve designing transparent systems they could have better considered
the understanding of a human stakeholder, technologists designing how to meet the goals of this key stakeholder group. For example,
transparent ADS must first consider the stakeholders of the system, a person may want to appeal being predicted low risk because they
before thinking about the system’s goals or the technical methods feel they are high risk for long-term unemployment and need access
for creating transparency. to better interventions.
The existing literature and taxonomies on ADS transparency
have identified a number of important stakeholders, which in-
4.3.2 Goals. There has been little consensus in the literature on
clude technologists, policymakers, auditors, regulators, humans-
how ADS goals should be classified. Some researchers have fo-
in-the-loop, and those individuals affected by the output of the
cused broadly, classifying the goals of ADS as evaluating, justify-
ADS [4, 45, 46]. While there is some overlap in how these stake-
ing, managing, improving, or learning about the outcome of an
holders may think about transparency, in general, there is no sin-
ADS [45]. Others have defined goals more closely to what can be
gle approach to designing transparent systems for these disparate
accomplished by known transparency methods, including building
stakeholder groups, and each of them has their own goals and pur-
trust, establishing causality, and achieving reliability, fairness, and
poses for wanting to understand an ADS [66]. In fact, even within
privacy [43]. Amarasinghe et al. identified five main goals (desig-
a stakeholder group there may be variations on how they define
nated as use-cases) of transparency specifically in a policy setting:
meaningful transparency [28]. Designers of ADS may also want
model-debugging, trust and adoption, whether or not to intervene,
to weight the needs of separate stakeholders differently. For exam-
improving intervention assignments, and for recourse. In this con-
ple, it may be more meaningful to meet the transparency needs of
text, the term intervention refers to a policy action associated with
affected individuals over AI managers or auditors.
the outcome of an ADS.
Importantly, by staking transparency on the needs of stakehold-
Notably, the goals of transparency are distinct from the purpose.
ers, technologists will be compliant with citizen-aware regulations
The purpose addresses a context-specific aim of the ADS. For ex-
like the Right to Explanation, and those that require audits of ADS.
ample, if an explanation is created for an ADS with the purpose
Running example. In the ADS used by IEFP in Portugal, there
of explaining to an individual why their loan was rejected, the
are four main stakeholders: the technologists who developed the
goal may be to offer individual recourse against the rejection. This
ADS, the policymakers who reviewed the ADS and passed laws for
distinction is made clear in 4.3.3.
its implementation, the job counselors who use the system, and
For our stakeholder-first approach we make two changes to the
the affected individuals who are assessed for long-term unemploy-
existing body of research work. First, we require that the goal of
ment. In the development of the AI, explanations were created to
transparent design must start with a stakeholder. Since all trans-
meet the varying goals of many of these stakeholders including
parency elements of an ADS are intended for a human audience,
practitioners, policymakers, and the job counselors. Unfortunately,
defining a stakeholder is implicit in defining goals. Second, we have
 Andrew Bell, Oded Nov, and Julia Stoyanovich

4.3.3 Purpose. Miller proposed that the purpose of transparency

is to answer a “why” question [47], and gives the following exam-
ple: In the context where a system is predicting if a credit loan
is accepted or rejected, one may ask, “why was a particular loan
rejected?” Liao et al. expanded on this significantly by creating a
“question bank” which is a mapping from a taxonomy of techni-
cal transparency methodology to different types of user questions.
Instead of just answering why questions, the works shows that
transparency can be used to answer 10 categories of questions:
questions about the input, output, and performance of the system,
how, why, why not, what if, how to be that, how to still be this,
and others [36]. These questions have two important character-
istics. First, they are context-specific and should address a direct
transparency goal of the stakeholder. Second, and importantly for
technologists, these questions can be mapped onto known methods
for creating explanations, meaning that a well-defined purpose for
transparency acts a bridge between the goals and methods.
Thoughtfully defining the goals and purpose of transparency in
ADS is critical for technologists to be compliant with regulators. It
is not sufficient to try and apply general, one-size-fits-all design
like simply showing the features that were used by an ADS. For
instance, both the proposed Algorithmic Accountability Act in the
United States and the Artificial Intelligence Act in the European
Union specifically mention that ADS should have transparency
Figure 1: A stakeholder-first approach for creating transpar- mechanisms that allow individuals to have recourse against a sys-
ent ADS. The framework is made up of four components: tem outcome. Researchers have noted that feature-highlighting
stakeholders, goals, purpose, and methods. We recommend transparency lacks utility when there is a disconnect between the
that transparency be thought of first by stakeholders, sec- explanation and real-world actions [7]. For instance, if someone is
ond by goals, before thirdly defining the purpose, and lastly rejected for a loan and the reason for that decision is the person’s
choosing an appropriate method to serve said purpose. Us- age, there is no action that they can effectively take for recourse
ing the framework is simple: starting at the top, one should against that decision.
consider each bubble in a component before moving onto Running example. In the long-term unemployment use case, there
the next component. were two main purposes of transparency: to understand why an
individual was assigned to a particular risk category, and to under-
stand what could be done to help high risk individuals lower their
chances of remaining long-term unemployed.

established 6 goal categories, which encompass those found in lit- 4.3.4 Methods. Once the stakeholders, goals, and purposes for al-
erature. These categories are validity, trust, learning and support, gorithmic transparency have been established, it is time for the tech-
recourse, fairness and privacy, and are defined in Table 2 alongside nologist to pick the appropriate transparency method (somtimes
concrete examples of how these goals may be implemented. called explainablity method). Over the past decade there has been
An important discussion surrounding goals are the justifications significant work in transparent ADS research (sometimes called
for pursuing them. For example, fairness and privacy goals may “explainable AI” research or XAI) on developing new methods for
be justified for humanitarian reasons (they are perceived by the understanding opaque ADS. There are several existing taxonomies
stakeholders as the “right thing to do”). Other justifications may of these methods, which show that explanations can be classified
be to prevent harm, like offering recourse to stakeholders against on a number of attributes like the scope (local or global), intrin-
an outcome of an ADS, or for a reward, like an explanation that sic or post-hoc, data or model, model-agnostic or model-specific,
supports a doctor’s correct diagnosis. For reasons of scope we will surrogate or model behavior, and static or interactive [6, 43, 48].
not delve into the issue of goal justification in this paper. Furthermore, researchers have created a number of different tools
Running example. In our case study, transparency is built into the to accomplish transparency in ADS [14, 14, 16, 41, 57].
ADS with the goal of offering learning and support to job counselors. In contrast to the complex classification of transparency methods
The ADS generates explanations about what factors contribute to by technologists, regulations have focused on two elements of ADS:
an individual being classified as low, medium, or high risk for long- (1) what aspect of the ADS pipeline is being explained (the data,
term unemployment, which job counselors use to help make better algorithm, or outcome)?, and (2) what is the scope of the explana-
treatment decision. Furthermore, the job counselor may also use tion (for one individual or the entire system)? Table 3 shows how
the explanation to offer recommendations for recourse against a different regulations speak to different combinations of pipeline
high risk score. and scope. In our stakeholder first-approach to transparency, we
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance

focus on these two main attributes. We will not discuss specific actually improve the perceived trust in the outcome of an algo-
methods in detail, but for the convenience of technologists we have rithm [8].
underlined them throughout this discussion. However, post-hoc transparency methods have been shown to
Data, algorithm, or outcome. Transparency methods have fo- have two weaknesses that technologists should be aware of: (1) in
cused on generating explanations for three different “points in time” many cases, these methods are at-best approximations of the black-
in an ADS pipeline: the data (pre-processing), the model/algorithm box they are trying to explain [81], and (2) these methods may
(in-processing, intrinsic), or the outcome (post-processing, post- be vulnerable to adversarial attacks and exploitation [65]. Some
hoc) [6, 74]. Importantly, transparency is relevant for each part of researchers have also called into question the utility of black-box
the machine learning pipeline because issues likes bias can arise models and post-hoc explanation methods altogether, and have
within each component [78]. cautioned against their use in real-world contexts like clinical set-
Transparency techniques that focus on the pre-processing com- tings [61].
ponent of the pipeline, that is, on the data used to create an ADS, Scope. There are two levels at which a transparent explanation
typically include descriptive statistics or data visualizations. about an ADS can operate: it either explains its underlying algo-
Data visualizations have proved useful for informing users and rithm fully, called a “global” explanation; or it explains how the
making complex information more accessible and digestible, and algorithm operates on one specific instance, called a “local” expla-
have even been found to have a powerful persuasive effect [50, 71]. nation. Molnar further subdivides each of these levels into two
Therefore, it is advisable to use data visualization if it can easily ad- sub-levels: global explanations can either be holistic (applying to
dress the purpose of an explanation. However, visualizations should an entire algorithm, which includes all of its features, and in the
be deployed thoughtfully, as they have the ability to be abused and case of an ensemble algorithm, all of the component algorithms) or
can successfully misrepresent a message through techniques like modular, meaning they explain on part of the holistic explanation
exaggeration or understatement [51]. and local explanations can either be applied to a single individual,
Techniques for creating in-processing or post-processing ex- or aggregated to provide local explanations for an entire group [48].
planations call into question the important consideration of us- The scope of an explanation is highly relevant to the stakeholder
ing explainable versus black-box algorithms when designing AI. and goals of an explanation, and is related to whether the stake-
The machine learning community accepts two classifications of holder operates at a system or individual level. Researchers found
models: those that are intrinsically transparent by their nature that the scope of explanation can influence whether or not an in-
(sometimes called directly interpretable or white-box models), and dividual thinks a model is fair [31, 36]. Policymakers and ADS
those that are not (called black box models) [43]. Interpretable compliance officers are more apt to be concerned with system level
models, like linear regression, decision trees, or rules-based mod- goals, like ensuring that the ADS is fair, respects privacy, and is valid
els, have intrinsic transparency mechanisms that offer algorithmic overall, while humans-in-the-loop and those individuals affected
transparency, like the linear formula, the tree diagram, and the set of by the outcome of an ADS are likely more interested in seeing local
rules, respectively. There are also methods like select-regress-round explanations to pertain to their specific cases. Technologists should
that simplify black-box models into interpretable models that use a consider both.
similar set of features [33]. Naturally, there is considerable overlap between stakeholders’
As an important design consideration for technologists, researchers scope needs (for example, an auditor may want to inspect a model
have studied the effect of the complexity of a model and how it globally and look at local cases), but generally, it is important which
impacts its ability to be understood by a stakeholder. A user study scope an explanation has. Therefore designers of ADS explanations
found that the understanding of a machine learning model is nega- should be thoughtful of how they select the scope of an explanation
tively correlated with it’s complexity, and found decision trees to based on a stakeholder and their goals.
be among the model types most understood by users [3]. An addi- Running-example. In the IEFP use case, SHAP factors were given
tional, lower-level design consideration is that model complexity is to job counselors to show the top factors influencing the score of
not fixed to a particular model type, but rather to the way that the a candidate both positively and negatively [80]. The transparency
model is constructed. For example, a decision tree with 1,000 nodes provided by SHAP provided a local explanation about the outcome
will be understood far less well than a tree with only 3 or 5 nodes. of the model. A bias audit was also conducted on the entire algo-
In contrast to in-process transparency, which is intrinsically rithm, and presented to policy officials within IEFP.
built into a model or algorithm, post-hoc transparency aims to Overall, researchers found that the explanations improved the
answer questions about a model or algorithm after is has already confidence of the decisions, but counter-intuitively, had a somewhat
been created. Some of the most popular post-hoc methods are negative effect on the quality of those decisions [80].
LIME, SHAP, SAGE, and QII [14, 16, 41, 57]. These methods are
considered model-agnostic because they can be used to create ex-
planations for any model, from linear models to random forests
4.4 Putting the Approach into Practice
to neural networks. Some methods create a transparent surrogate The stakeholder-first approach describe in Section 4.3 is meant
model that mimics the behavior of a black-box model. For exam- to act as a guide for technologists creating regulatory-compliant
ple, LIME creates a linear regression to approximate an underlying ADS. Putting this approach into practice is simple: starting at the
black-box model [41]. More work needs to be done in this direction, first component in Figure 1 (stakeholders), one should consider
but one promising study has shown that post-hoc explanations can each bubble, before moving onto the next component and again
considering each bubble. By the time one has finished worked
 Andrew Bell, Oded Nov, and Julia Stoyanovich

Data Algorithm Outcome

Right to Explanation gives indi- Both the proposed Algorithmic
GDPR (EU) gives individuals the
viduals the right to know how an Accountability Act (US) and Ar-
Local right to request a copy of any of
algorithm made a decision about tificial Intelligence Act (AI) give
their personal data
them individuals the right to recourse
EU Regulation 2019/115 requires
OPEN Government Data Act (US) that online stores and search en- NYC Int 1894-2020 requires hiring
Global mandates the government pub- gines to disclose the algorithmic algorithms be audited for biased
lishes public data parameters used to rank goods outcomes
and services on their site
Table 3: How different laws regulate the aspects the ADS pipeline (the data, algorithm or outcome), and within what scope
(local or global).

their way through the figure, they should have considered all the [2] Evgeni Aizenberg and Jeroen Van Den Hoven. Designing for human rights in ai.
possible stakeholders, goals, purposes, and methods of an ADS. An Big Data & Society, 7(2):2053951720949566, 2020.
[3] Hiva Allahyari and Niklas Lavesson. User-oriented assessment of classification
instantiation of the approach can be found throughout Section 4.3 model understandability. In Anders Kofod-Petersen, Fredrik Heintz, and Helge
in the running example of building an ADS that predicts the risk of Langseth, editors, Eleventh Scandinavian Conference on Artificial Intelligence,
SCAI 2011, Trondheim, Norway, May 24th - 26th, 2011, volume 227 of Frontiers in
long-term unemployment in Portugal. Artificial Intelligence and Applications, pages 11–19. IOS Press, 2011.
It’s important to note that our proposed stakeholder-first ap- [4] Kasun Amarasinghe, Kit T. Rodolfa, Hemank Lamba, and Rayid Ghani. Explain-
proach is only a high-level tool for thinking about ADS trans- able machine learning for public policy: Use cases, gaps, and research directions.
CoRR, abs/2010.14374, 2020.
parency through the perspective of stakeholders and their needs. [5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias.
Beyond this approach, there are meaningful low-level steps that propublica. See https://www. propublica. org/article/machine-bias-risk-assessments-
can be taken by technologists when it comes to actually implement in-criminal-sentencing, 2016.
[6] Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind,
transparency into ADS. One such step is the use of participatory Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mo-
design, where stakeholders are included directly in design conver- jsilovic, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John T. Richards,
Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney,
sations [2, 13, 21, 26]. In one promising study researchers used Dennis Wei, and Yunfeng Zhang. AI explainability 360: An extensible toolkit for
participatory design to successfully create better algorithmic expla- understanding data and machine learning models. J. Mach. Learn. Res., 21:130:1–
nations for users in the field of communal energy accounting [13]. 130:6, 2020.
[7] Solon Barocas, Andrew D Selbst, and Manish Raghavan. The hidden assumptions
behind counterfactual explanations and principal reasons. In Proceedings of the
2020 Conference on Fairness, Accountability, and Transparency, pages 80–89, 2020.
5 CONCLUDING REMARKS [8] Nadia El Bekri, Jasmin Kling, and Marco F. Huber. A study on trust in black box
models and post-hoc explanations. In Francisco Martínez-Álvarez, Alicia Tron-
If there is to be a positive, ethical future for the use of AI sys- coso Lora, José António Sáez Muñoz, Héctor Quintián, and Emilio Corchado,
tems, there needs to be stakeholder-driven design for creating editors, 14th International Conference on Soft Computing Models in Industrial
transparency algorithms — and who better to lead this effort than and Environmental Applications (SOCO 2019) - Seville, Spain, May 13-15, 2019,
Proceedings, volume 950 of Advances in Intelligent Systems and Computing, pages
technologists. Here we proposed a stakeholder-first approach that 35–46. Springer, 2019.
technologists can use to guide their design of transparent AI sys- [9] Andrew Bell, Alexander Rich, Melisande Teng, Tin Orešković, Nuno B Bras,
tems that are compliant with existing and proposed AI regulations. Lénia Mestrinho, Srdan Golubovic, Ivan Pristas, and Leid Zejnilovic. Proactive
advising: a machine learning driven approach to vaccine hesitancy. In 2019 IEEE
While there is still significant research that needs to be done in International Conference on Healthcare Informatics (ICHI), pages 1–6. IEEE, 2019.
understanding how the transparency of AI systems can be most [10] Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yun-
han Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley.
useful for stakeholders, and in the policy design of AI regulation, Explainable machine learning in deployment, 2020.
this paper aims to be a step in the right direction. [11] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy
There are several important research steps that could be taken disparities in commercial gender classification. In Sorelle A. Friedler and Christo
Wilson, editors, Conference on Fairness, Accountability and Transparency, FAT
to extend this work. First, the stakeholder-first approach described 2018, 23-24 February 2018, New York, NY, USA, volume 81 of Proceedings of Machine
here lays the foundation for creating a complete playbook to de- Learning Research, pages 77–91. PMLR, 2018.
signing transparent systems. This playbook would be useful to a [12] Dorte Caswell, Greg Marston, and Jørgen Elm Larsen. Unemployed citizen or
‘at risk’client? classification systems and employment services in denmark and
number of audiences including technologists, humans-in-the-loop, australia. Critical Social Policy, 30(3):384–404, 2010.
and policymakers. Second, a repository of examples and use cases [13] Florian Cech. Tackling algorithmic transparency in communal energy accounting
through participatory design. In C&T’21: Proceedings of the 10th International
of regulatory-compliant systems derived from this approach could Conference on Communities & Technologies-Wicked Problems in the Age of Tech,
be created, to act as a reference to technologists. pages 258–268, 2021.
[14] Ian Covert, Scott M. Lundberg, and Su-In Lee. Dblp:journals/corr/abs-2004-
00668 feature contributions through additive importance measures. CoRR,
REFERENCES abs/2004.00668, 2020.
[15] Jessica Dai, Sina Fazelpour, and Zachary Lipton. Fair machine learning under
[1] Ashraf Abdul, Christian von der Weth, Mohan Kankanhalli, and Brian Y Lim. partial compliance. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics,
Cogam: Measuring and moderating cognitive load in machine learning model and Society, pages 55–65, 2021.
explanations. In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems, pages 1–14, 2020.
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance

[16] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quanti- Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Ad-
tative input influence: Theory and experiments with learning systems. In 2016 vances in Neural Information Processing Systems 30: Annual Conference on Neural
IEEE symposium on security and privacy (SP), pages 598–617. IEEE, 2016. Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA,
[17] Inigo Martinez de Troya, Ruqian Chen, Laura O Moraes, Pranjal Bajaj, Jordan pages 4765–4774, 2017.
Kupersmith, Rayid Ghani, Nuno B Brás, and Leid Zejnilovic. Predicting, explain- [42] Gianclaudio Malgieri. Automated decision-making in the eu member states: The
ing, and understanding risk of long-term unemployment. In NeurIPS Workshop right to explanation and other “suitable safeguards” in the national legislations.
on AI for Social Good, 2018. Computer law & security review, 35(5):105327, 2019.
[18] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable [43] Ricards Marcinkevics and Julia E. Vogt. Interpretability and explainability: A
machine learning. arXiv preprint arXiv:1702.08608, 2017. machine learning zoo mini-tour. CoRR, abs/2012.01805, 2020.
[19] Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, [44] Simon Matty. Predicting Likelihood of Long-term Unemployment: The Development
David O’Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, of a UK Jobseekers’ Classification Instrument. Corporate Document Services, 2013.
et al. Accountability of ai under the law: The role of explanation. arXiv preprint [45] Christian Meske, Enrico Bunde, Johannes Schneider, and Martin Gersch. Ex-
arXiv:1711.01134, 2017. plainable artificial intelligence: Objectives, stakeholders and future research
[20] Lilian Edwards and Michael Veale. Enslaving the algorithm: From a “right to an opportunities. Information Systems Management, 12 2020.
explanation” to a “right to better decisions”? IEEE Security & Privacy, 16(3):46–54, [46] Marcia K Meyers, Susan Vorsanger, B Guy Peters, and Jon Pierre. Street-level
2018. bureaucrats and the implementation of public policy. The handbook of public
[21] Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike administration, pages 153–163, 2007.
Haug, and Heinrich Hussmann. Bringing transparency design into practice. In [47] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences.
23rd international conference on intelligent user interfaces, pages 211–223, 2018. CoRR, abs/1706.07269, 2017.
[22] Urs Gasser and Virgílio A. F. Almeida. A layered model for AI governance. IEEE [48] Christoph Molnar. Interpretable Machine Learning. 2019. https://christophm.
Internet Comput., 21(6):58–62, 2017. github.io/interpretable-ml-book/.
[23] Indermit S Gill. Policy approaches to artificial intelligence based technologies in [49] Austin Nichols, Josh Mitchell, and Stephan Lindner. Consequences of long-term
china, european union and the united states. 2020. unemployment. Washington, DC: The Urban Institute, 2013.
[24] Philip Gillingham. Can predictive algorithms assist decision-making in social [50] Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite,
work with children and families? Child abuse review, 28(2):114–126, 2019. and Enrico Bertini. The persuasive power of data visualization. IEEE Trans. Vis.
[25] David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Comput. Graph., 20(12):2211–2220, 2014.
Guang-Zhong Yang. Xai—explainable artificial intelligence. Science Robotics, [51] Anshul Vikram Pandey, Katharina Rall, Margaret L Satterthwaite, Oded Nov,
4(37), 2019. and Enrico Bertini. How deceptive are deceptive visualizations? an empirical
[26] Abhishek Gupta and Tania De Gasperis. Participatory design to build better analysis of common distortion techniques. In Proceedings of the 33rd annual acm
contact-and proximity-tracing apps. arXiv preprint arXiv:2006.00432, 2020. conference on human factors in computing systems, pages 1469–1478, 2015.
[27] Michael Hind. Explaining explainable ai. XRDS: Crossroads, The ACM Magazine [52] Jack Parker and David Danks. How technological advances can reveal rights. In
for Students, 25(3):16–19, 2019. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’19,
[28] Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven Mark page 201, New York, NY, USA, 2019. Association for Computing Machinery.
Drucker. Gamut: A design probe to understand how data scientists understand [53] Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, and Ece
machine learning models. In Stephen A. Brewster, Geraldine Fitzpatrick, Anna L. Kamar. What you see is what you get? the impact of representation criteria on
Cox, and Vassilis Kostakos, editors, Proceedings of the 2019 CHI Conference on human bias in hiring. CoRR, abs/1909.03567, 2019.
Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, [54] Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo
2019, page 579. ACM, 2019. Chakraborty. Stakeholders in explainable ai. arXiv preprint arXiv:1810.00184,
[29] Andreas Holzinger, André Carrington, and Heimo Müller. Measuring the quality 2018.
of explanations: the system causability scale (scs). KI-Künstliche Intelligenz, pages [55] Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury.
1–6, 2020. Where responsible ai meets reality: Practitioner perspectives on enablers for
[30] Johan Huysmans, Bart Baesens, and Jan Vanthienen. Using rule extraction to shifting organizational practices. arXiv preprint arXiv:2006.12358, 2020.
improve the comprehensibility of predictive models. 2006. [56] Jennifer Raso. Displacement as regulation: New regulatory technologies and
[31] Sheikh Rabiul Islam, William Eberle, Sheikh Khaled Ghafoor, and Mohiud- front-line decision-making in ontario works. Canadian Journal of Law and Society,
din Ahmed. Explainable artificial intelligence approaches: A survey. CoRR, 32(1):75–95, 2017.
abs/2101.09429, 2021. [57] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust
[32] Anna Jobin, Marcello Ienca, and Effy Vayena. Artificial intelligence: the global you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM
landscape of ethics guidelines. CoRR, abs/1906.11668, 2019. SIGKDD international conference on knowledge discovery and data mining, pages
[33] Jongbin Jung, Connor Concannon, Ravi Shroff, Sharad Goel, and Daniel G Gold- 1135–1144, 2016.
stein. Simple rules for complex decisions. arXiv preprint arXiv:1702.04690, 2017. [58] John T Richards, David Piorkowski, Michael Hind, Stephanie Houde, Aleksandra
[34] P. M. Krafft, Meg Young, Michael A. Katell, Karen Huang, and Ghislain Bugingo. Mojsilovic, and Kush R Varshney. A human-centered methodology for creating
Defining AI in policy versus practice. In Annette N. Markham, Julia Powles, Toby ai factsheets. IEEE Data Eng. Bull., 44(4):47–58, 2021.
Walsh, and Anne L. Washington, editors, AIES ’20: AAAI/ACM Conference on AI, [59] T Riipinen. Risk profiling of long-term unemployment in finland. In Power Point
Ethics, and Society, New York, NY, USA, February 7-8, 2020, pages 72–78. ACM, presentation at the European Commission’s “PES to PES Dialogue Dissemination
2020. Conference,” Brussels, September, pages 8–9, 2011.
[35] Maciej Kuziemski and Gianluca Misuraca. Ai governance in the public sec- [60] Kit T. Rodolfa, Hemank Lamba, and Rayid Ghani. Machine learning for public
tor: Three tales from the frontiers of automated decision-making in democratic policy: Do we need to sacrifice accuracy to make models fair?, 2020.
settings. Telecommunications Policy, 44(6):101976, 2020. Artificial intelligence, [61] Cynthia Rudin. Stop explaining black box machine learning models for high
economy and society. stakes decisions and use interpretable models instead. Nature Machine Intelligence,
[36] Q. Vera Liao, Daniel M. Gruen, and Sarah Miller. Questioning the AI: inform- 1(5):206–215, 2019.
ing design practices for explainable AI user experiences. In Regina Bernhaupt, [62] Philipp Schmidt, Felix Biessmann, and Timm Teubner. Transparency and trust in
Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy artificial intelligence systems. Journal of Decision Systems, 29(4):260–278, 2020.
Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjøn, Shengdong Zhao, Bri- [63] Anette Scoppetta and Arthur Buckenleib. Tackling long-term unemployment
ane Paul Samson, and Rafal Kocielnik, editors, CHI ’20: CHI Conference on Human through risk profiling and outreach. 2018.
Factors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020, pages 1–15. [64] Andrew Selbst and Julia Powles. "meaningful information" and the right to
ACM, 2020. explanation. In Sorelle A. Friedler and Christo Wilson, editors, Conference on
[37] Zachary C Lipton. The mythos of model interpretability: In machine learning, Fairness, Accountability and Transparency, FAT 2018, 23-24 February 2018, New
the concept of interpretability is both important and slippery. Queue, 16(3):31–57, York, NY, USA, volume 81 of Proceedings of Machine Learning Research, page 48.
2018. PMLR, 2018.
[38] Michele Loi and Matthias Spielkamp. Towards accountability in the use of artifi- [65] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.
cial intelligence for public administrations. In Proceedings of the 2021 AAAI/ACM Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods.
Conference on AI, Ethics, and Society, pages 757–766, 2021. In Annette N. Markham, Julia Powles, Toby Walsh, and Anne L. Washington,
[39] Artan Loxha and Matteo Morgandi. Profiling the unemployed: a review of oecd editors, AIES ’20: AAAI/ACM Conference on AI, Ethics, and Society, New York, NY,
experiences and implications for emerging economies. 2014. USA, February 7-8, 2020, pages 180–186. ACM, 2020.
[40] Joy Lu, Dokyun Lee, Tae Wan Kim, and David Danks. Good explanation for [66] Kacper Sokol and Peter A. Flach. One explanation does not fit all: The
algorithmic transparency. Available at SSRN 3503603, 2019. promise of interactive explanations for machine learning transparency. CoRR,
[41] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model abs/2001.09734, 2020.
predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M.
 Andrew Bell, Oded Nov, and Julia Stoyanovich

[67] Gregor Stiglic, Petra Povalej Brzan, Nino Fijacko, Fei Wang, Boris Delibasic, pages 151–157, 2018.
Alexandros Kalousis, and Zoran Obradovic. Comprehensible predictive modeling [75] Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. Transparent, explainable,
using regularized logistic regression and comorbidity based features. PloS one, and accountable ai for robotics. 2017.
10(12):e0144439, 2015. [76] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations
[68] Julia Stoyanovich and Ellen P Goodman. Revealing algorithmic rankers. Freedom without opening the black box: Automated decisions and the gdpr. Harv. JL &
to Tinker (August 5 2016), 2016. Tech., 31:841, 2017.
[69] Julia Stoyanovich, Bill Howe, and H.V. Jagadish. Responsible data management. [77] Ben Wagner. Liable, but not in control? ensuring meaningful human agency in
PVLDB, 13(12):3474–3489, 2020. automated decision-making systems. Policy & Internet, 11(1):104–122, 2019.
[70] Karolina Sztandar-Sztanderska and Marianna Zielenska. Changing social citizen- [78] Ke Yang, Biao Huang, Julia Stoyanovich, and Sebastian Schelter. Fairness-aware
ship through information technology. Social Work & Society, 16(2), 2018. instrumentation of preprocessing pipelines for machine learning. In HILDA
[71] Aner Tal and Brian Wansink. Blinded with science: Trivial graphs and formulas workshop at SIGMOD, 2020.
increase ad persuasiveness and belief in product efficacy. Public Understanding [79] Yiwei Yang, Eser Kandogan, Yunyao Li, Prithviraj Sen, and Walter S Lasecki. A
of Science, 25(1):117–125, 2016. study on interaction in human-in-the-loop machine learning for text analytics.
[72] UNICRI. Towards responsible artificial intelligence innovation. 2020. In IUI Workshops, 2019.
[73] Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse in linear [80] Leid Zejnilović, Susana Lavado, Íñigo Martínez de Rituerto de Troya, Samantha
classification. In Proceedings of the conference on fairness, accountability, and Sim, and Andrew Bell. Algorithmic long-term unemployment risk assessment in
transparency, pages 10–19, 2019. use: Counselors’ perceptions and use practices. Global Perspectives, 1(1), 2020.
[74] Elio Ventocilla, Tove Helldin, Maria Riveiro, Juhee Bae, Veselka Boeva, Göran [81] Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell.
Falkman, and Niklas Lavesson. Towards a taxonomy for interpretable and inter- " why should you trust my explanation?" understanding uncertainty in lime
active machine learning. In XAI Workshop on Explainable Artificial Intelligence, explanations. arXiv preprint arXiv:1904.12991, 2019.