7/26/2018 Of oaths and checklists - O'Reilly Media

ON OUR
RADAR AI BUSINESS DATA DESIGN ECONOMY JUPY TER SEE ALL
OPERATIONS

DATA SCIE NCE

Of oaths and checklists

Oaths have their value, but checklists will help put principles into practice.
By Mike Loukides, Hilary Mason, and DJ Patil. July 17, 2018

Check mark (source: Pixabay)

Check
Check out
Checkout the
outthe tutorial
thetutorial course
tutorialcourse "How
course"How to
"Howto be
tobe fair:
fair:aaatutorial
befair: tutorial for
tutorialfor beginners"
beginners" at Strata Data in New
forbeginners"
York, September 11 13, 2018. Hurry—early price ends July 27.

This post is part of a series

series on
serieson data
ondata ethics.
dataethics
ethics

"Oaths? We don't need no stinkin' oaths" (with apologies to Humphrey Bogart in “Treasure
of the Sierra Madre”).

https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 1/12
7/26/2018 Of oaths and checklists - O'Reilly Media

Over the past year, there has been a great discussion of data ethics, motivated in part by
discomfort over “fake news,” targeted advertising, algorithmic bias, and the effect that
data products have on individuals and on society. Concern about data ethics is hardly
new; the ACM
ACM, IEEE,
IEEE and the American
ACM IEEE American Statistical
AmericanStatistical Association
Association all have ethical codes that
StatisticalAssociation
address data. But the intensity with which we’ve discussed ethics shows that something
significant is happening: data science is coming of age and realizing its responsibilities. A
better world won’t come about simply because we use data; data has its dark underside.

The recent discussion frequently veers into a discussion of data

data oaths,
oaths looking back to the
dataoaths
ancient Hippocratic
Hippocratic Oath
Oath for doctors. Much as we appreciate the work and the thought
HippocraticOath
that goes into oaths, we are skeptical about their value. Oaths have several problems:

S T R ATA DATA C O N F E R E N C E

Strata Data Conference in New York, September 11 13,

2018
Early price ends July 27

https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 2/12
7/26/2018 Of oaths and checklists - O'Reilly Media

They're one-shots. You take the oath once (if at all), and that's it. There's no reason to
keep it in the front of your consciousness. You don’t recite it each morning. Or evaluate
regularly whether you’re living up to the ideals.

Oaths are a set of very general and broad principles. Discussions of the Hippocratic
Oath begin with the phrase "First, do no harm," words that don’t actually appear in the
oath. But what does “do no harm” mean? For centuries doctors did very little but harm
(many people died because doctors didn’t believe they needed to wash their hands).
The doctors just didn't know they were doing harm. Nice idea, but short on the
execution. And data science (like medicine) is all about execution.

Oaths can actually give cover to people and organizations who are doing unethical
work. It’s easy to think “we can’t be unethical, because we endorsed this oath.” It’s not
enough to say “don’t be evil.” You have to not be evil.

Oaths do very little to connect theories and principles to practice. It is one thing to say
"researchers must obtain informed consent"; it's an entirely different thing to get
informed consent at internet scale. Or to teach users what "informed consent" means.

We are not suggesting that the principles embodied in oaths aren't important, just that
they don't get us to the endpoint we want. They don't connect our ideas about what's
good or just to the practices that create goodness and justice. We can talk a lot about the
importance of being fair and unbiased without knowing about how to be fair and
unbiased. At this point, the oath actually becomes dangerous: it becomes a tool to
convince yourself that you're one of the good guys, that you're doing the right thing,
when you really don't know.

Oaths are good at creating discussion—and, in the past year, they have created quite a lot
of discussion. The discussion has been tremendously helpful in making people aware of
issues like algorithmic fairness. The discussion has helped software developers and data
scientists to understand that their work isn’t value-neutral, that their work has real
impact, both good and bad, on real people. And there has been a vigorous debate about
what self-government means for data scientists, and what guiding principles would last
longer than a few years. But we need to take the next step, and connect these ideas to
practice. How will we do that?

In 2009, Atul Gawande wrote The

The Checklist
TheChecklist Manifesto,
Manifesto a short book on how not to make
ChecklistManifesto
big mistakes. He writes a lot about his practice as a surgeon. In a hospital, everyone
knows what to do. Everyone knows that you're supposed to scrub down before the
https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 3/12
7/26/2018 Of oaths and checklists - O'Reilly Media

surgery. Everyone knows that you're not supposed to amputate the wrong leg. Everyone
knows that you're not supposed to leave sponges and other equipment in patients when
you close the incision.

But mistakes are made, particularly when people are in stressful environments. The
surgeon operates on the wrong leg; the sponge is left behind; and so on. Gawande found
that, simply by creating checklists for basic things you shouldn't forget, these mistakes
could be eliminated almost completely. Yes, there were some doctors who found the idea
of checklists insultingly simple; they were the ones who continued making mistakes.

Unlike oaths, checklists connect principle to practice. Everyone knows to scrub down
before the operation. That's the principle. But if you have to check a box on a form after
you've done it, you're not likely to forget. That's the practice. And checklists aren't one-
shots. A checklist isn’t something you read once at some initiation ceremony; a checklist
is something you work through with every procedure.

O'Reilly Data Newsletter

Get the O'Reilly Data Newsletter

Receive weekly insight from industry insiders—plus exclusive content,
offers, and more on the topic of data.

Your Email

Country

- Select Country -

Subscribe

Please read our Privacy Policy.

What would a checklist for data science and machine learning look like? The UK
UK
UK
Government’s
Government’s Data
Government’sData Ethics
DataEthics Framework
Framework and Data
EthicsFramework Data Ethics
DataEthics Workbook
Workbook is one approach. They
EthicsWorkbook
isolate 7 principles, and link to detailed discussions of each. The workbook asks a number
of open-ended questions to probe your compliance with these principles. Our criticism is
that their process imposes a lot of overhead. While anyone going through their entire
process will certainly have thought carefully about ethical issues, in practice, asking
https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 4/12
7/26/2018 Of oaths and checklists - O'Reilly Media

developers to fill out a workbook with substantive answers to 46 questions is an effective

way to ensure that ethical thought doesn’t happen.

We believe that checklists are built around simple, “have we done this?” questions—and
they are effective because they are simple. They don’t leave much room to wiggle. Either
you’ve analyzed how a project can be abused, or you haven’t. You’ve built a mechanism
for gathering consent, or you haven’t. Granted, it’s still possible to take shortcuts: your
analysis might be inadequate and your consent mechanism might be flawed, but you’ve
at least gone on record for saying that you’ve done it.

Here's a checklist for people who are working on data projects:

❏ Have we listed how this technology can be attacked or abused?

❏ Have we tested our training data to ensure it is fair and representative?

❏ Have we studied and understood possible sources of bias in our data?

❏ Does our team reflect diversity of opinions, backgrounds, and kinds of thought?

❏ What kind of user consent do we need to collect to use the data?

❏ Do we have a mechanism for gathering consent from users?

❏ Have we explained clearly what users are consenting to?

❏ Do we have a mechanism for redress if people are harmed by the results?

❏ Can we shut down this software in production if it is behaving badly?

❏ Have we tested for fairness with respect to different user groups?

❏ Have we tested for disparate error rates among different user groups?

❏ Do we test and monitor for model drift to ensure our software remains fair over time?

❏ Do we have a plan to protect and secure user data?

This checklist isn't without its problems, but it's a start; feel free to use it and modify in
your projects. It covers most of the bases that we’ve seen discussed in various data oaths.
https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 5/12
7/26/2018 Of oaths and checklists - O'Reilly Media

Go over the checklist when starting a product so the developers know what’s needed and
aren’t surprised by a new set of requirements at the last minute. Then work through it
whenever you release software. Go through it, and actually check off all the boxes before
your project hits the public.

Oaths and codes of conduct have their value. The value of an oath isn't the pledge itself,
but the process you go through in developing the oath. People who work with data are
now having discussions that would never have taken place a decade ago. But discussions
don’t get the hard work done, and we need to get down to the hard work. We don't want
to talk about how to use data ethically; we want to use data ethically. It's hypocritical to
talk about ethics, but never do anything about it. We want to put our principles into
practice. And that's what checklists will help us do.

Article image: Check mark (source: Pixabay).

Tags: Data Ethics. 

Share Tweet Share 34 Share

Mike Loukides
Mike Loukides is Vice President of Content Strategy for O'Reilly Media, Inc. He's edited many highly
regarded books on technical subjects that don't involve Windows programming. He's particularly
interested in programming languages, Unix and what passes for Unix these days, and system and
network administration. Mike is the author of System Performance Tuning and a coauthor of Unix
Power Tools. Most recently, he's been fooling around with data and data analysis, languages like R,
Mathematica, and Octave, and thinking about how to make books soc...
more

https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 6/12
7/26/2018 Of oaths and checklists - O'Reilly Media

Hilary Mason
Hilary Mason is the General Manager for Machine Learning at Cloudera. Previously, she founded Fast
Forward Labs, an applied machine learning research and advisory company, which was acquired by
Cloudera in 2017. Hilary is the Data Scientist in Residence at Accel Partners, and is on the board of the
Anita Borg Institute. Previously, she co-founded HackNY.org, a non-profit that helps engineering
students find opportunities in New York's creative technical economy, served on Mayer Bloomberg's
Technology Advisory Council, and was the Chief Scientis...
more

DJ Patil
DJ Patil is the first-ever U.S. Chief Data Scientist where he led the national strategy for data.
Currently he is building Devoted Health, a new health care company with the mission to take care of
every member like they were family. He is also a Senior Fellow at the Harvard Kennedy School and an
Advisor to Venrock Partners. Previously, DJ served as VP of Product at RelateIQ, which was acquired
by Salesforce.com, and Head of Data Products and Chief Security Officer at LinkedIn. DJ can be
reached on Twitter @dpatil and on LinkedIn.
more

https://www.oreilly.com/ideas/of-oaths-and-checklists?utm_medium=email&utm_source=topic+optin&utm_campaign=awareness&utm_content=2018… 7/12