2021 AI Index Report Master
2021 AI Index Report Master
2021 AI Index Report Master
Intelligence
Index Report 2021
Artificial Intelligence
Index Report 2021
INTRODUCTION TO THE
2021 AI INDEX REPORT
Welcome to the fourth edition of the AI Index Report! • Diversity and ethics data: We gathered more data
This year we significantly expanded the amount of data for this year’s report, although our investigation
available in the report, worked with a broader set of surfaced several areas where the AI community
external organizations to calibrate our data, and deepened currently lacks good information.
our connections with Stanford’s Institute for Human- • Country comparisons: Readers were generally
Centered Artificial Intelligence (HAI). interested in being able to use the AI Index for cross-
country comparisons. To support this, we:
The AI Index Report tracks, collates, distills, and visualizes
• gathered more data to allow for comparison
data related to artificial intelligence. Its mission is to
among countries, especially relating to
provide unbiased, rigorously vetted, and globally sourced
economics and bibliometrics; and
data for policymakers, researchers, executives, journalists,
• included a thorough summary of the various AI
and the general public to develop intuitions about the
strategies adopted by different countries and
complex field of AI. The report aims to be the world’s most
how they evolved over time.
credible and authoritative source for data and insights
about AI. P U B L I C DATA A N D TO O L S
The AI Index 2021 Report is supplemented by raw data
C OV I D A N D A I
and an interactive tool. We invite each member of the AI
The 2021 report shows the effects of COVID-19 on AI
community to use the data and tool in a way most relevant
development from multiple perspectives. The Technical
to their work and interests.
Performance chapter discusses how an AI startup used
• Raw data and charts: The public data and high-
machine-learning-based techniques to accelerate COVID-
resolution images of all the charts in the report are
related drug discovery during the pandemic, and our
available on Google Drive.
Economy chapter suggests that AI hiring and private
• Global AI Vibrancy Tool: We revamped the Global AI
investment were not significantly adversely influenced
Vibrancy Tool this year, allowing for better interactive
by the pandemic, as both grew during 2020. If anything,
visualization when comparing up to 26 countries
COVID-19 may have led to a higher number of people
across 22 indicators. The updated tool provides
participating in AI research conferences, as the pandemic
transparent evaluation of the relative position of
forced conferences to shift to virtual formats, which in turn
countries based on users’ preference; identifies
led to significant spikes in attendance.
relevant national indicators to guide policy priorities
CHANGES FOR THIS EDITION at a country level; and shows local centers of AI
In 2020, we surveyed more than 140 readers from excellence for not just advanced economies but also
government, industry, and academia about what they emerging markets.
found most useful about the report and what we should • Issues in AI measurement: In fall 2020, we
change. The main suggested areas for improvement were: published “Measurement in AI Policy: Opportunities
• Technical performance: We significantly expanded and Challenges,” a report that lays out a variety of
this chapter in 2021, carrying out more of our own AI measurement issues discussed at a conference
analysis. hosted by the AI Index in fall 2019.
2
Artificial Intelligence
Index Report 2021
Table of Contents
INTRODUCTION TO THE 2021 AI INDEX REPORT 2
TOP 9 TAKEAWAYS 4
AI INDEX STEERING COMMITTEE & STAFF 5
HOW TO CITE THE REPORT 6
ACKNOWLEDGMENTS 7
REPORT HIGHLIGHTS 10
APPENDIX 177
3
Artificial Intelligence
Index Report 2021
TOP 9 TAKEAWAYS
1 AI investment in drug design and discovery increased significantly: “Drugs, Cancer, Molecular,
Drug Discovery” received the greatest amount of private AI investment in 2020, with more than USD
13.8 billion, 4.5 times higher than 2019.
2 The industry shift continues: In 2019, 65% of graduating North American PhDs in AI went into
industry—up from 44.4% in 2010, highlighting the greater role industry has begun to play in AI
development.
3 Generative everything: AI systems can now compose text, audio, and images to a sufficiently high
standard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology.
4 AI has a diversity challenge: In 2019, 45% new U.S. resident AI PhD graduates were white—by
comparison, 2.4% were African American and 3.2% were Hispanic.
5 China overtakes the US in AI journal citations: After surpassing the United States in the total
number of journal publications several years ago, China now also leads in journal citations;
however, the United States has consistently (and significantly) more AI conference papers (which are also
more heavily cited) than China over the last decade.
6 The majority of the US AI PhD grads are from abroad—and they’re staying in the US:
The percentage of international students among new AI PhDs in North America continued to rise in
2019, to 64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States
and 8.6% have taken jobs outside the United States.
7 Surveillance technologies are fast, cheap, and increasingly ubiquitous: The technologies
necessary for large-scale surveillance are rapidly maturing, with techniques for image classification,
face recognition, video analysis, and voice identification all seeing significant progress in 2020.
8 AI ethics lacks benchmarks and consensus: Though a number of groups are producing a range
of qualitative or normative outputs in the AI ethics domain, the field generally lacks benchmarks
that can be used to measure or assess the relationship between broader societal discussions about
technology development and the development of the technology itself. Furthermore, researchers and
civil society view AI ethics as more important than industrial organizations.
9 AI has gained the attention of the U.S. Congress: The 116th Congress is the most AI-focused
congressional session in history with the number of mentions of AI in congressional record more
than triple that of the 115th Congress.
4
Artificial Intelligence
Index Report 2021
Co-Directors
Jack Clark Raymond Perrault
OECD, GPAI SRI International
Members
Erik Brynjolfsson James Manyika
Stanford University McKinsey Global Institute
Terah Lyons
Partnership on AI
AI Index Staff
Program Manager
Saurabh Mishra
Stanford University
5
Artificial Intelligence
Index Report 2021
The AI Index was conceived within the One Hundred Year Study on AI (AI100).
6
Artificial Intelligence
Index Report 2021
Acknowledgments
We appreciate the following organizations and individuals who provided data, analysis,
advice, and expert commentary for inclusion in the AI Index 2021 Report:
Organizations
arXiv LinkedIn
Jim Entwood, Paul Ginsparg, Guy Berger, Mar Carpanelli, Di Mo,
Joe Halpern, Eleonora Presani Virginia Ramsey
7
Artificial Intelligence
Index Report 2021
Individuals
8
Artificial Intelligence
Index Report 2021
Graduate Researchers
Ankita Banerjea, Yu-chi Tsao (Stanford University)
9
Artificial Intelligence
Index Report 2021
REPORT HIGHLIGHTS
C H A P T E R 1 : R E S E A R C H & D E V E LO P M E N T
• The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher percentage growth
than from 2018 to 2019 (19.6%).
• In every major country and region, the highest proportion of peer-reviewed AI papers comes from academic
institutions. But the second most important originators are different: In the United States, corporate-
affiliated research represents 19.2% of the total publications, whereas government is the second most
important in China (15.6%) and the European Union (17.2%).
• In 2020, and for the first time, China surpassed the United States in the share of AI journal citations in the
world, having briefly overtaken the United States in the overall number of AI journal publications in 2004 and
then retaken the lead in 2017. However, the United States has consistently (and significantly) more cited AI
conference papers than China over the last decade.
• In response to COVID-19, most major AI conferences took place virtually and registered a significant increase
in attendance as a result. The number of attendees across nine conferences almost doubled in 2020.
• In just the last six years, the number of AI-related publications on arXiv grew by more than sixfold, from 5,478
in 2015 to 34,736 in 2020.
• AI publications represented 3.8% of all peer-reviewed scientific publications worldwide in 2019, up from
1.3% in 2011.
C H A P T E R 2: T E C H N I CA L P E R F O R M A N C E
• Generative everything: AI systems can now compose text, audio, and images to a sufficiently high standard
that humans have a hard time telling the difference between synthetic and non-synthetic outputs for some
constrained applications of the technology. That promises to generate a tremendous range of downstream
applications of AI for both socially useful and less useful purposes. It is also causing researchers to invest in
technologies for detecting generative models; the DeepFake Detection Challenge data indicates how well
computers can distinguish between different outputs.
• The industrialization of computer vision: Computer vision has seen immense progress in the past decade,
primarily due to the use of machine learning techniques (specifically deep learning). New data shows that
computer vision is industrializing: Performance is starting to flatten on some of the largest benchmarks,
suggesting that the community needs to develop and agree on harder ones that further test performance.
Meanwhile, companies are investing increasingly large amounts of computational resources to train
computer vision systems at a faster rate than ever before. Meanwhile, technologies for use in deployed
systems—like object-detection frameworks for analysis of still frames from videos—are maturing rapidly,
indicating further AI deployment.
10
Artificial Intelligence
Index Report 2021
• Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has yielded AI
systems with significantly improved language capabilities that have started to have a meaningful economic impact
on the world. Google and Microsoft have both deployed the BERT language model into their search engines, while
other large language models have been developed by companies ranging from Microsoft to OpenAI. Progress in
NLP has been so swift that technical advances have started to outpace the benchmarks to test for them. This can
be seen in the rapid emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation
suite developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
• New analyses on reasoning: Most measures of technical problems show for each time point the performance
of the best system at that time on a fixed benchmark. New analyses developed for the AI Index offer metrics that
allow for an evolving benchmark, and for the attribution to individual systems of credit for a share of the overall
performance of a group of systems over time. These are applied to two symbolic reasoning problems, Automated
Theorem Proving and Satisfiability of Boolean formulas.
• Machine learning is changing the game in healthcare and biology: The landscape of the healthcare and
biology industries has evolved substantially with the adoption of machine learning. DeepMind’s AlphaFold
applied deep learning technique to make a significant breakthrough in the decades-long biology challenge of
protein folding. Scientists use ML models to learn representations of chemical molecules for more effective
chemical synthesis planning. PostEra, an AI startup used ML-based techniques to accelerate COVID-related drug
discovery during the pandemic.
C H A P T E R 3: T H E E C O N O M Y
• “Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI investment in 2020, with
more than USD 13.8 billion, 4.5 times higher than 2019.
• Brazil, India, Canada, Singapore, and South Africa are the countries with the highest growth in AI hiring from
2016 to 2020. Despite the COVID-19 pandemic, the AI hiring continued to grow across sample countries in 2020.
• More private investment in AI is being funneled into fewer startups. Despite the pandemic, 2020 saw a 9.3%
increase in the amount of private AI investment from 2019—a higher percentage increase than from 2018 to 2019
(5.7%), though the number of newly funded companies decreased for the third year in a row.
• Despite growing calls to address ethical concerns associated with using AI, efforts to address these concerns
in the industry are limited, according to a McKinsey survey. For example, issues such as equity and fairness
in AI continue to receive comparatively little attention from companies. Moreover, fewer companies in 2020
view personal or individual privacy risks as relevant, compared with in 2019, and there was no change in the
percentage of respondents whose companies are taking steps to mitigate these particular risks.
• Despite the economic downturn caused by the pandemic, half the respondents in a McKinsey survey said that
the coronavirus had no effect on their investment in AI, while 27% actually reported increasing their investment.
Less than a fourth of businesses decreased their investment in AI.
• The United States recorded a decrease in its share of AI job postings from 2019 to 2020—the first drop in six years.
The total number of AI jobs posted in the United States also decreased by 8.2% from 2019 to 2020, from 325,724
in 2019 to 300,999 jobs in 2020.
11
Artificial Intelligence
Index Report 2021
C H A P T E R 4 : A I E D U CAT I O N
• An AI Index survey conducted in 2020 suggests that the world’s top universities have increased their investment
in AI education over the past four years. The number of courses that teach students the skills necessary to build
or deploy a practical AI model on the undergraduate and graduate levels has increased by 102.9% and 41.7%,
respectively, in the last four academic years.
• More AI PhD graduates in North America chose to work in industry in the past 10 years, while fewer opted for jobs
in academia, according to an annual survey from the Computing Research Association (CRA). The share of new
AI PhDs who chose industry jobs increased by 48% in the past decade, from 44.4% in 2010 to 65.7% in 2019. By
contrast, the share of new AI PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019.
• In the last 10 years, AI-related PhDs have gone from 14.2% of the total of CS PhDs granted in the United States,
to around 23% as of 2019, according to the CRA survey. At the same time, other previously popular CS PhDs have
declined in popularity, including networking, software engineering, and programming languages. Compilers all
saw a reduction in PhDs granted relative to 2010, while AI and Robotics/Vision specializations saw a substantial
increase.
• After a two-year increase, the number of AI faculty departures from universities to industry jobs in North America
dropped from 42 in 2018 to 33 in 2019 (28 of these are tenured faculty and five are untenured). Carnegie Mellon
University had the largest number of AI faculty departures between 2004 and 2019 (16), followed by the Georgia
Institute of Technology (14) and University of Washington (12).
• The percentage of international students among new AI PhDs in North America continued to rise in 2019, to
64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States and 8.6% have
taken jobs outside the United States.
• In the European Union, the vast majority of specialized AI academic offerings are taught at the master’s level;
robotics and automation is by far the most frequently taught course in the specialized bachelor’s and master’s
programs, while machine learning (ML) dominates in the specialized short courses.
C H A P T E R 5: E T H I CA L C H A L L E N G E S O F A I A P P L I CAT I O N S
• The number of papers with ethics-related keywords in titles submitted to AI conferences has grown since 2015,
though the average number of paper titles matching ethics-related keywords at major AI conferences remains
low over the years.
• The five news topics that got the most attention in 2020 related to the ethical use of AI were the release of the
European Commission’s white paper on AI, Google’s dismissal of ethics researcher Timnit Gebru, the AI ethics
committee formed by the United Nations, the Vatican’s AI ethics plan, and IBM’s exiting the facial-recognition
businesses.
12
Artificial Intelligence
Index Report 2021
CHAPTER 6: DIVERSITY IN AI
• The percentages of female AI PhD graduates and tenure-track computer science (CS) faculty have remained low
for more than a decade. Female graduates of AI PhD programs in North America have accounted for less than
18% of all PhD graduates on average, according to an annual survey from the Computing Research Association
(CRA). An AI Index survey suggests that female faculty make up just 16% of all tenure-track CS faculty at several
universities around the world.
• The CRA survey suggests that in 2019, among new U.S. resident AI PhD graduates, 45% were white, while 22.4%
were Asian, 3.2% were Hispanic, and 2.4% were African American.
• The percentage of white (non-Hispanic) new computing PhDs has changed little over the last 10 years,
accounting for 62.7% on average. The share of Black or African American (non-Hispanic) and Hispanic computing
PhDs in the same period is significantly lower, with an average of 3.1% and 3.3%, respectively.
• The participation in Black in AI workshops, which are co-located with the Conference on Neural Information
Processing Systems (NeurIPS), has grown significantly in recent years. The numbers of attendees and submitted
papers in 2019 are 2.6 times higher than in 2017, while the number of accepted papers is 2.1 times higher.
• In a membership survey by Queer in AI in 2020, almost half the respondents said they view the lack of
inclusiveness in the field as an obstacle they have faced in becoming a practitioner in the AI/ML field. More than
40% of members surveyed said they have experienced discrimination or harassment at work or school.
C H A P T E R 7: A I P O L I CY A N D N AT I O N A L S T R AT E G I E S
• Since Canada published the world’s first national AI strategy in 2017, more than 30 other countries and regions
have published similar documents as of December 2020.
• The launch of the Global Partnership on AI (GPAI) and Organisation for Economic Co-operation and Development
(OECD) AI Policy Observatory and Network of Experts on AI in 2020 promoted intergovernmental efforts to work
together to support the development of AI for all.
• In the United States, the 116th Congress was the most AI-focused congressional session in history. The number
of mentions of AI by this Congress in legislation, committee reports, and Congressional Research Service (CRS)
reports is more than triple that of the 115th Congress.
13
Artificial Intelligence
Index Report 2021
CHAPTER 1:
Research &
Development
Artificial Intelligence
Index Report 2021
CHAPTER 1 PREVIEW 14
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH &
Index Report 2021 D E V E LO P M E N T
CHAPTER 1:
Chapter Preview
Overview 16 AI Patents 31
Chapter Highlights 17 Overview 31
arXiv Publications 32
1.1 PUBLICATIONS 18 Overview 32
Peer-Reviewed AI Publications 18 By Region 32
Overview 18 By Geographic Area 33
By Region 18 By Field of Study 34
By Geographic Area 20 Highlight: Deep Learning Papers on arXiv 35
By Institutional Affiliation 21
Academic-Corporate Collaboration 23 1.2 CONFERENCES 36
AI Journal Publications 25 Conference Attendance 36
Overview 25 Highlight: Corporate Representation
By Region 26 at AI Research Conferences 38
By Geographic Area 27
Citation 27 1.3 AI OPEN-SOURCE SOFTWARE
LIBRARIES 39
AI Conference Publications 28
GitHub Stars 39
Overview 28
By Region 29
By Geographic Area 30
Citation 30
CHAPTER 1 PREVIEW 15
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & OV E R V I E W
Index Report 2021 D E V E LO P M E N T
Overview
The report opens with an overview of the research and development (R&D)
efforts in artificial intelligence (AI) because R&D is fundamental to AI
progress. Since the technology first captured the imagination of computer
scientists and mathematicians in the 1950s, AI has grown into a major
research discipline with significant commercial applications. The number
of AI publications has increased dramatically in the past 20 years. The rise
of AI conferences and preprint archives has expanded the dissemination of
research and scholarly communications. Major powers, including China, the
European Union, and the United States, are racing to invest in AI research.
The R&D chapter aims to capture the progress in this increasingly complex
and competitive field.
CHAPTER 1 PREVIEW 16
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence CHAPTER
RESEARCH &
Index Report 2021 D E V E LO P M E N T HIGHLIGHTS
CHAPTER HIGHLIGHTS
• The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher
percentage growth than from 2018 to 2019 (19.6%).
• In every major country and region, the highest proportion of peer-reviewed AI papers
comes from academic institutions. But the second most important originators are
different: In the United States, corporate-affiliated research represents 19.2% of the total
publications, whereas government is the second most important in China (15.6%) and the
European Union (17.2%).
• In 2020, and for the first time, China surpassed the United States in the share of AI
journal citations in the world, having briefly overtaken the United States in the overall
number of AI journal publications in 2004 and then retaken the lead in 2017. However,
the United States has consistently (and significantly) more cited AI conference papers
than China over the last decade.
• In response to COVID-19, most major AI conferences took place virtually and registered
a significant increase in attendance as a result. The number of attendees across nine
conferences almost doubled in 2020.
• In just the last six years, the number of AI-related publications on arXiv grew by more
than sixfold, from 5,478 in 2015 to 34,736 in 2020.
CHAPTER 1 PREVIEW 17
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
AI publications include peer-reviewed publications, journal articles, conference papers, and patents. To track trends among these
publications and to assess the state of AI R&D activities around the world, the following datasets were used: the Elsevier/Scopus
database for peer-reviewed publications; the Microsoft Academic Graph (MAG) database for all journals, conference papers, and
patent publications; and arXiv and Nesta data for electronic preprints.
1.1 PUBLICATIONS
P E E R - R E V I E W E D A I P U B L I CAT I O N S
This section presents data from the Scopus database by among all peer-reviewed publications in the world. The
Elsevier. Scopus contains 70 million peer-reviewed research total number of publications grew by nearly 12 times
items curated from more than 5,000 international publishers. between 2000 and 2019. Over the same period, the
The 2019 version of the data shown below is derived from percentage of peer-reviewed publications increased from
an entirely new set of publications, so figures of all peer- 0.82% of all publications in 2000 to 3.8% in 2019.
reviewed AI publications differ from those in previous years’
By Region1
AI Index reports. Due to changes in the methodology for
Among the total number of peer-reviewed AI publications
indexing publications, the accuracy of the dataset increased
in the world, East Asia & Pacific has held the largest share
from 80% to 84% (see the Appendix for more details).
since 2004, followed by Europe & Central Asia, and North
Overview America (Figure 1.1.2). Between 2009 and 2019, South Asia
Figure 1.1.1a shows the number of peer-reviewed AI and sub-Saharan Africa experienced the highest growth
publications, and Figure 1.1.1b shows the share of those in terms of the number of peer-reviewed AI publications,
increasing by eight- and sevenfold, respectively.
NUMBER of PEER-REVIEWED AI PUBLICATIONS, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
Number of Peer-Reviewed AI Publications (in Thousands)
120
100
80
60
40
20
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.1a
1 Regions in this chapter are classified according to the World Bank analytical grouping.
CHAPTER 1 PREVIEW 18
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
4% 3.8%
Peer-Reviewed AI Publications (% of Total)
3%
2%
1%
0%
2000
2004
2006
2009
2008
2003
2005
2002
2001
2007
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.1b
40%
36.9% East Asia & Pacific
Peer-Reviewed AI Publications (% of Total)
30%
20%
17.0% North America
10%
8.8% South Asia
5.5% Middle East & North Africa
2.7% Latin America & Caribbean
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2019
2023
2016
2017
2018
2020
2021
2022
Figure 1.1.2
CHAPTER 1 PREVIEW 19
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Geographic Area
To compare the activity among the world’s major AI overtaking the European Union in 2017 (Figure 1.1.3).
players, this section shows trends of peer-reviewed AI It published 3.5 times more peer-reviewed AI papers
publications coming out of China, the European Union, in 2019 than it did in 2014—while the European Union
and the United States. As of 2019, China led in the share published just 2 times more papers and the United States
of peer-reviewed AI publications in the world, after 2.75 times more over the same period.
25%
Peer-Reviewed AI Publications (% of World Total)
22.4% China
20%
16.4% EU
15%
14.6% US
10%
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.3
CHAPTER 1 PREVIEW 20
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Institutional Affiliation
The following charts show the number of peer-reviewed could be affiliated with more than one type of institution.
AI publications affiliated with corporate, government,
The data suggests that, excluding academia, government
medical, and other institutions in China (Figure 1.1.4a),
institutions—more than those in other categories—
the European Union (Figure 1.1.4b), and the United States
consistently contribute the highest percentage of peer-
(Figure 1.1.4c).2 In 2019, roughly 95.4% of overall peer-
reviewed AI publications in both China and the European
reviewed AI publications in China were affiliated with the
Union (15.6% and 17.2 %, respectively, in 2019), while
academic field, compared with 81.9% in the European
in the United States, the highest portion is corporate-
Union and 89.6% in the United States. Those affiliation
affiliated (19.2%).
categories are not mutually exclusive, as some authors
4,352 Government
4,000
Number of Peer-Reviewed AI Publications
3,000
2,000
1,675 Corporate
1,000
382 Medical
14 Other
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.4a
2 Across all three geographic areas, the number of papers affiliated with academia exceeds that of government-, corporate-, and medical-affiliated ones; therefore, the academia affiliation is not shown,
as it would distort the graphs.
CHAPTER 1 PREVIEW 21
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
3,523 Government
Number of Peer-Reviewed AI Publications
3,000
2,000
1,594 Corporate
1,000
508 Medical
187 Other
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.4b
3,513 Corporate
Number of Peer-Reviewed AI Publications
3,000
2,277 Gov.
2,000
1,000
718 Medical
120 Other
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 1.1.4c
CHAPTER 1 PREVIEW 22
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
Academic-Corporate Collaboration
Since the 1980s, the R&D collaboration between United States produced the highest number of hybrid
academia and industry in the United States has grown academic-corporate, co-authored, peer-reviewed AI
in importance and popularity, made visible by the publications—more than double the amount in the
proliferation of industry-university research centers as European Union, which comes in second, followed by
well as corporate contributions to university research. China in third place.
Figure 1.1.5 shows that between 2015 and 2019, the
United States
European Union
China
United Kingdom
Germany
Japan
France
Canada
South Korea
Netherlands
Switzerland
India
Hong Kong
Spain
Italy
Figure 1.1.5
CHAPTER 1 PREVIEW 23
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
To assess how academic-corporate collaborations according to the world average. For example, an FWCI of
impact the Field-Weighted Citation Impact (FWCI) of 0.75 means 25% fewer citations than the world average.
AI publications from different geographic regions,
The chart shows the FWCI for all peer-reviewed AI
see Figure 1.1.6. FWCI measures how the number of
publications on the y-axis and the total number (on a log
citations received by publications compares with the
scale) of academic-corporate co-authored publications
average number of citations received by other similar
on the x-axis. To increase the signal-to-noise ratio of the
publications in the same year, discipline, and format
FWCI metric, only countries that have more than 1,000
(book, article, conference paper, etc.). A value of 1.0
peer-reviewed AI publications in 2020 are included.
represents the world average. More than or less than 1
means publications are cited more or less than expected,
Singapore
Hong Kong
Australia Switzerland
Netherlands
United Kingdom United States
2 Iran
Malaysia Canada
Germany
Italy South Korea
European Union
Spain France
Turkey
Japan China
Taiwan
1 Russia
Indonesia Brazil India
0
5 10 20 50 100 200 500 1,000 2,000
Number of Academic-Corporate Peer-Reviewed AI Publications (Log Scale)
Figure 1.1.6
CHAPTER 1 PREVIEW 24
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
A I J O U R N A L P U B L I CAT I O N S Overview
The next three sections chart the trends in the publication Overall, the number of AI journal publications in 2020 is 5.4
of AI journals, conference publications, and patents, as times higher than it was in 2000 (Figure 1.1.7a). In 2020, the
well as their respective citations that provide a signal number of AI journal publications increased by 34.5% from
for R&D impact, based on data from Microsoft Academic 2019—a much higher percentage growth than from 2018 to
Graph. MAG3 is a knowledge graph consisting of more than 2019 (19.6%). Similarly, the share of AI journal publications
225 million publications (at the end of November 2019). among all publications in the world has jumped by 0.4
percentage points in 2020, higher than the average of 0.03
NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report percentage points in the past five years (Figure 1.1.7b).
80
Number of AI Journal Publications (in Thousands)
60
40
20
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.7a
AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
AI Journal Publications (% of All Publications)
3%
2.2%
2%
1%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.7b
3 See “An Overview of Microsoft Academic Service (MAS) and Applications” and “A Review of Microsoft Academic Services for Science of Science Studies” for more details.
CHAPTER 1 PREVIEW 25
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Region
Figure 1.1.8 shows the share of AI journals—the dominant regions changes over time. In 2020, East Asia & Pacific held
publication entity in terms of numbers in the MAG the highest share (26.7%), followed by Europe & Central
database—by region between 2000 and 2020. East Asia Asia (13.3%) and North America (14.0%). Additionally,
& Pacific, Europe & Central Asia, and North America are in the last 10 years, South Asia, and Middle East & North
responsible for the majority of AI journal publications in Africa saw the most significant growth, as the number of
the past 21 years, while the lead position among the three AI journal publications in those two regions grew six- and
fourfold, respectively.
40%
AI Journal Publications (% of World Total)
30%
26.7%, East Asia & Pacific
20%
10%
4.9%, South Asia
3.1%, Middle East & North Africa
1.3%, Latin America & Caribbean
0.3%, Sub-Saharan Africa
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
Figure 1.1.8
CHAPTER 1 PREVIEW 26
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Geographic Area
Figure 1.1.9 shows that among the three major AI powers, China has had the largest share of AI journal publications in the
world since 2017, with 18.0% in 2020, followed by the United States (12.3%) and the European Union (8.6%).
25%
20%
18.0%, China
15%
12.3%, US
10%
8.6%, EU
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.9
Citation
In terms of the highest share of AI journal citations, Figure 1.1.10 shows that China (20.7%) overtook the United States
(19.8%) in 2020 for the first time, while the European Union continued to lose overall share.
AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
40%
AI Journal Citations (% of World Total)
30%
20.7%
China
20%
19.8%
US
10% 11.0%
EU
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.10
CHAPTER 1 PREVIEW 27
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
A I C O N F E R E N C E P U B L I CAT I O N S
Overview
Between 2000 and 2019, the number of AI conference publications increased fourfold, although the growth flattened
out in the past ten years, with the number of publications in 2019 just 1.09 times higher than the number in 2010.4
NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
50
Number of Publications (in Thousands)
40
30
20
10
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.11a
25%
AI Conference Publications (% of All Publications)
20.2%
20%
15%
10%
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.11b
4 Note that conference data in 2020 on the MAG system is not yet complete. See the Appendix for details.
CHAPTER 1 PREVIEW 28
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Region
Figure 1.1.12 shows that, similar to the trends in AI Asia & Pacific took the lead starting in 2004, accounting
journal publication, East Asia & Pacific, Europe & Central for more than 27% in 2020. North America overtook
Asia, and North America are the world’s dominant Europe & Central Asia to claim second place in 2018,
sources for AI conference publications. Specifically, East accounting for 20.1%, followed by 21.7% in 2020.
40%
AI Conference Publications (% of World Total)
30%
10%
2004
2006
2009
2008
2003
2005
2002
2020
2024
2026
2007
2023
2025
2022
1999
2001
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2021
Figure 1.1.12
CHAPTER 1 PREVIEW 29
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Geographic Area
China overtook the United States in the share of AI is almost nine times higher than it was in 2000. The
conference publications in the world in 2019 (Figure share of conference publications for the European Union
1.1.13). Its share has grown significantly since 2000. peaked in 2011 and continues to decline.
China’s percentage of AI conference publications in 2019
AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
25%
AI Conference Publications (% of World Total)
20% 19.4% US
15%
15.2% China
12.8% EU
10%
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.13
Citation
With respect to citations of AI conference publications, 21 years. The United States tops the list with 40.1% of
Figure 1.1.14 shows that the United States has held a overall citations in 2020, followed by China (11.8%) and
dominant lead among the major powers over the past the European Union (10.9%).
AI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
40.1% US
40%
AI Conference Citations (% of World Total)
30%
20%
11.8% China
10% 10.9% EU
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.14
CHAPTER 1 PREVIEW 30
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
A I PAT E N T S
Overview
The total number of AI patents published in the world around 2% in 2000 to 2.9% in 2020 (Figure 1.1.15b). The AI
has been steadily increasing in the past two decades, patent data is incomplete—only 8% of the dataset in 2020
growing from 21,806 in 2000 to more than 4.5 times that, includes a country or regional affiliation. There is reason
or 101,876, in 2019 (Figure 1.1.15a). The share of AI patents to question the data on the share of AI patent publications
published in the world exhibits a lesser increase, from by both region and geographic area, and it is therefore not
included in the main report. See the Appendix for details.
NUMBER of AI PATENT PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
100
Number of AI Patent Publications (in Thousands)
80
60
40
20
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.15a
2.9%
AI Patent Publications (% of All Publications)
3%
2%
1%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.1.15b
CHAPTER 1 PREVIEW 31
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
A R X I V P U B L I CAT I O N S Overview
In addition to the traditional avenues for In just six years, the number of AI-related publications on arXiv grew
publishing academic papers (discussed more than sixfold, from 5,478 in 2015 to 34,736 in 2020 (Figure 1.1.16).
above), AI researchers have embraced the
practice of publishing their work (often NUMBER of AI-RELATED PUBLICATIONS on ARXIV, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
pre–peer review) on arXiv, an online
repository of electronic preprints. arXiv
The analysis by region shows that while North America still holds the lead in the global share of arXiV AI-related
publications, its share has been decreasing—from 41.6% in 2017 to 36.3% in 2020 (Figure 1.1.17). Meanwhile, the share of
publications in East Asia & Pacific has grown steadily in the past five years—from 17.3% in 2015 to 26.5% in 2020.
40%
36.3% North America
30%
26.5% East Asia & Paci c
10%
4.0% South Asia
2.5% Middle East & North Africa
1.3% Latin America & Caribbean
0% 0.3% Sub-Saharan Africa
CHAPTER 1 PREVIEW 32
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Geographic Area
While the total number of AI-related publications on arXiv Figure 1.1.18b). The share of publication counts by the
is increasing among the three major AI powers, China is European Union, on the other hand, has remained largely
catching up with the United States (Figure 1.1.18a and unchanged.
11,280
12,000
US
Number of AI-Related Publications on arXiv
10,000
8,000
6,505
EU
6,000
5,440
China
4,000
2,000
0
2015 2016 2017 2018 2019 2020
Figure 1.1.18a
ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
AI-Related Publications on arXiv (% of World Total)
32.5% US
30%
20%
18.7% EU
15.7% China
10%
0%
2015 2016 2017 2018 2019 2020
Figure 1.1.18b
CHAPTER 1 PREVIEW 33
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
By Field of Study
Among the six fields of study related to AI on arXiv, Among the six fields of
the number of publications in Robotics (cs.RO) and
Machine Learning in computer science (cs.LG) have
study related to AI on arXiv,
seen the fastest growth between 2015 and 2020, the number of publications
increasing by 11 times and 10 times respectively
(Figure 1.1.19). In 2020, cs.LG and Computer Vision in Robotics (cs.RO) and
(cs.CV) lead in the overall number of publications, Machine Learning in
accounting for 32.0% and 31.7%, respectively, of all AI-
related publications on arXiv. Between 2019 and 2020, computer science (cs.
the fastest-growing categories of the seven studied
here were Computation and Language (cs.CL), by
LG) have seen the fastest
35.4%, and cs.RO, by 35.8%. growth between 2015 and
2020, increasing by 11 times
and 10 times respectively.
11,001 cs.CV
11,098 cs.LG
10,000
Number of AI-Related Publications on arXiv
8,000
6,000
5,573 cs.CL
4,000
2,571 cs.RO
2,000 1,923 cs.AI
1,818 stat.ML
743 cs.NE
0
2015 2016 2017 2018 2019 2020
Figure 1.1.19
CHAPTER 1 PREVIEW 34
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1.1 P U B L I CAT I O N S
Index Report 2021 D E V E LO P M E N T
7
Number of Deep Learning Publications on arXiv (in Thousands)
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 1.1.20
CHAPTER 1 PREVIEW 35
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1. 2 C O N F E R E N C E S
Index Report 2021 D E V E LO P M E N T
Conference attendance is an indication of broader industrial and academic interest in a scientific field. In the past 20 years,
AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends in
attendance at and submissions to major AI conferences.
1.2 CONFERENCES
C O N F E R E N C E AT T E N DA N C E
Last year saw a significant increase in participation levels Conference organizers
at AI conferences, as most were offered through a virtual
format. Only the 34th Association for the Advancement
report that a virtual
of Artificial Intelligence (AAAI) Conference on Artificial format allows for higher
Intelligence was held in person in February 2020.
Conference organizers report that a virtual format allows attendance of researchers
for higher attendance of researchers from all over the from all over the world,
world, though exact attendance numbers are difficult to
measure. though exact attendance
Due to the atypical nature of 2020 conference attendance numbers are difficult to
data, the 11 major AI conferences in 2019 have been
split into two categories based on 2019 attendance data:
measure.
large AI conferences with over 3,000 attendees and small
AI conferences with fewer than 3,000 attendees. Figure
1.2.1 shows that in 2020, the total number of attendees
across nine conferences almost doubled.5 In particular,
the International Conference on Intelligent Robots
and Systems (IROS) extended the virtual conference
to allow users to watch events for up to three months,
which explains the high attendance count. Because the
International Joint Conference on Artificial Intelligence
(IJCAI) was held in 2019 and January 2021—but not in
2020—it does not appear on the charts.
5 For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference; For the KR conference,
the attendance in 2020 is based on the number of registrations; For the ICPAS conference, the attendance of 450 in 2020 is an estimate as some participants may have used anonymous Zoom accounts.
CHAPTER 1 PREVIEW 36
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1. 2 C O N F E R E N C E S
Index Report 2021 D E V E LO P M E N T
25,719 IROS
25,000
22,011 NeurIPS
20,000
Number of Attendees
15,000
10,800 ICML
10,000
7,500 CVPR
5,000
4,884 AAAI
2020
2010
2014
2016
2019
2018
2013
2015
2012
2017
2011
Figure 1.2.1
5,600
ICLR
5,000
3,972
ACL
4,000
Number of Attendees
3,726
AAMAS
3,000
2,000
1,000 469
KR
450
0 ICAPS
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 1.2.2
CHAPTER 1 PREVIEW 37
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
RESEARCH & 1. 2 C O N F E R E N C E S
Index Report 2021 D E V E LO P M E N T
20%
% of Fortune Global 500 Tech-Affiliated Papers
10%
0%
40%
27.9% ICML
30% 25.6% ECCV 23.7% ICCV
21.9% CVPR
20%
10%
0%
40%
30%
19.3% AAAI 17.5% IJCAI
20%
10%
0%
CHAPTER 1 PREVIEW 38
TA B L E O F C O N T E N T S
CHAPTER 1: 1. 3 A I O P E N -
Artificial Intelligence
RESEARCH & S O U R C E S O F T WA R E
Index Report 2021 D E V E LO P M E N T LIBRARIES
A software library is a collection of computer code that is used to create applications and products. Popular AI-specific software
libraries—such as TensorFlow and PyTorch—help developers create their AI solutions quickly and efficiently. This section analyzes
the popularity of software libraries through GitHub data.
1.3 AI OPEN-SOURCE
SOFTWARE LIBRARIES
G I T H U B S TA R S
GitHub is a code hosting platform that AI researchers and TensorFlow (developed
developers frequently use to upload, comment on, and
download software. GitHub users can “star” a project by Google and publicly
to save it in their list, thereby expressing their interests
and likes—similar to the “like’’ function on Twitter and
released in 2017) is the
other social media platforms. As AI researchers upload most popular AI software
packages on GitHub that mention the use of an open-
source library, the “star” function on GitHub can be used
library. The second most
to measure the popularity of various AI programming popular library in 2020 is
open-source libraries.
Keras (also developed by
Figure 1.3.1 suggests that TensorFlow (developed by
Google and publicly released in 2017) is the most popular Google and built on top
AI software library. The second most popular library in
of TensorFlow 2.0).
2020 is Keras (also developed by Google and built on top
of TensorFlow 2.0). Excluding TensorFlow, Figure 1.3.2
shows that PyTorch (created by Facebook) is another
library that is becoming increasingly popular.
CHAPTER 1 PREVIEW 39
TA B L E O F C O N T E N T S
CHAPTER 1: 1. 3 A I O P E N -
Artificial Intelligence
RESEARCH & S O U R C E S O F T WA R E
Index Report 2021 D E V E LO P M E N T LIBRARIES
153 TensorFlow
150
Cumulative Github Stars (in Thousand)
100
51 Keras
50 46 PyTorch
45 Sckit-learn
31 BVLC/caffe
19 MXNet
17 Cntk
9 Theano
8 Caffe2
0
2014 2015 2016 2017 2018 2019 2020
Figure 1.3.1
51 Keras
50
46 PyTorch
45 Sckit-learn
Cumulative Github Stars (in Thousand)
40
31 BVLC/caffe
30
19 MXNet
20
17 Cntk
10 9 Theano
8 Caffe2
0
2014 2015 2016 2017 2018 2019 2020
Figure 1.3.2
CHAPTER 1 PREVIEW 40
TA B L E O F C O N T E N T S
Artificial Intelligence
Index Report 2021
CHAPTER 2:
Technical
Performance
Artificial Intelligence
Index Report 2021
CHAPTER 2 PREVIEW 41
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
T E C H N I CA L
Index Report 2021 PERFORMANCE
CHAPTER 2:
Chapter Preview
Overview 43 Face Detection and Recognition 61
Chapter Highlights 44 National Institute of Standards and
Technology (NIST) Face Recognition
Vendor Test (FRVT) 61
COMPUTER VISION 45
2.1 COMPUTER VISION—IMAGE 46
2.3 LANGUAGE 62
Image Classification 46
English Language Understanding Benchmarks 62
ImageNet 46
SuperGLUE 62
ImageNet: Top-1 Accuracy 46
SQuAD 63
ImageNet: Top-5 Accuracy 47
Commercial Machine Translation (MT) 64
ImageNet: Training Time 48
Number of Commercially Available
ImageNet: Training Costs 49
MT Systems 64
Highlight: Harder Tests Beyond ImageNet 50
GPT-3 65
Image Generation 51
STL-10: Fréchet Inception Distance
2.4 LANGUAGE REASONING SKILLS 67
(FID) Score 51
Vision and Language Reasoning 67
FID Versus Real Life 52
Visual Question Answering (VQA) Challenge 67
Deepfake Detection 53
Visual Commonsense Reasoning (VCR) Task 68
Deepfake Detection Challenge (DFDC) 53
Human Pose Estimation 54
2.5 SPEECH 69
Common Objects in Context (COCO):
Keypoint Detection Challenge 54 Speech Recognition 69
Common Objects in Context (COCO): Transcribe Speech: LibriSpeech 69
DensePose Challenge 55 Speaker Recognition: VoxCeleb 69
Semantic Segmentation 56 Highlight: The Race Gap in
Cityscapes 56 Speech Recognition Technology 71
Embodied Vision 57
2.6 REASONING 72
2.2 COMPUTER VISION—VIDEO 58 Boolean Satisfiability Problem 72
Activity Recognition 58 Automated Theorem Proving (ATP) 74
ActivityNet 58
ActivityNet: Temporal Action 2.7 HEALTHCARE AND BIOLOGY 76
Localization Task 58 Molecular Synthesis 76
ActivityNet: Hardest Activity 59 Test Set Accuracy for Forward Chemical
Object Detection 60 Synthesis Planning 76
You Only Look Once (YOLO) 60 COVID-19 and Drug Discovery 77
AlphaFold and Protein Folding 78
CHAPTER 2 PREVIEW 42
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L OV E R V I E W
Index Report 2021 PERFORMANCE
Overview
This chapter highlights the technical progress in various subfields of
AI, including computer vision, language, speech, concept learning, and
theorem proving. It uses a combination of quantitative measurements,
such as common benchmarks and prize challenges, and qualitative
insights from academic papers to showcase the developments in state-of-
the-art AI technologies.
CHAPTER 2 PREVIEW 43
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence CHAPTER
T E C H N I CA L
Index Report 2021 PERFORMANCE HIGHLIGHTS
CHAPTER HIGHLIGHTS
• Generative everything: AI systems can now compose text, audio, and images to a sufficiently high
standard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology. That promises to generate a tremendous
range of downstream applications of AI for both socially useful and less useful purposes. It is
also causing researchers to invest in technologies for detecting generative models; the DeepFake
Detection Challenge data indicates how well computers can distinguish between different outputs.
•T
he industrialization of computer vision: Computer vision has seen immense progress in the past
decade, primarily due to the use of machine learning techniques (specifically deep learning). New
data shows that computer vision is industrializing: Performance is starting to flatten on some of the
largest benchmarks, suggesting that the community needs to develop and agree on harder ones
that further test performance. Meanwhile, companies are investing increasingly large amounts
of computational resources to train computer vision systems at a faster rate than ever before.
Meanwhile, technologies for use in deployed systems—like object-detection frameworks for
analysis of still frames from videos—are maturing rapidly, indicating further AI deployment.
• Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has
yielded AI systems with significantly improved language capabilities that have started to have a
meaningful economic impact on the world. Google and Microsoft have both deployed the BERT
language model into their search engines, while other large language models have been developed
by companies ranging from Microsoft to OpenAI. Progress in NLP has been so swift that technical
advances have started to outpace the benchmarks to test for them. This can be seen in the rapid
emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation suite
developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
• New analyses on reasoning: Most measures of technical problems show for each time point the
performance of the best system at that time on a fixed benchmark. New analyses developed for
the AI Index offer metrics that allow for an evolving benchmark, and for the attribution to individual
systems of credit for a share of the overall performance of a group of systems over time. These
are applied to two symbolic reasoning problems, Automated Theorem Proving and Satisfiability of
Boolean formulas.
• Machine learning is changing the game in healthcare and biology: The landscape of the healthcare
and biology industries has evolved substantially with the adoption of machine learning. DeepMind’s
AlphaFold applied deep learning technique to make a significant breakthrough in the decades-long
biology challenge of protein folding. Scientists use ML models to learn representations of chemical
molecules for more effective chemical synthesis planning. PostEra, an AI startup used ML-based
techniques to accelerate COVID-related drug discovery during the pandemic.
CHAPTER 2 PREVIEW 44
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence COMPUTER
T E C H N I CA L
Index Report 2021 PERFORMANCE VISION
Computer Vision
Introduced in the 1960s, the field of computer vision has seen significant
progress and in recent years has started to reach human levels of
performance on some restricted visual tasks. Common computer
vision tasks include object recognition, pose estimation, and semantic
segmentation. The maturation of computer vision technology has unlocked
a range of applications: self-driving cars, medical image analysis, consumer
applications (e.g., Google Photos), security applications (e.g., surveillance,
satellite imagery analysis), industrial applications (e.g., detecting defective
parts in manufacturing and assembly), and others.
CHAPTER 2 PREVIEW 45
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
1 Though it is worth noting that the human baseline for this metric comes from a single Stanford graduate student who took roughly the same test as the AI systems took.
CHAPTER 2 PREVIEW 46
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
100%
80%
70%
60%
80..
.
Figure 2.1.1
ImageNet: Top-5 Accuracy
Top-5 accuracy asks whether the correct label is in at least the classifier’s top five predictions. Figure 2.1.2 shows that
the error rate has improved from around 85% in 2013 to almost 99% in 2020.2
90%
85%
80%
2 Note: For data on human error, a human was shown 500 images and then was asked to annotate 1,500 test images; their error rate was 5.1% for Top-5 classification. This is a very rough baseline, but it
gives us a sense of human performance on this task.
CHAPTER 2 PREVIEW 47
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
What follows are the results from MLPerf, a competition Distribution of Training Time: MLPerf does not just
run by the MLCommons organization that challenges show the state of the art for each competition period;
entrants to train an ImageNet network using a common it also makes available all the data behind each
(residual network) architecture, and then ranks systems entry in each competition cycle. This, in turn, reveals
according to the absolute “wall clock” time it takes them the distribution of training times for each period
to train a system.3 (Figure 2.1.3). (Note that in each MLPerf competition,
As shown in Figure 2.1.3, the training time on ImageNet competitors typically submit multiple entries that use
has fallen from 6.2 minutes (December 2018) to 47 different permutations of hardware.)
seconds (July 2020). At the same time, the amount of Figure 2.1.4 shows that in the past couple of years,
hardware used to achieve these results has increased training times have shortened, as has the variance
dramatically; frontier systems have been dominated by between MLPerf entries. At the same time, competitors
the use of “accelerator” chips, starting with GPUs in the have started to use larger and larger numbers of
2018 results, and transitioning to Google’s TPUs for the accelerator chips to speed training times. This is in line
best-in-class results from 2019 and 2020. with broader trends in AI development, as large-scale
training becomes better understood, with a higher
degree of shared best practices and infrastructure.
CHAPTER 2 PREVIEW 48
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
IMAGENET: TRAINING TIME and HARDWARE of the BEST IMAGENET: DISTRIBUTION of TRAINING TIME
SYSTEM Source: MLPerf, 2020 | Chart: 2021 AI Index Report
Source: MLPerf, 2020 | Chart: 2021 AI Index Report
10,000
6.2 Min
4,000 6
1,000
5
Number of Accelerators
Training Time
100
4,096
2,000 3
10
2
1.3 Min
1,000
47 Sec 1
1,024 1
640
0 0
12/2018 06/2019 07/2020 2018 2019 2020
2,000
1,000
500
Cost (U.S. Dollars; Log Scale)
200
100
50
20
10 $7.43
2
1
12/2017 06/2018 12/2018 06/2019 12/2019
Figure 2.1.5
CHAPTER 2 PREVIEW 49
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
What is the Time Table for Tracking This Data? As these benchmarks are relatively new, the plan is to
wait a couple of years for the community to test a range of systems against them, which will generate
the temporal information necessary to make graphs tracking progress overtime.
CHAPTER 2 PREVIEW 50
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
I M AG E G E N E R AT I O N
Image generation is the task of generating images STL-10: Fréchet Inception Distance (FID) Score
that look indistinguishable from “real” images. Image One way to measure progress in image generation is via a
generation systems have a variety of uses, ranging from technique called Fréchet Inception Distance (FID), which
augmenting search capabilities (it is easier to search for roughly correlates to the difference between how a given
a specific image if you can generate other images like AI system “thinks” about a synthetic image versus a real
it) to serving as an aid for other generative uses (e.g., image, where a real image has a score of 0 and synthetic
editing images, creating content for specific purposes, images that look similar have scores that approach 0.
generating multiple variations of a single image to help
designers brainstorm, and so on). Figure 2.1.6 shows the progress of generative models
over the past two years at generating convincing
In recent years, image generation progress has synthetic images in the STL-10 dataset, which is designed
accelerated as a consequence of the continued to test how effective systems are at generating images
improvement in deep learning–based algorithms, as well and gleaning other information about them.
as the use of increased computation and larger datasets.
45
40
Fréchet Inception Distance (FID) Score
35
30
25.4
25
20
01/2018 04/2018 07/2018 10/2018 01/2019 04/2019 07/2019 10/2019 01/2020 04/2020 07/2020
Figure 2.1.6
CHAPTER 2 PREVIEW 51
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
2018
2020
Figure 2.1.7
CHAPTER 2 PREVIEW 52
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
D E E P FA K E D E T E CT I O N
Advances in image synthesis have created new Deepfake Detection Challenge (DFDC)
opportunities as well as threats. For instance, in recent Created in September 2019 by Facebook, the Deepfake
years, researchers have harnessed breakthroughs Detection Challenge (DFDC) measures progress on
in synthetic imagery to create AI systems that can deepfake-detection technology. A two-part challenge,
generate synthetic images of human faces, then DFDC asks participants to train and test their models from
superimpose those faces onto the faces of other people a public dataset of around 100,000 clips. The submissions
in photographs or movies. People call this application are scored on log loss, a classification metric based on
of generative technology a “deepfake.” Malicious uses probabilities. A smaller log loss means a more accurate
of deepfakes include misinformation and the creation prediction of deepfake videos. According to Figure
of (predominantly misogynistic) pornography. To try 2.1.8, log loss dropped by around 0.5 as the challenge
to combat this, researchers are developing deepfake- progressed between December 2019 and March 2020.
detection technologies.
0.7
0.6
0.5
Log Loss
0.4
0.3
0.2
0.19
0.1
0.0
1/6/2020 1/16/2020 1/26/2020 2/5/2020 2/15/2020 2/25/2020 3/6/2020 3/16/2020 3/26/2020
Figure 2.1.8
CHAPTER 2 PREVIEW 53
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
90%
80% 80.8%
Average Precision (AP)
70%
60%
50%
07/2016 01/2017 07/2017 01/2018 07/2018 01/2019 07/2019 01/2020 07/2020
Figure 2.1.9
CHAPTER 2 PREVIEW 54
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
75%
72%
70%
Average Precision (AP)
65%
60%
55%
50%
03/2018 05/2018 07/2018 09/2018 11/2018 01/2019 03/2019 05/2019 07/2019 09/2019
Figure 2.1.10
CHAPTER 2 PREVIEW 55
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
S E M A N T I C S E G M E N TAT I O N
Semantic segmentation is the task of classifying each in the urban space, is crucial to the environmental
pixel in an image to a particular label, such as person, perception of autonomous vehicles. Cityscapes is useful
cat, etc. Where image classification tries to assign a for training deep neural networks to understand the urban
label to the entire image, semantic segmentation tries to environment.
isolate the distinct entities and objects in a given image,
One Cityscapes task that focuses on semantic
allowing for more fine-grained identification. Semantic
segmentation is the pixel-level semantic labeling task.
segmentation is a basic input technology for self-driving
This task requires an algorithm to predict the per-pixel
cars (identifying and isolating objects on roads), image
semantic labeling of the image, partitioning an image
analysis, medical applications, and more.
into different categories, like cars, buses, people, trees,
Cityscapes and roads. Participants are evaluated based on the
Cityscapes is a large-scale dataset of diverse urban street intersection-over-union (IoU) metric. A higher IoU score
scenes across 50 different cities recorded during the means a better segmentation accuracy. Between 2014 and
daytime over several months (during spring, summer, and 2020, the mean IoU increased by 35% (Figure 2.1.11). There
fall) of the year. The dataset contains 5,000 images with was a significant boost to progress in 2016 and 2017 when
high-quality, pixel-level annotations and 20,000 weekly people started using residual networks in these systems.
labeled ones. Semantic scene understanding, especially
90%
75%
70%
65%
60%
01/2015 01/2016 01/2017 01/2018 01/2019 01/2020
Figure 2.1.11
CHAPTER 2 PREVIEW 56
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .1 C O M P U T E R
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N — I M AG E
EMBODIED VISION
The performance data so far shows that computer vision • Thor (AI2, 2017) focuses on sequential abstract
systems have advanced tremendously in recent years. reasoning with predefined “magic” actions that are
Object recognition, semantic segmentation, and human applicable to objects.
pose estimation, among others, have now achieved
• Gibson (Stanford, 2018) focuses on visual navigation
significant levels of performance. Note that these visual
in photorealistic environments obtained with 3D
tasks are somewhat passive or disembodied. That
scanners.
is, they can operate on images or videos taken from
camera systems that are not physically able to interact • iGibson (Stanford, 2019) focuses on full interactivity
with the surrounding environment. As a consequence in large realistic scenes mapped from real houses and
of the continuous improvement in those passive tasks, made actable: navigation + manipulation (known in
researchers have now started to develop more advanced robotics as “mobile manipulation”).
AI systems that can be interactive or embodied—that
• AI Habitat (Facebook, 2019) focuses on visual
is, systems that can physically interact with and modify
navigation with an emphasis on much faster
the surrounding environment in which they operate: for
execution, enabling more computationally expensive
example, a robot that can visually survey a new building
approaches.
and autonomously navigate it, or a robot that can learn
to assemble pieces by watching visual demonstrations • ThreeDWorld (MIT and Stanford, 2020) focuses on
instead of being manually programmed for this. photorealistic environments through game engines,
as well as adds simulation of flexible materials, fluids,
Progress in this area is currently driven by the
and sounds.
development of sophisticated simulation environments,
where researchers can deploy robots in virtual spaces, • SEAN-EP (Yale, 2020) is a human-robot interaction
simulate what their cameras would see and capture, and environment with simulated virtual humans that
develop AI algorithms for navigation, object search, and enables the collection of remote demonstrations from
object grasping, among other interactive tasks. Because humans via a web browser.
of the relatively early nature of this field, there are few
• Robosuite (Stanford and UT Austin, 2020) is a modular
standardized metrics to measure progress. Instead, here
simulation framework and benchmark for robot
are brief highlights of some of the available simulators,
learning.
their year of release, and any other significant feature.
CHAPTER 2 PREVIEW 57
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2.2 COMPUTER
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N —V I D E O
Video analysis is the task of making inferences over sequential image frames, sometimes with the inclusion of an audio feed.
Though many AI tasks rely on single-image inferences, a growing body of applications require computer vision machines to reason
about videos. For instance, identifying a specific dance move benefits from seeing a variety of frames connected in a temporal
sequence; the same is true of making inferences about an individual seen moving through a crowd, or a machine carrying out a
sequence of movements over time.
50%
42.8%
40%
Mean Average Precision (mAP)
30%
20%
10%
0%
2016 2017 2018 2019 2020
Figure 2.2.1
CHAPTER 2 PREVIEW 58
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2.2 COMPUTER
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N —V I D E O
Polishing furniture
Putting in contact
lenses
Removing curlers
Rock-paper-scissors
Running a marathon
Shot put
Smoking a cigarette
Throwing darts
CHAPTER 2 PREVIEW 59
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2.2 COMPUTER
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N —V I D E O
O B J E C T D E T E CT I O N
Object detection is the task of identifying a given object since it was first published in 2015. Over time, YOLO has
in an image. Frequently, image classification and image been optimized along two constraints: performance and
detection are coupled together in deployed systems. inference latency, as shown in Figure 2.2.3. What this
One way to get a proxy measure for the improvement means, specifically, is that by measuring YOLO, one can
in deployed object recognition systems is to study the measure the advancement of systems that might not
advancement of widely used object detection systems. have the best absolute performance but are designed
around real-world needs, like low-latency inference
You Only Look Once (YOLO)
over video streams. Therefore, YOLO systems might
You Only Look Once (YOLO) is a widely used open source
not always contain the absolute best performance as
system for object detection, so its progress has been
defined in the research literature, but they will represent
included on a standard task on YOLO variants to give a
good performance when faced with trade-offs such as
sense of how research percolates into widely used open
inference time.
source tools. YOLO has gone through multiple iterations
80
70
65.2
Mean Average Precision (mAP50)
YOLOv4 PP-YOLO
60 (Resolution: 608) (Resolution: 608)
YOLOv3
(Resolution: 608)
50
YOLOv2
40 (Resolution: Unclear)
30
20
12/2016 04/2018 04/2020 11/2020
Figure 2.2.3
CHAPTER 2 PREVIEW 60
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2.2 COMPUTER
T E C H N I CA L
Index Report 2021 PERFORMANCE V I S I O N —V I D E O
FAC E D E T E CT I O N A N D
RECOGNITION
Facial detection and recognition is one of the use-cases recognition technologies used for a wide range of civil
for AI that has a sizable commercial market and has and governmental tasks (primarily in law enforcement
generated significant interest from governments and and homeland security), including verification of visa
militaries. Therefore, progress in this category gives photos, mug shot images, and child abuse images.
us a sense of the rate of advancement in economically
Figure 2.2.4 shows the results of the top-performing 1:1
significant parts of AI development.
algorithms measured on false non-match rate (FNMR)
National Institute of Standards and Technology across several different datasets. FNMR refers to the
(NIST) Face Recognition Vendor Test (FRVT) rate at which the algorithm fails when attempting to
The Face Recognition Vendor Tests (FRVT) by the match the image with the individual. Facial recognition
National Institute of Standards and Technology (NIST) technologies on mug-shot-type and visa photos have
provide independent evaluations of commercially improved the most significantly in the past four years,
available and prototype face recognition technologies. falling from error rates of close to 50% to a fraction of a
FRVT measures the performance of automated face percent in 2020.4
0.5
False Non-Match rate (FNMR; Log Scale)
0.2
0.1
0.05
0.02
0.01
0.0064, BORDER Photos FNMR @ FMR = 0.000001
0.005
0.0035, VISABORDER Photos FNMR@FMR ≤ 0.000001
0.0025, VISA Photos FNMR @ FMR ≤ 0.000001
0.0023, MUGSHOT Photos FNMR @ FMR ≤ 0.00001 DT>=12 YRS
0.002 0.0022, MUGSHOT Photos FNMR @ FMR ≤ 0.00001
Figure 2.2.4
4 You can view details and examples of various datasets on periodically updated FRVT 1:1 verification reports.
CHAPTER 2 PREVIEW 61
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 . 3 L A N G UAG E
Index Report 2021 PERFORMANCE
Natural language processing (NLP) involves teaching machines to interpret, classify, manipulate, and generate language.
From the early use of handwritten rules and statistical techniques to the recent adoption of generative models and deep
learning, NLP has become an integral part of our lives, with applications in text generation, machine translation, question
answering, and other tasks.
2.3 LANGUAGE
In recent years, advances in natural language processing a series of language understanding tasks on established
technology have led to significant changes in large-scale datasets. SuperGLUE replaced the prior GLUE benchmark
systems that billions of people access. For instance, in (introduced in 2018) with more challenging and diverse tasks.
late 2019, Google started to deploy its BERT algorithm
The SuperGLUE score is calculated by averaging scores on
into its search engine, leading to what the company said
a set of tasks. Microsoft’s DeBERTa model now tops the
was a significant improvement in its in-house quality
SuperGLUE leaderboard, with a score of 90.3, compared
metrics. Microsoft followed suit, announcing later in 2019
with an average score of 89.8 for SuperGLUE’s “human
that it was using BERT to augment its Bing search engine.
baselines.” This does not mean that AI systems have
E N G L I S H L A N G UAG E surpassed human performance on all SuperGLUE tasks, but
U N D E R S TA N D I N G B E N C H M A R KS it does mean that the average performance across the entire
suite has exceeded that of a human baseline. The rapid pace
SuperGLUE of progress (Figure 2.3.1) suggests that SuperGLUE may
Launched in May 2019, SuperGLUE is a single-metric need to be made more challenging or replaced by harder
benchmark that evaluates the performance of a model on tests in the future, just as SuperGLUE replaced GLUE.
SUPERGLUE BENCHMARK
Source: SuperGLUE Leaderboard, 2020 | Chart: 2021 AI Index Report
100
80
Score
70
60
50
Figure 2.3.1
CHAPTER 2 PREVIEW 62
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 . 3 L A N G UAG E
Index Report 2021 PERFORMANCE
SQuAD
The Stanford Question Answering Dataset, or SQuAD, the participants (mirroring the trend seen in GLUE and
is a reading-comprehension benchmark that measures SuperGLUE). SQuAD 2.0 combines the 100,000 questions
how accurately a NLP model can provide short answers in SQuAD 1.1 with over 50,000 unanswerable questions
to a series of questions pertaining to a small article written by crowdworkers to resemble answerable ones.
of text. The SQuAD test makers established a human The objective is to test how well systems can answer
performance benchmark by having a group of people questions and to determine when systems know that no
read Wikipedia articles on a variety of topics and then answer exists.
answer multiple-choice questions about those articles.
As Figure 2.3.2 shows, the F1 score for SQuAD 1.1
Models are given the same task and are evaluated on
improved from 67.75 in August 2016 to surpass human
the F1 score, or the average overlap between the model
performance of 91.22 in September 2018—a 25-month
prediction and the correct answer. Higher scores indicate
period—whereas SQuAD 2.0 took just 10 months to beat
better performance.
human performance (from 66.3 in May 2018 to 89.47
Two years after the introduction of the original SQuAD, in March 2019). In 2020, the most advanced models of
in 2016, SQuAD 2.0 was developed once the initial SQuAD 1.1 and SQuAD 2.0 reached the F1 scores of 95.38
benchmark revealed increasingly fast performances by and 93.01, respectively.
100
95.4 SQuAD 1.1
93.0 SQuAD 2.0
91.2 Human 1.1
80
70
60
07/2016 01/2017 07/2017 01/2018 07/2018 01/2019 07/2019 01/2020
Figure 2.3.2
CHAPTER 2 PREVIEW 63
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 . 3 L A N G UAG E
Index Report 2021 PERFORMANCE
Preview
25 Commercial
20
15
10
0
05/2017 07/2017 11/2017 03/2018 07/2018 12/2018 06/2019 11/2019 07/2020
0
Figure 2.3.3
CHAPTER 2 PREVIEW 64
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 . 3 L A N G UAG E
Index Report 2021 PERFORMANCE
G P T-3
In July 2020, OpenAI unveiled GPT-3, the largest known Figure 2.3.4, adapted from the GPT-3 paper,
dense language model. GPT-3 has 175 billion parameters demonstrates the impact of scale (in terms of model
and was trained on 570 gigabytes of text. For comparison, parameters) on task accuracy (higher is better) in zero-,
its predecessor, GPT-2, was over 100 times smaller, at one-, and few-shot learning regimes. Each point on the
1.5 billion parameters. This increase in scale leads to curve corresponds to an average performance accuracy,
surprising behavior: GPT-3 is able to perform tasks it aggregated across 42 accuracy-oriented benchmarks. As
was not explicitly trained on with zero to few training model size increases, average accuracy in all task regimes
examples (referred to as zero-shot and few-shot learning, increases accordingly. Few-shot learning accuracy
respectively). This behavior was mostly absent in the increases more rapidly with scale, compared with zero-
much smaller GPT-2. Furthermore, for some tasks (but shot learning, which suggests that large models can
not all; e.g., SuperGLUE and SQuAD2), GPT-3 outperforms perform surprisingly well given minimal context.
state-of-the-art models that were explicitly trained to
solve those tasks with far more training examples.
60% 57.4%
Few-Shot
50% 51.0%
One-Shot
40% 42.6%
Zero-Shot
Accuracy
30%
20%
10%
0%
1 10 100
Number of Parameters (Billions)
Figure 2.3.4
CHAPTER 2 PREVIEW 65
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 . 3 L A N G UAG E
Index Report 2021 PERFORMANCE
CHAPTER 2 PREVIEW 66
TA B L E O F C O N T E N T S
CHAPTER 2: 2 .4 L A N G UAG E
Artificial Intelligence
T E C H N I CA L REASONING
Index Report 2021 PERFORMANCE SKILLS
76.4% Accuracy
75%
Accuracy
70%
65%
60%
55%
10/2015 04/2016 10/2016 04/2017 10/2017 04/2018 10/2018 04/2019 10/2019 04/2020
Figure 2.4.1
CHAPTER 2 PREVIEW 67
TA B L E O F C O N T E N T S
CHAPTER 2: 2 .4 L A N G UAG E
Artificial Intelligence
T E C H N I CA L REASONING
Index Report 2021 PERFORMANCE SKILLS
100
85 Human Performance
80
70.5
Q->AR Score
60
40
20
11/2018 01/2019 03/2019 05/2019 07/2019 09/2019 11/2019 01/2020 03/2020 05/2020 07/2020
Figure 2.4.2
CHAPTER 2 PREVIEW 68
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2.5 SPEECH
Index Report 2021 PERFORMANCE
A major aspect of AI research is the analysis and synthesis of human speech conveyed via audio data. In recent years,
machine learning approaches have drastically improved performance across a range of tasks.
2.5 SPEECH
SPEECH RECOGNITION
Speech recognition, or automatic speech recognition (ASR), shifting from an absolute performance difference of more
is the process that enables machines to recognize spoken than seven points in late 2015 to a difference of less than
words and convert them to text. Since IBM introduced its one point in 2020. This reveals dramatic improvements in
first speech recognition technology in 1962, the technology the robustness of ASR systems over time and suggests that
has evolved with voice-driven applications such as we might be saturating performance on LibriSpeech—in
Amazon Alexa, Google Home, and Apple Siri becoming other words, harder tests may be needed.
increasingly prevalent. The flexibility and predictive power
of deep neural networks, in particular, has allowed speech Speaker Recognition: VoxCeleb
recognition to become more accessible. Speaker identification tests how well machine learning
systems can attribute speech to a particular person. The
Transcribe Speech: LibriSpeech VoxCeleb dataset, first introduced in 2017, contains over
LibriSpeech is a dataset, first introduced in 2015, made up a million utterances from 6,000 distinct speakers, and its
of 1,000 hours of speech from audiobooks. It has become associated speaker-identification task tests the error rate
widely used for the development and testing of speech for systems that try to attribute a particular utterance to
recognition technologies. In recent years, neural-network- a particular speaker. A better (lower) score in VoxCeleb
based AI systems have started to dramatically improve provides a proxy for how well a machine can distinguish
performance on LibriSpeech, lowering the word error rate one voice among 6,000. Evaluation method for VoxCeleb is
(WER; 0% is optimal performance) to around 2% (Figure Equal Error Rate (EER), a commonly used metric for identity
2.5.1a and Figure 2.5.1b). verification systems. EER provides a measure for both the
Developers can test out their systems on LibriSpeech in false positive rate (assigning a label incorrectly) and the
two ways: false negative rate (failing to assign a correct label).
• Test Clean determines how well their systems can In recent years, progress on this task has come from
transcribe speech from a higher-quality subset of the hybrid systems—systems that fuse contemporary deep
LibriSpeech dataset. This test gives clues about how learning approaches with more structured algorithms,
well AI systems might perform in more controlled developed by the broader speech-processing community.
environments. As of 2020, error rates have dropped such that computers
have a very high (99.4%) ability to attribute utterances to
• Test Other determines how systems can deal with a given speaker (Figure 2.5.2)
lower-quality parts of the LibriSpeech dataset. This
test suggests how well AI systems might perform in Still, obstacles remain: These systems face challenges
noisier (and perhaps more realistic) environments. processing speakers with different accents and in
differentiating among speakers when confronted with a
There has been substantial progress recently on both large dataset (it is harder to identify one person in a set
datasets, with an important trend emerging in the past two of a billion people than to pick out one person across the
years: The gap between performance on Test Clean and Test VoxCeleb training set of 6,000).
Other has started to close significantly for frontier systems,
CHAPTER 2 PREVIEW 69
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2.5 SPEECH
Index Report 2021 PERFORMANCE
LIBRISPEECH: WORD ERROR RATE, TEST CLEAN LIBRISPEECH: WORD ERROR RATE, TEST OTHER
Source: Papers with Code, 2020 | Chart: 2021 AI Index Report Source: Papers with Code, 2020 | Chart: 2021 AI Index Report
6 14
12
5
10
Word Error Rate (WER)
8
3
6
2
4
1.4
1 2.6
2
0 0
01/2016 01/2017 01/2018 01/2019 01/2020 01/2016 01/2017 01/2018 01/2019 01/2020
8%
6%
Equal Error Rate (EER)
4%
2%
0.6%
0%
2017 2018 2019 2020
Figure 2.5.2
CHAPTER 2 PREVIEW 70
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2.5 SPEECH
Index Report 2021 PERFORMANCE
TESTINGS on LEADIN
The Race Gap in TESTINGS on LEADING SPEECH-to-TEXT SERVICES:
WORD ERROR RATE by RACE and GENDER, 2019 WORD ERROR RATE b
Speech Recognition Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report Source: Koenecke et al., 2020 | Char
Technology 40%
25%
Researchers from Stanford University
Av..
Figure 2.5.3
through leading speech-to-text
services by Amazon, Apple, Google, TESTINGS on LEADING SPEECH-to-TEXT SERVICES:
TESTINGS on LEADING SPEECH-to-TEXT SERVICES:
WORD ERRORIBM, and
RATE byMicrosoft.
RACE and GENDER, 2019 WORD ERROR RATE by SERVICE and RACE, 2019
Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report
The results suggest that, on average,
systems made 19 errors every
40% Black Speakers
hundred words for white speakers 25%
White Speakers
and 35 errors for Black speakers—
Average Word Error Rate (WER)
Average Word Error Rate (WER)
Figure 2.5.4
CHAPTER 2 PREVIEW 71
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 .6 R E A S O N I N G
Index Report 2021 PERFORMANCE
This section measures progress on symbolic (or logical) reasoning in AI, which is the process of drawing conclusions from
sets of assumptions. We consider two major reasoning problems, Boolean Satisfiability (SAT) and Automated Theorem
Proving (ATP). Each has real-world applications (e.g., circuit design, scheduling, software verification, etc.) and poses
significant measurement challenges. The SAT analysis shows how to assign credit for the overall improvement in the field to
individual systems over time. The ATP analysis shows how to measure performance given an evolving test set.
All analyses below are original to this report. Lars Kotthoff wrote the text and performed the analysis for the SAT section.
Geoff Sutcliffe, Christian Suttner, and Raymond Perrault wrote the text and performed the analysis for the ATP section. This
work had not been published at the time of writing; consequently, a more academically rigorous version of this section (with
references, more precise details, and further context) is included in the Appendix.
2.6 REASONING
BOOLEAN SATISFIABILIT Y PROBLEM
Analysis and text by Lars Kotthoff in all previous years, and the performance of the median-
ranked solver in 2020 is almost on par with the top-ranked
The SAT problem considers whether there is an
solver in 2019.
assignment of values to a set of Boolean variables, joined
by logical connectives, that makes the logical formula it Performance improvements in SAT—and more generally,
represents true. Many real-world problems, such as circuit hard computational AI problems—come primarily
design, automated theorem proving, and scheduling, can from two areas of algorithmic improvements: novel
be represented and solved efficiently as SAT problems. techniques and more efficient implementations of existing
techniques. Typically, performance improvements arise
The performance of the top-, median-, and bottom-ranked
primarily from novel techniques. However, more efficient
SAT solvers was examined from each of the last five years
implementations (which can arise with performance
(2016–2020) of the SAT Competition, which has been
improvements in hardware over time) can also increase
running for almost 20 years, to measure a snapshot of
performance. Therefore, it is difficult to assess whether
state-of-the-art performance. In particular, all 15 solvers
performance improvements arise primarily from novel
were run on all 400 SAT instances from the main track of
techniques or more efficient implementations. To address
the 2020 competition and the time (in CPU seconds) it took
this problem, the temporal Shapley value, which is the
to solve all instances was measured.5 Critically, each solver
contribution of an individual system to state-of-the-art
was run on the same hardware, such that comparisons
performance over time, was measured (see the Appendix
across years would not be confounded by improvements
for more details).
in hardware efficiency over time.
Figure 2.6.2 shows the temporal Shapley value
While performance of the best solvers from 2016 to
contributions of each solver for the different competition
2018 did not change significantly, large improvements
years. Note that the contributions of the solvers in 2016
are evident in 2019 and 2020 (Figure 2.6.1). These
are highest because there is no previous state-of-the-art
improvements affect not only the best solvers but also
to compare them with in our evaluation and that their
their competitors. The performance of the median-ranked
contribution is not discounted.
solver in 2019 is better than that of the top-ranked solvers
5 Acknowledgments: The Advanced Research Computing Center at the University of Wyoming provided resources for gathering the computational data. Austin Stephen performed the computational
experiments.
CHAPTER 2 PREVIEW 72
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 .6 R E A S O N I N G
Index Report 2021 PERFORMANCE
TOTAL TIME to SOLVE ALL 400 INSTANCES for EACH SOLVER and YEAR (LOWER IS BETTER), 2016-20
Source: Kotthoff, 2020 | Chart: 2021 AI Index Report
2.00e+06
YalSAT 03r YalSAT PauSat
1.75e+06 CCAnrSim
1.50e+06
Total Time (CPUs)
MapleLCMDistChronoBT
glu_vc
1.20e+06 glue_alt
Candy
smallsat
expMC_VSIDS_LRB_Switch_2500
1.00e+06 MapleCOMSPS_DRUP Maple_LCM_Dist
Maple_mix
MapleLCMDiscChronoBT-DL-v3
8.00e+05
7.00e+05
Kissat-sc2020-sat
Figure 2.6.1
TEMPORAL SHAPLEY VALUE CONTRIBUTIONS of INDIVIDUAL SOLVERS to the STATE of the ART OVER TIME (HIGHER
IS BETTER), 2016-20
Source: Kotthoff, 2020 | Chart: 2021 AI Index Report
1.00e+06 MapleCOMSPS_DRUP
glue_alt
CCAnrSim
expMC_VSIDS_LRB_Switch_2500
Temporal Shapley Value
Maple_LCM_Dist
glu_vc MapleLCMDiscChronoBT-DL-v3
MapleLCMDistChronoBT
1.00e+04
PauSat
Maple_mix
1.00e+03
1.00e+02
YalSAT
Figure 2.6.2
CHAPTER 2 PREVIEW 73
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 .6 R E A S O N I N G
Index Report 2021 PERFORMANCE
According to the temporal Shapley value, in 2020 the best AUTOMATED THEOREM PROVING
solver contributes significantly more than the median- (ATP)
and bottom-ranked solvers do. The 2020 winner, Kissat, Analysis and text by Christian Suttner, Geoff Sutcliffe, and
has the highest temporal Shapley value of any solvers Raymond Perrault
excluding the first year. The changes it incorporates,
compared with those of previous solvers, are almost Automated Theorem Proving (ATP) concerns the
exclusively more efficient data structures and algorithms; development and use of systems that automate sound
Kissat thus impressively demonstrates the impact of reasoning, or the derivation of conclusions that follow
good engineering on the state-of-the-art performance. inevitably from facts. ATP systems are at the heart
of many computational tasks, including software
By contrast, smallsat, the solver with the largest verification. The TPTP problem library was used to
temporal Shapley value (but not the winner) in 2019, evaluate the performance of ATP algorithms from 1997 to
focuses on improved heuristics instead of a more efficient 2020 and to measure the fraction of problems solved by
implementation. The same is true of Candy, the solver any system over time (see the Appendix for more details).
with the largest temporal Shapley value in 2017, whose
main novelty is to analyze the structure of a SAT instance The analysis extends to the whole TPTP (over 23,000
and apply heuristics based on this analysis. Interestingly, problems) in addition to four salient subsets (each
neither solver ranked first in their respective years; both ranging between 500 and 5,500 problems)—clause
were outperformed by versions of the Maple solver, normal form (CNF), first-order form (FOF), monomorphic
which nevertheless contributes less to the state of the typed first-order form (TF0) with arithmetic, and
art. This indicates that incremental improvements, monomorphic typed higher-order form (TH0) theorems—
while not necessarily exciting, are important for good all including the use of the equality operator.
performance in practice. Figure 2.6.3 shows that the fraction of problems solved
Based on our limited analysis of the field, novel climbs consistently, indicating progress in the field. The
techniques and more efficient implementations have noticeable progress from 2008 to 2013 included strong
made equally important contributions to the state progress in the FOF, TF0, and TH0 subsets. In FOF, which
of the art in SAT solving. Incremental improvements has been used in many domains (e.g., mathematics,
of established solvers are as likely to result in top real-world knowledge, software verification), there were
performance as more substantial improvements of significant improvements in the Vampire, E, and iProver
solvers without a long track record. systems. In TF0 (primarily used for solving problems in
mathematics and computer science) and TH0 (useful in
subtle and complex topics such as philosophy and logic),
there was rapid initial progress as systems developed
techniques that solved “low-hanging fruit” problems. In
2014–2015, there was another burst of progress in TF0,
as the Vampire system became capable of processing
TF0 problems. It is noteworthy that, since 2015, progress
has continued but slowed, with no indication of rapid
advances or breakthroughs in the last few years.
CHAPTER 2 PREVIEW 74
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence
T E C H N I CA L 2 .6 R E A S O N I N G
Index Report 2021 PERFORMANCE
80%
49.2% All
40%
20%
0%
2000
2004
2006
2009
2008
2003
2005
2002
2020
2024
2007
2023
2022
2001
2010
1997
1998
1999
2011
2012
2013
2014
2015
2016
2017
2018
2019
2021
Figure 2.6.3
While this analysis demonstrates progress in ATP, there in the 1960s, while the former has become increasingly
is obviously room for much more. Two keys to solve important as large bodies of knowledge are encoded
ATP problems are axiom selection (given a large set of for ATP. In the last decade, there has been growing use
axioms, only some of which are needed for a proof of of machine learning approaches to addressing these
the conjecture, how to select an adequate subset of two key challenges (e.g., in the MaLARea and Enigma
the axioms); and search choice (at each stage of an ATP ATP systems). Recent results from the CADE ATP System
system’s search for a solution, which logical formula(e) Competition (CASC) have shown that the emergence of
should be selected for attention). The latter issue has machine learning is a potential game-changer for ATP.
been at the forefront of ATP research since its inception
CHAPTER 2 PREVIEW 75
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .7 H E A LT H CA R E
T E C H N I CA L
Index Report 2021 PERFORMANCE
A N D B I O LO GY
50%
25%
0%
12/2017 06/2018 11/2018 08/2019 11/2020
Human Prediction Benchmark Standard 500k Benchmark Figure 2.7.1
CHAPTER 2 PREVIEW 76
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .7 H E A LT H CA R E
T E C H N I CA L
Index Report 2021 PERFORMANCE
A N D B I O LO GY
15,545
15,000
Total Number of Moonshot Submissions
10,000
5,000
0
03/2020 04/2020 05/2020 06/2020 07/2020 08/2020 09/2020 10/2020 11/2020 12/2020
Figure 2.7.2
CHAPTER 2 PREVIEW 77
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence 2 .7 H E A LT H CA R E
T E C H N I CA L
Index Report 2021 PERFORMANCE
A N D B I O LO GY
100
ALPHAFOLD 2
80
Global Distance Test (GDT_TS)
ALPHAFOLD
60
40
20
0
CASP7 (2006) CASP8 (2008) CASP9 (2010) CASP10 (2012) CASP11 (2014) CASP12 (2016) CASP13 (2018) CASP14 (2020)
Figure 2.7.3
7 Currently most protein folding algorithms leverage multiple sequence alignments—many copies of a protein sequence representing the same protein across evolution—rather than just a
single sequence.
CHAPTER 2 PREVIEW 78
TA B L E O F C O N T E N T S
CHAPTER 2:
Artificial Intelligence EXPERT
T E C H N I CA L
Index Report 2021 PERFORMANCE
HIGHLIGHTS
EXPERT HIGHLIGHTS
This year, the AI Index asked AI experts to share their thoughts on the most significant technical AI breakthroughs
in 2020. Here’s a summary of their responses, along with a couple of individual highlights.
What aspect of AI technical progress, deployment, and development are you most excited to
see in 2021?
• “ It’s interesting to note the dominance of the Transformers architecture, which started for machine
translation but has become the de facto neural network architecture. More broadly, whereas NLP trailed
vision in terms of adoption of deep learning, now it seems like advances in NLP are also driving vision.” —
Percy Liang, Stanford University
• “ The incredible recent advancements in language generation have had a profound effect on the fields of NLP
and machine learning, rendering formerly difficult research challenges and datasets suddenly useless while
simultaneously encouraging new research efforts into the fascinating emergent capabilities (and important
failings) of these complex new models.” —Carissa Schoenick, Allen Institute of AI Research
CHAPTER 2 PREVIEW 79
TA B L E O F C O N T E N T S
Artificial Intelligence
Index Report 2021
CHAPTER 3:
The Economy
Artificial Intelligence
Index Report 2021
CHAPTER 3 PREVIEW 80
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
Index Report 2021 THE ECONOMY
CHAPTER 3:
Chapter Preview
Overview 82 3.3 CORPORATE ACTIVITY 98
Chapter Highlights 83 Industry Adoption 98
Global Adoption of AI 98
3.1 JOBS 84 AI Adoption by Industry and Function 99
AI Hiring 84 Type of AI Capabilities Adopted 99
AI Labor Demand 86 Consideration and Mitigation of
Risks from Adopting AI 101
Global AI Labor Demand 86
The Effect of COVID-19 103
U.S. AI Labor Demand: By Skill Cluster 87
Industrial Robot Installations 104
U.S. Labor Demand: By Industry 88
Global Trend 104
U.S. Labor Demand: By State 90
Regional Comparison 105
AI Skill Penetration 91
Earnings Calls 106
Global Comparison 91
Global Comparison: By Industry 92
3.2 INVESTMENT 93
Corporate Investment 93
Startup Activity 94
Global Trend 94
Regional Comparison 95
Focus Area Analysis 97
CHAPTER 3 PREVIEW 81
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
OV E R V I E W
Index Report 2021 THE ECONOMY
Overview
The rise of artificial intelligence (AI) inevitably raises the question of how much
the technologies will impact businesses, labor, and the economy more generally.
Considering the recent progress and numerous breakthroughs in AI, the field offers
substantial benefits and opportunities for businesses, from increasing productivity
gains with automation to tailoring products to consumers using algorithms,
analyzing data at scale, and more.
This chapter looks at the increasingly intertwined relationship between AI and the
global economy from the perspective of jobs, investment, and corporate activity.
It first analyzes the worldwide demand for AI talent using data on hiring rates
and skill penetration rates from LinkedIn as well as AI job postings from Burning
Glass Technologies. It then looks at trends in private AI investment using statistics
from S&P Capital IQ (CapIQ), Crunchbase, and Quid. The third, final section
analyzes trends in the adoption of AI capabilities across companies, trends in robot
installations across countries, and mentions of AI in corporate earnings, drawing
from McKinsey’s Global Survey on AI, the International Federation of Robotics
(IFR), and Prattle, respectively.
CHAPTER 3 PREVIEW 82
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3: CHAPTER
Index Report 2021 THE ECONOMY HIGHLIGHTS
CHAPTER HIGHLIGHTS
• “Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI
investment in 2020, with more than USD 13.8 billion, 4.5 times higher than 2019.
• Brazil, India, Canada, Singapore, and South Africa are the countries with the highest
growth in AI hiring from 2016 to 2020. Despite the COVID-19 pandemic, the AI hiring
continued to grow across sample countries in 2020.
• More private investment in AI is being funneled into fewer startups. Despite the pandemic,
2020 saw a 9.3% increase in the amount of private AI investment from 2019—a higher
percentage increase than in 2019 (5.7%), though the number of newly funded companies
decreased for the third year in a row.
• Despite growing calls to address ethical concerns associated with using AI, efforts to
address these concerns in the industry are limited, according to a McKinsey survey. For
example, issues such as equity and fairness in AI continue to receive comparatively little
attention from companies. Moreover, fewer companies in 2020 view personal or individual
privacy risks as relevant, compared with in 2019, and there was no change in the percentage
of respondents whose companies are taking steps to mitigate these particular risks.
• Despite the economic downturn caused by the pandemic, half the respondents in a
McKinsey survey said that the coronavirus had no effect on their investment in AI, while
27% actually reported increasing their investment. Less than a fourth of businesses
decreased their investment in AI.
• The United States recorded a decrease in its share of AI job postings from 2019 to 2020—
the first drop in six years. The total number of AI jobs posted in the United States also
decreased by 8.2%, from 325,724 in 2019 to 300,999 in 2020.
CHAPTER 3 PREVIEW 83
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
Attracting and retaining skilled AI talent is challenging. This section examines the latest trend in AI hiring, labor demand,
and skill penetration, with data from LinkedIn and Burning Glass.
3.1 JOBS
AI HIRING
How rapidly is the growth of AI jobs in different countries? This data suggests that the hiring rate has been
This section first looks at LinkedIn data that gives the increasing across all sample countries in 2020. Brazil,
AI hiring rate for different countries. The AI hiring rate India, Canada, Singapore, and South Africa are the
is calculated as the number of LinkedIn members who countries with the highest growth in AI hiring from 2016
include AI skills on their profile or work in AI-related to 2020 (Figure 3.1.1). Across the 14 countries analyzed,
occupations and who added a new employer in the same the AI hiring rate in 2020 was 2.2 times higher, on
month their new job began, divided by the total number of average, than that in 2016. For the top country, Brazil,
LinkedIn members in the country. This rate is then indexed the hiring index grew by more than 3.5 times. Moreover,
to the average month in 2016; for example, an index of 1.05 despite the COVID-19 pandemic, AI hiring continued its
in December 2020 points to a hiring rate that is 5% higher growth across the 14 sampled countries in 2020 (Figure
than the average month in 2016. LinkedIn makes month- 3.1.2).
to-month comparisons to account for any potential lags For more explorations of cross-country comparisons, see
in members updating their profiles. The index for a year is the AI Index Global AI Vibrancy Tool.
the average index over all months within that year.
Brazil
India
Canada
Singapore
South Africa
Germany
Australia
United States
Argentina
United Kingdom
Turkey
Italy
France
China
0 1 2 3
AI Hiring Index Figure 3.1.1
1 Countries included are a sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were also included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as in other countries, and should be interpreted accordingly.
CHAPTER 3 PREVIEW 84
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
2
1
0
3 2.3 South Africa 2.2 Germany 2.1 Australia 2.1 United States
2
1
AI Hiring Index
0
3
2.0 Argentina 1.8 United Kingdom 1.7 Italy
1.8 Turkey
2
1
0
3
2 1.6 France
1.3 China
1
0
CHAPTER 3 PREVIEW 85
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
AI LABOR DEMAND
This section analyzes the AI labor demand based on grown significantly in the last seven years (Figure 3.1.3).
data from Burning Glass, an analytics firm that collects On average, the share of AI job postings among all job
postings from over 45,000 online job sites. To develop postings in 2020 is more than five times larger than in
a comprehensive, real-time portrait of labor market 2013. Of the six countries, Singapore exhibits the largest
demand, Burning Glass aggregated job postings, growth, as its percentage of AI job postings across all job
removed duplicates, and extracted data from job posting roles in 2020 is 13.5 times larger than in 2013.
text. Note that Burning Glass updated the data coverage The United States is the only country among the six that
in 2020 with more job sites; as a result, the numbers in recorded a decrease in its share of AI job postings from
this report should not be directly compared with data in 2019 to 2020—the first drop in six years. This may be due
the 2019 report. to the coronavirus pandemic or the country’s relatively
Global AI Labor Demand more mature AI labor market. The total number of AI jobs
Demand for AI labor in six countries covered by Burning posted in the United States also decreased by 8.2%, from
Glass data—the United States, the United Kingdom, 325,724 in 2019 to 300,999 in 2020.
Canada, Australia, New Zealand, and Singapore—has
2.0%
1.5%
1.0%
0.8% United States
0.8% United Kingdom
0.7% Canada
0.5% 0.5% Australia
0.0%
2013 2014 2015 2016 2017 2018 2019 2020 2021
Figure 3.1.3
CHAPTER 3 PREVIEW 86
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by SKILL CLUSTER, 2013-20
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report
0.4%
0.3%
0.3% Artificial Intelligence
0.2%
0.1% Neural Networks
Figure 3.1.4
CHAPTER 3 PREVIEW 87
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by INDUSTRY, 2020
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report
Information
Professional, Scientific, Tech Services
Agriculture, Forestry, Fishing, Hunting
Manufacturing
Finance and Insurance
Public Administration
Mining, Quarrying, Oil/Gas Extraction
Management of Companies/Enterprises
Educational Services
Wholesale Trade
Utilities
Real Estate and Rental and Leasing
Other Services
Retail Trade
Transportation and Warehousing
Health Care and Social Assistance
Construction
Arts, Entertainment, and Recreation
Accommodation and Food Services
0.0% 0.5% 1.0% 1.5% 2.0% 2.5%
AI Job Postings (% of All Job Postings)
Figure 3.1.5
CHAPTER 3 PREVIEW 88
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by INDUSTRY, 2013-20
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report
1.0%
0.0%
3.0%
1.4%
2.0% 0.8%
Finance and Insurance 0.9% 0.8%
Mining, Quarrying, Oil/Gas Extraction
AI Job Postings (% of All Job Postings)
0.0%
3.0%
2.0%
0.8% 0.7% 0.6%
Educational Services Wholesale Trade
1.0% Utilities
0.5%
0.0% Other Services
3.0%
2.0%
0.5% 0.4%
1.0% Real Estate and Rental and Leasing 0.2% 0.2%
Retail Trade
Arts, Entertainment, and Recreation Construction
0.0%
3.0%
2.0%
Figure 3.1.6
CHAPTER 3 PREVIEW 89
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
AI JOB POSTINGS (TOTAL and % of ALL JOB POSTINGS) by U.S. STATE and DISTRICT, 2020
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report
2.0%
District of Columbia
Washington
1.5% Virginia
AI Job Postings (% of All Job Postings)
Massachusetts
California
New York
1.0% Maryland
0.0%
100 200 500 1,000 2,000 5,000 10,000 20,000 50,000 100,000
Number of AI Job Postings (Log Scale)
Figure 3.1.7
CHAPTER 3 PREVIEW 90
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
A I S K I L L P E N E T R AT I O N
How prevalent are AI skills across occupations? The AI skill penetration rate of 2 means that the average penetration
penetration metric shows the average share of AI skills of AI skills in that country is 2 times the global average
among the top 50 skills in each occupation, using LinkedIn across the same set of occupations.
data that includes skills listed on a member’s profile, Among the sample countries shown in Figure 3.1.8, the
positions held, and the locations of the positions. aggregated data from 2015 to 2020 shows that India (2.83
Global Comparison times the global average) has the highest relative AI skill
For cross-country comparison, the relative penetration penetration rate, followed by the United States (1.99 times
rate of AI skills is measured as the sum of the penetration the global average), China (1.40 times the global average),
of each AI skill across occupations in a given country, Germany (1.27 times the global average), and Canada (1.13
divided by the average global penetration of AI skills times the global average).2
across the same occupations. For example, a relative
India
United States
China
Germany
Canada
South Korea
Singapore
United Kingdom
France
Australia
Brazil
Italy
South Africa
0 1 2 3
Relative AI Skill Penetration Rate
Figure 3.1.8
2 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.
CHAPTER 3 PREVIEW 91
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3.1 J O B S
Index Report 2021 THE ECONOMY
0 1 2 3 0 1 2 3 0 1 2 3
0 1 2 3 0 1 2 3
3 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.
CHAPTER 3 PREVIEW 92
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3. 2 I N V E S T M E N T
Index Report 2021 THE ECONOMY
This section explores the investment activity of private companies by NetBase Quid based on data from CapIQ and
Crunchbase. Specifically, it looks at the latest trends in corporate AI investment, such as private investment, public
offerings, mergers and acquisitions (M&A), and minority stakes related to AI. The section then focuses on the private
investment in AI, or how much private funding goes into AI startups and which sectors are attracting significant
investment and in which countries.
3.2 INVESTMENT
C O R P O R AT E I N V E S T M E N T
The total global investment in AI, including private in 2020 are driving up the total corporate investment in
investment, public offerings, M&A, and minority stakes, AI. M&A made up the majority of the total investment
increased by 40% in 2020 relative to 2019 for a total of amount in 2020, increasing by 121.7% relative to 2019.
USD 67.9 billion (Figure 3.2.1). Given the pandemic, many Several high-profile acquisitions related to AI took
small businesses have suffered disproportionately. As a place in 2020, including NVIDIA’s acquisition of Mellanox
result, industry consolidation and increased M&A activity Technologies and Capgemini’s of Altran Technologies.
70,000 67,854
Private Investment
Total Investment (in Millions of U.S. Dollars)
60,000
Public O ering
Merger/Acquisition
48,851
50,000 Minority Stake
44,075 42,238
43,811
40,000
23,002
30,000 38,659
36,576
4,140
20,000 17,699
12,751
13,097 18,932
10,000 7,952 19,849
4,493 8,541
4,328
0
2015 2016 2017 2018 2019 2020
Figure 3.2.1
CHAPTER 3 PREVIEW 93
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3. 2 I N V E S T M E N T
Index Report 2021 THE ECONOMY
S TA R T U P ACT I V I T Y
The following section analyzed the trend of PRIVATE INVESTMENT in FUNDED AI COMPANIES, 2015-20
Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report
private investment in AI startups that have
received investments of over USD 400,000 in
the last 10 years. While the amount of private 40,000
2016
2017
2018
2019
2020
from 2019—compared with the largest increase
of 59.0%, observed between 2017 and 2018. Figure 3.2.2
4,000
Number of Companies
3,000
2,000
1,000
0
2015 2016 2017 2018 2019 2020
Figure 3.2.3
CHAPTER 3 PREVIEW 94
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3. 2 I N V E S T M E N T
Index Report 2021 THE ECONOMY
Regional Comparison
As shown in Figure 3.2.4, the United States remains the leading destination for private investment, with over USD 23.6
billion in funding in 2020, followed by China (USD 9.9 billion) and the United Kingdom (USD 1.9 billion).
United States
China
United Kingdom
Israel
Canada
Germany
France
India
Japan
Singapore
Australia
A closer examination of the three contenders leading 2018, its investment level in 2020 is less than half that of
the AI race—the United States, China, and the European the United States (Figure 3.2.5). It is important to note,
Union—further validates the United States’ dominant however, that China has strong public investments in
position in private AI investment. While China saw an AI. Both the central and local governments in China are
exceptionally high amount of private AI investment in spending heavily on AI R&D.4
4 See “A Brief Examination of Chinese Government Expenditures on Artificial Intelligence R&D” (2020) by the Institute for Defense Analyses for more details.
CHAPTER 3 PREVIEW 95
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3. 2 I N V E S T M E N T
Index Report 2021 THE ECONOMY
23,597
25,000 US
Total Investment (in Millions of U.S. Dollars)
20,000
15,000
9,933
China
10,000
6,662
Rest of the World
5,000
2,044
EU
0
2015 2016 2017 2018 2019 2020
Figure 3.2.5
CHAPTER 3 PREVIEW 96
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3:
3. 2 I N V E S T M E N T
Index Report 2021 THE ECONOMY
Figure 3.2.6
CHAPTER 3 PREVIEW 97
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3: 3. 3 C O R P O R AT E
Index Report 2021 THE ECONOMY AC T I V I T Y
This section reviews how corporations have capitalized on the advances in AI, using AI and automation to their advantage
and generating value at scale. While the number of corporations starting to deploy AI technologies has surged in recent
years, the economic turmoil and impact of COVID-19 in 2020 have slowed that rate of adoption. The latest trends in
corporate AI activities are examined through data on the adoption of AI capabilities by McKinsey’s Global Survey on
AI, trends in robot installations across the globe by the International Federation of Robotics (IFR), and mentions of AI in
corporate earnings calls by Prattle.
Developed Asia-Pacific
India
All Geographies
North America
Europe
Latin America
CHAPTER 3 PREVIEW 98
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3: 3. 3 C O R P O R AT E
Index Report 2021 THE ECONOMY AC T I V I T Y
High tech/Telecom
Financial Services
Healthcare/Pharma
Consumer Goods/Retail
Figure 3.3.2
CHAPTER 3 PREVIEW 99
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 3: 3. 3 C O R P O R AT E
Index Report 2021 THE ECONOMY AC T I V I T Y
% of Respondents
Figure 3.3.3
Other
Robotic
Autonomous Computer Conversation Deep NL NL Speech U NL Text Unde Machine Physical
Industry Process
Vehicles Vision al Interfaces Learning Generation nderstanding rstanding Learning Robotics
Automation
Techniques
All Industries 7% 18% 15% 16% 11% 12% 13% 23% 13% 22%
Automotive and Assembly 20% 33% 16% 19% 12% 14% 19% 27% 31% 33%
Financial Services 6% 18% 24% 19% 18% 19% 26% 32% 8% 37%
Healthcare/Pharma 1% 15% 10% 14% 12% 11% 15% 19% 10% 18%
High Tech/Telecom 9% 34% 32% 30% 18% 25% 33% 37% 14% 34%
% of Respondents
Figure 3.3.4
Cybersecurity
Regulatory
Compliance
Explainability
Personal/Individual
Privacy
Organizational
Reputation
Workforce/Labor
Displacement
Equity And Fairness
Physical Safety
Figure 3.3.5
Cybersecurity
Regulatory
Compliance
Personal/Individual
Privacy
Explainability
Workforce/Labor
Displacement
Organizational
Reputation
Physical Safety
Figure 3.3.6
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Figure 3.3.7
400
Number of Units (in Thousands)
300
200
100
0
2012 2013 2014 2015 2016 2017 2018 2019
Figure 3.3.8
6 For more insights on the adoption of AI and robots by the industry, read the National Bureau of Economic Research working paper based on the 2018 Annual Business Survey by the U.S. Census Bureau,
“Advancing Technologies Adoption and Use by U.S. Firms: Evidence From the Annual Business Survey” (2020).
7 Note that there is no information on the customer industry for approximately 20% of robots installed.
Regional Comparison
Asia, Europe, and North America—three of the largest Asia, Europe, and North
industrial robot markets—all witnessed the end of a six-
year growth period in robot installations (Figure 3.3.9). America—three of the
North America experienced the sharpest decline, of 16%,
largest industrial robot
in 2019, compared with 5% in Europe and 13% in Asia.
Asia
Europe
North Amercia
Others
Africa 2019
Figure 3.3.9
China
Japan
United States
2017
2018
South Korea 2019
Germany
Rest of the
World
E A R N I N G S CA L L S
Mentions of AI in corporate earnings calls have increased substantially since 2013, as Figure 3.3.11 shows. In 2020, the number
of mentions of AI in earning calls was two times higher than mentions of big data, cloud, and machine learning combined,
though that figure declined by 8.5% from 2019. The mentions of big data peaked in 2017 and have since declined by 57%.
5,000
4,000
Number of AI Mentions
3,000
2,000
1,000
652 Big Data
310 Cloud
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
Figure 3.3.11
CHAPTER 4:
AI Education
Artificial Intelligence
Index Report 2021
CHAPTER 4 PREVIEW 107
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 4:
Index Report 2021 A I E D U CAT I O N
CHAPTER 4:
Chapter Preview
Overview 109 New CS PhDs in the United States 114
Chapter Highlights 110 New CS PhDs by Specialty 115
New CS PhDs with AI/ML and
4.1 STATE OF AI EDUCATION IN HIGHER Robotics/Vision Specialties 117
EDUCATION INSTITUTIONS 111 New AI PhDs Employment
Undergraduate AI Course Offerings 111 in North America 118
Overview
As AI has become a more significant driver of economic activity, there has been
increased interest from people who want to understand it and gain the necessary
qualifications to work in the field. At the same time, rising AI demands from
industry are tempting more professors to leave academia for the private sector.
This chapter focuses on trends in the skills and training of AI talent through various
education platforms and institutions.
CHAPTER HIGHLIGHTS
• An AI Index survey conducted in 2020 suggests that the world’s top universities have
increased their investment in AI education over the past four years. The number of courses
that teach students the skills necessary to build or deploy a practical AI model on the
undergraduate and graduate levels has increased by 102.9% and 41.7%, respectively, in the
last four academic years.
• More AI PhD graduates in North America chose to work in industry in the past 10 years,
while fewer opted for jobs in academia, according to an annual survey from the Computing
Research Association (CRA). The share of new AI PhDs who chose industry jobs increased by
48% in the past decade, from 44.4% in 2010 to 65.7% in 2019. By contrast, the share of new AI
PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019.
• In the last 10 years, AI-related PhDs have gone from 14.2% of the total of CS PhDs granted in
the United States, to around 23% as of 2019, according to the CRA survey. At the same time,
other previously popular CS PhDs have declined in popularity, including networking, software
engineering, and programming languages. Compilers all saw a reduction in PhDs granted
relative to 2010, while AI and Robotics/Vision specializations saw a substantial increase.
• After a two-year increase, the number of AI faculty departures from universities to industry
jobs in North America dropped from 42 in 2018 to 33 in 2019 (28 of these are tenured faculty
and five are untenured). Carnegie Mellon University had the largest number of AI faculty
departures between 2004 and 2019 (16), followed by the Georgia Institute of Technology (14)
and University of Washington (12).
• The percentage of international students among new AI PhDs in North America continued to
rise in 2019, to 64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in
the United States and 8.6% have taken jobs outside the United States.
• In the European Union, the vast majority of specialized AI academic offerings are taught at
the master’s level; robotics and automation is by far the most frequently taught course in the
specialized bachelor’s and master’s programs, while machine learning (ML) dominates in the
specialized short courses.
1 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are—Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;
Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
2 See here for a list of keywords on practical artificial intelligence models provided to the survey respondents. A course is defined as a set of classes that require a minimum of 2.5 class hours (including
lecture, lab, TA hours, etc.) per week for at least 10 weeks in total. Multiple courses with the same titles and numbers count as one course.
3 For universities that have a cap on course registration, the number of students who attempted to enroll in the intro-level AI and ML courses are included.
in the European Union has gradually NUMBER of STUDENTS WHO ENROLLED or ATTEMPTED to ENROLL
in INTRO to AI and INTRO to ML COURSES, AY 2016-20
increased by 165% in the past four Source: AI Index, 2020 | Chart: 2021 AI Index Report
academic years, while such enrollment
10,000
in the United States has seen a clear
dip in growth in the last academic
year (Figure 4.1.3). Six of the eight
8,000
U.S. universities surveyed say that the
number of (attempted) enrollments for
Number of Students
the introductory AI and ML courses has 6,000
decreased within the last year. Some
universities cited students taking leaves
during the pandemic as the main cause 4,000
Figure 4.1.2
PERCENTAGE CHANGE in the NUMBER of STUDENTS WHO ENROLLED or ATTEMPTED to ENROLL in INTRO to AI and
INTRO to ML COURSES by GEOGRAPHIC AREA, AY 2016-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report
165.1%, EU
150%
Percentage Change (2016-17=1)
100%
50%
21.0%, US
0%
2016-17 2017-18 2018-19 2019-20
Figure 4.1.3
Number of Courses
Graduate Courses That Focus
on AI Skills
Graduate offerings that teach students
100
the skills required to build or deploy a
practical AI model increased by 41.7% in
the last four academic years, from 151
50
courses in AY 2016–17 to 214 in AY 2019–20
(Figure 4.1.4).
0
100
50
0
2016-17 2017-18 2018-19 2019-20
Figure 4.1.5
4 See here for a list of keywords on practical artificial intelligence models provided to the survey respondents. A course is defined as a set of classes that require a minimum of 2.5 class hours (including
lecture, lab, TA hours, etc.) per week for at least 10 weeks in total. Multiple courses with the same titles and numbers count as one course.
This section presents findings from the annual Taulbee Survey from the Computing Research Association (CRA). The annual
CRA survey documents trends in student enrollment, degree production, employment of graduates, and faculty salaries in
academic units in the United States and Canada that grant doctoral degrees in computer science (CS), computer engineer-
ing (CE), or information (I). Academic units include departments of computer science and computer engineering or, in some
cases, colleges or schools of information or computing.
30,000
Number of New CS Undergraduate Graduates
25,000
20,000
15,000
10,000
5,000
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Figure 4.2.1
5 New CS PhDs in this section include PhD graduates from academic units (departments, colleges, or schools within universities) of computer science in the United States.
N E W C S P H D S BY S P E C I A LT Y
Among all computer science PhD graduates in 2019, those Over the past 10 years, AI/ML and robotics/vision are the
who specialized in artificial intelligence/machine learning CS PhD specializations that exhibit the most significant
(22.8%), theory and algorithms (8.0%), and robotics/ growth, relative to 18 other specializations (Figure 4.2.3).
vision (7.3%) top the list (Figure 4.2.2). The AI/ML specialty The percentage of AI/ML-specialized CS PhD graduates
has been the most popular in the past decade, and the among all new CS PhDs in 2020 is 8.6 percentage points
number of AI/ML graduates in 2019 is higher than the (pp) larger than in 2010, followed by robotics/vision-
number of the next five specialties combined. Moreover, specialized doctorates at 2.4 pp. By contrast, the share
robotics/vision jumped from the eighth most popular of CS PhDs specializing in networks (-4.8 pp), software
specialization in 2018 to the third in 2019. engineering (-3.6 pp), and programming languages/
compilers (-3.0 pp) experienced negative growth in 2020.
PERCENTAGE POINT CHANGE in NEW CS PHDS in the UNITED STATES from 2010 to 2019 by SPECIALTY
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report
Figure 4.2.3
N E W C S P H D S W I T H A I / M L A N D R O B OT I C S/ V I S I O N S P E C I A LT I E S
Figure 4.2.4a and Figure 4.2.4b take a closer look at the number of recent AI PhDs specializing in AI/ML or robotics/
vision in the United States. Between 2010 and 2019, the number of AI/ML-focused graduates grew by 77%, while the
percentage of these new PhDs among all CS PhD graduates increased by 61%. The number of both AI/ML and robotics/
vision PhD graduates reached an all-time high in 2019.
NEW CS PHDS with AI/ML and ROBOTICS/VISION NEW CS PHDS (% of TOTAL) with AI/ML and
SPECIALTY in the UNITED STATES, 2010-19 ROBOTICS/VISION SPECIALTY in the UNITED STATES,
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report
300
30%
22.8%
200
20%
100
10%
7.3%
0 0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
2..
.
Number of Artificial Intelligence/Machine Learning New PhDs Artificial Intelligence/Machine Learning Specialty (% of New CS PhDs)
N E W A I P H D S E M P LOY M E N T I N N O R T H A M E R I CA
Where do new AI PhD graduates choose to work? This section captures the employment trends of new AI PhDs in
academia and industry across North America.6
65.7%
60%
150
Number of New AI PhDs
40%
100
20% 23.7%
50
0 0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
100
N.
Number of New AI PhDs to Academia New AI PhDs to Academia (% of Total New AI PhDs)
Number of New AI PhDs to Industry New AI PhDs to Industry (% of Total New AI PhDs)
6 New AI PhDs in this section include PhD graduates who specialize in artificial intelligence from academic units (departments, colleges, or schools within universities) of computer science, computer
engineering, and information in the United States and Canada.
70%
New International AI PhDs (% of Total New AI PhDs)
64.3%
60%
50%
40%
30%
20%
2010 2011 INTERNATIONAL
2012 2013 NEW AI
2014 2015PHDS2016
(% of TOTAL)
2017 in
2018the UNITED
2019 STATES
by LOCATION OF EMPLOYMENT, 2019
Figure 4.2.6
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report
This section presents research from the Joint Research Center at the European Commission that assessed the academic offerings
of advanced digital skills in 27 European Union member states as well as six other countries: the United Kingdom, Norway,
Switzerland, Canada, the United States, and Australia. This was the second such study,7 and the 2020 version addressed four
technological domains: artificial intelligence (AI), high performance computing (HPC), cybersecurity (CS), and data science (DS),
applying text-mining and machine-learning techniques to extract content related to study programs addressing the specific
domains. See the reports “Academic Offer of Advanced Digital Skills in 2019–20. International Comparison” and “Estimation of
Supply and Demand of Tertiary Education Places in Advanced Digital Profiles in the EU,” for more detail.
Master
30 Bachelor
Number of Specialized AI Programs
Short Courses
20
10
0
Luxembourg
Netherlands
Lithuania
Germany
Denmark
Romania
Hungary
Portugal
Slovenia
Slovakia
Bulgaria
Belgium
Czechia
Sweden
Croatia
Finland
Estonia
Greece
Cyprus
Austria
Ireland
Poland
France
Latvia
Malta
Spain
Italy
Figure 4.3.1
7 Note that the 2020 report introduced methodological improvements from the 2019 version; therefore, a strict comparison is not possible. Improvements include the removal of certain keywords and
the addition of others to identify the programs. Still, more than 90% of all detected programs in the 2020 edition are triggered by keywords present in the 2019 study.
Machine Learning
AI Applications
AI Ethics
Computer Vision
Philosophy Of AI
AI (Generic)
Audio Processing
I N T E R N AT I O N A L C O M PA R I S O N
The JRC report compared AI education in the 27 EU The United States appears
member states with other countries in Europe, including
Norway, Switzerland, and the United Kingdom, as well to have offered more
as Canada, the United States, and Australia. Figure 4.3.3 programs specialized
shows the total number of 1,680 specialized AI programs
in all countries considered in the 2019–20 academic in AI than any other
year. The United States appears to have offered more
programs specialized in AI than any other geographic
geographic area although
area, although EU27 comes in a close second in terms of EU27 comes in a close
the number of AI-specialized master’s programs.
second in terms of the
number of AI-specialized
master’s programs.
United States
United
Kingdom
EU27
Australia
Canada
Bachelor
Master
Norway
Short Courses
Switzerland
Figure 4.3.3
40
Tenured
Total Number of Departures
30
Number of Faculty
Untenured
20
10
0
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 4.4.1
NUMBER of AI FACULTY DEPATURES in NORTH AMERICA (with UNIVERSITY AFFILIATION) by UNIVERSITY, 2004-18
Source: Gofman and Jin, 2020 | Chart: 2021 AI Index Report
University of Toronto
Stanford University
University of Southern California
University of Michigan
Purdue University
Harvard University
0 2 4 6 8 10 12 14 16
Number of Faculty
Figure 4.4.2
CHAPTER 5:
Ethical Challenges
of AI Applications
Artificial Intelligence
Index Report 2021
CHAPTER 5 PREVIEW 125
TA B L E O F C O N T E N T S
CHAPTER 5:
Artificial Intelligence
E T H I CA L C H A L L E N G E S
Index Report 2021 O F A I A P P L I CAT I O N S
CHAPTER 5:
Chapter Preview
Overview 127
Chapter Highlights 128
Overview
As artificial intelligence–powered innovations become ever more prevalent in our
lives, the ethical challenges of AI applications are increasingly evident and subject
to scrutiny. As previous chapters have addressed, the use of various AI technologies
can lead to unintended but harmful consequences, such as privacy intrusion;
discrimination based on gender, race/ethnicity, sexual orientation, or gender
identity; and opaque decision-making, among other issues. Addressing existing
ethical challenges and building responsible, fair AI innovations before they get
deployed has never been more important.
This chapter tackles the efforts to address the ethical issues that have arisen
alongside the rise of AI applications. It first looks at the recent proliferation of
documents charting AI principles and frameworks, as well as how the media covers
AI-related ethical issues. It then follows with a review of ethics-related research
presented at AI conferences and what kind of ethics courses are being offered by
computer science (CS) departments at universities around the world.
The AI Index team was surprised to discover how little data there is on this topic.
Though a number of groups are producing a range of qualitative or normative
outputs in the AI ethics domain, the field generally lacks benchmarks that can be
used to measure or assess the relationship between broader societal discussions
about technology development and the development of the technology itself.
One datapoint, covered in the technical performance chapter, is the study by the
National Institute of Standards and Technology on facial recognition performance
with a focus on bias. Figuring out how to create more quantitative data presents
a challenge for the research community, but it is a useful one to focus on.
Policymakers are keenly aware of ethical concerns pertaining to AI, but it is easier
for them to manage what they can measure, so finding ways to translate qualitative
arguments into quantitative data is an essential step in the process.
CHAPTER HIGHLIGHTS
• The five news topics that got the most attention in 2020 related to the ethical use of AI
were the release of the European Commission’s white paper on AI, Google’s dismissal of
ethics researcher Timnit Gebru, the AI ethics committee formed by the United Nations,
the Vatican’s AI ethics plan, and IBM’s exiting the facial-recognition businesses.
45
Research/Professional Organization
Private Company
40
Intergovernmental Organization/Agency 13
Number of New AI Ethics Principles
Government Agency
30 28
6
23
19
4
20
17 8
11 2 5 15
10
11
3 9
2 2
4
0 2
Figure 5.1.1
45
East Asia & Pacific
6
Europe & Central Asia
40
Global
Number of New AI Ethics Principles
20
17
12
13
5
10 16
9 7
2 2 7
0
2015 2016 2017 2018 2019 2020
Figure 5.1.2
Guidance, Framework
Research, Education
Facial Recognition
Algorithm Bias
AI Explainability
Data Privacy
Enterprise Efforts
Figure 5.2.1
1 The methodology for this is looking for articles that contain keywords related to AI ethics as determined by a Harvard research study.
80
70
60
Number of Paper Titles
40
20
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 5.3.1
AVERAGE NUMBER of PAPER TITLES MENTIONING ETHICS KEYWORDS at SELECT LARGE AI CONFERENCES,
2000-19
Source: Prates et al., 2018 | Chart: 2021 AI Index Report
AAAI
ICML
0.06
Average Number of Keywords Matches
ICRA
IJCAI
IROS
0.04 NIPS/NeurIPS
0.02
0.00
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
Figure 5.3.2
0 2 4 6 8 10 12
Figure 5.4.1
2 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are—Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;
Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
3 The survey did not explicitly present “Ethics modules embedded into CS courses” as an option. Selections were filled in the “Others” option. This will be included in next year’s survey.
CHAPTER 6:
Diversity in AI
Artificial Intelligence
Index Report 2021
CHAPTER 6 PREVIEW 135
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 6:
Index Report 2021 DIVERSITY IN AI
CHAPTER 6:
Chapter Preview
Overview 137 New Computing PhDs in the
Chapter Highlights 138 United States by Race/Ethnicity 145
CS Tenure-Track Faculty
by Race/Ethnicity 146
6.1 GENDER DIVERSITY IN AI 139
Black in AI 146
Women in Academic AI Settings 139
Women in the AI Workforce 140
6.3 GENDER IDENTITY AND
Women in Machine Learning Workshops 141
SEXUAL ORIENTATION IN AI 147
Workshop Participants 141
Queer in AI 147
Demographics Breakdown 142
Demographics Breakdown 147
Experience as Queer Practitioners 149
6.2 RACIAL AND ETHNIC
DIVERSITY IN AI 144
New AI PhDs in the United States
by Race/Ethnicity 144
Overview
While artificial intelligence (AI) systems have the potential to dramatically affect society,
the people building AI systems are not representative of the people those systems are
meant to serve. The AI workforce remains predominantly male and lacking in diversity in
both academia and the industry, despite many years highlighting the disadvantages and
risks this engenders. The lack of diversity in race and ethnicity, gender identity, and sexual
orientation not only risks creating an uneven distribution of power in the workforce, but
also, equally important, reinforces existing inequalities generated by AI systems, reduces
the scope of individuals and organizations for whom these systems work, and contributes
to unjust outcomes.
This chapter presents diversity statistics within the AI workforce and academia. It draws
on collaborations with various organizations—in particular, Women in Machine Learning
(WiML), Black in AI (BAI), and Queer in AI (QAI)— each of which aims to improve diversity
in some dimension in the field. The data is neither comprehensive nor conclusive. In
preparing this chapter, the AI Index team encountered significant challenges as a result
of the sparsity of publicly available demographic data. The lack of publicly available
demographic data limits the degree to which statistical analyses can assess the impact
of the lack of diversity in the AI workforce on society as well as broader technology
development. The diversity issue in AI is well known, and making more data available
from both academia and industry is essential to measuring the scale of the problem and
addressing it.
There are many dimensions of diversity that this chapter does not cover, including AI
professionals with disabilities; nor does it consider diversity through an intersectional
lens. Other dimensions will be addressed in future iterations of this report. Moreover,
these diversity statistics tell only part of the story. The daily challenges of minorities and
marginalized groups working in AI, as well as the structural problems within organizations
that contribute to the lack of diversity, require more extensive data collection and analysis.
1 We thank Women in Machine Learning, Black in AI, and Queer in AI for their work to increase diversity in AI, for sharing their data, and for partnering with us.
CHAPTER HIGHLIGHTS
• The CRA survey suggests that in 2019, among new U.S. resident AI PhD graduates,
45% were white, while 22.4% were Asian, 3.2% were Hispanic, and 2.4% were African
American.
• The percentage of white (non-Hispanic) new computing PhDs has changed little
over the last 10 years, accounting for 62.7% on average. The share of Black or
African American (non-Hispanic) and Hispanic computing PhDs in the same period is
significantly lower, with an average of 3.1% and 3.3%, respectively.
• The participation in Black in AI workshops, which are co-located with the Conference
on Neural Information Processing Systems (NeurIPS), has grown significantly in recent
years. The numbers of attendees and submitted papers in 2019 are 2.6 times higher than
in 2017, while the number of accepted papers is 2.1 times higher.
• In a membership survey by Queer in AI in 2020, almost half the respondents said they
view the lack of inclusiveness in the field as an obstacle they have faced in becoming
a queer practitioner in the AI/ML field. More than 40% of members surveyed said they
have experienced discrimination or harassment as a queer person at work or school.
FEMALE NEW AI and CS PHDS (% of TOTAL NEW AI and CS PHDS) in NORTH AMERICA, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report
30%
Female New AI PhDs (% of All New AI PhDs)
25%
22.1% AI
20%
20.3% CS
15%
10%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Figure 6.1.1
WO M E N I N T H E A I WO R K F O R C E
Chapter 3 introduced the “global relative AI skills
This data suggests that
penetration rate,” a measure that reflects the prevalence
of AI skills across occupations, or the intensity with which across the majority of
people in certain occupations use AI skills. Figure 6.1.3
shows AI skills penetration by country for female and
select countries, the
male labor pools in a set of select countries.2 The data AI skills penetration
suggest that across the majority of these countries, the
AI skills penetration rate for women is lower than that rate for women is lower
for men. Among the 12 countries we examined, India, than it is for men.
South Korea, Singapore, and Australia are the closest to
reaching equity in terms of the AI skills penetration rate
of females and males.
India
United States
South Korea
Singapore
China
Canada
France
Germany
Australia
Italy
0 1 2 3
Relative AI Skills Penetration Rate
Figure 6.1.3
2 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.
WO M E N I N M AC H I N E L E A R N I N G
WO R KS H O P S
Women in Machine Learning, founded in 2006 by because of the pandemic and delivered on a new platform
Hanna Wallach, Jenn Wortman, and Lisa Wainer, is an (Gather.Town); these two factors may make attendance
organization that runs events and programs to support numbers harder to compare to those of previous years.
women in the field of machine learning (ML). This Figure 6.1.4 shows an estimate of 925 attendees in 2020,
section presents statistics from its annual technical based on the number of individuals who accessed the
workshops, which are held at NeurIPS. In 2020, WiML virtual platform.
also hosted for the first time a full-day “Un-Workshop” at
In the past 10 years, WiML workshops have expanded
the International Conference on Machine Learning 2020,
their programs to include mentoring roundtables, where
which drew 812 participants.
more senior participants offer one-on-one feedback and
professional advice, in addition to the main session that
Workshop Participants
includes keynotes and poster presentations. Similar
The number of participants attending WiML workshops at
opportunities may have contributed to the increase in
NeurIPS has been steadily increasing since the workshops
attendance since 2014. Between 2016 and 2019, the WiML
were first offered in 2006. According to the organization,
workshop attendance is on average about 10% of the
the WiML workshop in 2020 was completely virtual
overall NeurIPS attendance.
1,000
925
800
Number of Participants
600
400
200
0
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 6.1.4
Demographics Breakdown
The following geographic, professional position, and Among the
gender breakdowns are based only on participants at
the 2020 WiML workshop at NeurIPS who consented to participants, 89.5%
having the information aggregated and who spent at
least 10 minutes on the virtual platform through which
were women and/
the workshop was offered. Among the participants, or nonbinary, 10.4%
89.5% were women and/or nonbinary, 10.4% were men
(Figure 6.1.5), and a large majority were from North
were men, and a large
America (Figure 6.1.6). Further, as shown in Figure 6.1.7, majority were from
students—including PhD, master’s, and undergraduate
students—make up more than half the participants North America. Further,
(54.6%). Among participants who work in the industry, students—including
research scientist/engineer and data scientist/engineer
are the most commonly held professional positions. PhD, master’s, and
undergraduate
students—make up
PARTICIPANTS of WIML WORKSHOP at NEURIPS
(% of TOTAL) byofGENDER,
PARTICIPANTS 2020 at NEURIPS
WIML WORKSHOP more than half the
(%Women
Source: of TOTAL) by GENDER,
in Machine 2020
Learning, 2020 | Chart: 2021 AI Index Report
Source: Women in Machine Learning, 2020 | Chart: 2021 AI Index Report participants (54.6%).
10.4%10.4%
Man Man
89.5%
Woman and/or nonbinary
89.5%
Woman and/or nonbinary
Figure 6.1.5
North America
Europe
Asia
Africa
Middle East
PhD Student
Research Scientist/Engineer
MSc Student
Data scientist/Engineer
Undergraduate Student
Postdoctoral Researcher
Software Engineer
Professor (Pre-Tenure)
Professor (Post-Tenure)
Program/Product Manager
70%
60%
58.9% White (non-Hispanic)
New Computing PhDs, U.S. Resident (% of Total)
50%
40%
30%
24.4% Asian
20%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.2.2
B L AC K I N A I
Black in AI (BAI), founded in 2017 by Timnit Gebru participating at major AI conferences globally 40-fold.
and Rediet Abebe, is a multi-institutional and Figure 6.2.4 shows the number of attendees, submitted
transcontinental initiative that aims to increase the papers, and accepted papers from the annual Black in
presence of Black people in the field of AI. As of 2020, BAI AI Workshop, which is co-located with NeurIPS.4 The
has around 3,000 community members and allies, has numbers of attendees and accepted papers in 2019
held more than 10 workshops at major AI conferences, are 2.6 times higher than in 2017, while the number of
and has helped increase the number of Black people accepted papers is 2.1 times higher.
NUMBER OF ATTENDEES, SUBMITTED PAPERS, and ACCEPTED PAPERS at BLACK in AI WORKSHOP CO-LOCATED
with NEURIPS, 2017-19
Source: Black in AI, 2020 | Chart: 2021 AI Index Report
Number of
Attendees
Submitted
Papers
2017
2018
Accepted
Papers 2019
3 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;
Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
4 The 2020 data are clearly affected by the pandemic and not included as a result. For more information, see the Black in AI impact report.
Gay
Bisexual
Queer
Lesbian
Straight
Asexual
Pansexual
Others
Figure 6.3.1
5 Queer in AI presents the survey results at its workshop at the annual NeurIPS conference.
Cis Male
Cis Female
Gender Queer
Gender Fluid
Trans Female
Figure 6.3.2
QAI MEMBERSHIP SURVEY: HOW WOULD YOU DESCRIBE YOUR POSITION, 2018-20
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report
Student
Junior
Academic
Junior
Industry
2018
2019
2020
Others
Figure 6.3.3
QAI MEMBERSHIP SURVEY: WHAT ARE OBSTACLES YOU HAVE FACED in BECOMING a QUEER AI/ML
PRACTITIONER, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report
Lack of Community
Lack of Inclusiveness
Harrassment/Discrimination
Figure 6.3.4
QAI MEMBERSHIP SURVEY: HAVE YOU EXPERIENCED DISCRIMINATION/HARASSMENT as a QUEER PERSON at YOUR
JOB or SCHOOL, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report
0 times
1 time
2 times
5+ times
Others
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55%
% of Respondents
Figure 6.3.5
CHAPTER 7:
AI Policy and
National Strategies
Artificial Intelligence
Index Report 2021
CHAPTER 7 PREVIEW 151
TA B L E O F C O N T E N T S
C H A P T E R 7:
Artificial Intelligence
A I P O L I CY A N D
Index Report 2021 N AT I O N A L S T R AT E G I E S
CHAPTER 7:
Chapter Preview
Overview 153 Working Group 165
Chapter Highlights 154 Summits and Meetings 166
Bilateral Agreements 166
7.1 NATIONAL AND REGIONAL
AI STRATEGIES 155 7.3 U.S. PUBLIC INVESTMENT IN AI 167
Published Strategies 156 Federal Budget for Non-Defense AI R&D 167
2017 156 U.S. Department of Defense
2018 157 Budget Request 168
2019 159 U.S. Government Contract Spending 169
2020 161 Total Contract Spending 169
Strategies in Development Contract Spending by
(as of December 2020) 162 Department and Agency 169
Strategies in Public Consultation 162
Strategies Announced 163 7.4 AI AND POLICYMAKING 171
Highlight: National AI Strategies Legislation Records on AI 171
and Human Rights 164 U.S. Congressional Record 172
Mentions of AI and ML in
7.2 INTERNATIONAL Congressional/Parliamentary
COLLABORATION ON AI 165 Proceedings 172
Intergovernmental Initiatives 165 Central Banks 174
U.S. AI Policy Papers 176
Overview
AI is set to shape global competitiveness over the coming decades, promising
to grant early adopters a significant economic and strategic advantage. To
date, national governments and regional and intergovernmental organizations
have raced to put in place AI-targeted policies to maximize the promise of the
technology while also addressing its social and ethical implications.
This chapter navigates the landscape of AI policymaking and tracks efforts taking
place on the local, national, and international levels to help promote and govern AI
technologies. It begins with an overview of national and regional AI strategies and
then reviews activities on the intergovernmental level. The chapter then takes a
closer look at public investment in AI in the United States as well as how legislative
bodies, central banks, and nongovernmental organizations are responding to the
growing need to institute a policy framework for AI technologies.
CHAPTER HIGHLIGHTS
• Since Canada published the world’s first national AI strategy in 2017, more than 30 other
countries and regions have published similar documents as of December 2020.
• The launch of the Global Partnership on AI (GPAI) and Organisation for Economic
Co-operation and Development (OECD) AI Policy Observatory and Network of Experts
on AI in 2020 promoted intergovernmental efforts to work together to support the
development of AI for all.
• In the United States, the 116th Congress was the most AI-focused congressional session in
history. The number of mentions of AI by this Congress in legislation, committee reports, and
Congressional Research Service (CRS) reports is more than triple that of the 115th Congress.
This section presents an overview of select national and regional AI strategies from around the world, including details on
the strategies for G20 countries, Estonia, and Singapore as well as links to strategy documents for many others. Sources
include websites of national or regional governments, the OECD AI Policy Observatory (OECD.AI), and news coverage. “AI
strategy” is defined as a policy document that communicates the objective of supporting the development of AI while also
maximizing the benefits of AI for society. Excluded are broader innovation or digital strategy documents which do not focus
predominantly on AI, such as Brazil’s E-Digital Strategy and Japan’s Integrated Innovation Strategy.
COUNTRIES
WITH PUBLISHED
AI STRATEGIES: 32
COUNTRIES
DEVELOPING
AI STRATEGIES: 22
Published Strategies
2017
Canada
• AI Strategy: Pan Canadian AI Strategy
• Responsible Organization: Canadian Institute for
Advanced Research (CIFAR)
• Highlights: The Canadian strategy emphasizes
developing Canada’s future AI workforce, supporting major
AI innovation hubs and scientific research, and positioning
the country as a thought leader in the economic, ethical,
policy, and legal implications of artificial intelligence.
• Funding (December 2020 conversion rate): CAD 125
million (USD 97 million) a multi-stakeholder coalition consisting of academic
• In November 2020, CIFAR published its most recent institutions and private-sector players such as Tencent
annual report, titled “AICAN,” which tracks progress on and Baidu.
implementing its national strategy, which highlighted
substantial growth in Canada’s AI ecosystem, as well Japan
as research and activities related to healthcare and AI’s • AI Strategy: Artificial Intelligence Technology Strategy
impact on society, among other outcomes of the strategy. • Responsible Organization: Strategic Council for AI
Technology
China • Highlights: The strategy lays out three discrete phases of
• AI Strategy: A Next Generation Artificial Intelligence AI development. The first phase focuses on the utilization
Development Plan of data and AI in related service industries, the second
• Responsible Organization: State Council for the People’s on the public use of AI and the expansion of service
Republic of China industries, and the third on creating an overarching
• Highlights: China’s AI strategy is one of the most ecosystem where the various domains are merged.
comprehensive in the world. It encompasses areas • Funding: N/A
including R&D and talent development through • Recent Updates: In 2019, the Integrated Innovation
education and skills acquisition, as well as ethical norms Strategy Promotion Council launched another AI strategy,
and implications for national security. It sets specific aimed at taking the next step forward in overcoming
targets, including bringing the AI industry in line with issues faced by Japan and making use of the country’s
competitors by 2020; becoming the global leader in fields strengths to open up future opportunities.
such as unmanned aerial vehicles (UAVs), voice and
image recognition, and others by 2025; and emerging as Others
the primary center for AI innovation by 2030. Finland: Finland’s Age of Artificial Intelligence
• Funding: N/A United Arab Emirates: UAE Strategy for Artificial
• Recent Updates: China established a New Generation Intelligence
AI Innovation and Development Zone in February 2019
and released the “Beijing AI Principles” in May 2019 with
Published Strategies
2018
European Union
• AI Strategy: Coordinated Plan on Artificial Intelligence
• Responsible Organization: European Commission
• Highlights: This strategy document outlines the
commitments and actions agreed on by EU member
states, Norway, and Switzerland to increase investment
and build their AI talent pipeline. It emphasizes the value
of public-private partnerships, creating European data
spaces, and developing ethics principles.
• Funding (December 2020 conversation rate): At least
EUR 1 billion (USD 1.1 billion) per year for AI research and • Recent Updates: The French National Research Institute
at least EUR 4.9 billion (USD 5.4 billion) for other aspects for Digital Sciences (Inria) has committed to playing a
of the strategy central role in coordinating the national AI strategy and
• Recent updates: A first draft of the ethics guidelines was will report annually on its progress.
released in June 2018, followed by an updated version in
April 2019. Germany
• AI Strategy: AI Made in Germany
France • Responsible Organizations: Federal Ministry of
• AI Strategy: AI for Humanity: French Strategy for Artificial Education and Research; Federal Ministry for Economic
Intelligence Affairs and Energy; Federal Ministry of Labour and Social
• Responsible Organizations: Ministry for Higher Affairs
Education, Research and Innovation; Ministry of Economy • Highlights: The focus of the strategy is on cementing
and Finance; Directorate General for Enterprises; Public Germany as a research powerhouse and strengthening
Health Ministry; Ministry of the Armed Forces; National the value of its industries. There is also an emphasis
Research Institute for Digital Sciences; Interministerial on the public interest and working to better the lives of
Director of the Digital Technology and the Information people and the environment.
and Communication System • Funding (December 2020 conversion rate): EUR 500
• Highlights: The main themes include developing million (USD 608 million) in the 2019 budget and EUR
an aggressive data policy for big data; targeting four 3 billion (USD 3.6 billion) for the implementation up to
strategic sectors, namely health care, environment, 2025
transport, and defense; boosting French efforts in • Recent Updates: In November 2019, the government
research and development; planning for the impact of AI published an interim progress report on the Germany AI
on the workforce; and ensuring inclusivity and diversity strategy.
within the field.
• Funding (December 2020 conversion rate): EUR 1.5
billion (USD 1.8 billion) up to 2022
2018 (continued)
India
• AI Strategy: National Strategy on
Artificial Intelligence: #AIforAll
• Responsible Organization: National Institution for
Transforming India (NITI Ayog)
• Highlights: The Indian strategy focuses on both
economic growth and ways to leverage AI to increase
social inclusion, while also promoting research to
address important issues such as ethics, bias, and
privacy related to AI. The strategy emphasizes sectors
such as agriculture, health, and education, where public
investment and government initiative are necessary.
• Funding (December 2020 conversion rate): INR 7000 United Kingdom
crore (USD 949 million) • AI Strategy: Industrial Strategy: Artificial Intelligence
• Recent Updates: In 2019, the Ministry of Electronics and Sector Deal
Information Technology released its own proposal to • Responsible Organization: Office for Artificial
set up a national AI program with an allocated INR 400 Intelligence (OAI)
crore (USD 54 million). The Indian government formed • Highlights: The U.K. strategy emphasizes a strong
a committee in late 2019 to push for an organized AI partnership between business, academia, and the
policy and establish the precise functions of government government and identifies five foundations for a
agencies to further India’s AI mission. successful industrial strategy: becoming the world’s most
innovative economy, creating jobs and better earnings
Mexico potential, infrastructure upgrades, favorable business
• AI Strategy: Artificial Intelligence Agenda MX conditions, and building prosperous communities
(2019 agenda-in-brief version) throughout the country.
• Responsible Organization: IA2030Mx, Economía • Funding (December 2020 conversion rate): GBP 950
• Highlights: As Latin America’s first strategy, the Mexican million (USD 1.3 billion)
strategy focuses on developing a strong governance • Recent Updates: Between 2017 and 2019, the U.K.’s
framework, mapping the needs of AI in various industries, Select Committee on AI released an annual report on the
and identifying governmental best practices with an country’s progress. In November 2020, the government
emphasis on developing Mexico’s AI leadership. announced a major increase in defense spending of
• Funding: N/A GBP 16.5 billion (USD 21.8 billion) over four years, with
• Recent Updates: According to the Inter-American a major emphasis on AI technologies that promise to
Development Bank’s recent fAIr LAC report, Mexico is in revolutionize warfare.
the process of establishing concrete AI policies to further
implementation. Others
Sweden: National Approach to Artificial Intelligence
Taiwan: Taiwan AI Action Plan
Published Strategies
2019
Estonia
• AI Strategy: National AI Strategy 2019–2021
• Responsible Organization: Ministry of Economic Affairs
and Communications (MKM)
• Highlights: The strategy emphasizes actions necessary
for both the public and private sectors to take to increase
investment in AI research and development, while also
improving the legal environment for AI in Estonia. In
addition, it hammers out the framework for a steering
committee that will oversee the implementation and
monitoring of the strategy. the use of AI, developing practical measures to introduce
• Funding (December 2020 conversion rate): EUR 10 AI algorithms, providing neural network developers with
million (USD 12 million) up to 2021 competitive access to big data, and boosting private
• Recent Updates: The Estonian government released an investment in domestic AI industries.
update on the AI taskforce in May 2019.
Singapore
Russia • AI Strategy: National Artificial Intelligence Strategy
• AI Strategy: National Strategy for the Development of • Responsible Organization: Smart Nation and Digital
Artificial Intelligence Government Office (SNDGO)
• Responsible Organizations: Ministry of Digital • Highlights: Launched by Smart Nation Singapore, a
Development, Communications and Mass Media; government agency that seeks to transform Singapore’s
Government of the Russian Federation economy and usher in a new digital age, the strategy
• Highlights: The Russian AI strategy places a strong identifies five national AI projects in the following fields:
emphasis on its national interests and lays down transport and logistics, smart cities and estates, health
guidelines for the development of an “information care, education, and safety and security.
society” between 2017 and 2030. These include a • Funding (December 2020 conversion rate): While the
national technology initiative, departmental projects 2019 strategy does not mention funding, in 2017 the
for federal executive bodies, and programs such as the government launched its national program, AI Singapore,
Digital Economy of the Russian Federation, designed to with a pledge to invest SGD 150 million (USD 113 million)
implement the AI framework across sectors. over five years.
• Funding: N/A • Recent Updates: In November 2020, SNDGO published
• Recent Updates: In December 2020, Russian president its inaugural annual update on the Singaporean
Vladmir Putin took part in the Artificial Intelligence government’s data protection efforts. It describes the
Journey Conference, where he presented four ideas for AI measures taken to date to strengthen public sector data
policies: establishing experimental legal frameworks for security and to safeguard citizens’ private data.
2019 (continued)
United States
• AI Strategy: American AI Initiative
• Responsible Organization: The White House
• Highlights: The American AI Initiative prioritizes
the need for the federal government to invest in AI
R&D, reduce barriers to federal resources, and ensure
technical standards for the safe development, testing,
and deployment of AI technologies. The White House
also emphasizes developing an AI-ready workforce and
signals a commitment to collaborating with foreign
partners while promoting U.S. leadership in AI. The
initiative, however, lacks specifics on the program’s
• Funding (December 2020 conversion rate):
timeline, whether additional research will be dedicated
KRW 2.2 trillion (USD 2 billion)
to AI development, and other practical considerations.
• Recent Updates: N/A
• Funding: N/A
• Recent Updates: The U.S. government released its
year one annual report in February 2020, followed in
Others
November by the first guidance memorandum for federal
Colombia: National Policy for Digital Transformation
and Artificial Intelligence
agencies on regulating artificial intelligence applications
in the private sector, including principles that encourage
Czech Republic: National Artificial Intelligence
Strategy of the Czech Republic
AI innovation and growth and increase public trust and
confidence in AI technologies. The National Defense
Lithuania: Lithuanian Artificial Intelligence Strategy: A
Vision for the Future
Authorization Act (NDAA) for Fiscal Year 2021 called for a
National AI Initiative to coordinate AI research and policy
Luxembourg: Artificial Intelligence: A Strategic Vision
for Luxembourg
across the federal government.
Malta: Malta: The Ultimate AI Launchpad
South Korea Netherlands: Strategic Action Plan for Artificial
Intelligence
• AI Strategy: National Strategy for Artificial Intelligence
• Responsible Organization: Ministry of Science, ICT and
Portugal: AI Portugal 2030
Future Planning (MSIP)
Qatar: National Artificial Intelligence for Qatar
• Highlights: The Korean strategy calls for plans to
facilitate the use of AI by businesses and to streamline
regulations to create a more favorable environment for
the development and use of AI and other new industries.
The Korean government also plans to leverage its
dominance in the global supply of memory chips to build
the next generation of smart chips by 2030.
Published Strategies
2020
Indonesia
• AI Strategy: National Strategy for the Development of
Artificial Intelligence (Stranas KA)
• Responsible Organizations: Ministry of Research
and Technology (Menristek), National Research and
Innovation Agency (BRIN), Agency for the Assessment and
Application of Technology (BPPT)
• Strategy Highlights: The Indonesian strategy aims
to guide the country in developing AI between 2020
and 2045. It focuses on education and research, health
services, food security, mobility, smart cities, and public Others
sector reform. Hungary: Hungary’s Artificial Intelligence Strategy
• Funding: N/A Norway: National Strategy for Artificial Intelligence
• Recent Updates: None Serbia: Strategy for the Development of Artificial
Intelligence in the Republic of Serbia for the Period
Saudi Arabia 2020–2025
• AI Strategy: National Strategy on Data and AI (NSDAI) Spain: National Artificial Intelligence Strategy
• Responsible Organization: Saudi Data and Artificial
Intelligence Authority (SDAIA)
• Highlights: As part of an effort to diversify the country’s
economy away from oil and boost the private sector, the
NSDAI aims to accelerate AI development in five critical
sectors: health care, mobility, education, government,
and energy. By 2030, Saudi Arabia intends to train 20,000
data and AI specialists, attract USD 20 billion in foreign
and local investment, and create an environment that
will attract at least 300 AI and data startups.
• Funding: N/A
• Recent Updates: During the summit where the
Saudi government released its strategy, the country’s
National Center for Artificial Intelligence (NCAI) signed
collaboration agreements with China’s Huawei and
Alibaba Cloud to design AI-related Arabic-language
systems.
Strategies in Development
(AS OF DECEMBER 2020)
Brazil
• AI Strategy Draft: Brazilian Artificial Intelligence Strategy
• Responsible Organization: Ministry of Science,
Technology and Innovation (MCTI)
• Highlights: Brazil’s national AI strategy was announced
in 2019 and is currently in the public consultation stage.
According to the OECD, the strategy aims to cover
relevant topics bearing on AI, including its impact on the
economy, ethics, development, education, and jobs, and Others
to coordinate specific public policies addressing such Cyprus: National Strategy for Artificial Intelligence
issues. Ireland: National Irish Strategy on Artificial Intelligence
• Funding: N/A Poland: Artificial Intelligence Development Policy in
• Recent Updates: In October 2020, the country’s largest Poland
research facility dedicated to AI was launched in Uruguay: Artificial Intelligence Strategy for Digital
collaboration with IBM, the University of São Paulo, and Government
the São Paulo Research Foundation.
Italy
• AI Strategy Draft: Proposal for an Italian Strategy for
Artificial Intelligence
• Responsible Organization: Ministry of Economic
Development (MISE)
• Highlights: This document provides the proposed
strategy for the sustainable development of AI, aimed
at improving Italy’s competitiveness in AI. It focuses on
improving AI-based skills and competencies, fostering AI
research, establishing a regulatory and ethical framework
to ensure a sustainable ecosystem for AI, and developing
a robust data infrastructure to fuel these developments.
• Funding (December 2020 conversion rate): EUR 1
billion (USD 1.1 billion) through 2025 and expected
matching funds from the private sector, bringing the total
investment to EUR 2 billion.
• Recent Updates: None
Strategies Announced
Argentina
• Related Document: N/A
• Responsible Organization: Ministry of Science,
Technology and Productive Innovation (MINCYT)
• Status: Argentina’s AI plan is a part of the Argentine
Digital Agenda 2030 but has not yet been published. It is
intended to cover the decade between 2020 and 2030,
and reports indicate that it has the potential to reap huge
benefits for the agricultural sector.
Australia
• Related Documents: Artificial Intelligence Roadmap / on talent development, scientific research, ethics and
An AI Action Plan for all Australians inclusion, and digital infrastructure.
• Responsible Organizations: Commonwealth Scientific
and Industrial Research Organisation (CSIRO), Data 61, Others
and the Australian government Austria: Artificial Intelligence Mission Austria
• Status: The Australian government published a road (official report)
map in 2019 (in collaboration with the national science Bulgaria: Concept for the Development of Artificial
agency, CSIRO) and a discussion paper of an AI action Intelligence in Bulgaria Until 2030 (concept document)
plan in 2020 as frameworks to develop a national Chile: National AI Policy (official announcement)
AI strategy. In its 2018–19 budget, the Australian Israel: National AI Plan (news article)
government earmarked AUD 29.9 million (USD 22.2 Kenya: Blockchain and Artificial Intelligence Taskforce
million [December 2020 conversation rate]) over four (news article)
years to strengthen the country’s capabilities in AI and Latvia: On the Development of Artificial Intelligence
machine learning (ML). In addition, CSIRO published a Solutions (official report)
research paper on Australia’s AI Ethics Framework in 2019 Malaysia: National Artificial Intelligence (Al) Framework
and launched a public consultation, which is expected to (news article)
produce a forthcoming strategy document. New Zealand: Artificial Intelligence: Shaping a Future
New Zealand (official report)
Turkey Sri Lanka: Framework for Artificial Intelligence (news
• Related Document: N/A article)
• Responsible Organizations: Presidency of the Republic Switzerland: Artificial Intelligence (official guidelines)
of Turkey Digital Transformation Office; Ministry of Tunisia: National Artificial Intelligence Strategy (task
Industry and Technology; Scientific and Technological force announced)
Research Council of Turkey; Science, Technology and Ukraine: Concept of Artificial Intelligence Development
Innovation Policies Council in Ukraine AI (concept document)
• Status: The strategy has been announced but not yet Vietnam: Artificial Intelligence Development Strategy
published. According to media sources, it will focus (official announcement)
7.2 INTERNATIONAL
COLLABORATION ON AI
Given the scale of the opportunities and the challenges Working Groups
presented by AI, a number of international efforts have Global Partnership on AI (GPAI)
recently been announced that aim to develop multilateral • Participants: Australia, Brazil, Canada, France, Germany,
AI strategies. This section provides an overview of those India, Italy, Japan, Mexico, the Netherlands, New
international initiatives from governments committed to Zealand, South Korea, Poland, Singapore, Slovenia,
working together to support the development of AI for all. Spain, the United Kingdom, the United States, and the
European Union (as of December 2020)
These multilateral initiatives on AI suggest that
• Host of Secretariat: OECD
organizations are taking a variety of approaches to
• Focus Areas: Responsible AI; data governance; the future
tackle the practical applications of AI and scale those
of work; innovation and commercialization
solutions for maximum global impact. Many countries
• Recent Activities: Two International Centres of
turn to international organizations for global AI norm
Expertise—the International Centre of Expertise in
formulation, while others engage in partnerships or
Montreal for the Advancement of Artificial Intelligence
bilateral agreements. Among the topics under discussion,
and the French National Institute for Research in Digital
the ethics of AI—or the ethical challenges raised by current
Science and Technology (INRIA) in Paris—are supporting
and future applications of AI—stands out as a particular
the work in the four focus areas and held the Montreal
focus area for intergovernmental efforts.
Summit 2020 in December 2020. Moreover, the data
Countries such as Japan, South Korea, the United governance working group published the beta version of
Kingdom, the United States, and members of the European the group’s framework in November 2020.
Union are active participants of intergovernmental
OECD Network of Experts on AI (ONE AI)
efforts on AI. A major AI powerhouse, China, on the other
• Participants: OECD countries
hand, has opted to engage in a number of science and
• Host: OECD
technology bilateral agreements that stress cooperation
• Focus Areas: Classification of AI; implementing
on AI as part of the Digital Silk Road under the Belt
trustworthy AI; policies for AI; AI compute
and Road (BRI) initiative framework. For example, AI is
• Recent Activities: ONE AI convened its first meeting in
mentioned in China’s economic cooperation under the BRI
February 2020, when it also launched the OECD AI Policy
Initiative with the United Arab Emirates.
Observatory. In November 2020, the working group on
I N T E R G OV E R N M E N TA L the classification of AI presented the first look at an AI
I N I T I AT I V E S classification framework based on OECD’s definition of AI
Intergovernmental working groups consist of experts and divided into four dimensions (context, data and input, AI
policymakers from member states who study and report model, task and output) that aims to guide policymakers
on the most urgent challenges related to developing and in designing adequate policies for each type of AI system.
deploying AI and then make recommendations based on High-Level Expert Group on Artificial Intelligence (HLEG)
their findings. These groups are instrumental in identifying • Participants: EU countries
and developing strategies for the most pressing issues in AI • Host: European Commission
technologies and their applications. • Focus Areas: Ethics guidelines for trustworthy AI
•R ecent Activities: Since its launch at the recommendation
This section examines public investment in AI in the United States based on data from the U.S. Networking and Information
Technology Research and Development (NITRD) program and Bloomberg Government.
0
2020 (Request) 2020 (Enacted) 2021 (Request)
Figure 7.3.1
U.S. D E PA R T M E N T O F D E F E N S E A I The top five projects set to receive the highest amount of
R& D B U D G E T R E Q U E S T AI R&D investment in FY 2021:
While the official DOD budget is not publicly available, • Rapid Capability Development and Maturation, by the
Bloomberg Government has analyzed the department’s U.S. Army (USD 284.2 million)
publicly available budget request for research, • Counter WMD Technologies and Capabilities
development, test, and evaluation (RDT&E)— data that Development, by the DOD Threat Reduction Agency
sheds light on its spending on AI R&D. (USD 265.2 million)
• Algorithmic Warfare Cross-Functional Team (Project
With 305 unclassified DOD R&D programs specifying the use Maven), by the Office of the Secretary of Defense (USD
of AI or ML technologies, the combined U.S. military budget 250.1 million)
for AI R&D in FY 2021 is USD 5.0 billion (Figure 7.3.2). This • Joint Artificial Intelligence Center (JAIC), by the Defense
figure appears consistent with the USD 5.0 billion enacted Information Systems Agency (USD 132.1 million)
the previous year. However, the FY 2021 figure reflects • High Performance Computing Modernization Program,
a budget request, rather than a final enacted budget. by the U.S. Army (USD 99.6 million)
As noted above, once congressional appropriations are In addition, the Defense Advanced Research Projects
factored in, the true level of funding available to DOD AI R&D Agency (DARPA) alone is investing USD 568.4 million in AI
programs in FY 2021 may rise substantially. R&D, an increase of USD 82 million from FY 2020.
U.S. DOD BUDGET for AI-SPECIFIC RESEARCH DEVELOPMENT, TEST, and EVALUATION (RDT&E), FY 2018-20
Sources: Bloomberg Government & U.S. Department of Defense, 2020 | Chart: 2021 AI Index Report
5,000
Budget (in Millions of U.S. Dollars)
4,000
3,000
DOD Reported
2,000 Budget on AI R&D DOD Reported
Budget on AI R&D
0
2018 (Enacted) 2019 (Enacted) 2020 (Enacted) 2021 (Request)
Figure 7.3.2
Important data caveat: This chart illustrates the challenge of working with contemporary government data sources
to understand spending on AI. By one measure—the requests that include AI-relevant keywords—the DOD is requesting
more than USD 5 billion for AI-specific research development in 2021 . However, DOD’s own accounting produces a
radically smaller number: USD 841 million. This relates to the issue of defining where an AI system ends and another
system begins; for instance, an initiative that uses AI for drones may also count hardware-related expenditures for the
drones within its “AI” budget request, though the AI software component will be much smaller.
2,000 1,837
Contract Spending (in Millions of U.S. Dollars)
1,500
1,000
500
0
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 7.3.3
still in the early stages of driving DOD’s AI spending. In Warfighter program, and another to Deloitte Consulting for
2020, JAIC awarded two massive contracts, one to Booz a four-year, USD 106 million enterprise cloud environment
Allen Hamilton for the five-year, USD 800 million Joint for the JAIC, known as the Joint Common Foundation.
TOP 10 CONTRACT SPENDING on AI by U.S. GOVERNMENT DEPARTMENT and AGENCY, 2001-20 (SUM)
Source: Bloomberg Government, 2020 | Chart: 2021 AI Index Report
500 486
300
70
200
149
66
100 173
39
22 17 15 16
4 7 10 8 44
0
107Th 108Th 109Th 110Th 111Th 112Th 113Th 114Th 115Th 116th
(2001-2002) (2003-2004) (2005-2006) (2007-2008) (2009-2010) (2011-2012) (2013-2014) (2015-2016) (2017-2018) (2019-2020)
Figure 7.4.1
1 If a speaker or member mentioned artificial intelligence (AI) or machine learning (ML) multiple times within remarks, or multiple speakers mentioned AI or ML within the same event, it appears only
once as a result. Counts for AI and ML are separate, as they were conducted in separate searches. Mentions of the abbreviations “AI” or “ML” are not included.
140
129
120
120
28
28
Machine Learning
100
Artificial Intelligence 92
Number of Mentions
80 25
60
101
92
40
67
27
20
8 9 23
0 7 1 1 1 6
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 7.4.2
300
283
37
250
Machine Learning
Number of Mentions
150 138
246
100
179
158
138
51
50
42
7 4 5
0 0 1
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 7.4.3
40 38
Machine Learning 3
34
Artificial Intelligence
30
Number of Mentions
21
20 18
35
33
21
10
17
0 0 0 0 0 0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 7.4.4
C E N T R A L BA N KS
Central banks play a key role in conducting currency and Figure 7.4.5 shows a significant increase in the mention
monetary policy in a country or a monetary union. As of AI across 16 central banks over the past 10 years,
with many other institutions, central banks are tasked with the number reaching a peak of 1,020 in 2019. The
with integrating AI into their operations and relying on sharp decline in 2020 can be explained by the COVID-19
big data analytics to assist them with forecasting, risk pandemic as most central bank communications focused
management, and financial supervision. on responses to the economic downturn. Moreover,
the Federal Reserve in the United States, Norges Bank
Prattle, a leading provider of automated investment
in Norway, and the European Central Bank top the
research solutions, monitors mentions of AI in the
list for the most aggregated number of AI mentions in
communications of central banks, including meeting
communications in the past five years (Figure 7.4.6).
minutes, monetary policy papers, press releases,
speeches, and other official publications.
1,000
800
Number of Mentions
600
400
200 225
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Figure 7.4.5
2 See Science & Technology Review and Scientific American for more details.
MENTIONS of AI in CENTRAL BANK COMMUNICATIONS around THE WORLD by BANK, 2016-20 (SUM)
Source: Prattle/LiquidNet, 2020 | Chart: 2021 AI Index Report
Federal Reserve
Norges Bank
European Central Bank
Reserve Bank of India
Bank of England
Bank of Israel
Bank of Japan
Bank of Korea
Reserve Bank of Australia
Reserve bank of New Zealand
Bank of Taiwan
Bank of Canada
Sveriges Riksbank
Swedish Riksbank
Central Bank of the Republic of Turkey
Central Bank of Brazil
0 200 400 600 800 1,000 1,200 1,400 1,600 1,800 2,000
Number of Mentions
Figure 7.4.6
U.S. A I P O L I CY PA P E R S
What are the AI policy initiatives outside national and for policymakers. Primary topics mean that such a topic is
intergovernmental governments? We monitored 42 the main focus of the policy paper, while secondary topics
prominent organizations that deliver policy papers on mean that the policy paper either briefly touches on the
topics related to AI and assessed the primary topic as topic or the topic is a sub-focus of the paper.
well as the secondary topic on policy papers published Combined data for 2019 and 2020 suggests that the topics
in 2019 and 2020. (See the Appendix for a complete list of innovation and technology, international affairs and
of organizations included.) Those organizations are international security, and industry and regulation are
either U.S.-based or have a sizable presence in the United the main focuses of AI policy papers in the United States
States, and we grouped them into three categories: think (Figure 7.4.7). Fewer documents placed a primary focus
tanks, policy institutes and academia (27); civil society on topics related to AI ethics—such as ethics, equity and
organizations, associations and consortiums (9); and inclusion; privacy, safety and security; and justice and law
industry and consultancy (6). enforcement—which have largely been secondary topics.
AI policy papers are defined as research papers, research Moreover, topics bearing on the physical sciences, energy
reports, blog posts, and briefs that focus on a specific policy and environment, humanities, and democracy have
issue related to AI and provide clear recommendations received the least attention in U.S. AI policy papers.
Ethics
Physical Sciences
Appendix
Artificial Intelligence
Index Report 2021
APPENDIX 17 7
TA B L E O F C O N T E N T S
Artificial Intelligence
APPENDIX
Index Report 2021
Appendix
CHAPTER 1 Research and Development 179
APPENDIX 178
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
APPENDIX 179
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
AI Training Set
A training set of approximately 1,500 publications
defines the AI field. The set is only the EID (the Scopus
identifier of the underlying publications). Publications
can be searched and downloaded either from Scopus
directly or via the API.The training set is a set of
publications randomly selected from the initial seven
mio publications. After running the algorithm we verify
the results of the training set with the gold set (expert
hand-checked publications which are definitely AI).
APPENDIX 180
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
I C R O S O F T ACA D E M I C G R A P H :
M
M E T H O D O LO GY
Prepared by Zhihong Shen, Boya Xie, Chiyuan Huang, to conduct more web searches than a typical human
Chieh-Han Wu, and Kuansan Wang can complete. This helps disambiguate entities with
the same names. For example, for authors, MAG gets to
Source
additionally use all the CVs and institutional homepages
The Microsoft Academic Graph1 is a heterogeneous graph
on the web as signals to recognize and verify claims3.
containing scientific publication records and citation
MAG has found this approach to be superior to the results
relationships between those publications, as well as
of the best of the KDD Cup 2013 competition, which uses
authors, institutions, journals, conferences, and fields of
only data from within all publication records and Open
study. This graph is used to power experiences in Bing,
Researcher and Contributor Identifiers (ORCIDs).
Cortana, Word, and Microsoft Academic. The graph is
currently being updated on a weekly basis. Learn more Notes About the MAG Data
about MAG here. Conference Papers: After the contents and data sources
were scrutinized, it was determined that some of the 2020
Methodology
conference papers were not properly tagged with their
MAG Data Attribution: Each paper is counted exactly
conference venue. Many conference papers in the MAG
once. When a paper has multiple authors or regions,
system are under arXiv papers, but due to issues arising
the credit is equally distributed to the unique regions.
from some data sources (including delays in DBLP and
For example, if a paper has two authors from the
web form changes on the ACM website), they were possibly
United States, one from China, and one from the United
omitted as 2020 conference papers (ICML-PKDD, IROS, etc.).
Kingdom, then the United States, China, and the United
However, the top AI conferences (selected not in terms of
Kingdom each get one-third credit.
publication count, but rather considering both publication
Metrics: Total number of published papers (journal and citation count as well as community prestige) are
papers, conference papers, patents, repository2); total complete. In 2020, the top 20 conferences presented 103,000
number of citations of published papers. papers, which is 13.7% of all AI conference papers, and they
received 7.15 million citations collectively, contributing
Definition: The citation and reference count
47% of all citations received for all AI conference papers.
represents the number of respective metrics for AI
The number of 2020 conference publications is slightly
papers collected from all papers. For example, in
lower than in 2019. Data is known to be missing for ICCV
“OutAiPaperCitationCountryPairByYearConf.csv,” a row
and NAACL. About 100 Autonomous Agents and Multiagent
stating “China, United States, 2016, 14955” means that
Systems (AAMAS) conference papers are erroneously
China’s conference AI papers published in 2016 received
attributed to an eponymous journal.
14,955 citations from (all) U.S. papers indexed by MAG.
Unknown Countries for Journals and Conferences:
Curating the MAG Dataset and References: Generally
For the past 20 to 30 years, 30% of journal and conference
speaking, the robots sit on top of a Bing crawler to read
affiliation data lacks affiliation by country or region, due
everything from the web and have access to the entire
to errors in paper format, data source, and PDF parsing,
web index. As a result, MAG is able to program the robots
among others.
1 See “A Review of Microsoft Academic Services for Science of Science Studies” and “Microsoft Academic Graph: When Experts Are Not Enough” for more details.
2 Repository as a publication type in MAG refers to both preprints and postprints. In the AI domain, it predominantly comes from arXiv. See “Is Preprint the Future of Science? A Thirty Year Journey of
Online Preprint Services” for details.
3 See “Machine Verification for Paper and Author Claims” and “How Microsoft Academic Uses Knowledge to Address the Problem of Conflation/Disambiguation” for details.
APPENDIX 181
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
M I C R O S O F T ACA D E M I C G R A P H :
PAT E N T DATA C H A L L E N G E
As mentioned in the report, the patent data—especially jurisdictions. These multiple filings, while appear very
the affiliation information—is incomplete in the MAG different because the titles and inventor names are often
database. The reason for the low coverage is twofold. translated into local languages, are in fact the result of a
First, applications published by the patent offices often single invention. Raw patent counts therefore inflate the
identify the inventors by their residencies not affiliations. inventions in their respective domains. To remediate this
While patent applications often have the information issue, MAG uses the patent family ID feature to combine all
about the “assignees” of a patent, they do not necessarily filings with the original filing, which allows the database
mean the underlying inventions originate from the to count filings all around the world of the same origin
assignee institutions. Therefore, detected affiliations may only once.4 Conflating the multiple patent applications of
be inaccurate. In case a patent discloses the scholarly the same invention is not perfect, and over-conflations of
publications underlying the invention, MAG can infer patents are more noticeable in MAG than scholarly articles.
inventors’ affiliations through the scholarly publications.
These challenges raise questions about the reliability of
Second, to maximize intellectual property protection data on the share of AI patent publications by both region
around the globe, institutions typically file multiple and geographic area. Those charts are included below.
patent applications on the same invention under various
By Region
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
8%
AI Patent Publications (% of World Total)
6%
4%
3.2%, North America
2.6%, East Asia & Pacific
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
Figure 1.4.1
4 Read “Sharpening Insights into the Innovation Landscape with a New Approach to Patents” for more details.
APPENDIX 182
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
By Geographic Area
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
8%
AI Patent Publications (% of World Total)
6%
4%
3.2% US
2%
1.3% EU
0.4% China
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.4.2
Citation
AI PATENT CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
25%
AI Patent Citations (% of World Total)
20%
15%
10% 8.6% US
5% 2.1% EU
2.0% China
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.4.3
APPENDIX 183
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
M I C R O S O F T ACA D E M I C G R A P H :
MEASUREMENT CHALLENGES AND
A LT E R N AT I V E D E F I N I T I O N O F A I
As the AI Index team discussed in the paper “Measurement papers in the MAG database tagged with a field of study
in AI Policy: Opportunities and Challenges,” choosing how that is categorized in either the “artificial intelligence”
to define AI and correctly capture relevant bibliometric or the “machine learning” field of study as well as their
data remain challenging. Data in the main report is based subtopics in the MAG taxonomy.5 This is a more liberal
on a restricted definition of AI, adopted by MAG, that aligns definition than the one used by MAG, which considers only
with what has been used in previous AI Index reports. those publications tagged with “artificial intelligence”
One consequence is that such a definition excludes many as AI publications. For example, an application paper in
AI publications from venues considered to be core AI biology that uses ML techniques will be counted as an AI
venues. For example, only 25% of conference publications publication under the OECD definition, but not under the
in the 2020 AAAI conference are included in the original MAG definition unless the paper is specifically tagged in
conference dataset. the AI category.
To spur discussion on this important topic, this section Charts corresponding to those in the main text but using
presents the MAG data with an alternative definition of AI the OECD definition are presented below. The overall
used by the Organisation for Economic Co-operation and trends are very similar.
Development (OECD). OECD defines AI publications as
300
Number of AI Journal Publications (in Thousands)
250
200
150
100
50
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.1a
5 Read the OECD.AI Policy Observatory MAG methodological note for more details on the MAG-OECD definition of AI and “A Web-scale System for Scientific Knowledge Exploration” on the MAG Taxonomy.
APPENDIX 184
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
7.9%
8%
AI Journal Publications (% of All Publications)
6%
4%
2%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.1b
25%
23.0% East Asia & Pacific
AI Journal Publications (% of World Total)
20%
10%
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Figure 1.5.2
APPENDIX 185
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
20%
AI Journal Publications (% of World Total)
14.4%
China
15%
14.1%
US
10% 10.8%
EU
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.3
20%
AI Journal Publications (% of World Total)
15%
10% 8.4%
China
8.2%
US
5% 5.8%
EU
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.4
APPENDIX 186
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
100
Number of Publications (in Thousands)
80
60
40
20
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.5a
40%
AI Conference Publications (% of All Publications)
37.6%
35%
30%
25%
20%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.5b
APPENDIX 187
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
40%
AI Conference Publications (% of World Total)
30%
10%
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Figure 1.5.6
25%
AI Conference Publications (% of World Total)
18.9%
20% US
15.0%
China
15%
14.2%
EU
10%
5%
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.7
APPENDIX 188
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
14.3%
15% US
AI Conference Citation (% of World Total)
10%
4.2%
5% China
4.2%
EU
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.8
200
Number of AI Patent Publications (in Thousands)
150
100
50
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.9a
APPENDIX 189
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
5%
AI Patent Publications (% of All Publications)
4.6%
4%
3%
2%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.9b
10%
AI Patent Publications (% of World Total)
8%
6%
4%
3.3% North America
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
Figure 1.5.10
APPENDIX 190
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
10%
AI Patent Publications (% of World Total)
8%
6%
4%
3.2% US
2%
1.0% EU
0.4% China
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.11
25%
AI Patent Citation (% of World Total)
20%
15%
10%
5.6% US
5%
1.2% China
1.1% EU
0%
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Figure 1.5.12
APPENDIX 191
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
PA P E R S O N A R X I V
Prepared by Jim Entwood and Eleonora Presani arXiv is actively looking at ways to improve how it can
better support AI/ML researchers as the field grows
Source
and discovering content becomes more challenging.
arXiv.org is an online archive of research articles in
For example, there may be ways to create finer grained
the fields of physics, mathematics, computer science,
categories in arXiv for machine learning to help
quantitative biology, quantitative finance, statistics,
researchers in subfields share and find work more easily.
electrical engineering and systems science, and
The other rapidly expanding area is computer vision,
economics. arXiv is owned and operated by Cornell
where there is considerable overlap for ML applications
University. See more information on arXiv.org.
of computer vision.
Methodology
Nuance
Raw data for our analysis was provided by
• Categories are self-identified by authors—those shown
representatives at arXiv.org. The keywords we selected,
are selected as the “primary” category. Thus there is not
and their respective categories, are below:
a single automated categorization process. Additionally,
Artificial intelligence (cs.AI) the artificial intelligence or machine learning categories
Computation and language (cs.CL) may be categorized by other subfields or keywords.
Computer vision and pattern recognition (cs.CV)
• arXiv team members suggest that participation on
Machine learning (cs.LG)
arXiv can breed greater participation, meaning that an
Neural and evolutionary computing (cs.NE)
increase in a subcategory on arXiv could drive over-
Robotics (cs.RO)
indexed participation by certain communities.
Machine learning in stats (stats.ML)
APPENDIX 192
TA B L E O F C O N T E N T S
CHAPTER 1:
Artificial Intelligence
APPENDIX RESEARCH &
Index Report 2021 D E V E LO P M E N T
N E S TA G I T H U B S TA R S
Prepared by Joel Kliger and Juan Mateos-Garcia Source
GitHub: star-history (available at star history website)
Source
was used to retrieve the data.
Details can be found in the following publication:
Deep Learning, Deep Change? Mapping the Development Methodology
of the Artificial Intelligence General Purpose Technology The visual in the report shows the number of stars for
various GitHub repositories over time. The repositories
Methodology
include the following:
Deep learning papers were identified through a topic
apache/incubator-mxnet, BVLC/cafe, cafe2/cafe2, dmlc/
modeling analysis of the abstracts of arXiv papers in
mxnet, fchollet/keras, Microsoft/CNTK, pytorch/pytorch,
the CS (computer science) and stats.ML (statistics:
scikit-learn/scikit-learn, tensorflow/tensorflow, Theano/
machine learning category) arXiv categories. The data
Theano, Torch/Torch7.
was enriched with institutional affiliation and geographic
information from the Microsoft Academic Graph and Nuance
the Global Research Identifier. Nesta’s arXlive tool is The GitHub Archive currently does not provide a way
available here. to count when users remove a star from a repository.
Therefore, the reported data slightly overestimates the
Access the Code
number of stars. A comparison with the actual number
The code for data collection and processing can be found
of stars for the repositories on GitHub reveals that the
here; or, without the infrastructure overhead here.
numbers are fairly close and that the trends remain
unchanged.
APPENDIX 193
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
To highlight progress here in top-5 accuracy, we have taken scores from the following papers, without extra training data:
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Adversarial Examples Improve Image Recognition
OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks
Local Relation Networks for Image Recognition
Densely Connected Convolutional Networks
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Squeeze-and-Excitation Networks
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
MultiGrain: A Unified Image Embedding for Classes and Instances
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Billion-Scale Semi-Supervised Learning for Image Classification
GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
RandAugment: Practical Data Augmentation with No Separate Search
Fixing the Train-Rest Resolution Discrepancy
APPENDIX 194
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
To highlight progress here in top-5 accuracy, we have taken scores from the following papers, with extra training data:
Meta Pseudo Labels
Self-Training with Noisy Student Improves ImageNet Classification
Big Transfer (BiT): General Visual Representation Learning
ImageNet Classification with Deep Convolutional Neural Networks
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Xception: Deep Learning with Depthwise Separable Convolutions
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Self-training with Noisy Student Improves ImageNet Classification
To highlight progress here in top-1 accuracy, we have taken scores from the following papers, without extra training data:
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Adversarial Examples Improve Image Recognition
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Densely Connected Convolutional Networks
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Dual Path Networks
Res2Net: A New Multi-Scale Backbone Architecture
Billion-Scale Semi-Supervised Learning for Image Classification
Squeeze-and-Excitation Networks
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
MultiGrain: A Unified Image Embedding for Classes and Instances
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Billion-Scale Semi-Supervised Learning for Image Classification
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
RandAugment: Practical Data Augmentation with No Separate Search
Fixing the Train-Test Resolution Discrepancy
To highlight progress here in top-1 accuracy, we have taken scores from the following papers, without extra training data:
Meta Pseudo Labels
Sharpness-Aware Minimization for Efficiently Improving Generalization
An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Self-training with Noisy Student Improves ImageNet Classification
Big Transfer (BiT): General Visual Representation Learning
ImageNet Classification with Deep Convolutional Neural Networks
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Xception: Deep Learning with Depthwise Separable Convolutions
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Self-training with Noisy Student Improves ImageNet Classification
The estimate of human-level performance is from Russakovsky et al, 2015. Learn more about the LSVRC ImageNet
competition and the ImageNet data set.
APPENDIX 195
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
I M AG E N E T: T R A I N I N G T I M E I M AG E N E T: T R A I N I N G C O S T
Trends can also be observed by studying research papers Source
that discuss the time it takes to train ImageNet on any DAWNBench is a benchmark suite for end-to-end,
infrastructure. To gather this data, we looked at research deep-learning training and inference. Computation
papers from the past few years that tried to optimize time and cost are critical resources in building deep
for training ImageNet to a standard accuracy level while models, yet many existing benchmarks focus solely on
competing on reducing the overall training time. model accuracy. DAWNBench provides a reference set
of common deep-learning workloads for quantifying
Source
training time, training cost, inference latency, and
The data is sourced from MLPerf. Detailed data for runs for
inference cost across different optimization strategies,
specific years are available:
model architectures, software frameworks, clouds, and
2020: MLPerf Training v0.7 Results
hardware. More details available at DawnBench.
2019: MLPerf Training v0.6 Results
2018: MLPerf Training v0.5 Results
Note
The DawnBench data source has been deprecated for the
Notes
period after March 2020, and MLPerf is the most reliable
Data from MLPerf is available in cloud systems for rent.
and updated source for AI compute measurements.
Available On Premise systems contain only components
that are available for purchase. Preview systems must be
submittable as Available In Cloud or Available on Premise
C O C O : K E Y P O I N T D E T E CT I O N
The data for COCO keypoint detection data is sourced from
in the next submission round. Research, Development, or
COCO keypoints leaderboard.
Internal (RDI) contain experimental, in development, or
internal-use hardware or software. Each row in the results
table is a set of results produced by a single submitter us- C O C O : D E N S E P O S E E S T I M AT I O N
ing the same software stack and hardware platform. Each We gathered data from the CODALab 2020 challenge and
row contains the following information: read arXiv repository papers to build comprehensive data
on technical progress in this challenge. The detailed list of
Submitter: the organization that submitted the results papers and sources used in our survey include:
System: general system description DensePose: Dense Human Pose Estimation In the Wild
Processor and count: the type and number of CPUs used, if COCO-DensePose 2018 CodaLab
CPUs perform the majority of ML compute Parsing R-CNN for Instance-Level Human Analysis
Accelerator and count: the type and number of accel- Capture Dense: Markerless Motion Capture Meets Dense
erators used, if accelerators perform the majority of ML Pose Estimation
compute Slim DensePose: Thrifty Learning from Sparse Annotations
Software: the ML framework and primary ML hardware and Motion Cues
library used COCO-DensePose 2020 CodaLab
Benchmark results: training time to reach a specified tar- Transferring Dense Pose to Proximal Animal Classes
get quality, measured in minutes Making DensePose Fast and Light
Details: link to metadata for submission SimPose: Effectively Learning DensePose and Surface
Code: link to code for submission Normals of People from Simulated Data
Notes: arbitrary notes from the submitter
APPENDIX 196
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
AC T I V I T Y N E T: T E M P O R A L YO LO ( YO U O N LY LO O K O N C E )
LO CA L I Z AT I O N TA S K YOLO is a neural network model mainly used for the
In the challenge, there are three separate tasks, but detection of objects in images and in real-time videos.
they focus on the main problem of temporally localizing mAP (mean average precision) is a metric that is used
where activities happen in untrimmed videos from to measure the accuracy of object detectors. It is a
the ActivityNet benchmark. We have compiled several combination of precision and recall. mAP is the average of
attributes for the task of temporal localization at the the precision and recall calculated over a document. The
challenge over the last four rounds. Below is a link to performance of YOLO has increased gradually with the
the overall stats and trends for this task, as well as some development of new architectures and versions in past
detailed analysis (e.g., how has the performance for years. With the increase in size of model, its mean average
individual activity classes improved over the years? Which precision increases as well, with a corresponding decrease
are the hardest and easiest classes now? Which classes in FPS of the video.
have the most improvement over the years?). See the
We conducted a detailed survey of arXiv papers and
Performance Diagnosis (2020) tab for a detailed trends
GitHub repository to segment progress in YOLO across
update. Please see ActivityNet Statisticsin the public data
its various versions. Below are the references for original
folder for more details.
sources:
YOLOv1:
You Only Look Once: Unified, Real-Time Object Detection
YOLOv2:
YOLO9000: Better, Faster, Stronger
YOLO: Real-Time Object Detection
YOLOv3:
YOLOv3: An Incremental Improvement
Learning Spatial Fusion for Single-Shot Object Detection
GitHub: ultralytics/yolov3
YOLOv4:
YOLOv4: Optimal Speed and Accuracy of Object Detection
GitHub: AlexeyAB/darknet
YOLOv5:
GitHub: ultralytics/yolov5
PP-YOLO:
PP-YOLO: An Effective and Efficient Implementation of
Object Detector
POLY-YOLO:
Poly-YOLO: Higher Speed, More Precise Detection and
Instance Segmentation for YOLOV3
APPENDIX 197
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
V I S UA L Q U E S T I O N A N S W E R I N G PA P E R S W I T H C O D E :
( VQ A ) PA P E R A N D C O D E L I N K I N G
VQA accuracy data was provided by the VQA team. Learn We used paperswithcode (PWC) for referencing technical
more about VQA here. More details on VQA 2020 are progress where available. Learn more about PWC here and
available here.
see the public link here.
Methodology
Methodology
Given an image and a natural language question about the
For papers, we follow specific ML-related categories on
image, the task is to provide an accurate natural language
arxiv (see [1] below for the full list) and the major ML
answer. The challenge is hosted on the VQA Challenge
conferences (NeurIPS, ICML, ICLR, etc.). For code, we
website. The challenge is hosted on EvalAI. The challenge
follow GitHub repositories mentioning papers. We have
link is here.
good coverage of core ML topics but are missing some
The VQA v2.0 training, validation, and test sets, containing applications—for instance, applications of ML in medicine
more than 250,000 images and 1.1 million questions, or bioinformatics, which are usually in journals behind
are available on the download page. All questions are paywalls. For code, the dataset is fairly unbiased (as long
annotated with 10 concise, open-ended answers each. as the paper is freely available).
Annotations on the training and validation sets are
For tasks (e.g., “image classification”), the dataset has
publicly available.
annotated those on 1,600 state-of-the-art papers from the
VQA Challenge 2020 is the fifth edition of the VQA database, published in 2018 Q3.
Challenge. Results from previous versions of the VQA
For state-of-the-art tables (e.g., “image classification on
Challenge were announced at the VQA Challenge
ImageNet”), the data has been scraped from different
Workshop in CVPR 2019, CVPR 2018, CVPR 2017, and CVPR
sources (see the full list here), and a large number focusing
2016. More details about past challenges can be found
on CV and NLP were hand-annotated. A significant portion
here: VQA Challenge 2019, VQA Challenge 2018, VQA
of our data was contributed by users, and they have added
Challenge 2017, VQA Challenge 2016.
data based on their own preferences and interests. Arxiv
VQA had 10 humans answer each question. More details categories we follow:
about the VQA evaluation metric and human accuracy ARXIV_CATEGORIES = “cs.CV”, “cs.AI”, “cs.LG”, “cs.CL”, “cs.
can be found here (see Evaluation Code section) and NE”, “stat.ML”,”cs.IR”}
in sections three (“Answers”) and four (“Inter-Human
Process of Extracting Dataset at Scale
Agreement”) of the paper.
1) Follow various paper sources (as described above) for
See slide 56 for the progress graph in VQA in the 2020 new papers.
Challenge. The values corresponding to the progress graph 2) Conduct a number of predefined searches on GitHub
are available in a sheet. Here is the information about the (e.g., for READMEs containing links to arxiv).
teams that participated in the 2020 challenge and their 3) Extract GitHub links from papers.
accuracies. For more details about the teams, please refer 4) Extract paper links from GitHub.
to the VQA website. 5) Run validation tests to decide if links from 3) and 4) are
bona fide links or false positives.
6) Let the community fix any errors and/or add any missing
values.
APPENDIX 198
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
NIST FRVT
Source
There are two FRVT evaluation leaderboards available here: 1:1 Verification and 1:N Identification
The 1:1 and 1:N should be studied separately. The differences include algorithmic approaches, particularly fast search
algorithms are especially useful in 1:N whereas speed is not a factor in 1:1.
S U P E R G LU E
The SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about the SuperGLUE benchmark
are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks and evaluation metrics for SuperGLUE are:
V I S UA L C O M M O N S E N S E R E A S O N I N G ( VC R )
Technical progress for VCR is taken from the VCR leaderboard. VCR has two different subtasks:
• Question Answering (Q->A): A model is provided a question and has to pick the best answer out of four choices. Only
one of the four is correct.
• Answer Justification (QA->R): A model is provided a question, along with the correct answer, and it must justify it by
picking the best rationale among four choices.
The two parts with the Q->AR metrics are combined in which a model only gets a question right if it answers correctly
and picks the right rationale. Models are evaluated in terms of accuracy (%).
APPENDIX 199
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
VOXC E L E B B O O L E A N SAT I S F I A B I L I T Y
VoxCeleb is an audio-visual dataset consisting of short PROBLEM
clips of human speech, extracted from interview videos Analysis and text by Lars Kotthoff
uploaded to YouTube. VoxCeleb contains speech from
7,000-plus speakers spanning a wide range of ethnicities, Primary Source and Data Sets
accents, professions, and ages—amounting to over a The Boolean Satisfiability Problem (SAT) determines
million utterances (face-tracks are captured “in the wild,” whether there is an assignment of values to a set of
with background chatter, laughter, overlapping speech, Boolean variables joined by logical connectives that
pose variation, and different lighting conditions) recorded makes the logical formula it represents true. SAT was the
over a period of 2,000 hours (both audio and video). Each first problem to be proven NP-complete, and the first
segment is at least three seconds long. The data contains algorithms to solve it were developed in the 1960s. Many
an audio dataset based on celebrity voices, shorts, real-world problems, such as circuit design, automated
films, and conversational pieces (e.g., talk shows). The theorem proving, and scheduling, can be represented and
initial VoxCeleb 1 (100,000 utterances taken from 1,251 solved efficiently as SAT. The annual SAT competition is
celebrities on YouTube) was expanded to VoxCeleb 2 (1 designed to present a snapshot of the state-of-the-art and
million utterances from 6,112 celebrities). has been running for almost 20 years.
However, in earlier years of the challenge, top-1 and top-5 We took the top-ranked, median-ranked, and bottom-
scores were also reported. For top-1 score, the system is ranked solvers from each of the last five years (2016-2020)
correct if the target label is the class to which it assigns of the SAT competition. We ran all 15 solvers on all 400 SAT
the highest probability. For top-5 score, the system is instances from the main track of the 2020 competition. More
correct if the target label is one of the five predictions information on the competition, as well as the solvers and
with the highest probabilities. In both cases, the top score instances, is available at the SAT competition website.
is computed as the number of times a predicted label Results
matches the target label, divided by the number of data We ran each solver on each instance on the same
points evaluated. hardware, with a time limit of 5,000 CPU seconds per
The data is extracted from different years of the instance, and measured the time it took a solver to solve
submission challenges, including: an instance in CPU seconds. Ranked solvers always return
• 2 017: VoxCeleb: A Large-Scale Speaker Identification Dataset correct results, hence we do not consider correctness
• 2018: VoxCeleb2: Deep Speaker Recognition as a metric. Except for the 2020 competition solvers,
• 2 019: Voxceleb: Large-Scale Speaker Verification in the Wild we evaluated the performance of the SAT solvers on a
• 2020: Query ExpansionSystem for the VoxCeleb Speaker set of instances different from the set of instances they
Recognition Challenge 2020 competed on. Further, our hardware is different from what
was used for the SAT competition. The results we report
here will therefore differ from the exact results reported for
the respective SAT competitions.
APPENDIX 200
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
the Shapley value is used to distribute airport costs to its that logic is used to (attempt to) solve the problem. A
users, allocate funds to different marketing campaigns, key concern of ATP research is the development of more
and in machine learning, where it helps render complex powerful systems, capable of solving more difficult
black-box models more explainable. problems within the same resource limits. In order to
assess the merits of new techniques, sound empirical
In our context, it quantifies the contribution of a solver
evaluations of ATP systems are key.
to the state-of-the-art through the average performance
improvement it provides over a set of other solvers and over 2. Analysis
all subsets of solvers (Fréchette et al. (2016)). For a given set For the evaluation of ATP systems, there exists a large and
of solvers, we choose the respective best for each instance growing collection of problems called the TPTP problem
to solve. By including another solver and being able to library. The current release v7.4.0 (released June 10,
choose it, overall solving performance improves, with the 2020) contains 23,291 ATP problems, structured into 54
difference to the original set of solvers being the marginal topic domains (e.g., Set Theory, Software Verification,
contribution of the added solver. The average marginal Philosophy, etc.). Orthogonally, the TPTP is divided into
contribution to all sets of solvers is the Shapley value. Specialist Problem Classes (SPCs), each of which contains
problems with a specified set of logical, language, and
Quantifying the contribution of a solver through the
syntactic characteristics (e.g. first-order logic theorems
Shapley value compares solvers from earlier competitions
with some use of equality). The SPCs allow ATP system
to solvers in later competitions. This is often not a fair
developers to select problems and evaluate their
comparison, as later solvers are often improved versions
systems appropriately. Since its first release in 1993,
of earlier solvers, and the contribution of the solver to the
many researchers have used the TPTP as an appropriate
future state-of-the-art will always be low. The temporal
and convenient basis for ATP system evaluation. Over
Shapley value (Kotthoff et al. (2018)) solves this problem
the years, the TPTP has also increasingly been used as
by considering the time a particular solver was introduced
a conduit for ATP users to contribute samples of their
when quantifying its contribution to the state-of-the-art.
problems to ATP system developers. This exposes the
problems to ATP system developers, who can then
AU TO M AT E D T H E O R E M P R OV I N G improve their systems’ performances on the problems,
Analysis and text by Christian Suttner, Geoff Sutcliffe, and which completes a cycle to provide users with more
Raymond Perrault effective tools.
1. Motivation Associated with the TPTP is the TSTP solution library,
Automated Theorem Proving (ATP) (also referred to which maintains updated results from running all current
as Automated Deduction) is a subfield of automated versions of ATP systems (available to the maintainer) on
reasoning, concerned with the development and use of all the TPTP problems. One use of the TSTP is to compute
systems that automate sound reasoning: the derivation TPTP problem difficulty ratings: Easy problems, which are
of conclusions that follow inevitably from facts. ATP solved by all ATP systems, have a rating of 0.0; difficult
systems are at the heart of many computational tasks and problems, which are solved by some ATP systems, have
are used commercially, e.g., for integrated circuit design ratings between 0.0 and 1.0; unsolved problems, which are
and computer program verification. ATP problems are not solved by any ATP system, have a rating of 1.0. Note
typically solved by showing that a conjecture is or is not that the rating for a problem is not strictly decreasing, as
a logical consequence of a set of axioms. ATP problems different ATP systems and versions become available for
are encoded in a chosen logic, and an ATP system for populating the TSTP. The history of each TPTP problem’s
APPENDIX 201
TA B L E O F C O N T E N T S
C H A P T E R 2:
Artificial Intelligence
APPENDIX T E C H N I CA L
Index Report 2021 PERFORMANCE
ratings is saved with the problem, which makes it possible The table here in the public data folder shows the
to tell when the problem was first solved by any ATP breakdown of TPTP problems by content fields, as well
system (the point at which its rating dropped below as by SPCs used in the analysis. The totals are slightly
1.0). That information has been used here to obtain an larger than those shown in the analysis, as some
indication of progress in the field. problems were left out for technical reasons (no scores
available, problems revised over time, etc.).
The simplest way to measure progress takes a fixed set of
problems that has been available (and unchanged) in the
TPTP from some chosen initial TPTP release, and then for
the TPTP releases from then on, counts how many of the
problems had been solved from that release. The analysis
reports the fraction of problems solved for each release.
This simple approach is unambiguous, but it does not take
into account new problems that are added to the TPTP
after the initial release.
APPENDIX 202
TA B L E O F C O N T E N T S
Artificial Intelligence C H A P T E R 3:
APPENDIX
Index Report 2021 ECONOMY
CHAPTER 3: ECONOMY
LINKEDIN
Prepared by Mar Carpanelli, Ramanujam MV, and Nathan • Re-weight skill frequencies using a TF-IDF model to get
Williams the top 50 most representative skills in that entity. These
50 skills compose the “skill genome” of that entity.
Country Sample
• Compute the share of skills that belong to the AI skill
Included countries represent a select sample of eligible
group out of the top skills in the selected entity.
countries with at least 40% labor force coverage by
LinkedIn and at least 10 AI hires in any given month. Interpretation: The AI skill penetration rate signals the
China and India were included in this sample because of prevalence of AI skills across occupations, or the intensity
their increasing importance in the global economy, but with which LinkedIn members utilize AI skills in their
LinkedIn coverage in these countries does not reach 40% jobs. For example, the top 50 skills for the occupation of
of the workforce. Insights for these countries may not engineer are calculated based on the weighted frequency
provide as full a picture as other countries, and should be with which they appear in LinkedIn members’ profiles. If
interpreted accordingly. four of the skills that engineers possess belong to the AI
skill group, this measure indicates that the penetration of
Skills
AI skills is estimated to be 8% among engineers (e.g., 4/50).
LinkedIn members self-report their skills on their LinkedIn
profiles. Currently, more than 35,000 distinct, standardized Relative AI Skills Penetration
skills are identified by LinkedIn. These have been coded To allow for skills penetration comparisons across
and classified by taxonomists at LinkedIn into 249 skill countries, the skills genomes are calculated and a relevant
groupings, which are the skill groups represented in the benchmark is selected (e.g., global average). A ratio is then
dataset. The top skills that make up the AI skill grouping constructed between a country’s and the benchmark’s AI
are machine learning, natural language processing, data skills penetrations, controlling for occupations.
structures, artificial intelligence, computer vision, image
Interpretation: A country’s relative AI skills penetration
processing, deep learning, TensorFlow, Pandas (software),
of 1.5 indicates that AI skills are 1.5 times as frequent as in
and OpenCV, among others.
the benchmark, for an overlapping set of occupations.
Skill groupings are derived by expert taxonomists through
Global Comparison
a similarity-index methodology that measures skill
For cross-country comparison, we present the relative
composition at the industry level. Industries are classified
penetration rate of AI skills, measured as the sum of the
according to the ISIC 4 industry classification (Zhu et al.,
penetration of each AI skill across occupations in a given
2018).
country, divided by the average global penetration of AI
AI Skills Penetration skills across the overlapping occupations in a sample of
The aim of this indicator is to measure the intensity of AI countries.
skills in an entity (in a particular country, industry, gender,
Interpretation: A relative penetration rate of 2 means
etc.) through the following methodology:
that the average penetration of AI skills in that country
• Compute frequencies for all self-added skills by LinkedIn
is two times the global average across the same set of
members in a given entity (occupation, industry, etc.) in
occupations.
2015–2020.
APPENDIX 203
TA B L E O F C O N T E N T S
Artificial Intelligence C H A P T E R 3:
APPENDIX
Index Report 2021 ECONOMY
APPENDIX 204
TA B L E O F C O N T E N T S
Artificial Intelligence C H A P T E R 3:
APPENDIX
Index Report 2021 ECONOMY
it is found and is considered a duplicate and removed Language Processing, Natural Language Toolkit (NLTK),
in subsequent months. JOLTS data captures active Nearest Neighbor Algorithm, OpenNLP, Sentiment
postings: A posting appears in the data for every month Analysis/Opinion Mining, Speech Recognition, Text
that it is still actively posted, meaning the same posting Mining, Text to Speech (TTS), Tokenization, Word2Vec
can be counted in two or more consecutive months if it
Neural Networks: Caffe Deep Learning Framework,
has not been filled. To allow for apples-to-apples volume
Convolutional Neural Network (CNN), Deep Learning,
comparison in postings, the Burning Glass data needs
Deeplearning4j, Keras, Long Short-Term Memory (LSTM),
to be inflated to account for active postings, not only
MXNet, Neural Networks, Pybrain, Recurrent Neural
new postings. The number of postings from Burning
Network (RNN), TensorFlow
Glass can be inflated using the ratio of new jobs to active
jobs in Help Wanted OnLine™ (HWOL), a method used Machine Learning: AdaBoost algorithm, Boosting
in Carnevale, Jayasundera and Repnikov (2014). Based (Machine Learning), Chi Square Automatic Interaction
on this calculation, the share of jobs online as captured Detection (CHAID), Classification Algorithms, Clustering
by Burning Glass is roughly 85% of the jobs captured in Algorithms, Decision Trees, Dimensionality Reduction,
JOLTS in 2016. Google Cloud Machine Learning Platform, Gradient
boosting, H2O (software), Libsvm, Machine Learning,
The labor market demand captured by Burning Glass
Madlib, Mahout, Microsoft Cognitive Toolkit, MLPACK
data represents over 85% of the total labor demand.
(C++ library), Mlpy, Random Forests, Recommender
Jobs not posted online are usually in small businesses
Systems, Scikit-learn, Semi-Supervised Learning,
(the classic example being the “Help Wanted” sign in the
Supervised Learning (Machine Learning), Support
restaurant window) and union hiring halls.
Vector Machines (SVM), Semantic Driven Subtractive
Measuring Demand for AI Clustering Method (SDSCM), Torch (Machine Learning),
In order to measure the demand by employers of AI skills, Unsupervised Learning, Vowpal, Xgboost
Burning Glass uses its skills taxonomy of over 17,000
Robotics: Blue Prism, Electromechanical Systems,
skills. The list of AI skills from Burning Glass data are
Motion Planning, Motoman Robot Programming, Robot
shown below, with associated skill clusters. While some
Framework, Robotic Systems, Robot Operating System
skills are considered to be in the AI cluster specifically,
(ROS), Robot Programming, Servo Drives / Motors,
for the purposes of this report, all skills below were
Simultaneous Localization and Mapping (SLAM)
considered AI skills. A job posting was considered an AI
job if it requested one or more of these skills. Visual Image Recognition: Computer Vision, Image
Processing, Image Recognition, Machine Vision, Object
Artificial Intelligence: Expert System, IBM Watson,
Recognition
IPSoft Amelia, Ithink, Virtual Agents, Autonomous
Systems, Lidar, OpenCV, Path Planning, Remote Sensing
APPENDIX 205
TA B L E O F C O N T E N T S
Artificial Intelligence C H A P T E R 3:
APPENDIX
Index Report 2021 ECONOMY
APPENDIX 206
TA B L E O F C O N T E N T S
Artificial Intelligence C H A P T E R 3:
APPENDIX
Index Report 2021 ECONOMY
Methodology Source
The survey was conducted online and was in the field from Liquidnet provides sentiment data that predicts
June 9, 2020, to June 19, 2020, and garnered responses the market impact of central bank and corporate
from 2,395 participants representing the full range of communications. Learn more about Liquidnet here.
regions, industries, company sizes, functional specialties,
and tenures. Of those respondents, 1,151 said their
organizations had adopted AI in at least one function and
were asked questions about their organizations’ AI use.
To adjust for differences in response rates, the data are
weighted by the contribution of each respondent’s nation
to global GDP. McKinsey also conducted interviews with
executives between May and August 2020 about their
companies’ use of AI. All quotations from executives were
gathered during those interviews.
Note
Survey respondents are limited by their perception of their
organization’s AI adoption.
APPENDIX 207
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 4:
APPENDIX
Index Report 2021 A I E D U CAT I O N
CHAPTER 4: AI EDUCATION
C R A TAU L B E E S U R V E Y The CRA Taulbee Survey is sent only to doctoral
Prepared by Betsy Bizot (CRA senior research associate) departments of computer science, computer engineering,
and Stu Zweben (CRA survey chair, professor emeritus at and information science/systems. Historically, (a)
The Ohio State University) Taulbee covers 1/4 to 1/3 of total BS CS recipients in
the United States; (b) the percent of women earning
Source bachelor’s degrees is lower in the Taulbee schools than
Computing Research Association (CRA) members overall; and (c) Taulbee tracks the trends in overall CS
are 200-plus North American organizations active in production.
computing research: academic departments of computer
science and computer engineering; laboratories and Nuances
centers in industry, government, and academia; and • Of particular interest in PhD job market trends are
affiliated professional societies (AAAI, ACM, CACS/AIC, the metrics on the AI PhD area of specialization. The
IEEE Computer Society, SIAM USENIX). CRA’s mission categorization of specialty areas changed in 2008 and
is to enhance innovation by joining with industry, was clarified in 2016. From 2004-2007, AI and robotics
government, and academia to strengthen research and were grouped; from 2008-present, AI is separate; 2016
advanced education in computing. Learn more about clarified to respondents that AI includes ML.
CRA here. • Notes about the trends in new tenure-track hires
(overall and particularly at AAU schools): In the 2018
Methodology Taulbee Survey, for the first time, we asked how many
CRA Taulbee Survey gathers survey data during the fall new hires had come from the following sources: new
of each academic year by reaching out to over 200 PhD- PhD, postdoc, industry, and other academic. Results
granting departments. Details about the Taulbee Survey indicated that 29% of new assistant professors came
can be found here. Taulbee does not directly survey the from another academic institution.
students. The department identifies each new PhD’s area • Some may have been teaching or research faculty
of specialization as well as their type of employment. rather than tenure-track, but there is probably some
Data is collected from September to January of each movement between institutions, meaning the total
academic year for PhDs awarded in the previous number hired overstates the total who are actually new.
academic year. Results are published in May after data
collection closes. So the 2019 data points were newly
available last spring, and the numbers provided for 2020
will be available in May 2021.
APPENDIX 208
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 4:
APPENDIX
Index Report 2021 A I E D U CAT I O N
A I I N D E X E D U CAT I O N S U R V E Y
Prepared by Daniel Zhang (Stanford Institute for Human- • Practical Artificial Intelligence Models - Keywords:
Centered Artificial Intelligence) Adaptive learning, AI Application, Anomaly detection,
Artificial general intelligence, Artificial intelligence,
Methodology
Audio processing, Automated vehicle, Automatic
The survey was distributed to 73 universities online over
translation, Autonomous system, Autonomous vehicle,
three waves from November 2020 to January 2021 and
Business intelligence, Chatbot, Computational creativity,
completed by 18 universities, a 24.7% response rate. The
Computational linguistics, Computational neuroscience,
selection of universities is based on the World University
Computer vision, Control theory, Cyber physical steam,
Rankings 2021 and Emerging Economies University
Deep learning, Deep neural network, Expert system, Face
Rankings 2020 by The Times Higher Education.
recognition, Human-AI interaction, Image processing,
The 18 universities are: Image recognition, Inductive programming, Intelligence
• Belgium: Katholieke Universiteit Leuven software, Intelligent agent, Intelligent control, Intelligent
• Canada: McGill University software development, Intelligence system, Knowledge
• China: Shanghai Jiao Tong University, Tsinghua University representation and reasoning, Machine learning, Machine
• Germany: Ludwig Maximilian University of Munich, translation, Multi-agent system, Narrow artificial
Technical University of Munich intelligence, Natural language generation, Natural
• Russia: Higher School of Economics, Moscow Institute of language processing, Natural language understanding,
Physics and Technology Neural network, Pattern recognition, Predictive analysis,
• Switzerland: École Polytechnique Fédérale de Lausanne Recommender system, Reinforcement learning, Robot
• United Kingdom: University of Cambridge system, Robotics, Semantic web, Sentiment analysis,
• United States: California Institute of Technology, Service robot, Social robot, Sound synthesis, Speaker
Carnegie Mellon University (Department of Machine identification, Speech processing, Speech recognition,
Learning), Columbia University, Harvard University, Speech synthesis, Strong artificial intelligence,
Stanford University, University of Wisconsin–Madison, Supervised learning, Support vector machine, Swarm
University of Texas at Austin, Yale University intelligence, Text mining, Transfer learning, Unsupervised
learning, Voice recognition, Weak artificial intelligence
Key Definitions
(Adapted from: Joint Research Centre, European
• Major or a study program: a set of required and elective
Commission, p.68)
courses in an area of discipline—such as AI—that leads to
• AI Ethics - Keywords: Accountability, Consent,
a bachelor’s degree upon successful completion.
Contestability, Ethics, Equality, Explainability, Fairness,
• Course: a set of classes that require a minimum of 2.5
Non-discrimination, Privacy, Reliability, Safety, Security,
class hours (including lecture, lab, TA hours, etc.) per
Transparency, Trustworthy ai, Uncertainty, Well-being
week for at least 10 weeks in total. Multiple courses with
(Adapted from: Joint Research Centre, European
the same titles and numbers count as one course.
Commission, p.68)
APPENDIX 209
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 4:
APPENDIX
Index Report 2021 A I E D U CAT I O N
E U ACA D E M I C O F F E R I N G , J O I N T
R E S E A R C H C E N T E R, E U R O P E A N
COMMISSION
Prepared by Giuditta De-Prato, Montserrat López Cobo, are those with a strong focus in AI, e.g., “automation
and Riccardo Righi and computer vision” or “advanced computer science
(computational intelligence).” Broad programs target the
Source
addressed domain, but in a more generic way, usually
The Joint Research Centre (JRC) is the European
aiming at building wider profiles or making reference to
Commission’s science and knowledge service. The JRC
the domain in the framework of a program specialized in
employs scientists to carry out research in order to provide
a different discipline (e.g., biomedical engineering).
independent scientific advice and support to EU policy.
Learn more about JRC here. Due to some methodological improvements introduced
in this edition, namely the addition of new keywords, a
Methodology
strict comparison is not possible. Still, more than 90%
By means of text-mining techniques, the study identifies
of all detected programs in this edition are triggered by
AI-related education programs from the programs’
keywords present in the 2019 study.
descriptions present in JRC’s database. To query the
database, a list of domain-specific keywords is obtained The original source on which queries are performed is the
through a multistep methodology involving (i) selection Studyportals’ database, which is made up of over 207,000
of top keywords from AI-specific scientific journals; programs from 3,700 universities in over 120 countries.
(ii) extraction of representative terms of the industrial Studyportals collects information from institutions’
dimension of the technology; (iii) topic modeling; and (iv) websites, and their database is regularly updated. This
validation by experts. In this edition, the list of keywords source, although offering the widest coverage among
has been enlarged to better cover certain AI subdomains all those identified, still suffers from some lack of
and to expand to related transversal domains, such coverage, mostly because it only tracks English-language
as philosophy and ethics in AI. Then the keywords are programs. This poses a comparability issue between
grouped into categories, which are used to analyze the English-native-speaking countries and the rest, but also
content areas taught in the identified programs. The between countries with differing levels of incorporation
content areas used are adapted from the JRC report of English as a teaching language in higher education.
“Defining Artificial Intelligence: Towards an Operational Bachelor’s-level studies are expected to be more
Definition and Taxonomy of Artificial Intelligence,” affected by this fact, where the offer is mostly taught in
conducted in the context of AI Watch. a native language, unlike master’s, which attracts more
international audiences and faculties. As a consequence,
The education programs are classified into specialized
this study may be showing a partial picture of the level of
and broad, according to the depth with which they
inclusion of advanced digital skills in bachelor’s degree
address artificial intelligence. Specialized programs
programs.
APPENDIX 210
TA B L E O F C O N T E N T S
C H A P T E R 5 : E T H I CA L
Artificial Intelligence
APPENDIX CHALLENGES OF
Index Report 2021 A I A P P L I CAT I O N S
APPENDIX 211
TA B L E O F C O N T E N T S
C H A P T E R 5 : E T H I CA L
Artificial Intelligence
APPENDIX CHALLENGES OF
Index Report 2021 A I A P P L I CAT I O N S
ETHICS IN AI CONFERENCES
Prepared by Marcelo Prates, Pedro Avelar, and Luis C. The keywords chosen for the classical keywords category
Lamb were:
Cognition, Cognitive, Constraint Satisfaction, Game
Source
Theoretic, Game Theory, Heuristic Search, Knowledge
Prates, Marcelo, Pedro Avelar, Luis C. Lamb. 2018.
Representation, Learning, Logic, Logical, Multiagent,
On Quantifying and Understanding the Role of Ethics in
Natural Language, Optimization, Perception, Planning,
AI Research: A Historical Account of Flagship Conferences
Problem Solving, Reasoning, Robot, Robotics, Robots,
and Journals. September 21, 2018.
Scheduling, Uncertainty, and Vision.
Methodology
The curated trending keywords were:
The percent of keywords has a straightforward
Autonomous, Boltzmann Machine, Convolutional
interpretation: For each category (classical/trending/
Networks, Deep Learning, Deep Networks, Long Short
ethics), the number of papers for which the title (or
Term Memory, Machine Learning, Mapping, Navigation,
abstract, in the case of the AAAI and NeurIPS figures)
Neural, Neural Network, Reinforcement Learning,
contains at least one keyword match. The percentages do
Representation Learning, Robotics, Self Driving, Self-
not necessarily add up to 100% (e.g, classical/trending/
Driving, Sensing, Slam, Supervised/Unsupervised
ethics are not mutually exclusive). One can have a paper
Learning, and Unmanned.
with matches on all three categories.
The terms searched for were based on the issues exposed
To achieve a measure of how much Ethics in AI is
and identified in papers below, and also on the topics
discussed, ethics-related terms are searched for in the
called for discussion in the First AAAI/ACM Conference on
titles of papers in flagship AI, machine learning, and
AI, Ethics, and Society.
robotics conferences and journals.
J. Bossmann. Top 9 Ethical Issues in Artificial Intelligence.
The ethics keywords used were the following:
2016. World Economic Forum.
Accountability, Accountable, Employment, Ethic, Ethical,
Ethics, Fool, Fooled, Fooling, Humane, Humanity, Law, Emanuelle Burton, Judy Goldsmith, Sven Koenig,
Machine Bias, Moral, Morality, Privacy, Racism, Racist, Benjamin Kuipers, Nicholas Mattei, and Toby Walsh.
Responsibility, Rights, Secure, Security, Sentience, Ethical Considerations in Artificial Intelligence Courses. AI
Sentient, Society, Sustainability, Unemployment, and Magazine, 38(2):22–34, 2017.
Workforce.
The Royal Society Working Group, P. Donnelly, R.
The classical and trending keyword sets were compiled Browsword, Z. Gharamani, N. Griffiths, D. Hassabis, S.
from the areas in the most cited book on AI by Russell Hauert, H. Hauser, N. Jennings, N. Lawrence, S. Olhede,
and Norvig [2012] and from curating terms from the M. du Sautoy, Y.W. Teh, J. Thornton, C. Craig, N. McCarthy,
keywords that appeared most frequently in paper titles J. Montgomery, T. Hughes, F. Fourniol, S. Odell, W. Kay,
over time in the venues. T. McBride, N. Green, B. Gordon, A. Berditchevskaia, A.
Dearman, C. Dyer, F. McLaughlin, M. Lynch, G. Richardson,
C. Williams, and T. Simpson. Machine Learning: The
Power and Promise of Computers That Learn by Example.
The Royal Society, 2017.
APPENDIX 212
TA B L E O F C O N T E N T S
C H A P T E R 5 : E T H I CA L
Artificial Intelligence
APPENDIX CHALLENGES OF
Index Report 2021 A I A P P L I CAT I O N S
Codebase
The code and data are hosted in this GitHub repository.
APPENDIX 213
TA B L E O F C O N T E N T S
Artificial Intelligence CHAPTER 6:
APPENDIX
Index Report 2021 DIVERSITY IN AI
CHAPTER 6: DIVERSITY IN AI
LINKEDIN Global Comparison: By Gender
The relative AI skill penetration by country for gender
AI Skills Penetration
provides an in-depth decomposition of AI skills
The aim of this indicator is to measure the intensity of
penetration across female and male labor pools and
AI skills in an entity (in a particular country, industry,
sample countries.
gender, etc.) through the following methodology:
• Compute frequencies for all self-added skills by Interpretation: A country’s relative AI skill penetration
LinkedIn members in a given entity (occupation, rate of 2 for women means that the average penetration
industry, etc.) in 2015–2020. of AI skills among women in that country is two times
• Re-weight skill frequencies using a TF-IDF model to get the global average across the same set of occupations
the top 50 most representative skills in that entity. These among women. If, in the same country, the relative AI
50 skills compose the “skill genome” of that entity. skill penetration rate for men is 1.9, this indicates that
• Compute the share of skills that belong to the AI skill the average penetration of AI skills among women in
group out of the top skills in the selected entity. that country is 5% higher than that of men (calculated by
dividing 1.9 by 2 and then subtracting 1, or 2/1.9-1) for
Interpretation: The AI skill penetration rate signals the
the same set of occupations.
prevalence of AI skills across occupations, or the intensity
with which LinkedIn members utilize AI skills in their
jobs. For example, the top 50 skills for the occupation of
engineer are calculated based on the weighted frequency
with which they appear in LinkedIn members’ profiles. If
four of the skills that engineers possess belong to the AI
skill group, this measure indicates that the penetration of
AI skills is estimated to be 8% among engineers (e.g., 4/50).
APPENDIX 214
TA B L E O F C O N T E N T S
C H A P T E R 7:
Artificial Intelligence A I P O L I CY A N D
APPENDIX
Index Report 2021 N AT I O N A L S T R AT E G I E S
APPENDIX 215
TA B L E O F C O N T E N T S
C H A P T E R 7:
Artificial Intelligence A I P O L I CY A N D
APPENDIX
Index Report 2021 N AT I O N A L S T R AT E G I E S
APPENDIX 216
TA B L E O F C O N T E N T S
C H A P T E R 7:
Artificial Intelligence A I P O L I CY A N D
APPENDIX
Index Report 2021 N AT I O N A L S T R AT E G I E S
Institution, Carnegie Endowment for International Peace, security, national security, autonomous weapons
Cato Institute, Center for a New American Security, • Justice & Law Enforcement: civil justice, criminal justice,
Center for Strategic and International Studies, Council social justice, police, public safety, courts
on Foreign Relations, Georgetown Center for Security • Communications & Media: social media, disinformation,
and Emerging Technology (CSET), Harvard Belfer Center, media markets, deepfakes
Harvard Berkman Klein Center, Heritage Foundation, • Government & Public Administration: federal
Hudson Institute, MacroPolo, MIT Internet Policy Research government, state government, local government, public
Initiative, New America Foundation, NYU AI Now Institute, sector efficiency, public sector effectiveness, government
Princeton School of Public and International Affairs, RAND services, government benefits, government programs,
Corporation, Rockefeller Foundation, Stanford Institute public works, public transportation
for Human-Centered Artificial Intelligence (HAI), Stimson • Democracy: elections, rights, freedoms, liberties,
Center, Urban Institute, Wilson Center. personal freedoms
• Industry & Regulation: economy, antitrust, M&A,
Civil Society, Associations & Consortiums: Not-for profit
competition, finance, management, supply chain,
institutions including community-based organizations
telecom, economic regulation, technical standards,
and NGOs advocating for a range of societal issues. We
autonomous vehicle industry & regulation
included the following nine organizations: Algorithmic
• Innovation & Technology: advancements and
Justice League, Alliance for Artificial Intelligence in
improvements in AI technology, R&D, intellectual
Healthcare, Amnesty International, EFF, Future of Privacy
property, patents, entrepreneurship, innovation
Forum, Human Rights Watch, IJIS, Institute for Electrical
ecosystems, startups, computer science, engineering
and Electronics Engineers, Partnership on AI
• Education & Skills: early childhood, K-12, higher
Industry & Consultancy: Professional practices providing education, STEM, schools, classrooms, reskilling
expert advice to clients and large industry players. We • Workforce & Labor: labor supply and demand, talent,
included six prominent organizations in this space: Accenture, immigration, migration, personnel economics, future of
Bain & Co., BCG, Deloitte, Google AI, McKinsey & Company work
• Social & Behavioral Sciences: sociology, linguistics,
Methodology
anthropology, ethnic studies, demography, geography,
Each broad topic area is based on a collection of underlying
psychology, cognitive science
keywords that describes the content of the specific paper.
• Humanities: arts, music, literature, language,
We included 17 topics that represented the majority of
performance, theater, classics, history, philosophy,
discourse related to AI between 2019-2020. These topic
religion, cultural studies
areas and the associated keywords are listed below.
• Equity & Inclusion: biases, discrimination, gender,
• Health & Biological Sciences: medicine, healthcare
race, socioeconomic inequality, disabilities, vulnerable
systems, drug discovery, care, biomedical research,
populations
insurance, health behaviors, COVID-19, global health
• Privacy, Safety & Security: anonymity, GDPR,
• Physical Sciences: chemistry, physics, astronomy, earth
consumer protection, physical safety, human control,
science
cybersecurity, encryption, hacking
• Energy & Environment: Energy costs, climate change,
• Ethics: transparency, accountability, human
energy markets, pollution, conservation, oil & gas,
values, human rights, sustainability, explainability,
alternative energy
interpretability, decision-making norms
• International Affairs & International Security:
international relations, international trade, developing
countries, humanitarian assistance, warfare, regional
APPENDIX 217
TA B L E O F C O N T E N T S
Artificial Intelligence G LO BA L A I
APPENDIX
Index Report 2021 V I B R A N CY
GLOBAL AI VIBRANCY
OV E R V I E W
The tables below show the high-level pillar, sub-pillars, and indicators covered by the Global AI Vibrancy Tool. Each
sub-pillar is composed of individual indicators reported in the Global AI Vibrancy Codebook. There are 22 metrics in
total, with 14 metrics under Research and Development (R&D) pillar, 6 metrics under the Economy pillar, and 2 metrics
available under the Inclusion pillar specific to gender diversity. To aid data-driven decision-making to design national
policy strategies, the Global AI Vibrancy is available as a web tool.
R&D
SUB-PILLAR VARIABLE
Journal Publications > Deep Learning Number of Deep Learning papers per capita
ECONOMY INCLUSION
SUB-PILLAR VARIABLE SUB-PILLAR VARIABLE
APPENDIX 218
TA B L E O F C O N T E N T S
Artificial Intelligence G LO BA L A I
APPENDIX
Index Report 2021 V I B R A N CY
The webtool allows users to adjust weights to each CONSTRU CTION OF THE GLO BA L A I
metric based on their individual preference. The default V IBRANCY: COM POSITE M E ASU R E
settings of the tool allow the user to select between three Source
weighting options: The data is collected by AI Index using diverse datasets
that are referenced in the 2020 AI Index Report chapters.
All weights to midpoint
This button assigns equal weights to all indicators. Methodology
Step 1: Obtain, harmonize, and integrate data on
Only absolute metrics individual attributes across countries and time.
This button assigns maximum weights to absolute Step 2: Use Min-Max Scalar to normalize each country-year
metrics. Per capita metrics are not considered. specific indicator between 0-100.
Step 3: Take arithmetic Mean per country-indicator for a
Only per capita metrics
given year.
This button assigns maximum weights to per capita
Step 4: Build modular weighted for available pillars and
metrics. Absolute metrics are not considered.
individual indicators.
The user can adjust the weights to each metric based on Aggregate Measure
their preference. The charts automatically update when The AI Vibrancy Composite Index can be expressed in the
any weight is changed. following equation:
APPENDIX 219
TA B L E O F C O N T E N T S
Artificial Intelligence G LO BA L A I
APPENDIX
Index Report 2021 V I B R A N CY
R E S E A R C H A N D D E V E LO P M E N T I N D I CATO R S
ID PILLAR SUB-PILLAR NAME DEFINITION SOURCE
Number of AI
Research and Conference Total count of published AI conference papers Microsoft Academic
1 conference
Development Publications attributed to institutions in the given country. Graph (MAG)
papers*
Research and Journal Number of AI Total count of published AI journal papers attributed to Microsoft Academic
5
Development Publications journal papers* institutions in the given country. Graph (MAG)
Research and Journal Number of AI Total count of AI journal citations attributed to Microsoft Academic
7
Development Publications journal citations* institutions in the given country. Graph (MAG)
Research and Innovation > Number of AI Total count of published AI patents attributed to Microsoft Academic
9
Development Patents patents* institutions in the given country. Graph (MAG)
Research and Innovation > Number of AI Total count of published AI patents citations attributed Microsoft Academic
11
Development Patents patent citations* to institutions of originating patent filing. Graph (MAG)
APPENDIX 220
TA B L E O F C O N T E N T S
Artificial Intelligence G LO BA L A I
APPENDIX
Index Report 2021 V I B R A N CY
E C O N O M Y I N D I CATO R S
ID PILLAR SUB-PILLAR NAME DEFINITION SOURCE
Total Amount of Total amount of private investment funding received for Crunchbase, CapIQ,
17 Economy Investment
Funding* AI startups (nominal USD). NetBase Quid
Number of
Total number of AI companies founded in the given Crunchbase, CapIQ,
19 Economy Investment companies
country. NetBase Quid
funded*
I N C LU S I O N I N D I CATO R S
ID PILLAR SUB-PILLAR NAME DEFINITION SOURCE
Number of unique
Gender Number of unique AI occupations (or job titles) with LinkedIn Economic
22 Inclusion AI occupations
Diversity high AI skill penetration for females in a given country. Graph
(job titles), female
APPENDIX 221
TA B L E O F C O N T E N T S
Artificial
Intelligence
Index Report 2021