Research Report

JAMES RYSEFF, BRANDON DE BRUHL, SYDNE J. NEWBERRY

The Root Causes of Failure

for Artificial Intelligence
Projects and How They
Can Succeed
Avoiding the Anti-Patterns of AI

A
rtificial intelligence (AI) is widely recognized as technology with the potential to have a
transformative effect on organizations.1 Although AI was once reserved for advanced tech-
nology companies with the ability to hire top talent and spend millions of dollars, all types
of organizations are adopting AI today. Private-sector investment in AI increased 18-fold
from 2013 to 2022,2 and one survey found that 58 percent of midsize corporations3 had deployed
at least one AI model to production.4 Similarly, the U.S. Department of Defense (DoD) is spending
$1.8 billion each year on military applications for AI, and DoD leaders have identified AI as one of
the most crucial technologies to the future of warfare.5
AI is already making impacts across a wide variety of industries. Pharmaceutical companies are
using it to accelerate the pace and success rate of drug development.6 Retailers, such as Walmart, are
deploying AI for predictive analytics so that they know when to restock inventory and how to optimize
their end-to-end supply chains.7 Finally, in the defense realm, AI is piloting fighter jets,8 detecting
enemy submarines,9 and improving commanders’ awareness of the battlefield.10 These examples dem-
onstrate the relevance of AI to organizations in a variety of industries and for a variety of use cases.
However, despite the promise and hype around AI, many organizations are struggling to
deliver working AI applications. One survey found that only 14 percent of organizations responded
that they were fully ready to adopt AI, even though 84 percent of business leaders reported that
they believe that AI will have a significant impact on their business.11 Managers and directors find
themselves under enormous pressure to do something—anything—with AI to demonstrate to their
superiors that they are keeping up with the rapid advance of technology.12 But too many managers
have little understanding of how to translate this desire into action. By some estimates, more than
80 percent of AI projects fail.13 This is twice the already-high rate of failure in corporate information
technology (IT) projects that do not involve AI.14
SUMMARY
Background
Although leaders widely recognize the importance of artificial intelligence (AI), successfully implementing AI
projects remains a serious challenge.a According to one survey, 84 percent of business leaders responded that
they believe that AI will have a significant impact on their business, and 97 percent of business leaders reported
that the urgency to deploy AI-powered technologies has increased.b Despite this, the same survey found that
only 14 percent of organizations responded that they were fully ready to integrate AI into their businesses.

By some estimates, more than 80 percent of AI projects fail—twice the rate of failure for information technol-
ogy projects that do not involve AI.c Thus, understanding how to translate AI’s enormous potential into concrete
results remains an urgent challenge. In this report, we document lessons learned from those who have already
applied AI/ML so that U.S. Department of Defense leadership and others can avoid these failures or mitigate
risks in their planning.

Approach
To investigate why AI projects fail, we interviewed 65 experienced data scientists and engineers. Participants
had at least five years of experience building AI/ML models in industry or academia. We selected participants
across a variety of company sizes and industries to ensure that these findings would be broadly representative.
The output of these interviews is summarized in this analysis.

Takeaways
Our interviews highlighted five leading root causes of the failure of AI projects. First, industry stakeholders often
misunderstand—or miscommunicate—what problem needs to be solved using AI. Too often, trained AI models
are deployed that have been optimized for the wrong metrics or do not fit into the overall business workflow and
context. Second, many AI projects fail because the organization lacks the necessary data to adequately train
an effective AI model. Third, in some cases, AI projects fail because the organization focuses more on using the
latest and greatest technology than on solving real problems for its intended users. Fourth, organizations might
not have adequate infrastructure to manage their data and deploy completed AI models, which increases the
likelihood of project failure. Finally, in some cases, AI projects fail because the technology is applied to prob-
lems that are too difficult for AI to solve. AI is not a magic wand that can make any challenging problem disap-
pear; in some cases, even the most advanced AI models cannot automate away a difficult task.

Industry Recommendations
To overcome these issues, leaders should consider these five principles for success in AI projects:

• Ensure that technical staff understand the project purpose and domain context: Misunderstandings and
miscommunications about the intent and purpose of the project are the most common reasons for AI proj-
ect failure. Ensuring effective interactions between the technologists and the business experts can be the
difference between success and failure for an AI project.
• Choose enduring problems: AI projects require time and patience to complete. Before they begin any AI
project, leaders should be prepared to commit each product team to solving a specific problem for at
least a year. If an AI project is not worth such a long-term commitment, it most likely is not worth commit-
ting to at all.
• Focus on the problem, not the technology: Successful projects are laser-focused on the problem to be
solved, not the technology used to solve it. Chasing the latest and greatest advances in AI for their own
sake is one of the most frequent pathways to failure.

2
 • Invest in infrastructure: Up-front investments in infrastructure to support data governance and model
deployment can substantially reduce the time required to complete AI projects and can increase the volume
of high-quality data available to train effective AI models.
• Understand AI’s limitations: Despite all the hype around AI as a technology, AI still has technical limitations
that cannot always be overcome. When considering a potential AI project, leaders need to include technical
experts to assess the project’s feasibility.

Academia Recommendations
To overcome the issues described by our academic interviewees, leaders should consider these two
recommendations:

• Overcome data-collection barriers through partnerships with government: Partnerships between

academia and government agencies could give researchers access to data of the provenance needed for
academic research. The federal government should expand its investment in such programs as Data.gov
(the U.S. government’s open data site) and seek to increase the number of datasets available for research.
• Expand doctoral programs in data science for practitioners: Newer academics often feel pressure to
focus on research that leads to career success as opposed to research that has the most potential to solve
important problems. Computer science and data science program leaders should learn from disciplines,
such as international relations, in which practitioner doctoral programs often exist side by side at even the
top-ranked universities to provide pathways for the most-advanced researchers to apply their findings to
urgent problems.

a For this project, we focused on the machine learning (ML) branch of AI because that is the technology underpinning most
business applications of AI today. This includes AI models trained using supervised learning, unsupervised learning, or
re­inforcement learning approaches and large language models (LLMs). Projects that simply used pretrained LLMs (some-
times known as prompt engineering) were not included in the scope of this work.
b Cisco AI Readiness Index.
c Kahn, “Want Your Company’s AI Project to Succeed?”

The purpose of this exploratory analysis is to ture, and place in the overall organization). These two
help leaders and managers within all types of orga- elements enable organizations and AI tools to work
nizations who are struggling to understand how together to solve pressing business problems.16
to execute AI projects in their organization avoid IT-type projects can fail for many reasons not
some of the most common reasons for AI project related to the technology itself. For example, projects
failures. To do so, we interviewed 65 experienced AI can fail because of process failures (i.e., flaws in the
engineers and researchers across a variety of com- way the project is executed), interaction failures (i.e.,
panies and industries, as well as academia. From problems with how humans interact with the tech-
these interviews, we identified the most frequently nology), or expectation failures (i.e., a misalignment
reported anti-patterns of AI—common responses in the anticipated value of the project).17 Breakdowns
to recurring problems that are typically ineffective in any component could result in a project failure,
or even counterproductive.15 We hope to help orga- which results in increased costs for the sponsoring
nizations avoid making these common mistakes enterprise. There is a large body of literature on how
and to provide leaders and managers endeavoring IT projects fail. However, AI seems to have different
to understand AI with practical advice to help them project characteristics, such as costly labor and capi-
get started. tal requirements and high algorithm complexity, that
AI projects have two components: the technology make them unlike a traditional information system.18
as a platform (i.e., the development, use, and deploy- The high-profile nature of AI may increase the desire
ment of AI to complete some set of business tasks) and for stakeholders to better understand what drives the
the organization of the project (i.e., the process, struc- risk of IT projects related to AI.

3
 Most prior work on this topic has taken one of ended questions of experienced data scientists and
two forms. In some cases, an individual data scien- ML engineers allowed us to discover what these
tist or manager discusses their personal experiences professionals believe are the greatest problems and
and beliefs about what causes AI projects to fail.19 In challenges when attempting to execute AI projects.
other cases, consulting firms conduct a widespread However, because the majority of our interviewees
survey of IT leaders to discuss their experiences were nonmanagerial engineers instead of business
with AI.20 For example, McKinsey has conducted executives, the results may disproportionately reflect
an annual survey about AI for several years.21 Addi- the perspective of individuals who do not hold lead-
tionally, one study conducted a systematic literature ership positions. Thus, the results may be skewed
review and interviews with six experts to explore the toward identifying leadership failures.
factors that might cause general AI projects to fail.22
Our study differs from this prior work in several
Industry Participants
ways. First, we focus on the perspective of the indi-
viduals building AI applications as opposed to the We identified potential industry participants using
business leaders of the organization. A bottom-up the LinkedIn Recruiter tool and LinkedIn InMail
approach allows us to discuss why AI projects fail messages. Potential participants had at least five
from the point of view of the people who intimately years of AI/ML experience in industry and job titles
understand the specifics of the technology. Second, that indicated that they were either an individual
we conducted semistructured interviews as opposed contributor or a manager in the data science or ML
to relying on multiple-choice or short-answer survey engineering technical disciplines.23 We selected
questions. Although the burden of conducting participants to represent a variety of experiences
interviews means that the sample size of this study and backgrounds. In particular, we selected par-
is smaller compared with those of multiple-choice ticipants from different company sizes (start-ups,
survey studies, this approach allowed us to explore large companies, and medium-sized companies) and
the issues raised in greater nuance and depth. Finally, industries (technology, health care, finance, retail,
we conducted substantially more semistructured consulting, and others). Industry participants were
interviews with experts compared with prior authors offered a $100 honorarium for agreeing to take part
who took this approach. in a 45-minute interview.
A total of 379 potential industry candidates were
identified and contacted. Of these, 50 individuals
Methods ultimately participated in an interview, represent-
To gather data for this report, we conducted semi­ ing more than 50 unique organizations.24 Fourteen
structured interviews with experienced AI practitio- individuals sent a message declining to participate
ners in both industry and academia. During these in the study; these individuals were removed from
interviews, we defined the failure of an AI project as a the candidate pool and had no further contact from
project that was perceived to be a failure by the orga- the study team.25 Table 1 illustrates the percentages
nization. We included both technical failures and of potential candidates who either participated or
business failures within this definition. Each inter- declined to participate in the study.
viewee was asked to discuss the types of failures that Industry interviews used a consistent battery
they perceived to be the most frequent or impactful of questions, which is provided in Appendix A.
and what they believed the root causes of these fail- All interviews were conducted with a promise of
ures were. We then identified common root causes ­anonymity to ensure that participants felt free to
based on the interview responses. The interviews speak candidly about their experiences.
were conducted between August and December 2023.
The approach taken in this report has strengths
and weaknesses. Conducting interviews with open-

4
Academia Participants TABLE 1
Industry Candidate Response Rates
We conducted 15 interviews of academics drawn
from convenience samples during conferences and Candidate
Indicators Pool Accepted Declined
from individuals known to the research team. These
interviews ranged across school types (e.g., engi- Number of 379 50 14
candidates
neering programs and business schools) and degree
levels (e.g., tenure-track researcher, non–tenure-track Percentage 100 13.2 3.7

researcher, graduate student, and undergraduate

or research assistant). These interviews used a con- TABLE 2
sistent battery of questions, which is presented in Academic Candidate Response Rates
Appendix B. Our interviews were conducted with
Candidate
the promise of anonymity to allow non–tenure-track Indicators Pool Accepted Declined
academic researchers and nonresearcher engineers
Number of 37 15 22
who support the research efforts to have an opportu- candidates
nity to speak without attribution. Table 2 illustrates
Percentage 100 40.5 59.5
the academic candidate response rates.

failures caused by fundamental limitations in what

Findings from Industry AI can actually achieve. While these failure patterns
were cited less frequently than the two dominant root
Interviews
causes, they each were cited by a one-quarter to one-
Across all of the interviews conducted with experi- third of the interview participants.
enced AI practitioners from industry, five dominant
root causes emerged describing why AI projects
fail. Overall, interviewees expressed that the most Leadership-Driven Failures
common root cause of failure was the business More than any other type of issue, our interviewees
leader­ship of the organization misunderstanding noted that failures driven by the decisions and expec-
how to set the project on a pathway to success. Our tations of the organization’s business leadership were
interviewees also noted that these types of failures far and away the most frequent causes of project fail-
had the most impact on the ultimate outcome of the ure. Eighty-four percent of our interviewees cited one
project compared with the other root causes of fail- or more of these root causes as the primary reason
ure they discussed. that AI projects would fail. These leadership-driven
The other notable root cause of failure identified failures took several forms.
by interviewees was limitations in the quality and
utility of data available to train the AI models. These Optimizing for the Wrong Business Problem
two root causes were cited spontaneously by more
First, all too often, leadership instructs the data sci-
than one-half of the interviewees as the primary rea-
ence team to solve the wrong problem with AI. This
sons that AI projects failed or underperformed.
results in the data science team working hard for
In addition to the most frequent failure patterns
months to deliver a trained AI model that makes
cited, three other root causes were noted by a mean-
little impact on the business or organization. In
ingful number of interviewees.26 First, some inter-
many cases, this is due to a communication break-
viewees noted the lack of investment in infrastruc-
down between the data science team and the leaders
ture to empower the team. Second, some interviewees
of the organization.
discussed the difference between the top-down fail-
Few business leaders have a background in data
ures caused by leadership and the bottom-up failures
science; consequently, the objectives they set need to
caused by individual contributors on the data science
be translated by the technical staff into goals that can
team. Finally, some interviewees discussed project

5
be achieved by a trained AI model. In failed projects, that demand complex solutions. Regardless of the
either the business leadership does not make them- cause, while these types of projects might succeed in
selves available to discuss whether the choices made a narrow sense, they fail in effect because they were
by the technical team align with their intent, or they never necessary in the first place.
do not realize that the metrics measuring the success
of the AI model do not truly represent the metrics of Overconfidence in Artificial Intelligence
success for its intended purpose. For example, busi- Additionally, many senior leaders have inflated
ness leaders may say that they need an ML algorithm expectations of what AI can be expected to achieve.
that tells them the price to set for a product—but The rapid advancements and impressive achieve-
what they actually need is the price that gives them ments of AI models have generated a wave of hype
the greatest profit margin instead of the price that about the technology. Pitches from salespeople and
sells the most items. The data science team lacks this presentations by AI researchers add to the perception
business context and therefore might make the wrong that AI can easily achieve almost anything. In ­reality,
assumptions. These kinds of errors often become optimizing an AI model for an organization’s use
obvious only after the data science team delivers a case can be more difficult than these presentations
completed AI model and attempts to integrate it into make it appear. AI models developed by academic
day-to-day business operations. researchers might not work effectively for all of the
peculiarities of an organization’s business. Many
Using Artificial Intelligence to Solve Simple business leaders also do not realize that AI algo-
Problems rithms are inherently probabilistic: Every AI model
In other cases, business leaders demand that the tech- incorporates some degree of randomness and uncer-
nical team apply ML to a problem that does not truly tainty. Business leaders who expect repeatability and
require it. Not every problem is complex enough certainty can be disappointed when the model fails
to require an ML solution: As one interviewee to live up to their expectations, leading them to lose
explained, his teams would sometimes be instructed faith in the AI product and in the data science team.
to apply AI techniques to datasets with a handful of
dominant characteristics or patterns that could have Underestimating the Time Commitment
quickly been captured by a few simple if-then rules. Needed
This mismatch can happen for different reasons. In Finally, many interviewees (14 of 50) reported finding
some cases, leaders understand AI only as a buzz- that senior leaders often underestimated the amount
word and do not realize that simpler and cheaper of time that it would take to train an AI model that
solutions are available. In other cases, senior leaders was effective at solving their business problems.
who are far removed from the implementation details Even when an off-the-shelf AI model is available, it
demand the use of AI because they are confident has not been trained on an organization’s data and
that their business area must have complex problems thus it may not be immediately effective in solving
the specific business problems. Many leaders are not
prepared for the time and cost of acquiring, clean-
ing, and exploring their organization’s data. They
Many leaders are not expect AI projects to take weeks instead of months
to complete, and they wonder why the data science
prepared for the time team cannot quickly replicate the fantastic achieve-
ments they hear about every day. Even worse, in
and cost of acquiring, some organizations, senior leaders rapidly switch

cleaning, and exploring their priorities every few weeks or months. In these
cases, projects that are in progress can be discarded
their organization’s data. before they have the opportunity to demonstrate real

6
results, or completed projects can be ignored because Too Few Data Engineers
they no longer address what leadership views as the The lack of prestige associated with data engineer-
most important priorities of the company. Even when ing acts as an additional barrier: One interviewee
the project is successful, leaders may direct the team referred to data engineers as “the plumbers of data
to move on prematurely. As one interviewee put it, science.”29 Data engineers do the hard work of
“Often, models are delivered as 50 percent of what designing and maintaining the infrastructure that
they could have been.”27 ingests, cleans, and transforms data into a format
suitable for data scientists to train models on. Despite
Bottom-Up–Driven Failures this, often the data scientists training the AI models
are seen as doing “the real AI work,” while data
In contrast to the top-down failure patterns driven engineering is looked down on as a menial task.30
by the organization’s business leadership, many inter- The goal for many data engineers is to grow their
viewees (16 of 50) noted a different type of failure skills and transition into the role of data scientist;
pattern driven by the data scientists on the team. consequently, some organizations face high turnover
Technical staff often enjoy pushing the boundaries of rates in the data engineering group. Even worse,
the possible and learning new tools and techniques. these individuals take all of their knowledge about
Consequently, they often look for opportunities to the organization’s data and infrastructure when they
try out newly developed models or frameworks even leave. In organizations that lack effective documen-
when older, more-established tools might be a better tation, the loss of a data engineer might mean that
fit for the business use case. Individual engineers and no one knows which datasets are reliable or how the
data scientists also have a strong incentive to build meaning of a dataset might have shifted over time.
up their experience using the latest technological Painstakingly rediscovering that knowledge increases
advancements because these skills are highly desired the cost and time required to complete an AI project,
in the hiring market. AI projects often fail when they which increases the likelihood that leadership will
focus on the technology being employed instead of lose interest and abandon it.
focusing on solving real problems for their intended
end users. While it is important for an organization Lack of Suitable Data
to experiment with new technologies and provide its
Additionally, in some cases, organizations lack the
technical staff with opportunities to improve their
right kind of data to train AI models. This failure
skill sets, this should be a conscious choice balanced
pattern is particularly common when the business
against the other objectives of the organization.
is applying AI for the first time or to a new domain.
Interviewees noted that business leaders often
Data-Driven Failures would be surprised to learn that their organization
lacked sufficient data to train AI algorithms. As one
After leadership-driven failures, interviewees identi-
interviewee put it, “They think they have great data
fied data-driven failures as the second most common
because they get weekly sales reports, but they don’t
reason that AI projects end in failure. These difficul-
realize the data they have currently may not meet
ties manifested in a number of ways.
its new purpose.”31 In many cases, legacy datasets
Many interviewees (30 of 50) discussed persistent
were intended to preserve data for compliance or
issues with data quality. One interviewee noted,
logging purposes. Unfortunately, structuring data
80 percent of AI is the dirty work of data engi- for analysis can be quite different: It often requires
neering. You need good people doing the dirty considerable context about why things happened
work—otherwise their mistakes poison the
as opposed to simply what happened. For example,
algorithms. The challenge is, how do we con-
an e-commerce website might have logged what
vince good people to do boring work?28
links users click on—but not a full list of what items
appeared on the screen when the user selected one

7
or what search query led the user to see that item in Failures Due to Underinvestment in
the first place. This may mean that different fields Infrastructure
need to be preserved, or different levels of granular-
ity and quality may be necessary. Thus, even if an One contributing factor to the numerous difficulties
organization has a large quantity of historical data, that organizations face in making their data ready
that data may not be sufficient to train an effective for AI is the lack of investment in supporting infra-
AI algorithm. structure. Data engineering professionals need time
to build up pipelines that can automatically clean
data and continuously deliver fresh data to deployed
Unbalanced Data
AI models. Infrastructure investments ensure that
A related problem occurs when organizations have these pipelines are automatically monitored to
large quantities of data, but the data are unbalanced. determine whether a data source changes formats or
For example, in health care applications, datasets fails to arrive promptly. Organizations that quickly
may contain a large number of instances where a move from prototype to prototype often find that
medical test correctly confirmed the absence of a rare they are completely blind to failures that arise after
cancer but only a handful of cases where the cancer the AI model has been completed and deployed.
was actually present. These conditions raise the risk Robust infrastructure allows the engineering team
of overfitting the data: The algorithm might exces- to detect when a deployed model needs mainte-
sively correlate the detection of these rare conditions nance, which deployed models most urgently need
with random, unrelated data characteristics from the maintenance, and what kind of maintenance action
handful of known cases. Gathering enough data to is required for each.
detect rare real-world events requires time, money, Additionally, investments in operations infra-
and patience. structure ensure that AI models can be more quickly
and easily deployed to production. Interviewees rec-
Lack of Domain Understanding ommended investing in hiring ML engineers who
Finally, several interviewees (10 of 50) noted that have the specialized skills to build this infrastructure
their lack of domain understanding could cause and speed up model deployments. Some interviewees
the failure of AI projects. Data scientists are rarely noted that they had observed cases where AI models
experts in the topics for which they are building could not be deployed from test environments to
their models: They require the assistance of subject- production environments because the production
matter experts who can explain what the elements environments were incompatible with the require-
in the dataset mean and which ones are—and are ments of the model. In other cases, interviewees
not—important or might be unreliable. For example, noted significant delays in deploying their completed
a particular data field might appear at first glance models to end users because of a lack of robust infra-
to be highly relevant for training the AI model, structure to automate the deployments. Ultimately,
but the data might be unreliable because they were developing effective AI products requires more than
manually entered by users who had little incentive just a data science team. Investing in data engineers
to ensure that the data were of high quality. Unfor- and ML engineers can substantially shorten the time
tunately, in some cases, the subject-matter experts required to develop a new AI model and deploy it to
who are needed to support the AI team put up pas- a production environment, where it can actually help
sive resistance to AI projects because they believe end users.
that these projects are intended to replace their jobs.
In any case, without a detailed understanding of
what the organization’s data mean and which pieces
Failures Due to Immature Technology
of data are reliable and important, AI projects will Finally, interviewees observed that, in some cases, AI
often struggle to achieve the organization’s aspira- projects fail because some problems are still too diffi-
tions for them. cult for AI algorithms to solve. The frequency of this

8
type of failure varies significantly depending on the are too sensitive to upload to a cloud environment.
type of use case for AI. For example, AI models are This is particularly true in heavily regulated indus-
quite effective for many e-commerce or advertising tries, such as finance or health care. However, even
use cases, but some intended applications for com- in these industries, many companies have success-
puter vision resist even the most rigorous and well- fully migrated their operations to the cloud in a way
funded attempts to apply AI. One interviewee stated that preserves the security of their data. The second
that AI algorithms are poorly suited to automating exception occurs when companies are operating
the internal processes of an organization—especially at the edge of AI research. These are mostly large
when subjective human judgment is required to technology companies that are attempting to train
determine how those processes should function. their own LLMs. Even AI researchers who were not
Leaders of an organization need to recognize that working on LLMs found that compute power might
AI is not a magic tool that can fully automate any be rationed within the organization and that, in
­process or solve any problem. Some business use some cases, this would delay their ability to train or
cases are a better fit for AI than others; understand- test models for a few days. However, several of these
ing which problems are a good fit for AI and which interviewees (4 of 50) expressed the belief that this
are at or beyond the current state of the art can help would prove to be a temporary problem as graphics
organizations avoid costly and embarrassing failures. processing unit manufacturers ramp up production
of their products.

Two Special Cases: Compute Power

Availability of Artificial Intelligence Talent
and Availability of Talent
In contrast to the findings on compute power, when
Alongside data, talent and compute power are key asked, many interviewees expressed the belief that
prerequisites for the training of AI algorithms. No the availability of AI talent does inhibit their work
organization can expect to succeed in developing to some extent.32 Many interviewees noted that the
AI products without a strong foundation in each overall availability of talent has improved in recent
of these components. Unlike the situation for data, years as new master’s programs in data science and
relatively few interviewees identified issues with the bootcamps have produced graduates trained in the
availability of either talent or compute power as the basic skills required to train AI algorithms. However,
most frequent or impactful factors behind the failure interviewees often noted that finding quality talent
of AI projects. However, because of the importance
of these key inputs, we specifically asked the inter-
viewees to discuss their perception of whether short-
ages in either of these areas contribute to the failure
of AI projects. Understanding which
Compute Power problems are a good
Nearly all of the interviewees stated that compute
power was not a limiting factor in their work. Most
fit for AI and which
interviewees said that cloud computing provid-
ers offer substantial amounts of compute power
are at or beyond the
for purchase on demand. Consequently, as long as current state of the art
the organization had adequately budgeted for the
purchase of compute power, this was not a limiting can help organizations
factor in the development of AI algorithms. How-
ever, interviewees noted two exceptions to this rule. avoid costly and
First, in some cases, companies think that their data
embarrassing failures.
9
remains difficult. Many educational programs focus often amount to nothing more than fancy to-do
primarily on development of AI models as opposed lists—the technical team should communicate fre-
to related skills in how to clean data, identify poor quently with their business partners about the state
data, or deploy AI models to production environ- of the project. As one interviewee put it:
ments. Consequently, interviewees said that they Stakeholders want to be a part of the process.
find it difficult to determine which recent graduates They don’t like it when you say, “it’s taking
would be effective in a less pristine workplace envi- longer than expected; I’ll get back to you in
ronment where data might be dirty, undocumented, two weeks.” They are curious.36
or unavailable.
Open communication builds trust between the
Interviewees also observed that many companies
business stakeholders and the technical team and
want to hire AI workers with exposure to the latest
increases the likelihood that the project will ulti-
techniques and models, even though relatively few of
mately be successful.
these companies truly need workers with these skills.
Several interviewees (8 of 50) found that their organi-
zations were most successful at hiring AI talent when Industry Interview Takeaways
they were prepared to identify talent with the poten-
As Table 3 illustrates, our industry interviews high-
tial to grow into the job as opposed to only hiring
lighted five leading root causes resulting in the fail-
perceived “rockstars.”
ure of AI projects. First, business stakeholders often
Additionally, some interviewees noted the lack
misunderstand—or miscommunicate—what prob-
of consistency in industry titles as a barrier to hiring.
lem needs to be solved using AI. Too often, organi-
The role of data scientist, in particular, can have
zations deploy trained AI models only to discover
radically different expectations and responsibilities
that the models have optimized the wrong metrics
across organizations. Open communication about
or do not fit into the overall workflow and context.
exactly how the workplace functions is essential to
Second, often the organization lacks the neces-
ensure a good fit with potential new employees.
sary data to adequately train an effective AI model.
Third, in some cases, AI projects fail because they
Agile Software Development and focus more on using the latest and greatest technol-
Artificial Intelligence ogy than on solving real problems for their intended
users. Fourth, organizations often do not have ade-
Finally, several interviewees (10 of 50) expressed
quate infrastructure to manage their data and deploy
the belief that rigid interpretations of agile software
completed AI models, which increases the likelihood
development processes are a poor fit for AI projects.33
of project failure. Finally, in some cases, AI projects
While the agile software movement never intended
fail because the technology is applied to problems
to develop rigid processes—one of its primary tenets
that are too difficult for AI to solve. AI is not a magic
is that individuals and interactions are much more
wand that can make any challenging problem dis-
important than processes and tools34—many organi-
appear; in some cases, even the most advanced AI
zations require their engineering teams to universally
models cannot automate away a difficult task. These
follow the same agile processes. One interviewee
five root causes stood out in the industry interviews
noted that, in his experience, work items repeatedly
as the most common and most impactful reasons
had to either be reopened in the following sprint or
that data science teams in industry perceive AI proj-
made ridiculously small and meaningless to fit into
ects as failing.
a one-week or two-week sprint.35 In particular, AI
projects require an initial phase of data exploration
and experimentation with an unpredictable duration.
Interviewees recommended that instead of adopting
established software engineering processes—which

10
TABLE 3
Key Root Causes Highlighted in Industry Semistructured Interviews
Root Cause Description

Leadership-driven failures Leaders fail to communicate to the engineering team what problem they want to be solved and
what metrics they need to optimize to solve it. Additionally, many leaders change priorities too
rapidly to allow the engineering team to deliver effective AI models.

Data-driven failures Organizations often lack sufficient high-quality data to train performant AI models. Leaders
may not be prepared for the time and expense required to gather enough data to train an
effective AI model.

Bottom-up–driven failures Data scientists sometimes focus on using the most-advanced technology instead of finding the
most effective way to solve the business problem.

Underinvestment in Inadequate infrastructure can lead to lower-quality data and longer deployment times for
infrastructure completed models. Underinvesting in infrastructure increases the risk that an AI project will fail.

Immature technology In some cases, organizations attempt to apply AI to business problems that are beyond the
state of the art for the technology.

Results of Interviews with projects based on the public demand for those proj-
Representatives of Academia ects’ outcomes. The interviewees indicated that
higher-prestige projects often take priority over less
The academic AI research environment is differ- attention-driven areas—even when the researcher
ent from the business environment. Academic believes that these other projects would be more
research often focuses on developing new techniques useful or valuable (consequently making the project a
through an integrative experimental process. Failure failure from their perspective). This is not to say that
is hard to measure academically, as investigations the researchers personally believed the lower-prestige
into new computer algorithms or ML techniques projects held less value; rather, there was an oppor-
are grounded in highly uncertain research areas. tunity cost from focusing only on high-priority proj-
Unsurprisingly, we did not get a clear consensus on ects. In this context, a project was deemed successful
what AI failure is in academic research. During the if it resulted in prestigious outcomes.
interviews, we identified some root causes that might This research builds on some key applied ele-
influence how academic researchers view AI project ments of information theory. This classic foun-
failures. A plurality of the interviews mentioned dational research may have market implications.
activity prestige (defined in the next section), data Experience is important in determining which
structures, and publication incentives as trends that activities researchers perceive as prestigious among
would affect AI research. Additionally, we found their peers. Newer researchers are more focused on
that computing resources were not a large concern completing tenure-track requirements; consequently,
within the academic setting. We attribute this to the they described feeling pressure to undertake AI
use of smaller datasets, more-efficient algorithms, projects that would result in publications. In con-
and regular access to large computing, which are trast, recently tenured researchers often prioritized
common in academia. securing new or expanded funding sources. Finally,
the well-established researchers (tenured for more
Activity Prestige than five years) we interviewed emphasized impact
on new research lines as a success driver. From these
Given the demand for new AI projects, interviewees observations, we found that while some research-
reported prioritizing projects that grab headlines, ers found publications to be a motivator, the focus
improve reputation, and increase institutional on output-driven research overshadowed promis-
prestige. We refer to this as activity prestige, which ing research areas that are more complicated and
is the amount of positive attention given to some less linked to publishable outputs, making it less

11
likely that new researchers would presume pos- collection. This point was especially emphasized by
sible innovation because these innovations would those doing biomedical research.
not be linked to tenure-track progress or valued In research, there is a higher level of emphasis
within an academic institution. While these find- applied to diagnostics, performance, and measurement
ings are anecdotal, future research could include to highlight outcomes because the goal is to improve
non­academic research organizations, such as think knowledge of computer science, which is distinctly
tanks or government-funded research affiliates, to different than applying that knowledge to business
see whether the publication pressure was constant in challenges. Academia’s focus on the science of comput-
shaping the research areas. ers requires more-stringent procedures to ensure that
Researchers also face incentives to undertake the data are collected and reported to minimize harm
work that is more likely to result in boosts to their to the users, often going through an internal review
prestige. This means that they weigh the risk that board and, if applicable, oversight controls. An exam-
a project will not result in a publication or addi- ple might be the application of an ML tool to patients
tional funding when considering potential projects with cancer, where the AI program will learn how to
to undertake. Younger researchers also take into diagnose dark spots on the skin to predict the prob-
account the expected duration and then choose ability that they are cancerous. Patient data cannot be
projects that have a shorter time horizon so that collected or stored within an openly accessed unstruc-
they will accumulate a more impressive record by tured central repository but instead must be stored
the time they apply for tenure. Computer science subject to patient record laws. For the enterprise of sci-
research often focuses on improving technology, ence, the quality of the data is paramount to ground-
with little consideration of the practical applica- ing a theory or developing a new field.
tion. At a company, AI production is more about
practically applying the technology to business prob-
Publication Incentives
lems. This creates an incentive gap whereby aca-
demic researchers, especially newer ones, are more Pressure to publish was repeatedly mentioned as a
rewarded for taking on projects that have increased potential contributor to AI project failure. Many
publication chances rather than real-world benefits. interviewees noted that when seeking tenure or
Once tenured, researchers have more freedom to building a research agenda, publication equals suc-
take on riskier and longer-term projects that could cess. If an AI project did not result in a publication,
have a greater impact. then the project was not perceived as a success. It
should be noted that a publication in this case could
be any outward-facing engagement, such as a talk,
Improper Data Structures conference paper, or proceeding. Given the high level
Improper data structures are related to data, bias, and of attention being placed on AI, there is an enormous
collection. Data scale and distribution were marked demand for new ideas, concepts, and techniques,
as issues that could lead to a well-specified AI model. which further increases the institutional demand
The most-prominent technical challenges were often already placed on the researchers.
in collecting and organizing the data needed to test Some of the non–tenure-track respondents sug-
a set of theoretical hypotheses that the researchers gested that even if their AI projects were technically
were seeking to better understand. Furthermore, successful, they might not be successful in terms of
sometimes data collections were biased or poorly advancing the researchers’ academic career oppor-
constructed, as evidenced by many researchers tunities (making the project a failure from their per-
noting that they went out of their way to collect the spective). Frequently, failure to publish on a project
best possible samples. Sometimes, if data collection results from the discovery of a new and unforeseen
was infeasible, they would try to use synthetic or technical problem, even though the act of problem
simulated datasets rather than using a biased data identification itself is a contribution that might lead

12
to new insights or open up new avenues of research. the recent surge in interest in AI technology, which has
Sometimes, the only way to identify a problem is to driven opportunities for more research but has also
experiment and explore. However, even if a techni- increased the demand for teachers and course offer-
cal problem leads to a more promising research ings. Furthermore, at least one interviewee expressed
agenda, the interviewees noted that the project the belief that the increasing popularity of LLMs
would still be considered a failure unless it resulted could, in the long run, crowd out other types of AI
in an immediate publication, such as a conference research. However, other respondents noted that the
proceeding or paper. interest in LLMs was complementary to other types of
AI research. Many of the graduate students reported
being optimistic about future AI research and feeling
Other Findings
that it was an exciting time to be in the field.
Almost none of the scholars mentioned the issues
of access to computing resources, data storage, or a
skilled labor force as limiting factors in their work.
Academic Interview Takeaways
For access to computing, universities may have some Researcher participants noted that prestige, fund-
of the largest, most powerful computers available or ing, and publication incentives played a large role in
are dealing with smaller-scale datasets. Data stor- determining the success of an AI project within the
age was most often purchased via a private company academic space, as outlined in Table 4. While techni-
and therefore often used a secure cloud service. The cal problems persist, they are often overcome with
need for skilled labor in academic settings is easily access to university-sponsored resources, including
managed through the academic institution’s focus on graduate student labor, computing services, and new
training and managing graduate and undergraduate hardware. When technical challenges arise, they
students rather than seeking new sources of labor. often come from errors that are fixed during the
Overall, although no respondent discussed quantity research process or become new lines of research. In
of available personnel being an issue, some noted that our interviews, we learned that when AI projects fail,
because laboratory workers were at different stages in they do so because of a misalignment in incentives
their education, quality control was a key focus. rather than an overall technical barrier to product
Respondents varied greatly in the different types delivery. In short, overcoming an AI failure is more
of exogenous factors they identified. Several discussed about humans than the machines.

TABLE 4
Key Root Causes Highlighted in Academic Semistructured Interviews
Root Cause Description

Activity prestige Researchers face pressure to do work that will be perceived by their peers as prestigious as
opposed to work that could be impactful.

Improper data structures Academic data often are either older or not collected with AI activities in mind; thus, the
researchers will have issues with training data or insufficient amounts or quality of testing data.

Publication incentives Researchers identified a project as a failure if it did not result in a publication, conference
proceeding, or communication item, even if the project led to new AI research.

13
Industry Recommendations Organizations should rethink the processes that
they have in place to facilitate these connections and
Although AI projects can be challenging for any
interactions among the various team members and
organization, failure is not inevitable. Leaders who
stakeholders. Rigid interpretations of existing soft-
want to avoid the most common mistakes cited by
ware development processes rarely suit the cadence
our AI experts should consider the following five
of an AI project. Instead of forcing project teams to
recommendations that may help lead to successful
follow a uniform set of procedures designed for a
AI implementation.
different type of engineering, organizations should
empower their teams to adapt their processes to fit
Ensure That Technical Staffs their workloads. Ultimately, organizations will need
Understand Project Purpose and to rediscover how to make the agile software develop-
Domain Context ment process be adaptive and—truly—agile.

Misunderstandings and miscommunications about

the intent and purpose of the project cause more AI Choose Enduring Problems
projects to fail than any other factor. Both the busi- AI projects require time and patience to be completed
ness leaders and the engineers have a role to play in successfully. Data scientists and data engineers need
avoiding this outcome. Business leaders need to help space to explore, understand, and curate the available
their technical staff understand what they truly need data before attempting to train an AI model that will
the AI project to achieve and how the completed AI learn how to behave from those data. Rapidly shift-
product will ultimately be used. They cannot assume ing the team’s priorities and chasing after the crisis
that the engineering team can independently dis- or opportunity of the moment can lead to a string
cover which design choices will make their product of AI projects being abandoned before they have a
useful within its business context. At the same time, chance to deliver tangible results. Before they begin
AI researchers and engineers need to earn the trust of any AI project, leaders should be prepared to commit
their business stakeholders by keeping them apprised each product team to solving a specific problem for
of their progress and project status, as well as any at least a year. If an AI project is not worthy of such
interim discoveries. Business leaders are often just as a long-term commitment, it most likely is not worth
excited about the potential of AI as the engineers are, committing to at all—especially because an AI proj-
if not more so; appropriately including them in the ect with an overly accelerated timeline is likely to fail
journey helps ensure a successful outcome. without ever achieving its intended goal.

Focus on the Problem, not the

Technology
Before they begin any Experienced engineers told us that successful project
AI project, leaders teams kept a clear focus on the business problem
to be solved instead of the technology that would
should be prepared to be used to solve it. Chasing the latest and greatest
advances in AI for their own sake is one of the most
commit each product frequent pathways to failure. Instead, an organiza-

team to solving a tion’s leaders need to collaborate with the technolo-

gists to ensure that they select AI projects that are

specific problem for at both a good fit for the technology and that solve a
real problem for their intended user. No matter how
least a year. impressive a new technology may appear, ultimately
any technology—even AI—is simply a tool to be
wielded rather than an end in and of itself.

14
Invest in Infrastructure require rigorous data-collection standards that often
limit the amount of data that is available for research
Data-related problems are among the top reasons
or may subtly bias the distribution of data samples
AI projects fail. Building up data infrastructure to
collected. For example, collecting data for biometric
reliably clean, ingest, and monitor data streams can
tracking studies can require years of effort and still
substantially improve an organization’s data and
yield a relatively small dataset. In contrast, research-
ensure that more of its AI projects ultimately suc-
ers in industry routinely collect and analyze much
ceed. Additionally, investments in infrastructure to
larger datasets. Consequently, academics could par-
automatically deploy AI models allow organizations
ticularly benefit from large-scale datasets with well-
to deploy these models to production more rapidly
documented collection procedures.
and reliably, where they can deliver real benefits to
Local, state, and federal government agencies
real users. Too many businesses fail to recognize the
could play an important role in providing these foun-
value that these kinds of investments can provide;
dational training sets as a public good. Government
instead, they rapidly switch from one AI project to
datasets meet academia’s requirement that data must
another without taking the time to invest in common
be collected in ways that meet the legal and ethical
tools that would make their data science teams more
standards appropriate for academic research. Addi-
productive. Leaders often justify this strategy because
tionally, many government agencies collect data at the
technology and their businesses are changing too
scale required to train sophisticated AI algorithms.
rapidly to make these kinds of investments. In real-
In return, collaborating with academic researchers
ity, delaying investments in infrastructure makes AI
could help government agencies address their critical
projects take longer to complete and fail more often.
shortages of technical and AI talent. Such initiatives
as Data.gov should be expanded and better funded to
Understand Artificial Intelligence’s take advantage of these opportunities.
Limitations
Finally, despite all the hype around AI as a technology, Expand Doctoral Programs in Data
AI still has technical limitations that cannot always be Science for Practitioners
overcome. Leaders cannot treat AI as a magic wand
In higher education, different roles have different
that can solve any problem or automate any process.
incentives. Society benefits when academics are able
Instead, leaders need to collaborate with their techni-
to focus on basic research focusing on long-term
cal experts to choose projects that are a good fit for
problems. However, newer academics might need
AI’s capabilities and would deliver meaningful value
encouragement to tackle these challenging issues
to the organization. Leaders do not necessarily need to
because they face short-term pressures to publish
have a deep technical understanding of AI themselves,
papers as quickly and reliably as possible. More-
but they need to employ staff with a strong data sci-
established academics, who have more job security,
ence background when selecting objectives for their
have greater freedom to take academic risks that
AI product teams. Simply assuming that AI can solve
may not pay off for several years. Governments and
any problem risks setting the team up for failure.
corporations should consider funding fellowships
to enable innovative younger researchers to pursue
longer-term research projects and free them from the
Academic Recommendations
pressure to constantly publish. Computer science and
Overcome Data Collection Barriers data science programs could learn from other disci-
Through Partnerships with Government plines, such as international relations and security
studies, where practitioner doctoral programs often
As we found in our interviews, academic researchers
exist side by side at even the top-ranked universities
face some challenges collecting sufficient quantities
to provide pathways for the most-advanced research-
of data to train effective AI models compared with
ers to apply their findings to contemporary problems.
their colleagues in industry. Academic use cases

15
APPENDIX A 5. Have any of these areas ever caused problems
Industry Interview Template for your AI/ML projects?
Ȥ Availability of talent (unable to hire the
This appendix includes the discussion questions used
right people)
for industry interviewees.
Ȥ Availability of compute
1. Please introduce yourself. Please tell us a bit
6. Are there differences in bottom-up reasons
about your background and your experience
why AI/ML projects fail compared to top-
with AI.
down reasons why AI/ML projects fail?
2. In your experience, when an AI project fails
7. What kinds of problems have you encountered
or encounters significant difficulties, what
as AI/ML teams expand from a small core
are the most common root causes for those
group to a larger team?
problems?
8. Does the rapidly changing nature of AI/ML
3. What kinds of technical causes can contribute
itself cause problems for your AI/ML projects?
to the failure of an AI/ML project?
9. What else should we have asked you?
Ȥ Problems with data
Ȥ Using the wrong approach
Ȥ Other technical root causes?
4. What kinds of nontechnical causes can con-
tribute to the failure of an AI/ML project?
Ȥ Direction from leadership
Ȥ Internal politics/bureaucracy
Ȥ Other nontechnical root causes

16
APPENDIX B Ȥ Probe 3.1: How did data structure, col-
Academia Interview Template lection, and capture approaches affect the
failure of developing an AI system? Was
This appendix includes the discussion questions used
this a data problem?
for academic interviewees.
Ȥ Probe 3.2: Given the complexity of the
1. Please introduce yourself. Please tell us a bit modeling, was modeling choice or specifi-
about your background and some key ele- cation of selection result a significant con-
ments that establish your knowledge base tributing factor to an AI failure? Was this a
associated with AI technologies. model or development problem?
Ȥ Probe 1.1: How many projects involving AI Ȥ Probe 3.3: Do the resulting workflow,
have you worked on? What did those proj- development processes, and shifts in tech-
ects aim to accomplish? nology platforms present a challenge that
2. Tell us a story about your most relevant expe- significantly contributes to the likelihood
rience where an AI project was not success- of AI failure? Do changes in the technology
ful. We are interested in your experience in standards matter?
why an unsuccessful project has challenges 4. We now want to pivot to examining the
or roadblocks and ultimately was not able to organizational roots of AI failures within
move to completion. an organizational structure and the human,
Ȥ Probe 2.1: What do you think were the labor, and capital dynamics that might shape
root causes of that failure? Are there any AI failure.
elements that stand out as critical inflec- Ȥ Probe 4.1: What workforce challenges
tion points that put stress on the project? caused difficulties for your projects?
What, if any, were the results of these
5. Exogenous and systematic factors may shape
inflection points?
if an AI is more or less likely to fail. How, in
Ȥ Probe 2.2: What are the first-, second-,
your experience, do network externalities,
and third-order effects of the key failure
labor or capital shortages, market failures,
points? Are there clear patterns within the
market forces, and/or regulatory frameworks
failures that could be observed from your
influence AI failure?
vantage point?
6. In your view, what are the most common
3. Consider the difficulty of developing AI sys- reasons that you have observed or experienced
tems. We would like to explore root techni- that resulted in AI failure?
cal causes of AI failure. The complexity of 7. Thank you for coming today and for sharing
AI development cycles and the novelty of its your opinions with us. We hope you enjoyed
application are often a source of reported the discussion today.
technical difficulty.

17
 21
Notes Chui et al., “The State of AI in 2022—and a Half Decade in
Review.”
1 For this project, we focused on the machine learning (ML) 22 Schlegel, Schuler, and Westenberger, “Failure Factors of AI
branch of AI because that is the technology underpinning most
business applications of AI today. This includes AI models Projects.”
trained using supervised learning, unsupervised learning, or 23 Individual contributor is a common term used in industry to
reinforcement learning approaches and large language models denote an employee who is not responsible for managing other
(LLMs). Projects that simply used pretrained LLMs (sometimes employees. Individual contributors can have various levels of
known as prompt engineering) but did not attempt to train or seniority and work experience. For this project, the term man-
customize their own were not included in the scope of this work. ager includes all levels of management, including director, vice
2 Maslej et al., Artificial Intelligence Index Report 2023. The president, and other titles. It includes both managers of indi-
report notes that private-sector AI investment decreased in 2022 vidual contributors and managers of managers.
for the first time in a decade. 24Many interviewees had worked for more than one company
3 Midsize corporations are those with between $100 million and during their career.
$499 million in annual revenue. 25 The majority of the rest of our candidate pool were nonre-
4 Patience and Immerman, 2023 Global Trends in AI Report. sponders. Approximately two dozen individuals responded that
they were interested in participating in the study but did not set
5 Vincent, “Why the Pentagon Didn’t Request Higher Funding up an interview time before we ended data collection.
for AI in Fiscal 2025.” 26 Meaningful number here means more than one-quarter but
6 Anagnostopoulos et al., “How Artificial Intelligence Can less than one-half of the interviewees identified these root causes
Power Clinical Development.” as among the primary reasons for AI project failures.
7 Musani, “Decking the Aisles with Data: How Walmart’s 27 Study participant, interview with the authors, October 6, 2023.
­AI-Powered Inventory System Brightens the Holidays.” 28 Study participant, interview with the authors, September 12,
8 Copp, “An AI-Controlled Fighter Jet Took the Air Force Leader 2023.
for a Historic Ride.” 29Study participant, interview with the authors, September 28,
9 Seffers, “Undersea Combat Includes Way More Than 2023.
Submarines.” 30Study participant, interview with the authors, November 20,
10 Clark, “Pentagon Official Lays Out DoD Vision for AI.” 2023.
11 Cisco AI Readiness Index. 31 Study participant, interview with the authors, October 10, 2023.
12 McKendrick, “Executives and Managers Welcome AI, but 32 Seven of 50 interviewees stated that talent availability was a
Struggle with It.” difficulty in AI projects. An additional 19 of 50 stated that while
13 talent availability was not a problem overall, the lack of high-
Kahn, “Want Your Company’s AI Project to Succeed?”
quality talent was a limitation in their AI projects.
14 Bojinov, “Keep Your AI Projects on Track.” 33 Agile software development is defined by the U.S. Government
15 Budgen, Software Design; Ambler, Process Patterns. Accountability Office as “an approach to software development
16 Sjödin et al., “How AI Capabilities Enable Business Model in which software is built incrementally and is continuously
evaluated on functionality, quality, and customer satisfaction”
Innovation.” (U.S. Government Accountability Office, Science & Tech Spot-
17Standing et al., “The Attribution of Success and Failure in IT light: Agile Software Development).
Projects.” 34 Beck et al., “Manifesto for Agile Software Development.”
18 Westenberger, Schuler, and Schlegel, “Failure of AI Projects.” 35 In agile software development, sprints are a short, time-boxed
19 Bojinov, “Keep Your AI Projects on Track”; Schmelzer, “The period in which the engineering team commits to completing a
One Practice That Is Separating the AI Successes from the specific set of work. Sprints typically last one to four weeks.
Failures.” 36 Study participant, interview with the authors, October 6, 2023.
20 Mittal, Saif, and Ammanath, Deloitte’s State of AI in the Enter-
prise, 5th Edition Report.

18
References McKendrick, Joe, “Executives and Managers Welcome AI, but
Struggle with It,” Forbes, August 24, 2023.
Ambler, Scott W., Process Patterns: Building Large-Scale Systems
Using Object Technology, Cambridge University Press, 1998. Mittal, Nitin, Irfan Saif, and Beena Ammanath, Deloitte’s State of
AI in the Enterprise, 5th Edition Report, Deloitte, October 2022.
Anagnostopoulos, Chris, David Champagne, Thomas Devenyns,
Alex Devereson, and Heikki Tarkkila, “How Artificial Musani, Parvez, “Decking the Aisles with Data: How Walmart’s
Intelligence Can Power Clinical Development,” McKinsey & AI-Powered Inventory System Brightens the Holidays,” Walmart
Company, November 22, 2023. Global Tech, October 25, 2023.

Beck, Kent, Mike Beedle, Arie van Bennekum, Alistair Patience, Nick, and David Immerman, 2023 Global Trends in AI
Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Report, S&P Global, August 2023.
Jim Highsmith, Andrew Hunt, Ron Jeffries, et al., “Manifesto for Schlegel, Dennis, Kajetan Schuler, and Jens Westenberger,
Agile Software Development,” Agile Alliance, February 2001. “Failure Factors of AI Projects: Results from Expert Interviews,”
Bojinov, Iavor, “Keep Your AI Projects on Track,” Harvard International Journal of Information Systems and Project
Business Review, November–December 2023. Management, Vol. 11, No. 3, 2023.

Budgen, David, Software Design, 2nd ed., Addison-Wesley, 2003. Schmelzer, Ron, “The One Practice That Is Separating the AI
Successes from the Failures,” Forbes, August 14, 2023.
Chui, Michael, Bryce Hall, Helen Mayhew, Alex Singla, and Alex
Sukharevsky, “The State of AI in 2022—and a Half Decade in Seffers, George, “Undersea Combat Includes Way More Than
Review,” McKinsey & Company, December 6, 2022. Submarines,” Signal Magazine, February 1, 2022.

Cisco AI Readiness Index, homepage, undated. As of June 19, Sjödin, David, Vinit Parida, Maximilian Palmié, and Joakim
2024: Wincent, “How AI Capabilities Enable Business Model
https://www.cisco.com/c/m/en_us/solutions/ai/ Innovation: Scaling AI Through Co-Evolutionary Processes
readiness-index.html and Feedback Loops,” Journal of Business Research, Vol. 134,
September 1, 2021.
Clark, Joseph, “Pentagon Official Lays Out DoD Vision for AI,”
DoD News, February 21, 2024. Standing, Craig, Andrew Guilfoyle, Chad Lin, and Peter E. D.
Love, “The Attribution of Success and Failure in IT Projects,”
Copp, Tara, “An AI-Controlled Fighter Jet Took the Air Industrial Management and Data Systems, Vol. 106, No. 8,
Force Leader for a Historic Ride. What That Means for War,” October 2006.
Associated Press, May 3, 2024.
U.S. Government Accountability Office, Science & Tech Spotlight:
Kahn, Jeremy, “Want Your Company’s A.I. Project to Succeed? Agile Software Development, GAO-20-713SP, September 29, 2020.
Don’t Hand It to the Data Scientists, Says This CEO,” Fortune,
July 26, 2022. Vincent, Brandi, “Why the Pentagon Didn’t Request Higher
Funding for AI in Fiscal 2025,” DefenseScoop, March 11, 2024.
Maslej, Nestor, Loredana Fattorini, Erik Brynjolfsson, John
Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Westenberger, Jens, Kajetan Schuler, and Dennis Schlegel,
Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell “Failure of AI Projects: Understanding the Critical Factors,”
Wald, Jack Clark, and Raymond Perrault, Artificial Intelligence Procedia Computer Science, Vol. 196, 2022.
Index Report 2023, AI Index Steering Committee, Institute for
Human-Centered AI, Stanford University, 2023.

19
 RAND is a research organization that
develops solutions to public policy
About This Report challenges to help make communities
throughout the world safer and more
Leaders across all sectors are increasingly recognizing the importance of arti- secure, healthier and more prosperous.
ficial intelligence and machine learning (AI/ML) for their organization, believ- RAND is nonprofit, nonpartisan, and
committed to the public interest.
ing or hoping that AI will help them achieve formerly impossible feats and rise
above their peers. Yet all too often, AI projects flounder or never get off the Research Integrity

ground. To investigate why AI projects fail, RAND researchers conducted an Our mission to help improve policy and
decisionmaking through research and
exploratory study, interviewing data scientists and engineers with at least five analysis is enabled through our core
years of experience building AI/ML models in industry or academia and syn- values of quality and objectivity and our
unwavering commitment to the highest
thesized the experts’ experiences to develop a set of recommendations for smart level of integrity and ethical behavior. To
implementation of AI. The findings and recommendations of this report should help ensure our research and analysis
are rigorous, objective, and nonpartisan,
be of interest to the U.S. Department of Defense, which has been actively looking we subject our research publications to
for ways to use AI, along with other leaders in government and the private sector a robust and exacting quality-assurance
process; avoid both the appearance and
who are considering using AI/ML. The lessons from earlier efforts to build and
reality of financial and other conflicts of
apply AI/ML will be helpful for others to avoid the same pitfalls. interest through staff training, project
screening, and a policy of mandatory
The research reported here was completed in April 2024 and underwent security disclosure; and pursue transparency
in our research engagements
review with the sponsor and the Defense Office of Prepublication and Security
through our commitment to the open
Review before public release. publication of our research findings and
recommendations, disclosure of the
source of funding of published research,
RAND National Security Research Division and policies to ensure intellectual
independence. For more information, visit
This research was conducted within the Acquisition and Technology Policy www.rand.org/about/research-integrity.
Program of the RAND National Security Research Division (NSRD), which
RAND’s publications do not necessarily
operates the RAND National Defense Research Institute (NDRI), a federally reflect the opinions of its research clients
funded research and development center (FFRDC) sponsored by the Office of and sponsors. is a registered
trademark.
the Secretary of Defense, the Joint Staff, the Unified Combatant Commands,
the Navy, the Marine Corps, the defense agencies, and the defense intelligence Limited Print and Electronic
Distribution Rights
enterprise. This research was made possible by NDRI exploratory research fund-
This publication and trademark(s)
ing that was provided through the FFRDC contract and approved by NDRI’s contained herein are protected by law.
primary sponsor. This representation of RAND intellectual
property is provided for noncommercial
For more information on the RAND Acquisition and Technology Policy Program, use only. Unauthorized posting of this
publication online is prohibited; linking
see www.rand.org/nsrd/atp or contact the director (contact information is pro- directly to its webpage on rand.org is
vided on the webpage). encouraged. Permission is required from
RAND to reproduce, or reuse in another
form, any of its research products for
Acknowledgments commercial purposes. For information on
reprint and reuse permissions, please visit
We would like to thank Jeannette Tsuei for her assistance with this project and www.rand.org/pubs/permissions.
Christopher Mouton, Yun Kang, Caitlin Lee, and Megan McKernan for their
For more information on this publication,
support of the project. We would also like to thank our reviewers (Justin Lynch, visit www.rand.org/t/RRA2680-1.
Pete Schirmer, Michael Kennedy, Chad Ohlandt, Bonnie Triezenberg, Jeff
© 2024 RAND Corporation
Alstott, and Aaron Frank) for all of their helpful insights and questions. Finally,
we would like to thank all of our interviewees for their time and their perspec-
tives about this important topic. www.rand.org

RR-A2680-1