Guide to

Measuring Impact
with Agile Metrics

AGILE | DATA | DIGITAL | MODERNIZATION

In an Agile context, it’s important to collect data to inform decision-making and enable continuous improvement. Not
all metrics are created equal; however, some metrics can do more harm than good. One of the best recent sources of
information about what consistently high-performing organizations measure themselves against is the 2019 Accelerate
State of DevOps Report, compiled by Data Scientist Nicole Forsgren, who is also one of the authors of the book
Accelerate. The “big four” metrics from that report are among those that we recommend, and which we’ll describe further,
along with many others: 1. Deployment frequency; 2. Lead time for changes; 3. Change failure rate; 4. Time to restore
service.

At Excella, we have partnered with numerous federal agencies to help them on their Agile journey. We follow an adaptive
approach, continuously seeking to improve, by listening to our business partners, running experiments, and making data-
driven decisions based on the results from those experiments. Knowing what to measure, and what those measurements
can and cannot tell us, is an Excella core competency. Our expertise is based on our own experiences as practitioners,
along with careful application of metrics such as those that are included in the State of DevOps Report.

Let’s focus our attention on four categories of Agile metrics that

can provide actionable and meaningful insight, and help us focus on business outcomes:

Collaboration Delivery Quality Value

Metrics Metrics Metrics Metrics
 Examples of Collaboration Metrics:
Effectiveness
• Number of Cross-Team Dependencies Identified/Closed. Count
the number of dependencies that exist, where each additional

Collaboration
dependency significantly increases the risk that work will not be
finished when expected; and count the number of cross-team

Metrics
dependencies closed, where each elimination of a dependency
significantly decreases the risk that work will not be finished when
expected. (Focused Objective: Dependency Types and Impact)

Collaboration Metrics provide visibility into whether

• Mean Cycle Time of User Stories with Dependencies: Collect data
an Agile team is working together effectively, both
on how long it takes to complete each user story (its “cycle time”),
internally (intra-team), and externally (cross-team),
and compare the cycle times for user stories with and without
and also provide a pulse check on team morale.
dependencies. (Focused Objective: Calculating Throughput and
Cycle Time History)
Unlike many of the other metrics categories that we
describe, Collaboration Metrics tend to be leading
Happiness/Satisfaction
indicators, offering early warning signs of potential
areas of trouble. • Organizational Satisfaction Score: Collect data from a broad
cross-section of people in the organization and generate scores
that highlight which things are going well, and which things need
At Excella, we place great importance on attention, at the organizational level. (Crisp: Leadership and Team
partnering with our clients to gather data Health Checks)
that provides visibility into team health
and organizational health, and this sets us • Team Satisfaction Score: Collect data from individual teams and
generate scores that highlight which things are going well, and
apart from many of our competitors.
which things need attention, at the team level. (Atlassian: Health
Monitor)
Questions that Collaboration Metrics help answer:

Do team members proactively Do team members feel that they

Do team members feel empowered
seek to minimize the number of can safety express their views, and
to discover and develop solutions
dependencies that they have on are they confident that those views
to business problems?
other groups/teams? are heard and acted upon?

Collaboration Metrics to Try and Collaboration Metrics to Avoid

Try This Avoid This
Compare the cycle times of one team with the cycle times of
Evaluate cycle times, on an ongoing basis
another team

Single out individual teams because their morale appears to

Collect data about the happiness of people in the organization be lower (instead, look for root causes of broadly applicable
organizational dissatisfiers)
 Examples of Delivery Metrics:
Delivery Cadence
• Deployment Frequency: Collect data on the relative
frequency with which new functionality is made available

Delivery
to users. (CircleCI: How Often Does Your Team Deploy?)

• Lead Time: Collect data on how much time elapses from

Metrics the moment a customer first requests a feature to the

moment that the feature is available to that customer for
use. (GoCD:What Lead Time Is)
Delivery Metrics provide visibility into the
relative frequency of deployment of software System Availability
features to Production. As such, Delivery Metrics • Change Failure Rate: Collect data on the relative
focus on the ability of an organization to deliver frequency with which a deployment results in the
at a consistent, sustainable pace. immediate need to correct a problem caused by that
deployment. (CircleCI: What does change fail rate tell us?)
Delivery Metrics tend to be lagging indicators,
giving us insight into the maturity of our tools • Mean Time to Detect (MTTD)/Mean Time to Repair
and processes that govern the management (MTTR): Collect data on how long it takes to detect that a
of code artifacts and monitoring of physical or problem exists, and how long it takes to correct a problem
virtual infrastructure. after it has been detected. (AlertOps: MTTD vs MTTF vs
MTBF vs MTTR)
Questions that Delivery Metrics help answer:

How many running, tested units How stable and reliable are code
How stable and reliable is our
of work can we reliably deliver and other artifacts that we deploy
deployment pipeline?
during a Sprint? to Production?

Delivery Metrics to Try and Delivery Metrics to Avoid

Try This Avoid This
Compare the lead times of one team with the lead times of
Evaluate lead times, on an ongoing basis
another team

Leverage system availability data to identify areas for Use a deployment or system failure as a reason to single out an
improvement individual or team for blame
 Examples of Quality Metrics:
Defects
• Defect Density: Number of Defects by Module: Count the number of
defects that exist in a particular software module (often counted based
on number of defects per 1,000 Lines of Code (KLOC). (SeaLights: Defect
Density: Context is King)

• Defect Escape Rate: Count the number of defects that are opened in
a particular component, where the defects are opened after the Sprint
has ended during which the component was created or modified.
(LeadingAgile: Escaped Defects)

Quality Technical Debt

Metrics • Amount of Time Set Aside for Code Refactoring (per User Story): Capture
data on the amount of time during which code refactoring is done, during
initial development and/or during peer code review. (Mountain Goat
Quality Metrics provide visibility into the Software: The Economic Benefit of Refactoring | Tushar Sharma: How to
extent to which the products that we deliver Track Code Smells Effectively)
work as they are intended to work.
• Number of User Stories or Defects Created to Address Technical Debt:
Some quality metrics, such as unit test Count how many user stories or defects are opened to address shortcuts
coverage, are leading indicators. For that had to be taken due to time constraints; related metrics that can
instance, if we take it as a given that unit help uncover technical debt via code analysis tools include cyclomatic
tests are testing something meaningful, complexity, code coverage, SQALE rating, and rule violations. (Excella: The
execution of unit tests can prevent problems Technical Debt Management Plan | Christiaan Verwijs: How to Deal with
from surfacing later. Other quality metrics, Technical Debt in Scrum)
such as Defect Escape Rate, are lagging
indicators, because they tell us after the fact Test Coverage
that we have a gap in our testing approach. • Test Automation Coverage Level: Collect data on the percentage of the
code base that is covered by automated tests, by dividing the total test
coverage by the test automation coverage. (LogiGear: 5 Useful KPIs for
Test Automation)

• Unit Test Coverage Level: Collect data on the percentage of the code base
that is covered by unit tests. (SeaLights: Code Coverage vs Test Coverage)
Questions that Quality Metrics help answer:

To what extent are there

To what extent do we give
To what extent does the software automated checks in place to
ourselves sufficient time to build
work as intended? surface anomalies before they
things right the first time?
become significant problems?

Quality Metrics to Try and Quality Metrics to Avoid

Try This Avoid This

Open defects for anomalies that are detected during a Sprint
Seek to detect and correct anomalies in real time, during a Sprint
(fix them on the spot!)

Use multiple testing approaches to check for correct software/

Create incentives for opening as many defects as possible
system behaviors

Make time available for code refactoring, which greatly reduces Place blame on teams when it becomes difficult to improve or
the likelihood that technical debt will be created maintain existing code, due to the existence of technical debt

Assume that a high test coverage number is a guarantee of

Consider the extent to which different parts of the code base have
success (some tests may have been commented out or lack
sufficient test coverage
sufficient depth)
 Examples of Value Metrics:
Customer Satisfaction
• Customer Effort Score (CES): Capture data about a customer’s
experience, based on how much time was required by that customer
to achieve a goal, such as getting a question answered, an issue
resolved, or a product purchased/returned. (Retently: What is CES
and How to Measure It)

Value • Customer Satisfaction Score (CSAT): Capture customer survey

data, divide the positive responses by the negative responses, and

Metrics multiply by 100. (Emolytics: CSAT: The Happy Customer KPI)

• Net Promoter Score (NPS): Capture customer survey data for a

Value Metrics provide visibility into the extent given group, total the number of people in the group and divide it
to which the products that we deliver enable by the number of survey responses, and subtract the percentage
customers to achieve their goals, and the of respondents who are Detractors from the percentage of
extent to which our investments provide the respondents who are Promoters. (Hotjar: What is NPS?)
returns that we expect.
Value Delivery
Value metrics, especially those associated with
• Return on Investment (ROI): Divide the amount of money produced
Customer Satisfaction, are lagging indicators,
by a new product or improvement to an existing product by how
because some time needs to elapse to give
much money was spent to build or improve that product. (Xpedia:
customers sufficient time to interact with a
Calculating ROI for Software Development)
new product, or to experience changes to an
existing product.
• Units of Value Delivered per Release: Count the number of finished
work items in a Release, and assign a value score based on that
number. (Folding Burritos: Measuring Customer Value in a Software
Product)
Questions that Value Metrics help answer:

To what extent do we deliver To what extent are customers To what extent are internal
solutions that customers satisfied with the solutions that stakeholders satisfied with the
consider usable? we deliver? solutions that we deliver?

Value Metrics to Try and Value Metrics to Avoid

Try This Avoid This
Measure “success” only on the basis of traditional project
Collect customer satisfaction data on a consistent cadence
management measures, such as “on time and on budget”

Collect data on how much work (and by extension, how much Compare how much work one team completed with how much
value) each team finishes during a particular time frame work another team completed
Conclusion
By taking a balanced perspective, being sure to include Collaboration Metrics (which often get little if any attention),
and including Delivery, Quality, and Value Metrics, we have a strong foundation on which to build. As the authors
point out on page 9 of the 2019 Accelerate State of DevOps Report:

Remember to accelerate your transformation by starting with a solid foundation

and then focusing on the capabilities that are constraints: What capabilities cause
the biggest delays? What are the biggest headaches? Where are the biggest
problems? Pick three to five and dedicate resources to solving these first. Don’t
worry if you still have problems; by focusing on the biggest problems now, you
remove bottlenecks, discover synergies, and avoid unnecessary work.

Excella is an Agile technology firm helping leading organizations realize their future through the power of
technology. We work collaboratively to solve our clients’ biggest challenges and evolve their thinking to help
them prepare for tomorrow. Together we transform bold ideas into elegant technology solutions to create real
progress. Learn more at www.excella.com.

Find us on social

AGILE | DATA | DIGITAL | MODERNIZATION