Data Driven Product Manager Handbook

The Data-driven
Product Manager’s
Handbook
Using insight and guidance to deliver
exceptional product experiences
A Pendo eBook
Table of Contents
Introduction.....................................................................3
How to use this handbook?...........................................4

Chapter 1:
Product KPIs.................................................................. 5
Chapter 2:
Data-driven Roadmaps................................................. 8
Chapter 3:
Better Customer Feedback.......................................... 12
Chapter 4:
Actionable NPS............................................................ 15
Chapter 5:
Tailored Onboarding.................................................... 19
Chapter 6:
Ongoing Customer Education.................................... 23
Chapter 7:
Feature Awareness and Adoption............................. 26
Closing......................................................................... 30
The Data-Driven Product Manager’s Handbook A Pendo ebook 2

Introduction
Data is the cornerstone of many product team conversations, but what does it really mean to be
a data-driven product leader? All product managers strive to build products that matter, solve real
world problems, and have a positive impact in customers’ lives.
This handbook will help cut through the noise, revealing the seven critical elements of effective
data-driven product leadership.

We need more than buzzwords. We need concrete practices to help product leaders excel and
execute. That’s what this handbook will help you tackle.

Let’s get started!

How to use this handbook?
This handbook covers the core elements of effective data-driven product management, including
best practices for becoming more successful in your role.
Here are some ways you can put this information to work:
• Define the KPIs that help you demonstrate the impact of your role and measure and deliver
exceptional product experiences
• Create roadmaps that align your team to the highest impact opportunities
• Optimize your customer onboarding and user feedback processes
• Learn how to drive product feature awareness and adoption
• Crack the code on capturing NPS scores and, more importantly, how to make them actionable.
Whether you’re new to product management or a seasoned veteran, this handbook will arm you
with new ways to deliver products that your users love!

Chapter 1:
Product KPIs

Chapter 1: Product KPIs

You can’t manage what you can’t measure. That’s why product managers
=
need a clear set of KPIs to guide their overall performance and contribution.
In the past, the key metric for product teams was product delivery - shipping
features and products. Today, product managers are judged on their ability
to deliver continuous value to users. Measures now include how users adopt,
use, engage with and derive value from their products, as well as setting
appropriate benchmarks. Without setting clear KPIs, teams may be vulnerable
to subjective evaluation of what is and isn’t working for customers, users and
the business.
5% 25-95% A different set of KPIs and a unique approach to continuous measurement are
necessary.
Customer Retention Profits
The Downstream KPIs Revenue and Retention
"Increasing customer retention by 5% A fundamental measurement of any software product is whether or not it’s
can increase profits by 25-95%" making money. Many of the financial metrics used in the SaaS industry are
- Frederick Reichheld of Bain & Company (inventor of NPS) relevant to product teams:
ARR / MRR = Annual or monthly recurring revenue associated with subscribers

of a specific product. All contracted revenue is normalized in either a monthly
or annual view.
MAU / DAU = Monthly or daily active users. A count of the number of unique
SaaS companies with more advanced users in the product during a given time period.
product analytics programs have 2X to 5X ARPU = Average revenue per user. Total subscription revenue divided by total
higher ARR than their competitors subscribers. This metric helps to quantify the value of an individual customer or
user.
- The Product Data Advantage, Pendo Survey, 2016
Renewal / Retention Rate = Percentage of customers that renew a subscription
as well as the percentage of revenue retained with renewal.

These KPIs are valuable, but they are also lagging indicators. Too Frequency of Use
often,negative changes appear in this data long after customers have had
Understanding breadth of usage is necessary, but insufficient. You also need to
a negative experience. It’s typically too late to help them realize value from
understand how frequently individual users are engaging with your product--
your product.
and for how long. This is another key measure of product and customer health.
Leading KPIs: Behavior and Sentiment Frequency of use measures how often users (or users from specific accounts)
log-in to your product in a given time frame. Product teams should look at
Looking beyond financial metrics into how users behave in the product the average number of sessions within a given time frame and the average
can provide greater insight into the product experience, and provide more duration of each session.
actionable information for product teams. Some more additional metrics to
consider include: Efficiency of User Actions
How difficult is it for users to complete common tasks in an application? For
Product Breadth of Usage example, think of adding a new vendor in a procurement application or posting
The simple way to think of breadth of usage is to answer this question: a job requisition in a recruiting application. A common task represents a core
“How many active users do I have in my product?” But there is more to this function within the application and typically comprises several application
metric than meets the eye. features used in a specific sequence. To measure efficiency, product teams
should look at the total number of users who begin a task and see what
For B2B products, in particular, a more important measure is how many percentage successfully complete it. This measure can augment user testing
active users you have per account. Breadth of usage, particularly at an and help show the usability of key product functions.
account level, is a great indicator of how sticky your product has become
within your customer base. This understanding allows you to monitor Customer Satisfaction via Net Promoter Score (NPS)
account health and ensures that you have visibility into potential churn risk.
Net Promoter Score (NPS) provides a simple benchmark of customer advocacy
that can be tracked over time. Many companies use it to measure customer
Depth of Use satisfaction, benchmarking against their industry, their competitors and themselves.
It’s also important to assess what percentage of application features get
used within a specific time frame. Or more simply, how much of the product NPS is only one indicator of customer health, but it’s a very popular one that
and its core features are actually being utilized by your users? many SaaS companies have adopted. A recent Pendo survey suggests that 57%
of product managers are actively using NPS to understand how users feel about
For effective measurement, product teams should first define their key their products. You can send NPS surveys to product users and ask how likely they
features—those that make up the product’s core function or are most heavily are to recommend the product to other users to gauge overall satisfaction and
used by the happiest customers. Product depth is about looking at how sentiment. It’s important to track NPS over time rather than using the score as a
adept? your customers are at using all parts of your product. Are they using static snapshot. The goal, of course, is to move your NPS in a positive direction and
it to its fullest potential or are they using just a limited portion of the overall continue to improve it over time as satisfaction increases.
application? You want to have balanced usage of the product to relay full
value. Build these KPIs into your benchmarking methodology to help you effectively
measure value delivered and ensure exceptional product experiences for your
users.

Chapter 2:
Data-driven
Roadmaps

Chapter 2: Data-driven roadmaps
Chapter 2: Data-driven Roadmaps

A roadmap is a communication tool, and often a good one. It reflects the
product team’s plan for what is going to happen in the future, based on current
“We try to always back up our strategic priorities. It’s a summarized, often thematic view of the force-ranked backlog,
which provides a useful way for product teams to share current plans. It is,
decisions with data - whether that’s market as the safe harbor statement points out, non-binding and always ‘subject to
data, user research, or product data. Pendo change.’
feeds into our strategic decision-making. It The real question is: how should the product team arrive at the key priorities in
helps us validate from a usage perspective the first place? Product managers have no shortage of input. Vocal customers,
executives, and sellers all have opinions on what should be built next. The
which customers are most engaged and challenge is sorting through all that feedback to prioritize what truly delivers
customer value. This requires data.
active, and helps us prioritize support for this
strategic customer segment.” Data-driven product managers incorporate KPIs and strategic goals into their
roadmaps and backlog prioritization. They use data to identify opportunities
for product enhancement, to inform prioritization decisions, and to measure the
- Darren Card, uptake and impact of new features and new products.
Director of Product Management, Allocadia
The first of these steps involves identifying the right metrics to guide the
roadmaps.
Start with Vision and Strategy

Before prioritizing any feature backlog, it’s important for product teams to
ensure they are aligned with the overall business strategy and vision. It’s hard
to effectively prioritize what to build if the broad objectives are unclear. What’s
the market pain we’re trying to address? Who is the target customer—their
industry and persona? What is the market opportunity? Where do we see the
best opportunity for growth? These are key strategic questions that must be
addressed before any feature-level discussions can happen.
Each of the high-level strategic goals should have KPIs associated with it—
whether that’s market share, revenue, or customer acquisition. These metrics
serve as the baseline for any product initiative, and should be reflected in
roadmap priorities.

Prioritize Appropriately
 CUSTOMER  BUSINESS-ORIENTED
Effective prioritization is a continuing challenge for product teams. ORIENTED METRICS METRICS
Engineering resources are always limited, forcing product teams to focus
on maximizing customer value for every feature delivered. The question is
how to know which item on the roadmap will deliver the most value. Product usage/adoption (e.g., Cost to acquire a new customer
MAU, DAU) (CAC)
Data-driven product teams should consider both the alignment with
strategic objectives as well as customer behavior and input to arrive at
Percent of users who take a Lifetime value (LTV)
prioritization decisions. At a basic level, understanding how much an
specific action
existing feature or area of the product is used should inform whether to
invest additional development resources. Prioritizing feature updates for
things that are rarely used is not likely to deliver significant value. However, Feature usage Monthly recurring revenue
a feature may be underused because it doesn’t provide value, or because (MRR)
it’s difficult to use. Adding targeted customer feedback into the analysis
can help determine the “why” behind observed behavior, and further refine
feature prioritization. Retention or churn rate Average revenue per user
For net new products or features, it may be difficult to rely on historical Quality Conversion
customer behavior alone to drive prioritization. For these decisions, product
teams should rely more heavily on alignment with vision and strategy.
Does this feature further our strategic objectives better than the next best NPS/customer satisfaction Retention
option, and how would we measure that? is the key question. Though
any prioritization decision is at some level a “best guess”, measurement
is critical. Data can inform the decision, but it should also be leveraged to Business metrics like revenue, churn or conversion rate represent higher-level
evaluate the decision after the fact. outcomes. These are the goals that a product team is hoping to achieve, and
often the metrics they are measured against. Customer-oriented or usage
Assign Specific Metrics to Each Item in metrics are specific measures of behavior or user sentiment that can be leading
indicators of business outcomes. Roadmap initiatives should be associated with
your Roadmap specific goals for both business and usage.
A product investment has no place in the product roadmap if its value

can’t be assessed. Product teams should be sure to assign both business Measure your Impact
(e.g. revenue) and adoption/usage goals to each prioritized feature. This
chart can help you identify how the metrics impact your customers and the To understand impact, each roadmap feature should have baseline KPIs
business as a whole: associated with it. Those could be usage goals (this feature should be used by
X% of users in 30 days), usability and customer satisfaction (these UI changes

should reduce support requests by X%), or specific financial metrics (migrate

20 additional accounts to a paid tier). The KPIs should be set ahead of time,
and integrated within the roadmap itself. That way you can ensure that
any development required to support measurement is scoped as part of
the feature, and that the organization understands how the product team is
measuring success.
Communicating Priorities
The goal of a roadmap is to provide an organized view of development
priorities. It is only effective as a communication tool if it can convey the
“why” behind the priorities it illustrates. Effective product teams know
that incorporating goals and metrics into the roadmap documents helps
to illustrate that why. It also makes the team accountable for results. Too
often roadmaps are shared without any of this explanation or reasoning. If
details cannot be included, product leaders should simply avoid sharing a
roadmap until they’ve talked through the items and priorities with the rest
of the organization or stakeholders.
A roadmap is never “done.” Elements are always changing. And so, any
roadmap discussion is an opportunity to gain feedback. That feedback
isn’t always actionable, and may not cause a change in priorities, but it’s
an important way to see how the thing you’re building is perceived by the
market and your customers.

Chapter 3:
Better Customer
Feedback

Chapter 3: Better Customer Feedback

Most product leaders would suggest that talking to customers is one of
their most important functions. Customer feedback is vital to any successful
software product, and product teams should always strive to understand and
incorporate the voice of the customer in everything they build. The challenge,
“We add surveys to targeted feature of course, is reaching the right customers.
announcements asking users whether or not It can be difficult to recruit customers to provide feedback, especially those
they think the new feature would be useful with whom a Product Manager doesn’t have an existing relationship. Rather
than trying to connect with unfamiliar customers, many PMs will rely on the
for their clinic. Because we’re asking for ones they already know. Unfortunately, the most vocal customers aren’t
always the most representative ones. In fact, the things that make them vocal—
feedback in the application we see response such as an unusual or advanced use case—can actually make them the wrong
rates of 50% versus the 7 - 8% we’d get for customers to rely on for feedback.
emailed surveys.” The highest quality feedback typically comes from specific sets of users. For
example, input on how to improve a certain feature should come from its most
- Bradley Lafave, active users. Whereas feedback on the onboarding process should come from
Product Marketing, WebPT users who are new to the product. Targeted outreach based on identified user
behavior is key for getting quality feedback.
As a product leader, how can you gather better and more representative
customer feedback? First, get out of your comfort zone and find users who
accurately represent the customer population. And second, segment and target
those customers based on the area of the product you wish to improve.
Running High Value User Tests

The same considerations made when connecting with customers for feedback
should apply to those you are using for testing as well. User testing is
extremely valuable, and something that product teams should do as frequently
as possible. It provides valuable feedback on the user experience, and helps to
assess the impact of UI changes.
It’s vital to recruit quality candidates for usability testing. Remember that every
application has a “silent majority” of users. Look for great subjects that aren’t

necessarily already in your purview to get a full picture. You can also
rely on product data to identify quality participants. For example, how
much time users have spent in the product, whether they’ve used features
related to the ones you’re testing, how long they’ve been customers, their
functional or user roles or plan level can all be important. Take some
time to assemble a profile of the ideal test subject before beginning your
outreach.
Recruiting Testers for your Product:

There are two ways to go about recruiting testers for your product.
Traditional outreach includes the use of phone and email, but these
methods tend to have lower response rates. Customer advisory panels and
focus groups often have better results. But note that you may hear from
the loudest person in the room and miss valuable feedback from the “silent
majority.”
In-app outreach can be a more effective route. It lets you recruit users as a
seamless part of the product experience. Just be sure to target users based
on product use data rather than spamming your entire customer base.
What Should you Test?

Before you begin user testing, think about which tasks you’ll include, and
the key metrics you will need to collect. Will you be replacing an existing
workflow? Is your focus on high frequency or on low frequency tasks? What
are your key outcomes and goals? What is your testing sequence? Defining
these aspects will ensure that you are well prepared.

Chapter 4:
Actionable NPS

Chapter 4: Actionable NPS

Net Promoter score, or NPS as it is known, is a simple one- (or sometimes two)
question customer satisfaction survey that was developed by Fred Reichheld,
Bain & Company, and vendor Satmetrix. An NPS survey asks respondents
to rate whether they would recommend a product or service to friends or
coworkers on an 11-point scale from 0 to 10, with 10 being the most likely to
recommend. The responses are segmented into detractors (scores of 0 to 6),
passives (scores of 7 - 8), and promoters (scores of 9 - 10). An overall NPS score
is calculated by subtracting the percentage of detractors from the percentage
of promoters to return a single numerical score from -100 (everyone is a
detractor) to 100 (everyone is a promoter).
The idea is that this willingness to advocate is a measure of customer

satisfaction and perhaps loyalty.
According to a recent Pendo survey, 57% of product managers currently use

NPS to measure customer satisfaction with their products. Why? Net Promoter
surveys are simple. It is essentially a single question asked to your customers
that provides a reasonably clear, comparable benchmark of their satisfaction.
Yet, with all the variation in how responses can be collected, analyzed, and
followed-up on, implementing an NPS initiative can be trickier than you might
expect.
The reality is that, on its own, NPS isn’t a particularly actionable metric.
NPS provides a simple, consistent method- It shines light on the what--the likelihood of customers to advocate for a
ology that is widely adopted for quantifying product or brand–but not the why behind that sentiment. Without a deeper
understanding of customers’ behaviors, it’s nearly impossible to find the root
customer sentiment. The measurement is cause of positive or negative sentiment. The following chapter explains how to
make NPS actionable.
easily understood, and easily benchmarked
to track performance over time. 5 Tips to Make the Most of your NPS Strategy
1. Be Thoughtful in Your Targeting
Many organizations begin their NPS initiative by running a survey for their
entire user base. This can lead to some issues. If you’ve launched a survey

for your entire user base, will your team be able to follow up on all of the 3: Share your Results Broadly
responses?
NPS initiatives are successful when an organization is united around the
process. It’s difficult to effectively run NPS within a single department, as input
Start by defining the core segments you want to target, and then pull a
captured is important to many teams within the organization.
sample of those users to survey over time. For example, you may segment
by role, company size, industry, lifetime value or any number of other
The product team needs to understand how the experience delivered in the
characteristics. Knowing the difference in NPS score within and across
product impacts customer sentiment.
segments of customers or users allows you to make better inferences
about where your product is and isn’t delivering value.
The customer success team wants to know which customers are unhappy and
in need of intervention.
2: Use a Consistent Delivery Method
There are many ways to collect NPS survey responses. Historically, the The marketing team wants to know which customers are potential advocates.
most common method was via email. However many software companies
are starting to survey customers within their products. When everyone is engaged, the value of the program grows significantly.
Surveying users in-app ensures you reach active users while they’re “in Regardless of who initiates NPS, a best practice is to share the results broadly
the moment” using your product, which leads to higher response rates. for maximum impact to your organisation.
Different collection methods can have a significant impact on response
rates. Companies who move from email to in-app collection often see a 2x 4: Segment your Responses and Analyze Usage for Promoters
to as much as 10x increase in response rate. and Detractors
When running NPS surveys, it’s common to look at individual responses and, of
course, the overall score. What’s missed, sometimes, is the total NPS of specific
Companies who move from email to in- customers or customer segments. Rolling up the NPS responses for different
app feedback collection see response segments can yield important insights that would otherwise get lost in the
larger summary.
rates increase from 2x to as much as 10x.
One area to evaluate is different user roles. It’s common in enterprise software
to have different user roles using different parts of the software. Their
experiences, and thus the feedback they give you, can be quite different.
Your method of delivery can have a significant impact on the scores you Looking at responses from different roles can help you understand which
receive. Email may bias responses towards your most motivated (positively segments are underserved in your product, and provide a quick way to push up
and negatively) customers, while customers may be less likely to share your NPS score.
negative feedback to a live person on the phone. Even subtle changes can
make an impact. Make sure to stick with a consistent methodology as much Segmenting NPS responses also helps you examine the behavior of detractors,
as possible. A program that moves back and forth between email and in- passives, and promoters. For example, looking at which features are used by
app survey collection will introduce variability that can’t be attributed to different NPS respondents, and how often they are used, can tell you a lot
changes in customer sentiment, or improvements in service delivery. about how users realize value. Features that are used heavily by promoters

are likely providing significant value — especially if your neutrals and

passives don’t use them as actively. If you can identify a trend like this,
an easy way to improve customer satisfaction is to begin promoting those
features heavily to users who gave you a poor NPS rating.
5: Close the Loop

The NPS survey is only the first touch in a multi-step engagement with the
customer. After receiving the score, follow up to understand the reasons
behind it. The reality of many NPS implementations is that surveys are
pushed out and responses are collected, but only a small percentage of
customers receive follow-up over time.
To avoid this, consider putting a follow-up plan in place prior to survey

rollout with owners for each step of the process. Remember that your
support or customer success teams aren’t the only ones involved in this
process. Look to distribute that responsibility. For example, you may want
to have your marketing team handle follow-up with promoters — those
who give a score of 9 or 10. These are customers who are advocates, and
potential references. Product and UX teams should also get involved —
especially when usability issues or product shortcomings are cited in the
survey feedback.
If your team doesn’t have the bandwidth, try a triage method for your
follow-up. Look to connect with all of your detractors first, then follow-up
with those who take the time to provide additional feedback in the second
part of the survey. These people likely have the strongest opinions (both
positive and negative), and will give you the most valuable feedback. In
future surveys, consider reducing your sample size so that you won’t have
so many responses to follow up on afterwards.

Chapter 5:
Tailored
Onboarding

Chapter 5: Tailored Onboarding

When product teams talk about user onboarding, they typically mean the
user’s initial experience in an application. These onboarding experiences are
“Now when new users come into Insightly, designed to introduce the user to key features and functions of the product,
walking them through some sort of setup or training flow.
we use a custom guide design to present
users with a welcome modal that includes These simple onboarding experiences can be helpful, but they typically
don’t make the short list of engineering priorities for a couple of reasons.
a welcome video, Insightly basics videos First, the onboarding experience is often treated separately from the overall
user experience. It’s considered a ‘one-off’ and any updates or refinements
and getting started tasks. Prior to having this end up deprioritized on the roadmap. Second, onboarding initiatives are
onboarding modal we didn’t have any in- typically championed by sales, training or customer success teams within the
organization, rather than product management. Although product managers
product guidance on where to get started. will lead the implementation, they may not feel the same level of ownership or
Since deploying the modal, we’ve seen a accountability as they do for other product features.
1540% increase in viewership of the getting A Better Definition of Onboarding

started videos and average view duration
Part of the challenge stems from the fact that onboarding has a very narrow
increased 40%.” definition for most companies. Rather than thinking about onboarding as the
initial user experience in the application, companies should think about it as
the process by which users become proficient in the application. Perhaps an
- Teresa Roche, even better definition: the process by which users are able to quickly realize
Director of Customer Success, Insightly value from the product. This definition is helpful for a couple of reasons. First,
it opens the aperture beyond just the in-application experience to understand
that onboarding also includes any hands-on help, account setup, and training.
Secondly, and most importantly, it helps to clarify that onboarding doesn’t
just refer to a user’s initial experience in the application. It leads to all of the
experiences that create proficiency and, ultimately, habit.
The truth is that users will churn if they don’t realize value quickly. But it’s
equally true that users will churn if they don’t receive ongoing value. The SaaS
business model is built on the assumption of recurring revenue—customers
must renew and expand their usage for a product to succeed in the long run.
This means that product managers must consider how they deliver additional

capabilities at a velocity that meets customer expectations for both actual

and perceived value.
Only 40% of SaaS companies tailor their on-
Implications for the Onboarding Experience boarding experience for different user segments.
In addition to delivering against a high value, differentiated functional - The Power of Effective Onboarding, Pendo Survey, 2017
roadmap, product leaders must also deliver an exceptional onboarding
experience. New functionality is of no value if users are unable to discover,
and learn how to use these features. This means that product managers
must think through the onboarding experience. Taking over the UX for Measuring Onboarding
an introduction and initial walk-through is fine (if not ideal) for the user’s
first experience, but it definitely isn’t a workable approach for every Part of the reason that product teams don’t approach the onboarding
new capability that is introduced. Also, if onboarding never truly ends, experience broadly enough is because the measurements are too myopic.
product teams need to address how to manage the volume of onboarding Certainly, you want to understand how customers are engaging with the
content. Adding too much guidance or training to the user experience can onboarding content itself, but it’s also critical to think about broader user
add clutter and ultimately degrade the experience. The answer here is behavior, and how your onboarding is influencing the entire customer journey.
segmentation and targeting. Onboarding measurement can be thought of on three levels:
An effective approach is one that customizes both the onboarding Onboarding Content
content and the delivery to the user based on role context, sophistication
and learning style. User context includes behavioral and demographic At the first level, measure user engagement with the onboarding content itself.
information such as: Here, you want to understand how many users actually spend time with your
onboarding content, how much of it gets completed, and how much time it
• Time spent in the application takes users to get through the steps.
• Features used
A couple key measures to consider at this level include:
• Previous onboarding / guidance viewed
• Engagement rate (any usage) for onboarding content
• Functional job role
• Completion rate of onboarding walkthroughs or courses
• Application role (i.e. admin vs regular user)
• Average time spent in onboarding content
As a simple example, you probably wouldn’t want to offer help for a
particular feature to a user that has already used that feature several But these measures alone won’t paint a complete picture of the effectiveness
times. The offer would be intrusive, not helpful. This same principle applies of your onboarding. If users don’t engage with onboarding content, then it’s
to features and capabilities to which a user may not have access based likely not perceived to be helpful. If they don’t complete it, the content may be
on their entitlements or plan level. Any help for that feature should be too long or unfocused.
hidden from them as well. By limiting content like this, product managers
can ensure that their onboarding experience is as relevant as possible, and
doesn’t unnecessarily clutter the user experience.

complete the content, and those that don’t. Usage differences between
Companies that effectively onboarding new segments can help to pinpoint the specific outcomes associated with your
onboarding experience.
users into their products report 4x higher
revenue and 4x higher profitability than their Business Outcomes
The final level of measurement is business outcomes. These can be the
competitors. most difficult metrics to attribute to onboarding. Here, you are looking at
specific business and customer outcomes – things that ultimately drive the
- The Power of Effective Onboarding, Pendo Survey, 2017 success of your product. How much cost is associated with acquiring and
supporting customers? How satisfied are they? What is their retention rate? Key
measurements to consider at this level are:
• Trial or freemium conversion rate
Product Behavior • Number of support tickets per customer
The next level of user onboarding measurement shines light on the • Net Promoter Score or other customer satisfaction measure
user behavior within the product itself, not just the onboarding guides,
walkthroughs or other content. Here, your goal is to measure how users • Customer and ARR churn rates
engage with the product after they have engaged with the onboarding
content. You want to understand how much of your product is used and the Do customers that engage with onboarding content generate fewer support
adoption of specific features covered in the onboarding steps, as well as tickets? What about those that complete the content? Are there specific
how often users log into your application, and how much time they spend behaviors or features that, when used regularly, drive higher conversion or
in the product when they do. A couple of key measures to consider at this satisfaction rates? By carefully segmenting your user base, you can start to see
level are: the specific impact that onboarding is having on the overall product success.
• Depth of use or the percent of product features that are used by the Bringing these three tiers of onboarding metrics together can give you a clear
average user and complete picture of the effectiveness of your user onboarding. With better
• Usage rate of specific features covered in the onboarding training visibility, you can begin to optimize the user onboarding experience.
• Overall average time in the application or login frequency
This data will give you a more accurate picture of your onboarding
effectiveness. Ultimately the goal of onboarding is to increase user
proficiency, driving the types of activities in the product that are associated
with highly engaged, happy customers.
The best way to see if you’re having an impact is to measure the actual
user behavior in app. Look for behavioral differences between segments
that have and have not gone through the onboarding steps, those that

Chapter 6:
Ongoing Customer
Education

Chapter 6: Ongoing Customer Education
Chapter 6: Ongoing Customer

Education
When you launch new products or features, you have some teaching to do.
Your users need some education on the underlying problem and how your
“Our job in educating customers goes beyond product or feature solves it. They need to know why it’s better than the old
way, what value it brings and how it actually works.
product features. We have to educate people
on not just how to use the product, but quite As you define a strategy for ongoing customer education, consider the
following:
frankly, that they need the product.” • Users learn at different speeds. Some will pick things up immediately,
while others may require a bit of hand-holding. Try to make it fun, but take
- Adam Avramescu, Head of Customer the time to show them the different aspects of your product.
Education at Optimizely • Meet your users where they are. You can do this by educating customers
through every channel available. Team up with your marketing and
customer success organizations to get the message out and use email,
client platforms, press releases, articles, blogs and webinars. Promote
new products across a variety of outlets to reach as many customers as
possible.
• Illustrate the functionality. Allow users to see your product in action and
find the value themselves. You can do this with a live demo, a video or an
in-app guide letting users test things out themselves.
• Walk users through step-by-step. In-app guidance allows you to highlight
new features, drive preferred behavior, and provide in-context support.
Contextual, personalized guidance provides help when and where it’s
needed, simplifies any user experience, and improves overall usability.
There is no end date when it comes to customer education. That’s because

the time required for a user to become proficient really isn’t at all consistent.
For some products, that could happen during the first use, and for others it
happens through months of usage. It’s important for product and customer
success teams to be aware of this. Many product usage issues surface when
the content and experience is designed for a specific timeline that doesn’t align
with how the user actually learns.

Chapter 6: Ongoing Customer Education
Measuring Customer Education

WebPT reduced support tickets 50% by adding
As a product manager, it’s easy to fall into the trap of thinking that
education isn’t a product problem and should be owned by customer contextual tooltips to educate users in their
success or customer education teams. But while these teams may have product.
primary responsibility for customer education, product teams should make
sure that customers maintain proficiency in the product as new features
and updates are rolled out. Without the right measurements, however, it
can be difficult to keep this top of mind. There is no one-size-fits-all solution. But data-driven product managers can use
these insights to help deliver excellent and continued customer education.
There are three key ways that product teams can evaluate the
effectiveness of ongoing customer education:
• Engagement with training content: Product teams should measure
customer engagement with any content that is provided to train users
of new features, especially if that content is surfaced in-app. This
measure doesn’t necessarily indicate the effectiveness of the content,
but it shows whether users are curious and eager to learn about
updates.
• Support ticket volume: The number of support tickets associated
with new features or updates can be an excellent measure of the
effectiveness of ongoing education. If users cannot understand how
to use new features or navigate an updated interface, they will often
contact support. If significant updates are rolled out without growing
support volume, the education content was likely on point.
• Long-term retention: Customer time-to-value is an important measure
for initial conversion and sale, but continued proficiency is just as
important for SaaS products. If a customer is frustrated by interface
changes, or can no longer complete tasks as efficiently in a product,
the prospect of renewal is likely to decrease. Customer education
impacts this measure—retention should be considered an important
downstream measure of education effectiveness.
Remember that the right amount of customer education requires a fine

balance. Too much content can mean that the product isn’t intuitive and
can get overwhelming for users. Make sure that you are evaluating how
much help is needed and communicating accordingly.

Chapter 7:
Feature Awareness
and Adoption

Chapter 7: Feature Awareness and Adoption
Chapter 7: Feature Awareness and

Adoption
Product teams are naturally excited when they prepare to roll out a major new
feature. And in a perfect world, that feature is immediately and enthusiastically
“We have a calendar interface in the product, adopted by every customer. In reality, that’s rarely the case. Feature adoption
tends to be sporadic, and a lot of teams don’t have visibility into how their
but the ‘Sync my Calendar’ button is located product is used to truly understand how widely a feature is adopted.
in your personal settings so they’re on
Every product team wants to build features that provide value to customers, but
different pages. We deployed a small guide doing so requires effective customer feedback, appropriate measurements, and
that would pop up and and say ‘Hey – did an ability to rapidly drive awareness around new updates.
you know you can get these dates directly Why Product Success Depends on Feature
in your calendar?’. Users that saw the guide, Adoption
their usage of the feature went up by 2000%.
With the shift to subscription-based software licensing, software is no longer
It’s amazing how little announcements like just purchased once. Software products are purchased over and over again
this can have such a huge impact.” – sometimes as frequently as every month – as customers renew their
subscriptions. Each and every renewal is contingent on customers perceiving
and receiving ongoing value from the software. Each new product feature
presents an opportunity for additive value if customers are aware of and
- Kristen Hariton,
actively using them.
Senior Product Marketing Manager, SiteCompli
Unused features, however, can have an adverse effect. This is why product
teams are increasingly focused on overall product adoption. Every piece of
a product that isn’t used represents something a customer is paying for, but
not realizing value from. Underuse lowers perceived value and ultimately a
customer’s willingness to pay for a product.
Measuring Feature Adoption

At the surface, this seems simple enough: are or aren’t customers using
the feature? While that is definitely a measure of usage, it may not be the
best benchmark for adoption. Consider the following scenarios. Software

company A releases an update, and publicizes the update broadly to important information is to ask for it. Look for opportunities to collect feedback
the current user base. As a result, over 40% of users use the feature over when users are interacting with a new feature for the first few times. Some
the next week. However, a week later almost none of them continue to companies prefer open-ended feedback, while others use a number scale or yes/
use the feature. Software company B also releases and publicizes a new no questions to gather a baseline on perceived user value.
feature. Only a tiny percentage of users pick up the feature, but they
enthusiastically continue to use it.
Promoting Feature Launches
Both scenarios are examples of feature adoption, but neither would be
New software features will never see significant adoption if the user base
judged particularly successful. Neither feature provided significant ongoing
doesn’t know about them. So the announcement and discovery process are
value to customers. When measuring feature adoption, companies should
also important parts of driving adoption.
consider the following dimensions:
• Breadth of adoption: How widely has a feature been adopted across There’s no one ‘one size fits all’ way to announce features, but there are
the user base or targeted user segment? Has the feature been picked a couple considerations that can help to shape the strategy. The first one
up by a majority of the targeted users, or only a small percentage? is relevance. Users are much more likely to respond to announcements
Looking at the breadth of adoption shows the initial appeal of the new that matter to them. Software applications—especially business software
feature. applications—have diverse user bases with different roles, maturity,
• Time to adopt: How long does it take for users to begin using a new and technical proficiency. Very few features are deeply relevant to all
feature. When learning about a feature, do users immediately try it out users. Therefore, the announcement approach should be tailored to the
or do they wait several days or weeks before picking it up? Looking most appropriate user segments. Whether a new feature is relevant to
at adoption time provides input into motivation. The more quickly a prospective customers in addition to the current user base can also shape the
feature is adopted, the more likely it addresses a significant customer announcement strategy.
pain or usability problem.
The second consideration is desired action. What should users do upon
• Duration of adoption: How long do users continue to use a feature reading the announcement? Try it out? Read documentation about how to use
after learning about it? Do they try it out a few times, or continue to it? Provide feedback? The desired next action can also have an impact on the
use it regularly? This is an important measurement as it helps to show best way to announce a new feature.
whether a feature is providing any real value beyond its initial novelty.
In many cases, the product itself can provide a powerful channel for new
What constitutes successful adoption across these three dimensions is feature announcements. Delivering feature announcements or promotions in
obviously going to vary from use case to use case, but it’s important to the form of in-app modals or tooltips ensures that the announcement reaches
consider all three when assessing the outcome of any feature release. users at a highly relevant time (when they’re using the product). A best practice
is to segment the announcements for different groups of users for even greater
Feedback is a Critical Component of Adoption relevance.
Effective measurement will help product teams understand the extent to Often, the primary next action for users is to try the feature. If the
which new features are adopted, but it certainly won’t tell them why, or what announcement is served directly in the product, there’s nothing stopping
users really think about a particular feature. The only way to collect this the user from trying it out. With email or blog announcements, the user must

either immediately login to the product to try out the feature, or attempt to
remember the announcement the next time she uses the product.
Infusionsoft increased feature adoption by

1000% by adding in-app feature promotion.
Improving Feature Adoption

Increasing feature adoption ultimately comes down to the value that each
feature delivers. But understanding that value requires clear insight. To
understand adoption, product teams should measure the breadth, time,
and duration of feature adoption, and pair those metrics with direct user
feedback about specific features.
Discoverability also plays an outsized role in feature adoption. By

leveraging the right promotion strategy, product teams can ensure that
highly-actionable announcements reach the users for whom they will be
the most valuable.

Closing
“Listening offers data. Hearing offers empathy and intelligence. Activity, action, and
engagement steer perspective and encourage a sense of community and advocacy.”
- Brian Solis
Putting this handbook to work is one of the most important things you can do to achieve your long-
term goals as a data-driven product leader. Use this book to get everyone aligned and working like
a streamlined PM machine. Then, use these tips to deliver exceptional product experiences to your
customers, driving adoption, retention and overall product love.
We encourage you to take our guidelines and include them in your own organizational handbook.
If you don’t have one, use this book as a baseline, implementing the chapters that resonate with
your team, and adding your own practices too?
Be sure to visit Pendo for more product insight: www.pendo.io
© 2018 Pendo I/O. All rights reserved.

Data Driven Product Manager Handbook

Uploaded by

Copyright:

Available Formats

Data Driven Product Manager Handbook

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Driven Product Manager Handbook

Uploaded by

Copyright:

Available Formats

The Data-driven

How to use this handbook?...........................................4

The Data-Driven Product Manager’s Handbook A Pendo ebook 2

The Data-Driven Product Manager’s Handbook A Pendo ebook 3

The Data-Driven Product Manager’s Handbook A Pendo ebook 4

The Data-Driven Product Manager’s Handbook A Pendo ebook 5

Chapter 1: Product KPIs

ARR / MRR = Annual or monthly recurring revenue associated with subscribers

The Data-Driven Product Manager’s Handbook A Pendo ebook 6

The Data-Driven Product Manager’s Handbook A Pendo ebook 7

The Data-Driven Product Manager’s Handbook A Pendo ebook 8

Chapter 2: Data-driven Roadmaps

Start with Vision and Strategy

The Data-Driven Product Manager’s Handbook A Pendo ebook 9

A product investment has no place in the product roadmap if its value

The Data-Driven Product Manager’s Handbook A Pendo ebook 10

should reduce support requests by X%), or specific financial metrics (migrate

The Data-Driven Product Manager’s Handbook A Pendo ebook 11

The Data-Driven Product Manager’s Handbook A Pendo ebook 12

Chapter 3: Better Customer Feedback

Running High Value User Tests

The Data-Driven Product Manager’s Handbook A Pendo ebook 13

Recruiting Testers for your Product:

What Should you Test?

The Data-Driven Product Manager’s Handbook A Pendo ebook 14

The Data-Driven Product Manager’s Handbook A Pendo ebook 15

Chapter 4: Actionable NPS

The idea is that this willingness to advocate is a measure of customer

According to a recent Pendo survey, 57% of product managers currently use

The Data-Driven Product Manager’s Handbook A Pendo ebook 16

The Data-Driven Product Manager’s Handbook A Pendo ebook 17

are likely providing significant value — especially if your neutrals and

5: Close the Loop

To avoid this, consider putting a follow-up plan in place prior to survey

The Data-Driven Product Manager’s Handbook A Pendo ebook 18

The Data-Driven Product Manager’s Handbook A Pendo ebook 19

Chapter 5: Tailored Onboarding

1540% increase in viewership of the getting A Better Definition of Onboarding

The Data-Driven Product Manager’s Handbook A Pendo ebook 20

capabilities at a velocity that meets customer expectations for both actual

The Data-Driven Product Manager’s Handbook A Pendo ebook 21

The Data-Driven Product Manager’s Handbook A Pendo ebook 22

The Data-Driven Product Manager’s Handbook A Pendo ebook 23

Chapter 6: Ongoing Customer

There is no end date when it comes to customer education. That’s because

The Data-Driven Product Manager’s Handbook A Pendo ebook 24

Measuring Customer Education

Remember that the right amount of customer education requires a fine

The Data-Driven Product Manager’s Handbook A Pendo ebook 25

The Data-Driven Product Manager’s Handbook A Pendo ebook 26

Chapter 7: Feature Awareness and

Measuring Feature Adoption

The Data-Driven Product Manager’s Handbook A Pendo ebook 27

The Data-Driven Product Manager’s Handbook A Pendo ebook 28

Infusionsoft increased feature adoption by

Improving Feature Adoption

Discoverability also plays an outsized role in feature adoption. By

The Data-Driven Product Manager’s Handbook A Pendo ebook 29

© 2018 Pendo I/O. All rights reserved.