3 Product Metric Pitfalls You Must Keep away from

In my e book, The Insights Pushed Product Supervisor, I cowl why it’s essential to trace much less to create extra focus and spend extra time on extracting true insights out of your information.

The subsequent step is to be sure that on the nitty-gritty stage, you might be monitoring what I name “good high quality metrics.” This publish—an excerpt from chapter 7 of my e book—will concentrate on the best way to enhance the general high quality of your metrics, the best way to make them extra actionable, and what pitfalls to keep away from.

Pitfall #1: Self-importance metrics

Just a few years in the past I used to be engaged on a B2B SaaS product for managing workplace environments. We simply launched the product and began to run our first paid promoting campaigns, so I arrange a dashboard that tracked the full variety of sign-ups over 30 days:

The numbers gave the impression to be going up, so we had been fairly comfortable concerning the momentum.

The issue was that whereas this graph seemed good on shows, the arduous fact was that solely 4% of the brand new sign-ups became precise conversions and income, and consequently, we didn’t meet our month-to-month recurring income targets.

It’s a cumulative chart, so the worst case can be that the graph would simply plateau if we don’t purchase any new customers, however the quantity can by no means lower. It’s a traditional instance of a conceit metric:

• this graph made us really feel good.
• This metric was particularly helpful in stakeholder shows.
• It didn’t give us any perception into whether or not we had been truly doing nicely, or not.
• As a result of it didn’t give us any perception, it didn’t drive us to take motion to enhance the product or options in any approach.
• And regardless of taking a look at this metric each day, it took us two months (by the point all of the 30-day trials had completed and churned) to seek out out that there was a problem.

Whereas one a part of the issue was our lack of information on the best way to measure higher metrics at the moment, the issue in organizations usually lies lots deeper: most groups or stakeholders are merely not prepared to listen to the reality from their metrics, so we search for the numbers that make us look good.

In my interview with Crystal Widjaja, CPO at Kumu and author for Reforge, she summarized fantastically the best way to view information as a solution to capitalize on failures and drive enhancements as an alternative:

“When folks make errors (failed experiment, failed deployment, and many others), you’ve already paid that value. We must always consider information as a solution to capitalize on errors and study from them. Reasonably than ‘paying the tutoring of the failure’ and firing the person, use the info insights to inform us WHY it was a failure, study from it, and leverage it for the following iteration to be 10x higher than the primary.”

– Crystal Widjaja

With the intention to get extra insights out of your information you really want to cease monitoring vainness metrics, and as an alternative use information to uncover the reality and drive precise enhancements. For those who look carefully, it’s fascinating how usually groups are displaying very selective metrics to appease sure stakeholders or make the numbers sound higher than they really are. Be careful for different traditional examples of vainness metrics similar to:

• Variety of web page views or guests
• Variety of followers/likes
• Time spent on web site (session size)
• Variety of downloads

Metrics like web page views and session size are nonetheless closely utilized in web site analytics, the place the main focus is to measure site visitors, consciousness and preliminary engagement. They provide you some perception on what we name the highest of the funnel—the preliminary acquisition of shoppers—however not whether or not prospects are literally activating and interesting with the product, which could have a way more significant correlation along with your wider product and enterprise targets.

How one can do it higher: to actually perceive if a metric is sweet or dangerous, we have to put numbers into context. On the very minimal, you wish to attempt to examine a quantity over completely different time durations, similar to evaluating your sign-up numbers this month versus the earlier month.

One other efficient solution to make your metrics extra helpful is to make use of ratios as an alternative of complete numbers. Ratios are inherently comparative. For example, accountants don’t simply take a look at complete income, however usually examine the prices of manufacturing a product with the gross sales they comprised of it. This manner accountants can monitor their revenue margin (a terrific instance of a helpful ratio) over time to evaluate whether or not the enterprise is wholesome.

Examples of higher, extra comparable metrics:

• % of sign-ups per acquisition channel
• % of sign-ups who accomplished the total sign-up course of
• % of sign-ups who carried out a key activation metric
• % of customers utilizing the product after 4 weeks

Pitfall #2: Solely monitoring lagging metrics

A giant downside was the period of time it took us to seek out out whether or not we had been hitting our conversion targets (or not). The product had a 30-day free trial and our objective was to transform them to paying prospects after the top of the trial, so whereas the primary month seemed good by way of sign-ups, we might finally solely know by the top of the second month what number of of these sign-ups transformed to paying prospects.

It is a traditional instance of a lagging metric. Lagging metrics report retrospectively on previous outcomes. For instance, your income numbers for the yr are lagging metrics like most of your different operational metrics. You solely know whether or not you probably did nicely after you have the outcomes.

The true worth in monitoring person habits via your product analytics is which you could begin to search for earlier indicators than having to attend on your remaining income numbers. In case your main metrics don’t carry out nicely, you will have the prospect to course-correct earlier than it’s too late. Because of this I designed the Holistic Metrics One Pager in chapter 5 of my e book to incorporate each buyer habits and operational metrics, so groups can monitor a wholesome mixture of main and lagging metrics to get the total image.

One of the crucial highly effective main metrics is the activation metric. An excellent activation metric represents the proportion of shoppers who take a key motion of organising or beginning to use the product. Many corporations have found out that if customers do a sure motion inside their product throughout onboarding, they have an inclination to understand the true worth of the product which ends up in greater engagement additional down the road. Some name this activation step reaching the “aha second” of their product.

Listed below are some easy examples of main activation metrics:

• Social community product: a traditional instance was Fb’s first activation metric of including a minimal of seven pals in 10 days.
• Dashboard aggregation product: the worth proposition is to bundle a number of instruments into one view, so that you may discover that customers who add a minimal of two or three instruments throughout onboarding notice the total potential of the product.
• Utility product: your worth proposition is likely to be to simplify or digitize a job similar to monitoring gross sales conversations in a CRM, so you may monitor the variety of customers who full their first buyer entry as shortly as doable as an activation metric.
• Consideration product: in case your product is centered round leisure and content material you may monitor customers who consumed a specific amount of content material within the first week of signing up .

Lagging metrics should not inherently dangerous, by the way in which. The truth is, they’re a crucial a part of reporting, particularly for measuring enterprise metrics similar to your monetary outcomes. Their benefit is that they signify the ultimate outcomes, the actual information.

Main metrics then again usually embody some quantity of assumptions like the idea {that a} excessive quantity of chilly calls every single day will increase the variety of paying customers additional down the road. As you get extra information you need to take a look at whether or not these assumptions are literally true, however even then there may be nonetheless some uncertainty on whether or not the activation metric really prompted the rise in retention, or whether or not different elements contributed to it.

This implies main metrics won’t ever be as correct as lagging metrics, however they’re essential to getting true insights out of your metrics. They permit us to study from buyer habits and establish early indicators which will change our product choices to optimize for higher enterprise outcomes additional down the road. Utilizing the Holistic Metrics One Pager template forces you into monitoring each main and lagging indicators, in addition to to consider how these affect one another.

Pitfall #3: Metrics nobody understands

After I interview product managers I usually hear that analytics information and information insights get hidden away in darkish mysterious corners of places of work, with occasion names that nobody however a few extremely specialised analysts perceive. Each month these specialists would meet with varied product groups in an try and share and translate a few of their findings.

If we would like our product groups and stakeholders to create a shared understanding of our information and focus on enhancements to the product collaboratively, we have to actively work on democratizing our information, be sure our metrics are accessible to everybody and straightforward to grasp.

Intercom shared their learnings from doing an enormous occasions clean-up a couple of years in the past. That they had round 350 occasions for his or her product that form of seemed like this:

Does this look acquainted to you?

Intercom shared that they failed a key precept of analytics: they made little or no sense to anybody however the analytics staff. They redefined and rebuilt their whole naming construction of all their occasions to introduce higher readability as a key step to democratize their product analytics information.

It’s additionally essential to make stories extra accessible for varied stakeholders and groups within the group. Sadly, I usually see groups afraid of opening up their dashboards, as it might once more uncover the true engagement or acquisition numbers that won’t look nice to stakeholders. To keep away from uncomfortable conversations or pesky questions, it’s usually simpler for groups to cover behind a veneer of complexity.

How one can do it higher:

• Step 1: Work along with your engineering groups and analysts to simplify your product analytics occasion names: “Accomplished Onboarding” and “Added Dashboard Widget” are actions that everybody will perceive.
• Step 2: When you’ve got an analytics staff, embody them higher in your product groups. The extra context the analysts have of what your product staff is engaged on, which experiments you’re testing and which questions have to be answered, the higher they will help you to dig into the info to seek out probably the most related insights. It needs to be a collaboration reasonably than an outsourcing strategy.
• Step 3: Make your analytics dashboards and stories accessible to the broader group. Your dashboards ought to mirror your product’s key metrics (which you’ll be able to outline utilizing the Holistic Metrics One Pager from the e book). That is crucial for scale (your staff doesn’t wish to get flooded with guide reporting requests every single day) in addition to to really construct a extra data-driven tradition throughout the wider group.

“When groups are requested concerning the state of the enterprise, they will both go look it up or make up hypothetical guesses. It’s crucial to make the previous the best, default approach for management to answer these requests by constructing customized, easy-to-use drill-down dashboards for issues like cohorts, funnels, and person occasions.”

-Crystal Widjaja

Keep in mind that the job we employed our information to do is to uncover the reality so we are able to take motion and enhance our product experiences. Making your metrics simple to grasp and extra accessible are key steps to incorporate information insights into the day-to-day determination making in your group. A powerful product group needs to be extra motivated than ever to resolve these issues as soon as they know the place the issues lie.

How one can enhance your metrics utilizing the metrics guidelines

I created a easy guidelines that summarizes key traits of excellent high quality, actionable metrics that may enable you get extra insights out of your information. Use this guidelines to evaluate and enhance all of your current metrics:

It takes actual observe to really get your key metrics proper, and one can find the satan usually lies within the particulars. It’s completely regular, and actually inspired to continuously revisit the metrics you will have chosen, and to refine them a number of instances to make them extra helpful.

Be careful for the pitfalls of sharing vainness metrics, focusing an excessive amount of on lagging indicators the place you don’t have any time to course-correct, and ensure you simplify and democratize your metrics to really stage up the info maturity in your group.