Learning Russian

Use the Metrics Maturity Model to improve how you measure and report your progress.

“You can only make as well as you can measure.”
Joseph Whitworth, 1840

Successful founders, looking back, often talk about how they used data to track their progress and inform their decisions.

But how did they get started?

Is using data in this way a behaviour that you can learn?

In my experience, during the early days of a new venture, it’s far more common for both founders and investors to feel completely overwhelmed or outright misled by data than to feel informed or driven by it. Even if you understand the importance of looking at your numbers constantly and sending regular updates, starting with a completely blank spreadsheet is still a daunting prospect.

So please don’t panic if you feel like that! It’s completely normal.

Let’s map out the steps you can take, on the path towards metrics enlightenment.

M3: The Metrics Maturity Model

Level 1: Drowning in data.

“Start simply by asking what are the most important facts in your organisation and how many people know them”
— Hans Rosling, Factfulness

In the very beginning all you have are numbers.

The best analogy I have is the science fiction nightmare of waking up to find yourself locked in the control room of an abandoned Russian nuclear power plant that is just about to melt down.1 You can see the control panel, with all of the dials and switches, and you know you need to do something, but none of the labels make any sense.2 All you can do is rapidly try to learn which control is which by trial and error.

It’s impossible to know exactly which of the numbers you have are going to matter most to your venture as you grow, so to get started the best you can do is just start to track everything you can, beginning with a standard template of metrics that are applicable to your business model.

Make a list of all of the various systems you use that are collecting numbers about your venture: your finance or billing system, your website analytics, your customer support system, your sales pipeline system, etc etc. Then start to collate everything in one place. This will give you a foundation to build on when you come to more detailed analysis.

Action: create a spreadsheet, or use a tool like One Metric, to collect all of your numbers in one place.

Level 2: The habit of regular updates.

“I must write each day without fail, not so much for the success of the work, as in order not to get out of my routine.”
— Tolstoy

It’s tempting at this point to jump straight into analysis.

But, wait!

Often it’s not the complexity of the maths or how many three-letter-acronyms you have that makes the difference - it’s the consistency of the habits that are formed early around regularly updating your numbers, looking at the trends and sharing them with somebody who can help you decipher them. Maybe you can do that all by yourself, but more likely it will be somebody else on your team, or an external investor or advisor. It’s harder than you probably realise to punch holes in your own reality distortion field!

It’s shocking how many founders never get beyond this step - either because they don’t have numbers in a format that are easily shared or they are embarrassed by the numbers themselves (and then mistakenly think that the best fix is to keep quiet until the numbers are better!)

This is no different to the advice that is given to people who want to improve their diet or be more active. The most important thing is to be consistent and regular than it is to be amazing infrequently but neglectful the rest of the time.

If you regularly exercise it makes a big difference to your wellbeing and once the habit is established it’s much easier to do consistently.

Likewise, if you regularly review your numbers with somebody who can be objective it makes a big difference to your venture, and once the habit is established it will become something that you and your team do intuitively.

Action: find somebody else who is interested and share your metrics with them on a regular basis - possibly as often as weekly in the very early stages, but at least monthly as you grow.

Level 3: Crossing the streams

Once you have a handle on the basic metrics associated with your business, and you’ve formed the habit of updating and sharing them regularly, the next job is to be more savvy about the analysis you do.

Often the most interesting metrics are not the things that you can record directly - e.g. the number of unique visitors to your website, the number of paying customers you have, or the revenue you earned - but the combinations, derivatives and countermeasures.


What are the ratios that measure your progress?

Often these are more nuanced than the raw numbers, because they are more easily compared across different time periods and to industry benchmarks.

For example, if you have an online retail business, rather than looking at the number of new customers you gained or the amount you spend on advertising, combine those to calculate your cost of acquisition (often abbreviated to CAC) - that is, the amount you spent on average over the last period to acquire each new customer.

Or, for any kind of business that is not yet profitable, rather than recording how much cash you have in the bank, and how much cash you are burning (total expenses net of revenue), combine those to calculate your runway - i.e how many months you can continue in that mode before you run out of money?


What are the first or second derivatives of your basic metrics?

For example, if you have a SaaS business, in addition to measuring your average revenue per subscriber (often abbreviated to ARPU), calculate the change - is the ARPU increasing or decreasing? Or, if you want to be really fancy, calculate the acceleration - is the increase (or decrease) itself increasing or decreasing?

Often, to really get value from these kind of metrics you need to look back over a few periods, so you can really understand the trend and you are not distracted by one abnormal data point.


Whenever you have a metric that is looking really positive, it’s useful to ask yourself: what is the countermeasure?

Often when you are very focussed on one thing, such as growing the number of customers you have, you can lose sight of the collateral damage this is causing in other areas of the business. Having a good countermeasure will ensure that you have healthy growth.

A good way to identify countermeasures is to put on your tinfoil hat and ask: if somebody in the team was maliciously trying to juice this metric, how would you tell?

For example, if you have a marketplace business, then one of your key metrics is the inventory you have (i.e. the number of active listings on a classified site). But, it’s not just the raw amount of inventory that matters, but also the quality. So, you might also measure the ratio of sales to inventory (often called the sell-through rate), or the ratio of unique sellers to inventory (the listings per unique seller), as a way of checking that you have valuable listings and not just lots of listings.

Or, a classic example, if you measure the average time it takes to respond to a customer enquiry then, as a countermeasure, you also want some way to measure the quality of the responses, so that your customer service team don’t just rush off poorly-considered responses, but actually try to solve the problem the user is experiencing.3

Level 4: Lines in the sand

Once you’re collecting, analysing and sharing your metrics, the next step is to create feedback loops, so that you can constantly improve.

Choose the metrics that you think are most important and set yourself a target for each of these - either a level or range. These could be driven by your business model, your cash position or just arbitrary growth objectives. Either way, you should try to be realistic about how quickly and how impressively you can make changes and achieve these results.

Once you have goals defined, whenever you update your metrics compare your actual performance to where you imagined you would be, and think about the gaps - where have you made improvements?

This is a form of debrief, and if you do it regularly you’ll get much better at it.

Some good questions to ask are:

  • What went well?
  • What didn’t go well? (not who was at fault, but what and why?)
  • What surprised you? (and why?)
  • What did others do in response (good and bad)?
  • What did you learn?

You should write down these things as you review your metrics and consider why you did or didn’t hit your targets. Then you can look back at those in the future and ask if those reasons have stood the test of time, are recurring excuses (a problem!) or have been something that you were able to correct for subsequently.

Lagging vs Leading

Most of the things that are easily measured are lagging indicators, showing what has happened in the relatively recent past.

It’s much more valuable if you can uncover a leading indicator - i.e. something that helps you to know that you’re already on the right track.

Probably the most famous example of a leading indicator is the engagement measure used at Facebook. The growth team there discovered that if a new user has connected with seven friends within 10 days of creating an account they typically remain much more engaged as a user. That’s a hugely useful thing to understand, as it then focusses the team on how to get as many new users as possible across that threshold.

Action: set realistic targets for each of your key metrics and check back in future months to compare what you actually achieved to the original target.

Level 5: The One!

By now you should have a much better grip on which metrics matter the most to your venture. You’ll be sharing these regularly with others and comparing your actual results with predefined targets.

By creating these feedback loops you’ll start to narrow the long list of metrics you track down to a more focussed basket of metrics that capture the most important aspects of your venture and clearly show how you are currently tracking. These are likely to be quite specific to your specific business model and stage, and will almost certainly change over time as old problems get solved and you grow into new areas with their own new problems.

In their book Lean Analytics, Alistair Croll and Benjamin Yoskovitz define useful metrics as those that have these five attributes:

  • Simple - i.e. a single number or ratio
  • Immediate - i.e. can be captured quickly
  • Actionable - i.e. you can do something about it, once you know
  • Comparable - i.e. with itself, over time, or by benchmarking to other similar companies
  • Fundamental - i.e. is correlated with your overall success

To achieve metrics nirvana, you just need to choose one of these metrics to obsess about. And then do everything you can think of to improve it.

Focus in this situation means stripping your full list of metrics back to just those that indicate what you’re doing at the moment is working (or not!) - ie the things you can take action on.

Perhaps you’ve already identified the single metric that clearly correlates with your success.

Or maybe you’re still searching but know there is a specific area where you’re not doing as well as you think you can and want to really focus on that for a bit.

Often the best metric to choose will be a composite metric that captures several different and ideally competing aspects of your venture in a single number.

For example, in the early days of Trade Me we realised that one of the most impactful things we could to do drive our growth was increase the number of unique sellers listing items for sale. One seller listing lots of similar items was good, but lots of different sellers listing items was much better - more diverse listings meant more buyers, more feedback placed, more stories told by happy sellers, more word-of-mouth growth. We started tracking this metric daily and as a result were able to quickly identify a number of improvements we could make to increase this number - mostly by making the process of listing an item for sale much more obvious.

Another more recent SaaS example is Tomas Tunguz’s Cost of a Recurring Gross Profit Dollar metric (CRGPD). This is a great composite metric: it captures lots of different aspects of the SaaS business model in a single number (including the gross margin, acquisition cost and churn); it gives you a single number which is immediately meaningful (if it’s costing more than $1 to gain $1 of recurring gross profit then that should immediately raise concern); and by tracking the number over time you can quickly see progress (a lower cost is better than a higher cost).

Action: Determine what is your most important metric at the moment, and then obsess about what you need do to improve that number?

What are you waiting for?

They say that the hardest part of going for a run is putting on your shoes. Once you’ve done that then actually heading out the door is an obvious next step.

And it’s the same with all of this. Don’t let a blank spreadsheet scare you from taking the first step. Once you get started you’ll find that each of the levels are a natural progression from the last.

So get going and perhaps one day you’ll be one of those successful founders looking back talking about how much data helped you understand your venture and track your progress.

  1. Credit to Munjal Shah, the former founder and CEO of like.com, for this idea from a 2007 blog post that is sadly no longer online:

    “Launching a new site is like becoming the owner of a brand Russian nuclear power plant. You have a ton of dials with labels you can’t read. The only dial you can read is the amount of electricity (in our case revenue) and the temperature of the nuclear core (in our case number of clicks you are sending to merchants).”

  2. Obviously this doesn’t apply to any readers who speak Russian. In that case, please substitute whatever language would be most indecipherable to you. ↩︎

  3. In his book 21 Dog Years, Mike Daisey tells the story of the early customer support team at Amazon who were measured based on the average time taken to resolve a customer enquiry. They would hack this number by hanging up on customers who called with difficult questions. ↩︎

Related Ideas