Consider the Odds

I’m a closet statistics nerd. Good for me, because innovation is a numbers game. One of my favorite counterintuitive statistical truths is expressed by Bayes’ Theorem, which effectively states that one must consider prior probabilities when calculating the likelihood of an outcome or the relevance of another statistic.

I remember being stunned when I first encountered Bayes in undergrad. As a simple example of the power of prior probabilities, here’s a question for you: A test for a particular disease is 99.9% accurate. If you tested positive, how concerned should you be?

The answer, as with many things in life is, it depends. If the prior probability of contracting the disease is exceedingly rare (say 1/1,000,000), then you probably shouldn’t be concerned at all. “WHAT?!?!” I remember wondering as an undergrad, “How can that be?” Simply put, if those million people are tested, a 99.9% accurate test will be wrong 1 in 1,000 times; which is to say, the test will tell 1,000 people they’re “positive,” when the truth is, if the prior probability of contracting the disease is only one in a million, then most likely, 999 are “false positives.”

Prior probabilities matter a lot. (If you’re a statistician, pls forgive my loose description. I’m working from a dusty memory of a transformative learning moment, not immune to error. But I aim for approximate accuracy!)

I’m convinced very few people consider prior probabilities when it comes to innovation efforts. The truth is, very few things work. Which is to say, the prior probability of success is quite low. So what if you think you’ve got a really, really good idea? Like, you’re 99.9% sure it’s great?

(Pls see the above to map the math)

The truth is, folks’ expectations need to be ratcheted down significantly, and their expectation of at-bats required to get on base needs to be ratcheted up.

Even taking “success” out of the picture, it’s helpful to consider prior probabilities when crafting effective experiments. For example, just the other day in LaunchPad office hours, a prospective founder was crafting an experimental email campaign. He planned to send it to “5 or 6 potential users” to get feedback. Another student spoke up:

“The response rate on my best-performing campaign to date is ~12%.”

Which is to say what? If the prospective founder wanted to get a response (let alone a positive one!), even assuming it was a fantastic first attempt, he needed to email at least ~8 people. If it’s a more modest effort, perhaps he should double or triple that. Considering the odds — prior probabilities of success, response rates, etc — helps founders calibrate the proper scale of their efforts, and scope of their ambition, and expectations of cycles required to succeed.

Related: Create A Portfolio
Related: Judge Experiments Before Ideas
Related: Have Lots of Ideas

Click here to subscribe to Paint & Pipette, the weekly digest of these daily posts.