Judge Experiments Before Ideas

Innovation team: “We made the offer to 23 different customers, but no one took us up on it.”
Me: “So what does that tell us?”
Team: “The idea is no good, right?”
Me: “Well… it depends…”

What would you say? Does your idea stink if you design an experimental offer and no one takes you up on it?

It’s entirely possible. But before you judge the idea, you should judge the experiment. There’s a classic pitfall that trips innovation teams up, that keeps them from learning about the quality of their ideas: poorly designed experiments.

In this case, the team was testing a new service recovery offer, whereby a customer could make direct contact if the store messed up. They made the offer to 23 people, and no one attempted contact, despite being given an employee’s personal phone number. That’s pretty convincing data, I’ll grant.

The wrinkle, as we dug into the data, is that the team inadvertently botched the experiment: because the sales were conducted under the watchful eyes of a newly-deployed innovation team, they made sure every single order was perfect. Seems great, until you realize that no customers will avail themselves of a service recovery channel if the sale is perfectly executed! So the fact that zero out of 23 sales resulted in a new recovery interaction is entirely expected, and tells us nothing about the merits of the new recovery pathway.

This is a classic blunder, and far from being a terrible mistake, it was actually a real gift to the entire cohort of learners because it helped everyone understand some of the fundamentals of experimental design. But such errors will never get caught as long as teams are only evaluating their ideas. We’ve got to build in the discipline to evaluate experiments, too — and that evaluation should come before the ideas stand judgment.

Click here to subscribe to Paint & Pipette, the weekly digest of these daily posts.