Experiment Broadly

Shallow ideation and narrow experimentation are to blame for far more failures in industry than we can possibly imagine. Because of our limited definitions of “relevance,” we end up commissioning far fewer experiments than would be beneficial, and within a much narrower range than is likely to reveal a meaningful difference-makjer. Consider this fantastic example from Rory Sutherland, Vice Chairman of Ogilvy, from his delightful book, Alchemy.

“Once a year, volunteers for our client charity drop printed envelopes through millions of doors, and return a few weeks later to collect people’s donations. This year the envelopes contained a hurricane relief appeal, but some of these envelopes were randomly different from the rest: 100,000 of them announced that the envelopes had been delivered by volunteers; 100,000 encouraged people to complete a form which meant their donation will be boosted by a 25% tax rebate; 100,000 were in better-quality envelopes; and 100,000 were in portrait format (so the flap of the envelope was along the short side rather than the long one).

If you were an economist you would look at the results of the experiment and immediately conclude the people are completely insane. Logically, the only one of these changes that should affect whether people give is the one that reminds you that, for every $1 you donate, the government will give a further $.25. The other three tests are seemingly irrelevant; the paper quality, the orientation of the envelope and the fact that it was hand-delivered by a volunteer have nothing to the rational (economic) reasons to donate.

However, the results tell a different story. The ‘rational’ envelope in fact reduces donations by over 30 per cent compared to the plain control, while the other three tests increase donations by over 10 per cent. The higher quality paper also attracts a significantly higher number of more significant donations of $100 or more.”

Setting aside the cognitive bias Sutherland is writing to illuminate, what’s fascinating to me is the breadth of experiments that the Ogilvy team commissioned in attempting to increase charitable donations! It spoke to me of the importance of experimenting broadly: it’s only when we do things beyond the normal consideration set that we end up delivering unexpected results.

So next time you’re running an A/B test, definitely test with scientific precision — and definitely try more than one, considering the odds of any particular approach succeeding — but while you do, push yourself to consider some seemingly irrational options as well.

And if you’re auditing your portfolio of projects, put a critical eye on narrow experimentation.

Related: Have Lots of Ideas
Related: Consider The Odds
Related: Get Scientific
Related: Try More Than One
Related: Perform An Innovation Audit

Join over 10,147 creators & leaders who read Paint & Pipette each week