Reduce Experimental Fidelity
"The real measure of success is the number of experiments that can be crowded into 24 hours."
- Thomas Edison
Edison’s quote reminded me of how Marc Randolf (co-founder of Netflix) described the early days of new venture creation to Tim Ferriss in their fascinating conversation: “I had tons of ideas. The problem is that back then I was a bit of a perfectionist. And so the tests we put together would be these mini works of art: We’d do custom photography, or we would lovingly craft and argue all the copy. We would spell check it and have copy editors review it. And we would stress test the site and check every link. And as you can imagine, that would take two weeks to put together. And then the test would fail. And we’d look at each other and say, ‘We just wasted two weeks.’ And we’d say, ‘Okay, faster.’ And we’d cut some corners and do a test a week. And it would fail. And then we’d cut some more corners and we’d begin to do a test every other day. And then pretty soon a test every day. And soon we were doing four and five tests in the same day.”
Every innovator should know how to craft clever experiments; but one clever experiment is hardly sufficient. One of the things that both Edison and Randolf teach us is that quantity matters. And paradoxically, you can get to quantity in two ways: by dedicating more resources to experimentation at large, or by dedicating less resources to each individual experiment.
As you might imagine, I’m a big fan of the latter. I’m a firm believer that there’s always a way make each experiment cheaper.
A variation on this theme is to make experiments slightly less accurate (statistically speaking), which requires significantly less resources than experiments that are geared towards statistical significance. This thread by the fantastic Wharton professor Ethan Mollick cites a fascinating piece of research conducted by Wharton’s own Eduardo M. Azevedo and NYU’s Jose Montiel Olea, alongside a team of folks from Microsoft. As Mollick recounts, they discovered that, in many scenarios, the better strategy is to “run more small (sample size) & (thus) less accurate tests.”
Related: Make Experiments Cheaper
Related: Judge Experiments Before Ideas
Click here to subscribe to Paint & Pipette, the weekly digest of these daily posts.