Improve Your eROI

“My investors, my employees, my wife told me, ‘That’ll never work.’ And they were right. It didn’t work. And it took us a year and a half of one failed experiment after another, one test after the next.”

Marc Randolph is the co-founder of Netflix, and he gives a master-class in what we call “maximizing experimental ROI,” or eROI. His fascinating conversation with Tim Ferriss is one I’ll be revisiting countless times over the next few weeks, I’m sure. (Hat tip to my friend and thought partner, Signo, who is now 2 for 2 with his recommendations.) When Ferriss asks what the process was like during the 18 months that Netflix was running countless failed experiments, Randolph’s response blew me away. He speaks directly to the importance of designing simple but impactful experiments:

“The problem wasn’t that I didn’t have ideas of things I wanted to try. I had tons of ideas. The problem is that back then I was a bit of a perfectionist. And so the tests we put together would be these mini works of art:

“We’d do custom photography, or we would lovingly craft and argue all the copy. We would spell check it and have copy editors review it. And we would stress test the site and check every link. And as you can imagine, that would take two weeks to put together. And then the test would fail. And we’d look at each other and say, ‘We just wasted two weeks.’ And we’d say, ‘Okay, faster.’ And we’d cut some corners and do a test a week. And it would fail. And then we’d cut some more corners and we’d begin to do a test every other day. And then pretty soon a test every day. And soon we were doing four and five tests in the same day.”

What I especially love about this story is the clear admission that they didn’t start off perfectly, but they got better over time. It’s so critical to have a sense for, “What’s our experimental capacity? And how long is our learning cycle?” What’s counterintuitive to most people, though, is despite the fact that Netflix was increasing its rate of learning, that’s not to say that the experiments looked better. Far from it. Randolf continues:

“And as you can imagine, they’re getting pretty sloppy by then. There’s misspellings, there’s the wrong image, the watermark is still in it. We’re crashing the site and the links don’t work. I mean, it is so sloppy.

“But this learning, it was incredible because we realized it didn’t make a difference. That no matter how crappy this test was, that it was still going to surface things. So in other words, if it was a bad idea, even these beautiful two-week tests didn’t make it a good idea. But if we had stumbled on something interesting, even with these unbelievable mistakes and crashed sites and dead links, people still would raise their hand. They would shine the spotlight. They would call us, they would come to the door. And then you saw as plain as day what was working and what you had to then fix. And it always surprised us.

“And so this insight informed everything from then on. And this insight was that it was not about having good ideas, it was about building this system and this process and this culture for testing lots of bad ideas.”

So the clear implications for teams seeking to deliver innovation are: eliminate the tendency towards perfectionism - expand your capacity (and willingness!) to test bad ideas; and increase your rate of speed, recognizing that this will likely require lower-seeming quality of work. But the quality of learning will not suffer, rather, it will accelerate.

And the primary goal is speed of learning. So improve your eROI, and expand your capacity to experiment.

Click here to subscribe to Paint & Pipette, the weekly digest these daily posts.