“You can’t control the market. Worse, you can’t control whether you believe you can control the market. Remember, it’s called the ‘Invisible Hand’ for a reason.”
Tootly had weathered the Super Bowl storm; in fact, the controversy had actually improved brand awareness, according to some numbers the Social Media Ninjas had extracted from their collective asses. Either way, Jacob and Ben felt like they had overcome the first great challenge in their fledgling business. Unfortunately, the Super Bowl ad was nothing more than a red herring; the real risk wasn’t public relations or marketing, but something more fundamental. Around mid-March of 2009, Tootly’s downloads finally slowed down. By the end of March, they had practically stopped.
Ben was stunned. “We couldn’t believe it. We had nearly 800,000 downloads across all of our applications, but without any clear reason, people stopped downloading. We demanded that the Data Junkies and Social Media Ninjas use all those great metrics they had developed to explain what had happened.” After many all nighters, they came back with an elegantly simple presentation with a number of bulleted take-aways:
Jacob sat through the meeting intently, watching each overly wordy, animated slide come in and out of focus. As soon as they finished, Jacob looked at his data and marketing team, and pressed them with a simple inquiry. “That was a wonderful summary, but what the hell does it have to do with our download numbers? Do you have any idea why they’re slowing down?”
The team couldn’t help but look ashamed, and Jacob himself felt ashamed when he embarrassed them all. “Honestly, there’s no way they could have known why our numbers were falling off a cliff. You don’t know what you don’t know, you know? We had a wealth of data on current app users, but there was nothing we could know about people not engaging with our apps. We may as well have been asking them to read tea leaves.”
Ben asked the Social Media Ninjas to do outside market research, which was, in fact, about as accurate as reading tea leaves. They spent tens of thousands of dollars on outside marketing reports, focus groups, academic studies, and outsourced circus psychics (who called themselves ‘mobile app consultants,’ but may as well have claimed psychic powers). After a month of stagnant growth, all of these reports were compiled and reduced to a single page memo, which was discussed at length by the Committee of Chiefs. The last time Ben and Jacob had read such a disheartening conclusion was in a business school case study. In keeping that fine tradition for future entrepreneurs, here’s the depressing conclusion of the Social Media Ninjas’ report:
CONCLUSION: After numerous consumer sentiment studies, deep marketplace analysis, engaged focus groups, and in-depth competitor canvassing, we have determined that the novelty has worn off. We have saturated the market with farts, and new customers cannot possibly bear the smell. The successful, high-growth apps today are either much more sophisticated, tied into broad web services, or games.
While Jacob doubted their methodology (he was becoming increasingly distrustful of data analysis), he couldn’t discount the conclusion. “Unlike other data I had been shown, this lined up with my instinct. Ben agreed, but wanted confirmation through more in-house data; I would have preferred if we just accepted it and moved on.” Despite Jacob’s protestations, Ben demanded that the Data Junkies build “A/B testing capability” into the next updates of Tootly’s apps.
Famous in web services, A/B testing allows you to take two (or more) separate scenarios (whether it’s a new interface, user flow, or piece of functionality) and compare metrics on which scenario leads to better outcomes. Most famously used by Google to modify everything on their website, including colors; apparently a certain shade of blue leads to a ‘statistically significant’ engagement boost. A/B testing routinely gets taken to absurd levels (see Google’s obsession with colors), and invariably leads to “designing by numbers.” With an expansive A/B testing infrastructure, a company will outsource creative thinking by narrowly defining products into testable chunks, leading to nasty blindsides. For example, if A/B testing had existed at the turn of the century, horse-and-buggy manufacturers would have designed the most statistically perfect buggy, only to watch, in horror, as Henry Ford’s mass-produced automobile destroyed their business. Or if dynamic A/B testing existed in video games, you would have every Facebook game ever made. Sadly, few technology companies will ever understand this danger, unless there was some A/B test that could prove its validity with a high degree of statistical significance.
As it turns out, the Data Junkies had a great deal of expertise building A/B testing infrastructure, and rapidly constructed a beautiful web interface to modify aspects of the Tootly apps on the fly (different sounds, interface elements, descriptions, etc.). Of course, they didn’t come up with any conclusions that were entirely useful; they did however identify a particular color of blue that made the ‘FART’ button much more appealing. Blue was, apparently, quite the engagement work-horse. As is usually the case with A/B testing, many of those reading the conclusions had different interpretations of “statistically significant,” which led to more confusion and conflict. However, Ben eventually wound up satisfied with the evidence as it was presented (and interpreted by him), which demonstrated (in his mind anyway) that there was little they could do to improve future downloads or recommendations by current users. They did, however, immediately change all button colors to a very particular shade of blue.
Data in hand as April drew to a close, Ben and Jacob decided they needed to sound out their board of advisers (Tim and Jim) on how they should proceed. Their business seemed stuck and they needed advice, especially if they were going to survive long enough to receive a second round of funding.
The venture capitalists were not happy with the news. First and foremost, they demanded that Jacob and Ben reign in spending. Ben remembered the conversation vividly. “They understood the gamble behind the Super Bowl ad -- hell, they encouraged it -- but apparently they had been talking to the Chief Finance Dude without our knowledge and weren’t happy with our extravagant spending. We claimed it was part of the talent retention game for startups, but perhaps there was some personal motivation to having ‘Massage and In-N-Out Fridays.’ I like to think everyone benefited from that. Though I’ll admit the gourmet burger chef we hired may have been excessive.”
Jacob was more frustrated with their conversation. “Okay, so I understand that we weren’t being fiscally responsible, but we engaged in those expenses in the middle of incredible app growth! More importantly, I can't say I was pleased that some of our employees were informants -- then again, I should have been a bit more realistic and expected that the Chief People Person was selecting employees that were friendlier to the venture capitalists; she was from their network after all.” That conversation firmly convinced Jacob that the venture capitalists were not operating in their interest, but Ben was more empathetic. “They made an investment. While I know Jacob was upset about the breach of trust, if I were Tim and Jim, I would have been upset by the change of fortune and the lack of transparency in finances. I could see where they were coming from.”
Either way, Jacob and Ben yielded, and made a commitment to cut down their expenses. The venture capitalists also encouraged Tootly to consider an alternate, popularized path to product development and startup growth, which they called the “lean thinking” methodology. Tootly was at a crossroads, and the venture capitalists made it clear they’d use their considerable leverage to steer them in the “right direction.” Jacob didn’t like being manipulated to choose a particular path, but Ben made the choice more stark for him.
“We were going off a cliff, and we had to make an epic business decision.”
Like any decision, only more so.
Ben continued. “The outcome was clear in my mind. Either we adjust our business through this new process, or the venture capitalists ensure we wouldn’t receive funding in the future. What choice did we have, really?” Besides, Ben had heard about many of the scrappy startup successes from this new style of thinking, and he believed in its promise. Jacob didn’t, but he knew it didn’t matter. “Ben was right. We didn’t have much of a choice.” So they embraced being lean, no matter the outcome. Unfortunately, being lean also meant firing the burger chef. “Burgers...and chefs, I guess...were anything but lean,” Jacob recalled.
Now burger and masseuse-less, and with no large advertising spends in sight, Tootly concentrated on evolving their business...leanly. Inevitably, that meant they had to learn to “fail fast and cheap.” Jacob laughed when he thought of the phrase. “I thought we were already failing fast -- I mean, negative growth in downloads should be considered a failure, right? But no. Apparently, we were failing at marketing how we were failing.” His laughter continued; this time tinged with regret. “Looking back, I wish I had paid more attention in communicating with our venture capitalists...but on the other hand, their vision of ‘fast failure’ didn’t exactly work out for us either.”
Copyleft MMXIV Josh Cincinnati