Ideas Can Only Be Validated Forward, Never Backward

In 2007, our media company in Ukraine completed its IPO and would soon be valued at $100 million. For a rough and tumble organization that, just a few years before, was a relatively small business, it was exhilarating. We had big plans and were eager to execute them. It was a “damn the torpedoes, full speed ahead” moment.

We also had an innovative strategy that we thought was a clear winner—a bet on Ukrainian language media. Although the Russian language was dominant at the time, we thoroughly researched the idea and found that a large part of the market said they preferred Ukrainian. To grab the opportunity, we launched three major brands in a year.

It was a disaster. Although the first launch was cause for concern, we were moving so fast the other two were too far along to stop. Then the 2008 global financial crisis hit and we were soon struggling to stave off bankruptcy. It was a brutal lesson. You can research an idea, but you never really know what you have until you’ve actually tested it in the marketplace.

The Rule Following Paradox

The Philosopher Ludwig Wittgenstein famously wrote, “no course of action could be determined by a rule, because every course of action can be made out to accord with the rule.” He meant that every rule is subject to some interpretation and, given varying contexts, interpretations are bound to vary.

That’s essentially what happened to us. We did our research and combed through all the evidence. Television and advertising was, by law, in Ukrainian and not Russian. Consumer surveys consistently showed that a significant portion of the Ukrainian public preferred Ukrainian language media. There were plenty of signs that we were on to something.

Given that analysis, our course seemed clear. We should not only launch Ukrainian language products, we should proceed at a rapid pace so that we could move out ahead of the pack. Surely, once competitors saw how big the opportunity was, they would pounce and our opportunity would be squandered.

Except that there was no opportunity. We weren’t acting on facts, but our interpretation of them and that interpretation was horribly, drastically wrong. To make matters worse, all this was happening as the Ukrainian media market was hitting its peak and the world was about to head off a cliff into the worst financial crisis since the Great Depression.

Survivorship Bias

Business school professors and consultants gain fame—not to mention large fees—when they are able to define a novel concept or success factor. If you are able to isolate one thing that organizations should do differently, you have a powerful product to sell. A single powerful insight can make an entire career, which is probably why so many cut corners.

For example, in their study of 108 companies, distinguished INSEAD professors W. Chan Kim and Renée Mauborgne found that “blue ocean” products, those in new categories without competition, far outperform those in the more competitive “red ocean” markets. Their book, Blue Ocean Strategy, was an immediate hit, selling over 3.5 million copies.

Bain consultants Chris Zook and James Allen’ book, Profit from the Core, boasted even more extensive research encompassing 200 case studies, a database of 1,854 companies, 100 interviews of senior executives and an “extensive review” of existing literature. They found that firms that focused on their ”core” far outperformed those who strayed.

It doesn’t take too much thinking to start seeing problems. How can you both “focus on your core” and seek out “blue oceans”? It betrays logic that both strategies could outperform one another. Also, how do you define “core?” Core markets? Core capabilities? Core customers? While it’s true that “blue ocean” markets lack competitors, they don’t have any customers either. Who do you sell to?

Yet there is an even bigger, more insidious problem called survivorship bias. Notice how “research” doesn’t include firms that went out of business because there were no customers in those “blue oceans” or because they failed to diversify outside of their “core.” The data only pertains to those that survived.

The Problem With Case Studies

The gold standard for research is randomized, double blind trials in which some of the subjects receive some sort of intervention, a control group gets a placebo and no one, not even those conducting the study, know which subjects are in which group. This design minimizes the chance of bias affecting results.

Yet this type of design is impractical for studying real businesses that are competing in the marketplace. So researchers largely depend on case studies in which participants are interviewed after the fact. These can be helpful in that they offer first-person perspectives of events and their context, but have obvious problems.

First, much like in Wittgenstein’s rule-following paradox, a lot is left up to interpretation. There are rarely more than a half-dozen people interviewed and they tend to be insiders. We almost never hear from competitors, customers or lower level employees. Then the researchers themselves bring their own biases to what they see and hear.

There are also issues with survivorship bias. Clearly, key players will be much more forthcoming about successes than failures. So we tend to hear about strategies that worked when, for all we know, those same strategies may have failed in other organizations and other contexts. There’s simply no real way for us to know.

Incidentally, researchers did a series of case studies on our company and I was struck by how much they depended on who was interviewed. While there wasn’t anything factually wrong, a different sample of perspectives would have led to very different interpretations.

Adopting A Bayesian Strategy

Traditionally, strategy has been seen as a game of chess. Wise leaders survey the board of play, plan their moves carefully and execute flawlessly. That’s always been a fantasy, but it was close enough to reality to be helpful. Organizations could build up sustainable competitive advantage by painstakingly building up bargaining power within the value chain.

Yet as Mike Tyson pointed out, “everybody has a plan until they get hit.” We can research and plan all we want, but the real world is a messy place. The facts, as we see them, are really just interpretations of the data we have available to us. Invariably, there are other data we’re not seeing and, even that which we have in front of us, can be interpreted in multiple ways.

That’s why we need to take a more Bayesian approach to strategy, in which we don’t pretend that we have the “right strategy,” but endeavor to make it less wrong over time. As Rita Gunther McGrath has put it, it’s no longer as important to “learn to plan” as it is to “plan to learn.” We need to be more iterative, see what works and change course as needed.

Today, instead of thinking about strategy as a game of chess, we’d do better to envision an online role-playing game, in which you bring certain capabilities and assets and connect with others to go on quests and discover new things along the way. Unlike chess, where everyone knows that their objective is to capture the opponent’s king, we need to expect the rules to change over time and adapt accordingly.

Greg Satell is Co-Founder of ChangeOS, a transformation & change advisory, an international keynote speaker, and bestselling author of Cascades: How to Create a Movement that Drives Transformational Change. His previous effort, Mapping Innovation, was selected as one of the best business books of 2017. You can learn more about Greg on his website, GregSatell.com and follow him on Twitter @DigitalTonto

Like this article? Sign up to receive weekly insights from Greg!

Photo by 愚木混株 cdd20 on Unsplash