Are You Using Data To Analyze The Past Or To Create A New Future?

In Nate Silver’s Five Thirty Eight manifesto, he argues that the plural of anecdote is data.  It is through compiling and analyzing observations that we transform ordinary experiences into scientific conclusions.

Yet the concept of data is still relatively new.  According to Google’s Ngram, the use of the word only began to be heat up in the 1960’s.  So, although data is a term that gets thrown around a lot, it’s not something we’re really accustomed to using in a meaningful way.

That’s probably why many feel that data is in conflict with personal experience.  Yet in reality, data is most often used to codify experience.  Historical observations are aggregated and then analyzed to identify salient trends.  And that’s the real problem with how we use data.  History can be a trap.  It’s not the past that we need to worry about, but the future.

The Lure Of Historical Trends

My first real job out of college was on a trading desk, where I encountered the concept of “technical” market analysis.  Often known in the profession as “chartists,” practitioners scour historical data looking for patterns that they expect will give them clues to how things will play out in the future.

I was always amazed at the nearly religious fervor that many of the chartists had about their work, especially the Elliott wave theorists.  It was as if they were not merely performing analysis, but communing with nature and unlocking its secrets.  Chart analysis was a strange blend of the technical and the traditional.

I suspect that is part of what made the technique so popular.  Tradition has a romantic quality to it, as if we’re not merely forging our own path, but honoring the wisdom of those who came before us.  There has always been conflict between the medicine men, who put their faith in ageless insights and the visionaries who wish to break old molds.

The Poverty Of Historicism

In the 1930’s, Karl Popper began to speak out against excessive reverence of the past in a paper called The Poverty of Historicism, which was later published as a book.  He argued that, in fact, there was no reason to believe that future will look like what came before it.

 
Take a look at the chart above and you can see what he means.  For most of history, little changed, but when the Industrial Revolution came along the world hit an inflection point. It’s hard to argue that many trends in force before the shift remained valid afterwards. Everything had to be called into question.

One of Popper’s students, George Soros, became one of the worlds richest men by identifying similar, albeit smaller shifts.  People like Henry Ford, Bill Gates and Steve Jobs defined entire new industries by breaking with established norms.  It is the visionaries we admire—not the trend followers—because they’re the ones who make an impact.

Why Big Data Matters

One of Popper’s key concepts was that any assertion, in order to be scientifically viable, has to be falsifiable.  Unless it can be tested and proven wrong it was basically nonsense.  We admire great scientists like Darwin and Einstein not simply but because they had an idea, but because they stated it in a way that others could verify or disprove it.

For most of the 20th century, the testing process was deeply influenced by the methods of Ronald Fisher who set out rules for the design of experiments, confidence intervals and statistical significance, among other things.  Underlying his ideas was an emphasis on controls.  Put good data in and you would get good answers out.

Yet for all of their charms, Fisher’s techniques were still essentially historical.  You collected data over a period of time and determined mathematically whether they were meaningful or not.  You were still counting on the future looking like the past and, if it didn’t, you could find yourself wildly off the mark.

Today’s big data techniques focus less on the past and more on the present.  No longer are we trying to divine the rhythms of history to uncover eternal truths, but using a vast array of sensors and servers to gobble up data and analyze it in real time.  Truth is no longer measured in eons or even in months, but in milliseconds.  That’s why big data matters.

Managing For Uncertainty

In the old days, when executives began their careers they could be reasonably sure that the the rules that guided their industry would stay fairly constant, their skills would stay relevant and their business models would remain intact.  That’s no longer true.

It’s now becoming clear that business models don’t last and skills need to adapt.  We need to start taking a more Bayesian approach to strategy in which we no longer expect to uncover timeless truths, but instead value adaptation over steadfastness, learning over experience and preparation over planning.

History, even if properly construed, can blind us to important possibilities.  After all, it is not the rhythms of the past that define us, but our dreams for the future.

– Greg