Skip to content

Why Our Numbers Are Always Wrong

2012 October 28

Our data driven society requires hard numbers.  We take those numbers, plug them into models to create solid plans and execute those plans with ruthless efficiency.  If we do it right, things are supposed to go well.

The problem is that our numbers are fantasies, our models are broken and our budgets are farces.  We all know it, try to make allowances for it and the game goes on because, quite frankly, it is the only one we know how to play.

Somewhere along the way we became enamoured by certainty and obsessed with precision in the hopes that, if we only built better tools, we could conquer complexity.  That effort has failed miserably.  There is, however, another way that was abandoned long ago.  It has a rich history of solving the thorniest, most uncertain problems.  It’s time we returned to it.

The Guessing Game

Sometime in the 1740’s, Thomas Bayes, a minister and amateur scholar, had a brilliant idea. He wrote it down, tucked it away and there it stayed until his death.  His friend, Richard Price, found it among his papers, refined and published it in 1763.  The theory was later augmented and formalized by Laplace, the greatest mathematician of the age.

The idea, inverse probability, built on Abraham de Moivre’s work on the the Doctrine of Chances, which provided rules for predicting future events on present information and became a hit with gamblers.  Bayes wanted to reverse the process, to ascertain causes from events.  Could we, through observation, determine why things occur?

His solution, was to start with a guess.  Even if it was far off, there would still a quantified, working hypothesis that could be adjusted as new information came in.  What it lacked in precision, it made up for in common sense and was invaluable in solving problems like hunting German subs in World War II to determining who wrote the Federalist Papers.

The idea was controversial even at the time of its inception.  There’s just something about guessing that seems unscientific and unprofessional.  It was only a matter of time before Bayes and his simple idea fell into disrepute.

Hard Numbers, Soft Facts

The man who would lead the charge against the Bayesian method was the brilliant and famously cantankerous Ronald A Fisher, who railed against the guessing game.  He felt that science is only valuable when it is built on the solid edifice of clear data and established many of the methods that you find in standard statistics textbooks today.

The key to his approach was in the design of controlled experiments.  He vociferously advocated large, randomized samples which could then be analyzed using the Gaussian Bell Curve.  Sample data would be collected and the significance would be derived mathematically through the use of a confidence interval.

Because of his emphasis on samples, his method became known as the frequentist approach.  Guessing would be replaced by cold, hard facts augmented by complex mathematics (Fisher pioneered the use of modern techniques such as the z test, the t statistic and chi-squared).

By the middle of the 20th century, this frequentist approach became standard.  Controlled experiments would lead to scientifically verifiable conclusions that could be trusted and treated as fact.  Or so it was hoped.

The Problem of Uncertainty

To understand the difference between the two approaches, imagine a basketball tryout where a free throw test is used to measure skill.   Under the frequentist approach, a certain number of trials (say 100) would be required to establish confidence.  Under the Bayesian method, confidence increases with each shot and you just take as many as you need.

The problem, of course, is that the world is an uncertain place no matter how many Greek letter equations you affix to a problem.  It is extremely difficult, if not impossible, to create controlled experiments that match real life conditions.  In fact, a recent study in the journal Nature found that a majority of cancer research studies could not be replicated.

If highly trained scientists working in controlled lab settings can get it so wrong, what does that say about the billions of dollars spent on market research every year, which are not nearly as tightly controlled or, to be frank, as transparent?  What, for that matter, are we supposed to make of business planning based on market research?

The problem underlies the basic dilemma of frequentist statistics.  We take studies which, if done properly (often a generous assumption), tell us that we can be 95% confident that a result falls within a certain variance, and then treat that conclusion as if it were forever settled, never to be questioned or returned to.

What is possibly worse is that the frequentist approach leaves us no avenue of taking an assertion that is clouded in uncertainty and making it more concrete over time, causing us to miss opportunities in the name of “sound evidence.”

Micromotives and Macrobehavior

Finding himself bored on a plane one night, Nobel laureate Thomas Schelling would yield a more lethal blow to frequentist statistics; that of complexity.

He started, as you can see in the video below, by thinking about segregation.  He posited what would happen if people want to live in mixed neighborhoods, but preferred not to be outnumbered by people of a different race.  As they moved around to satisfy their seemingly reasonable preferences, they would end up with extreme segregation.

Schelling’s’ key insight was that because our decisions often affect the actions of others and theirs, in turn, affect ours, a small change in preferences can lead to large changes in behavior.  Controlled experiments using independent variables fail to account for this kind of feedback loop.

The problem, of course, goes far beyond the makeup of neighborhoods.  As I explained in a previous post, reliance on frequentist methods contributed to the recent financial crises and, as marketing practitioners increasingly rely on similar methods, we have ample reason for concern.

The world is a chaotic place, we need to account for the fact that anything can happen. Business planning based on the false certainty of “controlled” studies isn’t science, it is pseudoscience.  We need to return to a more iterative, less certain model for strategy.

Bayesian Strategy

In the world of statistics, Bayes is making a comeback.  Noted polling analyst Nate Silver, for one, is a strong advocate and many college textbooks are being revised to put greater emphasis on Bayesian inference.  However, business strategy is still largely mired in misleading conclusions driven by confidence intervals.

We need to move to an approach which becomes less wrong over time rather than the present paradigm of false certainty.  Merely extrapolating past data is not enough, we need to factor in new data as it becomes available.  Bayes rule gives us a mathematically viable way to do that.

Another development is the increased use of agent based models, Markov Chain simulations and other types of sequential analysis that have been built on Schelling’s work. While these won’t bring us certainty, they will enable us to account for interdependence between variables, uncover new insights and manage a dynamic marketplace.

In the end, our numbers will always be wrong.  It is our choice whether we want to blindly believe or continue to test and refine them.

– Greg

18 Responses leave one →
  1. October 28, 2012

    Super post Greg.

    I’ve been thinking that much of our relationship with change, planning and thinking about the future is the same as when someone learns about something – at first they are aware of the unknowns and dela with this – learning more as they go along. Over time they amass knowledge and become more confident and certain. The danger is that people can stop learning once they think they know it all. However, if people continue learning they may again appreciate what they don’t know and again consider uncertainty and complexity.

    I welcome the return of complexity and uncertainty into our thinking.

    Let keep learning!

  2. October 28, 2012

    Apparently, Bayesian thinking got popular at HBS in the 50’s, but then got cast aside once the statisticians got involved and implemented a more “scientific approach”


    – Greg

  3. October 28, 2012

    Hi Greg,

    Wonderful post. What you wrote about the frenquist approach reminded me of another great book – Black Swan. It was about the non-linearity of life. Output is not always proportional to input.

    And I like the way you showed a way out : agent based modelling, Markov chain simulations. I guess you are completely spot on about “bottom up” modelling.

    I guess the nearest thing to real life is “bottom up” approach where aggregates emerge out of interaction of the agents with decision rules.

    Marketing research can be useful if only it uncovers the decision rules and maps the market in terms of decision rules. (just like in Thomas Schelling thought experiment)

    The big data approach is always after the fact. What makes me uneasy about the big data is that you are trying to make sense of an emergent phenomena by looking at the “end product” ignoring the decision rules behind it. just like the segregation example, if the decision rules are not understood, the emergent result can not be reverse engineered.

  4. October 28, 2012

    All very true. Tesekkur ederim:-)

    I do think Nassim Taleb has a lot to say, although I don’t recommend the book because he tends to say it like such an asshole. On the other hand, Benoit Mandelbrot’s posthumous memoir is coming out at the end of the month and I’m buying that for sure!

    – Greg

  5. October 28, 2012

    Excellent article Greg. Well done again.


  6. October 28, 2012

    Thanks Brian. Have a great week!

    – Greg

  7. October 30, 2012

    Very interesting post! It reminded me to Nassim Taleb’s books, too, and his opinion about people who wants do describe everything with normal distributions. It is funny to find a good piece of science on a blog I follow for marketing. 🙂

  8. October 30, 2012

    Thanks Greg. Glad you liked it:-)

    – Greg

  9. November 3, 2012

    Greg, I wandered on to your site about 45 minutes ago and have been devouring your articles. Great stuff, and consistently insightful. I’ve been advocating for the application of Agile to marketing for a year or more, and we talk about an iterative, measured approach to marketing, often invoking the scientific method. But I hadn’t thought of the important differences between the Bayesian approach and the Frequentist approach. For most, if not all marketing, the inability to control all the variables makes the Bayesian approach more applicable, as you rightly point out.

    Keep up the great work!


  10. November 3, 2012

    Very kind words and I appreciate them. Thanks Jim!

    – Greg

  11. November 7, 2012

    Brilliant post! It reminds me of the joke I heard long back about numbers. Numbers are like bikini. What they hide are more interesting than what they reveal!

  12. November 7, 2012

    That’s an extremely thoughtful comment on Big data! Thanks!

  13. November 7, 2012

    Nice one:-) Thanks.

    – Greg

  14. November 8, 2012

    Great post Greg.
    I’m pleased to see the resurgence of Bayesian thinking, but to place the blame at the foot of the frequentist statisticians is naieve. Familiarity with tit for tat or prisoner’s dilemma games, or the group think work by Irving Janis all warned against complacency as a precursor to disaster. The problem isn’t the models, but our contentment with what works, our reluctance to take randomness seriously and belief that by minimizing error it disappears.

    Do hope you will continue to provide clarity on the options and the ability of the models to test assumptions and further our understanding. The more people who can understand there are ways to unravel complexity, the more likley they may find ways to fight their complacency. Win win in my book! Keep writing ad I’ll keep reading.

  15. November 9, 2012

    Glad you liked it. Thanks for sharing your thoughts.

    – Greg

  16. Hoon Lee permalink
    November 26, 2013

    Thank you for sharing your insight. Enjoyed this post.

  17. November 26, 2013

    Glad you liked it!

    – Greg

  18. wly185 permalink
    January 13, 2017

    great article

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS