Skip to content

The Problem with Pundits

2012 November 11

In the last few days leading up to the recent presidential election, long time political observers such as Peggy Noonan, Steve Forbes and Dick Morris were predicting that Mitt Romney would win in a landslide.

They were wrong, of course.  Horribly, drastically, overwhelmingly wrong about everything from the shape of the electorate, voter turnout and, of course, the direction of the result in the face of massive, publicly available evidence that showed Barack Obama as the clear favorite.

InTrade markets gave 7 to 3 odds that the President would prevail.  Nate Silver’s FiveThirtyEight blog (which analyzes polling data) calculated a 91% chance of an Obama win.  How could they get it so wrong?  It turns out that the answer lies not so much in partisan politics, but in the inherent flaws of our basic human decision making machinery.

Tetlock’s Revenge

It’s tempting to see the dismal failure of pundits in factional terms.  Given their sometimes hysterical backlash against any information that contradicted their convictions, right wing talking heads might seem completely irrational.  In retrospect, their misplaced confidence in their cause seems positively crazy.

However, according to Philip Tetlock, who embarked on a 20 year study of political pundits, expert predictions of all ideological persuasions are no better than flipping a coin. In other words, for all the time they spend studying an issue (and in fact, to some extent because they spend so much time on it), they might as well just throw darts.

Yet, there is something more going on here.  These pundits in particular not only got it wrong, but chose to ignore the most salient evidence – a puzzling, as well as troubling development.  Why would they reject the most obvious answer in favor of obscure arguments about poll skews and media bias?

The Tale of Two Systems

We can find part of the solution to the pundit puzzle in Nobel laureate Daniel Kahneman’s excellent book Thinking, Fast and Slow.  He describes our decision making process as a competition between two competing systems:

System 1: This is our more instinctual, automatic system.  It relies on rules of thumb, called heuristics, which enable us to act quickly and decisively.  It is fast and frugal, using little time and energy to arrive at a conclusion.

Incidentally, system 1 is very active in politics.  Studies by political scientists have found that voters often make judgments of candidates based on “rapid, unreflective inferences” about their appearance.

System 2:  This system reflects our more rational side.  We use it when we stop, think, make difficult calculations and weigh facts, which takes considerably more time and effort than simply going on instinct.  We only engage our second system when we realize that the first one is falling short.

Kahneman argues that we have a tendency to substitute tough system 2 questions (such as how to analyze an enormous amount of polling data) for an easier system 1 question (i.e. how one feels about the possibility of an outcome).  That, as we shall see, is where the trouble starts.

Substituting One Question For Another

We ‘re more comfortable with judgements we make on the basis of system 1 because it feels more natural.  We don’t live our lives based on polls or artificial markets, but mostly by gut feelings.  Does this seem like a good neighborhood?  Can I trust this salesman? Does that item on the menu look tasty?

These are the questions we ask ourselves everyday and no amount of controlled tests or sophisticated mathematical models will do us any good.  We go with what feels right, picking up subtle cues that will sway us one way or another because of how they jibe with our internalized experience.

Take a look at what longtime Wall Street Journal political columnist Peggy Noonan had to say about why she felt so strongly about a Romney win:

All the vibrations are right. A person who is helping him who is not a longtime Romneyite told me, yesterday: “I joined because I was anti Obama—I’m a patriot, I’ll join up But now I am pro-Romney.” Why? “I’ve spent time with him and I care about him and admire him. He’s a genuinely good man.” Looking at the crowds on TV, hearing them chant “Three more days” and “Two more days”—it feels like a lot of Republicans have gone from anti-Obama to pro-Romney.

Yep.  There’s her system 1 kicking in.  She can feel it in her bones, therefore it must be right, because what you feel is not only very real, it is reality itself.  Once your system 1 becomes convinced, it becomes very easy to reject the relatively weak impulses of system 2, which she does just a few paragraphs later.

Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us? Maybe that’s the real distortion of the polls this year: They left us discounting the world around us.

Of course, professionals can not judge based on feelings alone, they need facts to back them up, but that’s a relatively minor problem, for we can always find data to support a position.  After all, no president had been re-elected with such a high unemployment rate and 2008 was a wave election, reflecting a misshapen electorate.  Right?

When these comforting morsels agree with our system 1, they become very convincing. Behavioral economists call this confirmation bias and we all fall prey to it.

The Confidence Game

When the “vibrations are right” as they were for Peggy Noonan, doubt begins to fade and a dangerous confidence sets in.  Conservative columnist Ross Douthat describes what happens next:

You could see this belief at work in the confidence with which many conservatives insisted that the Obama presidency was not only embattled but self-evidently disastrous, in the way so many voices on the right sought to raise the ideological stakes at every opportunity, in the widespread conviction that the starker conservatives made the choice between left and right, the more votes they would win.

You could also see this conviction shaping the punditry and predictions that issued from conservatives in the days leading up this election. It was remarkable how many analysts not normally known for their boosterism (I’m thinking of Michael Barone and George Will in particular) were willing to predict that Romney would not only win but win sweepingly, capturing states that haven’t gone Republican since Reagan. But even less starry-eyed conservatives — like, well, myself — were willing to embrace models of the electorate that overstated the Republican base of support and downplayed the Democrats’ mounting demographic advantage.

Pundits tend to be homophilic.  They form schools of thought and think tanks, which creates an echo chamber for friendly conclusions.  There is nothing more satisfying than spending time with people who passionately agree with us.  Our brains release oxytocin, giving us the ultimate feeling of well being and reinforcing our system 1 even further.

We no longer merely think that we’re right, we’re absolutely sure of it.  Pundits who convey this confidence are especially convincing, which is what makes them compelling to audiences and increases the public’s demand for their services.  Surety sells.

The Steve Jobs Effect

We live in a data driven, technological age and this offends our system 1.  Data is never completely clear and we require complicated mathematics to digest it.  It is no accident that the polling analyst Nate Silver called his new book The Signal and the Noise.

We’ve found a hero in Steve Jobs, who famously never hired consultants or used research surveys.  He was more of an artist than an engineer and conjured big ideas.  He operated by instinct and that seems admirable.  Highly regarded columnists Michael Gerson and David Brooks both remarked that the reliance on polling data diminished big ideas.

This is the most dangerous trick that system 1 plays on us, for in many highly technical fields, system 1 is an enormous asset, allowing us to act quickly and confidently.  We don’t, after all, want a surgeon consulting a database when blood begins to spurt or a fireman running models in excel while flames engulf our homes.

However, as I explained in a previous post, these problems are not equivalent.  Just like Steve Jobs spent many hard years studying design, surgeons and fireman personally confront thousands of life or death situations.  That kind of immediate, direct feedback is wholly absent in the political realm.  Elections are rare and we experience them by proxy.

The Visceral Abstract

The issue goes far beyond political prognostication.  As I’ve argued before, we increasingly live in a world of the visceral abstract, where positively wacky ideas that are completely foreign to our experience have taken a central role in our everyday lives.  However, we don’t directly confront them, which makes it hard to take them into account.

Einstein’s theory of relativity says that time and space are illusions, which only have relevance to our particular perspective.  Quantum mechanics tells us that while we are most likely here, we could be over there; or on the dark side of the moon for that matter.

Preposterous?  Maybe, but our GPS navigation systems and our iPhones are living proof that these things are true.

It is only through our relatively boring and tedious system 2 that we can understand this abstract world.  The “vibrations” that stirred Peggy Noonan’s soul gave her no insight into the feelings of millions of freethinking individuals across hundreds of miles, just as the comfortable look and feel of our iPhone tells us nothing about subatomic physics.

And that’s the problem with pundits.  They are paid to dazzle us with their surety and it is that same confident conviction that makes them so very likely to get it wrong.

– Greg

7 Responses leave one →
  1. November 11, 2012

    Oh, you know I love this!

  2. November 11, 2012

    Excellent Greg. Couldn’t be said better.

  3. November 11, 2012


    You say it beautifully, as always. We are all so human.

    People, being people, tend to align themselves with those people they feel think like them, and thus self validate their own ideas, opinions and cause.

    Somewhere in the United States of the 1950s, a seed of the notion that culture could refer to smaller segments of the larger society, could refer to individual business, social or political groups within the society as a whole, was planted.

    The notion was tended, cultivated and fertilized, and it grew. Use of the word culture as the label for collective behavior and attitudes of a group became a holy mantra for many going forward.

    An amazing amount of language and lucre continues to be spent by truth seekers explaining to each other what culture is and what it all means.

    They say this thing, this culture, exhibits its existence, makes itself known through language and symbols, art and ideas, and often illuminates, awkwardly at times, in beliefs, ceremony, politics, institutions, rituals and taboos.

    Within this fragile integrated pattern of behaviour and belief, with a pinch of knowledge, our human pride compels us to learn and think and transmit this substance on to each unsuspecting succeeding generation ad infinitum. It is what we do so well.

    Obviously, I guess, the awareness and manipulation of this substance culture plays a crucial role in our human evolution, as harsh, and beautiful and glacier paced, as that evolution seems at times.

    Allowing us to adapt our present environment to our own purposes, rather than being reliant on natural selection, each particular culture allows each human society to achieve adaptive success, or we would not still be here.

    Within a subtle mist, before our very eyes, political, religious and socioeconomic change, and developed links with other human cultures, will influence new attitudes, beliefs, values, and away we go.

    It is all inside our heads. For better or for worse, we are more interested in our reflection than in reality.


  4. November 11, 2012

    Thx Burak!

    – Greg

  5. November 11, 2012

    Interesting perspective. Thanks Dan.

    – Greg

  6. Ebenezer permalink
    November 13, 2012

    I am still married to the same woman after twenty years because I wasn’t “romantic”. All my romantic friends have since divorced, some acrimoniously. I have learnt not to trust my feelings alone since I was a kid. It is certainly not 100% successful but my choices were always based on the measurable. As a conservative who dared to follow the facts I predicted Romney’s loss a while ago and all my GOP twitter followers grew silent, no one would engage me anymore because I refused go get caught up in the post debate euphoria.

    Your analysis helps me to explain to some of my republican friends why I did not join the “party”.

  7. November 13, 2012

    I wouldn’t go overboard. In many situations, you’re better off acting on gut feelings than thinking things through (i.e. tasks that you have repeated often and provide immediate feedback, like a surgeon on an operating table or a salesperson working a prospect).

    The problem comes when we try to apply gut feelings to situations where they are of no use. If you go to a great party and feel giddy, that tells you nothing about how the other 7 billion people in the world are feeling and it would be silly to think that it would. Nevertheless, Peggy Noonan not only used her feelings as a proxy for the entire country, but felt those feelings overode an quantitative analysis.

    Like I said, a bit silly…

    Thanks for you comment.

    – Greg

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS