The Fact of My Ignorance

I doubt I will ever write many articles that fit better into the theme and mission of this blog than this one.  And despite the fact that the piece it’s based on is over 3 years old, it couldn’t be more topical.  There was an article in the New Yorker in December of 2005 entitled “Everybody’s An Expert” that discusses a book by UC Berkeley Psychology Professor Philip Tetlock called “Expert Political Judgment: How Good Is It? How Can We Know?”.  The article is a fascinating read and I’d encourage you to go through it if you’ve got a few minutes.  But it is long so I’ve tried to draw out the more important and politically relevant points and I’ve added them below, along with a little of my own commentary.

The article (and the book) revolve around the information provided by the following study, performed by Tetlock over a 20-year period:

“He picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends,” and he started asking them to assess the probability that various things would or would not come to pass, both in the areas of the world in which they specialized and in areas about which they were not expert. Would there be a nonviolent end to apartheid in South Africa? Would Gorbachev be ousted in a coup? Would the United States go to war in the Persian Gulf? Would Canada disintegrate? (Many experts believed that it would, on the ground that Quebec would succeed in seceding.) And so on. By the end of the study, in 2003, the experts had made 82,361 forecasts…

“Tetlock got a statistical handle on his task by putting most of the forecasting questions into a “three possible futures” form. The respondents were asked to rate the probability of three alternative outcomes: the persistence of the status quo, more of something (political freedom, economic growth), or less of something (repression, recession). And he measured his experts on two dimensions: how good they were at guessing probabilities (did all the things they said had an xper cent chance of happening happen x per cent of the time?), and how accurate they were at predicting specific outcomes. The results were unimpressive. On the first scale, the experts performed worse than they would have if they had simply assigned an equal probability to all three outcomes—if they had given each possible future a thirty-three-per-cent chance of occurring. Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices.” (emphasis added)

So with so many political pundits throwing out predictions on a daily basis, somebody finally decided to check their accuracy.  And surprise, surprise, they’re extremely bad overall.  But they also collected a large amount of data about the personalities and perspectives of the their test subjects and they found that some pundits were much more accurate than others, and I think the associations they discovered are incredibly interesting:

“Tetlock uses Isaiah Berlin’s metaphor from Archilochus, from his essay on Tolstoy, “The Hedgehog and the Fox,” to illustrate the difference. He says:

‘Low scorers look like hedgehogs: thinkers who “know one big thing,” aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who “do not get it,” and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible “ad hocery” that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.’

“Tetlock did not find, in his sample, any significant correlation between how experts think and what their politics are. His hedgehogs were liberal as well as conservative, and the same with his foxes. (Hedgehogs were, of course, more likely to be extreme politically, whether rightist or leftist.) He also did not find that his foxes scored higher because they were more cautious—that their appreciation of complexity made them less likely to offer firm predictions. Unlike hedgehogs, who actually performed worse in areas in which they specialized, foxes enjoyed a modest benefit from expertise. Hedgehogs routinely over-predicted: twenty per cent of the outcomes that hedgehogs claimed were impossible or nearly impossible came to pass, versus ten per cent for the foxes. More than thirty per cent of the outcomes that hedgehogs thought were sure or near-sure did not, against twenty per cent for foxes.”   emphasis added

So the lowest scorers are those who are confident that they understand the world and can predict its future, they are the type who express anger at those who “do not get it”, they tend to be on one of the farther ends of the political spectrum, they have particular themes that they focus on and which they tend to see everywhere, they form “grand schemes” and conspiracies, and they tend to make extreme and probably startling predictions about the future.  Does this sound familiar?  These are the Rush Limbaughs and the Michael Moores of the world.  The Glenn Becks and the Bill Mahers.  In fact nearly all of our most popular pundits would probably qualify as hedgehogs and in fact that’s exactly what Tetlock’s study found:

“And the more famous the forecaster the more overblown the forecasts. ‘Experts in demand,’ Tetlock says, ‘were more overconfident than their colleagues who eked out existences far from the limelight.’…  ’Our system of expertise is completely inside out: it rewards bad judgments over good ones.’”

This probably shouldn’t be too surprising.  I mean can you imagine listening to a “fox” on talk radio?  There would be no angry rants and no apocalyptic predictions about the future.  There would probably be very few claims that their perspective was being silenced or persecuted, and there would be no group to demonize or blame the country’s problems on.  Rather than fitting the events in the world together into a cohesive conspiracy theory and sounding the warning they would be deconstructing the world’s problems and pointing out areas where we still need more information.  When asked to predict something they’d often be able to give their thoughts but they’d frequently have to say, “I don’t know exactly, its an extremely complex situation.  It could go this way, but who knows for sure?”

In other words it would be boring.  The fact is, we like to listen to people who seem like they have got it all figured out.  We are drawn to charismatic TV commentators who can point out the enemy and give us a target for our frustration and confusion.  We enjoy reading books and editorials from people who can fit all the pieces together and make it all seem so simple.  The problem, of course, is that the world is not actually simple at all, current events are the result of millions of actors each with a multitude of motivations, we don’t have half of the puzzle pieces let alone the ability to fit them together, and in the end these people are literally wrong more often than dart-throwing monkeys.

I guess the lesson is, and this seems to be a common refrain on this blog, to be wary of those who seem to think they’ve got the world figured out.  Be careful of those who are confident, popular, or heavily partisan.  I suppose we could say we should try to get our information from “foxes” but perhaps the best course of action is to strive to become one ourselves.  And that’s the conclusion that the article seems to come to as well.  Though Tetlock kind of downplays some of the implications of his study (he is a pundit himself after all) the author finishes off with this observation, which I’ll allow to end this post:

“But the best lesson of Tetlock’s book may be the one that he seems most reluctant to draw: Think for yourself.”

No related posts.