2005 December 12 Monday
Experts Have High Error Rates In Prognostications
Experts and commentators make predictions with high error rates and rarely get called on it.
It is the somewhat gratifying lesson of Philip Tetlock’s new book, “Expert Political Judgment: How Good Is It? How Can We Know?” (Princeton; $35), that people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be.
Go read the article for details on the research which Tetlock conducted to reach these conclusions. The whole article is worth your time.
What we really need: Studies like Tetlock's but designed to survey a much larger group of people in order to identify people who have much better rates of being right. We can't trust TV news show producers to wisely choose experts whose rates of accuracy are greater than sheer chance. In fact, the only thing we can count on is that the most famous prognosticators and commentators will be wrong more often than most people. If the talking head consensus is X then the truth is not X.
Consider what the article says and introspect about your own thinking. There are some obvious simple rules of thumb to follow such as resist committing to a position if you do not have strong proof for it. If you commit publically you will become less likely to recognize the incorrectness of a position if you feel your reputation is at stake. Also, just because a person spouts lots of details doesn't mean the details somehow prove the argument. Experts have tons of facts and terrible track records in predicting complex human events. When it comes to human affairs the factors influencing outcomes are so many and so little understood that the feeling of certainty is not something you should feel too often.
I found one part of the article gratifying: The bit about people who ascribe big single causes to explain events are wrong more often than the people who think many factors contribute. For example, I think the Iraq democracy project is doomed for a whole host of reasons including cousin marriage, Islam, ethnic divisions, low IQs, Arab culture, and still other factors. Contrast that with the position that liberal democracy is an inevitable historical force, a sort of Manifest Destiny.
Tetlock says big name commentators are so bad because their predictions are designed more to make people feel good about their side than to be correct.
Tetlock notes, sadly, a point that Richard Posner has made about these kinds of public intellectuals, which is that most of them are dealing in “solidarity” goods, not “credence” goods. Their analyses and predictions are tailored to make their ideological brethren feel good—more white swans for the white-swan camp. A prediction, in this context, is just an exclamation point added to an analysis.
This is a problem. The market of information for those who want to know the truth is an awful lot smaller than the market of information for those who want to feel good about themselves and their faction.
While I agree with the overall premis, I think that there might also be a selection skew happening - after all, media isn't actually interested in reporting a real expert because they will tend to qualify their statements. Instead, the media will select those who are a combination of (a) forcefully presented, and (b) predicting the worst. This filters out people who have complex thoughts about complex topics (a yawn to the vast majority of the population - present company entirely excepted).
A no-news day is a dream for readers and a nightmare for journalists, but a horror story is a dream for jounalists and a nightmare for readers. So, with that in mind which expert will a journalist select - the 'nothing to see here' guy, or the 'sky is falling' guy?
This an opportunity for a nonexpert, namely myself, to admit to some bad predictions that I made. One was that the US economy would be in recession by now, from the oil shock. Energy prices are still about as high as their peaks, yet there is no indication of recession at present. I predicted that the French riots would lead to troops being called back from the Ivory Coast, and that renewed war would inflame that country. Actually, the riots burned out on their own, without any such extraordinary measures.
You get an A+ for honesty. Maybe you should write for the op-ed page of the New York Times - it would be refreshing.
But the non-celebrity experts studied also did not have good track records at prediction.
I have a pretty good record. I think it pisses people off, more often than not. However, I doubt if my being right infuriates people anywhere near as much as the lazy, lying good-for-nothings who make up the pundits and decision-makers in the country infuriate me.
The self-proclaimed experts on this blog have an even worse record. All the worse for their boasting.
This is exactly what you expect, however. They remember the rare cases of being right, and forget the more common cases of being wrong.
Substance, Marvin! Substance! Where's the beef?
On a more serious note, Marvin, since you have obviously made some quantitative analysis of the accuracy of predictions around here, perhaps you could share a little more of your method and data than just the final conclusion.
Before the war, I predicted that Iraq had no nuclear program and not much of anything else. Before the war, I predicted that we'd find nothing significant in the WMD area and that the Administration would end up blaming it on the CIA. Before the war, I thought we'd probably see guerrilla resistance: including guerrilla resistance from groups that originally welcomed us, which has happened some and will happen more. Just before the war, I predicted we'd win the invasion phase in a breeze and lose on the order of 100 KIA. By June 2003, I figured that'd we see a low-level resistance to which we'd lose on the order of 1,000 KIA a year indefinteily - until we got sick of it and left. I figured, in less than one minute, than our decison to dissolve the Iraqi Army would have very bad consequences. By the summer of 2003, I figured that we weren't going to manage to export oil at a higher rate than before we invaded: oil exports are in fact noticeably lower nowadays. And Iraq is all about OIl, you know: they're worth nothing without it. I thought that catching Saddam's sons would not stop the resistance. I thought that catching Saddam would not stop the resistance, and might actually strengthen it (make it easier for people to join), as well making the Shi'ites feel free to cause trouble, with the bogeyman gone - which they did. I figured we'd end up spending tens of billions a yearon the occupation. I was a little low. By August 2003, a good friend of mine (this one I didn't manage to say first) thought that our best possible outcome was a Shi'ite Islamic Republic of Iraq, close to Iran, an outcome that would do the US no damn good at all.
Oh, and I thought we'd find no evidence of Iraqi connections with Al-Qaeda or 9-11, and of course we did not.
I feared (and expected) suicide bombings and car bombings) as in the West Bank and in Israel's occupation of southern Lebanon.
I expected that in general, we would execute the occupation of Iraq in a remarkably stupid way, because anyone stupid enough to invade a country for no fucking reason in the first place would certainly continue to be stupid. Pretty much everyone now admits this.
I figured that a lot of people I know to some extent (for example, John Derbyshire, at NR ) would gradually realize what a bust this was and want to withdraw: I told him that he would do so about a year and half before he actually did. Of course I was furious at him for taking so long and being such a lazy git in the first plac, as he had to have been to go along with this nonsense. I was irritable with people who should know better making what I conidered to be an obvious mistake: ask Randall.
Oh, what else.. I predicted that no country would be crazy enough to send more troops to aid in the occupation after things went sour, that we'd alienate much of the world, that public opinion in Arab and Moslem countries woud end up much more unfriendly to the US than before the invasion.
As for the idea that we're going unleash Democracy in the Middle East - I'd rather see the US invest a few hundred billion in magic beans than in arab democracy. The expected payoff is higher.
This must be the only US war _during which_ the Army War College has pointed out its absolute stupidity.
These are beings trues believer not thinker here. Is beating breast like ape, me, me. Dumabass being too stupid to know elections going on. Dumabass.
Elections don't necessarily mean much, especially in places like Iraq where less than half the population can even read (accordng to the CIA world factbook). And for those that can read, what do they read? Locke? Montesquieu? Jefferson? Somehow, I doubt it.
Some countries have had civil wars due to the outcomes of elections. Elections are not a magic dust that makes a populace into respecters of each others' rights.
In case you haven't noticed, the elections in Iraq have not decreased the level of violence there. But the war party has repeatedly predicted that each election would decrease the violence level. When the war party is continually wrong I discount the predictions of the war party. Seems a reasonable thing to do.
This is what the PAM (policy analysis market) trading system was supposed to rememdy. Pundits, "experts", acadamians, and other associated parasites would buy and sell futures based on predictions. Such a trading system would force these people to put their money where their mouths were. It is also known that market trading systems have a better track record at making predictions that have turned out to be true than some pundit or expert's claim.
Of course, the PAM got shot down because of emotional hysteria having no connection to reality. In any case, one could still start a PAM with private fiancing if one chose to do so.
Silchiuk, where are you from?