Philip Tetlock (screen shot from high-res Seminar video at members’ webpage)

Philip Tetlock recently presented a Seminar About Long-Term Thinking to the effect that confident forecasters ought to be ignored. Despite his research showing the profound unreliability of such speculation, it’s rare to find even a moderately systematic evaluation of political forecasts in the popular media. In this category comes an article from the online version of Radar magazine earlier this year, “The Iraq Gamble”, which scrutinises journalistic commentators’ predictions about the war in Iraq. In it, writer Jebediah Reed tracks the fortunes of predictions against those of the predictors, and is disturbed by what turns up.

[M]aybe something is amiss in the world of punditry. Are the incentives well-aligned? Surely those who warned us not to invade Iraq have been recognized and rewarded, and those who pushed for this disaster face tattered credibility and waning career prospects. Could it be any other way in America?
…
So we selected the four pundits who were in our judgment the most influentially and disturbingly misguided in their pro-war arguments and the four who were most prescient and forceful in their opposition.
…
Then we did a career check … and found that something is rotten in the fourth estate.

The suggestion is that American meritocracy seems to be in trouble when it comes to the careers of political pundits. Eight journalists’ profiles are presented under two headings: “Getting rich by being wrong”, and “Right but poor”. As one of those drafted into the latter category, Jonathan Schell, remarks: “There doesn’t seem to be a rush to find the people who were right about Iraq and install them in the mainstream media.” Meanwhile, certain confident voices have prevailed despite the question marks over their track record (a similar point is made in an article from FAIR.org last year, addressing the Iraq-related statements of columnist Thomas Friedman, who also comes into the firing line in the Radar piece).

Whether or not one agrees with the argument as applied to this politically contentious topic — and a larger sample would be required to draw firm conclusions — it seems clear that improving accountability (i.e., correspondence of reputation and record) for predictions made by public figures remains a major challenge in politics and the media alike. It’s an aspiration embedded in the Long Bets project, and any readers who think they can do as well as, or better than, these guys — on any topic of public interest — is encouraged to test their long-term prescience by placing a wager on the site.

Share This ▼

The online version of Popular Science magazine is offering a prediction market for science and technology. It uses token dollars instead of real money (in order to avoid gambling laws). Here is what they say about it:

Ready to bet on the future?
Join the PopSci Predictions Exchange.

Welcome to the PPX, the first place to bet on the future of science and technology. It’s easy and free: Log on, and we’ll give you POP$250,000 in our virtual PopSci Dollars. Use that money to buy propositions you think are likely to happen. If other traders also want to buy, that proposition’s price will go up, and you’ll make PopSci bucks. Expand your portfolio with bets on energy, space, consumer technology and extreme science, and compete against other players for prizes and bragging rights.

Here is a sample chart from bets on whether Internet Radio Survives

Share This ▼

This is a very short article on how economists are using prediction markets to predict weather at least as good as meteorologists, which is not very good.

Penn State Researchers Testing Futures Markets For Weather Forecasting

UNIVERSITY PARK, PA (March 21, 2007) – Economists at Penn State’s Smeal College of Business and College of Earth and Mineral Sciences are testing whether futures markets can be used to accurately forecast the weather, and, so far, they’ve found the markets to be just as accurate as major forecasting services.

The 60 participants in this predictions market experiment, which is in the midst of a two-year run at Smeal’s Laboratory for Economics Management and Auctions, are mostly students studying business or meteorology at Penn state. They use allotted funds to bet on what they believe the high and low temperatures will be in different U.S. cities on a given day. As the going rates for various temperatures fluctuate within the market, the researchers can weigh the market’s confidence in what temperatures will be reached.

(more…)

Share This ▼

It’s a bit rare that we see such clear and succinct predictions. These three by Ari Emanuel were published at the Huffington Post and were forwarded to me by Paul Saffo:

I have three predictions to make this morning: 1) John McCain will not be the Republican Party’s nominee. 2) Hillary Clinton will not be the Democratic Party’s nominee. 3) Before the end of Bush’s term, Condoleezza Rice will be Vice-President.

http://www.huffingtonpost.com/ari-emanuel/three-predictions_b_41851.html

Share This ▼

“What is it about politics that makes people so dumb?”

From his perspective as a pyschology researcher, Philip Tetlock
watched political advisors on the left and the right make bizarre
rationalizations about their wrong predictions at the time of the
rise of Gorbachev in the 1980s and the eventual collapse of the
Soviet Union. (Liberals were sure that Reagan was a dangerous idiot;
conservatives were sure that the USSR was permanent.) The whole
exercise struck Tetlock as what used to be called an
“outcome-irrelevant learning structure.” No feedback, no correction.

He observes the same thing is going on with expert opinion about the
Iraq War. Instead of saying, “I evidently had the wrong theory,” the
experts declare, “It almost went my way,” or “It was the right
mistake to make under the circumstances,” or “I’ll be proved right
later,” or “The evilness of the enemy is still the main event here.”

Tetlock’s summary: “Partisans across the opinion spectrum are
vulnerable to occasional bouts of ideologically induced insanity.”
He determined to figure out a way to keep score on expert political
forecasts, even though it is a notoriously subjective domain
(compared to, say, medical advice), and “there are no control groups
in history.”

So Tetlock took advantage of getting tenure to start a long-term
research project now 18 years old to examine in detail the outcomes
of expert political forecasts about international affairs. He
studied the aggregate accuracy of 284 experts making 28,000
forecasts, looking for pattern in their comparative success rates.
Most of the findings were negative— conservatives did no better or
worse than liberals; optimists did no better or worse than
pessimists. Only one pattern emerged consistently.

“How you think matters more than what you think.”

It’s a matter of judgement style, first expressed by the ancient
Greek warrior poet Archilochus: “The fox knows many things; the
hedgehog one great thing.” The idea was later expanded by essayist
Isaiah Berlin. In Tetlock’s interpretation, Hedgehogs have one grand
theory (Marxist, Libertarian, whatever) which they are happy to
extend into many domains, relishing its parsimony, and expressing
their views with great confidence. Foxes, on the other hand are
skeptical about grand theories, diffident in their forecasts, and
ready to adjust their ideas based on actual events.

The aggregate success rate of Foxes is significantly greater, Tetlock
found, especially in short-term forecasts. And Hedgehogs routinely
fare worse than Foxes, especially in long-term forecasts. They even
fare worse than normal attention-paying dilletantes— apparently
blinded by their extensive expertise and beautiful theory.
Furthermore, Foxes win not only in the accuracy of their predictions
but also the accuracy of the likelihood they assign to their
predictions— in this they are closer to the admirable discipline of
weather forecasters.

The value of Hedgehogs is that they occasionally get right the
farthest-out predictions— civil war in Yugoslavia, Saddam’s
invasion of Kuwait, the collapse of the Internet Bubble. But that
comes at the cost of a great many wrong far-out predictions— Dow
36,000, global depression, nuclear attack by developing nations.

Hedgehogs annoy only their political opposition, while Foxes annoy
across the political spectrum, in part because the smartest Foxes
cherry-pick idea fragments from the whole array of Hedgehogs.

Bottom line… The political expert who bores you with an cloud of
“howevers” is probably right about what’s going to happen. The
charismatic expert who exudes confidence and has a great story to
tell is probably wrong.

And to improve the quality of your own predictions, keep brutally
honest score. Enjoy being wrong, admitting to it and learning from
it, as much as you enjoy being right.

–Stewart Brand

(Iraq footnote. I asked Tetlock to opine on which experts were most
right about how things have gone in the Iraq War. He said the most
accurate in this case were the regional experts, who opposed the
invasion, and what they are predicting now is a partition of Iraq
into Kurdish, Shia, and Sunni areas.)

Stewart Brand
Seminars & downloads: http://www.longnow.org/projects/seminars/

Share This ▼

With the Republican control of the House and Senate, there is no scenario in which a Democrat can become President by August 2005 (the two-year horizon of my original Bet). Looking at the two arguments, Eno’s is persuasive in detail, mine clearly wrong.

The morning after the election I wrote Eno my summary of what happened and the prospects…

Dear Brian,

You win, we lose. My country is in a bad way and getting worse. Exit polls said that Bush supporters voted for his “moral values.” For “strong and wrong versus weak and right,” as Bill Clinton has put it.

Bin Laden won. He tried against all odds to set in motion a religious war, and he succeeded. The US majority WANTS a religious war of good against evil. Irrationality rules here, and as a result a very cynical Republican party rules— Presidency, Senate, House, judiciary, governorships, and state legislatures. They own America and are running it at a profit.

The unreported divide: educated Americans voted strongly for Kerry, the uneducated for Bush.

Kerry and the Democrats did everything as right as could be. They couldn’t have fielded a stronger candidate, nor worked harder individually to get out the huge vote. Bush’s war in Iraq went as badly as it possibly could, yet he was not held accountable.

Now I can only hope for your scenario to play out—Bush has to suffer the consequences of his own incompetence. But it will be a bleak four years. Irrationality feeds on failure. As things get worse, it just tries harder, gets shriller. I hate the prospect.

The turmoil will be good for the arts perhaps. We may see a revival of the religious left.

Economic wars can end; ideological wars can end. Religious wars go on and on. Long term I can imagine a new Enlightenment coming out of all this, and I will work for that.

–Stewart

Share This ▼

The Red Sox have won the Series and with it falls Mike Elliot’s argument around the speed of globalization vs. the Red Sox pitching depth.  Below is the write up in the New York Times.

Hey, Cliff Clavin, This Time Sam Malone’s the Smart One
By JONATHAN FUERBRINGER
The New York Times

Published: November 1, 2004

here once was a time when one of the (im)potent symbols of the luckless Red Sox Nation spent his time at a bar called Cheers. Sam “Mayday” Malone, a fictional former Boston pitcher ruined by drink, tended the bar he owned, chased his waitress, and probably placed some bets on his old team.

Now Ted Danson, the actor who portrayed Sam Malone, is making Red Sox fans proud for a different reason: as a result of Boston’s World Series victory last week, Mr. Danson won the first bet ever decided at Long Bets (longbets.org), an online prediction site that focuses on scores that may not be settled for 45 years, if ever.

The Web site, which is popular among the Silicon Valley digerati, is a spinoff from the Long Now Foundation in San Francisco, whose aim is to foster long-term thinking and stimulate discussion about the future. Members make predictions, typically about topics like immigration or artificial intelligence, and other members challenge them, accompanied by a minimum wager of $200. Unlike most betting parlors, however, these contest are rarely about sports. Alexander Rose, executive director of the Long Bets Foundation, said all the predictions had “some social or scientific value.”

The Red Sox bet slipped in because Michael Elliot, the editor of Time Asia, argued that the United States soccer team would win the World Cup before the Red Sox won the World Series. In an argument posted on the site, he said his larger point was that immigration and technology would improve the quality of American soccer, but that the curse of the Bambino was “one of those mystical truths that are beyond the reach of human intervention.”

Mr. Danson’s counter was also scientific. “Statistically, scoring goals is harder than hitting a home run and in the World Cup you have the whole WORLD against you,” he wrote. In baseball, he argued, “the Red Sox only really have to beat the Yankees.”

Mr. Danson, who was not available to comment, now gets to donate $2,000 (they each bet $1,000, but winners receive bragging rights but no cash, according to the site’s rules), plus interest accumulated since the wager was made in February 2002, to a charity of his choice. Mr. Elliot, a self-described Yankee fan, said in an e-mail response, “I feel sorta proud to be the first to have lost, actually!”

If Chicago Cubs fans are so inclined, there is now room for a bet on which century that team will end its World Series winless streak.

Share This ▼

This article was written by Peter Schwartz for Red Herring’s 02002 Scenarios issue. This is the original un-edited piece.

Yucca Mountain in southern Nevada is more a ridge than a mountain. It slowly rises from a height of four thousand feet to six thousand feet along its’ length of six miles. On February 28 seven colleagues of mine from the Board and staff of The Long Now Foundation rode in an open train into one of the biggest holes in the world bored into the face of Yucca Mountain. Beginning at the north portal of a five-mile long C shaped tunnel the train carried us about a mile and half into Yucca Mt. In 1997 the 25-foot diameter borer machine emerged from the face of the mountain to open the other end of the tunnel three miles south of the north portal. For most of its length the tunnel is about a thousand feet beneath the summit of the mountain and even more important a thousand feet above the water table. That’s important because, of course, this tunnel in Yucca Mountain is where the United States government is intending to store the nation’s high level nuclear waste for the next ten thousand years and beyond.

The question of the future of Yucca Mountain has become very current because on February 14, 2002 Secretary of Energy Spencer Abrams recommended the approval of Yucca Mt to the President who acted the next day to notify the Congress of his intention to go ahead. By the time you read this it is virtually certain that the Governor of Nevada will have acted to veto the project as the Nuclear Waste Policy Act of 1982 allows him to do. Congress then has 90 session days to override the governor. The issue in the Congress will undoubtedly be highly contentious in both the Republican controlled House and the Democrat controlled Senate with the outcome not predetermined. In the end it is likely that the Congress will go along with the President because no other state want’s nuclear waste in their backyards either.

The more than five miles of tunnels, cross drifts and alcoves that have been drilled so far are really part of what is called the Exploratory Studies Facility. It is a research program, costing $8 billion so far, intended to prove the safety of the repository for ten thousand years. If it morphs into the actual nuclear waste site then they will bore another sixty miles of tunnels branching off the main one where they will actually store the hot waste. Deep in the tunnel we saw one of the current research projects designed to test the consequences of the heating that the sealed in nuclear waste will produce. In a tunnel branching several hundred feet off the main tunnel we found the last half sealed off. Peering through a very hot pane of glass we could see along row of huge heaters lined up back into the end of the tunnel. The heaters had raised the temperature in the tunnel to several hundred degrees over four and had just been turned off a few weeks ago for their four year cool down.

What’s the urgency to get Yucca Mt on line? Today the country’s 104 nuclear plants and the nuclear weapons program have produced 40, 000 metric tons of spend fuel. By 2035 it will be two and a half times that. Most of that waste is currently stored in 33 states at a few Dept. of Energy sites and at the sites of 72 nuclear power plants in what are euphemistically called “swimming pools.” These were designed as temporary storage sites where the risks of dangerous failures are increasing over time. So something must be soon with the existing waste let alone what is to come. And even if we get started now it will be 2010 before any waste goes underground. It will take that long to build out the necessary infrastructure for handling this very nasty stuff.

This is very big science and truly great engineering at the service of bad politics. At Yucca Mt we met remarkably creative people who have spent much of their working lives in very harsh conditions trying to solve one of the toughest problems we have in this country. That the problem as posed is insoluble is not their doing. Politicians on both sides of the issue, proponents and opponents of nuclear power have engaged in the politics of illusion at great cost to the American people. The opponents that cannot be realistically achieved have set a target of perfect isolation for 10,000 years. This is their way of blocking nuclear power. So the proponents in turn design a deceptive process to validate the achievement of an unattainable goal.

So what are the options? We can leave them where scattered around the country in temporary facilities. This has very high risks of something wrong and no one finds it acceptable. We could recycle the fuel for reuse. So far, however, cost, its own environmental problems and most of all, the dangers of nuclear weapons proliferation, have stopped nuclear fuel reprocessing. Most current process for recycling nuclear waste yields plutonium that can be used for weapons.

Or as currently planned we can store somewhere for along time. That means Yucca Mt or somewhere else. No one wants nuclear fuel around but Yucca Mt has a few advantages. It sits at the edge of the Nevada Nuclear weapons testing site. Shortly after clearing the gate of the site if instead of turning left toward Yucca Mt we had turned right we would have encountered dozens of sites of nuclear weapons test both above ground and underground. As it was we crossed Jackass Flats where we tested a nuclear powered rocket motor in the late fifties and early sixties. This isn’t prime development real estate. Indeed some hint of the local attitude is the fact that there are two prisons on the 100-mile drive out from Las Vegas.

There are several possible scenarios for the future of Yucca Mt. The opponents could successfully block it indefinitely. It is not too hard to imagine opponents lying across the railroad tracks as the nuclear waste trains make their way to Nevada. It ends up like some other federal energy related projects, never being used. Something else would have to come along, like cheap safe recycling to make this an enduring scenario. Eventually you have to clean up the local mess one way or the other.

Of course, we could put the waste into Yucca Mt and remains their uneventfully for tens of thousands of years stretching on indefinitely into the future. However it is not impossible that something goes wrong relatively soon, say in the next thousand years. Perhaps the heat and the radioactivity lead to the breakdown of the storage vessels soon along with more rapid intrusion of water into the repository could lead to the poisoning of the aquifer. This would lead to big regrets.

Or as I think most likely we will put it in and take it out. There is likely no better answer in the short run. But I think we will be come so concerned about the consequences of burning hydrocarbons, especially the impact on climate change that we will want to revive nuclear power. We may come to want the usable fuel buried in the waste. The technology for fuel reprocessing and for nuclear plants themselves are both likely to improve dramatically in the future. We just don’t when. There are no risk free answers. But it appears that the balance of risks and the least regrets scenario is to store the waste at Yucca Mt and invest in better re-cycling technology to create future options for our children.

-Peter Schwartz 3/3/02002

Share This ▼