The New Yorker gets an F in statistics

Thanks to Don Luskin at “Conspiracy to Keep You Poor and Stupid” for steering me to the New Yorker article this post is about.

The 9/25/06 issue of the New Yorker contains an article, “Wagers of Sin” by Suroweicke. Suroweicke laments in part that the ban on online betting has serious adverse consequences. Here is an excerpt from his article.

Furthermore, the ban on online betting is hindering the development of new markets that could predict far more important outcomes than that of the N.B.A. finals. In the past few years, a host of prediction markets, as they’re usually called, have appeared online, offering people the chance to speculate on subjects ranging from the box-office performance of Hollywood films to the outcome of Presidential elections and the spread of bird flu. These markets’ forecasts have proved remarkably accurate—just as bettors collectively do an exceptionally good job of predicting sports results. (In 2004, for instance, Tradesports, a Dublin-based prediction market, called thirty-three out of thirty-four races in the Senate correctly, and called all fifty states correctly in the results for the electoral college.) But in the U.S. these markets have to use play money, because using real money would constitute gambling. The online gambling ban prevents these markets from getting bigger and more accurate.

I favor online betting and believe that there are times when a betting market provides valuable predictions. However, Suroweicke’s article is misleadingly optimistic.

The idea behind betting markets’ accuracy is, roughly, that each individual bettor’s estimate is unbiased, can be viewed as the sum of true value plus mean zero error, and that the errors across bettors are approximately independent. If this property holds, then the error associated with the average estimate across bettors becomes very small as the number of bettors increases.

If betting markets really reflected these assumptions, future events could be predicted as accurately as desired by using the average prediction of a large number of bettors. Suroweicke gives the impression that this possibility is likely. In fact, it is to be expected that it is unlikely, particularly for any prediction where great accuracy is valuable. And Suroweicke’s quoting a few lucky exceptions to the rule does not invalidate that fact.

One problem is that a prerequisite for an accurate prediction to be valuable is that an accurate prediction is not easy, or even possible. Another problem is that bettors prediction characteristics are not characterized by Suroweicke’s implicit estimation assumptions. Moreover, at best, unbiased estimates provide an accurate estimate of the mean. Where there is large volatility around the mean, there is still great uncertainty.

Bettors’ estimates are often biased in the same direction and highly correlated. This does a betting markets’ prediction accuracy in just where prediction accuracy is most valuable.

To drive the ridiculousness of Suroweicke’s optimism home, see if you agree that the following events can be predicted accurately by simply using the average forecast of a large number of forecasters.

Next year’s price of xyz stock.

Next year’s price of a broad stock market index.

Your year of death.

The year New York will suffer a nuclear attack by terrorists.