Tuesday, July 25, 2006

Freakonomics author Steven Levitt: Incompetent? Unethical?

Steven Levitt (along with Donohue), the well known economist author (along with Dubner) of Freakonomics, advocated the theory that the advent of legalized abortion in the 1970s is responsible for much of the steep and persistent drop in the crime rate during the 1990s. The research behind this claim was a paper by Donohue and Levitt. However, as Foote and Goetz [2005] showed, it turned out to be wrong.

Donohue’s and Levitt’s mistake is so elementary as to suggest incompetence or the intent to mislead. These two possibilities also are consistent with Donohue’s and Levitt’s criticism of John Lott’s research showing that allowing honest citizens to carry concealed weapons reduces violent crime.

Donohue has written copiously on Lott’s work, all of it negatively. However, as Plassman and Whitney show in their Stanford Law Review article “Confirming More Guns Less Crime”, Ayres and Donohue’s work appears to be substandard, including statistical errors and claims not supported by the data. Moreover, Donohue has continued to make negative claims in the face of several papers pointing out his errors. Here, too, the mistakes and continued apparently incorrect claims are so elementary and pervasive as to suggest incompetence or the intent to mislead.

Levitt has taken up Donohue’s cudgel and attacked Lott in Freakonomics, saying:

Then there was the troubling allegation that Lott actually invented some of the survey data that supports his more-guns/less-crime theory. Regardless of whether the data were faked, Lott’s admittedly intriguing hypothesis doesn’t seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don’t bring down crime.

In fact, Lott’s results have been replicated and supported by other researchers. This was known in the field. Indeed one researcher sent the following email to Levitt.

I also found the following citations – have not read any of them yet, but it appears they all replicate Lott’s research.

Levitt’s response was:

It was not a peer refereed edition of the Journal. For $15,000 he was able to buy an issue and put in only work that supported him. My best friend was the editor and was outraged the press let Lott do this.

Levitt’s response misstated the facts.

It seems that Levitt now has a record on at least two important issues, abortions and guns. In both cases he has confidently and repeatedly espoused a position that is probably wrong and that he should have known was probably wrong. In one case, he has compounded the error by apparently acting unethically, i.e., by possibly libeling a competent researcher in lieu of admitting his own likely mistake.

Here is Foote and Goetz’s abstract.

State‐level data are often used in the empirical research of both macroeconomists and microeconomists. Using data that follows states over time allows economists to hold constant a host of potentially confounding factors that might contaminate an assignment of cause and effect. A good example is a fascinating paper by Donohue and Levitt (2001, henceforth DL), which purports to show that hypothetical individuals resulting from aborted fetuses, had they been born and developed into youths, would have been more likely to commit crimes than youths resulting from fetuses carried to term. We revisit that paper, showing that the actual implementation of DL’s statistical test in their paper differed from what was described. (Specifically, controls for state‐year effects were left out of their regression model. ) We show that when DL’s key test is run as described and augmented with state‐level population data, evidence for higher per capita criminal propensities among the youths who would have developed, had they not been aborted as fetuses, vanishes. Two lessons for empirical researchers are, first, that controls may impact results in ways that are hard to predict, and second, that these controls are probably not powerful enough to compensate for the omission of a key variable in the regression model.

Here is the abstract from Ayres’s and Donohue’s paper “Shooting Down the More Guns, Less Crime Hypothesis”.

John Lott and David Mustard have used regression analysis to argue forcefully that “shall issue” laws (which give citizens an unimpeded right to secure permits for concealed weapons) reduce violent crime. This article shows that the claim has support from certain facially plausible statistical models, but that these are rejected by a variety of statistical tests. Estimating more statistically preferred disaggregated models on more complete data, we show that in most states shall issue laws have been associated with more crime. Using our expanded data set and our preferred jurisdiction-specific regression model, we show that more states have experienced an upturn in crime than have experienced a downturn in crime after enacting the law and that the apparent stimulus to crime tends to be especially strong for those states that adopted in the last decade. We estimate that on net the passage of the law in 24 jurisdictions has increased the annual cost of crime somewhere on the order of half a billion dollars. We also provide an illustration of how our jurisdiction-specific regression model has the capacity to generate more nuanced assessments concerning which states might profit from a particular legal intervention.

Here is an excerpt from the introduction to Plassman’s and Whitney’s Stanford Law Review article “Confirming More Guns Less Crime”.

Quite a few empirical papers have examined the impact of right-to-carry laws on crime rates. Most studies have found significant benefits, with some finding reductions in murder rates twice as large as the original research. Even the critics did not provide evidence that such laws have increased violent crime, accidental gun deaths, or suicides.

Unlike previous authors, Ian Ayres and John Donohue claim to have found significant evidence that right-to-carry laws increased crime. However, they have misread their own results. The most detailed results they report. following the change in crime rates on a year-by-year basis before and after right-to-carry laws were adopted.clearly show large drops in violent crime that occur immediately after the laws were adopted. Their hybrid results showing a small increase in crime immediately after passage are not statistically significant and are an artifact of fitting a straight line to a curved one. When one examines a longer period.from 1977 to 2000.even this type of result disappears.

Ayres and Donohue.s efforts have been valuable in forcing others to reexamine the evidence, extend the dataset over more years, and think of new ways to test hypotheses, and we appreciate their efforts. They are both highly regarded and well-known for their research, such as claiming that the legalization of abortion can account for half the drop in murder during the 1990s. Unfortunately, their research on this issue inaccurately describes the literature and also fails to address previous critiques of their work. For example, Ayres and Donohue claim that "[w]hen we added five years of county data and seven years of state data, allowing us to test an additional fourteen jurisdictions that adopted shall-issue laws, the previous Lott and Mustard findings proved not to be robust". All their tables report results for "Lott's Time Period (1977-1992)" and compare those estimates with the "Entire Period (1977-1997)". Yet, whatever differences in results arise, they are not due to the inclusion of more data for a longer period. Their paper gives a misleading impression as to how much their research extends the data period, since Lott's book and other work examined both the county and state data up through 1996. Ayres and Donohue's work thus extends the county-level data by one year, from twenty to twenty-one years.

3 comments:

Caveat B said...

Leavitt responded to the Foote and Goetz's 2005 paper, which can be found here:

http://pricetheory.uchicago.edu/levitt/Papers/ResponseToFooteGoetz2006.pdf

TOG said...

Here is a link

http://post.economics.harvard.edu/faculty/miron/papers/Comment_on_DL_FG.pdf

that puts Donohue and Levitt's response in perspective. As usual, Donohue and Levitt appear to still be trying to pull the wool over everyone's eyes. As the link reference notes:

"Our conclusion is that the kind of analysis considered in Table VII of DLII does not suggest a
quantitatively important effect of legalized abortion on crime. The best case for such an effect is the
IV results in columns (6) and (7); these imply that abortion legalization explain 24-25.9% of the
1991-1998 decline in violent crime and 7.1-8.1% of that in property crime. None of these
coefficients is statistically significant at conventional levels, however, and the results in column (8)
suggest they rely on an implausible mechanism relating abortion to crime."

In other words, Donohue and Levitt appear to be willing to data mine in order to avoid admitting their likely errors.

One thing to remember is that the statistical tests that everyone is quoting presume a one shot modeling try. Trying all sorts of combinations of variables and techniques and then quoting the most in line with what you want it to be is, in my view, outright fraud.

TOG said...

Another link worth looking at is to the Lott and Whitly refutation of Donohue and Levitt's abortion reduces crime thesis.

http://papers.ssrn.com/sol3/papers.cfm?cfid=669548&cftoken=22941286&abstract_id=270126#PaperDownload

Here is the conclusion from that paper. it is diplomatically put. The message is that Donohue and Levitt screwed up big time.


"There are many factors that reduce murder rates, but the legalization of abortion is not one of
them. Of the over six thousand regressions that we estimated here, only one regression
implied even a small reduction in murder rate. All the other estimates implied significant if
very small to modest increases in murder rates: legalizing abortion would increase murder
rates by around 0.5 to 7 percent. Yet, presumably the lower end estimates would not appear
to have a significant impact on the overall abortion debate. Donohue and Levitt suggest that
abortion reduces annual victimization costs by $30 billion, with most of this coming from
reductions in murder. Our results indicate that total victimization costs rose as a result of
abortion."

"Many times academics can not avoid using aggregate crime data. Yet, the linking of abortion
and crime is not such a situation: examining total crime rates and not directly linking
abortions and the crimes committed by individual cohorts missed catching obvious patterns.
Yet, even if Donohue and Levitt believe that the correct approach is to link crimes committed
by all ages with their aggregate “effective abortion rate,” sensible minor adjustments such as
allowing the share of crime committed by different ages to vary across states and years rather
than assuming that the weights are constant reverses the sign on their estimates."

"This is not to suggest that the hypothesis provided by Bouza-Morgentaler-Donohue-Levitt is
not plausible, but at least that it is not the most important part of the story. Abortion can
eliminate unwanted children and can benefit many women, but it can also make other women
who are unable to bring themselves to have an abortion worse-off and more likely to have
out-of-wedlock births. Like many laws there appear to both winners and losers, and here the
affect appears to be a net reduction in human capital."