An example of the dishonesty of some anti-gun academics

Here is a rejoinder by Carlisle Moody, John Lott, and Thomas Marvell to an American Law and Economics Review article by Abhay Aneja, John Donohue, and Alexandria Zhang criticizing - and even slandering Lott.

This kind of dishonest and/or incompetent research by Lott's academic critics is common.  Donohue, in particular, has written (alone and with co-authors) several academic papers criticizing Lott's work showing that allowing honest citizens to carry concealed firearms reduces the violent crime rate, including murders.  Yet, despite being shown the flaws in his research that invalidate his claims, Donohue and his co-authors have continued their unethical behavior of misrepresenting Lott's work.

Here is the rejoinder.
-----------------------------------------
In an American Law and Economics Review article published during 2011, Abhay Aneja, John Donohue III, and Alexandria Zhang (hereafter ADZ) examined Chapter 6 of Firearms and violence: ACriticalReview, a 2005 report from the National Research Council (hereafter NRC). The chapter examined by ADZ is concerned with the effect that right-to-carry laws have on crime. The laws are also known as shall-issue laws, and we employ that term. Shall-issue laws require authorities to issue concealed carry permits to all persons who meet certain legislated requirements. Aside from Illinois, states that have not passed shall-issue laws leave it up to the issuing authorities, typically local police or sheriff departments, to determine whether or not to grant the applicant a concealed weapons permit.Such states are known as “may-issue” states.It is usually the case that may-issue states, especially in urban cities and counties, issue very few concealed carry permits, and most of these go to celebrities, wealthy individuals, and politicians (Snyder 1997). An interesting policy question is whether shall-issue laws, which increase the number of concealed carry permits, increase or decrease crime. One theory is that criminals, knowing that some ordinary citizens may be carrying firearms and, being unable to tell those who are from those who aren’t, will be more likely to forego a violent crime for fear of being met with armed resistance. Under this theory, violent crime should go down as a result of the passage of shall-issue laws.

The original article in this area is by John Lott and David Mustard (1997), who found that states with shall-issue laws had significantly lower violent crime rates than may-issue states or states that ban concealed carry. The publication of Lott and Mustard’s article generated a controversy that continues to this day. The Lott and Mustard results have been tested many times: by our reckoning, there have been at least 29 peer-reviewed studies by economists and criminologists, with a majority finding some support for the hypothesis that shall-issue laws reduce crime, many (including the NRC report) not finding any significant effect on crime, and only a few finding that shall-issue laws cause an increase in one or more types of violent crime (Lott2010,284).

ADZ (2011) attempted to replicate the results of the 2005 NRC report with a data set that they received from NRC. Their attempts at replication failed. “We cannot replicate the NRC results using the NRC’s own data set…. [O]ur… estimates diverge wildly from the…estimates [that] appeared in the NRC report” (ADZ 2011, 583). In the conclusion of their article they discuss their problem in replicating the NRC results. We find their discussion murky. Here we quote at length the key passages of the discussion; the parenthetical remarksareADZ’s, but we have bolded some words:

Data reliability is one concern in the NRC study. We corrected several coding errors in the data that were provided to us by the NRC (which had originally been obtained from John Lott). Accurate data are essential to making precise causal inferences about the effects of policy and legislation—and this issue becomes particularlyimportant when we are considering topics as controversial as firearms and crime control. We attempted to mitigate any uncertainty over data reliability by re-collecting the data. However, when attempting to replicate the NRC specifications—on both the NRC’s and our own newly constructed data sets—we consistently obtained point estimates that differed substantially from those published by the committee. 

Thus, an important lesson for both producers and consumers of econometric evaluations of law and policy is to understand how easy it is to get things wrong. In this case, it appears that Lott’s data set had errors in it, which then were transmitted to the NRC committee for use in evaluating Lott and Mustard’s hypothesis. The committee then published tables that could not be replicated (on its data set or a new corrected data set), but which made at least Professor James Q. Wilson think (incorrectly it turns out—see our Tables2a–c) that running Lott Mustard regressions on both data periods (through 1992 and through 2000) would generate consistently significant evidence that RTC [or shall-issue] laws reduce murder. (ADZ2011,613-614)

There are two questions to ask about ADZ’s inability to replicate the NRC results. The first question is: What was the source of the inability to replicate? We now know that the source was ADZ’s estimation of a misspecified model, a fact later admitted by ADZ (ADZ 2012; Aneja, Donohue, Pepper, Wellford, and Zhang 2012). But at the time of the 2011 article, ADZ presumably thought they were estimating the same model as did the NRC, in which case they would have replicated the NRC results since the same programs applied to the same data would yield the same results.

The second question is: What was ADZ’s understanding of the source of their inability to replicate? And, correspondingly, what were they suggesting to readers was the import of that inability? It seems that ADZ either concluded that the same program applied to the same data generated different results, or they thought—and perhaps were suggesting—that two different data sets, or tables based on different data sets, both originating with Lott, had been in play. The latter interpretation might fit ADZ’s mention of“on its data set or a new corrected data set. ”Since, for the data sets they themselves constructed, ADZ had used the expression “our own newly constructed data sets,” perhaps “a new corrected data set” is meant to suggest a second Lott-originated data set.

If researchers receive reports that a data set is inconsistent and unreliable, that sows seeds of doubt about all the research that has made use of that data set. Many studies have used the Lott data in question. Since we now know that the source of  ADZ’s failure to replicate was their having estimated the wrong model, we know that the published articles using Lott’s data have not been invalidated because of critical data errors. The picture as sketched by ADZ (2011) is vague, but their speculation that “it appears that Lott’s data set had errors in it” turns out to be unfounded. In two items released in 2012, ADZ themselves admit their error. But they do so in a way that fails to take responsibility for or rectify the doubts they had sown about the data and, therefore, the studies using the data.

In October 2012, the American Law and Economics Review published an Erratum bearing the names of ADZ along with those of John V. Pepper and Charles F.Wellford.The Erratum begins:

In section four (pp.578-584)of the above referenced article[ADZ2011] the authors report their efforts to replicate some of the results of analyses conducted by a panel of the National Research Council (NRC) and reported in Firearms and Violence (2004) [sic]. Based on this analysis, Aneja, Donohue and Zhang (2011) conclude that they “cannot replicate the NRC results using the NRC’s own data set” (p.583) and that the NRC committee “published tables that could not be replicated.” (p. 614). Subsequent to the publication of this article, members of the NRC panel demonstrated to th eauthors that the results in question were replicable if the authors used the data and statistical models described in Chapter6 of the NRC (2004) report. The results presented in Tables1b and 2b of Section 4 of the article do not replicate the NRC results because different data and models were used in the attempted replication effort. Thus, the results reported in the article should not be interpreted to mean that if one uses the data and model used by the NRC panel the results they reported cannot be replicated. In fact, replication using the NRC’s data and models produces results that are identical to those reported by the NRC panel. (Aneja,Donohue,Pepper,Wellford,andZhang2012,601)

The Erratum says that ADZ’s replication efforts failed because in those efforts “different data and models were used. ”In fact, the problem was in the specification of the models estimated. Any concerns about the data that ADZ might have had, such as the “few small errors in the NRC data” they listed in the 2011 paper (ADZ 2011,585), could not have caused the failure to replicate. So ADZ, as coauthors of the Erratum, are not being candid about the source of the problem.

The Erratum then has a second and concluding paragraph. It speaks of the importance of making data and replication files publicly available, and then concludes:

Donohue, Aneja, and Zhang have now issued an amended version of the published paper, which removes the above inaccurate claim about replication of the NRC results and corrects some other minor errors that do not alter the major findings of the published paper: Donohue, John J., Aneja, Abhay and Zhang, Alexandria, The Impact of Right to Carry Laws and the NRC Report: The Latest Lessons for the Empirical Evaluation of Law and Policy (July 27, 2012). Available at SSRN: http://ssrn.com/abstract=2118893. (Aneja, Donohue, Pepper, Wellford,andZhang2012,602)

Thus, in August 2012, ADZ issued a National Bureau of Economic Research working paper, 93 pages in length, that is supposed to supersede their journal article published the preceding year. The title of the paper has changed by the addition of the word“ Latest. ”The 2012 working paper does not explain itself in relation to the article published the preceding year. In fact, the working paper acknowledges the existence of the 2011 journal article only in footnotes, the first of which appears on page 21 (ADZ2012,2 1n. 11, 23n. 14, 29n. 18). The journal article does not appear in the Reference section of the working paper(74-77).

In the text of the 2012 working paper, the authors write: “With the help of the NRC Committee members who provided the NRC 1977-2000 county data set,  we were ultimately able to generate the NRC panel data estimates ”(20-21). At the end of the sentence they hang a footnote, which reads in full:

The initial published version of this article—Aneja, Donohue, and Zhang(2011)—noted that we had originally failed to replicate the NRC results, with our efforts complicated because the Committee had misplaced the do files that generated the NRC estimates. After publication, we were informed of the precise specification the NRC had employed, which did generate the published NRC estimates (although these estimates are flawed in the manner described in the text).(2012,21n.11)

Whatever ADZ’s problems in obtaining the do files from NRC might have been, what is significant is that this admission is relegated to a footnote, and that it in no way acknowledges that they had suggested that their inability to replicate had arisen from misfeasance in the Lott-originated data. In the new working paper those accusations are now somewhat muted but not altogether eliminated. For example, “there were some errors in the data,whichwassuppliedbyLott”(ADZ2012,21), “clearerrorsintheLottandMustardspecification,data, and standard errors” (22), “inaccurate Lott data” (27), and, most egregiously, “pure data errors that entered into the NRC data set when Lott transmitted an imperfect data set” (71). Thus, ADZcontinue to mention “data errors” without noting that these errors involved minor judgment differences in the timing of when laws were enacted and, more importantly, did not materially affect the results.

We direct the reader to a more extensive discussion of these matters (Moody  et al.2012).In that paper we replicate the NRC tables using the same data set that Lott sent to the NRC and over a hundred researchers around the world, including one of the authors (Moody), who still has the original data set on a server. We also use the correct model specification to replicate the NRC tables using the data set that ADZ collected. We find that the results are essentially the same as the estimates based on the Lott data set.