The Economist on the Freakonomics Fiasco

The Economist on the Freakonomics Fiasco:

Oops-onomics
Dec 1st 2005
From The Economist print edition
Did Steven Levitt, author of “Freakonomics”, get his most notorious paper wrong?

But a paper published last week† by Christopher Foote and Christopher Goetz, two economists at the Federal Reserve Bank of Boston, finds an embarrassing hole in the evidence. Messrs Donohue and Levitt subjected the data to a battery of tests, some suggestive, others more systematic, in an effort to prove the links in the chain. The challenge is to distinguish the role of abortion from other potential influences on crime, many of which cannot be observed directly. Some of these rival factors vary year by year; others state by state. Messrs Foote and Goetz concentrate their fire on those that do both. They offer the crack epidemic, which rose and receded at different times in different places, as an example.

Messrs Donohue and Levitt claim to control for such effects in the final test of their paper. That exercise is meant to facilitate comparisons such as: did arrests of 20-year-olds in New York in 1992 diverge from those of 18-year olds in the same state and year? This automatically takes account of anything going on in the Empire state that year (such as a crack epidemic) that would have affected 18-year-olds and 20-year-olds alike. The principal difference between the two age groups is that one was born after the Supreme Court legalised abortion and the other before.

It was a good test to attempt. But Messrs Foote and Goetz have inspected the authors' computer code and found the controls missing. In other words, Messrs Donohue and Levitt did not run the test they thought they had—an “inadvertent but serious computer programming error”, according to Messrs Foote and Goetz.

Fixing that error reduces the effect of abortion on arrests by about half, using the original data, and two-thirds using updated numbers. But there is more. In their flawed test, Messrs Donohue and Levitt seek to explain arrest totals (eg, the 465 Alabamans of 18 years of age arrested for violent crime in 1989), not arrest rates per head (ie, 6.6 arrests per 100,000). This is unsatisfactory, because a smaller cohort will obviously commit fewer crimes in total. Messrs Foote and Goetz, by contrast, look at arrest rates, using passable population estimates based on data from the Census Bureau, and discover that the impact of abortion on arrest rates disappears entirely. “I am simply not convinced that there is a link between abortion and crime,” Mr Foote says.

It may be asking too much of the numbers to convince everybody. “The debate over abortion and crime will not be resolved within the parameters of our paper,” says Mr Donohue. He thinks the arrest figures are “muddy” and the state population data “sloppy”. Combining the two generates so much noise, it is hard for the statistical tests to hear anything. Ted Joyce, a professor at Baruch College (part of the City University of New York), who has had his own methodological disagreements with Messrs Donohue and Levitt, also thinks the debate is stretching the data too far. He points out that if you add controls for 50 states and 12 years—as Messrs Foote and Goetz do, and as Messrs Donohue and Levitt meant to do—you are, in effect, holding another 600 things constant. This robs the data of most of their variety, and of much of their ability to explain anything.

To say, as Mr Levitt does in “Freakonomics”, that “abortion was one of the greatest crime-lowering factors in American history” may be a bit strong. But the underlying thesis, however unpalatable to some, is not likely to be dispelled by a stroke of Mr Foote's computer key. Mr Levitt says his case is based on a “collage of evidence”, of which the flawed test is one small piece.

No, the flawed state-level econometric data was always Levitt's ace in the hole, the evidence that non-professionals didn't have the skills or resources to subject to critical analysis, since his "collage" of simpler evidence has been debunked repeatedly. But when he finally laid his cards on the table, it turned out he had been bluffing (presumably, by mistake).

He is, in particular, sceptical that crack undermines his thesis: it varied more by age group than by state, he says, hitting 17-year-olds in all states harder than 25-year-olds in any state. He is instead trying to improve his measures of abortion, to take account of the fact that people born under one state's abortion regime might later move elsewhere to commit their crimes.

So, the bottom line is what I've been saying for six years: nobody knows. You can make about as good a case that legalization drove up the crime rate as that it drove it down, but neither case is very persuasive.

The most surprising thing I've learned from this about legalized abortion is, despite the enormous political tumult over it, just how pointless it mostly proved. Legalized abortion turned out to be reminiscent of Homer Simpson's toast: "To alcohol! The cause of, and solution for, all of life's problems."

Legal abortion is a major cause of what it was supposed to solve -- unwanted pregnancies. Levitt himself notes that following Roe, "Conceptions rose by nearly 30 percent, but births actually fell by 6 percent …" So for every six fetuses aborted in the 1970s, five would never have been conceived except for Roe!

My published articles are archived at iSteve.com -- Steve Sailer