My dismissal of the methodology of Berkeley economist David Romer's paper claiming that NFL coaches are too risk-averse -- lacking sufficient sample size of teams going for the first down on fourth down, Romer assumed that they'd be as successful on fourth down as they are on third down -- has elicited this reply from a reader:
Did you even read the whole thing? He explicitly considers the difference between third and forth down, based on exactly the reasons you mention. It's on the bottom of page 20. The idea that he's unaware of these differences is simply false. Your failure to acknowledge this is completely misleading.
I read them and they  appeared to be empty rationalizations, not worth describing and refuting at 1:30  AM. The fact that Romer's model (Figure 5 - last  page) says you should be indifferent between going for it versus punting on  4th and 3 from your own 10 yard line is plenty of evidence that his use  of 3rd downs to estimate what would happen on 4th downs is close to uselessly  wrong, for the reasons I explicitly explained in my posting below -- the defense  tends to play farther back on third down than on fourth down to guard against a  long gain or touchdown.
His third down methodology however, might be quite useful for fourth-and-goal  situations (within 10 yards of a touchdown), or even just in the red zone  (inside your opponent's 20) because defenses won't play all that much different on  third and fourth downs.
He concludes that using third down is probably still meaningful, and he gives reasons why he believes this. Now, he might be wrong, and you might be right that his analysis could be improved. But did you actually go through the work to determine how large such an effect might be and how much it might change the analysis? Or did you just declare him to be an idiot and his findings to be "prima facie stupid" without doing any of the hard work?
Of course I didn't redo Romer's entire analysis for him. I pointed out the ridiculous predictions it makes and explained an obvious reason why it fails to offer plausible advice. My role in this is wholly negative -- pointing out how the it gives bad advice and pointing to a bad methodology. from a bad methodology. And I'm trying to do my part to shame economists into being a little less smug about these slapdash freakonomics analyses that are in fashion.
And do you *really* believe that football coaches already do everything optimally?
No, I said it was my vague  impression that NFL coaches are, just as Romer contends, too risk averse. But I don't  know enough about the NFL to offer anything more than speculation. I suspect  Romer and I are right and that coaches like Bill Walsh and Bill Belichick are  wrong about this, but there is a possibility that Romer and I might, strange as  it sounds, be wrong and Walsh and Belichick be right.
My objection is not to Romer's general prejudice, which I share. I simply find  his methodology close to useless.
However, he could salvage some utility from his study. UPDATE: What Romer should do is take his calculations and say, "All right,  I admit it, I don't know anything about the chance of making a first down on 4th  down. Obviously, my methodology of looking at 3rd down plays is useless for  understanding the chances of making it on 4th down, except probably near the  goal line. However, here are the expected return in all situations of making it  versus not making it. Then, you football coaches should take _your_ estimates of  the percentage chance of making it on different 4th down situations, which are  obviously better than mine, and multiply it by my estimate of the returns of  making it, which are pretty good, and the combined result of our efforts will be  useful information."
Haven't you read Bill James? (James, by the way, is someone who has done a lot of interesting work, but who could really benefit from a little more rigorous statistical training.)
Because James, the baseball  statistics guru now employed by the Boston Red Sox, is an old English major  lacking training in elaborate statistics and econometrics, he prefers quicks and  dirty simple analyses to elaborate modeling. There are obvious shortcomings to  lacking a Ph.D. in economics, but the humility and practical-orientation it  engenders in James have its uses too.
You can see an looming problem with an elaborate model like Romer's  --  you design a complicated methodology in your head, then you collect a lot of  data, then you work on it and, after infinite labors, it tells you to go for it  on fourth and two at your own 10 yard line. If you'd done a Bill James-style  quickie study and gotten that results, you'd say, uh-oh, that's wrong, obviously there's something wrong  with my model, what is it? And you'd wouldn't be so invested in this study, so  you'd be willing to walk away from it if you couldn't make it work. 
But if you've built a huge  model and it gives an irrational result due to a fatal, unfixable  flaw in your analytical approach, you'd be tempted, like Romer, to go ahead and  issue it anyway.
It's the same thing as Levitt and Donohue's abortion-cut-crime theory. They put  all this effort into the state-level analysis without putting much effort at all  into the simpler national level analysis. So when I pointed out to them in in  1999 that murder instead went up sharply among those born right after  legalization, their egos and reputations were too invested in their flawed study  to walk away from it. Instead, they ended up getting  humiliated in 2005 by Foote and Goetz showing that their state-level  analysis was based on two fatal flaws. Of course, by then Levitt was a rich  celebrity, so he cried all the way to the bank.
Seriously, speaking as someone who thinks you have a lot of good ideas, you could stand to show just a little tiny bit of humility now and then. Even the great Steve Sailer doesn't know everything. You'd probably find more support for your ideas if you weren't so quick to declare everyone else an idiot.
I don't have the ability to build vast econometric models, but I do have a certain talent for proposing reality checks on economists' popular theories, particularly when they wander out of their traditional domains. This is not likely to make me popular, but I suspect, or at least hope, that in the long run, my irritating knack serves the common good. It may even help economists by making them try harder to find the weaknesses in their ideas before they attempt to turn them into conventional wisdom.
UPDATE: If Romer really wants to continue to argue that teams should go for it on 4th and 2 on their own ten yard line, there is a source of data available. There are inner city high school football teams that seldom punt or try field goals because they can't persuade players to specialize in kicking. Romer could study these teams games. My impression is that lacking a punter is a huge problem, but I could be wrong.
My published articles are archived at iSteve.com -- Steve Sailer
 
 
 
 Posts
Posts
 
 
 
 
 
