My dismissal of the methodology of Berkeley economist David Romer's paper claiming that NFL coaches are too risk-averse -- lacking sufficient sample size of teams going for the first down on fourth down, Romer assumed that they'd be as successful on fourth down as they are on third down -- has elicited this reply from a reader:
Did you even read the whole thing? He explicitly considers the difference between third and forth down, based on exactly the reasons you mention. It's on the bottom of page 20. The idea that he's unaware of these differences is simply false. Your failure to acknowledge this is completely misleading.
I read them and they appeared to be empty rationalizations, not worth describing and refuting at 1:30 AM. The fact that Romer's model (Figure 5 - last page) says you should be indifferent between going for it versus punting on 4th and 3 from your own 10 yard line is plenty of evidence that his use of 3rd downs to estimate what would happen on 4th downs is close to uselessly wrong, for the reasons I explicitly explained in my posting below -- the defense tends to play farther back on third down than on fourth down to guard against a long gain or touchdown.
His third down methodology however, might be quite useful for fourth-and-goal situations (within 10 yards of a touchdown), or even just in the red zone (inside your opponent's 20) because defenses won't play all that much different on third and fourth downs.
He concludes that using third down is probably still meaningful, and he gives reasons why he believes this. Now, he might be wrong, and you might be right that his analysis could be improved. But did you actually go through the work to determine how large such an effect might be and how much it might change the analysis? Or did you just declare him to be an idiot and his findings to be "prima facie stupid" without doing any of the hard work?
Of course I didn't redo Romer's entire analysis for him. I pointed out the ridiculous predictions it makes and explained an obvious reason why it fails to offer plausible advice. My role in this is wholly negative -- pointing out how the it gives bad advice and pointing to a bad methodology. from a bad methodology. And I'm trying to do my part to shame economists into being a little less smug about these slapdash freakonomics analyses that are in fashion.
And do you *really* believe that football coaches already do everything optimally?
No, I said it was my vague impression that NFL coaches are, just as Romer contends, too risk averse. But I don't know enough about the NFL to offer anything more than speculation. I suspect Romer and I are right and that coaches like Bill Walsh and Bill Belichick are wrong about this, but there is a possibility that Romer and I might, strange as it sounds, be wrong and Walsh and Belichick be right.
My objection is not to Romer's general prejudice, which I share. I simply find his methodology close to useless.
However, he could salvage some utility from his study. UPDATE: What Romer should do is take his calculations and say, "All right, I admit it, I don't know anything about the chance of making a first down on 4th down. Obviously, my methodology of looking at 3rd down plays is useless for understanding the chances of making it on 4th down, except probably near the goal line. However, here are the expected return in all situations of making it versus not making it. Then, you football coaches should take _your_ estimates of the percentage chance of making it on different 4th down situations, which are obviously better than mine, and multiply it by my estimate of the returns of making it, which are pretty good, and the combined result of our efforts will be useful information."
Haven't you read Bill James? (James, by the way, is someone who has done a lot of interesting work, but who could really benefit from a little more rigorous statistical training.)
Because James, the baseball statistics guru now employed by the Boston Red Sox, is an old English major lacking training in elaborate statistics and econometrics, he prefers quicks and dirty simple analyses to elaborate modeling. There are obvious shortcomings to lacking a Ph.D. in economics, but the humility and practical-orientation it engenders in James have its uses too.
You can see an looming problem with an elaborate model like Romer's -- you design a complicated methodology in your head, then you collect a lot of data, then you work on it and, after infinite labors, it tells you to go for it on fourth and two at your own 10 yard line. If you'd done a Bill James-style quickie study and gotten that results, you'd say, uh-oh, that's wrong, obviously there's something wrong with my model, what is it? And you'd wouldn't be so invested in this study, so you'd be willing to walk away from it if you couldn't make it work.
But if you've built a huge model and it gives an irrational result due to a fatal, unfixable flaw in your analytical approach, you'd be tempted, like Romer, to go ahead and issue it anyway.
It's the same thing as Levitt and Donohue's abortion-cut-crime theory. They put all this effort into the state-level analysis without putting much effort at all into the simpler national level analysis. So when I pointed out to them in in 1999 that murder instead went up sharply among those born right after legalization, their egos and reputations were too invested in their flawed study to walk away from it. Instead, they ended up getting humiliated in 2005 by Foote and Goetz showing that their state-level analysis was based on two fatal flaws. Of course, by then Levitt was a rich celebrity, so he cried all the way to the bank.
Seriously, speaking as someone who thinks you have a lot of good ideas, you could stand to show just a little tiny bit of humility now and then. Even the great Steve Sailer doesn't know everything. You'd probably find more support for your ideas if you weren't so quick to declare everyone else an idiot.
I don't have the ability to build vast econometric models, but I do have a certain talent for proposing reality checks on economists' popular theories, particularly when they wander out of their traditional domains. This is not likely to make me popular, but I suspect, or at least hope, that in the long run, my irritating knack serves the common good. It may even help economists by making them try harder to find the weaknesses in their ideas before they attempt to turn them into conventional wisdom.
UPDATE: If Romer really wants to continue to argue that teams should go for it on 4th and 2 on their own ten yard line, there is a source of data available. There are inner city high school football teams that seldom punt or try field goals because they can't persuade players to specialize in kicking. Romer could study these teams games. My impression is that lacking a punter is a huge problem, but I could be wrong.
My published articles are archived at iSteve.com -- Steve Sailer
No comments:
Post a Comment
Comments are moderated, at whim.