Let's continue kibbitzing in the argument between Steven Pinker and Malcolm Gladwell over Gladwell's contention that "In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros."
The mass of evidence suggests that, yes, there is a correlation between where a quarterback is selected in the draft and how well he'll do. Let's note, however, that the correlation glass is half full. For example, Peyton Manning, winner of tonight's 35-34 come-from-behind win over Tom Brady's New England Patriots, was chosen first overall in the 1998 NFL draft. On the other hand, Brady, whose 4th and 2 pass on his own 28 with two minutes left, was juggled by the receiver, costing New England the win, was chosen 199th in the 2000 NFL draft.
I've now read the most recent paper by Gladwell's favorites, economists David J. Berri and Rob Simmons, "Catching a Draft:"
But, Berri is using a very, very slippery approach.
First, he likes to compare quarterbacks picked in the top 10 draft picks in a year to those picked 11 to 50 or to 11 to 100. (And, he ignores the many picked below # 100, where the accuracy of the draft becomes even more apparent.) But because the teams pick in inverse order of how well they did the previous season, those top ten draft picks are going to, on average, bad teams: the worst 10 teams in the league (leaving out trades of draft choices). In contrast, picks 11 to 50 or 11 to 100 will go, on average, to better teams. All else being equal, it’s easier to be successful on a good team than a bad team, if they let you play.
And here’s Berri's other major trick: he wants to measure success on a per play basis, rather than some more useful cumulative measure, such as Pro Bowl selections.
There are obvious problems with measuring success on a per play basis, such as if you’re no good, the coaches don’t let you get many plays. Here are all the quarterbacks drafted since 1980 with their career statistics.
They’re arranged per draft order for each year. You’ll notice that a high proportion of high draft choices played a lot. Some of the low draft choices played a lot, but a lot of them barely played at all in the NFL: the team didn’t invest much in them, and when they proved in practice, unsurprisingly, to be less than NFL starting quality, they went to the bench or into insurance sales.
So, there’s a huge selection bias built into Berri’s measure of success. If you turn out in training camp to be better than the NFL draft consensus (e.g., Tom Brady), they let you play. But if you are a low draft pick and you don't prove to be better than the NFL thought you were, they don't let you play.
For example, the year Brady was picked 199th, Tee Martin was picked 163rd. In Tee's career, he completed 6 passes in 16 attempts for 69 yards, 0 touchdowns, and 1 interception. In other words, Tee Martin proved to be exactly as mediocre as you would expect a 5th round draft choice to be.
So, Brady's statistics would weight much more heavily on a per play basis than Tee Martin's.
On the other hand, if they think you are such hot stuff that they'll burn a high draft choice and millions of dollars on you because they really need a new quarterback right now, well, then they make you play a fair amount at a young age, even if you aren't ready for the NFL, and even if you aren't as good as they thought you were.
Moreover, when those lower drafted quarterbacks did play, they played typically under conditions more fruitful for success per play. Typically, they weren’t thrown in as 22-year-old rookie starters on lousy teams. In their younger years, they probably played against second-string defenses in the last minutes of blowouts. Or the starter went down on a good team, and they stepped into the driver’s seat of a high-powered machine (like Matt Cassell taking over for Tom Brady last year, whose having a harder time this year in St. Louis where he can't just throw the ball in the general direction of Randy Moss.)
The mass of evidence suggests that, yes, there is a correlation between where a quarterback is selected in the draft and how well he'll do. Let's note, however, that the correlation glass is half full. For example, Peyton Manning, winner of tonight's 35-34 come-from-behind win over Tom Brady's New England Patriots, was chosen first overall in the 1998 NFL draft. On the other hand, Brady, whose 4th and 2 pass on his own 28 with two minutes left, was juggled by the receiver, costing New England the win, was chosen 199th in the 2000 NFL draft.
I've now read the most recent paper by Gladwell's favorites, economists David J. Berri and Rob Simmons, "Catching a Draft:"
Our analysis revealed that there was a relationship between aggregate performance and where a player was chosen. But when we looked at per play performance, the relationship between production and draft position was quite weak. In contrast, a much stronger relationship existed between how many plays a quarterback ran and where he was selected. In sum, draft position can get a quarterback on the field. But quarterbacks taken higher do not appear to perform any better.
But, Berri is using a very, very slippery approach.
First, he likes to compare quarterbacks picked in the top 10 draft picks in a year to those picked 11 to 50 or to 11 to 100. (And, he ignores the many picked below # 100, where the accuracy of the draft becomes even more apparent.) But because the teams pick in inverse order of how well they did the previous season, those top ten draft picks are going to, on average, bad teams: the worst 10 teams in the league (leaving out trades of draft choices). In contrast, picks 11 to 50 or 11 to 100 will go, on average, to better teams. All else being equal, it’s easier to be successful on a good team than a bad team, if they let you play.
And here’s Berri's other major trick: he wants to measure success on a per play basis, rather than some more useful cumulative measure, such as Pro Bowl selections.
There are obvious problems with measuring success on a per play basis, such as if you’re no good, the coaches don’t let you get many plays. Here are all the quarterbacks drafted since 1980 with their career statistics.
They’re arranged per draft order for each year. You’ll notice that a high proportion of high draft choices played a lot. Some of the low draft choices played a lot, but a lot of them barely played at all in the NFL: the team didn’t invest much in them, and when they proved in practice, unsurprisingly, to be less than NFL starting quality, they went to the bench or into insurance sales.
So, there’s a huge selection bias built into Berri’s measure of success. If you turn out in training camp to be better than the NFL draft consensus (e.g., Tom Brady), they let you play. But if you are a low draft pick and you don't prove to be better than the NFL thought you were, they don't let you play.
For example, the year Brady was picked 199th, Tee Martin was picked 163rd. In Tee's career, he completed 6 passes in 16 attempts for 69 yards, 0 touchdowns, and 1 interception. In other words, Tee Martin proved to be exactly as mediocre as you would expect a 5th round draft choice to be.
So, Brady's statistics would weight much more heavily on a per play basis than Tee Martin's.
On the other hand, if they think you are such hot stuff that they'll burn a high draft choice and millions of dollars on you because they really need a new quarterback right now, well, then they make you play a fair amount at a young age, even if you aren't ready for the NFL, and even if you aren't as good as they thought you were.
Moreover, when those lower drafted quarterbacks did play, they played typically under conditions more fruitful for success per play. Typically, they weren’t thrown in as 22-year-old rookie starters on lousy teams. In their younger years, they probably played against second-string defenses in the last minutes of blowouts. Or the starter went down on a good team, and they stepped into the driver’s seat of a high-powered machine (like Matt Cassell taking over for Tom Brady last year, whose having a harder time this year in St. Louis where he can't just throw the ball in the general direction of Randy Moss.)
My published articles are archived at iSteve.com -- Steve Sailer
Posts
11 comments:
Awesome game! I can't believe Belicheat actually went for it!!!
Yeah, I completely agree with you that Berri & Simmons's analysis is incredibly flawed. But it's really Gladwell who is to be blamed, for stating it so simplisticly. It's like if I said "among NBA canters who averaged 30+ minutes, height is only weakly correlated with scoring" and then he said "Anonymous found that height is not important for basketball players".
I feel the need to echo Scrutineer's quote of Bertrand Russell from the earlier thread:
"A stupid man's report of what a clever man says is never accurate because he unconsciously translates what he hears into something he can understand."
Berri's analysis is ridiculous -- the selection effect Steve points out completely ruins it. Say there were 100 quarterbacks drafted low in the draft, and 99 sucked and didn't play in the NFL, but one was really good and had a 20 year career. Then the good one's numbers would dominate the per play statistics and QBs picked late in the draft would look like great players.
Berri's NBA stats methodology is also pretty dubious.
Steve,
A minor correction. Matt Cassel is not in St. Louis. He is in Kansas City.
I missed it!!!!!!!
But related to the post, Tom Brady doesn't belong in the same sentence as Peyton Manning.
A guy who hadn't played a game of football in NINE years replaced Brady and threw for almost 25 TD's and was a borderline Pro Bowler. That about says it all.
Brady got by on Belicheck's coaching and a turnover producing defense. He only produced in 2007 with Moss and Welker (who is starting to look like a HOFer) and Stallworth and Ben Watson and Belicheck throwing about 60% of the time.
Gladwell isn't even in Sailer's league. Gladwell's success comes more from marketing himself as the crazy hair (he had a normal haircut once) "diverse" guy who speaks "troof to power" to college kids. Plus, he can write anything and the NY Times will praise it.
But if I had a good shot at draft pick #5-10, if I played real hard and risked more injury? I'd sure take it easy enough to be #11-15.
Is there any evidence of this happening?
A critical factor for a QB is his offensive line. Good offensive lines keep their man upright and give the receivers time to get open. An average QB can look pretty good with a good offensive line and pretty bad otherwise; e.g., Jim Plunkett was awful with the poor NE Patriot O-line in front of him and pretty good with the Raiders much better line.
well, the perennially bad teams in the modern NFL who get to pick high in the draft year after year, can certainly screw up the careers of potentially decent quarterbacks.
getting picked in the top 5 overall by teams like the lions is basically the end of your career before it even starts. it is exactly like being taken high by the clippers in the NBA draft. you should do everything in your power to avoid it, even willing to become a hated player by refusing to play for the team that drafted you.
matt stafford is totally wasted playing for the lions. he could do significantly better playing for a .500 team. instead, he's going to get pummelled, and look a lot worse than he is. even jamarcus russell, who has no business being an NFL starter, on any team, would do better on a .500 team than on the raiders, even though the raiders have much better players, man for man, than any other bad team.
Tom Brady clearly outplayed Manning in this matchup and a questionable 4th down spot and terrible call and strategy by Bellicheck handed the game to the Colts.
I would take Brady over Manning for many reasons, but this year and Brett Favre is the MVP. No one in the history of the NFL has had a higher touchdown to interception ratio in a season than Favre currently has.
Post a Comment