Pinker v. Gladwell on NFL quarterbacks

Was Steven Pinker correct when dismissing in the New York Times Malcolm Gladwell's New Yorker article "Most Likely to Succeed" with the words, "It is simply not true that a quarter­back’s rank in the draft is uncorrelated with his success in the pros"?

Gladwell's statement of his position is quite uncompromising:

This is the quarterback problem. There are certain jobs where almost nothing you can learn about candidates before they start predicts how they'll do once they're hired. So how do we know whom to choose in cases like that? ... The problem with picking quarterbacks is that [U. of Missouri quarterback] Chase Daniel's performance can't be predicted. The job he's being groomed for is so particular and specialized that there is no way to know who will succeed at it and who won't. In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros.

"No connection" is not, in fact, the position of economists David J. Berri and Rob Simmons, whose new paper "Catching a Draft" (gated and therefore I haven't read it) has the following abstract:

The reverse order college draft gives the worst teams in the National Football League (NFL) the opportunity to hire the best amateur talent. For it to work effectively, teams must be able to identify the “best” talent. Our study of NFL quarterbacks highlights problems with the draft process. We find only a weak correlation between teams’ evaluations on draft day and subsequent quarterback performance in the NFL. Moreover, many of the factors that enhance a quarterback’s draft position are unrelated to future NFL performance. Our analysis highlights the difficulties in evaluating workers in the uncertain environment of professional sports.

They find what they characterize as "only a weak correlation," which is different from "no connection." Moreover, what is a "weak correlation?" In the selection business, a seemingly "weak correlation" is quite different from no correlation.

With most things in the human sciences, the glass is roughly half empty and half full at the same time. For example, in the 1998 NFL draft, San Diego used a #2 pick in the first round to choose Ryan Leaf, a notorious bust. However, immediately before that legendary bad decision, Indianapolis had used the first pick in the draft to acquire Peyton Manning, who, as I write, is still gainfully employed in the Colt organization. So, looking at the single most famous pair of quarterback drafts in history, you come up with the usual glass half empty / half full situation.

Further, coming up with one Peyton Manning and one Ryan Lean with your top two picks is a lot better than picking at random among the 100+ college quarterbacks who were eligible for the draft that year.

When I analyzed Gladwell's thesis just before the last Super Bowl, using all 278 quarterbacks drafted in the 1980s and 1990s, I came up with the following table, looking at Pro Bowl honors as a stringent test of success in the NFL. (This partly gets around the problem that high draft picks are often given more years to fail than low draft picks.)

Draft Rank	Count	Average Pro Bowls
Top 50	54	1.50
51-100	43	0.28
101-200	71	0.17
201+	110	0.07

In other words, the 54 quarterbacks drafted among the top 50 players of their year averaged 1.50 Pro Bowl honors each, versus 0.07 Pro Bowls for the 110 quarterbacks drafted 201st or later. Per capita, the high draft picks were selected for the Pro Bowl more than 20 times as often as the low draft picks.

What about more recent experience?

Here’s a 2008 blog post byJosh Millet looking at the 2000-2004 quarterback drafts:

To take the most recent decade as an example, when one looks at all the quarterbacks (67 in all) who were drafted by NFL teams from 2000 to 2004, and compares their overall draft position to their statistics in their first four years in the league, it is clear that on balance NFL teams are very accurate in predicting statistical success in the NFL. Organizational psychologists measure the predictive validity of an employee selection technique by quantifying the strength of the relationship between selection measure and job performance; the strength of the association is expressed as a correlation coefficient. For the whole group, the correlation between draft order and passing yardage is very strong (-.73 — the coefficient is negative because the higher a player is drafted, the lower their draft rank).

For those concerned that a measure of total productivity such as passing yardage is somewhat correlated with opportunity, we can consider passer efficiency, as measured by QB rating. Only 51 of the 67 quarterbacks drafted attempted a pass in the NFL, a necessary requirement for calculating a QB rating: for this group there was a -.34 correlation between draft position and QB rating. This is still a strong association, and shows a clear, statistically significant correlation between draft order and future statistical success in the NFL.

Like the fans of the teams that drafted them, Gladwell has let the Ryan Leafs (a high draft choice that flopped) and the Tom Bradys (a low draft choice who became a superstar) of the world influence his thinking. These are outliers, a concept with which Gladwell should be familiar given the title of his latest book. (If you take Brady out of the mix the correlations strengthen considerably!) It turns out, in fact, that on average the NFL draft process is highly accurate at predicting QB success, and the draft is based entirely on things that Gladwell dismisses as useless--college performance, scouting, performance in the NFL combine.

If Gladwell had considered any quantitative measures at all relating to the efficacy of the draft he'd have no basis for his conclusion that "a prediction, in a field where prediction is not possible, is nothing but a prejudice." Gladwell, we fear, gets swept up in his own story telling, and in the process badly misconstrues the alleged "quarterback problem."

A commenter on that blog points out:

The best college QBs are typically assigned to the worst teams because of the rules of the NFL draft, which most likely hinders their chance for pro success, and weakens the association between draft position and pro success, at least for first- round draft picks.

This would tend to artificially strengthen the correlation (-0.73) between draft position and early career passing yardage and artificially weaken the correlation (-0.34) between draft position and passer rating, so the underlying “true” correlation is somewhere in between.

For example, in 1998 the Indianapolis Colts had “earned” the #1 draft pick by going 3-13 in 1997. For the 1998 season, they immediately plugged Manning in at starting quarterback at age 22. Having nothing else going for the team, they had him throw a league-leading 575 passes for 3,739 yards, but also a league-worst 28 interceptions. That gave him a passer efficiency rating of only 71.2, far below his career average of 95.3.

If Manning had been drafted by a better team, he probably would have only played as a rookie at the end of blow-outs against the opposition’s second string, and wouldn’t have inherited the starting job until he was mature. (The NFL career of Steve Young, who has the higher career passer rating, demonstrates this – one bad year at a young age in Tampa Bay, then a long sojourn as Joe Montana’s backup learning the job in the brilliant San Francisco organization before finally emerging as the highest rated QB in history) So, Peyton’s early career yardage would likely have been less if drafted lower, but his early career passer rating would have been higher. Thus, the “true” correlation between draft rank and NFL success is probably between -.73 and -.34.

Another thing to keep in mind is the semi-random role of injuries in reducing correlations. A lot of quarterbacks whose NFL careers are disappointments are simply too banged up to play up to their potential. In baseball, it’s widely accepted that a pitcher’s career is contingent on his arm staying healthy, but in football, there’s a certain amount of moralizing about how if the quarterback was tough enough, like Brett Favre, he would just shake it off and play through the pain.

High draft choices who are likely to be thrown in as starters before they are physically and mentally mature are more likely to get badly hurt early in their careers than a lower draft choice who doesn’t get the starting job until he knows what he’s doing and has a little extra muscle on him, and is mentally ready to have big years. So, that also lowers the correlations a little.

My published articles are archived at iSteve.com -- Steve Sailer