Gladwell strikes back

From Gladwell.com:

Steven Pinker reviewed my new book "What the Dog Saw," in the New York Times Book Review this past Sunday. I sent the following letter to the editor in response:

It is always a pleasure to be reviewed by someone as accomplished as Stephen [sic] Pinker, even if—in his comments on “What the Dog Saw” [which you can buy here] (Nov. 15)—he is unhappy with my spelling (rightly!) and with the fact that I have not joined him on the lonely ice floe of IQ fundamentalism. But since football has been on my mind these days, I do want to make one small observation about his comments.

I would suggest that the reason Gladwell is choosing to make a big deal over Pinker calling BS on Gladwell's assertion that performance as an NFL quarterback "can't be predicted" is because Malcolm senses that this minor issue is characteristic of his entire career as the foremost conduit to the public of wrong ideas.

He goes on:

In one of my essays, I wrote that the position a quarterback is taken in the college draft is not a reliable indicator of his performance as a professional.

"Not a reliable indicator" does not exactly get across what Malcolm actually wrote. Let's keep in mind that Malcolm's assertion in The New Yorker is quite uncompromising: there is no correlation:

This is the quarterback problem. There are certain jobs where almost nothing you can learn about candidates before they start predicts how they'll do once they're hired. ... The problem with picking quarterbacks is that [U. of Missouri quarterback] Chase Daniel's performance can't be predicted. The job he's being groomed for is so particular and specialized that there is no way to know who will succeed at it and who won't. In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros.

Pinker thinks of the term "can't be predicted" in the standard statistical sense of predictions not being better than random, that NFL teams are so bad at drafting quarterbacks that they might as well throw darts. Unsurprisingly, that's not true.

Gladwell is using it in the sense of, well, who knows?

Perhaps Gladwell is using "can't be predicted" to mean "can't always be predicted" -- as in, "How about that Ryan Leaf pick? Whattabout Tim Couch?" But everybody already knows that when it comes to drafting quarterbacks the glass is part empty as well as part full. So, if Malcolm comes out and tells the truth (NFL general managers are a lot better than random at drafting quarterbacks, but also lot worse than perfection), then he doesn't have much of a hook for his article.

But instead of Malcolm trying to laugh it off as him just being breezy and trying to hype his little magazine article, he instead gets all sanctimonious and tries to bring the hammer of academic authority down upon the head of Steven Pinker, Harvard College Professor and Johnstone Family Professor in the Department of Psychology of Harvard U.

See, what makes Malcolm so successful as a speaker at sales conferences is that he believes his own hype. Many people can smell insincerity, but Malcolm is sincere. He believes whatever he's peddling, no matter how obviously wrong it is.

Malcolm goes on in his letter to the New York Times:

That was based on the work of the academic economists David Berri and Rob Simmons, who, in a paper published the Journal of Productivity Analysis, analyze forty years of National Football League data. Their conclusion was that the relation between aggregate quarterback performance and draft position was weak. Further, when they looked at per-play performance—in other words, when they adjusted for the fact that highly drafted quarterbacks are more likely to play more downs—they found that quarterbacks taken in positions 11 through 90 [what Malcolm means here is the 90 draft positions of 11 through 100] in the draft actually slightly outplay those more highly paid and lauded players taken in the draft’s top ten positions. I found this analysis fascinating. Pinker did not. This quarterback argument, he wrote, “is simply not true.”

I wondered about the basis of Pinker’s conclusion, so I e-mailed him, asking if he could tell me where to find the scientific data that would set me straight. He very graciously wrote me back. He had three sources, he said. The first was Steve Sailer. [You can read my January 29, 2009 posting here.] Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people.

As a commenter below pointed out, Malcolm should be best known for his 1997 New Yorker article: The Sports Taboo: Why blacks are like boys and whites are like girls. (Actually, he should be: it's one of his better articles, back from when he was braver and poorer. In it, he, applies the same logic that got Larry Summers in so much trouble in 2005 to race. Unfortunately, like so many of Malcolm's ideas, it's wrong. )

Sailer’s “proof” of the connection between draft position and performance is, I’m sure Pinker would agree, crude: his key variable is how many times a player has been named to the Pro Bowl.

Why? It's a well-known measure of excellence for a single season. In my data set of 278 quarterbacks drafted during the Eighties and Nineties, there are 113 Pro Bowl selections, so the sample size is reasonably adequate.

The irony, however, is that the correlation between making the Pro Bowl and what draft pick a player was is less strong than the correlations for quite a few other important measures of accomplishment. That's not surprising. That's why I've emphasized Pro Bowls as measure recently -- because they are a more favorable measure for Malcolm's theory than most other plausible measures.

I've looked at the 278 quarterbacks drafted in the 1980s and 1990s, and here are the correlations between draft pick and various career statistics:

Draft and Pro Bowls: r = -0.33
Draft and Touchdown Passes: r = -0.45
Draft and Passing Yards: r = -0.48
Draft and Years Starting: r = -0.48
Draft and Games Played: r = -0.52

(The correlations are negative because, for example, Peyton Manning was picked #1 overall in his year and has, through 2008, 45,628 yards passing, while Randy Essington was picked #336 overall in his year and had 0 yards passing in his NFL career.)

So, the correlation between draft picks and Pro Bowls that Malcolm objected to turns out to be weaker than many other correlations, but it's still noticeable in real life.

(Are these correlations high or low? They're pretty normal for what you see in the social sciences. There is an old rule of thumb that correlations with an absolute value of 0.2 are low, 0.4 medium and 0.6 high.)

Pinker’s second source was a blog post [by Josh Millet, which you can read for yourself here], based on four years of data, written by someone who runs a pre-employment testing company, who also failed to appreciate—as far as I can tell (the key part of the blog post is only a paragraph long)—the distinction between aggregate and per-play performance. Pinker’s third source was an article in the Columbia Journalism Review [by Daniel Luzer, which you can read for yourself here], prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation” which in turn makes reference to a “study” of quarterbacks conducted by a fantasy football website. I have enormous respect for Professor Pinker, and his description of me as “minor genius” made even my mother blush. But maybe on the question of subjects like quarterbacks, we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google.

What Berri is doing, in effect, by using his "per-play" measure is comparing quarterbacks taken at the top of the draft (most of whom get a lot of plays in the NFL) to those taken lower in the draft who turned out to be surprisingly better than expected, and thus get a lot of plays. He's essentially leaving out of his analysis all those lower drafted quarterbacks who turned out to be as mediocre as expected and thus didn't get many plays. In other words, his methodology is pre-rigged to produce the conclusion that Malcolm likes.

Through 2008, among quarterbacks drafted from 1980-1999, top ten draftees averaged 2,975 pass attempts in their careers. Quarterbacks drafted 11th to 100th averaged 1,470 attempts, a little less than half as much. And quarterbacks drafted 101st or higher averaged only 387 attempts.

So, Berri is more or less throwing away the lousier half of the sample of quarterbacks drafted 11th-100th (and totally ignoring all the quarterbacks drafted after 100) and comparing them to all the quarterbacks drafted in the top ten.

When you actually count everybody drafted, you get the following figures for career yardage (through 2008):

Drafted	Mean Yards	Median Yards
Top 10	20,296	18,148
11-100	10,099	3,881
101+	2,614	0

The differences between the mean and the median (50th percentile) point out that the higher drafted players tend to be safer bets. The quarterback at the 50th percentile among the top ten draftees of his year goes on to have a fairly impressive NFL career, throwing for 18,148 yards. (The median top ten quarterback of 1980-1999 in career yardage was Jim McMahon, who led the Chicago Bears to the 1985 Super Bowl title.)

In contrast, the 50th percentile of the 11th to 100th picks of his year only accumulates 21% as much career yardage. The median quarterbacks of the 11-100 group are Mark Herrmann and Chuck Long.

And the 50th percentile of 101st plus picks never completes a pass in the NFL).

So, the top ten quarterbacks drafted in the eighties and nineties tended to be safer bets, which has its value. (General managers in this decade, however, might have gotten overconfident from a pretty decent run of luck with high draft pick quarterbacks in the two previous decades.)

On the other hand, there are lots of diamonds in the semi-rough of the 11-100 group, such as Brett Favre, Dan Marino, and Boomer Esiason. And in the 101+ group, there are diamonds in the real rough like Mark Brunell, Trent Green, and Matt Hasselbeck. (And that's not to mention the undrafteds, like Kurt Warner.)

To expand on what I pointed out in the comments to Gladwell's blog post:

Malcolm, the reason your reputation has plummeted in recent years as your net worth has risen is that you are too trusting of academics. As you blogged on August 29, 2006:

I will confess to having a slightly reverential attitude toward academia. I'm the son of an academic. Much of my writing involves taking academic research and trying to translate it for a more general audience. And I've always believed that if you set out to write about the work of academic specialists, you have a responsibility to treat that work with respect-- to acknowledge your own ignorance and, where appropriate, defer to the greater expertise of others.

You shouldn't be in awe of David J. Berri, Associate Professor of Economics at Southern Utah University in Cedar City. David J. Berri should be in awe of you, the (likely) highest-earning print journalist in America. You should make Professor Berri prove his theories to you by subjecting his ideas to rigorous reality checks. You have to do the work.

But it's not that hard. The Internet is chock full of data. You just copy and paste it into Excel. Get your tax accountant to show you how to use Excel. I'm sure he owes you a favor by now.

My published articles are archived at iSteve.com -- Steve Sailer