Was Steven Pinker correct when dismissing in the New York Times Malcolm Gladwell's New Yorker article "Most Likely to Succeed" with the words, "It is simply not true that a quarterback’s rank in the draft is uncorrelated with his success in the pros"?
Gladwell's statement of his position is quite uncompromising:
"No connection" is not, in fact, the position of economists David J. Berri and Rob Simmons, whose new paper "Catching a Draft" (gated and therefore I haven't read it) has the following abstract:
They find what they characterize as "only a weak correlation," which is different from "no connection." Moreover, what is a "weak correlation?" In the selection business, a seemingly "weak correlation" is quite different from no correlation.
With most things in the human sciences, the glass is roughly half empty and half full at the same time. For example, in the 1998 NFL draft, San Diego used a #2 pick in the first round to choose Ryan Leaf, a notorious bust. However, immediately before that legendary bad decision, Indianapolis had used the first pick in the draft to acquire Peyton Manning, who, as I write, is still gainfully employed in the Colt organization. So, looking at the single most famous pair of quarterback drafts in history, you come up with the usual glass half empty / half full situation.
Further, coming up with one Peyton Manning and one Ryan Lean with your top two picks is a lot better than picking at random among the 100+ college quarterbacks who were eligible for the draft that year.
When I analyzed Gladwell's thesis just before the last Super Bowl, using all 278 quarterbacks drafted in the 1980s and 1990s, I came up with the following table, looking at Pro Bowl honors as a stringent test of success in the NFL. (This partly gets around the problem that high draft picks are often given more years to fail than low draft picks.)
In other words, the 54 quarterbacks drafted among the top 50 players of their year averaged 1.50 Pro Bowl honors each, versus 0.07 Pro Bowls for the 110 quarterbacks drafted 201st or later. Per capita, the high draft picks were selected for the Pro Bowl more than 20 times as often as the low draft picks.
What about more recent experience?
Here’s a 2008 blog post byJosh Millet looking at the 2000-2004 quarterback drafts:
This would tend to artificially strengthen the correlation (-0.73) between draft position and early career passing yardage and artificially weaken the correlation (-0.34) between draft position and passer rating, so the underlying “true” correlation is somewhere in between.
For example, in 1998 the Indianapolis Colts had “earned” the #1 draft pick by going 3-13 in 1997. For the 1998 season, they immediately plugged Manning in at starting quarterback at age 22. Having nothing else going for the team, they had him throw a league-leading 575 passes for 3,739 yards, but also a league-worst 28 interceptions. That gave him a passer efficiency rating of only 71.2, far below his career average of 95.3.
If Manning had been drafted by a better team, he probably would have only played as a rookie at the end of blow-outs against the opposition’s second string, and wouldn’t have inherited the starting job until he was mature. (The NFL career of Steve Young, who has the higher career passer rating, demonstrates this – one bad year at a young age in Tampa Bay, then a long sojourn as Joe Montana’s backup learning the job in the brilliant San Francisco organization before finally emerging as the highest rated QB in history) So, Peyton’s early career yardage would likely have been less if drafted lower, but his early career passer rating would have been higher. Thus, the “true” correlation between draft rank and NFL success is probably between -.73 and -.34.
Another thing to keep in mind is the semi-random role of injuries in reducing correlations. A lot of quarterbacks whose NFL careers are disappointments are simply too banged up to play up to their potential. In baseball, it’s widely accepted that a pitcher’s career is contingent on his arm staying healthy, but in football, there’s a certain amount of moralizing about how if the quarterback was tough enough, like Brett Favre, he would just shake it off and play through the pain.
High draft choices who are likely to be thrown in as starters before they are physically and mentally mature are more likely to get badly hurt early in their careers than a lower draft choice who doesn’t get the starting job until he knows what he’s doing and has a little extra muscle on him, and is mentally ready to have big years. So, that also lowers the correlations a little.
Gladwell's statement of his position is quite uncompromising:
This is the quarterback problem. There are certain jobs where almost nothing you can learn about candidates before they start predicts how they'll do once they're hired. So how do we know whom to choose in cases like that? ... The problem with picking quarterbacks is that [U. of Missouri quarterback] Chase Daniel's performance can't be predicted. The job he's being groomed for is so particular and specialized that there is no way to know who will succeed at it and who won't. In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros.
"No connection" is not, in fact, the position of economists David J. Berri and Rob Simmons, whose new paper "Catching a Draft" (gated and therefore I haven't read it) has the following abstract:
The reverse order college draft gives the worst teams in the National Football League (NFL) the opportunity to hire the best amateur talent. For it to work effectively, teams must be able to identify the “best” talent. Our study of NFL quarterbacks highlights problems with the draft process. We find only a weak correlation between teams’ evaluations on draft day and subsequent quarterback performance in the NFL. Moreover, many of the factors that enhance a quarterback’s draft position are unrelated to future NFL performance. Our analysis highlights the difficulties in evaluating workers in the uncertain environment of professional sports.
They find what they characterize as "only a weak correlation," which is different from "no connection." Moreover, what is a "weak correlation?" In the selection business, a seemingly "weak correlation" is quite different from no correlation.
With most things in the human sciences, the glass is roughly half empty and half full at the same time. For example, in the 1998 NFL draft, San Diego used a #2 pick in the first round to choose Ryan Leaf, a notorious bust. However, immediately before that legendary bad decision, Indianapolis had used the first pick in the draft to acquire Peyton Manning, who, as I write, is still gainfully employed in the Colt organization. So, looking at the single most famous pair of quarterback drafts in history, you come up with the usual glass half empty / half full situation.
Further, coming up with one Peyton Manning and one Ryan Lean with your top two picks is a lot better than picking at random among the 100+ college quarterbacks who were eligible for the draft that year.
When I analyzed Gladwell's thesis just before the last Super Bowl, using all 278 quarterbacks drafted in the 1980s and 1990s, I came up with the following table, looking at Pro Bowl honors as a stringent test of success in the NFL. (This partly gets around the problem that high draft picks are often given more years to fail than low draft picks.)
| Draft Rank | Count | Average Pro Bowls |
| Top 50 | 54 | 1.50 |
| 51-100 | 43 | 0.28 |
| 101-200 | 71 | 0.17 |
| 201+ | 110 | 0.07 |
In other words, the 54 quarterbacks drafted among the top 50 players of their year averaged 1.50 Pro Bowl honors each, versus 0.07 Pro Bowls for the 110 quarterbacks drafted 201st or later. Per capita, the high draft picks were selected for the Pro Bowl more than 20 times as often as the low draft picks.
What about more recent experience?
Here’s a 2008 blog post byJosh Millet looking at the 2000-2004 quarterback drafts:
To take the most recent decade as an example, when one looks at all the quarterbacks (67 in all) who were drafted by NFL teams from 2000 to 2004, and compares their overall draft position to their statistics in their first four years in the league, it is clear that on balance NFL teams are very accurate in predicting statistical success in the NFL. Organizational psychologists measure the predictive validity of an employee selection technique by quantifying the strength of the relationship between selection measure and job performance; the strength of the association is expressed as a correlation coefficient. For the whole group, the correlation between draft order and passing yardage is very strong (-.73 — the coefficient is negative because the higher a player is drafted, the lower their draft rank).A commenter on that blog points out:
For those concerned that a measure of total productivity such as passing yardage is somewhat correlated with opportunity, we can consider passer efficiency, as measured by QB rating. Only 51 of the 67 quarterbacks drafted attempted a pass in the NFL, a necessary requirement for calculating a QB rating: for this group there was a -.34 correlation between draft position and QB rating. This is still a strong association, and shows a clear, statistically significant correlation between draft order and future statistical success in the NFL.
Like the fans of the teams that drafted them, Gladwell has let the Ryan Leafs (a high draft choice that flopped) and the Tom Bradys (a low draft choice who became a superstar) of the world influence his thinking. These are outliers, a concept with which Gladwell should be familiar given the title of his latest book. (If you take Brady out of the mix the correlations strengthen considerably!) It turns out, in fact, that on average the NFL draft process is highly accurate at predicting QB success, and the draft is based entirely on things that Gladwell dismisses as useless--college performance, scouting, performance in the NFL combine.
If Gladwell had considered any quantitative measures at all relating to the efficacy of the draft he'd have no basis for his conclusion that "a prediction, in a field where prediction is not possible, is nothing but a prejudice." Gladwell, we fear, gets swept up in his own story telling, and in the process badly misconstrues the alleged "quarterback problem."
The best college QBs are typically assigned to the worst teams because of the rules of the NFL draft, which most likely hinders their chance for pro success, and weakens the association between draft position and pro success, at least for first- round draft picks.
This would tend to artificially strengthen the correlation (-0.73) between draft position and early career passing yardage and artificially weaken the correlation (-0.34) between draft position and passer rating, so the underlying “true” correlation is somewhere in between.
For example, in 1998 the Indianapolis Colts had “earned” the #1 draft pick by going 3-13 in 1997. For the 1998 season, they immediately plugged Manning in at starting quarterback at age 22. Having nothing else going for the team, they had him throw a league-leading 575 passes for 3,739 yards, but also a league-worst 28 interceptions. That gave him a passer efficiency rating of only 71.2, far below his career average of 95.3.
If Manning had been drafted by a better team, he probably would have only played as a rookie at the end of blow-outs against the opposition’s second string, and wouldn’t have inherited the starting job until he was mature. (The NFL career of Steve Young, who has the higher career passer rating, demonstrates this – one bad year at a young age in Tampa Bay, then a long sojourn as Joe Montana’s backup learning the job in the brilliant San Francisco organization before finally emerging as the highest rated QB in history) So, Peyton’s early career yardage would likely have been less if drafted lower, but his early career passer rating would have been higher. Thus, the “true” correlation between draft rank and NFL success is probably between -.73 and -.34.
Another thing to keep in mind is the semi-random role of injuries in reducing correlations. A lot of quarterbacks whose NFL careers are disappointments are simply too banged up to play up to their potential. In baseball, it’s widely accepted that a pitcher’s career is contingent on his arm staying healthy, but in football, there’s a certain amount of moralizing about how if the quarterback was tough enough, like Brett Favre, he would just shake it off and play through the pain.
High draft choices who are likely to be thrown in as starters before they are physically and mentally mature are more likely to get badly hurt early in their careers than a lower draft choice who doesn’t get the starting job until he knows what he’s doing and has a little extra muscle on him, and is mentally ready to have big years. So, that also lowers the correlations a little.
My published articles are archived at iSteve.com -- Steve Sailer
I don't know about quarterbacks, but the liberal media's yardstick for hiring journalists, thinkers, and writers sucks big time. What was the standard used for promoting Gladwell as one of the 'premier' thinkers of our age?
ReplyDeleteSo, maybe Gladwell is projecting what goes on journalism and book publishing onto professional sports. If a jerkoff nitwit like him could succeed as an intellectual--well beyond others with much better credentials--, who's to say a low-ranked college quarterback might not become a hall-of-famer down the line?
Gladwell is a great storyteller but a bad sociologist. For this he is able to earn a much better living than many great sociologists. I'm sure there is a Gladwellism to explain why.
ReplyDeleteOne of the Gladwell's common errors is to assume all other variables are constant. When is that ever the case, especially in sociology?
Fact is a good QB on a great team will have a much better career than a great QB on a bad team. Sports fans argue about this all day but Gladwell just ignores this dilemma and casts his faith on a linear regression chart.
To improve analysis of this particular question one must take into account not just the draft position of the QB but how many quarterbacks were selected before him? It is one thing to say Tom Brady was drafted in the sixth round. It puts this pick in a whole different perspective to see he was the seventh QB drafted. In other words, the pros still thought at the time he was the seventh best quarterback in the world! That is a far cry from simply saying he was just one of many elite players.
Dan:
ReplyDeleteIn other words, the pros still thought at the time he was the seventh best quarterback in the world!
Don't you mean "in the draft pool"?
But to Gladwell, any correlation not equal to +/-1 means NO correlation, since correlation means A causes B EVERY time.
ReplyDeleteMalcolm Gladwell is a guy who knows how to write compellingly readable stories. The takeaway in his book Outliers The Story of Success is quite unRandian — it is that luck plays an enormous factor in out-sized success. That is a factor the Randians prefer to ignore.
ReplyDelete-- from Ritholtz Big Picture blog
Continuing to visit the Big Picture blog I'm becoming convinced that that little dustup awhile back had little or nothing to do with government housing policies and programs, just a convenience for a gut level dislike.
What is it in the water out east there, the NY-DC money-media-power axis that for something to gain traction, broad appeal, that gains a shared affirmation, must have either an element of fraudulence or whole cloth.
Fwiw- I don't take issue with the luck thingy of Gladwell's above. Many or most probably attribute good fortune
to their own doing and blaming something else for bad fortune. I find life's sorting process replete with good and bad luck. It was just Pinker's takedown of that popular author, in no uncertain terms, and telling of his fans.
The really big problem is that the pro defenses are so much more difficult to understand and deal with mentally than the college defenses. Teams get a very good idea of how tall, fast, accurate a quarterback is before the draft, but they cannot get a good test of their ability to deal with professional defensive schemes/personnel.
ReplyDeleteI'd take Payton Manning about a thousand draft picks ahead of Ryan Leaf (you've got a typo there, btw) in, say, the "who does my taxes" draft.
The real problem, as I've said before, is that the ability to read defensive schemes is so much more important in the pros that it makes much of the previous screening systems moot. Yes, a pro QB needs a certain arm strength and height, eliminating maybe 75% of the male population, but the ability to read defenses is so important that it winds up overriding mots physical concerns. The physical freaks who were the obvious best choices for high school and college quarterbacks aren't likely to have the most valuable pro skills, but there's no system to develop players with the exact skills and abilities that pro quarterbacks need.
It's like if minor league baseball only let pitchers who threw 99+ mph fastballs advance beyond rookie league.
Continuing to visit the Big Picture blog I'm becoming convinced that that little dustup awhile back had little or nothing to do with government housing policies and programs, just a convenience for a gut level dislike.
ReplyDeleteWhy is that? I often read visit Mr. Ritholz's blog, and I can't figure out what you mean.
Are you trying to imply that Mr. Ritholz was simply expressing anti-antiSemitism during his brief "drive by" visit here, or what?
Why else would Barry have a gut level dislike for iSteve.com? 'Cuz Steve's better looking and slimmer? I'm sure that Barry R. is a lot richer than Steve, even if B.R. is not as rich as Malcolm Gladwell.
This analysis deserves some meta-analysis.
ReplyDeleteGiven Gladwell's remarkable success, there are many many people sifting through every statement he has made and objecting to their literal meaning. This is just annoyingly foolish use of smart minds. A non-fiction writer in this day and age knows that every fact is a click or two away. Disaffected, or the merely curious, bloggers will dredge these up and throw them out into the web. When Gladwell says "no connection", it has to be interpreted in the sense of providing a fresh reader with a first-order perspective. Do we really think that his statement is going to send all those executives who read his books and pay to listen to him go scurrying out with agate eyes and trash whatever mental rulebook they have been following in hiring QBs? No. The idea is to suggest that a new perspective, a fresh perspective, is definitely worth something.
As you point out, weak correlations can mean anything, especially that the real variables might be hidden (or latent as the machine learning people say). All these armchair regression-wielding literalists pounce on that statement and show that it is wrong, but their analysis itself is imperfect. What if the null hypothesis, the baseline your regression must measure up against, is that a fundamental variable is being left out. The weak correlations do not disprove that; they can in fact imply that the variables being considered are related to the fundamental variable we ignored. And really, that is the ultimate message here: rethink your fundamental assumptions.
There's a lot of gloating from these regression warriors triumphantly waving their R and p values. These are the one-eyed crotch-grabbing grandiosely in front of the blind. What are the assumptions behind the regression? With 67 data points, how many outliers are too many and should make us doubt our variables? I don't see any such discussion.
How about some constructive re-framing of this problem? Imagine you are a generalist non-fiction writer who want to acquaint your readership to a new field of research or a new, perhaps startling, research finding. How would you go about this if you yourself are unfamiliar with this area? You are like the explorer journeying into exotic new lands and you are reporting back to your domestic readers. Do you, in the name of technical precision, meditate and observe in this new territory for long years gaining intimate expertise before you write up your experiences?
No, there are able experts already penning those, or will be following this surge of interest.
You suggest he is low on neuroticism, and high on openness and agreeableness. Isn't that exactly what we project when seeking new experiences?
Gladwell speaks to his audience, and he isn't falsifying, merely simplifying.
I remember the '98 draft debates regarding Leaf vs. Manning. I would note that while they were widely ranked 1 and 2 as quarterback prospects, and in terms of numerical evaluation they were virtually tied, they were never seen as anything like interchangable.
ReplyDeleteJust about everybody agreed that Leaf had the greater potential, the higher athletic "upside," if his already recognized volatility sometimes poor decision-making were brought under control. Leaf was bigger, stronger, and, if I remember correctly, more mobile.
Coming out of college, Manning was recognized as the more cool and cerebral, the better decision maker, the one less likely to lose a game for you. He didn't have Leaf's athleticism, but athleticism isn't necessarily the highest attribute for an NFL quarterback. Manning was the surer bet, the guy you'd want starting games in his rookie season.
Had Leaf not gone postal, he might've turned into a sort of Brett Farve. Not necessarily a great decision maker, but a gutsy playmaker who takes chances to win, and around whom the offense rallies. Didn't work out that way. But then again, this is a little like consulting your investment advisor, who says both these investments are attractive, but for different reasons: you could put your money in utilities stocks, or in a high tech start up. One is the safer bet, the other has greater earnings potential. If you wind up losing a lot of money in the start up, that doesn't mean that he didn't know what he was talking about.
David Davenport said...
ReplyDeleteWhy else would Barry have a gut level dislike for iSteve.com? 'Cuz Steve's better looking and slimmer? I'm sure that Barry R. is a lot richer than Steve, even if B.R. is not as rich as Malcolm Gladwell.
~~~~~~~~~~~~~~~~~~~~~~~~~~~
More in the vein of 'and the horse it rode in on.' Deep differences of view about the workings of the world, what's good and bad in it, right and wrong,
who are the good guys and who are the bad. Both BR and Stevil blindly grasped differing, conflicting parts of the elephant, as we all can only do,
the elephant that mindlessly conspired and colluded to effect yet another ever greater credit debacle/implosion/bailouts...and refuse to let go of incompatible, undermining versions. Stevil strikes a nerve of BR's. In my experience it is generally the same thing that does that, having been on both ends. On the giving end, the reaction to the truth is immediately apparent, on the receiving end sometimes years or decades.
TomV, yes I meant to say the pros considered Brady the 7th best QB in the draft pool, which in theory includes 6 billion people less children and those already in the NFL.
ReplyDeleteEven as a backup Brady was a success, holding a job million of young males wish they had but with only a few openings per year. Luck allowed him to get the chance to perform when he did but that does not prove he would not have succeeded otherwise. Consider Steve Young who had to wait several years for Montana to play out his contract with the 49ers. Despite or because of the wait he too excelled when his chance came.
Often, players who are "head cases" succeed later in their career after a change of scenery. A current example is running back Cedric Benson, who failed miserably with the Bears but is now having a splendid season with the Bengals. The looming threat of unemployment is one factor that motivates them to turn over a new leaf (apologies for the pun). Ryan Leaf did not get this opportunity because he had an arm injury early in his career. When he went to his second club, he was bouncing passes to his receivers. His arm was shot.
ReplyDeleteTo be sure, Leaf was astonishingly arrogant and disrespectful FOR A QUARTERBACK. His sort of behavior has become commonplace at other positions, notably wide receiver, but not at quarterback.
Steve,
ReplyDeleteHow much of your top 50 performance results from #1 picks (first round first pick)? They're a pretty dominant as a group, and the drop off in performance is quick even to the rest of the first round (enough that ex #1 picks the second round QBs has a higher likelihood of giving you a pro bowl season or All Pro season than the rest of the first round).
Gladwell responds on his blog:
ReplyDeletehttp://gladwell.typepad.com/gladwellcom/2009/11/pinker-on-what-the-dog-saw.html
I wondered about the basis of Pinker’s conclusion, so I e-mailed him, asking if he could tell me where to find the scientific data that would set me straight. He very graciously wrote me back. He had three sources, he said. The first was Steve Sailer. Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people. Sailer’s “proof” of the connection between draft position and performance is, I’m sure Pinker would agree, crude: his key variable is how many times a player has been named to the Pro Bowl. Pinker’s second source was a blog post, based on four years of data, written by someone who runs a pre-employment testing company, who also failed to appreciate—as far as I can tell (the key part of the blog post is only a paragraph long)—the distinction between aggregate and per-play performance. Pinker’s third source was an article in the Columbia Journalism Review, prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation” which in turn makes reference to a “study” of quarterbacks conducted by a fantasy football website.
In Gladwell's blog, he writes that Pinker acknowledges Sailer as a source:
ReplyDeletehttp://gladwell.typepad.com/gladwellcom/2009/11/pinker-on-what-the-dog-saw.html
impossible to evaluate quarterbacks? it is much easier to evaluate first time NFL quarterbacks and roughly predict their performance, than it is to evaluate first time head coach candidates and predict their performance.
ReplyDeletethere's nothing to go on, when it comes to some first time head coaches. this is the main reason that i think it was not discrimination, but unfamiliarity, that explains why there were so few black head coaches in the NFL in the past. how the heck do you estimate the capabilities of guys you don't even know, haven't even seen, never even heard of?
there's no mystery with players. you watch them play every week, you have a rough idea about how they would do on another team. with coaches, if you are not on the staff of the team, you get almost no information at all about what each particular guy is doing day in and day out. there's no way for a manager to evaluate the performance and capabilities of a couple black position coaches and coordinators on various teams around the league.
that's why a brief, initial period of affirmative action for hiring head coaches was definitely a good thing. there were a couple of black coaches better than the worst white coaches, and they got a chance to prove it. of course, AA has been co-opted into a black supremacy movement, which is the risk and the downside of such initiatives. AA is almost impossible to shut down once it has achieved a good and fair result. today it is used simply to pack the league with black americans in every job, in an ongoing and very deliberate takeover. it is using the cover of "minority" hiring to transform itself into a nearly 100% african enterprise.
"there were a couple of black coaches better than the worst white coaches,"
ReplyDeleteWell isn't that charitable of you, Jody.
"Well isn't that charitable of you, Jody."
ReplyDeletebut you would never, not in a MILLION years, agree that the number 1 white running back is better than the number 64 black running back. you would deny it up and down. if pressed, you would probably suggest that there are literally ONE THOUSAND black athletes who should be running backs in the NFL before the, LOL, "best" white athlete is considered for the regular duty of 20 carries a game. white guys carrying the football, that's 1940s style segregation crap. white guys aren't carrying the ball in the NFL simply because the best players play, and that's all there is to it!
as we can clearly see from jamarcus russell, that's a load. probably the most damning evidence this decade against the lie, THE LIE, that the best players play, was when nick saban wanted to cut wes welker from the dolphins in 06 to make room for marcus vick, michael vick's retarded brother. marcus vick has no place in NFL, wes welker was the best player on the dolphins and was the only guy generating any offense for what was the blackest NFL team in the history of the league. 21 out of 22 starters were black on that below average team, including the NFL's first all-black starting offense - welker was NOT EVEN ALLOWED TO START!
you're a troll.
"but you would never, not in a MILLION years, agree that the number 1 white running back is better than the number 64 black running back."
ReplyDeleteYes Jody, I would agree that Toby Gerhart is better than the 64th best black HB in college football; I don't know about Bruva #63 Doe, he nice!
I covered Welker on one of the other threads, and I appreciate you telling me what it is that I think; sometimes I forget.
Having white players in virtually all-black positions like RB and WR probably reduces team cohesion and invites more aggressive responses from blacks on the opposing team. It's probably worse than anything Jackie Robinson faced, if only by the fact that Robinson was at least allowed to play and fully backed by management.
ReplyDeleteIt may be like having openly gay unit members or fielding attractive blondes in 3rd world combat situations. Both possible and with some appeal to fairness, but with obvious consequences.
In this way, maybe the anti-white racism in the NFL is utilitarian in trying to minimize the overall negative consequence.
"Having white players in virtually all-black positions like RB and WR probably reduces team cohesion and invites more aggressive responses from blacks on the opposing team. It's probably worse than anything Jackie Robinson faced..."
ReplyDeleteOK, I think that pretty much kills the subject.