July 8, 2013

Does IQ testing work or not work?

My previous post cites an interesting new study of brain development that begins:
“IQ predicts many measures of life success ...

Here's an article in Salon by the author of a new book on the failures of IQ testing, which appears to be aimed at upper middle class parents of kids who are having trouble in school:
IQ tests hurt kids, schools — and don’t measure intelligence 
The research proves that IQ tests poorly predict learning disabilities. So why are schools still using them? 
BY SCOTT BARRY KAUFMAN

Can IQ testing both "predict" and "poorly predict" at the same time?

Can a glass be half-full and half-empty at the same time?

If it's your kid, of course you want more detail. You want to know your child's strengths as well as weaknesses. But one number IQ scores also have their uses, especially in social sciences when thinking about groups.

The half-full glass wisecrack that I use a lot seems snarky, but the truth is that I found learning enough about psychometrics in the 1990s to be able to write non-stupidly on the subject to be hard work. I noticed early in my writing career that writing X number of words on IQ was more mentally exhausting than writing X number of words on almost any other subject.

But, I don't regret the effort in that once you get the hang of thinking about IQ, you'll notice that you've learned a lot about how to think about the human world in general. The subject is so difficult that you need to improve your toolbox of helpful reductionist concepts, and you need to be able to hold opposing concepts like nature and nurture in mind at the same time, which isn't easy to do.

Back in 2007, I wrote up a Frequently Asked Questions list for IQ that discussed some of these issues:
Q. So, do IQ tests predict an individual's fate? 
A. In an absolute sense, not very accurately at all. Indeed, any single person's destiny is beyond the capability of all the tests ever invented to predict with much accuracy. 
Q. So, if IQ isn't all that accurate for making predictions about an individual, why even think of using it to compare groups, which are much more complicated?
A. That sounds sensible, but it's exactly backwards. The larger the sample size, the more the statistical noise washes out. 
Q. How can that be? 
A. If Adam and Zach take an IQ test and Adam outscores Zach by 15 points, it's far from impossible that Zach actually has the higher "true" IQ. A hundred random perturbations could have thrown the results off. Maybe if they took the test dozen times, Zach just might average higher than Adam. 
But for comparing the averages of large groups of people, the chance of error becomes vanishingly small. For example, the largest meta-analysis of American ethnic differences in IQ, Philip L. Roth's  2001 survey,[Ethnic group differences in cognitive ability in employment and educational settings: a meta-analysis, Personnel Psychology 54, 297–330.] aggregated 105 studies of 6,246,729 individuals. That's what you call a decent sample size. 
Q. So, you're saying that IQ testing can tell us more about group differences than about individual differences? 
A. If the sample sizes are big enough and all else is equal, a higher IQ group will virtually always outperform a lower IQ group on any behavioral metric.
One of the very few positive traits not correlated with IQ is musical rhythm—which is a reason high IQ rock stars like Mick Jagger, Pete Townshend, and David Bowie tell Drummer Jokes. 
Of course, everything else is seldom equal. A more conscientious group may well outperform a higher IQ group. On the other hand, conscientiousness, like many virtues, is positively correlated with IQ, so IQ tests work surprisingly well. 
Q. Wait a minute, does that mean that maybe some of the predictive power of IQ comes not from intelligence itself, but from virtues associated with it like conscientiousness? 
A. Most likely. But perhaps smarter people are more conscientious because they are more likely to foresee the bad consequences of slacking off. It's an interesting philosophical question, but, in a practical sense, so what? We have a test that can predict behavior. That's useful. 
Q. Can one number adequately describe a person's intelligence?
A. Sort of. 
Q. "Sort of"?!? What the heck kind of answer is that?
A. A realistic one. 
Q. How can something be true and not true at the same time?
A. How can the glass be half-full and half-empty at the same time? Most things about IQ testing are partly true and partly false at the same time. That's the nature of anything inherently statistical, which is most of reality. 
Humans are used to legalistic reasoning that attempts to draw bright lines between exclusive categories. For example, you are either old enough to vote or you aren't. There's no gray area. But the law is artificial and unlike most of reality. Many people have a hard time dealing with that fact, especially when it comes to thinking about IQ. 
Q. Enough epistemology! How can you rationalizing summing up something as multifaceted as intelligence in a single number? 
A. Think about SAT scores. Your total score says something about you, while breaking out your Math and Verbal scores separately says more. A kid who gets a total of 1400 out of 1600 (Math + Verbal) is definitely college material, while a kid who gets a 600 isn't. That's the big picture. For the fine detail, like which college to apply to, it helps to look at the subscores. A kid with a 1400 who got a 600 Math and an 800 Verbal would be better off at Swarthmore than at Cal Tech. 
A few years ago, the SAT added a third score, Writing, but many colleges aren't sure how useful it is, and there's some sentiment for dropping the Writing test as not worth the extra time or cost. In other words, there are diminishing marginal returns to more detail.

By the way, the FAQ was a mainstay of the Usenet era of Internet discussion groups before the invention of the World Wide Web. It was widely considered a terrific way to bring people up to speed. Yet, in this century, FAQs seem to have fallen dramatically out of fashion. Why? Does it have something to do with the rise of Wikipedia? 

The dialogue format for written instruction of difficult material has gone in and out of favor over the ages. Plato and Galileo used it, but at the moment it seems to be just not done.

22 comments:

  1. IQ naysayers dwell on cultural differences between groups, but I think they are best devoting their time to citing the differences between eras.

    A typical modern Flynn-Fantastic adulator who boasts of Millennial cerebral acuity is just as guilty of the same short-sighted thinking that the evo-relativists are in claiming that IQ tests are culturally biased (with the implication that groups in question are the same generation). If you enlarge the sample enough, any outstanding individual contribution is just that much less meaningful. Sorta the bread-and-butter of the modern corporacracy.

    I wonder how anyone can claim we are "smarter" now than we were 40 years ago. This is such such utter, obvious garbage. The only thing that has changed in 40 years are our cultural expectations and parental meddling. Hmmm!

    ReplyDelete
  2. Liberal arguments against g testing:

    1) Stephen J. Gould

    2) Dats raciss!!!!

    3) STEPHEN J. GOULD!!

    4) I heard some anecdote about there was this guy whose IQ varied 30 points

    5) STEPHEN J.GOULD godammit!!!!!!

    6) Some racist 115 years ago did it wrong

    7) IQ don't measure nothing, but your stoopid

    8) STEPHEN J. GOULDDDDD!!!!!!!!

    and so on

    ReplyDelete

  3. A few years ago, the SAT added a third score, Writing, but many colleges aren't sure how useful it is, and there's some sentiment for dropping the Writing test as not worth the extra time or cost. In other words, there are diminishing marginal returns to more detail.


    Writing was added to allow women to double count their verbal scores which are comparable to men's. It narrows the "gap" among the top scorers so to speak. More women than men take the test but the absolute number of men who get an 800 math score is about double the number of that of women.

    ReplyDelete
  4. @Steve

    "A. If Adam and Zach take an IQ test and Adam outscores Zach by 15 points, it's far from impossible that Zach actually has the higher "true" IQ.... Maybe if they took the test dozen times, Zach just might average higher than Adam."

    I would suggest that peak performance is the correct measure of IQ, not the average. A person might be ill on many occasions of test taking, and it is how they perform at their best that measures their underlying g.

    http://iqpersonalitygenius.blogspot.co.uk/2012/08/how-should-we-measure-general.html

    *

    "conscientiousness, like many virtues, is positively correlated with IQ, so IQ tests work surprisingly well."

    I don't think this is correct - at least not necessarily, and not as a generalization. Between ethnic groups there does seem to be a correlation between IQ and C - probably due to natural selection in complex agricultural societies, as described by Greg Clark - but this breaks down within ethnicities - indeed several studies by Adrian Furnham (of University College London) in college students found an inverse relationship between IQ and C.

    It also breaks down between men and women - on average, men are higher in IQ and women in C -

    http://iqpersonalitygenius.blogspot.co.uk/2012/08/do-elite-us-colleges-choose-personality.html

    ReplyDelete
  5. Dialogue requires two interlocutors to come to some kind of consensus, which is usually a synthesis of two positions, but a synthesis in which some interlocutors need to concede more than others.

    No one's too interested in doing that sort of thing right now. Moral outrage and social justice do-goodery are more satisfying.

    ReplyDelete
  6. http://www.nypost.com/p/news/business/corzine_off_the_crook_t3VpDFmfEsx9Qd7VdtCLvM?utm_medium=rss&utm_content=Business


    And the GOP Sucks up to the superrich.

    ReplyDelete
  7. "Writing was added to allow women to double count their verbal scores which are comparable to men's. It narrows the "gap" among the top scorers so to speak. More women than men take the test but the absolute number of men who get an 800 math score is about double the number of that of women."

    Interesting. Yet, it seems that men still write most of the "important" books. "My favorite book of 2011 was Homesickness: An American History by Susan J. Matt." How often does Steve refer to a book by a woman? Are times changing? Are women hitting their stride? Or do women still defer to men, like the Bronte sisters to their undistinguished brother?

    ReplyDelete
  8. Writing was also added to give scorers a chance to add points for race and/or leftism. Check out some actual SAT writing prompts. Note how they beg the writer to step on an ideological banana peel, while also admitting first-person signalling of favored backgrfounds.

    ReplyDelete
  9. "One of the very few positive traits not correlated with IQ is musical rhythm—which is a reason high IQ rock stars like Mick Jagger, Pete Townshend, and David Bowie tell Drummer Jokes. "

    What are the others? Reading the HBD stuff is depressing to me. I read the entirety of what you just posted, but it is very hard sometimes to stomach a lot of the implications of this material.

    I mean reading this, it's like you meet the average guy on the street and have to think "You will never be good at anything, ever."

    It's be nice to see a list of positive things that aren't correlated to higher IQ.

    ReplyDelete
  10. Steve, I have long appreciated your FAQs on Race and IQ. Have borrowed heavily from them when debating the topics on various forums. Your FAQ on race has been particularly useful in refuting foolish "Race is a social construct." blather, so thanks for those.

    ReplyDelete
  11. "I mean reading this, it's like you meet the average guy on the street and have to think "You will never be good at anything, ever.""

    Anecdata: I offer up my father, age 78: IQ of 95, excellent mechanic, encyclopedic knowledge of aircraft, fantastic improvisational cook and musician, conversationally fluent in any language you can think of after 3 days among the natives. No abstract capability at all. (I wrote about him earlier, here, just to show I didn't invent him: http://educationrealist.wordpress.com/2013/01/03/acquiring-content-knowledge-without-hirschs-help/ (second paragraph only))

    We'd just heard about the Asiana flight and I was reading the first witness account, saying that the tail had fallen off and then the plane had crashed.

    "Naw," said dad, "It's SFO. The pilot came in too low, hit the sea wall, knocked off the tail. Check the debris when the photos come out--you'll see it strewn all over the tarmac. Betcha anything he tried to abort."

    This was an hour after the crash, within the next few hours a number of aviation experts showed up in comments sections and quotes saying the same thing.

    And, for that matter, he had two kids with IQs over 125 (mine over 145), another two with IQs north of 100. That's an outlier situation, but it happens.


    The problem, of course, is that we are wiping out the gainful employment for people like my father. We're spending tons of time in school trying to educate them using abstractions and forcing critical thinking and literature analysis down their throats, rather than teach them interesting content and give them meaningful jobs after high school.

    ReplyDelete
  12. " Piper said.."
    And Piper said it brilliantly. I took the SAT way before the written component. The prompts he linked to are so obviously steering the student toward the "right" answer, and the right kind of student will catch the cue. The students who had the right kind of teachers have such a big advantage, having been inculcated with The Narrative.

    ReplyDelete
  13. Re Usenet: This isn't a FAQ in the Usenet sense; it's a dialogue. These so-called FAQs seem to have arisen with the death of Usenet.

    You seem inconsistent on the "IQ measures intelligence" thing. You acknowledge that maybe "some of the predictive power of IQ comes not from intelligence itself" (e.g. conscientiousness), but then you say that IQ "sort of" describes someone's intelligence. But if the only way to tell if a particular measurement describes a property is through prediction ("external validation"), then IQ tests also "describe" conscientiousness.

    Some critics have said that they have nothing against IQ tests, as long as you don't call them intelligence tests. I think there's something to that. The tests measure some vaguely defined property called "intelligence," along with other vaguely defined or undefined things. I think research on biological correlates of g might be a way out of that, but it's nowhere near that stage yet.

    Some minor points. "That's what you call a decent sample size." Sample size is rarely an issue with these studies. The question is how representative the sample is.

    Typo: I think you meant that rhythm is negatively correlated with IQ, not uncorrelated. Otherwise, no Drummer Jokes, right?

    ReplyDelete
  14. [T]he truth is that I found learning enough about psychometrics in the 1990s to be able to write non-stupidly on the subject to be hard work. I noticed early in my writing career that writing X number of words on IQ was more mentally exhausting than writing X number of words on almost any other subject.

    Seems to me the real IQ test is writing correctly about heritability. Rule of thumb: If you think you understand heritability, you don't.

    ReplyDelete
  15. Right, writing accurately and readably about the heritability of IQ is very hard.

    ReplyDelete
  16. Or heritability of anything. It's no harder to understand/describe heritability of IQ than heritability of height. Both are very hard if not impossible to understand. People who should know better, like Razib Khan and (if he was quoted correctly) Charles Murray have said outrageously wrong things about heritability.

    The philosopher Ned Block put it best: "Heritability is a lousy scientific concept." But it's not meaningless, and it's the best concept we've got for now.

    ReplyDelete
  17. Maximo Macaroni7/9/13, 5:37 AM

    If IQ predicts nothing, why are people still worried about the effects of lead on the IQ of children?

    ReplyDelete
  18. Perhaps you or one of your readers should start "FAQipedia", to parallel Wikipedia. I'm serious. They'd both be good, each in their own way. Wikipedia's organizational style could be exactly imitated. Or, more simply, Wikipedia could add a FAQ section to each article.

    One thing about the FAQ format that is good is that it makes addressing myths and misconceptions easier. In a Wikipedia or other article, it's awkward to insert a one-sentence paragraph saying, "Contrary to what many believe, poisonous mushrooms do not tarnish a silver spoon."


    ReplyDelete
  19. Eric Rasmusen writes:

    "One thing about the FAQ format that is good is that it makes addressing myths and misconceptions easier. In a Wikipedia or other article, it's awkward to insert a one-sentence paragraph saying, "Contrary to what many believe, poisonous mushrooms do not tarnish a silver spoon."

    Right. It makes the writing style choppy and it may well be that readers miss the point of many of these random interpolations. With an FAQ, readers can skip over what strike them as stupid questions or say, "Yeah, I was always kind of wondering that ..."

    ReplyDelete
  20. Perhaps you or one of your readers should start "FAQipedia"

    Faqs.org. Doesn't have the inter-linkage that put WikiPedia on the map, though.

    ReplyDelete
  21. Somehow the lack of efficacy of IQ tests just doesn't penetrate high stakes measurement of persons such as that basic to efficacious placements in the military and in national security positions. Beria, reportedly, used the Raven
    Advanced Progressive Matrices, to assess applicants for his KGB operation, even if use of such tests was illegal for anyone other than the chief honcho of the police state! The use of IQ testing by the American military has been highly regarded all over the world in placement challenges in military situations, in industry, and so forth. Leftist
    verbal engineers from cafe society are recurrently disvoering that intelligence is a fraud.

    ReplyDelete
  22. Favorite drummer joke:
    Q: How does the roadie tell that the drum riser is level?
    A: The drool comes out of both sides of the drummer's mouth.

    ReplyDelete

Comments are moderated, at whim.