September 16, 2011

SAT score trends

The Unsilenced Silence blog has a good graph of Asian v. white SAT scores from 1996 to 2010 in terms of gaps in standard deviations:
So, like most things involving test scores, the gap was stable in the  later 1990s. However, Asians improved through much of the 2000s, especially on the Writing test, a new section of the SAT that strikes me as easier to game because it uses subjective (but very quick) grading. 

One possibility is that globalization of American colleges has led to more Asian-in-Asia elites having their kids take the SAT, which could be increasing the pool of smart Asians taking the test. I'm also concerned about test security, however, when the test is administered in Asia. The Graduate Record Exam has been plagued by instances of cheating in Asia. Also, the new Writing test is rather like the tests that the Chinese Empire conducted for millennia to pick mandarins. 

Another possibility is that the legal immigration system is working in bringing in smart Asians, who are becoming a larger fraction of the Asian-American population over time. But what about regression toward the mean? Amy Chua is very smart, for example, but she's nowhere near as smart as her father, the conceptualizer of the memristor.

The take home lesson for Americans would seem to me to be that Americans shouldn't credulously trust assertions by the College Board / ETS / ACT that their tests can't really be gamed through test prep or other means. The SAT and ACT are, relatively speaking, very good tests, but we live in a highly competitive, globalized world full of people who are smart, hard-working, and less trusting and less believing in Fair Play than naive Americans.

Now, here's his graph of the white-black SAT gap, which has been widening:

The sharp decline on The Gap in  2006 on the Verbal test (renamed Critical Reading) may have something to do with changes in that test, perhaps intended to make it easier on blacks. But, the general trend has been for The Gap to get worse, especially since the early 1990s. Yet, since the early 1990s (the Crack Years) were not a halcyon period for young blacks, the widening of the gap probably reflects concerted efforts since then by the Great and the Good to get more blacks to take the SAT (scraping the bottom of the barrel harder). 

The process might work like this: the University of California, prevented from using racial quotas by Prop. 209) demands changes in the SAT Verbal that it hopes will boost black and Hispanic scores. A small amount of initial success in narrowing The Gap, however, leads elites to then over-confidently subsidize more black test-takers. Scraping the bottom of the barrel harder drives The Gap in Verbal back up, and increases The Gap in the other two sections to new heights. (Or I may be over-interpreting this.)

Also, it's likely that as test prep and gaming of the SAT has increased, that whites have benefited more from this than blacks. My mental model of who is doing more and more effectual test prep is Asians > whites > blacks and/or Mexicans > Other Hispanics (many of whom just arrived from Guatemala and are pretty clueless about the kinds of things that lead to higher scores that seem obvious to, say, a Hong Kong Chinese family in San Marino, CA).

Anyway, the White-Black gap graph shows the stability in gaps that is the overwhelming main lesson of generations of test score studies. Despite intense efforts by society since the mid-1960s, we still see about a one standard deviation gap between whites and blacks on a host of tests of cognitive ability (La Griffe du Lion's Fundamental Constant of American Sociology). But, that makes the recent change in Asian v. white scores even more interesting and more deserving of study.

Unsilenced Science also has informative graphs showing trends by year in SAT scores (i.e., not standard deviations).

12 comments:

  1. Yeah, I think even realizing that you should obsess about SAT scores, at the level that causes you to buy some test prep software and prepare, plan to retake the test a couple times if necessary, go to test prep classes, etc., is very much depenent on who your parents and friends are. If your parents are an electrician and hairdresser from
    El Salvador, they probably aren't going to realize that they should be pushing you to do any of that stuff, so you can go to Hopkins instead of University of Maryland.

    I'd also assume that there's more and more cheating, use of performance enhancing drugs, and less obvious gaming of the tests as you move up the ladder toward people who *really* want their kids to get into Yale. I gather that top private schools very much want to be able to talk about how many of their kids get into Ivy League schools, and so I expect they face incentives that aren't dissimilar to the incentives of public schools whose performance is measured by high-stakes NCLB tests.

    ReplyDelete
  2. I generally agree with everything you mentiond, particularly the caution on cheating. I did want to make a couple more points.

    First, the essay can be "gamed", as I described, but there's very little granularity in the scores--it's just 1-6, with two readers and a total score out of 12. It's actually hard for someone to get a 6 unless they are an exceptional writer/reasoner or memorizing many sentences that they spew out in the right order with a few prompts (something it's quite possible is happening in Korea). The rest of the scoring can't be gamed.

    Second, I think your point about the impact of test prep is about right--the tests are good, but they weren't designed for the onslaught of Asian style prep. But I did want to point out that coaching or no, it's hard to get a kid out of his range. That is, a kid who isn't particularly good at math or reading can't prep his scores to an 800 in each test, no matter how many hours of test prep he takes. But most kids will get to the top of their range with test prep, and I suspect that most East Asians are, with prep, getting to the top of their range. That would have a huge impact on scores overall, even if some of the kids were just upping their scores from a 400 to a 500.

    Whites, on the other hand, don't do prep in the same numbers. Moreover, the emphasis on grades--which was invented to help blacks, Hispanics, and girls--is really hurting smart white boys. It is genuinely astonishing to me how many suburban white boys I know with solid to good math skills--easily 600 range with no prep--aren't even taking the SAT and going to community college. Because they aren't interested in grades per se, they've been deemed not ready for college, so they've been shunted off. I should say it's not just white boys that it happens to, but the ratio of white boys in the group is a shock. (I know a Vietnamese kid who was just celebrating the fact he wasn't taking the SAT, and going to community college. I asked him if he was actually an alien from Zorkon.)

    ReplyDelete
  3. The take home lesson for Americans would seem to me to be that Americans shouldn't credulously trust assertions by the College Board / ETS / ACT that their tests can't really be gamed through test prep or other means. The SAT and ACT are, relatively speaking, very good tests, but we live in a highly competitive, globalized world full of people who are smart, hard-working, and less trusting and less believing in Fair Play than naive Americans.


    All true as far as it goes, but I hardly think that studying for a test violates the principles of "Fair Play".

    The question should not be "Why do Asians cram for the SAT"s?" but rather "Why don't whites do so?".

    In other breaking news, I can now reveal - right here on this site! - that you can "game" the Cisco network adminsitrator exam by studying for it. And there are rumors that this same magical process works for passing the bar and all sorts of other testing.

    Maybe what's really hampering American whites is the belief that their ability is fixed by their genes and cannot be improved on by old-fashioned hard work?

    ReplyDelete
  4. You can absolutely "game" the SAT. What that means is simply taking a clever test-prep program like Princeton Review, which teaches how to get correct answers even if you don't know the answer. (One of the tricks on the reading comprehension tests is to always go with the politically correct answer!) (There is always -- ALWAYS -- a reading passage or five about a brave, bold, good-hearted, fearless, super smart minority person. The answer to "Tyrone is..." must always be a positive attribute.)

    You also learn a lot of effective test taking techniques. Some students have test skills intuitively, but many don't, and many are really stupid at test strategies (like spending ten minutes on the first question and then running out of time for the last 20 -- it's amazing how many people make this mistake).

    I would expect Asians flock to these test prep classes, while blacks don't go and Mexicans don't even know what they are.

    ReplyDelete
  5. If these tests can be so easily gamed, then surely white Americans can game them even better than Asians can? I mean after all, two of the three sections on the SAT(reading and writing) clearly favor native speakers.

    ReplyDelete
  6. re: "I'm also concerned about test security, however, when the test is administered in Asia. The Graduate Record Exam has been plagued by instances of cheating in Asia."SS

    Gaming, cheating is a large part of answer with actual tracking education ( smart segregated with smart ) accounting for most of the rest.

    My son is a recent Ph.D. in Mechanical Engineering who was the lone white out of seven in his Ph.D. cohort. All of the rest were from Red China. This was at a top Research University in the USA.

    It according to my son was common knowledge that Graduate Record Exam when given in Asia was being gamed. The Chinese students who told my son about the cheating were not bothered by it because they considered American students UNEDUCATED in math and science because of the abysmal primary and secondary schooling American students underwent such that American undergraduates were in "catch-up" mode throughout their college years and therefore were really not deserving or capable of Graduate School.

    My son, who smoked the Chinese students in the comprehensives as he was the only one who passed the two days of written exams and the oral on the first try, was accused by a few of them of having been educated in Germany, given his German surname, before Graduate school because Americans could not be as good at math as my son was because of the horrible primary and secondary American schools. All of the Ph.D. candidates taught undergraduates as part of the Ph.D. curriculum so that they experienced on a regular basis the poor preparation of the undergrads for the engineering courses at this elite university.

    ReplyDelete
  7. "The take home lesson for Americans would seem to me to be that Americans shouldn't credulously trust assertions by the College Board / ETS / ACT that their tests can't really be gamed through test prep or other means."

    The ACT can be gamed, the science part in particular, but really all of it. The format has been so by the numbers for so long, that any student with initiative can pore through half a dozen retired test and come up with a game plan. The best "game the ACT" guide I've seen is the Princeton Review's. The edition I read a couple of years back was very realistic, bordering on a cynical tone. Teachers all over are being trained to use Princeton method for ACT prep classes that students actually get high school elective credit for. It has lowered the relevance of this test in areas where these practices are going on. The ACT is becoming more and more a racket.

    The only part that does much to separate the wheat from the chaff is the reading portion. Because it consists of four passages (Humanities, Social Science, Natural Science, Literature) that are excerpts to which the test taker is given no context, success on this part shows one vital trait: it separates the readers from the nonreaders. A kid who has read a lot of genres can instantly contextualize the passage and not get bogged down (you only have 35 minutes for the whole section).

    ReplyDelete
  8. Steve,

    A lot of the Verbal gap between asians and Whites is simply due to many Asians being 1st Generation. As the percentage of Asians who grow up with English rises the Gap will narrow and probably disappear.

    I deal with large numbers of Asian-American & Business people. And guess what? They really are better in math, verbally they are equal.

    ReplyDelete
  9. I added the new data to the old graphs in my new blog post. I also created a graph for the Hispanic-White SAT gaps. It shows that the math gap grew to align with the size of the verbal gap. The Educational Testing Service statistician claimed that the 1995 nonlinear recentering is what made those scores align in the raw scores, but I do not have raw score data prior to the recentering. The recentering was also supposed to reduce racial gaps, but the gaps grew in terms of Cohen’s d standard deviations.

    ReplyDelete
  10. If this is how Koreans play video games, then lazy white teen boys have no hope of competing:
    http://www.youtube.com/watch?v=YbpCLqryN-Q

    "Steady and persistent practice seems to be the only way."

    ReplyDelete
  11. "But what about regression toward the mean? Amy Chua is very smart, for example, but she's nowhere near as smart as her father, the conceptualizer of the memristor."

    It goes both ways. Very smart people can have less smart kids, and less smart parents can have a supersmart kid.
    Same with athletics. A child of an athlete may not be much of an athlete, but not-so-athletic people can give birth to a super athlete. Genes work in a funny way.

    ReplyDelete
  12. fwiw,
    I was a National Merit Scholar about 30 years ago.
    My (nearly all white, working class) high school told us not to study for the PSAT. I studied, did lots of test prep on my own. Practically read the dictionary. Scored best in my class and did well on SAT, Nation Merit Scholarship and a ton of money.

    Now, everybody knows about it and does it. Doubt the prep helps much, but it's necessary.
    the score differential will remain as those who know how to work the system succeed.
    This is not a comment about group intellect, bell curves etc.
    The differences exist, if you don't help your kid with extras like SAT prep you are handicapping them.
    And if you care and do the prep, the prep didn't matter as much as being a parent who cared.
    African American and Hispanics do not care as much. That is the saddest fact, but it is a fact.

    ReplyDelete

Comments are moderated, at whim.