April 16, 2014

SAT: The new test hasn't been tested

Looking through the couple hundreds pages of verbiage that the College Board has released about their revisions to the SAT, I haven't found any evidence that they've tested the new test they've announced. It wouldn't be terribly hard to carry out research to see what kind of questions predict college performance best, but they don't seem to have done any research whatsoever involving potential questions. They've conducted various market research studies (focus groups, surveys, etc.) of what various people say they want in the SAT, but they have done nothing to see if what they've announced will actually work. 

There's an amusing irony here: the SAT is a test used to predict how individuals do. But, as for predicting how the predictor is going to work, well, we'll just have to wing it. This strikes me as fundamentally irresponsible -- nearly a couple of million kids per year take the SAT -- but all too typical of contemporary elites in America. 

Here's a revealing passage from the College Board:
The sat has been redesigned to better align to what research shows students need to know and be able to do in order to be prepared for college and careers. This goal has led to a more focused sat with a balance across fluency, conceptual understanding, and application. In these and other ways, such as embedding mathematical practices, the redesigned sat is also a good reflection of college- and career-ready standards. 
We will continue to be guided by research and evidence as we develop the redesigned sat. In the months leading up to its release, for example, we may find through research that we need to adjust elements described in this document, such as time limits, number of questions or tasks, or scores reported. When and if we make these or other changes, we will do so solely to enhance the validity evidence supporting the test for its intended purposes, and we will communicate those changes as widely as possible and in a timely manner.

In other words, the College Board is flying blind here. David Coleman feels like these are good changes to make, but nobody has actually tested the new boss's brainstorms. They may quietly deepsix some of the innovations. Or then again, they may find they have too much prestige invested in the "reforms" they trumpeted in 2014 to get rid of them, so they may keep them to avoid admitting mistakes. After all, they're just playing around with the lives of young people, so who needs to be careful?
Because the redesigned sat is a different test than the current sat, a numerical score on one test will not be equivalent to the same numerical score on the other. Therefore, to help higher education admission officers, k–12 educators and counselors, and students and parents transition to the new test scores, we will be providing a concordance between the scores on the current sat and the redesigned sat that shows how to relate the scores of one test to the scores of the other. ... The concordance information will be released immediately after the first operational administration of the redesigned sat in 2016. 

Ready, Fire, Aim.

The good news is that it's hard to screw up an IQ-type test completely. As Robert Gordon says, Life is an IQ test, and there are such large differences in IQ among individuals that just about any collection of questions will sort people in a rough rank order of smartness. But, still ... shouldn't we be getting smarter about intelligence, not stupider?


Reg C├Žsar said...

To put a (Bengal) Tiger Mom spin on this, here are some replies to a question recently posted on Quora.com:

What is a job that exists only in your country?

Coaching classes to crack the entrance exam of bigger coaching classes that further coach you to crack various engineering/ medical examinations. This is the Indian version of Inception.
21 Feb

Sujay Muzumdar
In Hyderabad, you will find one step ahead of this. There are coaching institutes for 6th and 7th class students so that they can make it to a coaching institute (popularly called Sharma's) for 8th, 9th and 10th class students. Sharma coaches people to get into IIR Study Circle (popularly known as Ramaiah) which coaches students for IIT-JEE. :)
12 Mar

Kumar Atul
Coaching for a coaching for a coaching. We need to dig deeper. :P
12 Mar

Laugh, but it beats some of the other unique-to-our-country jobs such as carrying pupils across a raging stream in plastic bags (Vietnam) and walking behind cars to block congestion-enforcement cameras' view of license plates (Iran).

TheLRC said...

Steve, you've hit upon one of the prime dangers with centralized educational authority. 'Progress' that is too easy to implement messes with people, in serious, life-altering ways.

I live in Hong Kong, which as Educational Hothouse is exceeded maybe by South Korea. In the past couple of years the Education Department here came up with a whole new structure for late secondary education, extended degree-level study from three to four years, changed the pre-university exams, and more.

People with kids in their teens have of course been going crazy trying to figure out what subjects their darlings should take (i.e. which subjects will get them into university), schools have to alter their curricula and learn how to teach to whole new tests, and more.

It's analogous to the effects far-reaching but ill-defined laws (such as Obamacare) have on the economic climate. Uncertainty breeds stress, which breeds an unhelpful reactionary slowdown/overly-cautious torpor.

It's in some ways better to have some totally wacky but *stable* system that everybody's used to and can prepare for, both practically and psychologically.

This is one of the foundation stones of conservatism that even lots of conservatives don't really try to defend, because trying to do so means you often must speak up for leaving alone something that's eminently reformable.

Anonymous said...

By not making the questions public as early as possible by pre testing the questions they will not let the test prep industry full prepare.

Steve Sailer said...

So, perhaps the College Board thinks there are moles within the Educational Testing Service that sell upcoming exams to Korean test prep services?

Anonymous said...

I did military service with a guy who notoriously stole the national Matriculation papers and got caught. The 'agencies' eventually 'worked together' because on the eve of passing out parade he was given stripes on his arm. Officers were required to have matric or higher. That was not the problem, He had terrific grades. It was the criminal record that cost him birdshit on his shoulders.

Thinking about it, there is a Mike Judge type script about guys who steal the SATs ans still fail.

Steve Sailer said...

I'd watch that.

Anonymous said...

So the people with the high IQs, e.g., Koreans and East Asians, who spent all that money for test prep and long hours after school and on weekends through high school, don't know it really doesn't help?! Go and pick up a Koreatown newspaper on Sunday and look at how much of it is devoted to academic tutoring and SAT test prep. They have a 50-page education section devoted to test prep services. They will list the names of former client students (in English and Korean), show his/her SAT score, and the college accepted to (e.g., Princeton, Johns Hopkins, Yale,...).

Anonymous said...

There is one and only one answer to all of this schizophrenic sociopathic nihilistic insanity.


Education Realist said...

"In other words, the College Board is flying blind here. David Coleman feels like these are good changes to make, but nobody has actually tested the new boss's brainstorms. They may quietly deepsix some of the innovations. Or then again, they may find they have too much prestige invested in the "reforms" they trumpeted in 2014 to get rid of them, so they may keep them to avoid admitting mistakes. "

and (from comments):

"So, perhaps the College Board thinks there are moles within the Educational Testing Service that sell upcoming exams to Korean test prep services?"

Okay, now you're just hurting my feelings, man. Lord knows I get tons of link love from you, but my last post is pretty directly on these points.


Education Realist said...

Incidentally, I was working at Kaplan when the 2005 changes came out. We got quite a bit of information from the College Board, and I know that they did do field testing because (as I say in my piece) I distinctly remember getting an email from Kaplan saying that the essay weight was being dropped from 50 to 33%. (Can't find substantiation of that, but it was much publicized at Kaplan).

There were test question samples, and our prep materials were very much able to prepare students. Kids were taking practice tests before March 2005.

Anonymous said...

This "redesign" has everything to do with aligning the test with the Common Core curriculum.

If the SAT tests are CC based, kids will be coerced to learn the particular curriculum -- and abdicate control of their personal information -- to get into college.

Back-handed coercion by the ed-school masters.

Dr. Stephen J. Krune III said...

Doesn't it seem like the SAT should have a prefix for its scores based on which version of the test it is?

Anthony said...

They had to do something soon, probably for corporate reasons (in other words, they didn't *really* have to do something that soon). However, switching to adaptive, computer-based testing, will allow them to undo any of their mistakes on the QT. Perhaps the next year or so of SATs are just public beta testing for the adaptive test.

Anonymous said...

In several of these threads, people have been suggesting reasons for the hate-fest on the SAT.

Clearly race is one reason.

Another major reason that no one seemed to have put their finger on: the SAT is an objective test that wealthy and intelligent parents can't manipulate to benefit their children disproportionately. Consider that the children of your average high achieving couple will be intelligent, but less intelligent than the parents due to regression toward the mean (because the heritability of IQ is <100%). These parents can compensate for their children's relative deficiencies in IQ by coaching and helping them with things like homework, extra-curricular activities, admissions essays, and so on. Basically GPA and the other factors considered in college admission are more amenable to improvement by good parents. However, they cannot take the SAT for their kids, and coaching for the test just isn't that effective.

Jerry said...

TheLRC said...
I live in Hong Kong...

That makes two isteve readers in Hong Kong. We should get together and start a club.

Seriously, education in Hong Kong is a mess, but there is a third way between local schools (the perimeter walls stylishly decked out with barbed wire for some reason... I am not kidding) and $$$ private snob-schools. In China of course homeschooling is prohibited, but here there is some leeway.

Anyway, all these parents trying to get their kids into "good" universities are fighting the last war. It's sad to watch. No profession has any guarantee of a prosperous lifestyle in the decades ahead, perhaps excepting some tiny yearly number of specialists in medicine.

Assistant Village Idiot said...

If you make the real rules of the test less obvious, then the smart people will pick up on the strategies more quickly. The semi-smart people will remain stuck on what they believe the rules for success "should" be.

Compare: Schools public and private favor girls at something between a half and a full letter grade, on average, K-12. That is because the schools (female dominated) use what it believes the rules for success in life "should" be, and that favors girls. But the actual rules for life are different, and the smarter boys figure that out sooner. More than one way to skin a cat, and all that. The main victims are less-intelligent boys and eccentric girls. They are the ones who need to be told that life's rules are different, because they won't see it on their own.

Some of the smarter girls catch on, the rest become permanently pissed at the unfairness of a system that won't reward them for what they had been told were the rules.

As long as the SAT does measure IQ, the smart kids will know how to exploit that credential. Okay, only some will at first, but it will find its level. As soon as it stops doing that, the smart kids will start figuring out what credential really does work. The semi-smarts will bang their head against the not-your-father's-SAT wall for years after.

Anonymous said...

For all this testing, I wonder if we're doing any better at identifying high IQ kids than back in ye olde england (or similar) when "knowing your Euclid" separated out the "donkeys", as I think the saying went, for about 1000 years.

Successfully learning Euclid's Elements through to the end in high school (I suppose you'd consider that advanced geometry today) I'd guess is highly correlated with G and, like the SAT and GRE today, everybody knew it.

Ah, much more polite to say Pons Asinorum.

"...Pons asinorum (Latin for "bridge of donkeys") is the name given to Euclid's fifth proposition in Book 1 of his Elements ...

...the more popular explanation is that it is the first real test in the Elements of the intelligence of the reader and functions as a "bridge" to the harder propositions that follow.[2]

Whatever its origin, the term is also used as a metaphor for a problem or challenge which will separate the sure of mind from the simple, the fleet thinker from the slow, the determined from the dallier; to represent a critical test of ability or understanding."

nsam said...

Wasn't one of the early Euclid theorems called Asses Bridge? Asses failed to understand it and couldn't cross the bridge. Aha, it is here: http://www.britannica.com/EBchecked/topic/724634/The-Bridge-of-Asses
"Medieval schoolboys did not usually go beyond the Bridge of Asses, which thus marked their last obstruction before liberation from the Elements."

Jorn said...


Anonymous said...

I still think the biggest benefactor in this whole thing will be the ACT owners

Anonymous said...


"Secret Military Test, Coming Soon to Your Spanish Class:
A powerful, precise language aptitude test is entering civilian life"
, Michael Erard, Nautilus, issue 12, April 17, 2014:

"... the Hi-LAB (or “High Level Language Aptitude Battery”)... was developed by University of Maryland researchers working on a government contract in order to predict a person’s ability to learn a language to a very high level. Since its release in 2012, the Hi-LAB has been rolled out to government agencies and military training schools and will eventually be available for civilians as well.

...to become highly skilled in a second language, simply devoting the 10,000 hours of practice that Malcolm Gladwell made famous in Outliers isn’t enough. ... a person needs high-performing cognitive hardware...

... Defense Language Institute used the aptitude test... to track people toward certain languages...

...One might think that listening abilities would be a central part of high-level learning, but the results eventually showed otherwise....

...a good working memory correlated with successful second language learning. ..."