David Coleman is redesigning the SAT. Does he know what he's doing?

In 1994, Richard Herrnstein and Charles Murray pointed out that the SAT was such an effective college admission test that it was having profound effects on society as a whole (with Herrnstein more positive than Murray about the net impact).

Ever since, the College Board (the sponsor of the SAT -- the Educational Testing Service writes it up for the CB) has been trying to do something about that.

For example, one problem with the SAT back in 1994 was that it was too easy to get an 800 on the Math part of the test. When I was at Rice U., it was common to hear the better STEM majors say things like, "Well, sure I got an 800 on the Math SAT, but I'm not a real 800 like Joe or Bill are." In contrast, back then the SAT verbal test (on which only double digits scored 800 annually) was, as Henry Harpending has pointed out, perhaps the best high end IQ test in the world.

So, in 1995, the College Board changed the scoring. But not to make the Math scoring more like the Verbal, but vice-versa.

In the early 21st Century, the chancellor of the College Board's biggest customer, the University of California, demanded changes in the SAT. So the analogies questions were thrown out because they were like, well, you know, they were kind of like ... well, I don't know what they were like, but they were annoying. And a Writing section was added, featuring a large essay which has proven over the last decade to be a probing test of how legible and swift is one's penmanship (or pencilmanship, in this case), although perhaps not much else.

The old head of the College Board, former West Virginia governor Gaston Caperton, resigned last year around the time of his third divorce to pursue his fortune on Wall Street. The new head is David Coleman, who is also the author of the Common Core standards for K-12. He soon announced he was going to revamp the SAT by 2015.

In other words, the educational establishment has decided to bet the country, at least the K-16 part, on one guy.

Coleman sounds like not the worst choice for The Guy. Ideologically, I see a lot of similarity between Coleman and E.D. Hirsch, of the Core Knowledge Foundation.

Hirsch was a top academic literary critic who got involved in trying to help local schoolchildren learn. He decided that the reason average children's reading comprehension was so bad was because they didn't know anything -- they were constantly stumbling upon the text's references to events or concepts that they didn't know anything about, and were thus losing the thread of what the author was saying.

So, Hirsch argued that schools should teach less poetry and fiction and instead use more informative nonfiction texts that would help students build up their core of common knowledge. Hirsch published long lists of the facts that he felt came up the most often as the assumed background knowledge of the reader in the better sort of writing.

This is not a bad idea, although it's not as much of a panacea as Hirsch thought it was.

Politically, Hirsch came out of the liberal Jewish mainstream of, say, 1960. But by the late 1980s poor Hirsch was denounced as a flaming rightwing advocate of Dead White European Males. (Here's a 1990 Christopher Hitchens article on Hirsch.)

Perhaps because of all the vitriol spewed at Hirsch's Core Knowledge campaign, there have been many denials that Coleman's Common Core has the slightest to do with Hirsch. But, my impression is that they are pretty similar: coming out of the right of center side of the liberal Jewish mainstream.

Coleman's philosophy basically appears to be the same as every other educational reformer: Be Like Me. Coleman is a bright, cultured former McKinsey consultant, and thus his Common Core write-up would be a good outline for the home schooling of David Coleman Jr.

Unfortunately, I haven't seen much evidence that he knows much at all about, say, psychometrics. 

Details were not terribly forthcoming about his New SAT, but Inside Higher Ed has now reported:

Hints on the New SAT 

September 20, 2013

Scott Jaschik 

TORONTO -- David Coleman, president of the College Board, was fairly general here Thursday in describing the changes coming to the SAT. His theme was that the new SAT would be more closely tied to high school and college curriculums and less coachable than is the current version. 

Aren't those tradeoffs?

I've long argued that the rise in test prepping should lead to more weight in college admissions being given to Advanced Placement test because if you are going to test prep for years, you might as well learn something, like chemistry or French literature. So, maybe this is brilliant ju-jitsu on Coleman's part by applying my AP idea to the SAT.

Or maybe he just doesn't get it.

The point of having standardized college admission tests is to fill in the obvious shortcoming of just using high school GPA. Making the SAT more like high school seems pointless. If we are all supposed to be into "critical thinking," maybe we should make high school more like the old SAT?

Coleman presumably thinks his Common Core is going to revolutionize American K-12 school by making it much more rigorous, so, assuming that will happen real soon now, well, then of course we should change the SAT in 2015 to be more like his Common Core.

More likely, the Common Core will just be another K-12 fad, like the Small Learning Communities boondoggle that Bill Gates wasted $2 b-b-b-billion on in the 2000s, but Coleman will manage to do some genuine damage to the SAT.

But in his remarks at the annual meeting of the National Association for College Admission Counseling, he seemed most comfortable in predicting big changes for the essay portion of the test. He admitted that there are serious flaws in the test now (namely that test-takers are not rewarded for being factual and that in fact a common strategy for students to do well is to ignore the facts). Further, he floated a specific idea for changing the essay. (He was not specific in describing most other changes.) 

Coleman asked attendees to imagine replacing the current essay (in which students respond to a prompt) with a writing assignment in which students analyze information. "What if you were analyzing a source" in a short essay "and using evidence from that source?" he asked. Might such an essay prompt "celebrate analytic writing?"

On the AP history tests, they have data-based essay questions where they give you eight or ten brief documents to use in answering a question. Extending that to the SAT isn't the worst idea.

Of course, that's pretty much how the Fire Department of New York hiring test worked: they gave you a few paragraphs of fire fighting instructional material -- e.g., what kind of chain saw blades to use on steel doors and which kind on wooden doors so you wouldn't accidentally rip your face off when the chainsaw bucked -- then asked questions based on the presented material. Test-takers could do well either through good reading comprehension while taking the test or by learning the firefighting material beforehand. As a potential fire victim, that seems pretty okay to me, but the whole test was junked by a federal judge in 2009 because blacks and Hispanics didn't score as well on it.

But, AP essay questions are about the AP subjects such as European History, while the SAT is supposed to be a more general test. One of its functions is to find the "diamonds in the rough" who haven't necessarily had every educational advantage.

The essential problem with essay questions are small sample sizes, on multiple dimensions, which leads to scores being less reliable. In other words, the endlessly denounced multiple choice question has some advantages, such as that you can ask a lot of them, thus giving a reasonable sample size.

He said that he was bothered by the status quo, where the essay "does not grade you on the correctness of what you write." And he described feeling somewhat ashamed when he spoke to a friend who taught SAT test-prep in Hong Kong when she told him how she helped students who asked about how to come up with examples to back up their points in their essays on the SAT. "You make them up," she said. 

In the decade since the College Board announced and then started giving the essay portion of the SAT, reaction has been decidedly mixed. College Board officials predicted that the addition of the writing test would send a message to high schools to take writing seriously. 

But almost from the start, many writing experts questioned the kind of writing the College Board was promoting, saying that it emphasized using a few impressive words and paying little attention to facts or logic. A writing professor at the Massachusetts Institute of Technology famously coached students on how to write laughably untrue statements in SAT essays that would receive good grades from the College Board. Many colleges have pointedly not used the writing scores of the SAT. 

Until Coleman became president of the College Board last year, however, officials there have defended the essay.

Coleman didn't focus on the issue in his prepared remarks, but did so when urged by Jon Reider of San Francisco University High School to "get rid of the essay." 

Reider said that the essay test has forced counselors like him to tell students to do things that are educationally unsound -- just so they can get a decent score on the essay. 

He said that when smart students who are good writers talk to him about disappointing scores on the SAT writing test, he tells them that they are doing poorly because "you are a good English student and you've been taught to stop and think before you write, but that's not an asset on this test."

Sheer amount of verbiage emitted into the test booklet correlates with essay scores.

... It's also wrong, Reider said, for the College Board to promote writing without any relationship to facts. "I challenge anyone in the room: Have you ever sat down for professional purposes to write about a question you have never seen before, in which the accuracy of what you write is totally and utterly meaningless?" 

Coleman's response to Reider was that "you've got a real point. You really do." 

But Coleman cautioned that eliminating the essay, instead of fixing it, might send the wrong "message."

Fixing essay grading sounds extremely expensive. Are they really going to pay graders to stop grading and go look things up on Wikipedia to see if the student knows what he's talking about?

He did not dispute any of Reider's criticism, but said that "there may be alternatives beyond dropping it." 

Coleman first announced plans to redesign the SAT in February. 

Most of his comments here were consistent with the goals he announced previously for the shift -- that he wants to more closely align the SAT with the high school curriculum, for example.

The SAT college admission test was long dominant over its Iowa competitor, the ACT, but recently the ACT has surpassed the SAT in sheer numbers. The ACT positions itself (I don't know how accurately) as the meat and potatoes test that's an achievement test of what you learned in high school, not some fancy-pantsy "aptitude" test.

On Thursday, he took swipes at how the SAT has evolved, noting that the test has created a huge test-prep industry and that students aren't necessarily being encouraged to focus on important learning. 

He noted that when someone says a person has just used "an SAT word," the idea isn't that the person has shown eloquence or clarity but that "they have used a word they would never use again." 

Coleman also talked about how a better SAT shouldn't be something one has to learn at a test-prep service. The revised SAT "shouldn't be a sudden departure" from what students learn in class, he said. A student's plan to take the SAT "should inspire excellence in the classroom." 

One questioner (and a number of people in comments after the presentation) suggested that Coleman was describing a test more like the ACT, which has long claimed that it is more curricularly based than the SAT. And the ACT has gained substantially against the SAT in recent years in market share. Amid all these discussions, one questioner here, warning the audience to prepare to be shocked, said that he liked the current SAT, and feared the new SAT might be something like New Coke.

Ouch.

Coleman responded that "I'm not interested in a Coke/Pepsi kind of debate here."

You may not find the New Coke fiasco of 1985 interesting, Mr. Coleman, but fiascos are interested in finding you.

And he asserted that what he was talking about was a new kind of test, one that would promote educational values. 

This could be ugly.

Presumably, there are some good psychometricians in the backrooms at ETS whose jobs are mostly to keep the politicans running the College Board from messing up too badly.

Another question -- from someone who used to work in admissions at an elite university -- highlighted how challenging that may be. The questioner said that his instructions at the university -- straight from the president's office -- were to increase average SAT scores and to increase minority enrollments. He said that he found it impossible to do both.