May 20, 2009

How to make your SAT prep business look good

Give the kids a really hard practice test when they walk in the door. Measure progress against that. From the WSJ:
SAT Coaching Found to Boost Scores--Barely

My published articles are archived at iSteve.com -- Steve Sailer

23 comments:

Veracitor said...

Yeah, but the key thing is that "a ten-point difference in the Verbal score* or 20-point difference in Math means a significantly higher chance of admission to a highly-selective college."

*I assume this means: for white/asian applicants already scoring in the 700's, and Verbal is weighted more because asians do worse on it (I think Verbal used to be more g-loaded, but I'm not sure that's still true).

Mr Lomez said...

There's a lot of false promises made in the SAT prep industry, no doubt. Any company that guarantees a score increase -- of any magnitude -- is full of shit.

This is true mostly because whether or not these courses are helpful is entirely dependent on the student herself. If she is unmotivated (despite spending thousands on this stuff, students very often are), and/or too dumb (ie - lacking in the 'g' department), there isn't much an SAT prep class can do for them. In my extensive experience in this field, I'd say about half of my students fall into one of these categories. Very often, they go hand in hand.

Another 1/4 of the students are motivated but not bright enough for it to make much difference (meat of the bell-curve, I'm looking at you). I can teach these students to approach the test strategically -- looking out for common pitfalls, avoiding questions they are likely to get wrong, and focusing on how to best allocate their time. These gains are modest, but not insignificant (say 50 - 150 points total out of 2400).

The remaining 1/4 of the students (roughly) are bright and motivated. These are the students for whom Prep classes are most effective. I'm talking about the kids who score around 620 - 700 per section, before they've taken any tutoring. Usually I can identify a few of their weaknesses, drill home some basic skills to shore up those problem areas, show them how to apply optimum strategy, get them attuned to the tendencies of the SAT content, and by the end of a standard 8 week course I not only hope for, but expect their scores to go up anywhere from 80 - 120 points per section. I see a handful of my students do this on every test.

Quick caveat: For those students with IQ's 3+ SD's above the mean, like for students with average or below average IQ's, SAT prep courses are redundant at best. Those kids, even if they don't get perfect scores, will do just fine, no matter what. But we knew that already.

And a final note. As much as anything, these prep classes are peace of mind. Parents and kids sleep better knowing they've done everything in their means to improve the chances of getting into the right college. $4,000 for private lessons is absurd. Exploitative, even. It's shameful what some of these companies do. But that doesn't make the entire industry without merit. Even when scores don't dramatically improve, those dollars are buying something. Else, why would people do it?

Anonymous said...

Those tests are so fair, so reliable, valid and so broad, that there is no way to game them. That is why they use them. Way back in the 80's when the top SAT was 1600, we never studied or even thought there was anything to study. Either you knew it or you didn't. We even went out the night before the test. I was not one to study and my parents didn't make me. I took no honors courses and still got a 1210, which was good enough to be accepted to the engineering program. My very sweet, straight-laced, honor society sister got a 980. She got into college, barely, because of grades. She got married straight out of college to a doctor. She is very hardworking, diligent, a great person. College material? Barely.

SKT said...

I took Kaplan and it definitely helped. I came home from school each and day and played video games. I don't think I stayed after for a single extra circular activity until my senior year. I was a good student, but not in the top 10. Yet my 1490 thanks to Kaplan got me into Cornell. It was time and money well spent.

Anonymous said...

Anecdotally, I had a girlfriend who told me she took it and got 1250, which was not good enough for her mother. Her mother sent her to a prep course and she got 1420. I took it without any prep and got 1370- 740 verbal, 630 math. I think I could have done significantly better on math with good prep, particularly since I did not take geometry in high school. Or maybe I'm just kidding myself and I'm not as smart as I would like to think I am.

Anonymous said...

Oldest trick in the book.

That doesn't mean the SAT can't be coached, though. It just takes a lot longer than most test-prep outfits would like you to believe. From my own experience I found it takes about 6 months of rigorous daily practice to boost your SAT or LSAT score significantly. (Of course if you've never seen a standardized test before, a test-prep outfit can teach you some of the most common tricks used by the testmakers, which can boost your score in a short period. But if, like most students, you're already familiar with these tricks, boosting your score much further will take a lot more work.)

Lover of Wisdom said...

Steve:

Suppose we could replace the SAT and ACT as part of the college entrance requirements with a really good IQ test. Do you think a new industry of IQ test prep would emerge, and would IQ scores improve? Perhaps like the Flynn effect but much faster?

Anonymous said...

This thread is going to turn into a SAT/IQ bragfest, so I might as well get things started. I got a 1550 in 1995 (the year after "recentering", but well before they added the essay section, and I think before they took away antonyms). My studying consisted of practicing on a bunch of those old tests from the "10 real SATs" books. Familiarity with the test, and especially with the kind of answers the reading section questions were looking for, was easily worth 50-100 points to me.

Anomalous Female said...

The PSAT, not the SAT, is the test everyone should be concerned about. It's full name is PSAT/NMSQT -- that last part means "National Merit Scholarship Qualifying Test." Though their numbers seem to be decreasing, there are still many colleges, among them Baylor, Auburn, Oklahoma State, and U of Houston, which *automatically* give out full-ride four-year scholarships to National Merit Finalists. All you have to do to become one is score high enough on the official PSAT, the one you take your junior year of high school, and submit some paperwork. Oh, and your Verbal score counts twice, Math only once.

My IQ is apparently at or higher than two SDs above the white norm, so though my brain was atrophied from K-8 mindless mush (e.g. "gifted" programs where we got to basically just screw around all day), I proved a quick study when I realized the importance of the PSAT Verbal. I missed two questions and got a 740 (this was prior to recentering). That plus my OK math score got me the NM Finalist distinction and a flood of letters from colleges offering scholarships.

To those who say SAT prep doesn't work -- I disagree. I took a practice PSAT sophomore year without studying and got something like an 1100 (I don't remember). By using the Princeton Review study guides, the 10 SATs practice tests, and by memorizing over a hundred dictionary definitions of vocabulary words, I raised my Verbal score to a nearly perfect one.

If you need further proof, how's this -- through studying the Princeton Review's PSAT Math section and their Math Smart book, I raised my Math score from 540 on the PSAT to 680 on the SAT a year later. And please know at that time I believed in the false dichotomy between math and humanities and I in no way considered myself a "math person."

Of course the caveats are that I was born with a good brain and I was extremely motivated at the time. However, at the same time I had neglected my brain for many years, had no intellectual curiosity, and scorned Math.

There is absolutely no way I could have become a National Merit Finalist if I hadn't studied for the PSAT. So, for those who are both smart enough (probably at least 115 IQ) and motivated enough, PSAT prep can be a wise investment. I am really quite surprised people would say otherwise.

Anonymous said...

It's well-known among LSAT teachers (like me) that your diagnostic score is an accurate predictor of your final score.

Diagnostic of 150: you'll be lucky to hit 160.

Diagnostic of 155: you might be able to hit 163. Outside chance of 165.

Diagnostic of 160: you'll be able to get into the 170s and go to a school that's actually worth what you'll pay.

Diagnostic of 170: you'll break the test.

I scored in the low 160s on my first, cold test. Hit the high 170s on test day. Assuming adequate prep, you can definitely improve, but almost everyone tops out at a predictable level.

-bushrod

testing99 said...

Anon is quite right, SAT can be coached.

Interestingly enough, the charity I'm involved with, is increasingly seeing Midwest and Eastern schools demand ACT scores in addition to the SAT.

[Our kids are bright/poor, nearly all Hispanic, a few Asian, kids we select for after-school enhanced curriculum in Math and Science. The kids get seminars from industry speakers (JPL, Architects, Beckman, Medtronics, etc) and lots and lots of coaching on SAT/ACT, college prep, manners/teamwork etc.]

Most of the kids go on to advanced degrees in things like Bio-medical or genetics research (MIT, Stanford) etc.

Point being we see fairly large gains in Math at least with coaching on the test over the two year cycle (we take them start of Junior Year). Gains are around 50-80 points on the Math. Only about 10-20 on the Verbal. Most of the kids don't read that much, which is probably the best prep for the Verbal part of the SAT.

Anyway, we are seeing at least a lot more schools asking for ACT out of concern that the SAT is over-coached and therefore gameable. The schools are all Midwest/Eastern.

Antioco Dascalon said...

As I previously mentioned, I teach SAT, LSAT, GRE and GMAT. The biggest flaw I see in these studies is that there is no control. They should have a randomly selected group take a $4000 prep class and another just take 10 practice SATs.
I suspect that there would be little or no difference between the groups.

Anonymous said...

"I got a 1550 in 1995 (the year after "recentering", but well before they added the essay section, and I think before they took away antonyms). My studying consisted of practicing on a bunch of those old tests from the "10 real SATs" books. Familiarity with the test, and especially with the kind of answers the reading section questions were looking for, was easily worth 50-100 points to me."

I got a 1600 and didn't study a bit!

Anonymous said...

"I got a 1600 and didn't study a bit!"

Yeah, well, I didn't study at all and then tried my hardest to flunk damn thing and I STILL got a 1900 (they gave me 300 extra points against my wishes). Can you imagine how amazing my score would have been if I'd tried to ace it?

Mr. Toad said...

I have several aunts who ought to be washerwomen, and each one of them scored 1600 on her SATs.

Anonymous said...

The ability of the test prep industry to boost scores is helping to fuel the SAT and ACT test optional movement.

http://www.fairtest.org/university/optional

On the other hand, the ability of the SAT and ACT tests to accurately measure g also give the test optional movement a boost.

Anonymous said...

testing99 said...

Our kids are bright/poor, nearly all Hispanic, a few Asian, kids we select for after-school enhanced curriculum in Math and Science. The kids get seminars from industry speakers (JPL, Architects, Beckman, Medtronics, etc) and lots and lots of coaching on SAT/ACT, college prep, manners/teamwork etc.

T99, these kids come from intact families, have the brains to succeed and still need after school help from someone like you. Don't you see this as a problem? Lucius has already pointed out on one thread that these children are almost freaks, considering that they are almost one in 9000.

Your entire charity appears to be a waste of resources directed at children who could either do well on their own or who are so rare as to be unable to improve the communities from which they come from. You are helping these exceptional children move to a Midwestern college where the young students there will think they are representative of the average Hispanic, when in fact they are literally 1 in 9000.

Don't you see what a waste your program is?

Richard Hoste said...

"Yeah, well, I didn't study at all and then tried my hardest to flunk damn thing and I STILL got a 1900 (they gave me 300 extra points against my wishes). Can you imagine how amazing my score would have been if I'd tried to ace it?"

LOL, these people are funny. I wonder what satisfaction they get talking about their supposedly high scores under the name "anonymous" on a blog.

You might as well just scribble it on a bathroom stall.

rob said...

Dilligent test prep for long periods of time raises scores a bit. That's actually a good a thing, it shows concientiousness and the ability to learn and retain material: people who are lazy or have high time preference won't do that.

I wouldn't have been surprised if the new SAT didn't show strong pracice effects. They've removed g loaded material(analogies) and replaced it with knowledge based material(vocabulary) to reduce various gaps.

How much studying for standardized tests matters depends on where you started.

For my GRE
Practice Test Real
percentile % scaled
Verbal ~83 97 710
Quant ~32 83 750

I could be off by 1 or 2 from the practice. What happened?

Mostly, I forgot how to do long division fast, lost or added a pi in geometric formulae, that sort of thing, in the years since high school. On the other hand, I took the practice test early in the morning, and I was tired.

Anonymous said...

The ACT measures knowledge, not intelligence.

Anonymous said...

Im not sure of my current IQ but when I was a kid they said I was smart. I got a 1270 on the SAT in 1994 when I ws 12 (JHU CTY program). Even split. I was sent downstairs to the high school to take a one day prep course when I was 11. first scores 630 340M. ONE DAY LATER my math had gone up to the high 600s. For some reason I did worse on the actual test than on any of the practices they gave us. Anyhow the old sat at least is definately coachable.

Anonymous said...

Richard Hoste said...
LOL, these people are funny. I wonder what satisfaction they get talking about their supposedly high scores under the name "anonymous" on a blog
.

Pffft. Spoken like some cloth-headed dullard who got a piddling 1580 on his SAT. I never even took the stupid thing and I got a score so high that it killed nearly half the people working at the ETS. It was years before they could allow anyone back on the premises, and to this day my score is kept under tight security at a military installation deep underground in a Nevada nuclear test site.

Suck on that, you patzers.

Richard Hoste said...

So, I took the GRE yesterday and got Verbal 650 and Math 780. I left the testing center telling myself I wouldn't go brag about it on ISteve but what the hey, I'm happy.

But there's a practical reason why I bring it up. Internet says I've got an IQ of 145-148 yet I have doubts about the conversions that you find online. The one I saw uses 115 as the average test taker's IQ. Charles Murray says that the actual college grad's IQ is only 105 but I assume that the average IQ of a GRE taker is higher than that of your normal college grad so 115 for the average test taker is close enough.

Both my scores are at about the 90th percentile. If we assume that GRE test takers represent the top third of the population (the amount that have college degrees) then the top ten percent of the top third only puts me at about the 97 percentile for the population in general. If that's correct then my IQ is only about 128, the same as John McCain's. He strikes me as a disgusting, blithering idiot and I really want to be smarter than him.

Here is the conversion chart that I used. All others I've found seem to be based on that.