From PsychFileDrawer:
What are important studies that your field of Psychology gives credence to, but which--as far as you know--have not been replicated in any published follow-up work? (Up to 3 votes per person--see links below to vote for one of the studies already nominated--or nominate & vote for another study of your choosing.) (What is the goal in creating this list?)
| Rank | Citation | Finding | Discussion of Study |
| 1 | Jaeggi, S., Buschkuehl, M., Jonides, J., Perrig, W. (2008). Improving fluid intelligence with training on working memory. PNAS 105(19), 6829-6833. | 10 - 20 sessions of doing n-back task raises fluid IQ as measured with Ravens | Discuss (2 postings) |
| 2 | Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006). Reducing the racial achievement gap: A social-psychological intervention. Science, 313, 1307-1310. | Writing brief paragraph on important values improves academic achievement over subsequent year | Discuss (1 postings) |
| 3 | Ramirez, G. & Beilock, S. L. (2011). Writing about testing worries boosts exam performance in the classroom. Science, 331, 211-213. | Huge exam-score increases resulting from anxiety-reducing interventions deserve replication. I | Discuss |
Via David Z. Hambrick in the NYT, IQ Points for Sale, Cheap.
How about that Abecedarian Early Intervention Project? Has it been replicated? Criticized? What does Charles Murray have to say about it? Given those promising results you would think we would be hearing more about it.
ReplyDeleteSeriously? Three out of three of them are influential, all assert that cognitive ability can be expanded through simply and easy intervention, and none can be replicated?
ReplyDeleteI'm tickled but not surprised.
(to copy an earlier email of mine)
ReplyDeleteI don't think this website is very up to date. Take #1, Jaeggi 2008. You want positive replications? Here you go: http://www.gwern.net/DNB%20FAQ#support You want some failures to replicate? Hey, I've got those too: http://www.gwern.net/DNB%20FAQ#criticism
So, uh...
"Seriously? Three out of three of them are influential, all assert that cognitive ability can be expanded through simply and easy intervention, and none can be replicated?"
ReplyDeleteI wouldn't classify n-back training as easy. It is tedious and requires plenty of motivation to keep up daily for any length of time. Whether or not it increases general cognitive ability is a subject of heated debate.
Wow, Diederik Stapel sure was robbed here. He did lots of Studies so great nobody's ever replicated em
ReplyDeleteJust the other day I thought of creating a website like this. It seems like there's so many pointless words published in the social sciences. Shouldn't psychologists be fighting over each other to create and replicate these studies on things that are actually important like cognitive improvement? What is wrong with the academics who just waste time and money by publishing so much useless nonsense?
ReplyDeleteI have to say, I just don't place any real credit in studies that support stereotype threat. Perhaps under quite constrained circumstances stereotype threat might induce real changes in performance. Yet I simply have to believe that most of those studies have little more plausibility than do reports of paranormal cognition -- which, of course, once upon a time likewise seemed to some sympathetic psychologists to have been established by (supposedly) careful experiment.
ReplyDeleteIs the desire to believe in the purely cultural basis of the IQ gap really less intensely felt by many in the social sciences than the desire to believe in telepathy among paranormal investigators?
Stereotype threat: It's certainly plausible that you can induce people to slack off on low stakes tests. But it wouldn't be ethical for experimenters to try to lower performance on high stakes tests. I don't think this conundrum is soluble.
ReplyDeleteWell, yes, there is an available explanation for the stereotype threat experiments, namely, as you indicate, that they are low stakes tests and stereotype threat diminishes motivation enough to affect performance.
ReplyDeleteBut, frankly, I don't even believe most of those results so explained.
To me, the situation is as if virtually the entire discipline of social sciences was under the sway of an overpower need to believe in telepathy. Don't you think that almost certainly we would be reading about experiment after experiment finding evidence of telepathy? Again, once upon a time, some sympathetic psychologists found supposedly powerful evidence of telepathy. They were debunked only by the strong resistance and countervailing experiments conducted by the much larger segment of the social science community which emphatically believed otherwise. How do you think that would have turned out had that opposing community not existed? How much longer would the debunking have required, if it were ever even to have come to pass?
The Test Score Culture has produced America's most incompetent elite so far. The Peter Principle is metastasizing like cancer because of test score resumes. We are catapulting noxious phony geniuses with high test scores to the top of all of our professions.
ReplyDeleteRobert McNamara was not the high water mark but just the beginning. Lincoln fired the Failures in the 1860's and won the damn war. By the 1960's it was already becoming difficult to fire the Failures. Flash forward to GW Bush and hardly anyone got fired. Obama continues the trend.
In the private sector Jack Welch last week addressed a conference of high test score zombies who rejected his policy of firing the Failures at GE.
"Stereotype threat: It's certainly plausible that you can induce people to slack off on low stakes tests. But it wouldn't be ethical for experimenters to try to lower performance on high stakes tests. I don't think this conundrum is soluble."
ReplyDeleteNot to mention getting the wrong findings would be politically untenable.
Steve:
ReplyDeleteYou could make the test somewhat higher stakes by offering to pay a reward for the highest scores. Make the reward relatively generous (say, $500) and you can probably get a good effort from a bunch of college students.
Alternatively, randomly assign your high-stakes testees to a treatment group and a control group. Give the treatment group whatever pre-test interventions are predicted by the theory and low-stakes tests to help. (In the worst case, offer the people in the treatment group a second chance at the test for free a month later.). I'm not in the social sciences, but it sure seems like you could do this ethically.
It seems "stereotype threat" would affect groups who have the lowest self-confidence such that their performance would be depressed by the negative opinions of others.
ReplyDeleteWomen and Asians are the two groups that purportedly have the lowest self-esteem. Has anyone tested these groups in comparison to other groups?
"Eeyore said...
ReplyDeleteIt seems "stereotype threat" would affect groups who have the lowest self-confidence such that their performance would be depressed by the negative opinions of others.
Women and Asians are the two groups that purportedly have the lowest self-esteem. Has anyone tested these groups in comparison to other groups?"
AIR a stereotype threat experiment on east-Asian women doing maths tests could significantly raise their performance by reminding them they were east-Asian (good at maths) and lower their performance by reminding them they were female (bad at maths).
i can teach folks how to do better at a matrix reasoning task (& smarter folks will make more gains from my teaching than dumber folks!) BUT their slightly higher score on a matrix task will NOT GENERALIZE to being smarter in the real world! ...these are simple parlor tricks to avoid the elephant in the closet - you are pre-set to have the level of cognitive skill you have.
ReplyDeleteI think the premise of this topic is exactly backwards. In psychology progress is not a struggle against ignorance. It is a struggle against resistance. It's not that people don't know, it's that people don't want to know.
ReplyDeleteI read Jensen's 1969 article in The Harvard Education Review on the prospect of educating black kids. His findings and conclusions could not have been more clear. For most of my lifetime it seems everyone has tried to replicate that paper out of existence.
That same year the Westinghouse Study on Head Start was published. Again the results were clear. Head Start didn't work. Yet just yesterday the Obama administration published their "Julia" ads attacking Mitt Romney over his lack of enthuiastic support for Head Start. Romney is right in backing away from Head Start but he will probably have to recant.
The very first item in Julia's life is enrollment in Head Start. That means that Obama is running for President on his unwavering support of this long discredited program. Actual truth and facts don't seem to count for anything.
The American people desparately want early childhood intervention to work. It's an idea that resonates with the electorate. Americans want everyone to have an equal chance and that's just what Head Start promises. It is not important at all that it doesn't actually work.
Albertosaurus
gwern, you say that the first study on the list has been replicated--making the top-20 list generated by users of the website "not up to date". Uhhh, I clicked your link, and all it does is to mention something called "Brain Workshop", which seems to be a game of some sort. You understand that saying "Brain Workshop" is not the same as providing a citation to a published article in the scientific literature, right? So how does what you've said imply the people voting on the website are not up to date?
ReplyDeleteAnyway, the comments on the website provide what sounds like extremely up to date (but also relevant) information on the replicability of the study:
http://www.psychfiledrawer.org/top-20/discuss.php?nomination=1
If kids can be fooled that Air Jordans will help them jump higher, maybe someone should market Air Heads in the hope of making kids think faster.
ReplyDeleteHow about silicon implants for heads?
ReplyDelete"I seek the grail."
ReplyDelete"How about that Abecedarian Early Intervention Project? Has it been replicated? Criticized? What does Charles Murray have to say about it? Given those promising results you would think we would be hearing more about it."
ReplyDeleteI was surprised that Abecedarian was cited. The result have been contested. See, for example, Spitz (1997) "Some questions about the results of the Abecedarian Early Intervention Project cited by the APA Task Force on Intelligence." The basic problem, as Spitz points out, was that the control-experient difference was found just months after intervention. You have a 6 point increase on initial testing and no subsequent change. While it's possible that the beneficial effect occurred early on -- the study researchers didn't thinks so -- it's also possible that the groups weren't as random as thought.
"Comments on the results used by U. Neisser et al (see record 83-26553) in the American Psychological Association (APA) task force study on intelligence, which cited the Carolina Abecedarian Project (C. T. Ramey and F. A. Campbell, 1987) as an example of a successful early intervention program that produced larger and more lasting effects on intelligence scores than did Head Start programs. The author claims that Neisser et al "shortchange" the Carolina project by mentioning results occurring only at or after age 2, although startling test score differences were apparent in Ss by age 18 mo. (PsycINFO Database Record (c) 2012 APA, all rights reserved)"
Shampoo: please don't be condescending when I obviously know the topic far better than you. Brain Workshop implements dual n-back among other modes, and has been used in studies since it is Free and customizable.
ReplyDeletepsychology shills for environmental arguments to keep itself alive. psychology forgets to remember (sir William of) occam's razor:
ReplyDeleteGENETICS > elaborate sociocultural-environmental hypotheses!
cheers! panjoomby
I had been very hopeful that the Daviess County long-running experiment of "brain teaching" - every kid had chess, music, and a foreign language for their entire K-12 career (folk dancing was added later) would show at least modest results. At first, it looked that they were on track for that.
ReplyDeleteBut it ultimately didn't pan out.
I keep hoping.
please don't be condescending when I obviously know the topic far better than you.
ReplyDeleteTry not to post obvious nonsense.
the stanford prison experiment.
ReplyDeleteHow about that whole inoculation and smallpox thing? Has it ever been replicated?
ReplyDeleteThe Test Score Culture has produced America's most incompetent elite so far.
ReplyDeleteWas just reading some Charles Murray on assortative mating and social stratification, and had exactly the same thought. They're marrying each other, getting richer than ever, blocking others out from the plums -- and screwing up like never before. As Glenn Reynolds (yes Steve, I know) keeps saying, they're the worst elites we've ever had.
I added Eyferth to their list...
ReplyDeleteHow about that whole inoculation and smallpox thing? Has it ever been replicated?
ReplyDeleteSo many times that we'll never do it again. Barring acts of malice and God.
>they're the worst elites we've ever had.<
ReplyDeleteThis is true only if you assume that the job of the elites is to look after us. They are doing just fine for themselves. Their own world is a good one.
"Avondlander said...
ReplyDeleteI added Eyferth to their list..."
Good one. This study of the IQs of half-black children of NATO soldiers in Germany was published over a half century ago, so it's sort of about time for it to be replicated.
About the Eyferth study, forget replication. I want a followup. Adoption studies in the US have shown that minimizing the contact a black child has with other blacks can raise his IQ to near-white levels during childhood. Follow-up research on the study participants showed that blacks and mulattoes didn't stay uplifted as adults and were no brighter than black-raised blacks and half-blacks.
ReplyDeleteFinding out how well the Eyferth kids did later in life could go a long way to demonstrating that the white-black IQ gap is partly environmental. That's the sorta thang could make a scientist all famous and shit. I'd bet a dollar that there's been some followup, the results weren't what Eyferth wanted to see, and the study sits in a drawer somewhere in his office. Assuming the first study was legit, that is.