How can you be confident that local officials didn't pull any fast ones with their PISA results? Well, you can't, but you can get some sense of how much room there is to pull the wool over your eyes by looking at the response rate.
Large countries have to test at least 4,500 students, and the sample is supposed to be carefully designed to represent the entire country's 15-year-olds. But projected coverage usually turns out less than perfect. For example, countries can exclude students with disabilities. This sounds reasonable -- it's hard for a blind person to take a pencil and paper test. But, what about cognitive disabilities, such as not being very bright? From the federal government's website on PISA:
PISA 2012 is designed to be as inclusive as possible. The guidelines allowed schools to be excluded for approved reasons (for example, schools in remote regions, very small schools, or special education schools). Schools used the following international guidelines on student exclusions:
Students with functional disabilities. These were students with a moderate to severe permanent physical disability such that they cannot perform in the PISA testing environment.
Students with intellectual disabilities. These were students with a mental or emotional disability and who have been tested as cognitively delayed or who are considered in the professional opinion of qualified staff to be cognitively delayed such that they cannot perform in the PISA testing environment.
Students with insufficient language experience. These were students who meet the three criteria of not being native speakers in the assessment language, having limited proficiency in the assessment language, and having less than 1 year of instruction in the assessment language.
Overall estimated exclusions (including both school and student exclusions) were to be under 5 percent of the PISA target population.
Buried in a PISA appendix entitled Annex 2A are PISA figures for what percentage of the target populations of 15-year-olds didn't get tested. America didn't come close to getting 95% representation, and many Third World countries were far worse.
"Coverage Index 3: Coverage of 15-year-old population" shows what percentage of the cohort are represented if the test taking sample was projected to the whole country. I subtracted this percentage from 100% to come up with the % Missing index. For example, Costa Rica only managed to test half the people they were supposed to, and Albania only tested 55%. Vietnam, which made a splashy PISA debut with high scores, somehow couldn't find 44% of their 15-year-olds. At the other end, the dutiful Dutch managed to test slightly more students than were thought to be around.
| % Missing | |
| Costa Rica | 50% |
| Albania | 45% |
| Vietnam | 44% |
| Mexico | 37% |
| Colombia | 37% |
| Indonesia | 37% |
| Turkey | 32% |
| Brazil | 31% |
| Thailand | 28% |
| Peru | 28% |
| Uruguay | 27% |
| Liechtenstein | 25% |
| Bulgaria | 23% |
| Shanghai-China | 21% |
| Malaysia | 21% |
| Argentina | 20% |
| Kazakhstan | 19% |
| Macao-China | 19% |
| Hungary | 18% |
| United Arab Emirates | 17% |
| Canada | 17% |
| Chile | 17% |
| Hong Kong-China | 16% |
| Czech Republic | 15% |
| Serbia | 15% |
| Latvia | 15% |
| Lithuania | 14% |
| Jordan | 14% |
| Australia | 14% |
| Italy | 14% |
| Greece | 13% |
| New Zealand | 12% |
| Korea | 12% |
| Austria | 12% |
| Portugal | 12% |
| Spain | 12% |
| France | 12% |
| United States | 11% |
| Chinese Taipei | 11% |
| Poland | 11% |
| Luxembourg | 11% |
| Montenegro | 10% |
| Israel | 9% |
| Denmark | 9% |
| Japan | 9% |
| Ireland | 9% |
| Slovak Republic | 9% |
| Tunisia | 9% |
| Switzerland | 9% |
| Norway | 8% |
| Estonia | 8% |
| Russian Federation | 8% |
| Iceland | 7% |
| Sweden | 7% |
| United Kingdom | 7% |
| Slovenia | 6% |
| Qatar | 6% |
| Croatia | 6% |
| Germany | 5% |
| Singapore | 5% |
| Belgium | 5% |
| Finland | 4% |
| Romania | 4% |
| Cyprus | 3% |
| Netherlands | -1% |
In general, Third World countries were bad at getting good coverage, suggesting that the First World v. Third World gap is even larger than the test scores imply.
Top scorer Shanghai missed 21%, so we should take its flashy scores with a few grains of salt.
America was at 11% missing, down from 18% missing in 2009, which may account for the slight decline in U.S. scores?
Consistent high-flier Finland had only 4% missing, so they aren't cheating on this measure more than the competition is.
Top scorer Shanghai missed 21%, so we should take its flashy scores with a few grains of salt.
America was at 11% missing, down from 18% missing in 2009, which may account for the slight decline in U.S. scores?
Consistent high-flier Finland had only 4% missing, so they aren't cheating on this measure more than the competition is.
A major question is how random were the missing test-takers. If the missing were purely random, then no harm no foul. But of course, many of the missing are dropouts, or in special day classes, or in juvy hall, or whatever.
This may help excuse slightly Argentina's horrible scores. The Argentineans misplaced only 20% of their 15-year-olds compared to the 37% of Mexicans who went missing.
Costa Rica 50%
ReplyDeleteROTFL
This comment has been removed by the author.
ReplyDeleteI don't believe for a minute that only 21% of Shanghai's kids were missing from the data. Migrant laborers are ubiquitous in China's big cities, yet are not counted as residents. They have higher fertility than native Shanghainese - who have the lowest fertility in China, if not the entire world - and far lower standards when it comes to education.
ReplyDeleteMy bet is that they weren't even included as residents for the purposes of this test. The rational basis behind their exclusion would be that since they don't have the hukou they aren't Shanghainese, but they do in fact live there. The results would change the scores in a similar manner to removing all the illegal immigrants' children's scores from the data in Los Angeles.
China has a residence permit "Hukou" system which allows migrants to work in the cities, but forces them to live in suburbs. Their children probably don't go to school in Shanghai because they're not registered in the city. I'd also guess that the typical migrant couldn't afford to live in the city even if they had a residence permit. I know it's common for Chinese workers to leave their children with their spouse and migrate for work.
ReplyDeleteShanghai is an exceptionally expensive city and the financial capital of the country. Those being tested are probably from the top 5-10% of China's socioeconomic ladder.
A better question is how well Chinese in the rural villages would score. There was some data from the last PISA which indicates Chinese village children scoring near the Western mean, but it's not clear what percent of the children were tested. In China, it's common for village kids to leave school at a young age to work, so I'm not sure how well a more representative sample would score.
I've seen data which show rural Chinese kids are extremely underrepresented in China's top colleges, which are entered into through test scores. I've seen other data which show an enormous (larger than the black-white gap) cognitive gap between even young kids from rural and urban China. It'd be interesting to find out how much of this is nurture (higher accessibility to cram schools and tutors among the urbanites) and how much is nature (ie IQ).
Taiwan was built by Chinese peasants, but they were mainly from Fujian. Fujian and the other southeastern provinces historically were the wealthiest in the nation and produced the majority of the successful imperial exam takers. How representative they are of the interior, I don't know.
I was surprised to Vietnam score so highly on the PISA. I wonder if the test was disproportionately given in predominately Chinese schools in the big cities. Remember, a substantial percentage of Vietnam's urban population are from the ethnic Chinese minority. When I read that only half of Vietnam's pool of potential test takers were tested, it introduces a strong possibility that of the government deliberately skewing the results.
In 2015 China will reportedly participate en bloc, though I'm pretty sure if they score well the haters will still find an excuse or thirty!
ReplyDeleteThis may help excuse slightly Argentina's horrible scores. The Argentineans misplaced only 20% of their 15-year-olds compared to the 37% of Mexicans who went missing.
ReplyDeleteWhat's the deal with Romania? They only missed 4% and scored below Hispanics and the UAE. They almost scored as low as African Americans.
Fujianese and Cantonese Chinese are also the dominant groups in the Chinese diaspora. For instance, almost everyone in my building (I live in Flushing, NYC) is Fujianese. Almost all Chinese in Singapore and Malaysia were native Hokkein (a Fujianese dialect) speakers 50 years ago. The Singaporean government Mandarinized the entire country for economic reasons. This included surnames--my husband is Malaysian Chinese and the Malaysian side of the family still uses the Hokkein pronunciation of their name; the ones who are in Singapore use the Mandarin pronunciation.
ReplyDeleteI don't know how representative people from Fujian and Guandong provinces are of China as a whole.
Ethnic Chinese (Hoa) were mostly run out of Vietnam when the commies took over, so they're less than 1% of the population now, compared to 14% in Thailand. It'd take some horrific testing methodology for that to be a factor.
ReplyDeleteI'm Vietnamese-American myself and went to schools that were about a quarter Vietnamese-American and a quarter white, and there were always way more of us in the honors classes. My impression is we're a bit more slack than the Chinese or Koreans, but we're still solidly in the Asian cluster when it comes to academics.
Steve, I might look at a linear regression equation later to try and predict what the PISA score "would've" been adjusted for participation rates in different countries.
ReplyDeleteI'll bet the Asian countries'll still come through fairly strong, but we may see them fall to have a smaller advantage over the FinnStonians.
""""What's the deal with Romania? They only missed 4% and scored below Hispanics and the UAE. They almost scored as low as African Americans."""
ReplyDeleteSince they only missed 4% that means they included lots of Roma who are not known for their intellectual skills.
But with the EU now allowing Roma to move to richer countries with richer pick pocketing opportunities the Romanian scores might be increasing. Might be a good time to be a minister of education in the Romanian government who can take credit for the increase in scores.
on group testing days (& more so if the tests take a few days) the least able are the least likely to show up for all the testing. SO some data must be "weighted" e.g., you count a low scoring kid as equal to 1.4 kids instead of 1. this is done with some IQ tests, b/c it's harder to get low ability/low SES kids to show up!!
ReplyDelete-1% missing in Holland! Do kids magically appear out of nowhere on test day or do they score the smart kids twice?
ReplyDeleteSomeone on AR noticed that black Americans > Mexico. "God help us" was his reaction.
ReplyDeleteIf 37% of Mexico's 15-yos were missing in terms of random distribution of this test, it means that Mexico is even worse than it seems.
I'm surprised Costa Rica heads the list with 50%. This is not a 3rd world country. It is a country that made a decision decades ago to devote its budget to culture and education, instead of military. Generally they are well educated. I think perhaps they have different ideas of whether it's worth testing.
ReplyDeleteTalk about jumping to conclusion, Steve seem to be running out of reasons to dismiss the gap between Asians and white.
ReplyDeleteYou don't get the point. Nobody cares about PISA and TIMSS. Nobody is trying to manipulate scores at all.
ReplyDeleteYour data shows that 21% Malaysia was absent on PISA day. Malaysia is 28% Chinese. And has a PISA score of 420 with 0.9% reaching level 5 and 6.
By any possible estimate, the Chinese score is below 475. Even if I look at the distribution of score in Malaysia and only consider top 30%, it will still show 475.
If I consider that bottom 21% students in Malaysia were absent on PISA day, the scores for Chinese in Malaysia will be below African Americans at 430. And this is the maximum possible score (the abosulte upper limit by assuming that "all Chinese outscore all Malays").
It does not make sense that East Asians can score below African Americans or even Europeans.
Most probably the schools that were tested declared "school vacations" for 15 year old that were not supposed to participate in PISA. And those kids went on a vacation and you had mass bunk at many schools.