December 11, 2012

How much can you trust international school achievement tests?

Commenter TH compares 2011 results on TIMSS / PERLS to 2009 results on PISA:
I calculated some correlations between the PISA 2009 (15-y/o's), TIMSS 2011 (8th grade), and PIRLS 2011 (10-y/o's). 
The correlation between PISA 2009 math and TIMSS 2011 math is 0.87 (n=26). 
In both studies, East Asians are at the top, white-majority countries at the middle, and others at the bottom. 
However, if you look only at white-majority countries, the correlation is 0.19 (n=13). Russia and Israel do particularly well in the TIMSS compared to the PISA. The former is supposed to be a more math-heavy test compared to the latter which is a test of "mathematics literacy".

So, on the big picture, PISA and TIMSS are pretty much in agreement on the global racial hierarchy of math smarts. On the other hand, on the small picture of how white countries are doing, it's pretty much of a mess. There could be a lot of reasons for this, some inevitable (the test has to choose what to test on by a certain grade, which might not be what that country teaches up to that point), some potentially fixable (one country might try really hard to get students to work diligently on the test, another might treat it as just another test, and a lower stakes one than most).
In the US, the racial breakdown of the TIMSS scores in grade 8 is as follows (SD=100, global baseline = 500)
White 530
Black 465
Hispanic 485
Asian 568
Multiracial 513 
Hispanics slightly outscore Norway and Sweden in the TIMSS, while Norway and Sweden score only slightly (0.1 SD or so) higher than US blacks. In the PISA math test, Norway and Sweden outscored US Hispanics by 0.3-0.4 SD and US blacks by about 0.7 SD. 

What about for reading?
The correlation between PISA 2009 reading and PIRLS 2011 is 0.81 (n=36). Among white-majority countries (n=18) the correlation is 0.24. 
In the US, the racial breakdown of the PIRLS scores is as follows (SD=100): 
White 575
Black 522
Hispanic 532
Asian 588
Multiracial 578 
The black average is higher than that of, for example, France, Spain, Norway, and Belgium. In the PISA reading test, each of those four countries outscored US blacks by more than 0.5 SD.


10 comments:

  1. The lower correlation is probably a result of smaller sample sizes and less overall variance. The range of scores between wealthy white countries isn't nearly so large as the range of scores overall, so the correlations are weaker.

    Imagine a chart made by firing 4 or 5 shotgun blasts at different points along a diagonal line. The x-axis is one test's scores, the y-axis is the other test's scores. Overall, the correlation will be relatively strong, but within each grouping, the correlation will be pretty low.

    Being surprised by the lower correlation is an example of ignorance of De Moivre's equation.

    ReplyDelete
  2. Could it be that each country decides on what level of competence is competent?

    ReplyDelete
  3. Reminds me of those millions of "engineers" educated in China and India we heard about a few years ago.
    In the US, an engineer has a four year Bachelor of Science degree.
    China and India counted air conditioning techs, auto mechanics, computer repair techs, etc. as engineeers.
    Vivek Wadhwa is on YouTube somewhere talking about this in great detail.

    ReplyDelete
  4. In the US, the racial breakdown of the PIRLS scores is as follows (SD=100):

    White 575
    Black 522
    Hispanic 532
    Asian 588
    Multiracial 578


    Multiracial is surprisingly high. Could it be comprised mainly of mixed white-Asian young people?

    Peter

    ReplyDelete
  5. Well, Conseravtives that like immirgation for developers like Perry will have a headache. A article below explains.

    The more than 600 Texas school districts suing the state over funding wrapped up their case Wednesday, with a pair of witnesses saying that low-income students and those requiring additional English-language instruction are the most likely to fall through the cracks amid deep budget cuts.
    The case has hinged on the fact that Texas has seen a major spike in the number of minority students and those from poor families over the past decade. A booming Hispanic population has meant a dramatic rise in the number of students who need extra instruction in English.
    Districts say such students cost more to educate, but the state Legislature voted in 2011 to cut $5.4 billion in funding to public schools.
    That prompted a series of lawsuits from districts educating about three-fourths of the state’s more than 5 million students. They say the cuts have been especially devastating because they come amid enrollment increases and as Texas implements new student accountability measures built on a more-difficult standardized test.

    Read more here: http://www.star-telegram.com/2012/12/05/4463006/plaintiffs-wrap-up-case-in-school.html#storylink=cpy

    Share this:
    Facebook



    ReplyDelete
  6. Is Texas lying , how could you go from 53 percent to 82 percent of hispanic graduation?

    For the second straight year, Texas has the lowest percentage of high school graduates in the nation, according to a U.S. Census Bureau study released Tuesday.

    For Texans over age 25 the rate of having a high school diploma is white 91%, Asian, 89%, black, 83%, and Hispanic 51%. Note that the Houston Chronicle labels whites as "Anglos" even though I'm sure many of the whites in Texas are Saxons, Juts, and maybe even French!

    While some of these Hispanics came to the US as adults without high school educations many more were either born here or went to school here after coming to the US legally or illegally.


    "Nationally 85 percent of young people under 18 who are Hispanic are born in the U.S.," Montecel said. "The issue of dropouts is an issue of how well schools in the state are able to educate Latino students rather than a question of immigration. It is in our schools. That's where we need to focus."

    Notice Montecel's spin: The question not whether Hispanics are willing or able to learn. It is whether the schools have the right abilities needed to teach them.

    The article says California ranks 42nd in the nation in graduation rates. California is bifurcating into a high tech highly skilled upper class and a third world lower class. My guess is that this is not a sustainable condition. Higher taxes, crowding, and crime will continue to drive out the white middle class. The resulting loss of tax revenues will lower the quality of services while at the same time leading to higher taxes.

    The 51% figure for Hispanics is in line with the 53% figure measured for national high Hispanic school graduation rates. Note that 53% figure is a measure of how many kids from grade school make it through to a high school diploma. So it can not be attributed to the immigration of adults who lack education (though of course if we had high minimal educational standards for immigrants their children would do better in school than the children of most of our current immigrants).

    Unfortunately, as Samuel P. Huntington has noted, later generations of Hispanics do not improve their scholastic performance by much.

    Update: When I see poor academic performance of some group I think of costs. Aside from higher taxes, crime, and crowding how does the demographic trend caused by immigration create costs? One of the biggest costs that does not get as much attention as it deserves is the rising costs of racial quotas for jobs, college slots, and other opportunities. There is the opportunity cost of jobs that more qualified people will be shut out of. But there is also the cost in inefficiency due to the inability of organizations to fire people in important positions. Plus there is the cost in the increasing divisiveness of American politics as the political parties split over race. That is going to approach levels that Americans today can scarcely imagine.



    More Sharing ServicesShare|Share on facebookShare on twitterShare on emailShare on print

    By Randall Parker at 2004 July 08 02:48 PM Immigration Societal Decay
    Graduation rates for white students increased 0.4 percentage points to 92 percent, and 2011 Hispanic graduation rates increased 3 percentage points to 81.8 percent. Black student graduation rates rose to 80.9 percent from 78.8 percent in 2010





    Comments

    John S Bolton said at July 8, 2004 10:34 PM:

    ReplyDelete
  7. In Texas, public schools lose funding when students drop out. So administrators are motivated to find creative ways to account for kids who leave their schools.

    The state has a history of funny dropout numbers.

    Back in 2000, Houston schools were top performers nationwide, hailed as a Texas “miracle,” and used as the model for the federal No Child Left Behind Act. President George W. Bush, impressed by Houston’s success, appointed the city’s school superintendent, Rod Paige, as his secretary of education in 2001.

    But in 2003, Robert Kimball, then an assistant principal at Houston’s Sharpstown High School, learned that his school was planning to report zero dropouts, even though he knew there were dozens of kids who had left. He went to the press.

    His charges spurred a state audit of 16 Houston high schools that found more than 3,000 dropouts hiding in the data. “The pressure on school principals is intense,” Kimball told FRONTLINE. “If they don’t get that dropout rate down, and they don’t get their students performing well, then they get fired.”

    Since then, Kimball said, the culture among top officials in the district has changed — administrators have shown much lower tolerance for false data. Schools also must provide documentation for where students go, making it tougher to fudge the numbers.

    But the problem hasn’t gone away. Earlier this year, two Houston principals made headlines when they were indicted for falsifying dropout data at a middle and high school. Investigators alleged they were forging documents to show that students had transferred out of state.

    “Every time we crack down on some of these practices of skirting around the true dropout rates, people come up with a new out,” said Robert Sanborn, the president of Children at Risk, a nonprofit education-policy group based in Texas.

    Sanborn said that overuse of the “private school” code has become one of the latest ways that Texas high schools can lower their dropout numbers.

    But these private schools aren’t what you think.

    The state is home to at least 10 programs that offer a high-school diploma for a few hundred dollars and some completed workbooks, or an exam, submitted online.

    In Miles’ program, students pay $300 to take a 200-question test. One they pass, they receive a diploma. Miles said his staff offers to go over the test material ahead of time in the office with the students, to ensure they’re prepared to pass the test. He’s graduated about 2,000 people so far.

    ReplyDelete
  8. The lower correlations among majority-white countries is of course partly due to the fact that most them have rather similar scores, so their rankings easily change places. However, I included comparisons with US racial minorities because they show that there are actually substantial differences between the results of different tests. Some countries, including Russia, Israel, and the US, do a lot better in the TIMSS and PIRLS than in the PISA when compared to many Western European countries. The TIMSS and PIRLS (both conducted by the IEA) are supposed to test "core curriculum skills", whereas the PISA (conducted by the OECD) is supposed to be more abstract and less dependent on what is taught in different school systems.

    A big problem with these international comparisons is that from a purely psychometric perspective such comparisons are often invalid. This study showed that the TIMSS math test is frequently not measurement invariant across different countries, especially if cultural differences are large. This means that the test is consistently biased against some countries when compared to others, and it therefore cannot be used to show that there are real differences in math skills unless measurement invariance is tested and found to hold.

    ReplyDelete
  9. "ironrailsironweights said...

    In the US, the racial breakdown of the PIRLS scores is as follows (SD=100):

    White 575
    Black 522
    Hispanic 532
    Asian 588
    Multiracial 578

    Multiracial is surprisingly high."


    This is probably because it is higher-IQ mixed people who most are likely to identify as multiracial (as I often do).

    ReplyDelete
  10. From Anonymous's comment about Texas dropouts and fraud:

    His charges spurred a state audit of 16 Houston high schools that found more than 3,000 dropouts hiding in the data. [....] Since then, Kimball said, the culture among top officials in the district has changed — administrators have shown much lower tolerance for false data. [....] But the problem hasn’t gone away. Earlier this year, two Houston principals made headlines when they were indicted for falsifying dropout data at a middle and high school. [....] “Every time we crack down on some of these practices of skirting around the true dropout rates, people come up with a new out,” said Robert Sanborn [....]

    So when the first guy says the culture has changed, what he really means is that it hasn't changed at all. Everybody got that?

    ReplyDelete

Comments are moderated, at whim.