Report: International tests severely misrank U.S. students


Authors confused why PISA releases its first data sets on averages; say its misleading.
“What’s puzzling is why international tests like PISA release overall average scores first, then more nuanced data weeks or months later, since this promotes misleading analyses,” said a co-author of the report.

Prominent international tests skew comparisons of test scores, and U.S. student performance actually ranks much higher than believed, according to a new report released by the Economic Policy Institute (EPI).

The truth, says the report, is that—when comparing apples to apples in weighing U.S. student performance against that of other industrialized countries—U.S. students don’t rank 25th in math, but 10th; and in reading, the country is not 14th, but 4th.

The report, “What do international tests really show about U.S. student performance?,” is the first of its kind to detail what it claims is an inaccurate analysis of student performance on international tests such as the Program on International Student Assessment (PISA) and Third International Mathematics and Science Study (TIMSS).

The report’s analysis found that average U.S. scores in reading and math on the PISA are low partly because a “disproportionately greater share of U.S. students come from disadvantaged social class groups, whose performance is relatively low in every country.”

When differences in countries’ social class compositions are adequately taken into account, the report says, the performance of U.S. students in relation to students in other countries improves markedly. “Errors in selecting sample populations of test-takers and arbitrary choices regarding test content contribute to results that appear to show U.S. students lagging,” it says.

(Next page: In-depth findings)

“We disaggregated the numbers by social class,” said Richard Rothstein, a research associate at EPI and co-author of the report. “What’s puzzling is why international tests like PISA release overall average scores first, then more nuanced data weeks or months later, since this promotes misleading analyses.”

Rothstein revealed that, when compared to other countries, the achievement gap between students in the U.S. is extremely small, and in fact, the most disadvantaged students’ scores have been increasing, and continue to increase, over the years; whereas disadvantaged students’ scores from top-performing countries like Finland have not—they’ve decreased.

“What this report does is challenge the easy assumption that following Finnish education leads to success. When you adjust the data so that disadvantaged and lunch-eligible students are re-weighted, U.S. student performance goes from 25th place in math to 10th; and from 14th place in reading to 4th,” he said.

“We’re not attacking any tests,” said Martin Carnoy, a professor of education at Stanford and co-author of the report. “We’re simply showing that disaggregating by social class reveals much more policy-relevant information.”

For example, Carnoy said, according to PISA’s reading scores, economically advantaged students in the U.S. score as high as Finland, Canada, and Korea—PISA’s top-performing countries. Also, disadvantaged U.S. students score higher than France, Germany, and the U.K.

“You can’t compare nations’ test scores without looking at the social class characteristics of students who take the test in different countries,” said Carnoy. “Nations with more lower social class students will have lower overall scores, because these students don’t perform as well academically, even in good schools. Policy makers should understand how our lower and higher social class students perform in comparison to similar students in other countries before recommending sweeping school reforms.”

Carnoy suggested that U.S. education policy might be influenced by inaccurate data, noting that because Education Secretary Arne Duncan “talks about how we’re making small gains,” he’s not receiving accurate information—and therefore policy directions “should be changed.”

Conclusions based on these averages “ignore the complexity of test results and may lead policy makers to pursue inappropriate and even harmful reforms,” said Rothstein, who also is senior fellow at the Chief Justice Earl Warren Institute of Law and Social Policy at the University of California-Berkeley School of Law.

The report describes how, in TIMSS reports from 1999 to 2007 (2011 data will be analyzed when available), disadvantaged U.S. students have made consistent gains.

“While it’s not my place to comment on policy changes, I will say that average scores are not sufficient on which to base sound education policy,” said Hans Wagemaker, executive director of the International Association for the Evaluation of Educational Achievement. “What’s useful about this report is that the discussion moves away from averages and uses trend data as compositional data over time.”

(Next page: ‘Arbitrary’ decisions about content by designers of the tests?)

The report’s authors say the differences in average scores on these international tests reflect arbitrary decisions about content by the tests’ designers.

For example, although it has been widely reported that U.S. 15-year-olds perform worse on average than students in Finland in math, U.S. students perform better than Finnish students in algebra but worse in number properties (such as fractions). If algebra had greater weight in these tests and numbers less weight, test scores could show that overall U.S. student performance was superior to that of Finland.

The report also found:

  • The highest social class students in the U.S. do worse than their peers in other nations, and this gap widened from 2000 to 2009 on the PISA.
  • U.S. PISA scores are depressed partly because of a sampling flaw resulting in a disproportionate number of students from high-poverty schools among the test-takers. About 40 percent of the PISA sample in the U.S. was drawn from schools where half or more of the students are eligible for the free lunch program, though only 23 percent of students nationwide attend such schools.
  • There is an achievement gap between more and less disadvantaged students in every country; surprisingly, that gap is smaller in the U.S. than in similar post-industrial countries, and not much larger than in the very highest scoring countries.

The EPI sent its report to Andreas Schleicher, deputy director for education and special advisor on education policy to the Organization for Economic Cooperation and Development (OECD), which produces the PISA.

“OECD/PISA shares the view expressed in the paper about the importance of examining countries’ performance levels from various perspectives. Indeed, one of the five volumes in which the initial results from PISA 2009 are published is dedicated to the examination of performance by various background characteristics of students, schools, and countries,” said Schleicher. “This is not acknowledged in the paper. More importantly, the Carnoy/Rothstein paper contains several fundamental misunderstandings and misinterpretations of the PISA data. In particular, the paper claims that there are flaws in PISA samples, which is simply incorrect and unsupported in the paper.”

To read Schleicher’s detailed response, as well as EPI’s counter-response, click here (PDF).

To read EPI’s full report, click here.

EPI’s report comes as the administrators of TIMSS are preparing to make public more detailed data underlying the 2011 test results, following last month’s release of average national scores. PISA plans to release detailed data about its 2012 test in December 2013.

Carnoy said he and Rothstein then will be able to supplement their current report by comparing the most recent TIMSS and PISA results by social class across countries. He invites other researchers to conduct similar analyses, to see if their findings confirm those of the current report.

Meris Stansbury

Want to share a great resource? Let us know at submissions@eschoolmedia.com.

Comments are closed.