New report IDs problems with international school rankings


A new study exposes mistakes often made when interpreting test data.

A new report that tackles a number of hot-button education issues argues that U.S. academic performance might not be as poor as originally thought when compared to other countries—and that the Common Core standards might not have the impact many are hoping.

The 2012 Brown Center Report on American Education is organized into three sections: Common Core State Standards, achievement gaps in the National Assessment of Educational Progress (NAEP), and international test scores and rankings.

In the report, author Tom Loveless, senior fellow at the Brown Center on Education Policy, notes that education has not yet been an important issue in the Republican nomination campaign and “is unlikely to be a prominent issue in the fall general election,” although the report covers topics that will need presidential attention.

International assessments

The report says educators and policy makers often misinterpret international test scores because of dubious conclusions of causality, rankings, and what Loveless calls the “A+ Country Fallacy.”

The errors are usually committed by advocates of a particular policy position who selectively use data to support an argument, he noted.

For more education reform news, see:

Beyond ‘Superman’: Leading Responsible School Reform

“Dubious causal conclusions” refers to attributing a change in test scores to a particular policy change.  The case of Poland is used to illustrate. It accomplished large gains on the PISA reading test. But the theory that tracking reform produced the gains is not supported by the evidence, the report argues.

When it comes to international rankings, Loveless demonstrates how rankings can distort a nation’s relative standing by exaggerating small changes in test scores or the reverse, making large changes appear less significant than they really are.

“Rankings are not equal interval—they differ in various parts of the distribution—so a nation may jump several rankings with a gain that is actually smaller than that of a country whose ranking stays the same,” Loveless notes.

In the study, Loveless uses 2007 Trends in International Mathematics and Science Study math scores from a number of countries. While the U.S. ranks near the bottom, a closer look at the data reveals that “the U.S. performance is indistinguishable statistically” from some of the countries included in the rankings—essentially boosting U.S. performance.

“A statement that we can be confident in making, then, is that the U.S. scored below eight countries, the same as four countries, and above twenty-three countries in mathematics,” Loveless writes.

The “A+ Country Fallacy” refers to the habit of pointing to high-performing countries and assuming that their policies must be good. Loveless calls this a “misguided” step, because it combines the errors of dubious causal claims and misusing rankings. Combining those two mistakes with a third error of omitting or ignoring evidence from low- or average-scoring countries leads to further misinterpretation, he writes.

For more education reform news, see:

Beyond ‘Superman’: Leading Responsible School Reform

Common standards

The Common Core State Standards are intended to improve U.S. education by identifying a more rigorous curriculum, setting higher expectations for states and improving efficiency through standardization.

Critics have argued that the common standards’ “one-size-fits-all, centrally controlled curriculum does not make sense,” the report notes.

While 46 states and the District of Columbia have signed on to the Common Core standards, Loveless argues that the standards will have “little to no impact on student learning.”

This conclusion emerged after analysis of states’ previous experiences with standards, combined with years of National Assessment of Education Progress (NAEP) scores.

The report examines data on the effectiveness of state curriculum standards and reveals that:

•    The quality of state standards is not related to state achievement.
•    The rigor of state standards is also unrelated to achievement.
•    The ability of standards to reduce variation in achievement is also weak.

“If you look at the past history of how standards affect achievement, we conclude the Common Core will probably have very little effect,” Loveless said.

The achievement gap

The NAEP actually has two different tests—the Long-Term Trend NAEP and the Main NAEP. Loveless aims to determine if the two tests report similar achievement gaps and how these data might affect education policy.

The Main NAEP consistently reports larger socioeconomic status achievement gaps than the Long-Term Trend NAEP. The study examines gaps between students who qualify for free and reduced lunch and those who do not; black and white students; Hispanic and white students; and English language learners and students who are not English language learners.

Loveless argues: “The biggest discrepancy between the tests is with [English language learner] students. That suggests that the role language plays on the two tests—which is quite different, even in math—may be influencing the magnitude of the gaps.”

Content differences might be a factor in determining why the tests differ in measuring socioeconomic status achievement gaps.

“The Main NAEP was designed to assess different skills and concepts than the LTT NAEP, which had a nearly twenty-year track record when the Main NAEP was first launched in 1990. In math, for example, the LTT NAEP focuses more on computing with whole numbers and fractions; the Main NAEP on how students apply mathematics to solve problems. In reading, the LTT NAEP presents shorter passages, more vocabulary words in isolation, and more items asking students to identify the main idea of a passage. The Main NAEP has a broader selection of literary forms and asks students to compare multiple texts,” Loveless writes.

For more education reform news, see:

Beyond ‘Superman’: Leading Responsible School Reform

Want to share a great resource? Let us know at submissions@eschoolmedia.com.