By Sarah Butrymowicz

A report released Tuesday aims to debunk claims that the United States lags substantially behind the international competition in education. The study, released by the Stanford Graduate School of Education and the Economic Policy Institute, argues that looking only at the United States’ average score on international exams is problematic and can lead to unwarranted policy conclusions.

By disaggregating the data of the Programme for International Student Assessment (PISA), researchers concluded that U.S. students fared better on the exam in 2009 than most had originally believed. In particular, low-income students appear to have gained ground on disadvantaged students in other countries.

“Our main message is a cautionary tale,” said Martin Carnoy, a professor of economics and education at Stanford who coauthored the report. “If you don’t make some attempt to look at everything by social-class groups, you are headed for lots of mistakes … in your policy conclusions.”

The exam, given to 15-year-olds, ranked the United States 14th in reading and 25th in math out of the 33 countries in the Organisation for Economic Co-operation and Development (OECD). (Students in six countries achieved significantly higher scores in reading than U.S. students did; in math, Americans were substantially outperformed by other 15-year-olds in 17 countries.) The lackluster performance of U.S. students, especially when compared to that of their peers in Shanghai, prompted some policymakers to compare the results to the Soviet launch of Sputnik in 1957.

The United States has a larger proportion of economically disadvantaged students than do higher-performing countries. Finland, for example, reports that 4 percent of its students live in low-income families. In the United States, nearly a quarter of children live in poverty.

Carnoy and his coauthor Richard Rothstein of the Economic Policy Institute also contend that low-income students were oversampled in the U.S. results on the 2009 PISA test. About 40 percent of American PISA-takers attended a school where half or more of students were eligible for free or reduced-priced lunch, although nationwide only 23 percent of students attend such schools.

After adjusting for both of these factors, they estimated that the United States would place fourth in reading and 10th in math—which demonstrates that educational reforms alone aren’t enough for the United States to catch up to the international competition, Carnoy said.

“If you do policy that significantly reduces poverty in the U.S., I guarantee you, you will reduce the distance between top and bottom in our own country … and you’ll certainly raise those kids relative to kids in Finland, [South] Korea and Canada,” he said. “If you really want to go after the academic issues in the U.S. schools, [poverty] is a part of it.”

The researchers criticized the Trends in International Mathematics and Science Study (TIMSS) for publishing average country scores five weeks before the disaggregated data are made publicly available. The full set of data for the 2011 TIMSS will be released later this week.

“We would be the first to argue that average scores in and of themselves are not sufficient on which to base sound educational policy or thinking,” Hans Wagemaker, executive director of the International Association for the Evaluation of Educational Achievement, which administers TIMSS, said on a press call. Wagemaker praised the report for moving the discussion away from average scores.

Carnoy and Rothstein compared American results by social class to the three top performers on PISA—Canada, Finland and South Korea—and three economically similar peers, England, France and Germany. They found that the most disadvantaged students in the United States, as measured by the number of books in children’s homes, have been improving in reading and math since PISA was first given in 2000. By contrast, test scores among similar students in Canada, Finland and South Korea have been dropping.

“The whole idea that we’re not doing very well for the bottom kids—we would say, ‘Keep doing what you’re doing rather than keep criticizing what you’re doing,’ ” Carnoy said. “We’re making progress with the kids at the bottom.”

The most economically advantaged students in America, however, are slipping compared to their peers in the other countries that Carnoy and Rothstein analyzed. And at each level of social class, U.S. students perform worse than their peers in the corresponding group in the three top-performing countries.

The data also reveal that Germany is improving faster across all income levels than the United States, in large part due to gains among the country’s immigrants, while Finland’s scores have been falling at all levels.

Still, the report’s authors warn against making sweeping conclusions about any country’s performance, pointing to a variety of factors that influence scores, from demographics to the tests themselves and how they are scored. There can be stark differences in a country’s performance over time on PISA and TIMSS, according Carnoy.

“It’s a conundrum,” he said, adding that he was surprised by the discrepancies. “It just throws a note of caution into this whole thing.”

This post was produced by The Hechinger Report, a non-profit journalism organization housed at Columbia University in New York.