The Corner

Can We Trust the PISA Data?

I recently urged caution in interpreting the U.S.’s mediocre PISA scores. The post was meant to help preempt all the hand-wringing that was expected when the scores became public. But it turns out that the blogosphere didn’t need my infinite wisdom after all: Skepticism about the PISA abounds, and it’s coming from across the ideological spectrum, from Diane Ravitch to Rick Hess.

What’s especially encouraging is that the conversation has focused as much on the reliability of the PISA data as on the interpretation. Gone are the days, hopefully, when a major organization can release a report and expect as a matter of course that the media will treat the data and conclusions as authoritative.

I previously mentioned the inappropriately strong lessons the OECD attempts to draw from thin data. In addition, Hess points to the mixing of city and country data — “Comparing U.S. performance to that of Shanghai isn’t apples and oranges; it’s applesauce and Agent Orange” – the inexplicable fall of Finland, and the extreme sensitivity of the rankings to the choice of test questions.

What about the mechanics of PISA test administration? Did every country follow the same strict procedures? Almost certainly not. Consider sample drop-out, which is one of the most frequent problems we confront in program evaluation. Even an impeccable research design won’t produce meaningful results if a significant number of people don’t get measured. This is an especially common issue in educational interventions, when the least gifted students in the treatment group are mysteriously absent on test day.

PISA response rates vary widely from country to country. For instance, Finland tested 96 percent of its nationally-representative sample, but Mexico somehow tested only 63 percent of its own. One need not be a cynic to suspect some gamesmanship there.

This isn’t the first time that I’ve encountered questionable OECD data. Its earlier 2013 report on teachers contained some comparisons of relative pay and work time that were simply invalid. As I wrote at the time, “The truth is that large-scale international comparisons are almost inevitably plagued by inconsistent and unreliable data.” The PISA is no exception.

Jason Richwine — Jason Richwine is a public-policy analyst and a contributor to National Review Online.

Most Popular

PC Culture

‘White Women’ Becomes a Disparaging Term

Using “white men” as a putdown is no longer extreme enough for the Left. Now it is moving on to doing the same for “white women.” How rapidly this transpired. It was less than two years ago that the approximately 98.7 percent of white women working in media who were openly rooting for Hillary Clinton ... Read More
Politics & Policy

The World Keeps Not Ending

We were not supposed to have made it this far. George Orwell saw night descending on us in 1984. Orwell was, on paper, a radical, but in his heart he was an old-fashioned English liberal. He dreamed of socialism but feared socialists. He feared them because he knew them. I was in the sixth grade in 1984, but I ... Read More
Culture

A Free People Must Be Virtuous

Dear Reader (Even those of you who didn’t seem to notice or care that I failed to file this “news”letter on Friday), So I’m sitting here at Gate C6 at O’Hare waiting for my flight home. I am weary, pressed for time, in desperate need of a shower, and filled with a great sense of dread for the work ... Read More