The Corner

Can We Trust the PISA Data?

I recently urged caution in interpreting the U.S.’s mediocre PISA scores. The post was meant to help preempt all the hand-wringing that was expected when the scores became public. But it turns out that the blogosphere didn’t need my infinite wisdom after all: Skepticism about the PISA abounds, and it’s coming from across the ideological spectrum, from Diane Ravitch to Rick Hess.

What’s especially encouraging is that the conversation has focused as much on the reliability of the PISA data as on the interpretation. Gone are the days, hopefully, when a major organization can release a report and expect as a matter of course that the media will treat the data and conclusions as authoritative.

I previously mentioned the inappropriately strong lessons the OECD attempts to draw from thin data. In addition, Hess points to the mixing of city and country data — “Comparing U.S. performance to that of Shanghai isn’t apples and oranges; it’s applesauce and Agent Orange” – the inexplicable fall of Finland, and the extreme sensitivity of the rankings to the choice of test questions.

What about the mechanics of PISA test administration? Did every country follow the same strict procedures? Almost certainly not. Consider sample drop-out, which is one of the most frequent problems we confront in program evaluation. Even an impeccable research design won’t produce meaningful results if a significant number of people don’t get measured. This is an especially common issue in educational interventions, when the least gifted students in the treatment group are mysteriously absent on test day.

PISA response rates vary widely from country to country. For instance, Finland tested 96 percent of its nationally-representative sample, but Mexico somehow tested only 63 percent of its own. One need not be a cynic to suspect some gamesmanship there.

This isn’t the first time that I’ve encountered questionable OECD data. Its earlier 2013 report on teachers contained some comparisons of relative pay and work time that were simply invalid. As I wrote at the time, “The truth is that large-scale international comparisons are almost inevitably plagued by inconsistent and unreliable data.” The PISA is no exception.

Jason Richwine is a public-policy analyst and a contributor to National Review Online.

Most Popular

White House

The Trivialization of Impeachment

We have a serious governance problem. Our system is based on separation of powers, because liberty depends on preventing any component of the state from accumulating too much authority -- that’s how tyrants are born. For the system to work, the components have to be able to check each other: The federal and ... Read More
U.S.

‘Texodus’ Bodes Badly for Republicans

‘I am a classically trained engineer," says Representative Will Hurd, a Texas Republican, "and I firmly believe in regression to the mean." Applying a concept from statistics to the randomness of today's politics is problematic. In any case, Hurd, 42, is not waiting for the regression of our politics from the ... Read More
Culture

Feminists Have Turned on Pornography

Since the sexual revolution of the 1960s, the feminist movement has sought to condemn traditional sexual ethics as repressive, misogynistic, and intolerant. As the 2010s come to a close, it might be fair to say that mainstream culture has reached the logical endpoint of this philosophy. Whereas older Americans ... Read More
Elections

Put Up or Shut Up on These Accusations, Hillary

Look, one 2016 candidate being prone to wild and baseless accusations is enough. Appearing on Obama campaign manager David Plouffe’s podcast, Hillary Clinton suggested that 2016 Green Party candidate Jill Stein was a “Russian asset,” that Republicans and Russians were promoting the Green Party, and ... Read More
PC Culture

Defiant Dave Chappelle

When Dave Chappelle’s Netflix special Sticks & Stones came out in August, the overwhelming response from critics was that it was offensive, unacceptable garbage. Inkoo Kang of Slate declared that Chappelle’s “jokes make you wince.” Garrett Martin, in the online magazine Paste, maintained that the ... Read More