This week a new empirical study claiming to show that public schools do a better job than private schools has made a big media splash. But the study is deeply misleading. The authors make claims their statistical method can’t possibly justify. And if you guessed that the study got off the ground with help from the educational status quo, you’d be right.
If there’s one thing education research has shown, it’s that private schools do a better job than public schools. The consensus in favor of this among empirical studies is as strong as on anything in education-policy research. Indeed, this is just the sort of thing that makes people wonder why we social scientists spend so much time doing empirical studies to prove things that everybody already knows.
Well, one reason we do it is to counteract the effects of propaganda and bad research. This new study, conducted by researchers at the University of Illinois, was first published by an openly anti-voucher think tank located at Columbia University’s Teachers College. It’s getting media play now because it was picked up by the journal of Phi Delta Kappa, a professional organization for teachers.
The authors themselves make no bones about what their real target is. One author told the Christian Science Monitor that their study “really undercuts a lot of those choice-based reforms.” Translation: People only support vouchers because they don’t realize that private schools are actually worse than public schools.
Another reason why the study is getting attention is owing to its size: The data set includes 23,000 students. People tend to assume that a big study is automatically good. But the same rule applies in research as in so many other things: size does matter, but technique matters a lot more.
The researchers take raw test scores from isolated years and apply statistical controls for race, socioeconomic status, and disabilities. While the raw scores are higher in private schools, they find that once you apply the statistical controls public-school students actually have higher test scores.
They characterize this as evidence that public schools do a better job than private schools. In fact, it shows nothing of the kind.
The main problem is that they use scores from isolated years. That is, they take a snapshot of student achievement rather than tracking achievement over time. While they do take snapshots from different years, they have no way to track students from one snapshot to the next, which is no better in practice than taking just one snapshot.
This is important because if you don’t track students over time, you can’t establish a causal connection between the type of school a student attends (public or private) and test scores. In other words, their data have nothing to say about the relative quality of public and private schools.
A much more likely explanation for the latest study’s results is that when students enter private schools, they tend to have test scores a little lower than other students of their race and socioeconomic status. That seems counterintuitive, because people are used to thinking of private-school students as privileged. And so they are–because of their race and socioeconomic status. But that’s precisely what this study controls for.
In fact, it makes perfect sense that within each racial and socioeconomic group it’s the low performers whose parents will be motivated to make the sacrifices necessary to put them in private schools. What counts is whether those students make better or worse gains over time after they enter private school–and that’s just what this study can’t tell us.
I could go on, but instead I’ll let the authors explain it for themselves. Buried in the back of the study, they write:
NAEP data [the test score set they use] do not allow for examinations of growth in achievement over time, nor do they include information about student movement between school sectors. Therefore, correlations between school sector and achievement are not demonstrably causal. In other words, one cannot conclude from this analysis that public schools are more effective at promoting student growth than private schools.
Read that last sentence again: One cannot conclude from this analysis that public schools are more effective at promoting student growth than private schools.
So what about all the huffing and puffing in the front of the study–”At this time when market-style reforms are changing the public school landscape, this study offers fresh evidence that challenges common assumptions about the general superiority of private schools,” etc.? It’s just smoke and mirrors.
As it happens, there’s a large body of very high-quality research that does allow us to evaluate the causal connection between school type and student achievement, and it overwhelmingly finds that private schools do better. The most convincing evidence comes from seven studies using “random assignment,” the same method used in medical trials. In all seven studies, students who won a random lottery to use a school voucher at a private school had significantly greater test-score gains than similar students who lost the lottery and stayed in public schools. Numerous studies using other methods have also produced a very strong consensus in favor of this finding.
As a general rule, whenever a researcher announces that his study finds something that contradicts all the other empirical evidence, and the finding just happens to coincide with the self-interest of powerful political groups, it’s a good idea to do a reality check. One can only hope this study doesn’t damage the chances that more students will be empowered to attend superior private schools through voucher programs.
–Greg Forster is a senior fellow at the Milton and Rose D. Friedman Foundation.