The biggest difficulty for defenders of the government’s school monopoly is the overwhelming consensus in the empirical research finding that school choice works. They deal with this little problem primarily by ignoring the evidence and changing the subject, but it also helps that they have a stable of professors ready to distort, confuse, and obfuscate the research.
A new article in Perspectives on Politics, a prominent academic journal published by the American Political Science Association, shows how low they’ll sink. Written by Kevin Smith of the University of Nebraska-Lincoln, “Data Don’t Matter? Academic Research and School Choice” is a warped and unfair review of the research on school choice: It’s full of innuendo, misdirection, and selective omissions.
The academic effects of vouchers have been studied eight times with random-assignment methods, the gold standard of social science. But Smith, following standard procedure for opponents of vouchers, doesn’t even acknowledge the existence of most of these studies. This may be because seven of the eight studies found statistically significant positive effects from vouchers and no significant negative effects. The eighth study also found positive effects, and only failed to achieve statistical significance by watering down the data with unorthodox methods, some of which violate federal research guidelines.
Smith also follows the standard anti-choice procedure in failing to acknowledge the research consensus in favor of school choice on other questions, such as the effect of choice on public schools and whether choice students learn values like tolerance. Not one empirical study has ever found that outcomes at U.S. public schools exposed to any form of school choice have worsened, and quite a few have found that they improve. Similarly, there is a large body of empirical studies finding that choice improves students’ levels of tolerance and other civic values, while very few studies find the reverse.
There is certainly lots of room for legitimate discussion about the limitations of these studies. However, for such discussion to be honest it must acknowledge the preponderance of empirical studies supporting choice, and evaluate them on their merits. Smith carefully keeps most of these studies offstage. Instead, his primary tactic is to question the motives of those whose findings are favorable to school choice. And the substantive comments he does make on the content of the research are shockingly unfair.
Bias produced by researchers’ beliefs and motives is a delicate problem. There’s nothing wrong with researchers’ developing a point of view about the things they study. And though we need to be on guard against biased research, we also need to avoid dismissing as bad scholarship any study produced by a researcher who has a point of view on the things he studies. Ironically, Smith himself acknowledges the difficulty of this problem at one point in the article, and even provides the correct answer: He says that the important question is not whether the researcher has a point of view, but rather this: “Were the data treated fairly? Fair means that the researcher offers demonstrable assurance that he or she has adhered to scholarly conventions designed to minimize the influence of [the researcher’s] preferences.”
Too bad Smith didn’t stick to this rule. When it comes time to evaluate the research, he is more concerned with attributing positive findings for choice to the motives of researchers and organizations who support choice than he is with determining whether the data were treated fairly. For example, a systematic review of all available empirical studies found an overwhelming consensus that private schools and choice programs improve tolerance and other civic values among students, but Smith dismisses the review out of hand because many of the studies were sponsored by Harvard University’s pro-reform Program on Education Policy and Governance. (Full disclosure: Smith includes my employer, the Friedman Foundation, on his list of suspect organizations, as well as my former employer, the Manhattan Institute.)
Smith also employs misdirection. He dismisses some positive school choice findings because the effect identified is small, but a positive effect that is small over one year can look a lot bigger when you compound it over the twelve years students are in school. He points out that not all voucher programs are identical, so a study finding that vouchers work in Milwaukee doesn’t necessarily prove that they work elsewhere. This clearly leads the reader to believe that the findings on voucher programs in different cities are mixed, when in fact the findings of the best studies are similarly positive across all cities.
Finally, Smith mischaracterizes scholarly debates. One major voucher study found significant gains only for black students; Smith paints this as implausible because “there is no satisfactory causal explanation” for this result. In fact, researchers have pointed to several perfectly satisfactory possible explanations, including that the black students were more severely underserved by their public schools and thus had more to gain from vouchers, and that the much smaller number of non-black participants in the study may have prevented their results from achieving statistical significance. Smith likewise dismisses as inexplicable another study’s finding of significant gains in math but not in reading, but it is perfectly plausible that math achievement is more affected by schools than reading achievement, since kids learn more about math at school, and reading in the home. To disagree with these explanations is fine, but to pretend that they don’t exist is blatantly unfair.
In one of his most egregious distortions, Smith reviews several critiques of a study by my former colleague Jay Greene. What Smith doesn’t tell you is that Greene later published new analyses showing that his findings aren’t affected by those criticisms. He also doesn’t tell you that two independent studies confirm the findings.
This is only a sampling of the innuendo, errors, misdirection, and injustices in Smith’s article. This phony research review will undoubtedly reinforce the myth that the research on school choice is mixed or worse, when in fact school choice is as well supported by the research as any education policy. But I prefer to look on the bright side: If it weren’t for people like Smith, people like me would be out of a job.
– Greg Forster is a senior fellow at the Milton and Rose D. Friedman Foundation.