Skeptics of government social programs have long been frustrated by a familiar pattern of media coverage. It starts with a new social program portrayed as the key to fixing problems like poverty or low test scores. The media proceed to either ignore the long history of similar initiatives found to have little effect, or they acknowledge the history but report that this program is fundamentally different from everything that has come before.
Either way, the media trumpet the positive results from early evaluations of the program, even when those results are based on small samples, short time horizons, non-experimental data, or all of the above. Years later, researchers find that a randomized experiment shows that the program has no effect, or that the effects quickly fade, or that the effects cannot be replicated on a larger scale. These findings are quietly reported in a technical article, with no fanfare and little public attention. By that time, the media have already moved on to promoting another new social program.
Nicholas Kristof has been a prime offender. Earlier this month, he devoted his New York Times column to Oklahoma’s state-run pre-school program for four-year-olds. Kristof extolled the program’s open access and high-quality instruction. He cited journal articles showing the big test-score gains it was generating. If only every state did this!
Despite Kristof’s enthusiasm, the Oklahoma pre-school program has never been subjected to an experimental evaluation. Its backers are relying on hope more than evidence. As Russ Whitehurst points out, the existing analyses mainly use “regression discontinuity” to isolate the effect of preschool. It’s a standard non-experimental technique, but in this case it can’t tell us how the children would have fared in the absence of the program.
What should be sobering to supporters of Oklahoma preschool is the recent experimental evaluation of a similar preschool program in Tennessee, one that boasts the same well-trained teachers and small class sizes that Oklahoma’s does. The results of the evaluation are depressingly familiar: no effect. I have yet to see the Tennessee finding reported on the front page of the New York Times or discussed in a Kristof column. But, no matter – did you hear about the exciting new program in Georgia . . .