Tampering or Experiments?

In a review of Jim Manzi’s excellent new book Uncontrolled, Arnold Kling offers the following aside:

I was once seated at a dinner table next to an official of the Department of Education involved in education research. I made an impassioned plea for more controlled experiments in education. The official responded by asking, “Would you want your child to be the subject of an experiment?” At this, my jaw dropped, and I sputtered, “They do it to my children all the time! They constantly introduce curriculum changes, scheduling changes, and changes in teacher methods. They just don’t bother to evaluate whether or not it works.”

Statistical quality control guru W. Edwards Deming used the term “tampering” to describe this process of introducing changes without rigorously evaluating results. Tampering and experiments are two ways of disturbing the status quo. But only experiments are designed with the intent of producing reliable measurements of success or failure.

Like my dinner companion, most policymakers view experiments as at best costly and at worst immoral. Even though tampering is just as bad, if not worse, it somehow escapes such criticisms.

I often encounter arguments against decentralized trial-and-error. The alternative to trial-and-error, however, is centralized trial-and-error, in which errors are far more costly and far more difficult to reverse. But it is very easy to mask these costs: all you need to do is fail to engage in the rigorous evaluation of results, as Kling suggests. 

On a separate note, an education professor recently made the case that choice-based reform in New York city has exacerbated the advantages of savvy, affluent parents. She makes a number of entirely legitimate points. Yet it is important to reckon with the counterfactual, e.g., had New York city not embraced (decidedly imperfect) choice-based reform, would we have a more egalitarian, high-performing educational system? Or would we have somewhat more middle-class flight? And if we had the latter, would this have materially improved outcomes for students in New York city public schools — or would it have eroded the tax base, undermined the political investment of middle-class voters in the school system, and contributed to a general deterioration of the quality of life in New York city? This seems like a question worth asking.

Reihan Salam — Reihan Salam is executive editor of National Review and a National Review Institute policy fellow.

Most Popular


Courage: The Greatest of Virtues

EDITOR’S NOTE: The following is Jonah Goldberg’s weekly “news”letter, the G-File. Subscribe here to get the G-File delivered to your inbox on Fridays. Dear Reader (Or Listener), As the reporter assigned the job of writing the article about all of Sidney Blumenthal’s friends and supporters told his ... Read More

My American Dream

This morning, at 8 a.m., I did something I’ve wanted to do for as long as I can remember: I became an American. I first applied for a visa in early 2011, and since then I have slowly worked my way through the system — first as a visa-holder, then as a permanent resident (green card), and, finally, as a ... Read More

The Gun-Control Debate Could Break America

Last night, the nation witnessed what looked a lot like an extended version of the famous “two minutes hate” from George Orwell’s novel 1984. During a CNN town hall on gun control, a furious crowd of Americans jeered at two conservatives, Marco Rubio and Dana Loesch, who stood in defense of the Second ... Read More