A new “quasi-experimental” study of KIPP schools claims that “estimated impacts are frequently large enough to substantially reduce race- and income-based achievement gaps within three years of entering KIPP.” Sound familiar?

Fortunately, we’ll have a real study soon — an actual experiment, with students randomly assigned to KIPP and non-KIPP schools:

The evaluators are going to take advantage of the fact that they can conduct a true randomized experiment — something that is normally difficult to do — using an existing KIPP lottery system. Then, they can use the more robust method to confirm that their model is accurate. Or as the report states, “If the nonexperimental impact estimates match the experimental impact estimates, we will have evidence that the nonexperimental design can produce unbiased impact estimates.” An interim report is due in mid-2012 with the final report due in 2014 after four years of follow-up.

My guess: The experiment will probably show some improvement, but not as dramatic as that seen in the non-experimental study.

Also, it’s worth noting that only a small subset of parents care enough to apply to KIPP schools and put up with the hassles such schools impose. An experimental design can show whether the KIPP schools help the kids who get into them, relative to other kids whose parents also applied. It cannot show whether KIPP schools would help children from less dedicated families.