All newcomers to social policy should receive a mandatory inoculation against easy answers and outsized expectations. Reading through decade after decade of disappointing results from randomized experiments — probably while attached to one of those torture devices that holds the eyelids open — is the best way to do it.
In a new NBER working paper, three researchers use a randomized experiment to evaluate a program that encourages second- and third-graders to read books over the summer. Their results help illustrate the limits of intervention.
As background, the “number of books in the home” is sometimes considered an important variable in explaining why educational outcomes for children vary so widely. Summer break is thought to exacerbate educational inequalities, since children with few books in the home cannot supplement that with school books. Could the solution be a social program that gives books to students and encourages them to read over the summer?
The researchers set up a large-scale experiment involving over 6,000 second- and third-graders across seven different school districts in North Carolina. In the spring, the treatment group received special lessons on solo reading. Then, over the summer, they received a book in the mail once per week for ten weeks, along with comprehension questions designed to encourage engagement with the books. Children in the control group received math (not reading) lessons in the spring and were not sent any books.
The intervention caused students to report reading only about one more book over the summer compared with the control group. In addition, the treatment group showed no statistically significant gain in reading comprehension relative to the control group when tested in the fall.
When the researchers divided the results by grade, they found no significant reading-comprehension effects for either second- or third-graders. After further subdividing, they again found no significant effects for second-grade boys, second-grade girls, or third-grade boys. They did, however, find a positive effect for third-grade girls.
Equivalent to about 9 percent of the achievement gap between poor and non-poor families, that one positive effect is not small, at least as interventions go, and the researchers suggest some interesting avenues for further investigation. But given the failure to boost achievement for the overall sample, it’s clear that achievement differences go far beyond simple access to books. The quality of reading instruction that schools provide may have reached a point that additional gains are hard to come by. In any event, the results here are another reminder for students of social policy to keep their expectations modest.