While reading a number of Thorfinn posts on the new Dale-Krueger study (see here, here, and here), I was struck by the extreme credulousness of most of the reporters who’ve been covering it. I found this Thorfinn passage particularly interesting:

In looking at why their results changed for this paper, Krueger and Dale find that their effects already diminish when using the sample of Colleges used for this paper as opposed to the sample from the old paper; and diminish even more when using government income rather than self-reported income. This tells us two things. One, the schools dropped for this paper (Denison, Hamilton, Kenyon, Rice, UNC) may matter a lot for future income, or else the inclusion of two historically black Colleges might affect the results. Second, it’s puzzling to think of why the results would change dramatically depending on the source of income. We know from other studies that individuals systematically under-report income both to surveys and in official government data. It’s not clear that the government data is “better” in the sense of getting a more accurate picture. The authors also exclude income received from capital gains, which doesn’t strike me as a good exclusion. Either students who went to elite schools lie more about their income, or are better at hiding it from the government (or else receive more of it in the form of capital gains). All that we can seriously say is that the conclusion you draw depends enormously on the data source you use for income and set of Colleges. [Emphasis added.]

Given that capital income has been a huge driver of wealth concentration in the United States, and that there is reason to believe that highly selective schools are somewhat overrepresented in the ranks of elite financiers, this makes for a not-so-good exclusion indeed.

Then there is the following:

The results for both tuition and selectivity still show sizable effects for the 1976 cohort. Their Table 5 breaks out the effects of tuition and College selectivity by years. While none of these regressions are statistically significant on their own, the net effect is quite large. I applied the estimates on wages to the actual median wages in each time period (interpolating when the authors did not provide actual wage statistics). I estimate that a one percent increase in 1976 tuition (perhaps $100 total over four years) results in roughly a two percent increase in overall compensation through 2007 (assuming that you work for all 24 years), or $43k in non-inflation adjusted dollars. Alternately, a category shift in the Barron’s selectivity criteria (ie, from Highly Competitive to Most Competitive) is associated with $45k more in lifetime income. The effects of both selectivity and tuition grow over time, and are at their highest for wages observed in 2003-2007 (at this point, a one percent increase in tuition paid in 1976 gets you roughly $4k more a year per year. Presumably, this will rise even more by the time this cohort retires.
While the results from any one regression may not be statistically significant, that may simply be due to their sample size. The cumulative effect appears rather large in magnitude for both of the measures that were quite important in earlier drafts of the paper.

Yet most reporters simply parroted the findings Dale and Krueger chose to highlight.

Arpit Gupta reminded me of the Neumark-Wascher study that reevaluated the still-famous Card-Krueger analysis of the impact of an increase in the minimum wage. Greg Mankiw wrote a post summarizing the findings back in 2006:

Gene Sperling, former economic adviser to Bill Clinton, tries to get President Bush to endorse a minimum-wage increase. Gene dismisses worries about adverse effects on employment. He writes:
No one has yet rebutted convincingly David Card and Alan Krueger’s study that compared fast-food jobs on the border of New Jersey and Pennsylvania, and found no decrease in lower-wage jobs after New Jersey raised its state minimum wage.
The key word here is “convincingly.” Gene is, apparently, not convinced by the Neumark-Wascher study that reevaluated the Card-Krueger work:
estimates of the employment effect of the New Jersey minimum wage increase from the payroll data lead to the opposite conclusion from that reached by CK.

The post continues with links to a number of other well-regarded studies. It turns out that the Card-Krueger study relied on a potentially problematic telephone survey method:

We re-evaluate the evidence from Card and Krueger’s (1994) New Jersey-Pennsylvania minimum wage experiment, using new data based on actual payroll records from 230 Burger King, KFC, Wendy’s, and Roy Rogers restaurants in New Jersey and Pennsylvania. We compare results using these payroll data to those using CK’s data, which were collected by telephone surveys.

And it turns out that the payroll method yields a different result:

For comparable sets of restaurants, differences-in-differences estimates using CK’s data imply that the New Jersey minimum wage increase (of 18.8 percent) resulted in an employment increase of 17.6 percent relative to the Pennsylvania control group, an elasticity of 0.93. In contrast, estimates based on the payroll data suggest that the New Jersey minimum wage increase led to a 4.6 percent decrease in employment in New Jersey relative to the Pennsylvania control group. This decrease is statistically significant at the five-percent level and implies an elasticity of employment with respect to the minimum wage of -0.24.

Yet how many of the (bright!) people who continue cite the Card-Krueger study are aware of the relative vices and virtues of telephone surveys and payroll data, or that Neumark and Wascher offered a follow-up?