When newspaper reporters and bloggers across the nation skewered the Obama administration’s estimates of jobs “created” or “saved” in November 2009, they focused on the obvious target: the thousands of jobs supposedly saved in congressional districts that didn’t exist. Many took this as evidence of bureaucratic incompetence at best and, at worst, fraud. But there was a bigger story that went largely unremarked. The errors, far from being a one-time failure, provided a dramatic reminder that President Obama’s pledge to enact policies guided by empirical analysis was — as any such pledge must be — hollow.
The notion of “evidence-based public policy” has guided progressive thought since the 19th century, and even Republicans now embrace it. To be sure, evidence is important and sometimes compelling. But it is rarely decisive in the world of economic and social policy, and what is commonly presented as “evidence” is more often pretense: Savvy politicians invoke data to mask the fact that their initiatives are driven, not by evidence, but by ideology.
President Obama is squarely in this tradition. On the campaign trail, discussing everything from stem-cell research to the wars in Afghanistan and Iraq to job creation, he wooed independent and fiscally conservative voters by saying that public policy should be based on what works, and that programs that don’t work should be abandoned. While still a senator and dark-horse presidential contender, he was among the first and most prominent public officials to sign a “transparency pledge” urging forthrightness in government spending and an objective approach in evaluating government programs’ effectiveness.
And, to be fair, in creating Recovery.gov, a website devoted to tracking the effectiveness of the American Recovery and Reinvestment Act (ARRA), the Obama administration made a good-faith effort to uphold this pledge. In May 2009, the White House unveiled guidelines concerning how it would tally up jobs “saved or created” by the ARRA, as reported on Recovery.gov. Christina Romer, chairwoman of the president’s Council of Economic Advisers, explained: “We convened a meeting of top economists from the agencies [responsible for implementing the ARRA] to begin developing a simple, conservative, and — most important — accurate model for measuring progress.” Those recommendations called for recipients of ARRA funds to report how many jobs the funds had saved or created, as a way to “check on” the accuracy of the administration’s macroeconomic forecasts.
But after using the promise of “evidence based” economic policies to enact the ARRA, the administration ducked accountability for its initiative. Its forecasts at the time the package was passed predicted the unemployment rate would peak at 8 percent in late 2009 and then fall to 7 percent by the end of 2010. In a May 2009 briefing with an on-background “senior administration official,” a journalist asked whether the administration still hoped to meet this target. The official responded that the prediction no longer counted because “the baseline deteriorated between when we did our first report in January and over the first couple of months of the year.” But — fear not — the real goal, to create or save 3.5 million jobs, was still attainable. So the pressure was on to report progress in the creation and retention of jobs, and this may have contributed to the endless inaccuracies in the White House’s claims.
This administration’s bait-and-switch, and the subsequent exaggerations of the ARRA’s impact, was more than a case of government error in an important prediction. It was a clear example of the problems with “evidence-based public policy” — not to mention a good reason to doubt the same official’s promise that the government would “monitor this thing” by “doing another [macroeconomic] forecast” to “make sure we’re doing what we need to do to put the economy back to the track we want to be on.”
Obama’s campaign was a continuous, thinly veiled effort on the part of the candidate to distance himself from the Bush administration, which he regularly accused of ideologically motivated impracticality. Rather than cut taxes on businesses and high earners to stimulate investment — an ideological approach, he alleged — government should build up the incomes of the unfortunate through redistributive policies such as unemployment insurance, higher minimum wages, and expanded health-care coverage. This, Obama implied, would be pragmatic.
Obama seemed to be drawing the classic distinction between “politics” and “administration,” a dichotomy deeply rooted in the progressive movement of the early 20th century. Indeed, in an 1887 Political Science Quarterly article, Woodrow Wilson argued that there “should be a science of administration [that] shall seek to straighten the paths of government, to make its business less unbusinesslike, to strengthen and purify its organization, and to crown its dutifulness.” Frederick Taylor, the father of “scientific management,” went even farther in testimony before Congress in 1912, calling for the systematic application of scientific approaches to management as a replacement for old “rules of thumb.” These thinkers advocated the idea that empirical methods are sufficient to evaluate the effectiveness of government programs.
Why does this approach — so appealing in theory — fail in practice? Why can’t evidence-based policy substitute for political judgment?
One reason is that we simply don’t know enough about the way the economy and society work to make reliable — or, in some cases, even useful — predictions on purely empirical grounds about the effects of public policy. That’s why macroeconomic forecasting tends to be inaccurate. We can’t model all the layers of human decision-making and activity that determine macroeconomic outcomes, even when our equations include hundreds of variables.
But even the data we do have are often suspect — as the White House learned when it tried to evaluate the ARRA. Jobs were reported in nonexistent congressional districts all over the nation and in its territories, in such places as Montana, Arizona, Connecticut, Iowa, the Northern Mariana Islands, and Puerto Rico. All politics is local, the old saying goes, and we might add the corollary that all data are local too. It is simply unreasonable to suppose that federal bureaucrats will be able to investigate in fine detail the conditions of cities and towns far from Washington.
And never mind the fine detail; the federal government has proved incapable of detecting even gross reporting error, as witness those imaginary congressional districts. Ohio Watchdog, the investigative-reporting arm of the Buckeye Institute in Columbus, discovered ten “new” districts in its home state when it dug into the ARRA data. Districts with outlandish numbers such as 49, 69, 85, 87, and 99 had appeared out of nowhere. Many of these cases likely resulted from computer-programming errors (“99” is a common coding term for missing data), but the problem was widespread enough to undermine the credibility of the entire ARRA reporting process. The Government Accountability Office, in a systemic review of the reports, found that 3,978 of them, while showing no dollar amount received or spent, together claimed more than 50,000 jobs saved or created. On the other hand, more than 9,200 reports showed no jobs saved or created, but together claimed spending of almost $1 billion.
Almost everyone involved in the program recognizes that there is no easy way to ensure the accuracy of the data, so complex are the reporting procedures and so great are the obstacles to collecting accurate information. And without reliable data, no evidence-based evaluation is possible.
That is the nut of the problem. Social-scientific methods simply cannot provide the definitive answers progressives claim to seek or, worse, to have. At the end of the day, the data will be inconclusive. Policymakers will then have to make judgments, and these judgments will inevitably reflect ideology. The Obama administration is no exception, and its talk of “evidence-based public policy” is little more than a pretext for doing what it wants to do anyway.
– Mr. Staley is director of urban-growth and land-use policy at the Reason Foundation. He teaches urban and regional economics at the University of Dayton.