Numbers reveal a lot, but they do not contain the whole truth.
Numbers reveal a lot, but they do not contain the whole truth.
D ata analysis was back in fashion again this spring, with a nation hanging on the shape of infection curves and the reliability of predictive mathematical models. Then, it was back out of fashion, as the lockdowns of March gave way to the primal street violence of May. Both experiences should humble our view of the power of data.
Using data to inform the tasks of making public policy and persuading voters of its merits is a good thing, especially where hard, falsifiable science is concerned. It is good for conservatives to cultivate leaders, pundits, and spokesmen who are fluent in and comfortable dealing with such data, and bad news when they cannot compete on that field. Facts, knowledge, and evidence are always important. Without them, we are flying blind.
But “data” is not always the same thing as facts, knowledge, or evidence. This is doubly so when dealing with social-science data, which after all is the study of human experience. A subset of data here and a hidden-agenda-driven report there, and suddenly people are saying they have all the facts they will ever need. The effort to turn “Data” into a shibboleth with the sole power of command over politics is not only anti-democratic; it is ultimately anti-empirical and commits us to an unduly narrow field of vision of the world. It’s the same hubristic project that humanity has been chasing ever since French revolutionaries built the Temple of Reason.
1. “Competence, Not Ideology”: The Mirage of Politics Driven by Data
The first reason why voters are, and should be, skeptical of data-driven politicians and political arguments is that there is a very long history of people in politics — especially politicians of a statist bent and center-left or left ideology — masquerading their ideological commitments under the neutral banner of data. The pervasive habit of using data as a drunk uses a lamppost — for support, rather than illumination — calls to mind Mark Twain’s epigram:
Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: “There are three kinds of lies: lies, damned lies, and statistics.”
If we’re honest with ourselves, we know that most people do not use data at all objectively. Arguments about data are often just arguments about ideology dressed up with numbers. Almost nobody who supports or opposes legal abortion or same-sex marriage is likely to be persuaded to a different view by any amount of data; those issues are almost universally viewed as matters of principle. Junk data that feels right, such as oft-debunked gender/pay statistics, can prove unkillable. Ironically, most political-science studies of voter behavior confirm this: “The data” literally show that the data don’t matter all that much.
In 1988, Michael Dukakis ran for the presidency on a slogan of “Competence, not Ideology,” in an effort to rebrand the Democrats from liberalism to technocratic management. George H. W. Bush (hardly a natural ideologue) responded with a campaign of cultural wedge issues to demonstrate that Dukakis was, in fact, a doctrinaire Massachusetts liberal passing off his ideological preferences as the outputs of “competence.” Culturally, the same tendency manifests itself in people such as Neil deGrasse Tyson and Bill Nye, who mask themselves as just “nerds” who “F***ing Love Science.”
Data should, in theory, be more persuasive in assessing large government programs such as Medicaid or Head Start. Yet nearly all the data cited or discussed in political debates, punditry, and “explanatory journalism” are data that support the position of the person making the argument. Since I’m a lawyer, that’s familiar turf to me. An adversarial system in which both sides marshal the best case they have in order to persuade the undecided, and occasionally change the minds of partisans at the margins, is a good system for ferreting out the truth. But an adversarial clash of arguments is not the posture of the Dataphiles, who instead pose as neutral arbiters reporting objective reality. All who disagree are tarred as “deniers” who won’t accept evidence. There is nothing wrong with using data to support your ideas. The problem is in dishonestly pretending that data are the only reason your conclusions always match your preferences.
Consider which claims are examined as questions of data, and which are not. For example, most progressives would be horrified at using cost-benefit analysis to enact policies against statistically riskier sexual behaviors, because they regard sexual liberty as a good too important to be put under a green eyeshade. Yet, when the subject turns to drinking large sodas, or smoking cigarettes, or firing a gun, or riding a motorcycle without a helmet, suddenly the value of freedom is not even in the equation. In the Black Lives Matter debate, it is considered heretical to actually examine the number of unarmed black Americans killed by the police in comparative statistical terms. You may agree or disagree with the relative hierarchy of values that this implies, but make no mistake: These are value judgments, not merely scientific statements. Someone, at some point, has to decide what things matter before trying to count them, and what things matter too much to be counted.
Data analysis is a tool, to be deployed in making arguments over the relative weight of contending values. It is not a trump card that makes values and principles obsolete.
2. Looking Under the Streetlamps: The Bias toward Quantifiable Data
Closely related is the overvaluation of factors that can be reduced to numbers. Not everything of value in life can be quantified, even if you wanted it to be. Even if honestly pursued, comprehensive data are not always available on all aspects of public policy. The fact that we’ve counted some things creates a bias toward treating them as the only variables worth considering and assuming away the rest. This creates a bias toward elevating economics and social utilitarianism over more nebulous values such as love, faith, family, freedom, art, loyalty, community, patriotism, self-reliance, dignity, and pride. The things that move us most are often the hardest to tabulate.
This is why conservatives are more suspicious of dataphilia than are liberals or progressives. The lockdown debate threw this divide into sharp relief. Deaths and infections can be quantified. So can some of the costs of lockdowns, such as lost jobs. We can count increased suicides and missed cancer screenings but can only estimate how much they are due to lockdowns. But how do we measure the quantified loss of life against the unquantified loss of living? That is fundamentally a question of values. How do we measure the buildup of tensions that exploded in the George Floyd protests? You don’t have to be anti-lockdown to see that “trust the data” puts a thumb on the scales.
Look at social-science literature on divorce. There is a cottage industry of studies of children of divorce, many of which seem tailored to reassure divorced parents that their kids will, statistically speaking, turn out just fine in the long run. Studies, however, cannot truly quantify the pain and anguish that children of divorce routinely experience. If you have been through a divorce or know anyone who has, or if you’ve spent any time listening to a child of divorce, you know this. What is the quantifiable value of a happy childhood? If you can’t put a number on happiness, does it still exist?
Or what about home ownership? We can measure the relative value of a home as an investment, or its contribution to family wealth, or suburban sprawl, or property-tax bases, or pollution, or even educational outcomes. But the value of a home, a place where your children can be raised and nobody can tell you what to do: What is the value of that?
Overreliance on available data is also a problem when the data are quantifiable but just aren’t in the data set being studied. Many of the models that valued mortgage-backed securities leading up to the 2008 credit crisis were built around data that didn’t go back to the last major crash in the market. If the numbers weren’t in the model, they may as well have never existed.
3. Small Sample Sizes: Tradition and Experience versus Data
A core insight of statistical analysis is the importance of sample sizes: The more observations a study has, the greater your confidence in its conclusions. Ironically, however, using data as the exclusive path to understanding the world means missing a lot of observations. Human history is a vastly larger sample than any study. Much of the world’s knowledge and wisdom predates social science and can be found instead in the humanities, or in tradition. The Founding Fathers relied on historical example, not numerical data, when they wrote the Declaration of Independence and the Constitution.
Social, cultural, political, religious, legal, and economic traditions carry the vast freight of thousands of years of trial and error by billions of individuals. As G. K. Chesterton said, tradition is the true democracy, superior to the tyranny of whatever generation happens to be walking around at a particular moment. People have habits for reasons they do not even think about because someone before them tried and saw what happened. A “pragmatism” that discards tradition deprives itself of the raw material to test whether a solution actually works. Tradition is pragmatic: It is constant experimentation with the new and comparison of its results to the old. It is the gradual accrual of the life lessons of countless individuals. It changes when new things are proven to work, and when old things are found to have become unuseful. It is human natural selection: That which works over time prospers, and that which does not falls by the wayside. Tradition protects us from the tyranny of small sample sizes by delivering to us the lessons drawn from experience of prior generations. It preserves moral and empirical progress from a process of Brownian motion in which nothing learned today has any guarantee against being unlearned tomorrow. No study can replicate it.
4. No Room for the Common Man: A World Ruled by Experts and Their Values
Government by expertise and data is inherently undemocratic, which is why it has grown together with the twin Wilsonian projects of judicial activism and the administrative state. It is designed to place decisions beyond the reach of the ordinary citizen.
A core premise of democracy, as with the jury system, is that decisions of great importance to life, liberty, and property should be made with the input of ordinary citizens. One reason is to legitimate decisions by making people feel involved — a goal that is subverted by privileging social-science data over popular sentiment. More importantly, decisions made with the input of the broad diversity of ordinary citizens are, in the long run, likely to be better decisions. Ordinary citizens as a group have broader and more varied experience than particular individuals. Moreover, every individual is likely to know more about his or her own circumstances than even the most rigorous expert. These advantages are especially pronounced when dealing with matters of family, child-rearing, education, and lifestyle that are well within the life experience of most adults. The common man is valuable not because there is great virtue in being “common,” but precisely because the common man, being more numerous than the uncommon man, has more human experiences — and more opportunities to learn from his mistakes. This brings to bear the greatest possible amount of accumulated wisdom and dilutes the folly of any particular individual. As George Orwell wrote of Nazi Germany, “So long as the common man can get a hearing, such elementary rules as not fighting all your enemies simultaneously are less likely to be violated.”
Any society will have intellectual and political elites who are properly expected to provide leadership. Leaders can and should consult the best available data. But “shut up and listen to the data” is both undemocratic and ultimately unpersuasive. Indeed, one German study found that people were more likely to be skeptical when scientific pronouncements claimed to put a debate at its end. Even in situations such as a pandemic, where public-health experts know vastly more about disease than the general public, the other side of the ledger — the costs of economic and social lockdowns — involves a lot of the kinds of considerations that demand broader democratic input than social-science data alone can provide.
5. The Problem of the Non-Average Citizen
Data are aggregate, but human lives are not lived in the aggregate. Dataphiles are fond of reminding us that “anecdote is not the singular of data,” but the reverse is true as well: Data is not the plural of the individual. Even if banning or subsidizing some activity would benefit the average person, not everyone is the average person. Individual liberty lets people explore what is best for them, not best for The Average American. Opponents of school choice, for example, often cite studies claiming to show that the average student does no better in private than public schools. But even if we accept this as a premise, what about a mother who sees choice as better for her children? Dealing with public-policy questions solely in aggregate terms overlooks the ways in which “facts” that are true of society as a whole are false in the lives of individuals, families, and communities.
6. Who Watches the Watchers?: Appeals to Authority and Biased Experts
Public-policy data can be manipulated by people with biases, agendas, and axes to grind. Not only can studies be misused by politicians or based on bad data, they can also mislead deliberately. Any courtroom lawyer can tell you how easy it is to hire a credentialed expert for either side of any given debate, and people will do things for power that they would never do for money. Experts bearing data should not be able to wave away skepticism by brandishing credentials.
Every form of expertise and data still needs civilian oversight. Given power, or the ability to influence those in power, scientists and social scientists will act the way human beings have always acted around power. The temptation of the unrestrained expert comes in two stages. First, the expert in anything is prone to tunnel vision. The greater the expertise, the greater the risk. The expert is apt to have a limitless appetite for resources while ignoring competing social priorities. Experts may demand policies that maximize the ends sought by their discipline, while ignoring countervailing considerations and interests.
The expert who learns that the recitation of jargon and the appeal to authority effectively exempt him or her from moral or social scrutiny has made the most dangerous discovery known to man: the ability to get away with virtually anything. If people will let you talk your way into money and influence with good data on the grounds that they do not understand it or have no right to obstruct it, what is to stop the expert from using bad data, if the layman isn’t equipped to tell the difference between the two?
Oversight, however, is often lacking. Study after study pours out of journals in fields ranging from economics to medicine, many of which cannot be replicated. (One study concluded that “most current published research findings are false,” yet the incentive structure of publication rewards them.) Consumers of the studies may never realize this. In one depressing example, the Pentagon was inundated with so many studies that it commissioned a study in 2010 to determined how much it cost to produce all those studies; two years later, the GAO produced a study of the study about studies, and concluded that the Pentagon was unable to “readily retrieve documentation” for six of the nine studies it attempted to analyze.
7. The Law of Unforeseen Consequences: Static Data in a Dynamic World
The current relationship between points of data can also change when policies introduce new incentives. Changes to taxes and benefits are well-trod examples. Projections that the government will “lose” a fixed sum of money from cutting taxes, or that a program will cost a certain sum, are presented as fact. But if the rules change, people’s behavior will, too. Phil Gramm once illustrated the absurdity of Congress’s tax-scoring system by asking it to score the increased revenue if taxes went to 100 percent of income: The system was required to assume that everyone would work just as hard for no money. “Dynamic” scoring nods in the right direction, but the real answer is that there are almost always too many moving parts to project the effect of changes with enough confidence to hang a dollar sign on it.
Critics of lockdowns and social distancing fall into this trap as well, triumphantly claiming that declining infection rates show that those measures were not needed, when what is really shown is that they worked. The same is true of critics of incarcerating criminals as crime rates fall. The Bush-era term “reality-based community” referred to people who were stuck measuring the world instead of observing how it changed in response to actions; it was embraced, without apparent irony, by many of the self-proclaimed Dataphiles. In the real world, policy changes cause changes in the reality the data measures.
8. The Unreliable Narrator: Comparing Data from Deceitful or Inconsistent Sources
Comparing international data on the coronavirus should teach us a lesson that applies in many other areas: Beware of Chinese data. And it’s not just China. Many sources of information at home and abroad are unreliable. Garbage in, garbage out. In areas such as opinion-poll aggregation, a sound methodology for balancing out the random biases of polls assumes that the biases are random and the polls are genuine — giving bad actors an incentive to game the system unless some subjective controls are used to weed out bad data.
The politicization of every corner of American life these days, especially from the Left, leads inevitably to taboos on even collecting data that might prove politically inconvenient. Again, the coronavirus provides examples. New York mayor Bill de Blasio has instructed contact tracers not to ask about whether people attended Black Lives Matter protests, because it might be embarrassing to discover that such protests spread the virus. Clinical trials of hydroxychloroquine have been stymied because anti-Trump scientists did not want to risk positive findings on a drug touted by Trump, while leading medical journals and policymakers relied instead on “studies” by a company that appears to have no business producing science.
It’s not just dishonesty, but also different standards for gathering data. One of the things that have been clear from efforts to drill into the international data on infections and deaths is that different countries have different levels of testing and different standards for classifying deaths as related to the coronavirus. A related issue plagues climate-change debates: The resistance of leading climate modelers to make their underlying hard data transparent sets off alarm bells about the reliability of the data itself. Even when that kind of behavior is not cover for outright cooking of the data, it often conceals the gray area between hard data and assumptions. In the climate area, for example, there are extensive disputes about comparing modern global temperature measurements (which are collected by sensitive instrumentation) to historical data, which must be extrapolated from indirect evidence.
9. Custer’s Last Data Point: The Need to Act on Uncertainty
A big challenge for leaders today is whether to take dramatic actions, such as ordering the closure of businesses, churches, and schools, on the basis of mathematical models that are — even in the best of times — educated guesses resting on a lot of assumptions. These are not the best of times. The rapid spread of infectious disease means that by the time we have solidly grounded models to work with, it is already too late. “But the models are bad” misses the point: No model could ever be good enough fast enough. In a fast-moving crisis, we must fall back on instinct and tradition because the data we have are too fragmentary and evolving. Waiting for good data is not an option; leaders must act, or decide not to act, on the basis of the information they have and fill in the gaps with their judgment. Fortunately, human beings are wired to make decisions about threats without adequate data; our ability to do that has kept us alive as a species. There are still occasions when we need to trust that instinct.
People who pride themselves on working from data can find themselves at sea when there is not time to process it. Barack Obama was asked a question during the 2008 campaign about how he would handle a passenger jet barreling toward the Capitol, and his first response was, “What you want is somebody who is, first of all, going to get all the facts and gather up good intelligence. The second thing you want is somebody who is able to analyze the situation, the costs and benefits of action.” That reflex led him to wait eight months after the CIA reported a “strong possibility of the location of Osama bin Laden” before he authorized a raid. Had bin Laden moved in the interim, Obama’s instinct to wait for better information would have lost the opportunity.
10. No Data in Foxholes: The Problem of Violence
The persuasive force of data tends to be particularly inadequate when people feel themselves physically threatened. Much of politics deals in the primal fears that government is expected to hold at bay: crime, war, terrorism. Humans are programmed by long evolutionary experience to respond to such threats with a variety of mechanisms: fight, flight, closing ranks with our tribes, making a show of strength. Understanding the relative balance of threats is important, but we see persistently in debates ranging from terrorism to hate crimes to illegal immigration to Black Lives Matter that the power of these instincts regularly overwhelms cold-eyed analysis of threats. That is not always a bad thing when dealing with foreign threats; violence is itself a breaking of the norms of rationality, and those who pursue it should always live with the fear of provoking an irrationally excessive response. In democratic terms, policymakers need to account for the reality that voter concerns about violence will not just go away when exposed to data.
11. Believing Your Own Baloney: The Fallacy of Precise Predictions
Good predictive models are nothing more than predictions of probability, yet they often get presented to the public as if they were far more definite. The difference between “a 50/50 chance” and “a 63.5 percent chance” is that the latter fools you into thinking it’s more scientific.
People can be shocked when the likeliest outcome doesn’t happen. In 2008 and 2012, Nate Silver projected election outcomes with a high level of certainty and was right. In 2016, his final projection said that there was a 28.6 percent chance of a Trump victory. Given that Trump basically won by drawing to an inside straight, winning Michigan, Pennsylvania, and Wisconsin by less than a point each, the outcome is entirely consistent with something that happens a little more than one time in four. Yet Silver got roasted for not “calling” a Trump win.
In 2009, the Obama White House pushed a projection of what would happen to the unemployment rate without the stimulus. This could not be projected with any kind of precision, but conservatives dined out for years comparing the projected “without stimulus” unemployment curve to the actual, higher unemployment rate. The White House felt compelled to compile statistics of jobs “created or saved,” which counted as many existing jobs as needed as “jobs saved.” The entire fiasco started with trying to put hard numbers on hope.
12. Selection Bias in a Watched World
Both social science and opinion polls frequently suffer from an inability to study a representative sample of the public. Exit polls may reach more younger and educated people who like answering questions. Psychology experiments tell us what college undergraduates volunteering for studies will do. This problem gets worse when the people doing the reporting know they’re being watched. People not only give answers they want heard, they are more likely to sign up for studies if they and the people running the study have the same goals. When self-reporting turns to self-selection, alarm bells should go off.
13. Temporary Data, Permanent Conclusions: Turning Changeable Data into Unchangeable Policy
In 1954, the Supreme Court cited social-science data on educational outcomes in its Brown v. Board of Education decision striking down school segregation. Brown should not have needed data to say that the government may not discriminate on the basis of race. The mischief of giving a starring role to data has multiplied in later cases. Roe v. Wade rested largely on the state of neonatal medicine circa 1972, which is why its trimester framework had to be junked 20 years later — yet the Court did not change its conclusions. The run-up to the Obergefell same-sex-marriage decision was festooned with rushed efforts to build a temporary scholarly “consensus” on same-sex family life, with the understanding that it wouldn’t matter what further studies came out after the decision. The nature of constitutional decision-making is that precedents don’t get easily revisited, so building them around the scientific data available at a particular moment can cast data in stone long after it has been debunked. The changeable nature of any honest dataset makes it a poor basis for the writing of permanent law.
14. Schrödinger’s Data: The Scoring System Drives the Policy
Washington policymaking presents a special hazard: the need to build policy proposals around getting a required budget number, rather than around what will work best. The congressional process for scoring the projected cost of some program or other is so much hokum, using artificial frameworks such as evaluating a permanent program by its ten-year cost. Yet lawmakers let it dictate how they write laws. The Bush tax cuts delayed their implementation — against the best economic judgment of virtually everyone who wanted tax cuts — in order to be “scored” as costing less. Obamacare scored ten years of revenue against six years of spending and included all sorts of other accounting gimmicks that nobody would ever have included if the goal was just the best health-care bill. Letting the scoring system drive the policy is backwards.
15. Passing the Torch: Losing the Habit of Principle
Overrelying on data gets you out of the habits of mind that require going back to first principles to remember what is really important. Humans naturally think in narratives and morality. The clinical recitation of data leaves us cold. Data analysis is a tool, and we can use it to apply principles and clarify the lessons taught by history, tradition, and experience. But it is not a substitute for those principles and lessons. Data can aid in persuasion, but the mind will not move where the heart does not desire to go. Our government is not merely a matter of technocratic administration by experts; it is and must remain a democracy because everything it touches raises deep claims of liberty, security, property, and justice. Data can measure the progress of our journeys, but it cannot tell us where we should go.