In May, the New York Times highlighted a study—or, rather, a summary of a study—by political scientists Lee Epstein, Christopher M. Parker, and Jeffrey A. Segal that purported to show that in First Amendment speech cases “the votes of both liberal and conservative justices tend to reflect their preferences toward the ideological groupings of the speaker.” Further, the study summary contended that for many justices—especially Scalia and Thomas but also Roberts, Kennedy, O’Connor, and Stevens—there was “a statistically significant difference … between support for the free expression claim when the speaker is liberal versus conservative.”
In my post about the article, I identified two obvious flaws in the study. First, Epstein, Parker, and Segal seem blind to the possibility that there may be factors other than speaker-favoritism or speaker-hostility in a large category of the cases (e.g., the institutional interests of the government), and those factors may have a very large disparate impact one way or the other in those cases. Second, they have no basis for assessing when a vote difference is statistically significant. (I note that their full study makes no assertions of statistical significance; it seems that they may have tarted up the summary that they specially prepared for the New York Times.)
Well, it turns out that the Epstein-Parker-Segal report is far shoddier than I imagined and that it suffers from what law professor Todd E. Pettys summarizes in his devastating critique as a “surprisingly sizable combination of coding errors, superficial case readings, and questionable judgments about litigants’ ideological affiliations.” Among the lowlights (from Pettys’s review of Epstein-Parker-Segal’s “treatment of roughly a quarter of the more than 500 cases in their study”):
In a number of cases, Epstein, Parker, and Segal made simple errors in coding justices’ votes. (Pettys, pp. 15-17.)
In numerous cases, the authors incorrectly classified speakers’ ideological identities. In several cases, for example, they seem not to have looked beyond the speaker named in the case caption to discover that the First Amendment claimants included multiple speakers of various ideologies. (Pettys, pp. 18-20.) In many others, they seem to have made superficial assumptions (e.g., a case with college students must involve liberal speech) that more careful scrutiny of the particular speakers undermines. (Pettys, pp. 20-23.)
In plenty of other cases (Pettys, pp. 24-33), the authors’ classification of speakers’ ideologies is questionable. As Pettys points out, this is no minor flaw:
If one cannot be reasonably sure whether a justice perceived a given speaker as a member of an ideological in-group or out-group, one cannot confidently use the justice’s treatment of that speaker to measure the justice’s tendency toward ideological in-group bias.
Two amazing examples: David Dawson (of Dawson v. Delaware) was a convicted murderer on death row who belonged to a white racist prison gang and who called himself a disciple of Satan. Epstein, Parker, and Segal classify him as a conservative speaker because they code racist communication and racist behavior as conservative. For the same bizarre reason, they code as conservative an African-American man “whose sentence … had been enhanced because he selected his victim (a young white boy) on the basis of his race.” (Pettys, p. 25.) As Pettys asks, are we really to believe that any conservative justice would see either speaker as part of his own ideological in-group?
The authors’ errors are so pervasive—and splatter in so many different directions—that it’s anyone’s guess whether they have any impact on the authors’ (already dubious) bottom-line assessments. Looking at the manageable universe of votes of Roberts and Alito, Pettys concludes that it is the campaign-finance cases—in which they have both “treated ideologically diverse litigants even-handedly”—that account for most or all of the seeming disparity in their voting records. (Pettys, pp. 43-44.) (He also points out that sexual-speech cases, in which Scalia’s originalism amply explains his votes, may account for much of the supposed disparity—again, unmeasurable against any objective benchmark—in Scalia’s votes. Pettys, p. 49 n. 349.)
Pettys sums things up very nicely:
In light of the difficulties in Professors Epstein, Parker, and Segal’s study, we find ourselves confronting an ironic twist: working in tandem with the temptations of motivated reasoning, the very same sort of biases that the authors aimed to measure on the Court may have helped predispose many writers and readers to be too quick to embrace the study’s uniformly damning critique of the Court’s currently sitting conservatives.