Earlier this year I raised some concerns about a study reported in the Washington Post. It had found that when counties hosted a Trump rally in 2016, they saw a 226 percent increase in hate incidents. The Post had allowed the researchers to present the findings to the public despite the fact that their full paper had not yet been released. And the study relied on data from the Anti-Defamation League that had been called into question.
When the paper was released in draft form, I became even more skeptical, for technical reasons I tentatively laid out in a Twitter thread. (Tentatively because I am a lowly journalist, not a statistician.) For one thing, while the authors claimed that counties experienced an “increase” in hate incidents following a rally, the methods don’t actually measure an increase within counties; they compare rally counties to other counties. And two, in making this comparison, the authors didn’t fully account for the fact that rally counties have higher populations than other counties on average and would be expected to have a higher raw number of hate incidents for that reason alone.
I emailed the authors of the paper, who were happy to discuss concerns about their methods but said they couldn’t provide their data yet because the study is still under peer review. I put the project on the backburner. But then the Harvard Ph.D. candidates Matthew Lilley and Brian Wheaton did the yeoman’s work of reassembling the data set from scratch and posting it online for all to see and download. In Reason they wrote that if you apply the authors’ same methods to Clinton rallies, you find a similar “effect” on hate incidents. And now they’re out with a more detailed paper applying methods that really do measure changes within counties — finding no effect from the rallies.
I find their paper compelling, but as is always the case with these things, we’re probably in store for a very long debate over the best way to analyze these data. The authors of the original study sent me this response to the Reason piece last night:
We have spent a considerable amount of time evaluating the positions of Lilley and Wheaton, specifically their methodological approach, their critiques of our approach, and their comments on the state of academia and the media. We intend this to be our official response to any set of inquiries on this issue.
First and foremost, we have run numerous models controlling for different demographic variables that include county population, the percentage urban of a population, and the size of the target population. All of those models find that Trump rallies have the effects consistent with our initial findings. This has been further confirmed by using data from the government’s official documentation of hate crimes in the FBI UCR hate crime report. Additionally, none of the variables in any of our models have been transformed. This includes a fixed-effect model that looks at within county variation thus making any population or logged population variable irrelevant — counter to Lilley and Wheaton. Further, we contend it is incorrect to include monthly controls in a fixed-effects model where the dataset is already set as panel data.
Second, the primary model we used in our analysis already accounts for overdispersion. Therefore, the decision to log the population is counterintuitive and questionable from our perspective. If logging the population is the only way Lilley and Wheaton make the Trump rally effect insignificant, then this is particularly problematic when referencing the numerous other models we used to identify the initial finding. Further, Lilley and Wheaton should provide an accurate explanation — theoretical and empirical — of the need to use the log transformation of the county-level population.
Third, we are unsurprised that logging the population will washout the effect of Trump rallies on hate crime. This is because the logged population serves as a mediator variable. The importance and impact of mediating variables is well documented (e.g. Baron and Kenny, 1986). We followed all four steps necessary to determine whether the logged population is a mediator and established that it is, in fact, mediating the relationship between Trump rallies and hate crimes. This auxiliary analysis confirms that regardless of the transformation of the population variable, the impact of Trump rallies on county-level hate incidents is positive and significant.
Fourth, theoretically, we contend that controlling for the percentage urban of a population represents a better control variable than the logged population of a county. Hate incidents are more common in urban areas than in rural areas. When using this as a control in auxiliary analysis, the findings remain directionally consistent and statistically significant.
Fifth, Lilley and Wheaton, and individuals citing their piece have engaged in a variety of commentary regarding the makeup of the media and academia. This is, of course, their right. Ultimately, what individual scholars, the media, and policymakers choose to be of interest to them is their right, and is also open to scrutiny. However, any insinuation that our individual analysis is a result of our bias or bias within these enterprises is at best inaccurate. We studied identity issues, prejudice, and hate crime long before Trump was a candidate for political office, let alone the President. Our research was motivated by an academic research question that falls squarely within our discipline: does certain political rhetoric have behavioral consequences? We engaged in our research without being clouded by any political biases. Regardless of the findings of our analysis, we would have actively pursued the publication of our research, indicated by our collective history publishing dozens of peer-reviewed articles. In the same vein, one of us has published an article showing that the inclusion of far-right political parties in governing coalitions has no effect on anti-Semitic incident variation in Europe. We fully and passionately reject any notion outright that our research is biased or was motivated by bias.
Finally, it is worth noting with tremendous irony that as a consequence of Lilley and Wheaton’s article, we have been on the receiving end of vile comments and threats from self-identified angry Trump supporters via email and telephone. Some of these comments have used extremely offensive prejudicial language to target our perceived ethnicity and gender.
Update: I don’t want to turn this post into a letters exchange in an econ journal, but here’s a brief response from Lilley and Wheaton:
Feinberg et al. continue to claim that it is somehow acceptable to believe that Trump rallies cause hate crimes without explaining why their methods also find that Clinton rallies ostensibly had the same effect, or even more absurdly, that both future Trump and Clinton rallies cause hate crimes in the past. The fact that these nonsensical results arise using their methods shows that these methods are flawed. That they stand by their original claim while refusing to engage with these inconvenient results is tantamount to saying that it is acceptable to pick and choose which statistically-significant results to believe and which to discard.
Much of the remainder of their statement is dominated by a laundry list of mistaken arguments about statistical methodology. While we have responses to each, this risks detracting from the central point. At no point have Feinberg et al. been able to explain, nor appeared particularly interested in engaging with, why their method finds that Clinton rallies had equivalent effects. By contrast, we do have a simple explanation for these results. Their methodology is flawed, and flawed models produce erroneous results.