Interesting points from a reader:
I’d be inclined to take Green’s state-level findings with a grain of salt.
Let’s put it this way: 3,500 is a terrific sample size that allows you to confidently draw conclusions with a very low error — for the sample as a whole. But let’s say he divided (i.e., stratified) that sample across the 50 states. That would leave just 70 interviews per state. Then let’s say he further stratified this by, say, the six religious segments mentioned in the article.* Prof. Green would then average only 12 people in each sub-sample.
So, he could then be 95% certain of the following:
That his overall voter preference estimate is accurate to within +/- 2 pct pts
That his religion-level voter preference estimate is accurate to within +/- 4 pct pts
That his state-level voter preference estimate is accurate to within +/- 12 pct pts
That his state-level AND religion-level voter preference estimate is accurate to within +/- 28 pct pts
In other words, Professor Green can very credibly say “Kerry beats Bush” overall among the survey respondents (I’d hope he screened for likely voters). But he’s skating on thin ice when he says, “Kerry is beating Bush among Centrist Catholics 45 percent to 41 percent,” since it’s statistically a wash. And he’s really, really out on a limb making any claim whatsoever about, say, white evangelicals in Missouri.
Now, he would probably respond to this in two ways:
1. That he is making statistical claims for the religion sub groups only, and drawing statewide conclusions based on their known representation in the state. This would assume that religious affiliation is the only thing that drives voting behavior, not geography… and that’s a stretch.
2. That he didn’t divide all 50 states equally, and instead gave more weight to states with larger populations. This strengthens the results for California, but makes the sample sizes for states like Iowa even smaller. Unless he had the foresight to disproportionately weight the sample sizes of the contested states, which I doubt.
My overall point: Professor Green’s study certainly raises questions for Rove and company about their strategy. But let’s not get carried away with the statistics just yet.