The summer reading assignment for my introductory statistics class was a short book entitled How to Lie with Statistics. Before we ever took our first random sample or ran our first regression line, we learned how questions and numbers can be used to affirm any position (and yet for some reason the professor docked us points if we tried to do such on exams).
In the recent Wisconsin budget saga, pundits and politicos on both sides of the debate attempted to reinforce their positions with public-opinion numbers. Rachel Maddow has led off numerous shows in the past few weeks with the latest poll from Wisconsin, and Nate Silver has critiqued Rasmussen’s early poll on Walker and unions. This month, the Wisconsin Policy Research Institute added another statewide poll to the mix. Partisans have cherry-picked numbers out of it — that “65 percent think Governor Walker should compromise” or that a majority disapprove of Senate Democrats’ leaving the state. Fortunately, there is more information within the poll release than partisan positioning.
Two aspects of WPRI’s release make it notable over previous polls on the subject: They included two differently worded questions in an “experiment to examine responses to different ways of framing a key element of the budget repair bill,” specifically collective bargaining; and they released the full crosstabs of their data, revealing the beliefs of different demographic groups. Question framing and wording are often cited as reasons to dismiss poll results when they are unagreeable, though rarely is there a direct comparative sample to see if such dismissals are correct.
The first question in the experiment asks if respondents strongly favor, somewhat favor, strongly oppose, or somewhat oppose the following aspect of the bill, as described in a Walker-friendly statement (let’s call it statement A):
Limiting most public employees’ ability to negotiate over non-wage issues in order to prevent local union affiliates from obstructing the budgeting process for local governments.
The results found a statistical tie: 47 percent strongly or somewhat in favor of the statement, and 50 percent somewhat or strongly opposed.
But when the other half of the sample was given a union-friendly version of the statement (statement B) …
Stripping most public employees of their right to collectively bargain over benefits and working conditions as part of a ploy to eliminate public employee unions altogether
… there was a 25-point swing in the favor/oppose margin. Only 32 percent favored the second statement, and 58 percent opposed it.
An interesting demographic aspect of this swing is the shift in opinion among the college-educated. Forty percent of those with an undergraduate or higher degree favor statement A. When we turn to statement B — where overall favorability fell by 15 points — the favorability of those with college or graduate degrees marginally increased (42 percent).
Gallup discovered something similar regarding the wording and results in their union questions. On February 21, they found that only 33 percent of Americans favored a Wisconsin bill “that would take away some of the collective bargaining rights of most public unions, including the state teachers’ union,” and 61 percent opposed it. Less than two weeks later (March 3–6), a question on the same subject found 49 percent favored “changing state laws to limit the bargaining power of state employee unions,” and 45 percent opposed. The 34-point swing between the two Gallup questions is even larger than the swing between the two WPRI experiment questions.
As the union debate has spread to Ohio, so has this polling discrepancy. Quinnipiac was even more explicit than WPRI with their experiment question. They asked:
As you may know, there is a proposed law in Ohio that would limit collective bargaining [rights] for public employees. Do you support or oppose limiting collective bargaining [rights] for public employees?
The question was asked in two forms (to half the sample respectively), one with the word “rights” and one omitting it. When “rights” was included in the wording, 54 percent opposed the law, a 19-point margin over the 35 percent that supported it. When the word “rights” was omitted, opposition fell to 48 percent and the margin fell to only seven points (41 percent favored the law). That’s a twelve-point swing in the difference of opinion when only six letters are omitted!
The essential difference in these questions is the wording “collective bargaining rights.” Statements with those words, in some order, suggest clear support for unions, whereas questions omitting the words “collectively” or “right” show mixed opinion. Gallup observes that their “differing results likely reflect Americans’ sensitivity to nuances in how the debate can be framed. They may also indicate the high and low boundaries of support for setting new limits on collective bargaining.”
The wording of questions matters, not just the percentages that follow them. So, too, does looking beyond the state-level responses to see how different groups react. The certain takeaway from these polls reaffirms the old adage popularized by Mark Twain, that there are “lies, damned lies, and statistics.”