"Pollsters' Dirty Little Secrets" by John Leo, from the U.S. News & World Report This is a meditation on opinion polls, and how some polls are more honest and more valuable than others. A week ago, as I sat down for a panel discussion on the "Sensation" exhibit at the Brooklyn Museum of Art, someone handed me a fresh poll commissioned by our host, the Freedom Forum's First Amendment Center. The pollsters said it showed that the public was overwhelmingly on the side of the museum -- 85 percent believed Americans have the right to judge controversial exhibits for themselves, and 59 percent said government should not be able to ban such exhibits at museums receiving public funds. Later, after I took a long look at the survey, the results seemed much more ambiguous. Early questions showed that majorities, ranging from 54 percent to 64 percent, disagreed with the statement that "people should be allowed to display in a public place art that has content that might be offensive" to minorities, women, religious groups, or simply "to others." This opinion, if applied rigorously, would probably empty out all public spaces except children's museums. But, in the context of the dispute, it either meant that a lot of people were backing Mayor Giuliani or at least had mixed feelings about deeply offensive art. This aversion to offensive art, however, was cut short by a number of changes in the language of later questions. The phrase "people should be allowed to display" was replaced with "government should be able to ban art in public museums" (67 percent said no). No surprise there. Faith in government is low, so questions about "government" controlling anything at all are likely to draw heavy negatives, particularly when the word "ban" is nearby. These negatives might have been avoided by simply dropping the G-word: "Should museums be allowed to display" the art. Like a lawyer leading a witness, the poll asked whether government has the right to ban books from public libraries and plays from public auditoriums. While interviewees were wallowing in outrage over the idea of our arrogant government even thinking of such stupidity, the pollsters decided to spring the first question about public funding: "Government should be able to cut funding" to museums staging offensive exhibits (61 percent said no, actually a low figure given the set-up questions). Another verbal maneuver: Pollsters know that questions beginning with weak words like "regardless" and "whatever" cue people to ignore the clause that follows and agree with the emphatic main part of the statement, which comes last. So it's no surprise that 73 percent agreed that "Regardless of how I feel about the art exhibit itself, banning art in public places is something that violates Americans' right of free expression." Remember, a majority had already said that people shouldn't be allowed to display offensive art in a public place. Here the "regardless" sentence structure and the strong language of "ban," "right" and "violate" both work to get a different result. The spin put on the survey by the pollster, the Center for Survey Research at the University of Connecticut, did not reflect the public's conflict and confusion over the issue. Even by the center's own numbers, the public was saying that the Brooklyn Museum had a right to put on the show, but it really shouldn't have done it. Regardless of one's opinion on the Brooklyn Museum's exhibition (whatever), this sort of thing is almost enough to make us call for a national center to monitor polling excesses. Pollsters, however, do monitor one another now and then. After a "whatever" question in a National Law Journal poll produced 75 support for jury nullification, pollsters at Frederick Schneiders Research in Washington, D.C., showed that when you reverse the "whatever" sentence structure, you also reverse the poll result: 75 percent oppose nullification. In a 1993 poll by Roper Starch Worldwide, 22 percent of Americans said yes to a question on the Holocaust framed in tortured language featuring a double negative ("Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"). Gallup got a similar result using the Roper language. But when Gallup asked the question in plain language, less than 0.5 percent said the Holocaust definitely never happened, and only 2 percent had real doubts. Yankelovich Partners did a good job of deflating Ross Perot's 1993 poll showing strong support for Perot positions on issues such as debt reduction and trade agreements. Perot got 67 percent to support a debt-reduction plan calling for $2 in spending cuts for each dollar of tax increase. But the poll gave no clue as to where the cuts would come from or what the social costs might be. Yankelovich's poll showed that half of public approval disappeared when the question added the words "even if it meant cuts in domestic programs like Medicare and education?" One problem with polls is that all questions and answers are usually distilled down to what the pollster says they all mean. This is distilled down even further to a press release that will pretty much determine how reporters treat the story. Another common ploy is not making the raw data available to the media, thus forcing reporters to work with some sketchy material and an overstated press release. "There are all sorts of dirty little secrets in polling," says Robert Lichter of the Statistical Assessment Service. "Readers have to be their own editors. Don't look at the interpretation. Look at what people actually said."
Return to rants