"Pollsters' Dirty Little Secrets"

by John Leo, from the U.S. News & World Report

     

     This is a meditation on opinion polls, and how some polls are more
honest and more valuable than others.

     A week ago, as I sat down for a panel discussion on the
"Sensation" exhibit at the Brooklyn Museum of Art, someone handed me a
fresh poll commissioned by our host, the Freedom Forum's First
Amendment Center.  The pollsters said it showed that the public was
overwhelmingly on the side of the museum -- 85 percent believed
Americans have the right to judge controversial exhibits for
themselves, and 59 percent said government should not be able to ban
such exhibits at museums receiving public funds.

     Later, after I took a long look at the survey, the results seemed
much more ambiguous.  Early questions showed that majorities, ranging
from 54 percent to 64 percent, disagreed with the statement that
"people should be allowed to display in a public place art that has
content that might be offensive" to minorities, women, religious
groups, or simply "to others."

     This opinion, if applied rigorously, would probably empty out all
public spaces except children's museums.  But, in the context of the
dispute, it either meant that a lot of people were backing Mayor
Giuliani or at least had mixed feelings about deeply offensive art.

     This aversion to offensive art, however, was cut short by a number
of changes in the language of later questions.  The phrase "people
should be allowed to display" was replaced with "government should be
able to ban art in public museums" (67 percent said no).  No surprise
there.  Faith in government is low, so questions about "government"
controlling anything at all are likely to draw heavy negatives,
particularly when the word "ban" is nearby.  These negatives might have
been avoided by simply dropping the G-word: "Should museums be allowed
to display" the art.

     Like a lawyer leading a witness, the poll asked whether government
has the right to ban books from public libraries and plays from public
auditoriums.  While interviewees were wallowing in outrage over the
idea of our arrogant government even thinking of such stupidity, the
pollsters decided to spring the first question about public funding:
"Government should be able to cut funding" to museums staging offensive
exhibits (61 percent said no, actually a low figure given the set-up
questions).

     Another verbal maneuver:  Pollsters know that questions beginning
with weak words like "regardless" and "whatever" cue people to ignore
the clause that follows and agree with the emphatic main part of the
statement, which comes last.  So it's no surprise that 73 percent
agreed that "Regardless of how I feel about the art exhibit itself,
banning art in public places is something that violates Americans'
right of free expression."

     Remember, a majority had already said that people shouldn't be
allowed to display offensive art in a public place.  Here the
"regardless" sentence structure and the strong language of "ban,"
"right" and "violate" both work to get a different result.

     The spin put on the survey by the pollster, the Center for Survey
Research at the University of Connecticut, did not reflect the public's
conflict and confusion over the issue.  Even by the center's own
numbers, the public was saying that the Brooklyn Museum had a right to
put on the show, but it really shouldn't have done it.

     Regardless of one's opinion on the Brooklyn Museum's exhibition
(whatever), this sort of thing is almost enough to make us call for a
national center to monitor polling excesses.  Pollsters, however, do
monitor one another now and then.  After a "whatever" question in a
National Law Journal poll produced 75 support for jury nullification,
pollsters at Frederick Schneiders Research in Washington, D.C., showed
that when you reverse the "whatever" sentence structure, you also
reverse the poll result: 75 percent oppose nullification.

     In a 1993 poll by Roper Starch Worldwide, 22 percent of Americans
said yes to a question on the Holocaust framed in tortured language
featuring a double negative ("Does it seem possible or does it seem
impossible to you that the Nazi extermination of the Jews never
happened?").  Gallup got a similar result using the Roper language. 
But when Gallup asked the question in plain language, less than 0.5
percent said the Holocaust definitely never happened, and only 2
percent had real doubts.

     Yankelovich Partners did a good job of deflating Ross Perot's 1993
poll showing strong support for Perot positions on issues such as debt
reduction and trade agreements.  Perot got 67 percent to support a
debt-reduction plan calling for $2 in spending cuts for each dollar of
tax increase.  But the poll gave no clue as to where the cuts would
come from or what the social costs might be.  Yankelovich's poll showed
that half of public approval disappeared when the question added the
words "even if it meant cuts in domestic programs like Medicare and
education?"

     One problem with polls is that all questions and answers are
usually distilled down to what the pollster says they all mean.  This
is distilled down even further to a press release that will pretty much
determine how reporters treat the story.  Another common ploy is not
making the raw data available to the media, thus forcing reporters to
work with some sketchy material and an overstated press release.

     "There are all sorts of dirty little secrets in polling," says
Robert Lichter of the Statistical Assessment Service.  "Readers have to
be their own editors.  Don't look at the interpretation.  Look at what
people actually said."
  

Return to rants