The Perils of Polling

During the past six months, 1,700 of the stories we published in The Oregonian's news sections contained some reference to polls or surveys.

Surprised? Don't be. We're more typical than not. When Arnold Ismach, dean of the University of Oregon journalism school, asked students to check five days' worth of news stories in 14 papers, they discovered that nearly 1 in 5 had some quantitative research element. Usually, that means a poll or a survey.

Survey research is a big deal these days. It influences elections, helps determine government policy and shapes our view of the world. The volume just keeps growing. More and more, survey research makes news, and not just on regular news beats. Surveys play an essential part in arts and music criticism, lifestyle coverage, medical writing and business reporting.

Now that we're headed into the heat of the election season, we'll be faced with an especially intense onslaught of polls. It follows that we should be as savvy and skeptical of survey research as we are of more traditional news sources.

We have good reason to be. Even respectable pollsters are often way off the mark. Early polling on Washington state's initiative to limit terms of political office showed 70 percent support. It failed. Missouri surveys on an initiative to increase school taxes showed 2-1 support. It failed, too. And recent polls on political races involving Jesse Helms, Texas Gov. Ann Richards and David Duke were off by large margins, as well.

We'd do a better job if we knew more about polling and included more safeguards in our reporting.

The AP Stylebook lists several suggestions for handling survey research, and anybody who hasn't checked it lately should consult the "Polls and Surveys" entry before writing or editing a poll story. In addition, you should pay special attention to these key questions:

• Is it really a survey?
  We're not terribly careful when it comes to how we use the terms "poll" and "survey." Those two words are, for the most part, interchangeable, although "polls" usually appear in the context of a political campaign. The important point is that both involve scientific sampling as a way of estimating figures for a larger population. Man-on-the-street interviews are neither polls nor surveys. Neither are call-in votes. Remember, too, that haphazard or self-selected samples like those are not random samples, which are scientifically selected so that every respondent has a known probability of being included.
  
  Note that a huge self-selected sample is usually less accurate than a much smaller probability sample. The infamous 1936 Literary Digest poll, which picked Alf Landon for president, drew a sample of more than 2 million.
  
  That doesn't mean that we shouldn't do man-on-the-street interviews, call-in programs or our own non-scientific studies. After the Nathan Thomas shooting, we contacted 12 police departments to check their records on use of deadly force. That wasn't a scientific survey. But it sure was good reporting.
  

  The important thing is that we don't represent non-scientific samples as anything more than what they actually are. So we shouldn't ever refer to one of our phone-in reader-participation efforts as a poll or a survey.

  
  The important thing is that we don't represent non-scientific samples as anything more than what they actually are. So we shouldn't ever refer to one of our phone-in reader-participation efforts as a poll or a survey.
  
  And don't forget that counting everybody, as the State of Oregon did when it checked to find out how many homeless people were in shelters on a single night, isn't a poll or a survey either. It's a census. We overlooked that fact we when we headlined the story, "One-night survey finds 3,200 people homeless."
  
  
• Who sponsored the poll?
  One of the stylebook's most important points is that we should always identify the poll's sponsor, which bears directly its credibility.
  
  An AP report we published this fall (11/15/'91, A15) is a case in point. The story's main conclusion? "Americans who think they'll be sitting pretty when it comes to covering health costs during their retirement may instead have to turn quickly to welfare." The source? A study conducted by Northwestern National Life Insurance Co., which had an obvious self-interest in the subject.
  
  On its face, the "study" is nonsense. The only survey-research result reported was that of workers over 40, "most believe they will be well-prepared for retirement." The insurance company also reported that only a minority of private companies provide after-retirement medical insurance. Then they proceeded to wave the bogeyman of welfare around while they conveniently ignored the fact that virtually all retirees are covered by Medicare, Medicade and, often, some form of personal health insurance. We helped them out by passing along a one-source story rife with conflict of interest.
  
  If some local insurance salesman had walked into the office with a similar claim, we'd have laughed him right out into the street.
  
  At least we named the sponsor, even though we failed to highlight its direct self-interest in the study's outcome. All too often, we allow sources to toss around completely anonymous poll results.
  
  
• Do the data justify the conclusions?
  Not only do we often swallow anonymous polls or dubious surveys from self-interested sponsors, but we also have a weakness for inflated conclusions that go way beyond a study's hard data. When we reported that a UC San Diego study of 242 upper-middle-class women showed that those with jobs had lower cholesterol and blood-sugar levels, our headline concluded that Work is healthy.
  
  Tell that to a coal miner.
  
  
• Is our math OK?
  We also have a hard time reporting numbers accurately. For example, we frequently confuse percentages and percentage points. We reported that the number of full professors in the Oregon State System of Higher Education had increased 5 percent over 10 years. (1/26/92, B4) In fact, the number had increased from 6 percent of the total to 11 percent, an increase of 5 percentage points. Assuming that total number of full professors was fairly stable, that's an increase of nearly 100 percent.
  

  When we do report numbers accurately, we sometimes fail to keep them in perspective.

  
  
• Is our report balanced?
  When we do report numbers accurately, we sometimes fail to keep them in perspective. The rate of pregnant Oregon women carrying the AIDS virus doubled from 1989 to 1991, and our story's headline and lead focused on that fact. But the total number of infected mothers-to-be was 34, a tiny, tiny percentage of the total number of both pregnant women and infected Oregonians. Each case is a tragedy. But was this angle the news in this story?
  
  
• What were the actual questions?
  We almost always leave out the exact questions used in a poll or survey, even though the national association of professional pollsters asks that they always be included. Sometimes question wording is critical. Lyndon Johnson flim-flammed the press in 1968 when he released a poll that showed him leading four Republicans in New York state. What most news stories failed to note was that the poll questions didn't mention Nelson Rockefeller, the state's favorite son.
  
  Differences in question wording also explain why the president usually gets a better job-performance rating from Gallup than from Harris. Gallup asks respondents whether they approve or disapprove of the job the president is doing. Harris asks respondents to choose a rating from among four choices ("excellent," "pretty good," "only fair," "poor"). That spreads out the responses and puts fewer into the positive categories.
  
  Inept or malicious question wording can influence results, too. Watch out for double-barreled questions, imprecise questions and loaded questions.
  
  
• What's the margin of error?
  Probably the best-known source of error in virtually all polls is simple chance -- the likelihood that the poll sample doesn't accurately reflect the entire population. We express that likelihood as the level of confidence and the margin of error. Usually pollsters reveal that there's a 95 percent chance (the confidence level) that the poll results are within plus or minus X percentage points (the margin of error) of the actual values for the whole population. Both the stylebook and the professional pollsters say we should include both figures in every poll or survey story.
  
  But we often leave them out, even when they're essential to interpreting the results. For example, when we reported the results of a New York Times/CBS News poll on the Clarence Thomas nomination, we said that 22 percent of the men and 18 percent of the women polled were against the nomination. But we reported no margin of error. Chances are that it was at least 3 percent. That means the figure for the men could have been as low as 19 percent, and the figure for women could have been as high as 21 percent. So it's well within the realm of possibility that, in fact, more women than men were against the nomination.
  
  
• When was the poll taken?
  Public opinion is volatile, and it often keeps changing right up to election day. That's why it looked as though David Duke would do much better than he actually did.
  
  Early polls are especially likely to mislead on initiative campaigns because supporters generate most of the early publicity. Once the opposition gets organized, things can change in a hurry. Which is why the early reports on public opinion in the Washington state term limits initiative were so far off the mark....
  
  
• What was the response rate?
  Lots of commercial mail surveys draw an extremely low percentage of returns. And the lower the rate of return, the more likely that the responses represent some systematic error. Maybe most of the replies came from those with the most extreme opinions. Or from the richest respondents. Or from Republicans.
  
  In general, be suspicious of response rates below 70 or 80 percent.
  
  Lots of other variables affect survey results, and no news story will mention them all. We don't want to bore readers by mentioning every little thing that went into a poll. But we should nonetheless ask plenty of skeptical questions before we decide that a poll is worth reporting.
  
  Remember, for example, that respondents sometimes lie when they have good reason to do so. Only 19 percent of Louisiana's voters admitted that they cast ballots for David Duke in last fall's primary. But Duke got 32 percent of the vote.
  
  The order of the questions matters, too. One survey asked respondents to agree or disagree with the statement that "it's all right for Japan to impose import quotas." The rate of positive responses was far higher when the statement followed on that said "it's O.K. for the United States to impose import quotas."
  
  Intensity matters as well. If a small group believes something passionately, that can have more impact than if a lot of people feel lukewarm about something. Jesse Jackson almost always gets more votes than the polls predict because his supporters are the most likely to vote.
  
  Lots of other things go into the mix. The kind of interviewing -- phone, mail or in-person -- may have an impact. And a skeptical reporter will always ask whether the folks who administered the questionnaires were audited to make sure they didn't make up answers or bias results. We should also always ask whether a survey turned up anything that the pollster isn't reporting. Even accurate results can be misleading if they only reveal part of the picture.
  
  
• And, lastly, we should remember that survey research isn't the same thing as common sense.
  Philip Meyer, an early proponent of using survey research methods in journalism, points out that no computer ever managed a political campaign. And he cites an old story about Harvard professor Edward Banfield and his comments to the students in his voting-behavior seminar. No matter what computer tables told them about the peculiarities of Chicago voters, Banfield told his students, they still would know less than Mayor Daley.

Back to Numeracy