When Numbers Lie Figures don't lie, goes the adage, but liars can figure. Which is precisely why we need to be a lot more skeptical of the numbers we publish.... Especially numbers that supposedly measure things such as nutrition, environmental hazards and health risks. One gripe about modern newspapers is that they carry an unrelenting onslaught of threatening -- and often-contradictory -- information. This study says salt causes high blood pressure ... that study says it doesn't. This scientist says an earthquake will strike ... that scientist says the prediction is nonsense. This nutritionist says you should avoid butter ... that one says you shouldn't bother. And so on, a daily deluge of threats that leave readers feeling badgered, bullied and bewildered. Who can blame them for throwing down the paper in disgust? But there are ways to sort through all those claims. Victor Cohn's "News and Numbers" is a guide to dealing with the conclusions reached by all the scientists, doctors and demographers who are out there counting, weighing and timing everything in sight. All of them have vested interests in proving their points. Sometimes they're liars who figure. Sometimes they're just humanitarians blinded by their own beliefs. Once in a while, they're actually right. We have "News and Numbers," in our Writer's Library. And we also have "Reporting on Risk," another Victor Cohn book that suggests ways for getting past the numbers to find the truth. Cohn, now a senior writer at the Washington Post, was the Post's science editor. So he brings a journalist's perspective and skepticism to the subject. One or both of his books are essential reading for every one of us who deals with science, medicine, nutrition or any other field heavily influenced by scientific research. Copy editors who deal with wire stories on research results should be especially wary. We pass along too much dubious information that serves only to frighten readers despite sketchy or nonexistent evidence. * * * The point is well illustrated by the catchy story we put on A1 a couple of months ago, the one that claimed left-handed Americans lived, on average, nine years less than right-handers. The study no doubt got 'em talking, and it's great when we can get something that lively and provocative on the front page. But the L.A. Times-Washington Post version we published on March 4 gave little basis for deciding whether or not there was any truth to this startling -- and frightening -- claim. The story was based on a study scheduled for publication in the New England Journal of Medicine, which gave it a certain amount of credibility. And it did hedge the basic claim, quoting several scientists and doctors who warned of possible sampling error and inherent improbability. What it didn't do was give the basic facts Cohn says should accompany any such story. Unlike many stories on scientific studies, it did describe the sample -- death certificates for 1,000 recently deceased folks from the San Bernardino area. It didn't say how many of them were left-handed, although the story did reveal that about 10 percent of the overall population is left-handed. If we project from the national figures to this sample, we can conclude that we gave front-page play to a startling claim based on figures for about 100 lefties in a single American city. Generalizing to the whole American population of left-handers -- something like 25 million people -- must involve a huge amount of risk. Normally, you would need a sample of about 3,000 before you could safely generalize to such a huge universe. Unfortunately, we didn't report the amount of risk involved in this particular generalization. So we left out a statistic that Cohn calls for in every story on based on sampling. We express statistical risks in two basic figures: margin of error and confidence level. The margin of error is the amount -- plus or minus -- that the sample figure is likely to vary from the actual figure for the whole population. And the confidence level is the likelihood that it varies by that much. The usual confidence level for reputable scientific research is 95 percent. That is, the scientist is 95 percent sure that his numbers are within a certain amount of the numbers for the whole population, sometimes known as the universe. Here's our suggested form for reporting those two numbers in conversational terms: "The chances are 95 out of 100 that the actual figures are within X amount of the sample figures." In this case the sample measured life expectancy. So the margin of error would have been expressed in months or years. * * * We published a similar A1 story last December 13. The story also was from the Times-Post service and also reported on the results of a study due to be published in the New England Journal of Medicine. It, too, made a startling claim: For the first time, according to the lead researcher, scientific evidence linked colon cancer to a diet heavy in animal fats. The basic claim was that women who ate beef, pork or lamb every day as a main dish had a risk of developing colon cancer 2.5 times higher than women who consumed red meat less than once a month. On the surface, the evidence seemed good. The scientists based their study on what appeared to be a huge sample -- nearly 90,000 women -- and they studied the women in the sample over a six-year period, which meant they relied less on the respondents' memories.
But our report still lacked certain critical information. Once again, it lacked margins of error and confidence levels. Even more importantly, it lacked the crucial base figure. It told us that women who ate red meat regularly ran 2.5 times the risk of women who hardly ever at red meat. But what was the basic risk? If it was tiny, then a risk 2.5 times as great was still tiny. But we nonetheless gave front-page play to the story, suggesting the risk was significant. Our story may, in fact, have persuaded large numbers of readers to change their eating habits significantly. And for what? By the way, that's a flaw in a high percentage of our medical stories. We say that a such and such a behavior involves X times the risk of another behavior. But we seldom say what the risk actually is. And that number is critical to readers who want to make up their own minds about the risks they're running. Smoking and cancer stories are notorious for just this kind of loose end. A number buried deep in this particular story suggests that it hardly deserved the credence we gave it. During the entire six years of the study, only 150 cases of colon cancer turned up among the 120,000 women in the sample group. (The story never explained the difference between that figure and the figure given for the actual sample size.) Presumably, only some of those 150 cases qualified as women who ate red meat every day. And only some qualified as women who ate red meat less than once a month. Of 150 women picked at random, how many would eat red meat less than once a month? Five? Ten? Fifteen at the most? Let's say 15. And let's say that 37 of the women -- 2.5 times as many -- ate red meat every day. So do some simple math. If you eat red meat every day, your chances of developing colon cancer over a six-year period are 37 out of 120,000 -- something like 1 in 3,500. And your chances if you eat virtually no red meat? One in 8,000. Of course, it isn't quite that simple. And we're working on the basis of multiple assumptions forced by the sketchy information the story provided. Furthermore, your risk over the vulnerable part of your adult lifetime will be higher than your risk over six years. And how much faith can we place in odds that apparently were calculated on the basis of 15 cases out of 120,000 women? But put all that aside for a moment. Accept the odds we've calculated. Do they sound like the kind of risk that would persuade you to have no more than one steak a month for the rest of your life?
|