World Federation of
Science Journalists

Statistical help for reporters

May 12, 2011 posted in Sci.Journalism
Does this ring true, or what?

“While journalists generally are very good at spotting political non-sequiturs and flawed arguments, their immune systems are not geared against fraudulent numbers and statistical data. They do not even know the right questions to ask.”

These thoughts come from Hans van Maanen, a veteran Dutch science journalist based in Amsterdam. He’s the author of a new lesson on statistics to augment the online course for science journalists, and it’s available now (see lesson 9).

Gap Minder
Hans Rosling TED talk:
Hans Rosling shows the best stats you've ever seen'
He’s also the author of the recently published Goochelen met getallen, which roughly translates from Dutch into English as Juggling with Numbers.

The statistics chapter is rich in how-to-think-about statistical numbers, and how-to-write-about them. It should help anyone avoid being bamboozled or flummoxed by the freightload of statistics that accompany reports by research institutes, university science faculties and industry.

When asked, van Maanen comes up with an ample sampling of recent examples of possible bamboozle and muddled thinking.

He refers to a recent story on alcohol and cancer online in the British Medical Journal. For men, every “extra” glass of alcohol over the current recommended “moderate” drinking limit increases your risk of cancer by 10 per cent. For women, the number is 33 per cent. Van Maanen says, “True, but this does not mean a thing in absolute terms, it only shows that current alcohol recommendations lead to cancer cases, too.”

The BMJ article says, “These data support current political efforts to reduce or to abstain from alcohol consumption to reduce the incidence of cancer,” according to the research team.

This underlines van Maanen’s idea that “numbers, like pictures, have been collected and chosen with a purpose, and sometimes edited, beautified and trimmed to better suit the needs of the presenter. This may have been done with the best intentions, but sometimes the “data shopping” is more treacherous and misleading.”

He should know, because he’s a veteran teacher of science writing and what he calls “quantitative journalism” at several Dutch universities.

His examples drawn upon journalistic experience, not university mathematics classes. He repeats the oh-so-familiar advice to always check your numbers, but says reporters should make the numbers human by providing scale and context that a reader can understand. “We can easily live for a million seconds – that is less than 12 days – but a billion seconds is 32 years and a trillion seconds is 32,000 years – and 32,000 years ago, the last Neanderthal died.”

He suggests some large numbers may have been made up on the spot. The number of visitors at a political rally, the number of girls being abducted each year, the television audience at Olympic Games... good questions for reporters are: How do they know? is it even possible to know? And is it a lot or not, in this context?

And, the lesson warns to avoid assuming too much, even about something as commonplace as percentages: “Not everybody knows what ‘40 per cent’ means,” the lessons says. Really? Yes. A German study shows this turns out to be ready knowledge for just a bit more than half the population (54 per cent). What is clear for a reporter, may not be so obvious for readers.

And percentages are seldom simple. “The scientist tells us that anti-depressants lead to problems of a sexual nature (like impotence or loss of libido) in 40 per cent of cases. But does that mean that one user will have 40 per cent more problems with intercourse, or that 40 per cent of users will experience problems?”

Another tricky commonplace is averages. Van Maanen’s example is that in many developed countries, the average age of mothers having their first baby is going up. So the headline reads: “More older mothers.” But then come the questions all reporters should ask. Is the average going up because women are having their first child later in life or because there are fewer teen pregnancies? Or both?

There are rich topics in the new Chapter 9 on the online course, but we’ll finish our sampling with one about luck.

“If there is one thing that distinguishes statisticians from ordinary people, it is luck. Statisticians know that luck, randomness, plays a much bigger part in our lives and in all kinds of events than we would think possible. If someone wins the lottery twice in a row, it will make headlines – but the statistician will note that, with all the lotteries going on in the world, this is bound to happen sometime, somewhere …

“Moreover, people always tend to see patterns, even where there are none. That is the way our brains are wired – we see faces in the moon, tigers in the bush, and anything that happens three times in a row is ‘remarkable.’ Especially if we can think of a causal link: remarkably more cases of leukemia near a power plant, and the headlines read ‘Power plants cause leukemia’ …

“Science, one could say, is the endeavour to separate real occurrences from chance occurrences, real relationships from chance relationships.”

The WFSJ online course’s new statistics lesson is part of a general updating of the whole course under the direction of Frank Nuijens, the editor of the university newspaper and its science alumni magazine at Delft University of Technology in the Netherlands, where he is also a guest lecturer in science journalism.

“We’ve commissioned two new chapters – one on statistics, which Hans has written so beautifully … we are very happy to now have a chapter in the online course where science journalists can learn the basic concepts and how to avoid the most common pitfalls when dealing with statistics in a story.”

Earlier in his career Nuijens was a science journalist and television researcher for popular science programs on Dutch public television and radio. He was also on the executive committee of the World Conference of Science Journalists 2009 in London, and is on the program committee of the WCSJ 2011 in Doha June 27–29.

Van Maanen says he welcomed the invitation from Nuijens to write the chapter because, “I had just finished my book on misleading statistics and numbers in science and the media, so it did not take me too long. I had my examples ready and I knew how I wanted to order them. There’s nothing like teaching to learn how not to explain things!”

And in van Maanen’s world, there is no shortage of numbers and statistics that are at best baseless, and often just plain wrong. He says many stories on genes are inaccurate “because journalists do not understand the difficult statistics behind them. So they will write that a gene doubles the risk, while in reality it only raises the risk by, say, 12 per cent.

There is also gullible reporting because the story is so irresistible. He points to a study which concluded that women’s tears lower men’s libido.

“They measured testosterone in saliva in men after they were exposed either to women’s tears or to plain salt water. If the men were exposed to tears, on average their testosterone went down, if they were exposed to water, the average did not change. But that was because one or two men had their testosterone go up a lot, which influenced the data enormously: without these two, the average would have gone down. And nobody knows what it means when testosterone in saliva goes up, yet.”

For the newly numerate readers of the statistics lesson, there is insight into social networks soon to come – in the next newly commissioned section in the WFSJ online course.

Visit our online courses section


Commenting is close