Every week a headline announces that something you do is either saving or shortening your life. Most of these headlines describe real studies. Most of them also describe those studies inaccurately. The gap between the two is predictable enough to be taught.
Correlation is not cause
Suppose researchers find that students who eat breakfast score higher on tests. Three explanations fit that finding equally well.
Breakfast might improve performance. Or higher-performing students might come from households with settled morning routines — the third factor causes both. Or the direction might be reversed in some subtler way.
An observational study watches what people already do and can only establish that two things occur together. To claim cause, you need an experiment: participants assigned at random to conditions, so the groups differ only in the thing being tested. When a headline says causes, check whether the study was capable of showing cause at all.
Ask how many, and who
A finding from 40 volunteers is a suggestion. A finding from 40,000 people followed for a decade is closer to evidence. Small samples produce dramatic results by chance, and dramatic results are the ones that get reported.
Ask also who was studied. Research on adults aged 60 to 75 tells you little about teenagers. Research on mice tells you about mice.
Relative and absolute risk
This is where most readers are misled, and it is easily fixed.
"Doubles your risk" is a relative risk. If the original risk was 1 in 10,000, doubling it makes it 2 in 10,000 — an extra case in every ten thousand people. Reported as "doubles", it sounds alarming. Reported as "increases from 0.01% to 0.02%", it sounds like what it is.
Whenever you meet a percentage change in a risk, look for the underlying numbers. If the article does not give them, that absence is itself information.
Read past the headline
Headlines are usually written by someone other than the reporter, and their job is to earn a click. The article often contains the qualification the headline dropped — in mice, among older adults, the effect was small. The researchers'' own summary, in the study''s abstract, is usually the most cautious version of all.
Four questions
Before you repeat a finding, answer these:
- Was it an experiment or an observation?
- How many participants, and who were they?
- Is the risk given in relative or absolute terms?
- Who paid for the study, and does the funder benefit from the result?
None of this requires distrusting science. It requires distrusting summaries of science — including, sometimes, the ones produced by scientists'' own press offices.