‘Most of the trouble in life comes from misunderstanding’ as Willy Loman said in Oscar Wilde’s masterpiece An Inspector Calls. Grating isn’t it? But also a reasonable facsimile of how science reporting often reads, at least to those of us self identifying as scientists. Pre-Covid it was so. With Covid the problem has become, well, an epidemic. Recent reporting about the efficacy of masks is a classic case in point.
You might remember the study in question: it certainly made headlines when it came out. ‘Mask-wearing cuts Covid incidence by 53 per cent, says global study’, stated the Guardian: ‘results from more than 30 studies from around the world were analysed in detail’. The New Scientist agreed: ‘Covid-19 news: Mask wearing cuts infections by 53 per cent’, but instead referenced a greater number of evidence bases: ‘using data from 72 studies’. A more cautious Times reported the same results as ‘Masks more effective than distancing and handwashing’. It goes for ‘more than 30 studies from around the world’ which ‘were analysed in detail’. Bloomberg claimed ‘Masks Cut Covid Risk in Half’, now based on a far-lower number: eight studies.
Let’s leave aside the nitpicking details (infections and risk are not the same thing, for example) — surely these reports are usefully communicating the gist of the new evidence, are they not? I’m afraid not.
The BMJ paper at the centre of these reports is a meta-analysis. That is to say: a summary, an attempt to combine the evidence contained in several scientific studies to arrive at an objective assessment of the state of knowledge on a topic.
The analysis identified 72 studies that might potentially have provided evidence on the effectiveness of masks, social distancing and hand washing. Of those, just six (not eight, 30 or 72) were sufficiently relevant — and of sufficient quality — that they could provide any useful information on mask efficacy.