The BBC reported that AstraZeneca and Pfizer are credited with together saving more than 12 million lives in the first year of Covid vaccination. To substantiate this claim, the BBC refers to Airfinity, a ‘disease forecasting company’.
Models do not fit anywhere in the pathway for establishing effectiveness
Airfinity used an Imperial College London study, which calculated that Covid vaccines saved 20 million lives between December 2020 and December 2021. Using a mathematical model, the Imperial team assumed that vaccination conferred protection against Covid infection (mRNA vaccines were estimated to have given 88 per cent protection against infection after the second dose) and the development of severe disease requiring hospital admission. The team also assumed that those who develop infection are less infectious than unvaccinated individuals.
We downloaded the supplementary data from GitHub (where academics often store their computer code) to examine the country-specific estimates and assess the plausibility of the numbers for the UK, Italy, and the USA.
In the UK, with vaccine coverage of 68 per cent, 507,200 (382,200 to 789,900) deaths were estimated to be averted because of vaccination (52 deaths prevented per 10,000 vaccinated). In Italy, the modellers predicted that, with a coverage of 73 per cent, 491,300 (444,800 to 544,400) deaths were averted (53 per 10,000 vaccinated). In the USA, with 61 per cent coverage, 1,902,000 (1,737,000 to 2,069,000) deaths were averted (44 per 10,000 vaccinated).
The Imperial team fitted a model to all-cause excess mortality, using more modelling from the Economist. They also assumed that each country would follow the same time-varying reproductive number (Rt) trend since the start of the pandemic. Despite an absence of detailed vaccination data for most countries and an assumption that the relationship between age and the infection fatality ration was the same for each country, the BBC still used the numbers.
In 2021, there were 667,479 deaths in the UK, 22,150 fewer than the 689,629 deaths in 2020. By our reckoning, the modellers want us to believe that in the absence of vaccination, there would have been 1,174,679 deaths in the UK in 2021.
Similarly, in Italy, 701,346 deaths occurred in 2021; the model predicts the number would have been 1,192,646 and 452,329 more deaths than in 2020. In the USA, a total of 3,464,231 resident deaths were registered in 2021, 80,502 more than in 2020. The modellers assumed there would have been 5,366,231 deaths in the US in 2021 if vaccination had not been implemented.
Sadly, many journalists don’t check their numbers or facts: many of the assumptions in the model are incorrect, and the estimated number of deaths averted by vaccines is implausible.
This isn’t surprising. As in medicine, models do not fit anywhere in the pathway for establishing effectiveness. Regulators don’t use them for approval, and decision-makers like the National Institute for Health and Care Excellence use economic models with reliable estimates of effect and credible costs. They do not appear in the The Centre for Evidence-Based Medicine levels of evidence or the Scottish Intercollegiate Guidelines Network system for grading evidence, as they are irrelevant to answering therapeutic questions. Clinical trials are the primary study type to determine the effectiveness of medicines or vaccinations; anything else is just a bad guess.
This post was written by two old geezers who will explain why observational data-based estimates of effect should be taken with a grain of salt. Models simply should not be used. Large, well-designed, well-reported, and data-accessible trials should be used for global public health interventions. So, why are we using models to justify decisions?
This article first appeared on the Trust the Evidence Substack
Comments