Be careful about quantifying vaccination benefits

Vaccinated people tend to be richer and better educated: they live longer anyway

Whenever anyone posts a version of these charts of all-cause age-standardised mortality by vaccination status, two things happen.¹

1. A set of well-meaning people copy one or both into the replies of the most unhinged antivaxxers, making everyone’s mentions unusable for days.

2. At least one person asks the following - good - question. Does the area between the lines represent the number of people the vaccine saved (or more precisely, the number saved per 100k per year)?

The answer to Question 2 is “no”. But it’s “no” for interesting reasons.²

---

(Note that in the UK, vaccination proceeded by an assessment of vulnerability - so e.g., the only 18-39 year olds vaccinated before May 2021 were those in the Clinically Extremely Vulnerable (CEV) and immunosuppressed groups. These groups have a high all-cause mortality rate due to their conditions.)

---

At first sight, it looks as though - in both charts - the area between the red and blue lines should indeed give the number of people the vaccine saved - it’s the difference in overall mortality rates over time between the two groups. And, so - at a quick calculation - this gives us roughly than 500,000 lives saved over the 26 months covered (i.e., the age-standardised difference between the two groups, multiplied by the number of people who were actually vaccinated).

But this is almost certainly wrong.

For to assign causation to this, we must (amongst other things) assume the groups are overwise relevantly identical - that is, that there are no confounding factors for mortality.

Now, it’s true that the largest confounding factor - age - has been accounted for in this analysis, by age-standardising the rates. But there are others.

---

For various reasons, when you give people free access to vaccination, as England and most other countries did with COVID, there are those who get vaccinated quickly and those who delay, or do not do so at all.

When you take a balanced survey of the population in England then empirically, we find that those who have chosen not to get vaccinated for COVID tend to be:

• Less educated

• In the lower socio-economic categories (i.e., poorer)

• Unemployed

There is also an ethnic/country-of-origin effect, where white European and Asian groups tend to seek out vaccination, and Middle Eastern and Afro-Caribbean communities tend not to. All these - and several other - characteristics and how they relate to the chance of being unvaccinated or unvaccinated, are laid out clearly in ONS’ report on the associations of socio-economic characteristics with vaccination status.

And these effects are enormous: you can you see that a long-term/permanently unemployed person is more than 3x more likely to be unvaccinated than one working as a highly paid professional.

---

But this brings a big problem for us - the characteristics associated with seeking out vaccination: especially socio-economic status and education, are known to correlate positively with better health and a lower risk of death, over any timeframe. So, although we’ve adjusted for age, we still have a series of strong confounders lurking in this data.

Or, in other words, if we imagine for a moment that the vaccine did precisely nothing, then we should still expect the red (vaccinated) line to sit above the blue (unvaccinated) line, simply because they were healthier to begin with.

---

In fact, we’ve got an illustration of this in the original dataset. The ONS split the deaths into those where COVID was mentioned on the death certificate (i.e., COVID was judged a causal factor by the attending physician) and those where it was not. And you find that in both those counts, death rates were higher in the unvaccinated

This is the ratio of non-COVID death rates between the groups: i.e., how much less dying there is in the vaccinated group, and while the orange line shows the sky-high difference for COVID-related deaths (+10x when you get big COVID waves, and before everyone gets natural immunity) the green line still shows a difference even when COVID is taken out of the equation - it may be much smaller, but it’s not zero!

It’s obviously a pretty significant effect at the start - i.e., the vaccinated group are very, very much less likely to die, and then quickly declines, until the two are fairly comparable towards the end (i.e., the green line goes nearly to zero: it’s just +7%). My interpretation would be that this is catch-up. The better educated, richer, healthier people got the vaccines earlier, but outreach efforts eventually made it to harder-and-harder to reach groups. This effect has been long-known for any kind of vaccination programme - it is called the “healthy vaccinee effect” and simply reflects that people in better health are often more able to access effective healthcare.

By the way, the green line probably does overstate the issue somewhat. Not every COVID case was diagnosed, and it is very possible that COVID sequelae have led to deaths, where it was not obvious enough that COVID was a causal factor to appear on the death certificate. But it is pretty implausible that this accounts for the entire effect.

---

So, what if we do want to find how effective the vaccines were overall, and how many lives they saved/didn’t? Well, the obvious answer is to look at a properly conducted, randomised study, where all these effects were removed by proper randomisation of the control (no vaccine) and effect (vaccine) groups. But these studies were done in 2020, when the original Wuhan strain was the main one in circulation, so they will not tell us how vaccines worked in the much more complicated variant soup of 2021-3.

For the more complex picture, you can look at UKHSA’s real-world vaccine effectiveness studies, and this is probably the best answer on UK data. They do their best to remove all the confounding effects, though they will be the first to state where they still see weaknesses in their methodology.

---

But let’s say we want to get the best approximation given the data we have in front of us - this mortality data from the ONS.

I would suggest that a reasonable approach might be to assume for a moment that the non-COVID death rate difference between the two groups is giving us a reasonable proxy for this “healthy vacinee effect”. And so if we subtract that effect from the COVID deaths difference we see in the two groups, we should get a reasonable estimate as to the “true” difference driven by the vaccine rather than the underlying health differences.

That is, by May 2023, non-COVID deaths were +7% higher in the unvaccinated group than the unvaccinated one. If we assume that this effect will also appear as part of the “deaths due to COVID” numbers, simply because the vaccinated group was healthier to start with, we can subtract it from that difference, and see what remains.

Do this month by month, sum up this remainder, and you get a much smaller answer: around 150,000 lives saved in England, over the 26 months of data we have.

My feeling is that the 500,000 age-standardised lives mentioned earlier is a clear overestimate (basically, assuming “high” at all points where one might have a difference) and can be thought of as an upper limit, while this 150,000 number is the opposite (it assumes “low” at all points) and so might be thought of as a lower limit.³

My own guess is that the true answer - if any such thing even makes sense - lies a little closer to the lower limit than the upper, but somewhere around this point we should do better to acknowledge the weaknesses of this data, and look for more evidence. Fortunately there is plenty elsewhere.

1. Unless they just copy and paste them without credit, of course.

2. There are other points that are much less interesting.
Someone will always claim there must have been a delay between vaccination and when people are placed into different groups: this is uninteresting because it’s a straightforward lie, they’re counted from the moment of vaccination.
Someone else will always claim that because the ONS are only looking at those where the medical records can be linked, the rates may be very different in the ones they can’t link. This is true (e.g., the unlinked will almost certainly include homeless and others “off the grid”, which may have very low vaccination rates, and extremely poor outcomes). However, since the ONS have linked 91.6% of the Census 2021 population it’s somewhat pointless, since there’s very little the remaining 8.4% would be able to swing.

3. However, note that all of these are dwarfed by the wider counterfactual that without any vaccines, the rates of infection would have been very much higher, especially in early 2021 when they were at maximum effectiveness. This would have saved many more lives. But then the status of NPIs would have had to be very different: we could not have opened up in early 2021, nor could we have progressed to full re-opening so fast, and then Omicron would have hit us much harder. These fiendishly complex counterfactuals are not at all dealt with here - we have just assumed that the transmission of the virus proceeded as it did in the actual world.