Rolling your own ASMRs for New Zealand

I get a lot of this sort of thing from anonymous accounts on twitter.

They’re slightly tiring, and one reason I’m not really there any more.

They’re tiring because:

1. If the claim were true, it’s extraordinary and important. In particular, it would indeed be pretty clear evidence that vaccines were causing harms, and even killing people

2. The claim takes some effort to check. “Excess deaths” are a difference between actual deaths and those “expected”. They are therefore hard work (working with the data by age) and highly dependent on modelling assumptions (constructing a good, age-adjusted baseline)

Given these two points, a particular variety of attention-seeking liar spends a lot of time inventing or repeating these sorts of claims. They find them attractive because they are striking enough to be spread around, while being sufficiently hard to check that they are very unlikely to be challenged.

This particular message popped up just as I was thinking that I wanted to make sure I had fully understood age-standardisation and its effects on excess deaths estimates, and was vaguely thinking I might test this by doing some from scratch.

Fine, I’ll do New Zealand.

Age standardising, step by step.

The point of age standardisation mortality rates that the numbers of deaths are massively dependent on the age profile of a population. And this makes comparisons of raw rates particularly tricky. An aging population (such as you get in most Western countries) will tend to have higher death rates over time, even if the state of public health stays the same: over time there will be just more 90 year olds, and they tend to die more. So you want to strip that effect out, but leave any true signals in there.

The way you do this is age standardisation. It’s pretty simple: it’s just a weighted average of death rates by age, using weights from some standard age profile. Comes in three steps.

1. Count the deaths (usually in 5-year increments: 0-4s, 5-9s, 10-14s etc)

2. Put this over the population in each of those to get an age-specific death rate per 100k for each age band.

3. Select a “standard” population profile, and weight-average the rates by this standard population to get a comparable overall number.

Let’s step through this, starting from raw death and population numbers and ending up with ASMRs. I’ll give all the links to the raw data, so anyone interested should be able to reproduce every step from source.

Step 1: Deaths. The New Zealand Statistical Authority has what you need: Births and Deaths by month, from 2010-to June 2023. Monthly, but by age. Also by sex, and by ethnicity, particularly pulling out Maori and Asian. This more detaild information might indeed be helpful, especially if you were looking at the interplay between socio-economic status and mortality, but this requires knowing a lot more about the state and history of society in New Zealand than I do. We’ll use the totals.

Reminder: New Zealand is a small country:

Step 2: Rates per 100k

You can find the raw numbers on population by age, by year from the same New Zealand Statistical Authority: in fact they have a nice tool called Infoshare where you can download all kinds of metrics including population estimates for each year, by age. Put the deaths over the populations, and you have age-specific rates by age (5 year bands). Here’s bands from 65-69 up (the others get too squashed on this scale to see).

Note that in the oldest ages you get a very typical pattern of a regular winter spike each year from flu and other respiratory infections. All very familiar, except for those of us in the Northern hemisphere have to remember that we’re in New Zealand, so the winter is in June-July.

Step 3: Select a standard population profile and aggregate

Now we take each of these rates over time, and weight-average them together into one rate, given a “standardised” age profile. The choice of standard age-profile is actually not that important - you just need to choose something not totally different to the actual population (if you do - e.g., with too many older people - it will accentuate aspects that aren’t terribly relevant to New Zealand, though they might be if you were analysing Japan). I’ll choose a standard worldwide population pyramid used by the WHO since 2001 - it’s a bit older than the old Segi worldwide pyramid, but not as old as the European population pyramids.

It seems appropriate for New Zealand, but do feel free to change it to another. Using the WHO standard population profile we combine all those age-specific all-cause mortality rates into one combined age-standardised all-cause mortality rate. From 2010 to 2023, it looks like this.

Excess deaths - by eye

To work out “excess deaths” from these rates, we should now design an expected “baseline” for the years we are interested in. But there are a lot of choices to make in this design. Should we include that big spike in 2017 for example (bad flu year)? How far should we go back? Should we include additional trends such as medical advances, and if so, how should we project them forward?

All these choices can all be challenged. So I’m not going to use any of them. We’re just going to use our eyes, and overlay each previous year in grey - to give a visual indicator of where our standardised death rates have been in the past. Then we can see what changes the recent years have brought to New Zealand. Here’s to 2019 to show how similar they are (no, not going to tell you which year is which, but you can pick out 2017 for example).

Now, let’s add 2020, the first year of the global pandemic.

Wow. This started off normal, and then had massively, absurdly … lower mortality for the rest of the year.

The explanation isn’t hard to come by. New Zealand locked down successfully, had no COVID in 2020, and also had no flu wave in the winter. Many people stayed at home, and without the usual seasonal pathogens, mortality plummeted. We don’t need to worry about any specifics of constructing a baseline rate: any sensible way of calculating excess deaths would find massively negative number in 2020 for New Zealand.

---

Now 2021

Not quite as low as 2020, but lower than just about any other year on record. Again, a negative excess death number would result from just about any sensible way of constructing a baseline.

---

Now 2022

Hmm. Not so good. After a good start to the year, 2022 was a high-ish mortality year, particularly in Q2 and Q3. Depending on your precise approach to creating a baseline, you’d probably find some significant excess deaths in 2022 - quite how many would be down to the details of your approach, but there is definitely something that took deaths back to historical trend and then worse for a few months.

---

For completeness, let’s add the 6 months of 2023 we have in the NZ Stats dataset.

More standard now. A pretty typical year. I’d expect most sensible approaches to be giving around zero excess deaths now.

---

Were they telling the truth?

All we need now to judge the truth of this original claim, is to know when COVID vaccination, and also COVID itself came to New Zealand. To remind, here’s that original claim:

Here’s the dates:

COVID vaccination started on February 2021 in New Zealand, doses peaked in August, followed by boosters. Here’s the Te Whatu Ora Health Ministry schedules:

To remind, 2021 looked like this in terms of all-cause mortality. Excess deaths were negative.

OK, and how about COVID cases themselves? When did these come to New Zealand?

Again, Te Whatu Ora are the authoritative source for case counts, but let’s use the chart from OurWorldInData, which is internationally comparable.

COVID arrived in New Zealand (in a pretty spectacular fashion) a few months into 2022.

And, once again, to remind, here’s 2022 in mortality terms vs previous years, showing how some fairly clear excess death patterns start in Q2.

Oh. They’re straightforwardly lying. What a shock.

It’s the opposite of what they say. The vaccination programme was in 2021, when there were no excess deaths. In fact, they were negative. Then COVID arrived in 2022, whereupon there were some excess deaths (though they were pretty low - almost certainly because everyone was vaccinated by then).

So, with a small sigh - we discover that once again, that in response to someone who probably spent 10 seconds inventing their lie, we’ve had to spend several minutes working from the bottom-up to demonstrate that the truth is the opposite.

Still, we found out a few things. We dived into age-standardisation, working out how to do it from scratch, looked into various choices of standard population pyramids, thought a bit about the uncertainties in excess deaths, and showed a way of plotting historical data that mean you can cut out controversy about baselines. We learned something.

I bet they didn’t.