About this episode
As Britain enters its second week of lockdown, I became curious about how the government made the decision to lock the country down. One research team that's been extremely influential in the response is the Imperial College team led by Professor Neil Ferguson. Their model predicted that an unchecked COVID-19 epidemic would overwhelm the NHS and result in half a million deaths in the UK, and suggested we may need to be in some form of social distancing for 12 out of the next 18 months.
More recently, an Oxford University team led by Professor Sunetra Gupta published their model online. Any model of COVID-19 has to make assumptions. Imperial looked at the deaths we've had so far and assumed that COVID-19 hadn't infected much of the UK, but had quite a high death rate. Oxford assumed the opposite — they built a model in which COVID-19 had already infected most of the population, but had a relatively low death rate. This was picked up in outlets like the Financial Times, with the headline "coronavirus may have already infected half of the UK."
To find out how exactly these types of models work, what their limitations are, and how we should interpret them, I called up Dr James Hay, a computational epidemiologist at Harvard School of Public Health. For the record, this conversation was recorded on the 31st of March 2020. I hope you enjoy.
In this conversation
- The core contrast, explained cleanly: Imperial's 13,000-line agent-based model that simulates every individual in the country, versus Oxford's stripped-down two-equation ODE model — and why each level of complexity fits a different question.
- Why a high-resolution model's greatest strength is also its weakness: every output is conditional on fixed, sometimes arbitrary-looking assumptions, like "75% adherence from people over 70."
- The real story behind the Financial Times "half the UK may already be infected" headline — a Dropbox-hosted thought experiment about model uncertainty, mangled on the way to the public.
- A candid take on scientific rigour under pressure: even landmark COVID models weren't open-source, and code routinely goes unchecked — "I wouldn't have expected my supervisor to check my code line for line."
- The one distinction Hay wants every reader to hold onto: a "case" is defined differently across countries and changed six or seven times in China, but an infection is an infection.
Transcript AI-generated
In the UK there have been a few models that take what we know about COVID-19 and try to predict the future of the pandemic — the Imperial College model, which has been quite influential on UK policy, and the recently published Oxford model. Broadly speaking, how do these models work?
They take two very different modelling styles. The Imperial model is based on a framework Neil Ferguson has been working on for most of his career. It uses what we call an agent-based model, where every individual in the population is their own object — you look at the properties of each person individually. It's meant to be as realistic as possible, so it takes into account things like population density, where people make contact — schools, workplaces — and how they move between cities, using travel data. It's meant to be a really close representation of reality.
That's quite a contrast to the Oxford model, which is about the simplest population-level model you can come up with — a deterministic ODE model. That means you're only modelling population-level quantities; you're not interested in individuals. It's just a couple of equations.
Professor Neil Ferguson has said the Imperial model is built from 13,000 lines of code, and from what you've said it does sound like quite a complex model.
The amount of complexity you need comes down to the question you're trying to answer. Imperial are trying to answer some quite nuanced questions about specific interventions, so the model has to take into account things like age structure and the location of contacts — because we're talking about closing schools, or targeting people over 70. That level of complexity really matters, especially if you want a high-resolution picture of what's going on in a country: is it going to spread to this city at this time, or to rural areas at that time? Then you need those places modelled explicitly.
If you're more interested in what I'd think of as thought experiments, that level of detail doesn't matter so much. That's why the Oxford model wasn't worried about high-resolution detail — they were just asking, at the broadest level, if we run this thought experiment, what sort of results do we see?
So in the Imperial model, are you saying there's a separate data point for every person in the population, and they're all being modelled?
Sort of. If you think about it in terms of the code — I don't know how much your listeners are into programming — every individual is their own little bit of computer memory. In software terms this is object-oriented programming: every time you want an individual in the model, you generate a new bit of data for them. In contrast, the Oxford model only records the total numbers in the population.
And that's because the Oxford model is trying to be a what-if, hypothetical statement, whereas the Imperial model is actually influencing policy?
That's right. The way I've understood it, Imperial has been called on to answer very particular questions. I think the Oxford group felt — quite rightly — that it's been a difficult time for the rest of the scientific community and the public to keep a handle on what scientific advice is going into government. When that's happening quickly and behind closed doors, as it has to, you want some level of scrutiny on the assumptions informing policy. So the Oxford group wanted to put their own perspective out there: if we took a more critical approach to the assumptions Imperial made, how broad could the possible scenarios be?
Speaking specifically about the Imperial model — what are the limitations of a model like that?
“There's a lot of expertise going into fixing those assumptions — but every result that comes out is conditional on those assumptions.”
James
The biggest limitation comes down to the fact that it's such a high-resolution model — everything in it has to be underpinned by fixed assumptions about parameter values. If you read the report, you'll see a lot of numbers that seem fairly arbitrary. For example, we might assume 75% adherence from people over the age of 70. It can be quite unsatisfactory to place so much faith in these fixed numbers, and in such a complex model it's hard to iterate over every possible parameter. You have to run scenario-based analyses — assume 25%, assume 50%, assume 75% — but with so many parameters, you can't run enough scenarios to cover all your bases. So you really are trusting a lot of the assumptions going in.
That said, the Imperial team is huge. There are dozens of reports and preprints coming out, with teams looking at each of these assumptions individually, so there's a lot of expertise going into fixing them. But every result that comes out is conditional on those assumptions.
There hasn't been widespread community testing in the UK in the way there has been in countries like South Korea, so presumably we don't have a good grasp of a lot of these assumptions — what percentage of the population has been infected, how many are asymptomatic, what percentage are being hospitalised. What impact is that having on the validity of the overall model?
A lot of the parameters being fixed you can get a good idea of without UK-specific data, because they're independent of COVID-19 — things like how frequently and where people make contact, how often they move, the population demographics. That's fixed regardless of this outbreak. And the parameters related to the natural history of the disease — we're getting a pretty good idea from other countries that are a bit further ahead than us, particularly China, South Korea and Italy.
So it's not really a question of how applicable these assumptions are to the UK; it's more a question of how reliable the data from other countries are. Everywhere has the same limitation — nowhere has yet done really large-scale testing of the general population, other than places like Iceland. It won't be until we get serological testing that we'll have a good idea of the true asymptomatic fraction and the true prevalence in the population.
And once we do get this testing, is that the silver bullet? Does it mean we can input that into these models and get a really good idea of what the pandemic is going to look like?
It's going to help a lot. I don't know that it's necessarily a silver bullet. All of these assumptions hinge on how quickly the virus is spreading and what proportion of infected people get sick, to various degrees. As soon as you fix that denominator — the total number infected — it calibrates the model so much better.
One of the predictions in the original Imperial paper was that the pandemic could last 12 to 18 months, and we'd need to be in social isolation on and off for around two thirds of that time. What's your opinion of that — do you think it's going to happen?
“As an epidemiologist you're not often put in a position to think about all these different types of cost. Your mission is normally to minimise mortality, and to disregard the costs.”
James
This is an interesting question, because as an epidemiologist you're not often put in a position to think about all these different types of cost. Your mission is normally to find a model or an intervention that minimises mortality or morbidity, and to disregard the costs. That's really the crux of a lot of the controversy around Imperial's model — they say in the paper that they don't consider the costs; their aim is to minimise deaths.
So from an epidemiological perspective, what do I think about switching interventions on and off for 18 months? It seems like a very logical strategy for minimising deaths. Whether it's feasible as a long-term policy isn't really something I could comment on.
Going back to the Oxford team's model — it was reported in the Financial Times with the headline "coronavirus may have infected half of the UK." Is that a fair representation of what they were trying to say?
For somebody with the expertise to read the paper and understand the analysis, absolutely not. We're living in an unprecedented time for a researcher, where you don't know who's going to read what you write or how it'll be taken. There were a few turns of phrase in the paper where you can understand why someone without the expertise would misinterpret it. It's such a problematic time for science communication, because there's so much pressure to get results out quickly. The paper that was commented on was on Dropbox — it was a pre-preprint. Six months ago even preprints were met with scepticism; the gold standard was publication. So scientists are having to adapt on the fly to how they communicate their results.
How would you say we should interpret the Oxford paper and the modelling they did?
The point they were trying to make is that we have to be aware our results hinge on very particular assumptions. If we massively relax the assumption about what proportion of the population is at risk of severe disease, we can generate a huge range of scenarios. As a thought experiment, that was very valuable. A lot of the gripes people have had are that we know we're not in that unconstrained space — we do have a reasonable idea of what proportion of the population is at risk of severe disease, more than the Oxford model was showing. So as a thought experiment it's very useful; as a model whose results should inform policy, probably less directly useful.
And when you wrote about it, you framed it like this: look at the data we have on deaths in the UK from COVID-19. Those deaths could be due to very high infectivity but a low death rate — meaning most of the population has already been infected — or to very low infectivity but a high death rate. They were asking how each of those assumptions would look.
That's exactly right, and it's a very valuable and valid thought experiment to run. In terms of how we should interpret the results, we have good external data suggesting we're in a place where a low proportion of the population has been affected and a high proportion is at risk of severe disease. I'm pretty confident we're closer to that reality — but as an exercise in model uncertainty, it was valuable.
You've talked about how, in the current climate, we need preprints and Dropbox uploads of papers. But the code for these models — the Imperial model in particular, but also Oxford — isn't open source; it's not available for other people like yourself to peer review. Do you see that as an issue?
“As academics there's so much pressure to publish and to generate results, and if you're a computational scientist, the way you do that is by writing code. But when you write your code, maybe only one or two other people ever look at it.”
James
It definitely is an issue, and it's not a new one for this setting. As academics there's so much pressure to publish and to generate results, and if you're a computational scientist, the way you do that is by writing code. But when you write your code, maybe only one or two other people ever look at it. During my PhD I wouldn't really have expected my supervisor to check my code line for line. There's a lot of trust, because we don't really have time to be checking — and more senior people especially don't have time to run and check code. There's no incentive or reward structure to do it. There probably are solutions, but it's not something we're going to solve during the course of this outbreak.
The last thing I wanted to ask about is what common mistakes you're seeing in the reporting of COVID-19. Is there anything, from your perspective as a computational epidemiologist, that's being reported incorrectly?
One thing I've seen done wrong a few times is distinguishing between the definition of a case and the definition of an infection. When you're comparing case counts across countries, it's really hard to have comparable numbers unless you know what definition each country is using for a case. This was particularly difficult early on in China, when the case definition changed six or seven times. From one day to the next the number of "cases" would jump massively — just because the definition changed from needing a PCR confirmation to requiring evidence of severe respiratory damage on a CT scan. So it's really important to know what the number represents. The definition of a case changes, but the definition of an infection doesn't. That's one really useful thing to distinguish.
I hope you liked that episode — make sure you subscribe if you did. If you have any thoughts or feedback, the best way to get in touch is on Twitter, @mustafasultan. Links to everything mentioned are in the show notes. Thank you.