About this episode
The Economist's health policy editor, Natasha Loder, tweeted: "I'm going to go out on a limb and say that AI isn't going to get us out of the COVID-19 crisis." Well, some researchers have been trying just that. They've made predictive models which aim to either diagnose COVID-19 or to predict how the disease will progress in a patient, using medical data such as CT scans and various blood markers.
These are different to the epidemiological models which have received a lot of attention. Those aim to predict the spread of COVID-19 across a population, whereas the models we're talking about this time are diagnostic and prognostic models which aim to diagnose or work out the prognosis of an individual patient.
Professor Laure Wynants from Maastricht University in the Netherlands and her team conducted a systematic review of all available predictive COVID-19 models. Understandably, there's been an explosion in these types of models — but Laure and her team found that they were at a very high risk of bias, they were poorly reported, and they were very optimistic about their results. The paper is very interesting and you can find it in the show notes or on the BMJ website, available open access.
In this conversation
- The provocation that opens the episode: The Economist's health policy editor bet publicly that AI wouldn't get us out of COVID — and Laure Wynants's team set out to check whether the flood of prediction models actually delivered.
- The damning headline finding: every COVID-19 diagnostic and prognostic model they reviewed was at high risk of bias, poorly reported and over-optimistic — none ready for real patients, despite many already running in hospitals and apps.
- Why a wrong model is worse than no model: drawing on a retrospective study from the 2009 swine flu pandemic, Wynants explains how triage models that looked sensible underestimated mortality and misled decisions.
- A practical builder's lesson: in small pandemic datasets, complex models overfit to noise and inflate their own scores — a simple regression model you can actually inspect often beats a black box.
- The fix she argues for: open data-sharing platforms and head-to-head cross-validation between teams, so the field converges on one trustworthy model instead of everyone racing to publish their own.
Transcript AI-generated
In normal circumstances — so when there's no corona crisis — we make these models to inform decision making. Now in this crisis you also have that purpose, but there's more to it, because it can also help in the allocation of scarce resources.
For example, if from a diagnostic model based on symptoms and age a patient has a clearly very high or very low risk of having COVID, then you could decide not to do any time-consuming diagnostic testing — skip that, and reserve those resources for the patients in whom the risk is more unclear. The same for prognosis. If you have to decide who to admit to ICU — in Belgium the guidelines say that's based on prognosis, but they don't say how you get that prognosis. Well, a prediction model integrates all the clinical factors, all the tests that are relevant, and gives you one number for that prognosis.
You mention in the paper that with models like these, sometimes they can do more harm than good in a pandemic.
Obviously, if you estimate a 95% chance of dying but in reality it's only 60%, and you base your decision making as a doctor on that, you may make the wrong decision for this patient. But that statement in the paper is actually referring to more technical work — work that shows the utility of decisions made with a model can be lower than without a model, if the predictions of the model are wrong.
Yep. So it can be giving you the wrong answers, but then you perhaps have more confidence in them than you should.
Yes, exactly. There was earlier work based on the 2009 swine flu pandemic, where they used prediction models that were intended for mortality prediction in critically ill patients to triage people at the intensive care units. Afterwards — it was a retrospective study — they calculated whether using those models would have been beneficial during that pandemic, using actual patient data.
And they saw that those prediction models did not work as well as expected in populations outside of that pandemic. They underestimated the mortality, and they didn't do a very good job at discriminating between people who would and would not die. So that's an indication that even though models intuitively might seem to make sense, if you actually test them, they could be worse than you'd expect.
Wow. So perhaps we should not be as confident in these models as just a glance at the paper might suggest.
“Papers might seem very impressive based on the fancy modelling techniques they used — but unless you have good underlying data and a sensible validation strategy, I would not trust them just based on the looks of it.”
Laure
Yes. Papers might seem very impressive based on the fancy modelling techniques they used — but unless you have good underlying data, a sensible modelling strategy and a sensible validation strategy, I would not trust them just based on the looks of it.
Did any of the models you looked into — are any of them ready to be used in clinical practice, on real patients?
In terms of availability, a lot of them are. In the paper we read that some of these models were already implemented in Chinese hospitals, and we saw a lot of online applications and apps, so ready for use. In principle, any doctor in Europe could use these tools already. But in terms of risk of bias, none of the ones we've seen — and that's all that have been published — were ready for use in clinical practice.
Interesting. And were there any of the models you looked into that stood out?
They were all at high risk of bias, as it stands now. And I did make the effort — I dove back into the papers, had a very good look at them, and tried to decide for myself: if it were me, which one would I put my money on? Which model would I like to be used on me? There was one — a paper by Gong and colleagues. It had a high risk of bias, but it had two small-scale validation studies in representative patients — a non-selected set of patients — and it still performed pretty well. Now, this is very preliminary; you would have to validate it again in larger and local databases. But I think this one is maybe worth looking into more.
So one of the keys is to validate your model using patients and datasets that you haven't used when training the model.
Yes — but you can do this without necessarily needing a completely new dataset. In times like these, where data is scarce, you can have very intelligent cross-validation or bootstrap schemes which accomplish much the same thing. It's not the same as a real external validation from another centre, but it makes efficient use of the data you do have. And I think that's the next best thing.
And in terms of the data, is there international collaboration going on? Or are all these datasets siloed, with people only able to access data that's immediately available to them geographically?
We've seen a lot of small-scale initiatives, and data that's open on GitHub, for example. There's a registry by the World Health Organization. There's LEOSS, which is from Germany but is also open internationally. In the Netherlands you have CAPACITY, and also a registry by Maastricht University and Amsterdam — and they're collecting data internationally too. So in terms of data collection, yes, there is coordination. In terms of access to the data, I think you would still have to apply to get access for modelling or validating existing models.
When you're comparing these different models, you use a score called the C-index. What exactly is the C-index?
The C-index is one of the things we look at when we validate a model, and it's the one that's most frequently reported by studies. It quantifies discrimination — how good a job your model does at distinguishing between people who have the outcome you're looking for and those who don't. So, for example, who has the disease and who doesn't, or those who will die versus those who will live. A value of one is perfect, and a value of 0.5 means your model is doing no better than chance.
The technical interpretation is this: take a random pair, one with the outcome of interest and one without. The C-index is the probability that the one with the outcome had a higher estimated risk than the one without.
So in layman's terms, if you were looking at a diagnostic COVID-19 model, a score of 0.5 would mean it's equivalent to chance — not a very good model — and a score of one would mean it's perfect at distinguishing between COVID and non-COVID.
Yeah. And we've seen a lot of values very close to one — but there's optimism here as well. If you didn't validate properly, you might estimate the C-index higher than it will be in practice. And we suspect that was what was going on in these studies.
So why is that?
“One reason is model overfitting. That means you're picking up patterns in the data that are just random noise — they're related to the outcome, but only in your dataset.”
Laure
One reason is model overfitting. That means you're picking up patterns in the data that are just random noise — they're related to the outcome, but only in your dataset. It's a coincidence, if you like. And that inflates your performance, and it inflates your model coefficients. Typically, if you have an overfitted model, it will perform very well in your dataset. But if you try it again — apply it in practice, or validate it on new patients — it will perform less well.
So with overfitting, your model becomes really good at distinguishing between COVID and non-COVID just in that specific population — but not for a good reason. It's picking up on random artefacts and random pieces of noise, and it won't be useful when you take it out of that population and use it on another.
Yes, exactly.
And is this a particular problem at the moment because we've got such small datasets? Does it become less of a problem when you've got huge datasets?
Yes, indeed. There's always a balance between the complexity of your modelling and the amount of data that you have. And if you apply very complex modelling to very small datasets, that's a good recipe for an overfitted model.
So if you've got a small dataset, you're saying it's probably better to have a less complex model?
Yes, definitely.
That's interesting — it doesn't quite make intuitive sense. Why not? I guess you'd think the more complex the model, the better in general.
No, I think simplicity is a good thing. If you have a simple regression model, you can look at the coefficients and see for yourself whether they align with expectations or previous knowledge. If you build something very complex, it may become a black box and you have no way of checking.
From what you've told me, it sounds like the current models aren't that useful right now for patients, and they're at a high risk of bias as well. What do you think the solution is for this pandemic?
“The first step is collaboration between research teams that have data, so that larger datasets can be used for model development and models can be cross-validated.”
Laure
Well, I think the first step is collaboration between research teams that have data, so that larger datasets can be used for model development and models can be cross-validated — developed on one dataset and tested on the other, and vice versa. If research teams could collaborate that way, you could also get head-to-head comparisons of competing models for the same purpose, and just see which one performs best. And if you did this on an open data-sharing platform, that would really speed things up. We could get a good model — with good evidence that it works — much faster than when everybody is competing and trying to get their own model out there.
Say you've got a diagnostic model that's been made in China. If the authors don't have an interest in the model being used internationally and they're just focused on their own patients, is there much benefit for them to cross-validate and take this open, collaborative approach? Or is it better for them to focus on their own population and demographics?
That's a good point. But even if you were only interested in making a model for Chinese patients, you'd still have to validate it on other Chinese patients than those you used to build it, ideally. So I think it would be good to have more data either way.
And what do you think the challenges are in getting this collaborative approach going? Is it a legal and ethical problem with data sharing, or is it something else?
It seems the legal and ethical regulations have become more lenient now, because we're in a crisis situation. So I think it's technical — the platforms weren't there — and maybe also, unfortunately, a cultural issue, in that everybody's trying to advance their own models instead of collaborating.
I hope you liked that episode — make sure you subscribe if you did, and do leave a review. Links to everything mentioned can be found in the show notes. Thank you.