AI Too Smart for Clinical Trials: SPIRIT-AI and CONSORT-AI

There's a lot of fantastic research going on in this space — but there's just a lot of research going on, full stop. So the danger is that the highest-quality work gets diluted out. There's been concern about this for a while, because what tends to float to the top is the most sensational. It almost becomes a competition to see who can make the most astounding claim, because that's what gets picked up by the mainstream media.

The best-quality research in this space is not necessarily what gets the main headlines. We wanted to understand the gap between the very real hope coming through AI-type interventions in health and the hype that seemed to surround it. So a lot of our work has been trying to answer exactly that question: how good are these AI applications for healthcare? How ready are we for them? How do they fit into our pathways? What's a safe, assured, efficient way of getting the best research and the best applications through to benefit patients? That's what our whole programme of work is based on — addressing those issues in a very practical way, to help researchers, clinicians and patients.

That's absolutely right, and I'd completely agree. Alastair and I come at this from the perspective of clinicians who see patients who could benefit from this on a day-to-day basis. That's been our end goal from the very beginning, and it should be the end goal for everyone working in this space. So how we measure quality centres solely around that. You can have a brilliant AI algorithm that's 99.999% accurate, but if it doesn't actually end up affecting patient care, then to us it's irrelevant.

The other thing I'll say is that if you'd asked us this a year ago, we'd have given a slightly different answer. We published a systematic review in the Lancet Digital Health in September last year that looked at precisely this issue. We took all the deep-learning algorithms for medical imaging and looked at how accurate they were compared to humans. At the time there were lots of issues around reporting, transparency and risk of bias. The long story short is that in the very small selection of studies where we could actually compare the accuracy of the algorithms versus humans, we found it was at best equivalent.

Having said that, some of the issues we identified in that review we're seeing less and less of in the really high-profile studies over the last year. There's much more emphasis now on quality issues: are these algorithms being assessed in a realistic environment? Is the comparator appropriate? Has the ground truth been properly defined by experts? So the quality is improving a lot as we go along.

The first thing is to understand the problem. The meta-analysis Xiao mentioned, which gained a lot of traction, was really the first benchmark for where we were as a community on AI and health. As you said, of the more than 20,000 studies we identified, less than 1% were what we'd consider trustworthy — in the sense that they provided enough information, were designed to minimise bias, and gave a head-to-head comparison of the AI system versus the human healthcare professional in the same setting. So less than 1% were studies you could take to an evaluation and say, "yes, on this basis this test or intervention could be deployed."

So the first thing is to understand the issue. The next phase — which we're really excited about — is to ask, well, how can we make this better? In terms of improving the standards of design and reporting, we set up an initiative working with something called the EQUATOR Network, who are the standard-bearers for how trials of different types should be conducted and reported. They're a really important organisation for ensuring that things like randomised clinical trials are done and reported to a certain standard. Working with that fantastic team, we set up CONSORT-AI and SPIRIT-AI.

Xiao, do you want to talk through a bit more about what this meant in practice, and the last year we've had working with this team?

We announced the initiative with an article in Nature Medicine in October last year, and it's just been the most phenomenal coming-together of experts from around the world contributing to defining what good practice looks like.

The way we went about it was a Delphi study — a consensus methodology where you begin with a scoping exercise and propose a number of items you think should go into the guidelines, then run several rounds of surveys. We did this electronically, with 169 participants: a mixture of epidemiologists, researchers, clinicians, machine-learning engineers, regulators, journal editors and so on. They looked at the proposed items and said whether each should be included. Then in January this year we held a consensus meeting here at the University of Birmingham, where we invited 31 of the participants to finalise the guidelines. We put them through a round of piloting to make sure they were usable, understandable and clear, and pulled out examples of good practice.

So now we're at the stage where the guidelines are available for the first time. That's really exciting, because we have a framework people can use for reporting their clinical trials of AI interventions. Now, SPIRIT-AI and CONSORT-AI are a start to tackling this transparency issue, but they're specific to randomised controlled trials. That's only part of the story, because many algorithms will be assessed in diagnostic-accuracy trials and observational studies, which have their own transparency issues — and those need to be tackled with other upcoming guidelines, like TRIPOD-ML and STARD-AI. These are for diagnostic-accuracy studies and prediction-model studies with machine-learning considerations.

The first thing is that we asked exactly that question. Just because something's AI, does it actually need its own anything? We shouldn't assume it's always different from, say, software as a medical device that doesn't use an AI component. So the very first question we put into the Delphi process was to lay out, in a really systematic way, what was already there — and then invite people to critically ask what wasn't covered. It was very striking: there was a really rich return of challenges that people felt the current SPIRIT and CONSORT guidelines didn't recognise.

Some of this would be true for any software intervention, but some is specific to the kind of data feed an AI system needs. There are also issues around clarity. Compare it to a drug: in a randomised controlled trial of drug A, you provide the name and everybody knows what you're talking about. Here, it's just a software title with an algorithm that's completely hidden. How does anyone know exactly what it is? How do they assure it? How could someone repeat the study to check the finding is repeatable? There were numerous items people felt weren't covered, reflecting a broad set of stakeholders — hardcore machine-learning experts, people in industry, academics, clinicians. We had patients there, representing what they'd want reported. We had regulators, because this is key: how do we ensure the best products get through to patients? And we had journal editors, because one of the reasons for doing this was that editors and reviewers were saying, "we're being flooded with these papers, but we don't really know how to evaluate them."

One thing we absolutely did not intend to do was reinvent guidelines that already exist. So we pulled in as much as possible any reporting items that were generic, and we haven't changed any of the original CONSORT guidelines. CONSORT has had a number of extensions developed for different interventions and study designs, and what's customary is that you keep the original guidelines and then, where certain items need extensions or elaborations, you add on top. That's what we've done. One example is the inclusion and exclusion criteria of a randomised controlled trial: we've added an extra item to separate that into the participant level and the input-data level, because data quality and input-data characteristics can affect how well an algorithm works. We thought it would be useful to make investigators really think — are you causing selection bias, perhaps not at the participant level, but by choosing certain input data and not others?

That's a very real problem. You'll have seen studies that report a fantastic performance on diagnostic accuracy, but then, if you look, they've rejected 30% of scans because they weren't good enough quality. That doesn't mean the diagnostic is worthless — if 70% of scans can be fully automated, that's still useful. But what's the difference? Is it just reporting the easy scans that humans would have read quickly anyway, leaving 30% to be read by humans differently? How do you compare a technology that performs really well on 70% and fails on 30% versus one that performs slightly less well across 95%? These data are hidden in many papers. So, as Xiao says, that's a really important distinction — and one of the things Xiao mentioned right at the top is having a clear understanding of the data-flow pathway. Increasingly, what interests us is understanding, before deployment — say, before you put something into the NHS — how sure we are that the intervention would perform the same way in a different population. Rather than just saying "we used dataset X," it's about knowing: what was that population? How were they selected? Is it a totally routine population? How likely is it that this generalises to the population I'm considering it for? In a sense that's true of any health intervention, but there are reasons it's a specific challenge in AI systems.

The original SPIRIT guidelines are for clinical trial protocols — they cover what you should lay out in your protocol before you've done the trial. The CONSORT guidelines are for clinical trial reports — how you report the results afterwards. SPIRIT-AI and CONSORT-AI have items that cross both, because some considerations — input data for your algorithm, how you set up the IT infrastructure of your test site, selection criteria — are common to both. That's why we developed them at the same time; you use SPIRIT for the protocol and CONSORT for the report.

It matters, because if you set out what you intend to do, it can be reviewed and critically appraised even before you run the trial — which is a really good thing. Someone else can look and say, "yes, you've designed this appropriately." It also stops you changing course halfway through, which we still see happening: people go, "actually I'm not getting the result I hoped for, but this other thing looks interesting, so we'll report that instead," and then sort of hope everyone believes that's what they always meant to report. You'll be aware of the multiple-testing issue: if you test lots and lots of things, some will turn up apparently significant just by chance. So a protocol is really important both for design and to keep you honest when reporting.

The audience is also slightly different. The protocol is particularly important for regulators and ethics boards considering whether to approve a trial; the report is more for people thinking about implementing. And in CONSORT-AI we specifically look at what was meant to happen versus what actually happened — because you have the results, you can start doing analyses on things like failure cases. So one of the CONSORT-AI items is: was there any intention to do a failure-case analysis, and if not, why not?

Sure. We added 14 items for CONSORT-AI and 15 for SPIRIT-AI. Most are elaborations rather than extensions, meaning they're not new concepts — they apply existing CONSORT and SPIRIT concepts to AI-specific scenarios. To give a few examples: we ask authors to state that it's an AI intervention in the title. We ask them to specify how the AI intervention fits within a clinical pathway — what's set up around it, what leads to it being applied, and what decision-making happens afterwards.

We talk a lot about the inputs and outputs of the algorithm. On the input side: are there specifications for input-data quality? Any pre-processing steps? On the output side: how does the algorithm show its output — is it a diagnosis, a probability, a heat map? How does it get shown to the user? And we ask authors to specify the details of the human-AI interaction — is training required, and what kind? All of these help the reader understand whether the findings are generalisable: could we implement this in the real world? Do the details give us enough to replicate it? And if there were multiple trial sites, were these things done consistently?

This is a universal concept — whenever we think about any health intervention, we can't think about it in isolation. Forget AI for a second: if we're talking about doing an ECG, there's a whole pathway that leads up to a patient getting one, and then the result may lead to treatment or further tests. The population moving through that pathway depends on what's happening around it. In diagnostic testing, sensitivity and specificity are intrinsically tied to disease prevalence. So if we don't describe which patients are getting the test and what happens as a result, we don't really understand the population these accuracy metrics apply to, and it becomes very difficult to generalise about who the algorithm will and won't work well for.

So we need to understand the population, and the context in which the test is used. And — going back to my earlier point about improving patient outcomes — we need to know what happens after the algorithm too. Most current AI algorithms are diagnostic, but having a diagnosis doesn't necessarily improve your outcomes. It's intuitive that it probably does, but that depends on what you do with it. We need to know the therapeutic decisions afterwards, whether it's acceptable to the patient, whether it's acceptable to the health system — all the considerations that can make or break an algorithm.

If I could bring in two analogies that might help. One: there's a lot of debate right now about whether you provide a percentage, a probability, or an actual recommendation. It's a bit like — do you want to know there's a 43% chance of rain, or do you just want to be told to bring an umbrella? It's often not easy to answer. Sometimes both is helpful, but does that cause confusion? And it depends: is this going straight to a patient, to a health professional, or is it creating a decision automatically in a pathway, under the bonnet, unseen by a human until further downstream?

The other analogy: whatever use case something has been evaluated in, you can't always trust humans to keep it within that box. It's exactly like driver-assisted cars versus true driverless cars — people become so comfortable with cruise control and steering working that they start using their computer in the driving seat, or climbing into the back, and then crash. So we have to be really clear about the setting something's been validated in and what the appropriate use case is. This doesn't just apply to reporting trials — it's the regulatory space and real-world implementation too. One of our big areas of interest and concern, which goes beyond the guidelines, is that real-world implementation space: how do you take something that works brilliantly, with fantastic papers behind it in one setting, and put it into another? What's the ongoing real-world evaluation that reassures patients and the health community that it's still working as it should?

Luke's brilliant, and I strongly recommend your listeners follow him — there's always a good debate in his tweets. So I'm going to do that typical doctor thing and say: it depends.

There's real value in providing clear guidance, a clear recommendation, that isn't just human-dependent. The danger is that how I interpret 73% might be different from how you or someone else interprets it, even though we're in the same role making the same decision. That difference is just our personality and risk-aversion coming into play — not extra data. We've been presented with the same information, the same page in front of us; there's no logical reason to reach different decisions except our own inherent biases. So having a clear series of cut-offs — if X%, then decision Y — informed by guidelines, has real value.

There's an argument, though, that you need almost a safety-override switch, because there may be data we as professionals elicit from patients that isn't captured in the algorithm's inputs — outside the model. In those cases you'd say, "the recommendation is X based on this model, but through patient preference, or a specific factor, or something unexpected — COVID's turned up, say — I'm going to make this alternative decision." Again, it's like a pilot overriding autopilot: but there needs to be a really clear reason. It can't just be, "I don't like the thresholds, they don't allow for risk." It should be something specific. The last thing I'd say is there's an argument that the model itself should report a percentage, but the recommendation is a next step on top. So there's transparency: the model doesn't generate the recommendation, it generates a probability, and over the top sits an agreed set of recommendations defined by some process — whether a systematic review and meta-analysis, or human consensus where experts say, "we think these are reasonable."

The human-AI interaction matters on both the input and output side, because both are influencing factors that can affect model performance and downstream outcomes. On the input side: if a human is making some selection on image quality, they're essentially determining the selection criteria for the model. If the human tends to choose a certain group of patients as having poor image quality and excludes them, you're introducing a bias. That's fine if it's what needs to happen for the model to work well — but it needs to be reported and transparent, so we know that in this group of patients the algorithm isn't likely to work, and that we can't really be sure how well it works for them because they were excluded by a human.

On the output side, when an algorithm gives an output, the next step in the pathway is usually a human action. If it's a diagnosis of pneumonia, that human might decide to prescribe antibiotics. If it's a probability of pneumonia, the human has to interpret whether that means they should give antibiotics. So on either end of the algorithm there's an interaction where the human can influence not the technical performance, but the overall clinical efficacy — because each part of the pathway is so intrinsically tied together.

That's a really big question, with answers at multiple levels. The first thing is that this was our way of capturing the issue that the model evaluated in a paper might then undergo further iterations as it emerges for deployment. Ongoing updating of an algorithm with more and more data is fantastic and can improve performance — but the underlying algorithm is evolving, and you can't assume the same performance metrics in later iterations. This is carried to an extreme with an adaptive or continuously-learning algorithm. People get very excited about these, because in theory performance is perpetually improving — every new data point is informing and improving it. But there's a risk it may be improving in one domain while starting to underperform, or even catastrophically fail, in others. So we'll need evaluation systems that can cope with continuously-learning algorithms in future. I'd say we don't need to be there yet, because to our knowledge none of that type are being deployed into patient-facing settings — but that will presumably come. What we do need now is the recognition that companies will quite rightly be updating algorithms in a continuous product life-cycle, so everyone must be clear which version they're using and what evaluation process it's been through.

That brings us back to deployment, safety measures and validation — something we really want to highlight to your audience. There's concern that algorithms are currently being developed on quite narrow datasets. They may be very large but represent quite small portions of the population — data acquired in one city of one country as a particular research study, or made available by one very large hospital or network. That's fantastic, and at an academic level it's perhaps all you need. But if that's how your model learns, its performance will reflect its ability to process data from a narrow cohort — possibly predominantly one ethnic group, one country, one socio-demographic profile. We know, for example, that people who volunteer for research tend to be more affluent and more educated; ethnic minorities and people from poorer socio-demographic quintiles are underrepresented.

We've become increasingly concerned about what we call health data poverty: the idea that, quite apart from other disadvantages, there may be sectors of society — and, globally, whole countries — that don't have data representing them or people like them. It's a bit like Google Maps, where parts of the globe are just unmapped: your map app works brilliantly most places, and then there are no-go areas where it'll fail if you try to drive in. We're concerned we'll see the same with AI systems — people saying, "it works over here, so it'll be fine," taking something developed in San Francisco, dropping it into West Africa and saying, "what's the problem, these are all people with diabetic retinopathy?" But it fails — not just underperforms, it can fail catastrophically. Will we have the processes in place to watch for that?

So health data poverty is a really important and under-recognised concept, both within a country where particular sectors are underrepresented, and across countries at the global level — we've been working with the Global Eye Health Commission on this. Part of my role is directing a data hub called INSIGHT, the UK's health data research hub for eye health. It's part of Health Data Research UK — a partnership across the NHS, academia and industry. Rather than generating a massive bespoke research cohort, INSIGHT uses real-world, routinely-collected data from patients coming through the NHS. Patients can opt out — that's true across the NHS — but there's an assumption that routinely-collected, safely anonymised data will be used to improve services, for audit and for research. The advantage is that we can provide an eye-health database for exactly this kind of validation. So if you came with an algorithm from San Francisco and wanted to test it pre-deployment, you could get access to something like 10 million OCT scans or retinal fundus photographs and check it before it goes into the real world. You'd still need ongoing real-world evaluation, but you'd have real confidence it would work.

To begin with, I hope reporting will improve — that's the main goal. It'll bring to light considerations that investigators may be thinking about anyway but only reporting partially, or missing entirely, and hopefully make that reporting complete.

The other thing I hope is that it builds trust. We've done this through a huge international collaborative effort, from people with real expertise across the different dimensions of this problem. Knowing the guidelines were developed this way, and considered by people with the right expertise, I hope we can start building trust in AI. Because that lack of trust is a problem — I understand why it's there, and it's good to be cautious, but it can also stop really good research from being recognised for its real-world potential.

And the next thing for us: we recognise we're unlikely to have got everything right first time. This is a rapidly evolving field, and guidelines get updated as new applications and study designs come through. We'll do the same with SPIRIT-AI and CONSORT-AI — we'll be revisiting them within the next 18 months to two years for sure. What's brilliant is that through the consensus process we've built a network of people who now understand what good reporting looks like, whose expertise we can draw on for the next iteration. In the long term, I hope we'll start defining what good AI RCT methodology looks like. These are minimum reporting guidelines — the bare minimum that should be reported — but there's a whole load of good practice we haven't included, because we didn't want to be too prescriptive or too cumbersome, or they wouldn't be adopted. So what we want to do is pull out really good examples of methodology in these AI RCTs, showcase them, and define how you conduct a really robust trial.

All right — we'll give you one each, so two takeaways in total. Mine would be that transparency is really key for AI studies going forward. Completeness and transparency. This applies to any medical research, but particularly to AI, where we know there's a lot of hype and there have been issues of transparency and bias in the past. What we want to do is build frameworks that have gone through proper due process from the right experts, so people can put their trust in them — so that even if I don't have the technical knowledge to assess AI, don't know everything about how a model is built or about neural networks, these are the key areas that, in a clinical trial context, I need to understand. We'd hope SPIRIT-AI and CONSORT-AI are a big step towards that.

And for me: if we want AI systems and digital health applications to transform health on a global scale, and to benefit those least well served in our societies and across the world, we really need to take health data poverty seriously. Without that, these applications may lead to an exaggerated digital divide in health. As more and more care is delivered through digital or digitally-enhanced applications, if there are whole chunks of society who can't make use of them — or in whom they actually underperform or fail — that would be terrible. So we need to understand this concept of health data poverty and address it through routinely-collected datasets, like those provided by INSIGHT and other health data research hubs in the UK, and by other real-world data sources across the world.