Machine Learning for Medics 101

So AI is broadly the term we use when we try to perform a task that we'd consider to require human intelligence, but it's being performed by a computer. Machine learning is a particular way we can try to achieve that. There are other approaches to artificial intelligence that are not machine learning. In machine learning, we don't tell the algorithm what to do — we just give it data, and we expect it to interpret that data and give us an output. The alternative, the non-machine-learning route to artificial intelligence, is to give it rules to follow, and by following those rules it can perform on what we'd consider to be a human-intelligence task.

That actually raises a really interesting point, because one thing that's been criticised is that the goalposts keep moving for what counts as AI. At a certain point in time, a calculator would definitely have been considered AI. Now, because we think it's so trivial, when you talk about AI people wouldn't really be thinking about a calculator — they think about things much more advanced. But technically, in the strict definition of the word, yes, a calculator would be classed as an artificial intelligence.

Yes, that's right.

Deep learning is a really exciting subset of machine learning, and it involves a specific type of algorithm called neural networks. Without going into too much technical detail, these neural networks are loosely based on the human brain: you have large numbers of neurons and connections between them, and by giving it lots of examples it gradually learns the patterns that produce the intelligent output you want. So classically in medicine, you might give it a million chest X-rays, and it will learn to recognise the different pathologies by strengthening and weakening different connections within the network.

Yes, loosely. It was initially inspired by the human brain, although I think the techniques used in deep learning have diverged from that somewhat — the architectures used in these algorithms are quite different from what we find in the human brain. But it follows a similar principle of reinforcing the connections based on feedback.

Convolutional neural networks, or CNNs as they're sometimes called, are a specific type of neural network, and therefore a type of deep learning. They're specifically good for imaging. They're a particular way of structuring the network so it can take in the pixels from an image and process them in a way that makes it quite reliable at identifying what's in that image and what classification it should be put into.

In terms of deep learning, the other major algorithm is the recurrent neural network. It's a different type of neural network with a structure that makes it very good at analysing what we call sequential data — patterns of data points separated by time. For example, you might use a recurrent neural network to analyse text, because in text every word has a word that goes after it and a word before it. Or you might use it to analyse patterns in blood tests, because you have a temporal sequence there. So that's another avenue of deep learning.

Potentially, yes. There are other methods you can use for that kind of data that aren't recurrent neural networks, but broadly, yes, that's true.

The first thing to appreciate is just how many different components there are to the job of a doctor. In a typical shift, say in the accident and emergency department, you'll see a new patient and talk to them — a human interaction to get as much information as you can about their disease and what might have brought them into hospital. You'll then perform a physical examination, bring those findings together, look at blood results, form some potential diagnoses, and identify what investigations make sense to do next. And throughout that whole process you're talking to your colleagues, your seniors, other members of the team. So it's a very multifaceted job — and that's just an example in A&E. Different specialties operate slightly differently, and there are a lot of other components beyond the direct doctor-patient interaction.

So when you look at this question of automation, it's important to be quite specific about what exactly we're saying the AI is going to, or might, automate. You break it down into the tasks a doctor does and ask which ones AI may have some role in. I think you can definitely make the argument that there are particular elements of medical practice where AI will play a significant role and have the technical ability to perform well. But whether that leads to replacement of doctors in those domains is another question — because one thing we're seeing in the literature is that a lot of the time AI has been shown to be advantageous, it's actually not as a replacement for the doctor, but as a support to the doctor.

For example, one of the most advanced studies at the moment is using AI to detect polyps on colonoscopy. To my knowledge there have been three randomised controlled trials looking at AI as a screener during colonoscopy. The AI obviously can't perform the colonoscopy itself, but it can work alongside the doctor to flag polyps the colonoscopist might have missed. In combination, they detect more polyps, and hopefully that leads to better outcomes, because you have a higher chance of catching cancerous ones.

The other thing to say is that the state of research applying machine learning to healthcare is still quite in its infancy. When you look at the types of studies being done — and this is something we found in a systematic review I did with a number of colleagues a couple of months ago — the vast majority of published studies applying machine learning to healthcare are retrospective. And you see a dip in performance when you go from a retrospective study to a randomised controlled trial. We closed our review last June — June 2019 — and found there were 81 studies, of which only nine were prospective. That makes sense to some extent, because deep learning is a relatively new field, so there's a lag before people design these studies and run them. Naturally you'd want to run a retrospective study before a prospective one, so you have an idea of whether your algorithm is going to be effective. But you can't conclude that your algorithm will definitely work in clinical practice from a retrospective study — you need to validate it in a prospective study, ideally a randomised controlled trial.

Classically there's a dip in performance when you go from retrospective to prospective, and the reason is that it's so much harder to control for potential confounding variables retrospectively. In a retrospective study you try to think as hard as you can about confounders and eliminate them, so that anything you're measuring is directly down to the algorithm itself — but it's very difficult to eliminate every potential cause of lower performance. So you ideally want to run it prospectively, because by the very fact of running it that way you reduce the risk of confounders making your performance look higher than it actually would be on new, incoming, real-world data.

Yes. The main thing is that there's a higher chance of real-world noise being contained within that incoming information, and your algorithm needs to be able to respond to real-world noise. In a retrospective study, less of that noise might be present — just as an artefact of how the data was collected or brought together.

One of the other things we found, using a tool called PROBAST that assesses the risk of bias in a study, was that a lot of the studies were at quite high risk of bias — particularly in the analysis section. Some of the main reasons: the studies often wouldn't comment on the need to prevent overfitting, or on the methods they used to prevent it. (We haven't covered overfitting yet, so maybe we can come to that.) Ultimately there were several aspects of the study design that either weren't done or weren't commented on, which placed the papers at risk of biased findings.

Secondly, we used a checklist called TRIPOD, which looks at the reporting standards of papers — how well they cover the points that should be covered when you report a study, specifically one looking at a diagnostic or prognostic model. On the whole, these papers didn't do a fantastic job of adhering to that guideline; they didn't always cover all the bases they should have when reporting the findings from the model.

In particular — and remember we specifically looked at studies comparing an AI algorithm to clinicians — we found a lot of studies reported superior performance but didn't include the caveats, such as the fact that it was a retrospective study, or that the clinician control group was quite small. For example, the median size of the clinician control group was four doctors, which places it at risk of variation: you can have a lot of variation within a group of four, and if you're averaging that performance it's not necessarily representative of the clinician population as a whole. So it weakens the conclusions you can draw. On the whole, the papers didn't do a fantastic job of acknowledging the potential weaknesses of their methodology.

Sure. Once you've defined exactly what you're trying to do and what type of study it is — let's say we're trying to classify a diagnosis on a chest X-ray, so technically this is a classification task — you'd define the outcomes the algorithm is going to produce, and then acquire a data set containing examples of each of those outcomes. Take a real study performed by Stanford a couple of years ago: they looked at 12 different pathologies on chest X-rays and found a data set in the region of around 1.4 million chest X-rays labelled with those 12 pathologies.

A key stage is that you define the truth of the study — the technical term is the ground truth. That's the label we're considering true for the study. So if you have a chest X-ray that shows a pneumonia, and you bring together a group of clinicians who all say they think it's pneumonia, then that becomes the ground truth for that X-ray. When you show it to the AI algorithm, you train it on the belief that it's a pneumonia. There's a small room for error there — if you labelled it with two radiologists and they both got it wrong, as they might on a small percentage of cases, you'd train the algorithm on an incorrect label. But that's the idea: you establish this ground truth.

Exactly. For the purposes of the algorithm, the ground truth is what we're saying is the truth, and the algorithm can't deviate or look outside of that, because we're giving it the answer. So you have this data set labelled with the ground truth for these different pathologies, and then you take the appropriate algorithm architecture — in this case, because we're doing imaging, a convolutional neural network. You feed the data set into the algorithm, and it updates itself with every example. For each image in turn, it comes up with a prediction and gets feedback on how right or wrong it was, and it updates itself over time. So by the time it's seen a certain number of examples, it's very good at classifying an example into the right category. It sees a new chest X-ray and says it thinks it's a pneumonia; because this is labelled data, it gets feedback that it was indeed a pneumonia. Then it gets another X-ray, maybe with a pleural effusion, and gets feedback again. The idea is that by the time you present it with new data it's never seen before, it's learned to recognise the key features that distinguish one pathology from another, so it can classify images it's never seen.

Yeah, exactly — that's a good description.

Sure. One of the important things when you're training an algorithm is that you want it to perform on data it's never seen before, because ultimately the end goal is to give it real-world data it's never seen and have it make a prediction you can trust. So if you have a training set of, say, one million chest X-rays, you don't want the algorithm to see all one million, because then you wouldn't be able to reliably test its performance. If you let it see all of them and then tested it on one, you can imagine the algorithm just remembering the answer for that particular X-ray — maybe there was some nuance on it that told the algorithm it was a certain pathology, but that might not be the right characteristic.

Yes, exactly. So to get a realistic understanding of the algorithm's performance, you keep back some data as the test set. You present that test set to the algorithm without telling it the answer, and you quantify its performance — because you know the answer, but the algorithm doesn't. That gives you a more realistic understanding of how well it performs.

One reason this matters is that sometimes the algorithm picks up things that aren't what you want it to. A classic example, where there was a pitfall because they didn't separate the data appropriately: they trained a convolutional neural network to recognise how serious a pathology was, but what they didn't realise was that the algorithm was recognising that it was an erect chest X-ray rather than an AP chest X-ray — and it knew people with erect chest X-rays were more sick, so it predicted those as more severe. That's not necessarily what you'd want it to be looking at in the real world. There's another example where an algorithm was being trained to identify pneumothoraces, but it actually started looking for the presence of chest drains, which was a giveaway that a pneumothorax was present. That's not what you want, because you want it to recognise a pneumothorax without there needing to be a chest drain in place.

Some of it is stuff we've touched on — not using appropriate caveats in the analysis when you're comparing an AI against clinicians. Another is taking the appropriate measures to prevent overfitting: making sure you have a diverse data set that's representative of the population you're going to test on, that it's big enough, and that you're using appropriate methods to reduce the risk of overfitting.

Overfitting is when the algorithm performs very well on the data you've trained it with, but it's learned to recognise noise within that data set — noise that works in that data set but doesn't work on new data. So while it performs well on the training data, it won't perform well on the test data, and it won't perform well on new real-world data. It's one of the drawbacks of machine learning: these algorithms are very good at spotting nuanced patterns, which is a strength, but it creates the risk of overfitting. Say you have a thousand data points, and by chance a hundred of them — the ones with a particular diagnosis — happen to have a slightly elevated blood marker that actually isn't relevant to the disease. The algorithm might learn that this marker being ever so slightly elevated suggests that category, but it was completely by chance, and on new data it's not a useful marker. That's overfitting.

Yes, that would be an example of overfitting too.

Another important thing to be aware of is that when these algorithms are applied to healthcare problems, the performance metrics that get reported typically look at how well the algorithm discriminates between different pathologies. But that doesn't directly translate into a positive clinical impact, and it's important we don't conflate the two. There's a lot of focus in these papers on reporting metrics and trying to get the highest AUC score — one of the common metrics — but the algorithm with the highest AUC doesn't necessarily have a positive clinical impact.

That's something that has to be borne out by evidence — by prospective trials and trials in a clinical setting — so we can identify whether it actually leads to positive clinical impact.

An AUC is one of the popular metrics used to quantify the performance of machine learning algorithms. It's quite complicated and a bit tricky to explain, but fundamentally you're testing the algorithm at different thresholds and seeing how well it performs at what we call high sensitivity versus high specificity. It's hard to explain what these terms mean for people who aren't familiar with them, but there's a really great blog post by Luke Oakden-Rayner that explains the different performance metrics in a really accessible, helpful way. I'd highly recommend anyone who isn't familiar with these take some time to read it and go over it in their head until it's clear — because then when you read machine learning studies, you'll be in much better stead to understand exactly what they found.

Yes. It's important to be aware that good performance metrics don't always lead to better clinical performance. For example, there was a study in Japan looking at a chatbot to support people with mental health. The chatbot was very effective at having these conversations, prompting people about their mental health and giving advice, and the idea was that it would help treat depression and anxiety. Alternative chatbots have been developed that were shown to be quite effective. But one thing they found in this study was that in a subset of patients using the chatbot, there was an increase in alcohol consumption. The reason they thought this was the case was that by prompting people to reflect on their negative thoughts more often, it led to more impulsive behaviour — drinking alcohol, if they were predisposed to that or had past problems with it. So while the algorithm may have been very effective at doing what we designed it to do, that doesn't necessarily mean it leads to a positive clinical impact. We always need to consider how the algorithm is going to integrate into clinical workflows — how it fits into the hospital, into the way clinicians assimilate information and make decisions — and make sure it's actually going to have a positive impact. You can't purely rely on the performance metrics.

Yes. The key thing is to define the metrics you want to measure, and then test whether it has a positive clinical impact by doing a randomised controlled trial with the appropriate outcome measures. Using the polyp-detection colonoscopy example, the outcome they wanted was adenoma detection rates, and they saw that was higher when they used the algorithm. Another metric you could look at is the number of adverse outcomes from colonoscopy — because if you're using the AI, maybe the colonoscopy takes a little longer, since they're stopping to pick up these extra polyps, and feasibly that extra time might lead to a slight increase in negative outcomes. Maybe that negative impact outweighs the benefit of picking up more polyps. It's something you need to quantify across the relevant outcomes, and that lets you make a balanced judgement call on whether it's having a positive clinical impact.

Yeah, exactly.

The first thing would be to understand the performance metrics, which comes back to that blog post from Luke Oakden-Rayner — a really great starting point. If anything from that is unclear, look elsewhere and try to conceptually understand: what is an AUC score, what is an F1 score, what are sensitivity, specificity, precision, recall, and how do they all fit into each other? Another thing I'd recommend anyone Google, if they're in a position to right now, is the extended confusion matrix — there's a really nice diagram that shows where it all fits together: true negatives, false negatives, sensitivity, specificity, recall, F1 and so on. So the first thing is just having a really good conceptual understanding of what these performance metrics are actually quantifying.

The second thing would be to have a broad understanding of the different types of algorithm used in these studies. You don't necessarily need to understand the full technical details of how, say, a convolutional neural network works — but as long as you know a CNN is a specialised neural network in the area of deep learning that recognises images, that's very useful. And if you understand that there are certain types of task you can perform — classification, regression — and that within those there'll be different algorithms, that helps too. Some common classification algorithms are random forests, logistic regression, support vector machines, and you'll see them come up time and time again. If you have a superficial understanding that they're all different types of classifier, without a deep insight into how they work, that makes it much easier to read and understand these papers. Honestly, for some of them it's not even entirely clear why, say, a support vector machine performs better than a random forest in a particular scenario. So following the Pareto principle, if you just understand that they're all classification algorithms, that puts you in pretty good stead to get the main gist of what the paper is getting at.

Another point would be to make sure you understand what sort of data they're using. It's really helpful to look at wherever they've described their data set, because a huge amount of these studies comes down to what data they're feeding in — is it imaging data, or various features of a patient used to predict something like length of stay or likelihood of going to intensive care? Often — and it might be in the appendix — they'll have a list of all the features that were included: gender, age, comorbidities, blood markers. Looking through those gives you a really good idea of exactly what information the algorithm had in order to perform the task.

Yes. The first step is to understand a lot of the principles underlying these studies — things like performance metrics and the algorithms. To some extent that just comes from different sources on the internet, Googling around and finding good explanations of specific concepts that happen to make things click in your mind.

A few specific resources I personally found really helpful: Luke Oakden-Rayner's blog is great for certain concepts, like the post on performance metrics, as well as a lot of the general perspectives on AI applied to healthcare. Another person who's made a lot of helpful resources is Eric Topol — he's written a book called Deep Medicine, which is pretty accessible and gives a high-level overview while also going into a lot of detail about ways machine learning is being applied to healthcare. He's also very active on Twitter and shares a lot of useful information, very on the pulse with studies coming out at the moment, so for people who use Twitter, following him is a good place to start. He and a couple of other people have also set up something called Doctor Penguin — a newsletter where every week they send out the top machine learning and healthcare papers they've read. That's another good way to keep your finger on the pulse.

For people who want to go more technical and learn things in a more systematic way — I've found there's not one really satisfying resource I can recommend. One I really enjoyed is a course called Machine Learning on Coursera by Professor Andrew Ng from Stanford. It gives a pretty good overview of the key technical details, but I've had feedback that some people find it quite hard, and it does get quite technical at times. So one thing I've tried to do is create resources that are a bit more accessible and explain some of the technical details, but not to as much depth — more tailored to people trying to understand its applications in medicine. I've created various videos on my website, and run some courses in London, that try to provide that non-technical introduction. Another course I've seen recently, released a week or two ago, is AI for Medicine on Coursera from Stanford. I haven't checked it out personally yet, and I've had feedback from a couple of friends that it's a little more technical, maybe aimed towards people coming from more of a technical background — so for people who have experience coding or some theoretical understanding of machine learning, that could be a great course to do. It may also be accessible to people from a medical background.

And for people who really want to go deeper, I think there are four main domains you can learn to gain a really deep understanding of the area: the mathematical aspects, the machine learning theory, the practical coding and programming, and how these all apply to medicine. I wrote a blog post a while back at chrislovejoy.me/foundation which recommends different resources across those four domains for people who want to go much deeper and gain a more solid foundational understanding of how these algorithms apply to medicine.

A lot of the hype is around algorithms trying to compete with clinicians — going for a diagnostic algorithm. In my opinion it's probably never going to be the case that we allow an algorithm to fully replace a human in a diagnostic process; it may work as a second opinion or as support in some way. We hear a lot about how this algorithm can now diagnose pneumonia on a chest X-ray better than humans, or this one is better at diagnosing a retinal pathology — and while that might be interesting, they're probably not going to fully replace doctors in those tasks.

What I think is arguably more interesting, and maybe doesn't receive as much attention, is applying data science and machine learning to the more operational aspects of healthcare — things like optimising theatre usage, waiting lists or clinic scheduling, predicting who's not going to attend their appointment, increasing adherence to medication. All these areas are a lot less sexy, but there's a lot of ways we could use interesting machine learning insights to overcome these problems — and they may well have a pretty significant, scalable, positive health impact.

Probably the best place is my personal website, www.chrislovejoy.me, and from there I link to a lot of the other things I'm working on. The main two are a YouTube channel where I share videos on machine learning applied to healthcare, and a weekly email newsletter, which I write on a mix of topics but often on machine learning and healthcare. So the website is the best first place, and it links to everything else.