Digital Twins in Clinical Trials

So, I like to say that I'm a reluctant entrepreneur. I never set out to start a company or anything like that — I was a scientist. I did my bachelor's degree in biophysics at the University of Michigan and a PhD in biophysics at Harvard. Then I went the postdoc route, assuming I was going to go on to become a professor or some sort of researcher. I was a postdoc in physics at Boston University, and then a Philippe Meyer fellow in theoretical physics at the École Normale Supérieure in Paris.

This whole time I'm just being an academic scientist, working on applications of machine learning to problems across different areas of biology — everything from protein folding to genomics, to even looking at ecology and how ecosystems function. But at the end of my second postdoc, I was curious about industry. When you're in academia, industry is sort of this shadowy land where you don't really know what goes on. So I thought I'd take a job in industry to gain the experience and see what it was like. A good friend of mine, a collaborator, was working at Pfizer as a computational scientist. So I went to Pfizer and worked as a machine learning scientist.

Then I made the jump from Pfizer to the one thing I've done that was not in biology — moving out to San Francisco, Silicon Valley, and working at a startup on machine learning for virtual reality. Which is a real left turn from the rest of my career. And then eventually I started Unlearn, about four years ago now, leaving that job to start it. So that's a rough overview of the journey.

Yes and no. I think it was a general sense of frustration with the ability to have an impact in academia. Academia is a very interesting, special environment where it's incredibly competitive — and you're constantly arguing. You're arguing at the staff meeting, arguing at the seminar, arguing with the reviewers. It's this constant arguing for things. And the pace of change is frequently very, very slow, particularly when you think about how it translates to things that regular people actually have to interact with. So I wanted to see whether moving to industry would let me effect that change in a more efficient way.

The lack of academic positions is certainly something people talk about a lot. I applied for a few faculty positions in my life, but I never seriously went through the whole process, so I can't speak to that part directly. But there's a famous saying — I think it's John Maynard Keynes, but I don't know for sure — that academic politics is so petty because the stakes are so low. And I really do believe that. People are constantly arguing about stuff that ultimately isn't going to impact the world. That's just the culture.

On the second part of your question — the idea that you can make a bigger pie and everyone can succeed — I think that's partially true, but it's a bit harsh to put it that way. You can always start another company, and of course there's competition between companies, so it's not like competition goes away. But an enormous amount of what happens in industry is actually quite collaborative.

For example, I meet many, many other biotech startups when I'm at events or meeting with investors, and I absolutely want every single one of those startups to succeed. Because for me, as a company that helps biotech and pharma bring their drugs to market faster, every time someone has a successful drug, that's a new potential customer. And from their perspective, they want us to succeed because we can help them get their drugs to market faster. So if I look at the companies in the pharma, biotech and health tech space, 99.9% of them are collaborators and 0.1% are potential competitors. It really flips the perspective around: in business, most people are actually potential collaborators, not competitors.

Oh yes. There are two answers to that. One is the incredible experiences from academia that translated over perfectly to what I needed in industry — in particular, public speaking. I'd had to speak at group meetings since I was an undergrad, since I was 18 years old, giving a presentation once a month until I was 30. Twelve years of monthly presentations to a hostile audience — not necessarily hostile, but that argumentative style, where no one just lets you get away with anything. Having drilled myself for years on how to give presentations and describe things translated to industry perfectly.

What didn't translate was coding. In academia, you write a program that's meant to work for your paper — it supports the analysis in that paper and nothing else. Academic software engineering standards are about as far down as you can push standards, whereas industry software engineering is really, really high. So there's a huge gap, and I had to completely relearn coding and software engineering. When I think about people who want to get into data science, computational biology or bioinformatics, the number one thing I'd recommend is to really learn good software engineering practices — not just coding and scripting, but actual real software engineering.

It's helpful to clarify what physics-style thinking is, because it's actually really juxtaposed to traditional biological thinking. Physicists look for commonalities between things that appear to be very different. We try to get a model that describes 90% of both things at the same time — maybe 10% isn't perfect, but 90% of both things that seem very different will be described by the same model. So you capture the majority of the important behavior. Biologists tend to do the opposite: they focus on the things that are different, separating things out to search for the differences.

To give you an example, one of the papers I wrote as a postdoc took mathematics used in physics to describe magnetic materials and applied it to studying ecosystems and how species interact. You actually get really interesting predictions out of it, and it's gone on to be a fairly well-cited paper in ecology — two totally different things described by the same mathematics.

So when we look at disease areas, and I want to build machine learning methods to understand the progression of different diseases, we're asking: how can we bring in all these commonalities between diseases and use them to get better predictions about this particular disease? Instead of trying to find smaller and smaller groups of patients to learn from, we say, let's make the group of patients we're learning from as big as possible, and try to learn things that are relatively universal.

The same is true in clinical trials. Say we're running an Alzheimer's trial with 200 patients — but there are millions of patients with Alzheimer's all over the earth, and a lot of data available from them. Why would we not use that? Why would you not say you could learn more by also incorporating those data? So we're trying to find the commonalities between things that traditionally would be thought of as quite different.

Sure — these are actually kind of different things, so I'll start with the digital twin. When we ask whether some treatment has an effect, you first have to ask what it even means to say a treatment has an effect. We think about this in terms of a framework people call potential outcomes. The basic idea is: I take a patient who has a given disease, and at that point in time there are two possible futures. If I give them the treatment, whatever happens under that scenario happens. Or I don't give them the treatment, and whatever happens under that alternative scenario happens. The treatment effect is the difference between those two outcomes — what happens if they got the drug, minus what happens if they didn't.

There's this thing people call the fundamental problem of causal inference: you can only ever observe one of those outcomes. I can't simultaneously give a person a drug and not give them the drug and see both. I only ever observe one of them.

So the idea of a digital twin is: let's take machine learning and AI-based methods and predict the other one. I can take a patient, give them this new experimental treatment, and observe what happens. Then I use a mathematical model to predict what would have happened if I hadn't given them the treatment. And the reason I can build that second prediction is that I usually have lots and lots of data from people with that disease who are receiving the current standard of care — data from tens of thousands of people like that. So the basic idea of a digital twin is that I simulate one outcome and I observe the other. And then you can incorporate these predicted outcomes into clinical trials using statistical methods that actually guarantee certain statistical properties of the trial. We can cover that more later, but that's the overall concept.

In terms of the founding story of Unlearn — the company was founded by three theoretical physicists: myself; Jon Walsh, who was a high-energy theorist working on particle physics; and Aaron Smith, who's technically a mathematician, working on mathematical physics. We were all doing machine learning at the time, but with this physics background. And it started as a pure machine learning company. The way we looked at it: 99.9% of machine learning research is controlled by 10 or 12 companies on earth — Google, Facebook, Baidu, Amazon, Microsoft. They put all the resources into the research, they pay for it all, and that means the research is directed at the problems they care about. Very little has actually been invested in real machine learning research for medicine — very little, if you look at it dollar for dollar.

So our view was: no one's really ever worked on machine learning for medicine. Maybe if we focused on it, because it's a new area, we'd discover interesting machine learning. That was the original idea — we'll focus on this unsolved area and learn about machine learning in the process. And then the journey from that starting point to where we are today, with digital twins in clinical trials, was really a lot of trial and error — in the R&D, inventing the methods needed to create these digital twins, and with customers too: working with pharma companies, talking to the FDA, getting feedback and making our products better.

A digital twin isn't exactly a replacement for subjects in the control arm. People often think, oh, you removed 20% of the control-arm subjects and replaced them with digital twins. But that's not how it works. The reason is that they're twins: it's literally an individual person for whom you're making a prediction, whereas a randomized controlled trial's control group is a group-wise comparison. So it ends up being a bit different.

All of the methods we're proposing today involve incorporating digital twins into randomized controlled trials. You're still randomizing patients to both a treatment and a control group. You create digital twins of everybody — predicted outcomes — and then incorporate those into the analysis. And what we can show is that, depending on how you do the statistical analysis, you can run trials that are vastly smaller. If you're doing what people call a frequentist analysis — p-values and all the things people typically see — you might be able to run a trial with 25% fewer subjects. If you're willing to try another very sound approach, a Bayesian one — where you're talking about the probability the treatment is effective, as opposed to the probability of the data if the treatment is not effective — then you could decrease the size of your trial by two-thirds. So you could run a trial with a third of the patients you'd typically need. But it's still a randomized controlled trial.

That's important. In most cases we're not proposing to eliminate the control group. The reason is that any statistical method you use to bring in data from other sources can prove you can make the conclusions of the trial insensitive to any variable you've measured.

So if there's some patient characteristic that could be different, that could relate to the outcome and confound your results, and you've measured it, then you can de-confound things. But what do you do if there's a patient characteristic you haven't measured? We don't understand that much about lots of diseases — we don't know the biology enough to say we've measured every important variable, that's totally ridiculous. The only method we know of that lets you say this inference will be robust even to unmeasured things we don't know about yet is randomization. Randomization is the only tool we have to give those truly robust answers.

They are statistically equivalent, yes. When we think about how we interpret trials, there are two kinds of mistakes we don't want to make. We could just be right — the drug doesn't work and we say it doesn't, or the drug works and we say it does. But there are two ways to be wrong. The first is that we're testing a drug that doesn't do anything, but we declare the trial positive — people call that a type 1 error. And we can guarantee the type 1 error rate ahead of running the trial. There's no possible way it could be otherwise, unless we had a bug in our code — if we follow the steps we say we're going to follow, we can guarantee it.

The other way is that the drug does work but you fail to detect the difference — a type 2 error, which is related to statistical power. Higher power, lower type 2 error rate. And the power you can get depends on how predictive the digital twin is of a patient's outcome: the more predictive it is, the more power you get. Power is a trade-off for sample size. So you have a choice. You could use that power to detect smaller and smaller effect sizes — but at some point it's not a clinically meaningful effect anymore, so you don't need to. The other thing you can do is say: this is the effect size I want to detect, so let me find the fewest patients I need to detect it. Those are the types of designs we focus on. We can guarantee type 1 error rate control in trials that use digital twins, and we can generally guarantee high power — though the guarantee on power is a little bit less mathematically rigorous than the guarantee on type 1 error control.

This is ultimately an empirical question. Our goal isn't necessarily to match every individual baseline characteristic of you — it's to match all the important characteristics that let us make accurate predictions about what will happen to you in the future. If something isn't useful — I like to use astrology as the example — do I need to know your astrology sign to put into the digital twin? Probably not. So we only really need the things that are important for predicting your outcomes, and that varies from one therapeutic area to another. The models are constructed for individual diseases to be predictive of patients with that disease. For Alzheimer's you might have brain imaging, MRI, blood tests in some cases. In oncology it might be very different — transcriptomic or genomic samples from a biopsy. So it's the features that are useful for predicting outcomes in that particular disease area.

The biggest obstacle is education — explaining our technology, the approach, its pros and cons, in a way people can relate to, so they understand the value and understand what we mean when we say certain things are guaranteed. We guarantee a kind of safety for the use of these approaches. If you look at regulators, there's current regulatory guidance from the FDA and the EMA that would both lead you to believe what we're proposing is acceptable now. It's generally acceptable in pretty much all clinical trials with a continuous outcome. We're in the process right now of going through a submission with the EMA to get their opinion on that. Our view is that their current guidance already makes it acceptable, and we can go through the process to confirm they agree.

So from the regulatory perspective, I actually think there's no regulatory risk. What we're proposing is a fundamentally better way to run a clinical trial. We can take all these additional data, put that knowledge into a model, incorporate the predictions into a trial, guarantee the type 1 error rate, increase power and efficiency, and do this generally across any therapeutic area — and prove the mathematical properties of the trial will be preserved. It's a fundamentally better way of running clinical trials. It should just be used in every single one. We have proofs that it's not only better — we prove it's optimal, the best possible thing you can do. So the biggest thing for us is a hearts-and-minds campaign. We publish academic papers, give presentations at conferences, talk to regulators — all with the goal of getting people to realize, you know what, they're right, this is the best way to run a clinical trial. That's our main focus right now: getting that shift in people's perceptions.

It depends very much which investors you're talking to. There are roughly three groups in our vicinity: tech investors, life sciences investors, and a handful of people in the Venn diagram in the center who invest in both. When we talk to tech people, it's often pretty challenging just to explain what the problem is. Many times I've had to start by saying, this is what a clinical trial is — there's a phase one where you test safety, a phase two where you do this. If I have to walk you through what a clinical trial is, it's probably going to be hard for you to understand why this is an important problem to solve, because you don't know the market.

On the other hand, the real life science investors don't really understand the business model. Even if they believe in the tech — and many times they don't understand the technology, but put that aside — they tend to focus on therapeutics, where their style is: this is a therapeutic, it's an asset I can value, and if it's successful it will be worth so many billion dollars. That's just how they work. Whereas we're a software company. Our goal isn't to have an asset worth billions, but to have recurring revenues through the use of our software. So we have a SaaS-like business, but we're working in this life sciences market. That's why we tend to focus on the group of investors who do both — invest in tech businesses and in life sciences businesses. With that particular group, it's been very easy to explain what we're doing and get buy-in on the technology.

I think it's a combination. In terms of the value proposition, the opportunity to decrease trial sizes by 25, 50, maybe even 75% if you have very high predictive power — that's an enormous gain for a pharmaceutical company. If you decrease your trial by 25%, you can cut a year off your time to market. In something like Alzheimer's, where trials are very long, a year off is a really big deal. If you have a drug that's going to sell $10 billion a year, then one extra year is an extra $10 billion. The value is very clear.

It's more about getting them to understand that the approach is statistically sound, that regulators will accept it, and that people can interpret the studies in the end. When you read those two trials next to each other, you can interpret a study that uses digital twins without any special statistical knowledge. You see the power, you see the p-value, and you interpret it like any other clinical trial. You don't need any special insight into AI. It's a perfectly interpretable study. Once everyone's on the same page with that — and given our approach is the optimal, best possible approach — it's something that really should be widely used. We're confident we'll get people on board.

My personal biggest critic, or the biggest critic of Unlearn? Because those might be different answers.

So let me tell you the thing that, for me at least, I want to see us evolve. Right now we build models that are kind of incredible in their predictive power. They don't just predict the mean — I don't just predict what the mean of something is going to be. I predict its variance, its correlation with every other thing, and I predict that over time. So we really predict an immense amount of information about each individual patient. But the statistical methods we use right now for incorporating digital twins into trials only use a small amount of that information. So it's like our predictive models are a Ferrari, and we're driving it at 25 miles an hour.

That's how I'd describe it. We do that at first because we can — at 25 miles an hour we can guarantee this is going to be safe. You're going slow enough that everything's safe, it's going to be fine. That's where we're starting. But as we get more people comfortable with the approach, there are ways to start ratcheting up the speed. What I'd really like to be doing over time is getting to the more cutting-edge statistical approaches where we use more of the information coming out of the digital twins than we currently do — because you could decrease trial sizes a ton in that situation. But to do so, you have to have more belief in the quality of the digital twin models.

I mentioned giving lots of talks — that's probably one of the most important. The only other thing I've always done, that's a bit unusual, is that I pretty much spend the first 30 minutes to an hour every single morning reading preprints. Preprints are new in biology and medicine, but they've been around for 50 years in physics. So going back — even as an undergraduate, for the last 12 years — the first hour of every day, I'm on the arXiv, and now bioRxiv, reading through the latest postings and what people are working on. And I try to do it in a way that's really diverse. I don't focus just on my area of research. I'll even read economics papers — academic papers in economics — and then, being a physicist, try to bring in that commonality. I might learn a new econometric statistical technique that turns out to be useful in my own research in a totally different area. So I read this wide scope of papers and then try to find ways to use those things myself.

Oh, man. Books on clinical trials and things? One thing I'd say — a higher-level book, maybe not on clinical trials — is The Book of Why by Judea Pearl. It's a book just on thinking about causality. And causality is really what we're trying to get at when we look at clinical trials: did this treatment cause this difference? It's a popular-level book, and it's focused on by a Nobel Prize winner — well, not actually the Nobel Prize, it's the equivalent for computer scientists — on how to think about causality. It's really relevant to understanding when a medical treatment is actually having an effect, and how to think about those kinds of experiments.