Even before generative AI took off in the mainstream, AI has been playing a significant role in healthcare. Medicine is a domain where our frail, human systems fall terribly short. Disease is an indefatigable foe that inflicts much suffering. The more progress we make against sickness and injury, the harder the healthcare system problem gets: the care gets more expensive, overstretched clinicians are asked to stretch further still, patients and families are required to navigate an even-more-bewildering landscape.

AI has had a bit of success in alleviating some of these bottlenecks, such as narrowing candidates for drug development, predicting which patients who are at risk of things like sepsis or ICU step-up, and identifying cancer from radiology scans. But so far, these solutions haven’t done much to alleviate an over-burdened system. The lengthy human process of clinical trials remains the limiting factor on bringing new treatments to market. Early warning systems lead to better outcomes for patients, but do little to lighten the load of overworked healthcare providers. The number of radiologists has actually increased since the advent of AI.

Generative AI in Healthcare: What could go right

There is potential that generative AI may finally change the calculus, and make healthcare work better as a system. It’s early days, but anecdotes from clinicians allow us to imagine a world where GenAI achieves improvement across multiple dimensions:

• better patient outcomes
• lessen physician burden
• ease patient access and navigation
• reduce cost

AI-enabled charting is one possible driver. Especially since the advent of electronic health records (EHRs), charting and notekeeping have become an enormous time-sink and a significant contributor to burnout. That calculus is changing as AI scribes become more prevalent. Dana Suskind describes how AI scribing has totally changed the atmosphere of her clinic. Exam rooms that used to be silent as physicians typed at a computer screen during patient visits have once again become a nexus of human interaction: clinicians and patients can have in-depth conversations while an AI takes care of the EHR input.

AI likewise has the potential to make an enormous difference in the realm of diagnosis. A lot of what doctors do is pattern-matching: given the symptoms a patient is presenting with, what is the most likely diagnosis and the treatment options most likely to succeed?

Diagnosis has become increasingly difficult with the pace of medical innovation. Over 1-million new articles are published each year in life sciences and medicine, with an overarching trend towards personalized treatments. No human can keep up. Generative AI is a first-rate pattern-matcher, and can really help here by surfacing considerations that might have been overlooked. There are many AI-enabled products focused on this task (here is a good recent summary), as well as individual physicians using Open Evidence to support their practice. ​

You can imagine a world where AI represents a real inflection point, allowing healthcare providers to focus on the human-side of care while AIs carry the administrative burden and support diagnosis and treatment, while making it more efficient and less error-prone.

Sadly, you can also imagine a world where none of this comes to pass. EHRs are something of a cautionary tale. In theory, they were going to reduce costs, improve patient care, and generate efficiencies for clinicians. In practice, EHRs have created no notable cost savings or patient benefits, while contributing significantly to physician burnout. EHRs have largely proven themselves to be a tool of bureaucracy, requiring clinicians to go through box-checking exercises at the cost of in-person clinical care.

All to say that a positive outcome for AI in healthcare is far from a foregone conclusion. The introduction of new technology often has unexpected results as people respond to incentives in surprising ways.

Rats, Cobras, and Incentives

In Hanoi around the turn of the century, the French colonial government wanted to reduce the rat population, so they offered a bounty on rat tails. Soon, officials noticed the sewers were filled with tail-less rats, as rat-catchers severed the tails but left the rats to run free and reproduce. The British Raj allegedly stumbled into a similar trap by offering a bounty on venomous cobras, which eventually led people to breed cobras as a source of income and increase the cobra population. It has come to be known as the cobra effect* – given a solution to a problem, competing or unanticipated incentives within an ecosystem inadvertently make the problem worse.

In many ways, the implementation of EHRs is an example of the cobra law in action. Digitizing health records was supposed to make the system more efficient and reduce admin. But because of competing incentives within the healthcare system, EHRs actually make the problem worse. We should anticipate similar cobra effects as AI gets embedded into healthcare.

Introducing Goodhart’s Law

Goodhart’s Law is one way of formalizing cobra effects. In its original formulation, the law states:

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

That is, we can observe that a declining cobra population and dead cobras generally go together (i.e. the statistical regularity). But if we try to get a greater decline in population by producing or encouraging more dead cobras (i.e. the pressure being applied for control purposes), the relationship between the two collapses.

“Applying pressure for control purposes” sounds nefarious, but it’s actually standard practice in a healthcare context. If we know from data or experience that a particular set of symptoms generally resolves with a particular intervention, that is a statistical regularity. When a new patient presents with those symptoms and we apply the intervention, we are “applying pressure” hoping that the symptoms will subside.

In order to rely on a statistical regularity without falling prey to Goodhart’s Law, we need to know that the relationship is definitely causal (the symptoms abated because of treatment, rather than the two things simply co-occurring). We also need to know that no additional, unintended effects occurred to make the situation worse (treatment creating side effects worse than the original symptoms). Of course, this is exactly why we have randomized controlled trials in medicine. Because we have studied a new treatment in a controlled environment, we can be confident that the statistical regularity will hold and the treatment will be safe and effective.

Generative AI in healthcare is relying on (very sophisticated) statistical regularities to produce its output. When we use that output to inform clinical care, we are, inevitably, applying pressure on those regularities for control purposes. Of course, generative AI in healthcare is trained on published clinical trial results. But what makes AI such a powerful tool is its ability to identify many additional statistical regularities that have not been tested in a clinical trial, but that we nonetheless use to inform clinical care. Goodhart’s Law has entered the chat.

Goodhart’s Law applied to generative AI in healthcare

There are four different variants of Goodhart’s Law, which help us break down the risks and challenges of over-reliance on statistical regularities that AI has found†.

1. Stakeholders with competing goals
2. Misattribution of causation
3. Application in new domains
4. Omitting relevant variables

1. Stakeholders with Competing Goals

The challenge of stakeholders with competing goals is closest to the classic cobra effect. In healthcare, this is a constant challenge because there are so many stakeholders with somewhat different goals. At the highest level, clinicians, administrators, payors, caregivers, and patients all want high-quality, cost-efficient patient care. But when we define that goal in more detail, or make it specific to an individual with a particular health condition, we find that people make different trade-offs. Even for the same person, the trade-offs often shift over time. Thus, for any AI in healthcare, we need to be aware for whom its goals were defined, and how they might differ from the goals of other relevant stakeholders.

2. Misattribution of Causation

In medicine, misattribution of causation is surprisingly common, which is why bloodletting was central to medical practice for 2,000 years. More recently, radical mastectomies were a common treatment for breast cancer, until it was definitively shown they inflict serious side effects on patients without any clinical benefit. Many of today’s cutting edge cancer treatments work so well for some patients that they no longer have any evidence of disease. For other patients, the drugs don’t seem to have any effect on the cancer. Medical science does not know why some tumours respond and others do not, which is another way of saying we don’t yet understand the underlying human biology enough to identify the causal chain.

AI certainly has the potential to help us refine our causal understanding, but it also has the potential to rapidly propagate practices based on incorrect or incomplete causal understanding. History tells us that once a practice becomes the standard of care, it’s difficult to change. We should therefore be attuned to the risk of generative AI suggesting interventions based on statistical regularities rather than an established causal link.

3. Application in new domains

Domain shifting is particularly salient in healthcare, because health outcomes can be very different in different populations and healthcare practices vary across institutions and jurisdictions. And yet, applying knowledge to a new domain is in many ways fundamental to the practice of medicine. For example:

• Treating patients from populations that have been under-studied in clinical trials (racial minorities, pregnant people, the very young and the very old).
• Shifting a procedure from an academic hospital to the community, or from an inpatient procedure to an outpatient.
• Treating rare disease patients based on experience and data with more common conditions.
• Applying learnings from a treatment in a later line of therapy to first-line treatment, or vice versa.

Human clinicians are generally quite aware when they are shifting the domain, and proceed accordingly. When we are working with an AI model, domain shifts can be less obvious. A 2022 MIT study at Beth Israel Deaconess Hospital looked at an AI tool for predicting risk of sepsis in the ICU. Initially, prediction accuracy was quite good, but it degraded significantly over time. The culprit was two-fold:

1. A new version of the International Classification of Diseases (ICD) changed clinician input to the system, and
2. The model started to be used at affiliated suburban hospitals with different patient populations.

This example is from an older technology that wasn’t using generative AI, and we might expect more modern approaches to adapt better to changes in the data. But it does suggest we should be clear on the training domain for AI in healthcare, and be alert to the possible implications of domain shifts in order to interpret AI outputs appropriately.

4. Omitting relevant variables.

A very simplistic example of this challenge is using low blood pressure as a proxy for good health, and ignoring other clinical indicators. This is the basic issue with older alert systems that often gave clinicians useless warnings based on a very small number of data points. In this simplest form, it’s not a very big challenge for modern AI systems, which are great at weighing enormous amounts of input and unlikely to over-index on a single variable.

However, AI cannot weigh information it doesn’t have. For instance, Dr. Robert Wachter outlines the risks of patients interacting directly with an AI for diagnosis: Patients are not experts at communicating the most important details of their case, which can lead the AI to provide bad advice based on missing information. Even when a clinician is using AI to support a case, they won’t necessarily have all the relevant clinical data. In healthcare, data collection is not free. Some diagnostics are expensive; others might require long waits. All of them require the patient to spend more time undergoing procedures they might rather avoid.

Further, healthcare decision-making is not just a matter of clinical data points — human values play a role too. Values encompass weighty questions around end-of-life, but they also include how people experience the hassle and traffic of a treatment they can only get at a big hospital downtown. Because the universe of potentially relevant variables is so vast, it’s very likely that AI will be missing some of them, and any AI outputs should be interpreted through that lens.

The point is not that Goodhart’s Law exists, so AI in healthcare will never work. Nor is it that we can’t do anything about our metaphorical rats and cobras. Generative AI holds so much potential for the healthcare system, and ever-more-sophisticated generative AI models might be able to self-correct for Goodhart’s Law. But especially as AI becomes increasingly agentic and humans are in the loop less often, it behooves us to understand the different manifestations of Goodhart’s Law, and how they might cause problems in a healthcare context.