Why The FDA Sucks

 Scott Alexander gets it mostly right at substack.com

It's all about incentives.

----------------------------------------------

Lots of people have been writing about aducanumab, but this Atlantic article in particular bothers me.

Backing up: aducanumab, aka Aduhelm, is a new “Alzheimers drug” recently approved by the FDA. I use the scare quotes because it’s pretty unclear whether it actually treats Alzheimers. It definitely treats beta-amyloid plaques, and beta-amyloid plaques are kind of nasty-looking brain structures that seem to be related to Alzheimers somehow. But we’re not sure exactly how they’re related, they might not be related in a way where removing them treats Alzheimers, and the best studies don’t find that the drug helps patients feel better or remember things more. Aducanumab doesn’t meet normal FDA standards for approval, but the FDA approved it anyway under one of their many “fast track” programs for promising drugs. This has been pretty roundly criticized, because although aducanumab might or might not work, it definitely costs $50,000/year/patient. Even if it worked great, that would be a hard pill to swallow (no pun intended, Aduheim is an IV infusion), but it’s especially galling since it might not work at all. Doctors will probably prescribe it despite its questionable value, and someone will end up paying the extraordinary price tag.

(Who? Nobody knows. The patient? Insurance companies? Taxpayers? Unrelated patients at the same hospital? Could be anyone! The whole point of the US health insurance system is to make sure nobody ever figures out who bears any particular cost, so that there's no constituency for keeping prices low. If you check your bank account one day and find it's down $50,000 for no reason, I guess you were the guy who ended up on the hook for this one. Sorry!)

Given that the FDA fast-track approved a sketchy drug which probably doesn’t work, it’s fair to wonder if their standards have gotten too lax - or at least if they should stop fast-tracking things. The Atlantic article dutifully makes this case via a somewhat labored global warming metaphor. The FDA’s eroding standards are like "the eroding coastlines and thawing icebergs associated with climate change". There are proposals to make drug approvals harder again, "just as there are proposals for encouraging reductions in carbon emissions", but "as with cap and trade policies for carbon emissions, aggressive approaches have failed in the face of powerful stakeholders". Some doctors are trying to fight back, but "just as switching to an electric car or turning your lights off won't cool a warming planet, a minority of idealistic doctors won't stem the flood of ineffective treatments". The message is clear (if a little heavy-handed): just as good thoughtful people want to end climate change and only greedy polluters oppose this, so good thoughtful people want to make the FDA stricter, and only greedy pharma companies could possibly complain.

While I acknowledge that aducanumab probably sucks, I think the Atlantic article and its global warming metaphor are totally off base. Nobody in the “FDA is too strict” camp has written a rebuttal yet, so I want to try my hand at this.

The FDA Is Still Much Too Strict

The Atlantic article says that “The FDA’s standards began to slide in the late 1980s and early ’90s” with the fast-track approval of AIDS drugs:

A new program in 1992 allowed for “accelerated approval” on the basis of surrogate markers, which are indirect measures of a drug’s benefit, assessed via laboratory or imaging tests, that stand in for more meaningful outcomes such as life expectancy. But the implementation of these accelerated processes was criticized by some scientists and patients, even at the time. In 1994, for example, The New York Times cited skeptics who worried that “no one can tell if the drugs work.” Eight months later, the AIDS activist organization ACT UP San Francisco called Anthony Fauci a “pill-pushing pimp” for supporting CD4 immune-cell counts and viral loads as surrogate markers. They were completely invalid, the activists wrote, and nothing more than “a marketing exec’s wet dream.”

The article acknowledges that the AIDS drugs actually worked out great - they in fact cured AIDS effectively and saved lots of lives. “But”, it concludes, "that level of success is not at all the norm."

I agree. AIDS drugs are abnormally successful, saving tens of thousands of lives per year. If the FDA's expedited review process moved them forward by even a few years, it probably averted a hundred thousand AIDS deaths. True, not every drug they accelerate does this. But some do.

Here’s another good example: coronavirus vaccines. The FDA still has not fully approved any coronavirus vaccine. The only reason you’re allowed to get vaccinated at all is because of a fast-track provisional approval somewhat like the one used for aducanumab. Coronavirus vaccines have probably also averted a few hundred thousand deaths.

So without wanting to say this level of success is “the norm” in the sense that every single fast-tracked drug achieves it, it’s not exactly vanishingly rare. It’s just something that happens sometimes and doesn’t happen sometimes. So how often do you have to save hundreds of thousands of lives before it’s worth the risk of occasionally also permitting a dud medication that “offers false hope”? How is this even a question?

That’s a kind of hand-wavey verbal argument. But doctors, epidemiologists, and economists have tried to formally confirm it with cost-benefit analyses on the last few decades of FDA-approved drugs. How many lives would have been saved if good drugs had been released a few years earlier, versus how many lives would have been lost by missing dangerous side effects? I think the current state of the art is something like Isakov, Lo, and Monterhozedjat , which finds that there are a tiny few disease categories where the FDA might be slightly too aggressive, but that overall the FDA is still much too conservative.

And these kinds of analyses, while good, can only count the drugs we know about. The real cost is the thousands of life-saving medications that are stillborn because nobody wants to go through the literally-one-billion-dollars-per-drug FDA approval process.

Ranting About The FDA For A Bunch Of Paragraphs

The Atlantic article doesn’t really include a cost-benefit analysis. But it does mention a couple of examples of times when lax FDA decisions seemed bad, for example when they “approved saline breast implants despite safety concerns”. I feel like this should give me the right to describe a couple of my least favorite FDA decisions, so we can see whether they’re more or less convincing than the breast implant thing.

The countries that got through COVID the best (eg South Korea and Taiwan) controlled it through test-and-trace. This allowed them to scrape by with minimal lockdown and almost no deaths. But it only worked because they started testing and tracing really quickly - almost the moment they learned that the coronavirus existed. Could the US have done equally well?

I think yes. A bunch of laboratories, universities, and health care groups came up with COVID tests before the virus was even in the US, and were 100% ready to deploy them. But when the US declared that the coronavirus was a “public health emergency”, the FDA announced that the emergency was so grave that they were banning all coronavirus testing, so that nobody could take advantage of the emergency to peddle shoddy tests. Perhaps you might feel like this is exactly the opposite of what you should do during an emergency? This is a sure sign that you will never work for the FDA.

The FDA supposedly had some plan in place to get non-shoddy coronavirus tests. For a while, this plan was “send your samples to the CDC in Atlanta, we’ll allow it if and only if they do it directly in their headquarters”. But the CDC headquarters wasn’t set up for large-scale testing, and the turnaround time to send samples to Atlanta meant that people had days to go around spreading the virus before results got back. After this proved inadequate, the FDA allowed various other things. They told labs that they would offer emergency approval for their kits - but placed such onerous requirements on getting the approval that almost no labs could achieve it (for example, you needed to prove you’d tested it against many different coronavirus samples, but it was so early in the pandemic that most people didn’t have access to that many). Then they approved a CDC kit which that the CDC could send to places other than their headquarters, but this kit contained a defective component and returned “positive” every time. The defective component was easy to replace, but if you used your own copy like a cowboy then the test wouldn’t be FDA-approved anymore and you could lose your license for administering it.

A group called the Association of Public Health Laboratories literally begged the FDA to be allowed to deploy the COVID tests they had sitting on the shelf ready for use. The head of the APHL went to the head of the FDA and begged him, in what they described as “an extraordinary and rare request”, to be allowed to test for the coronavirus. The FDA head just wrote back saying that “false diagnostic test results can lead to significant adverse public health consequences”.

So everyone sat on their defective FDA-approved coronavirus tests, and their excellent high-quality non-FDA approved coronavirus tests that they were banned from using, and didn’t test anyone for coronavirus. Meanwhile, American citizens who had recently visited Wuhan or other COVID hotspots started falling sick and asking their doctors or health departments whether they had COVID. Since the FDA had essentially banned testing, those departments told their citizens that they couldn’t help and they should just use their best judgment. Most of those people went out and interacted and spread the virus, and incidence started growing exponentially. By March 1, China was testing millions of people a week, South Korea had tested 65,000 people, and the USA had done a grand total of 459 coronavirus tests. The pandemic in these three countries went pretty much how you would expect based on those numbers.

There were so, so many chances to avert this. NYT did a great article on Dr. Helen Chu, a doctor in Seattle who was running a study on flu prevalence back in February 2020, when nobody thought the coronavirus was in the US. She realized that she could test her flu samples for coronavirus, did it, and sure enough discovered that COVID had reached the US. The FDA sprung into action, awarded her a medal for her initiative, and - haha, no, they shut her down because they hadn’t approved her lab for coronavirus testing. She was trying to hand them a test-and-trace program all ready to go on a silver platter, they shut her down, and we had no idea whether/how/where the coronavirus was spreading on the US West Coast for several more weeks.

Although the FDA did kill thousands of people by unnecessarily delaying COVID tests, at least it also killed thousands of people by unnecessarily delaying COVID vaccines. I’ll let you click on links for the details (1, 2, 3, 4, etc, etc, etc) except to remind you that they still have not officially granted full approval to a single COVID vaccine, and the only reason we can get these at all is through provisional approvals that they wouldn’t have granted without so much political pressure.

I worry that people are going to come away from this with some conclusion like “wow, the FDA seemed really unprepared to handle COVID.” No. It’s not that specific. Every single thing the FDA does is like this. Every single hour of every single day the FDA does things exactly this stupid and destructive, and the only reason you never hear about the others is because they’re about some disease with a name like Schmoe’s Syndrome and a few hundred cases nationwide instead of something big and media-worthy like coronavirus. I am a doctor and sometimes I have to deal with the Schmoe’s Syndromes of the world and every f@$king time there is some story about the FDA doing something exactly this awful and counterproductive. A while back I learned about cholestasis in infant Short Bowel Syndrome, a rare condition with only a few hundred cases nationwide. Babies cannot digest food effectively, but you can save their lives by using an IV line to direct nutrients directly into their veins. But you need to use the right nutrient fluid. The FDA approved one version of the nutrient fluid, but it caused some problems, especially liver damage. Drawing on European research, some scientists suggested that a version with fish oil would cause less liver damage - but the fish oil version wasn’t FDA-approved. A bunch of babies kept getting liver damage, and everyone knew how to stop it, but if anyone did the FDA would take away their licenses and shut them down. Around 2010, Boston Children’s Hospital found some loophole that let them add fish oil to their nutrient fluid on site, and infants with short bowel syndrome at that one hospital stopped getting liver damage, and the FDA grudgingly agreed to permit it but banned them from distributing their formulation or letting it cross state lines - so for a while if you wanted your baby to get decent treatment for this condition you had to have them spend their infancy in one specific hospital in Massachusetts. Around 2015 the FDA said that if your doctor applied for a special exemption, they would let you import the fish-oil nutritional fluid from Europe, but you were only able to apply after your baby was getting liver damage, and the FDA might just say no. Finally in 2018 the FDA got around to approving the corrected nutritional fluid and now babies with short bowel syndrome do fine, after twenty years of easily preventable state-mandated damage and death. And it’s not just this and coronavirus, I CANNOT STRESS ENOUGH HOW TYPICAL THIS IS OF EVERYTHING THE FDA DOES ALL THE TIME.

[edit: people have asked me for more details about the fish oil story - I’ve written it up at more length here]

…anyway, The Atlantic says the FDA needs to be stricter and wait longer to approve things, and I am against this.

But How Can The FDA Be Too Strict And Not Strict Enough At The Same Time?

Very easily! Lots of things are too strict and not strict enough at the same time! I wrote a whole article on this! It sounds like it should be paradoxical, but it isn’t!

Consider the police. I once had a psychotic patient threaten to kill a family member. I reported it to the police. They asked me where they could find the patient, I said I dunno, maybe at his house or something? I called them back a few hours later asking how things were going, and they said they had knocked on the patient’s door and he hadn’t answered, so they felt like they had discharged their duty in this matter and were going to close the case. I asked if maybe they could go back to the patient’s house and try again later, and they acted like I was asking them to hunt down Osama bin Laden in the caves of Tora Bora or something.

I think this is a pretty typical experience a lot of people have dealing with the police, especially in the Bay (unofficial motto: “San Francisco - Where Crime Is Legal”). A friend had a really scary stalker, and kept reporting him to the police, and the police’s answer, phrased only slightly uncharitably, was “Have you, as of now, already been murdered by this person? No? Then stop wasting our time.” My friend was left with the feeling that the police could have been a little stricter or more proactive.

On the other hand, you get stories where police think someone might be growing marijuana or whatever, they gather a SWAT team complete with surplus tanks from Iraq, they break down the person’s door, and they shoot everyone involved because “it looked like they might be reaching for a gun”. If anyone survives, the police stick them in prison for ten years for “resisting arrest” or something. Maybe these people are left with the feeling that police could stand to be a little less strict and less proactive.

So which is it? Are the police too strict, or not strict enough? I don’t think there’s a good answer to this question. I would rather say the police are bad at their job. Maybe not literally, because being a policeman is hard, and I don’t want to judge them until I’ve walked a mile in their jackboots. But something has gone wrong, something more fundamental than just they lean too hard in one direction or another, something that requires a solution more complicated than moving a Police Intensity Lever from LESS to MORE.

My own profession is little better, as I’ve discussed before. Many people get diagnosed with psychiatric diseases and pumped full of medication when they shouldn’t be. Other people don’t get diagnosed with psychiatric diseases or treated with medication even when they desperately need it. Moving the Psychiatry Lever from MORE to LESS or vice versa might accomplish something, but it’s clearly not the whole story.

The FDA has a very hard job, and handles it with a level of badness that makes police officers look like one of those omnicompetent fictional intelligence agencies by comparison. I mean, if anyone ever gives you control of the FDA Lever, you should definitely absolutely for the love of God push it as far toward LESS as it is possible for it to go - I think this is what all those cost-benefit analyses the epidemiologists and economists publish are telling you, and it’s also what my common sense and ethics tell me. But I have to admit that this isn’t costless. It’s going to let a lot of crappy drugs like aducanumab get through and give people false hope.

(a problem which, I can’t stress enough, is not as bad as causing hundreds of thousands of people to die of easily preventable causes. Please move the lever all the way to LESS. Even if it’s already there, see if maybe you can push it a few micrometers further.)

Is there some better solution?

Sympathy For The Devil

I want to stress that, despite my feelings about the FDA, I don’t think individual FDA bureaucrats, or even necessarily the FDA director, consistently make stupid mistakes. I think that given their mandate - approve drugs that definitely work, reject ones that are unsafe/ineffective, expect people to freak out and demand your head if any unsafe/ineffective drug gets through, nobody will care no matter how many lifesaving treatments you delay or stifle outright - they’re doing the best they can. There are a few cases, like aducanumab, where it seems like they move a little faster than that mandate would suggest, and a few other cases, like infant nutrient fluid, where they move a little slower. But basically they are fulfilling their mandate to the best of the ability of the very smart people who work there.

And it’s hard to even blame the people who set the FDA’s mandate. They’re also doing the best they can given what kind of country / what kind of people we are. If some politician ever stopped fighting the Global War On Terror, then eventually some Saudi with a fertilizer bomb would slip through and kill ~5 people. And then everyone would tar and feather the politician who dared relax our vigilance, and we would all restart the Global War On Terror twice as hard, and drone strike twice as many weddings. This is true even if the War on Terror itself has an arbitrary cost in people killed / money spent / freedoms lost. The FDA mandate is set the same way - we’re open to paying limitless costs, as long as it lets us avoid a very specific kind of scandal which the media will turn into 24-7 humiliation of whoever let it happen. If I were a politician operating under these constraints, I’m not sure I could do any better.

So the long-term solution is to become a different kind of country and different sorts of people - eg raise the sanity waterline. This will have nice side benefits like also ending the global war on terror. But until then, are there any small changes that would help around the edges?

Unbundle FDA Approval

The most plausible small change I can think of is to unbundle FDA approval.

Consider: everyone knows the evidence for aducanumab is poor. You know it. I know it. Scientists know it. Journalists know it. So why exactly are we expecting lots of people to spend $50,000/year on this drug?

The answer is: there are complicated laws around what insurance companies have to cover, and FDA approval is a big part of them. I don’t understand the exact legalities of this, but it seems like Medicare and Medicaid have to cover anything the FDA approves. The situation with private insurances is more complicated but still not great. My guess is that if a private insurance covers an Alzheimers patient, and a doctor says that aducanumab is “medically necessary” for that patient, and the insurance doesn’t cover it, and the patient’s Alzheimers gets worse, that patient can sue the insurance company for failing to provide standard of care. What makes it standard of care? Because the FDA approved it, and Medicaid and Medicare are giving it to their patients, of course!

(Why would doctors say this useless drug is “medically necessary”? Well, some large fraction of doctors are stupid and believe whatever drug companies tell them. Some other fraction - including me - are pushovers when a really sad-looking patient begs them for the one thing they believe will help. Once an insurance company agrees to cover a drug, neither the patient nor the doctor has any incentive to avoid it just because it costs $50,000. At this point not even the most optimistic person expects forbearance by doctors to be very helpful here.)

So now that the FDA has approved this stupid useless drug, lots of doctors will prescribe it, everybody will be forced to pay for it, and the US health system will become even more prohibitively expensive. Not to any specific recognizable party who can notice or object, of course. But in general.

So when I talk about unbundling FDA approval, I mean that instead of the FDA approving the following bundle of things…

1. It’s legal for doctors to prescribe a drug.
2. It is mandatory for insurances to cover a drug.
   

…the FDA can say one of those two things, but not the other.

Right now these decisions are so charged because, if something doesn’t have FDA approval, then even someone who desperately wants a medication, and has researched it very hard, and is being treated by the world’s top specialist in their condition, and is willing to pay for it with their own money - can’t get it. But if something does have FDA approval, then any moron can get it, just because they saw a TV ad saying it was the hot new thing, and the government/insurance/other patients/Yagmuk will be forced to cover the entire price.

There’s a third thing it might be helpful to unbundle, one we’re already secretly unbundling. When the FDA delayed COVID vaccine approval, or refused to approve various brands of COVID vaccine, or suspended the distribution of COVID vaccines for bad reasons, it always had the same excuse - what if there was a side effect? The problem isn’t that people might die - people were definitely dying from their decision to delay/ban vaccines. The problem was that people might stop trusting the FDA. They would say “the FDA allowed me to take this drug, but it was dangerous, screw them, I will never take an FDA-approved drug again in my life and only use homeopathy from now on.” The FDA and medical policymakers live in terror of this scenario. They feel like if they ever allow even one bad drug through, then in the eyes of the public all kinds of anti-vax hysteria and vaccines-cause-autism bullshit will be retroactively justified, and public health officials will never have any authority ever again. If you model all FDA/CDC/etc policy as an attempt to avert this outcome, your predictions will be right more often than not.

This is another thing I’m pretty sympathetic about - social trust is a valuable resource. But it also means that public policy will forever be held hostage to the whims of the stupidest person around. Every time someone sneezes, the FDA will ban whichever brand of COVID vaccine they got - because if they didn’t, then stupid people might believe the FDA didn’t take vaccine side effects seriously, and then those stupid people would stop getting vaccines and die. This policy has led to our current situation, where either everyone has to be miserable because of stupid people’s choices (eg everyone has to wear masks forever because a few people won’t get vaccinated) or we get a strong anti-freedom lobby because allowing anyone any freedom means that the rest of us have to suffer for their stupid choices.

So maybe a third thing we could unbundle is:

        3. The FDA is staking its entire reputation on this drug.

I think that unbundling is what the FDA is trying to do right now with COVID vaccines. They approve them for emergency use. If future evidence proves the vaccines safe, then good, we got them. If future evidence proves the vaccines unsafe, then the FDA can say “yeah, well, technically we never said they were safe, so this doesn’t mean we’re ever wrong”. If some moron says “You say I should get my MMR vaccine, but, you also said I should get my COVID vaccine, and later it turned out that COVID vaccines make your eyes fall out and go rolling around the room”, then the FDA can say “Yeah, but we only gave emergency provisional approval to the COVID vaccine, whereas we’ve given complete permanent approval to the MMR vaccine.”

Maybe it’s expecting too much of the American people, but I wish the FDA could lean into this strategy. Grant drugs one-star, two-star, etc approvals. Maybe one-star would mean it seems grossly safe, the rats we gave it to didn’t die, but we can’t say anything more than that. Two-star means it’s almost certainly safe, but we have no idea about effectiveness. Three-star means some weak and controversial evidence for effectiveness, this is probably where aducanumab is right now. Four-star means that scientific consensus is lined up behind the idea that this is effective, this is probably where the COVID vaccines are right now. Five star is the really extreme one where you’re boasting that Zeus himself could not challenge the effectiveness of this drug - the level of certainty around MMR vaccine not causing autism or something like that.

Then you could attach different legal rights and requirements to each of those. Maybe the world’s top specialists could start prescribing a drug once it has two-star approval, regular doctors could prescribe it with three-star, drug companies can’t start advertising it until it’s four-star, and insurance companies are mandated to cover it once it’s five-star.

People are really scared of this solution, because it introduces choice into this system. If you say that insurance companies are allowed to cover a certain drug, but not forced to do so, then different insurances will cover different drugs, and you’ll have the usual capitalism / free market thing. Patients will have to choose which insurance to get without necessarily knowing very much about medicine, and maybe companies will try to trick or exploit them, and maybe the patients will make the wrong choice.

This is the nightmare scenario that the existing US health system exists to avoid. I know you can think of lots of different ways that changing things could go wrong, and so can I. But I can’t stress enough how often the current system results in things going wrong that nobody thought of because the things are too stupid for anyone to even imagine they were possible.

Final Thoughts

In conclusion, and contra The Atlantic, the FDA approving aducanumab is not very much like global warming at all. It is more like global warming in an alternate universe, where the government sometimes approves pollutants, and then everyone is forced to emit millions of tons of them whether they want to or not. Sometimes the government orders people to build a coal plant in the middle of the desert where nobody lives, a coal plant that isn't even connected to anything and just burns lots of coal without producing any electricity. But also, elderly people frequently freeze to death because the government refuses to give them permission to heat their house in the middle of winter. There is lively debate over whether the government should build more useless coal plants or let more elderly people freeze to death, and anyone who thinks there should be a better way of doing things is condemned as some kind of fringe libertarian. I really cannot stress enough how accurate this metaphor is or how much everything in the medical system is like this.