180 Comments

User's avatar
Yehuda Isseroff's avatar

Wow this was an awesome read! Congrats on your beautiful son!! I agree with nearly everything you said and feel about AI, you encapsulated it very well.

Kenny Easwaran's avatar

What a read - I see what you did there!

sclmlw's avatar

Yes, the pun was good. It's in calling out a pun that the joke is ruined.

Californian's avatar

I still don't see the pun

sclmlw's avatar

That's how you know it's good. Only a handful of people will read it and deduce the play on words.

vorkosigan1's avatar

No, what makes it good is its unobviousness until it’s pointed out.

Shaked Koplewitz's avatar

Congratulations on having him

Andy B's avatar

This was delightful, thanks for taking the time to write this up!

Gustavo José Zambrano's avatar

A very perceptive observation. It is rare to see discussions connect foundational theory with practical reality so coherently.

Demarquis's avatar

A human using an AI has the potential to become more powerful than either a human or an AI alone.

UlyssesB's avatar

That was true of chess engines too.

Demarquis's avatar

"In real world settings" I should have said.

Lórien's avatar

Self-driving cars being the obvious counter-example...

Nutrition Capsule's avatar

IIRC LLMs already beat both GPs and GPs using LLMs in diagnostics. These will absolutely be a moment when a human is mostly in the way, and we might have already passed that moment.

(I'm a GP)

Kennedy N's avatar

I haven’t followed the comparison studies. Have they really become this good at diagnostics?

Nutrition Capsule's avatar

Yes. And they're improving, unlike most practicing physicians. You can look it up if you like. An LLM can easily summarize it - I won't flood the comment section.

Demarquis's avatar

Am I the only one noticing the irony here?

Nutrition Capsule's avatar

I'm not sure what's ironic. The fact that as a practicing physician I think LLMs are better than me? Or the fact that I think LLMs improve while thinking that most physicians don't? Or me suggesting to ask an LLM to summarize data pointing to LLM superiority in diagnostics in case KN didn't want to dive too deep into it? Or me saying I won't flood the comment section with a summary I'd anyway ask an LLM to do if such summary was requested?

Sin's avatar

Are you saying the GP's knowledge and experience actually misleads them from the correct answer when using an LLM, and the LLM on its own is more likely to be correct?

Nutrition Capsule's avatar

That's how I interpret it, yes.

AnthonyCV's avatar

Many GPs are really just fancy autocomplete, rounding off the rich and detailed and messy information that comes from the real world into whatever they see as the most likely diagnosis from the statistics of their training distribution. They lack the intrinsic curiosity to pull apart the layers of the puzzle and achieve real understanding and insight.

Sin's avatar

Even so it's a little surprising that a GP using an LLM would perform worse than an LLM by itself. Wouldn't they just autocomplete whatever the LLM told them?

AnthonyCV's avatar

Should we be surprised by that? That wasn't true for basic checklists telling them to wash their hands.

Muskwalker's avatar

The GP would do a looks-good-to-me test on the LLM output, which may have unacceptable false negative rates.

Also if the GP is working interactively with the LLM, it may trigger unrelated LLM problems that degrade its diagnostic ability (such as sycophancy towards the GP).

Probably someone good at both being a GP and working with LLMs might do better with an LLM, where the average GP with average LLM skills might not.

Demarquis's avatar

Yeah, I was assuming a human skilled at using AI, not current humans with an AI dumped in their lap.

Nutrition Capsule's avatar

If you're assuming a honed human skilled at using an AI instead of current humans, why not then compare it to a honed AI skilled at using itself instead of current AIs?

DrMcleod's avatar

Autocomplete with a good bedside manner is probably better than autocomplete alone.

AnthonyCV's avatar

I agree! Isn't it great how consistently LLMs roleplay good bedside manner compared to some humans? Still below the top of the human range, though - they haven't saturated that benchmark yet.

Demarquis's avatar

Nah, that's institutional incentives. Left to themselves, most GP's would choose to spend their time talking to patients all day.

Nutrition Capsule's avatar

Most patients, yes. Some of them I can't wait to get rid of. I apologize, but that's just the way it is.

Demarquis's avatar

Diagnostics is not the entirety of providing healthcare.

Nutrition Capsule's avatar

Agreed. However, it's one of the few places where we have been able to compare humans and AI. There's a long way to go, for sure - my family's financial future depends on that - but I don't think there are many things most doctors do which can't be automated. Robotics lags decades behind, so most things done by hand (outside an operating theater) are relatively safe for now.

A psychiatrist I know routinely uses LLMs to carve treatment plans for his patients, and typically those plans are given good feedback from his seniors. For now, that process needs him to use the LLMs, but I see little reason to believe this necessarily remains the case in, say, a decade or two.

Demarquis's avatar

Also, one thing no one seems to be taking into account here are economic incentives. Once a large proportion of patients are consulting AI when choosing medical care, the incentives start pushing the AI's (or rather their controllers) to promote the most profitable choices.

not Bob Cobb's avatar

Congrats on your son and great writing skills. U put this really well: "Can we somehow stop everything right here"

Nutrition Capsule's avatar

While I partly agree, I still want that life-extending nanotech without that all-life-consuming nanotech.

Violets and Roses's avatar

I don't want to stop until we have post scarcity.

Rosten's avatar

We have that, it's just badly distributed

not Bob Cobb's avatar

Facts. Tho if i wanted to be pedantic i could say we could use better AI's help improving wild animal welfare

Jerry's avatar

And we're super wasteful, like feeding most our crops to animals and losing ~90% of the calories in the process. Not to mention that giant animal torture factories are bad in and of themself

Matthew V. Milone's avatar

I strongly disagree. I think that people who say things like this tend to conflate the momentary alleviation of first-world poverty with the sustainable alleviation of global scarcity. The first is counterproductive, and the latter is practically impossible.

What specific policy are you advocating for?

Hedonic Escalator's avatar

Distribution is a highly difficult and resource-intensive problem which we cannot solve because we do not have post-scarcity.

TonyZa's avatar

Post scarcity is impossible.

Ch Hi's avatar

Post-scarcity is probably impossible except on a temporary basis without strong population control. The speed of light limit means that unfettered growth will always overcrowd the environment. (Other limits would probably come sooner, but that one's really difficult to avoid.)

Jo-Anne Rosen's avatar

The most enjoyable essay on LLM and ChatGPT I have yet to read. Dr. Reid was very well behaved, and IMO, the medical establishment is more screwed up than AI presently. I was very recently introduced to Chat as a tool for historical research. Our conversations frequently make me laugh out loud. Congratulations on your beautiful son.

Ross Levatter's avatar

6 cm and it was missed by all those ultrasounds? Disappointed radiologist here.

User was indefinitely suspended for this comment. Show
zinjanthropus's avatar

Why was user suspended for this comment? Seems innocuous?

Neike Taika-Tessaro's avatar

I reported another comment by the same user further up the page. My rationale was that I'm pretty sure it's spam - it's not actually engaging with the comment it's responding to, it doesn't have any actual message content on its own, and there's a convenient link to the person's substack attached to the comment.

zinjanthropus's avatar

Yes, looks like you are right.

Dane Garrus's avatar

It is a bot account that has made 100+ nearly-identical comments on other blogs, with 70+ of those comments coming in a ~3 hour timespan. And yes, dumping all of those comments into a MD and having Pangram scan it shows it to be 100% AI generated:

https://www.pangram.com/history/9d15c970-c4db-49c0-880a-3b449330d960?ucc=PJF3ToqZnGm

zinjanthropus's avatar

I see. I apologize, that didn’t occur to me.

davep's avatar
1hEdited

“Seems innocuous?”

It’s meaningless slop. It’s not even wrong.

(It seems like spam. It has nothing that makes it a specific reply. That is, it would have looked the same as a reply to many of the other posts here. It doesn’t relate in any real way to the post it’s replying to.)

Dane Garrus's avatar

For anyone wondering why this user was indefinitely suspended for this comment: The ban has nothing to do with the substance of the comment itself. Rather, it is a bot account:

Look at the post history, and you will see 100+ comments in the past 2 days that all follow the exact same rhetorical structure of "Brief laudatory noun phrase. A longer sentence further elaborating on the thing being praised." (They occasionally break from this pattern to self-promote.)

Within this, there is also extremely abnormal behavior that is obviously not the work of a human commenter: on October 1, the account posted 73 comments within the span of 3 hours and 18 minutes. Obviously Mr. Zambrano did not find the time within 3 hours to read 73 different posts and compose similarly-phrased comments remarking on their insight.

And yes, running his body of work through Pangram shows it to be 100% AI-generated: https://www.pangram.com/history/9d15c970-c4db-49c0-880a-3b449330d960?ucc=PJF3ToqZnGm

davep's avatar

“The ban has nothing to do with the substance of the comment itself.”

There is no “substance”.

This one post looks like spam.

Halle's avatar

That’s what I thought when I read it. That’s comparable to the size of the uterus itself in some dimensions. How do you miss something that large on an ultrasound, let alone a bunch of different ultrasounds? I’ve got a dozen or so fibroids half that size and smaller, and they found them right away with a simple ultrasound.

Henry Howard's avatar

That struck me as very unusual also. The hesitation to recommend an MRI seemed fair given the very low chance of it finding anything relevant that the ultrasound wouldn't have seen.

Is the chance of missing a 6cm mass on ultrasound higher than I think?

Ross Levatter's avatar

Well that depends on how likely you think it is.😁

It depends in part on patient habitus. Fat is not a friend of ultrasound. However, using transvaginal probes should lead to excellent visualization of the uterus even in the obese.

HaPea's avatar

Agreed. OBGYN and I have no idea how this was missed, I really couldn’t wrap my head around it. For context an average uterus is 7 cm so a 6 cm would be like a whole extra organ… seems crazy to have gone though five transfers and no one noticed?

Sin's avatar
12hEdited

Nobody wants to say it because we like to think of medical professionals as tireless and perfect lifesaving angels, but like in all professions, there is a range of competence there.

Paula Amato's avatar

Agree. And after five failed embryo transfers, the patient surely would have had a hysteroscopy where it would have been difficult to miss a large submucosal fibroid. But in any case, it should have been visible on ultrasound or sonohysterogram. Given that it was removed laparoscopically, my guess is that it was intramural. The data is mixed on whether these impact pregnancy rates in IVF. Obviously a single anecdote doesn’t prove anything. Maybe she would have gotten pregnant anyway on her next transfer without a myomectomy or maybe not. Large trials are needed to answer the question.

Ch Hi's avatar

How are you going to get a large trial on something like that? I feel you're asking for the impossible.

Poodoodle's avatar

She’s not really asking, just stating epistemology.

Bart S's avatar

As a total noob to the topic I can only wonder: is there a chance it had only recently reached this size but had been impeding pregnancies for a long time?

Ross Levatter's avatar

I of course wonder whether it was read by a radiologist or a gynecologist.😉

Ellen's avatar

also very confused by this as a sufferer of fibroids and endometriosis. If this was a story about how endometriosis was preventing successful pregnancies it would be one thing. But fibroids are not hard to spot. Even a manual pelvic exam could have found a fibroid that large, right?

Drew Housman's avatar

Author here -- yeah, it's wild right? Here are a few more details for the interested. The problem fibroid was intramural with a submucosal component (mostly in the muscle wall), so I guess those are hard to see on ultrasound? Pathology report says 6 cm, I double checked. I'm tempted to share a pic, that thing was big and gnarly looking.

She didn’t get a hysteroscopy until after the MRI, doing that sooner would have been a good idea but it was never presented as an option.

She had other fibroids that were seen on ultrasound and removed during a surgery for endometriosis, so it's not like we didn't know fibroids could be an issue.

Obesity was not a factor, she's 5'6 120 pounds.

Ross Levatter's avatar

It might be interesting to ask your AI “why would a 6 cm fibroid be missed on multiple technically competent ultrasounds?”

User was indefinitely suspended for this comment. Show
Melvin's avatar

Oh sure you mock now, but if GPT4 can get a young woman pregnant with one baby, then surely GPT5 can get a 50-year-old woman pregnant with twins, and GPT6 will get men pregnant with quadruplets. By GPT7, we'll all be having octuplets who themselves are already pregnant with octuplets, and the lightcone will be filled with babies by 2035.

Presto's avatar

cuteness-maximizer

Violets and Roses's avatar

That explains the catgirls.

Don Geddis's avatar

AI safety is not "idiotic". The worst predictions may or may not come true ... but that's the case with any risk assessment. There are very reasonable and justifiable reasons to be concerned about opaque super intelligent AIs running our society. It is not "embarrassing" to be involved in a sober careful analysis of possible future risks.

Kenny Easwaran's avatar

One should stop making stupid anti-AI jokes or comments. Safetyism can sometimes consist in stupid anti-AI jokes or comments, but it can also consist in thoughtful discussion of the pros and cons of AI. We should try to do a lot more of the latter than the former.

Ch Hi's avatar

"Should" isn't a very useful think to say to a population most of whom aren't listening. It doesn't work very well even to those paying attention. Actually, it doesn't work all that well even when directed at oneself...and that's really where it should be directed as "should" is a modal form of "shall".

UlyssesB's avatar

I don't think you should conflate water hysteria, which we already know to be false with fears about growing AI capabilities.

nominative indecisiveness's avatar

I postulate the existence of a future god of humour who will torture you for all eternity if you don't laugh at funny anti-AI jokes today. (He will give you a pass for unfunny jokes, and if you at least crack a smile sometimes then he'll just prank you once a century.)

DrMcleod's avatar

This is lowkey true.

Odd anon's avatar

I desperately, desperately hope you are right. I would relish the little thing that is the feeling of embarrassment, overwhelmed by the feeling of relief that life continues on, that all my fears were wrong, a passing thing, that children will grow up, that there's a future to look forward to.

I just don't think you are.

Bugmaster's avatar

This was both heartwarming and depressing. Heartwarming, because of the happy ending. But also depressing, because the AI bot didn't perform any superhuman feats of AGI mental might. It just did the bare minimum that that entire health care system refused to do. People complain about losing their jobs to AI, but if you stubbornly refuse to do your job, year after year, then maybe you *should* be replaced.

bird's avatar

Now do you understand how low the bar is for AGI? These things do not sleep, or suffer, or desire trivial things, and they can be duplicated as long as the compute necessary is provided.

They do not need to be perfect. They just need to be equivalent in terms of intelligence, and every other advantage they have will create a gap the size of the ocean between them and the people they will replace.

Bugmaster's avatar

> Now do you understand how low the bar is for AGI?

First of all, my entire point was that we appear to be accomplishing the "equivalent in terms of intelligence" part backwards: by making humans effectively dumber. Maybe all those medical professionals involved in the story were smart in some abstract sense, but they sure didn't do anything useful until forced to do so, so their equivalent intelligence was near zero until that point. Ok, that is maybe a bit unfair; they did after all run some tests that ruled out a lot of conditions, and that is a useful result, but still.

Secondly, if a tool that can perform knowledge lookups on demand incredibly quickly, and apply them to a narrowly-defined task with some degree of reliability counts as "AGI", then sure, let's call it "AGI" -- but with the understanding that such "AGI" is still incredibly sub-par compared to the intelligence of the average motivated human; cannot operate without human guidance; and on its own poses neither an existential threat nor the promise of utopia. It merely amplifies these capabilities in its human users, much like Excel or Photoshop or BLAST, or indeed a hammer, only more so.

Nutrition Capsule's avatar

>> Ok, that is maybe a bit unfair

That's totally unfair. Yes I agree they should have taken the MRI earlier (I mean, anyone can tell they should have! That was the solution!), but real life medicine is hard. Most of the time - I guarantee, literally almost every time - that MRI won't do squat. And when it shows nothing, the patient usually doesn't relent, but instead doubles down and demands even more imaging studies to be done.

I know, because I'm the guy who didn't go down the path of "let's try to convince them they don't need the study done on them". I gave them everything they wanted. And most of the time, I ended up wasting everyone's time and money.

No mistake, sometimes scanning a person entirely and putting them through the full works is the solution, but please, PLEASE don't reduce this to doctors being unwilling to do their jobs. Get over here and do it yourself - without the LLM - if you think it's that easy.

For clarity, I'm not arguing against doing imaging studies in general - I love MRIs, and would love to have all studies done on myself. My point is that in real life, most of the time (especially when the first tests come up negative), pinpointing the correct diagnostic path is not easy except post hoc.

Sam Hartman's avatar

If anything, this showed both the pro and con of talking to an LLM about a medical condition. You can ask it 100 questions without it getting tired, at any hour of the day, with no irritation or shutting down the conversation. That's a major pro!

But LLM's also default to pleasing the user. Newer models can be better about this, but I still find that if I push them hard enough, they'll tell me the answer I "wanted" to hear. That's a con, particularly if your LLM is giving medical advice on a topic you're emotionally invested in.

Violets and Roses's avatar

> They do not need to be perfect. They just need to be equivalent in terms of intelligence, and every other advantage they have will create a gap the size of the ocean between them and the people they will replace.

This makes me think of self-driving cars. They don't need to be a better driver than an average human. Just better than a human whose too tired or in emotional turmoil, because the AI never will be.

Bugmaster's avatar

> Just better than a human whose too tired or in emotional turmoil, because the AI never will be.

This is not entirely accurate. Yes, LLMs and other ML models do not suffer emotional turmoil; but they are still subject to hallucinations, mis-identifications, and other glitches. They do it all the time, in fact; but you are correct that if we could get their glitching rate below that of humans, we'd score a huge win in traffic safety.

Ch Hi's avatar

If I read (and remember) correctly, WayMo is already claiming that. It's not clear that this would be true in all conditions, however, because they're operating in a restricted environment.

Viliam's avatar
12hEdited

Sometimes I ask myself "what would Claude do?" and the answer is that Claude would never give up and would keep trying... and then start hallucinating... so I guess the difference is that AIs don't know when it is rational to give up, but humans often give up too soon, at least I do.

Ch Hi's avatar

Yeah, it's definitely true that current AI's tend to go off the rails when asked to solve an impossible (under the conditions) problem, and then aren't allowed to give up.

Mister_M's avatar

If I work persistently for days without sleep, I will also hallucinate

Randomstringofcharacters's avatar

I think the big benefit is making the previously scarce resource of expert attention less scarce. It's not out of malice that the medical system only gives you 30 mins with a specialist, who mostly works with a basic flowchart, but because that's an efficient use of resources at scale.

Alex Selby's avatar

It's funny because when I got to this point I thought "Oh cool, it's human-written". (Yes I know someone could ask their LLM to include small errors, but they don't, OK?, so let's keep this trick a secret between us.)

Richard Meadows's avatar

Congratulations! What a blessing. I don't have any dramatic stories, but Dr Claude has been an absolute godsend this winter for tracking and treating all of the various lurgs and maladies that my two little kids bring home.

Average Man's avatar

Can you elaborate on that? I’m curious of the benefits vs just dealing with common kid illnesses, flu, norovirus, hand foot mouth, etc

Richard Meadows's avatar

Yeah so it's stuff like:

- maintaining a detailed history of all my and my kids past health issues

- giving me the optimal treatment plan for my specific situation

- tracking when I gave what kid which medication or painkiller

- assuaging anxiety, giving me hard gates for when I should escalate to a real doctor

So none of it is revolutionary, it's more so just incredibly convenient and useful to have a highly engaged and responsive doctor-equivalent in your pocket.

To give a couple of example:

1. in the early stages of the flu, it proactively reminded me to start using a certain nasal spray that helps avoid sinus infection, which takes several days to build up effectiveness, and which I'd totally forgotten about.

2. I had some unfamiliar pain that it helped me narrow down to trapped gas, and suggested a medicine I'd never heard of before, which worked well.

Meanwhile it typically takes weeks to see a GP if it's not an emergency, and they may or may not solve the problem within my 15 minute consultation. In general there's just a lot of health stuff that never rises to the level of *needing* to visit the doctor, but where you still benefit a lot from getting good advice.

SMK's avatar

What a beautiful boy -- congratulations!

Drea's avatar

Beautiful, and congratulations to you both.

And as a resident of the Wild and Wonderful state of West Virginia, I can assure your friends that people drink dirty water not because of data centers (which are never near the bad water) but because government is incompetent and corrupt from bottom to top (and maybe because someone promised them city water in a country location, then couldn't fund the maintenance with the shrinking tax base).

Some Guy's avatar

He is a handsome fellow and I am glad to hear you and your wife came to a good end. Every day I find some new use for it. Lately I’ve been making my children little books. There is an upside as real as the dangers are.

Paulo Esteves's avatar

This was awesome. Thanks for writing this up.

BD's avatar

My wife and I have the same unexplained infertility diagnosis. The difficult part in my situation is that she is not as inquisitive or answer seeking as I am. I feel lost and ill-equipped to replicate your process. I don't even know if she has records of her visits and imaging.

Ebrima Lelisa's avatar

You can do it man. Believe in yourself. Think - today is day one. You can start today

Mister_M's avatar

I often make this sort of exhortation to Claude before I go to bed. He gets stuff done.

Eremolalos's avatar

If your doctor uses a patient gateway the scans and reports on them will be on the gateway.

Sniffnoy's avatar

Congratulations! And hello Reed! :D

Catherine Caldwell-Harris's avatar

Like everyone else, I really enjoyed Drew Houseman's write-up of his pregnancy journey. I had a related experience with Google back in 2009 when I received IVF with egg donation due to, as they called it, "advanced maternal age." I was doing fertility tourism in Turkey. My wonderful doctors were highly competent and empathetic but only explained the outline of what was being done at each visit and never gave me even a single sentence of written material. So after every clinic appointment, I attacked Google. Asking all my questions to learn what was being done to me. I suspect google sufficed for my case because mine was just a routine IVF. But with this background, it was a special pleasure to hear the saga of the parallel but more advanced research and problem-solving you received with your TV AI doctor.

Substratespecificity's avatar

I am curious about how you thought of some of the privacy concerns. I also have tons of medical questions for my doctors but I am really hesitant to let Sam and Dario know too much about my personal medical situation. In principle I guess this isn't too different from googling everything but it feels really different

Nutrition Capsule's avatar

I'm wondering about this - why would Sam and Dario (as in the companies they lead) give a flying duck about your health concerns? What are the negative scenarios you are thinking of?

vorkosigan1's avatar

Blackmail. Customized pricing.

Nutrition Capsule's avatar

I don't follow. Why would ultra rich companies alienate their user base and push them towards competition by blackmailing them? And why would they change their pricing based on the user's health data in particular?

Frikgeek's avatar

Ultra rich companies already do this. Once your users are "locked in" you start ripping them off as hard as you can, and all sorts of personal data can help you do that more effectively. Outright blackmail is not going to happen but customized pricing definitely is. And if a company like OpenAI, Antrophic, or Google can't directly sell you things they'll sell your data to someone else who does and can profit off it, like insurers, Amazon, or even airlines.

As for why, probably because it makes them more money. But the key part is locking in your users first, if you start enshittifying before that then your users will just leave.

Nutrition Capsule's avatar

What sort of blackmail are we talking about? And how does the health data fit in on custom pricing LLM services?

Frikgeek's avatar

LLMs are being practically given away for free so price gouging is not going to happen there. But Anthropic can create an accurate profile of what your needs are and then sell that to everyone else so they can price gouge you.

If you have an important appointment with a specialist that requires you fly suddenly tickets are three times more expensive. Supplements or OTC meds you've been ordering online are up in price too, it's not like you can afford not to buy them.

Nutrition Capsule's avatar

So, a cartel the size of the market. I suppose it's possible, but unlikely. Willing to bet this won't happen anything close to the scale you describe.

Frikgeek's avatar

What do you mean it's possible? It's literally already happening. I genuinely wonder what world you live in if you think surveillance pricing isn't a big deal.

And yes, obviously it's done a lot by companies that have a de-facto monopoly. Amazon, Uber, Tinder. MultiPlan and Express scripts are also #2 and #3 on the Surveillance pricing index, both with high confidentiality. So yes, your personal data is used to price gouge you on healthcare. Right now. Not in theory, not in the future.

https://surveillancepricingindex.org/#companies

Nutrition Capsule's avatar

I'm 100% sure that we aren't living in a world where health data gathered by LLMs is used to blackmail or customize prices for LLM users in a cartel the size of the market.

I'm sorry if you interpret otherwise, but that's where this discussion started.

Frikgeek's avatar

OpenAI and Anthropic don't sell user data yet, but Google absolutely does. Everything you say in a chat with Gemini can be sold off to a data broker and eventually used to build a detailed profile of your needs and habits for the explicit purpose of price gouging you. And selling user data is an easy way to boost profits when these companies are burning money, it's only a matter of time before they give in as well.

vorkosigan1's avatar

Customized pricing is already being used.

Matthew V. Milone's avatar

Thank you for the link! I've been wishing for something like that for a year or two.

Substratespecificity's avatar

I think am less worried about the AI companies themselves blackmailing me but more so the data being given to health insurance companies to deny coverage, given to employers to do medical discrimination etc.. You could image a deal where Linkedin and openAI sign a deal to do AI assessments of people's competence and interests based on chatbot messages and then the AI discriminates against people with mental illness.

Nutrition Capsule's avatar

I see a lot of issues with such a scenario. To begin with, a huge "delete all chats" flood for all suggested LLMs if said policies were actually threatened to be applied.

That would make me a person with a couple of dozen different cancers, several autoimmune diseases and both sexes. And a stoner and a crazy genius and a severely disabled person. How on Earth are they going to differentiate personal health chats from interests, interests from trolling, trolling from fooling around and all of the aforementioned from another person using the account?

Such behavior would also severely favor providers who promised to share less data, unless people believe said data is good for them (in which case they wouldn't mind).

"Profile hacking" would become so common sharing such data would become practically meaningless.

They also wouldn't update on future negative health outcomes as people would hide those from LLMs. So the trick only works once, and on a fixed population at that.

And so on, and on. I don't see this as a realistic future, but perhaps a will shall find a way.

Presto's avatar

What a great post. Congrats !

As a French man with a pregnant wife, I'm considering naming the baby either Claude or Astra...

Eszti's avatar

I highly recommend to look up what happened to everyone called Alexa

Martin Sustrik's avatar

A friend of mine, a doctor, had a case where they failed to diagnose a cancer and a young woman died because of that. Afterwards, she fed the symptoms to ChatGPT and it suggested checks that would have diagnosed the problem. Obvious stuff, like "ask about the cancer history in the family". Her point was that doctors often fail not because they are incapable, but because they are overworked and things just fall through the cracks. And that the AI could help, not because it is som kind of a super-human doctor, but because it has time that doctors don't. It can speak to the patient for hours. It can diligently check the literature. Etc. This may well have been the case here.

Xpym's avatar

> not because it is som kind of a super-human doctor, but because it has time that doctors don't

Having something that humans don't seems like a decent definition of "superhuman".

vorkosigan1's avatar

Then people who are in a permanent coma are superhuman?

Xpym's avatar

Clearly. Their ability to be unbothered by stuff is unmatched!

Viliam's avatar

Some Buddhists spend their entire lives training to maybe get halfway there.

Lórien's avatar

Patients also hate having to fill out a fourth questionnaire with the same information as the last three, minus one question about family history and plus two about MRI history, just in time to receive their fifth questionnaire (no one knows what happened to the fourth). Conditional on being fine with this sort of thing, as OP clearly is, LLMs are pure net-gain. And LLM's instant response time and good personality helps a lot. But not all people are willing to sit through that, and as others have noted, not even unlimited scans can find everything.

Andrew McDonald's avatar

But the conclusion of the piece is - taking away the verbiage and shy handwringing- that ‘I got what I wanted, and I don’t know what should be done to mitigate the ghastly problems that will follow if everybody else uses A.I. to get what they want, so…there you go, eh?’ Plus baby pic, all OK with that?? What spinach.

DrMcleod's avatar

Or... Maybe it is pointing about that all technology, from fire onwards, provides both benefits and risks and we should work to maximise the former whilst minimising the latter.

Randomstringofcharacters's avatar

Not everything is zero sum. If we can get everyone accurate medical diagnoses faster that is just a general good.

TGGP's avatar

The piece does not claim that "ghastly problems [...] will follow if everybody else uses A.I. to get what they want".

Brandon Fishback's avatar

AI isn’t poisoning the water supply. The main existing downsides are aesthetics. So it’s that plus potential problems versus very real benefits.

Larry's avatar

Brilliant! Was going to write something similar. But you beat me to the punch!

Jennifer B.'s avatar

This story made me very happy for you and I loved it.

DrMcleod's avatar

Exactly related: https://www.nytimes.com/2026/09/29/health/maha-summit-kennedy-vance.html

"A.I. Is ‘Better Informed’ Than Doctors, Kennedy Tells Industry-Backed MAHA Summit"

DrMcleod's avatar

Incidentally, this is exactly why I ask my LLM to critique my software development plans before they get implemented. It knows more than I do, makes good suggestions, and points out potential problems with my scheme, and there is no point denying it.

Viliam's avatar

Just following a checklist sometimes helps a lot, but people still don't do it, because it is annoying. LLM can go through a checklist easily.

Again with a Pen's avatar

I am confused about the mental model underneath this.

The story, as told, makes it seem as if an MRI is highly unusual in this situation given that doctors do not actively suggest it and oppose it when suggested.

So what happened here?

- Over time and several conversations the AI got enough context about this particular case that it recommended a highly unusual diagnostic for this case specifically. (AI discovered something about this case that doctors missed)

- MRIs are good in any case of unexplained fertility, regardless of the particular case. (AI discovered a blind spot in the system / the system failed the author)

- MRIs are something doctors in medical TV do a lot so if you ask an AI to write a script for long enough at some point it will recommend an MRI, no matter what the diagnosis, let alone the particular case is. (AI got lucky)

- Something else.

Viliam's avatar

Great question!

I could give you a "Bayesian" argument that after you tried the usual things and they failed, you need to consider the unusual things (proportionally to their prior probabilities and the evidence you have received so far), which could push MRI from "unlikely to be useful" to "likely to be useful".

(Something like -- numbers completely made up -- there is only a 1 in 1000 chance that MRI would help a random person. But we already tried things that would have helped 995 out of 1000 people, so now you are one of the remaining 5, and there is a 20% chance you are the one who would benefit from the MRI.)

But why the LLM actually did it, that is a completely different question.

Matthew V. Milone's avatar

Thank you for mentioning this. I'm optimistic about AI in medical diagnostics, but Drew's methodology seems very unsound.

For one thing, asking an AI to roleplay, especially using an explicit reference to popular fiction, seems likely to elicit character-driven behavior, with all of its flaws, rather than true rationality. I have a similar critique of the use of AI for personal coaching.*

For another, some of the questions that Drew asks the AI are presumptive and leading. Asking "is there a problem?" is a lot different from asking "what's the best evidence for a problem?", and I think that the question in the third screenshot ("[W]hat might [you] see that's concerning?" seems closer to the latter.

*https://matthewmilone.substack.com/p/companions-are-not-coaches

Neurology For You's avatar

There’s a saying in medicine that even a blind pig finds an acorn now and then. I’m surprised an AI prompted with “Dr House” didn’t suggest an MRI much earlier, Dr House orders them like DoorDash burritos.

Nan Deng's avatar

The MRI episode is a useful example of why fluent medical assistance needs a verification boundary: the model surfaced a diagnostic path, but the clinician still had to validate it and act. The strongest workflow seems less like replacing expertise and more like giving patients a persistent, context-aware question generator whose suggestions are explicitly checked against records and guidelines.

Violets and Roses's avatar

Congratulations on the baby!

Tracy W's avatar

Oh wow this is so cute!

Pre-AI I had my own experience with infertility which eventually resolved in first a +ve on a pee stick and then a baby with all the normal number of limbs and digits but it was a weird time.

Tasty_Y's avatar

Great tale! They should make a movie out of it (at least an AI-generated short film) where the actual events are portrayed realistically, but the "Dr. Reid" interactions are depicted as a cheesy TV drama, with stilted writing (but very good and specific medical advice!) and obvious AI artifacts all over the place. And Dr. Reid also gets a happy ending where the happy parents show him the baby named after him.

Julia D.'s avatar

This could be an entertaining and more lighthearted than usual episode of Black Mirror.

Raphaël Roche's avatar

I'm happy for the author and his wife. Still, I always feel ambivalent about this kind of story. The natural conclusion seems very favorable to AI and very unfavorable to human doctors and the medical establishment.

But the same story could have been told twenty years ago with no AI in the loop. Desperate patients determined not to give up would have kept seeking information from other healthcare professionals, patient associations, and so on. They would have read and reread the medical file and stumbled upon the document suggesting an MRI, or someone else would have suggested it. They would have pressed the doctor in charge of the case until they got it, and the outcome would have been the same. It's worth keeping in mind that medical errors, or plain negligence, have always existed.

Even setting fault aside, it's also a matter of how much one invests. How much time, energy and money is it reasonable to commit to a goal whose outcome is uncertain? To take a reductio ad absurdum: a patient who exhausts all their time and resources trying to get to the bottom of their infertility, to the point of losing their job and ending up on the street. If they fail, we might be tempted to call them unreasonable. If, against all odds, their efforts paid off and they later got back on their feet and lived a good life with their children, we might admire them instead. We are quick to judge a situation by its outcome, and the author's story is no exception. He probably wouldn't have published it had it ended in failure. There is inevitably some bias here, since this is an individual story with no counterfactual.

In short, it's hard to conclude anything from a single anecdote, let alone get excited about AI assistance on that basis.

That said, this view coexists in my mind with a less guarded one. There's no denying that technology brings a great deal to medicine. In particular, the very existence of MRI may matter even more than AI in this story. It would be naive to dismiss what technology contributes. The surgical robot is an asset, and so is AI.

At the time of the events the author describes, I again don't think AI was entirely a game changer, since he could probably have reached the same outcome by combing through the medical file again and again and continuing to consult various people. The result is mainly due to the couple's commitment and determination. Back then, GPT-4 was not unambiguously superhuman in medicine.

However, the trend lines keep going, and yes, there will come a point where AI is vastly superhuman in medicine. I wouldn't be surprised if that were already the case. What's at stake is whether this technology will empower doctors, as MRI and surgical robots have, rather than disempowering them. That remains an open question. There is also the concern that this empowerment period could be very short before desempowerment or a tragic loss of control.

TGGP's avatar

They didn't need to exhaust all their time & resources, because the AI made it easier for them to talk to a "doctor" whenever they wanted.

Doctor Mist's avatar

I don’t know if you meant this, but your “what’s at stake” paragraph suggests that the important thing is the consequences to doctors, rather than the benefits to patients.

Theragra Chalcogramma's avatar

So nice outcome. I'm very happy for the author and his wife.

Ehh.

I'm ill for 11 years, in a terrible neurological pain all over my body. (Allodynia?)

Nobody knows why. Unfortunately, it does not look that my issues can be solved by a measly MRI. (I did several).

I tried the same yesterday, asking an AI chatbot. Unfortunately, most things it outlines I already did try.

Usual pain relievers, SSRI, tricyclics, gabanoids, massage, exercise, avoiding stress, not working, sleeping a lot, yoga, swimming.

The only things that helps is opiates, and this is the road in one direction. I'm going there, and death is closer with each step.

Maybe I need to do more tests for nerve damage (but how to treat it anyway?)

Eremolalos's avatar

Your best hope is to make yourself as expert as a layman can be on your problem. I recommend you get on one of the really good chatbots (I use GPT Astra) and have it do a thorough "consult," aiming first to identify possible causes of your pain. Give it an initial brief summary of the problem then ask it to interview you in depth and ask for any records it needs to see, aiming at this point to figure out what the possible diagnoses are, not what treatments might work. Upload any records it asks for. Ask it to make a list of the likeliest causes of your pain, and for each to name the features of your case that make it a good fit, and the ones that don't, and what tests are needed to determine whether you have that diagnosis, and what treatments there are for each, and how effective. And it is worthwhile, for each possible diagnosis, to ask whether there are any novel, effective treatments for it or any old effective ones that are currently overlooked. Get the AI to summarize the research on diagnosis, how illness works, and treatment, asking that it use only reputable research sources.

This process will take several hours. It will be time well spent.

Then take it from there. You should probably see a doctor at that point, using the AI to find doctors with special expertise in the problem you suspect you have.

Selfmaker's avatar

Very heartwarming <3

George H.'s avatar

Wow, this was great, thx. (I can feel a small tear starting in the corner of my eye... :)

FlyingPhaser913's avatar

Congratulations on your baby!

Our life is all about our son right now and it's hard to think about how close a call it was for him to actually come to being. Our doctors were really good all along the way, which has been part of it -- they were super knowledgeable and gave us plenty of time at each visit. Even then, a lot of things had to go right for us to have our healthy baby (in particular some well-timed scans that discovered a severe fetal growth restriction).

Arminius's avatar

Congratulations! Heart-warming story

2irons's avatar

Beautiful story. Thanks

Parker Smith, PMHNP-BC's avatar

The part that got me is the AI didn't out-diagnose anyone. It just read everything, which is the one thing nobody in the system has time to do. Six years, one fibroid, zero fresh eyes until a $20/month chatbot. That's either an indictment or a business model, and I'm not sure which... either way kinda scary

MichaeL Roe's avatar

Thanks for the article!

It’s unfortunate you have to use jailbreaks / workarounds to get an answer out of ChatGPT.

Deepseek will usually answer medical question for me without needing jailbreaks, although it does sometimes get scared looking at my data. (To be fair on DeepSeek here, both my GP and my endocrinologist are kind of twitchy about my test results, for reasons where DeepSeek is thinking ahead of them)

hazard's avatar

You could have bootstrapped Reed to ASI with the Dr. House persona if you told it to continuously pretend to take Vicodin and spin up other LLMs there to act exasperated at him and talk about how all the tests have failed and surely there's no solution while making meta comments that they are approaching the last 10 minutes of the episode.

Vermillion's avatar

Congrats! Similar if less dramatic story, my wife and I have been using Claude to generate social stories for our 4 y/o who can have a lot of difficulty navigating new situations and encounters flexibly. We upload a few pictures and a prompt and it starts chugging away. It's part of an ongoing project so it can also reference previous social stories and all our other notes on him so it's very personalized. Our son loves them and I think it's also helpful.

So yeah, thanks robots, that's good, please don't get murderous.

Drew Housman's avatar

Thank you! And haha yep, no murders just helpful stories please. That’s a really cool use case

Micaela's avatar

This is absolutely insane! Thank you for sharing & congrats on your beautiful baby!

Matthew V. Milone's avatar

Congratulations! This reminds me of a story about someone who consulted with and LLM in (what turned out to be) a medical emergency:

https://benorenstein.substack.com/p/chatgpt-sent-me-to-the-er

Unrelatedly, I've struggled to convince EAs and rationalists to think of prenatal humans as people, so I was encouraged to see you casually refer to your frozen embryos as "babies".

Paula Amato's avatar

Challenging for sure! It would need to be multi-center, would probably take years to enroll enough subjects, and would cost a lot of money.

LightlySearedOnRealitysGrill's avatar

Congrats on your cute son! Nice write up. To me, this seems like a story about medical incompetence than anything else.

Protothecosis's avatar

As others have said, this was not a particularly sophisticated diagnostic suggestion. I definitely support using AI to double check your physicians plan. Although have certainly seen a couple go off the deep end asking for research-level testing for mostly theoretical conditions. I’m surprised by how few of my patients seem to use it so far.

(p)bloom's avatar

I has a similar experience with ChatGPT and IVF!

I did my first round just doing exactly what the doctor told me. We got 8 euploids, which is not bad, but I felt we could do better. Next round I let Dr. Chat take the wheel, and asked my human doctor to follow Dr. Chat's recommendations. We got 14 euploids! And for round 3, 16 euploids!

I think we are officially at the point where the shoggoths are just better than the average doctor (not surprising since real world doctors have time constraints whereas Dr. Chat does not).

I always run my medical stuff through Dr. Chat now, even for minor stuff

(Big changes it suggested were longer stim period with a lower amount of stimulation hormone, and conventional fertilisation rather than ICSI)

Drew Housman's avatar

That’s so awesome! Good luck on the next steps of the journey

Neurology For You's avatar

There’s a kid out there named Opus, probably.

MRIs are one of those technologies that is great for individuals but not for populations, necessarily — not everybody needs an MRI but Drew’s wife did, and does my friend whose cancer was detected after they insisted on getting an MRI — but there’s tons of useless MRIs too.

Ready for more?