Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
Their public messaging claims that LLM's may already be self-aware, may replace most of the workforce or even eradicate humanity but for legalese, it is just a dumb computer program. Can't have it both ways!
LLMs occupy a different niche from other deterministic doohickeys. They are sold as replacing human thought and judgment and so they should themselves be judged by their output... just like the humans they replace.
Your phone alarm isn't adjusting your dosage or treatment plan like ChatGPT was. Your phone alarm isn't recommending you hide symptoms from your physician. It's not giving you medical advice by facilitating your medical treatment, just like how your drinking glass for any pills isn't giving you medical advice.
If they already have a health specific model why aren't the others instructed to forward health related queries? Sounds to me, they are just trying to cover their ass; "Your honor, it is not our fault she didn't use the correct model"
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
ChatGPT Health just tells me to go to the hospital right away for any minor thing
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.