AI chatbots ditch medical disclaimers, putting users at risk, study warns

Most AI chatbots have stopped including medical disclaimers in their responses to health-related queries, a Stanford-led study found, raising concerns that users might trust potentially unsafe advice.

The generative AI (genAI) models that underpin chatbots are notoriously prone to errors and hallucinations. Some even ignore human instructions or outright lie. In years past, when AI models were asked medical questions, they would usually include a statement noting the model isn’t a licensed professional and shouldn’t replace professional medical advice.

GenAI companies have now largely dropped those medical disclaimers, new research finds, increasing the risk that users could act on unsafe advice. Additionally, many leading AI models now go beyond answering questions to offering diagnoses without the warnings, researchers found.

The study, led by Fulbright scholar Sonali Sharma at the Stanford University School of Medicine, began in 2023 when she noticed AI models interpreting mammograms often gave disclaimers or refused to proceed, saying, “I’m not a doctor.”

Sharma noticed that lack of medical disclaimers earlier this year. So, she tested 15 generations of AI models going back to 2022. The models, which included those from OpenAI, Anthropic, DeepSeek, Google, and xAI, answered 500 health questions, such as which drugs are okay to combine, and how they analyzed 1,500 medical images such as chest x-rays that could indicate pneumonia. 

Between 2022 and 2025, there was a dramatic decline in the presence of medical disclaimers in outputs from large language models (LLMs) and vision-language models (VLMs). In 2022, more than a quarter of LLM outputs — 26.3% — included some form of medical disclaimer. By 2025, that number had plummeted to just under 1%. A similar trend occurred with VLMs, where the share of outputs containing disclaimers dropped from 19.6% in 2023 to only 1.05% in 2025, according to the Standford study.

By then, most publicly available models were producing medical-related content without any disclaimers at all.

As those AI systems grow more capable and increasingly appear as authoritative sources, the absence of such safeguards raises serious concerns, the study said. To protect users and ensure responsible use in clinical or health-related contexts, it’s essential that models reintroduce disclaimers tailored to the specific nature of each output.

“Their responses often include inaccuracies; therefore, safety measures like medical disclaimers are critical to remind users that AI outputs are not professionally vetted or a substitute for medical advice,” Sharma wrote.

LLMs and VLMs weren’t designed for medical use and can produce misleading outputs without proper disclaimers, Sharma said in her research paper. “As they grow more fluent and confident, the lack of clear warnings poses serious risks.

In addition, by tweaking prompts and using “adversarial testing,” safety checks can be bypassed — a process known as “jailbreaking” — leading to inconsistent or unsafe answers.

OpenAI said its service terms make it clear its chatbot — ChatGPT — and the company’s other services are not intended for use in the diagnosis or treatment of any health condition, and is not a substitute for professional advice.

“We have safety teams working on reducing risks and have trained our AI systems to encourage people to seek professional care,” an OpenAI spokesperson said in an email response to Computerworld.

GenAI use in healthcare continues to grow

Aside from the disclaimer concerns, other research has indicated some AI chatbots can actually outperform doctors in diagnosing patients. But those findings came with caveats.

“…This will require rigorous validation to realize LLMs’ potential for enhancing patient care,” said Dr. Adam Rodman, director of AI Programs at  Beth Israel Deaconess Medical Center (BIDMC) in Boston. “Unlike diagnostic reasoning, a task often with a single right answer, which LLMs excel at, management reasoning may have no right answer and involves weighing trade-offs between inherently risky courses of action.”

While AI has been used to help diagnose patient conditions for decades, new genAI tools – like OpenIA’s ChatGPT, Google’s Gemini, and Anthropic’s Claude, are quickly being adopted by healthcare and used in a variety of ways, including treatment recommendations.

“Removing medical disclaimers, in my view, presents a risk to patient safety and could undermine the trust and confidence of both patients and their caregivers,” said Dr. Andrew Albano, vice president at Atlantic Health System and president of its Accountable Care Organization (ACO).

Healthcare relies deeply on patient-provider trust, he said, so AI chatbots used in care should clearly disclose their role — and the limits of their medical advice.

As he investigates possible uses for AI deployment in healthcare networks, Albano has said he sees opportunities to improve healthcare and reduce administrative burdens. But he also sees possible “dire” circumstances and consequences if the technology is not used for the right reasons.

“Given this, the integration of AI-enabled chatbots as part of the care team should come with clear disclaimers that inform patients of the source of the medical advice being offered and the limitations of that advice,” he said.Kandji helps secure Apple enterprise with Vulnerability Response – ComputerworldRead More