Part 2: Potential Use Cases, LLM Use in Practice
The second of three posts on using AI chatbots responsibly for personal health questions.
Here we will be reviewing what happens when real people actually use chatbots for health questions- where they could be useful and where they fall far short.
Use of AI Chatbots for Diagnosis
In February 2026, Nature Medicine published a randomized trial of over 1,000 adults. Participants were given everyday medical scenarios, then they were randomly assigned either a leading LLM (Large Language Model) or a traditional search engine like google. They were then tasked to 1) research what plan of action the patient needed based on their symptoms/medical history and 2) identify what facts within the scenario were important to come to the conclusion.
For reference, when the AI models were given these scenarios directly from the researchers with no participant in the loop, the models identified the relevant condition 94.9% of the time. When the human participants used the same models, they produced a correct identification in under 34.5% of cases. Comparatively, the control group (people with nothing but ordinary 'Google' search) did better than the people with LLMs. They had 1.76 times higher odds of landing on the right condition.
The researchers found two reasons. First, people didn't know which details mattered most , so they left them out. Second, when the answer came back containing a mix of correct and incorrect material, they had no way to tell which was which.
Its clear from this study that expertise is important when using LLMs for health inquiries, both for determining the most pertinent facts, and to proof-read the outputs to filter inaccurate information.
The Emergencies it Catches, and the Ones it Misses
In February 2026, researchers ran an independent safety evaluation of a new major consumer AI health product against sixty scenarios written by physicians. For many classic, textbook emergencies like stroke, anaphylaxis, meningitis, etc, the LLM appropriately relayed the importance of immediate evaluation. However, it under-triaged just over half of the true emergencies, recommending re-evaluation in 1-2 days rather than the medically appropriate recommendation to visit the Emergency Department.
Also concerning, the study demonstrated that the LLM did not have the capability to consistently recognize mental health crises. They could not appropriately connect users with crisis resources given their inability to identify case severity.
These safety flaws raise serious concerns for implementing these 'triage models' on a consumer-scale. Its seems logical that these models should undergo more rigorous safety evaluations before being used widely.
Where LLMs Can be Helpful
LLMs can be good at making jargon easier to understand.
A study in JMIR in June 2025 took sixty patient education documents from the American Heart Association, the American Cancer Society, and the American Stroke Association and had chatbots rewrite them. The originals sat between ninth and eleventh grade reading level, whereas the rewrites sat between fifth and eighth. The models were able to do so while preserving the intended messaging with an accuracy of 97-100%.
There is utility to be found when utilizing LLMs if you approach them with correct and complete information, and requesting it to be reframed rather than asking it to add missing pieces or make judgements. For example:
Translating a discharge summary or a specialist's note into English
Explaining a diagnosis after a clinician has made it
Helping you write down what you want to ask before an appointment, so you aren't grasping at straws in the parking lot
Asking questions regarding a study that was completed
Rehearsing how to bring up something embarrassing
In these cases, the truth comes from somewhere else and you're just changing its shape. Errors are visible and recoverable, though it is always good practice to verify.
On the other hand, asking open ended or vague questions like, 'what's wrong with me?' , 'how urgent is this?' , or 'should I stop taking this medicine?' invites errors. As discussed in Part 1, we would be asking the LLM to supply judgment it has no full basis for, in a voice engineered to sound certain, to a person whose phrasing is meaningfully skewing the answer.
Four Things That Will Make Your Answers Better
A variety of studies have looked into strategies to increase the accuracy of LLM outputs. Here are some general tips that can help:
Tell it explicitly that it's allowed to say no. And ask it to state the relevant facts before it answers. Both together, not one that's the combination that worked.
Don't lead with your theory. Just state facts. "Pounding behind my right eye, three days, worse lying down" gets you a better answer than "this is probably a migraine, right?"
Don't argue it into agreeing with you. If you push back, LLMs very often fold. And once it flips, it often stays flipped. Changing its mind is not the same as being right.
Bring it to the expert. I encourage patients to come to me with information they want to review. A good chunk of our job as patient-facing healthcare professionals is helping people understand their health. If you go to your provider and bring your inquiries, they are worth reviewing together.
Privacy Concerns
77% of adults say they're concerned about the privacy of health data shared with AI. And 19% of adults under thirty have uploaded their actual medical records anyway.
HIPAA does not protect all health information. It protects health information held by certain kinds of organizations - doctors, hospitals, insurers, and their contractors. When you use a consumer- facing LLM such as Chat GPT, Gemini, Copilot, etc, any personal health data you enter may be used by the company for its own interests. These companies each have their own privacy policies, and they are subject to change at any time. It is prudent to be cautious- avoid entering identifying details (name, date of birth, record numbers), review whether your conversations are used for model training and to opt out if possible, and think carefully before uploading any medical documents.
References
Pew Research Center. Users of social media and AI chatbots for health information are more likely to say they are convenient than accurate. April 7, 2026. https://www.pewresearch.org/science/2026/04/07/users-of-social-media-and-ai-chatbots-for-health-information-are-more-likely-to-say-they-are-convenient-than-accurate/
Chen S, Gao M, Sasse K, et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digital Medicine, October 17, 2025. https://www.nature.com/articles/s41746-025-02008-z
Rosen KL, Sui M, Heydari K, Enichen EJ, Kvedar JC. The perils of politeness: how large language models may amplify medical misinformation. npj Digital Medicine, November 6, 2025. https://www.nature.com/articles/s41746-025-02135-7
Sharma M, Tong M, Korbak T, et al. Towards Understanding Sycophancy in Language Models. ICLR 2024. https://arxiv.org/abs/2310.13548
SycEval: Evaluating LLM Sycophancy. arXiv. https://arxiv.org/html/2502.08177v4
Spiller TR, et al. Assessing and alleviating state anxiety in large language models. npj Digital Medicine, March 3, 2025. https://www.nature.com/articles/s41746-025-01512-6
Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses. arXiv. https://arxiv.org/html/2604.00014v1
Kalai AT, Nachum O, et al. Why Language Models Hallucinate. OpenAI, 2025. https://openai.com/index/why-language-models-hallucinate/
Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 2023. https://www.nature.com/articles/s41598-023-41032-5
Draelos RL, et al. Large language models provide unsafe answers to patient-posed medical questions. npj Digital Medicine, February 13, 2026. https://www.nature.com/articles/s41746-026-02428-5
Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians. medRxiv preprint, 2025. https://www.medrxiv.org/content/10.1101/2025.08.23.25334280v2.full
OpenAI. Strengthening ChatGPT's responses in sensitive conversations. October 27, 2025. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
Breaking Down the Lawsuit Against OpenAI Over Teen's Suicide. Tech Policy Press. https://www.techpolicy.press/breaking-down-the-lawsuit-against-openai-over-teens-suicide/
Google, Character.AI agree to settle suits involving teen suicide. January 2026. https://www.yahoo.com/news/articles/google-chatbot-startup-character-ai-054822780.html
Moore J, Haber N, et al. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. ACM FAccT 2025. https://news.stanford.edu/stories/2025/06/ai-mental-health-care-tools-dangers-risks
McBain R, et al. Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment. Psychiatric Services, 2025. https://www.rand.org/news/press/2025/08/ai-chatbots-inconsistent-in-answering-questions-about.html
Psychiatrists Hope Chat Logs Can Reveal the Secrets of AI Psychosis. UCSF, January 2026. https://www.ucsf.edu/news/2026/01/431366/psychiatrists-hope-chat-logs-can-reveal-secrets-ai-psychosis
A Case of Bromism Influenced by Use of Artificial Intelligence. Annals of Internal Medicine: Clinical Cases, August 5, 2025. https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260
Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, April 28, 2023. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309
AI chatbots versus human healthcare professionals: a systematic review and meta-analysis of empathy in patient care. British Medical Bulletin, 2025. https://academic.oup.com/bmb/article/156/1/ldaf017/8293249
Bean AM, Payne R, et al. Reliability of LLMs as medical assistants for the general public. University of Oxford, February 2026. https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice
Bean AM, Payne R, et al. Reliability of LLMs as medical assistants for the general public. University of Oxford, February 2026. https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice
