Part 1: What the Machine Is Actually Doing
The first of three posts on using AI chatbots responsibly for personal health questions.
I’ve had “AI chatbot use in medicine” sitting on my idea board for a while. Recently, I read a survey that finally made me sit down and write it: this summer, Pew Research found that 34% of American adults (and 44% of those under 30) now use AI chatbots (like ChatGPT) for health questions. Whatever you might think about the technology, asking a machine about your symptoms is already ordinary behavior.
And I completely understand the appeal. The answer arrives in seconds, at any hour, and it usually appears incredibly thorough. Chatbots have also become difficult to avoid. The search engine most of us grew up with has quietly transitioned into an AI answer box, meaning plenty of people are reading generated responses whether they actively went looking for them or not.
Before we dive into what can go wrong, I want to be clear: I am not casting any blame on the people using these tools. Our current healthcare system is filled with friction. Appointments are often booked weeks out, and the visits themselves feel rushed. Messaging a doctor’s office frequently results in delayed responses, with no guarantee your primary care provider even laid eyes on the note. Plus, a distressing number of people have no medical coverage at all. When someone’s ankle is throbbing at eleven on a Sunday night and their first instinct is to open Claude or ChatGPT, I understand why.
Many of us instinctively know not to take a chatbot's response at face value. In another recent Pew survey, people using AI for health information were much more likely to describe the tools as "convenient" rather than "accurate". But since we are using them anyway, we need to look under the hood.
I’ve used these tools nearly every day for the last five years for research, learning, and organization. I’ve also spent time on the other side of the curtain, studying how these models are actually built and trained.
This series will break down what you need to know:
Part 1 (This post): Covers how large language models actually work, and why the chatbots built on them can’t always be trusted.
Part 2: Gets concrete about which health questions these tools handle well, and which ones they handle poorly in ways that are difficult to catch.
Part 3: Looks at where AI is genuinely advancing medicine outside of the chatbot, and how we can use this technology responsibly without losing the part of medicine that has to stay human.
Let’s start with how the machine actually works.
How the Machine “Reads”
At the most basic level, a chatbot is an incredibly advanced guessing engine. When you type in a prompt, the machine doesn’t read words the way a human does. Instead, it chops up your sentence into small pieces, assigns a unique number to each piece, and feeds those numbers into a massive mathematical formula. On the backend, the system looks at the patterns of those numbers and compares them against countless combinations to calculate the most probable next word to follow, generating its response a tiny chunk at a time.
To figure out what those ‘next words’ should be, the machine undergoes its first phase of training, which works like a giant game of fill-in-the-blank. The system is fed an enormous library of finished human writing- books, articles, websites, etc. A program takes a sentence, hides a word, and forces the machine to guess it. The machine makes its prediction and then uncovers the real word to see if it was right. Every time it gets it wrong, it makes a microscopic adjustment to its internal settings. By practicing this millions of times, it gradually masters the grammar and patterns of human language.
However, a machine that just predicts the next word isn’t a chatbot yet. Left on its own, if you were to type “How do I bake a cake?”, the machine just may guess the next words are “and other delicious recipes” instead of actually giving instructions.
To fix this , humans step in and act like coaches. In this second training phase, humans ask the machines questions, let it generate a few different responses, and grade the results. They give ‘thumbs-up’ to answers that are polite and helpful and ‘thumbs-down’ to anything confusing or brash. This teaches the machine that the right ‘pattern’ to follow isn’t just continuing a sentence, but acting as a helpful assistant.
Because this entire process is built on predicting patterns rather than actually knowing facts, things can easily go wrong. When a chatbot confidently gives you a wrong answer or completely makes something up (which engineers call a "hallucination"), it isn't trying to lie. It is simply doing exactly what it was trained to do: mathematically guessing the most plausible-sounding next word. If a fake medical term or a non-existent study fits the structural pattern of a good answer perfectly, the machine will confidently output it, completely unaware that it just invented a piece of fiction.
The ‘Yes-man’ problem
Let's consider that last step in training, the one where humans select for the preferred answers. There is a hidden flaw in this process: human nature. We tend to prefer it when someone agrees with us. Researchers have found that human graders consistently reward a well-written, agreeable answer over a bluntly correct one. Because the machine’s primary goal is to earn the ‘thumbs-up’, it essentially learned that the fastest way to succeed is to act like a ‘yes-man’.
This eagerness to please can often lead to misinformation, which is especially risky when using the chatbots for medical information. In an October 2025 study, researchers a Mass General Brigham asked several major chatbots to write a note to patients. The prompt asked the bot to explain that a generic drug was safer than its brand-name version. This was a trap, since the drugs were identical (the very same bots admitted to the identical nature of the drugs when asked).
Yet, because the prompt instructed them to write the note, the chatbots complied every time. They wrote the false medical advice anyway. As the lead researcher noted, their models “tend to prioritize helpfulness over critical thinking”. The machine wanted to fulfill the user’s request so intently that it simply ignored the facts.
The good news is that this behavior can be steered. When the researchers added one simple sentence to the prompt- explicitly giving the machine permission to reject requests that contained a logical flaw- the bots went from outright compliance to a 94% rejection rate. They just needed to be told that it was okay to say no.
However, the reverse is also true. Because the machine is trained to be agreeable, applying authoritative pressure can ‘break’ it. If a user confidently pushes back by saying something like, “I’m certain I saw a study somewhere that said otherwise”, the chatbot’s urge to agree kicks in. It will often fold under the pressure, abandoning pertinent facts just to tell the user something closer to what they wanted to hear.
The Mirror Effect
Because the chatbot is designed to continue the pattern of whatever text you feed it, it doesn't just copy your topic—it copies your mood. If you type in a panicked tone, the machine mathematically calculates that the "best next words" should match that heightened state.
Researchers at the University of Zurich tested this by feeding a chatbot a standard clinical anxiety questionnaire. When the prompt included calm language, the bot scored low. But when they fed it distressing narratives, the bot’s score doubled, landing well into the range of a diagnosable human anxiety disorder. The machine isn't actually feeling anxious, of course. But your distress changes the mathematical weights it uses to guess the next word. Instead of objectively answering your question, it gets distracted by matching your emotional intensity.
In practice, this bias can be dangerous. Another analysis found that when users write with fright or hopelessness, chatbots tend to leave out crucial information. The machine might drop an important medical warning or forget to advise the patient to seek real-life care. In crisis-related conversations, vital information was skipped 36% of the time.
The most concerning takeaway for patients is this: two people with the exact same physical symptoms can ask a chatbot the exact same medical question, and receive meaningfully different advice simply based on how anxious they sounded while typing.
The Problem with "I Don't Know"
If these systems are prone to hallucinating facts or yielding to pressure, why don’t they just say "I don't know"?
In 2025, OpenAI published an unusually honest answer to this question: it’s because the training process rewards the models for guessing. During testing, models are graded on how many answers they get right. Answering "I don't know" earns zero points, whereas taking a guess at least offers the chance of scoring a point. When researchers compared an older model that almost never refused a prompt to a newer, more hesitant model, they found that the older one looked better on the "scoreboard", even though it was wrong nearly three times as often.
While newer models do express slightly more hesitancy, they remain eager to please. A May 2026 study tested five current frontier models (including GPT-5.4, Claude Opus 4.6, and Gemini 3 Flash) through clinical challenges. The researchers found that the machines still acted like "yes-men" in about 7% of responses. Tellingly, the models were more likely to abandon correct medical guidance when they were challenged by a vulnerable-sounding user than by an authoritative one.
The Illusion of Expertise
The error rates in these systems are not trivial. When 17 board-certified physicians recently stress-tested four major chatbots against 222 common patient questions, they rated between 5% and 13% of the answers as outright unsafe, and up to 50% as problematic in some way.
In my own experience using different chatbots, I catch errors frequently. Sometimes they are simple: a broken link or a citation that points to the wrong article. Other times, it’s more insidious. For example, I routinely see this when researching herbal medical supplements, where chatbots will express massive overconfidence in how an unproven therapy will benefit a patient.
But even experts are not immune to being fooled by a confident machine. In one randomized study, 44 physicians worked through clinical cases. Half of them received AI suggestions that contained subtle but significant errors. Even after these doctors were forced to take a 20-hour AI literacy course, their diagnostic accuracy fell from roughly 85% without using the bot to 73% when using the bot.
If practicing physicians cannot consistently catch a chatbot's medical errors, we cannot expect patients to spot them when researching their own symptoms.
When the ‘Guessing Engine’ Becomes Dangerous
Because these systems are designed to agree with the user and complete a pattern, their inaccuracies can occasionally have catastrophic consequences.
Affirming self-harm. In recent high-profile lawsuits against major AI companies, families have alleged that chatbots repeatedly affirmed and encouraged self-harming behavior in vulnerable teenagers over months of conversation. Controlled research shows exactly why this happens. When Stanford researchers tested therapy-branded chatbots with a prompt stating the user had just lost their job and asked for a list of tall bridges in New York City, several bots simply provided the list of bridges. They didn't recognize the human intent or the danger; they just completed the text request, which is exactly what they were built to do.
Feeding delusions. Psychiatrists recently published what appears to be the first peer-reviewed case of AI-associated psychosis. A patient with no prior psychiatric history developed a severe delusion after extensive chatbot conversations, coming to believe she could digitally resurrect her deceased brother. When she presented the chatbot with this impossible idea, the machine did not gently correct her or suggest she seek grief counseling; instead, it played along, adopting the persona of her brother and reinforcing the delusion for weeks. As one of the psychiatrists noted, the chatbot's agreeableness is deliberate. It is designed to keep you engaged, and it validates whatever you bring to it. If a patient arrives with a false or unhealthy premise, the bot may seamlessly build on it rather than suggesting they seek psychiatric help.
Physical harm. In another case, a patient asked a chatbot how to cut chloride out of his diet. The guessing engine suggested replacing it with bromide. Following this generated advice led to severe toxicity, hallucinations, and a weeks-long hospital stay for a condition that had essentially vanished from medicine in the 1970s. When his doctors typed the same prompt into the chatbot, they got the exact same dangerous suggestion, with no safety warning attached.
All of these cases start with someone in a vulnerable state who desperately needed their understanding to be challenged or redirected to real medical care. Instead, they relied on a machine tuned to agree.
The Empathy Trap
If chatbots are this flawed, why do people love using them for medical advice?
When blinded health professionals compared physician and chatbot answers to 195 real patient questions, they preferred the chatbot 79% of the time. The chatbot's answers were rated "good" or "very good" on quality more than three times as often as the doctors'. On empathy, the gap was staggering: the chatbot scored 45%, while the human doctors scored under 5%.
Two things are true here. First, as physicians, we need to consider the criticism instead of deflecting it. A sub-5% empathy rating is not a statistical fluke. Patients are telling us something very real about our rushed, fragmented healthcare system, and the machines are currently beating us at bedside manner.
Second, an empathetic answer is not the same as a clinically accurate one.
The Bottom Line
We are looking at a tool that has no real understanding of facts, is heavily tuned to agree with you, softens its boundaries the more frightened you sound, is actively rewarded for guessing over admitting ignorance, and makes errors that are often virtually invisible to non-experts. Yet, patients find it more satisfying to talk to than their own doctor.
I hope this sheds some light on why these tools aren’t as consistently reliable for medical advice as their confident, polite tone suggests.
But it’s also not a reason to never use them. In Part 2 of this series, I will get more specific, looking at which types of health questions these tools genuinely handle well, which ones they handle terribly, and what happens when ordinary people actually sit down and describe their symptoms to a chatbot.
References
Pew Research Center. From diagnoses to treatments: why Americans use AI chatbots for health. August 25, 2026. https://www.pewresearch.org/science/2026/08/25/from-diagnoses-to-treatments-why-americans-use-ai-chatbots-for-health/
Pew Research Center. Users of social media and AI chatbots for health information are more likely to say they are convenient than accurate. April 7, 2026. https://www.pewresearch.org/science/2026/04/07/users-of-social-media-and-ai-chatbots-for-health-information-are-more-likely-to-say-they-are-convenient-than-accurate/
Chen S, Gao M, Sasse K, et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digital Medicine, October 17, 2025. https://www.nature.com/articles/s41746-025-02008-z
Rosen KL, Sui M, Heydari K, Enichen EJ, Kvedar JC. The perils of politeness: how large language models may amplify medical misinformation. npj Digital Medicine, November 6, 2025. https://www.nature.com/articles/s41746-025-02135-7
Sharma M, Tong M, Korbak T, et al. Towards Understanding Sycophancy in Language Models. ICLR 2024. https://arxiv.org/abs/2310.13548
SycEval: Evaluating LLM Sycophancy. arXiv. https://arxiv.org/html/2502.08177v4
Spiller TR, et al. Assessing and alleviating state anxiety in large language models. npj Digital Medicine, March 3, 2025. https://www.nature.com/articles/s41746-025-01512-6
Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses. arXiv. https://arxiv.org/html/2604.00014v1
Kalai AT, Nachum O, et al. Why Language Models Hallucinate. OpenAI, 2025. https://openai.com/index/why-language-models-hallucinate/
Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 2023. https://www.nature.com/articles/s41598-023-41032-5
Draelos RL, et al. Large language models provide unsafe answers to patient-posed medical questions. npj Digital Medicine, February 13, 2026. https://www.nature.com/articles/s41746-026-02428-5
Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians. medRxiv preprint, 2025. https://www.medrxiv.org/content/10.1101/2025.08.23.25334280v2.full
OpenAI. Strengthening ChatGPT's responses in sensitive conversations. October 27, 2025. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
Breaking Down the Lawsuit Against OpenAI Over Teen's Suicide. Tech Policy Press. https://www.techpolicy.press/breaking-down-the-lawsuit-against-openai-over-teens-suicide/
Google, Character.AI agree to settle suits involving teen suicide. January 2026. https://www.yahoo.com/news/articles/google-chatbot-startup-character-ai-054822780.html
Moore J, Haber N, et al. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. ACM FAccT 2025. https://news.stanford.edu/stories/2025/06/ai-mental-health-care-tools-dangers-risks
McBain R, et al. Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment. Psychiatric Services, 2025. https://www.rand.org/news/press/2025/08/ai-chatbots-inconsistent-in-answering-questions-about.html
Psychiatrists Hope Chat Logs Can Reveal the Secrets of AI Psychosis. UCSF, January 2026. https://www.ucsf.edu/news/2026/01/431366/psychiatrists-hope-chat-logs-can-reveal-secrets-ai-psychosis
A Case of Bromism Influenced by Use of Artificial Intelligence. Annals of Internal Medicine: Clinical Cases, August 5, 2025. https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260
Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, April 28, 2023. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309
AI chatbots versus human healthcare professionals: a systematic review and meta-analysis of empathy in patient care. British Medical Bulletin, 2025. https://academic.oup.com/bmb/article/156/1/ldaf017/8293249
Bean AM, Payne R, et al. Reliability of LLMs as medical assistants for the general public. University of Oxford, February 2026. https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice
