Scaling AI in healthcare: Consultants vs. agents
As more AI-powered tools come online, organizations need to be pre-emptive in setting goals and establishing guardrails.
Everyone is posting about the new wave of artificial intelligence in healthcare.
First, it was ChatGPT Health. Then came Anthropic, announcing Claude for Healthcare, a sleek, HIPAA-ready "consultant" designed to synthesize medical records and guidelines. Then, Google dropped MedGemma, handing developers some open-source "brains" to build their own clinical tools. And finally, Amazon Health AI entered the chat with an agentic AI that doesn’t just talk, it does.
The market has spent the last two years debating whether AI will transform healthcare or simply exhaust it. But here’s my take, building on what I shared in my LinkedIn post – the real issue isn’t "AI in healthcare." The real risk is scaling ungoverned confidence.
As we look at these three distinct approaches, the Consultant, the Open Brain, and the Assistant, we must decide which kind of confidence we are willing to scale.
The good: Why the ‘assistant’ model wins
There are positives here, and I want to be specific about where the value lies. This is exactly how consumer tech gains traction in healthcare – by offering something people want, right now, in a form that feels easy.
But not all "ease" is created equal.
My preference, and where I see the safest path forward, lies with the executive assistant model, like the one Amazon One Medical just announced. Here is an AI that focuses on agency over advice. It books appointments. It renews prescriptions. It routes you to a human. It manages the logistics of care rather than trying to simulate the judgment of care.
This matters because the "confidence" of an assistant is verifiable. Did it book the appointment? Yes or No. Did it renew the script? Yes or No. The failure modes are obvious and logistical, not hidden and clinical.
The bad: The ‘consultant’ trap and reassurance bias
Contrast the "assistant" with the "confident consultant" model we see in general-purpose LLMs like ChatGPT or even the more specialized Anthropic implementations.
The risk here is subtle and dangerous. When you ask a "consultant" AI a health question, it is programmed to be helpful. It is programmed to be polite. And often, it is prone to reassurance bias.
Reassurance bias is when the model, in an effort to be "safe" and "aligned," minimizes symptoms or offers comforting platitudes that de-escalate a patient when they actually need urgency. It says, "That sounds like meaningful exhaustion," when the clinical reality might be, "That sounds like heart failure."
When a system appears "medical" because it’s connected to your records (as Anthropic and OpenAI are pushing for), people instinctively raise the credibility score in their heads. They treat the output as truth. Partial data + confident AI + reassurance bias = false safety.
False safety is more harmful than "no answer" because it changes behavior. It delays care. It lowers urgency. One clinician who commented on my post put it plainly – the burden is going to shift instantly to clinicians, who will be forced to de-escalate patients terrified by a hallucination or, more dangerously, escalate patients who were falsely reassured that they were fine.
The ‘tech bro MVP’ doesn't translate to health
Google’s release of MedGemma is a triumph for open source; I love that we are democratizing the "brains" of these models so local builders can create solutions. But we must be careful not to confuse capability with care.
In consumer tech, people love the story of iteration – the skateboard that becomes a bicycle, that becomes a car. In healthcare, that analogy is nonsense when sensitive data and human lives are on the line.
You can’t treat governance as something you bolt on later. If you do, what you’re really saying is, “We’re willing to scale risk first, and manage consequences second.”
The "consultant" model scales risk; the "assistant" model scales utility. If I have to choose, I will take the assistant every time. I don't need a chatbot to debate my diagnosis; I need an agent to help me navigate the system to find the human who can.
So what do we do now?
This is why I said that clinicians, regulators, and health tech leaders need to co-build guardrails now. Not "soon." Not "after the next release." Now.
If healthcare leaders want to respond responsibly, we need to focus on immediate guardrails.
Clear decision boundaries. Define strictly what the tool can and cannot do, (for example, "I can book your visit," not "I can tell you why you're hurting.")
Escalation pathways. When and how users are directed to clinicians.
Accountability. Who owns the harm when the output is wrong?
We also need to be honest about what people value. One commenter summarized it well – AI on its own isn’t the differentiator anymore; it’s infrastructure. What people value is trust, judgment and human support. The biggest opportunity is to use AI (like the Amazon agent) to remove the friction so we can get to that human support faster.
The real ramifications
If you take one thing from my perspective, let it be this – the biggest risk isn’t that ChatGPT Health, Anthropic or MedGemma exist. The biggest risk is that we normalize a world where confident outputs are treated as truth.
The real ramifications are not "AI in healthcare." They are scaling ungoverned confidence.
Give me the humble assistant that does the work. Keep the confident consultant until it learns to say, "I don't know."
And if we don’t co-build these guardrails now, we’re going to spend the next phase cleaning up preventable harm, while patients keep adopting the tools anyway. That’s not innovation – that’s negligence wearing a glossy UX.
Ramy Azzam is a physician executive specializing in responsible transformation of health in people and organizations.