Ready, set, go – or no go? The complex challenges of assessing AI
Healthcare organizations are wrestling with a variety of fluctuating factors in determining whether AI is ready to use.
The acceptance of artificial intelligence for healthcare applications isn’t a given. It’s a challenge with multiple dimensions.
One of the major considerations considered is the maturity of the tools, matched against the application for which they’re envisioned. This has been a basic consideration for all forms of emerging technologies applied to healthcare, whether it’s EHRs or electronic prescribing. The capacity of technology often lags behind the vision for how to use it, waiting for Moore’s Law to improve the odds and match reality to the vision.
Then, another challenge is matching the readiness of the technology, as constrained by current computational limitations, to an appropriate use case. In healthcare, that may mean targeting current iterations of AI at low-hanging fruit that both are well within the technology’s capabilities and meet a vexing need. With AI, that may mean applying it to claims processing, agentic note-taking or administrative tasks, before plunging it directly into patient care.
Finally, a key piece is ensuring that users have faith in what AI can safely and perfectly do. Without this critical buy-in – based on experience, scientific experimentation and testing, and users’ first-hand exposure and confidence – AI will have a long road ahead. The risk is high as Gartner predicts that 40 percent of agentic AI projects will be cancelled, primarily because of non-technology reasons.
With at least these three components, all fluctuating rapidly in real-time, healthcare organizations are looking for the best uses of AI technologies, both for immediate implementation. They’re concurrently researching what’s possible now, as well as what isn’t ready now, but may be ready in the near future.
The investigative phase
Some recent studies and use cases show both the attraction of applications that work, as well as concern about potential pitfalls and unmitigated risks for too-early adoption.
In one use case where AI seems ready to provide quantifiable benefits, the Mount Sinai Health System is planning to collaborate with Midstream Health, an AI-powered financial action platform, to use real-time intelligence to aid financial operations and improve workflow efficiency.
Mount Sinai says the goal is to simultaneously achieve cost savings while ensuring better services and care for patients. The use of AI is expected to help the health system improve billing, supply chain management, contract management and more.
Midstream is tasked with pulling together fragmented, complex financial and contract data into a single source to create custom-built datasets to fill critical information gaps, using the information to prioritize cost-saving and margin opportunities. Mount Sinai will use the platform to more efficiently and quickly model outcomes, source documents, and continuously monitor and analyze operations strategies.
In the realm of the possible, studies are suggesting that AI can play a role in enhancing clinical conversations through diving deeper into patient-clinician discussions.
That was part of the rationale behind Google’s decision to develop the Articulate Medical Intelligence Explorer (AMIE), a research AI system based on a large language model and optimized for diagnostic reasoning and conversations. The multi-phase effort aimed to investigate the potential for conversational diagnostic AI, seeking to test it in a pilot to assess diagnostic conversations in areas such as history-taking, diagnostic accuracy, clinical management, clinical communication skills, relationship fostering and empathy.
This seminal work strongly suggests that “AI-driven clinical conversation is emerging as a foundational layer of healthcare delivery,” according to conclusions from MakeWell, a Santa Barbara, Calif.-based technology provider of a clinical communication platform. The company says its technology facilitates structured conversations that capture information in clinically meaningful contexts.
"Healthcare has spent decades digitizing records, but not the conversation that produces the most important clinical data," said Daniel Carroll, founder and CTO at MakeWell. "The AMIE research confirms what we've believed -- the next frontier of healthcare AI is not just analysis of records — it's intelligent dialogue with patients that produces better data in the first place."
Other niche solutions hold promise. For example, Experity Health, a Machesney Park, Ill.-based technology provider, is developing an AI-powered platform that can be a digital “front door” for patients, intended to provide seamless experiences and better care coordination. It’s touting a “Clinic of Tomorrow,” which it says shows how AI-powered documentation, automated workflows and integrated prescription fulfillment through Amazon Pharmacy “could transform urgent care operations.”
When clinicians are assured of benefits from AI, they are more likely to adopt tools. For example, data analyzed by Trilliant Health indicates that hospitals that have adopted AI scribing tools are billing a much higher share of patient visits at the highest-intensity evaluation and management codes. “With technology enablement, AI can now capture clinical encounters more thoroughly and accurately than humans could,” the company contends.
But AI readiness isn’t universal
For all the hype, much work needs to be done when it comes to complex care, and clinicians will need to be convinced of its accuracy and efficacy.
For example, a new peer-reviewed study published in the March edition of PLOS Digital Health evaluates how physicians assess AI-generated clinical content in everyday practice. The study’s authors suggest that large language models “frequently miss critical clinical nuances when addressing complex medical queries, sometimes sounding convincing while providing incomplete, misaligned or irrelevant evidence.”
Authors conclude that “the results emphasize the need for a new approach in clinical AI development centered on transparency, verifiable literature and human oversight. The study was conducted by researchers at Soroka University Medical Center in Be'er Sheva, Israel, in collaboration with the clinical team at MedINT.
And concerns are growing as some providers increasingly trust AI to play pivotal roles in healthcare. For example, the Veterans Administration is rolling out AI tools, but some experts are sounding alarms over the potential impact on clinician oversight and patient safety.
Benjamin Krause, a veterans’ rights attorney, is voicing concern about whether AI errors could directly impact vets, where oversight gaps can have serious consequences and how serious issues could arise from overt dependence on the tools.
These and other examples show healthcare organizations are dealing with a sliding scale of capabilities that require careful assessment of many relevant factors that are crucial to success.
Fred Bazzoli is the Editor in Chief of Health Data Management.