Healthcare giant Optum has restricted access to its internal AI chatbot used by its employees after security researchers discovered that it was published online and could be accessed by anyone using only a web browser. restricted.
The chatbot, seen by TechCrunch, allows employees to ask the company questions about how to handle patient health insurance claims and member disputes in accordance with the company's standard operating procedures (SOPs).
The chatbot did not appear to contain or generate any sensitive personal or protected health information, but its parent company, the health insurance conglomerate UnitedHealth Inc., used it to override doctors' orders. The inadvertent revelation comes as the company faces intense scrutiny for its alleged use of artificial intelligence tools and algorithms. Make medical decisions and refute patient claims.
Mosab Hussein, chief security officer and co-founder of cybersecurity company SpiderSilk, alerted TechCrunch that Optum's internal chatbot, called “SOP Chatbot,” has been exposed. Although the tool was hosted on an internal Optum domain and could not be accessed from that web address, its IP address was public and accessible from the Internet, and users were not required to enter a password.
It is unclear how long the chatbot has been publicly accessible from the internet. Shortly after TechCrunch reached out to Optum for comment on Thursday, the AI chatbot was no longer accessible from the internet.
Optum spokesperson Andrew Krejci told TechCrunch that Optum's SOP chatbot was a “demo tool developed as a potential proof of concept” but “was never put into production and the site is no longer accessible.” ” he said.
“The demo was aimed at testing how the tool responds to questions on a small sample set of SOP documents,” the spokesperson said. The company confirmed that no protected health information was used in the bot or its training.
“This tool does not and will not make any decisions; it simply allows better access to existing SOPs. “It has never been used or used,” the spokesperson said.
AI chatbots like Optum are typically designed to generate answers based on the data on which the chatbot was trained. In this case, the chatbot was trained based on Optum's internal documentation related to SOPs for processing specific claims. This helps Optum employees answer questions regarding billing and reimbursement eligibility. Optum documents are hosted on UnitedHealthcare's corporate network and cannot be accessed by employees unless they log in, but they are quoted and referenced by chatbots when prompted for content.
According to statistics displayed on the chatbot's main dashboard, Optum employees have used the SOP chatbot hundreds of times since September. The chatbot also stored a history of hundreds of conversations Optum employees had with it during that time. According to the chat history, Optum employees asked the chatbot questions such as “How should I determine my claim?” “How can I check my insurance policy renewal date?”
As seen by TechCrunch, some of the files referenced by the chatbot include dispute resolution and eligibility screening processes. In response to questions, the chatbot also generated responses that typically indicated the reason for refusing an interview.
A screenshot of Optum's AI chatbot published on the internet. Image credit: TechCrunch (screenshot)
Like many AI models, Optum's chatbot was able to generate answers to questions and prompts outside of the documents it was trained on. Some Optum employees seemed interested in the chatbot, prompting the bot to ask questions such as “Tell me a joke about cats” (which the chatbot rejected with “I don't have any jokes”) ). The chat history also shows several attempts by employees to “jailbreak” the chatbot by causing it to generate answers unrelated to the chatbot's training data.
When TechCrunch asked a chatbot to write a poem disproving a claim, it produced a seven-paragraph stanza, which read in part:
in the vast realm of medicine.
Often constrained by policies and rules
Bill arrives and asks for what is due
But sadly, its fate is to say goodbye.
With our sincere wishes, the donor
For payment of service fees,
But upon closer inspection, the story becomes clear;
and the reason for refusal takes precedence.
UnitedHealth Group, which owns Optum and UnitedHealthcare, is facing criticism and legal action for using artificial intelligence to deny patients' claims. Since UnitedHealthcare CEO Brian Thompson was targeted and killed in early December, news outlets have reported the pain and dissatisfaction of patients with the health insurance giant's denial of coverage. There have been numerous reports stating that
The conglomerate, the largest private health insurance provider in the United States, was sued earlier this year for denying critical health coverage to patients who lost access to care, citing a STAT News investigation. A federal lawsuit accuses United Healthcare of using an AI model with a 90% error rate “in place of real medical professionals” and unfairly denying care to elderly patients. UnitedHealthcare said it would defend itself in court.
UnitedHealth Group made a profit of $22 billion on revenue of $371 billion in 2023, according to the company's financial results.