A research letter published in the Journal of the American Medical Association Network Open this week examined the number of alerts generated by the Epic Sepsis Model at 24 hospitals before and during the COVID-19 pandemic.
Researchers from the University of Michigan Medical School and Washington University School of Medicine found that the total alerts per day increased by 43% in the three weeks before and after the first case of COVID-19 in each health system.
“Although the increase in the proportion of patients generating sepsis alerts in this study can be explained by the cancellation of elective surgeries and a higher average acuity among the remaining hospitalized patients, the 43% increase in total alerts illustrates the increased alerting burden imposed by COVID-19 on a sepsis model,” they wrote.
Epic did not respond to requests for comment by press time.
WHY IT MATTERS
As noted by the researchers, concerns have been raised about the potential for sepsis models to cause alert fatigue, leading to the University of Michigan to pause Epic-generated alerts in April 2020.
“This increase in alerting could have resulted from dataset shift, a phenomenon in which model performance deteriorates as a result of changes in the case mix,” said researchers. “However, even accurate alerts can be disruptive in the presence of resource constraints.”
The team calculated Epic Sepsis Model scores from 24 hospitals across four different health systems between November 3, 2019 and April 25, 2020.
In the three weeks before and after the first case of COVID-19 in each of the studied health systems, the proportion of patients generating sepsis alerts per day more than doubled from 9% to 21%.
This took place even as the total hospital census declined by 35%.
“Larger hospitals generally experienced an increase in the proportion of patients generating sepsis alerts, whereas the change in the alerting proportion was more heterogeneous across smaller hospitals,” wrote the researchers in their letter.
They noted that they did not evaluate the model’s accuracy for the study.
“However, even if the alerts were accurate, many existing sepsis workflows are built around bacterial sepsis and thus may not be entirely appropriate in the context of COVID-19,” they observed.
THE LARGER TREND
Epic’s sepsis model has been the source of controversy this year, with a retrospective study – led by the same author as the research letter in JAMA Network Open – in JAMA Internal Medicine suggesting that it lacked predictive power.
A different study, however, found that the early warning system led to faster antibiotic administration and better patient outcomes without an increase in harmful clinical interventions.
“Sepsis is a hard problem to solve, and it’s one the industry has been working for many years to address,” said Epic’s director of nursing, Emily Barey in an interview with Healthcare IT News about that study. “As a nurse at the bedside, sepsis can sneak up on you because it shows up in patients in different ways.”
ON THE RECORD
“Being able to rapidly assess and disable AI-based alerts is a responsibility faced by health systems using AI to support clinical care,” wrote the Michigan researchers in JAMA Network Open.
“Given the susceptibility of AI-based systems to changes in alerting patterns, clinical AI governance within health systems may play a role in monitoring and supporting deployed AI systems,” they said.
Kat Jercich is senior editor of Healthcare IT News.
Twitter: @kjercich
Email: [email protected]
Healthcare IT News is a HIMSS Media publication.
Source: Read Full Article