Credit: Michigan Medicine/Synthetaic
Michigan Medicine was working toward a goal: to develop a decision support tool that would improve pathologists’ ability to accurately diagnose brain tumors in the operating room, allowing them to both diagnose more quickly and more accurately.
THE PROBLEM
However, this is no small feat. Historically, tumor diagnosis is a challenging task because the images of the brain that pathologists are reading while in surgery are complex, and the time available to read them is limited.
“We wanted to develop a computer vision model that could identify regions that are likely to be diagnostic and provide a tentative diagnosis for the pathologist to consider when making their final interpretation,” said Dr. Todd Hollon, neurosurgeon and principal investigator of the machine learning in neurosurgery laboratory at Michigan Medicine.
“By improving across both of these vectors, we believed we could accelerate diagnosis and improve accuracy in a way that still allowed pathologists to be part of the process and make the final diagnostic call.”
To develop the first iterations of their computer vision model, staff were using the latest deep neural network models and had collected a pathology data set from the University of Michigan for model training.
“However, we were limited to using data that we had collected just from our own medical center,” Hollon noted. “We would take slides that had been warehoused and digitize them as a training data set to train the network. All in all, that resulted in about 300 patients that we were able to pull with five different brain tumor diagnoses represented. We are fortunate in that we have performance computing resources at Michigan, but the process end-to-end would still take about a week.”
Hollon and his team then aimed to validate the model using a multi-institutional data set to ensure good performance across multiple medical centers. However, they noted an unexpected drop in accuracy when testing their model on images from other medical centers.
“For example, after we tested our model on new data from another medical center from Ohio State, accuracy dropped to 50%,” he recalled.
“We were able to correctly diagnose 90% of the most challenging brain tumor class, primary central nervous system lymphomas, compared to only 70% without Synthetaic’s methods.”
Dr. Todd Hollon, Michigan Medicine
“However, we didn’t have a fantastic explanation for why the model didn’t perform as well, other than that the processing of the images was different – the slides were stained differently and scanned using a different type of slide scanner. And even with this working theory on why the accuracy dropped, we weren’t able to address the issue through the existing data or models alone.”
Still, the team knew they needed to address this accuracy drop before moving forward because they wanted the technology to be applicable to any medical center. They knew they needed a different strategy for model training, and that’s when they turned to synthetic data vendor Synthetaic.
PROPOSAL
Synthetaic was able to provide Michigan Medicine with a technique that improved how well its models performed on new, unseen images from its own and other medical centers. The big part of Synthetaic’s technique lies in the creation of synthetic data. The company’s expertise lies in closing the statistical gaps in AI training by generating high-quality, high-fidelity training data.
“By using Synthetaic’s synthetic data, which was generated from very large pathology data sets, our model is now better able to learn what to look for in our pathology images,” Hollon explained. “Putting it simply, by studying more images, the model was able to get ‘smarter’ and therefore improve its diagnostic accuracy.”
To get more technical, the model was able to improve because synthetic data allowed the team to augment the amount of data available for model training. In particular, the team needed more data for specific brain tumor types that are uncommon or that were getting disproportionately high diagnostic error rates. Synthetaic was able to generate synthetic images around these two use-cases to solve this issue.
“The problem of too little data is a common and major challenge with training computer vision models for clinical decision support,” Hollon said. “However, we saw firsthand how synthetic data can help alleviate this problem by creating more data. Fortunately, the result is improved model training and diagnostic accuracy.”
MEETING THE CHALLENGE
Using the methods developed by Synthetaic, the Michigan Medicine team drastically exceeded its previous diagnostic accuracy on both its own data set and pathology data from other medical centers. Most important, the team was able to correctly classify challenging tumors that were incorrectly classified by clinical pathologists at the time of surgery.
“These results demonstrate how our intraoperative decision support tool could assist surgeons and pathologists to interpret challenging brain tumor specimens,” Hollon noted. “Synthetaic also developed an AI dashboard that allowed for real-time image interpretation with full decision support integration.
“This of course isn’t being used clinically yet, as you need FDA approval for decision support tools, even if something is wholly validated,” he continued. “So right now, the primary users are neuropathologists. Surgeons likewise benefit because they’re incorporating this data into their decisions in the operating room.”
It’s also worth noting that the same intraoperative pathology workflow applies across all disciplines – everything from neuro-oncology to gyno-oncology.
“There’s nothing special about neuro except we’re the first people to get this up and running, so it’s reasonable to expect that more labs across the country will leverage synthetic data-assisted pathology support tools over the coming years,” Hollon predicted.
At Michigan Medicine specifically, the team sees major value across both diagnostic speed and accuracy with the solution.
“Those are orthogonal values to a technology like this, and you have to be careful,” Hollon cautioned. “For example, it’s possible to develop a model that is attempting to increase the speed at which you achieve a diagnosis, but that as you can imagine as speed increases, there is a proportional decrease in accuracy.
“That’s why we were very careful to measure both metrics in tandem,” he added. “We knew we had to achieve improvements across both speed and accuracy to make this a valuable tool. Now that we’ve solved the engineering challenge of speed and the diagnostic challenge of accuracy, we’re focused on how we can implement this solution in real cases as a clinical tool. We are specifically focused on tumors, where what is diagnosed in the OR really influences what the surgery is going to look like.”
RESULTS
Hollon and his collaborator, Siri Khalsa, both were skeptical about how well synthetic data would work. That skepticism stemmed from the idea that one could only get so much out of the data.
“We can train new models, but the data doesn’t change, right?” he noted. “So I was pleasantly surprised with how well the synthetic data boosted the brain tumor diagnoses. It’s a testament to Synthetaic and the use of generative adversarial networks in a constructive way to improve computer-aided diagnostic systems.
“Our new model was able to achieve 96% accuracy across the major brain tumor types included in our study,” he continued. “This was a massive jump in performance compared to 68% accuracy without the use of synthetic data. Moreover, we were able to correctly diagnose 90% of the most challenging brain tumor class, primary central nervous system lymphomas, compared to only 70% without Synthetaic’s methods.”
These results include correctly classifying five out of six lymphomas that were misclassified by board-certified neuropathologists at the time of surgery. These results demonstrate the synergistic effect decision support tools and computer-aided diagnostics can have on improving patient care, Hollon concluded.
Twitter: @SiwickiHealthIT
Email the writer: [email protected]
Healthcare IT News is a HIMSS Media publication.
Source: Read Full Article