JAIC, in partnership with DIU, Delivering AI-Enabled Cancer Diagnostics at the Point of Care

The U.S. military has a long history of being the greatest fighting force in the world, but it’s also been quite adept at advancing medicine in ways that have improved battlefield survivability and the health of all people. Among these breakthroughs are mass vaccinations, blood transfusions, blood banking, and the widespread use of electronic health records.

Now, the Department of Defense is set to take the lead on another critical medical innovation: leveraging artificial intelligence capabilities to more quickly and accurately diagnose cancer.

This month, the Joint Artificial Intelligence Center’s Warfighter Health Mission Initiative, in partnership with the Defense Innovation Unit (DIU), deployed the DoD’s first-ever augmented reality microscope (ARM) loaded with AI algorithms that can detect metastatic breast cancer cells on digital images. This first device will be used by pathologists at Brooke Army Medical Center (BAMC) in San Antonio, Tex., and additional microscopes will be deployed to military treatment facilities (MTFs) all over the world.

This project, known as Predictive Health, will develop AI applications that improve readiness and save lives.

“This is not an abstract thing,” stated U.S. Navy Capt. (Dr.) Hassan Tetteh, Warfighter Health Mission Chief. “What we are doing with these augmented reality microscopes is delivering AI capability at the point of care. We’ve leveraged computer vision that’s built into the device that can examine the samples and identify the abnormal cancer cells by drawing rings around them even before the pathologist looks at it.”

The AI capability within the microscopes is critical to not just identifying cancer but also prioritizing cases, optimizing workflow, and decreasing the amount of time it takes to get a patient definitively diagnosed and into treatment.

Maj. Scott McKeithen, Warfighter Health Mission Manager, likens the future of this particular AI capability to having a senior medical faculty on staff—only it’s always available and never sleep-deprived.

“When the pathologist arrives for their shift, they will already have, as a result of the AI capabilities, a set of cases that have been identified as positive for cancer,” he explained. “So instead of working off of a queue in no particular order, the pathologist immediately knows which cases are most urgent and which are more likely routine. That step alone is going to significantly decrease the time to diagnosis, which is critical to increasing a cancer patient’s chances for a positive outcome.”

Kicking Into High Gear

The Warfighter Health Mission Initiative (MI) was created by the JAIC in September 2019 to deliver AI capabilities to the Military Health System (MHS), the military services, and other DoD components that can enable improvements to combat medicine, human performance, and warfighter readiness at scale. Since then, the team has expanded and kicked off nearly 30 AI initiatives within all three mission areas.

The Predictive Health project is one of its most advanced product lines. It is a multi-agency, multi-disciplinary effort that started at Navy Medical Center San Diego (NMSD), where a team digitized thousands of pathology samples and labeled them. The Warfighter Health MI has since taken those images and used them to train AI models and develop AI algorithms that can recognize and identify specific types of cancer cells. Through a competitive selection process, the DIU worked with the JAIC to award contracts to build a prototype of an augmented reality microscope that could overlay that AI capability onto current case samples.

The key to this effort is the massive amount of historical medical data held by the DoD. The Joint Pathology Center (JPC) under the Defense Health Agency (DHA), for example, houses more than 55 million pathology slides representing 7.4 million specimens at its location in Silver Spring, Maryland. The DHA is now moving ahead to digitize and label a subset of the samples in the JPC repository. As more pathology samples are digitized, the Warfighter Health team will work with partners and stakeholders to build additional AI models and train algorithms that can detect other types of cancer and diseases. Soon, a similar AI capability will also be extended to radiology images. There are millions of digitized radiographic images currently archived.

At present, the JAIC and DIU are collaborating to house data, but the Warfighter Health MI team expects to transition to the JAIC’s Joint Common Foundation (JCF) AI development platform (as it matures) or “at least create an application programming interface (API) within that will utilize specific digital data cards to continue to advance the algorithms/technology of pathology,” Maj. McKeithen said.

This collaboration represents an exciting opportunity to accelerate AI-enabled medical innovation. Combining commercial technology with the DoD’s unique and large data sets can significantly advance AI capabilities for healthcare delivery. DoD health data is not only one of the largest datasets, it is also among the most unique because it contains extensive rare edge cases and lengthy time series.

Cmdr. (Dr.) Niels Olson, who now serves as the Chief Medical Officer at DIU, began laying the groundwork for this while a medical resident at NMCSD. “I’ve been working on this project for nine years and we would not have been able to move the ball this far without the JAIC’s significant support,” said Olson. “Had all these efforts remained siloed we would not be at this pivotal inflection point where we are bringing the best of DoD and the best of commercial industry together to partner over a truly shared value: saving lives.”

Getting AI to the Frontlines

Another challenge is increasing the scope and scale of deployment so the team can deliver algorithms and augmented reality microscopes out to the field and other remote locations. This is critical to more effectively care for deployed troops. Presently, frontline clinics and even mid-sized medical facilities don’t always have pathologists or even radiologists on staff, which can delay diagnosis and treatment.

As a surgeon stationed on a U.S. Navy aircraft carrier, for example, Tetteh frequently cared for patients who developed suspicious skin lesions as a result of spending so much time on deck in the harsh sun. He took biopsies for the purpose of diagnosis, but he would have to wait weeks and sometimes months until the ship reached port before he could transmit those samples to a pathologist at a large-scale medical center. Then, he would wait for the results.

“It was not uncommon for some of those lesions to come back as early cancers,” Tetteh recalled, noting that in some cases, by the time he got the diagnosis, the sailors would have already transferred to a new duty station and he would have to track them down. “You can imagine how that kind of time delay can impact the ability to treat a cancer case optimally.”

With the new microscopes and the availability of new AI algorithms, medical staff deployed on ships or battlefield locations may be able to one day take an image and upload it for analysis on a cloud-based server, or even do the analysis on-site, with AI algorithms that can examine the entire slide exhaustively for the presence of cancer cells and then report a probability of whether cancer has been detected, what type of cancer, and the grade.

This information would give the deployed staff the confidence they need to begin the process of developing a care plan for that patient.

“You can see how that gives us great utility in the field, and that’s just one use case,” said Tetteh. “There’s a lot of capability and a lot of potential use and application for how AI can be used to advance our diagnostics.”

In fact, the potential is so great that Tetteh believes that the Predictive Health project, with the volume, variety, veracity, and value of pathology data collected over 100 years, is akin in scope and scale to the Human Genome Project, which sequenced and mapped human DNA. The knowledge and the AI capabilities being developed through the data digitization and AI algorithm training processes will eventually be tied in with the JAIC’s medical records analysis project and other initiatives. And that will enable the impact to go well beyond diagnostics and eventually lead to more effective treatments and possibly even cures for certain cancers.

“When you think about being able to digitize all of those various samples and then think of the potential insights that can be gleaned and harnessed from all of that information, you really now have a cornucopia of opportunity for the research community, for the drug development community, and for so many benefits to be realized from this resource that we’re curating, nurturing, and making available,” Tetteh stated. “It’s very, very exciting.”