AI catches aneurysms hiding in brain scans

On 55 brain scans read during routine clinical care, an aneurysm was present but absent from the radiologist’s report. An AI system running silently in the background picked it up.

Had its findings been added to the reports, the number of aneurysms detected would have risen by 39% compared with the radiologists’ results alone. Yet the machine also missed 30 aneurysms that the radiologists found, making the study a powerful case for combining their abilities.

Researchers at Northwell Health, a large American healthcare network, tested an algorithm developed by Israeli medical imaging company Aidoc on 3,856 brain CT angiography examinations. These scans use X-rays and an injected contrast agent to create detailed images of blood vessels, allowing doctors to look for an intracranial aneurysm: a weakened section of an artery wall that has bulged outwards.

The study was published this month in the Journal of the American College of Radiology. It adds an unusually practical piece of evidence to the debate over AI in medicine because the algorithm examined scans from everyday hospital work, instead of a carefully assembled set of test images. (pubmed.ncbi.nlm.nih.gov)

An aneurysm can remain hidden for years, with the patient experiencing no symptoms. If it ruptures, it can cause bleeding around the brain, with potentially devastating consequences. Finding it early gives doctors an opportunity to assess the danger, monitor it and decide if treatment is needed.

“Many of the additional aneurysms identified by AI in our study were among the smallest lesions,” said senior author Pina Sanelli, professor of radiology and vice-chair of research at the Zucker School of Medicine at Hofstra/Northwell. “Early detection of small aneurysms provides an opportunity for risk assessment, surveillance, and, when appropriate, treatment before a life-threatening haemorrhage occurs.”

The researchers used what they call “shadow mode”. Aidoc’s US Food and Drug Administration-cleared algorithm processed each examination alongside the normal hospital system, while radiologists read the scans as they usually would. Its alerts remained hidden from them and had no influence on patient care during the study.

When the AI and radiologist differed, independent specialist neuroradiologists reviewed the images to decide which finding was correct. This allowed the researchers to measure what each could add to the other without changing the way the original scans were handled.

Radiologists and the software agreed in more than 96% of cases. The disagreements revealed their different strengths.

The AI detected 84.6% of aneurysms that were present, compared with 71.8% for the radiologists. In medical testing, this measure is called sensitivity. The radiologists performed better when the question was how often an aneurysm alert proved correct: 92.7% of their positive findings were accurate, compared with 78.2% for the algorithm.

The software produced 101 findings that had been absent from the original reports. Expert review confirmed 55 as real aneurysms and rejected 46 as false alarms. In the other direction, radiologists found 30 aneurysms missed by the AI.

This turns the familiar contest between doctor and machine into a poor way of reading the result. The algorithm cast a wider net and picked up more of the smallest lesions, while the radiologists made more reliable positive calls and caught cases the software overlooked. Combining their findings produced the strongest result.

“The algorithm found additional true-positive aneurysms that enhanced overall detection performance, while radiologists identified important aneurysms that the algorithm missed,” said lead author Shlomit Goldberg-Stein, professor of radiology at the Zucker School of Medicine and director of artificial intelligence in Northwell Health’s radiology department. “Together, they achieved better results than either could alone.”

The location of the patient within the health system made a striking difference. Among inpatients, the AI found 18 additional aneurysms and generated seven false-positive alerts. It also performed favourably in the emergency department. In outpatient scans, it added only four detections and produced more false positives than confirmed findings.

Matthew Barish, chief medical information officer of clinical shared services at Northwell Health, said the difference may reflect the more complex examinations encountered in wards and emergency units. Those settings create more opportunities for the software to contribute alongside a radiologist.

That variation may prove as useful to hospitals as the headline detection figure. FDA clearance establishes that a medical device has met the regulator’s requirements for its intended use. A hospital still needs to learn how it performs with its own patients, scanners, staff and mix of clinical cases.

Northwell’s shadow test offers a way to do that before AI alerts enter the reporting process. It can reveal where the software adds enough confirmed findings to justify the extra cases a radiologist must review, and where the false alarms may create more work than value.

The study covered one healthcare system and one commercial algorithm, so its percentages cannot be transferred automatically to every hospital. Its more durable finding lies in the pairing: the software found real aneurysms that people missed, and the people found real aneurysms that escaped the software.

“This study shows that AI can deliver meaningful clinical value by helping radiologists find additional aneurysms while also revealing important differences in performance across care settings,” said co-author Elizabeth Rula, executive director of the Harvey L. Neiman Health Policy Institute. “These findings underscore the importance of ongoing monitoring and evaluation after implementation.”