While there are currently a dozen approved AI systems for breast cancer screening, it has been difficult to evaluate their clinical performance objectively. Now, however, there is a Swedish validation platform ready to compare how well AI systems detect signs of breast cancer—and its development has been led by researchers at Karolinska Institutet.

The pilot project started in 2021 and the platform was ready for trial last year. So far it has been used to start the evaluation of three different companies' algorithms based on about 40,000 mammograms from three Swedish regions.

"It is important to evaluate the diagnostic accuracy of the AI algorithms that might be used clinically," says project leader Fredrik Strand, researcher at the Department of Oncology-Pathology, Karolinska Institutet, and radiologist at Karolinska University Hospital. "Even if they meet the regulatory requirements, it doesn't mean they work in all contexts."

In a paper published in the Journal of Medical Imaging, the researchers describe how they have designed the platform in the hope that more teams will be able to develop similar platforms.

Enabling objective evaluation of AI systems

"By making our code freely accessible, we hope that it will be of use in implementing similar platforms for algorithm evaluation for other kinds of cancer than breast cancer," says Fredrik Strand, adding that objective evaluations of this kind have been keenly awaited. Several of the country's regions are already considering to start using AI for mammography screening.

"There's an imminent need for a system that can evaluate algorithms under locally relevant conditions," he says. "It's important that each hospital chooses the right system to suit its requirements and particular circumstances to avoid missing a breast cancer or recalling an excessive number of healthy women."

There is currently no established evaluation procedure on designated datasets to assess how accurate AI algorithms are in medical diagnostics. Manufacturers of AI systems have trained and tested their algorithms on disparate datasets, and those results are used for regulatory approval. In the platform, all algorithms process the same images and the results are compared with data on actual cancer diagnoses extracted from the National Quality Register for Breast Cancer (NKBC), revealing differences in the algorithms. Since today's AI algorithms have been trained on the specific population groups to which the developers have had access, they may contain biases that should be identified.

"The platform will be able to show an algorithm's bias, particularly when it comes to age, geographical origin and socioeconomic status of the screened individual," says Fredrik Strand.

Platform may also appeal to manufacturers

It is hoped that this means of validating algorithms will also appeal to the manufacturers, who will have an independent assessment that can help them improve their products.

"Hospitals will have to do their bit by demanding that the manufactures take part in independent tests, maybe before procurement," he says.

The development of the validation platform took place under a research project due to conclude in 2024. Before this, the project group is to have presented a proposal on how the platform can be permanently implemented for national use. Meanwhile, more regions are being invited to use the platform.

"We want more regions to take this opportunity, also for evaluating other algorithms than the three already installed," says Fredrik Strand.

Also involved in the project, alongside Karolinska Institutet and Karolinska University Hospital, are Lund University, Region Skåne, Linköping University, Region Östergötland, KTH Royal Institute of Technology, the Swedish Breast Cancer Association and Medtech4Health AB.