Malignant tumors often form metastases that spread to other parts of the body via the lymphatic system. High surgical skills are required to identify the precise location of the affected lymph nodes, enabling them to be completely removed. Fraunhofer researchers have developed a navigation aid that simplifies such interventions. 3D-ARILE is an augmented reality (AR) system that superimposes a virtual image of the exact position of the lymph nodes using data glasses.

Despite advances in medical science, it is still difficult for doctors to determine the precise anatomical location of sentinel lymph nodes during surgery and to check that the affected lymph nodes have been completely removed. 3D-ARILE, developed by researchers at the Fraunhofer Institute for Computer Graphics Research IGD, provides doctors with a navigation aid for lymph node removal. The Clinic for Dermatology at Essen University Hospital and Trivisio Prototyping GmbH collaborated with the Fraunhofer researchers on this project.

The novel augmented reality system, based on the use of data glasses, helps surgeons locate lymph nodes with the aid of virtual markers. What makes these AR glasses special is that they work in combination with powerful medical navigation software, a stereoscopic, near infrared (NIR) camera system, and indocyanine green (ICG) fluorescent dye. “To make the affected lymph nodes visible, the fluorescent dye is injected into the patient in the direct vicinity of the primary tumor.

It then spreads along the lymphatic pathways and collects in the sentinel lymph nodes,” explained Fraunhofer IGD research scientist Dr. Stefan Wesarg. The dye fluoresces when exposed to infrared light, in this case generated by infrared LEDs. NIR cameras capture the fluorescence and produce a 3D reconstruction of the affected lymph node. This virtual image showing their exact position is superimposed in real time and can be viewed by the surgeon through the data glasses. The necessary software was developed by the research team in Darmstadt. “In our case, the malignant tissue is marked in green. In this way, the surgeon can verify that every last trace has been removed,” said Wesarg.

Until now, doctors used technetium-99m as a medical tracer. By replacing this radioisotope with ICG fluorescent dye, patients can be treated without risk of radiation side-effects. Time is another essential factor. The level of radiation emitted by lymph nodes containing the radioactive marker is very low. This made it necessary to use scintillation (or gamma) cameras that required around 30 minutes to capture the images needed to determine the precise location of a lymph node. The data glasses, by contrast, display the site of the affected lymph node instantly. This greatly relieves the stress on the OR staff, who no longer need to watch an additional monitor and compare the image on the screen with that on the camera’s display. “As a result, the doctor can focus entirely on the patient and carry out surgeries without hassle,” said Wesarg.

Another advantage of the AR glasses is that they are light and extremely comfortable to wear, as dermatologists at Essen University Hospital have confirmed after carrying out numerous tests. The partners who developed the system remained in close contact throughout the project, enabling them to design a navigation aid optimized to the surgeon’s needs.

The augmented reality system comprises hardware and software components. The hardware was developed by Trivisio Prototyping GmbH. In addition to the data glasses with an integrated camera and two displays, designed specifically for medical applications, the company also developed two infrared and two visual cameras. These four optical devices are integrated in a cube-shaped unit suspended over the operating table.

The Fraunhofer IGD research team was responsible for developing the software. It includes an image processing system which detec