Multispectral imaging system improves image quality for tumor surgeries

Hospitals are increasingly implementing multispectral camera-based image processing systems during surgery to visualize cancers. One such example is Quest Medical Imaging BV (Middenmeer, The Netherlands;www.quest-mi.com), which developed an imaging system using a camera, a programmable frame grabber, and accompanying software from Silicon Software (Mannheim, Germany;https://silicon.software).

The Quest Spectrum platform is an image processing system for hospitals based on a proprietary multispectral camera. The area scan camera features a Camera Link interface based on a three-chip design to capture color (Bayer RGB chip) and near infrared (two monochrome chips) images at HDresolution.

Incident light filters through a spectral beam splitter into the five desired frequency spectra (red, green, blue, and two spectra in thenear infrared range) along the same optical axis. In doing so, three congruent images for various spectral ranges are generated, which can be aligned using color space conversion. The camera processes data at a rate of 255 MB/s, at a maximumof 60fps.

Silicon Software’s microEnable 5 marathon VCL frame grabber features a Camera Link interface and is programmable.

Images captured by the camera display on a monitor during surgery, enabling the surgeon to distinguish between a tumor and surrounding tissue during both invasive (open) and minimally-invasive laparoscopic surgeries. Prior to surgery, a patient is injected with a fluorescent marker at targeted points. The marker attaches to the tumor cell and NIR light is used to excite the cells. Meanwhile, the multispectral camera captures an HD color image in the RGB color space and two fluorescent images in the NIR image. One of the NIR images then combines with the color image for image output.

Live images during surgery

In the new system, a microEnable 5 marathon VCL Camera Link frame grabber from Silicon Software executes the entire process on the FPGA. The images captured by the camera are processed pixel-for-pixel with higher parallelism and two of the images are merged together for the finalimage.

In the new system, live images enable the surgeon to identify the fluoresced tumor cells and better differentiate them from healthy tissue, while also imaging lymph nodes and blood vessels—something not possible with the naked eye.

A multispectral camera is used for high-definition image acquisition.

Instead of switching back and forth during surgery from black-and-white to color images, the surgeon chooses between two camera operating modes. One option is viewing the color and fluorescent images side by side, and the other is viewing as a combined image. In doing so, the surgeon can better identify small tumors and control them with surgicaldevices.

FPGA algorithms with no hardware programming

One additional reason Quest Medical Imaging chose the frame grabber is its programmability via its VisualApplets graphical programming environment. With this software, appropriate image processing algorithms are generated as data flow diagrams, simulated, and implemented in very little time with no cumbersome or expensive VHDL programming by hardwarespecialists.

Following a short training period, the team graphically programmed a hardware applet and loaded it onto the frame grabber’s FPGA. The applet contained all the algorithms for image preprocessing, such as shading correction, noise reduction, Debayering color reconstruction, color correction, color space conversion, and artifact reduction in RGB and NIR images. It also uses tone mapping for image output to represent three contrast images and regulates the combination of two synchronized color and NIR images into one resulting image with a greenoverlay.

The Quest Spectrum Platform provides real-time result images.

Image preprocessing provides the benefit of data reduction, even before transfer to the evaluation computer, so that no bandwidth or storage space problems arise. With the FPGA, cumbersome computations such as color conversions are accelerated. Preprocessing parallel to image acquisition enables a high frame rate, spares the host computer processing, and generally allows for a more cost-effective systemconfiguration.

Quest Medical Imaging’s project team switched over to the new system within two months, and without any significant manufacturersupport.

“The hard- and software can be implemented for any markers and thus for any type of tumor. It merely requires slight modification for any tumor type, for example, in relation to the required color spectra,” says Project Director Sander de Jonge, Engineering Manager at Quest Medical Imaging. “Our requirements were all implemented, especially the real-time processing with the most minimal latencies, relieving the CPU, and acquiring better image quality for the surgeon. The time to market has been markedly reduced by virtue of the short implementationtime.”

With an eye toward potential future requirements, the entire system can be easily modified. This includes the addition of cameras, camera interfaces, or expanded software functionalities.