Where is the Boundary: Multimodal Sensor Fusion Test Bench for Tissue Boundary Delineation

TL;DR

This paper introduces a modular test bench for multimodal sensor fusion in tissue boundary detection, combining visual, acoustic, and force data to improve accuracy in surgical applications.

Contribution

A novel, scalable platform for evaluating and integrating multimodal sensors to enhance tissue boundary delineation in robot-assisted surgery.

Findings

Multimodal fusion improves tissue classification accuracy.

The system supports real-time data visualization and analysis.

Experimental results validate the effectiveness of sensor integration.

Abstract

Robot-assisted neurological surgery is receiving growing interest due to the improved dexterity, precision, and control of surgical tools, which results in better patient outcomes. However, such systems often limit surgeons' natural sensory feedback, which is crucial in identifying tissues -- particularly in oncological procedures where distinguishing between healthy and tumorous tissue is vital. While imaging and force sensing have addressed the lack of sensory feedback, limited research has explored multimodal sensing options for accurate tissue boundary delineation. We present a user-friendly, modular test bench designed to evaluate and integrate complementary multimodal sensors for tissue identification. Our proposed system first uses vision-based guidance to estimate boundary locations with visual cues, which are then refined using data acquired by contact microphones and a force sensor. Real-time data acquisition and visualization are supported via an interactive graphical interface. Experimental results demonstrate that multimodal fusion significantly improves material classification accuracy. The platform provides a scalable hardware-software solution for exploring sensor fusion in surgical applications and demonstrates the potential of multimodal approaches in real-time tissue boundary delineation.