Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes

TL;DR

This paper introduces a novel framework that integrates 3D acoustic localization with visual data to create dynamic, multimodal representations of surgical scenes, enhancing understanding of surgical activities.

Contribution

It is the first approach to spatial sound localization in dynamic surgical scenes, combining acoustic and visual data for richer contextual understanding.

Findings

Successfully localizes surgical acoustic events in 3D space

Accurately associates acoustic events with visual scene elements

Demonstrates robust fusion of multimodal data in realistic settings

Abstract

Purpose: Surgical scene understanding is key to advancing computer-aided and intelligent surgical systems. Current approaches predominantly rely on visual data or end-to-end learning, which limits fine-grained contextual modeling. This work aims to enhance surgical scene representations by integrating 3D acoustic information, enabling temporally and spatially aware multimodal understanding of surgical environments. Methods: We propose a novel framework for generating 4D audio-visual representations of surgical scenes by projecting acoustic localization information from a phased microphone array onto dynamic point clouds from an RGB-D camera. A transformer-based acoustic event detection module identifies relevant temporal segments containing tool-tissue interactions which are spatially localized in the audio-visual scene representation. The system was experimentally evaluated in a realistic operating room setup during simulated surgical procedures performed by experts. Results: The proposed method successfully localizes surgical acoustic events in 3D space and associates them with visual scene elements. Experimental evaluation demonstrates accurate spatial sound localization and robust fusion of multimodal data, providing a comprehensive, dynamic representation of surgical activity. Conclusion: This work introduces the first approach for spatial sound localization in dynamic surgical scenes, marking a significant advancement toward multimodal surgical scene representations. By integrating acoustic and visual data, the proposed framework enables richer contextual understanding and provides a foundation for future intelligent and autonomous surgical systems.