Sonar-MASt3R: Real-Time Opti-Acoustic Fusion in Turbid, Unstructured Environments

TL;DR

This paper introduces Sonar-MASt3R, a real-time opti-acoustic fusion system that enhances underwater perception in turbid environments by combining dense optical correspondences with acoustic data for improved 3D reconstruction.

Contribution

The paper presents a novel real-time opti-acoustic fusion method, Sonar-MASt3R, capable of dense 3D underwater reconstruction in turbid conditions, outperforming existing approaches.

Findings

Improved robustness to turbidity over baseline methods

Real-time dense 3D reconstruction achieved

Effective in environments with turbidity up to >12 NTU

Abstract

Underwater intervention is an important capability in several marine domains, with numerous industrial, scientific, and defense applications. However, existing perception systems used during intervention operations rely on data from optical cameras, which limits capabilities in poor visibility or lighting conditions. Prior work has examined opti-acoustic fusion methods, which use sonar data to resolve the depth ambiguity of the camera data while using camera data to resolve the elevation angle ambiguity of the sonar data. However, existing methods cannot achieve dense 3D reconstructions in real-time, and few studies have reported results from applying these methods in a turbid environment. In this work, we propose the opti-acoustic fusion method Sonar-MASt3R, which uses MASt3R to extract dense correspondences from optical camera data in real-time and pairs it with geometric cues from an acoustic 3D reconstruction to ensure robustness in turbid conditions. Experimental results using data recorded from an opti-acoustic eye-in-hand configuration across turbidity values ranging from <0.5 to >12 NTU highlight this method's improved robustness to turbidity relative to baseline methods.