SonarSweep: Fusing Sonar and Vision for Robust 3D Reconstruction via Plane Sweeping

TL;DR

SonarSweep is an end-to-end deep learning framework that fuses sonar and vision data using a plane sweep algorithm to achieve robust 3D reconstruction in underwater environments with poor visibility.

Contribution

It introduces a novel deep learning approach for cross-modal sonar-vision fusion, overcoming limitations of prior heuristic methods for underwater 3D reconstruction.

Findings

Outperforms state-of-the-art methods in simulation and real-world tests.

Produces dense, accurate depth maps in high turbidity conditions.

Demonstrates robustness and effectiveness of the fusion approach.

Abstract

Accurate 3D reconstruction in visually-degraded underwater environments remains a formidable challenge. Single-modality approaches are insufficient: vision-based methods fail due to poor visibility and geometric constraints, while sonar is crippled by inherent elevation ambiguity and low resolution. Consequently, prior fusion technique relies on heuristics and flawed geometric assumptions, leading to significant artifacts and an inability to model complex scenes. In this paper, we introduce SonarSweep, a novel, end-to-end deep learning framework that overcomes these limitations by adapting the principled plane sweep algorithm for cross-modal fusion between sonar and visual data. Extensive experiments in both high-fidelity simulation and real-world environments demonstrate that SonarSweep consistently generates dense and accurate depth maps, significantly outperforming state-of-the-art methods across challenging conditions, particularly in high turbidity. To foster further research, we will publicly release our code and a novel dataset featuring synchronized stereo-camera and sonar data, the first of its kind.