STARS: Zero-shot Sim-to-Real Transfer for Segmentation of Shipwrecks in Sonar Imagery

TL;DR

This paper introduces STARS, a novel segmentation network enabling zero-shot sim-to-real transfer for shipwreck detection in sonar imagery, achieving significant performance improvements without real data training.

Contribution

STARS is a new segmentation approach that fuses deformation prediction and anomaly detection to improve zero-shot transfer from simulation to real sonar images.

Findings

20% performance increase over baselines

Effective zero-shot transfer without real data fine-tuning

Generalizes well to real sonar images in shipwreck segmentation

Abstract

In this paper, we address the problem of sim-to-real transfer for object segmentation when there is no access to real examples of an object of interest during training, i.e. zero-shot sim-to-real transfer for segmentation. We focus on the application of shipwreck segmentation in side scan sonar imagery. Our novel segmentation network, STARS, addresses this challenge by fusing a predicted deformation field and anomaly volume, allowing it to generalize better to real sonar images and achieve more effective zero-shot sim-to-real transfer for image segmentation. We evaluate the sim-to-real transfer capabilities of our method on a real, expert-labeled side scan sonar dataset of shipwrecks collected from field work surveys with an autonomous underwater vehicle (AUV). STARS is trained entirely in simulation and performs zero-shot shipwreck segmentation with no additional fine-tuning on real data. Our method provides a significant 20% increase in segmentation performance for the targeted shipwreck class compared to the best baseline.