MULTIAQUA: A multimodal maritime dataset and robust training strategies for multimodal semantic segmentation

TL;DR

This paper introduces MULTIAQUA, a comprehensive multimodal maritime dataset with synchronized sensor data, and proposes training strategies for robust semantic segmentation under challenging visibility conditions, especially at night.

Contribution

The paper presents a new multimodal maritime dataset and robust training methods that enable effective scene understanding in poor visibility conditions, reducing the need for nighttime data collection.

Findings

Multimodal methods outperform unimodal ones in low-visibility scenarios.

Training with daytime images alone can achieve robustness in nighttime conditions.

The dataset facilitates development of resilient maritime perception systems.

Abstract

Unmanned surface vehicles can encounter a number of varied visual circumstances during operation, some of which can be very difficult to interpret. While most cases can be solved only using color camera images, some weather and lighting conditions require additional information. To expand the available maritime data, we present a novel multimodal maritime dataset MULTIAQUA (Multimodal Aquatic Dataset). Our dataset contains synchronized, calibrated and annotated data captured by sensors of different modalities, such as RGB, thermal, IR, LIDAR, etc. The dataset is aimed at developing supervised methods that can extract useful information from these modalities in order to provide a high quality of scene interpretation regardless of potentially poor visibility conditions. To illustrate the benefits of the proposed dataset, we evaluate several multimodal methods on our difficult nighttime test set. We present training approaches that enable multimodal methods to be trained in a more robust way, thus enabling them to retain reliable performance even in near-complete darkness. Our approach allows for training a robust deep neural network only using daytime images, thus significantly simplifying data acquisition, annotation, and the training process.