MODS -- A USV-oriented object detection and obstacle segmentation benchmark

TL;DR

This paper introduces MODS, a comprehensive benchmark dataset and evaluation protocol for obstacle detection and segmentation tailored for small unmanned surface vehicles, addressing the lack of standardized maritime perception datasets.

Contribution

The paper presents a new diverse maritime dataset with 81k stereo images and over 60k annotated objects, along with a novel evaluation protocol for obstacle detection and segmentation in USV scenarios.

Findings

Evaluation of 19 state-of-the-art methods using the new protocol.

Benchmark results highlight current method performances and challenges.

Public availability of dataset and evaluation routines to foster research.

Abstract

Small-sized unmanned surface vehicles (USV) are coastal water devices with a broad range of applications such as environmental control and surveillance. A crucial capability for autonomous operation is obstacle detection for timely reaction and collision avoidance, which has been recently explored in the context of camera-based visual scene interpretation. Owing to curated datasets, substantial advances in scene interpretation have been made in a related field of unmanned ground vehicles. However, the current maritime datasets do not adequately capture the complexity of real-world USV scenes and the evaluation protocols are not standardised, which makes cross-paper comparison of different methods difficult and hinders the progress. To address these issues, we introduce a new obstacle detection benchmark MODS, which considers two major perception tasks: maritime object detection and the more general maritime obstacle segmentation. We present a new diverse maritime evaluation dataset containing approximately 81k stereo images synchronized with an on-board IMU, with over 60k objects annotated. We propose a new obstacle segmentation performance evaluation protocol that reflects the detection accuracy in a way meaningful for practical USV navigation. Nineteen recent state-of-the-art object detection and obstacle segmentation methods are evaluated using the proposed protocol, creating a benchmark to facilitate development of the field. The proposed dataset, as well as evaluation routines, are made publicly available at vicos.si/resources.