Assessing Vision-Language Models for Perception in Autonomous Underwater Robotic Software

TL;DR

This paper empirically evaluates vision-language models for perception tasks in autonomous underwater robots, focusing on their ability to detect underwater trash amidst challenging conditions.

Contribution

It provides an assessment of VLM performance and uncertainty in underwater environments, aiding software engineers in selecting suitable models.

Findings

VLMs show potential for underwater perception tasks.

Uncertainty correlates with detection performance.

Performance varies across different VLMs and conditions.

Abstract

Autonomous Underwater Robots (AURs) operate in challenging underwater environments, including low visibility and harsh water conditions. Such conditions present challenges for software engineers developing perception modules for the AUR software. To successfully carry out these tasks, deep learning has been incorporated into the AUR software to support its operations. However, the unique challenges of underwater environments pose difficulties for deep learning models, which often rely on labeled data that is scarce and noisy. This may undermine the trustworthiness of AUR software that relies on perception modules. Vision-Language Models (VLMs) offer promising solutions for AUR software as they generalize to unseen objects and remain robust in noisy conditions by inferring information from contextual cues. Despite this potential, their performance and uncertainty in underwater environments remain understudied from a software engineering perspective. Motivated by the needs of an industrial partner in assurance and risk management for maritime systems to assess the potential use of VLMs in this context, we present an empirical evaluation of VLM-based perception modules within the AUR software. We assess their ability to detect underwater trash by computing performance, uncertainty, and their relationship, to enable software engineers to select appropriate VLMs for their AUR software.