AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysis

TL;DR

AquaticCLIP is a novel contrastive vision-language model designed for underwater scene understanding, enabling various tasks without ground-truth annotations by leveraging a large-scale underwater image-text dataset.

Contribution

The paper introduces AquaticCLIP, a new unsupervised framework with a prompt-guided vision encoder for underwater scene analysis, setting a new benchmark in aquatic vision-language tasks.

Findings

Outperforms existing methods in zero-shot underwater vision tasks

Enables segmentation, classification, detection, and object counting without annotations

Constructed a 2 million image-text paired dataset from diverse sources

Abstract

The preservation of aquatic biodiversity is critical in mitigating the effects of climate change. Aquatic scene understanding plays a pivotal role in aiding marine scientists in their decision-making processes. In this paper, we introduce AquaticCLIP, a novel contrastive language-image pre-training model tailored for aquatic scene understanding. AquaticCLIP presents a new unsupervised learning framework that aligns images and texts in aquatic environments, enabling tasks such as segmentation, classification, detection, and object counting. By leveraging our large-scale underwater image-text paired dataset without the need for ground-truth annotations, our model enriches existing vision-language models in the aquatic domain. For this purpose, we construct a 2 million underwater image-text paired dataset using heterogeneous resources, including YouTube, Netflix, NatGeo, etc. To fine-tune AquaticCLIP, we propose a prompt-guided vision encoder that progressively aggregates patch features via learnable prompts, while a vision-guided mechanism enhances the language encoder by incorporating visual context. The model is optimized through a contrastive pretraining loss to align visual and textual modalities. AquaticCLIP achieves notable performance improvements in zero-shot settings across multiple underwater computer vision tasks, outperforming existing methods in both robustness and interpretability. Our model sets a new benchmark for vision-language applications in underwater environments. The code and dataset for AquaticCLIP are publicly available on GitHub at xxx.