ORCA: Object Recognition and Comprehension for Archiving Marine Species

TL;DR

ORCA introduces a comprehensive multi-modal dataset and benchmark for marine species recognition and understanding, addressing domain-specific challenges and facilitating advancements in marine visual AI research.

Contribution

It provides a large, annotated dataset and evaluates multiple models across key tasks, establishing a systematic benchmark for marine visual understanding.

Findings

Species diversity and morphological overlap pose significant challenges.

Current models struggle with domain-specific demands.

The benchmark reveals key gaps in marine visual AI capabilities.

Abstract

Marine visual understanding is essential for monitoring and protecting marine ecosystems, enabling automatic and scalable biological surveys. However, progress is hindered by limited training data and the lack of a systematic task formulation that aligns domain-specific marine challenges with well-defined computer vision tasks, thereby limiting effective model application. To address this gap, we present ORCA, a multi-modal benchmark for marine research comprising 14,647 images from 478 species, with 42,217 bounding box annotations and 22,321 expert-verified instance captions. The dataset provides fine-grained visual and textual annotations that capture morphology-oriented attributes across diverse marine species. To catalyze methodological advances, we evaluate 18 state-of-the-art models on three tasks: object detection (closed-set and open-vocabulary), instance captioning, and visual grounding. Results highlight key challenges, including species diversity, morphological overlap, and specialized domain demands, underscoring the difficulty of marine understanding. ORCA thus establishes a comprehensive benchmark to advance research in marine domain. Project Page: http://orca.hkustvgd.com/.