MarineEval: Assessing the Marine Intelligence of Vision-Language Models

TL;DR

MarineEval introduces a large-scale marine-specific VLM dataset and benchmark, revealing current models' limitations in domain-specific marine question answering and highlighting the need for further advancements.

Contribution

This work creates the first comprehensive marine VLM dataset and benchmark, enabling evaluation of VLMs in marine domain expertise and identifying their current shortcomings.

Findings

Existing VLMs perform poorly on marine domain questions.

The dataset covers diverse tasks and capacities, ensuring broad evaluation.

Significant room for improvement in domain-specific VLM performance.

Abstract

We have witnessed promising progress led by large language models (LLMs) and further vision language models (VLMs) in handling various queries as a general-purpose assistant. VLMs, as a bridge to connect the visual world and language corpus, receive both visual content and various text-only user instructions to generate corresponding responses. Though great success has been achieved by VLMs in various fields, in this work, we ask whether the existing VLMs can act as domain experts, accurately answering marine questions, which require significant domain expertise and address special domain challenges/requirements. To comprehensively evaluate the effectiveness and explore the boundary of existing VLMs, we construct the first large-scale marine VLM dataset and benchmark called MarineEval, with 2,000 image-based question-answering pairs. During our dataset construction, we ensure the diversity and coverage of the constructed data: 7 task dimensions and 20 capacity dimensions. The domain requirements are specially integrated into the data construction and further verified by the corresponding marine domain experts. We comprehensively benchmark 17 existing VLMs on our MarineEval and also investigate the limitations of existing models in answering marine research questions. The experimental results reveal that existing VLMs cannot effectively answer the domain-specific questions, and there is still a large room for further performance improvements. We hope our new benchmark and observations will facilitate future research. Project Page: http://marineeval.hkustvgd.com/