J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

TL;DR

This paper introduces J-EDI QA, a benchmark dataset for deep-sea organism image understanding using multimodal LLMs, revealing current models' limitations in expert-level deep-sea species comprehension.

Contribution

The paper presents a new benchmark dataset for deep-sea organism image understanding in Japanese, enabling evaluation of multimodal LLMs in this specialized domain.

Findings

OpenAI o1 achieved 50% accuracy on the benchmark.

Current models are not yet at expert-level understanding of deep-sea species.

The benchmark highlights the need for specialized deep-sea LLMs.

Abstract

Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.