Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Study

TL;DR

This study evaluates GPT-4V's capabilities in marine analysis, revealing current limitations in domain-specific understanding and setting a benchmark for future multi-modal large language models in specialized fields.

Contribution

It provides a systematic assessment of GPT-4V's performance in marine research, highlighting gaps and establishing a standard for future domain-specific MLLM development.

Findings

GPT-4V responses are insufficient for marine domain needs.

Current GPT-4V models lack domain-specific expertise.

The study offers a benchmark for future marine-focused MLLMs.

Abstract

Large language models (LLMs) have demonstrated a powerful ability to answer various queries as a general-purpose assistant. The continuous multi-modal large language models (MLLM) empower LLMs with the ability to perceive visual signals. The launch of GPT-4 (Generative Pre-trained Transformers) has generated significant interest in the research communities. GPT-4V(ison) has demonstrated significant power in both academia and industry fields, as a focal point in a new artificial intelligence generation. Though significant success was achieved by GPT-4V, exploring MLLMs in domain-specific analysis (e.g., marine analysis) that required domain-specific knowledge and expertise has gained less attention. In this study, we carry out the preliminary and comprehensive case study of utilizing GPT-4V for marine analysis. This report conducts a systematic evaluation of existing GPT-4V, assessing the performance of GPT-4V on marine research and also setting a new standard for future developments in MLLMs. The experimental results of GPT-4V show that the responses generated by GPT-4V are still far away from satisfying the domain-specific requirements of the marine professions. All images and prompts used in this study will be available at https://github.com/hkust-vgd/Marine_GPT-4V_Eval