Benchmarking Large Language Models for Image Classification of Marine Mammals

TL;DR

This paper introduces a new benchmark dataset of 1,423 images of 65 marine mammal species, evaluates various AI models including LLMs and a novel multi-agent system for classification, and demonstrates their strengths and improvements.

Contribution

The work creates the first marine mammal-specific dataset and benchmarks multiple AI approaches, including a novel LLM-based multi-agent system for classification.

Findings

Traditional models perform well in certain aspects

LLMs show strong zero-shot classification abilities

The multi-agent system enhances classification accuracy

Abstract

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.