LIME: Less Is More for MLLM Evaluation

TL;DR

LIME is a curated, efficient benchmark for evaluating multimodal large language models that filters out uninformative samples, reducing evaluation time and improving the assessment of model capabilities.

Contribution

We introduce LIME, a semi-automated pipeline that creates a more effective and efficient benchmark by filtering uninformative samples and focusing on image-based understanding tasks.

Findings

LIME reduces sample size by 76% and evaluation time by 77%.

Traditional metrics like CIDEr are inadequate for captioning evaluation.

Excluding caption scores improves overall model performance assessment.

Abstract

Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.