Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens

TL;DR

This paper introduces AbilityLens, a unified benchmark for evaluating vision perception abilities of multimodal large language models, revealing strengths, weaknesses, and training phenomena to guide future development.

Contribution

The paper presents AbilityLens, a comprehensive and robust benchmark for perception abilities in MLLMs, addressing evaluation variance and providing insights into model performance and training dynamics.

Findings

Identifies performance gaps between open-source and closed-source MLLMs.

Reveals stability patterns and ability conflicts during training.

Suggests fine-tuning and model merging as strategies to mitigate ability conflicts.

Abstract

As multimodal large language models (MLLMs) advance rapidly, rigorous evaluation has become essential, providing further guidance for their development. In this work, we focus on a unified and robust evaluation of \textbf{vision perception} abilities, the foundational skill of MLLMs. We find that existing perception benchmarks, each focusing on different question types, domains, and evaluation metrics, introduce significant evaluation variance, complicating comprehensive assessments of perception abilities when relying on any single benchmark. To address this, we introduce \textbf{AbilityLens}, a unified benchmark designed to evaluate MLLMs in six key perception abilities (ranging from counting, OCR, to understanding structural data), focusing on both accuracy and stability, with each ability encompassing diverse types of questions, domains, and metrics. With the assistance of AbilityLens, we: (1) identify the strengths and weaknesses of current main-stream MLLMs, highlighting stability patterns and revealing a notable performance gap between state-of-the-art open-source and closed-source models; (2) uncover interesting ability conflict and early convergence phenomena during MLLM training; (3) reveal the primary reason of ability conflict is data mixing ratio and LLM model size; and (4) discuss the effectiveness of some straightforward strategies \eg, fine-tuning and model merging, to solve the ability conflict. The benchmark and online leaderboard is released in https://github.com/Chenfeng1271/AbilityLens.