ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks

TL;DR

ReForm-Eval introduces a unified benchmark for evaluating large vision-language models by reformulating existing benchmarks into LVLM-compatible formats, enabling comprehensive and automated assessment of their capabilities.

Contribution

The paper proposes a systematic method to reformulate existing benchmarks into a unified format for LVLM evaluation, reducing manual effort and enabling comprehensive assessment.

Findings

Extensive experiments reveal strengths and weaknesses of current LVLMs.

ReForm-Eval provides a large, unified dataset for LVLM evaluation.

Analysis uncovers key factors influencing LVLM performance.

Abstract

Recent years have witnessed remarkable progress in the development of large vision-language models (LVLMs). Benefiting from the strong language backbones and efficient cross-modal alignment strategies, LVLMs exhibit surprising capabilities to perceive visual signals and perform visually grounded reasoning. However, the capabilities of LVLMs have not been comprehensively and quantitatively evaluate. Most existing multi-modal benchmarks require task-oriented input-output formats, posing great challenges to automatically assess the free-form text output of LVLMs. To effectively leverage the annotations available in existing benchmarks and reduce the manual effort required for constructing new benchmarks, we propose to re-formulate existing benchmarks into unified LVLM-compatible formats. Through systematic data collection and reformulation, we present the ReForm-Eval benchmark, offering substantial data for evaluating various capabilities of LVLMs. Based on ReForm-Eval, we conduct extensive experiments, thoroughly analyze the strengths and weaknesses of existing LVLMs, and identify the underlying factors. Our benchmark and evaluation framework will be open-sourced as a cornerstone for advancing the development of LVLMs.