TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models

TL;DR

This paper introduces Tiny LVLM-eHub, a lightweight, comprehensive evaluation framework for large vision-language models like Google's Bard, assessing multiple multimodal capabilities with improved accuracy and ease of use.

Contribution

It presents a systematic, multi-capability evaluation method for LVLMs, including a new lightweight benchmark and analysis approach that enhances evaluation robustness and practicality.

Findings

Bard outperforms previous LVLMs in most capabilities

Tiny LVLM-eHub evaluates six multimodal categories using 42 benchmarks

The ChatGPT Ensemble Evaluation improves alignment with human judgment

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated significant progress in tackling complex multimodal tasks. Among these cutting-edge developments, Google's Bard stands out for its remarkable multimodal capabilities, promoting comprehensive comprehension and reasoning across various domains. This work presents an early and holistic evaluation of LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the vanilla version, Tiny LVLM-eHub possesses several appealing properties. Firstly, it provides a systematic assessment of six categories of multimodal capabilities, including visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, and embodied intelligence, through quantitative evaluation of $42$ standard text-related visual benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and accurate evaluation and exhibits improved alignment with human evaluation compared to the word matching approach. Thirdly, it comprises a mere $2.1$K image-text pairs, facilitating ease of use for practitioners to evaluate their own offline LVLMs. Through extensive experimental analysis, this study demonstrates that Bard outperforms previous LVLMs in most multimodal capabilities except object hallucination, to which Bard is still susceptible. Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages innovative strategies aimed at advancing multimodal techniques. Our project is publicly available at \url{https://github.com/OpenGVLab/Multi-Modality-Arena}.