Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

TL;DR

This paper introduces Vision-Language Bootstrapping (VLB), a dynamic evaluation protocol for LVLMs that reduces data contamination and adapts to models' evolving capabilities by generating new multimodal samples.

Contribution

The paper presents a novel dynamic evaluation method that generates diverse, consistent multimodal samples to assess LVLMs more robustly and flexibly than static benchmarks.

Findings

VLB reduces data contamination in evaluations.

VLB exposes limitations of current LVLMs.

VLB adapts to models' evolving capabilities.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.