Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation

TL;DR

This paper introduces AutoConverter, an automated framework for converting open-ended visual questions into challenging multiple-choice questions, enabling more reliable and scalable evaluation of vision language models.

Contribution

AutoConverter automates the creation of multiple-choice questions from existing datasets, facilitating consistent and scalable evaluation of VLMs.

Findings

AutoConverter generates challenging multiple-choice questions with high accuracy.

VLMs perform similarly or worse on AutoConverter-generated questions compared to human-made ones.

VMCBench provides a comprehensive benchmark for evaluating 33 state-of-the-art VLMs.

Abstract

The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce AutoConverter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly multiple-choice question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using AutoConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.