Towards Flexible Evaluation for Generative Visual Question Answering

TL;DR

This paper introduces a semantics-based evaluation framework for VQA, proposing a flexible evaluator that better captures open-ended responses and outperforms existing methods, enhancing the assessment of multimodal models.

Contribution

It develops a novel Semantically Flexible VQA Evaluator (SFVE) and a dataset for analyzing VQA evaluators, addressing limitations of exact match metrics.

Findings

SFVE surpasses existing semantic evaluators significantly.

Model-based evaluation is feasible and effective.

The training scheme generalizes across different encoder architectures.

Abstract

Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different than the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we proposes three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation. Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM.