AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions

TL;DR

This paper introduces AQuA, a dataset for ambiguous visual question answering, enabling models to recognize ambiguity levels and select appropriate response strategies, thus improving their ability to handle real-world VQA challenges.

Contribution

The paper presents AQuA, a novel dataset categorizing ambiguity in VQA and demonstrates fine-tuning models for strategy-aware responses, advancing beyond existing benchmarks.

Findings

Models trained on AQuA better recognize ambiguity levels.

Fine-tuned models select context-appropriate response strategies.

Enhanced models outperform baselines in strategic response generation.

Abstract

Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.