Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, Giedrius Burachas

TL;DR
This paper introduces ConVQA, a dataset and metrics for evaluating VQA consistency, and proposes a CTM data augmentation method that improves model consistency by generating and training on entailed questions.
Contribution
It presents a new dataset and metrics for assessing VQA consistency, and introduces a novel CTM method for enhancing model grounding through entailed question generation.
Findings
CTM improves VQA model consistency on ConVQA datasets.
Generated entailed questions help train models to produce logically consistent answers.
The approach sets a new baseline for VQA consistency enhancement.
Abstract
While models for Visual Question Answering (VQA) have steadily improved over the years, interacting with one quickly reveals that these models lack consistency. For instance, if a model answers "red" to "What color is the balloon?", it might answer "no" if asked, "Is the balloon red?". These responses violate simple notions of entailment and raise questions about how effectively VQA models ground language. In this work, we introduce a dataset, ConVQA, and metrics that enable quantitative evaluation of consistency in VQA. For a given observable fact in an image (e.g. the balloon's color), we generate a set of logically consistent question-answer (QA) pairs (e.g. Is the balloon red?) and also collect a human-annotated set of common-sense based consistent QA pairs (e.g. Is the balloon the same color as tomato sauce?). Further, we propose a consistency-improving data augmentation module, a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
