CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li, Fei-Fei, C. Lawrence Zitnick, Ross Girshick
TL;DR
The paper introduces CLEVR, a diagnostic dataset designed to evaluate and analyze the reasoning capabilities of AI systems in visual question answering, addressing biases and providing detailed annotations for targeted assessment.
Contribution
CLEVR is a novel diagnostic dataset with minimal biases and detailed reasoning annotations, enabling precise evaluation of visual reasoning models.
Findings
Modern visual reasoning systems show specific strengths and weaknesses.
CLEVR reveals model limitations in compositional reasoning.
The dataset facilitates targeted analysis of reasoning abilities.
Abstract
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Topic Modeling