TallyQA: Answering Complex Counting Questions

TL;DR

This paper introduces TallyQA, a large dataset for complex counting in visual question answering, and proposes a relation network-based algorithm that achieves state-of-the-art results on multiple benchmarks.

Contribution

The paper presents TallyQA, the largest dataset for complex counting questions, and a novel relation network-based algorithm optimized for high-resolution images.

Findings

Achieved state-of-the-art results on TallyQA and HowMany-QA benchmarks.

Demonstrated the effectiveness of relation networks with region proposals for complex counting.

Showed improved performance over baseline systems on complex counting tasks.

Abstract

Most counting questions in visual question answering (VQA) datasets are simple and require no more than object detection. Here, we study algorithms for complex counting questions that involve relationships between objects, attribute identification, reasoning, and more. To do this, we created TallyQA, the world's largest dataset for open-ended counting. We propose a new algorithm for counting that uses relation networks with region proposals. Our method lets relation networks be efficiently used with high-resolution imagery. It yields state-of-the-art results compared to baseline and recent systems on both TallyQA and the HowMany-QA benchmark.