Text-Aware Dual Routing Network for Visual Question Answering

TL;DR

The paper introduces a Text-Aware Dual Routing Network for VQA that effectively handles questions requiring text understanding in images, improving accuracy especially on number-related questions.

Contribution

It proposes a dual-branch network with dynamic routing to better handle text and non-text questions in visual question answering tasks.

Findings

Outperforms existing methods on VQA v2.0 dataset

Significantly improves accuracy on number-related questions

Incorporates OCR features for better text understanding

Abstract

Visual question answering (VQA) is a challenging task to provide an accurate natural language answer given an image and a natural language question about the image. It involves multi-modal learning, i.e., computer vision (CV) and natural language processing (NLP), as well as flexible answer prediction for free-form and open-ended answers. Existing approaches often fail in cases that require reading and understanding text in images to answer questions. In practice, they cannot effectively handle the answer sequence derived from text tokens because the visual features are not text-oriented. To address the above issues, we propose a Text-Aware Dual Routing Network (TDR) which simultaneously handles the VQA cases with and without understanding text information in the input images. Specifically, we build a two-branch answer prediction network that contains a specific branch for each case and further develop a dual routing scheme to dynamically determine which branch should be chosen. In the branch that involves text understanding, we incorporate the Optical Character Recognition (OCR) features into the model to help understand the text in the images. Extensive experiments on the VQA v2.0 dataset demonstrate that our proposed TDR outperforms existing methods, especially on the ''number'' related VQA questions.