ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks

TL;DR

ChartNet is a neural network model that combines visual perception and reasoning to interpret statistical charts, capable of answering both predefined and open-ended questions about chart images.

Contribution

The paper introduces ChartNet, a novel MAC-Network based system that performs reasoning over statistical charts, including localization of textual answers for open-ended questions.

Findings

ChartNet outperforms existing methods on chart reasoning tasks.

It effectively handles both in-vocabulary and out-of-vocabulary answers.

The dataset of chart images and questions demonstrates the model's practical applicability.

Abstract

Despite the improvements in perception accuracies brought about via deep learning, developing systems combining accurate visual perception with the ability to reason over the visual percepts remains extremely challenging. A particular application area of interest from an accessibility perspective is that of reasoning over statistical charts such as bar and pie charts. To this end, we formulate the problem of reasoning over statistical charts as a classification task using MAC-Networks to give answers from a predefined vocabulary of generic answers. Additionally, we enhance the capabilities of MAC-Networks to give chart-specific answers to open-ended questions by replacing the classification layer by a regression layer to localize the textual answers present over the images. We call our network ChartNet, and demonstrate its efficacy on predicting both in vocabulary and out of vocabulary answers. To test our methods, we generated our own dataset of statistical chart images and corresponding question answer pairs. Results show that ChartNet consistently outperform other state-of-the-art methods on reasoning over these questions and may be a viable candidate for applications containing images of statistical charts.