Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams

TL;DR

This paper introduces a dynamic graph-generation network that analyzes diagrams by integrating visual and textual information, enabling automatic understanding and knowledge extraction from multi-modal, arbitrarily laid out diagrams.

Contribution

It presents a novel unified diagram-parsing network utilizing dynamic memory and graph theory, achieving state-of-the-art results on diagram understanding tasks.

Findings

Outperforms existing baselines on diagram datasets

Demonstrates effectiveness in question answering applications

Utilizes gated recurrent units for dynamic information modeling

Abstract

In this work, we introduce a new algorithm for analyzing a diagram, which contains visual and textual information in an abstract and integrated way. Whereas diagrams contain richer information compared with individual image-based or language-based data, proper solutions for automatically understanding them have not been proposed due to their innate characteristics of multi-modality and arbitrariness of layouts. To tackle this problem, we propose a unified diagram-parsing network for generating knowledge from diagrams based on an object detector and a recurrent neural network designed for a graphical structure. Specifically, we propose a dynamic graph-generation network that is based on dynamic memory and graph theory. We explore the dynamics of information in a diagram with activation of gates in gated recurrent unit (GRU) cells. On publicly available diagram datasets, our model demonstrates a state-of-the-art result that outperforms other baselines. Moreover, further experiments on question answering shows potentials of the proposed method for various applications.