GoG: Relation-aware Graph-over-Graph Network for Visual Dialog

TL;DR

This paper introduces GoG, a relation-aware graph-over-graph network that models complex dependencies in visual dialog, improving understanding of dialog history, questions, and images for better conversational AI performance.

Contribution

The paper proposes a novel multi-graph neural network architecture that explicitly models coreference, dependency, and object relations in visual dialog tasks, addressing limitations of previous graph models.

Findings

Outperforms strong baselines in generative and discriminative settings

Effectively captures coreference and dependency relations in dialog history and questions

Enhances image representation based on full question understanding

Abstract

Visual dialog, which aims to hold a meaningful conversation with humans about a given image, is a challenging task that requires models to reason the complex dependencies among visual content, dialog history, and current questions. Graph neural networks are recently applied to model the implicit relations between objects in an image or dialog. However, they neglect the importance of 1) coreference relations among dialog history and dependency relations between words for the question representation; and 2) the representation of the image based on the fully represented question. Therefore, we propose a novel relation-aware graph-over-graph network (GoG) for visual dialog. Specifically, GoG consists of three sequential graphs: 1) H-Graph, which aims to capture coreference relations among dialog history; 2) History-aware Q-Graph, which aims to fully understand the question through capturing dependency relations between words based on coreference resolution on the dialog history; and 3) Question-aware I-Graph, which aims to capture the relations between objects in an image based on fully question representation. As an additional feature representation module, we add GoG to the existing visual dialogue model. Experimental results show that our model outperforms the strong baseline in both generative and discriminative settings by a significant margin.