Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing

TL;DR

This paper introduces VL-MPAG Net, a graph neural network that leverages scene graphs and message passing to improve object grounding in natural images, outperforming existing methods on multiple datasets.

Contribution

The paper proposes a novel visio-lingual message passing graph neural network for scene graph grounding, integrating semantic relationships to enhance object localization.

Findings

Significant performance improvements over baselines on four datasets.

Effective use of scene graph relationships for contextual object grounding.

Robustness across varied scene complexities.

Abstract

This paper presents a framework for jointly grounding objects that follow certain semantic relationship constraints given in a scene graph. A typical natural scene contains several objects, often exhibiting visual relationships of varied complexities between them. These inter-object relationships provide strong contextual cues toward improving grounding performance compared to a traditional object query-only-based localization task. A scene graph is an efficient and structured way to represent all the objects and their semantic relationships in the image. In an attempt towards bridging these two modalities representing scenes and utilizing contextual information for improving object localization, we rigorously study the problem of grounding scene graphs on natural images. To this end, we propose a novel graph neural network-based approach referred to as Visio-Lingual Message PAssing Graph Neural Network (VL-MPAG Net). In VL-MPAG Net, we first construct a directed graph with object proposals as nodes and an edge between a pair of nodes representing a plausible relation between them. Then a three-step inter-graph and intra-graph message passing is performed to learn the context-dependent representation of the proposals and query objects. These object representations are used to score the proposals to generate object localization. The proposed method significantly outperforms the baselines on four public datasets.