Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation

TL;DR

This paper introduces a subgraph-based framework for scene graph generation that significantly improves efficiency and accuracy by reducing computation and maintaining spatial information, outperforming existing methods.

Contribution

The paper proposes a novel subgraph-based connection graph and clustering method to enhance scene graph generation efficiency and accuracy without relying on external data.

Findings

Outperforms state-of-the-art in accuracy on Visual Relationship Detection and Visual Genome datasets.

Achieves faster inference speed compared to previous methods.

Effectively maintains spatial information through subgraph features.

Abstract

Generating scene graph to describe all the relations inside an image gains increasing interests these years. However, most of the previous methods use complicated structures with slow inference speed or rely on the external data, which limits the usage of the model in real-life scenarios. To improve the efficiency of scene graph generation, we propose a subgraph-based connection graph to concisely represent the scene graph during the inference. A bottom-up clustering method is first used to factorize the entire scene graph into subgraphs, where each subgraph contains several objects and a subset of their relationships. By replacing the numerous relationship representations of the scene graph with fewer subgraph and object features, the computation in the intermediate stage is significantly reduced. In addition, spatial information is maintained by the subgraph features, which is leveraged by our proposed Spatial-weighted Message Passing~(SMP) structure and Spatial-sensitive Relation Inference~(SRI) module to facilitate the relationship recognition. On the recent Visual Relationship Detection and Visual Genome datasets, our method outperforms the state-of-the-art method in both accuracy and speed.