SGTR: End-to-end Scene Graph Generation with Transformer

TL;DR

This paper introduces SGTR, a transformer-based end-to-end method for scene graph generation that formulates the task as bipartite graph construction, achieving state-of-the-art results with higher efficiency.

Contribution

The paper proposes a novel transformer-based framework for scene graph generation, including entity-aware predicate representation and a graph assembling module for end-to-end inference.

Findings

Achieves state-of-the-art or comparable performance on benchmarks.

Surpasses most existing approaches in accuracy.

Offers higher inference efficiency.

Abstract

Scene Graph Generation (SGG) remains a challenging visual understanding task due to its compositional property. Most previous works adopt a bottom-up two-stage or a point-based one-stage approach, which often suffers from high time complexity or sub-optimal designs. In this work, we propose a novel SGG method to address the aforementioned issues, formulating the task as a bipartite graph construction problem. To solve the problem, we develop a transformer-based end-to-end framework that first generates the entity and predicate proposal set, followed by inferring directed edges to form the relation triplets. In particular, we develop a new entity-aware predicate representation based on a structural predicate generator that leverages the compositional property of relationships. Moreover, we design a graph assembling module to infer the connectivity of the bipartite scene graph based on our entity-aware structure, enabling us to generate the scene graph in an end-to-end manner. Extensive experimental results show that our design is able to achieve the state-of-the-art or comparable performance on two challenging benchmarks, surpassing most of the existing approaches and enjoying higher efficiency in inference. We hope our model can serve as a strong baseline for the Transformer-based scene graph generation. Code is available: https://github.com/Scarecrow0/SGTR