EGTR: Extracting Graph from Transformer for Scene Graph Generation

TL;DR

This paper introduces EGTR, a lightweight one-stage scene graph generation model that extracts relation graphs from self-attention layers of DETR, utilizing relation smoothing and auxiliary connectivity prediction for improved performance.

Contribution

The paper proposes a novel method to extract relation graphs from self-attention in DETR, with relation smoothing and auxiliary tasks enhancing scene graph generation.

Findings

Effective relation graph extraction from self-attention layers.

Improved scene graph generation performance on Visual Genome and Open Image V6.

Efficient and lightweight model suitable for real-world applications.

Abstract

Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. After DETR was developed, one-stage SGG models based on a one-stage object detector have been actively studied. However, complex modeling is used to predict the relationship between objects, and the inherent relationship between object queries learned in the multi-head self-attention of the object detector has been neglected. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. By fully utilizing the self-attention by-products, the relation graph can be extracted effectively with a shallow relation extraction head. Considering the dependency of the relation extraction task on the object detection task, we propose a novel relation smoothing technique that adjusts the relation label adaptively according to the quality of the detected objects. By the relation smoothing, the model is trained according to the continuous curriculum that focuses on object detection task at the beginning of training and performs multi-task learning as the object detection performance gradually improves. Furthermore, we propose a connectivity prediction task that predicts whether a relation exists between object pairs as an auxiliary task of the relation extraction. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets. Our code is publicly available at https://github.com/naver-ai/egtr.