Single-Stage Visual Relationship Learning using Conditional Queries

TL;DR

This paper introduces TraCQ, a single-stage scene graph generation model using conditional queries and DETR architecture, achieving superior performance and efficiency over existing methods.

Contribution

Proposes a novel single-stage SGG model with conditional queries that simplifies multi-task learning and reduces parameters, outperforming existing methods.

Findings

TraCQ reduces parameters by 20% compared to state-of-the-art.

Outperforms existing single-stage SGG methods on Visual Genome.

Beats many two-stage methods while enabling end-to-end training.

Abstract

Research in scene graph generation (SGG) usually considers two-stage models, that is, detecting a set of entities, followed by combining them and labeling all possible relationships. While showing promising results, the pipeline structure induces large parameter and computation overhead, and typically hinders end-to-end optimizations. To address this, recent research attempts to train single-stage models that are computationally efficient. With the advent of DETR, a set based detection model, one-stage models attempt to predict a set of subject-predicate-object triplets directly in a single shot. However, SGG is inherently a multi-task learning problem that requires modeling entity and predicate distributions simultaneously. In this paper, we propose Transformers with conditional queries for SGG, namely, TraCQ with a new formulation for SGG that avoids the multi-task learning problem and the combinatorial entity pair distribution. We employ a DETR-based encoder-decoder design and leverage conditional queries to significantly reduce the entity label space as well, which leads to 20% fewer parameters compared to state-of-the-art single-stage models. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset, yet is capable of end-to-end training and faster inference.