RepSGG: Novel Representations of Entities and Relationships for Scene Graph Generation

TL;DR

RepSGG introduces a dynamic, fine-grained approach to scene graph generation that improves representation flexibility and balances class distribution, achieving state-of-the-art results efficiently.

Contribution

The paper proposes RepSGG, a novel architecture with flexible entity and relationship representations and a run-time logit adjustment strategy for better generalization.

Findings

Achieves state-of-the-art performance on Visual Genome and Open Images V6 datasets.

Provides faster inference speed compared to previous methods.

Effectively balances performance across common and rare classes.

Abstract

Scene Graph Generation (SGG) has achieved significant progress recently. However, most previous works rely heavily on fixed-size entity representations based on bounding box proposals, anchors, or learnable queries. As each representation's cardinality has different trade-offs between performance and computation overhead, extracting highly representative features efficiently and dynamically is both challenging and crucial for SGG. In this work, a novel architecture called RepSGG is proposed to address the aforementioned challenges, formulating a subject as queries, an object as keys, and their relationship as the maximum attention weight between pairwise queries and keys. With more fine-grained and flexible representation power for entities and relationships, RepSGG learns to sample semantically discriminative and representative points for relationship inference. Moreover, the long-tailed distribution also poses a significant challenge for generalization of SGG. A run-time performance-guided logit adjustment (PGLA) strategy is proposed such that the relationship logits are modified via affine transformations based on run-time performance during training. This strategy encourages a more balanced performance between dominant and rare classes. Experimental results show that RepSGG achieves the state-of-the-art or comparable performance on the Visual Genome and Open Images V6 datasets with fast inference speed, demonstrating the efficacy and efficiency of the proposed methods.