Improving Scene Graph Generation with Superpixel-Based Interaction Learning

TL;DR

This paper introduces Superpixel-based Interaction Learning (SIL), a novel approach that models fine-grained scene interactions at the superpixel level, significantly improving scene graph generation accuracy over traditional box-level methods.

Contribution

The paper proposes a generic superpixel-based paradigm for SGG that captures detailed interactions, enhancing existing box-level approaches with a plug-and-play method.

Findings

SIL outperforms previous state-of-the-art methods on Visual Genome and Open Image V6 benchmarks.

SIL improves the PredCls task by an average of 2.0% mR, up to 3.4%.

The method effectively models intra- and cross-entity superpixel interactions.

Abstract

Recent advances in Scene Graph Generation (SGG) typically model the relationships among entities utilizing box-level features from pre-defined detectors. We argue that an overlooked problem in SGG is the coarse-grained interactions between boxes, which inadequately capture contextual semantics for relationship modeling, practically limiting the development of the field. In this paper, we take the initiative to explore and propose a generic paradigm termed Superpixel-based Interaction Learning (SIL) to remedy coarse-grained interactions at the box level. It allows us to model fine-grained interactions at the superpixel level in SGG. Specifically, (i) we treat a scene as a set of points and cluster them into superpixels representing sub-regions of the scene. (ii) We explore intra-entity and cross-entity interactions among the superpixels to enrich fine-grained interactions between entities at an earlier stage. Extensive experiments on two challenging benchmarks (Visual Genome and Open Image V6) prove that our SIL enables fine-grained interaction at the superpixel level above previous box-level methods, and significantly outperforms previous state-of-the-art methods across all metrics. More encouragingly, the proposed method can be applied to boost the performance of existing box-level approaches in a plug-and-play fashion. In particular, SIL brings an average improvement of 2.0% mR (even up to 3.4%) of baselines for the PredCls task on Visual Genome, which facilitates its integration into any existing box-level method.