Visual Relationship Detection with Relative Location Mining

TL;DR

This paper introduces a novel visual relationship detection framework that leverages relative location information of object pairs throughout the process, enhancing predicate recognition and overall accuracy.

Contribution

The work proposes a new framework that deeply mines and utilizes relative location data, including a GGNN for better predicate relevance measurement, improving detection performance.

Findings

Significant performance improvement on VRD and VG datasets.

Enhanced clustering of spatially similar predicates.

Increased top n recall accuracy.

Abstract

Visual relationship detection, as a challenging task used to find and distinguish the interactions between object pairs in one image, has received much attention recently. In this work, we propose a novel visual relationship detection framework by deeply mining and utilizing relative location of object-pair in every stage of the procedure. In both the stages, relative location information of each object-pair is abstracted and encoded as auxiliary feature to improve the distinguishing capability of object-pairs proposing and predicate recognition, respectively; Moreover, one Gated Graph Neural Network(GGNN) is introduced to mine and measure the relevance of predicates using relative location. With the location-based GGNN, those non-exclusive predicates with similar spatial position can be clustered firstly and then be smoothed with close classification scores, thus the accuracy of top $n$ recall can be increased further. Experiments on two widely used datasets VRD and VG show that, with the deeply mining and exploiting of relative location information, our proposed model significantly outperforms the current state-of-the-art.