PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN

TL;DR

This paper introduces PPR-FCN, a novel weakly supervised model for detecting subject-predicate-object relations in images, efficiently handling large region pair sets with shared computation and a new score map for context.

Contribution

The paper proposes a parallel FCN architecture with pairwise RoI pooling and a position-role-sensitive score map for improved weakly supervised visual relation detection.

Findings

PPR-FCN outperforms baseline methods on relation detection benchmarks.

The model effectively captures context with the position-role-sensitive score map.

Shared computation reduces complexity in processing large region pairs.

Abstract

We aim to tackle a novel vision task called Weakly Supervised Visual Relation Detection (WSVRD) to detect "subject-predicate-object" relations in an image with object relation groundtruths available only at the image level. This is motivated by the fact that it is extremely expensive to label the combinatorial relations between objects at the instance level. Compared to the extensively studied problem, Weakly Supervised Object Detection (WSOD), WSVRD is more challenging as it needs to examine a large set of regions pairs, which is computationally prohibitive and more likely stuck in a local optimal solution such as those involving wrong spatial context. To this end, we present a Parallel, Pairwise Region-based, Fully Convolutional Network (PPR-FCN) for WSVRD. It uses a parallel FCN architecture that simultaneously performs pair selection and classification of single regions and region pairs for object and relation detection, while sharing almost all computation shared over the entire image. In particular, we propose a novel position-role-sensitive score map with pairwise RoI pooling to efficiently capture the crucial context associated with a pair of objects. We demonstrate the superiority of PPR-FCN over all baselines in solving the WSVRD challenge by using results of extensive experiments over two visual relation benchmarks.