Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation

TL;DR

This paper introduces PRISE, a new method that combines concise interaction graphs and discriminative scene features to improve social relation inference from images, outperforming existing approaches.

Contribution

The paper proposes a simple relational graph convolutional network and a contrastive learning task for holistic scene understanding, along with a large-scale dataset for social relation inference.

Findings

PRISE achieves 6.8% improvement in domain classification on PIPA.

The method outperforms state-of-the-art approaches.

The large-scale dataset enhances social relation inference.

Abstract

There has been a recent surge of research interest in attacking the problem of social relation inference based on images. Existing works classify social relations mainly by creating complicated graphs of human interactions, or learning the foreground and/or background information of persons and objects, but ignore holistic scene context. The holistic scene refers to the functionality of a place in images, such as dinning room, playground and office. In this paper, by mimicking human understanding on images, we propose an approach of \textbf{PR}actical \textbf{I}nference in \textbf{S}ocial r\textbf{E}lation (PRISE), which concisely learns interactive features of persons and discriminative features of holistic scenes. Technically, we develop a simple and fast relational graph convolutional network to capture interactive features of all persons in one image. To learn the holistic scene feature, we elaborately design a contrastive learning task based on image scene classification. To further boost the performance in social relation inference, we collect and distribute a new large-scale dataset, which consists of about 240 thousand unlabeled images. The extensive experimental results show that our novel learning framework significantly beats the state-of-the-art methods, e.g., PRISE achieves 6.8$\%$ improvement for domain classification in PIPA dataset.