Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information

TL;DR

This paper enhances visual relationship prediction by using label clustering to reduce class complexity and incorporating depth information to improve model accuracy, validated through extensive experiments.

Contribution

It introduces an unsupervised label clustering method and integrates depth data into the relationship prediction model, improving accuracy on the PIC dataset.

Findings

Significant accuracy improvements with label clustering.

Depth information enhances relationship prediction.

Effective mitigation of class imbalance issues.

Abstract

In this paper, we investigate the use of an unsupervised label clustering technique and demonstrate that it enables substantial improvements in visual relationship prediction accuracy on the Person in Context (PIC) dataset. We propose to group object labels with similar patterns of relationship distribution in the dataset into fewer categories. Label clustering not only mitigates both the large classification space and class imbalance issues, but also potentially increases data samples for each clustered category. We further propose to incorporate depth information as an additional feature into the instance segmentation model. The additional depth prediction path supplements the relationship prediction model in a way that bounding boxes or segmentation masks are unable to deliver. We have rigorously evaluated the proposed techniques and performed various ablation analysis to validate the benefits of them.