Improving Human-Object Interaction Detection via Virtual Image Learning

TL;DR

This paper introduces Virtual Image Learning (VIL) with a novel dataset creation method and a teacher-student training framework to improve Human-Object Interaction detection, especially addressing data imbalance issues.

Contribution

The paper proposes a new virtual image generation approach and a training framework that enhances HOI detection performance and can be integrated with existing methods.

Findings

Significant performance improvements on benchmarks

Achieved new state-of-the-art results

Effective handling of data imbalance in HOI detection

Abstract

Human-Object Interaction (HOI) detection aims to understand the interactions between humans and objects, which plays a curtail role in high-level semantic understanding tasks. However, most works pursue designing better architectures to learn overall features more efficiently, while ignoring the long-tail nature of interaction-object pair categories. In this paper, we propose to alleviate the impact of such an unbalanced distribution via Virtual Image Leaning (VIL). Firstly, a novel label-to-image approach, Multiple Steps Image Creation (MUSIC), is proposed to create a high-quality dataset that has a consistent distribution with real images. In this stage, virtual images are generated based on prompts with specific characterizations and selected by multi-filtering processes. Secondly, we use both virtual and real images to train the model with the teacher-student framework. Considering the initial labels of some virtual images are inaccurate and inadequate, we devise an Adaptive Matching-and-Filtering (AMF) module to construct pseudo-labels. Our method is independent of the internal structure of HOI detectors, so it can be combined with off-the-shelf methods by training merely 10 additional epochs. With the assistance of our method, multiple methods obtain significant improvements, and new state-of-the-art results are achieved on two benchmarks.