Learning Human-Object Interaction Detection using Interaction Points

TL;DR

This paper introduces a novel fully-convolutional method for human-object interaction detection that predicts interaction points and vectors, achieving state-of-the-art results on V-COCO and HICO-DET benchmarks.

Contribution

It proposes the first approach to treat HOI detection as a keypoint detection and grouping problem, improving accuracy over previous instance-centric methods.

Findings

Sets new state-of-the-art on V-COCO and HICO-DET datasets.

Introduces a fully-convolutional network predicting interaction points and vectors.

Demonstrates the effectiveness of keypoint-based HOI detection.

Abstract

Understanding interactions between humans and objects is one of the fundamental problems in visual classification and an essential step towards detailed scene understanding. Human-object interaction (HOI) detection strives to localize both the human and an object as well as the identification of complex interactions between them. Most existing HOI detection approaches are instance-centric where interactions between all possible human-object pairs are predicted based on appearance features and coarse spatial information. We argue that appearance features alone are insufficient to capture complex human-object interactions. In this paper, we therefore propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Paired with the densely predicted interaction vectors, the interactions are associated with human and object detections to obtain final predictions. To the best of our knowledge, we are the first to propose an approach where HOI detection is posed as a keypoint detection and grouping problem. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET. Our approach sets a new state-of-the-art on both datasets. Code is available at https://github.com/vaesl/IP-Net.