HOTR: End-to-End Human-Object Interaction Detection with Transformers

TL;DR

HOTR introduces an end-to-end transformer-based framework for human-object interaction detection that directly predicts interaction triplets, outperforming existing methods with faster inference and better accuracy.

Contribution

The paper proposes a novel transformer encoder-decoder architecture for direct HOI triplet prediction, eliminating the need for post-processing and improving efficiency.

Findings

Achieves state-of-the-art performance on HOI detection benchmarks.

Inference time is under 1 millisecond after object detection.

Effectively exploits semantic relationships in images.

Abstract

Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred to by HOTR, which directly predicts a set of <human, object, interaction> triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.