ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos

TL;DR

This paper introduces ST-HOI, a novel spatial-temporal architecture for detecting human-object interactions in videos, emphasizing the importance of temporal context and proposing a new benchmark dataset.

Contribution

The paper presents a simple yet effective spatial-temporal model for video HOI detection and introduces the VidHOI benchmark dataset.

Findings

Naive temporal-aware models face feature-inconsistency issues.

ST-HOI effectively utilizes trajectories and spatial-temporal features.

Proposed method outperforms static image-based approaches on VidHOI.

Abstract

Detecting human-object interactions (HOI) is an important step toward a comprehensive visual understanding of machines. While detecting non-temporal HOIs (e.g., sitting on a chair) from static images is feasible, it is unlikely even for humans to guess temporal-related HOIs (e.g., opening/closing a door) from a single video frame, where the neighboring frames play an essential role. However, conventional HOI methods operating on only static images have been used to predict temporal-related interactions, which is essentially guessing without temporal contexts and may lead to sub-optimal performance. In this paper, we bridge this gap by detecting video-based HOIs with explicit temporal information. We first show that a naive temporal-aware variant of a common action detection baseline does not work on video-based HOIs due to a feature-inconsistency issue. We then propose a simple yet effective architecture named Spatial-Temporal HOI Detection (ST-HOI) utilizing temporal information such as human and object trajectories, correctly-localized visual features, and spatial-temporal masking pose features. We construct a new video HOI benchmark dubbed VidHOI where our proposed approach serves as a solid baseline.