Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions

TL;DR

This paper presents a novel weakly supervised learning approach for detecting human-object interactions in videos using contrastive spatiotemporal region association and a new curated dataset, improving over baseline methods.

Contribution

The work introduces a contrastive weakly supervised training loss and a semi-automatically curated dataset for human-object interaction detection in videos.

Findings

Improved performance over weakly supervised baselines.

Effective spatiotemporal region association for interaction detection.

A new dataset with 6.5k videos for this task.

Abstract

We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object. To address these challenges, we introduce a contrastive weakly supervised training loss that aims to jointly associate spatiotemporal regions in a video with an action and object vocabulary and encourage temporal continuity of the visual appearance of moving objects as a form of self-supervision. To train our model, we introduce a dataset comprising over 6.5k videos with human-object interaction annotations that have been semi-automatically curated from sentence captions associated with the videos. We demonstrate improved performance over weakly supervised baselines adapted to our task on our video dataset.