Detecting and Recognizing Human-Object Interactions

TL;DR

This paper introduces InteractNet, a novel end-to-end model for detecting human-object interactions in images by leveraging human appearance cues to localize objects and jointly recognizing triplets in challenging photos.

Contribution

The paper presents a human-centric model that predicts action-specific object locations and jointly detects humans and objects for interaction recognition.

Findings

Achieves state-of-the-art results on V-COCO and HICO-DET datasets.

Effectively localizes objects based on human appearance cues.

Demonstrates the importance of joint detection and interaction modeling.

Abstract

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.