Subjects and Their Objects: Localizing Interactees for a Person-Centric View of Importance

TL;DR

This paper introduces a method to localize the object of a person's action in images by predicting the likely position and scale of interactees, independent of specific actions, enhancing understanding of image context.

Contribution

It proposes a novel, action-independent approach to predict interactee locations using pose, gaze, and scene cues, and introduces a new dataset for evaluation.

Findings

Effective in localizing interactees across diverse images

Improves performance in object detection and scene understanding tasks

Demonstrates utility in natural language scene description

Abstract

Understanding images with people often entails understanding their \emph{interactions} with other objects or people. As such, given a novel image, a vision system ought to infer which other objects/people play an important role in a given person's activity. However, existing methods are limited to learning action-specific interactions (e.g., how the pose of a tennis player relates to the position of his racquet when serving the ball) for improved recognition, making them unequipped to reason about novel interactions with actions or objects unobserved in the training data. We propose to predict the "interactee" in novel images---that is, to localize the \emph{object} of a person's action. Given an arbitrary image with a detected person, the goal is to produce a saliency map indicating the most likely positions and scales where that person's interactee would be found. To that end, we explore ways to learn the generic, action-independent connections between (a) representations of a person's pose, gaze, and scene cues and (b) the interactee object's position and scale. We provide results on a newly collected UT Interactee dataset spanning more than 10,000 images from SUN, PASCAL, and COCO. We show that the proposed interaction-informed saliency metric has practical utility for four tasks: contextual object detection, image retargeting, predicting object importance, and data-driven natural language scene description. All four scenarios reveal the value in linking the subject to its object in order to understand the story of an image.