Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions

TL;DR

This paper introduces a novel graph neural network approach to predict human and object future states during interactions with large objects, utilizing a new dataset and object dynamic descriptors to improve accuracy and generalization.

Contribution

The study presents a new dataset for full-body interactions with large objects, and proposes HO-GCN, a graph neural network that fuses motion data with object dynamic descriptors for improved prediction.

Findings

HO-GCN achieves state-of-the-art prediction accuracy.

Object dynamic descriptors enhance generalization to unseen objects.

Predicted interactions facilitate better human-robot collaboration.

Abstract

Understanding human intentions during interactions has been a long-lasting theme, that has applications in human-robot interaction, virtual reality and surveillance. In this study, we focus on full-body human interactions with large-sized daily objects and aim to predict the future states of objects and humans given a sequential observation of human-object interaction. As there is no such dataset dedicated to full-body human interactions with large-sized daily objects, we collected a large-scale dataset containing thousands of interactions for training and evaluation purposes. We also observe that an object's intrinsic physical properties are useful for the object motion prediction, and thus design a set of object dynamic descriptors to encode such intrinsic properties. We treat the object dynamic descriptors as a new modality and propose a graph neural network, HO-GCN, to fuse motion data and dynamic descriptors for the prediction task. We show the proposed network that consumes dynamic descriptors can achieve state-of-the-art prediction results and help the network better generalize to unseen objects. We also demonstrate the predicted results are useful for human-robot collaborations.