Egocentric Prediction of Action Target in 3D

TL;DR

This paper introduces a new egocentric vision task to predict the target location of object manipulation in 3D, supported by a large multimodal dataset and baseline methods, aiming to advance research in human-robot interaction.

Contribution

The paper presents a novel egocentric prediction task, a large multimodal dataset with annotations, and baseline recurrent neural network methods for early target prediction in 3D workspace.

Findings

Baseline models achieve promising results on the new task.

The dataset enables comprehensive evaluation with high-quality labels.

The task encourages further research in robotics and vision communities.

Abstract

We are interested in anticipating as early as possible the target location of a person's object manipulation action in a 3D workspace from egocentric vision. It is important in fields like human-robot collaboration, but has not yet received enough attention from vision and learning communities. To stimulate more research on this challenging egocentric vision task, we propose a large multimodality dataset of more than 1 million frames of RGB-D and IMU streams, and provide evaluation metrics based on our high-quality 2D and 3D labels from semi-automatic annotation. Meanwhile, we design baseline methods using recurrent neural networks and conduct various ablation studies to validate their effectiveness. Our results demonstrate that this new task is worthy of further study by researchers in robotics, vision, and learning communities.