COPILOT: Human-Environment Collision Prediction and Localization from Egocentric Videos

TL;DR

COPILOT is a transformer-based model that predicts and localizes human-environment collisions from multi-view egocentric videos, enabling collision avoidance in VR, AR, and robotics, trained on a large synthetic dataset.

Contribution

The paper introduces COPILOT, a novel transformer model with a 4D attention mechanism for collision prediction and localization from egocentric videos, and provides a large synthetic dataset for training.

Findings

COPILOT generalizes well to unseen scenes.

The model accurately predicts collision regions.

Outputs improve downstream collision avoidance.

Abstract

The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. In this work, we introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. Solving this problem requires a generalizable perception system that can classify which human body joints will collide and estimate a collision region heatmap to localize collisions in the environment. To achieve this, we propose a transformer-based model called COPILOT to perform collision prediction and localization simultaneously, which accumulates information across multi-view inputs through a novel 4D space-time-viewpoint attention mechanism. To train our model and enable future research on this task, we develop a synthetic data generation framework that produces egocentric videos of virtual humans moving and colliding within diverse 3D environments. This framework is then used to establish a large-scale dataset consisting of 8.6M egocentric RGBD frames. Extensive experiments show that COPILOT generalizes to unseen synthetic as well as real-world scenes. We further demonstrate COPILOT outputs are useful for downstream collision avoidance through simple closed-loop control. Please visit our project webpage at https://sites.google.com/stanford.edu/copilot.