# Forecasting Time-to-Collision from Monocular Video: Feasibility, Dataset, and Challenges

## TL;DR

This paper demonstrates the feasibility of using monocular video and deep learning to accurately predict the time to collision with pedestrians, providing a fine-grained approach for collision avoidance in dynamic environments.

## Contribution

It introduces a novel dataset and a multi-stream CNN model for predicting exact time to collision from monocular video, surpassing binary detection methods.

## Key findings

- Average prediction error of 0.75 seconds
- Multi-stream CNN outperforms other models
- Dataset of over 13,000 indoor video segments

## Abstract

We explore the possibility of using a single monocular camera to forecast the time to collision between a suitcase-shaped robot being pushed by its user and other nearby pedestrians. We develop a purely image-based deep learning approach that directly estimates the time to collision without the need of relying on explicit geometric depth estimates or velocity information to predict future collisions. While previous work has focused on detecting immediate collision in the context of navigating Unmanned Aerial Vehicles, the detection was limited to a binary variable (i.e., collision or no collision). We propose a more fine-grained approach to collision forecasting by predicting the exact time to collision in terms of milliseconds, which is more helpful for collision avoidance in the context of dynamic path planning. To evaluate our method, we have collected a novel dataset of over 13,000 indoor video segments each showing a trajectory of at least one person ending in a close proximity (a near collision) with the camera mounted on a mobile suitcase-shaped platform. Using this dataset, we do extensive experimentation on different temporal windows as input using an exhaustive list of state-of-the-art convolutional neural networks (CNNs). Our results show that our proposed multi-stream CNN is the best model for predicting time to near-collision. The average prediction error of our time to near collision is 0.75 seconds across the test videos.