Collision Replay: What Does Bumping Into Things Tell You About Scene Geometry?

TL;DR

This paper introduces collision replay, a method where collision data is used to train neural networks to infer scene geometry and navigational features from images, enhancing scene understanding.

Contribution

The paper proposes a novel collision replay technique that leverages collision examples to supervise neural networks for scene geometry prediction from images.

Findings

Collision replay effectively predicts scene geometry.

The approach improves understanding of navigational affordances.

Neural networks trained with collision replay outperform baseline methods.

Abstract

What does bumping into things in a scene tell you about scene geometry? In this paper, we investigate the idea of learning from collisions. At the heart of our approach is the idea of collision replay, where we use examples of a collision to provide supervision for observations at a past frame. We use collision replay to train convolutional neural networks to predict a distribution over collision time from new images. This distribution conveys information about the navigational affordances (e.g., corridors vs open spaces) and, as we show, can be converted into the distance function for the scene geometry. We analyze this approach with an agent that has noisy actuation in a photorealistic simulator.