Deep Tracking: Seeing Beyond Seeing Using Recurrent Neural Networks

TL;DR

This paper introduces a novel end-to-end deep learning approach using recurrent neural networks for real-time object tracking directly from raw sensor data, capable of handling occlusions and noise without feature engineering.

Contribution

It presents the first end-to-end object tracking system that maps raw sensor input to environment state using recurrent neural networks and unsupervised learning with input dropout.

Findings

Successfully tracks multiple objects despite occlusions

Learns from raw sensor data without ground-truth annotations

Operates in real-time with noisy sensor inputs

Abstract

This paper presents to the best of our knowledge the first end-to-end object tracking approach which directly maps from raw sensor input to object tracks in sensor space without requiring any feature engineering or system identification in the form of plant or sensor models. Specifically, our system accepts a stream of raw sensor data at one end and, in real-time, produces an estimate of the entire environment state at the output including even occluded objects. We achieve this by framing the problem as a deep learning task and exploit sequence models in the form of recurrent neural networks to learn a mapping from sensor measurements to object tracks. In particular, we propose a learning method based on a form of input dropout which allows learning in an unsupervised manner, only based on raw, occluded sensor data without access to ground-truth annotations. We demonstrate our approach using a synthetic dataset designed to mimic the task of tracking objects in 2D laser data -- as commonly encountered in robotics applications -- and show that it learns to track many dynamic objects despite occlusions and the presence of sensor noise.