Learning to Track with Object Permanence

TL;DR

This paper introduces a novel end-to-end trainable multi-object tracking method that incorporates object permanence reasoning through a spatio-temporal memory module, improving robustness to occlusions in videos.

Contribution

It extends the CenterTrack architecture with a recurrent memory for reasoning over long video sequences and proposes new training strategies using synthetic data for tracking behind occlusions.

Findings

Outperforms state-of-the-art on KITTI and MOT17 datasets.

Robust to occlusions due to object permanence reasoning.

Effective training with synthetic and real data.

Abstract

Tracking by detection, the dominant approach for online multi-object tracking, alternates between localization and association steps. As a result, it strongly depends on the quality of instantaneous observations, often failing when objects are not fully visible. In contrast, tracking in humans is underlined by the notion of object permanence: once an object is recognized, we are aware of its physical existence and can approximately localize it even under full occlusions. In this work, we introduce an end-to-end trainable approach for joint object detection and tracking that is capable of such reasoning. We build on top of the recent CenterTrack architecture, which takes pairs of frames as input, and extend it to videos of arbitrary length. To this end, we augment the model with a spatio-temporal, recurrent memory module, allowing it to reason about object locations and identities in the current frame using all the previous history. It is, however, not obvious how to train such an approach. We study this question on a new, large-scale, synthetic dataset for multi-object tracking, which provides ground truth annotations for invisible objects, and propose several approaches for supervising tracking behind occlusions. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI and MOT17 datasets thanks to its robustness to occlusions.