Multi-Object Tracking and Segmentation with a Space-Time Memory Network

TL;DR

This paper introduces MeNToS, a multi-object tracking and segmentation method that uses a space-time memory network to improve long-term data association, especially during occlusions or long gaps, outperforming re-identification approaches.

Contribution

The paper presents a novel memory-based data association mechanism for multi-object tracking that enhances long-term tracking robustness using a space-time memory network.

Findings

Outperforms re-identification methods in long-term association

Improves HOTA metric scores on KITTIMOTS and MOTSChallenge datasets

Demonstrates robustness in tracking objects with long occlusions

Abstract

We propose a method for multi-object tracking and segmentation based on a novel memory-based mechanism to associate tracklets. The proposed tracker, MeNToS, addresses particularly the long-term data association problem, when objects are not observable for long time intervals. Indeed, the recently introduced HOTA metric (High Order Tracking Accuracy), which has a better alignment than the formerly established MOTA (Multiple Object Tracking Accuracy) with the human visual assessment of tracking, has shown that improvements are still needed for data association, despite the recent improvement in object detection. In MeNToS, after creating tracklets using instance segmentation and optical flow, the proposed method relies on a space-time memory network originally developed for one-shot video object segmentation to improve the association of sequence of detections (tracklets) with temporal gaps. We evaluate our tracker on KITTIMOTS and MOTSChallenge and we show the benefit of our data association strategy with the HOTA metric. Additional ablation studies demonstrate that our approach using a space-time memory network gives better and more robust long-term association than those based on a re-identification network. Our project page is at \url{www.mehdimiah.com/mentos+}.