Lost and Found: Overcoming Detector Failures in Online Multi-Object Tracking

TL;DR

This paper introduces BUSCA, a versatile online framework that improves multi-object tracking by recovering missed objects without altering past results or using future data, achieving state-of-the-art performance.

Contribution

BUSCA is a novel, detector-agnostic framework that enhances online multi-object tracking by leveraging multimodal information and a decision Transformer, trained solely on synthetic data.

Findings

Consistent performance improvements across five trackers.

Establishes new state-of-the-art results on three benchmarks.

Operates fully online without future frame access.

Abstract

Multi-object tracking (MOT) endeavors to precisely estimate the positions and identities of multiple objects over time. The prevailing approach, tracking-by-detection (TbD), first detects objects and then links detections, resulting in a simple yet effective method. However, contemporary detectors may occasionally miss some objects in certain frames, causing trackers to cease tracking prematurely. To tackle this issue, we propose BUSCA, meaning `to search', a versatile framework compatible with any online TbD system, enhancing its ability to persistently track those objects missed by the detector, primarily due to occlusions. Remarkably, this is accomplished without modifying past tracking results or accessing future frames, i.e., in a fully online manner. BUSCA generates proposals based on neighboring tracks, motion, and learned tokens. Utilizing a decision Transformer that integrates multimodal visual and spatiotemporal information, it addresses the object-proposal association as a multi-choice question-answering task. BUSCA is trained independently of the underlying tracker, solely on synthetic data, without requiring fine-tuning. Through BUSCA, we showcase consistent performance enhancements across five different trackers and establish a new state-of-the-art baseline across three different benchmarks. Code available at: https://github.com/lorenzovaquero/BUSCA.