TAO: A Large-Scale Benchmark for Tracking Any Object
Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, Deva, Ramanan

TL;DR
TAO introduces a large-scale, diverse benchmark dataset for tracking any object, significantly expanding vocabulary and environmental variety to advance multi-object tracking research.
Contribution
The paper presents TAO, a new large-scale, diverse dataset with 2,907 videos and 833 object categories, enabling research on open-world, large-vocabulary object tracking.
Findings
Existing trackers struggle with large-vocabulary, open-world tracking.
Detection-based multi-object trackers are competitive with user-initialized methods.
The dataset reveals challenges and opportunities for improving tracking algorithms.
Abstract
For many years, multi-object tracking benchmarks have focused on a handful of categories. Motivated primarily by surveillance and self-driving applications, these datasets provide tracks for people, vehicles, and animals, ignoring the vast majority of objects in the world. By contrast, in the related field of object detection, the introduction of large-scale, diverse datasets (e.g., COCO) have fostered significant progress in developing highly robust solutions. To bridge this gap, we introduce a similarly diverse dataset for Tracking Any Object (TAO). It consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. Importantly, we adopt a bottom-up approach for discovering a large vocabulary of 833 categories, an order of magnitude more than prior tracking benchmarks. To this end, we ask annotators to label objects that move at any…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
