TPT-Bench: A Large-Scale, Long-Term and Robot-Egocentric Dataset for Benchmarking Target Person Tracking

TL;DR

This paper introduces TPT-Bench, a comprehensive large-scale dataset for target person tracking in complex, real-world environments, enabling better benchmarking and development of robust tracking algorithms for autonomous robots.

Contribution

The paper presents a novel, large-scale, multi-modal dataset for target person tracking in unstructured environments, including extensive annotations and real-world scenarios, addressing limitations of existing benchmarks.

Findings

Existing SOTA methods struggle with occlusions and long-term re-identification.

The dataset reveals significant performance gaps in current tracking algorithms.

Analysis suggests directions for improving robustness in real-world tracking.

Abstract

Tracking a target person from robot-egocentric views is crucial for developing autonomous robots that provide continuous personalized assistance or collaboration in Human-Robot Interaction (HRI) and Embodied AI. However, most existing target person tracking (TPT) benchmarks are limited to controlled laboratory environments with few distractions, clean backgrounds, and short-term occlusions. In this paper, we introduce a large-scale dataset designed for TPT in crowded and unstructured environments, demonstrated through a robot-person following task. The dataset is collected by a human pushing a sensor-equipped cart while following a target person, capturing human-like following behavior and emphasizing long-term tracking challenges, including frequent occlusions and the need for re-identification from numerous pedestrians. It includes multi-modal data streams, including odometry, 3D LiDAR, IMU, panoramic images, and RGB-D images, along with exhaustively annotated 2D bounding boxes of the target person across 48 sequences, both indoors and outdoors. Using this dataset and visual annotations, we perform extensive experiments with existing SOTA TPT methods, offering a thorough analysis of their limitations and suggesting future research directions.