RealTraj: Towards Real-World Pedestrian Trajectory Forecasting

TL;DR

RealTraj introduces a robust, real-world applicable pedestrian trajectory forecasting framework that minimizes data collection and annotation costs through self-supervised pretraining and weakly-supervised fine-tuning, outperforming existing methods.

Contribution

The paper presents a novel trajectory forecasting model, Det2TrajFormer, invariant to tracking noise, with a training approach that reduces reliance on costly annotations and enhances real-world robustness.

Findings

Outperforms state-of-the-art methods on multiple datasets

Effective with limited real-world data and detection-based training

Robust to tracking noise and perception errors

Abstract

This paper jointly addresses three key limitations in conventional pedestrian trajectory forecasting: pedestrian perception errors, real-world data collection costs, and person ID annotation costs. We propose a novel framework, RealTraj, that enhances the real-world applicability of trajectory forecasting. Our approach includes two training phases -- self-supervised pretraining on synthetic data and weakly-supervised fine-tuning with limited real-world data -- to minimize data collection efforts. To improve robustness to real-world errors, we focus on both model design and training objectives. Specifically, we present Det2TrajFormer, a trajectory forecasting model that remains invariant to tracking noise by using past detections as inputs. Additionally, we pretrain the model using multiple pretext tasks, which enhance robustness and improve forecasting performance based solely on detection data. Unlike previous trajectory forecasting methods, our approach fine-tunes the model using only ground-truth detections, reducing the need for costly person ID annotations. In the experiments, we comprehensively verify the effectiveness of the proposed method against the limitations, and the method outperforms state-of-the-art trajectory forecasting methods on multiple datasets. The code will be released at https://fujiry0.github.io/RealTraj-project-page.