MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses

TL;DR

This paper introduces MX-LSTM, a novel model that jointly forecasts pedestrian trajectories and head poses by integrating tracklets and vislets, significantly improving accuracy especially in challenging scenarios.

Contribution

The paper presents a new framework called MX-LSTM that jointly models tracklets and vislets with a novel covariance optimization, advancing trajectory and head pose forecasting.

Findings

Achieves state-of-the-art trajectory forecasting accuracy on multiple datasets.

Significantly improves predictions when pedestrians slow down.

Joint modeling of tracklets and vislets enhances long-term forecast reliability.

Abstract

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.