Pedestrian Trajectory Prediction with Structured Memory Hierarchies

TL;DR

This paper introduces a structured memory hierarchy framework using multimodal data for improved pedestrian trajectory prediction, inspired by neuroscience, and demonstrates its effectiveness on new and existing datasets.

Contribution

It proposes a novel structured memory component with hierarchical LSTM cells for multimodal trajectory prediction, capturing both short-term and long-term context.

Findings

Outperforms existing methods on the new multimodal dataset.

Achieves better prediction accuracy on the New York Grand Central dataset.

Effectively integrates multimodal data without supervision.

Abstract

This paper presents a novel framework for human trajectory prediction based on multimodal data (video and radar). Motivated by recent neuroscience discoveries, we propose incorporating a structured memory component in the human trajectory prediction pipeline to capture historical information to improve performance. We introduce structured LSTM cells for modelling the memory content hierarchically, preserving the spatiotemporal structure of the information and enabling us to capture both short-term and long-term context. We demonstrate how this architecture can be extended to integrate salient information from multiple modalities to automatically store and retrieve important information for decision making without any supervision. We evaluate the effectiveness of the proposed models on a novel multimodal dataset that we introduce, consisting of 40,000 pedestrian trajectories, acquired jointly from a radar system and a CCTV camera system installed in a public place. The performance is also evaluated on the publicly available New York Grand Central pedestrian database. In both settings, the proposed models demonstrate their capability to better anticipate future pedestrian motion compared to existing state of the art.