MUSE-VAE: Multi-Scale VAE for Environment-Aware Long Term Trajectory Prediction

TL;DR

MUSE-VAE introduces a multi-scale probabilistic framework using cascade VAEs for environment-aware long-term trajectory prediction, effectively modeling uncertainty and interactions in complex scenes.

Contribution

It proposes a novel multi-factor, coarse-to-fine VAE architecture that jointly models environment and agent dynamics for improved long-term trajectory forecasting.

Findings

Outperforms state-of-the-art on nuScenes and SDD benchmarks.

Demonstrates effective modeling of environment-agent interactions.

Provides diverse, accurate long-term trajectory predictions.

Abstract

Accurate long-term trajectory prediction in complex scenes, where multiple agents (e.g., pedestrians or vehicles) interact with each other and the environment while attempting to accomplish diverse and often unknown goals, is a challenging stochastic forecasting problem. In this work, we propose MUSE, a new probabilistic modeling framework based on a cascade of Conditional VAEs, which tackles the long-term, uncertain trajectory prediction task using a coarse-to-fine multi-factor forecasting architecture. In its Macro stage, the model learns a joint pixel-space representation of two key factors, the underlying environment and the agent movements, to predict the long and short-term motion goals. Conditioned on them, the Micro stage learns a fine-grained spatio-temporal representation for the prediction of individual agent trajectories. The VAE backbones across the two stages make it possible to naturally account for the joint uncertainty at both levels of granularity. As a result, MUSE offers diverse and simultaneously more accurate predictions compared to the current state-of-the-art. We demonstrate these assertions through a comprehensive set of experiments on nuScenes and SDD benchmarks as well as PFSD, a new synthetic dataset, which challenges the forecasting ability of models on complex agent-environment interaction scenarios.