MoST: Multi-modality Scene Tokenization for Motion Prediction

TL;DR

This paper introduces MoST, a novel multi-modality scene tokenization method that encodes scene elements using pre-trained models, improving motion prediction accuracy and robustness over traditional symbolic and raw sensor approaches.

Contribution

MoST proposes a scene tokenization framework leveraging pre-trained models for open-vocabulary scene encoding, enhancing interpretability and performance in motion prediction.

Findings

Significant performance improvements on Waymo dataset

Efficient encoding with a few hundred tokens

Compatibility with transformer architectures

Abstract

Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this approach suffers from the lack of interpretability and requires significantly more training resources. In this work, we propose tokenizing the visual world into a compact set of scene elements and then leveraging pre-trained image foundation models and LiDAR neural networks to encode all the scene elements in an open-vocabulary manner. The image foundation model enables our scene tokens to encode the general knowledge of the open world while the LiDAR neural network encodes geometry information. Our proposed representation can efficiently encode the multi-frame multi-modality observations with a few hundred tokens and is compatible with most transformer-based architectures. To evaluate our method, we have augmented Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open Motion Dataset show that our approach leads to significant performance improvements over the state-of-the-art.