Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMs

TL;DR

This paper introduces WM-MoE, a novel world model-based motion forecasting framework that leverages large language models and mixture of experts to improve autonomous vehicle safety in rare corner cases, outperforming existing methods.

Contribution

The paper presents WM-MoE, integrating perception, temporal memory, and decision making with LLMs and MoE to address corner cases in autonomous driving, a novel approach in the field.

Findings

Outperforms state-of-the-art baselines on multiple datasets.

Robust in corner-case and data-missing scenarios.

Introduces nuScenes-corner benchmark for evaluation.

Abstract

Accurate and reliable motion forecasting is essential for the safe deployment of autonomous vehicles (AVs), particularly in rare but safety-critical scenarios known as corner cases. Existing models often underperform in these situations due to an over-representation of common scenes in training data and limited generalization capabilities. To address this limitation, we present WM-MoE, the first world model-based motion forecasting framework that unifies perception, temporal memory, and decision making to address the challenges of high-risk corner-case scenarios. The model constructs a compact scene representation that explains current observations, anticipates future dynamics, and evaluates the outcomes of potential actions. To enhance long-horizon reasoning, we leverage large language models (LLMs) and introduce a lightweight temporal tokenizer that maps agent trajectories and contextual cues into the LLM's feature space without additional training, enriching temporal context and commonsense priors. Furthermore, a mixture-of-experts (MoE) is introduced to decompose complex corner cases into subproblems and allocate capacity across scenario types, and a router assigns scenes to specialized experts that infer agent intent and perform counterfactual rollouts. In addition, we introduce nuScenes-corner, a new benchmark that comprises four real-world corner-case scenarios for rigorous evaluation. Extensive experiments on four benchmark datasets (nuScenes, NGSIM, HighD, and MoCAD) showcase that WM-MoE consistently outperforms state-of-the-art (SOTA) baselines and remains robust under corner-case and data-missing conditions, indicating the promise of world model-based architectures for robust and generalizable motion forecasting in fully AVs.