HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models

TL;DR

HEAT introduces a trajectory-guided learning approach with a world model to enable a single autonomous driving model to perform well across diverse, heterogeneous environments without domain-specific retraining.

Contribution

The paper proposes a novel trajectory-driven training paradigm with a world model to improve multi-domain autonomous driving performance.

Findings

Substantial performance improvements across three benchmarks.

Unified model maintains strong performance across diverse domains.

Trajectory-guided learning reduces domain-induced biases.

Abstract

End-to-end autonomous driving has emerged as a compelling alternative to traditional modular pipelines by directly mapping raw sensor data to driving actions. While recent approaches achieve strong performance on single-domain datasets, their performance degrades significantly when trained jointly across multiple heterogeneous domains. In practice, however, autonomous systems must operate across diverse environments with heterogeneous distributions, including different cities, sensor configurations, and traffic patterns, without domain-specific retraining. This gap highlights a key challenge in multi-domain learning: domain-specific variations across heterogeneous domains introduce conflicting learning signals, driving models toward compromised solutions that are suboptimal across domains. To address this, we propose a trajectory-driven learning paradigm that organizes training around planning trajectories, enabling the model to capture domain-invariant representations of driving intent. Furthermore, we incorporate a world model that predicts future latent features conditioned on ego actions, improving feature consistency and mitigating domain-induced biases. We evaluate our approach on three benchmarks, nuScenes, NAVSIM, and the Waymo end-to-end dataset, and show substantial improvements over existing methods across all domains. Our results demonstrate that a single unified model can be trained on heterogeneous datasets while maintaining strong performance within each domain, highlighting a step toward scalable real-world deployment. We will make our code publicly available.