Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges

TL;DR

This paper introduces a comprehensive framework for latent world models in automated driving, organizing design choices, proposing evaluation methods, and outlining future research directions to enhance robustness, generalization, and efficiency.

Contribution

It unifies recent advances into a taxonomy, defines evaluation protocols, and highlights open challenges for latent world models in automated driving.

Findings

Proposes a unifying taxonomy for latent world models.

Introduces evaluation metrics including a closed-loop metric suite.

Identifies key research directions for robustness and resource efficiency.

Abstract

Emerging generative world models and vision-language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning, reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.