Latent Chain-of-Thought World Modeling for End-to-End Driving

TL;DR

This paper introduces LCDrive, a latent chain-of-thought model for autonomous driving that improves reasoning and decision-making by using a learned latent space instead of natural language, leading to better performance.

Contribution

The work presents a novel latent CoT reasoning framework for driving that unifies decision making and reasoning in an action-aligned latent space, enhancing inference speed and trajectory quality.

Findings

LCDrive achieves faster inference than baselines.

It produces higher quality driving trajectories.

Reinforcement learning further improves reasoning capabilities.

Abstract

Recent Vision-Language-Action (VLA) models for autonomous driving explore inference-time reasoning as a way to improve driving performance and safety in challenging scenarios. Most prior work uses natural language to express chain-of-thought (CoT) reasoning before producing driving actions. However, text may not be the most efficient representation for reasoning. In this work, we present Latent-CoT-Drive (LCDrive): a model that expresses CoT in a latent language that captures possible outcomes of the driving actions being considered. Our approach unifies CoT reasoning and decision making by representing both in an action-aligned latent space. Instead of natural language, the model reasons by interleaving (1) action-proposal tokens, which use the same vocabulary as the model's output actions; and (2) world model tokens, which are grounded in a learned latent world model and express future outcomes of these actions. We cold start latent CoT by supervising the model's action proposals and world model tokens based on ground-truth future rollouts of the scene. We then post-train with closed-loop reinforcement learning to strengthen reasoning capabilities. On a large-scale end-to-end driving benchmark, LCDrive achieves faster inference, better trajectory quality, and larger improvements from interactive reinforcement learning compared to both non-reasoning and text-reasoning baselines.