ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Driving

TL;DR

ALN-P3 introduces a unified framework that aligns vision and language modules across perception, prediction, and planning in autonomous driving, improving performance and interpretability without extra inference costs.

Contribution

It proposes a novel co-distillation framework with three alignment mechanisms that unify vision and language reasoning in autonomous driving systems.

Findings

Achieves state-of-the-art results on four benchmarks.

Significantly improves driving decision accuracy.

Enhances language reasoning capabilities in autonomous systems.

Abstract

Recent advances have explored integrating large language models (LLMs) into end-to-end autonomous driving systems to enhance generalization and interpretability. However, most existing approaches are limited to either driving performance or vision-language reasoning, making it difficult to achieve both simultaneously. In this paper, we propose ALN-P3, a unified co-distillation framework that introduces cross-modal alignment between "fast" vision-based autonomous driving systems and "slow" language-driven reasoning modules. ALN-P3 incorporates three novel alignment mechanisms: Perception Alignment (P1A), Prediction Alignment (P2A), and Planning Alignment (P3A), which explicitly align visual tokens with corresponding linguistic outputs across the full perception, prediction, and planning stack. All alignment modules are applied only during training and incur no additional costs during inference. Extensive experiments on four challenging benchmarks-nuScenes, Nu-X, TOD3Cap, and nuScenes QA-demonstrate that ALN-P3 significantly improves both driving decisions and language reasoning, achieving state-of-the-art results.