Unifying Language-Action Understanding and Generation for Autonomous Driving

TL;DR

This paper presents LinkVLA, a unified model for autonomous driving that improves language-action alignment and efficiency through shared codebooks, bidirectional training, and a fast coarse-to-fine decoding method, leading to better performance and lower latency.

Contribution

The paper introduces LinkVLA, a novel architecture unifying language and action tokens, and a two-step coarse-to-fine generation approach, significantly enhancing alignment and inference speed in autonomous driving.

Findings

Improved instruction following accuracy in driving benchmarks.

Reduced inference time by 86% with C2F decoding.

Enhanced cross-modal consistency and semantic understanding.

Abstract

Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.