LaViPlan : Language-Guided Visual Path Planning with RLVR

TL;DR

LaViPlan introduces a reinforcement learning-based fine-tuning method for vision-language models to improve autonomous driving planning, especially in out-of-distribution scenarios, by aligning language reasoning with low-level trajectory planning.

Contribution

The paper presents RLVR, a novel fine-tuning approach that enhances VLMs for planning tasks in autonomous driving, addressing the misalignment issue between language reasoning and trajectory generation.

Findings

Improved planning performance on in-domain and out-of-domain datasets.

Slight decrease in linguistic fidelity post-fine-tuning, but outputs remain coherent.

Ablation studies reveal the impact of sampling ratio and reasoning guidance on performance.

Abstract

Out-of-distribution (OOD) scenarios in autonomous driving pose critical challenges, as planners often fail to generalize beyond their training experience, leading to unsafe or unexpected behavior. Vision-Language Models (VLMs) have shown promise in handling such scenarios by providing high-level scene understanding and user-aligned decisions. However, existing VLMs often exhibit a misalignment between their language-based reasoning and the low-level trajectories required for action-level planning. In this paper, we propose LaViPlan, a framework that leverages Reinforcement Learning with Verifiable Rewards (RLVR) to fine-tune VLMs using planning-oriented metrics. Experimental results show that LaViPlan improves planning performance across both in-domain and out-of-domain datasets. While linguistic fidelity slightly decreases after RLVR-based fine-tuning, qualitative evaluation indicates that the outputs remain coherent. We also conduct ablation studies to analyze the effects of sampling ratio and reasoning guidance, highlighting how these design choices influence performance. These findings demonstrate the potential of RLVR as a post-training paradigm for aligning language-guided reasoning with action-level planning in autonomous driving.