Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

TL;DR

This paper introduces Curious-VLA, a two-stage framework that enhances exploration in autonomous vehicle learning models by generating diverse trajectories and prioritizing high-diversity samples, leading to state-of-the-art results.

Contribution

The paper proposes a novel two-stage approach with FTE and ADAS strategies to overcome narrow policy limitations in VLA models, improving exploration and driving performance.

Findings

Achieves state-of-the-art results on Navsim benchmark

Effectively enhances exploration in VLA models

Demonstrates significant performance improvements

Abstract

We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.