IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model

TL;DR

This paper introduces IRL-VLA, a novel framework for training vision-language-action models in autonomous driving using a reward world model and reinforcement learning, addressing limitations of previous open-loop and simulation-dependent methods.

Contribution

The paper presents a three-stage training paradigm combining imitation learning, inverse reinforcement learning, and reinforcement learning with a reward world model for VLA in autonomous driving.

Findings

Achieved state-of-the-art performance on NAVSIM v2 benchmark.

Secured 1st runner-up in CVPR2025 Autonomous Grand Challenge.

Demonstrated effective close-loop training without high-fidelity simulation.

Abstract

Vision-Language-Action (VLA) models have demonstrated potential in autonomous driving. However, two critical challenges hinder their development: (1) Existing VLA architectures are typically based on imitation learning in open-loop setup which tends to capture the recorded behaviors in the dataset, leading to suboptimal and constrained performance, (2) Close-loop training relies heavily on high-fidelity sensor simulation, where domain gaps and computational inefficiencies pose significant barriers. In this paper, we introduce IRL-VLA, a novel close-loop Reinforcement Learning via \textbf{I}nverse \textbf{R}einforcement \textbf{L}earning reward world model with a self-built VLA approach. Our framework proceeds in a three-stage paradigm: In the first stage, we propose a VLA architecture and pretrain the VLA policy via imitation learning. In the second stage, we construct a lightweight reward world model via inverse reinforcement learning to enable efficient close-loop reward computation. To further enhance planning performance, finally, we design specialized reward world model guidence reinforcement learning via PPO(Proximal Policy Optimization) to effectively balance the safety incidents, comfortable driving, and traffic efficiency. Our approach achieves state-of-the-art performance in NAVSIM v2 end-to-end driving benchmark, 1st runner up in CVPR2025 Autonomous Grand Challenge. We hope that our framework will accelerate VLA research in close-loop autonomous driving.