From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving

TL;DR

This paper compares imitation learning and offline reinforcement learning for autonomous driving, demonstrating that offline RL significantly improves policy robustness and safety over behavioral cloning in large-scale real-world datasets.

Contribution

It introduces a comprehensive pipeline for offline learning in autonomous driving, showing that offline RL with CQL outperforms behavioral cloning in robustness and safety on real-world data.

Findings

CQL achieves 3.2x higher success rate than BC

CQL reduces collision rate by 7.4x

Offline RL enhances robustness in long-horizon driving scenarios

Abstract

Learning robust driving policies from large-scale, real-world datasets is a central challenge in autonomous driving, as online data collection is often unsafe and impractical. While Behavioral Cloning (BC) offers a straightforward approach to imitation learning, policies trained with BC are notoriously brittle and suffer from compounding errors in closed-loop execution. This work presents a comprehensive pipeline and a comparative study to address this limitation. We first develop a series of increasingly sophisticated BC baselines, culminating in a Transformer-based model that operates on a structured, entity-centric state representation. While this model achieves low imitation loss, we show that it still fails in long-horizon simulations. We then demonstrate that by applying a state-of-the-art Offline Reinforcement Learning algorithm, Conservative Q-Learning (CQL), to the same data and architecture, we can learn a significantly more robust policy. Using a carefully engineered reward function, the CQL agent learns a conservative value function that enables it to recover from minor errors and avoid out-of-distribution states. In a large-scale evaluation on 1,000 unseen scenarios from the Waymo Open Motion Dataset, our final CQL agent achieves a 3.2x higher success rate and a 7.4x lower collision rate than the strongest BC baseline, proving that an offline RL approach is critical for learning robust, long-horizon driving policies from static expert data.