Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills

TL;DR

This paper introduces a hierarchical skill-based framework utilizing a variational autoencoder to improve offline reinforcement learning for long-horizon autonomous vehicle planning, demonstrating superior performance and interpretability in simulation.

Contribution

The work develops a novel skill learning approach with a two-branch VAE to enhance offline RL for extended vehicle planning tasks, addressing long-horizon challenges.

Findings

Outperforms strong baselines in CARLA simulations

Enables long-term reasoning with learned skills

Shows interpretability and transferability of skills

Abstract

Learning-based vehicle planning is receiving increasing attention with the emergence of diverse driving simulators and large-scale driving datasets. While offline reinforcement learning (RL) is well suited for these safety-critical tasks, it still struggles to plan over extended periods. In this work, we present a skill-based framework that enhances offline RL to overcome the long-horizon vehicle planning challenge. Specifically, we design a variational autoencoder (VAE) to learn skills from offline demonstrations. To mitigate posterior collapse of common VAEs, we introduce a two-branch sequence encoder to capture both discrete options and continuous variations of the complex driving skills. The final policy treats learned skills as actions and can be trained by any off-the-shelf offline RL algorithms. This facilitates a shift in focus from per-step actions to temporally extended skills, thereby enabling long-term reasoning into the future. Extensive results on CARLA prove that our model consistently outperforms strong baselines at both training and new scenarios. Additional visualizations and experiments demonstrate the interpretability and transferability of extracted skills.