Privileged to Predicted: Towards Sensorimotor Reinforcement Learning for Urban Driving

TL;DR

This paper explores how to develop sensorimotor reinforcement learning agents for urban driving by approximating privileged environment representations with vision-based models, aiming to bridge the gap with privileged RL agents.

Contribution

It introduces vision-based models to approximate privileged representations, identifies key aspects of state representation, and analyzes generalization challenges in RL for autonomous driving.

Findings

Privileged RL agents outperform sensorimotor agents in urban driving tasks.

Vision-based models can approximate privileged representations but face generalization issues.

State representation choices significantly impact RL agent performance.

Abstract

Reinforcement Learning (RL) has the potential to surpass human performance in driving without needing any expert supervision. Despite its promise, the state-of-the-art in sensorimotor self-driving is dominated by imitation learning methods due to the inherent shortcomings of RL algorithms. Nonetheless, RL agents are able to discover highly successful policies when provided with privileged ground truth representations of the environment. In this work, we investigate what separates privileged RL agents from sensorimotor agents for urban driving in order to bridge the gap between the two. We propose vision-based deep learning models to approximate the privileged representations from sensor data. In particular, we identify aspects of state representation that are crucial for the success of the RL agent such as desired route generation and stop zone prediction, and propose solutions to gradually develop less privileged RL agents. We also observe that bird's-eye-view models trained on offline datasets do not generalize to online RL training due to distribution mismatch. Through rigorous evaluation on the CARLA simulation environment, we shed light on the significance of the state representations in RL for autonomous driving and point to unresolved challenges for future research.