Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning

TL;DR

This paper introduces an automatic curriculum learning framework that dynamically generates driving scenarios tailored to an autonomous agent’s learning progress, significantly improving training efficiency and policy robustness in reinforcement learning-based driving tasks.

Contribution

The proposed framework autonomously creates and mutates driving scenarios based on agent performance, removing the need for manual curriculum design and enhancing generalization and training efficiency.

Findings

Achieved +9% success rate in low traffic density scenarios.

Achieved +21% success rate in high traffic density scenarios.

Faster convergence with fewer training steps compared to baselines.

Abstract

This paper addresses the challenges of training end-to-end autonomous driving agents using Reinforcement Learning (RL). RL agents are typically trained in a fixed set of scenarios and nominal behavior of surrounding road users in simulations, limiting their generalization and real-life deployment. While domain randomization offers a potential solution by randomly sampling driving scenarios, it frequently results in inefficient training and sub-optimal policies due to the high variance among training scenarios. To address these limitations, we propose an automatic curriculum learning framework that dynamically generates driving scenarios with adaptive complexity based on the agent's evolving capabilities. Unlike manually designed curricula that introduce expert bias and lack scalability, our framework incorporates a ``teacher'' that automatically generates and mutates driving scenarios based on their learning potential -- an agent-centric metric derived from the agent's current policy -- eliminating the need for expert design. The framework enhances training efficiency by excluding scenarios the agent has mastered or finds too challenging. We evaluate our framework in a reinforcement learning setting where the agent learns a driving policy from camera images. Comparative results against baseline methods, including fixed scenario training and domain randomization, demonstrate that our approach leads to enhanced generalization, achieving higher success rates: +9% in low traffic density, +21% in high traffic density, and faster convergence with fewer training steps. Our findings highlight the potential of ACL in improving the robustness and efficiency of RL-based autonomous driving agents.