Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learning

TL;DR

This paper presents a hierarchical deep reinforcement learning framework for complex traffic scenarios, enabling better decision-making and control in autonomous driving through decomposed subtasks and separate training of controllers.

Contribution

Introduces a hierarchical DRL framework with a two-step training process for complex traffic environments, improving exploration and control capabilities.

Findings

Hierarchical controller outperforms baseline in complex highway scenarios.

Separate training of high-level and low-level controllers enhances learning efficiency.

Framework demonstrates improved handling of long-term delayed rewards.

Abstract

Developing an automated driving system capable of navigating complex traffic environments remains a formidable challenge. Unlike rule-based or supervised learning-based methods, Deep Reinforcement Learning (DRL) based controllers eliminate the need for domain-specific knowledge and datasets, thus providing adaptability to various scenarios. Nonetheless, a common limitation of existing studies on DRL-based controllers is their focus on driving scenarios with simple traffic patterns, which hinders their capability to effectively handle complex driving environments with delayed, long-term rewards, thus compromising the generalizability of their findings. In response to these limitations, our research introduces a pioneering hierarchical framework that efficiently decomposes intricate decision-making problems into manageable and interpretable subtasks. We adopt a two step training process that trains the high-level controller and low-level controller separately. The high-level controller exhibits an enhanced exploration potential with long-term delayed rewards, and the low-level controller provides longitudinal and lateral control ability using short-term instantaneous rewards. Through simulation experiments, we demonstrate the superiority of our hierarchical controller in managing complex highway driving situations.