Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving

TL;DR

This paper introduces a spatially hierarchical reinforcement learning approach for autonomous highway driving, enabling safer and more efficient path planning in complex, unpredictable environments.

Contribution

It proposes a novel spatial hierarchy in reinforcement learning that improves path planning in autonomous driving by considering regions in state and policy space.

Findings

Outperforms baseline hierarchical RL in complex road scenarios

Finds nearly optimal policies from early episodes

Produces trajectories similar to human driving strategies

Abstract

Despite advances in hierarchical reinforcement learning, its applications to path planning in autonomous driving on highways are challenging. One reason is that conventional hierarchical reinforcement learning approaches are not amenable to autonomous driving due to its riskiness: the agent must move avoiding multiple obstacles such as other agents that are highly unpredictable, thus safe regions are small, scattered, and changeable over time. To overcome this challenge, we propose a spatially hierarchical reinforcement learning method for state space and policy space. The high-level policy selects not only behavioral sub-policy but also regions to pay mind to in state space and for outline in policy space. Subsequently, the low-level policy elaborates the short-term goal position of the agent within the outline of the region selected by the high-level command. The network structure and optimization suggested in our method are as concise as those of single-level methods. Experiments on the environment with various shapes of roads showed that our method finds the nearly optimal policies from early episodes, outperforming a baseline hierarchical reinforcement learning method, especially in narrow and complex roads. The resulting trajectories on the roads were similar to those of human strategies on the behavioral planning level.