Behavior Planning at Urban Intersections through Hierarchical Reinforcement Learning

TL;DR

This paper introduces a hierarchical reinforcement learning approach for autonomous vehicle behavior planning at urban intersections, outperforming heuristic rules and traditional RL in efficiency and decision quality.

Contribution

It presents a novel hierarchical RL framework for urban behavior planning, improving decision-making and training efficiency over existing methods.

Findings

Outperforms heuristic-rule-based methods in complex urban scenarios

Achieves faster convergence to optimal policies than traditional RL

More sample-efficient due to hybrid reward and heuristic exploration

Abstract

For autonomous vehicles, effective behavior planning is crucial to ensure safety of the ego car. In many urban scenarios, it is hard to create sufficiently general heuristic rules, especially for challenging scenarios that some new human drivers find difficult. In this work, we propose a behavior planning structure based on reinforcement learning (RL) which is capable of performing autonomous vehicle behavior planning with a hierarchical structure in simulated urban environments. Application of the hierarchical structure allows the various layers of the behavior planning system to be satisfied. Our algorithms can perform better than heuristic-rule-based methods for elective decisions such as when to turn left between vehicles approaching from the opposite direction or possible lane-change when approaching an intersection due to lane blockage or delay in front of the ego car. Such behavior is hard to evaluate as correct or incorrect, but for some aggressive expert human drivers handle such scenarios effectively and quickly. On the other hand, compared to traditional RL methods, our algorithm is more sample-efficient, due to the use of a hybrid reward mechanism and heuristic exploration during the training process. The results also show that the proposed method converges to an optimal policy faster than traditional RL methods.