Learning to Drive Safely with Hybrid Options

TL;DR

This paper applies the options framework to autonomous highway driving, defining hierarchical options for safety and comfort, leading to more interpretable and adaptable policies that outperform baseline methods.

Contribution

It introduces a hierarchical options-based approach tailored for autonomous driving, integrating domain knowledge and enabling flexible, interpretable control policies.

Findings

Hierarchical options improve driving policy performance.

Hybrid options outperform baseline policies.

Policies are more interpretable than classical continuous action policies.

Abstract

Out of the many deep reinforcement learning approaches for autonomous driving, only few make use of the options (or skills) framework. That is surprising, as this framework is naturally suited for hierarchical control applications in general, and autonomous driving tasks in specific. Therefore, in this work the options framework is applied and tailored to autonomous driving tasks on highways. More specifically, we define dedicated options for longitudinal and lateral manoeuvres with embedded safety and comfort constraints. This way, prior domain knowledge can be incorporated into the learning process and the learned driving behaviour can be constrained more easily. We propose several setups for hierarchical control with options and derive practical algorithms following state-of-the-art reinforcement learning techniques. By separately selecting actions for longitudinal and lateral control, the introduced policies over combined and hybrid options obtain the same expressiveness and flexibility that human drivers have, while being easier to interpret than classical policies over continuous actions. Of all the investigated approaches, these flexible policies over hybrid options perform the best under varying traffic conditions, outperforming the baseline policies over actions.