A Risk-Averse Preview-based $Q$-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

TL;DR

This paper introduces a risk-averse preview-based Q-learning algorithm for autonomous vehicle highway navigation, utilizing sensor preview data, stochastic reachability, and hybrid automaton models to improve safety and performance in dynamic traffic conditions.

Contribution

It presents a novel risk-averse Q-learning framework that incorporates preview information and hybrid automaton models for proactive and safe autonomous highway driving.

Findings

Effective in varying traffic densities

Reduces fluctuation of Q-values for safety

Enables proactive reaction to environmental changes

Abstract

A risk-averse preview-based $Q$-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A risk assessment unit module is then presented that leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the Q values, which can jeopardize the vehicle's safety and/or performance. The overall hybrid automaton model of the system is leveraged to develop a feasibility check unit module that detects unfeasible plans and enables the planner system to react proactively to the changes of the environment. Finally, to verify the efficiency of the presented algorithm, its implementation on two highway driving scenarios of an autonomous vehicle in a varying traffic density is considered.