Self-Improving Safety Performance of Reinforcement Learning Based Driving with Black-Box Verification Algorithms

TL;DR

This paper introduces a self-improving framework for reinforcement learning-based autonomous driving that uses black-box verification to identify safety failures and iteratively retrains the agent to enhance safety performance, especially in critical scenarios.

Contribution

The paper presents a novel framework combining black-box verification with transfer learning to systematically improve RL-based driving safety, addressing training data limitations.

Findings

Efficient discovery of safety failure scenarios in RL driving agents.

Significant reduction in vehicle collisions through iterative retraining.

Enhanced generalization of RL agents in safety-critical situations.

Abstract

In this work, we propose a self-improving artificial intelligence system to enhance the safety performance of reinforcement learning (RL)-based autonomous driving (AD) agents using black-box verification methods. RL algorithms have become popular in AD applications in recent years. However, the performance of existing RL algorithms heavily depends on the diversity of training scenarios. A lack of safety-critical scenarios during the training phase could result in poor generalization performance in real-world driving applications. We propose a novel framework in which the weaknesses of the training set are explored through black-box verification methods. After discovering AD failure scenarios, the RL agent's training is re-initiated via transfer learning to improve the performance of previously unsafe scenarios. Simulation results demonstrate that our approach efficiently discovers safety failures of action decisions in RL-based adaptive cruise control (ACC) applications and significantly reduces the number of vehicle collisions through iterative applications of our method. The source code is publicly available at https://github.com/data-and-decision-lab/self-improving-RL.